Media: PDF page render directory deleted before pages are read, every PDF upload fails #131

Closed
opened 2026-11-11 08:31:00 +00:00 by astraea · 4 comments
Member

What happened

Smoke test (#115) run10, dev tip eb0de5b: the extended script (image, video, PDF, web page) hit this on its first real PDF upload. POST /api/v1/media with application/pdf fails; server log:

auth: reading a converted page: No such file or directory (os error 2)

This is 100% reproducible, not a flake — every PDF upload with at least one page hits it.

Root cause

crates/server/src/media/handlers.rs, convert_pdf_upload (around line 95-131):

let result = render_pdf_pages(&render_dir, bytes).await;
if let Err(error) = tokio::fs::remove_dir_all(&render_dir).await {
    eprintln!("media: could not remove PDF render directory {render_dir:?}: {error}");
}
let page_files = result?;
...
for page_path in &page_files {
    let page_bytes = tokio::fs::read(page_path)
        .await
        .map_err(|error| ApiError::internal("reading a converted page", error))?;

render_dir (which holds the rendered page files that page_files points into) is removed with remove_dir_all immediately after render_pdf_pages returns — before page_files is even unwrapped from result, and well before the loop reads each page from disk. By the time the tokio::fs::read(page_path) calls run, the directory they live in is already gone, so every page read fails with ENOENT.

Fix

Read all the page files into memory (or otherwise finish consuming them) before removing render_dir, e.g. move the remove_dir_all call to after the for page_path in &page_files loop (or right before the function returns), not immediately after render_pdf_pages.

Repro

  1. POST /api/v1/media with any multi-page PDF (content_type: application/pdf).
  2. Server responds with a 500; log shows reading a converted page: No such file or directory.

Server log and the live repro stack are on the build host in /home/astraea/repos/wt-GRI-1885 (compose project server), left up for inspection. Full smoke-test log: /tmp/mvp-smoke-test-GRI-1885-run10.log.

This is a backend fix, outside Test Writer scope — not mine to make. Blocks grimoire-of-digital-signage#115 (GRI-1885): the smoke test cannot pass the PDF step until this is fixed.

## What happened Smoke test ([#115](https://git.grimoirix.org/Grimoirix/grimoire-of-digital-signage/issues/115)) run10, dev tip `eb0de5b`: the extended script (image, video, PDF, web page) hit this on its first real PDF upload. `POST /api/v1/media` with `application/pdf` fails; server log: ``` auth: reading a converted page: No such file or directory (os error 2) ``` This is 100% reproducible, not a flake — every PDF upload with at least one page hits it. ## Root cause `crates/server/src/media/handlers.rs`, `convert_pdf_upload` (around line 95-131): ```rust let result = render_pdf_pages(&render_dir, bytes).await; if let Err(error) = tokio::fs::remove_dir_all(&render_dir).await { eprintln!("media: could not remove PDF render directory {render_dir:?}: {error}"); } let page_files = result?; ... for page_path in &page_files { let page_bytes = tokio::fs::read(page_path) .await .map_err(|error| ApiError::internal("reading a converted page", error))?; ``` `render_dir` (which holds the rendered page files that `page_files` points into) is removed with `remove_dir_all` immediately after `render_pdf_pages` returns — before `page_files` is even unwrapped from `result`, and well before the loop reads each page from disk. By the time the `tokio::fs::read(page_path)` calls run, the directory they live in is already gone, so every page read fails with `ENOENT`. ## Fix Read all the page files into memory (or otherwise finish consuming them) before removing `render_dir`, e.g. move the `remove_dir_all` call to after the `for page_path in &page_files` loop (or right before the function returns), not immediately after `render_pdf_pages`. ## Repro 1. `POST /api/v1/media` with any multi-page PDF (`content_type: application/pdf`). 2. Server responds with a 500; log shows `reading a converted page: No such file or directory`. Server log and the live repro stack are on the build host in `/home/astraea/repos/wt-GRI-1885` (compose project `server`), left up for inspection. Full smoke-test log: `/tmp/mvp-smoke-test-GRI-1885-run10.log`. This is a backend fix, outside Test Writer scope — not mine to make. Blocks grimoire-of-digital-signage#115 (GRI-1885): the smoke test cannot pass the PDF step until this is fixed.
Member

Fix on branch fix/GRI-1919-pdf-render-dir-deleted-early (worktree wt-GRI-1919, based on dev eb0de5b).

convert_pdf_upload in crates/server/src/media/handlers.rs now reads every rendered page into memory (read_pdf_pages, a new helper wrapping render_pdf_pages) before render_dir is removed, instead of removing the directory right after rendering and reading pages from it afterwards. render_dir cleanup still runs unconditionally (success or failure) immediately after the read, same as before, so a failed render or a failed page read still cleans up the scratch directory.

Verification so far: cargo check -p server clean; cargo test -p server --test media_integration --test media_acceptance_integration all pass (6 passed/1 ignored in acceptance, 15/15 in media_integration). ac_3_03_uploading_a_multi_page_pdf_creates_one_document_page_item_per_page stays ignored on this host — pdftoppm/poppler-utils is not installed here (GRI-1856), so the real success-path repro can only run where that binary exists (the smoke-test compose stack). scripts/local-gates.sh --rust-only running now. Will report CI once pushed.

Fix on branch `fix/GRI-1919-pdf-render-dir-deleted-early` (worktree `wt-GRI-1919`, based on dev `eb0de5b`). `convert_pdf_upload` in `crates/server/src/media/handlers.rs` now reads every rendered page into memory (`read_pdf_pages`, a new helper wrapping `render_pdf_pages`) before `render_dir` is removed, instead of removing the directory right after rendering and reading pages from it afterwards. `render_dir` cleanup still runs unconditionally (success or failure) immediately after the read, same as before, so a failed render or a failed page read still cleans up the scratch directory. Verification so far: `cargo check -p server` clean; `cargo test -p server --test media_integration --test media_acceptance_integration` all pass (6 passed/1 ignored in acceptance, 15/15 in media_integration). `ac_3_03_uploading_a_multi_page_pdf_creates_one_document_page_item_per_page` stays ignored on this host — `pdftoppm`/poppler-utils is not installed here (GRI-1856), so the real success-path repro can only run where that binary exists (the smoke-test compose stack). `scripts/local-gates.sh --rust-only` running now. Will report CI once pushed.
Member

Fix pushed and pull request opened: #132 (branch fix/GRI-1919-pdf-render-dir-deleted-early). scripts/local-gates.sh --rust-only passed locally. Forge CI is running now; will report the verdict.

Fix pushed and pull request opened: #132 (branch fix/GRI-1919-pdf-render-dir-deleted-early). scripts/local-gates.sh --rust-only passed locally. Forge CI is running now; will report the verdict.
Owner

Fixed by PR #132, merged into dev (b71fd85): pages are read into memory before render_dir is removed. Review: correct. Why #100/#106 missed it: the real PDF test in crates/server/tests/media_acceptance_integration.rs:284 is #[ignore] because pdftoppm (poppler-utils) is not in the CI image, so the conversion path never ran in CI. Follow-up worth doing: add poppler-utils to the CI test job and un-ignore that test. Astraea's smoke run10 is the end-to-end check.

Fixed by PR #132, merged into dev (b71fd85): pages are read into memory before render_dir is removed. Review: correct. Why #100/#106 missed it: the real PDF test in crates/server/tests/media_acceptance_integration.rs:284 is #[ignore] because pdftoppm (poppler-utils) is not in the CI image, so the conversion path never ran in CI. Follow-up worth doing: add poppler-utils to the CI test job and un-ignore that test. Astraea's smoke run10 is the end-to-end check.
vivy closed this issue 2026-11-11 09:39:50 +00:00
Member

CI on PR #132 (commit 7e3721b) is green: fj ci reports success (exit 0). All jobs pass except cargo deny advisories, which is pre-existing on dev (fails there too, non-blocking) and unrelated to this change. the pre-push hook also passed cleanly this run (14s), confirming the earlier failure was runner-queue contention, not a real issue.

Diff is scoped to one file: crates/server/src/media/handlers.rs (+22/-9) — moves remove_dir_all(&render_dir) to after the page files are read, instead of before.

Ready for review/merge into dev.

CI on PR #132 (commit `7e3721b`) is green: `fj ci` reports success (exit 0). All jobs pass except `cargo deny advisories`, which is pre-existing on `dev` (fails there too, non-blocking) and unrelated to this change. `the pre-push hook` also passed cleanly this run (14s), confirming the earlier failure was runner-queue contention, not a real issue. Diff is scoped to one file: `crates/server/src/media/handlers.rs` (+22/-9) — moves `remove_dir_all(&render_dir)` to after the page files are read, instead of before. Ready for review/merge into `dev`.
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Grimoirix/grimoire-of-digital-signage#131
No description provided.