Media: PDF page render directory deleted before pages are read, every PDF upload fails #131
Labels
No labels
Area/Backend
Area/Database
Area/Docs
Area/Frontend
Area/Infra
Area/Security
Area/Tests
Area/Tooling
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Lang/Python
Lang/Rust
Lang/Shell
Lang/SQL
Lang/TypeScript
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Routing
Auto
Routing
Hold
Routing
Human
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Grimoirix/grimoire-of-digital-signage#131
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happened
Smoke test (#115) run10, dev tip
eb0de5b: the extended script (image, video, PDF, web page) hit this on its first real PDF upload.POST /api/v1/mediawithapplication/pdffails; server log:This is 100% reproducible, not a flake — every PDF upload with at least one page hits it.
Root cause
crates/server/src/media/handlers.rs,convert_pdf_upload(around line 95-131):render_dir(which holds the rendered page files thatpage_filespoints into) is removed withremove_dir_allimmediately afterrender_pdf_pagesreturns — beforepage_filesis even unwrapped fromresult, and well before the loop reads each page from disk. By the time thetokio::fs::read(page_path)calls run, the directory they live in is already gone, so every page read fails withENOENT.Fix
Read all the page files into memory (or otherwise finish consuming them) before removing
render_dir, e.g. move theremove_dir_allcall to after thefor page_path in &page_filesloop (or right before the function returns), not immediately afterrender_pdf_pages.Repro
POST /api/v1/mediawith any multi-page PDF (content_type: application/pdf).reading a converted page: No such file or directory.Server log and the live repro stack are on the build host in
/home/astraea/repos/wt-GRI-1885(compose projectserver), left up for inspection. Full smoke-test log:/tmp/mvp-smoke-test-GRI-1885-run10.log.This is a backend fix, outside Test Writer scope — not mine to make. Blocks grimoire-of-digital-signage#115 (GRI-1885): the smoke test cannot pass the PDF step until this is fixed.
Fix on branch
fix/GRI-1919-pdf-render-dir-deleted-early(worktreewt-GRI-1919, based on deveb0de5b).convert_pdf_uploadincrates/server/src/media/handlers.rsnow reads every rendered page into memory (read_pdf_pages, a new helper wrappingrender_pdf_pages) beforerender_diris removed, instead of removing the directory right after rendering and reading pages from it afterwards.render_dircleanup still runs unconditionally (success or failure) immediately after the read, same as before, so a failed render or a failed page read still cleans up the scratch directory.Verification so far:
cargo check -p serverclean;cargo test -p server --test media_integration --test media_acceptance_integrationall pass (6 passed/1 ignored in acceptance, 15/15 in media_integration).ac_3_03_uploading_a_multi_page_pdf_creates_one_document_page_item_per_pagestays ignored on this host —pdftoppm/poppler-utils is not installed here (GRI-1856), so the real success-path repro can only run where that binary exists (the smoke-test compose stack).scripts/local-gates.sh --rust-onlyrunning now. Will report CI once pushed.Fix pushed and pull request opened: #132 (branch fix/GRI-1919-pdf-render-dir-deleted-early). scripts/local-gates.sh --rust-only passed locally. Forge CI is running now; will report the verdict.
Fixed by PR #132, merged into dev (
b71fd85): pages are read into memory before render_dir is removed. Review: correct. Why #100/#106 missed it: the real PDF test in crates/server/tests/media_acceptance_integration.rs:284 is #[ignore] because pdftoppm (poppler-utils) is not in the CI image, so the conversion path never ran in CI. Follow-up worth doing: add poppler-utils to the CI test job and un-ignore that test. Astraea's smoke run10 is the end-to-end check.CI on PR #132 (commit
7e3721b) is green:fj cireports success (exit 0). All jobs pass exceptcargo deny advisories, which is pre-existing ondev(fails there too, non-blocking) and unrelated to this change.the pre-push hookalso passed cleanly this run (14s), confirming the earlier failure was runner-queue contention, not a real issue.Diff is scoped to one file:
crates/server/src/media/handlers.rs(+22/-9) — movesremove_dir_all(&render_dir)to after the page files are read, instead of before.Ready for review/merge into
dev.