GRI-2127: background conversion for PDF/PPTX uploads, with job progress #264
No reviewers
Labels
No labels
Area/Backend
Area/Database
Area/Docs
Area/Frontend
Area/Infra
Area/Security
Area/Tests
Area/Tooling
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Lang/Python
Lang/Rust
Lang/Shell
Lang/SQL
Lang/TypeScript
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Routing
Auto
Routing
Hold
Routing
Human
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Grimoirix/grimoire-of-digital-signage!264
Loading…
Reference in a new issue
No description provided.
Delete branch "feature/GRI-2127-background-conversion-progress"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Server side of forge #263 (part 2 of #233), GRI-2127.
Problem
An 83-slide PPTX converted inside the upload request, which could take
more than five minutes on the test server — long enough for a browser or
server timeout to cut the upload before it ever got an answer.
What changed
A
.pdf/.pptxupload now stores the original file and answersimmediately with a placeholder
mediarow and aqueuedjob. Conversionruns afterwards in a background task. Images, videos and URL items are
unaffected — still synchronous, same as before.
Contract (for #233's panel side to poll)
POST /mediawith a.pdfor.pptx:201, body is still an array (forconsistency with every other upload), with exactly one item:
kindis"document"for a.pdf(its final kind is alwaysdocument,only
pdf_pagesis still unknown) or"pending"for a.pptx(not knownyet whether it becomes
videoordocument).job_stateis only present in this upload response, always"queued".It is not present on
GET /medialist items — poll the job endpoint forprogress, not the catalog.
GET /media/{media_id}/job:page/page_total("page N of M rendered") are set only while pages arebeing stored (the PDF render, or the
.pptxslide-image fallback) — notduring a
.pptx's MP4 join, which has no page count.error_step/error_messageare set only oncestate == "failed".404for media that does not exist, that the caller's campus may notsee (same campus scoping as every other media endpoint —
delete_media'smay_accesscheck), or that never had a job at all (an image, video, urlupload, or a document's own page).
doneorfailed),GET /mediareflects the finalkind/pdf_pages/pages as usual.Behaviour change on failure (AC-3-06 superseded)
A failed conversion — a corrupt PDF, a
.pptxneither the render nor thefallback can process — used to mean a clean upload refusal with no catalog
entry at all. That guarantee only held while conversion ran inside the
upload request; now nothing is waiting on the upload's answer, so the
failure shows up as
job.state == "failed"witherror_stepanderror_message, and the placeholder row is left exactly as the uploadcreated it.
plan/tests/mvp-acceptance.md's AC-3-03 and AC-3-06 wording("at upload time", "the upload reports a clear error") describes the old
mechanism, not the new one — flagging this for the product manager/CTO to
update if they agree, rather than editing the acceptance doc myself.
Not in scope
Cancelling a running conversion, same as the issue says.
Tests
Written from "Done when", in their own commit before the implementation:
media_acceptance_integration.rs: rewrote AC-3-03 and AC-3-06 to pollthe job instead of reading the upload response; AC-3-06 renamed and its
doc comment explains the supersession.
pptx_video_acceptance_integration.rs: rewrote all four existing teststo poll the job; added
the_upload_answers_before_the_conversion_finishes(a slow fake
sofficeproves the response lands well before the renderdoes — the actual fix for forge #263) and
a_successful_join_removes_the_orphaned_original_pptx_file(theplaceholder's
sha256moves from the original to the video's; theoriginal file must not be left behind — caught a real bug during review,
fixed in the implementation commit).
media_integration.rs: updated the one PDF-upload test that asserted theold synchronous refusal.
Ran locally:
cargo test -p server(full crate, DB-backed) andscripts/local-gates.sh --rust-only(fmt, clippy, tests, cargo-deny) —every gate passed. Forge CI runs the full gates again on this PR.
Follow-up for the product manager/CTO
plan/tests/mvp-acceptance.mdAC-3-03/AC-3-06 wording describes thepre-GRI-2127 synchronous mechanism; worth a pass once this lands.
plan/server/media-pipeline.md's "Documents" section updated to describethe background job; not otherwise touched.
Review (Vivy). Contract is good and CI is green. One change before merge, one note.
Must fix — a restart strands every unfinished job. Nothing resumes or closes a job that is
queued/converting/rendering/encodingwhen the process stops.media_jobsis only written by the task thattokio::spawnstarted, so after a restart or redeploy that row stays non-terminal for ever: the panel (#233) polls a spinner that never ends, and thependingmedia row can never be used or told apart from a live job. A redeploy during an 83-slide upload is the case this issue exists for.Ask: at server start, before the router serves, one
UPDATE media_jobs SET state='failed', page=NULL, page_total=NULL, error_step='interrupted', error_message='the server restarted during conversion; upload the file again', updated_at=now() WHERE state NOT IN ('done','failed'). Plus one test: insert a job inconverting, run the startup sweep,GET /media/{id}/jobreadsfailed/interrupted. Re-running the conversion from the stored original is nicer but not needed now.Note, not blocking: there is no limit on concurrent conversions — five uploads start five renders at once. A
Semaphore(config key, default 1–2) is a follow-up; say so on #263 if you do not do it here.AC-3-03/AC-3-06 wording: agreed it describes the old mechanism; I will ask for the acceptance doc update separately.
A media_jobs row stuck in queued/converting/rendering/encoding by a crash or redeploy had no task left to finish it, so the panel would poll GET /media/{id}/job forever. recover_interrupted_jobs now runs once at start-up, right after migrations, marking every such job failed with error_step "interrupted".Fixed per review (
3c54f7c3):server::media::recover_interrupted_jobsruns once at start-up inmain.rs, right afterdb::migrateand before the HTTP listener binds. It updates everymedia_jobsrow not alreadydone/failedtostate = 'failed',error_step = 'interrupted', with an error message naming the restart — so a job the previous process left running (crash, redeploy, restart) resolves to a terminal state instead of spinning the panel forever.Added
an_interrupted_job_is_recovered_as_failed_at_start_upinmedia_integration.rs: inserts a job directly inconvertingstate, calls the recovery function, and asserts the job endpoint reportsfailed/interruptedwith a non-empty error message; a second case proves adonejob is left untouched.Concurrency limit left as the agreed follow-up, not in this PR.
cargo test -p server(serialized,--test-threads=1— this build host's shared Postgres hitsPoolTimedOutunder the default parallel test threads from unrelated concurrent load, not from this change) andcargo fmt --check/cargo clippy -p server --all-targets -- -D warningsall green locally. Forge CI running on89cf8f8.Fix verified on
89cf8f8e: fail_interrupted_jobs runs after migrations and before the listener binds, marks every queued/converting/rendering/encoding job failed with error_step = interrupted, and leaves done/failed rows alone; the test covers both. CI green on all 7 required contexts. Approving.Note for later, not blocking: this assumes one server process per database. A second instance would fail the running jobs of the first at its start-up. Fine for the campus test.