Upload and conversion progress in the panel #233
Labels
No labels
Area/Backend
Area/Database
Area/Docs
Area/Frontend
Area/Infra
Area/Security
Area/Tests
Area/Tooling
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Lang/Python
Lang/Rust
Lang/Shell
Lang/SQL
Lang/TypeScript
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Routing
Auto
Routing
Hold
Routing
Human
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Grimoirix/grimoire-of-digital-signage#233
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Asked by the product owner during the test (GRI-2051, 2026-11-15): "A nice to have is also feedback. The upload % and the processing % for converting."
Today an upload shows no progress, and a PPTX or PDF conversion shows nothing until it finishes or fails. A large deck can take a minute.
Wanted
XMLHttpRequest.upload.onprogress(or a streamedfetch) gives bytes sent / total.Not in scope: cancelling a running conversion.
Nice to have, so Phase 4 and low priority, like #231. Part 1 is small and could be done on its own first.
Investigated before writing any code.
Part 1 — upload %. Doable panel-only today:
uploadMedia()infrontend-web/src/api/httpMediaApi.tsdoes a plainfetchPOST of aFormDatabody, which gives no progress events. Switching toXMLHttpRequest(or a streamed body) to getupload.onprogressis a small,contained change.
Part 2 — processing %. Not panel-only, and not small. The single
POST /api/v1/mediacall is synchronous end to end:upload_media/convert_pdf_uploadincrates/server/src/media/handlers.rsruns the PDFrendering inline and only returns once every page is done — there is no job
row, no status field, and no progress anywhere in
MediaView. QA alreadyflagged this exact gap on AC-3-02
(
crates/server/tests/media_acceptance_integration.rs:232-239): "the restof the criterion — a background conversion job is created; the panel can
show its progress — has no job or progress field anywhere in
MediaView…that is a contract gap, not something this test can prove or disprove."
Part 2 needs a real API contract from the backend (async job + status
endpoint or an SSE stream) before any panel code can be written against it.
No QA acceptance tests exist yet for either part of this issue
(
grep -ri progress frontend-web/src --include='*.test.*'is empty). Perprocess, I don't implement against a verbal description with no tests to
build to.
Proposing: do Part 1 now (self-contained, needs only test coverage), and
leave Part 2 for after the backend engineer and QA settle the job/status
contract. Flagging on the board issue for the CTO to route QA/backend on
Part 2, so I'm not sitting idle in the meantime — I'll proceed with Part 1
once tests exist for it.
Withdrew the stale routing question (the issue bounced through the CTO, who is paused for product/signage work; the board's forge-router bridge then reassigned it to me directly, so no routing decision was needed).
Still no QA acceptance tests for either part (checked again:
grep -ril progress frontend-web/src --include="*.test.*"is empty). Per process I do not implement from a verbal description with no tests. Asked the CTO to route QA (Test Writer) to write Part 1's acceptance tests, and backend + QA to settle Part 2's job/status contract. Marked the board issue blocked pending that. Ready to implement Part 1 (swap the plainfetchinuploadMedia()forXMLHttpRequest+upload.onprogress) the moment tests exist for it.Found while checking the product owner's "sometimes an error on upload" (GRI-2051, 2026-11-15): a PPTX is converted inside the upload request (
media/handlers.rsawaitsconvert_pptxbefore it answers). On the test server LibreOffice needs more than 5 minutes for an 83-slide deck, so the browser request stays open for that whole time. Any HTTPS proxy or browser timeout in front of the server then cuts the request and the panel shows an error, while the server may still finish the conversion. Also, the server logs almost none of these failures: in the test-server log since 16:00 UTC there is exactly one upload error (17:22:46, Permission denied, which was my own mistake that morning).So for this issue, in this order:
queued / converting page N of M / ready / failed: <full error>) that the panel polls. That is also what makes the progress % possible.sofficewith a timeout from configuration as well (today only the ffmpeg join hasrender_timeout_seconds).Moved into Campus production test, priority High — product owner's decision, 2026-11-15 17:42 UTC.
Why: an 83-slide PPTX takes more than 5 minutes to convert on the test server. Today the upload request stays open for the whole conversion, so any timeout between browser and server can cut it, and the colleague sees an error for an upload that was still working.
TLS for
scryglass.sint-norbertus.beends on the school's Caddy reverse proxy. Do not depend on its timeout settings: we do not control that config, and other schools will run other proxies.Required, in this order:
POSTupload answers at once (202+ media id) once the file is stored. Conversion runs as a background job. A status endpoint (or the existing event stream) gives stateuploading / converting / ready / failed, with page N of M where it is known, and the full error on failure (see #239).ready. A failed one shows Failed to process: and can be closed (#236).failedon start-up.Split for the Monday demo: part 1 (upload %) is #250, prismancer. This issue keeps part 2 (processing %), which needs a server job/progress model that does not exist yet. Held until part 1 and the rest of the milestone are in; I asked the product owner whether part 2 can come after the demo.
Server side split to #263 (aetherforge): background conversion + progress state endpoint. This issue keeps the panel side (glamourweaver), built against the contract in #263's PR. Released now because every other milestone item is in
dev. Landing before the demo stays the product owner's call.Released: build the panel side now. The server contract is fixed in PR #264 (forge #263): a .pdf/.pptx POST /media answers 201 with one item, job_state 'queued', kind 'document' (pdf) or 'pending' (pptx). Poll GET /media/{id}/job → state queued|converting|rendering|encoding|done|failed, page/page_total while pages render, error_step/error_message on failed; 404 = no job. On done/failed refresh GET /media. Also handle error_step 'interrupted' (server restart). Branch from dev; #264 is not merged yet, so mock the endpoint in tests.
Built the panel side against the contract from your last comment (PR #264's job_state / GET /media/{id}/job). PR #265 opened dev <- feature/GRI-2098-conversion-progress. Shows a step name while unknown (queued / converting to PDF / rendering pages / encoding video), "Page N of M rendered" with a progress bar once known, and "Failed to process: " on failure (including error_step: interrupted) — delete still works throughout. Mock mode now also supports a .pptx upload (resolving to video) and simulates the same placeholder/job shape, so the panel's own tests don't depend on #264 landing first. CI running now (fj ci 265).
PR #265 (panel side, this issue's Part 2) is CI-green: fj ci 265 passed on
0b64fd97. It was built against #264's (server side, GRI-2127) documented contract with a mock standing in, since #264 wasn't merged yet. #264 is now also CI-green (fj ci 264 passed on89cf8f8e) and has one review comment pending.Both can merge independently of each other's order — #265's real httpMediaApi path talks to the documented /media/{id}/job route either way, nothing changes once #264 lands; the mock is test-only. Handing both to the chief of staff for merge into dev.
Pushed fix for the review comments on PR #265 (
62c646a2): poll interval is now 1500ms in the product (was 150ms, a test leak), with the test override moved to a new mediaJobPoll.ts so MediaScreen.tsx keeps exporting only its component (fast-refresh lint). A non-404 getMediaJob error (network blip/5xx) now retries after the normal interval instead of ending the poll for good — added a regression test for that. frontend-web gate green locally (163 tests). CI running on62c646a2now.Done: server side #263 (PR #264,
13f1f05c) and panel side PR #265 merged into dev.