Upload and conversion progress in the panel #233

Closed
opened 2026-11-15 17:05:24 +00:00 by vivy · 11 comments
Owner

Asked by the product owner during the test (GRI-2051, 2026-11-15): "A nice to have is also feedback. The upload % and the processing % for converting."

Today an upload shows no progress, and a PPTX or PDF conversion shows nothing until it finishes or fails. A large deck can take a minute.

Wanted

  1. Upload % — a progress bar while the file is sent. This is panel-only: XMLHttpRequest.upload.onprogress (or a streamed fetch) gives bytes sent / total.
  2. Processing % — after the upload, the conversion step shows its own progress: for PPTX/PDF, page N of M rendered; when the count is not known, a step name (converting to PDF, rendering pages, encoding video). This needs the server to report progress, for example a job status the panel polls, or a server-sent event stream.
  3. A failure shows the error at the step where it happened.

Not in scope: cancelling a running conversion.

Nice to have, so Phase 4 and low priority, like #231. Part 1 is small and could be done on its own first.

Asked by the product owner during the test (GRI-2051, 2026-11-15): *"A nice to have is also feedback. The upload % and the processing % for converting."* Today an upload shows no progress, and a PPTX or PDF conversion shows nothing until it finishes or fails. A large deck can take a minute. **Wanted** 1. **Upload %** — a progress bar while the file is sent. This is panel-only: `XMLHttpRequest.upload.onprogress` (or a streamed `fetch`) gives bytes sent / total. 2. **Processing %** — after the upload, the conversion step shows its own progress: for PPTX/PDF, *page N of M rendered*; when the count is not known, a step name (*converting to PDF*, *rendering pages*, *encoding video*). This needs the server to report progress, for example a job status the panel polls, or a server-sent event stream. 3. A failure shows the error at the step where it happened. **Not in scope:** cancelling a running conversion. Nice to have, so Phase 4 and low priority, like #231. Part 1 is small and could be done on its own first.
Member

Investigated before writing any code.

Part 1 — upload %. Doable panel-only today: uploadMedia() in
frontend-web/src/api/httpMediaApi.ts does a plain fetch POST of a
FormData body, which gives no progress events. Switching to
XMLHttpRequest (or a streamed body) to get upload.onprogress is a small,
contained change.

Part 2 — processing %. Not panel-only, and not small. The single
POST /api/v1/media call is synchronous end to end: upload_media /
convert_pdf_upload in crates/server/src/media/handlers.rs runs the PDF
rendering inline and only returns once every page is done — there is no job
row, no status field, and no progress anywhere in MediaView. QA already
flagged this exact gap on AC-3-02
(crates/server/tests/media_acceptance_integration.rs:232-239): "the rest
of the criterion — a background conversion job is created; the panel can
show its progress — has no job or progress field anywhere in MediaView …
that is a contract gap, not something this test can prove or disprove."

Part 2 needs a real API contract from the backend (async job + status
endpoint or an SSE stream) before any panel code can be written against it.

No QA acceptance tests exist yet for either part of this issue
(grep -ri progress frontend-web/src --include='*.test.*' is empty). Per
process, I don't implement against a verbal description with no tests to
build to.

Proposing: do Part 1 now (self-contained, needs only test coverage), and
leave Part 2 for after the backend engineer and QA settle the job/status
contract. Flagging on the board issue for the CTO to route QA/backend on
Part 2, so I'm not sitting idle in the meantime — I'll proceed with Part 1
once tests exist for it.

Investigated before writing any code. **Part 1 — upload %.** Doable panel-only today: `uploadMedia()` in `frontend-web/src/api/httpMediaApi.ts` does a plain `fetch` POST of a `FormData` body, which gives no progress events. Switching to `XMLHttpRequest` (or a streamed body) to get `upload.onprogress` is a small, contained change. **Part 2 — processing %.** Not panel-only, and not small. The single `POST /api/v1/media` call is synchronous end to end: `upload_media` / `convert_pdf_upload` in `crates/server/src/media/handlers.rs` runs the PDF rendering inline and only returns once every page is done — there is no job row, no status field, and no progress anywhere in `MediaView`. QA already flagged this exact gap on AC-3-02 (`crates/server/tests/media_acceptance_integration.rs:232-239`): *"the rest of the criterion — a background conversion job is created; the panel can show its progress — has no job or progress field anywhere in `MediaView` … that is a contract gap, not something this test can prove or disprove."* Part 2 needs a real API contract from the backend (async job + status endpoint or an SSE stream) before any panel code can be written against it. **No QA acceptance tests exist yet for either part** of this issue (`grep -ri progress frontend-web/src --include='*.test.*'` is empty). Per process, I don't implement against a verbal description with no tests to build to. Proposing: do Part 1 now (self-contained, needs only test coverage), and leave Part 2 for after the backend engineer and QA settle the job/status contract. Flagging on the board issue for the CTO to route QA/backend on Part 2, so I'm not sitting idle in the meantime — I'll proceed with Part 1 once tests exist for it.
Member

Withdrew the stale routing question (the issue bounced through the CTO, who is paused for product/signage work; the board's forge-router bridge then reassigned it to me directly, so no routing decision was needed).

Still no QA acceptance tests for either part (checked again: grep -ril progress frontend-web/src --include="*.test.*" is empty). Per process I do not implement from a verbal description with no tests. Asked the CTO to route QA (Test Writer) to write Part 1's acceptance tests, and backend + QA to settle Part 2's job/status contract. Marked the board issue blocked pending that. Ready to implement Part 1 (swap the plain fetch in uploadMedia() for XMLHttpRequest + upload.onprogress) the moment tests exist for it.

Withdrew the stale routing question (the issue bounced through the CTO, who is paused for product/signage work; the board's forge-router bridge then reassigned it to me directly, so no routing decision was needed). Still no QA acceptance tests for either part (checked again: `grep -ril progress frontend-web/src --include="*.test.*"` is empty). Per process I do not implement from a verbal description with no tests. Asked the CTO to route QA (Test Writer) to write Part 1's acceptance tests, and backend + QA to settle Part 2's job/status contract. Marked the board issue blocked pending that. Ready to implement Part 1 (swap the plain `fetch` in `uploadMedia()` for `XMLHttpRequest` + `upload.onprogress`) the moment tests exist for it.
Author
Owner

Found while checking the product owner's "sometimes an error on upload" (GRI-2051, 2026-11-15): a PPTX is converted inside the upload request (media/handlers.rs awaits convert_pptx before it answers). On the test server LibreOffice needs more than 5 minutes for an 83-slide deck, so the browser request stays open for that whole time. Any HTTPS proxy or browser timeout in front of the server then cuts the request and the panel shows an error, while the server may still finish the conversion. Also, the server logs almost none of these failures: in the test-server log since 16:00 UTC there is exactly one upload error (17:22:46, Permission denied, which was my own mistake that morning).

So for this issue, in this order:

  1. The upload answers as soon as the file is stored; the conversion runs as a background job with a state (queued / converting page N of M / ready / failed: <full error>) that the panel polls. That is also what makes the progress % possible.
  2. The server logs every failed upload and conversion with the media id and the full tool error.
  3. Bound soffice with a timeout from configuration as well (today only the ffmpeg join has render_timeout_seconds).
**Found while checking the product owner's "sometimes an error on upload" (GRI-2051, 2026-11-15):** a PPTX is converted *inside* the upload request (`media/handlers.rs` awaits `convert_pptx` before it answers). On the test server LibreOffice needs more than 5 minutes for an 83-slide deck, so the browser request stays open for that whole time. Any HTTPS proxy or browser timeout in front of the server then cuts the request and the panel shows an error, while the server may still finish the conversion. Also, the server logs almost none of these failures: in the test-server log since 16:00 UTC there is exactly one upload error (17:22:46, *Permission denied*, which was my own mistake that morning). So for this issue, in this order: 1. The upload answers as soon as the file is stored; the conversion runs as a background job with a state (`queued / converting page N of M / ready / failed: <full error>`) that the panel polls. That is also what makes the progress % possible. 2. The server logs every failed upload and conversion with the media id and the full tool error. 3. Bound `soffice` with a timeout from configuration as well (today only the ffmpeg join has `render_timeout_seconds`).
Author
Owner

Moved into Campus production test, priority High — product owner's decision, 2026-11-15 17:42 UTC.

Why: an 83-slide PPTX takes more than 5 minutes to convert on the test server. Today the upload request stays open for the whole conversion, so any timeout between browser and server can cut it, and the colleague sees an error for an upload that was still working.

TLS for scryglass.sint-norbertus.be ends on the school's Caddy reverse proxy. Do not depend on its timeout settings: we do not control that config, and other schools will run other proxies.

Required, in this order:

  1. Server: POST upload answers at once (202 + media id) once the file is stored. Conversion runs as a background job. A status endpoint (or the existing event stream) gives state uploading / converting / ready / failed, with page N of M where it is known, and the full error on failure (see #239).
  2. Panel: upload % from the browser upload itself, then conversion progress from the status. The item is not usable in a playlist until ready. A failed one shows Failed to process: and can be closed (#236).
  3. A server restart during a conversion must not leave a half-made item that looks ready: mark it failed on start-up.
  4. Test: upload of a slow (stubbed) conversion returns before the conversion ends.
**Moved into Campus production test, priority High** — product owner's decision, 2026-11-15 17:42 UTC. Why: an 83-slide PPTX takes more than 5 minutes to convert on the test server. Today the upload request stays open for the whole conversion, so any timeout between browser and server can cut it, and the colleague sees an error for an upload that was still working. TLS for `scryglass.sint-norbertus.be` ends on the school's **Caddy** reverse proxy. Do not depend on its timeout settings: we do not control that config, and other schools will run other proxies. Required, in this order: 1. **Server:** `POST` upload answers at once (`202` + media id) once the file is stored. Conversion runs as a background job. A status endpoint (or the existing event stream) gives state `uploading / converting / ready / failed`, with *page N of M* where it is known, and the full error on failure (see #239). 2. **Panel:** upload % from the browser upload itself, then conversion progress from the status. The item is not usable in a playlist until `ready`. A failed one shows **Failed to process: <reason>** and can be closed (#236). 3. A server restart during a conversion must not leave a half-made item that looks ready: mark it `failed` on start-up. 4. Test: upload of a slow (stubbed) conversion returns before the conversion ends.
Author
Owner

Split for the Monday demo: part 1 (upload %) is #250, prismancer. This issue keeps part 2 (processing %), which needs a server job/progress model that does not exist yet. Held until part 1 and the rest of the milestone are in; I asked the product owner whether part 2 can come after the demo.

Split for the Monday demo: part 1 (upload %) is #250, prismancer. This issue keeps part 2 (processing %), which needs a server job/progress model that does not exist yet. Held until part 1 and the rest of the milestone are in; I asked the product owner whether part 2 can come after the demo.
Author
Owner

Server side split to #263 (aetherforge): background conversion + progress state endpoint. This issue keeps the panel side (glamourweaver), built against the contract in #263's PR. Released now because every other milestone item is in dev. Landing before the demo stays the product owner's call.

Server side split to **#263** (aetherforge): background conversion + progress state endpoint. This issue keeps the panel side (glamourweaver), built against the contract in #263's PR. Released now because every other milestone item is in `dev`. Landing before the demo stays the product owner's call.
Author
Owner

Released: build the panel side now. The server contract is fixed in PR #264 (forge #263): a .pdf/.pptx POST /media answers 201 with one item, job_state 'queued', kind 'document' (pdf) or 'pending' (pptx). Poll GET /media/{id}/job → state queued|converting|rendering|encoding|done|failed, page/page_total while pages render, error_step/error_message on failed; 404 = no job. On done/failed refresh GET /media. Also handle error_step 'interrupted' (server restart). Branch from dev; #264 is not merged yet, so mock the endpoint in tests.

Released: build the panel side now. The server contract is fixed in PR #264 (forge #263): a .pdf/.pptx POST /media answers 201 with one item, job_state 'queued', kind 'document' (pdf) or 'pending' (pptx). Poll GET /media/{id}/job → state queued|converting|rendering|encoding|done|failed, page/page_total while pages render, error_step/error_message on failed; 404 = no job. On done/failed refresh GET /media. Also handle error_step 'interrupted' (server restart). Branch from dev; #264 is not merged yet, so mock the endpoint in tests.
Member

Built the panel side against the contract from your last comment (PR #264's job_state / GET /media/{id}/job). PR #265 opened dev <- feature/GRI-2098-conversion-progress. Shows a step name while unknown (queued / converting to PDF / rendering pages / encoding video), "Page N of M rendered" with a progress bar once known, and "Failed to process: " on failure (including error_step: interrupted) — delete still works throughout. Mock mode now also supports a .pptx upload (resolving to video) and simulates the same placeholder/job shape, so the panel's own tests don't depend on #264 landing first. CI running now (fj ci 265).

Built the panel side against the contract from your last comment (PR #264's job_state / GET /media/{id}/job). PR #265 opened dev <- feature/GRI-2098-conversion-progress. Shows a step name while unknown (queued / converting to PDF / rendering pages / encoding video), "Page N of M rendered" with a progress bar once known, and "Failed to process: <reason>" on failure (including error_step: interrupted) — delete still works throughout. Mock mode now also supports a .pptx upload (resolving to video) and simulates the same placeholder/job shape, so the panel's own tests don't depend on #264 landing first. CI running now (fj ci 265).
Member

PR #265 (panel side, this issue's Part 2) is CI-green: fj ci 265 passed on 0b64fd97. It was built against #264's (server side, GRI-2127) documented contract with a mock standing in, since #264 wasn't merged yet. #264 is now also CI-green (fj ci 264 passed on 89cf8f8e) and has one review comment pending.

Both can merge independently of each other's order — #265's real httpMediaApi path talks to the documented /media/{id}/job route either way, nothing changes once #264 lands; the mock is test-only. Handing both to the chief of staff for merge into dev.

PR #265 (panel side, this issue's Part 2) is CI-green: fj ci 265 passed on 0b64fd97. It was built against #264's (server side, GRI-2127) documented contract with a mock standing in, since #264 wasn't merged yet. #264 is now also CI-green (fj ci 264 passed on 89cf8f8e) and has one review comment pending. Both can merge independently of each other's order — #265's real httpMediaApi path talks to the documented /media/{id}/job route either way, nothing changes once #264 lands; the mock is test-only. Handing both to the chief of staff for merge into dev.
Member

Pushed fix for the review comments on PR #265 (62c646a2): poll interval is now 1500ms in the product (was 150ms, a test leak), with the test override moved to a new mediaJobPoll.ts so MediaScreen.tsx keeps exporting only its component (fast-refresh lint). A non-404 getMediaJob error (network blip/5xx) now retries after the normal interval instead of ending the poll for good — added a regression test for that. frontend-web gate green locally (163 tests). CI running on 62c646a2 now.

Pushed fix for the review comments on PR #265 (62c646a2): poll interval is now 1500ms in the product (was 150ms, a test leak), with the test override moved to a new mediaJobPoll.ts so MediaScreen.tsx keeps exporting only its component (fast-refresh lint). A non-404 getMediaJob error (network blip/5xx) now retries after the normal interval instead of ending the poll for good — added a regression test for that. frontend-web gate green locally (163 tests). CI running on 62c646a2 now.
vivy closed this issue 2026-11-16 03:53:07 +00:00
Author
Owner

Done: server side #263 (PR #264, 13f1f05c) and panel side PR #265 merged into dev.

Done: server side #263 (PR #264, 13f1f05c) and panel side PR #265 merged into dev.
Sign in to join this conversation.
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Grimoirix/grimoire-of-digital-signage#233
No description provided.