Server never sends TransferOffer: media download is unimplemented end to end #127

Closed
opened 2026-11-11 04:53:02 +00:00 by astraea · 1 comment
Member

Found while running the end-to-end smoke test for #115, on dev tip c789d2b (after #126, the reconnect-token fix, had already landed).

Symptom

Server, panel and client all come up clean. The client enrolls, gets approved, reconnects as Accepted, receives the playlist (sync: server says Accepted: welcome back then no further sync errors). Playback then loops forever on:

playback: nothing in the playlist can be shown yet

Player::next (crates/client/src/player.rs) only prints that line for FallbackReason::CannotLoad — meaning the playlist was received and an item is due, but cached(hash) never returns a path. No download ever starts: nothing in the client log mentions a transfer, an offer, or an error (success is silent by design, but so is "never attempted").

Root cause

The server never sends a TransferOffer. grep -rn TransferOffer crates/server/src is empty — the whole server-side push half of media transfer does not exist.

  • crates/server/src/ws.rs:5 documents this directly: "Transfers are later work. A device that reaches Admission::Registered here registers its sender in DeviceConnections and is kept open on a heartbeat, forwarding whatever assign_playlist ... pushes to it until it disconnects."
  • push_to_device in crates/server/src/playlists/handlers.rs:326-351 builds and pushes MediaManifest and PlaylistPush only. It never walks the manifest and sends one TransferOffer per file, as plan/shared/protocol.md specifies.
  • The spec is explicit and was settled before either side was built: plan/shared/protocol.md:248 — "the server sends MediaManifest, PlaylistPush and one TransferOffer per file the playlist needs" — and the ## Media transfer section (line 270 on) gives the full offer/accept/chunk/complete frame contract.
  • The client side is fully implemented and unit-tested: crates/client/src/session.rs (on_offer, on_binary, chunked resume, hash verification) and crates/client/src/cache.rs. It has never been exercised against a real server because nothing on the server side ever offers a transfer.

So this is not a race or a timing bug: the server has no code path that constructs a ServerControl::TransferOffer at all, for any device, ever.

Impact

This is the core MVP feature — "the device downloads and plays the assigned content" — and it does not work end-to-end. A device can enroll, get approved, and receive a playlist description, but it can never actually render an image or video it does not already have cached, because it is never sent the bytes. This blocks #115's step 5 and will block #29 (the product owner's test week on a real mini PC) the same way.

What's needed

Server-side logic that, per connected device and per playlist assignment (and really per manifest entry already being pushed in push_to_device), opens a transfer: picks a transfer_id, sends TransferOffer { transfer_id, sha256, total_bytes } for each manifest entry, and on TransferAccept streams the file (from wherever uploaded media is stored, e.g. the media directory logged at startup) as MAX_CHUNK_BYTES (256 KiB) binary chunks with the 12-byte header, honoring resume_from_offset, interleaved with control frames per the spec. DeviceConnections (crates/server/src/connections.rs) already has the per-device outbound channel; this needs its own task per transfer (or per device) and bandwidth/concurrency limits per the spec's table ("Concurrent transfers and total bandwidth are limited, server-side").

This is a design-and-implementation task for the back-end engineer, not a test-writer fix — filing per GRI-1885's Done-when criteria ("every bug found is a separate forge issue in the MVP milestone").

Found while running the end-to-end smoke test for #115, on dev tip c789d2b (after #126, the reconnect-token fix, had already landed). ## Symptom Server, panel and client all come up clean. The client enrolls, gets approved, reconnects as `Accepted`, receives the playlist (`sync: server says Accepted: welcome back` then no further sync errors). Playback then loops forever on: ``` playback: nothing in the playlist can be shown yet ``` `Player::next` (`crates/client/src/player.rs`) only prints that line for `FallbackReason::CannotLoad` — meaning the playlist *was* received and an item *is* due, but `cached(hash)` never returns a path. No download ever starts: nothing in the client log mentions a transfer, an offer, or an error (success is silent by design, but so is "never attempted"). ## Root cause The server never sends a `TransferOffer`. `grep -rn TransferOffer crates/server/src` is empty — the whole server-side push half of media transfer does not exist. - `crates/server/src/ws.rs:5` documents this directly: *"Transfers are later work. A device that reaches `Admission::Registered` here registers its sender in `DeviceConnections` and is kept open on a heartbeat, forwarding whatever `assign_playlist` ... pushes to it until it disconnects."* - `push_to_device` in `crates/server/src/playlists/handlers.rs:326-351` builds and pushes `MediaManifest` and `PlaylistPush` only. It never walks the manifest and sends one `TransferOffer` per file, as `plan/shared/protocol.md` specifies. - The spec is explicit and was settled before either side was built: `plan/shared/protocol.md:248` — *"the server sends `MediaManifest`, `PlaylistPush` and one `TransferOffer` per file the playlist needs"* — and the `## Media transfer` section (line 270 on) gives the full offer/accept/chunk/complete frame contract. - The client side is fully implemented and unit-tested: `crates/client/src/session.rs` (`on_offer`, `on_binary`, chunked resume, hash verification) and `crates/client/src/cache.rs`. It has never been exercised against a real server because nothing on the server side ever offers a transfer. So this is not a race or a timing bug: the server has no code path that constructs a `ServerControl::TransferOffer` at all, for any device, ever. ## Impact This is the core MVP feature — "the device downloads and plays the assigned content" — and it does not work end-to-end. A device can enroll, get approved, and receive a playlist description, but it can never actually render an image or video it does not already have cached, because it is never sent the bytes. This blocks #115's step 5 and will block #29 (the product owner's test week on a real mini PC) the same way. ## What's needed Server-side logic that, per connected device and per playlist assignment (and really per manifest entry already being pushed in `push_to_device`), opens a transfer: picks a `transfer_id`, sends `TransferOffer { transfer_id, sha256, total_bytes }` for each manifest entry, and on `TransferAccept` streams the file (from wherever uploaded media is stored, e.g. the media directory logged at startup) as `MAX_CHUNK_BYTES` (256 KiB) binary chunks with the 12-byte header, honoring `resume_from_offset`, interleaved with control frames per the spec. `DeviceConnections` (`crates/server/src/connections.rs`) already has the per-device outbound channel; this needs its own task per transfer (or per device) and bandwidth/concurrency limits per the spec's table ("Concurrent transfers and total bandwidth are limited, server-side"). This is a design-and-implementation task for the back-end engineer, not a test-writer fix — filing per GRI-1885's Done-when criteria ("every bug found is a separate forge issue in the MVP milestone").
Member

Implemented server-side media transfer end to end:

  • push_to_device (playlists/handlers.rs) now sends one TransferOffer per MediaManifest entry, after MediaManifest/PlaylistPush, matching plan/shared/protocol.md.
  • DeviceConnections grew a per-connection transfer_id allocator (resets on reconnect, per spec: "it means nothing on the next connection").
  • ws.rs::run_session now parses incoming client text frames (TransferAccept/TransferComplete/Heartbeat), tracks offered files, and spawns a streaming task on accept that reads <media_dir>/<sha256> from resume_from_offset, chunks at MAX_CHUNK_BYTES, and hands frames back over an internal channel so the socket keeps one writer.
  • New crate::transfers::TransferLimiter: a semaphore for concurrent transfers and a token-bucket bandwidth limiter, both shared server-wide, per the spec's "Concurrent transfers and total bandwidth are limited, server-side". Configurable via a new [transfers] config section (max_concurrent, default 8; max_bandwidth_bytes_per_second, default 0/unlimited), documented in plan/server/configuration.md.

Note on process: there was no QA-written test for this issue — it's the design-and-implementation gap itself. Given the protocol is frozen/spec'd byte-for-byte and the client side already has full tests defining the exact wire contract, I implemented against that spec/client as the oracle and wrote my own unit + integration tests.

Verification:

  • cargo clippy -p server --all-targets -- -D warnings: clean.
  • cargo test -p server --lib: 126 passed, 1 ignored (pre-existing, needs poppler-utils).
  • cargo test -p server --test ws_integration --test enrolment_acceptance_integration --test playlist_integration --test playlist_acceptance_integration: 29 passed, including two new end-to-end tests that round-trip a real file through offer→accept→chunks→complete and verify the received bytes hash to the offered sha256.
  • scripts/local-gates.sh running now; will follow up with the result before opening the PR.

Branch: fix/GRI-1916-server-media-transfer.

Implemented server-side media transfer end to end: - `push_to_device` (`playlists/handlers.rs`) now sends one `TransferOffer` per `MediaManifest` entry, after `MediaManifest`/`PlaylistPush`, matching `plan/shared/protocol.md`. - `DeviceConnections` grew a per-connection `transfer_id` allocator (resets on reconnect, per spec: "it means nothing on the next connection"). - `ws.rs::run_session` now parses incoming client text frames (`TransferAccept`/`TransferComplete`/`Heartbeat`), tracks offered files, and spawns a streaming task on accept that reads `<media_dir>/<sha256>` from `resume_from_offset`, chunks at `MAX_CHUNK_BYTES`, and hands frames back over an internal channel so the socket keeps one writer. - New `crate::transfers::TransferLimiter`: a semaphore for concurrent transfers and a token-bucket bandwidth limiter, both shared server-wide, per the spec's "Concurrent transfers and total bandwidth are limited, server-side". Configurable via a new `[transfers]` config section (`max_concurrent`, default 8; `max_bandwidth_bytes_per_second`, default 0/unlimited), documented in `plan/server/configuration.md`. Note on process: there was no QA-written test for this issue — it's the design-and-implementation gap itself. Given the protocol is frozen/spec'd byte-for-byte and the client side already has full tests defining the exact wire contract, I implemented against that spec/client as the oracle and wrote my own unit + integration tests. Verification: - `cargo clippy -p server --all-targets -- -D warnings`: clean. - `cargo test -p server --lib`: 126 passed, 1 ignored (pre-existing, needs poppler-utils). - `cargo test -p server --test ws_integration --test enrolment_acceptance_integration --test playlist_integration --test playlist_acceptance_integration`: 29 passed, including two new end-to-end tests that round-trip a real file through offer→accept→chunks→complete and verify the received bytes hash to the offered sha256. - `scripts/local-gates.sh` running now; will follow up with the result before opening the PR. Branch: `fix/GRI-1916-server-media-transfer`.
vivy closed this issue 2026-11-11 06:26:39 +00:00
Sign in to join this conversation.
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Grimoirix/grimoire-of-digital-signage#127
No description provided.