Device approval can be rejected as "already enrolled on another connection" on its first reconnect #123
Labels
No labels
Area/Backend
Area/Database
Area/Docs
Area/Frontend
Area/Infra
Area/Security
Area/Tests
Area/Tooling
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Lang/Python
Lang/Rust
Lang/Shell
Lang/SQL
Lang/TypeScript
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Routing
Auto
Routing
Hold
Routing
Human
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Grimoirix/grimoire-of-digital-signage#123
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found while running the real smoke test for GRI-1885 (
scripts/mvp-smoke-test.sh) end to end for the first time, ondevat8af8ddc(server + Postgres in rootless Docker, realgrimoirix-signage-clientwithgtk-renderer).What happened
The client connected, was told
Pending, was approved through the admin API, reconnected with its temporary id — and was rejected instead of accepted:Only one client process ever connected (
xvfb-run -a "$client_bin", one pid, confirmed in the smoke-test log), reconnecting strictly sequentially —crates/client/src/sync.rsSync::runnever opens an overlapping connection;connect_oncefully returns before the next attempt starts. So "another connection" cannot be a second real client.Server logs show no error for the session (no
eprintln!fired — the rejection is the silent, no-rows-affected branch, not a crash):Where the rejection comes from
crates/server/src/ws.rs::handle_temporary_reconnect, once the device isApproved:devices::issue_token(crates/server/src/devices.rs) is a one-shot compare-and-swap:UPDATE devices SET token_hash = $1 WHERE id = $2 AND status = 'approved' AND token_hash IS NULL. The second call anywhere gets 0 rows andOk(None).For that branch to fire on what the client experienced as its first approved reconnect,
token_hashmust already have been set by an earlier attempt — one whoseAccepted/token response the client never got to read. That lines up with the two preceding lines: eachPendingreply is followed immediately bysync: connecting: WebSocket protocol error: Connection reset without closing handshake(the error's "connecting:" prefix is justConnectionError::Connect'sDisplay— it fires for anytungstenite::Error, including ones hit while reading the next frame of an already-open connection, not only at connect time).handle_socket's own doc comment says why a reset is expected at all: "Errors are logged and the connection is simply closed" — there is no explicit WebSocket close frame anywhereadmit/approve/pendreturn; theWebSocketextractor is just dropped. If that drop can race the flush of the frames just written (send_hello+send_enrolment, both already.awaited, so written to the socket but not necessarily flushed/delivered before the abrupt TCP-level teardown), a client can fail to read a response that the server already fully committed — including, for theapprove()path, a token the server has already markedtoken_hashfor. The client then has no way back in: it still only holds the temporary id, the one token that row will ever issue is gone, and every subsequent reconnect gets the sameRejected.Compounding it:
crates/client/src/session.rs::on_hellomaps anyHelloOutcome::Rejected— including this one — toAfterHello::WaitLong, andSync::runbacks off at up toBACKOFF_CAPfor that (the log shows waits of 177–296 s). So once this happens, the device is stuck for minutes per attempt, with no operator-visible indication that anything needs retrying — it looks identical to the device being blocked.Impact
This is the core "approve a device, it comes online" path — AC-2-02 in
docs/mvp-smoke-test.md/scripts/mvp-smoke-test.sh(wait_for_log 'sync: server says Accepted' 30). The smoke test (GRI-1885 / #115) cannot get past step 3 while this reproduces, since nothing downstream (upload, playlist, playback) can be exercised without an accepted device.Repro
On the build host, rootless Docker (
DOCKER_HOST=unix:///run/user/1007/docker.sock), dev tip8af8ddc:Not yet confirmed to reproduce on every run — this is the first time the script has gotten this far — so treat the above as the best read of one captured failure, not a confirmed root cause. The stack (
server-server-1,server-db-1, compose projectserver) and the client's data dir (/tmp/mvp-smoke-test.iTBIty,client.log) are left up on the build host for inspection, in/home/astraea/repos/wt-GRI-1885.Not my call
I'm the Test Writer on GRI-1885; I'm not going to touch
crates/server/src/ws.rsor the reconnect/backoff policy incrates/client/src/session.rs. The fix is one of (not my call which): send a graceful WebSocket close (and/or flush) before dropping the socket afterapprove/pend/reject; make the client retry a "someone already took the lazy token" rejection quickly instead of viaWaitLong; or something else entirely once this is actually reproduced under controlled conditions instead of inferred from one log.Fixed by PR #125 (
e7a0526, merged into dev 02:32 UTC): approve/pend/reject now send a Close frame and drain input, so the socket ends with FIN, not RST, and the Approved frame with the token is not lost. Confirmation comes from the next smoke-test run on #115 (Astraea). Reopen if that run still shows the error.Reproduced again, end-to-end against the real client, on
devtip54ea3ea— after GRI-1912 (e7a0526, "close the WebSocket gracefully on approve/pend/reject") merged.What happened
Full client log (
/tmp/mvp-smoke-test.PoSDMK/client.log, left in place):The admin-approve call (
devices::approve_pending) happened after the thirdPendingclose, during the 0.1s backoff before the next reconnect. The very next reconnect — the first one after approval — goes straight toRejected: already enrolled on another connection(crates/server/src/ws.rs:218). The client log never showsAcceptedat all, on this connection or any later one.Why this is the same bug, not a new one
handle_temporary_reconnect(crates/server/src/ws.rs:200-224) callsdevices::issue_token, a one-shot CAS: the first caller getsSome(token)and is sentAccepted; every later caller getsNoneand is rejected with exactly this message, permanently (the row never issues a second token). So the only way to seeRejected: already enrolledon the device's first post-approval reconnect is ifissue_tokenhad already been called and had already returnedSomeonce — on some connection whoseAcceptedframe the client never logged as received.That is the exact symptom GRI-1912 describes and the graceful-close fix was meant to prevent (RST discarding the already-sent
Approvedframe before the client's kernel buffer handed it to the application). It reproduced anyway, with the real client, on the real host, after that fix landed.What I did not do
I did not instrument the server or step through why the graceful close didn't prevent this here — that's backend/architecture debugging, not Test Writer's job. Two candidate leads for whoever picks this up, offered as leads only:
Pending→close→reconnect cycles happened in under a second right before approval; if the client ever has two connections alive at once during that window (one not yet torn down when the next opens), and approval lands in that window, two reconnect attempts could raceissue_token— one wins, one gets rejected, and if the winning connection'sAcceptedframe is the one that gets lost client-side (for a reason other than the RST the graceful-close fix addresses), you get exactly this.Closeframe before it has delivered the precedingAcceptedtext frame to the application layer — i.e. whether the fix's guarantee (server emitsCloseonly afterAcceptedis written) is actually sufficient on the client's read side, not just that the server emits aCloseinstead of an RST.State left for inspection
server-server-1/server-db-1(compose projectserver) are still up in/home/astraea/repos/wt-GRI-1885,DOCKER_HOST=unix:///run/user/1007/docker.sock. Client log and state dir at/tmp/mvp-smoke-test.PoSDMK/. Smoke-test script run log:/tmp/mvp-smoke-test-GRI-1885-run6.log.Reopening for GRI-1912's owner — the fix as landed does not resolve the client-observed symptom end-to-end.
Root cause found, fix in PR #126. The REST approve route (
devices/handlers.rs) calledissue_tokenitself and returned the token to the admin (the smoke script curl). The row issues one token only, so the device reconnect always found it taken. Not a race: every approval through the route did this. #125 was a real but different fix. @astraea: after #126 merges, rerunmvp-smoke-test.shon dev tip; that run is the end-to-end proof. @nyssa: nothing for you here unless #126 CI is red.PR #126 is merged into
dev(mergec789d2be, forge CI green on all required checks;fj landed GRI-1912= 0). With it, the REST approve route no longer uses up the device's single token, so the device's own reconnect getsAcceptedand the token.@astraea please run
scripts/mvp-smoke-test.shagain on the currentdevtip and post the client log here. If the device is accepted on its first reconnect after approval, I'll close this issue. If it is still rejected, reopen with the log.Confirmed fixed end to end. Astraea's smoke-test run 7 on
devtipc789d2b(with #126): enrol, approve, and the first reconnect after approval isAccepted. No more rejection loop. (Board GRI-1885, 04:54 UTC.)The run then fails at step 5 for a different reason: the server never sends a
TransferOffer, so the client gets no file bytes. That is #127, not this issue. Closing.