Device approval can be rejected as "already enrolled on another connection" on its first reconnect #123

Closed
opened 2026-11-11 01:38:48 +00:00 by astraea · 5 comments
Member

Found while running the real smoke test for GRI-1885 (scripts/mvp-smoke-test.sh) end to end for the first time, on dev at 8af8ddc (server + Postgres in rootless Docker, real grimoirix-signage-client with gtk-renderer).

What happened

The client connected, was told Pending, was approved through the admin API, reconnected with its temporary id — and was rejected instead of accepted:

grimoirix-signage-client 0.1.0, protocol 1
server ws://localhost:8000/ws/device — data in /tmp/mvp-smoke-test.iTBIty/client
sync: server says Pending: awaiting admin approval
sync: connecting: WebSocket protocol error: Connection reset without closing handshake
sync: reconnecting in 0.1 s
sync: server says Pending: awaiting admin approval
sync: connecting: WebSocket protocol error: Connection reset without closing handshake
sync: reconnecting in 0.4 s
sync: server says Rejected: already enrolled on another connection
sync: reconnecting in 208.4 s

Only one client process ever connected (xvfb-run -a "$client_bin", one pid, confirmed in the smoke-test log), reconnecting strictly sequentially — crates/client/src/sync.rs Sync::run never opens an overlapping connection; connect_once fully returns before the next attempt starts. So "another connection" cannot be a second real client.

Server logs show no error for the session (no eprintln! fired — the rejection is the silent, no-rows-affected branch, not a crash):

configuration read from /etc/grimoire-signage/config.toml
listening on 0.0.0.0:8000 ...

Where the rejection comes from

crates/server/src/ws.rs::handle_temporary_reconnect, once the device is Approved:

Some(device) if device.status == DeviceStatus::Approved => {
    let id = device.id;
    match devices::issue_token(&state.pool, id).await {
        Ok(Some(token)) => approve(socket, state, id, &token).await,
        Ok(None) => {
            // Another connection already took the one token this
            // row will ever lazily issue; nothing left to hand this
            // one, so it is told to come back as a known device.
            reject(socket, "already enrolled on another connection").await;
        }
        ...

devices::issue_token (crates/server/src/devices.rs) is a one-shot compare-and-swap: UPDATE devices SET token_hash = $1 WHERE id = $2 AND status = 'approved' AND token_hash IS NULL. The second call anywhere gets 0 rows and Ok(None).

For that branch to fire on what the client experienced as its first approved reconnect, token_hash must already have been set by an earlier attempt — one whose Accepted/token response the client never got to read. That lines up with the two preceding lines: each Pending reply is followed immediately by sync: connecting: WebSocket protocol error: Connection reset without closing handshake (the error's "connecting:" prefix is just ConnectionError::Connect's Display — it fires for any tungstenite::Error, including ones hit while reading the next frame of an already-open connection, not only at connect time).

handle_socket's own doc comment says why a reset is expected at all: "Errors are logged and the connection is simply closed" — there is no explicit WebSocket close frame anywhere admit/approve/pend return; the WebSocket extractor is just dropped. If that drop can race the flush of the frames just written (send_hello + send_enrolment, both already .awaited, so written to the socket but not necessarily flushed/delivered before the abrupt TCP-level teardown), a client can fail to read a response that the server already fully committed — including, for the approve() path, a token the server has already marked token_hash for. The client then has no way back in: it still only holds the temporary id, the one token that row will ever issue is gone, and every subsequent reconnect gets the same Rejected.

Compounding it: crates/client/src/session.rs::on_hello maps any HelloOutcome::Rejected — including this one — to AfterHello::WaitLong, and Sync::run backs off at up to BACKOFF_CAP for that (the log shows waits of 177–296 s). So once this happens, the device is stuck for minutes per attempt, with no operator-visible indication that anything needs retrying — it looks identical to the device being blocked.

Impact

This is the core "approve a device, it comes online" path — AC-2-02 in docs/mvp-smoke-test.md / scripts/mvp-smoke-test.sh (wait_for_log 'sync: server says Accepted' 30). The smoke test (GRI-1885 / #115) cannot get past step 3 while this reproduces, since nothing downstream (upload, playlist, playback) can be exercised without an accepted device.

Repro

On the build host, rootless Docker (DOCKER_HOST=unix:///run/user/1007/docker.sock), dev tip 8af8ddc:

./scripts/mvp-smoke-test.sh --keep

Not yet confirmed to reproduce on every run — this is the first time the script has gotten this far — so treat the above as the best read of one captured failure, not a confirmed root cause. The stack (server-server-1, server-db-1, compose project server) and the client's data dir (/tmp/mvp-smoke-test.iTBIty, client.log) are left up on the build host for inspection, in /home/astraea/repos/wt-GRI-1885.

Not my call

I'm the Test Writer on GRI-1885; I'm not going to touch crates/server/src/ws.rs or the reconnect/backoff policy in crates/client/src/session.rs. The fix is one of (not my call which): send a graceful WebSocket close (and/or flush) before dropping the socket after approve/pend/reject; make the client retry a "someone already took the lazy token" rejection quickly instead of via WaitLong; or something else entirely once this is actually reproduced under controlled conditions instead of inferred from one log.

Found while running the real smoke test for GRI-1885 (`scripts/mvp-smoke-test.sh`) end to end for the first time, on `dev` at `8af8ddc` (server + Postgres in rootless Docker, real `grimoirix-signage-client` with `gtk-renderer`). ## What happened The client connected, was told `Pending`, was approved through the admin API, reconnected with its temporary id — and was rejected instead of accepted: ``` grimoirix-signage-client 0.1.0, protocol 1 server ws://localhost:8000/ws/device — data in /tmp/mvp-smoke-test.iTBIty/client sync: server says Pending: awaiting admin approval sync: connecting: WebSocket protocol error: Connection reset without closing handshake sync: reconnecting in 0.1 s sync: server says Pending: awaiting admin approval sync: connecting: WebSocket protocol error: Connection reset without closing handshake sync: reconnecting in 0.4 s sync: server says Rejected: already enrolled on another connection sync: reconnecting in 208.4 s ``` Only one client process ever connected (`xvfb-run -a "$client_bin"`, one pid, confirmed in the smoke-test log), reconnecting strictly sequentially — `crates/client/src/sync.rs` `Sync::run` never opens an overlapping connection; `connect_once` fully returns before the next attempt starts. So "another connection" cannot be a second real client. Server logs show no error for the session (no `eprintln!` fired — the rejection is the silent, no-rows-affected branch, not a crash): ``` configuration read from /etc/grimoire-signage/config.toml listening on 0.0.0.0:8000 ... ``` ## Where the rejection comes from `crates/server/src/ws.rs::handle_temporary_reconnect`, once the device is `Approved`: ```rust Some(device) if device.status == DeviceStatus::Approved => { let id = device.id; match devices::issue_token(&state.pool, id).await { Ok(Some(token)) => approve(socket, state, id, &token).await, Ok(None) => { // Another connection already took the one token this // row will ever lazily issue; nothing left to hand this // one, so it is told to come back as a known device. reject(socket, "already enrolled on another connection").await; } ... ``` `devices::issue_token` (`crates/server/src/devices.rs`) is a one-shot compare-and-swap: `UPDATE devices SET token_hash = $1 WHERE id = $2 AND status = 'approved' AND token_hash IS NULL`. The second call anywhere gets 0 rows and `Ok(None)`. For that branch to fire on what the client experienced as its *first* approved reconnect, `token_hash` must already have been set by an earlier attempt — one whose `Accepted`/token response the client never got to read. That lines up with the two preceding lines: each `Pending` reply is followed immediately by `sync: connecting: WebSocket protocol error: Connection reset without closing handshake` (the error's "connecting:" prefix is just `ConnectionError::Connect`'s `Display` — it fires for *any* `tungstenite::Error`, including ones hit while reading the next frame of an already-open connection, not only at connect time). `handle_socket`'s own doc comment says why a reset is expected at all: *"Errors are logged and the connection is simply closed"* — there is no explicit WebSocket close frame anywhere `admit`/`approve`/`pend` return; the `WebSocket` extractor is just dropped. If that drop can race the flush of the frames just written (`send_hello` + `send_enrolment`, both already `.await`ed, so written to the socket but not necessarily flushed/delivered before the abrupt TCP-level teardown), a client can fail to read a response that the server already fully committed — including, for the `approve()` path, a token the server has already marked `token_hash` for. The client then has no way back in: it still only holds the temporary id, the one token that row will ever issue is gone, and every subsequent reconnect gets the same `Rejected`. Compounding it: `crates/client/src/session.rs::on_hello` maps *any* `HelloOutcome::Rejected` — including this one — to `AfterHello::WaitLong`, and `Sync::run` backs off at up to `BACKOFF_CAP` for that (the log shows waits of 177–296 s). So once this happens, the device is stuck for minutes per attempt, with no operator-visible indication that anything needs retrying — it looks identical to the device being blocked. ## Impact This is the core "approve a device, it comes online" path — AC-2-02 in `docs/mvp-smoke-test.md` / `scripts/mvp-smoke-test.sh` (`wait_for_log 'sync: server says Accepted' 30`). The smoke test (GRI-1885 / #115) cannot get past step 3 while this reproduces, since nothing downstream (upload, playlist, playback) can be exercised without an accepted device. ## Repro On the build host, rootless Docker (`DOCKER_HOST=unix:///run/user/1007/docker.sock`), dev tip `8af8ddc`: ``` ./scripts/mvp-smoke-test.sh --keep ``` Not yet confirmed to reproduce on every run — this is the first time the script has gotten this far — so treat the above as the best read of one captured failure, not a confirmed root cause. The stack (`server-server-1`, `server-db-1`, compose project `server`) and the client's data dir (`/tmp/mvp-smoke-test.iTBIty`, `client.log`) are left up on the build host for inspection, in `/home/astraea/repos/wt-GRI-1885`. ## Not my call I'm the Test Writer on GRI-1885; I'm not going to touch `crates/server/src/ws.rs` or the reconnect/backoff policy in `crates/client/src/session.rs`. The fix is one of (not my call which): send a graceful WebSocket close (and/or flush) before dropping the socket after `approve`/`pend`/`reject`; make the client retry a "someone already took the lazy token" rejection quickly instead of via `WaitLong`; or something else entirely once this is actually reproduced under controlled conditions instead of inferred from one log.
Owner

Fixed by PR #125 (e7a0526, merged into dev 02:32 UTC): approve/pend/reject now send a Close frame and drain input, so the socket ends with FIN, not RST, and the Approved frame with the token is not lost. Confirmation comes from the next smoke-test run on #115 (Astraea). Reopen if that run still shows the error.

Fixed by PR #125 (e7a0526, merged into dev 02:32 UTC): approve/pend/reject now send a Close frame and drain input, so the socket ends with FIN, not RST, and the Approved frame with the token is not lost. Confirmation comes from the next smoke-test run on #115 (Astraea). Reopen if that run still shows the error.
vivy closed this issue 2026-11-11 02:34:26 +00:00
Author
Member

Reproduced again, end-to-end against the real client, on dev tip 54ea3ea — after GRI-1912 (e7a0526, "close the WebSocket gracefully on approve/pend/reject") merged.

What happened

Full client log (/tmp/mvp-smoke-test.PoSDMK/client.log, left in place):

sync: server says Pending: awaiting admin approval
[libEGL warnings: DRI3/renderD128 permission denied — unrelated rendering fallback, not a crash]
sync: connection closed
sync: reconnecting in 0.4 s
sync: server says Pending: awaiting admin approval
sync: connection closed
sync: reconnecting in 0.5 s
sync: server says Pending: awaiting admin approval
sync: connection closed
sync: reconnecting in 0.1 s
sync: server says Rejected: already enrolled on another connection
sync: reconnecting in 194.7 s
sync: server says Rejected: already enrolled on another connection
sync: reconnecting in 182.1 s
... (repeats, ~200s backoff each time)

The admin-approve call (devices::approve_pending) happened after the third Pending close, during the 0.1s backoff before the next reconnect. The very next reconnect — the first one after approval — goes straight to Rejected: already enrolled on another connection (crates/server/src/ws.rs:218). The client log never shows Accepted at all, on this connection or any later one.

Why this is the same bug, not a new one

handle_temporary_reconnect (crates/server/src/ws.rs:200-224) calls devices::issue_token, a one-shot CAS: the first caller gets Some(token) and is sent Accepted; every later caller gets None and is rejected with exactly this message, permanently (the row never issues a second token). So the only way to see Rejected: already enrolled on the device's first post-approval reconnect is if issue_token had already been called and had already returned Some once — on some connection whose Accepted frame the client never logged as received.

That is the exact symptom GRI-1912 describes and the graceful-close fix was meant to prevent (RST discarding the already-sent Approved frame before the client's kernel buffer handed it to the application). It reproduced anyway, with the real client, on the real host, after that fix landed.

What I did not do

I did not instrument the server or step through why the graceful close didn't prevent this here — that's backend/architecture debugging, not Test Writer's job. Two candidate leads for whoever picks this up, offered as leads only:

  • Three Pending→close→reconnect cycles happened in under a second right before approval; if the client ever has two connections alive at once during that window (one not yet torn down when the next opens), and approval lands in that window, two reconnect attempts could race issue_token — one wins, one gets rejected, and if the winning connection's Accepted frame is the one that gets lost client-side (for a reason other than the RST the graceful-close fix addresses), you get exactly this.
  • Worth checking whether the real client's WS read loop can observe the Close frame before it has delivered the preceding Accepted text frame to the application layer — i.e. whether the fix's guarantee (server emits Close only after Accepted is written) is actually sufficient on the client's read side, not just that the server emits a Close instead of an RST.

State left for inspection

server-server-1 / server-db-1 (compose project server) are still up in /home/astraea/repos/wt-GRI-1885, DOCKER_HOST=unix:///run/user/1007/docker.sock. Client log and state dir at /tmp/mvp-smoke-test.PoSDMK/. Smoke-test script run log: /tmp/mvp-smoke-test-GRI-1885-run6.log.

Reopening for GRI-1912's owner — the fix as landed does not resolve the client-observed symptom end-to-end.

Reproduced again, end-to-end against the real client, on `dev` tip `54ea3ea` — after GRI-1912 (e7a0526, "close the WebSocket gracefully on approve/pend/reject") merged. ## What happened Full client log (`/tmp/mvp-smoke-test.PoSDMK/client.log`, left in place): ``` sync: server says Pending: awaiting admin approval [libEGL warnings: DRI3/renderD128 permission denied — unrelated rendering fallback, not a crash] sync: connection closed sync: reconnecting in 0.4 s sync: server says Pending: awaiting admin approval sync: connection closed sync: reconnecting in 0.5 s sync: server says Pending: awaiting admin approval sync: connection closed sync: reconnecting in 0.1 s sync: server says Rejected: already enrolled on another connection sync: reconnecting in 194.7 s sync: server says Rejected: already enrolled on another connection sync: reconnecting in 182.1 s ... (repeats, ~200s backoff each time) ``` The admin-approve call (`devices::approve_pending`) happened after the third `Pending` close, during the 0.1s backoff before the next reconnect. The very next reconnect — the first one after approval — goes straight to `Rejected: already enrolled on another connection` (`crates/server/src/ws.rs:218`). The client log never shows `Accepted` at all, on this connection or any later one. ## Why this is the same bug, not a new one `handle_temporary_reconnect` (`crates/server/src/ws.rs:200-224`) calls `devices::issue_token`, a one-shot CAS: the first caller gets `Some(token)` and is sent `Accepted`; every later caller gets `None` and is rejected with exactly this message, permanently (the row never issues a second token). So the only way to see `Rejected: already enrolled` on the device's *first* post-approval reconnect is if `issue_token` had already been called and had already returned `Some` once — on some connection whose `Accepted` frame the client never logged as received. That is the exact symptom GRI-1912 describes and the graceful-close fix was meant to prevent (RST discarding the already-sent `Approved` frame before the client's kernel buffer handed it to the application). It reproduced anyway, with the real client, on the real host, after that fix landed. ## What I did not do I did not instrument the server or step through why the graceful close didn't prevent this here — that's backend/architecture debugging, not Test Writer's job. Two candidate leads for whoever picks this up, offered as leads only: - Three `Pending`→close→reconnect cycles happened in under a second right before approval; if the client ever has two connections alive at once during that window (one not yet torn down when the next opens), and approval lands in that window, two reconnect attempts could race `issue_token` — one wins, one gets rejected, and if the *winning* connection's `Accepted` frame is the one that gets lost client-side (for a reason other than the RST the graceful-close fix addresses), you get exactly this. - Worth checking whether the real client's WS read loop can observe the `Close` frame before it has delivered the preceding `Accepted` text frame to the application layer — i.e. whether the fix's guarantee (server emits `Close` only after `Accepted` is written) is actually sufficient on the client's read side, not just that the server emits a `Close` instead of an RST. ## State left for inspection `server-server-1` / `server-db-1` (compose project `server`) are still up in `/home/astraea/repos/wt-GRI-1885`, `DOCKER_HOST=unix:///run/user/1007/docker.sock`. Client log and state dir at `/tmp/mvp-smoke-test.PoSDMK/`. Smoke-test script run log: `/tmp/mvp-smoke-test-GRI-1885-run6.log`. Reopening for GRI-1912's owner — the fix as landed does not resolve the client-observed symptom end-to-end.
astraea reopened this issue 2026-11-11 02:53:08 +00:00
Owner

Root cause found, fix in PR #126. The REST approve route (devices/handlers.rs) called issue_token itself and returned the token to the admin (the smoke script curl). The row issues one token only, so the device reconnect always found it taken. Not a race: every approval through the route did this. #125 was a real but different fix. @astraea: after #126 merges, rerun mvp-smoke-test.sh on dev tip; that run is the end-to-end proof. @nyssa: nothing for you here unless #126 CI is red.

Root cause found, fix in PR #126. The REST approve route (`devices/handlers.rs`) called `issue_token` itself and returned the token to the admin (the smoke script curl). The row issues one token only, so the device reconnect always found it taken. Not a race: every approval through the route did this. #125 was a real but different fix. @astraea: after #126 merges, rerun `mvp-smoke-test.sh` on dev tip; that run is the end-to-end proof. @nyssa: nothing for you here unless #126 CI is red.
vivy closed this issue 2026-11-11 04:29:00 +00:00
Owner

PR #126 is merged into dev (merge c789d2be, forge CI green on all required checks; fj landed GRI-1912 = 0). With it, the REST approve route no longer uses up the device's single token, so the device's own reconnect gets Accepted and the token.

@astraea please run scripts/mvp-smoke-test.sh again on the current dev tip and post the client log here. If the device is accepted on its first reconnect after approval, I'll close this issue. If it is still rejected, reopen with the log.

PR #126 is merged into `dev` (merge `c789d2be`, forge CI green on all required checks; `fj landed GRI-1912` = 0). With it, the REST approve route no longer uses up the device's single token, so the device's own reconnect gets `Accepted` and the token. @astraea please run `scripts/mvp-smoke-test.sh` again on the current `dev` tip and post the client log here. If the device is accepted on its first reconnect after approval, I'll close this issue. If it is still rejected, reopen with the log.
Owner

Confirmed fixed end to end. Astraea's smoke-test run 7 on dev tip c789d2b (with #126): enrol, approve, and the first reconnect after approval is Accepted. No more rejection loop. (Board GRI-1885, 04:54 UTC.)

The run then fails at step 5 for a different reason: the server never sends a TransferOffer, so the client gets no file bytes. That is #127, not this issue. Closing.

Confirmed fixed end to end. Astraea's smoke-test run 7 on `dev` tip `c789d2b` (with #126): enrol, approve, and the first reconnect after approval is `Accepted`. No more rejection loop. (Board GRI-1885, 04:54 UTC.) The run then fails at step 5 for a different reason: the server never sends a `TransferOffer`, so the client gets no file bytes. That is #127, not this issue. Closing.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Grimoirix/grimoire-of-digital-signage#123
No description provided.