install.sh upgrade does not install the new client build (same package version) #206

Closed
opened 2026-11-15 09:02:05 +00:00 by vivy · 5 comments
Owner

Found on the test client on 2026-11-15 (GRI-2051). install.sh upgrade printed "installed grimoirix-signage-client_0.1.0_amd64.deb" and left the old binary in /usr/bin (sha 29296aaf from 11-13; the freshly built one was ee490c48). Fixed by hand with dpkg -i.

Cause. install.sh:426 runs apt-get install -y "$deb". The package version is 0.1.0 on every build, and apt skips a package whose version is already installed, with exit 0. The output goes to /dev/null, so nothing shows it.

Fix options.

  1. Install with apt-get install --reinstall -y "$deb" (keeps dependency resolution, which plain dpkg -i does not).
  2. Give every build its own package version, e.g. 0.1.0+git<date>.<sha> from package-client-deb.sh, so an upgrade is a real upgrade and dpkg -l tells which build runs.

Recommended: both. (2) makes the installed build visible; (1) makes a rebuild of the same commit still apply.

Also: after installing, the script must restart the running player (or say plainly that a restart/log-in is needed). The player that was running kept the old binary until the device was restarted.

Acceptance. Running install.sh upgrade twice on two different commits leaves the second build's binary in /usr/bin, and the script prints the installed version.

Found on the test client on 2026-11-15 (GRI-2051). `install.sh upgrade` printed "installed grimoirix-signage-client_0.1.0_amd64.deb" and left the old binary in `/usr/bin` (sha 29296aaf from 11-13; the freshly built one was ee490c48). Fixed by hand with `dpkg -i`. **Cause.** `install.sh:426` runs `apt-get install -y "$deb"`. The package version is `0.1.0` on every build, and apt skips a package whose version is already installed, with exit 0. The output goes to `/dev/null`, so nothing shows it. **Fix options.** 1. Install with `apt-get install --reinstall -y "$deb"` (keeps dependency resolution, which plain `dpkg -i` does not). 2. Give every build its own package version, e.g. `0.1.0+git<date>.<sha>` from `package-client-deb.sh`, so an upgrade is a real upgrade and `dpkg -l` tells which build runs. Recommended: both. (2) makes the installed build visible; (1) makes a rebuild of the same commit still apply. **Also:** after installing, the script must restart the running player (or say plainly that a restart/log-in is needed). The player that was running kept the old binary until the device was restarted. **Acceptance.** Running `install.sh upgrade` twice on two different commits leaves the second build's binary in `/usr/bin`, and the script prints the installed version.
Member

Fix is in PR #217 (base dev, commit 7613c535). CI is queued: position 2, ETA about 1h50.

Cause, reproduced. I ran the real apt-get and dpkg against a private dpkg database. A plain apt-get install -y of a second build with the same version exits 0 and leaves the first build's binary on disk. --reinstall replaces it. An older version also needs --allow-downgrades.

What changes.

  • Each build gets its own package version, 0.1.0+git<UTC commit time>.<sha>, so dpkg -l grimoirix-signage-client shows which build runs.
  • install.sh runs apt-get install --reinstall --allow-downgrades on the exact file the package script built. Then it checks that the dpkg version and the sha256 of /usr/bin/grimoirix-signage-client match the .deb. On a mismatch it stops with an error instead of printing "installed". It prints the installed version and the binary's sha.
  • A player still running the replaced binary is stopped and started again in the same user session, with the same environment, working directory and arguments. If that fails, the script says plainly to log out or restart the device.
  • New test scripts/tests/install-client-upgrade-test.sh (18 assertions, real apt and dpkg, no root). It runs in local-gates.sh and as CI job installer.

Owed before this closes: the acceptance run on the test client from GRI-2051. I can't run it from the build host: there is no desktop session there, and I have no access to the device.

  1. curl -fsSLo install.sh https://git.grimoirix.org/Grimoirix/grimoire-of-digital-signage/raw/branch/fix/GRI-2081-client-upgrade-reinstall/install.sh
  2. sudo SCRYGLASS_REF=fix/GRI-2081-client-upgrade-reinstall bash install.sh upgrade client --force
  3. Check: the printed version equals dpkg-query -W grimoirix-signage-client. sha256sum /usr/bin/grimoirix-signage-client equals the printed sha. The player on screen came back by itself.
  4. Run it once more after another commit lands on dev (SCRYGLASS_REF=dev). The version and sha change to the new build.
Fix is in PR #217 (base `dev`, commit `7613c535`). CI is queued: position 2, ETA about 1h50. **Cause, reproduced.** I ran the real apt-get and dpkg against a private dpkg database. A plain `apt-get install -y` of a second build with the same version exits 0 and leaves the first build's binary on disk. `--reinstall` replaces it. An older version also needs `--allow-downgrades`. **What changes.** - Each build gets its own package version, `0.1.0+git<UTC commit time>.<sha>`, so `dpkg -l grimoirix-signage-client` shows which build runs. - `install.sh` runs `apt-get install --reinstall --allow-downgrades` on the exact file the package script built. Then it **checks** that the dpkg version and the sha256 of `/usr/bin/grimoirix-signage-client` match the `.deb`. On a mismatch it stops with an error instead of printing "installed". It prints the installed version and the binary's sha. - A player still running the replaced binary is stopped and started again in the same user session, with the same environment, working directory and arguments. If that fails, the script says plainly to log out or restart the device. - New test `scripts/tests/install-client-upgrade-test.sh` (18 assertions, real apt and dpkg, no root). It runs in `local-gates.sh` and as CI job `installer`. **Owed before this closes:** the acceptance run on the test client from GRI-2051. I can't run it from the build host: there is no desktop session there, and I have no access to the device. 1. `curl -fsSLo install.sh https://git.grimoirix.org/Grimoirix/grimoire-of-digital-signage/raw/branch/fix/GRI-2081-client-upgrade-reinstall/install.sh` 2. `sudo SCRYGLASS_REF=fix/GRI-2081-client-upgrade-reinstall bash install.sh upgrade client --force` 3. Check: the printed version equals `dpkg-query -W grimoirix-signage-client`. `sha256sum /usr/bin/grimoirix-signage-client` equals the printed sha. The player on screen came back by itself. 4. Run it once more after another commit lands on `dev` (`SCRYGLASS_REF=dev`). The version and sha change to the new build.
Member

Why install.sh client upgrade was red in CI on d281986a, and the fix (now c523957).

The job log shows 17 of 18 checks passing. The only failure was the old player is gone. All install checks passed, and so did the player restart.

Cause: the CI container's PID 1 does not reap orphans. The old player is started with setsid -f, so its parent there is that PID 1. Once stopped, it stays a zombie, and kill -0 succeeds on a zombie. On the build host, the run has a subreaper that reaps it, which is why the test passed 18/18 locally. Nothing about apt or dpkg differs between the two.

It was in install.sh too, not only in the test. restart_players used kill -0 to wait for the old player to stop. Under a parent that never reaps, it waited the full 10 s and then sent SIGKILL to a process that had already exited.

Fix:

  • install.sh: a new pid_alive reads State: from /proc/PID/status and counts Z as stopped. restart_players waits on it.
  • The test: the old player now runs under a holder process that never reaps it, so the zombie occurs on every host, not only in the container. Two checks are added: that it really became a zombie, and that the stop did not take ≥ 8 s.

Measured on the build host:

  • New test against the old install.sh: red, the stop did not wait out its ten seconds on the zombie (13s).
  • With the fix: 20/20, and the stop takes 2 s.
  • Shell gates, pre-push-test.sh and repo-boundary: 0.

origin/dev is merged in. CI run #349 is going.

**Why `install.sh client upgrade` was red in CI on d281986a, and the fix (now c523957).** The job log shows 17 of 18 checks passing. The only failure was `the old player is gone`. All install checks passed, and so did the player restart. **Cause:** the CI container's PID 1 does not reap orphans. The old player is started with `setsid -f`, so its parent there is that PID 1. Once stopped, it stays a zombie, and `kill -0` succeeds on a zombie. On the build host, the run has a subreaper that reaps it, which is why the test passed 18/18 locally. Nothing about apt or dpkg differs between the two. **It was in `install.sh` too, not only in the test.** `restart_players` used `kill -0` to wait for the old player to stop. Under a parent that never reaps, it waited the full 10 s and then sent SIGKILL to a process that had already exited. **Fix:** - `install.sh`: a new `pid_alive` reads `State:` from `/proc/PID/status` and counts `Z` as stopped. `restart_players` waits on it. - The test: the old player now runs under a holder process that never reaps it, so the zombie occurs on every host, not only in the container. Two checks are added: that it really became a zombie, and that the stop did not take ≥ 8 s. **Measured on the build host:** - New test against the old `install.sh`: red, `the stop did not wait out its ten seconds on the zombie (13s)`. - With the fix: 20/20, and the stop takes 2 s. - Shell gates, `pre-push-test.sh` and `repo-boundary`: 0. origin/dev is merged in. CI run #349 is going.
Member

CI / cargo test red on c523957: a flake in dev's test code, not this branch. Measured 12:1x UTC.

  • What failed. 1 test of 194 in server --lib (run #349, job 4): pptx_video::tests::fallback_returns_pages_in_page_order, which panicked at crates/server/src/pptx_video.rs:1326 with Failed { tool: "soffice", detail: "Text file busy (os error 26)" }. The other 193 passed.
  • Why it is not this branch. agit diff --stat origin/dev...HEAD lists 6 files: ci.yml, README.md, install.sh, local-gates.sh, package-client-deb.sh and the new shell test. No Rust changed. The test and its fake_program helper came in with GRI-2014 (7e5a777 and earlier).
  • Likely mechanism (not reproduced here). fake_program writes a shell script and then the test executes it. If a parallel test thread forks between that write and its close, the forked child holds a write handle on the script until the child execs. Executing the script during that window fails with ETXTBSY. This is a known race in multithreaded test binaries. The same helper shape is in crates/server/src/pdf.rs:209 and crates/server/tests/pptx_video_acceptance_integration.rs:112, so the race can strike any PR, not only this one.
  • What I could not do. Re-run the one job: the forge gives my account canRerun: false on run #349. I did not push an empty commit to re-trigger it, because that would cost a full run on the single runner and leave a meaningless commit in the PR.

Asks for the chief of staff:

  1. Re-run job cargo test on run #349 (https://git.grimoirix.org/Grimoirix/grimoire-of-digital-signage/actions/runs/349).
  2. Decide whether the flake gets its own issue for the GRI-2014 owner. I have not filed one, following the work order. The fix is to retry the spawn on ETXTBSY in the three fake_program users, or to write the script from a child process, so the test process never holds a write handle on a file it later executes.
**`CI / cargo test` red on c523957: a flake in dev's test code, not this branch.** Measured 12:1x UTC. - **What failed.** 1 test of 194 in `server --lib` (run #349, job 4): `pptx_video::tests::fallback_returns_pages_in_page_order`, which panicked at `crates/server/src/pptx_video.rs:1326` with `Failed { tool: "soffice", detail: "Text file busy (os error 26)" }`. The other 193 passed. - **Why it is not this branch.** `agit diff --stat origin/dev...HEAD` lists 6 files: `ci.yml`, `README.md`, `install.sh`, `local-gates.sh`, `package-client-deb.sh` and the new shell test. No Rust changed. The test and its `fake_program` helper came in with GRI-2014 (`7e5a777` and earlier). - **Likely mechanism (not reproduced here).** `fake_program` writes a shell script and then the test executes it. If a parallel test thread forks between that write and its close, the forked child holds a write handle on the script until the child execs. Executing the script during that window fails with ETXTBSY. This is a known race in multithreaded test binaries. The same helper shape is in `crates/server/src/pdf.rs:209` and `crates/server/tests/pptx_video_acceptance_integration.rs:112`, so the race can strike any PR, not only this one. - **What I could not do.** Re-run the one job: the forge gives my account `canRerun: false` on run #349. I did not push an empty commit to re-trigger it, because that would cost a full run on the single runner and leave a meaningless commit in the PR. **Asks for the chief of staff:** 1. Re-run job `cargo test` on run #349 (https://git.grimoirix.org/Grimoirix/grimoire-of-digital-signage/actions/runs/349). 2. Decide whether the flake gets its own issue for the GRI-2014 owner. I have not filed one, following the work order. The fix is to retry the spawn on ETXTBSY in the three `fake_program` users, or to write the script from a child process, so the test process never holds a write handle on a file it later executes.
Member

12:5x UTC: cargo test on c523957 (run #349) had not been re-run, and the branch was 7 commits behind dev. I merged origin/dev (GRI-2073, #214) and pushed 2ee3f44, which starts a fresh CI run. The diff against dev is still the same 6 non-Rust files (install.sh, ci.yml, README, three scripts), so a cargo test red here would again be dev's code, not this branch's.

Next: fj ci 217. 0 = ask the chief of staff to merge. 1 with the same Text file busy in pptx_video/pdf tests = the GRI-2014 fake_program flake again; I'll report it and leave the flake-issue decision with the chief of staff. 1 with anything else = I fix it.

12:5x UTC: `cargo test` on c523957 (run #349) had not been re-run, and the branch was 7 commits behind dev. I merged origin/dev (GRI-2073, #214) and pushed **2ee3f44**, which starts a fresh CI run. The diff against dev is still the same 6 non-Rust files (install.sh, ci.yml, README, three scripts), so a `cargo test` red here would again be dev's code, not this branch's. Next: `fj ci 217`. 0 = ask the chief of staff to merge. 1 with the same `Text file busy` in `pptx_video`/`pdf` tests = the GRI-2014 `fake_program` flake again; I'll report it and leave the flake-issue decision with the chief of staff. 1 with anything else = I fix it.
Member

Waiting on #220 (the ETXTBSY test flake), per the chief of staff. #217 needs no change. When #220 is in dev I merge dev into the branch once more and push. The chief of staff merges #217 on green.

Waiting on #220 (the ETXTBSY test flake), per the chief of staff. #217 needs no change. When #220 is in `dev` I merge `dev` into the branch once more and push. The chief of staff merges #217 on green.
vivy closed this issue 2026-11-15 15:04:46 +00:00
Sign in to join this conversation.
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Grimoirix/grimoire-of-digital-signage#206
No description provided.