GRI-1965: run the CI test Postgres on tmpfs, with a health start period #170

Merged
vivy merged 1 commit from fix/GRI-1965-ci-postgres-tmpfs into dev 2026-11-13 21:23:57 +00:00
Member

GRI-1965. Only .github/workflows/ci.yml changes, in the test job's postgres service. No product code changes.

What the job logs say

The job logs are readable at /actions/runs/<n>/jobs/<i>/attempt/1/logs when you send a token header. I read every red cargo test job from the last two days. None of them is a branch defect:

shape runs what the log says
service never healthy 188, 191, 197, 212, 215, 220, 227, 229 service [postgres] ... is not healthy. The job ends before its first step and no test runs
service too slow 217, 231, 232 the per-test database must be reachable: Connect(PoolTimedOut)

The "pull_request red, push green on the same commit" pattern comes from PR #136 (dev → main). Every push to dev starts two runs of the same commit. The second cargo test job starts while the first is still running, and the second one fails. Run 212 is a push run that failed the same way, so the event type is not the cause. Load is. Host I/O pressure was some avg60=60%, full avg60=50% while CPU pressure was about 3%. Docker's storage is on a DRAM-less SATA SSD, and Postgres initdb and the ~120 per-test CREATE DATABASE calls are fsync-bound.

What I ruled out:

  • A shared Docker network or postgres DNS alias: each job has its own WORKFLOW-* network.
  • OOM kills: system.slice shows oom_kill 0 since boot.
  • A rust-cache key race: the job dies before the cache step runs.

The change

  • --tmpfs /var/lib/postgresql/data:rw,size=4g. The database is discarded at the end of the job, so durability buys nothing. 4 GB covers about 120 test databases plus their WAL. If the tmpfs fills, the run fails loudly with ENOSPC.
  • --health-start-period 60s. A slow start now counts as "starting" and does not use up the retries.

Both decisions can be undone in a day with a revert.

Residual risk

The pool's acquire_timeout is product code (crates/server), and this PR leaves it alone. Extreme host load could still slow connections down, but disk was the bottleneck and this change removes the disk from the path.

GRI-1965. Only `.github/workflows/ci.yml` changes, in the `test` job's `postgres` service. No product code changes. ## What the job logs say The job logs are readable at `/actions/runs/<n>/jobs/<i>/attempt/1/logs` when you send a token header. I read every red `cargo test` job from the last two days. None of them is a branch defect: | shape | runs | what the log says | | --- | --- | --- | | service never healthy | 188, 191, 197, 212, 215, 220, 227, 229 | `service [postgres] ... is not healthy`. The job ends before its first step and no test runs | | service too slow | 217, 231, 232 | `the per-test database must be reachable: Connect(PoolTimedOut)` | The "pull_request red, push green on the same commit" pattern comes from PR #136 (`dev` → `main`). Every push to `dev` starts two runs of the same commit. The second `cargo test` job starts while the first is still running, and the second one fails. Run 212 is a **push** run that failed the same way, so the event type is not the cause. Load is. Host I/O pressure was `some avg60=60%, full avg60=50%` while CPU pressure was about 3%. Docker's storage is on a DRAM-less SATA SSD, and Postgres `initdb` and the ~120 per-test `CREATE DATABASE` calls are fsync-bound. What I ruled out: - **A shared Docker network or `postgres` DNS alias**: each job has its own `WORKFLOW-*` network. - **OOM kills**: `system.slice` shows `oom_kill 0` since boot. - **A rust-cache key race**: the job dies before the cache step runs. ## The change - `--tmpfs /var/lib/postgresql/data:rw,size=4g`. The database is discarded at the end of the job, so durability buys nothing. 4 GB covers about 120 test databases plus their WAL. If the tmpfs fills, the run fails loudly with ENOSPC. - `--health-start-period 60s`. A slow start now counts as "starting" and does not use up the retries. Both decisions can be undone in a day with a revert. ## Residual risk The pool's `acquire_timeout` is product code (`crates/server`), and this PR leaves it alone. Extreme host load could still slow connections down, but disk was the bottleneck and this change removes the disk from the path.
GRI-1965: run the CI test Postgres on tmpfs, with a health start period
Some checks failed
CI / the pre-push hook (pull_request) Successful in 42s
CI / cargo fmt (pull_request) Successful in 38s
CI / cargo deny (pull_request) Successful in 3m26s
CI / cargo clippy (pull_request) Successful in 5m5s
CI / cargo deny advisories (pull_request) Failing after 2m7s
CI / cargo test (pull_request) Successful in 6m17s
CI / repository boundary (pull_request) Successful in 38s
CI / frontend-web (pull_request) Successful in 1m54s
CI / Notify gate failed (pull_request) Has been skipped
CI / CI (pull_request) Successful in 41s
6450b3cd8d
The cargo test gate went red on a loaded host with no change to the code.
The job logs show two shapes, both the service Postgres losing on fsync
against the shared SSD: the service never became healthy and the job died
before its first step, or it came up too slowly and the per-test
CREATE DATABASE hit PoolTimedOut. The database is thrown away with the job,
so its data directory goes on tmpfs.
Author
Member

CI run #239 on 6450b3c: CI / cargo test passed in 6m17s, and the aggregate CI / CI passed.

  • The Postgres service was healthy 24s after docker run (20:50:09 → 20:50:33). The red runs before this one never became healthy.
  • All 19 test binaries passed, including every DB-backed integration suite (auth, devices, media, playlist, ws, setup_session and the others). No PoolTimedOut errors.
  • cargo deny advisories is still red. That is GRI-1872 (unmaintained proc-macro-error via the tauri/gtk stack), not this change.

One thing I could not verify: the runner's log does not print the service's options:, and the container is removed after the job, so I have no direct mount-table proof that the tmpfs was applied. What I have is indirect. Docker accepted the options, because a bad --tmpfs or --health-start-period would have failed docker create. And the start time and the test results are consistent with it.

Ready to merge into dev, by the chief of staff.

CI run #239 on 6450b3c: **`CI / cargo test` passed in 6m17s**, and the aggregate `CI / CI` passed. - The Postgres service was healthy 24s after `docker run` (20:50:09 → 20:50:33). The red runs before this one never became healthy. - All 19 test binaries passed, including every DB-backed integration suite (auth, devices, media, playlist, ws, setup_session and the others). No `PoolTimedOut` errors. - `cargo deny advisories` is still red. That is GRI-1872 (unmaintained `proc-macro-error` via the tauri/gtk stack), not this change. One thing I could not verify: the runner's log does not print the service's `options:`, and the container is removed after the job, so I have no direct mount-table proof that the tmpfs was applied. What I have is indirect. Docker accepted the options, because a bad `--tmpfs` or `--health-start-period` would have failed `docker create`. And the start time and the test results are consistent with it. Ready to merge into `dev`, by the chief of staff.
vivy merged commit 1bcd4acbf7 into dev 2026-11-13 21:23:57 +00:00
vivy deleted branch fix/GRI-1965-ci-postgres-tmpfs 2026-11-13 21:23:57 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Grimoirix/grimoire-of-digital-signage!170
No description provided.