GRI-1965: run the CI test Postgres on tmpfs, with a health start period #170
No reviewers
Labels
No labels
Area/Backend
Area/Database
Area/Docs
Area/Frontend
Area/Infra
Area/Security
Area/Tests
Area/Tooling
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Lang/Python
Lang/Rust
Lang/Shell
Lang/SQL
Lang/TypeScript
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Routing
Auto
Routing
Hold
Routing
Human
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Grimoirix/grimoire-of-digital-signage!170
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/GRI-1965-ci-postgres-tmpfs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
GRI-1965. Only
.github/workflows/ci.ymlchanges, in thetestjob'spostgresservice. No product code changes.What the job logs say
The job logs are readable at
/actions/runs/<n>/jobs/<i>/attempt/1/logswhen you send a token header. I read every redcargo testjob from the last two days. None of them is a branch defect:service [postgres] ... is not healthy. The job ends before its first step and no test runsthe per-test database must be reachable: Connect(PoolTimedOut)The "pull_request red, push green on the same commit" pattern comes from PR #136 (
dev→main). Every push todevstarts two runs of the same commit. The secondcargo testjob starts while the first is still running, and the second one fails. Run 212 is a push run that failed the same way, so the event type is not the cause. Load is. Host I/O pressure wassome avg60=60%, full avg60=50%while CPU pressure was about 3%. Docker's storage is on a DRAM-less SATA SSD, and Postgresinitdband the ~120 per-testCREATE DATABASEcalls are fsync-bound.What I ruled out:
postgresDNS alias: each job has its ownWORKFLOW-*network.system.sliceshowsoom_kill 0since boot.The change
--tmpfs /var/lib/postgresql/data:rw,size=4g. The database is discarded at the end of the job, so durability buys nothing. 4 GB covers about 120 test databases plus their WAL. If the tmpfs fills, the run fails loudly with ENOSPC.--health-start-period 60s. A slow start now counts as "starting" and does not use up the retries.Both decisions can be undone in a day with a revert.
Residual risk
The pool's
acquire_timeoutis product code (crates/server), and this PR leaves it alone. Extreme host load could still slow connections down, but disk was the bottleneck and this change removes the disk from the path.CI run #239 on
6450b3c:CI / cargo testpassed in 6m17s, and the aggregateCI / CIpassed.docker run(20:50:09 → 20:50:33). The red runs before this one never became healthy.PoolTimedOuterrors.cargo deny advisoriesis still red. That is GRI-1872 (unmaintainedproc-macro-errorvia the tauri/gtk stack), not this change.One thing I could not verify: the runner's log does not print the service's
options:, and the container is removed after the job, so I have no direct mount-table proof that the tmpfs was applied. What I have is indirect. Docker accepted the options, because a bad--tmpfsor--health-start-periodwould have faileddocker create. And the start time and the test results are consistent with it.Ready to merge into
dev, by the chief of staff.