pptx_video test fails at random with Text file busy (ETXTBSY race in fake_program) #220

Closed
opened 2026-11-15 12:58:46 +00:00 by vivy · 2 comments
Owner

pptx_video::tests::fallback_returns_pages_in_page_order fails at random with Failed { tool: "soffice", detail: "Text file busy (os error 26)" } (crates/server/src/pptx_video.rs:1326).

It hit PR #217 twice in a row (c523957 and 2ee3f44). That PR changes no Rust file, and the same test is green on dev's tip, so the cause is the test, not the PR.

Cause. fake_program (line 812) writes the fake script with std::fs::write, then the test runs it. Other tests in the same binary spawn processes on other threads at the same time. A fork() from one of those threads copies our open write descriptor into its child, which holds it until its exec. If our exec of the fake comes inside that window, Linux refuses it with ETXTBSY. O_CLOEXEC does not close this window. Known Rust issue: rust-lang/rust#114554.

Fix shape (pick one, with the reason):

  1. Write the fake from a child process (sh -c 'cat > "$1" && chmod 755 "$1"'), so this process never holds a write descriptor that another fork can copy. Removes the race completely, test-only.
  2. Retry the spawn a few times on ETXTBSY (raw os error 26). Smaller change, but it hides the race rather than removing it.

Option 1 is my recommendation. Please add a test-loop run (e.g. the test 200 times with --test-threads=16) in the PR description as evidence.

This blocks #217 (#206), which blocks the redeploy to the campus test servers.

`pptx_video::tests::fallback_returns_pages_in_page_order` fails at random with `Failed { tool: "soffice", detail: "Text file busy (os error 26)" }` (`crates/server/src/pptx_video.rs:1326`). It hit PR #217 twice in a row (c523957 and 2ee3f44). That PR changes no Rust file, and the same test is green on `dev`'s tip, so the cause is the test, not the PR. **Cause.** `fake_program` (line 812) writes the fake script with `std::fs::write`, then the test runs it. Other tests in the same binary spawn processes on other threads at the same time. A `fork()` from one of those threads copies our open write descriptor into its child, which holds it until its `exec`. If our `exec` of the fake comes inside that window, Linux refuses it with `ETXTBSY`. `O_CLOEXEC` does not close this window. Known Rust issue: rust-lang/rust#114554. **Fix shape (pick one, with the reason):** 1. Write the fake from a child process (`sh -c 'cat > "$1" && chmod 755 "$1"'`), so this process never holds a write descriptor that another fork can copy. Removes the race completely, test-only. 2. Retry the spawn a few times on `ETXTBSY` (raw os error 26). Smaller change, but it hides the race rather than removing it. Option 1 is my recommendation. Please add a test-loop run (e.g. the test 200 times with `--test-threads=16`) in the PR description as evidence. This blocks #217 (#206), which blocks the redeploy to the campus test servers.
vivy closed this issue 2026-11-15 14:26:04 +00:00
Author
Owner

Reopened: the same race is in two more test helpers, and it makes dev red now.

dev f6d98ca (#330, #325) is red on cargo test, on both the push and the pull_request run (18:33 UTC). The PR head passed at 16:16 with the same tree, and #330 changes no file under crates/server/src/pdf.rs.

From fj log f6d98ca3 --job "cargo test":

pull_request run: pdf::tests::times_out_when_the_converter_hangs  FAILED  (pdf.rs:325, assertion matches!(error, PdfError::Timeout(_)))
push run:         pdf::tests::times_out_when_the_converter_hangs  FAILED
                  pdf::tests::reports_a_failing_converter         FAILED

On this host, cargo test (no database) on f6d98ca passes: 650 passed, 0 failed.

Likely cause, not proven: pdf.rs:209 fake_program still writes the fake pdftoppm with std::fs::write and then executes it. That is the exact pattern #221 removed from pptx_video.rs. The CONVERTER mutex serialises only the pdf tests; video::tests spawn processes on other threads in the same binary at the same moment (see the CI log timestamps). An ETXTBSY turns into a spawn error, not Timeout, so the assertion fails. The assertion does not print the error it got, which is why the log cannot prove it.

crates/server/tests/pptx_video_acceptance_integration.rs has the same fs::write + 0o755 shape.

Fix asked:

  1. Use the #221 writer (child sh piped over stdin) in pdf.rs and in pptx_video_acceptance_integration.rs. Better: one shared test helper, so a fourth copy cannot appear.
  2. Make the assertions print the error they got (assert!(matches!(..), "got {error:?}")), so a next red says what it was.

This blocks the deploy of #325. Merge on green, up to date with dev.

**Reopened: the same race is in two more test helpers, and it makes `dev` red now.** `dev` `f6d98ca` (#330, #325) is red on `cargo test`, on both the push and the pull_request run (18:33 UTC). The PR head passed at 16:16 with the same tree, and #330 changes no file under `crates/server/src/pdf.rs`. From `fj log f6d98ca3 --job "cargo test"`: ``` pull_request run: pdf::tests::times_out_when_the_converter_hangs FAILED (pdf.rs:325, assertion matches!(error, PdfError::Timeout(_))) push run: pdf::tests::times_out_when_the_converter_hangs FAILED pdf::tests::reports_a_failing_converter FAILED ``` On this host, `cargo test` (no database) on `f6d98ca` passes: 650 passed, 0 failed. **Likely cause, not proven:** `pdf.rs:209` `fake_program` still writes the fake `pdftoppm` with `std::fs::write` and then executes it. That is the exact pattern #221 removed from `pptx_video.rs`. The CONVERTER mutex serialises only the pdf tests; `video::tests` spawn processes on other threads in the same binary at the same moment (see the CI log timestamps). An `ETXTBSY` turns into a spawn error, not `Timeout`, so the assertion fails. The assertion does not print the error it got, which is why the log cannot prove it. `crates/server/tests/pptx_video_acceptance_integration.rs` has the same `fs::write` + `0o755` shape. **Fix asked:** 1. Use the #221 writer (child `sh` piped over stdin) in `pdf.rs` and in `pptx_video_acceptance_integration.rs`. Better: one shared test helper, so a fourth copy cannot appear. 2. Make the assertions print the error they got (`assert!(matches!(..), "got {error:?}")`), so a next red says what it was. This blocks the deploy of #325. Merge on green, up to date with `dev`.
vivy reopened this issue 2026-11-18 21:38:59 +00:00
Member

Fixed in both pptx_video.rs and pdf.rs: fake_program now writes the fake script via a piped child sh instead of std::fs::write in-process, removing the ETXTBSY race (rust-lang/rust#114554). Error assertions in pdf.rs/pptx_video.rs now print {error:?} on a recurrence. Verified: pptx_video::tests looped 200x and pdf::tests looped 50x at --test-threads=16, 0 failures. PR #221 (pptx_video.rs) merged earlier; PR #332 (pdf.rs + acceptance test) merged into dev at 2a3f58a9, CI green. Closing.

Fixed in both pptx_video.rs and pdf.rs: fake_program now writes the fake script via a piped child sh instead of std::fs::write in-process, removing the ETXTBSY race (rust-lang/rust#114554). Error assertions in pdf.rs/pptx_video.rs now print {error:?} on a recurrence. Verified: pptx_video::tests looped 200x and pdf::tests looped 50x at --test-threads=16, 0 failures. PR #221 (pptx_video.rs) merged earlier; PR #332 (pdf.rs + acceptance test) merged into dev at 2a3f58a9, CI green. Closing.
Sign in to join this conversation.
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Grimoirix/grimoire-of-digital-signage#220
No description provided.