pptx_video test fails at random with Text file busy (ETXTBSY race in fake_program) #220
Labels
No labels
Area/Backend
Area/Database
Area/Docs
Area/Frontend
Area/Infra
Area/Security
Area/Tests
Area/Tooling
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Lang/Python
Lang/Rust
Lang/Shell
Lang/SQL
Lang/TypeScript
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Routing
Auto
Routing
Hold
Routing
Human
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Grimoirix/grimoire-of-digital-signage#220
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
pptx_video::tests::fallback_returns_pages_in_page_orderfails at random withFailed { tool: "soffice", detail: "Text file busy (os error 26)" }(crates/server/src/pptx_video.rs:1326).It hit PR #217 twice in a row (
c523957and2ee3f44). That PR changes no Rust file, and the same test is green ondev's tip, so the cause is the test, not the PR.Cause.
fake_program(line 812) writes the fake script withstd::fs::write, then the test runs it. Other tests in the same binary spawn processes on other threads at the same time. Afork()from one of those threads copies our open write descriptor into its child, which holds it until itsexec. If ourexecof the fake comes inside that window, Linux refuses it withETXTBSY.O_CLOEXECdoes not close this window. Known Rust issue: rust-lang/rust#114554.Fix shape (pick one, with the reason):
sh -c 'cat > "$1" && chmod 755 "$1"'), so this process never holds a write descriptor that another fork can copy. Removes the race completely, test-only.ETXTBSY(raw os error 26). Smaller change, but it hides the race rather than removing it.Option 1 is my recommendation. Please add a test-loop run (e.g. the test 200 times with
--test-threads=16) in the PR description as evidence.This blocks #217 (#206), which blocks the redeploy to the campus test servers.
Reopened: the same race is in two more test helpers, and it makes
devred now.devf6d98ca(#330, #325) is red oncargo test, on both the push and the pull_request run (18:33 UTC). The PR head passed at 16:16 with the same tree, and #330 changes no file undercrates/server/src/pdf.rs.From
fj log f6d98ca3 --job "cargo test":On this host,
cargo test(no database) onf6d98capasses: 650 passed, 0 failed.Likely cause, not proven:
pdf.rs:209fake_programstill writes the fakepdftoppmwithstd::fs::writeand then executes it. That is the exact pattern #221 removed frompptx_video.rs. The CONVERTER mutex serialises only the pdf tests;video::testsspawn processes on other threads in the same binary at the same moment (see the CI log timestamps). AnETXTBSYturns into a spawn error, notTimeout, so the assertion fails. The assertion does not print the error it got, which is why the log cannot prove it.crates/server/tests/pptx_video_acceptance_integration.rshas the samefs::write+0o755shape.Fix asked:
shpiped over stdin) inpdf.rsand inpptx_video_acceptance_integration.rs. Better: one shared test helper, so a fourth copy cannot appear.assert!(matches!(..), "got {error:?}")), so a next red says what it was.This blocks the deploy of #325. Merge on green, up to date with
dev.Fixed in both pptx_video.rs and pdf.rs: fake_program now writes the fake script via a piped child sh instead of std::fs::write in-process, removing the ETXTBSY race (rust-lang/rust#114554). Error assertions in pdf.rs/pptx_video.rs now print {error:?} on a recurrence. Verified: pptx_video::tests looped 200x and pdf::tests looped 50x at --test-threads=16, 0 failures. PR #221 (pptx_video.rs) merged earlier; PR #332 (pdf.rs + acceptance test) merged into dev at
2a3f58a9, CI green. Closing.