truf-server/docs/extended-live-validation-2026-09-26.md

Extended Live Validation

Date: 2026-09-26 Run: e6ac8aec-ec20-4ba4-a924-abe5ee95d82c Conclusion: pass_with_documented_deviations

Scope

The production validation ran one native Windows worker slot and one WSL/Docker worker slot through discovery, assignment, execution, progress, diagnostics, bundle ingestion, projection, capacity release, and scheduled keycheck. The run started at 2026-09-26T00:29:30.500768Z; terminal server evidence was captured at 2026-09-26T03:54:04.573328Z.

This report contains derived counts, classifications, and cryptographic identities only. Raw provider values and worker payloads remain in private evidence storage.

Outcome

• 314 reservations reached terminal outcomes: 309 acknowledged and 5 refunded.
• All 309 accepted bundles were ingested in one attempt, projected, settled, and released; no unresolved reservation or duplicate receipt remained.
• The accepted cohort covered DockerHub (105), GitLab (102), and Hugging Face (107), split across Windows (167) and WSL (147).
• 3,042 progress events covered all 309 accepted reservations with no duplicate reservation sequence.
• All 348 projection jobs completed and released in one attempt with no error.
• The sealed terminal cut at controller revision 143 had no live assignment, pre-commit bundle, capacity use, publication outbox item, quarantine row, waiting lock, or blocker.

The 309 scan outcomes were 174 clean, 75 degraded, 56 error, and 4 found. Five findings were recorded and none were verified. The run recorded 100 classified scan errors, led by 71 download failures; source distribution was GitLab 92, DockerHub 7, and Hugging Face 1.

Diagnostics And Keycheck

The validation captured 61 worker diagnostics: 52 result errors, 4 stage timeouts, 4 runner protocol failures, and 1 client process failure. Fifty-one were retryable and ten were nonretryable. The records remain available in the private raw bundle for exact-body investigation.

The captured keycheck cohort contained 39 candidates. All completed in one attempt, released capacity, linked to a result, and projected. Results were 35 invalid or revoked and 4 no-balance; 37 came from API execution and 2 from cache. At the terminal cut the global keycheck queue had 184 completed and no pending, leased, or deferred candidate.

Nine pending DockerHub discovery retry rows with zero attempts remained as expected durable discovery backlog, not as leaked worker-pipeline work.

Evidence Integrity

• Server chain: 202 valid records, sequence -1..199, with all 575 referenced objects (1,245,586,992 bytes) verified.
• Server NDJSON SHA-256: a6beb1864c540fc5f22b2b647730a39e45dfddb10cf5ceff5e0181eb1a8f0bd8.
• Server run SHA-256: 5ae58df5f67a8d2a8d4e84d73262a6e7009e03dcbcc9ea176537b384d4e693c3.
• Worker chain: 158 valid records, sequence 0..157, with all 2,136 referenced objects (7,733,582,955 bytes) verified.
• Worker NDJSON SHA-256: 03a04a0da5a2db6bd02f2aaed41c191554b135ec287fbc1e2d9998d77da59898.
• Worker final payload SHA-256: f652e5f98fb23f9aa2fb4e44e10168a931ea973be616fb09dbdb72001ce961ad.

The machine-readable verification manifest is at build/extended-live-validation/runs/e6ac8aec-ec20-4ba4-a924-abe5ee95d82c/verification-manifest.json. The complete server evidence root is retained in private production storage; the local run root contains compact server artifacts and complete worker evidence.

Observed Deviations

The authenticated recheck all operation was accidentally used where only two pending candidates should have been rechecked. This caused 35 broad provider checks contrary to KEYCHECK-001. The command processed all 35, skipped none, returned success, produced no failed operation, and all resulting effects settled. This is an operator-scope deviation, not an approved workflow change.

Six monitor samples encountered transient database statement timeouts. Every monitor error recovered, and the evidence chains remained valid. The runtime and edge each recorded zero restart during the validation.

Remaining Defects

• Windows GitLab filename-too-long checkout recovery exists locally but is not deployed.
• Invalid API-key classification has a local fix that is not deployed.
• A roughly 20-second WSL clock-domain monotonic failure remains open.
• Generic WorkerContractError diagnostics can lose structured field detail.
• Monitor aggregate queries can exceed their statement timeout under load.

Focused regression coverage for Git checkout recovery, Hugging Face long paths, and direct remote credentials passed: 145 tests in 155.26 seconds.

Restore

Production was restored with compare-and-swap transitions from revision 143 to 146: cancel the validation drain, resume discovery, then resume dispatch. Final state was dispatch open, discovery open, drain normal, healthy core services and producers, and exactly one live assignment on each worker. No further validation probe is required.