sashatrask
106 lines
4.9 KiB
Markdown
106 lines
4.9 KiB
Markdown
# Extended Live Validation
|
|
|
|
Date: 2026-09-26
|
|
Run: `e6ac8aec-ec20-4ba4-a924-abe5ee95d82c`
|
|
Conclusion: `pass_with_documented_deviations`
|
|
|
|
## Scope
|
|
|
|
The production validation ran one native Windows worker slot and one WSL/Docker
|
|
worker slot through discovery, assignment, execution, progress, diagnostics,
|
|
bundle ingestion, projection, capacity release, and scheduled keycheck. The run
|
|
started at `2026-09-26T00:29:30.500768Z`; terminal server evidence was captured
|
|
at `2026-09-26T03:54:04.573328Z`.
|
|
|
|
This report contains derived counts, classifications, and cryptographic
|
|
identities only. Raw provider values and worker payloads remain in private
|
|
evidence storage.
|
|
|
|
## Outcome
|
|
|
|
- 314 reservations reached terminal outcomes: 309 acknowledged and 5 refunded.
|
|
- All 309 accepted bundles were ingested in one attempt, projected, settled, and
|
|
released; no unresolved reservation or duplicate receipt remained.
|
|
- The accepted cohort covered DockerHub (105), GitLab (102), and Hugging Face
|
|
(107), split across Windows (167) and WSL (147).
|
|
- 3,042 progress events covered all 309 accepted reservations with no duplicate
|
|
reservation sequence.
|
|
- All 348 projection jobs completed and released in one attempt with no error.
|
|
- The sealed terminal cut at controller revision 143 had no live assignment,
|
|
pre-commit bundle, capacity use, publication outbox item, quarantine row,
|
|
waiting lock, or blocker.
|
|
|
|
The 309 scan outcomes were 174 clean, 75 degraded, 56 error, and 4 found. Five
|
|
findings were recorded and none were verified. The run recorded 100 classified
|
|
scan errors, led by 71 download failures; source distribution was GitLab 92,
|
|
DockerHub 7, and Hugging Face 1.
|
|
|
|
## Diagnostics And Keycheck
|
|
|
|
The validation captured 61 worker diagnostics: 52 result errors, 4 stage
|
|
timeouts, 4 runner protocol failures, and 1 client process failure. Fifty-one
|
|
were retryable and ten were nonretryable. The records remain available in the
|
|
private raw bundle for exact-body investigation.
|
|
|
|
The captured keycheck cohort contained 39 candidates. All completed in one
|
|
attempt, released capacity, linked to a result, and projected. Results were 35
|
|
invalid or revoked and 4 no-balance; 37 came from API execution and 2 from
|
|
cache. At the terminal cut the global keycheck queue had 184 completed and no
|
|
pending, leased, or deferred candidate.
|
|
|
|
Nine pending DockerHub discovery retry rows with zero attempts remained as
|
|
expected durable discovery backlog, not as leaked worker-pipeline work.
|
|
|
|
## Evidence Integrity
|
|
|
|
- Server chain: 202 valid records, sequence `-1..199`, with all 575 referenced
|
|
objects (1,245,586,992 bytes) verified.
|
|
- Server NDJSON SHA-256:
|
|
`a6beb1864c540fc5f22b2b647730a39e45dfddb10cf5ceff5e0181eb1a8f0bd8`.
|
|
- Server run SHA-256:
|
|
`5ae58df5f67a8d2a8d4e84d73262a6e7009e03dcbcc9ea176537b384d4e693c3`.
|
|
- Worker chain: 158 valid records, sequence `0..157`, with all 2,136 referenced
|
|
objects (7,733,582,955 bytes) verified.
|
|
- Worker NDJSON SHA-256:
|
|
`03a04a0da5a2db6bd02f2aaed41c191554b135ec287fbc1e2d9998d77da59898`.
|
|
- Worker final payload SHA-256:
|
|
`f652e5f98fb23f9aa2fb4e44e10168a931ea973be616fb09dbdb72001ce961ad`.
|
|
|
|
The machine-readable verification manifest is at
|
|
`build/extended-live-validation/runs/e6ac8aec-ec20-4ba4-a924-abe5ee95d82c/verification-manifest.json`.
|
|
The complete server evidence root is retained in private production storage;
|
|
the local run root contains compact server artifacts and complete worker
|
|
evidence.
|
|
|
|
## Observed Deviations
|
|
|
|
The authenticated `recheck all` operation was accidentally used where only two
|
|
pending candidates should have been rechecked. This caused 35 broad provider
|
|
checks contrary to `KEYCHECK-001`. The command processed all 35, skipped none,
|
|
returned success, produced no failed operation, and all resulting effects
|
|
settled. This is an operator-scope deviation, not an approved workflow change.
|
|
|
|
Six monitor samples encountered transient database statement timeouts. Every
|
|
monitor error recovered, and the evidence chains remained valid. The runtime
|
|
and edge each recorded zero restart during the validation.
|
|
|
|
## Remaining Defects
|
|
|
|
- Windows GitLab filename-too-long checkout recovery exists locally but is not
|
|
deployed.
|
|
- Invalid API-key classification has a local fix that is not deployed.
|
|
- A roughly 20-second WSL clock-domain monotonic failure remains open.
|
|
- Generic `WorkerContractError` diagnostics can lose structured field detail.
|
|
- Monitor aggregate queries can exceed their statement timeout under load.
|
|
|
|
Focused regression coverage for Git checkout recovery, Hugging Face long paths,
|
|
and direct remote credentials passed: 145 tests in 155.26 seconds.
|
|
|
|
## Restore
|
|
|
|
Production was restored with compare-and-swap transitions from revision 143 to
|
|
146: cancel the validation drain, resume discovery, then resume
|
|
dispatch. Final state was dispatch open, discovery open, drain normal, healthy
|
|
core services and producers, and exactly one live assignment on each worker.
|
|
No further validation probe is required.
|