sashatrask
103 lines
5.3 KiB
Markdown
103 lines
5.3 KiB
Markdown
# Pipeline Quarantine Audit
|
|
|
|
Initial snapshot and remediation: `2026-08-15`
|
|
|
|
## Current Impact
|
|
|
|
- PostgreSQL quarantine rows: `177`
|
|
- Accounted capacity: `4354 items / 1,817,503,389 bytes`
|
|
- Configured admission limit: `10000 items / 1,073,741,824 bytes`
|
|
- New scan admission is closed because the byte limit is exceeded.
|
|
- `49,215` admission intents have already ended with `quarantine_admission_closed`.
|
|
|
|
`pipeline: ready` means that PostgreSQL and workers are healthy. It does not mean that new scan admission is open.
|
|
|
|
## Result Bundles
|
|
|
|
Two rows account for `4004 items / 1,212,153,856 bytes`.
|
|
|
|
| Quarantine ID | Source | Original error | Physical size | Read-only validation now |
|
|
|---|---|---|---:|---|
|
|
| `29` | Hugging Face `spaces` | Transient Windows `Permission denied` | `2,870 B` | Valid; 3 frames, no findings/errors/candidates |
|
|
| `91` | Docker Hub query `tokenizer` | Transient Windows `Permission denied` | `5,354 B` | Valid; 8 frames, 1 finding, 4 errors, no candidates |
|
|
|
|
The bundle contents are valid. Their large capacity cost comes from worst-case reservations transferred into quarantine, not their physical file sizes.
|
|
|
|
Current code now reports a temporarily unavailable private bundle as an availability error. Result ingester defers it instead of classifying it as invalid content.
|
|
|
|
Recommendation: recover both bundles through an audited offline path rather than discard them. ID `91` contains one finding and must not be deleted without an explicit decision.
|
|
|
|
## Keycheck Quarantine
|
|
|
|
There are `175` pending keycheck quarantine rows.
|
|
|
|
| Class | Rows | Distinct credentials | Assessment |
|
|
|---|---:|---:|---|
|
|
| Repeated DeepSeek unconsumed rechecks | `104` | `1` | Duplicate hourly retries; current state is now `NO_CONTEXT` |
|
|
| DeepSeek unconsumed findings | `15` | `6` | Legitimate historical candidates filtered by routing |
|
|
| Azure Foundry unconsumed | `15` | `12` | Historical provider-consumption issue |
|
|
| Hugging Face unconsumed | `7` | `5` | Historical provider-consumption issue |
|
|
| Replicate unconsumed | `6` | `3` | Historical provider-consumption issue |
|
|
| xAI unconsumed | `3` | `3` | Historical provider-consumption issue |
|
|
| DeepSeek route mismatch | `24` | `14` | Misrouted non-DeepSeek detectors; source findings remain in PostgreSQL |
|
|
| Azure route mismatch | `1` | `1` | Historical Azure Foundry route mismatch; current state is `UNKNOWN` |
|
|
|
|
The active growth came from one DeepSeek credential whose current state was `NETWORK`. Hourly network retry created a fresh candidate, provider routing silently rejected it, and the candidate entered quarantine after three unconsumed attempts.
|
|
|
|
## Preventive Changes
|
|
|
|
- Non-DeepSeek routing decisions now complete as `NO_CONTEXT` instead of leaving a leased candidate unconsumed.
|
|
- DeepSeek has a canonical `deepseekNoContext.txt` status projection.
|
|
- Recheck generation now refuses to enqueue a credential while it has an unresolved `provider_candidate_unconsumed` or `candidate_provider_route_mismatch` quarantine row.
|
|
- Temporarily unavailable result bundle files are deferred instead of quarantined as validation failures.
|
|
|
|
Verification:
|
|
|
|
- Targeted tests: `60 passed`.
|
|
- Manual `recheck deepseek network`: code `0`, no candidates processed.
|
|
- DeepSeek unconsumed quarantine remained exactly `119`; no new row was created.
|
|
|
|
## Approved Remediation
|
|
|
|
1. Stop sources and acquire the offline migration guard.
|
|
2. Recover bundle IDs `29` and `91` through deterministic re-ingestion.
|
|
3. Discard the `104` duplicate DeepSeek retry rows after exact manifest review.
|
|
4. Discard the `24` confirmed DeepSeek route-mismatch rows after exact manifest review.
|
|
5. Requeue one current candidate per distinct credential from the remaining legitimate unconsumed groups; keep source attribution.
|
|
6. Review the single Azure route mismatch separately.
|
|
7. Resolve superseded duplicate candidate rows with an audited discard manifest.
|
|
8. Verify physical artifacts, capacity accounting, projections and reopened scan admission before restarting sources.
|
|
|
|
All destructive decisions must use an exact private `truf-pipeline-quarantine-review-v1` manifest containing quarantine ID, reason code, payload hash and action. The offline review command requires both `--apply` and `--sources-stopped`.
|
|
|
|
## Applied Result
|
|
|
|
The user approved recovery plus selective cleanup.
|
|
|
|
- Manifest: `runtime/control/quarantine-remediation-20260815.json`
|
|
- Manifest SHA-256: `c3143e6b0e15e07b6afacffd50b449444b9932e75001f633ac82b5b79e21fe3f`
|
|
- Reviewed: `177`
|
|
- Approved retry: `32`
|
|
- Audited discard: `145`
|
|
- Duplicate/conflicting reviews: `0`
|
|
- Final quarantine capacity: `0 items / 0 bytes`
|
|
|
|
Bundle outcomes:
|
|
|
|
| Quarantine ID | Final reservation | Final bundle | Target disposition | Preserved contents |
|
|
|---|---|---|---|---|
|
|
| `29` | `acknowledged` | `acknowledged` | `done` | Clean empty result |
|
|
| `91` | `acknowledged` | `acknowledged` | `deferred` | `1 finding`, `4 errors` |
|
|
|
|
Keycheck outcomes from the retried unique credentials:
|
|
|
|
| Service | Final current statuses |
|
|
|---|---|
|
|
| Azure | `12 FOUNDRY_UNRESOLVED`, `1 UNKNOWN` |
|
|
| DeepSeek | `6 NO_CONTEXT` |
|
|
| Hugging Face | `5 NO_CONTEXT` |
|
|
| Replicate | `3 NO_CONTEXT` |
|
|
| xAI | `3 NO_CONTEXT` |
|
|
|
|
The malformed legacy provider candidates are now terminal current-state records instead of repeatedly deferred/quarantined candidates. Scan admission reopened and scanner workers returned to `3/3` active operation.
|