truf-server/app/JSONL_RECONCILIATION.md

Offline JSONL Reconciliation

PostgreSQL is authoritative. JSONL and provider status files are asynchronous, bounded, rebuildable compatibility projections and may lag. Normal keychecks never consume found_secrets.jsonl; reconciliation is explicit offline compatibility work only.

Provider *Checked.txt files are rebuilt from PostgreSQL by the keycheck runner and are not append streams owned by the JSONL projector.

1. Stop the supervisor and acquire the same cluster authority used by migrate_runtime_safety.py.
2. Make an immutable backup of the current file, every numbered segment, the manifest, the publication ledger, and any *.torn-tail.bin file.
3. Validate every retained segment as newline-terminated UTF-8 JSON. Quarantine, rather than concatenate, any final partial record.
4. Do not use keycheck input_state.json as a retention checkpoint. PostgreSQL candidates and current state are authoritative; immutable compatibility generations may be retired by the configured generation limit.
5. For pre-v2 multi-GiB history, do not raise online bounds or backfill it during startup. Preserve it and use a separately reviewed, bounded offline rebuild/import operation.
6. The singleton projector recovers prepared appends by exact generation, offset, length, and SHA-256; partial tails are quarantined and truncated to the prepared offset before retry.
7. Rotation renames the active generation atomically and never copies full history. A deterministic poison job is quarantined individually and later jobs continue.