truf-server/openspec/changes/rescan-updated-core-targets/design.md

Context

GitHub, GitLab, and HuggingFace discovery return stable target identities together with remote update timestamps. The runner currently discards those timestamps and the PostgreSQL queue permanently deduplicates targets by (source, normalized_target). A completed repository or Space is therefore never scanned again even when its content changes.

Production discovery is already saturated: less than one percent of fetched records are new target identities and the core queue is often empty. The change must restore changed-content coverage without turning every rediscovery into a rescan, growing one queue row per revision, or creating an unbounded backlog.

Goals / Non-Goals

Goals:

• Preserve a bounded remote content-update signal for GitHub, GitLab, and HuggingFace targets.
• Rescan a completed target only when discovery observes content newer than the revision covered by its latest claim.
• Coalesce multiple remote updates into one mutable queue row and one pending follow-up.
• Bound changed-target admission and expose it separately from new-target admission.
• Roll out without treating every legacy row as changed.

Non-Goals:

• Requeue terminal failures or actively leased/deferred targets.
• Rescan every known target on a timer.
• Change DockerHub's digest-based identity and refresh behavior.
• Guarantee that provider activity timestamps always represent content changes.
• Enable inactive historical sources or enlarge global scan concurrency.

Decisions

Carry one normalized discovery record

The runner will represent eligible discovery results internally as a target plus an optional UTC remote_modified_at. GitHub uses pushed_at, GitLab uses last_activity_at, and HuggingFace uses lastModified. Missing, malformed, or non-monotonic timestamps remain valid discovery results but cannot trigger an updated-target rescan.

HuggingFace discovery will use a newest-modified feed so old Spaces changed recently are observable. Identity-only known-page stopping will be disabled when updated-target rescans are enabled; hard page and result limits remain the discovery bound.

Alternatives rejected:

• Repository updated_at on GitHub, because metadata-only edits are not content pushes.
• A HEAD-SHA request per repository, because it multiplies API traffic and rate-limit exposure.
• Revision-aware early stopping, because a known first page does not prove later pages contain no changed targets.

Keep one queue row and two remote timestamps

target_queue will gain nullable remote_modified_at and scan_remote_modified_at columns. Discovery monotonically advances remote_modified_at. Claiming atomically copies the currently observed value into scan_remote_modified_at, recording what that scan covers.

The separate claim snapshot is required because a remote update can arrive while a scan is running. On completion, a newer observed timestamp remains ahead of the claimed timestamp and is eligible for exactly one later scan. Encoding revisions into normalized_target was rejected because it would grow queue rows and weaken queue authority.

For a legacy completed row with no claim snapshot, completed_at is the rollout baseline. It is eligible only when the first valid remote timestamp observed is newer than that completion. This prevents a migration surge while still admitting updates that occurred after the historical scan.

Observe and requeue atomically under a hard budget

A PostgreSQL transaction will upsert discovery observations and requeue at most updated_rescan_max_per_cycle eligible rows for one source. Eligibility requires:

• status='done';
• a strictly newer observed timestamp than scan_remote_modified_at, or than completed_at for a legacy row;
• elapsed updated_rescan_cooldown_seconds since completion;
• no lease, reservation, claim, or resolver authority.

Eligible rows are locked with FOR UPDATE SKIP LOCKED. Requeue resets only retry/completion scheduling fields required for a normal pending claim. Failed, pending, deferred, in-progress, unresolved, unchanged, and invalid-timestamp rows are never promoted by this path.

The initial production setting is one updated target per source cycle. Existing backlog-first behavior remains enabled, so a source drains its admitted work before discovery can admit more; refresh_registry is not enabled by this change.

Alternatives rejected:

• Global requeue_done=True, because it requeues unchanged rows on every cycle.
• Comparing only remote time with local completion time forever, because provider and host clocks differ and a mid-scan update can be lost.
• A separate maintenance queue, because it duplicates existing lease, reservation, and completion authority.

Account for updated targets separately

source_cycles will gain queued_updated_count. queued_new_count keeps its current meaning. Source logs and dashboard aggregation will report changed-target admissions separately so rollout volume and yield can be audited.

Risks / Trade-offs

• [GitLab activity can change without a repository push] -> Use a one-target-per-cycle cap and cooldown; report updated admissions separately.
• [Provider clock skew] -> Require strict monotonicity and use the claimed remote timestamp after the first revision-aware scan.
• [More discovery API traffic after disabling identity-only early stop] -> Keep existing hard page/per-page limits and source intervals.
• [Changed targets consume capacity without useful findings] -> Start at one per cycle and compare updated-target yield before increasing the cap.
• [Schema rollout while runtime is active] -> Stop the authority-managed runtime, apply schema through the normal initialization path, run tests, then restart through start_runtime.ps1.
• [Rollback leaves nullable columns] -> Disable the feature in source configuration; nullable columns and metrics are backward-compatible and can remain.

Migration Plan

1. Add nullable queue columns, the source-cycle counter, and a partial eligibility index through idempotent schema initialization.
2. Deploy code and tests with updated-target rescans disabled by default.
3. Enable GitHub, GitLab, and HuggingFace with a cap of one and a conservative cooldown; leave DockerHub unchanged.
4. Restart the authority-managed runtime and verify discovery, queue authority, projection, and source health.
5. Observe changed-target admission, completion, findings, and worker occupancy before changing any cap.

Rollback is configuration-first: disable updated-target rescans and restart the managed runtime. Existing pending work completes under normal queue semantics; no destructive data migration is required.

Open Questions

• Whether production evidence supports different cooldowns per source after the initial canary.
• Whether a later change should add provider-specific immutable revisions when APIs can supply them without extra requests.