Initial server source import

openspec/changes/add-postman-source/tasks.md

@@ -0,0 +1,69 @@
+## 1. Source Wiring
+
+- [x] 1.1 Add `postman` to CLI `--source` and `--platform` choices and source-to-platform mapping.
+- [x] 1.2 Add `postman` queue file support through the existing `queue_files_for_args`, prepare, and mark-checked flow.
+- [x] 1.3 Add `postman` to supervisor source configuration and dashboard source lists.
+- [x] 1.4 Add disabled-by-default `sources.postman` configuration with GitHub auth pool, search kinds, backfill/tail controls, cache path, and rate-limit settings.
+
+## 2. Target Model And Cache
+
+- [x] 2.1 Define Postman target JSON formats for GitHub code search, npm package, PyPI package, local cache, and future URL targets.
+- [x] 2.2 Implement Postman target parsing and normalization in `console_runner.py` and `scanner_db.py`.
+- [x] 2.3 Implement durable Postman cache path resolution under the configured runtime directory.
+- [x] 2.4 Implement safe cache writes with SHA-256 content hashing, max artifact size checks, and origin metadata preservation.
+
+## 3. GitHub Code Search Discovery
+
+- [x] 3.1 Implement GitHub code search queries for `filename:postman_collection.json <query>` and `filename:postman_environment.json <query>` based on `search_kinds`.
+- [x] 3.2 Implement bounded pagination using configured `pages` and `per_page`, respecting the GitHub 1000-result search cap.
+- [x] 3.3 Convert GitHub code search items into Postman target JSON containing repository, path, SHA, kind, API URL, and HTML URL.
+- [x] 3.4 Implement latest path commit lookup for `max_file_age_days` filtering.
+- [x] 3.5 Integrate existing known-page early stop behavior for Postman tail scans.
+
+## 4. GitHub Token Pool And Rate Limits
+
+- [x] 4.1 Build a source-local GitHub token pool from configured `auth_pool` entries and fallback token settings.
+- [x] 4.2 Rotate tokens per GitHub code search, commit lookup, and content download request.
+- [x] 4.3 Mark only the failing token unavailable on primary rate limit, secondary rate limit, auth invalid, or auth forbidden responses.
+- [x] 4.4 Sleep until earliest known reset time, or configured fallback cooldown, when all GitHub tokens are unavailable.
+- [x] 4.5 Record token cooldown status in the source runtime state without exposing token values in logs or database snapshots.
+
+## 5. Postman Artifact Scanning
+
+- [x] 5.1 Implement Postman content download from GitHub Contents API and cache it before scanning.
+- [x] 5.2 Implement `scan_postman_target()` to stage cached JSON in a temporary directory and run TruffleHog filesystem scanning.
+- [x] 5.3 Add Postman branch to `scan_targets_batch()` and pass timeout, detectors, excluded detectors, and verification flags.
+- [x] 5.4 Preserve nearby file context and apply existing noisy finding filters to Postman scan results.
+- [x] 5.5 Ensure Postman findings, errors, skipped reasons, and clean scans are persisted through existing JSONL and scanner database writes.
+
+## 6. npm And PyPI Harvesting
+
+- [x] 6.1 Add a Postman artifact finder for extracted package directories that matches collection and environment filename patterns.
+- [x] 6.2 Cache npm package Postman artifacts before package temp directory cleanup and attach npm origin metadata.
+- [x] 6.3 Cache PyPI package Postman artifacts before package temp directory cleanup and attach PyPI origin metadata.
+- [x] 6.4 Enqueue harvested package artifacts into `todo_postman.txt` after package scan batches without failing the original package scan.
+- [x] 6.5 Deduplicate harvested package artifacts by content hash before enqueueing.
+
+## 7. Postman-Aware Enrichment
+
+- [x] 7.1 Parse Postman collection and environment JSON into request, auth, header, query, body, and variable context maps.
+- [x] 7.2 Correlate TruffleHog finding locations or nearby context with Postman context maps.
+- [x] 7.3 Classify credential kind and provider using DetectorName, value shape, auth/header type, variable name, and endpoint host.
+- [x] 7.4 Detect common placeholders and assign placeholder or low-confidence classification.
+- [x] 7.5 Persist enrichment fields using existing finding enrichment/database columns where possible.
+
+## 8. Observability And Configuration
+
+- [x] 8.1 Add Postman source cycle metrics, queue snapshots, target scan records, findings, and errors to existing database flows.
+- [x] 8.2 Add Postman queue counts to dashboard current queues and source health views.
+- [x] 8.3 Add redaction coverage for Postman/GitHub auth pool settings in config snapshots and logs.
+- [x] 8.4 Add backfill-friendly and tail-friendly config examples in `config.yaml` comments.
+
+## 9. Verification
+
+- [x] 9.1 Run a small Postman GitHub discovery cycle with `pages: 1`, `per_page: 10`, and `max_targets` set.
+- [x] 9.2 Re-run the same cycle and verify duplicate targets are skipped through `todo_postman.txt` and `checked_postman.txt`.
+- [x] 9.3 Verify all-token rate-limit fallback with a simulated or controlled token-unavailable state.
+- [x] 9.4 Verify npm and PyPI harvesting using a package fixture containing collection and environment JSON files.
+- [x] 9.5 Verify database and dashboard visibility for Postman source cycles, queues, target scans, findings, and errors.
+- [x] 9.6 Run `openspec status --change add-postman-source` and ensure all implementation tasks are complete before archive.