Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,32 @@
+## Why
+
+Public Postman collections and environments are a high-signal source for leaked API credentials because they often preserve request auth settings, headers, variables, and example payloads close to real API usage. The existing scanner already supports multi-source discovery, queues, TruffleHog filesystem scans, token rotation, and observability, so adding Postman can reuse the current architecture while expanding coverage beyond repositories, packages, containers, and HuggingFace Spaces.
+
+## What Changes
+
+- Add a new `postman` source that discovers, queues, scans, and records Postman collection/environment artifacts.
+- Seed Postman targets from GitHub code search using public `*.postman_collection.json` and `*.postman_environment.json` files.
+- Support a one-time backfill mode that scans up to the GitHub Search API result limit per query while filtering out artifacts older than a configured age window.
+- Support a daily tail mode that fetches recently indexed pages and stops early when all targets on consecutive pages are already known.
+- Use the configured GitHub auth pool for Postman discovery, rotating across tokens and sleeping when all tokens are rate-limited.
+- Add durable Postman artifact caching so targets discovered from GitHub, npm, and PyPI can be scanned after temporary extraction directories are removed.
+- Harvest Postman artifacts from npm and PyPI packages during existing package extraction flows and enqueue them into the shared Postman queue.
+- Add Postman-aware result enrichment that classifies credentials using TruffleHog findings plus Postman auth/header/query/body/environment context.
+
+## Capabilities
+
+### New Capabilities
+
+- `postman-source`: Discovery, queueing, scanning, caching, and enrichment for Postman collection and environment artifacts.
+
+### Modified Capabilities
+
+- None.
+
+## Impact
+
+- Affected scanner paths: `app/scanner.py`, `app/console_runner.py`, `app/scanner_db.py`, `app/dashboard.py`, and `app/config.yaml`.
+- Adds runtime files under `runtime/queues/` for `todo_postman.txt` and `checked_postman.txt`.
+- Adds durable artifact storage under a runtime Postman cache directory.
+- Uses existing GitHub auth pools from `secrets.yaml`; no new secret format is required for GitHub discovery.
+- Uses existing TruffleHog filesystem scanning and keychecker follow-up flows; no breaking changes to current sources are expected.