sashatrask
33 lines
2.3 KiB
Markdown
33 lines
2.3 KiB
Markdown
## Why
|
|
|
|
Public Postman collections and environments are a high-signal source for leaked API credentials because they often preserve request auth settings, headers, variables, and example payloads close to real API usage. The existing scanner already supports multi-source discovery, queues, TruffleHog filesystem scans, token rotation, and observability, so adding Postman can reuse the current architecture while expanding coverage beyond repositories, packages, containers, and HuggingFace Spaces.
|
|
|
|
## What Changes
|
|
|
|
- Add a new `postman` source that discovers, queues, scans, and records Postman collection/environment artifacts.
|
|
- Seed Postman targets from GitHub code search using public `*.postman_collection.json` and `*.postman_environment.json` files.
|
|
- Support a one-time backfill mode that scans up to the GitHub Search API result limit per query while filtering out artifacts older than a configured age window.
|
|
- Support a daily tail mode that fetches recently indexed pages and stops early when all targets on consecutive pages are already known.
|
|
- Use the configured GitHub auth pool for Postman discovery, rotating across tokens and sleeping when all tokens are rate-limited.
|
|
- Add durable Postman artifact caching so targets discovered from GitHub, npm, and PyPI can be scanned after temporary extraction directories are removed.
|
|
- Harvest Postman artifacts from npm and PyPI packages during existing package extraction flows and enqueue them into the shared Postman queue.
|
|
- Add Postman-aware result enrichment that classifies credentials using TruffleHog findings plus Postman auth/header/query/body/environment context.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
|
|
- `postman-source`: Discovery, queueing, scanning, caching, and enrichment for Postman collection and environment artifacts.
|
|
|
|
### Modified Capabilities
|
|
|
|
- None.
|
|
|
|
## Impact
|
|
|
|
- Affected scanner paths: `app/scanner.py`, `app/console_runner.py`, `app/scanner_db.py`, `app/dashboard.py`, and `app/config.yaml`.
|
|
- Adds runtime files under `runtime/queues/` for `todo_postman.txt` and `checked_postman.txt`.
|
|
- Adds durable artifact storage under a runtime Postman cache directory.
|
|
- Uses existing GitHub auth pools from `secrets.yaml`; no new secret format is required for GitHub discovery.
|
|
- Uses existing TruffleHog filesystem scanning and keychecker follow-up flows; no breaking changes to current sources are expected.
|