truf-server/docs/worker-parallelism-validation-2026-09-23.md

Worker Parallelism Validation, 2026-09-23

Scope

This report records the production validation performed on sec using the local Windows computer:

• bounded discovery for GitLab, DockerHub, and HuggingFace;
• a native Windows protocol-2 worker with client and server parallelism 3;
• a dual-worker run with native Windows parallelism 1 and the existing trusted WSL production worker at cap 1;
• authoritative reconciliation from discovery through queue, reservation, receipt, bundle ingestion, scan settlement, projection, and JSONL append;
• restoration of production source/capacity settings and normal controls.

No provider credential, device token, raw target, raw finding, runtime YAML, or worker command line is included in this report.

Result

The core validation passed.

• Native Windows reached exact unfinished concurrency 3 and never exceeded 3.
• The dedicated Windows cohort accounted for 180 accepted assignments across all three sources.
• Windows and WSL independently reached concurrency 1 at the same time, for exact combined concurrency 2, and never exceeded their individual caps.
• All accepted results were ingested, queue-settled, projected, and appended as required.
• Final cohort and global lineage checks reported zero violations.
• No pipeline quarantine or failed hold was created.
• Production controls and the WSL worker were restored.

The originally requested 10-hour observation was not completed continuously. After successful active parallelism-3 work, 14,105 seconds (about 3 hours 55 minutes) of detached idle stability observation completed before the user changed the objective to the dual-worker test. The active concurrency evidence itself is complete; the residual limit is soak duration, not functional coverage.

Baseline

The fresh pre-test snapshot was stored root-only on sec:

• Path: /opt/truf-remote-server/staging/windows-p3-before2.json
• SHA-256: 469bcd69888cacce55523fef17508ee48191c4c879a6d39f2251abad4aa0efdb
• Active config SHA-256: e48797689ab0ee7328d21cf5c23d0ffedb979dba492bf49d1da3afe4b075cf5b
• Config bytes: 37,233
• Controls: revision 56, discovery open, dispatch open, drain normal
• Existing blocker: one stale production assignment, allowed to settle naturally
• Runtime image: sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf
• Failed hold: absent
• All recorded orphan/mismatch invariants: zero

Baseline high-water values included reservation 929, scan 926, projection job 926, projection append 984, error 1919, finding 127, and zero quarantine rows.

The existing WSL assignment was never canceled or directly mutated. Dispatch was paused and the lease was allowed to resolve before changing worker caps.

Windows Package

The trusted run artifact was built from pinned local caches:

• Package directory: build/worker-parallelism3-20260922/package-fixed1
• Archive: build/worker-parallelism3-20260922/truf-worker-windows-x86_64-fixed1.zip
• Archive SHA-256: 6ca460a47d31606c430e8c6f081443530892332cfc183fc0c6edfe7568a454cc
• Manifest file SHA-256: cd55820e0a04a04ded9ba340b9c6e5c6259422aab80989180a6ba28ff3b77103
• Canonical code-manifest SHA-256: e981da19283aad5971fc2865268bb8151b3540d71fbd057ae56d9d2379993374
• Platform: windows-x86_64
• Protocol and bundle schema: v2
• Capabilities: GitLab exact_git_v1, DockerHub docker_direct_v1, HuggingFace huggingface_space_v1

The registered Windows v2 manifest was stale relative to the current production authority files. It was not overwritten. The fixed package was registered as the new root-owned windows-worker-package-v3.json trust authority.

Reproducibility defect

Two initial package builds differed only in four timestamp bytes in each pip-generated Windows launcher executable and the corresponding RECORD hashes. The package builder now normalizes the embedded launcher ZIP timestamps to the DOS epoch and recomputes the RECORD rows.

Changed local source:

• app/worker_package_builder.py
• tests/test_worker_package.py

Verification:

• Two post-fix archives were byte-identical.
• python -m pytest tests/test_worker_package.py -q: 16 passed.

Discovery

The temporary discovery candidate was derived from the exact active config.

GitLab used 16 reviewed keyword queries, one API page, one result per page, and one target maximum per query. DockerHub used 16 reviewed keyword queries, one repository result per query, and one image per repository.

HuggingFace was deliberately reported differently: the configured implementation does not perform keyword search. It requests the newest-modified Spaces with a fixed API limit of 100. pages: 1 therefore means one real recent page, not one keyword result. Repeated polls occurred while GitLab and DockerHub rotated their queries; deduplication bounded the admitted rows.

Final bounded discovery evidence:

• GitLab: 16/16 distinct successful query rotations, 7 new queue rows
• DockerHub: 16/16 distinct successful query rotations, 10 new queue rows
• HuggingFace: 17 successful recent-page cycles, 31 new queue rows
• Discovery failures: 0
• Hard limits: GitLab 20, DockerHub 20, HuggingFace 120; none exceeded

Native Windows Parallelism 3

An isolated temporary server user and device were created with typed, audited AdminService operations. The one-time token was transferred privately, the server transfer file was deleted after successful launch, and the native worker used a dedicated protected LOCALAPPDATA state tree.

The first foreground launch exposed a local automation issue: the tool runner retained the descendant process and did not return. The worker itself survived. Subsequent starts used Win32_Process.Create so the worker was genuinely detached. No process command line was inspected.

Capacity findings

Client --parallelism 3 and server user cap 3 were not by themselves enough to permit three physical scans.

The first run reached only one active reservation because the production config had:

• global.max_active_scans = 1
• 64 MiB maximum event size
• 256 MiB projection backlog maximum
• 128 MiB projection headroom

The capacity model reserves twice the event maximum per assignment, plus headroom. Three slots therefore require 512 MiB. The temporary test candidate changed only:

• max_active_scans: 1 -> 3
• projection_backlog_max_bytes: 256 MiB -> 512 MiB

A second run still reached only one assignment. Admission evidence showed pipeline_capacity_closed on the keycheck axis. The stable keycheck backlog was 127 items and each assignment reserved up to 2,000 items. The production limit of 4,096 allowed one reservation but not three. The temporary candidate changed:

• keycheck_queue_max_items: 4096 -> 8192

No unrelated limit was increased.

Successful run

After both capacity corrections:

• Exact max unfinished concurrency: 3
• Never exceeded: 3
• Issued: 180
• Accepted: 180
• Prebundle failures: 0
• Expired: 0
• Unfinished after settlement: 0
• Accepted-not-ingested: 0
• Queue-not-settled: 0
• Projection-not-completed: 0

Source totals:

Source	Accepted
DockerHub	63
GitLab	65
HuggingFace	52

Scan outcomes were transport-successful but not necessarily target-clean:

Source	Outcome	Count
DockerHub	clean	44
DockerHub	degraded	18
DockerHub	error	1
GitLab	clean	57
GitLab	error	8
HuggingFace	clean	43
HuggingFace	error	9

These scan statuses are provider/target results. They are not lost transport, ingestion, or projection records. All 180 authoritative results settled.

Stability Observation

After the 180-assignment cap closed dispatch for the test identity, the worker remained online at cap 0 under detached monitoring.

• Successful checks: 43
• Elapsed observation: 14,105 seconds
• Runtime restarts: 0
• Edge restarts: 0
• Native worker OOM/restart: none
• Pipeline debt: 0
• Quarantine: 0
• Global invariants: 0

A single strict-health probe can observe a discovery producer during its normal short startup window. Monitors were corrected to require failure across three attempts with 20-second delays rather than treating one transient sample as a production defect.

Dual Worker Validation

The user then requested a second topology:

• native Windows worker: real client parallelism 1 and server cap 1;
• existing trusted production WSL worker: cap 1;
• desired combined concurrency: 2.

The native worker was cleanly stopped at cap 0, relaunched from the same trusted package and protected state with parallelism 1, and verified by PID/resource evidence. The production WSL token and command line were never inspected.

Monitor corrections

Several fail-closed attempts improved the monitoring policy without losing or canceling work:

• A scanner subprocess can remain active without an HTTP contact until it reports. Contact age alone must not fail an identity with unfinished work.
• A worker may honor a prior cap-0 Retry-After after caps change. Arming uses a separate 600-second threshold; the 180-second idle threshold applies only after dispatch has opened.
• Operation IDs are one-use and identity-bound. Every retry used a fresh audited operation namespace.
• A transient PostgreSQL connection timeout while reopening a read-only monitor connection terminated observability after caps and gates were already safely closed. The monitor now retries database opens three times with 20-second delays. A separate read-only settlement monitor completed the active run.

Attempt 1 evidence

The first active dual run already proved max concurrency Windows 1, WSL 1, combined 2. It fail-closed because of the old contact policy while WSL was doing a long DockerHub scan.

• Windows: 19 accepted assignments
• WSL: 2 accepted HuggingFace assignments
• WSL: 1 DockerHub reservation naturally expired after its two-hour lease
• No assignment was canceled

Final attempt 4

The final bounded run used baseline reservation ID 1131.

• Issued: 30
• Windows issued/accepted: 29/29
• WSL issued: 1
• WSL accepted: 0
• WSL expired/refunded: 1 long DockerHub assignment after its two-hour lease
• Max Windows concurrency: 1
• Max WSL concurrency: 1
• Max combined concurrency: 2
• Accepted-not-ingested: 0
• Queue-not-settled: 0
• Projection-not-completed: 0
• Cohort integrity violations: 0
• Global invariant violations: 0
• Failed holds in the cohort window: 0
• Quarantine bytes/items: 0/0

Windows source/outcome totals in the final run:

Source	Outcome	Count
DockerHub	clean	12
GitLab	clean	7
HuggingFace	clean	9
HuggingFace	error	1

The WSL expiry does not invalidate the concurrency result: both independent clients simultaneously held one real reservation and never exceeded cap. Earlier in attempt 1, the same WSL path also completed two accepted HuggingFace results.

Authoritative dual evidence:

• /opt/truf-remote-server/staging/windows-dual-worker-report.json
• SHA-256: 8fd03bf79960393406ef1a0f788d90db3ed3521d32ffa83f802c9d1c2c3509bf

Data Reconciliation

The validation distinguished assignment acceptance, bundle ingestion, queue settlement, and projection completion.

Checks covered:

• accepted reservation has a durable receipt;
• accepted reservation has exactly one matching bundle;
• reservation/bundle/scan event IDs and hashes agree;
• target scan references the same queue row;
• queue completion is applied;
• projection job exists, matches the event identity, and is completed;
• required scan_results append exists;
• required found_secrets append exists when requested by the mask;
• projection append event identity matches the job;
• no orphan bundle, scan, finding, error, or projection job exists;
• JSONL files are present, newline-terminated, and cursor offsets match.

All cohort and global mismatch counters were zero at the final report points.

Code Defects Fixed

Integer cap zero

AdminService._validate_cap() used str(value or ''), converting integer zero to an empty string even though cap 0 is valid. It now distinguishes None from zero.

Changed local source:

• app/admin_api.py
• tests/test_admin_api.py

Verification:

• python -m pytest tests/test_admin_api.py -q: 62 passed.

The currently deployed runtime image predates this source fix. Run helpers used canonical string "0" through the same typed/audited API; no direct database mutation was used.

Operational helpers

Run-specific helpers were added under build/sec-deploy/ for config lifecycle, controls, typed worker administration, monitoring, observation, local launch, safe stop, settlement, and reporting. Python helpers passed py_compile and Ruff; PowerShell helpers passed parser validation before use.

Configuration Restoration

Temporary active config identities were:

Stage	SHA-256
Bounded discovery + v3 profile	ec2ae20f9a23ee9ec423c28f7da0b45db3eaff17abb3f484e1a1f0c9227fada4
Parallel scan/projection capacity	4fee05f3a9ae001767f2de0388dcd5db75183dd954be8e024e98f47d4b64b6b1
Parallel keycheck capacity	b3509aacc53ab0bfa1e6b13dd789f889d9fea34daf0f680579d2a54697393bf9
Restored production semantics + v3 trust	c0966cac4f7f0610a813fa8732e91953f2e3e838ad880f91fd1a9437096925c7

The final config restores the original production discovery/source settings, global scan capacity, projection capacity, keycheck capacity, and supervisor interval. The only intentional permanent semantic difference from the initial config is the Windows compatibility profile path changing from stale v2 to the reproducible/current v3 manifest. Therefore the final config hash intentionally differs from the initial hash while retaining the same 37,233-byte size.

All four managed config apply operations succeeded and reconciled. The final apply operation was:

• a9943967-372e-522a-8386-4f9465cc03f9

The complete root-only operation export contains 69 operations for the run actor, all with status succeeded:

• /opt/truf-remote-server/staging/windows-p3-operations.json
• SHA-256: 318aaa56cc2ba348e905b72fd63d7795e24cbd1aa9e84d0a75fa341744ef0acf

Final Production State

The temporary native process was stopped only after its local bundle/work trees and authoritative pipeline were empty.

Typed cleanup completed:

• temporary Windows cap: 0
• temporary device: revoked
• temporary user: disabled
• server plaintext token transfer: absent
• production WSL cap: 1

Controls after restoration:

• revision: 98
• discovery: open
• dispatch: open
• drain: normal
• effective gates: open

The production WSL container was idle but still honoring a previous cap-0 retry delay. With zero unfinished work it was safely restarted using the same container, token, and state. It immediately made a fresh server contact and received a new production assignment, proving restored admission.

At the final snapshot:

• production WSL contact was fresh and one real production assignment was active;
• blocker count 1 was therefore expected, not a drain defect;
• WSL container running, OOM false, restart count 0;
• runtime healthy, restart count 0;
• edge running, restart count 0;
• host-agent, host Caddy, and X-UI active;
• strict runtime health: ACTIVE/READY with DockerHub, GitLab, HuggingFace, janitor, JSONL projector, result ingester, and Worker API;
• failed hold absent.

Runtime image remained:

sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf

Final Snapshot

• Path: /opt/truf-remote-server/staging/windows-p3-final.json
• SHA-256: 1e3ab6bdb50c5be1416974bf56235c220166a6b731e76a87d8abebb947aae074
• Bytes: 6,883
• Config SHA-256: c0966cac4f7f0610a813fa8732e91953f2e3e838ad880f91fd1a9437096925c7
• Reservation high-water: 1165
• Scan high-water: 1159
• Projection job high-water: 1159
• Projection append high-water: 1239
• Error high-water: 1945
• Finding high-water: 127
• Quarantine rows: 0
• All six recorded global orphan/mismatch invariants: 0
• scan_results.jsonl: 4,592,879 bytes, cursor exact, newline-terminated
• found_secrets.jsonl: 146,366 bytes, cursor exact, newline-terminated

Final strict-health evidence:

• /opt/truf-remote-server/staging/windows-p3-final-health.json
• SHA-256: 68c2b0dcd16591b2407fefcb5c3921aa7f58443011539ff912e2cb19b9fdef13

Retained Local Evidence

Per the user's explicit instruction on 2026-09-23, no additional local files were deleted during final reporting.

The following remain intentionally retained:

• protected local one-time token file;
• dedicated native worker state directory;
• worker stdout/stderr and PID evidence;
• fixed package and both reproducibility builds;
• local run-specific helpers and reports.

The token is no longer usable because the server device is revoked and user is disabled, but the local protected file remains until the user authorizes deletion.

Residual Limits

• The continuous 10-hour soak was shortened by the later dual-worker request.
• HuggingFace configured discovery is recent-page discovery, not keyword search.
• Long DockerHub work twice reached the fixed two-hour WSL lease and expired; expiry/refund behavior was correct, but those targets did not produce bundles.
• The local source fixes for package reproducibility and integer cap zero are tested in this checkout but were not deployed as a new production runtime image during this validation.
• Local sensitive/state evidence is deliberately retained and requires a later explicit cleanup instruction.