Files
truf-server/docs/defect-worker-network-oserror-mislabeled-local-io-2026-09-25.md
T
2026-09-30 20:30:56 +03:00

2.1 KiB

Worker network OSError is logged as local I/O

Status: open, reproducible from the error-classification path.

Observed behavior

The accepted Linux worker logged the following safe summary during the live operator-experience validation:

2026-09-25T13:10:05.909Z worker slot 0: local I/O operation failed

The complete worker journal shows that reservation 1689 had already entered assigned at 13:09:46.641Z. No runner phase had started. After the configured error delay, the worker retried the same durable assignment, entered preparing at 13:10:22.215Z, completed it, and received one bundle_accepted receipt at 13:10:28Z.

The only operation between the successful assigned event and runner startup is the authenticated assignment-status request in WorkerSlot.step() (app/remote_worker_client.py, around lines 2141-2153). The HTTPS client lets socket and transport OSError exceptions propagate. safe_worker_error_summary() (app/remote_worker_client.py, around lines 128-135) maps every OSError to local I/O operation failed, including network socket failures.

Impact

  • No assignment, result, or progress data was lost in the observed incident.
  • The durable retry behavior worked and did not rescan the target.
  • The operator-facing message misclassifies a transient network failure as a local storage/filesystem problem, which can send diagnosis in the wrong direction.
  • The original exception type is not retained in the safe worker log, so the transport subtype cannot be recovered after the fact.

Evidence

  • build/live-trace-20260925/linux-worker-state-interim.tar.gz
  • Worker event sequences 1292-1301 and the matching history row for reservation 1689
  • app/remote_worker_client.py status-read and safe-summary paths

Expected correction

Classify network/socket failures before the broad OSError branch and emit a safe transport-specific summary. Keep filesystem/storage OSError failures as local I/O. Add a regression test covering an OSError raised by WorkerHTTPClient.status() and verify that retry behavior remains unchanged.