sashatrask
51 lines
2.1 KiB
Markdown
51 lines
2.1 KiB
Markdown
# Worker network OSError is logged as local I/O
|
|
|
|
Status: open, reproducible from the error-classification path.
|
|
|
|
## Observed behavior
|
|
|
|
The accepted Linux worker logged the following safe summary during the live
|
|
operator-experience validation:
|
|
|
|
```text
|
|
2026-09-25T13:10:05.909Z worker slot 0: local I/O operation failed
|
|
```
|
|
|
|
The complete worker journal shows that reservation 1689 had already entered
|
|
`assigned` at `13:09:46.641Z`. No runner phase had started. After the configured
|
|
error delay, the worker retried the same durable assignment, entered
|
|
`preparing` at `13:10:22.215Z`, completed it, and received one
|
|
`bundle_accepted` receipt at `13:10:28Z`.
|
|
|
|
The only operation between the successful `assigned` event and runner startup
|
|
is the authenticated assignment-status request in
|
|
`WorkerSlot.step()` (`app/remote_worker_client.py`, around lines 2141-2153).
|
|
The HTTPS client lets socket and transport `OSError` exceptions propagate.
|
|
`safe_worker_error_summary()` (`app/remote_worker_client.py`, around lines
|
|
128-135) maps every `OSError` to `local I/O operation failed`, including network
|
|
socket failures.
|
|
|
|
## Impact
|
|
|
|
- No assignment, result, or progress data was lost in the observed incident.
|
|
- The durable retry behavior worked and did not rescan the target.
|
|
- The operator-facing message misclassifies a transient network failure as a
|
|
local storage/filesystem problem, which can send diagnosis in the wrong
|
|
direction.
|
|
- The original exception type is not retained in the safe worker log, so the
|
|
transport subtype cannot be recovered after the fact.
|
|
|
|
## Evidence
|
|
|
|
- `build/live-trace-20260925/linux-worker-state-interim.tar.gz`
|
|
- Worker event sequences 1292-1301 and the matching history row for reservation
|
|
1689
|
|
- `app/remote_worker_client.py` status-read and safe-summary paths
|
|
|
|
## Expected correction
|
|
|
|
Classify network/socket failures before the broad `OSError` branch and emit a
|
|
safe transport-specific summary. Keep filesystem/storage `OSError` failures as
|
|
local I/O. Add a regression test covering an `OSError` raised by
|
|
`WorkerHTTPClient.status()` and verify that retry behavior remains unchanged.
|