Don't trace commands that never finish #86

Merged
Patrick merged 1 commit from patrick/no-trace-for-daemons into main 2026-11-19 21:58:51 -05:00 AGit
Owner

Dallas's v0.3.30 upgrade timed out because Horizon crash-looped: Memory limit exceeded: Using 65/64MB, exit 12, with OpenTelemetry reporting missing call to Scope::detach() for scope #3096.

Cause: command tracing opens a root span on CommandStarting and closes it on CommandFinished. horizon (and the horizon:supervisor / horizon:work processes it starts), queue:work, reverb:start and schedule:work never finish, so their span stayed open for the life of the pod. Every job and query nested under it, and nothing was ever exported. Tracing only became effective on Dallas once its endpoint pointed at the real Tempo, which is why this didn't show before.

Fix: those commands open no span. Each job a worker runs now gets its own trace through the queue seam, which is what job tracing intended.

Checked: 3 new tests in TracingTest; the two daemon ones fail against the old code. Full suite: 516 passed, 8 skipped. Pint clean.

🤖 Generated with Claude Code

**Dallas's v0.3.30 upgrade timed out** because Horizon crash-looped: `Memory limit exceeded: Using 65/64MB`, exit 12, with OpenTelemetry reporting `missing call to Scope::detach() for scope #3096`. **Cause:** command tracing opens a root span on `CommandStarting` and closes it on `CommandFinished`. `horizon` (and the `horizon:supervisor` / `horizon:work` processes it starts), `queue:work`, `reverb:start` and `schedule:work` never finish, so their span stayed open for the life of the pod. Every job and query nested under it, and nothing was ever exported. Tracing only became effective on Dallas once its endpoint pointed at the real Tempo, which is why this didn't show before. **Fix:** those commands open no span. Each job a worker runs now gets its own trace through the queue seam, which is what job tracing intended. **Checked:** 3 new tests in TracingTest; the two daemon ones fail against the old code. Full suite: 516 passed, 8 skipped. Pint clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Don't trace commands that never finish
All checks were successful
Application Tests / frontend (pull_request) Successful in 50s
Application Tests / security-audit (pull_request) Successful in 2m40s
Application Tests / php (pull_request) Successful in 3m22s
Application Tests / tag (pull_request) Has been skipped
Application Tests / publish-images (., Dockerfile, laravel) (pull_request) Has been skipped
Application Tests / publish-images (agent-worker, Dockerfile, agent-worker) (pull_request) Has been skipped
Application Tests / publish-images (docker/asterisk, Dockerfile, asterisk) (pull_request) Has been skipped
Application Tests / publish-images (docker/asterisk-config-sync, Dockerfile, asterisk-config-sync) (pull_request) Has been skipped
Application Tests / publish-images (docker/haraka, Dockerfile, haraka) (pull_request) Has been skipped
Application Tests / publish-images (docker/whisper-local, Dockerfile, whisper-local) (pull_request) Has been skipped
Application Tests / scan-images (agent-worker) (pull_request) Has been skipped
Application Tests / scan-images (asterisk) (pull_request) Has been skipped
Application Tests / scan-images (asterisk-config-sync) (pull_request) Has been skipped
Application Tests / scan-images (haraka) (pull_request) Has been skipped
Application Tests / scan-images (laravel) (pull_request) Has been skipped
Application Tests / scan-images (whisper-local) (pull_request) Has been skipped
Application Tests / publish-helm (pull_request) Has been skipped
Application Tests / publish-archives (pull_request) Has been skipped
9b86ba6751
A command's root span closes on CommandFinished. horizon, its
supervisor/worker processes, queue:work, reverb:start and
schedule:work run until the pod stops, so their span stayed open
forever with every job and query nested under it, and nothing was
exported. Once Dallas's tracing endpoint pointed at the real Tempo,
the Horizon master passed its 64 MB memory_limit within minutes and
crash-looped (exit 12), failing the v0.3.30 upgrade.

These commands now open no span. Each job a worker runs gets its own
trace through the queue seam instead of folding into the endless one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
calltheory/orbital!86
No description provided.