Queue Boundaries
A TaskQ queue is an operational control boundary. It is not a database-table boundary and it is not just a label for a task type.
Jobs in separate queues may safely use the same application tables when the application enforces transactions, fencing, idempotency, and domain invariants. The queue isolates the controls around that work: pause/resume, drain, depth, flow limits, circuit breakers, worker ownership, credentials, and recovery.
One-owner decision test
Put workloads in the same queue only when one operational owner can safely make the same decision for every job in it:
- pause, resume, drain, purge, and redrive;
- worker placement, concurrency, and rollout timing;
- depth, claim rate, circuit breaker, and recovery ramp;
- credentials, downstream quotas, and provider spend;
- failure classification, recovery procedure, and service objective.
If two workloads need independent answers to any item, use separate queues. A controller that pauses or drains a queue controls every task type in that queue; task-type filtering does not make a queue-wide action safe.
Share related stages deliberately
Related stages may share one queue when they intentionally have the same owner, capacity budget, availability target, and recovery lifecycle. A render pipeline can share a render queue when a render-wide pause is supposed to stop prepare, execute, verify, and upload together.
Do not create one queue per job, handler, workflow step, process, or container. Use:
| Need | TaskQ mechanism |
|---|---|
| Dispatch different payloads | Task types and registries |
| Express stage dependencies | Workflows and follow-ups |
| Place work on a lane | Worker queue/task filters |
| Limit a real shared resource | Concurrency or flow keys |
| Control a workload independently | A separate queue |
Isolation does not create capacity
Separate queues protect control and admission. They still share the TaskQ installation's Postgres CPU, storage I/O, API capacity, network, and any common provider quotas.
For concurrent queues:
- set an explicit per-queue
max_running, claim rate, and depth; - bound worker concurrency and credential request budgets;
- measure planning/enqueue cost as well as handler cost;
- load-test cross-queue latency under shared database saturation;
- use another TaskQ installation only when physical resource or failure-domain isolation is required.
Safe bootstrap and deployment
Queue catalog reconciliation must preserve every existing queue's open or paused state. A newly created queue can start paused for its canary; bootstrap must not pause or resume unrelated queues as a side effect.
Treat queue names as stable operational API. Give producers, workers, and controllers least-privilege access to their queues. Run one supervised scheduler clock per TaskQ installation; the scheduler does not own every queue.
Durable workers require an always-on host and an external service supervisor with restart policy and observable health. Laptop batteries, lid state, sleep-prevention tools, and interactive login sessions are not production availability controls. Use a laptop only for attended, bounded work whose interruption and lease recovery are understood.
Prove the boundary
Before opening multiple queues in production, test against the real TaskQ database and authorization layer:
- pause/drain queue A while queue B stays open and settles work;
- reconcile the catalog without changing any existing queue state;
- deny every producer, worker, and controller on queues it does not own;
- run both queues together and verify independent presence, counters, depth, limits, and breakers;
- terminate a worker and prove supervised restart or lease-expiry recovery without duplicate external effects;
- saturate shared capacity and verify backpressure without control coupling.
Keep the queue-to-workload ownership map beside the worker registry and deployment manifest. Review it whenever a new controller, provider, worker family, or service objective is introduced.