Flow control

The flow-control plane (SQL contracts 0.4.0–0.6.6) regulates how fast and how much work a queue takes on, and stops the fleet from hammering a failing downstream. Every feature is off by default and configured per queue — a queue that sets nothing gets the same control behavior as before: nothing gates, throttles, ages, or breaks its work. When a gate declines work it returns a typed throttled claim verdict carrying retry_after_seconds, so workers sleep exactly as told rather than guessing.

Circuit breaker

Trips a queue open after N consecutive terminal failures, so the whole fleet stops claiming from a dying downstream instead of each worker burning jobs (and paid proxy spend) rediscovering the outage. After a cooldown it admits exactly one probe; a success closes it (and slow-starts via the ramp), a failure re-opens it.

A configured breaker trips on any of three triggers:

• Streak (always on) — N consecutive terminal failures. Fast; catches a hard-down downstream. A single success resets it, so it never catches a flaky one.
• Rate (optional, 0.6.3) — a sustained failure ratio over a rolling window. Add it for downstreams that fail intermittently.
• Latency (optional, 0.6.4) — a rolling-window average execution latency over a threshold. The one that catches a downstream that is slow but still succeeding; a slow success counts toward the window even at a 0% failure rate.

taskq queue set-breaker mail --failure-threshold 5 --cooldown-seconds 30 --half-open-successes 1
# Optional rate trip: >=50% of the last 60s' settles failed, min 20 settles.
taskq queue set-breaker-rate mail --failure-ratio 0.5 --window-seconds 60 --min-volume 20
taskq queue set-breaker-rate mail --off   # remove the rate trip (keep streak)
# Optional latency trip: avg execution latency over the last 60s >= 2000ms, min 10 settles.
taskq queue set-breaker-latency mail --threshold-ms 2000 --window-seconds 60 --min-volume 10
taskq queue set-breaker-latency mail --off  # remove the latency trip (keep streak/rate)
taskq queue trip-breaker mail             # force open (planned downstream maintenance)
taskq queue close-breaker mail            # force closed + slow-start recovery
taskq queue set-breaker mail --off        # disable the whole breaker

SELECT taskq.set_breaker_config('mail', 5, 30, 1, 'operator:admin');
SELECT taskq.set_breaker_rate('mail', 0.5, 60, 20, 'operator:admin');
SELECT taskq.set_breaker_latency('mail', 2000, 60, 10, 'operator:admin');

Start with a conservative streak threshold (5–10) — too low false-trips a heterogeneous queue. For the rate trip, keep min-volume high enough (≥10–20) that a quiet queue's few failures don't trip it. For the latency trip, set threshold-ms well above the downstream's healthy p50 so normal jitter doesn't trip it, and keep min-volume ≥10 — it measures average latency per settle, so sustained slowness trips it while one slow job among many won't.

In-flight cap (max_running)

Caps concurrently-running jobs per queue, across the whole fleet. Use it — not worker concurrency — when the limit belongs to the downstream (a connection pool, an API that 429s past N in-flight).

SELECT taskq.ensure_queue('mail', '{"max_running": 8}'::jsonb, 'operator:admin');

The cap is advisory under concurrency: a simultaneous burst can briefly overshoot, then settles. It is exact in aggregate.

Rate limits

Two independent keyspaces, both metering claims (work starts):

• Queue-level — claim_rate_per_minute / claim_burst on the queue profile (GCRA).
• Key-level — a flow_key that jobs carry, orthogonal to the queue. Unknown keys are unlimited (a politeness limiter must not serialize the world by default).

taskq queue set-flow-limit provider.acme --rate-per-minute 120 --burst 10

SELECT taskq.set_flow_limit('provider.acme', 120, 10, 'operator:admin');

Slow-start ramp

ramp_seconds on the profile: after a resume (or a breaker close), the queue's effective max_running and claim rate scale from near-zero to full over the window, so a just-recovered downstream isn't stampeded. It composes automatically with the breaker.

Job TTL

default_ttl_seconds on the profile, or p_ttl_seconds at enqueue. Expired queued/blocked jobs are settled cancelled / outcome='expired_ttl' by the tick. Running jobs are never TTL-killed — the lease governs in-flight work. Use it for work that is worthless if stale.

Smear (anti-stampede)

• Redrive smear — taskq.redrive_failed(queue, limit, actor, smear_seconds) spreads redriven jobs' scheduled_at across [now, now+smear) instead of releasing them all at once.
• Schedule smear — a deterministic per-schedule firing offset so co-scheduled jobs (many cron entries at :00) de-align instead of stampeding.

taskq schedule set-smear nightly-report --smear-seconds 300

SELECT taskq.set_schedule_smear('nightly-report', 300, 'operator:admin');

Priority aging

A waiting job's effective claim priority improves with age, so a sustained high-priority flood cannot starve low-priority work forever. Opt-in per queue; fresh work is never inverted.

taskq queue set-aging render --aging-seconds 60   # +1 priority step per 60s waited
taskq queue set-aging render --off                # strict priority

SELECT taskq.set_priority_aging('render', 60, 'operator:admin');

Notify mode

notify_mode = 'on_idle_transition' on the profile fires the wake-up NOTIFY only when the queue was idle before an enqueue, cutting notify volume on busy queues. Leave it 'always' (default) unless NOTIFY volume is itself a problem.

Observing flow control

Since 0.6.2 a tripped breaker surfaces as the breaker_open health verdict (with breaker state in the health detail) and as job events on each automatic transition. Use the health surface for "is it open now" and events for "when did it trip."

-- Is any breaker open right now? (verdict + breaker detail)
SELECT queue, verdict, detail -> 'breaker' AS breaker
FROM taskq.queue_health(NULL) WHERE verdict = 'breaker_open';

-- Breaker timeline (automatic transitions): opened / reopened / closed.
SELECT e.created_at, e.event_type, e.data
FROM taskq.job_events e JOIN taskq.jobs j ON j.id = e.job_id
WHERE j.queue = 'mail' AND e.event_type LIKE 'breaker_%'
ORDER BY e.created_at DESC;

-- Full flow state for a queue; live levels + throughput.
SELECT breaker_state, breaker_failure_streak, breaker_opened_total, breaker_tripped_at,
       priority_aging_seconds, ramp_started_at
FROM taskq.queue_flow WHERE queue = 'mail';
SELECT * FROM taskq.queue_counters WHERE queue = 'mail';

Operator audit log

Since 0.6.5, every queue-scoped operator verb (set-breaker, set-breaker-rate, set-breaker-latency, trip-breaker, close-breaker, set-aging) records an append-only audit row with its actor and a {before, after} detail — config-history plus the manual-trip trail that job_events never captured. A failed verb writes nothing (the row rolls back with the action).

taskq queue audit mail                    # last 50 operator actions, newest first
taskq queue audit mail --before-id 1234   # page: entries with an id below 1234

SELECT id, event_type, actor, detail, created_at
FROM taskq.list_queue_audit('mail', 50, NULL);

Read access is taskq_operator + taskq_observer.

The log is append-only; cap its growth with the maintenance prune verb (taskq_housekeeper or taskq_operator), safe to run on a schedule:

taskq maintenance prune-audit --older-than-hours 2160 --yes   # destructive: --yes required

SELECT taskq.prune_queue_audit(2160);  -- returns the number of rows removed

Worker-side, throttled verdicts (from any gate) surface as throttle counts in the worker snapshot — they are not errors and do not trip the claim-error backoff.