Operations

Standalone Scheduler

Bind the database target, apply paused manifests, and run one safe scheduler clock.

TaskQ 0.1.0a36 separates recurring-work ownership from FastAPI. The standalone scheduler is a clock: it evaluates recurrence and durably enqueues ordinary TaskQ jobs. It never imports an application registry or executes task handlers.

production topology

one scheduler clock -> queued jobs -> one or more application workers -> handlers

Process	Needs	Does not need
Scheduler	Restricted housekeeper DSN, static target expectations	Application imports or handler secrets
Manifest operator	Restricted operator DSN, reviewed YAML	Runner or migration-owner rights
Worker	Runner DSN or HTTP credentials, task registry, handler dependencies	Scheduler or operator rights
API / facade	Producer and optional facade credentials	A recurring scheduler loop

Run one logical scheduler per database/environment. Worker placement is an independent application decision: a supervised local worker, a consolidated multi-queue process, or a cloud worker are all valid when their dependencies, secrets, availability, and failure boundaries support that choice.

1. Install and bind the target

Use a dedicated owner/migration credential for this section only. Put only its environment-variable name in a secret-free CLI context:

~/.config/taskq/config.toml

version = 1

[contexts.staging]
transport = "sql"
dsn_env = "TASKQ_STAGING_DSN"
expected_environment = "staging"
actor = "operator:release-agent"

Terminal

export TASKQ_STAGING_DSN='postgresql://taskq_migration:...@db-host/app'

taskq --context staging db plan -o json
taskq --context staging --yes db migrate --plan-digest "$PLAN_DIGEST"
taskq --context staging target show -o json

On a fresh scheduler installation, migration 0019 commits and 0020 refuses to activate while the target is unbound. The first migrate exits 2 with stable code CLI_TARGET_BINDING_REQUIRED; it is not a retryable outage. target show intentionally permits this unbound bootstrap state even though the context declares the future environment. Record the returned installation UUID outside the target database configuration, then bind the reviewed environment:

Terminal

taskq --context staging --yes target bind staging \
  --expected-installation-id '<installation-uuid>' \
  --expected-binding-version 0

taskq --context staging db plan -o json
taskq --context staging --yes db migrate --plan-digest "$PLAN_DIGEST"
taskq --context staging db verify

Binding production additionally requires --allow-production. After migrate/bind/verify, remove the owner credential from process scope and use separate restricted scheduler, operator, and worker credentials.

Restores and clones

A same-environment disaster-recovery restore retains its installation identity and deployment pin. A clone used for another environment must rotate identity explicitly:

Terminal

taskq --context staging --yes target bind staging \
  --expected-installation-id '<current-installation-uuid>' \
  --expected-binding-version '<current-version>' \
  --rotate \
  --reason 'production clone promoted to staging'

Rotation invalidates prior scheduler and worker pins.

2. Pin runtime identity

Target expectations come from static deployment configuration, never from a database read performed during startup:

Environment

TASKQ_DSN=postgresql://taskq_scheduler:...@db-host/app
TASKQ_EXPECTED_ENV=staging
TASKQ_EXPECTED_INSTALLATION_ID=<staging-installation-uuid>

The installation UUID pin is optional outside production but strongly useful for catching restored or copied databases. Production requires both the pin and explicit opt-in:

Environment

TASKQ_EXPECTED_ENV=production
TASKQ_EXPECTED_INSTALLATION_ID=<production-installation-uuid>
TASKQ_ALLOW_PRODUCTION=true

A process cannot make a production DSN safe by labeling itself development. Schedule mutation, scheduler runtime, housekeeping, and direct worker claims establish database-attested expectations inside each protected transaction.

Use distinct DSNs, task namespaces, queues, and supervisor configuration for development, staging, and production. Never fall back to a production DSN when local configuration is missing.

3. Create a paused manifest

Manifests are versioned, source-owned desired state:

schedules.yaml

version: 1
namespace: myapp
source: api-deployment
schedules:
  maintenance:
    display_name: Application maintenance
    task: myapp.maintenance
    queue: maintenance
    interval_seconds: 300
    catchup: fire_once
    overlap: forbid
    max_lateness_seconds: 900
    state: paused
    payload: {}

Field	Production guidance
`namespace` + key	Stable schedule identity (`myapp.maintenance`)
`source`	Stable owner of this desired state
`catchup`	Explicit `fire_once` or `fire_all` in the current release
`overlap`	Start with `forbid`; use `allow` only when every occurrence must enqueue
`max_lateness_seconds`	Drop work that is too stale to remain useful
`state`	Start `paused`; activation is a separate reviewed change
`payload` / headers	Never place credentials or other secrets here

Do not use catchup: skip for recurring application work in the current SQL contract. The released SQL contract advances the schedule without selecting an occurrence, including during ordinary polling. Use explicit fire_once or fire_all until a later SQL contract ships corrected skip-missed semantics.

fire_once coalesces backlog to the latest due instant. fire_all selects oldest-first up to max_catchup. With fire_all plus overlap: forbid, the first selected occurrence can enqueue and later occurrences in the same batch record overlap_skipped. Use overlap: allow plus an execution concurrency_key when every occurrence must queue but handlers must serialize.

4. Plan and apply without firing

Use a restricted operator credential through the same target-pinned context shape (with dsn_env pointing at the operator credential):

Terminal

export TASKQ_STAGING_DSN='postgresql://taskq_operator:...@db-host/app'

taskq --context staging schedule manifest plan schedules.yaml -o json
taskq --context staging schedule manifest apply schedules.yaml \
  --plan-digest "$PLAN_DIGEST" -o json

plan never mutates. apply uses compare-and-swap versions and can create, update, or report unchanged/drift. Missing keys are not pruned. Retire one owned key explicitly:

Terminal

taskq --context staging --yes schedule manifest retire schedules.yaml maintenance \
  --reason 'maintenance moved to another owner'

Production apply/retire requires the static production installation pin and an explicit --allow-production flag.

5. Start the executor and clock

Start the application worker first, while the manifest is still paused. One host-native worker can subscribe to several compatible queues through a combined registry; TaskQ does not require a container per task, queue, or schedule.

Then validate and start the clock:

Terminal

export TASKQ_DSN='postgresql://taskq_scheduler:...@db-host/app'
export TASKQ_EXPECTED_ENV='staging'
export TASKQ_EXPECTED_INSTALLATION_ID='<staging-installation-uuid>'

taskq --context staging scheduler doctor -o json
taskq --context staging scheduler run

For scale-to-zero databases or platform timers, use bounded mode:

Terminal

taskq --context staging scheduler once -o json

Bounded mode exits 0 for nothing_due or successful firing, 2 for target or configuration refusal, 3 for unavailable/version/capability failures or a budget-exhausted pass, and 1 for an unexpected internal failure.

Bounded worker pulse

Workers normally remain supervised on an existing local worker host. When a handler must reach dependencies available only on the API host's private network, a platform timer may start a bounded worker from the existing application image instead of adding one permanently running container.

Coolify scheduled task

timeout --signal=INT --kill-after=30s 50s taskq --context staging worker run \
  --registry myapp.tasks:registry \
  --queue maintenance \
  --environment staging

Run the pulse more frequently than the task's acceptable claim latency, keep its bounded runtime below the timer cadence, and give the platform a slightly longer outer timeout. Treat intentional timeout/SIGINT shutdown as success only after confirming the worker received a graceful stop. Durable queued work survives the gap between pulses, but this topology is unsuitable for queues that require continuously low claim latency.

Each bounded invocation normally has a distinct worker identity and therefore leaves a historical presence row. Monitor recent online workers and last_seen_at, not the raw number of presence rows. Do not infer multiple live workers merely because several completed pulses remain observable.

This is a documented dependency-placement exception, not the default. Do not move ordinary workers into containers simply because the scheduler clock runs there, and do not create one worker container per task or queue.

6. Canary and activate

Verify isolation

Confirm the expected target fingerprint, the paused manifest, one scheduler clock, the intended worker registry/queues, and no stale jobs waiting in the target queue.

Run an immediate canary

Enqueue one ordinary job through the host producer path. Do not activate a recurring schedule merely to get an immediate test.

Activate desired state

Change the manifest to state: active, run plan, review the diff, and apply.

Wait one complete interval

Activation and resume are from now. The first scheduler pass anchors an interval schedule; the first job becomes due after one full interval. Activation does not enqueue immediately.

Observe the whole path

Verify the schedule decision, occurrence, queued job, worker claim, and handler result before ending the attended pilot.

Monitor independently of the API

Terminal

taskq --context staging scheduler doctor -o json

Monitor:

scheduler process presence and restart rate;
last_decision_at advancement and oldest due time;
due-schedule count and due lag;
jobs enqueued, evaluation errors, and bounded-mode exhaustion;
auto-paused schedules and overlap/lateness skips;
application worker presence and job outcomes.

Health totals can include TaskQ package-owned schedules such as the janitor. Filter decisions and jobs by the application namespace/queue before asserting that exactly one application schedule fired.

The host API /health endpoint does not prove that the scheduler clock or application worker is healthy. Three consecutive deterministic definition or calendar errors auto-pause a schedule; transient infrastructure errors do not increment that counter.

Roll back stop-first

Stop the scheduler process or platform timer.
Change the manifest to state: paused, then plan and apply it.
Allow an in-flight handler to finish or drain it, then stop the worker.
Inspect queued jobs and durable schedule decisions before changing topology.
Restore an embedded API loop only as a temporary, explicitly single-owner compatibility measure.

Stopping the scheduler does not cancel queued or running jobs. Pausing a manifest does not fix an incorrectly configured worker DSN, so environment identity and worker supervision remain separate safety controls.

OutlabsAuth maintenance recipe

OutlabsAuth 0.1.0a29+ exposes typed run_maintenance_once() while retaining the raw run_background_jobs_once() compatibility method. Keep background_job_mode="disabled" in every API replica, register a host task such as myapp.auth.maintenance, and let its worker call the typed one-shot API. Translate report.ok=false into a retry outcome.

Use a paused manifest with catchup: fire_once, overlap: forbid, and a queue concurrency of one. The executor needs the host auth Postgres/Redis access; the TaskQ scheduler does not. Persist only secret-free aggregate report fields and require empty missing_steps, zero reported_errors, and a successful TaskQ job outcome.

See the complete OutlabsAuth Background Maintenance guide.

Workflow Continuations

Grow sealed workflows through compiled, hash-pinned member policies.

Housekeeping

Tick, lease reaping, schedule firing, and wake behavior.