Cluster-wide
Add as many workers as you like. They share the work through your database.
Every worker is a peer: the same process loop, talking to the same rows. There is no control plane to run, no broker to stand up, and nothing to elect. Add a process and it starts claiming. Kill one and the others carry on, and the work it held comes back through a recovery pass any of them can run.
One namespace, as the database sees it.
Three workers on build 1 share the namespace. Six jobs are due, J4 and J5 carry the concurrency key rebuild:acme, and J6 belongs to a definition the next build drops. Step through a claim, a heartbeat, a kill, the recovery pass, a shared gate, and a rolling deploy.
| job | status | leased by | lease | execution | note |
|---|---|---|---|---|---|
| J1 | Ready | · | · | 0 | |
| J2 | Ready | · | · | 0 | |
| J3 | Ready | · | · | 0 | |
| J4 | Ready | · | · | 0 | key rebuild:acme |
| J5 | Ready | · | · | 0 | key rebuild:acme |
| J6 | Ready | · | · | 0 | handler in build 1 only |
Workers find new rows by polling SQL, a 1 s safety poll by default. The optional Redis wake-up only shortens that wait; it never carries truth.
Each worker registers in the namespace, heartbeats, claims due rows, runs handlers, and refreshes its leases. Nothing else coordinates them: no control plane, no broker, no election. The workers table is the only membership list, and every worker is in it as an equal.
Times are the defaults: 45 s heartbeat, 180 s lease, 315 s before a silent worker is marked dead. Execution stays at-least-once, so a handler with external effects is idempotent on every path. SQLite is the embedded single-node provider; the fleet story is PostgreSQL and SQL Server. Production guide · 705-worker-crash-recovery
sys.retention, sys.alerts and sys.outbox are recurring slots like any schedule: each fires once per cadence, run by whichever worker claims it. Crash recovery rides the heartbeat instead, so it cannot get stuck behind the crash it repairs. No worker is special, so there is nothing to fail over.
The heartbeat interval is the only coordination knob. The lease window is four heartbeats and the dead-worker threshold seven, both read-only, so replicas that agree on the heartbeat agree on everything else by construction.
The rc.5 certification ran 24 worker processes of 64 executors each, 1,536 concurrent slots on one database, with a worker killed every five seconds. Every one of a million jobs finished and no recorded step ran twice, checked afterwards from the database's own ledger, on PostgreSQL and on SQL Server. What one run showed, not a capacity claim. Certification records
Also: how one worker runs a job·which job runs next·order, concurrency, and rate