Inside a worker

How one worker claims, runs, and finalizes a job.

Three agents and one table. The claimer takes rows from the database, the executors run handlers, the finalizer writes what happened, and an in-flight registry holds every claim from the moment it is taken until its completion has committed. The heartbeat renews exactly what the registry holds, so a row nothing holds runs out its lease and goes back to recovery.

The pipeline

The pipeline inside one worker.

One worker process. Every arrow into the database is one round trip. The green call carries every completion of a batch together, whatever its shape. The amber loop is the backpressure: when the database is slow, intake stops instead of memory growing.

What each stage decides

What each stage decides.

Claimer: Waits for one permit, then takes every other free one, and claims that many rows in one call, each moved straight to Executing. After a claim that found work, it first lets the executors take those rows and collects every permit they free, so the next claim asks for all of them instead of the one or two free at that instant. Unused permits go back at once. An empty claim sleeps until the next row is due, at least 50 ms and at most 1 s, jittered up to 100 ms, and a wakeup ends the sleep early. A failed claim backs off a full second.
The recovery pass: The heartbeat that finds the namespace's recovery pass due moves it one lease on in the same call and runs the pass beside the loop, so one worker runs each pass and a worker whose executors are all busy still sweeps. A worker that dies mid-pass holds nothing back.
Executor: Skips an entry whose deadline already fired. Then admission (concurrency slot, rate meter), the handler, and the completion record. It hands the entry to the finalizer and only then returns its permit.
Finalizer: One per worker. Reads one completion, then everything already waiting, up to 128 completions or 4 MiB of results; if that first read found a backlog, it waits up to 50 ms more to fill the batch, and a lone completion never waits. They all go into one complete_executions call, whatever their shape: a success or failure, a retry, a signal wait, a recurring fire, a child telling its parent, a lane handing on. The write repeats until it lands; then the wakes, the metric, and the entry is removed.
Heartbeat: Every 45 s it snapshots the registry and makes one call with the held (job, execution) pairs and lock holds. A renewed row re-arms its entry's deadline. A row locked by another transaction keeps its deadline for the next beat. A row not answered cancels a Claimed or Executing entry, while a Finalizing entry is left to its write. A lock not renewed cancels its attempt. When the namespace's newest definition change moved, it reloads the policy.

The lease clock

Every deadline comes from one setting: the heartbeat interval.

The lease clock of one claim with the default heartbeat of 45 s. Every number derives from that one setting: the lease is four beats, and the deadline sits halfway between one beat and the lease, so a healthy attempt never comes near it and a cancelled one has 67.5 s to stop before another worker may take the row. Recovery charges the reclaimed row a failure only if it had started; a row that was only claimed goes back uncharged.

Every cap and threshold

Every cap and threshold, and what it bounds.

Values are the defaults. Only the heartbeat interval, the executor count, and the safety poll are settable; everything else is a constant or derives from them.

Threshold	Value	What it bounds
Claiming
`MaxConcurrentExecutors`	4 per core, 8 to 64	Executor loops per worker. A 2-core container gets 8, an 8-core host 32, anything from 16 cores up gets 64.
Permits	N	Claims held and not yet handed to the finalizer. A claim takes a row only for a free executor.
`Claimed` channel	= permits	Sized so the claimer's hand-off never blocks.
`SafetyPollInterval`	1 s	The longest idle sleep, the back-off after a failed claim, and the delay on a claim handed back for want of a handler.
Poll floor · idle jitter	50 ms · ≤ 100 ms	The shortest idle sleep, and the spread that keeps idle workers from claiming in lockstep.
Executing
Execution timeout	300 s default	Per attempt, set per definition; a timeout is retried under the failure budget.
Concurrency-key bounce	2 s	How soon a job refused its concurrency slot is due again.
Finalizing
`Completed` channel	128 + N	Room for a full batch and the executors behind it; when it fills, intake stops.
Batch size	128	Completions in one `complete_executions` call.
Batch bytes	4 MiB	Result bytes in one call; a larger single result is written alone.
Batch window	50 ms	Extra wait to fill a batch, only when its first read found a backlog.
Write repeat	1 s → 30 s, ±25 %	Back-off between tries of a failed completion: Warning for three tries, then Error, until it lands or the worker stops.
Leases and recovery
`HeartbeatInterval`	45 s	Renewal cadence, and the one timing value an operator sets.
Lease	180 s, 4 beats	How long a row stays this worker's without a renewal.
Attempt deadline	112.5 s	Halfway between one beat and the lease, counted from the last confirmed renewal's start: the handler is cancelled there.
`WorkerDeadAfter`	315 s, 7 beats	No heartbeat for this long and recovery marks the worker Dead.
Recovery interval	180 s, the lease	How often a namespace's recovery pass runs, taken by whichever heartbeat finds it due.

A row's status

A job's status, and its entry in the worker.

In the ledger

Ready → Executing → Succeeded | Failed | Ready

The claim itself starts the execution, so there is no separate start write. A claim whose answer was lost is treated as a crash right after it.

In the registry

Claimed → Executing → Finalizing → removed

Reported by acta.jobs.inflight, tagged by state; batch sizes by acta.completions.batch.

Also: which job runs next·order, concurrency, and rate·many workers, no coordinator