1Give the job a retry budget, because a crash spends an attempt
Nothing in the handler recovers the job. A worker that dies stops heartbeating, the lease on its job lapses, and another worker reclaims the job as a failed attempt and runs it again.
[Job("crash-recovery-probe", MaxAttempts = 3, Backoff = "0s")]
public async Task Handle(CrashRecoveryProbe input, CancellationToken ct)
{
Console.WriteLine($"[{role.Name}] executing {input.ProbeId}; ordinary handler code is at-least-once.");
if (role.Name == "worker-a" && input.SessionId == role.SessionId)
{
Console.WriteLine("[worker-a] worst moment reached: side effect may have happened; process disappears before completion.");
Console.Out.Flush();
Environment.FailFast("Intentional worker-a crash for the SQL recovery lab.");
}
if (role.Name == "worker-a")
{
var staleSession = input.SessionId is { Length: >= 8 } ? input.SessionId[..8] : "legacy";
Console.WriteLine($"[worker-a] stale probe from session {staleSession} completed without crashing the current session.");
return;
}
await Task.Delay(100, ct);
Console.WriteLine("[worker-b] replay completed; production code would need idempotency or durable steps.");
}2Shorten the heartbeat only to watch the recovery
One knob shortens the whole coordination triple in proportion: a 1s beat gives a 4s lease and a 7s dead-worker window, which is what makes the kill below observable in seconds instead of minutes.
builder.Services.UseActa(j =>
{
j.UseLocalDatabase(builder.Configuration, applyMigrations: mode == "worker-a");
if (mode is "worker-a" or "worker-b")
{
j.Run<WorkerCrashRecoveryJobs>(jobNamespace);
j.ConfigureOptions(o =>
{
o.HeartbeatInterval = TimeSpan.FromSeconds(1);
o.DeploymentVersion = $"{mode}:{sessionId[..8]}";
});
}
else
{
j.Reference<WorkerCrashRecoveryJobs>(jobNamespace);
}
});3Run it
From a clone of the Acta repository, with the .NET 10 SDK:
dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recoveryProven by other suites. The release check does not run this program. Needs PostgreSQL or SQL Server and two worker processes, one killed mid-job while the other reclaims its work; Verify's chaos scenario and WorkerCrashRecoveryChaosSpec prove the same behaviour.
Remember
- Run it in four terminals against one PostgreSQL or SQL Server database, each with
dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- <mode>:worker-a,enqueue,inspect, thenworker-b. With no mode the program prints these commands and exits. - On SQLite a restart brings a lost attempt back at once: a starting worker releases the work of processes on the same machine that have ended. A server database waits out the lease: three to about seven minutes with the default heartbeat.
- The reclaim is
job.execution-orphanedon the timeline, with the reasonlease-expired, orprocess-endedafter a SQLite restart. - A crash counts against the retry budget, and a reclaimed job is due again at once, without the backoff.
- A job that is waiting or sleeping holds no lease and needs no recovery.
The lab behind it: SQL-backed crash recovery without a leader
This program is an Engineering Lab: beside the how-to it prints the rows that show what happened, and its notes weigh the design against the alternatives.
The problem
Replacing a broker means the database must handle claim races, ownership, leases, dead workers, and replay after the process disappears at the worst moment.
Common approaches
- Use a broker for delivery and keep execution state elsewhere.
- Elect a coordinator or run a dedicated recovery control plane.
- Let peer workers coordinate through one relational authority.
- Use an in-process queue and accept loss on restart.
Why this design
Acta's claim, worker heartbeat, lease, and recovery transitions are SQL operations. Recovery rides every worker's heartbeat: the heartbeat that finds the namespace's pass due runs it, so no worker is a permanent leader. The lab kills worker A mid-attempt and lets worker B recover the same durable identity.
Trade-offs
Claim and completion traffic consume database capacity and compete with application traffic. Lease recovery is at-least-once, so handler work can repeat. This design targets application jobs, not high-throughput event streaming.
Run the experiment
This is intentionally a real distributed-provider lab, since its two workers run at once. One SQLite host recovers across restarts without waiting out the lease: its next start finds a predecessor on the same machine whose process has ended, marks it dead, and reclaims its job at once, as 220 shows with its crash and recover runs. Set ACTA_LOCAL_PROVIDER=postgres or sqlserver plus ACTA_TEST_PG/ACTA_TEST_MSSQL, then use separate terminals:
dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- worker-a
dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- enqueue
dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- inspect
dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- worker-bWorker A intentionally exits non-zero. Both workers run short lab-only lease timings, and worker B's heartbeat runs the real recovery pass after expiry. Starting worker A creates a new local session token; enqueue, inspect, and worker B reuse it. Worker A crashes only the probe carrying that token, so an unfinished probe from an aborted older session cannot terminate the new experiment. Each enqueue also updates a local current-job marker, making the sequence repeatable against the same shared database.
Rows to inspect
Run with --all-columns to execute the visible SELECT * Explore query first. The default Notice queries select the fields that prove the lesson, and the text below explains their meaning.
jobs_view moves from executing under A to done under B with a higher execution number. workers_view is filtered to the current session's deployment versions and preserves its heartbeat/liveness evidence, while events_view records the orphaned attempt with reason lease-expired. Application code should use IJobs; the views are curated operator surfaces.
Break it
Start two worker-B processes before recovery and observe that only one claims the recovered row. Move a non-idempotent side effect before worker A's crash and observe why recovery cannot promise exactly once.
When not to use
Use a broker or log when event throughput, fan-out, replayable streams, or independent consumer offsets are the primary problem. Use a dedicated workflow/control plane when recovery authority must be separate from application workers.
Source trail
The whole program
Every file of learn/700-topology-and-deployment/705-worker-crash-recovery, as it is compiled.
UseLocalDatabase and LearnLab are helpers of the Acta repository, not of the packages. In your own app, register the database with UseSqlite, UsePostgres or UseSqlServer, as in how to run your first job.
worker-crash-recovery.cs224 lines
// How to recover the work of a killed worker
// A worker process is killed in the middle of a job, and another worker reclaims the job once its lease has lapsed.
// The program runs on PostgreSQL or SQL Server.
using Acta;
using Acta.Labs;
using Acta.Learn.WorkerCrashRecovery;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
const string jobNamespace = "worker-crash-recovery";
var currentJobFile = Path.Combine(Path.GetTempPath(), "acta-worker-crash-recovery-current-job.txt");
var currentSessionFile = Path.Combine(Path.GetTempPath(), "acta-worker-crash-recovery-current-session.txt");
var mode = args.FirstOrDefault(a => !a.StartsWith("--", StringComparison.Ordinal))?.ToLowerInvariant();
if (mode is not ("worker-a" or "worker-b" or "enqueue" or "inspect"))
{
PrintCommands();
return;
}
// Keep provider/configuration switches, but do not pass the bare role to .NET's configuration parser.
var hostArgs = args.Where(a =>
!string.Equals(a, mode, StringComparison.OrdinalIgnoreCase) && a is not "--brief" and not "--pause" and not "--all-columns"
)
.ToArray();
var builder = Host.CreateApplicationBuilder(hostArgs);
var provider = LocalDatabase.ResolveProvider(builder.Configuration);
if (LocalDatabase.IsSqlite(provider))
{
Console.WriteLine("This lab intentionally requires PostgreSQL or SQL Server so two processes are real peer workers.");
Console.WriteLine("Set ACTA_LOCAL_PROVIDER=postgres (or sqlserver) and the matching ACTA_TEST_* connection string.");
PrintCommands();
return;
}
string sessionId;
if (mode == "worker-a")
{
sessionId = Guid.CreateVersion7().ToString("N");
File.WriteAllText(currentSessionFile, sessionId);
}
else if (mode is "worker-b" or "enqueue" or "inspect")
{
if (!File.Exists(currentSessionFile) || string.IsNullOrWhiteSpace(sessionId = File.ReadAllText(currentSessionFile).Trim()))
{
Console.WriteLine("No current crash-recovery session exists. Start worker-a first.");
return;
}
}
else
{
sessionId = "none";
}
var lab = new LearnLab(builder.Configuration, args);
builder.Services.AddSingleton(new WorkerRole(mode, sessionId));
builder.Services.UseActa(j =>
{
j.UseLocalDatabase(builder.Configuration, applyMigrations: mode == "worker-a");
if (mode is "worker-a" or "worker-b")
{
j.Run<WorkerCrashRecoveryJobs>(jobNamespace);
j.ConfigureOptions(o =>
{
// learn: 2. Shorten the heartbeat only to watch the recovery
// One knob shortens the whole coordination triple in proportion: a 1s beat gives a 4s lease
// and a 7s dead-worker window, which is what makes the kill below observable in seconds
// instead of minutes.
o.HeartbeatInterval = TimeSpan.FromSeconds(1);
// learn: end
o.DeploymentVersion = $"{mode}:{sessionId[..8]}";
});
}
else
{
j.Reference<WorkerCrashRecoveryJobs>(jobNamespace);
}
});
using var host = builder.Build();
await host.StartAsync();
var jobs = host.Services.GetRequiredService<IJobs>();
if (mode == "enqueue")
{
var enqueued = await jobs.EnqueueAsync(new CrashRecoveryProbe($"probe-{sessionId}", sessionId), o =>
o.DeduplicationKey(DeduplicationKey.ForDefinition("crash-recovery-probe", $"crash-recovery-probe-{sessionId}"))
);
File.WriteAllText(currentJobFile, enqueued.JobRef.ToString());
Console.WriteLine($"Enqueued {enqueued.JobRef}. Worker A should claim it and terminate at the marked failure point.");
await host.StopAsync();
return;
}
if (mode == "worker-a")
{
Console.WriteLine($"Worker A is ready for session {sessionId[..8]}. In another terminal run enqueue; this process will fail fast.");
await host.WaitForShutdownAsync();
return;
}
if (!File.Exists(currentJobFile) || !JobRef.TryParse(File.ReadAllText(currentJobFile).Trim(), out var currentJobRef))
{
Console.WriteLine("No current probe marker exists. Start worker-a and run enqueue first.");
await host.StopAsync();
return;
}
var probe = await jobs.GetAsync(currentJobRef);
if (probe is null)
{
Console.WriteLine($"The marker names {currentJobRef}, but that job is not in this database. Use the same provider for every command.");
await host.StopAsync();
return;
}
if (mode == "inspect")
{
await ShowStateAsync(lab, probe.JobId, sessionId);
await host.StopAsync();
return;
}
await ShowStateAsync(lab, probe.JobId, sessionId);
Console.WriteLine("Worker B is live. Waiting for A's short lab lease to lapse; B's heartbeat takes the recovery pass once a lease...");
using var recoveryTimeout = new CancellationTokenSource(TimeSpan.FromSeconds(30));
while ((await jobs.GetAsync(JobLookup.ById(probe.JobId), recoveryTimeout.Token))?.Status.IsFinished != true)
{
await Task.Delay(100, recoveryTimeout.Token);
}
await ShowStateAsync(lab, probe.JobId, sessionId);
await host.StopAsync();
static async Task ShowStateAsync(LearnLab lab, long jobId, string sessionId)
{
await lab.ShowAllAsync(
"Explore the complete recoverable job record",
"""
SELECT *
FROM jobs_view
WHERE job_id = @jobId
""",
new { jobId }
);
await lab.ShowAsync(
"The database is the shared authority for ownership and recovery",
"""
SELECT job_ref, job_status, execution_number, failure_count, leased_by_worker_id, leased_by_worker_host_name, lease_expires_at_utc
FROM jobs_view
WHERE job_id = @jobId
""",
new { jobId }
);
await lab.ShowAsync(
"Worker heartbeats are ordinary inspectable rows",
"""
SELECT worker_id, worker_status, deployment_version, host_name, process_id, last_seen_at_utc
FROM workers_view
WHERE namespace_name = @jobNamespace AND deployment_version IN (@workerADeployment, @workerBDeployment)
ORDER BY worker_id
""",
new
{
jobNamespace = "worker-crash-recovery",
workerADeployment = $"worker-a:{sessionId[..8]}",
workerBDeployment = $"worker-b:{sessionId[..8]}",
}
);
await lab.ShowAsync(
"The event ledger retains claim loss and replay",
"""
SELECT event, execution_number, details_text, worker_id
FROM events_view
WHERE entity = 'job' AND entity_id = @jobId
ORDER BY event_id
""",
new { jobId }
);
}
static void PrintCommands()
{
Console.WriteLine("Run in this order against one PostgreSQL or SQL Server database:");
Console.WriteLine(" dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- worker-a");
Console.WriteLine(" dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- enqueue");
Console.WriteLine(" dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- inspect");
Console.WriteLine(" dotnet run --project learn/700-topology-and-deployment/705-worker-crash-recovery -- worker-b");
}
namespace Acta.Learn.WorkerCrashRecovery
{
public sealed record WorkerRole(string Name, string SessionId);
public sealed record CrashRecoveryProbe(string ProbeId, string? SessionId = null);
public sealed class CrashRecoveryJob(WorkerRole role)
{
// learn: 1. Give the job a retry budget, because a crash spends an attempt
// Nothing in the handler recovers the job. A worker that dies stops heartbeating, the lease on its job lapses,
// and another worker reclaims the job as a failed attempt and runs it again.
[Job("crash-recovery-probe", MaxAttempts = 3, Backoff = "0s")]
// learn: end
public async Task Handle(CrashRecoveryProbe input, CancellationToken ct)
{
Console.WriteLine($"[{role.Name}] executing {input.ProbeId}; ordinary handler code is at-least-once.");
if (role.Name == "worker-a" && input.SessionId == role.SessionId)
{
Console.WriteLine("[worker-a] worst moment reached: side effect may have happened; process disappears before completion.");
Console.Out.Flush();
Environment.FailFast("Intentional worker-a crash for the SQL recovery lab.");
}
if (role.Name == "worker-a")
{
var staleSession = input.SessionId is { Length: >= 8 } ? input.SessionId[..8] : "legacy";
Console.WriteLine($"[worker-a] stale probe from session {staleSession} completed without crashing the current session.");
return;
}
await Task.Delay(100, ct);
Console.WriteLine("[worker-b] replay completed; production code would need idempotency or durable steps.");
}
}
}