Acta. Durable execution for .NET

learn actadecide what happens when work fails

How to set how long a job waits between retries

Backoff on the job sets how the wait between retries grows: 1s..8s x2 exact starts at one second, doubles with each failure, and stops growing at eight.

A how-to for Acta, durable execution for .NET. New to Acta? Run your first durable job in five minutes.

Checked every release learn/300-failure-and-recovery/312-backoff-curve
Read as

1Write the backoff on the job

Exponential backoff: delays double per failure (1s, 2s, 4s) up to MaxAttempts=4. exact makes the curve deterministic so NextRunAtUtc gaps are observable.

backoff-curve.csexcerpt
public static class FlakyTransferJob
{
    [Job("flaky-transfer", MaxAttempts = 4, Backoff = "1s..8s x2 exact")]
    public static Task Handle(FlakyTransfer input) =>
        throw new InvalidOperationException($"Transfer {input.Id} failed: upstream unavailable.");
}

2Run it and check the output

From a clone of the Acta repository, with the .NET 10 SDK:

Terminal
dotnet run --project learn/300-failure-and-recovery/312-backoff-curve
Its output must contain
failure #3: next retry at
job ended: status=Failed failureCount=4

Checked every release. Before each release, Acta Verify runs this program and checks that its output contains these lines.

Remember

  • With no Backoff, retries back off from a minute to a day over 15 attempts, about 4.4 days in all.

The whole program

Every file of learn/300-failure-and-recovery/312-backoff-curve, as it is compiled.

UseLocalDatabase is a helper of the Acta repository, not of the packages. In your own app, register the database with UseSqlite, UsePostgres or UseSqlServer, as in how to run your first job.

backoff-curve.cs74 lines
backoff-curve.cs
// How to set how long a job waits between retries
// `Backoff` on the job sets how the wait between retries grows: `1s..8s x2 exact` starts at one second, doubles with
// each failure, and stops growing at eight.
using Acta;
using Acta.Learn.BackoffCurve;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

var builder = Host.CreateApplicationBuilder(args);

builder.Services.UseActa(j =>
{
    j.UseLocalDatabase(builder.Configuration);
    j.Run<BackoffCurveJobs>("backoff-curve");
});

// Quiet expected-failure noise; the handler always throws on purpose.
builder.Logging.AddFilter("Acta", LogLevel.Error);

using var host = builder.Build();
await host.StartAsync();

var jobs = host.Services.GetRequiredService<IJobs>();

Console.WriteLine("Enqueueing a flaky job - it always throws to exhaust MaxAttempts=4.");
Console.WriteLine(
    "Curve: initial=1s, multiplier=2, max=8s, jitter=0 -> 3 retry gaps ~1s, 2s, 4s after MaxAttempts=4 is exhausted, then Failed."
);

var outcome = await jobs.EnqueueAsync(new FlakyTransfer("txn-1"));

// Poll until the job reaches a terminal status, printing NextRunAtUtc after each rearm.
int lastFailureCount = 0;
while (true)
{
    await Task.Delay(200);
    var snapshot = await jobs.GetAsync(outcome);
    if (snapshot is null)
        break;

    if (snapshot.FailureCount != lastFailureCount && snapshot.NextRunAtUtc.HasValue)
    {
        var gap = snapshot.NextRunAtUtc.Value - DateTime.UtcNow;
        Console.WriteLine(
            $"failure #{snapshot.FailureCount}: next retry at {snapshot.NextRunAtUtc.Value:HH:mm:ss.fff} UTC (in ~{gap.TotalSeconds:F1}s)"
        );
        lastFailureCount = snapshot.FailureCount;
    }

    if (snapshot.Status is JobStatusCode.Failed or JobStatusCode.Succeeded or JobStatusCode.Cancelled)
    {
        Console.WriteLine($"job ended: status={snapshot.Status} failureCount={snapshot.FailureCount}");
        break;
    }
}

await host.StopAsync();

namespace Acta.Learn.BackoffCurve
{
    public sealed record FlakyTransfer(string Id);

    public static class FlakyTransferJob
    {
        // learn: Write the backoff on the job
        // Exponential backoff: delays double per failure (1s, 2s, 4s) up to MaxAttempts=4.
        // exact makes the curve deterministic so NextRunAtUtc gaps are observable.
        [Job("flaky-transfer", MaxAttempts = 4, Backoff = "1s..8s x2 exact")]
        // learn: end
        public static Task Handle(FlakyTransfer input) =>
            throw new InvalidOperationException($"Transfer {input.Id} failed: upstream unavailable.");
    }
}

Where to go next

New to Acta?

Acta is durable execution for .NET: workflows and jobs that survive restarts, as a library inside your application with its durable state in your SQL database.