Acta. Durable execution for .NET

learn actasplit work across child jobs

How to fan out to child jobs with Map or WaitChildren

Two ways to fan out: EnqueueChildAsync with WaitChildrenAsync for a few named children, and MapAsync for one child per item of a list.

A how-to for Acta, durable execution for .NET. New to Acta? Run your first durable job in five minutes.

Checked every release learn/200-durable-execution/215-map-wait-children
Read as

1Enqueue a few named children and wait for them together

Independent analyzes, none needs another's result: enqueue each, and they run at once.

map-wait-children.csexcerpt
var exif = await context.EnqueueChildAsync("exif", new ExtractExif(photo.AssetId, photo.SourceUrl), ct);
var blurhash = await context.EnqueueChildAsync("blurhash", new ComputeBlurhash(photo.AssetId, photo.SourceUrl), ct);
var moderation = await context.EnqueueChildAsync("moderation", new ScanModeration(photo.AssetId, photo.SourceUrl), ct);
var analyze = await context.WaitChildrenAsync([exif, blurhash, moderation], ct);

analyze.ThrowIfAnyFailed();

2Map a list onto children, one per item

Map: one resize per width, keyed by width so replay dedupes onto the same children.

map-wait-children.csexcerpt
analyze.ThrowIfAnyFailed();

int[] widths = [256, 512, 1024, 2048];
var resized = await context.MapAsync(
    "resize",
    widths,
    itemKey: width => width,
    child: width => new Resize(photo.AssetId, photo.SourceUrl, width),
    ct
);

var renditionsReady = resized.Items.Count(i => i.Outcome.Succeeded);

3Count what landed instead of failing the parent

Resizes are best effort: report how many landed rather than failing the photo.

map-wait-children.csexcerpt
var renditionsReady = resized.Items.Count(i => i.Outcome.Succeeded);

// Two independent side effects, enqueued and waited for the same way.
var cdn = await context.EnqueueChildAsync("publish-cdn", new PublishCdn(photo.AssetId), ct);
var index = await context.EnqueueChildAsync("update-index", new UpdateIndex(photo.AssetId), ct);
var published = await context.WaitChildrenAsync([cdn, index], ct);

4Run it and check the output

From a clone of the Acta repository, with the .NET 10 SDK:

Terminal
dotnet run --project learn/200-durable-execution/215-map-wait-children
Its output must contain
photo asset-42: 4 renditions ready, published=True

Checked every release. Before each release, Acta Verify runs this program and checks that its output contains these lines.

Remember

  • A child's name on the job reads is its DeduplicationKey: the call's name for EnqueueChildAsync, {map}/{key} for a map's child.
  • A group waits for every child. It never fails fast and never cancels siblings.
  • The parent wakes only when the child it is waiting on lands, so it can run several times before the group is done. Keep the work before the wait in a step.

The whole program

Every file of learn/200-durable-execution/215-map-wait-children, as it is compiled.

UseLocalDatabase is a helper of the Acta repository, not of the packages. In your own app, register the database with UseSqlite, UsePostgres or UseSqlServer, as in how to run your first job.

map-wait-children.cs135 lines
map-wait-children.cs
// How to fan out to child jobs with Map or WaitChildren
// Two ways to fan out: `EnqueueChildAsync` with `WaitChildrenAsync` for a few named children, and `MapAsync` for one
// child per item of a list.
using Acta;
using Acta.Learn.MapWaitChildren;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateApplicationBuilder(args);

builder.Services.UseActa(j =>
{
    j.UseLocalDatabase(builder.Configuration);
    j.Run<MapWaitChildrenJobs>("photos");
});

using var host = builder.Build();
await host.StartAsync();

var jobs = host.Services.GetRequiredService<IJobs>();

// Children are ordinary jobs plus child-completion latches: replay finds the same children by name and
// enqueues none twice; a child that had not finished runs on as its own job.
//   EnqueueChildAsync + WaitChildrenAsync - a few named children at once,
//   MapAsync                              - one child per item, keyed by a stable key.
var outcome = await jobs.CallAsync(
    MapWaitChildrenJobs.ProcessPhoto(new ProcessPhoto("asset-42", "https://example.com/photos/asset-42/original.jpg"))
);

var result = outcome.Value!;
Console.WriteLine($"photo {result.AssetId}: {result.RenditionsReady} renditions ready, published={result.Published}");

await host.StopAsync();

namespace Acta.Learn.MapWaitChildren
{
    public sealed record ProcessPhoto(string AssetId, string SourceUrl);

    public sealed record PhotoResult(string AssetId, int RenditionsReady, bool Published);

    public sealed record ExtractExif(string AssetId, string SourceUrl);

    public sealed record ComputeBlurhash(string AssetId, string SourceUrl);

    public sealed record ScanModeration(string AssetId, string SourceUrl);

    public sealed record Resize(string AssetId, string SourceUrl, int Width);

    public sealed record PublishCdn(string AssetId);

    public sealed record UpdateIndex(string AssetId);

    public sealed class PhotoJobs
    {
        [Job("process-photo")]
        public async Task<PhotoResult> Process(ProcessPhoto photo, JobContext context, CancellationToken ct)
        {
            // learn: 1. Enqueue a few named children and wait for them together
            // Independent analyzes, none needs another's result: enqueue each, and they run at once.
            var exif = await context.EnqueueChildAsync("exif", new ExtractExif(photo.AssetId, photo.SourceUrl), ct);
            var blurhash = await context.EnqueueChildAsync("blurhash", new ComputeBlurhash(photo.AssetId, photo.SourceUrl), ct);
            var moderation = await context.EnqueueChildAsync("moderation", new ScanModeration(photo.AssetId, photo.SourceUrl), ct);
            var analyze = await context.WaitChildrenAsync([exif, blurhash, moderation], ct);
            // learn: end

            analyze.ThrowIfAnyFailed();

            // learn: 2. Map a list onto children, one per item
            // Map: one resize per width, keyed by width so replay dedupes onto the same children.
            int[] widths = [256, 512, 1024, 2048];
            var resized = await context.MapAsync(
                "resize",
                widths,
                itemKey: width => width,
                child: width => new Resize(photo.AssetId, photo.SourceUrl, width),
                ct
            );
            // learn: end

            // learn: 3. Count what landed instead of failing the parent
            // Resizes are best effort: report how many landed rather than failing the photo.
            var renditionsReady = resized.Items.Count(i => i.Outcome.Succeeded);
            // learn: end

            // Two independent side effects, enqueued and waited for the same way.
            var cdn = await context.EnqueueChildAsync("publish-cdn", new PublishCdn(photo.AssetId), ct);
            var index = await context.EnqueueChildAsync("update-index", new UpdateIndex(photo.AssetId), ct);
            var published = await context.WaitChildrenAsync([cdn, index], ct);

            return new PhotoResult(photo.AssetId, renditionsReady, published.Succeeded);
        }

        [Job("exif")]
        public async Task ReadExif(ExtractExif request, CancellationToken ct)
        {
            await Task.Delay(50, ct);
            Console.WriteLine($"exif read for {request.AssetId}");
        }

        [Job("blurhash")]
        public async Task Blurhash(ComputeBlurhash request, CancellationToken ct)
        {
            await Task.Delay(50, ct);
            Console.WriteLine($"blurhash computed for {request.AssetId}");
        }

        [Job("moderation")]
        public async Task Moderate(ScanModeration request, CancellationToken ct)
        {
            await Task.Delay(50, ct);
            Console.WriteLine($"moderation cleared for {request.AssetId}");
        }

        [Job("resize")]
        public async Task Scale(Resize request, CancellationToken ct)
        {
            await Task.Delay(50, ct);
            Console.WriteLine($"resized {request.AssetId} to {request.Width}px");
        }

        [Job("publish-cdn")]
        public async Task Publish(PublishCdn request, CancellationToken ct)
        {
            await Task.Delay(50, ct);
            Console.WriteLine($"published {request.AssetId} to cdn");
        }

        [Job("update-index")]
        public async Task Index(UpdateIndex request, CancellationToken ct)
        {
            await Task.Delay(50, ct);
            Console.WriteLine($"search index updated for {request.AssetId}");
        }
    }
}

Where to go next

New to Acta?

Acta is durable execution for .NET: workflows and jobs that survive restarts, as a library inside your application with its durable state in your SQL database.