Acta. Durable execution for .NET

learn actasplit work across child jobs

How to fan out over a list and merge the results

MapAsync starts one child job per item of a list, waits for them all, and hands back each item's result for the parent to merge.

A how-to for Acta, durable execution for .NET. New to Acta? Run your first durable job in five minutes.

Checked every release learn/200-durable-execution/212-fan-out-join
Read as

1Map the list onto child jobs

MapAsync starts one child per chunk and waits for all of them. Each child is keyed by its range, so a replay finds the same children by name. A child given as its generated call types the map, so each item's outcome carries the child's result and the merge reads no child by id.

fan-out-join.csexcerpt
[Job("add-numbers")]
public async Task<SumResult> Handle(AddNumbers request, JobContext context, CancellationToken ct)
{
    var chunks = await context.MapAsync(
        "chunk",
        request.Chunks,
        itemKey: chunk => $"{chunk.From}-{chunk.To}",
        child: chunk => FanOutJoinJobs.SumRange(chunk),
        ct
    );

    if (!chunks.Succeeded)
    {
        await context.FailAsync("a chunk failed", ct);
    }

    return new SumResult(chunks.Items.Sum(item => item.Outcome.ValueOrThrow().Value));
}

2Stop if a child failed, then merge the results

The map does not throw for a failed child; the parent asks, and decides. FailAsync ends the handler there, so the merge runs only when every chunk succeeded.

fan-out-join.csexcerpt
[Job("add-numbers")]
public async Task<SumResult> Handle(AddNumbers request, JobContext context, CancellationToken ct)
{
    var chunks = await context.MapAsync(
        "chunk",
        request.Chunks,
        itemKey: chunk => $"{chunk.From}-{chunk.To}",
        child: chunk => FanOutJoinJobs.SumRange(chunk),
        ct
    );

    if (!chunks.Succeeded)
    {
        await context.FailAsync("a chunk failed", ct);
    }
 
    return new SumResult(chunks.Items.Sum(item => item.Outcome.ValueOrThrow().Value));
}

3Run it and check the output

From a clone of the Acta repository, with the .NET 10 SDK:

Terminal
dotnet run --project learn/200-durable-execution/212-fan-out-join
Its output must contain
sum 1..1000 = 500500

Checked every release. Before each release, Acta Verify runs this program and checks that its output contains these lines.

Remember

  • A child's name on the job reads is its DeduplicationKey: {map}/{key} for a map's child. List a parent's children with acta.Ledger.ListJobsAsync(new ListJobsQuery(Parent: parent.JobRef)).
  • A failed or cancelled child never throws in the parent by itself. The wait answers an outcome: call ThrowIfAnyFailed() to fail with it, or branch on Failed.
  • The parent wakes only when the child it is waiting on lands, so it can run several times before the group is done. Keep the work before the wait in a step.
  • A group does not limit how many of its children run at once; that is ConcurrencyLimit on the child's definition.

The whole program

Every file of learn/200-durable-execution/212-fan-out-join, as it is compiled.

UseLocalDatabase is a helper of the Acta repository, not of the packages. In your own app, register the database with UseSqlite, UsePostgres or UseSqlServer, as in how to run your first job.

fan-out-join.cs85 lines
fan-out-join.cs
// How to fan out over a list and merge the results
// `MapAsync` starts one child job per item of a list, waits for them all, and hands back each item's result for the
// parent to merge.
using Acta;
using Acta.Learn.FanOutJoin;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateApplicationBuilder(args);

builder.Services.UseActa(j =>
{
    j.UseLocalDatabase(builder.Configuration);
    j.Run<FanOutJoinJobs>("fan-out-join");
});

using var host = builder.Build();
await host.StartAsync();

var jobs = host.Services.GetRequiredService<IJobs>();

// Map-reduce as child jobs: replay finds the same children by name, so no chunk is summed twice.
// Compare 601-fan-out, where the children are fire-and-forget and nothing merges.
var outcome = await jobs.CallAsync(FanOutJoinJobs.AddNumbers(new AddNumbers([new(1, 250), new(251, 500), new(501, 750), new(751, 1000)])));

Console.WriteLine($"sum 1..1000 = {outcome.Value!.Total}");

await host.StopAsync();

namespace Acta.Learn.FanOutJoin
{
    public sealed record AddNumbers(SumRange[] Chunks);

    public sealed record SumRange(long From, long To);

    public sealed record PartialSum(long Value);

    public sealed record SumResult(long Total);

    public sealed class AddNumbersJobs
    {
        [Job("add-numbers")]
        public async Task<SumResult> Handle(AddNumbers request, JobContext context, CancellationToken ct)
        {
            // learn: 1. Map the list onto child jobs
            // MapAsync starts one child per chunk and waits for all of them. Each child is keyed by its range,
            // so a replay finds the same children by name. A child given as its generated call types the map,
            // so each item's outcome carries the child's result and the merge reads no child by id.
            var chunks = await context.MapAsync(
                "chunk",
                request.Chunks,
                itemKey: chunk => $"{chunk.From}-{chunk.To}",
                child: chunk => FanOutJoinJobs.SumRange(chunk),
                ct
            );
            // learn: end

            // learn: 2. Stop if a child failed, then merge the results
            // The map does not throw for a failed child; the parent asks, and decides. `FailAsync` ends the handler
            // there, so the merge runs only when every chunk succeeded.
            if (!chunks.Succeeded)
            {
                await context.FailAsync("a chunk failed", ct);
            }
 
            return new SumResult(chunks.Items.Sum(item => item.Outcome.ValueOrThrow().Value));
            // learn: end
        }

        [Job("sum-range")]
        public async Task<PartialSum> Sum(SumRange range, CancellationToken ct)
        {
            await Task.Delay(150, ct);

            var sum = 0L;
            for (var n = range.From; n <= range.To; n++)
            {
                sum += n;
            }

            Console.WriteLine($"chunk {range.From}..{range.To} = {sum}");
            return new PartialSum(sum);
        }
    }
}

Where to go next

New to Acta?

Acta is durable execution for .NET: workflows and jobs that survive restarts, as a library inside your application with its durable state in your SQL database.