Add durable execution Step + Wait end-to-end

[GarrettBeatty]

#2216

What

This is the first real end-to-end slice of the Amazon.Lambda.DurableExecution SDK. After this PR, you can write a workflow that calls StepAsync and WaitAsync, deploy it as a Lambda, and have it run against the actual durable-execution service with replay-aware checkpointing.

Public API introduced:

Type	Purpose
DurableFunction.WrapAsync	Entry point; handles the durable-execution envelope (input hydration, output construction, status mapping). Four overloads: typed-input/typed-output and typed-input/void, each with and without an explicit IAmazonLambda client.
IDurableContext	User-facing context: StepAsync (typed-result and void overloads), WaitAsync, LambdaContext, Logger, ExecutionContext.
StepConfig	Per-step configuration. Empty in this PR — RetryStrategy and Semantics get wired in #2363.
DurableExecutionException	Base exception type for durable-execution failures.

Step input/output and workflow envelope serialization both go through the ILambdaSerializer registered on ILambdaContext.Serializer (added in #2378)

Why

Durable execution lets a Lambda function suspend and resume across invocations by checkpointing each side-effect to the service. This PR lays down the minimum needed to build everything that comes after — retries, callbacks, parallelism, the Annotations integration, the test runner package — all of it builds on the Step + Wait primitives and the replay machinery here.

I kept the scope narrow on purpose. Anything that does not block real-Lambda execution is pushed to follow-up PRs (see Out of scope) so this stays reviewable.

How

The runtime runs a Task.WhenAny race in DurableExecutionHandler between the user's workflow task and a suspension signal. Every StepAsync / WaitAsync call goes to a per-operation class (StepOperation / WaitOperation) that checks ExecutionState (built from operations the service replayed) before deciding what to do:

• Replay hit — the operation has a SUCCEEDED / FAILED record from a prior invocation. Return the cached result (or rethrow) without re-running user code.
• Cache miss — run user code, enqueue a SUCCEED/FAIL checkpoint, return.
• Wait pending — the wait was scheduled previously but has not expired. Call TerminationManager.SuspendAndAwait to win the WhenAny race, returning Pending so the service re-invokes us when the timer fires.

Replay determinism comes from OperationIdGenerator, which produces stable IDs from the workflow's call sequence so the same step always lands on the same record across invocations.

Serialization is unified on ILambdaContext.Serializer. Both WrapAsync (workflow input/output envelope) and StepOperation (step result checkpointing) read the registered ILambdaSerializer off the per-invocation context and use it for all JSON conversion. AOT-safety is determined entirely by which serializer the caller registers with LambdaBootstrapBuilder.Create — register SourceGeneratorLambdaJsonSerializer<TContext> for AOT/trimmed deployments, or DefaultLambdaJsonSerializer for reflection-based scenarios. StepConfig is intentionally empty in this PR — it's the configuration carrier for RetryStrategy (#2363) and future per-step knobs.

Checkpoint flushing goes through CheckpointBatcher, a Channel-based queue with a single background worker. Each EnqueueAsync returns a Task that completes when the worker has flushed the containing batch to the service. WrapAsyncCore sets up the batcher, threads it into DurableContext, and awaits DrainAsync() before returning to Lambda. Defaults match the Java SDK (MaxBatchOperations = 200, MaxBatchBytes = 750 KB, FlushInterval = 0 — flush as soon as the queue drains). There's a TODO for the async-flush overload that Map/Parallel/Child Context will eventually need.

Key files:

• DurableFunction.cs — envelope + replay-state hydration + batcher lifecycle; pulls ILambdaSerializer from ILambdaContext.Serializer
• DurableContext.cs — facade; constructs the right per-operation class
• Internal/DurableOperation.cs — abstract base with StartAsync / ReplayAsync template methods
• Internal/StepOperation.cs / Internal/WaitOperation.cs — per-op replay logic; StepOperation serializes step results via the registered ILambdaSerializer
• Internal/CheckpointBatcher.cs / Internal/CheckpointBatcherConfig.cs — checkpoint queue + worker
• Internal/ExecutionState.cs — operation lookup + replay-mode flag
• Internal/OperationIdGenerator.cs — deterministic IDs
• Internal/TerminationManager.cs — suspension trigger
• Internal/DurableExecutionHandler.cs — the WhenAny race
• Services/LambdaDurableServiceClient.cs — service client wrapper

Testing

Unit tests in Amazon.Lambda.DurableExecution.Tests cover: enums, exceptions, models, OperationIdGenerator, TerminationManager, ExecutionState, the handler race, both Step and Wait paths through DurableContext, DurableFunction.WrapAsync, and CheckpointBatcher (enqueue/flush, batching within window, overflow splitting, error propagation, drain, dispose, token updates, concurrency).

End-to-end integration tests in Amazon.Lambda.DurableExecution.IntegrationTests build each test workflow into a Docker container, deploy it as a real Lambda on provided.al2023, and run against the durable-execution service:

• StepWaitStep — basic Step → Wait → Step sequence
• MultipleSteps — several sequential steps
• WaitOnly — wait without any steps
• LongerWait — wait that spans multiple invocations
• ReplayDeterminism — verifies stable operation IDs across replays
• StepFails — verifies a failed step surfaces correctly

The AOT smoke test publishes the sample workflow with the source-generator serializer and produces zero IL2026 / IL3050 warnings.

Out of scope (follow-up PRs)

• IRetryStrategy, ExponentialRetryStrategy, retry decision factories, StepConfig.RetryStrategy / StepConfig.Semantics (Adds retry support to the Amazon.Lambda.DurableExecution #2363)
• StepException and the per-step failure exception type
• DurableLogger replay-suppression (currently returns NullLogger)
• Callbacks, InvokeAsync, ParallelAsync, MapAsync, RunInChildContextAsync, WaitForConditionAsync — the interface intentionally does not declare these yet
• Annotations source-generator integration (including a [DurableExecution] marker attribute)
• DurableTestRunner / Amazon.Lambda.DurableExecution.Testing package
• dotnet new lambda.DurableFunction blueprint

[GarrettBeatty]

needed to add another overload here for native AOT

[normj]

I think we need to revisit the JSON serializing and having a separate ICheckpointSerializer instead of reusing the ILambdaSerializer but I'm good with that being a separate PR.

[normj]

Since the SDK call can fail for a variety of reasons like permissions. I wonder if we should wrap the SDK call in a try/catch and then wrap the SDK's exception in a DurableExecutionException and then you can give an error message with the context of what the SDK was being used for with the SDK exception message. I'm wanting users to not see an SDK stack trace and exception without any context of what it was being used for.

[GarrettBeatty]

updated

[normj]

Since the SDK call can fail for a variety of reasons like permissions. I wonder if we should wrap the SDK call in a try/catch and then wrap the SDK's exception in a DurableExecutionException and then you can give an error message with the context of what the SDK was being used for with the SDK exception message. I'm wanting users to not see an SDK stack trace and exception without any context of what it was being used for.

[GarrettBeatty]

updated

[normj]

typo iff

[GarrettBeatty]

should say if and only if

[normj]

Wasn't sure if threading for this is a concern when you get to the parallel steps. If so then maybe this line should be:

var counter = System.Threading.Interlocked.Increment(ref _counter);

[GarrettBeatty]

fixed

[normj]

Since you have the SDK you could potentially use the SDK's ToHex method Amazon.Util.AWSSDKUtils.ToHex().

[GarrettBeatty]

fixed

[normj]

Any reason you can't use the enum OperationAction.FAIL instead of the string literal?

[GarrettBeatty]

fixed

[normj]

Use enum

[GarrettBeatty]

fixed

[Copilot]

Pull request overview

Copilot reviewed 72 out of 72 changed files in this pull request and generated 2 comments.

[philasmar]

Well-architected first slice of the durable execution SDK. The separation of concerns (DurableFunction entry point → DurableExecutionHandler race → per-operation classes →
CheckpointBatcher) is clean, replay semantics align with the Java/Python/JS reference SDKs, and test coverage is extensive. A few items worth addressing:

---

Important

1. TrackReplay is O(n) per operation call, making replay O(n²)

ExecutionState.TrackReplay iterates all operations on every call to check if all terminal non-EXECUTION ops have been visited. For a workflow with hundreds of steps, this
becomes noticeable.

Consider precomputing the count at LoadFromCheckpoint time and decrementing on each visit:

private int _remainingReplayOps;

// in LoadFromCheckpoint:
_remainingReplayOps = _operations.Values
    .Count(op => op.Type != OperationTypes.Execution && IsTerminalStatus(op.Status));

// in TrackReplay:
if (_visitedOperations.Add(operationId)
    && _operations.TryGetValue(operationId, out var op)
    && op.Type != OperationTypes.Execution
    && IsTerminalStatus(op.Status))
{
    if (--_remainingReplayOps <= 0)
        _isReplaying = false;
}

2. ExtractUserPayload silently returns default! on missing EXECUTION operation

If the service envelope arrives without an EXECUTION-type operation (malformed input, wire issue), the workflow receives null/default input with no signal. This would surface
as a confusing NullReferenceException deep in user code rather than a clear "durable execution envelope is malformed" error. Consider throwing DurableExecutionException when no
EXECUTION op is found and TInput is a non-nullable reference type.

3. No CancellationToken forwarding in WrapAsyncCore

The state-pagination loop (while (!string.IsNullOrEmpty(nextMarker))) and DrainAsync don't observe any cancellation token. If the Lambda runtime signals shutdown mid-hydration,
these operations could delay termination beyond the runtime's patience. Consider threading lambdaContext's remaining time through as a CancellationToken.

4. UpperSnakeCaseEnumConverter is public

This converter lives in the Amazon.Lambda.DurableExecution namespace (not Internal) and is public sealed. It seems like an implementation detail that shouldn't be on the public
API surface — locking in its behavior as a public contract limits future flexibility. Consider making it internal or moving it to the Internal namespace.

---

Nits

• DurableFunctionDeployment.ShortId() uses [..Math.Min(8, 32)] — Math.Min always evaluates to 8; could just be [..8].
• CheckpointBatcherConfig.MaxBatchBytes is declared but unenforced (documented via TODO — confirming this is intentional for this PR scope).
• The DurableExecutionHandler.RunAsync XML doc body references RetryScheduled / CallbackPending / InvokePending patterns that aren't wired yet. Minor doc-vs-shipped-code
  consistency point.

---

Questions

• IDurableContext.Logger always returns NullLogger.Instance — are there plans to wire DurableLogger before GA, or is this acceptable for the initial release?

[GarrettBeatty]

IDurableContext.Logger always returns NullLogger.Instance — are there plans to wire DurableLogger before GA, or is this acceptable for the initial release?

this will be in follow up PR. will address your other comments though

[GarrettBeatty]

noo CancellationToken forwarding in WrapAsyncCore The state-pagination loop (while (!string.IsNullOrEmpty(nextMarker))) and DrainAsync don't observe any cancellation token. If the Lambda runtime signals shutdown mid-hydration,
these operations could delay termination beyond the runtime's patience. Consider threading lambdaContext's remaining time through as a CancellationToken.

i dont think this matters too much, im going to keep this as-is for now

[GarrettBeatty]

@philasmar addressed your comments

[normj]

Can you update the design doc now that for Native AOT we are relying on the user to properly configure ILambdaSerializer?

[GarrettBeatty]

850c901

[normj]

Don't say "(typically configured via LambdaBootstrapBuilder.Create(handler, serializer)" because most users use the class library programming model which relies on the assembly attribute.

[GarrettBeatty]

81380c9

[normj]

You need to rework the wording here because must user use the class library programming model which likes on the assembly attribute.

[GarrettBeatty]

81380c9

[normj]

You need to rework the wording here because must user use the class library programming model which likes on the assembly attribute.

[GarrettBeatty]

81380c9

[normj]

I don't think internal properties will get set for serialization using a JsonSerializerContext when that JsonSerializerContext is defined in a different assembly then the type with internal properties. You should double check my thinking but we might need to handle this different to support source generator based serialization.

[GarrettBeatty]

i just checked this and you are right

[GarrettBeatty]

72f8824 made it public

[philasmar]

Approved, assuming you address Norm's comments