Adds retry support to the Amazon.Lambda.DurableExecution

[GarrettBeatty]

#2216

What

Adds retry support to Amazon.Lambda.DurableExecution on top of #2360. A step that throws can now be retried with configurable backoff and jitter. The Lambda suspends between attempts and is re-invoked by the service when the retry timer fires, so compute is not billed during the wait.

Public API:

Type	Purpose
IRetryStrategy	Decides whether a failed step should retry, with what delay.
RetryDecision	Output of IRetryStrategy.ShouldRetry — ShouldRetry flag plus Delay.
RetryStrategy	Static factory: Default, Transient, None, Exponential(...), FromDelegate(...).
JitterStrategy	None / Half / Full for exponential backoff.
StepSemantics	AtLeastOncePerRetry (default) / AtMostOncePerRetry.
StepConfig.RetryStrategy, StepConfig.Semantics	Per-step retry configuration.
StepInterruptedException	Surfaced to user retry strategies when a prior attempt crashed mid-execution under AtMostOncePerRetry.

How

When a step throws, StepOperation.HandleStepFailureAsync calls IRetryStrategy.ShouldRetry(ex, attemptNumber). If the strategy says retry, the SDK writes a RETRY checkpoint with NextAttemptDelaySeconds and suspends — RunAsync returns Pending. The service holds the execution until the delay elapses, then re-invokes us. On replay, StepOperation.ReplayAsync sees the PENDING status and either re-suspends (timer not yet up) or re-executes the step with an incremented attempt counter.

 Invocation N                 Service                Invocation N+1
 ────────────                 ───────                ──────────────
 step throws
   │
   ▼
 ShouldRetry(ex, n)? ── no ──► FAIL checkpoint ──►  done
   │ yes
   ▼
 RETRY checkpoint   ─────────► schedule timer
 (NextAttemptDelay)            (delay seconds)
   │
   ▼
 SuspendAndAwait
 returns Pending    ─────────► ...wait...
                               timer fires ──────► ReplayAsync
                                                     │
                                                     ▼
                                                   PENDING + elapsed?
                                                     │
                                                     ▼
                                                   re-execute step
                                                   (attempt n+1)

AtLeastOncePerRetry (default)

For idempotent steps. The SDK writes the START checkpoint as fire-and-forget — user code runs immediately, no waiting on the network round-trip. On success the SDK writes SUCCEED. The cached result is returned on every subsequent replay; user code never re-executes after success.

If the Lambda crashes mid-attempt (before SUCCEED is recorded), replay sees STARTED and re-runs the same attempt under the same attempt counter. "At least once per retry" because the user's logic may run more than once for a single attemptNumber if the host dies between START and the terminal record. This is the right default when the step is safe to repeat (a read, an idempotent PUT, a calculation).

AtMostOncePerRetry

For non-idempotent steps (charging a card, sending an email, posting to a non-idempotent API). The SDK writes the START checkpoint synchronously — user code does not run until the service has acknowledged the START. The flush is correctness-load-bearing: a queued-but-unflushed START would be indistinguishable from "never ran" if the Lambda dies, and replay would re-execute the side effect.

If the Lambda crashes between user code and the SUCCEED flush, the service sees a STARTED record with no terminal counterpart on the next invocation. Instead of re-running the step, the SDK synthesizes a StepInterruptedException and routes it through the retry strategy — the strategy decides whether attempt N+1 should run or whether to give up. The user's code is invoked at most once per attemptNumber: either it ran to completion (SUCCEED/FAIL recorded), or the host died and that attempt is closed out as failed.

User retry strategies can pattern-match on StepInterruptedException to decide whether crash-recovery should retry or surface the failure — useful when the side effect is non-idempotent enough that you'd rather fail than risk replaying.

Choosing between them

• Default is AtLeastOncePerRetry. Use it unless your step has a side effect you can't safely repeat.
• Switch to AtMostOncePerRetry when the step calls a non-idempotent external API. The trade-off: one extra synchronous round-trip per attempt for the START flush.

ExponentialRetryStrategy supports max attempts, initial/max delay, backoff rate, jitter, and exception filtering by type or message regex. Built-in factories: Default (6 attempts, 5s/60s, 2× backoff, full jitter), Transient (3 attempts, 1s/5s, half jitter), None. RetryStrategy.FromDelegate(...) covers arbitrary policies, including ones that branch on StepInterruptedException.

Testing

21 new unit tests in Amazon.Lambda.DurableExecution.Tests (130 total, up from 109 in #2360):

• RetryStrategyTests (14) — exponential backoff math, jitter, max-attempt exhaustion, exception-type and message-pattern filtering, delegate strategies.
• DurableContextTests retry block (6) — checkpoint-and-suspend on retry, fail-without-strategy, retry exhaustion, future/past PENDING replay, AtMostOnce start-flush ordering, STARTED replay routing through the retry handler.

Integration tests in Amazon.Lambda.DurableExecution.IntegrationTests run end-to-end against the durable-execution service:

• RetryTest — flaky step recovers on attempt 3.
• RetryExhaustionTest — step always throws, exhausts retries, surfaces FAILED with the original exception.
• AtMostOnceCrashReplayTest — Lambda is killed mid-attempt under AtMostOncePerRetry; service re-invokes, SDK routes through retry strategy, attempt 2 succeeds.
• LongRetryChainTest — five failures across multiple invocations, validates the wire-format Attempt counter is monotonic and matches IStepContext.AttemptNumber.

Out of scope (follow-up PRs)

• MapAsync / ParallelAsync / RunInChildContextAsync / WaitForConditionAsync
• CallbackAsync, InvokeAsync
• DefaultJsonCheckpointSerializer
• DurableLogger replay-suppression (currently NullLogger)
• Annotations source-generator integration / [DurableExecution] attribute
• DurableTestRunner / Amazon.Lambda.DurableExecution.Testing package
• dotnet new lambda.DurableFunction blueprint

[Copilot]

Pull request overview

Builds on PR #2360 to add retry support to the Amazon.Lambda.DurableExecution SDK. Failed steps can now be retried with configurable backoff and jitter via service-mediated retries (the SDK checkpoints a RETRY operation and suspends the Lambda so the user is not billed during backoff). Adds at-most-once semantics for non-idempotent steps via a synchronously-flushed START checkpoint that allows crash detection on replay.

Changes:

• New public retry API: IRetryStrategy, RetryDecision, RetryStrategy factories (Default/Transient/None/Exponential/FromDelegate), JitterStrategy, StepSemantics, and StepConfig.RetryStrategy/StepConfig.Semantics.
• StepOperation adds PENDING (retry-timer) and STARTED (AtMostOnce crash-recovery) replay arms, a HandleStepFailureAsync decision tree, and START-checkpoint emission (sync for AtMostOnce, fire-and-forget for AtLeastOnce).
• 21 new unit tests plus integration-test updates asserting StepStarted events and richer history logging.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
Config/IRetryStrategy.cs	New strategy interface + RetryDecision struct
Config/RetryStrategy.cs	ExponentialRetryStrategy, DelegateRetryStrategy, JitterStrategy, StepSemantics, factories
Config/StepConfig.cs	Adds RetryStrategy and Semantics properties
Internal/StepOperation.cs	PENDING/STARTED replay arms, retry decision tree, START-checkpoint emission
Internal/TerminationManager.cs	Adds RetryScheduled termination reason
Internal/CheckpointBatcher.cs	Doc-only update describing fire-and-forget semantics
Tests/RetryStrategyTests.cs	14 unit tests for exponential math/jitter/filters/delegate
Tests/DurableContextTests.cs	6 retry/AtMostOnce/Pending replay tests
Tests/DurableFunctionTests.cs	Updated to assert START + SUCCEED + WAIT-START flat sequence
IntegrationTests/*.cs	Add StepStarted-event assertions; richer history dump in DurableFunctionDeployment

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[GarrettBeatty]

now that we are emitting START steps (which are needed for retries) we are asserting them in the IT tests

[GarrettBeatty]

in previous PR only SUCCEEDED or FAILED mattered. But now for replays, we need to keep track of how many times the function was executed, which is done via the number of STARTED steps.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

[GarrettBeatty]

these defaults and values match javascript

[GarrettBeatty]

just updating docs here and everywhere to remove stuff like "similar to python/js"

[Copilot]

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

Libraries/src/Amazon.Lambda.DurableExecution/RetryStrategy.cs:117

• RegexOptions.Compiled is not trim/AOT-safe and can trigger IL3050 (RequiresDynamicCode) warnings, which this project treats as errors. Consider removing Compiled or gating it on RuntimeFeature.IsDynamicCodeSupported, and prefer a non-compiled regex option set (e.g., CultureInvariant/IgnoreCase as needed).

        _backoffRate = backoffRate;
        _jitter = jitter;
        _retryableExceptions = retryableExceptions;
        _retryableMessagePatterns = retryableMessagePatterns?
            .Select(p => new Regex(p, RegexOptions.Compiled))
            .ToArray();

[GarrettBeatty]

Our class	SDK equivalent	Wire values match?	Why we have it
OperationTypes	Amazon.Lambda.OperationType	Yes ("STEP", "WAIT", etc.)	We need string constants for switch arms on Operation.Type (string from JSON deserialization). The SDK type is class OperationType : ConstantClass, not string — its constants can't appear in C# switch cases against a string.
OperationStatuses	Amazon.Lambda.OperationStatus	Yes — but the SDK has TIMED_OUT and we don't (we never observe it on the wire today).	Same reason as above.
OperationSubTypes	doesn't exist in the SDK	n/a	The SDK ships no OperationSubType constant class. The wire values are PascalCase ("Step", "Wait") and were verified against the JS SDK definition.
OperationAction	Amazon.Lambda.OperationAction	Yes	We use the SDK's class directly (e.g., OperationAction.START) because we're constructing an Amazon.Lambda.Model.OperationUpdate — its Action property is typed as OperationAction, so the SDK constant is the right thing to assign.

some of these constants i duplicated since we just need the string value. but since im adding more now, wondering if we want to use the ones in the model everywhere? it would require doing switch (op.Type) { case var t when t == OperationType.STEP.Value: ... }. V or similar everywhere to access the string value of each ConstantClass.

[GarrettBeatty]

this max batch bytes thing is still a todo item in another pr

[GarrettBeatty]

this change catches this scenario

1. Deploy v1: workflow calls StepAsync(name: "fetch") first.
2. The step fails, the SDK writes a RETRY checkpoint, the service stores Operation { Id: hash("1"), Type: "STEP", Status: "PENDING" }, and re-invokes after the delay.
3. Before re-invoke, deploy v2: workflow now calls WaitAsync(name: "fetch") first instead. (The user shouldn't do this, but it's the exact scenario ValidateReplayConsistency is supposed to catch.)
4. Service re-invokes Lambda. The checkpoint envelope contains the PENDING step record at hash("1"). _isReplaying = false (no terminal ops, only the PENDING one).
5. User code reaches the first await: WaitAsync(name: "fetch") at hash("1"). ValidateReplayConsistency is called.

With the old if (!_isReplaying) return; short-circuit at the top, this validation would return early — even though _operations[hash("1")] exists with Type = "STEP" and our user is asking for type "WAIT". The mismatch slips through silently. Then ReplayAsync runs against a STEP record while user code expects a WAIT, producing weird downstream behavior.

[philasmar]

Important

1. RegexOptions.Compiled is not AOT/trim-safe (RetryStrategy.cs:117)

_retryableMessagePatterns = retryableMessagePatterns?
    .Select(p => new Regex(p, RegexOptions.Compiled))
    .ToArray();

RegexOptions.Compiled emits IL at runtime, which triggers IL3050 (RequiresDynamicCode) in trimmed/AOT deployments. Since this SDK explicitly targets Lambda AOT scenarios (the
AOT smoke test in #2360 checks for zero IL2026/IL3050 warnings), this will either break AOT builds or require a suppression. Consider removing Compiled (the performance
difference is negligible for retry-time regex matching) or gating on RuntimeFeature.IsDynamicCodeSupported.

2. Wall-clock comparison in ReplayPending (StepOperation.cs:~138)

if (DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() < scheduledMs)

Using DateTimeOffset.UtcNow during replay means clock skew between invocations could cause a step to re-suspend when the service intended it to run (or vice versa). In
practice, the service should transition the operation to READY when the timer fires, so this PENDING path is only hit if the service re-invokes slightly early. But the fallback
to ExecuteFunc when the timer is past means a slightly fast clock would skip re-suspension -- which seems fine. Worth a brief comment noting the service's READY transition is
the authoritative "timer fired" signal, and this check is a safety net for early re-invokes.

3. ExecutionState.ValidateReplayConsistency now runs regardless of IsReplaying (ExecutionState.cs:97)

The author's inline comment explains why: in-progress ops (PENDING/READY/STARTED) don't set IsReplaying=true but their type/name still needs validation against code drift. This
is a correctness improvement. However, it means every StepAsync call now does a dictionary lookup even in the common "fresh, no prior state" path. The cost is trivial
(dictionary lookup on empty or small map), so this is fine -- just noting the behavioral change is intentional and explained.

---

Nits

1. Double minimum-delay enforcement: CalculateDelay applies Math.Max(1, ...) at the end, and HandleStepFailureAsync also clamps to >= 1s with a warning log. The double
   enforcement is harmless (defense in depth), but the log message could confuse users of the built-in strategies since CalculateDelay already guarantees >= 1s. Only custom
   FromDelegate strategies would hit the warning.
2. Dockerfile USER warnings: Semgrep flagged all 4 test Dockerfiles for missing USER directive. Since these run inside the Lambda provided:al2023 base image (which manages the
   runtime user), this is a false positive for the Lambda execution context. No action needed, but a suppression comment could silence CI noise.
3. await Task.CompletedTask pattern in test lambdas: Several test functions use async (ctx) => { await Task.CompletedTask; ... } to satisfy the async signature. A
   ValueTask-returning signature or Task.FromResult would avoid the async state machine allocation, but this is test code so it doesn't matter.

Questions

1. Is there a plan to add a CancellationToken to IRetryStrategy.ShouldRetry? Currently the strategy is synchronous, which is fine for built-in strategies, but user delegates
   that want to check circuit breaker state or external config would benefit from an async overload. (Probably fine as follow-up if needed.)
2. For the AtMostOncePerRetry + StepInterruptedException path: if the retry strategy returns DoNotRetry for a StepInterruptedException, the SDK writes a FAIL checkpoint. But
   the actual step result is unknown (it may have succeeded before the crash). Is there guidance for users on how to handle this ambiguity? (e.g., check external state in a
   subsequent step before proceeding