Workflow replay

On this page 7

Workflow replay is the ability to re-execute an automation run from its recorded trigger data, inputs or a saved checkpoint, either in full or from the step that failed. In automation reliability, workflow replay is the main recovery tool after an outage, a bug fix or a downstream failure, because it lets operators process the affected items again without asking the original sender to resend anything.

The term covers two related uses. Operational replay reruns failed or missed executions to recover lost work. Deterministic replay, used by durable execution engines, reconstructs the state of a running workflow by re-reading its event history, so a crashed process can continue exactly where it stopped.

How workflow replay works

Workflow replay depends on one precondition: the platform must persist the input of every execution, and ideally the output of every step. Without the original payload, there is nothing to replay, which is why short retention of execution logs is a common reason replay fails when it is needed most.

When an operator triggers a replay, the engine loads the stored trigger data, creates a new execution linked to the original, and runs the steps again with the current workflow definition or with the version that originally ran. Some engines also allow replay from a specific step, reusing stored outputs of earlier steps so that completed work is not repeated.

Replay formStarting pointTypical use
Full replayThe original trigger payloadRerun after fixing a bug in an early step
Replay from failed stepStored outputs up to the failureRecover after a downstream API outage
Bulk replayA filtered set of past executionsReprocess every order affected by an incident
Dead letter queue redriveMessages parked after repeated failuresReturn poison-free messages to processing
Deterministic history replayThe workflow’s event historyResume a durable workflow after a worker crash
Forms of workflow replay

Workflow replay compared with retries and checkpointing

Workflow replay is often confused with automatic retries, but they operate at different moments. A retry policy repeats a single failing step within the same execution, seconds or minutes after the error. Workflow replay is usually a deliberate, later action on executions that have already ended in failure, or a reprocessing of successful runs whose results turned out to be wrong.

MechanismTriggerScopeWho decides
Retry policyTransient errorOne step, same runEngine, automatically
Exponential backoffRepeated transient errorsTiming of retriesEngine, automatically
CheckpointingEach completed stepSaved state for resumptionEngine, automatically
Workflow replayFailure, fix or incidentWhole run or from a stepOperator or recovery job
Compensating actionPartial completion to undoReverses side effectsSaga logic or operator
Replay and neighboring reliability mechanisms

Checkpointing is what makes partial replay efficient: if each step’s output is saved, a replay can begin at the failed step instead of the trigger. Without checkpoints, every replay is a full replay, which repeats calls that already succeeded.

Why idempotency decides whether replay is safe

The central risk of workflow replay is duplicate side effects. If a run created an invoice, charged a card or sent an email before failing, replaying it from the start can do all of that a second time. Replay is safe only when steps are idempotent, meaning that repeating them produces the same outcome as running them once.

  • Idempotency keys: pass a stable key, such as the original execution ID plus step name, so the receiving API ignores a repeated request.
  • Check-before-write: look up whether a record already exists before creating it.
  • Upserts instead of inserts: update an existing record when the key matches, rather than adding a new one.
  • Skip completed steps: replay from the failed step when stored outputs show earlier steps succeeded.
Step typeReplay riskSafeguard
Read or lookupNoneNo action needed
Record update with keyLowUpsert by business key
Record creationDuplicatesIdempotency key or existence check
Payment or chargeDouble chargeProvider idempotency key, never blind replay
NotificationRepeated messagesSent-message log checked before sending
Replay safety by step type

Deterministic replay in durable execution engines

Durable execution engines, such as Temporal or Azure Durable Functions, use replay internally. The workflow code is re-executed from the beginning after a restart, but each completed activity returns its recorded result from history instead of running again. This rebuilds in-memory state precisely, which is why such workflow code must be deterministic.

Determinism means the code must not read the system clock directly, generate random values, or call external services outside recorded activities. Changing workflow code while executions are in flight can cause non-determinism errors during replay, so these engines provide versioning mechanisms to run old and new logic side by side.

Designing workflows for safe replay

Replay should be designed in from the start, not added during an incident. The practical checklist is short: retain execution data long enough, make writes idempotent, and record enough context to identify which executions an incident affected.

  • Set retention deliberately: keep execution payloads at least as long as the typical time to detect an incident, often days or weeks rather than hours.
  • Mask sensitive data in stored payloads, since replay storage can hold personal or financial information.
  • Version workflows, so operators know whether a replay uses the fixed or the original logic.
  • Rate-limit bulk replay to avoid overwhelming downstream systems that just recovered.

In plain words

Automated processes sometimes fail halfway, for example because another system was briefly offline. Workflow replay means running that process again using the same data it received the first time, so nothing has to be re-entered by hand.

It resembles reheating a recipe from where it went wrong rather than shopping for ingredients again. The catch is that some steps must not happen twice, such as charging a customer, so a good replay knows which steps were already finished.

A real-world example

An online subscription business runs a workflow that receives payment confirmations, activates accounts and sends welcome emails. One afternoon, the account service returns errors for 40 minutes, and about 1,200 executions fail at the activation step, leaving paying customers without access.

Because the platform stores every execution’s input and step outputs for 30 days, the operations team filters the failed runs by time window and error code and starts a bulk replay from the failed step, throttled to 20 runs per second. The payment step is never repeated, and the email step checks a sent-message log first. All affected accounts are activated within an hour of the service recovering, and no customer receives a duplicate charge or email.