Skip to content

ADR-0024: Durable execution substrate for attempts and workflows

Decision

This page records a decision and its status. A Proposed decision is not binding; an Accepted decision is binding on the Factory until amended or superseded.

  • Status: Accepted 2026-10-12 by the product owner (decision D46); proposed 2026-10-09 as the recommendation of the durability spike, design plan P2
  • Date: 2026-10-09 (proposed) · 2026-10-12 (accepted)
  • Deciders: Dmitry Khaymov (product owner); proposed by the 2026-10 design plan
  • Tags: durability, orleans, agent-framework, workflows, checkpoints

Context

Long-running agent runs must survive process restarts, node failures, model-provider outages, tool failures and human approval delays (original handoff §3.6, §16). ADR-0014 fixes the run, step and attempt vocabulary and makes Run Control the single writer of truth with a recovery (not replay) promise. ADR-0012 makes the MessagingModel port the messaging seam with an Orleans outbox/inbox profile as the default. The Agent Mesh needs a substrate for attempt checkpoints, leases with fencing tokens, exactly-once outcome reports, durable timers and a deterministic workflow engine, and it must work in the saas-default, hybrid and on-prem profiles (ADR-0017), the last without a broker.

Three candidates were compared in the spike: the Microsoft Agent Framework durable and workflow extensions, Orleans grains with a journal, and a process manager (MassTransit saga) over a broker. The Agent Framework durable extension was not checked against its primary documentation: its capabilities in the spike are the author's reading of the 1.21.0 pins, which is why its first-class leases, fencing and exportable checkpoints are revisit triggers rather than rejected facts (open item in the review folder).

Decision

  1. Orleans grains are the durability substrate. AttemptGrain (one per attempt) and WorkflowRunGrain (one per run) serialise their transitions, persist state rows in runtime-sql, commit each transition with its outbox record (ADR-0012 §4) and use Orleans reminders for timers, lease heartbeats and budgets.
  2. The Microsoft Agent Framework is the execution adapter inside an attempt. The agent loop runs in-process under the grain; the framework's checkpoint is stored as an opaque blob with an adapterVersion, referenced by the attempt's Checkpoint entity. Resume is allowed only from a compatible checkpoint; otherwise Run Control starts a new attempt.
  3. Workflow definitions are data, the engine is ours. Workflow releases (BC-04) carry the DSL validated at publication; WorkflowEngineService drives the DAG and emits StepReady; Run Control dispatches and records. The framework's workflow graphs may be used inside one agent step, never as the run-level engine.
  4. Broker sagas are not used for run or workflow state. The MassTransit and NServiceBus profiles of the messaging port remain available as transports; sagas in the agent template are scaffolding and are removed from the design.
  5. Truth stays in Run Control. The runtime's grains hold projections and checkpoints; the run journal in BC-03 is the restore anchor (D30 restore order), and the runtime store is restored after it.
  6. A second execution adapter is admitted by implementing the attempt adapter contract (start, step, checkpoint, resume, interrupt) and declaring its checkpoint format; the contract suite towards Model & Compute, Tools & MCP and Knowledge does not change.

Consequences

  • Positive: reuses what ConnectSoft has adopted in source (Orleans and the Orleans messaging adapter: DerechHaLev, seven service projects reference ConnectSoft.Extensions.MessagingModel.Orleans, with an approved design pack; deployment and production operation are not verified in this repository); no broker dependency on-prem; leases, fencing and exactly-once reporting are natural in single-threaded grain activations; checkpoints are portable across framework upgrades because they are versioned blobs owned by the Factory.
  • Negative: the workflow engine (DAG, conditions, timers, compensation) is Factory code to design and maintain rather than a framework feature; framework-native durable features that assume their own store are not used.
  • Neutral: replay stays unpromised (ADR-0014); the broker profile stays for installations that have one.

Revisit triggers

The framework's durable extension gaining first-class leases, fencing and an exportable checkpoint format; Orleans unavailable for a target topology; or the spike's assumptions contradicted by the first implementation evidence.