AIFACTORY-FEAT-091: Durable execution substrate verified against pinned versions¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Engineering readiness · Increment I0 Readiness · State elaborated · Route solution-architect · Build order 46 of 185 (Factory code)
ADR-0024 chose Orleans grains with the Agent Framework for durable execution without checking the framework's durable extension against the versions the estate pins. This feature proves the substrate with one run that survives a process restart.
| Delivers | — |
| Stubs | — |
| Realizes | adrs: ADR-0024 |
| Requirements | — |
| Depends on | — |
| Sources | docs/architecture/capability-dependencies.md, docs/ai-software-factory/current-state.md |
AIFACTORY-US-370: Durability spike: a run with one step and one attempt runs as WorkflowRunGrain and AttemptGrain on the pinned Orleans and Agent Framework versions and persists every transition¶
enabling · persona platform-admin
As a platform admin, I want the smallest host in which a run, its step and its attempt live as grains on the versions the estate pins, so that the durable execution decision can be tested by running code and not by reading documentation.
ADR-0024 decision 1 makes Orleans grains the durability substrate, with AttemptGrain per attempt and WorkflowRunGrain per run, and decision 2 runs the Microsoft Agent Framework in-process under the attempt grain with its checkpoint stored as an opaque blob with an adapterVersion. The amendment of the ADR on the primary documentation check (decision D89) records that no code was run: of twenty-six claims five differ from the documentation and five were not found, and several of them can only be answered by a first implementation. This PBI builds the smallest host that can answer them. It is a spike project in the repository ConnectSoft.AI.SoftwareFactory.AgentRuntimeService. No other task of this increment creates that repository, so the spike does: after the approval of the owner it is created by the checklist of feature repository-and-pipeline-conventions with dotnet new connectsoft-ai-agent -n ConnectSoft.AI.SoftwareFactory.AgentRuntimeService --agent-host-kind attempt-runtime, the host kind that AIFACTORY-US-138 of feature component-agent-template delivers with the silo-kind configuration of one Orleans cluster and without any grain. This is the one creation of the repository, and the service feature of increment I1 takes it as it stands. The spike lives in a folder of its own outside the product solution and the package, so nothing of it is built into the service or shipped with it. The spike adds two grains. WorkflowRunGrain holds a run and its one step; AttemptGrain holds the one attempt. Each serialises its transitions, persists a state row per transition in a relational store through the grain persistence of the host, and commits the transition together with its outbox record through the Orleans messaging profile of the messaging port, as ADR-0012 decision 4 requires; AIFACTORY-US-147 of feature component-agent-template-factory-access makes that profile the default of a Factory host kind, and this PBI depends on it. Under the attempt grain a scripted agent of the Agent Framework runs with a scripted chat client, so no model provider is called and no key is needed, and it writes its checkpoint as an opaque blob with its adapterVersion after each step of its loop. The versions are the ones the estate pins: the Orleans packages and Microsoft.Agents.AI at the versions of the recorded pin set of the Factory libraries, which feature component-extensions-ai aligns. No preview package is referenced, so the Durable Extension, which ships outside the pins, is not part of the spike. The spike is evidence and not product code: its grains carry identifiers, a state name, a counter and a checkpoint reference only, nothing of it is the implementation that AgentRuntimeService and WorkflowEngineService receive in increment I1, the spike folder with its grains, which carry the names of the product grains, is removed when the service feature of I1 delivers the grains, and since Run Control does not exist yet the spike keeps no run journal and claims nothing about the restore order of ADR-0024 decision 5. Its store is a local database whose name the owner agreed before it was created.
Acceptance criteria
| AC1 | Given the spike host on the pinned versions, when a run with one step and one attempt is started, then WorkflowRunGrain and AttemptGrain persist a state row for each transition, the package versions in use equal the recorded pin set and no preview package is referenced (ADR-0024 decision 1 and its amendment on the primary documentation check; docs/planes/agent-runtime/durability-spike.md). |
| AC2 | Given the scripted agent under the attempt grain, when its loop advances, then a checkpoint is written after each step as an opaque blob with its adapterVersion and is referenced by the state of the attempt, and a loop that runs longer than the default timeout of a grain call proceeds without a grain call that waits for it to end (ADR-0024 decision 2; the spike page lists the agent loop inside a grain among the points that stay unknown). |
| AC3 | Given a transition of the attempt grain, when it commits, then its state row and the outbox record of its event are committed together through the Orleans messaging profile, and a planted failure between the two leaves neither behind (ADR-0024 decision 1, ADR-0012 decision 4). |
| Traces: adrs | ADR-0012, ADR-0024 |
| Depends on | feature:component-agent-template, feature:component-extensions-ai, feature:component-agent-template-factory-access, feature:repository-and-pipeline-conventions |
| Sources | docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/durability-spike.md, docs/architecture/capability-dependencies.md, docs/ai-software-factory/current-state.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-146 Spike: a run, a step and an attempt as grains that survive a restart |
implementation | The repository ConnectSoft.AI.SoftwareFactory.AgentRuntimeService is created by the checklist after the approval of the owner with dotnet new connectsoft-ai-agent -n ConnectSoft.AI.SoftwareFactory.AgentRuntimeService --agent-host-kind attempt-runtime; this is the one creation of the repository. The spike lives in a folder of its own outside the product solution and the package, and is removed when the service feature of I1 delivers the grains. It holds WorkflowRunGrain for the run and its step and AttemptGrain for the attempt; each transition persists a state row and commits with its outbox record through the Orleans messaging profile of the host, the scripted agent checkpoints as an opaque blob with its adapterVersion, and the recorded result names the package versions used and any deviation from ADR-0024. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |
AIFACTORY-TASK-1534 Scripted agent, scripted chat client and a harness that runs the silo as its own process |
fixture | A fixture of the spike: a scripted agent of the Agent Framework with a scripted chat client that calls no provider, a loop of a configurable number of steps and duration, and a harness that starts the silo as a separate process against the agreed local database, kills that process without a graceful stop at a chosen step and starts it again. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |
AIFACTORY-TASK-1535 Verify the persisted transitions, the checkpoints and the joint commit |
verification | Run dotnet test -c Release --filter Category=Spike on the spike project; the state rows per transition, the checkpoint with its adapterVersion, the long loop and the planted failure between state row and outbox record pass and prove AC1 to AC3 of AIFACTORY-US-370 (ADR-0024, ADR-0012). |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |
AIFACTORY-US-371: Durability spike: after a killed silo the grains reactivate with their last state, the attempt resumes from a compatible checkpoint, a reminder fires once and a stale activation cannot write¶
evidence · persona platform-admin
As a platform admin, I want the points the documentation check left open to be answered by probes against a killed and restarted silo, so that the first services are built on a substrate whose recovery was seen and measured.
This PBI is the proof of the feature. Each probe runs against the spike host of AIFACTORY-US-370 with the silo in a process of its own that the harness kills without a graceful stop and starts again. The first probe is the restart itself: the process is killed while the scripted agent is in the middle of its loop, and after the restart WorkflowRunGrain and AttemptGrain reactivate with their last committed state and the attempt resumes from its last checkpoint, so that a step completed before the checkpoint is not executed again. This is recovery and not replay, which ADR-0014 decision 5 does not promise. The time from the start of the new process to the resumed loop is measured and recorded. The second probe plants a checkpoint whose adapterVersion the adapter does not accept: ADR-0024 decision 2 allows a resume only from a compatible checkpoint, so the attempt is not resumed, ends with the reason and reports that a new attempt is needed. The third probe concerns reminders. The run grain registers a reminder; a tick that falls due while the silo is down is skipped and is not delivered late, and the next tick fires once after the restart and never twice. The documentation check found that reminders are documented for periods of minutes or longer, so the probe records the shortest period that held and whether a grain timer kept alive by the reminder serves a shorter heartbeat. The fourth probe forces two activations of one attempt grain, which the default grain directory allows for a short time: the write of the stale activation is refused by the consistency check of the state store and one outcome is recorded. The check found that leases and fencing are Factory design and not an Orleans feature; the probe shows what the store gives and what the design has to add. A probe that fails is recorded as failed with what it shows. It is a finding for the owner and is neither hidden nor worked around, and the spike decides nothing about it.
Acceptance criteria
| AC1 | Given a run whose attempt is in the middle of its loop, when the silo process is killed and started again, then WorkflowRunGrain and AttemptGrain reactivate with their last committed state, the attempt resumes from its last checkpoint without executing a completed step again, and the time from the start of the process to the resumed loop is measured and recorded (ADR-0024 decisions 1 and 2; ADR-0014 decision 5 promises recovery and not replay). |
| AC2 | Given a stored checkpoint whose adapterVersion the adapter does not accept, when the attempt grain reactivates, then the attempt is not resumed, it ends with the reason and reports that a new attempt is needed (ADR-0024 decision 2). |
| AC3 | Given a reminder registered by the run grain, when the silo is down while a tick falls due and is then restarted, then the tick that fell due while the silo was down is not delivered, the next tick fires once after the restart and never twice, and the probe records the shortest period that held and the behaviour of a grain timer kept alive by the reminder (ADR-0024 decision 1; docs/planes/agent-runtime/durability-spike.md records the documented granularity of reminders and that a tick due during an outage is skipped). |
| AC4 | Given two activations of one attempt grain, when both write their state, then the write of the stale activation is refused by the consistency check of the state store, one outcome is recorded, and the result states what the fencing token and the outcome report of the Factory design have to add to that check (ADR-0024 amendment on the primary documentation check). |
| Traces: adrs | ADR-0014, ADR-0024 |
| Depends on | AIFACTORY-US-370 |
| Sources | docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/durability-spike.md, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-1536 Probes for the restart, the incompatible checkpoint, the reminder and the duplicate activation |
fixture | Four probes in the spike project, each a test of the category Spike that uses the harness of AIFACTORY-TASK-1534: a kill in the middle of the loop with the measured time to the resumed loop, a planted checkpoint with an adapterVersion the adapter refuses, a reminder whose tick falls due while the silo is down, and a forced second activation of one attempt grain whose stale write is refused. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |
AIFACTORY-TASK-1537 Verify the four probes against a killed and restarted silo |
verification | Run dotnet test -c Release --filter Category=Spike on the spike project; the result of each of the four probes, passed or failed, is kept with the package versions, the measured recovery time and the shortest reminder period that held, which proves AC1 to AC4 of AIFACTORY-US-371 (ADR-0024) or names the criterion a failed probe contradicts. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |
AIFACTORY-US-052: A run, a step and an attempt survive a silo restart¶
enabling · persona platform-admin
As a platform admin, I want proof that a run, a step and an attempt modelled as grains keep their state across a silo restart, so that the durable execution decision (ADR-0024) stands on evidence before the first services are built on it.
This is the integration PBI of the feature. ADR-0024 chose Orleans grains with the Agent Framework as the execution adapter for durable execution, and its amendment on the primary documentation check states that the claims were read and never run. A platform admin gets the missing evidence here: AIFACTORY-US-370 builds a spike host in which a run with its step lives in WorkflowRunGrain and its attempt in AttemptGrain, on the Orleans and Agent Framework versions the estate pins, and AIFACTORY-US-371 kills and restarts the silo under four probes. This PBI turns the results into a record. The durability spike page of this site gains the result of the run: the package versions, the outcome of each probe, the measured recovery time, the shortest reminder period that held and every deviation from ADR-0024. Each of the five differences that the documentation check listed for the owner is marked as answered by a probe, with the probe named, or as still open. The record then confirms ADR-0024 or proposes an amendment of it for the owner to accept. The spike decides nothing: a revisit of the decision, the wording of the allowance for the durable extension inside an attempt and the acceptance of an amendment are decisions of the owner, and they stay open in the record until the owner takes them. The feature is done when the first services of increment I1 can be built on grains with a recorded answer to whether a run survives a restart. The spike project is not the implementation of AgentRuntimeService; that is delivered by its own feature in I1.
Acceptance criteria
| AC1 | Given a run with one step and one attempt persisted as grains, when the silo process is killed and restarted, then the grains reactivate with their last state and the attempt resumes from its last checkpoint (ADR-0024 decisions 1 and 2; recovery and not replay, ADR-0014 decision 5). |
| AC2 | Given a reminder registered by the run grain, when the silo restarts, then the tick that fell due during the outage is not delivered and the next tick fires once after restart and never twice (ADR-0024 decision 1; docs/planes/agent-runtime/durability-spike.md). |
| AC3 | Given the outcome of the spike, when it is recorded on docs/planes/agent-runtime/durability-spike.md, then the record names the package versions, the result of each probe, the measured recovery time and the open risks, marks each of the five differences of the documentation check as answered by a named probe or as still open, and confirms ADR-0024 or proposes an amendment that only the owner accepts. |
| Traces: adrs | ADR-0024, ADR-0014 |
| Depends on | AIFACTORY-US-370, AIFACTORY-US-371 |
| Sources | docs/architecture/capability-dependencies.md, docs/ai-software-factory/current-state.md, docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/durability-spike.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-147 Record the outcome as an ADR amendment or confirmation |
documentation | docs/planes/agent-runtime/durability-spike.md gains the result of the spike: package versions, the result of each probe, the measured recovery time, the shortest reminder period that held and every deviation; each of the five differences of the documentation check is marked answered by a named probe or still open; ADR-0024 is confirmed, or an amendment is written as a proposal that the owner accepts or rejects. | ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-714 Verify run-survives-a-restart-spike by running the declared command |
verification | Run dotnet test -c Release --filter Category=Spike on the spike project; the run completes and its results prove AC1 and AC2 of AIFACTORY-US-052 or name the probe that contradicts them; AC3 is proven by AIFACTORY-TASK-147. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |