Skip to content

AIFACTORY-FEAT-011: AgentRuntimeService: complete delivery

Generated from the architecture model

This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.

A design artefact, not a tracker

This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).

Epic Agent Mesh (PF-02) · Increment I1 Generate a microservice · State elaborated · Route backend-developer · Build order 57 of 185 (Factory code)

AgentRuntimeService is the service of bounded context BC-06 (Execution). Executes one attempt of one step under a lease from Run Control — loads the pinned release, assembles context, runs the plan-call-observe-verify loop on the Microsoft Agent Framework adapter, checkpoints before every gated call, asks for human decisions, delegates sub-agents and reports exactly one outcome (BC-06). Its own repository is ConnectSoft.AI.SoftwareFactory.AgentRuntimeService.

Delivers AgentRuntimeService (complete)
Stubs —
Realizes adrs: ADR-0014, ADR-0021, ADR-0022, ADR-0023
Requirements FR-03, FR-05, FR-06, FR-07, FR-10, FUX-04, FUX-05, FUX-07, NFR-05, NFR-06, NFR-11, NFR-13, XL-03, XL-04, XL-05, XL-06, XL-10
Depends on —
Sources model/services/agent-mesh/AgentRuntimeService.yaml

Scenarios to trace (15, catalog): SCN-AgentRuntimeService-001 ✓, SCN-AgentRuntimeService-002 ✓, SCN-AgentRuntimeService-003 ✓, SCN-AgentRuntimeService-004 ✓, SCN-AgentRuntimeService-005 ✓, SCN-AgentRuntimeService-006 ✓, SCN-AgentRuntimeService-007 ✓, SCN-AgentRuntimeService-008 ✓, SCN-AgentRuntimeService-009 ✓, SCN-AgentRuntimeService-010 ✓, SCN-AgentRuntimeService-011 ✓, SCN-AgentRuntimeService-012 ✓, SCN-AgentRuntimeService-013 ✓, SCN-AgentRuntimeService-014 ✓, SCN-AgentRuntimeService-015 ✓

AIFACTORY-US-007: An agent step runs on a model profile and leaves a receipt

behavior · persona requester As a requester, I want the vision architect to clarify my intent and draft the Vision Blueprint on the model profile of its role, so that I get a first blueprint and can see which model produced it at what cost.

This is the integration PBI of the feature: it proves on a deployed instance that an attempt of the first workflow runs end to end. A requester submitted a vision brief; Run Control dispatched the step clarify-intent of vision-to-blueprint, and later draft-vision, to the agent vision-architect. AgentRuntimeService, the attempt-runtime silo kind of the Agent Mesh cluster (ADR-0024, ADR-0028), claims the lease of the dispatched attempt (AIFACTORY-US-464), loads and verifies the pinned release (AIFACTORY-US-463), receives the context package of the pinned artifacts from the thin ContextBuilderService, runs the loop on the Microsoft Agent Framework adapter with a checkpoint before every gated call (AIFACTORY-US-465), and reports exactly one outcome with its hash and the fencing token of the lease (AIFACTORY-US-467). Every model call goes through the model gateway with the Factory model client of the pinned AI libraries: ModelRouterService/ModelGateway.ResolveRoutePlan resolves the pinned profile and ModelRouterService/ModelGateway.Invoke returns the completion with an invocation receipt that names the profile, the actual model, the token usage and the run and attempt; the runtime keeps the receipt reference on the call record and never the prompt or the completion. Every tool call, in this chain the read tools knowledge-search, graph-query, list-files and read-range, goes through the tool gateway with a policy decision id and the fencing token. When the provider of the profile fails, the fallback chain of the profile is applied by ModelRouterService and the receipt says which model answered; the runtime neither picks a provider nor repeats a call outside the profile. The repository and its store come from AIFACTORY-US-460, the hosted surface from AIFACTORY-US-461 and the workers from AIFACTORY-US-462; human decisions (AIFACTORY-US-466), interruption (AIFACTORY-US-468), lease keeping and recovery (AIFACTORY-US-469), delegation (AIFACTORY-US-470) and the timeline reads (AIFACTORY-US-471) complete the delivery. The runtime completes, fails or cancels nothing on its own authority: it reports an attempt outcome and Run Control records the truth (ADR-0024 decision 5). In I1 the releases are the seeded Draft releases the thin AgentRegistryService serves, which run under the waiver of the amendment of ADR-0021, and the context is the pass-through package of the thin ContextBuilderService. Every model call of the two steps passes ModelGatewayHost, the model gateway in front of ModelRouterService, and the usage of each receipt is recorded for the run by CostUsageService from ModelRouterService/ModelInvoked, which this PBI reads back through CostUsageService/GetRunUsage. The task contract (agent-task-contract-v1) and the content of the snapshot (resolved-execution-snapshot-v1) of each attempt are written before the attempt exists and never by the runtime (decision D121): WorkflowEngineService stores the task contract of a step in ArtifactService when it announces the step ready, and WorkflowEngineService/StepReady carries its reference and its hash, which RunControlService copies to each attempt of the step; RunControlService stores the snapshot once, when it accepts the run. The task contract describes the step and no attempt, so every attempt of a step reads the same contract and takes its ids and the reference of its snapshot from the lease answer (decision D123). The answer of RunControlService/RunControl.AcquireLease carries a reference and a hash for each, and the service reads their content from ArtifactService and verifies the hash before the loop starts (AIFACTORY-US-464, decision D119). AC1 is therefore proven on the deployed instance with the contracts of clarify-intent and draft-vision that the deployed WorkflowEngineService stored and with the snapshot that RunControlService stored, read by the references of their lease answers, and no seeded source stands in for them. An attempt whose task contract or snapshot cannot be read, or whose content does not match its hash, ends Failed before it runs with the reason and never runs on defaults. A delegation submits its child run through the child run operation of RunControlService (AIFACTORY-US-470); the two steps of this PBI delegate nothing. What this delivery does not hold is stated in the PBI it belongs to and reported to the owner: the proof of the three release event handlers against a publishing registry, because the thin registry of I1 publishes no release event; the handling of the nine events by their consumers (StudioBff, DashboardService, QualityScoreService, MemoryPromotionService, StudioNotificationService, AuditService), which each consumer proves in its own suite; and the platform gateway AgentRuntimeWorker, which fronts the service from I3. The handler of DeletionOrderIssued is built and proven in I4 with RetentionResidencyService, and this feature registers none (decision D119). Repository: ConnectSoft.AI.SoftwareFactory.AgentRuntimeService.

Acceptance criteria

AC1 Given the step vision-to-blueprint/clarify-intent dispatched to vision-architect for a seeded brief, when the attempt runs on the deployed service, then the agent receives exactly the inputs the workflow definition maps, from the task contract that WorkflowEngineService stored before it announced the step through WorkflowEngineService/StepReady and that the service read from ArtifactService by the reference of the answer of RunControlService/RunControl.AcquireLease and verified against its hash, and the context package of its pinned artifacts, AgentRuntimeService/AttemptStarted is published once with attemptId, runId, stepId, tenantId, releaseId, snapshotId and fencingToken, and exactly one AgentRuntimeService/AttemptOutcomeReported with state Succeeded follows (ADR-0024).
AC2 Given the agent calls the model, when ModelRouterService/ModelGateway.Invoke returns, then the call passed ModelGatewayHost and no other path, the invocation receipt names the profile, the actual model, the token usage and the run and attempt identifiers, ModelRouterService/ModelInvoked is published by the router, CostUsageService/GetRunUsage shows the usage of the receipt recorded for the run by CostUsageService, and the call record of the AgentRuntimeService/ExecutionAttempt holds the request hash and the receipt reference and neither prompt nor completion text (ADR-0014).
AC3 Given the agent uses one of the read tools knowledge-search, graph-query, list-files or read-range, when the tool executes, then the call went through the tool gateway with a policy decision id and the fencing token of the lease and is a call record of the attempt with its receipt reference; no tool ran inside the host (ADR-0022, FR-05).
AC4 Given the provider of the pinned profile fails and the profile has a fallback chain, when the model is called after ModelRouterService/ModelGateway.ResolveRoutePlan, then the fallback is applied by ModelRouterService, the receipt says which model answered, and the attempt neither selected a provider nor repeated the call outside the profile (XL-04, model/services/agent-mesh/AgentRuntimeService.yaml).
AC5 Given the step vision-to-blueprint/draft-vision dispatched after the first step succeeded, when its attempt ends, then one outcome was reported under the fencing token of its lease, Run Control recorded it, and the AgentRuntimeService/ExecutionAttempt and its AgentRuntimeService/LoadedRelease show which release ran against which adapter version (ADR-0024, FR-03).
Traces: services AgentRuntimeService, RunControlService, ModelRouterService, ModelGatewayHost, ToolAdapterService, ContextBuilderService, CostUsageService, ArtifactService, WorkflowEngineService
Traces: aggregates AgentRuntimeService/ExecutionAttempt, AgentRuntimeService/LoadedRelease, ModelRouterService/InvocationReceipt, ModelRouterService/RoutePlan
Traces: apis RunControlService/RunControl.AcquireLease, ModelRouterService/ModelGateway.ResolveRoutePlan, ModelRouterService/ModelGateway.Invoke, CostUsageService/GetRunUsage
Traces: events AgentRuntimeService/AttemptStarted, AgentRuntimeService/AttemptOutcomeReported, ModelRouterService/ModelInvoked, WorkflowEngineService/StepReady
Traces: steps vision-to-blueprint/clarify-intent, vision-to-blueprint/draft-vision
Traces: agents vision-architect
Traces: tools knowledge-search, graph-query, list-files, read-range
Traces: adrs ADR-0014, ADR-0021, ADR-0022, ADR-0024, ADR-0028
Traces: requirements FR-03, FR-05, XL-04
Relates: screens run-detail, run-timeline
Depends on AIFACTORY-US-460, AIFACTORY-US-461, AIFACTORY-US-462, AIFACTORY-US-463, AIFACTORY-US-464, AIFACTORY-US-465, AIFACTORY-US-466, AIFACTORY-US-467, AIFACTORY-US-468, AIFACTORY-US-469, AIFACTORY-US-470, AIFACTORY-US-471, feature:run-control-service, feature:workflow-engine-service, feature:agent-registry-service-thin, feature:model-router-service, feature:model-gateway-host, feature:cost-usage-service-complete, feature:tool-adapter-service, feature:context-builder-service-thin, feature:artifact-service, feature:wf-vision-to-blueprint
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/workflows/vision-to-blueprint.yaml, docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#purpose-and-responsibilities, docs/planes/agent-runtime/services/agent-runtime-service/index.md#sagas-and-long-running-flows

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-202 Acceptance fixture of the vision architect steps with a seeded brief, the expected task contracts and scripted provider answers fixture The acceptance suite holds a seeded vision brief, the vision-architect release as feature wf-vision-to-blueprint authors it and the thin registry serves it, never a copy of its instructions, the expected documents of the task handoff, that is one agent-task-contract-v1 and one resolved-execution-snapshot-v1 for each of the skills vision-architect-clarification and vision-architect-author-the-vision-blueprint-document, built from the input mappings, the limits and the validation of model/workflows/vision-to-blueprint.yaml and used only to compare with what the service read from ArtifactService by the references of the lease answer, never as a source the service reads, and a scripted provider behind the model gateway with two scripts: one yields a schema-valid draft for clarify-intent and draft-vision, the other fails the first provider once so that the fallback of the profile answers. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1853 Deployment values of the attempt-runtime silo: replica range, silo kind, pool, workload identity and secret references operations The deployment manifest of the service states the replica range 2 to 20, the cluster id and the silo kind name of the attempt-runtime kind, Runtime:Host:PoolId, Runtime:Host:MaxConcurrentAttempts, Runtime:Adapter:Kind maf, the workload identity of ConnectSoftIdentityPlatform:WorkloadIdentity, OrleansMessaging:ServiceName agent-runtime and the secret references messaging-transport and workload-identity-key, as model/services/agent-mesh/AgentRuntimeService.yaml records them; the same image runs in the saas-default, hybrid, on-prem and hybrid-compute topologies. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-658 Verify agent-step-with-receipt by running the declared command verification Run dotnet test -c Release --filter "Category=Acceptance&Feature=AgentStepWithReceipt" against the deployed service with Run Control, WorkflowEngineService, ArtifactService, the thin registry, the thin context builder, the model gateway with the scripted provider, CostUsageService and the tool gateway; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-007, that is the mapped inputs and the context package, the receipt of ModelRouterService/ModelGateway.Invoke on the call record with its usage read back through CostUsageService/GetRunUsage, the governed read tool call, the fallback named by the receipt and the single reported outcome of each step. AC1 is proven with the task contract of each step, stored by the deployed WorkflowEngineService before its WorkflowEngineService/StepReady, and the snapshot of the run, stored by Run Control when it accepted the run, both read from ArtifactService by the references of the answer of RunControlService/RunControl.AcquireLease and verified against its hashes (AIFACTORY-TASK-1896), and the content read equals the expected documents of AIFACTORY-TASK-202; the runtime writes neither document and no seeded contract source is bound. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-460: AgentRuntimeService repository: the attempt-runtime host as the durability spike created it, with the ExecutionAttempt and LoadedRelease aggregates and the store agent-runtime-sql

enabling The repository ConnectSoft.AI.SoftwareFactory.AgentRuntimeService is not created here. It was created once, in I0, by AIFACTORY-TASK-146 of feature durable-execution-spike with dotnet new connectsoft-ai-agent -n ConnectSoft.AI.SoftwareFactory.AgentRuntimeService --agent-host-kind attempt-runtime, and this feature takes it as it stands. The first task confirms what it takes: the committed product solution equals a fresh generation of the attempt-runtime host kind from the agent template package at the version the publication gate of feature component-agent-template-release published (AIFACTORY-US-150); when the spike generated from an earlier package, the solution is regenerated at that version and the difference is committed as one change. The repository records the template version. What the host kind generates is the work of the template features and is consumed by that pin, never rewritten or forked here: the silo-kind configuration of one Orleans cluster with the cluster id and the silo kind name (feature component-agent-template), the single execution port with the Factory scope on every entry, the agent identity (feature component-agent-template-identity), the FactoryAgentBase and the mapping of its lifecycle stages onto the attempt events of this service (feature component-agent-template-contract), and the model client through the model gateway, the tool client through the tool gateway and the Orleans messaging profile with outbox and inbox as the default of the host kind (feature component-agent-template-factory-access). The base template arrives through the submodule of the agent template with its Factory profile and its closed findings (features component-base-template-factory-profile and component-base-template-findings): scope propagation, role policies, the descriptor surface, health and telemetry. The agent template takes no UseFactoryProfile switch: it selects its Factory shape with --agent-host-kind, so the scaffold command of AIFACTORY-TASK-146, which carries the host kind only, is the complete command (decision D119); the boolean UseFactoryProfile of the amendment of ADR-0028 belongs to the templates that take it, the gateway template of AgentRuntimeWorker among them. The run event envelope and the event type names come from the contracts package by a version pin as well. The spike folder with its WorkflowRunGrain and AttemptGrain, which lives outside the product solution, is removed in the change that adds the product AttemptGrain, as the spike PBI states. On that host this PBI adds the domain and the store. The ExecutionAttempt aggregate holds the attempt, run, step, tenant, project and optional environment identifiers, the lease id and the fencing token, the snapshot id, the release, agent and model profile ids, state, phase, correctionCount, maxCorrections, the last checkpoint, the context snapshot reference, outcomeHash, the time cancellation was observed, start and end times and a version, with the entities Checkpoint, CallRecord, DecisionRequest and SubAgentDelegation and the value object AttemptLimits. It enforces the invariants an aggregate can hold alone: correctionCount never exceeds maxCorrections; exactly one outcome is recorded, a repeat with the same outcomeHash is the same outcome and a different one is refused; after cancellation is observed no call record of kind Model or Tool is opened; the sequence of a checkpoint and of a call is unique within the attempt. The invariants that need another service (the lease, the policy decision, the checkpoint before a gated call) are enforced where they act, in the behaviour PBIs. The state machine accepts the twenty-two transitions of the model. Three of them, Running, WaitingForTool and WaitingForDecision to Cancelled under the trigger ReleaseRevokedWithPolicyRevoke, end an attempt that a revocation with the in-flight policy revoke interrupted (decision D123, AIFACTORY-US-463). Four of them let an attempt leave Claimed and Paused for a terminal state (decision D119). Claimed to Failed under the trigger StartFailed: the release was rejected, the context stayed unavailable, or the task contract or the snapshot could not be read or did not match its hash before the start. Claimed to Cancelled, Paused to Cancelled and Paused to Uncertain under the trigger OrphanReconciled: the lease is gone, and the attempt is settled Uncertain when a call was in flight and Cancelled otherwise. Two statements of the same model still need more than the table lists: the handler of RunCancelled settles the attempts of a cancelled run, and the orphan reconciler settles any non-terminal attempt. The aggregate therefore accepts two modelled triggers on transitions the table lists under another trigger, and one transition the table does not list. CancellationObserved is accepted on Claimed to Cancelled and on Paused to Cancelled: an interruption reached an attempt that had not started or that was paused. OrphanReconciled is accepted on the three modelled transitions to Cancelled from Running, WaitingForTool and WaitingForDecision, because an attempt that is abandoned or orphaned observed no cancellation. Running to Uncertain under the trigger EffectUnconfirmed of the model is the one transition beyond the table: the lease is gone while a call record of the Running attempt is open. A WaitingForDecision attempt has no call in flight and never moves to Uncertain. The unit tests name the two further triggers and the one transition as additions that await the model, every other transition is refused, and the three additions are reported to the owner as a correction of the table. The LoadedRelease aggregate holds releaseId, tenantId, manifestHash, adapterVersion, contentRefs, state (Verified, Rejected or Revoked), verifiedAt and the optional inFlightPolicy; it is usable only in state Verified and never deletes a manifest that attempts loaded. The store agent-runtime-sql is owned by this service alone (ADR-0028 decision 5) and holds the six tables of the model, ExecutionAttempts, Checkpoints, CallRecords, DecisionRequests, SubAgentDelegations and LoadedReleases, with the unique index over the lease, the unique indexes over attempt and sequence of checkpoints and call records, the unique index over the child run of a delegation, the composite key of release and adapter version, the row version of ExecutionAttempts and the tenant filter in every mapping; a prompt, a completion or a tool payload is a column of no table. Nothing is observable for a caller yet; the surface is hosted by AIFACTORY-US-461.

Acceptance criteria

AC1 Given the repository as AIFACTORY-TASK-146 created it and the agent template package at the published version, when a fresh dotnet new connectsoft-ai-agent -n ConnectSoft.AI.SoftwareFactory.AgentRuntimeService --agent-host-kind attempt-runtime is compared with the committed product solution and the pipeline runs, then the committed solution equals the generation plus the files this feature added, the solution builds in Release, the generated tests pass, a container image is published, the repository records the template version, and the spike folder is absent once the product AttemptGrain exists (ADR-0028, ADR-0024, model/template-parameters/connectsoft-ai-agent.yaml).
AC2 Given the AgentRuntimeService/ExecutionAttempt aggregate, when its unit tests run, then each of the twenty-two transitions of model/services/agent-mesh/AgentRuntimeService.yaml is accepted, among them Claimed to Failed under StartFailed, Claimed to Cancelled, Paused to Cancelled and Paused to Uncertain under OrphanReconciled and Running, WaitingForTool and WaitingForDecision to Cancelled under ReleaseRevokedWithPolicyRevoke, the three additions are accepted and marked as additions, that is CancellationObserved on Claimed to Cancelled and on Paused to Cancelled, OrphanReconciled on the three modelled transitions to Cancelled from Running, WaitingForTool and WaitingForDecision, and Running to Uncertain under EffectUnconfirmed, every other transition is refused, WaitingForDecision to Uncertain among them, a correctionCount above maxCorrections is refused, a second outcome with another outcomeHash is refused while the same hash is the same outcome, and no call record of kind Model or Tool opens after cancellation was observed.
AC3 Given the AgentRuntimeService/LoadedRelease aggregate, when its unit tests run, then a release is usable only in state Verified, a rejection keeps its explicit reason, and a revocation sets the in-flight policy and keeps the manifest hash and the content references that attempts already loaded (docs/planes/agent-runtime/release-loading.md).
AC4 Given agent-runtime-sql created by the migrations of the repository on an isolated test database, when the mapping tests run, then the six tables exist with their keys and indexes, a second attempt row for one lease violates the unique index over the lease, a repeated sequence of a checkpoint or a call within one attempt violates its unique index, a query made for one tenant never returns a row of another, and no column holds a prompt, a completion or a tool payload (ADR-0015, docs/planes/agent-runtime/security.md).
Traces: services AgentRuntimeService
Traces: aggregates AgentRuntimeService/ExecutionAttempt, AgentRuntimeService/LoadedRelease
Traces: adrs ADR-0015, ADR-0024, ADR-0028
Traces: requirements FR-03, NFR-05
Depends on feature:durable-execution-spike, feature:component-agent-template, feature:component-agent-template-identity, feature:component-agent-template-contract, feature:component-agent-template-factory-access, feature:component-agent-template-release, feature:component-base-template-factory-profile, feature:component-base-template-findings, feature:contracts-package
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/template-parameters/connectsoft-ai-agent.yaml, model/stores.yaml, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md, docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/durability-spike.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#domain-model

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-030 Take the repository ConnectSoft.AI.SoftwareFactory.AgentRuntimeService as the durability spike created it and align it with the published agent template package implementation The repository that AIFACTORY-TASK-146 of feature durable-execution-spike created is taken as it stands and is not created again; its product solution is compared with a fresh dotnet new connectsoft-ai-agent -n ConnectSoft.AI.SoftwareFactory.AgentRuntimeService --agent-host-kind attempt-runtime from the package version the gate AIFACTORY-US-150 published and is regenerated in one committed change when it differs; the repository records that template version and pins the contracts package; the solution builds and the generated tests pass in the repository pipeline, which publishes a container image; the spike folder is removed in the change that adds the product AttemptGrain. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-031 Domain model: ExecutionAttempt with Checkpoint, CallRecord, DecisionRequest, SubAgentDelegation and AttemptLimits, and LoadedRelease implementation The ExecutionAttempt aggregate with its four entities and the AttemptLimits value object enforces the correction bound, the single outcome idempotent on outcomeHash, the stop of Model and Tool calls after an observed cancellation and the unique sequences; it accepts the twenty-two transitions of the model, Claimed to Failed under StartFailed, Claimed to Cancelled, Paused to Cancelled and Paused to Uncertain under OrphanReconciled and the three transitions to Cancelled from Running, WaitingForTool and WaitingForDecision under ReleaseRevokedWithPolicyRevoke among them, and the three additions one by one, CancellationObserved on Claimed to Cancelled and on Paused to Cancelled, OrphanReconciled as a further trigger of the three modelled transitions to Cancelled from Running, WaitingForTool and WaitingForDecision, and Running to Uncertain under EffectUnconfirmed, and refuses every other; the LoadedRelease aggregate is usable only when Verified and keeps what attempts loaded; unit tests cover every invariant, each transition, each addition and the refusals. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-032 Persistence in agent-runtime-sql: ExecutionAttempts, Checkpoints, CallRecords, DecisionRequests, SubAgentDelegations, LoadedReleases migration Mappings and migrations for the six modelled tables with UX_Attempts_Lease, IX_Attempts_Run_Step, IX_Attempts_Tenant_State, UX_Checkpoints_Attempt_Sequence, UX_Calls_Attempt_Sequence, UX_Delegations_ChildRun, the composite key of LoadedReleases, the row version of ExecutionAttempts and the tenant filter in every mapping; no table has a column for prompt, completion or tool payload text; mapping tests run against an isolated test database, never a shared one. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1854 Verify the taken repository, the two aggregates and the store of AgentRuntimeService verification Run dotnet build -c Release and dotnet test -c Release --no-build in the repository pipeline; the generated tests, the unit tests of the two aggregates and the mapping tests on an isolated database pass, and a diff of the committed product solution against a fresh generation at the recorded template version shows only the files this feature added, which proves AC1 to AC4 of AIFACTORY-US-460 (ADR-0028, ADR-0024). ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-461: AgentRuntimeService host: three REST routes and three gRPC methods behind scope policies, the agent-runtime messaging profile, tenancy, audit evidence and telemetry

enabling This PBI turns the host of AIFACTORY-US-460 into the surface on which the behaviour PBIs of the feature land. From the service descriptor the host exposes the three REST operations of the model for the platform route (GW-03), GetAttemptTimeline and ListCheckpoints under factory.runs.read and GetLoadedRelease under factory.assets.read, and the gRPC service Runtime with the methods Interrupt, AnswerDecision and WatchAttempt, each with its contract document, an OpenAPI document and a proto. A user token is validated through the resource-server library the profile wires, with the audience of the gateway of the platform, AgentRuntimeWorker, which forwards the user token unchanged (ADR-0015 amendment on the forwarded user token), and must hold the scope of the route; the tenant comes from the token. A gRPC method requires a workload identity whose token holds the service scope that names AgentRuntimeService; a workload token carries service scopes and no tenant claim, so the service reads the tenant and the project a workload forwards only under that scope (ADR-0015 amendment of D114). An attempt or a release of another tenant is invisible under the tenant row filter and is answered 404, over gRPC NOT_FOUND, the same answer as for an identifier that does not exist; a missing scope inside the tenant of the caller, and a caller whose token grants no role in the project of the attempt, is answered 403, over gRPC PERMISSION_DENIED (ADR-0015 amendment of D113). Every modelled refusal code has a problem response. In I1 the surface is reached on the internal address of the service; the platform gateway AgentRuntimeWorker fronts it from I3 (feature agent-runtime-worker) and changes nothing of it. The messaging profile of the service is named agent-runtime (OrleansMessaging:ServiceName), and its inbox and outbox belong to this service alone (ADR-0028). The outbox writer enlists in the transaction of the grain transition or the command that raises an event, and the dispatcher publishes the nine events of the model in the run event envelope of the pinned contracts package, with event id, schema version, tenant, project, run, step and attempt, correlation id and producer, each declared with its topic in the OrleansMessaging section. The same section declares one topic beside the nine of the model, audit.evidence.v1.evidence-submitted, on which the audit evidence event of this service is published; a publish of any other event is refused. The host emits no attempt event beyond the seven the model gives the ExecutionAttempt (AttemptStarted, AttemptPhaseChanged, AttemptCheckpointed, AttemptDecisionRequested, AttemptCancellationObserved, AttemptOutcomeReported and SubAgentDelegated) and the two of the LoadedRelease (ReleaseLoaded and ReleaseLoadRejected): the lifecycle publisher of the template maps the stages of the library base onto these names, and no event of the unmapped library family leaves the host. The inbox subscribes, with deduplication on the event id, to AttemptDispatched, ApprovalDecided, RunCompleted, RunFailed, RunCancelled and LeaseExpired of RunControlService, to AssetReleasePublished, AssetReleaseDeprecated and AssetReleaseRevoked of AgentRegistryService and to ToolEffectUncertain of ToolAdapterService; RunCancelled reaches two handlers, the one that settles the attempts of a cancelled run and the one that settles the delegation of a cancelled child run, and what each handler does is delivered by the behaviour PBIs. The handler of the eleventh consumed event of the model, DeletionOrderIssued of RetentionResidencyService, is built and proven in I4 in this repository by feature retention-residency-service, and this feature registers none (decisions D119 and D137). Audit evidence is an event: where a PBI of this feature records evidence, the Factory sink of the pinned audit library (feature component-audit-factory-evidence) writes an audit-evidence-v1 record to the outbox in the transaction of the command, on the topic audit.evidence.v1.evidence-submitted with the event type AuditEvidenceSubmitted. Telemetry follows the single registration of feature component-observability-factory-conventions: tenant and project are on metrics by default, run, step and attempt are on activities and log records, and connectsoft.factory.agent_id is written while the execution port holds the trace context call for the attempt; a prompt, a completion or a tool payload is on no span and in no log record. The eleven metrics of the model are registered in two groups. The eight attempt and release metrics (attempts_active, attempt_duration, calls, corrections, checkpoints_written, decisions_waiting, uncertain_outcomes and release_load_rejected under connectsoft.factory.runtime) carry tenant and project by default beside their modelled dimensions. The three host-level metrics describe the host and no tenant and carry exactly their modelled dimensions: connectsoft.factory.runtime.attempt_slots pool and replica, connectsoft.factory.runtime.inbox_lag service and connectsoft.factory.runtime.control_plane_rtt method. Beyond the metrics, the ten options are bound and validated at start, the transport credential and the workload identity key are read from the secret references messaging-transport and workload-identity-key, and readiness reports the store, the silo, the messaging profile and the required dependencies.

Acceptance criteria

AC1 Given the hosted service, when AgentRuntimeService/GetAttemptTimeline, AgentRuntimeService/ListCheckpoints and AgentRuntimeService/GetLoadedRelease are each called without a token and with a token that lacks the scope of the route, then the answers are 401 and 403 with a problem response; when AgentRuntimeService/Runtime.Interrupt, AgentRuntimeService/Runtime.AnswerDecision or AgentRuntimeService/Runtime.WatchAttempt is called without a workload identity or with a workload token that lacks the service scope, then the call is refused with UNAUTHENTICATED or PERMISSION_DENIED; no state changes (ADR-0015).
AC2 Given a transition of the AgentRuntimeService/ExecutionAttempt or a command on the AgentRuntimeService/LoadedRelease that raises one of the nine declared events, when its transaction commits, then one outbox record in run-event-envelope-v1 with event id, tenant id, project id, run id, step id and attempt id where they apply, correlation id and payload is committed with it; when the transaction rolls back, neither the change nor the record exists; the OrleansMessaging section declares the nine topics of the model and, for the audit evidence event of AC5, audit.evidence.v1.evidence-submitted, a publish of an event the section does not declare throws, and the dispatcher retries a transport outage with backoff without losing or duplicating a record under its event id (ADR-0012, ADR-0014).
AC3 Given the inbox of the agent-runtime profile, when one of the ten subscribed events is delivered twice with the same event id, then each of its handlers runs once; a message without the tenant and project scope of the envelope is dead-lettered and no handler runs; DeletionOrderIssued has no subscription (ADR-0014, model/services/agent-mesh/AgentRuntimeService.yaml).
AC4 Given the service starts, when liveness and readiness are probed, then readiness answers ready only after agent-runtime-sql is migrated, the silo joined the cluster and the messaging profile started, and names the dependency while it is unready; the ten options of model/services/agent-mesh/AgentRuntimeService.yaml are bound and an invalid value stops the start with the key named; the eleven metrics are registered, the eight attempt and release metrics with tenant and project beside their modelled dimensions and connectsoft.factory.runtime.attempt_slots, connectsoft.factory.runtime.inbox_lag and connectsoft.factory.runtime.control_plane_rtt with exactly their modelled dimensions and no tenant; activities and log records of an attempt carry run, step, attempt and connectsoft.factory.agent_id and no prompt, completion or tool payload (ADR-0015, docs/planes/agent-runtime/observability.md).
AC5 Given a command that records audit evidence, when its transaction commits, then one audit-evidence-v1 record is in the outbox with the event type AuditEvidenceSubmitted for the topic audit.evidence.v1.evidence-submitted, with tenant and actor from the trusted scope, and a rollback leaves none (ADR-0014, model/services/governance/AuditService.yaml).
Traces: services AgentRuntimeService
Traces: aggregates AgentRuntimeService/ExecutionAttempt, AgentRuntimeService/LoadedRelease
Traces: apis AgentRuntimeService/GetAttemptTimeline, AgentRuntimeService/ListCheckpoints, AgentRuntimeService/GetLoadedRelease, AgentRuntimeService/Runtime.Interrupt, AgentRuntimeService/Runtime.AnswerDecision, AgentRuntimeService/Runtime.WatchAttempt
Traces: descriptors run-event-envelope-v1, audit-evidence-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015, ADR-0028
Traces: requirements FR-03, NFR-05
Depends on AIFACTORY-US-460, feature:component-base-template-factory-profile, feature:component-agent-template-contract, feature:component-agent-template-factory-access, feature:contracts-package, feature:component-observability-factory-conventions, feature:component-audit-factory-evidence
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/gateways.yaml, model/services/governance/AuditService.yaml, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#apis, docs/planes/agent-runtime/services/agent-runtime-service/index.md#messages, docs/planes/agent-runtime/observability.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-033 API hosting skeleton: REST and gRPC hosts, scope enforcement, problem responses and contract documents for the 3 REST operations and 3 gRPC methods implementation The three REST routes GetAttemptTimeline, ListCheckpoints and GetLoadedRelease and the gRPC methods Runtime.Interrupt, Runtime.AnswerDecision and Runtime.WatchAttempt are hosted from the service descriptor with token validation, scope enforcement per route, the workload identity and service scope check on every gRPC method, problem responses for the modelled codes 403 and 404, and the OpenAPI and proto documents; the behaviour of each operation is delivered by AIFACTORY-US-463, AIFACTORY-US-466, AIFACTORY-US-468 and AIFACTORY-US-471, not here. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-034 Messaging skeleton: outbox and inbox wiring of the agent-runtime profile for 9 published and 10 subscribed events implementation The outbox writer enlists in the transaction of a grain transition or a command and the dispatcher of the agent-runtime messaging profile publishes AttemptStarted, AttemptPhaseChanged, AttemptCheckpointed, AttemptDecisionRequested, AttemptCancellationObserved, AttemptOutcomeReported, SubAgentDelegated, ReleaseLoaded and ReleaseLoadRejected in the run event envelope of the pinned contracts package, each declared with its topic in the OrleansMessaging section, which also declares the topic audit.evidence.v1.evidence-submitted of the audit evidence event beside the nine topics of the model; the lifecycle publisher maps the library stages onto these names and emits no other attempt event; the inbox subscribes to AttemptDispatched, ApprovalDecided, RunCompleted, RunFailed, RunCancelled, LeaseExpired, AssetReleasePublished, AssetReleaseDeprecated, AssetReleaseRevoked and ToolEffectUncertain with deduplication on the event id, RunCancelled with its two handlers; DeletionOrderIssued is not subscribed, its handler being built in I4 with RetentionResidencyService; what each handler does is delivered by the behaviour PBIs of this feature. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-036 Security, tenancy, audit evidence sink and telemetry of AgentRuntimeService implementation The two scopes of the model are enforced, a route that addresses an attempt admits a call only when the token grants a role in the project of the attempt, every query is tenant-filtered, the tenant and project of a workload are read only under the service scope, the transport credential and the workload identity key come from the secret references messaging-transport and workload-identity-key, the ten options are bound and validated at start, readiness reports the store, the silo, the messaging profile and the required dependencies, the Factory sink of the pinned audit library writes audit-evidence-v1 records to the outbox, and the eleven metrics are registered through the single observability registration, the eight attempt and release metrics with tenant and project beside their modelled dimensions and the three host-level metrics attempt_slots, inbox_lag and control_plane_rtt with exactly their modelled dimensions, with run, step, attempt and connectsoft.factory.agent_id on activities and log records; the 404 that follows from the tenant filter is proven by the behaviour PBIs that read an attempt or a release. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1855 Verify the hosted surface of AgentRuntimeService with a two-tenant fixture verification Run dotnet test -c Release --no-build --filter "Category=Acceptance&Area=Hosting"; a filter that matches no test counts as a failure; the tests call every REST route without a token and without its scope and every gRPC method without a workload identity and without the service scope, commit and roll back a transition that publishes, deliver a subscribed event twice and one without scope, probe readiness with the store stopped, read the telemetry of one attempt and commit and roll back a command that records evidence, which proves AC1 to AC5 of AIFACTORY-US-461 (ADR-0015, ADR-0014). ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-462: AgentRuntimeService workers: the four jobs of the model on the Workers.Runtime engine with scoped idempotency, failed work, the singleton lease and the checkpoint compactor

enabling The model gives AgentRuntimeService four workers: LeaseHeartbeat on a timer of 15 seconds, OrphanedAttemptReconciler every two minutes, CheckpointCompactor once a night and ReleaseCacheWarmer on the event AssetReleasePublished. They run inside the deployable of the service (ADR-0028), not in a host of their own, and they are not written as four private loops: the host registers the worker engine ConnectSoft.Workers.Runtime at the version feature component-workers-runtime-library published, the way its guide shows for a service on the base template, with drain wired to the stop of the host, the idempotency and failed-work stores on agent-runtime-sql and the singleton lease, so that two replicas never run the same job instance. The tables of the engine are created by a migration of this repository from the schema script of the engine in its scoped shape: a processed-message row and a failed-work row carry tenant and project, with run, step and attempt where the work has them, and the key of a processed message is tenant, project and message id, as the Factory profile of the worker template defines it (ADR-0028 amendment of D114); the same message id of a second tenant is therefore stored and not rejected. The service is scaffolded from the agent template and not from the worker template, so it consumes no generated code of feature component-worker-template-factory-profile: what it consumes from that feature is the definition the feature proves on a generated worker, the migration that keys processed messages by tenant, project and message id and the role rule of the worker operations, which the migration of this repository reproduces on the scoped key option of the engine. Each job is registered with the trigger, the retry policy, the idempotency key and the partitioning the model states. LeaseHeartbeat has three fixed attempts five seconds apart, the key lease id plus renewal sequence and one run at a time per attempt. OrphanedAttemptReconciler has five exponential attempts from 30 seconds, the key attempt id plus outcome hash and two parallel runs partitioned by tenant. CheckpointCompactor has three fixed attempts 300 seconds apart, the key checkpoint id and one run at a time per tenant. ReleaseCacheWarmer has five exponential attempts from 30 seconds, the key release id plus adapter version and two parallel runs partitioned by tenant. A trigger repeated under the same key has no second effect, and work that exhausts its retries leaves one open failed-work record with its scope. The periodic jobs tick only under the SQL lease provider. Run and attempt state never moves into the engine: the attempt lives in AttemptGrain (ADR-0024), and the two reminders of the grain, call-timeout and attempt-budget, are grain reminders and no jobs of the engine. What three of the jobs do belongs to behaviour PBIs: lease renewal and orphan settlement to AIFACTORY-US-469 and release warming to AIFACTORY-US-463. This PBI delivers the fourth, which has no caller and publishes no event: CheckpointCompactor deletes the superseded checkpoint rows of terminal attempts after the retention window and keeps the last checkpoint of every attempt and every checkpoint referenced by evidence. The model names no window for checkpoints and none of the ten options holds one. It states a retention for the table ExecutionAttempts only, one year by default under the retention of the project, and this delivery takes that value from model/services/agent-mesh/AgentRuntimeService.yaml as the fixed default window of the compactor; the value of a project arrives with RetentionResidencyService, which is delivered in a later increment, and the missing window and its missing option are reported to the owner. The model names no operation of ArtifactService that deletes the blob a checkpoint row referenced, so the compactor deletes rows only, the blob is left to the retention of the artifact store, and the gap is reported to the owner. The worker operations follow the amendment of ADR-0015 on worker operations: the job catalog, the triggers and failed work without a scope need the Platform admin role, and an Operator sees and replays only the failed work of own projects, a row of another tenant being answered 404. The model lists no operation of AgentRuntimeService that serves the job catalog or the failed work, and the engine package holds no operations surface, so this delivery exposes none and proves the rule on the rows: every failed-work row of an attempt carries its tenant and project. Which surface an operator uses to list and replay the failed work of this service is reported to the owner.

Acceptance criteria

AC1 Given the engine registered in the host, when the service starts on an isolated database, then the tables of the engine exist in agent-runtime-sql in the scoped shape with the processed-message key of tenant, project and message id, the four jobs of model/services/agent-mesh/AgentRuntimeService.yaml are in the catalog with their triggers, retry policies, idempotency keys and partitioning, and a periodic job without the SQL lease provider stops the start (ADR-0028, docs/planes/agent-runtime/workers.md).
AC2 Given two replicas and the conformance suite of the engine, when the suite runs against the registration, then a stop request drains in-flight work, a singleton job runs once, a trigger repeated under its idempotency key has no second effect, the same message id of a second tenant is processed and not rejected, and exhausted retries leave one open failed-work record that carries tenant and project (ADR-0015, ADR-0024).
AC3 Given terminal attempts of two tenants with checkpoints older and younger than the fixed default window of one year, which is the retention the model states for ExecutionAttempts, when CheckpointCompactor runs twice, then the superseded checkpoint rows older than the window are deleted, the last checkpoint of every AgentRuntimeService/ExecutionAttempt and every checkpoint referenced by evidence remain, checkpoints of non-terminal attempts are untouched, no event is published, and the second run deletes nothing more (model/services/agent-mesh/AgentRuntimeService.yaml).
Traces: services AgentRuntimeService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: adrs ADR-0015, ADR-0024, ADR-0028
Traces: requirements NFR-06
Depends on AIFACTORY-US-461, feature:component-workers-runtime-library, feature:component-worker-template-factory-profile
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/components.yaml, docs/planes/agent-runtime/workers.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#workers-and-jobs, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-035 Worker hosting: LeaseHeartbeat, OrphanedAttemptReconciler, CheckpointCompactor and ReleaseCacheWarmer registered on the Workers.Runtime engine implementation The host registers the pinned ConnectSoft.Workers.Runtime engine with drain on host stop, the idempotency and failed-work stores on agent-runtime-sql and the singleton lease under the SQL lease provider; the four jobs are in the catalog with the modelled triggers (timer of 15 seconds, every two minutes, nightly, the event AssetReleasePublished), retry policies, idempotency keys and partitioning, and a duplicate trigger has no second effect; what LeaseHeartbeat, OrphanedAttemptReconciler and ReleaseCacheWarmer do is delivered by AIFACTORY-US-469 and AIFACTORY-US-463. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1856 Engine tables in agent-runtime-sql from the schema script of the engine in the scoped shape migration A migration of the repository applies the idempotent schema script of the pinned engine under the schema name of the service in the scoped shape: the processed-message, failed-work, lease and job tables exist, processed messages are keyed by tenant, project and message id, and failed work carries tenant, project, run, step and attempt; the migration test runs on an isolated database. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1857 CheckpointCompactor over superseded checkpoints of terminal attempts implementation The job selects the checkpoints of terminal attempts older than the retention window per tenant, the window being the fixed default of one year that model/services/agent-mesh/AgentRuntimeService.yaml states as the retention of ExecutionAttempts and no option of the service, deletes the superseded rows, keeps the last checkpoint of every attempt and every checkpoint referenced by evidence, deletes no blob because the model names no delete operation of ArtifactService, publishes nothing and is idempotent on the checkpoint id. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1858 Verify the worker registration, the scoped engine tables and the compactor verification Run dotnet test -c Release --no-build --filter "Category=Acceptance&Area=Workers" with two replicas on an isolated database; a filter that matches no test counts as a failure; the catalog test, the conformance suite of the engine with a second tenant and the compactor test over two tenants pass and prove AC1 to AC3 of AIFACTORY-US-462 (ADR-0024, model/services/agent-mesh/AgentRuntimeService.yaml). ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-463: Load a release once per adapter version, verify its hashes and its compatibility, and apply the in-flight policy when it is deprecated or revoked

behavior · persona asset-maintainer As an asset maintainer, I want the runtime to run a release only after it verified the manifest against its content and against the adapter of the worker, so that an attempt executes exactly what was released, and a release that does not match fails with a reason and never runs degraded.

Publication belongs to the Control plane; the runtime consumes releases, keeps a read model and has no write path to publication. The command LoadRelease fetches the release an attempt pinned through AgentRegistryService/AssetPublication.LoadRelease with the workload token of the service: a server stream that delivers the manifest of release-manifest-v1 and then one message per closure entry with kind, asset id, exact version, hash and content reference; a release is used only after its stream completed. The loader verifies the hash of every element (agent instructions, skill contents, tool descriptors, workflow definition, policy revision, model profile requirement), fetching content by its reference from ArtifactService, and checks the runtime compatibility the manifest declares against this worker: the adapter version, the checkpoint format derived from Runtime:Adapter:Kind and the tool schema versions. A release that passes becomes a LoadedRelease row in state Verified, keyed by release id and adapter version, with manifestHash, the content references and verifiedAt, and ReleaseLoaded is written in the same transaction. A hash mismatch or an unsatisfied compatibility writes the row in state Rejected and ReleaseLoadRejected with the explicit reason in the same transaction; the counter connectsoft.factory.runtime.release_load_rejected moves, AuditService records the event as evidence in its own PBI, and every attempt of that release fails with the reason and never runs on a degraded release. Loading is idempotent on release id plus adapter version: a second load of a Verified release reads the row, calls nobody and publishes nothing. The registry is a fail-fast dependency: when it is unavailable, or its stream ends without completion, what was received is discarded, no row is written and the caller repeats the load. NOT_FOUND, the answer of the registry for an unknown release and for a release of another tenant alike, gives the waiting attempt the failure class permanent (ADR-0015 amendment of D114). The model states an explicit reason for a rejected release and no failure class; the class a hash mismatch or an incompatible adapter carries is reported to the owner, and until it is decided the suite asserts the outcome and the reason. The invariant of the read model speaks of a Published release. In I1 the thin AgentRegistryService serves only the seeded Draft releases, which run under the waiver of the amendment of ADR-0021; the loader verifies them the same way, records what the manifest states and adds no gate of its own, because the waiver is a policy decision of the run and not of the loader. A load is triggered two ways. An attempt that is being claimed has its release loaded when no Verified row exists for the adapter version of the replica (AIFACTORY-US-464). The worker ReleaseCacheWarmer loads a release before its first dispatch when AssetReleasePublished arrives for an agent or a workflow, through the inbox handler WarmReleaseCache, idempotent on release id plus adapter version, with five exponential retries. AssetReleaseDeprecated and AssetReleaseRevoked reach the handler ApplyInFlightPolicy: it records the in-flight policy of the event (finish, drain or revoke) on the row and moves a revoked release to Revoked so that no new attempt loads it. What a revocation does to work in flight is decided (decision D123). The handler applies the in-flight policy of AssetReleaseRevoked to the running attempts of every run whose pinned release closure holds the revoked release, not only to the attempts that pinned that release themselves. Under revoke each of those attempts is interrupted through the path of AIFACTORY-US-468: it stops at the next safe point, moves from Running, WaitingForTool or WaitingForDecision to Cancelled under the trigger ReleaseRevokedWithPolicyRevoke and reports its single outcome Cancelled with the failure class cancellation. Under drain the running attempt finishes and reports its own outcome, and the runtime interrupts nothing; cancelling the run after the step that is running belongs to RunControlService. Under finish nothing is done to an attempt. In every case the manifest and the content references that attempts already loaded are kept and never deleted. The handler is idempotent on the event id, so a redelivered revocation interrupts no attempt twice. The owner decided how the revoked release is found (D125): the resolved execution snapshot of a run names in pinnedReleases the release id of the pinned release and of every member of its closure, so the handler selects the running attempts whose snapshot names the releaseId of the event; the suite proves the three policies on a run that pinned the revoked release itself and on a run whose snapshot names it only as a closure member. The thin registry of I1 publishes none of these three events: AssetReleasePublished arrives with feature agent-registry-service-publication in I2 (D140), and AssetReleaseDeprecated and AssetReleaseRevoked arrive with feature agent-registry-service-complete in I4; the two handlers are delivered here and proven with events built from the pinned contracts package, and their proof against the publishing registry belongs to the increment that publishes each event. A caller whose scope carries factory.assets.read reads what this worker verified through GetLoadedRelease: for each adapter version the release id, the manifest hash, the state, the time of verification and the in-flight policy. A token without the scope is answered 403; a release that was never loaded and a release of another tenant are both answered 404; nothing is written.

Acceptance criteria

AC1 Given a release whose stream from AgentRegistryService/AssetPublication.LoadRelease completed and whose hashes and compatibility hold, when LoadRelease runs, then an AgentRuntimeService/LoadedRelease in state Verified exists for release id and adapter version and AgentRuntimeService/ReleaseLoaded is written in the same transaction and published on execution.releases.v1.release-loaded in the run event envelope with releaseId, tenantId, manifestHash and adapterVersion; a second load of the same release and adapter version calls nobody and publishes nothing, and a transport outage is retried with backoff without loss or duplicate under the event id (SCN-AgentRuntimeService-014, ADR-0012).
AC2 Given a manifest with a hash that does not match its content, or a compatibility this worker does not satisfy, when LoadRelease runs, then the row is written in state Rejected, AgentRuntimeService/ReleaseLoadRejected is written in the same transaction and published on execution.releases.v1.release-load-rejected with releaseId, tenantId, reason and adapterVersion, connectsoft.factory.runtime.release_load_rejected counts it by reason, and an attempt of that release fails with the reason and never runs (SCN-AgentRuntimeService-015, docs/planes/agent-runtime/release-loading.md).
AC3 Given the registry is unavailable, its stream ends without completion, or it answers NOT_FOUND, when LoadRelease runs, then in the first two cases nothing received is used, no AgentRuntimeService/LoadedRelease row is written and the caller repeats the load, and in the third the waiting attempt fails with the failure class permanent, the same for an unknown release and for a release of another tenant (ADR-0015, model/services/agent-mesh/AgentRuntimeService.yaml).
AC4 Given a caller with factory.assets.read and a release this worker loaded, when AgentRuntimeService/GetLoadedRelease is called, then the response is 200 with release id, manifest hash, adapter version, state, verification time and in-flight policy; a token without the scope is answered 403, a release never loaded and a release of another tenant are both answered 404 with a problem response, and nothing is written (SCN-AgentRuntimeService-003, ADR-0015).
AC5 Given AgentRegistryService/AssetReleasePublished, AgentRegistryService/AssetReleaseDeprecated and AgentRegistryService/AssetReleaseRevoked built from the pinned contracts package, when each is delivered, the first twice, then ReleaseCacheWarmer loads the published release once under the key release id plus adapter version, ApplyInFlightPolicy records the in-flight policy on the row, a revoked release is in state Revoked and is loaded by no new attempt, and the policy of AssetReleaseRevoked is applied to the running attempts of every run whose pinned release closure holds the revoked release: under revoke each such attempt is interrupted, ends Cancelled under ReleaseRevokedWithPolicyRevoke and reports one outcome with the failure class cancellation, under drain it finishes with its own outcome, under finish nothing changes for it, an attempt of a run whose closure does not hold the release is untouched, a redelivered event interrupts nothing twice, and the manifest hash and the content references stay in every case (ADR-0021, model/services/agent-mesh/AgentRuntimeService.yaml).
Traces: services AgentRuntimeService, AgentRegistryService, ArtifactService
Traces: aggregates AgentRuntimeService/LoadedRelease, AgentRuntimeService/ExecutionAttempt
Traces: apis AgentRuntimeService/GetLoadedRelease, AgentRegistryService/AssetPublication.LoadRelease
Traces: events AgentRuntimeService/ReleaseLoaded, AgentRuntimeService/ReleaseLoadRejected, AgentRegistryService/AssetReleasePublished, AgentRegistryService/AssetReleaseDeprecated, AgentRegistryService/AssetReleaseRevoked
Traces: descriptors release-manifest-v1, run-event-envelope-v1
Traces: adrs ADR-0012, ADR-0015, ADR-0021, ADR-0024
Traces: requirements NFR-11, NFR-13
Traces: scenarios SCN-AgentRuntimeService-003, SCN-AgentRuntimeService-014, SCN-AgentRuntimeService-015
Relates: screens release-detail
Depends on AIFACTORY-US-461, AIFACTORY-US-462, feature:agent-registry-service-thin, feature:artifact-service
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/services/control-plane/AgentRegistryService.yaml, docs/planes/agent-runtime/release-loading.md#rules, docs/reference/schemas/release-manifest.schema.json, docs/architecture/adr/0021-quality-gates.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1859 Release loader with hash and compatibility verification, the LoadedRelease read model and GetLoadedRelease implementation The command LoadRelease streams the manifest and the closure through AssetPublication.LoadRelease under the workload token, uses a release only after the stream completed, verifies every hash against the content fetched from ArtifactService and the declared compatibility against the adapter version, checkpoint format and tool schema versions of the worker, writes the row Verified with ReleaseLoaded or Rejected with ReleaseLoadRejected and its reason in one transaction, is idempotent on release id plus adapter version, writes nothing when the registry is unavailable and maps NOT_FOUND to the failure class permanent; GetLoadedRelease returns the rows of a release under the tenant filter with 403 for a missing scope and 404 for an unknown or foreign release. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1860 Inbox handlers WarmReleaseCache and ApplyInFlightPolicy with the ReleaseCacheWarmer job implementation WarmReleaseCache hands AssetReleasePublished of an agent or a workflow to the ReleaseCacheWarmer job, which loads the release once under the key release id plus adapter version with five exponential retries; ApplyInFlightPolicy records finish, drain or revoke on the row for AssetReleaseDeprecated and AssetReleaseRevoked, moves a revoked release to Revoked and, idempotently on the event id, applies the policy of AssetReleaseRevoked to the running attempts of every run whose pinned release closure holds the revoked release, found as the attempts whose resolved execution snapshot names the releaseId of the event in pinnedReleases (D125): under revoke it interrupts each of them through the interruption path of AIFACTORY-US-468, so that the attempt ends Cancelled under ReleaseRevokedWithPolicyRevoke with the failure class cancellation, under drain it lets the attempt finish and under finish it does nothing; nothing an attempt loaded is deleted. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1861 Fake registry with valid, mismatching and incompatible manifests and release events from the contracts package fixture The acceptance suite holds a fake of AssetPublication.LoadRelease that streams a valid release, a release with one wrong hash, a release with an unsatisfied compatibility, a stream that ends without completion and a NOT_FOUND, content for each reference in a fake of ArtifactService, and AssetReleasePublished, AssetReleaseDeprecated and AssetReleaseRevoked events built from the pinned contracts package for two tenants, the revocation once with each in-flight policy, finish, drain and revoke, together with running attempts of a run that pinned the revoked release, of a run whose snapshot names it in pinnedReleases as a closure member (D125) and of a run whose snapshot does not name it. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1862 Verify release loading, rejection, the read and the in-flight policy verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-AgentRuntimeService-003\|Scenario=SCN-AgentRuntimeService-014\|Scenario=SCN-AgentRuntimeService-015)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-463: the Verified row with ReleaseLoaded and the silent second load, the Rejected row with ReleaseLoadRejected and its reason, the discarded incomplete stream, the permanent class after NOT_FOUND, the 200, 403 and 404 of GetLoadedRelease over two tenants, the single warm load of a published release, the Revoked state, the attempt interrupted under revoke that ends Cancelled with the failure class cancellation, the attempt that finishes under drain, the attempt left alone under finish, the attempt of a run whose closure does not hold the release left alone under every policy, and the redelivered revocation that interrupts nothing twice. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-464: Claim a dispatched attempt under its lease, build its context and start the loop once

behavior Work reaches the runtime one way only: RunControlService/AttemptDispatched arrives in the inbox of the agent-runtime profile with runId, stepId, attemptId, leaseId, fencingToken, agentId and the optional skillId, in the run event envelope that carries the trusted tenant and project. The handler ClaimAndStart is idempotent on the event id and on the attempt: a redelivery finds the attempt row and starts nothing twice. It activates the AttemptGrain of the attempt, which serialises everything that follows, and claims the lease through RunControlService/RunControl.AcquireLease with acquire-lease-v1, naming the attempt, the leaseId it saw in the event and, as holder, the value of ConnectSoftIdentityPlatform:WorkloadIdentity. That value is the one identity of the service and the same on every replica: exclusivity between replicas comes from the single activation of the AttemptGrain and from the fencing token, never from a holder per replica, and a grain that re-activates on another replica renews the lease the first one claimed (AIFACTORY-US-469). The dispatcher created the lease Unclaimed; this first call moves it to Claimed and starts its clock, and the reply lease-v1 carries the lease id, the expiry, the fencing token, the snapshotId and, for the task handoff, taskContractRef, taskContractHash, snapshotRef and snapshotHash. It also carries attemptToken, a short-lived token bound to the run, the step and the attempt that the Authorization Server issued and that lives no longer than the lease (D125): the runtime hands it to the attempt, which presents it on the OpenAI-compatible route of the model gateway as the proof that its model call belongs to the run, keeps it in memory only and never writes it to a row, a checkpoint, a log or a trace. The attempt row is written in state Claimed with phase Resolving and with the lease id and the fencing token of the reply; the fencing token from then on accompanies every outbound call and the outcome report. When the lease was already claimed by another holder the call is refused and the handler stops without a row: the other holder runs the attempt. When Run Control answers NOT_FOUND, which is its answer for an unknown attempt and for an attempt of another tenant alike, the handler stops and nothing is written. When Run Control is unreachable the handler fails fast and the inbox redelivers; a lease nobody claimed within RunControl:Leases:ClaimTimeoutSeconds is expired by Run Control and the same attempt is dispatched again under a new Unclaimed lease. Because the claim names the lease the handler saw, a claim made from the stale AttemptDispatched names a superseded lease: Run Control answers FAILED_PRECONDITION and changes nothing (decisions D119 and D121), the handler stops and writes no row, and the second AttemptDispatched, which names the current lease, is claimed and starts the attempt once. The attempt needs its task contract (agent-task-contract-v1: objective, inputs, allowed tools, model profile, expected output, validation and limits) and the content of its snapshot (resolved-execution-snapshot-v1: release, policy revision, model profile and limits). Neither document travels in the dispatch event or in the reply: the reply carries a reference and a hash for the task contract of the attempt and for the resolved execution snapshot of the run (decision D119, amendment of ADR-0014). The runtime writes neither document (decision D121): WorkflowEngineService stored the task contract in ArtifactService when it announced the step ready, and the reference and the hash of the reply are the ones StepReady carried for the step and Run Control copied to the attempt; RunControlService stored the snapshot once, when it accepted the run. The task contract describes the step and no attempt (decision D123): attemptId, the stepId of RunControlService and snapshotId are no required members of agent-task-contract-v1 and the runtime expects none of them in the document; the attempt takes its ids and the reference of its snapshot from the answer of AcquireLease, and a retried attempt of the step reads the same contract under the same reference and hash. The handler reads both through the task contract port of the service, whose one binding fetches the content from ArtifactService by reference through ArtifactService/Artifacts.Fetch under the workload token of the service, hashes what it read, compares the result with the hash of the reply and validates each document against its schema; the acceptance suite runs the same binding against a fake of ArtifactService and binds no fixture to the port. releaseId, modelProfileId, the limits and maxCorrections of the attempt row come from the two documents. Content that does not match its hash is never used: an attempt whose task contract or snapshot does not match its hash, fails its schema, or cannot be read because ArtifactService holds nothing under the reference or does not answer, ends Failed before it runs (trigger StartFailed) with the reason and never runs on defaults. With the contract in hand the grain passes the phases LoadingRelease and BuildingContext: it has the release loaded and verified (AIFACTORY-US-463) and asks ContextBuilderService/ContextAssembly.Assemble for the context of the step within Runtime:Context:MaxContextTokens; in I1 the thin ContextBuilderService passes the pinned artifacts of the step through as its context package, without budget, ranking or provenance. The limits of the attempt (AttemptLimits) and maxCorrections come from the contract, and Runtime:Attempts:DefaultMaxCorrections applies when the contract sets none. When release and context are ready the grain writes the first checkpoint, moves the attempt from Claimed to Running (trigger ReleaseLoadedAndContextBuilt) and writes AttemptStarted with attemptId, runId, stepId, tenantId, releaseId, snapshotId and fencingToken in the same transaction; the claim latency objective of the model, from dispatch to AttemptStarted within 5 seconds at the 95th percentile, is measured from the two timestamps. Two further failures end an attempt before it runs. A rejected release ends it Failed with the reason of the rejection. An unavailable ContextBuilderService, a dependency of the degrade kind, is retried within the budget of the attempt and ends it Failed with the failure class transient when the budget is exhausted. Both report their outcome as AIFACTORY-US-467 specifies and use the transition Claimed to Failed under the trigger StartFailed of the model, as does the attempt whose task contract or snapshot cannot be used. Runtime:Host:MaxConcurrentAttempts bounds the attempts one replica executes at once and the free slots are published as connectsoft.factory.runtime.attempt_slots; the registration of that capacity with the execution capacity pool named by Runtime:Host:PoolId is an optional dependency of the degrade kind, and its absence blocks no claim.

Acceptance criteria

AC1 Given a dispatched attempt whose lease is Unclaimed, when RunControlService/AttemptDispatched is handled by ClaimAndStart, then RunControlService/RunControl.AcquireLease is called once with the leaseId of the event and the value of ConnectSoftIdentityPlatform:WorkloadIdentity as holder, the same value on every replica, the AgentRuntimeService/ExecutionAttempt is written in state Claimed with the lease id and the fencing token of lease-v1, its task contract and snapshot content are read through ArtifactService/Artifacts.Fetch by the references of lease-v1 and match the hashes of lease-v1, a task contract that describes the step and carries no attemptId, no stepId of RunControlService and no snapshotId is accepted, the ids of the attempt and the reference of its snapshot being taken from lease-v1, a second attempt of the same step reads the same contract, and after the release is verified and ContextBuilderService/ContextAssembly.Assemble returned the context package the attempt moves to Running with its first checkpoint and AgentRuntimeService/AttemptStarted written in the same transaction (SCN-AgentRuntimeService-007, ADR-0024).
AC2 Given the attempt started, when the outbox dispatcher publishes AgentRuntimeService/AttemptStarted on execution.attempts.v1.attempt-started, then the event is sent in the run event envelope with attemptId, runId, stepId, tenantId, releaseId, snapshotId and fencingToken, a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id, and a redelivered RunControlService/AttemptDispatched starts no second attempt and publishes no second event (SCN-AgentRuntimeService-007, ADR-0012).
AC3 Given a lease already claimed by another holder, an attempt for which Run Control answers NOT_FOUND, or an unreachable Run Control, when ClaimAndStart runs, then in the first two cases the handler stops, no attempt row exists and no event is published, and in the third the handler fails fast and the inbox redelivers; given the lease expired Unclaimed and the same attempt was dispatched again under a new lease, when the claim of the first RunControlService/AttemptDispatched arrives late, then it names the superseded lease, Run Control answers FAILED_PRECONDITION and nothing changes, no row is written for it, and the claim of the second dispatch event, which names the current lease, starts the attempt once with the lease id and the fencing token of its reply lease-v1 (SCN-AgentRuntimeService-007, ADR-0015).
AC4 Given a claimed attempt whose release is rejected, whose context assembly stays unavailable until the budget of the attempt is exhausted, or whose task contract or snapshot cannot be read from ArtifactService or does not match the hash of lease-v1, when the grain gives up, then no AgentRuntimeService/AttemptStarted is written, the attempt ends Failed under StartFailed with the reason, in the second case with the failure class transient, content that does not match its hash is never used, no default stands in for a missing contract, and its single outcome is reported (FR-07, model/services/agent-mesh/AgentRuntimeService.yaml).
AC5 Given a lease-v1 answer of RunControlService/RunControl.AcquireLease that carries an attemptToken, when the AgentRuntimeService/ExecutionAttempt runs and makes a model call, then the attempt presents that token on the OpenAI-compatible route of the model gateway, and the token appears in no row, no checkpoint, no log record and no trace span the service writes, the logs and traces of the attempt being searched for its value by the test (D125, model/services/agent-mesh/AgentRuntimeService.yaml).
Traces: services AgentRuntimeService, RunControlService, ContextBuilderService, ArtifactService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis RunControlService/RunControl.AcquireLease, ContextBuilderService/ContextAssembly.Assemble, ArtifactService/Artifacts.Fetch
Traces: events AgentRuntimeService/AttemptStarted, RunControlService/AttemptDispatched
Traces: descriptors acquire-lease-v1, lease-v1, agent-task-contract-v1, resolved-execution-snapshot-v1, context-package-v1, run-event-envelope-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015, ADR-0024
Traces: requirements FR-03, FR-07
Traces: scenarios SCN-AgentRuntimeService-007
Depends on AIFACTORY-US-461, AIFACTORY-US-463, feature:run-control-service, feature:context-builder-service-thin, feature:artifact-service
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/services/control-plane/RunControlService.yaml, docs/reference/schemas/attempt-outcome.schema.json, docs/reference/schemas/agent-task-contract.schema.json, docs/reference/schemas/resolved-execution-snapshot.schema.json, docs/planes/agent-runtime/services/agent-runtime-service/index.md#sagas-and-long-running-flows

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-200 ClaimAndStart with the AttemptGrain, the lease claim, the context and the first checkpoint implementation The inbox handler ClaimAndStart activates the AttemptGrain, claims the lease through RunControl.AcquireLease with the leaseId of the event and the value of ConnectSoftIdentityPlatform:WorkloadIdentity as holder, the same on every replica, writes the attempt Claimed with the lease id and the fencing token of the reply lease-v1, reads the task contract and the snapshot through the task contract port by the references of the reply and uses them only when they match its hashes (AIFACTORY-TASK-1896), has the release loaded, asks ContextAssembly.Assemble for the context within Runtime:Context:MaxContextTokens, writes the first checkpoint and moves the attempt to Running with AttemptStarted in one transaction; it hands the attemptToken of the reply to the attempt in memory for its model calls and writes it to no row, checkpoint, log or trace, a redaction filter of the logging pipeline removing the member wherever lease-v1 is logged (D125); a redelivery, a lease claimed by another holder and a NOT_FOUND start nothing, a claim made from a stale dispatch event names a superseded lease, is answered FAILED_PRECONDITION by Run Control and writes no row, the dispatch event of the current lease then starts the attempt once, an unreachable Run Control fails fast, and a rejected release, an exhausted context retry, or a task contract or a snapshot that cannot be read or does not match its hash ends the attempt Failed under StartFailed with its reason. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1896 Task handoff read: the task contract and the snapshot from ArtifactService by the references of the lease answer, verified against their hashes implementation The one binding of the task contract port fetches the agent-task-contract-v1 and the resolved-execution-snapshot-v1 of an attempt from ArtifactService through Artifacts.Fetch by taskContractRef and snapshotRef of the reply lease-v1 under the workload token of the service, hashes the content, compares it with taskContractHash and snapshotHash and validates both documents against their schemas, where the task contract describes the step and needs no attemptId, no stepId of RunControlService and no snapshotId, the ids of the attempt and the reference of its snapshot coming from the reply lease-v1; a mismatch, a schema failure, a reference under which ArtifactService holds nothing and an ArtifactService that does not answer end the attempt Failed under StartFailed with the reason, and content that does not match its hash is never used; no seeded source and no second binding exists, no option selects the binding, and a test proves the verified read, the mismatch, the schema refusal and the missing reference. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1863 Fakes of Run Control, the context builder and the artifact store for the claim and the task handoff fixture The acceptance suite holds a fake of RunControl.AcquireLease that claims an Unclaimed lease for the leaseId the claim names and answers lease-v1 with taskContractRef, taskContractHash, snapshotRef, snapshotHash and a recognisable attemptToken (D125), refuses a second holder, answers NOT_FOUND, can be stopped and can expire an Unclaimed lease and dispatch the same attempt again, after which a claim that names the superseded lease is answered FAILED_PRECONDITION and a claim that names the new lease is answered with lease-v1, a fake of ContextAssembly.Assemble that returns the pinned artifacts as a context package or stays unavailable, AttemptDispatched events built from the pinned contracts package, and a fake of Artifacts.Fetch that holds the agent-task-contract-v1 of a step, written without attemptId, stepId and snapshotId and returned under the same reference for every attempt of the step, and the snapshot content under the references of the fake lease answer and can return content that does not match the hash, hold nothing under a reference or be stopped. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1864 Verify the claim, the task handoff, the start and the failures before the start verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-AgentRuntimeService-007"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-464: the single claim that names the lease of the event, the Claimed row, the task contract and the snapshot read by the references of the reply and matching its hashes, the step contract without attemptId, stepId and snapshotId accepted and read again by a second attempt of the step, the first checkpoint and AttemptStarted in one transaction, the payload and the retry of the publication, the redelivered dispatch, the lease of another holder, the NOT_FOUND, the unreachable Run Control, the redispatched lease, where a late claim of the first dispatch event names the superseded lease, is answered FAILED_PRECONDITION and writes no row and the second dispatch event starts the attempt once, the holder value read from ConnectSoftIdentityPlatform:WorkloadIdentity, and the attempt that fails before it runs for a rejected release, an exhausted context retry, content that does not match its hash and a reference under which the artifact store holds nothing, and the attemptToken of the fake lease answer presented on the model call of the attempt and found in no row, checkpoint, log record or trace span. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-465: Run the plan, call, observe and verify loop with a checkpoint before every gated call and bounded self-correction

behavior A Running attempt executes the loop of the service on the execution adapter, the Microsoft Agent Framework under Runtime:Adapter:Kind maf, in-process under its AttemptGrain and through the single execution port of the host. The loop moves through the phases Planning, Calling, Observing, Verifying, Correcting, Checkpointing and Reporting, and every move writes AttemptPhaseChanged with phase and state in the transaction of the transition. A model call goes through the model gateway with the Factory model client of the pinned AI libraries (feature component-extensions-ai): ModelRouterService/ModelGateway.Invoke is called with the pinned model profile, the scope of the attempt and the fencing token and returns the completion with its invocation receipt. The runtime never names a provider or a model; when the gateway is degraded the fallback chain of the profile applies inside ModelRouterService, and when the chain is exhausted the attempt ends Failed with the failure class transient. A tool call goes through the tool gateway with the Factory tool client. The grain first asks PolicyEngineService/Policy.Decide for the decision on the proposed call; a call to a tool outside the allowed tools of the task contract, or a call with a denied decision, is never sent and is a call record with the status failed and the reason of the refusal, which the agent receives as its observation; an allowed call is issued through ToolAdapterService/ToolExecutor.Invoke with the policy decision id and the fencing token. The runtime opens no connection to an external system itself and executes no tool in the host (ADR-0022). Every call is a CallRecord with its sequence, kind, target, request hash, policy decision id and, once observed, the receipt reference and the status issued, observed, uncertain or failed; a prompt, a completion or a tool payload is stored nowhere in agent-runtime-sql. Issuing a tool call moves the attempt from Running to WaitingForTool (trigger ToolCallIssued) and the observed result moves it back to Running (trigger ToolResultObserved). A tool call that the tool gateway reports as failed is an observation and no terminal error: its call record holds the status failed with the receipt reference, the attempt returns to Running, a checkpoint follows, and the agent plans again within its budgets. A checkpoint is written before every gated call, that is every call of class shared-write or above, after every observed outcome, and in addition every Runtime:Attempts:CheckpointEveryCalls calls: the adapter state is stored through ArtifactService as an opaque blob, the Checkpoint row records sequence, phase, the blob reference, the adapter version and its hash, and AttemptCheckpointed with checkpointId, sequence and adapterVersion is written in the same transaction; the objective of the model for a checkpoint write is 500 milliseconds at the 95th percentile. ArtifactService, PolicyEngineService and ToolAdapterService are fail-fast dependencies, and the grain does not wait for one of them. Without the checkpoint or without the decision the gated call is not issued, and without the tool gateway no tool call is. The proposed call is a call record with the status failed and a reason that names the unavailable dependency, and the attempt ends Failed (trigger CorrectionsExhaustedOrTerminalError) with that reason; the terminal state is committed without a further checkpoint. The model names no failure class for an unavailable fail-fast dependency; the class decides whether Run Control retries the attempt, so it is reported to the owner, and until it is decided the suite asserts the state and the reason only. When the agent produced its output, the output is verified against the expected output and the validation of the task contract. A failed verification starts a correction with the validator feedback and counts correctionCount; when the count reaches maxCorrections the attempt ends Failed with the failure class validation and the feedback attached, never as a silent success. The limits of the attempt are enforced by the grain, and each has one observable result. When maxModelCalls or maxToolCalls is reached, the next call of that kind is not issued and the attempt ends Failed (trigger CorrectionsExhaustedOrTerminalError) with a reason that names the exhausted budget; the model names no failure class for it, which is reported to the owner, and the suite asserts the state and the reason. When the reminder call-timeout fires for a model call, the call is abandoned, its call record is failed with the reason, and the attempt ends Failed with the failure class transient, which is the recovery the model states for a degraded model gateway, detected by invocation errors and latency. When it fires for a tool call, the call is interrupted through ToolAdapterService/ToolExecutor.Interrupt and settled by its receipt: observed when the receipt confirms the effect or its absence, upon which the loop continues with that observation, and uncertain when the effect cannot be confirmed, upon which the attempt moves to Uncertain (trigger EffectUnconfirmed). The reminder attempt-budget moves a Running attempt to TimedOut (trigger AttemptBudgetExceeded) when maxMinutes elapse. An exception the adapter does not handle never leaves the attempt without an end: ADR-0014 decision 2 requires the exception path to emit a terminal Failed, so the grain ends the attempt Failed through the same trigger with the reason and a diagnostics reference in the error of the receipt, without prompt or payload text; the class of an unexpected adapter error is not named by the model either and is reported with the two above. When ToolAdapterService announces ToolEffectUncertain for a call of the attempt, the handler SuspendOnUncertainEffect settles the call record as uncertain and the attempt moves from WaitingForTool to Uncertain (trigger EffectUnconfirmed): the runtime retries nothing, connectsoft.factory.runtime.uncertain_outcomes counts it, the outcome Uncertain is reported as it is, and the runtime waits for nothing itself: for a step of class shared-write or above RunControlService opens the operator decision for the Operator of the project, who records the finding on the effect there, applied or not-applied, and chooses retry, compensate or fail; RunControlService records that finding in ToolAdapterService and refuses the choice retry unless the finding is not-applied, and the runtime records no finding and confirms no effect itself (decisions D119 and D121). So that the operator decision knows which effect is in question, the reported outcome marks the tool invocation whose effect is uncertain: the invocationId of the event is written to the optional member uncertainToolInvocationId of attempt-outcome-v1, and RunControlService keeps that id on the operator decision it opens (decision D123). A gated call that needs a human approval stops the loop as AIFACTORY-US-466 specifies; the verified output, the failure, the timeout and the uncertain settlement are reported as AIFACTORY-US-467 specifies.

Acceptance criteria

AC1 Given a Running attempt, when the loop moves between phases, then each move writes AgentRuntimeService/AttemptPhaseChanged in the transaction of the transition and the dispatcher publishes it on execution.attempts.v1.attempt-phase-changed in the run event envelope with attemptId, runId, tenantId, phase and state, retried with backoff on a transport outage without loss or duplicate under its event id (SCN-AgentRuntimeService-008, ADR-0012).
AC2 Given a model call and a tool call of the attempt, when they are issued, then the model call went through ModelRouterService/ModelGateway.Invoke with the pinned profile and the fencing token and its receipt reference is on the call record, the tool call was preceded by PolicyEngineService/Policy.Decide and went through ToolAdapterService/ToolExecutor.Invoke with the policy decision id and the fencing token, the AgentRuntimeService/ExecutionAttempt moved to WaitingForTool and back to Running, a call to a tool outside the allowed tools or with a denied decision was never sent and is a call record with the status failed and the reason of the refusal, a tool call the gateway reported as failed is a call record with the status failed and its receipt reference after which the loop continued, and no prompt, completion or tool payload is stored (ADR-0022, FR-05).
AC3 Given a call of class shared-write or above is about to be issued, or an outcome was observed, when the grain proceeds, then a checkpoint with the next sequence, the phase, the blob reference, the adapter version and its hash exists before the call leaves, AgentRuntimeService/AttemptCheckpointed is written in the same transaction and published on execution.attempts.v1.attempt-checkpointed with attemptId, runId, tenantId, checkpointId, sequence and adapterVersion; when ArtifactService, PolicyEngineService or ToolAdapterService is unavailable the call is not issued, it is a call record with the status failed and a reason that names the dependency, and the attempt ends Failed with that reason and reports its single outcome (SCN-AgentRuntimeService-009, ADR-0024).
AC4 Given an output that fails its verification, when corrections reach maxCorrections, then the attempt ends Failed with the failure class validation and the validator feedback attached and never Succeeded; given the model gateway exhausted the fallback chain of the profile, or call-timeout fired for a model call, then the attempt ends Failed with the failure class transient; given maxModelCalls or maxToolCalls is reached, or the adapter throws an exception it does not handle, then no further call is issued and the attempt ends Failed with a reason that names the cause; given call-timeout fired for a tool call, then the call is interrupted through ToolAdapterService/ToolExecutor.Interrupt and the loop continues when its receipt confirms the effect or its absence, while the attempt moves to Uncertain when it does not; given maxMinutes elapse, then the reminder attempt-budget moves the attempt to TimedOut (XL-04, ADR-0014, model/services/agent-mesh/AgentRuntimeService.yaml).
AC5 Given a tool call in flight for which ToolAdapterService/ToolEffectUncertain arrives, when SuspendOnUncertainEffect handles it, then the call record is uncertain, the attempt moves from WaitingForTool to Uncertain, no model or tool call follows, connectsoft.factory.runtime.uncertain_outcomes counts it, the runtime retries nothing, its single outcome Uncertain is reported with the invocationId of the event in uncertainToolInvocationId of attempt-outcome-v1, and the attempt is terminal and waits for no operator, the operator decision for the step being opened by Run Control and never by the runtime, and the finding on the effect being recorded there by the Operator and never by the runtime (XL-06, docs/planes/agent-runtime/services/agent-runtime-service/index.md).
Traces: services AgentRuntimeService, ModelRouterService, ToolAdapterService, PolicyEngineService, ArtifactService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis ModelRouterService/ModelGateway.Invoke, ToolAdapterService/ToolExecutor.Invoke, ToolAdapterService/ToolExecutor.Interrupt, PolicyEngineService/Policy.Decide
Traces: events AgentRuntimeService/AttemptPhaseChanged, AgentRuntimeService/AttemptCheckpointed, ToolAdapterService/ToolEffectUncertain
Traces: descriptors model-invocation-v1, tool-invocation-v1, policy-decision-v1, agent-task-contract-v1, run-event-envelope-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0022, ADR-0024
Traces: requirements FR-05, NFR-05, XL-04, XL-06
Traces: scenarios SCN-AgentRuntimeService-008, SCN-AgentRuntimeService-009
Depends on AIFACTORY-US-464, feature:component-extensions-ai, feature:component-agent-template-factory-access, feature:model-router-service, feature:model-gateway-host, feature:tool-adapter-service, feature:tool-executor-host, feature:policy-engine-service, feature:artifact-service
Sources model/services/agent-mesh/AgentRuntimeService.yaml, docs/architecture/adr/0022-mcp-as-tool-protocol.md, docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/security.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#sagas-and-long-running-flows

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-201 Model and tool calls of the attempt through the Factory clients with receipts, policy decisions and the fencing token implementation The execution port issues every model call through the Factory model client of the pinned AI libraries to ModelGateway.Invoke with the pinned profile, the scope of the attempt and the fencing token, and every tool call through the Factory tool client to ToolExecutor.Invoke after Policy.Decide, with the policy decision id and the fencing token; each call is a CallRecord with request hash, policy decision id, receipt reference and status; a call to a tool outside the allowed tools or with a denied decision is never sent and is a CallRecord with the status failed and the reason of the refusal; a tool call the gateway reports as failed is a CallRecord with the status failed and its receipt reference, and the loop continues with it as the observation; the profile resolution, the provider call and the fallback stay with ModelRouterService and are not repeated here. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1865 Loop phases, checkpoints, verification with bounded correction and the limits of the attempt in the AttemptGrain implementation The AttemptGrain drives the phases Planning to Reporting on the Agent Framework adapter with AttemptPhaseChanged on every move, writes a checkpoint through ArtifactService before every gated call, after every observed outcome and every Runtime:Attempts:CheckpointEveryCalls calls with AttemptCheckpointed in the same transaction, verifies the output against the task contract, corrects up to maxCorrections and then fails with the class validation, and settles a call as uncertain and the attempt as Uncertain when SuspendOnUncertainEffect receives ToolEffectUncertain, after which the attempt reports that outcome with the invocationId of the event in uncertainToolInvocationId, waits for nothing and holds no request for the operator decision that Run Control opens. Each failure branch has one result: when ArtifactService, PolicyEngineService or ToolAdapterService is unavailable the call is not issued, it is a CallRecord with the status failed and a reason that names the dependency, and the attempt ends Failed under CorrectionsExhaustedOrTerminalError with that reason; a reached maxModelCalls or maxToolCalls issues no further call and ends the attempt Failed with a reason that names the budget; call-timeout on a model call abandons the call and ends the attempt Failed with the class transient; call-timeout on a tool call interrupts it through ToolExecutor.Interrupt and continues the loop when the receipt confirms the effect or its absence, or moves the attempt to Uncertain; attempt-budget moves the attempt to TimedOut; an exception the adapter does not handle ends the attempt Failed with the reason and a diagnostics reference and never leaves it without an outcome. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1866 Scripted agent, scripted model gateway and fakes of the tool gateway, the policy engine and the artifact store fixture The acceptance suite holds a scripted agent of the Agent Framework, a fake of ModelGateway.Invoke that answers from a script, fails or exhausts its fallback, a scripted agent step that throws an exception the adapter does not handle, a fake of ModelGateway.Invoke that does not answer within the call budget, a fake of ToolExecutor.Invoke for a read tool and a shared-write tool that can leave a call in flight, report a call as failed or be stopped, with a fake of ToolExecutor.Interrupt whose receipt confirms or cannot confirm the effect, a fake of Policy.Decide that allows, denies or is stopped, a fake of the artifact store that can be stopped, and ToolEffectUncertain events built from the pinned contracts package. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1867 Verify the loop, its checkpoints, the correction bound and the uncertain effect verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-AgentRuntimeService-008\|Scenario=SCN-AgentRuntimeService-009)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-465: one AttemptPhaseChanged per move, the model call with its receipt, the tool call after its policy decision, the refused call as a call record with the status failed, the tool call reported as failed after which the loop continues, the checkpoint that precedes every gated call with AttemptCheckpointed, the call withheld and the attempt Failed with its reason while the artifact store, the policy engine or the tool gateway is stopped, the validation failure at maxCorrections, the transient failure after an exhausted fallback and after a model call that exceeded its budget, the attempt Failed with its reason at maxModelCalls, at maxToolCalls and after an unhandled exception of the adapter, the tool call that exceeded its budget with a confirmed and with an unconfirmed receipt, the timeout of the attempt and the Uncertain settlement; for the three causes whose failure class the model does not name the tests assert the state and the reason only. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-466: Stop for a human decision: an approval bound to the exact proposal, a missing input or an escalation, and resume, fail or expire on the answer

behavior · persona reviewer As a reviewer, I want an attempt to stop before a gated action and wait for my decision on the exact proposal, or for the input and the choice it asks for, so that nothing I did not decide is executed and an attempt that nobody answers ends instead of waiting for ever.

An attempt stops for a human in three cases, the three values of DecisionKind. Approval: the policy decision for a proposed call requires a human approval, which is the case for calls of class shared-write, external, production, publication and spend above the threshold. The grain writes a checkpoint, asks Run Control through RunControlService/RunControl.RequestApproval with approval-request-v1, which carries the action class, the proposal kind and the hash of the exact proposal under the fencing token of the lease, and receives approval-ref-v1 with the approval id. The approval is owned by Run Control; the runtime decides nothing and only references it. Input: the task cannot proceed without information the requester holds. Escalation: the agent cannot choose between options and lays them before a human with a recommendation. This kind is a request the agent raises inside a running attempt; the failure policy Escalate of a step is another matter, settled in Run Control after the attempt reported its outcome, and the runtime waits for nothing there (decision D119). In each case the grain adds a DecisionRequest with its kind, the status pending and, by kind, the approval id and the proposal hash or the options and the recommendation, moves the attempt from Running to WaitingForDecision (trigger DecisionRequested) and writes AttemptDecisionRequested with decisionId, kind and the optional approvalId in the same transaction; connectsoft.factory.runtime.decisions_waiting counts the waiting attempt by kind, and the lease is renewed while the attempt waits. The answers arrive two ways. An approval is answered by RunControlService/ApprovalDecided in the inbox, handler ResumeAfterApproval. The event carries one of the states of ApprovalState, and the handler has one branch for each state a decided request can carry. An event whose approvalId matches no pending decision request of this service is acknowledged and changes nothing; that is the case for a request whose subject is not a run, a provisioning run, an evaluation run, an asset release or a promotion, which a service that owns the subject requested (D129, D131, D135), for a run-level spend approval, which Run Control opens itself and which names no step and no lease, and for an operator decision, which Run Control opens itself for a step that failed terminally under the failure policy Escalate or for an Uncertain outcome on a step of class shared-write or above and which names no lease (decision D119); the choice retry of the Operator reaches the runtime only as a new AttemptDispatched. Approved, for the proposal hash of the pending request: the request is marked answered, the attempt moves back to Running (trigger DecisionReceived) and the gated call is issued from the checkpoint. A decision that names another proposal hash than the pending request never releases the call. Rejected: the request is marked answered and the attempt ends Failed (trigger DecisionRejectedOrExpired) with the failure class policy-denied, the class the amendment of ADR-0014 gives a run failed by a rejected or expired approval. Expired: the request is marked expired and the attempt ends Failed through the same trigger with the same class. Run Control settles an expired request by the on-expiry rule of the approval policy, fail or re-request. Until ApprovalPolicyService, which is delivered with the approval rules in I4, supplies a rule, every request carries onExpiry fail, so in this delivery an expired approval always fails the run and the attempt with it. ApprovalDecided does not tell the runtime whether Run Control reopened an expired request under a new approval id; that gap is reported to the owner, because once a rule allows a re-request the runtime would otherwise fail an attempt whose run still waits. ChangesRequested: the request is marked answered, the attempt returns to Running (trigger DecisionReceived) with the reason of the reviewer as feedback and without releasing the call, the agent regenerates the proposal, and the grain asks for a new approval with the new hash, which opens a new decision request. Stale: the proposal the request was bound to changed, and Run Control keeps the run WaitingForApproval until the lease holder asks again. The runtime is that lease holder, and nothing else would end the wait, so the grain stays WaitingForDecision, calls RunControlService/RunControl.RequestApproval again for the current proposal hash and replaces approvalId and proposalHash of the pending request with the new reference; no call is released. Cancelled: Run Control cancels a pending request when the run was cancelled or when it failed for another reason. ResumeAfterApproval closes the decision request, releases no call and ends nothing: the attempt does not end Failed and carries no class policy-denied, because the amendment of ADR-0014 gives that class to a rejected or expired approval and to a retired environment and not to a cancellation. The attempt of a cancelled run is settled by the interruption path of AIFACTORY-US-468, through Runtime.Interrupt or RunCancelled, as Cancelled with the class cancellation, whichever of the two events arrives first. When the request was cancelled because the run failed for another reason, the model names no outcome for the attempt: it stays WaitingForDecision until the interrupt or the expiry of its lease (AIFACTORY-US-469) settles it, and the point is reported to the owner. The status of a DecisionRequest in the model is pending, answered or expired and has no value for a cancelled approval; the closed request is stored as expired, and the missing value is reported. An input or an escalation choice is delivered through the internal gRPC method Runtime.AnswerDecision, of class shared-write, by a service with a workload identity whose token holds the service scope that names AgentRuntimeService: the request names the attempt, the decision and the answer, the grain marks the request answered with answeredAt, the attempt returns to Running and the reply is ack-v1. The method is applied once: the same answer repeated for a decision already answered is acknowledged without a second effect, and a different answer is refused with FAILED_PRECONDITION. A call without workload identity is refused with UNAUTHENTICATED and without the service scope with PERMISSION_DENIED; an invalid request, and an answer for a decision of kind Approval, which arrives through ApprovalDecided only, with INVALID_ARGUMENT; an unknown attempt or decision and one of another tenant with NOT_FOUND (ADR-0015); an answer for an attempt that is not WaitingForDecision with FAILED_PRECONDITION. No refusal changes state. An input or an escalation that nobody answers within Runtime:Decisions:InputTimeoutMinutes, 1440 by default, is marked expired and the attempt ends Failed through the same trigger; the model names no failure class for an expired input, which is reported to the owner. Decision requests are audited (docs/planes/agent-runtime/security.md): the model lists no AuditService consumer of AttemptDecisionRequested, so the transaction that opens a request also writes an audit-evidence-v1 record through the sink of AIFACTORY-US-461, and the difference between the page and the model is reported. The owner decided who delivers the answer of a requester (D137): RunControlService calls Runtime.AnswerDecision for its public route RunControlService/AnswerDecision, which a member holding the user scope factory.runs.operate calls behind ControlPlaneApi and through which the Studio answers, and RunControlService passes on only the answer of an input or an escalation, never an approval. The model still gives the method the approval schema as request, which has no member for an input or a choice, so the proto of this PBI names attempt, decision and answer and the descriptor is reported to the owner. StudioBff, StudioNotificationService and DashboardService consume the event in their own PBIs.

Acceptance criteria

AC1 Given a Running attempt whose proposed call needs an approval, when the grain calls RunControlService/RunControl.RequestApproval with approval-request-v1, then a checkpoint exists before the request, a pending decision request of kind Approval holds the approvalId of approval-ref-v1 and the proposal hash, the AgentRuntimeService/ExecutionAttempt is WaitingForDecision and AgentRuntimeService/AttemptDecisionRequested is written in the same transaction; a missing input and an escalation open a request of kind Input or Escalation the same way without calling Run Control (SCN-AgentRuntimeService-010, FR-06).
AC2 Given a decision request was opened, when the outbox dispatcher publishes AgentRuntimeService/AttemptDecisionRequested on execution.attempts.v1.attempt-decision-requested, then the event is sent in the run event envelope with attemptId, runId, tenantId, decisionId, kind and, for an approval, approvalId, a transport outage is retried with backoff without loss or duplicate under its event id, and one audit-evidence-v1 record of the request was written in the transaction that opened it (SCN-AgentRuntimeService-010, ADR-0012).
AC3 Given an attempt waiting for an approval, when RunControlService/ApprovalDecided arrives, then state Approved for the proposal hash of the pending request returns the attempt to Running and the gated call is issued from the checkpoint; state Rejected, and state Expired, which in this delivery always belongs to a request with onExpiry fail because no approval rule exists yet, end the attempt Failed with the failure class policy-denied; a decision for another proposal hash releases nothing; an event whose approvalId matches no pending decision request of this service, such as a run-level spend approval or an operator decision that Run Control opened for the step, is acknowledged and changes nothing; and a redelivered event has no second effect (FUX-05, ADR-0014).
AC4 Given an attempt waiting for an approval, when RunControlService/ApprovalDecided arrives with state ChangesRequested, Stale or Cancelled, then on ChangesRequested the attempt returns to Running with the reason as feedback, no call is released and a new approval is requested through RunControlService/RunControl.RequestApproval with the hash of the regenerated proposal; on Stale the attempt stays WaitingForDecision, RunControlService/RunControl.RequestApproval is called again for the current proposal hash and the pending request holds the new approvalId and proposalHash; on Cancelled the decision request is closed, no call is released, the attempt neither ends Failed nor carries the class policy-denied, and it stays WaitingForDecision without an outcome until the interruption path or the expiry of its lease settles it; that a cancelled run then ends the attempt Cancelled with the class cancellation is proven by AIFACTORY-US-468 (FR-06, model/services/control-plane/RunControlService.yaml).
AC5 Given an attempt waiting for an input or an escalation and a caller with a workload identity and the service scope, when AgentRuntimeService/Runtime.AnswerDecision is called with the attempt, the decision and the answer, then the reply is ack-v1, the decision request is answered with answeredAt, the attempt is Running and the same answer repeated is acknowledged without a second effect; a call without workload identity, without the service scope, with an invalid request or for a decision of kind Approval, for an unknown attempt or one of another tenant, with a different answer for an answered decision or for an attempt that is not WaitingForDecision is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, NOT_FOUND or FAILED_PRECONDITION respectively and changes no state; and a request nobody answers within Runtime:Decisions:InputTimeoutMinutes is expired and the attempt ends Failed (SCN-AgentRuntimeService-005, ADR-0015).
Traces: services AgentRuntimeService, RunControlService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis AgentRuntimeService/Runtime.AnswerDecision, RunControlService/RunControl.RequestApproval
Traces: events AgentRuntimeService/AttemptDecisionRequested, RunControlService/ApprovalDecided
Traces: descriptors approval-request-v1, approval-ref-v1, ack-v1, audit-evidence-v1, run-event-envelope-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-06, FUX-05, XL-05
Traces: scenarios SCN-AgentRuntimeService-005, SCN-AgentRuntimeService-010
Relates: screens approval-card, run-timeline
Depends on AIFACTORY-US-465, feature:run-control-service, feature:policy-engine-service, feature:component-audit-factory-evidence
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/services/control-plane/RunControlService.yaml, docs/reference/schemas/approval.schema.json, docs/planes/agent-runtime/security.md#audit, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1868 Decision requests of the three kinds with the approval request to Run Control and ResumeAfterApproval implementation The grain opens a DecisionRequest of kind Approval, Input or Escalation, for an approval after a checkpoint and RunControl.RequestApproval with the action class, the proposal kind and the proposal hash, moves the attempt to WaitingForDecision with AttemptDecisionRequested and an audit-evidence-v1 record in one transaction, and the inbox handler ResumeAfterApproval has one branch per state of ApprovalDecided: Approved for the pending proposal hash returns the attempt to Running and releases the gated call from the checkpoint; Rejected and Expired end it Failed with the class policy-denied, and no other state does; ChangesRequested returns it to Running with the reason as feedback, releases nothing and leads to a new RunControl.RequestApproval with the hash of the regenerated proposal; Stale keeps it WaitingForDecision, calls RunControl.RequestApproval again for the current proposal hash and replaces approvalId and proposalHash of the pending request; Cancelled closes the decision request, releases nothing and ends nothing, the attempt being settled by the interruption path of AIFACTORY-US-468 or by the expiry of its lease; a decision for another hash releases nothing, and an event whose approvalId matches no pending request of this service is acknowledged without a change. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1869 Runtime.AnswerDecision applied once and the expiry of an unanswered input or escalation implementation The gRPC method Runtime.AnswerDecision, called by RunControlService for its public route AnswerDecision (D137), marks a pending Input or Escalation request answered with answeredAt, returns the attempt to Running and replies ack-v1; the same answer repeated has no second effect; the refusals UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, NOT_FOUND and FAILED_PRECONDITION change nothing; a request unanswered after Runtime:Decisions:InputTimeoutMinutes is marked expired and the attempt ends Failed. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1870 Verify the decision requests, the two answer paths and the expiry verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-AgentRuntimeService-005\|Scenario=SCN-AgentRuntimeService-010)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-466 with a fake of RunControl.RequestApproval that records every request and ApprovalDecided events of each of the six decided states, for the pending proposal hash, for another proposal hash and for an approval id this service never requested: the three kinds of request, the published event and its evidence record, the attempt resumed on Approved, the attempt Failed with policy-denied on Rejected and on Expired, the second approval request with the new hash after ChangesRequested, the repeated approval request and the replaced approvalId and proposalHash after Stale, the Cancelled case, in which the request is closed, no call is released and the attempt is neither Failed nor policy-denied and stays WaitingForDecision without an outcome, its settlement as Cancelled with the class cancellation being proven by AIFACTORY-TASK-1875, the unmatched approval id that changes nothing, the answered input, the repeated answer, each refusal of Runtime.AnswerDecision over two tenants and the expired input request. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-467: Report exactly one outcome per attempt with its hash, its failure class and the fencing token, and take the answer of Run Control as final

behavior An attempt ends with one report and nothing else. When the loop has a verified output (trigger OutputVerified, state Succeeded), when corrections are exhausted or a terminal error occurred (trigger CorrectionsExhaustedOrTerminalError, state Failed), when the budget of the attempt elapsed (trigger AttemptBudgetExceeded, state TimedOut), when cancellation was observed (state Cancelled, AIFACTORY-US-468) or when an effect stayed unconfirmed (state Uncertain), the command ReportOutcome builds attempt-outcome-v1: the attempt id, the lease id, the fencing token, the outcome, the failure class where the outcome has one and the receipt with its hash, the artifacts the attempt stored through ArtifactService, the receipt ids of its model and tool invocations, the usage and the last checkpoint reference. An outcome Uncertain that a tool call caused also marks the tool invocation whose effect is uncertain, in the optional member uncertainToolInvocationId, so that the operator decision of RunControlService keeps that invocation id (decision D123); every other outcome leaves the member out. The grain commits the terminal state, outcomeHash and endedAt and writes AttemptOutcomeReported with state, the optional failureClass, outcomeHash, corrections, modelCalls and toolCalls in one transaction, and then calls RunControlService/RunControl.ReportAttemptOutcome under its workload identity. Run Control accepts an outcome once and only from the holder of the current lease with its fencing token and records the authoritative AttemptOutcomeRecorded; the event of this service is the timeline of the runtime and never the truth of the run (ADR-0024 decision 5). The failure class is one of the six values of the contract, and this PBI binds each value to an outcome in one table. attempt-outcome-v1 lists the six values without binding them to an outcome, and the amendment of ADR-0014 speaks of the class of a Failed attempt; the reading taken here, used in every PBI of the feature and reported to the owner, is that a Failed attempt carries one of transient, permanent, policy-denied or validation wherever the model names its class, a Cancelled attempt carries cancellation, an Uncertain attempt carries uncertain, and a Succeeded or TimedOut attempt carries none. For a Failed attempt the class is transient when the model gateway exhausted the fallback chain, when a model call exceeded its budget or when the context assembly stayed unavailable within the budget; validation when corrections were exhausted; policy-denied when an approval was rejected or expired, and never when it was cancelled; and permanent when a dependency answered NOT_FOUND for a resource the attempt addressed, which is also its answer for a resource of another tenant (ADR-0015 amendment of D114). Seven causes end an attempt Failed for which the model names no class: an unavailable fail-fast dependency (ArtifactService, PolicyEngineService or ToolAdapterService), a reached maxModelCalls or maxToolCalls, an exception the adapter does not handle, a rejected release, a task contract or a snapshot that cannot be read or does not match its hash, an input or an escalation nobody answered in time, and a checkpoint that is incompatible at resume. Each is a row of the table with its reason and with the class left to the owner; until the owner decides, the report of such an attempt carries the reason in the error of the receipt, and the suite asserts the state and the reason only. The runtime never retries an attempt. Run Control retries a Failed attempt only when its class is transient and the step is of class read or sandbox, or of class shared-write with a declared idempotency key (ADR-0014 amendment of D114), so a wrong class would repeat or block an effect, and each class has a test. Nor does the runtime wait for a human after its report. When a step fails terminally under the failure policy Escalate, and when an Uncertain outcome is recorded on a step of class shared-write or above, RunControlService opens an operator decision for the Operator of the project, who chooses retry, compensate or fail (decision D119); for an Uncertain outcome of a tool call the Operator records the finding on the effect there, applied or not-applied, and RunControlService records it in ToolAdapterService (decision D121). The runtime reports such an attempt exactly as any other, Failed with its class or Uncertain with the class uncertain, softens neither into another outcome, holds no decision request for the operator decision, and learns of the choice retry only as a new AttemptDispatched. The report is idempotent on the outcome hash: a repeated ReportOutcome with the same hash sends the same report again and writes no second event, and a different outcome for an attempt that already holds one is refused by the aggregate. Run Control is a fail-fast dependency: when the call fails, the committed outcome stands and the report is repeated with the same hash until it is answered or the lease is gone; no outcome is invented and none is changed. The answers of Run Control are final for the runtime. ack-v1 accepted closes the attempt. FAILED_PRECONDITION, its answer to a stale fencing token because a newer lease exists, and ack-v1 carrying the conflict outcome, its answer for an attempt it already settled, both mean that the truth was recorded without this report: the grain stops repeating, keeps its own row as the timeline of what it did and logs the difference with the attempt id. NOT_FOUND ends the repetition as well. The failure mode of the model for a checkpoint that is incompatible after an adapter upgrade names a failure class infrastructure, which is no value of attempt-outcome-v1; such an attempt is reported Failed with its reason and is one of the seven rows above whose class awaits the owner. MemoryPromotionService, DashboardService and QualityScoreService consume the event in their own PBIs.

Acceptance criteria

AC1 Given a Running attempt whose output was verified, when ReportOutcome runs, then the AgentRuntimeService/ExecutionAttempt is Succeeded with outcomeHash and endedAt, AgentRuntimeService/AttemptOutcomeReported is written in the same transaction, and RunControlService/RunControl.ReportAttemptOutcome is called with attempt-outcome-v1 carrying the lease id, the fencing token, the outcome and the receipt hash and answers ack-v1 accepted (SCN-AgentRuntimeService-012, ADR-0024).
AC2 Given the outcome was committed, when the outbox dispatcher publishes AgentRuntimeService/AttemptOutcomeReported on execution.attempts.v1.attempt-outcome-reported, then the event is sent in the run event envelope with attemptId, runId, tenantId, state, the optional failureClass, outcomeHash, corrections, modelCalls and toolCalls, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-AgentRuntimeService-012, ADR-0012).
AC3 Given attempts that end for each cause of the class table, when they report, then a Failed attempt carries transient for an exhausted model gateway, a model call beyond its budget or an exhausted context assembly, validation for exhausted corrections, policy-denied for a rejected or expired approval and permanent after NOT_FOUND from a dependency; a Cancelled attempt carries cancellation, an Uncertain attempt carries uncertain and, when a tool call caused it, the id of that tool invocation in uncertainToolInvocationId, and a Succeeded or TimedOut report carries none; and an attempt Failed by an unavailable fail-fast dependency, a reached call budget, an unhandled exception of the adapter, a rejected release, a task contract or a snapshot that cannot be read or does not match its hash, an unanswered input or an incompatible checkpoint reports the state Failed with its reason in the error of the receipt, its class being the row the owner decides; every attempt of the table is terminal once it reported and waits for no operator, also when Run Control then opens an operator decision for its step (ADR-0014, ADR-0015).
AC4 Given a committed outcome, when ReportOutcome is repeated with the same hash, when Run Control is unreachable, and when another outcome is attempted, then the same report is sent again without a second AgentRuntimeService/AttemptOutcomeReported, the report is repeated with the same hash after Run Control recovered, and the different outcome is refused by the aggregate (SCN-AgentRuntimeService-012, FR-03).
AC5 Given RunControlService/RunControl.ReportAttemptOutcome answers FAILED_PRECONDITION for a stale fencing token, ack-v1 with the conflict outcome for a settled attempt, or NOT_FOUND, when the answer arrives, then the grain stops repeating the report, changes nothing of its committed outcome and logs the difference with the attempt id (XL-06, model/services/control-plane/RunControlService.yaml).
Traces: services AgentRuntimeService, RunControlService, ArtifactService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis RunControlService/RunControl.ReportAttemptOutcome
Traces: events AgentRuntimeService/AttemptOutcomeReported
Traces: descriptors attempt-outcome-v1, ack-v1, run-event-envelope-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015, ADR-0024
Traces: requirements FR-03, FUX-07, XL-06
Traces: scenarios SCN-AgentRuntimeService-012
Depends on AIFACTORY-US-465, feature:run-control-service, feature:artifact-service
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/services/control-plane/RunControlService.yaml, docs/reference/schemas/attempt-outcome.schema.json, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#failure-and-recovery

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1871 ReportOutcome with the outcome hash, the failure class mapping and the handling of every answer of Run Control implementation The command ReportOutcome builds attempt-outcome-v1 with lease id, fencing token, outcome, failure class and the receipt with its hash, artifacts, invocation receipt ids, usage and checkpoint reference and, on an Uncertain outcome that a tool call caused, the id of that tool invocation in uncertainToolInvocationId, commits the terminal state with outcomeHash and AttemptOutcomeReported in one transaction and then calls RunControl.ReportAttemptOutcome; the mapping of causes to failure classes is one table with a test per row: for a Failed attempt transient (exhausted fallback chain, model call beyond its budget, exhausted context assembly), validation (exhausted corrections), policy-denied (rejected or expired approval, never a cancelled one) and permanent (NOT_FOUND of a dependency), cancellation for a Cancelled attempt, uncertain for an Uncertain attempt, none for Succeeded and TimedOut, and seven rows whose class awaits the owner and whose report carries the reason in the error of the receipt (unavailable ArtifactService, PolicyEngineService or ToolAdapterService, reached maxModelCalls or maxToolCalls, unhandled exception of the adapter, rejected release, task contract or snapshot that cannot be read or does not match its hash, unanswered input or escalation, incompatible checkpoint); a repeat with the same hash sends the same report and writes no second event; an unreachable Run Control is retried with the same hash; FAILED_PRECONDITION, the conflict outcome and NOT_FOUND stop the repetition without changing the committed outcome. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1872 Verify the single outcome, its failure classes and the answers of Run Control verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-AgentRuntimeService-012"; a zero-match filter counts as a failure; with a fake of RunControl.ReportAttemptOutcome that accepts, is stopped, answers FAILED_PRECONDITION, the conflict outcome and NOT_FOUND, the tests prove AC1 to AC5 of AIFACTORY-US-467: the committed outcome with its event, the payload and the retry of the publication, one case per row of the class table, among them a Cancelled attempt with cancellation, an Uncertain attempt with uncertain and with the id of the uncertain tool invocation in uncertainToolInvocationId, a Succeeded and a TimedOut attempt without a class and, asserted by state and reason only, the unavailable fail-fast dependency, the reached call budget and the unhandled exception of the adapter, the repeat with the same hash, the refused second outcome and the end of the repetition on each final answer. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-468: Observe an interruption: stop at the next safe point, settle the calls in flight and report the observed termination

behavior · persona requester As a requester, I want an attempt of a run I cancelled to stop at once and to tell me which effects it could not confirm, so that the Factory stops spending on the run and never pretends that an effect already made did not happen.

Cancellation is two facts, the request and the observed termination (ADR-0014 decision 2). Run Control records the intent and asks the lease holder to stop through the internal gRPC method Runtime.Interrupt, of class shared-write, after its own Cancelling transition committed; the same path is taken inside the service when a release is revoked with the in-flight policy revoke (AIFACTORY-US-463). The caller presents a workload identity whose token holds the service scope that names AgentRuntimeService, and the request names the attempt and the reason, cancellation or revocation. The method records the request on the attempt and answers ack-v1 at once; it does not wait for the stop. The command Interrupt takes effect at the next safe point of the grain: the cancellation token of the execution port is signalled, the loop stops within the cancellation budget of the host options, and after cancellation is observed no further model or tool call is made. A call in flight is settled: a model call is abandoned; a tool call is interrupted through ToolAdapterService/ToolExecutor.Interrupt and settles as observed when its receipt confirms the effect or its absence, and as uncertain when the effect cannot be confirmed. The grain sets cancellationObservedAt, moves the attempt from Running, WaitingForTool or WaitingForDecision to Cancelled (trigger CancellationObserved, and trigger ReleaseRevokedWithPolicyRevoke when the interruption comes from a revocation with the in-flight policy revoke, decision D123), or from WaitingForTool to Uncertain when a call stayed uncertain (trigger EffectUnconfirmed), and writes AttemptCancellationObserved with uncertainCalls, the number of calls it could not confirm, in the same transaction. The outcome Cancelled with the failure class cancellation, or the outcome Uncertain with the class uncertain, is then reported once as AIFACTORY-US-467 specifies. An attempt that waits for an approval is settled the same way whichever event of a cancelled run arrives first: when ApprovalDecided with the state Cancelled closed its decision request before the interrupt arrived (AIFACTORY-US-466), the attempt is still WaitingForDecision and ends Cancelled with the class cancellation, never Failed and never policy-denied. Completed effects stay visible with their receipts and are never rolled back silently. The request is applied once: a repeated Runtime.Interrupt for an attempt whose interruption is already recorded, or that is already terminal, is acknowledged with ack-v1 and changes nothing. A call without workload identity is refused with UNAUTHENTICATED, without the service scope with PERMISSION_DENIED, an invalid request with INVALID_ARGUMENT, and an unknown attempt and an attempt of another tenant alike with NOT_FOUND (ADR-0015); no refusal changes state. For its caller the method is fail-fast: when this service is unavailable, the Cancelling state of Run Control stands and the attempt is settled later by this path or by the expiry of its lease. One event leads to the same observation without the method: RunCancelled of RunControlService, handler ObserveCancellation, settles every attempt of the cancelled run that this service still holds as non-terminal the same way and leaves terminal attempts as they are. An attempt may leave Claimed and Paused for a terminal state (decision D119): a cancellation that reaches an attempt still Claimed or Paused settles it through the transitions Claimed to Cancelled and Paused to Cancelled of the model, which the aggregate of AIFACTORY-US-460 accepts under the trigger CancellationObserved. Cancellation observations are audited (docs/planes/agent-runtime/security.md): the model lists no AuditService consumer of AttemptCancellationObserved, so the transaction of the observation also writes an audit-evidence-v1 record through the sink of AIFACTORY-US-461, and the difference between the page and the model is reported. The model gives Runtime.Interrupt the request schema of the attempt outcome, which has no member for an interrupt request; the proto of this PBI names the attempt and the reason, and the schema is reported to the owner. StudioBff and DashboardService consume the event in their own PBIs.

Acceptance criteria

AC1 Given a Running attempt and a caller with a workload identity and the service scope, when AgentRuntimeService/Runtime.Interrupt is called, then the reply is ack-v1 before the stop, the request is recorded on the AgentRuntimeService/ExecutionAttempt, the loop stops within the cancellation budget, no model or tool call is made after the observation, the attempt is Cancelled with cancellationObservedAt and AgentRuntimeService/AttemptCancellationObserved is written in the same transaction (SCN-AgentRuntimeService-004, ADR-0014).
AC2 Given an attempt WaitingForTool with a call in flight, when the interruption is observed, then the call is interrupted through ToolAdapterService/ToolExecutor.Interrupt, settles as observed when its receipt confirms the effect or its absence and the attempt is Cancelled, or settles as uncertain and the attempt is Uncertain with uncertainCalls counting it; an attempt WaitingForDecision is Cancelled, also when RunControlService/ApprovalDecided with the state Cancelled closed its decision request before the interrupt arrived; in each case one outcome is reported, Cancelled with the failure class cancellation or Uncertain with the class uncertain and never Failed with policy-denied (SCN-AgentRuntimeService-004, XL-06).
AC3 Given the interruption was observed, when the outbox dispatcher publishes AgentRuntimeService/AttemptCancellationObserved on execution.attempts.v1.attempt-cancellation-observed, then the event is sent in the run event envelope with attemptId, runId, tenantId and uncertainCalls, a transport outage is retried with backoff without loss or duplicate under its event id, and one audit-evidence-v1 record of the observation was written in the same transaction (SCN-AgentRuntimeService-011, ADR-0012).
AC4 Given a call to AgentRuntimeService/Runtime.Interrupt without workload identity, without the service scope, with an invalid request, or for an unknown attempt or an attempt of another tenant, when it arrives, then it is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT or NOT_FOUND respectively and no state changes; given the same request repeated, or a request for a terminal attempt, then it is acknowledged with ack-v1 and nothing changes (SCN-AgentRuntimeService-004, ADR-0015).
AC5 Given RunControlService/RunCancelled for a run of which this service holds one non-terminal and one terminal attempt, when ObserveCancellation handles it twice, then the non-terminal attempt is settled once as in AC1, the terminal attempt is unchanged and the redelivery has no second effect (SCN-AgentRuntimeService-011, FR-10).
Traces: services AgentRuntimeService, RunControlService, ToolAdapterService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis AgentRuntimeService/Runtime.Interrupt, ToolAdapterService/ToolExecutor.Interrupt
Traces: events AgentRuntimeService/AttemptCancellationObserved, RunControlService/RunCancelled, RunControlService/ApprovalDecided
Traces: descriptors ack-v1, audit-evidence-v1, run-event-envelope-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-10, FUX-07, XL-06
Traces: scenarios SCN-AgentRuntimeService-004, SCN-AgentRuntimeService-011
Relates: screens run-detail, run-timeline
Depends on AIFACTORY-US-465, AIFACTORY-US-466, AIFACTORY-US-467, feature:run-control-service, feature:tool-adapter-service, feature:component-audit-factory-evidence
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/services/control-plane/RunControlService.yaml, docs/planes/agent-runtime/security.md#audit, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#context-and-boundaries

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1873 Runtime.Interrupt and the Interrupt command with settlement of the calls in flight implementation The gRPC method Runtime.Interrupt records the request and replies ack-v1 at once, is applied once and refuses with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT and NOT_FOUND without a change; the command Interrupt signals the cancellation token of the execution port, stops the loop within the cancellation budget, interrupts a tool call in flight through ToolExecutor.Interrupt, settles each call as observed or uncertain, sets cancellationObservedAt, moves the attempt to Cancelled or Uncertain and writes AttemptCancellationObserved with uncertainCalls and an audit-evidence-v1 record in one transaction, after which the outcome is reported once. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1874 Inbox handler ObserveCancellation for the attempts of a cancelled run implementation The handler of RunCancelled finds the attempts of the run that this service holds as non-terminal, settles each through the Interrupt command, leaves terminal attempts unchanged and is idempotent on the event id. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1875 Verify the interruption, the settled calls and the observation event verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-AgentRuntimeService-004\|Scenario=SCN-AgentRuntimeService-011)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-468 with a tool fake that confirms or cannot confirm a call in flight: the immediate ack-v1, the stop within the budget, the Cancelled and the Uncertain settlement with uncertainCalls, the attempt that waits for an approval and receives ApprovalDecided with the state Cancelled before Runtime.Interrupt, which ends Cancelled with the class cancellation and never Failed with policy-denied, the published event with its evidence record, each refusal over two tenants, the repeated request and the RunCancelled event delivered twice. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-469: Keep the lease while an attempt works, pause without the control plane, recover after a replica crash and settle attempts whose lease is gone

behavior · persona operator As an operator, I want an attempt to keep its lease while it works, to pause when the control plane is away and to be settled when its lease or its replica is gone, so that no attempt runs without a lease, no outcome is invented and an effect that cannot be confirmed is shown to me before anything is repeated.

A lease is a time-boxed right, and the runtime keeps it only by renewing it. The worker LeaseHeartbeat runs every 15 seconds over the attempts in Running, WaitingForTool, WaitingForDecision or Paused and renews each lease through RunControlService/RunControl.AcquireLease, naming the lease and as the holder that claimed it, with three fixed retries five seconds apart and the idempotency key lease id plus renewal sequence, one renewal at a time per attempt; Run Control extends the expiry for the same holder and refuses any other caller. The holder is the value of ConnectSoftIdentityPlatform:WorkloadIdentity, the one identity of the service and the same on every replica, so any replica renews the lease another replica claimed; exclusivity between replicas comes from the single activation of the AttemptGrain and from the fencing token. The worker input of the model lists Paused beside Running, WaitingForTool and WaitingForDecision (decision D119): the trigger LeaseRenewed moves a Paused attempt back to Running, which needs a renewal, so the heartbeat keeps renewing the lease of a Paused attempt. The heartbeat renews through the AttemptGrain of each attempt, and that call is what activates the grain again on a surviving replica after a crash. The round trip is recorded as connectsoft.factory.runtime.control_plane_rtt, and the objective of the model for renewal success is 99.9 percent over one hour. When the renewals of an attempt fail because the Control plane is unreachable, the attempt pauses at its next checkpoint: it moves from Running to Paused (trigger ControlPlaneUnreachable), AttemptPhaseChanged announces the state, and no model or tool call is issued while it is Paused. A renewal that succeeds again moves it back to Running (trigger LeaseRenewed) and the loop continues from the checkpoint. When the Control plane stays away past the lease and its renewal grace, Run Control reconciles the attempt on its side; the runtime invents no outcome for it. LeaseExpired of RunControlService, handler AbandonAttempt, tells the runtime that the Control plane reclaimed a lease. When the event names an attempt this service holds as non-terminal, the grain stops work at once, makes no further model or tool call and settles its row without a lease, as Uncertain when a call was in flight and otherwise as Cancelled; it sends no report under the expired fencing token, because Run Control already settled the attempt and would refuse the report. A renewal that Run Control refuses because the lease is no longer held by this service, or answers with FAILED_PRECONDITION because the lease it names is superseded (decision D121), is treated the same way. An event for a lease that expired Unclaimed carries an empty holder and names an attempt this service never started, so the handler has nothing to stop. A replica crash is detected by Orleans membership and by lease expiry. The AttemptGrain of an attempt whose lease is still valid re-activates on another replica when the heartbeat of a surviving replica calls it for the next renewal, renews the lease under the same holder, and resumes from its last checkpoint when the adapter version and the checkpoint format of that checkpoint are compatible with the replica; work since the last checkpoint is lost, and a step completed before the checkpoint is not executed again. When the checkpoint is not compatible, which is the case after an adapter upgrade, the attempt is not resumed: it ends Failed with the reason, and Run Control starts a new attempt from the step input when its retry rule allows it. The worker OrphanedAttemptReconciler runs every two minutes over the non-terminal attempts that have neither a live grain nor a valid lease, partitioned by tenant with five exponential retries. It settles an attempt with an open call record as Uncertain and any other as Cancelled, writes AttemptOutcomeReported and reports through the path of AIFACTORY-US-467 under the idempotency key attempt id plus outcome hash, so a second pass reports nothing twice. The runtime retries nothing and waits for nothing after that report: for an Uncertain outcome on a step of class shared-write or above RunControlService opens an operator decision, and the Operator of the project records the finding on the effect there, applied or not-applied, before anything is repeated; RunControlService records the finding in ToolAdapterService and refuses the choice retry unless the finding is not-applied (decisions D119 and D121). Settling without a lease uses the trigger OrphanReconciled of the model, under which an attempt leaves Claimed for Cancelled and Paused for Cancelled or Uncertain (decision D119), and the additions of AIFACTORY-US-460: the same trigger to Cancelled from Running, WaitingForTool and WaitingForDecision, and Running to Uncertain under EffectUnconfirmed. The jobs run on the engine registration of AIFACTORY-US-462.

Acceptance criteria

AC1 Given attempts in Running, WaitingForTool, WaitingForDecision and Paused, when LeaseHeartbeat ticks, then each lease is renewed once through RunControlService/RunControl.AcquireLease with the value of ConnectSoftIdentityPlatform:WorkloadIdentity as holder, the same value on every replica, a tick repeated under the same renewal sequence renews nothing twice, and the round trip is recorded as connectsoft.factory.runtime.control_plane_rtt (model/services/agent-mesh/AgentRuntimeService.yaml).
AC2 Given a Running attempt whose renewals fail because Run Control is unreachable, when it reaches its next checkpoint, then the AgentRuntimeService/ExecutionAttempt is Paused, AgentRuntimeService/AttemptPhaseChanged carries the state, no model or tool call is issued while it is Paused, the heartbeat keeps trying to renew the lease of the Paused attempt, a renewal that succeeds moves it back to Running, and no outcome is reported for an attempt that only paused (FR-10).
AC3 Given RunControlService/LeaseExpired for an attempt this service holds as non-terminal, when AbandonAttempt handles it, then the grain stops without a further call, the row is settled Uncertain when a call was in flight and otherwise Cancelled, no report is sent under the expired fencing token, and an event with an empty holder for an attempt that never started changes nothing (XL-06, ADR-0014).
AC4 Given a replica killed in the middle of a loop while the lease is valid, when the renewal call of LeaseHeartbeat re-activates the AttemptGrain on the surviving replica, then that replica renews the lease through RunControlService/RunControl.AcquireLease under the same holder and is not refused, and the grain resumes from the last checkpoint without executing a completed step again when the adapter version and the checkpoint format are compatible, and ends the attempt Failed with the reason without resuming when they are not (ADR-0024, NFR-06).
AC5 Given three non-terminal attempts without a live grain and without a valid lease, a Claimed one, a Paused one and a Running one with an open call record, when OrphanedAttemptReconciler runs twice, then the Claimed and the Paused attempt are settled Cancelled and the third Uncertain, AgentRuntimeService/AttemptOutcomeReported is written once for each and each is reported once under attempt id plus outcome hash (docs/planes/agent-runtime/services/agent-runtime-service/index.md).
Traces: services AgentRuntimeService, RunControlService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis RunControlService/RunControl.AcquireLease
Traces: events AgentRuntimeService/AttemptPhaseChanged, AgentRuntimeService/AttemptOutcomeReported, RunControlService/LeaseExpired
Traces: descriptors acquire-lease-v1, lease-v1
Traces: adrs ADR-0014, ADR-0024
Traces: requirements FR-10, NFR-06, XL-06
Depends on AIFACTORY-US-462, AIFACTORY-US-465, AIFACTORY-US-467, feature:run-control-service
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/services/control-plane/RunControlService.yaml, docs/planes/agent-runtime/workers.md, docs/planes/agent-runtime/durability-spike.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#failure-and-recovery

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1876 LeaseHeartbeat with renewal as holder and the pause and resume of an attempt implementation The job LeaseHeartbeat renews the lease of every attempt in Running, WaitingForTool, WaitingForDecision or Paused through RunControl.AcquireLease, through the AttemptGrain of the attempt, with the value of ConnectSoftIdentityPlatform:WorkloadIdentity as holder, the same on every replica, under the key lease id plus renewal sequence and records connectsoft.factory.runtime.control_plane_rtt; after failed renewals the grain moves the attempt to Paused at its next checkpoint with AttemptPhaseChanged and issues no call, the heartbeat keeps renewing the lease of the Paused attempt, and a renewal that succeeds moves it back to Running. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1877 AbandonAttempt, resume from a compatible checkpoint and the OrphanedAttemptReconciler implementation The inbox handler AbandonAttempt stops the grain of an attempt whose lease expired and settles the row as Uncertain or Cancelled under OrphanReconciled without a report, as does a renewal refused because the lease is no longer held by this service or is superseded; the renewal call of the heartbeat is what re-activates the AttemptGrain of a crashed replica on a surviving one, which renews the lease under the same holder; the re-activated AttemptGrain resumes from the last checkpoint only when adapter version and checkpoint format are compatible and otherwise ends the attempt Failed with the reason; the job OrphanedAttemptReconciler settles non-terminal attempts without a live grain and a valid lease, Claimed and Paused ones included, as Uncertain or Cancelled, writes AttemptOutcomeReported and reports once under attempt id plus outcome hash. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1878 Harness that runs two silo processes, kills one and stops Run Control fixture The acceptance suite starts two replicas of the service as separate processes against an isolated database, kills one without a graceful stop at a chosen step of a scripted loop, plants a checkpoint whose adapter version the replica does not accept, stops and restarts the fake of Run Control, and builds LeaseExpired events with and without a holder from the pinned contracts package. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1879 Verify lease renewal, pause, abandonment, recovery and orphan settlement verification Run dotnet test -c Release --filter "Category=Acceptance&Feature=LeaseAndRecovery"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-469 against RunControlService/RunControl.AcquireLease and RunControlService/LeaseExpired: one renewal per tick with the holder value of ConnectSoftIdentityPlatform:WorkloadIdentity, the Paused attempt that issues no call, whose lease the heartbeat keeps renewing and which resumes, the abandoned attempt without a report, the surviving replica that renews the lease of the resumed attempt under the same holder and is not refused, the resume from a compatible checkpoint and the refusal of an incompatible one, and the three orphaned attempts, a Claimed and a Paused one among them, settled and reported once. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-470: Delegate a sub-agent as a child run within the depth limit and announce the delegation

behavior A supervisor attempt may hand a part of its task to another agent when its task contract and its release allow it. A delegation is not a call from agent to agent inside the attempt: the child is a run of its own, under the same tenant and project scope and the same approval semantics as its parent, and inside one runtime worker the handoff stays a step of the flow and never an A2A call (ADR-0023 decision 1). The execution port of the host raises the delegation. The grain checks the depth of the new child against the limit of the attempt, maxSubAgentDepth of AttemptLimits, with Runtime:Attempts:MaxSubAgentDepth, 2 by default, as its bound, and refuses a delegation beyond it as a failed call on the timeline without submitting anything. An allowed delegation submits its child run through RunControlService/RunControl.SubmitChildRun, the internal child run operation of Run Control (decision D119), under the workload identity of the service: the request run-submission-v1 carries the run of the attempt as parentRunId and the attempt as parentAttemptId, and the reply run-accepted-v1 names the child run. The child inherits the tenant, the project, the policy revision and the pinned release closure of its parent; the runtime sets none of them. The child gets no reservation of its own: its usage is recorded with the child run id and counts on the reservation of the parent run, and the submission waits for no budget reservation and opens no spend approval (decision D121). The operation is of class shared-write and returns the same child run for the same idempotency key with the same input, so a submission repeated after a lost answer creates no second child; the idempotency key of a delegation is the delegationId (decision D121). The accepted delegation is recorded as a CallRecord of kind SubAgent with the child agent as target and as a SubAgentDelegation with delegationId, childRunId, childAgentId, purpose, depth and the status submitted, and SubAgentDelegated with childRunId, childAgentId and depth is written in the same transaction. A child run id is unique among delegations, so a delegation recorded twice is one. Run Control is a fail-fast dependency of the service. A submission it refuses is a call record with the status failed and the reason of the refusal: no delegation is recorded, no event is written, and the supervisor continues with that observation. Under a parent run that is terminal or Cancelling Run Control answers FAILED_PRECONDITION and no child run exists (decision D121); the runtime treats that answer as such a refusal. A submission that gets no answer is repeated under the same key until it is answered or the reminder call-timeout fires, upon which the call record is failed with the reason. One point of the contract is reported to the owner: run-submission-v1 names a workflow and has no member for the child agent or the purpose of a delegation. The agent never spawns an agent outside the scope of its parent run, and the runtime never executes the child inside the parent attempt. While the child runs, the parent attempt keeps its lease and its limits. The outcome of the child returns by event, and the grain follows no stream: RunCompleted, RunFailed and RunCancelled of RunControlService reach the inbox handler SettleDelegation, which finds the delegation by its child run id, sets it to completed, failed or cancelled and returns the outcome of a completed child or the failure of a failed one to the waiting supervisor attempt. The outcome of a completed child is read through the optional outputRef of RunCompleted, the reference of the outcome of the child run in ArtifactService beside its hash in outcomeHash (decision D121). A child run stopped for an overrun of its budget arrives as RunCancelled with the reason budget-exceeded and is a failed child: its delegation is set to failed, not to cancelled, and the supervisor attempt observes a failure (decision D121); a child run stopped because a release of its pinned closure was revoked arrives with the reason release-revoked and is a failed child in the same way (decision D125). The stopped run itself ends Cancelled and not Failed, and only its parent treats it as failed (decision D123). The result is observed like the result of a call, and a checkpoint follows it. A terminal run event that names no child run of a delegation of this service changes nothing in this handler, a redelivered event has no second effect, and RunCancelled reaches this handler beside ObserveCancellation of AIFACTORY-US-468. A parent run that ends Failed or Cancelled has its open child runs cancelled by Run Control, which owns their runs; the runtime cancels no child itself. StudioBff and DashboardService consume the event in their own PBIs.

Acceptance criteria

AC1 Given a supervisor attempt whose depth is below the limit, when its agent delegates, then RunControlService/RunControl.SubmitChildRun is called once with run-submission-v1 carrying the run of the attempt as parentRunId and the attempt as parentAttemptId, under the delegationId as idempotency key, a call record of kind SubAgent and a delegation with the child run id of run-accepted-v1 and the status submitted exist on the AgentRuntimeService/ExecutionAttempt and AgentRuntimeService/SubAgentDelegated is written in the same transaction with attemptId, runId, tenantId, childRunId, childAgentId and depth (SCN-AgentRuntimeService-013, ADR-0023).
AC2 Given the delegation was recorded, when the outbox dispatcher publishes AgentRuntimeService/SubAgentDelegated on execution.attempts.v1.sub-agent-delegated, then the event is sent in the run event envelope with the payload of AC1 and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-AgentRuntimeService-013, ADR-0012).
AC3 Given a delegation that would exceed Runtime:Attempts:MaxSubAgentDepth or the maxSubAgentDepth of the attempt, when it is raised, then it is refused and recorded as a failed call, no child run is submitted and no event is written; given RunControlService/RunControl.SubmitChildRun refuses the submission, as it does with FAILED_PRECONDITION under a parent run that is terminal or Cancelling, when a delegation is raised, then it is a call record with the status failed and the reason of the refusal, no child run exists, no delegation is recorded and no event is written; given a submission repeated under the same idempotency key, the delegationId, after a lost answer, or the same child run recorded twice, then the same child run is returned and one delegation exists (SCN-AgentRuntimeService-013, model/services/agent-mesh/AgentRuntimeService.yaml).
AC4 Given a submitted child run, when RunControlService/RunCompleted, RunControlService/RunFailed or RunControlService/RunCancelled names it and SettleDelegation handles the event, the first twice, then the delegation is completed, failed or cancelled, the result of a completed child is read through the outputRef of RunCompleted, a RunCancelled with the reason budget-exceeded or release-revoked (D125) sets the delegation to failed and the supervisor observes a failed child, although the child run itself ended Cancelled, a checkpoint follows the observation, the parent loop continues with the result or the failure of the child, the redelivery has no second effect, and a terminal run event that names no child run of this service changes no delegation (ADR-0024, ADR-0014).
Traces: services AgentRuntimeService, RunControlService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis RunControlService/RunControl.SubmitChildRun
Traces: events AgentRuntimeService/SubAgentDelegated, RunControlService/RunCompleted, RunControlService/RunFailed, RunControlService/RunCancelled
Traces: descriptors run-submission-v1, run-accepted-v1, run-event-envelope-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0023, ADR-0024
Traces: scenarios SCN-AgentRuntimeService-013
Depends on AIFACTORY-US-465, feature:run-control-service
Sources model/services/agent-mesh/AgentRuntimeService.yaml, model/services/control-plane/RunControlService.yaml, docs/reference/schemas/run-submission.schema.json, docs/architecture/adr/0023-a2a-interoperability.md, docs/planes/agent-runtime/security.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1880 Delegation in the AttemptGrain with the depth limit, the child run submission and the settlement of the child implementation The grain refuses a delegation beyond maxSubAgentDepth as a failed call, submits an allowed one through RunControl.SubmitChildRun with run-submission-v1 carrying the run of the attempt as parentRunId and the attempt as parentAttemptId under the delegationId as idempotency key, records the accepted one as a CallRecord of kind SubAgent and a SubAgentDelegation with the child run id of run-accepted-v1 and the status submitted, writes SubAgentDelegated in the same transaction, records a refused submission as a failed call with its reason without a delegation or an event, the FAILED_PRECONDITION of a parent run that is terminal or Cancelling among the refusals, sets no budget and asks for no reservation for the child, and repeats an unanswered one under the same key until the call budget is exhausted; the inbox handler SettleDelegation finds the delegation of the child run that RunCompleted, RunFailed or RunCancelled names, sets it to completed, failed or cancelled, sets the delegation of a child that RunCancelled names with the reason budget-exceeded or release-revoked (D125) to failed, returns the outcome, read through the outputRef of RunCompleted, or the failure to the waiting supervisor attempt, after which a checkpoint is written, ignores a run that is no child of this service and is idempotent on the event id; a child run id recorded twice is one delegation. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1881 Fake of the child run submission and terminal run events of a child run fixture The acceptance suite holds a fake of RunControl.SubmitChildRun that answers run-accepted-v1 with a child run id, returns the same child run for the same idempotency key, records what was submitted with its idempotency key, can refuse a submission, with FAILED_PRECONDITION for a parent run that is terminal or Cancelling among its refusals, and can be stopped, and RunCompleted events with an outputRef, RunFailed events and RunCancelled events, one of them with the reason budget-exceeded and with the reason release-revoked, built from the pinned contracts package for a child run of a delegation and for a run that is no child. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1882 Verify the delegation, its depth limit and the observed child verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-AgentRuntimeService-013"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-470: the submission through RunControl.SubmitChildRun with the run and the attempt as parent and the delegationId as idempotency key, the recorded delegation with SubAgentDelegated in one transaction, its payload and the retry of the publication, the refused delegation beyond the depth limit, the submission Run Control refuses recorded as a failed call with its reason, the FAILED_PRECONDITION of a terminal and of a Cancelling parent run among them, the same child run for a submission repeated under its key and the single delegation for a repeated child run id, and the completed, the failed and the cancelled child settled by RunCompleted, RunFailed and RunCancelled, the result of the completed child read through the outputRef of RunCompleted, the child stopped with the reason budget-exceeded and one with the reason release-revoked settled as failed delegations, a redelivered event and a terminal event of a run that is no child. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService

AIFACTORY-US-471: Read the timeline and the checkpoints of an attempt and watch its phases, by hash and reference and never by content

behavior · persona operator As an operator, I want to read what an attempt did, call by call and checkpoint by checkpoint, and to watch it while it runs, so that I can explain an outcome and decide on a resume from recorded facts and never from the content of a prompt.

A caller whose scope carries factory.runs.read reads one attempt through GetAttemptTimeline: the identity of the attempt (run, step, release, agent and model profile), its state and phase, the phases it passed, its checkpoints, every call with sequence, kind, target, request hash, policy decision id, receipt reference, status and times, its decision requests with kind, status and approval id, its delegations, and its outcome with the outcome hash. Calls are shown by hash and receipt reference and never by content: the service stores no prompt, no completion and no tool payload, so the response cannot hold one. The same caller lists the checkpoints of the attempt through ListCheckpoints, in sequence order, each with sequence, phase, adapter version, hash and creation time and without the adapter state, which stays an opaque blob; an operator reads the list before deciding whether an attempt can be resumed. Both are operations of class read with natural idempotency and write nothing. Every query runs under the tenant row filter: an attempt of another tenant is answered 404, the same answer as for an attempt id that does not exist, and the answer never confirms that the attempt exists; a token without the scope, and a caller whose token grants no role in the project of the attempt, is answered 403 (ADR-0015 amendments of D113 and D114). The Studio reads both through the platform route (GW-03). A subscriber with a workload identity whose token holds the service scope that names AgentRuntimeService opens the internal server stream Runtime.WatchAttempt for one attempt and receives the phase changes and the checkpoints of that attempt in commit order, each as a run-event-envelope-v1, until the attempt is terminal or the subscriber cancels. The stream is of class read and changes nothing. The model does not say whether the stream first replays what was committed before the subscription; this PBI starts at the subscription and leaves the past to the timeline, and the point is reported. A caller without workload identity is refused with UNAUTHENTICATED, without the service scope with PERMISSION_DENIED, an invalid request with INVALID_ARGUMENT, and an unknown attempt and an attempt of another tenant alike with NOT_FOUND. The summary of the model names the Studio BFF as the receiver of the stream while its inbound dependencies name RunControlService for it; the method admits any workload that holds the service scope, and the difference is reported to the owner. A read returns what was committed: the timeline of the runtime is never the truth of the run, which Run Control holds.

Acceptance criteria

AC1 Given a caller with factory.runs.read and a role in the project of an attempt that has checkpoints, calls, a decision request and an outcome, when AgentRuntimeService/GetAttemptTimeline is called, then the response is 200 with state, phase, the checkpoints, every call by request hash and receipt reference, the decision requests, the delegations and the outcome with its hash, and no field holds a prompt, a completion or a tool payload (SCN-AgentRuntimeService-001, FUX-04).
AC2 Given the same caller, when AgentRuntimeService/ListCheckpoints is called, then the response is 200 with the checkpoints of the attempt in sequence order, each with sequence, phase, adapter version, hash and creation time and without the adapter state (SCN-AgentRuntimeService-002).
AC3 Given a token without factory.runs.read, a token that grants no role in the project of the attempt, an attempt id that does not exist, or an attempt of another tenant, when either read is called, then the answers are 403, 403, 404 and 404, the foreign attempt answered exactly as the unknown one, each with a problem response and no state change (SCN-AgentRuntimeService-001, SCN-AgentRuntimeService-002, ADR-0015).
AC4 Given a subscriber with a workload identity and the service scope, when AgentRuntimeService/Runtime.WatchAttempt is opened for a running attempt, then each later phase change and checkpoint arrives once and in commit order as run-event-envelope-v1 until the attempt is terminal, the stream ends then or when the subscriber cancels, and nothing is written (SCN-AgentRuntimeService-006).
AC5 Given a call to AgentRuntimeService/Runtime.WatchAttempt without workload identity, without the service scope, with an invalid request, or for an unknown attempt or an attempt of another tenant, when it arrives, then it is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT or NOT_FOUND respectively, the foreign attempt exactly as the unknown one, and no stream is opened (SCN-AgentRuntimeService-006, ADR-0015).
Traces: services AgentRuntimeService
Traces: aggregates AgentRuntimeService/ExecutionAttempt
Traces: apis AgentRuntimeService/GetAttemptTimeline, AgentRuntimeService/ListCheckpoints, AgentRuntimeService/Runtime.WatchAttempt
Traces: descriptors run-event-envelope-v1
Traces: adrs ADR-0015, ADR-0028
Traces: requirements FR-03, FUX-04
Traces: scenarios SCN-AgentRuntimeService-001, SCN-AgentRuntimeService-002, SCN-AgentRuntimeService-006
Relates: screens run-timeline, run-detail
Depends on AIFACTORY-US-461, AIFACTORY-US-465
Sources model/services/agent-mesh/AgentRuntimeService.yaml, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md, docs/planes/agent-runtime/security.md, docs/planes/agent-runtime/services/agent-runtime-service/index.md#apis

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1883 GetAttemptTimeline and ListCheckpoints read models under the tenant filter and the project role implementation GetAttemptTimeline returns the attempt with phases, checkpoints, calls by request hash and receipt reference, decision requests, delegations and the outcome; ListCheckpoints returns the checkpoints in sequence order without the adapter state; both apply the tenant row filter so that an attempt of another tenant is answered 404 like an unknown one, answer 403 for a missing scope and for a token without a role in the project of the attempt, and write nothing. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1884 Runtime.WatchAttempt server stream of phase changes and checkpoints in commit order implementation The gRPC method Runtime.WatchAttempt streams one run-event-envelope-v1 per committed phase change and checkpoint of the attempt from the subscription on, in commit order, ends when the attempt is terminal or the subscriber cancels, writes nothing and refuses with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT and NOT_FOUND before a stream is opened. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService
AIFACTORY-TASK-1885 Verify the two reads and the stream with a two-tenant fixture verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-AgentRuntimeService-001\|Scenario=SCN-AgentRuntimeService-002\|Scenario=SCN-AgentRuntimeService-006)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-471: the timeline by hash and reference without content, the ordered checkpoints without adapter state, the 403 and the 404 of each read over two tenants and two projects, the ordered stream that ends with the attempt and each refusal of the stream. ConnectSoft.AI.SoftwareFactory.AgentRuntimeService