Skip to content

AIFACTORY-FEAT-006: RunControlService: complete delivery

Generated from the architecture model

This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.

A design artefact, not a tracker

This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).

Epic Control Plane (PF-01) · Increment I1 Generate a microservice · State elaborated · Route backend-developer · Build order 55 of 185 (Factory code)

RunControlService is the service of bounded context BC-03 (Run Control). Single authority for run, step, attempt and lease state; commits every transition and its outbox record in one transaction (BC-03). Its own repository is ConnectSoft.AI.SoftwareFactory.RunControlService.

Delivers RunControlService (complete)
Stubs —
Realizes adrs: ADR-0012, ADR-0014, ADR-0015
Requirements FR-03, FR-06, FR-10, FUX-04, FUX-05, FUX-07, NFR-04, NFR-05, NFR-06, XL-03, XL-05, XL-06
Depends on —
Sources model/services/control-plane/RunControlService.yaml

Scenarios to trace (24, catalog): SCN-RunControlService-001 ✓, SCN-RunControlService-002 ✓, SCN-RunControlService-003 ✓, SCN-RunControlService-024 ✓, SCN-RunControlService-004 ✓, SCN-RunControlService-005 ✓, SCN-RunControlService-006 ✓, SCN-RunControlService-007 ✓, SCN-RunControlService-022 ✓, SCN-RunControlService-008 ✓, SCN-RunControlService-009 ✓, SCN-RunControlService-010 ✓, SCN-RunControlService-011 ✓, SCN-RunControlService-012 ✓, SCN-RunControlService-013 ✓, SCN-RunControlService-014 ✓, SCN-RunControlService-015 ✓, SCN-RunControlService-021 ✓, SCN-RunControlService-023 ✓, SCN-RunControlService-016 ✓, SCN-RunControlService-017 ✓, SCN-RunControlService-018 ✓, SCN-RunControlService-019 ✓, SCN-RunControlService-020 ✓

AIFACTORY-US-005: Submit a vision and receive a run

behavior · persona requester As a requester, I want to submit a vision brief for my project and receive a run I can follow, so that the Factory starts working on my product without anyone starting workflows by hand.

A requester whose scope carries factory.runs.write submits a vision brief as run-submission-v1 through the gateway route of RunControlService/SubmitRun, with an Idempotency-Key. The project of the run comes from the trusted scope of the authenticated context, never from the body (ADR-0015): run-submission-v1 names a workflow, a release alias or release id and a model profile and may name an environmentId, the environment of the project the run targets, and the caller must be a member of the project the scope names. The service resolves the release alias to an immutable release through AgentRegistryService, evaluates the active policy revision through PolicyEngineService, pins releaseId, policyRevision and snapshotId on the Run, copies the optional environmentId of the submission to Run.environmentId, writes the resolved execution snapshot of the run (resolved-execution-snapshot-v1) to ArtifactService through ArtifactService/Artifacts.Store, once, and only then commits the run in state Accepted, with the reference and the hash of the stored snapshot in snapshotRef and snapshotHash, together with its RunAccepted outbox record in one transaction; in that transaction the release ids the snapshot names in pinnedReleases, the pinned release and every member of its closure, are written to the table RunPinnedReleases, through which the handler of a later revocation finds the runs whose pinned closure holds a revoked release (D125, AIFACTORY-US-736 of feature agent-registry-service-complete); the response is 202 with the run location. A run is never accepted without a stored snapshot (D121): when ArtifactService is unreachable the submission fails fast, nothing is committed or published, and the caller repeats it under the same Idempotency-Key; the store is idempotent on tenant and hash, so the repeat leaves one snapshot. A repeated submission under the same key and the same input hash returns the same run; a different input under the same key is refused as a conflict with 409; an invalid body is refused with 400, and so is a body that carries a parent field, parentRunId or parentAttemptId, because a parent is named only through the internal RunControl.SubmitChildRun (AIFACTORY-US-656, D121); a caller without the scope or who is not a member of the project is refused with 403. A service submits a run through the same route with a service token that holds the service scope factory.runs.submit and names the project of the submission, as QualityScoreService does for the evaluation task runs of its SuiteRunner (D131): the run records the submitting service in submittingService and the requester the token names in requestedBy, and it passes the same gates of the project, the environment and the release, the same budget reservation and the same spend approval as a run a user submitted; a service token that names another project or lacks the scope is refused with 403 and writes nothing. Five gates also answer 409, each with a problem body that names the gate: the project is suspended, closing or closed (GateRunsByProject), states a project reaches through SuspendProject and CloseProject of ProjectService; the alias points at a deprecated release (BlockNewPins); the release the submission would pin is Revoked, because a revoked release is never pinned (ApplyRevocationPolicy, D119), which is also the answer for as long as the alias of the submission points at a Revoked release: the alias stays in place, its resolution names the revoked release, and the answer stays 409 until an administrator moves the alias (D121); the environment that environmentId names is retired (GateRunsByEnvironment); or, the production gate, the environment that environmentId names is of production class and the release the submission pinned is not Published (D117). A gated submission writes no run and publishes nothing. The gates read the last announced state of the project, the release and the environment from the table GateStates, which those four inbox handlers keep from the events of ProjectService, AgentRegistryService and EnvironmentService; for an environment the row keeps the announced class beside the announced state, and the production gate compares that class with the state of the release that the resolution of the release at this submission returned. SubmitRun calls no other service to evaluate a gate, and a run without an environmentId is never gated by an environment. How a project, a release or an environment without a row in GateStates is treated is an open point of the model that the owner has not decided, and this PBI does not decide it. This PBI proves what the publisher can prove: the RunAccepted outbox record is written with the run and published under the canonical envelope, without the optional parentRunId that only the event of a child run carries (D121) and with the optional submittingService for a run a service submitted (D131); WorkflowEngineService starts the vision-to-blueprint workflow run, CostUsageService reserves the budget and AuditService records the submission as evidence from that event, which is the evidence path of a run submission (D135), in their own PBIs. Holding the accepted runs of a frozen environment, releasing them when it is active again and failing them when it is retired is the other half of GateRunsByEnvironment and is delivered by AIFACTORY-US-168. The requester follows the run through the Studio run detail or the CLI, which submits the brief from a file, prints the run identifier and watches the state until the run ends. Of the 26 consumed events of the model the feature registers the handlers of 18, among them BindChangeProposal of ChangeProposalOpened and ChangeProposalMerged (AIFACTORY-US-1606), RefreshSourceScope of SourceScopeChanged (AIFACTORY-US-1607) and PinActivePolicyRevision of PolicyRevisionActivated, which keeps the activated revision of a scope in GateStates for the runs accepted from then on while a running run keeps the revision it pinned (AIFACTORY-TASK-6671, D141). The handlers of five events whose producers arrive in I2 are built and proven in I2 in this repository by the features of those producers (D140): SettleWaitingStep of PipelineRunCompleted by feature pipeline-execution-service-complete (AIFACTORY-US-1608), SettleWaitingStep of PromotionCompleted and PromotionFailed by feature environment-promotion-service (AIFACTORY-US-1621), each looking up the waiting step directly by the runId and stepRunId the event echoes from the requesting step (D141), and HoldDispatchOnQuota of QuotaExceeded and ResumeDispatchOnQuota of QuotaChanged by feature execution-capacity-service-complete (AIFACTORY-US-1609, AIFACTORY-US-1610). The handlers of the three events of RetentionResidencyService, DeletionOrderIssued, LegalHoldChanged and RetentionRuleChanged, are built and proven in I4 in this repository by feature retention-residency-service (D121, D137, D140), and this feature registers none of these eight. The retention sweep of the recovery journal, which applies RunControl:Journal RetentionDays and the handler ApplyRetentionRule, therefore arrives with feature retention-residency-service in I4 (D140): until then the journal keeps every entry and no row of RecoveryJournal is deleted. The table RecoveryJournal is partitioned by the month of OccurredAt from its first migration; only a separate retention identity, used by that sweep (AIFACTORY-US-1459 of feature retention-residency-service), may drop whole partitions older than RunControl:Journal RetentionDays, and the service identity keeps DELETE denied, so the journal stays append-only for the service (D141). When the store run-control-sql is lost it is restored by the runbook of AIFACTORY-TASK-6514. Repository: ConnectSoft.AI.SoftwareFactory.RunControlService, created in I0 by the repository checklist of feature repository-and-pipeline-conventions and taken here as it stands.

Acceptance criteria

AC1 Given a caller whose trusted scope names a project the caller is a member of and a valid run-submission-v1 body, when RunControlService/SubmitRun is called with an Idempotency-Key, then the response is 202 with run-accepted-v1, the RunControlService/Run is in state Accepted with projectId taken from the trusted scope, releaseId, policyRevision and snapshotId pinned and environmentId copied from the submission when the body names one, the resolved execution snapshot of the run was stored through ArtifactService/Artifacts.Store before the run was committed and the run keeps its reference and hash in snapshotRef and snapshotHash (D121), the release ids of pinnedReleases of the snapshot are written to RunPinnedReleases (D125), and the RunControlService/RunAccepted outbox record is written in the same transaction; given instead a service token that holds factory.runs.submit and names the project of the submission, then the run is accepted the same way, after the same gates, and records the submitting service in submittingService (SCN-RunControlService-001, ADR-0014, ADR-0015, D131).
AC2 Given the same submission repeated with the same Idempotency-Key and the same input hash, when it arrives at RunControlService/SubmitRun, then the same run is returned and no second RunControlService/Run exists; given a submission whose snapshot cannot be stored because ArtifactService/Artifacts.Store fails, as when ArtifactService is unreachable, then the call fails fast, no RunControlService/Run is written and nothing is published, and the repeat under the same Idempotency-Key is accepted once the snapshot is stored, so that no run exists without a stored snapshot (SCN-RunControlService-001, D121).
AC3 Given a different body under the same Idempotency-Key, an invalid body or a body that carries a parent field, parentRunId or parentAttemptId (D121), or a caller without factory.runs.write or not a member of the project in scope, or a service token that lacks factory.runs.submit or names another project (D131), when it arrives, then RunControlService/SubmitRun refuses with 409, 400 or 403 respectively; given a project that is suspended, closing or closed, an alias that points at a deprecated release, a release that is Revoked (D119), which includes an alias that points at a Revoked release for as long as an administrator has not moved it (D121), an environmentId that names a retired environment, or an environmentId that names an environment of production class while the pinned release is not Published, the state of the project, the release and the environment and the class of the environment read from GateStates without a call to another service and the state of the pinned release taken from its resolution at this submission, it refuses with 409 and a problem body that names the gate, and a submission without environmentId is never refused by an environment gate; no refusal writes a run or publishes anything (SCN-RunControlService-001).
AC4 Given the run was accepted, when the outbox dispatcher of the run-control messaging profile publishes RunControlService/RunAccepted on run-control.runs.v1.run-accepted, then the event is sent under the canonical envelope with runId, tenantId, projectId, workflowId, releaseId, policyRevision, modelProfileId and requestedBy, without the optional parentRunId, which only the event of a child run carries (D121), and with the optional submittingService only for a run a service submitted (D131), and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-011, ADR-0012).
AC5 Given the run has progressed, when I ask for it through RunControlService/GetRun or the CLI status and watch commands, then I see its state, its steps and the attempt of each step as the single truth of the run (FUX-04).
Traces: services RunControlService, ControlPlaneApi, ProjectService, AgentRegistryService, PolicyEngineService, ArtifactService
Traces: aggregates RunControlService/Run
Traces: apis RunControlService/SubmitRun, RunControlService/GetRun, ArtifactService/Artifacts.Store
Traces: events RunControlService/RunAccepted
Traces: workflows vision-to-blueprint
Traces: descriptors run-submission-v1, run-accepted-v1, resolved-execution-snapshot-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015, ADR-0024
Traces: requirements FR-03, FUX-04, XL-03
Traces: scenarios SCN-RunControlService-001, SCN-RunControlService-011
Relates: screens run-submit, run-detail
Relates: nfrs NFR-05
Depends on feature:control-plane-api, feature:project-service, feature:agent-registry-service-thin, feature:policy-engine-service, feature:artifact-service, feature:repository-and-pipeline-conventions
Sources model/services/control-plane/RunControlService.yaml, model/workflows/vision-to-blueprint.yaml, docs/reference/schemas/run-submission.schema.json, docs/reference/schemas/resolved-execution-snapshot.schema.json, docs/planes/control/services/run-control-service/index.md#apis

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-194 SubmitRun with idempotency and scope checks implementation The operation takes the project from the trusted scope, resolves the release alias, pins release, policy revision and snapshot, copies the optional environmentId of run-submission-v1 to the run, has the resolved execution snapshot stored before the commit (AIFACTORY-TASK-2931), persists the run and its RunAccepted outbox record in one transaction and returns 202 with the run location; the 400, 403 and 409 refusals, the 400 for a body that carries parentRunId or parentAttemptId among them (D121), the idempotent repeat and the five 409 gates (a suspended, closing or closed project, an alias at a deprecated release, a release that is Revoked, which is also the answer while the alias of the submission points at a Revoked release and until an administrator moves it (D121), a retired target environment, and a target environment of production class with a pinned release that is not Published), each answered with a problem body that names the gate, are covered by acceptance tests; the gates read GateStates without a call to another service, the production gate comparing the class GateStates keeps for the environment with the state of the release that the resolution at this submission returned; a submission without environmentId is never refused by an environment gate; a service token that holds the service scope factory.runs.submit and names the project of the submission is accepted beside the user token, the run keeps the submitting service in the nullable column SubmittingService of Runs and the requester the token names in RequestedBy, RunAccepted carries the optional submittingService, and a service token that names another project or lacks the scope is answered 403 (D131). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-1491 BlockNewPins inbox handler and the GateStates rows the submission gates read implementation The handler of AssetReleaseDeprecated records the deprecated release in GateStates, idempotently on the event id, and the handler of AssetReleaseRevoked keeps the state Revoked of a revoked release there the same way, so that a revoked release is never pinned (D119); together with the project rows kept by GateRunsByProject (AIFACTORY-TASK-887) and the environment rows kept by GateRunsByEnvironment (AIFACTORY-TASK-890), which hold the announced class of the environment beside its state, GateStates holds the last announced state of each subject by tenant, subject kind and subject id with its announcedAt, and SubmitRun evaluates its five 409 gates from these rows and from the release state its own resolution returned. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2931 Resolved execution snapshot stored through Artifacts.Store before a run is accepted implementation Before SubmitRun or SubmitChildRun commits a run, the service builds the resolved-execution-snapshot-v1 document of the run as docs/reference/schemas/resolved-execution-snapshot.schema.json defines it, stores it once through a typed gRPC client of Artifacts.Store of ArtifactService and keeps the returned reference and the content hash in SnapshotRef and SnapshotHash of the run (D121); a failed store fails the call fast, commits nothing and publishes nothing; a repeat under the same idempotency key stores the same content under the same tenant and hash, so one snapshot exists for the run; the snapshot is never rewritten after acceptance and RunControlService writes no task contract; in the transaction that accepts the run, the release ids the snapshot names in pinnedReleases are written to the table RunPinnedReleases, keyed by run id and release id with the tenant and pinnedAt, for a child run as for any run (D125); a fake Artifacts.Store endpoint that records the call and can be made unavailable serves the acceptance suite. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-195 GetRun read model with steps and attempts implementation GetRun returns state, steps and attempts for the caller's trusted tenant and project scope only; ListRuns is delivered by AIFACTORY-TASK-861 of AIFACTORY-US-161. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-196 CLI commands submit, status and watch implementation The client submits a brief from a file, prints the run identifier and follows the state until it ends. ConnectSoft.AI.SoftwareFactory.Cli
AIFACTORY-TASK-702 Verify submit-a-vision by running the declared command verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-001\|Scenario=SCN-RunControlService-011)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-005, that is the acceptance, refusals and idempotent repeat of SubmitRun, the snapshot stored through the fake Artifacts.Store endpoint of AIFACTORY-TASK-2931 before the run is accepted, the rows of RunPinnedReleases written with the run, no run when that store fails, the run a service token with factory.runs.submit submits with its submittingService and the 403 of a service token that names another project, the 400 for a body that carries a parent field, the 409 that stays while the alias points at a Revoked release, and the publication and payload of RunAccepted without parentRunId; consumer handling is proven by the suites of WorkflowEngineService and CostUsageService. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-023 Take the repository ConnectSoft.AI.SoftwareFactory.RunControlService as the repository checklist created it in I0 implementation The repository is not created here. It was created in I0 by the repository checklist of feature repository-and-pipeline-conventions, which proves the checklist on this repository (AIFACTORY-TASK-190) and on a repository of a second kind (AIFACTORY-TASK-1514): its first commit is the unchanged output of the scaffold command with UseFactoryProfile and its pipeline is green from the shared steps. This feature takes the repository as it stands: the recorded template version is confirmed, the solution builds, the template tests pass in the repository pipeline, and the work of the feature is added on top without a second creation and without a second scaffold. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-024 Domain model: Run, Step, Attempt, Lease, ApprovalRequest implementation Aggregates with their invariants and state transitions as modelled; unit tests cover every invariant and every transition. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-025 Persistence in run-control-sql: Runs, CancellationIntents, RecoveryJournal, GateStates, RunPinnedReleases, Steps, Attempts, Leases, ApprovalRequests, ApprovalVotes implementation Mappings and migrations for the modelled tables with tenant filters and indexes; RecoveryJournal is created partitioned by the month of OccurredAt, and the service principal is denied UPDATE and DELETE on it, the drop of a whole partition being left to the separate retention identity of the sweep of feature retention-residency-service in I4 (D141); the tables ChangeProposalBindings and AnnouncedSourceScopes are created by AIFACTORY-TASK-6501 and AIFACTORY-TASK-6504 (D141); mapping tests run against an isolated test database, never a shared one. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-026 API hosting skeleton: REST and gRPC hosts, scope enforcement, problem responses and contract documents for the 7 REST and 5 gRPC operations implementation The seven REST routes, AnswerDecision among them (D137), and the five gRPC methods, the internal SubmitChildRun among them (D119), are hosted with JWT validation, scope enforcement per route, problem responses for every modelled refusal code, the Idempotency-Key filter of SubmitRun, workload identity checks on the gRPC methods, and the OpenAPI and proto documents; the behaviour of each operation is delivered by AIFACTORY-US-005, AIFACTORY-US-161 to AIFACTORY-US-167, AIFACTORY-US-656, AIFACTORY-US-1491 and AIFACTORY-US-1492, not here. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-027 Messaging skeleton: outbox and inbox wiring of the run-control profile for 12 published and 18 of the 26 consumed events implementation The outbox writer enlists in the run transaction and the dispatcher of the run-control messaging profile publishes the 12 modelled events under the canonical envelope, ProjectRunsSettled and StepSettled among them; the inbox registers 18 of the 26 consumed events of the model with idempotency on the event id, SpendApprovalRequired and BudgetExceeded of CostUsageService, ChangeProposalOpened and ChangeProposalMerged of PipelineExecutionService and SourceScopeChanged of ProjectService among them; the handlers of PipelineRunCompleted, PromotionCompleted, PromotionFailed, QuotaChanged and QuotaExceeded are built, registered and proven in I2 in this repository by features pipeline-execution-service-complete, environment-promotion-service and execution-capacity-service-complete (D140), PromotionCompleted and PromotionFailed by AIFACTORY-US-1621 of feature environment-promotion-service (D141), and the handlers of DeletionOrderIssued, LegalHoldChanged and RetentionRuleChanged of RetentionResidencyService in I4 by feature retention-residency-service (D121, D137, D140), so this feature registers none of these eight; what each registered handler does is delivered by the behaviour PBIs of this feature, BindChangeProposal by AIFACTORY-US-1606 and RefreshSourceScope by AIFACTORY-US-1607, and PinActivePolicyRevision by AIFACTORY-TASK-6671 (D141). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6671 PinActivePolicyRevision inbox handler of PolicyRevisionActivated implementation The inbox of the run-control profile registers PolicyRevisionActivated of PolicyEngineService with idempotency on the event id; PinActivePolicyRevision keeps the activated revision of its scope, a tenant or a project of the tenant, in GateStates under the subject kind policy-scope with the revision as its state and the time of the envelope, replacing it only with a newer activation; SubmitRun and SubmitChildRun pin that revision on a run accepted from then on and ask Policy.GetActiveRevision of PolicyEngineService only for a scope of which no activation was heard yet, both naming the same revision; a run already accepted keeps the revision it pinned, the handler moves no run, writes no RecoveryJournal entry and publishes nothing, and a revision of one tenant is never read for a run of another (D141). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-028 Worker hosting skeleton: schedules, partitioning and retry policies of LeaseExpiryReconciler, ApprovalExpiryReconciler and RunDispatcher implementation The three workers are hosted with their modelled triggers, retry policies, partitioning by tenant and idempotency keys, and a duplicate trigger has no second effect; what each worker does is delivered by AIFACTORY-US-163, AIFACTORY-US-164, AIFACTORY-US-165 and AIFACTORY-US-167. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-029 Security, tenancy and telemetry of RunControlService implementation Scopes and roles of the model are enforced, the service scopes factory.runs.submit (D131) and factory.approvals.request (D129, D135) and the user scope factory.runs.operate (D137) among them, every query is tenant-filtered, secrets come from secret references, traces, metrics and health checks follow the observability conventions, and the seven metrics of the model are emitted with their dimensions, where connectsoft.factory.runs.active moves with every non-terminal state change with its tenant and state dimensions, connectsoft.factory.approvals.pending_age records the age of an approval request at its decision or its expiry with the dimensions tenant and actionClass, and connectsoft.factory.runs.outbox_backlog reports the number of unpublished outbox records with the dimension host, so that it rises while the transport is unavailable and falls back once publication succeeds (D140). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6514 Recovery runbook of the failure mode Control store restore after loss operations The repository documentation holds the runbook of the failure mode Control store restore after loss of model/services/control-plane/RunControlService.yaml (D30, D140): readiness of the service stays unready while run-control-sql is unavailable; the store is restored from backup to a restore point within the recovery point of 15 minutes of transitions and within the recovery time of 240 minutes; the restore order is the run journal first, the table RecoveryJournal, then the approval requests with their votes, then the release rows that the gates keep in GateStates and RunPinnedReleases, then the outbox and the inbox; after the restore the LeaseExpiryReconciler reconciles every attempt that held an open lease at the loss as Uncertain and the runbook surfaces those runs to the operators, a step of class shared-write or above through the operator decision that the Uncertain outcome opens for the Operator of the project, so that no such attempt is retried automatically; the runbook names the evidence of each step, the restore point, the reconciled attempts and the operator decisions listed by ListApprovals; it holds no credential, connection string or secret value. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-161: Follow a run through its read model, its list and its state stream

behavior · persona requester As a requester, I want to read one run with its steps, attempts, lease and pending approvals, list the runs of my scope and watch a run change state, so that the run state I see in the Studio and the CLI is the single truth the service holds, never an assumed one.

A caller whose scope carries factory.runs.read reads one run through RunControlService/GetRun and receives run-view-v1: the run state and version, the pinned release, policy revision and snapshot, the steps in ordinal order with the attempts of each, the current lease of the open attempt and the pending approval requests. The same caller lists runs through RunControlService/ListRuns and receives run-list-v1, paged, with state and freshness, filtered by the trusted tenant and the projects in scope (ADR-0015). Every query runs under the tenant row filter of the service, so a run of another tenant is never listed and is not found when addressed: the read answers 404, the same answer as for a runId that does not exist, and never confirms that the run exists (ADR-0015). A caller without the scope is refused with 403; nothing is written by any of these reads. A subscriber with a workload identity (StudioBff or a runtime worker, each in its own increment) opens the server stream RunControlService/RunControl.WatchRun with a run-ref-v1 and receives one run-event-v1 per run and approval state change of that run in commit order, until the run is terminal or the subscriber cancels; over gRPC the not-found answer is the status NOT_FOUND, given alike for an unknown run and for a run of another tenant (ADR-0015). Every transition of the Run between two states of RunState is announced by RunControlService/RunStateChanged with fromState, toState and cause, written in the same transaction as the transition and published under the canonical envelope; the views of StudioBff and DashboardService that consume it belong to their own PBIs.

Acceptance criteria

AC1 Given a caller with factory.runs.read and a run in the caller's tenant and project with steps, an open attempt under lease and a pending approval, when RunControlService/GetRun is called, then the response is 200 with run-view-v1 carrying state, version, releaseId, policyRevision, snapshotId, the steps with their attempts, the current lease and the pending approval (SCN-RunControlService-002).
AC2 Given a caller without factory.runs.read, or a runId that exists only in another tenant, when RunControlService/GetRun is called, then the response is 403 for the missing scope and 404 for the foreign run, the same answer as for a runId that does not exist, each with a problem response and no state change (SCN-RunControlService-002, ADR-0015).
AC3 Given runs of several projects and tenants, when RunControlService/ListRuns is called with a page cursor, then the response is 200 with run-list-v1 holding only the runs of the caller's trusted tenant and projects in scope, each with state and freshness, and runs of other tenants never appear (SCN-RunControlService-003, ADR-0015).
AC4 Given a subscriber with a workload identity and a run-ref-v1 of a run in its scope, when RunControlService/RunControl.WatchRun is called, then the stream delivers one run-event-v1 per run and approval state change in commit order until the run is terminal, a caller without identity is refused with Unauthenticated, and an unknown run and a run of another tenant are both refused with the gRPC status NOT_FOUND, which never confirms that a foreign run exists (SCN-RunControlService-010, ADR-0015).
AC5 Given the RunControlService/Run moved between two states through a modelled transition, when the outbox dispatcher publishes RunControlService/RunStateChanged on run-control.runs.v1.run-state-changed, then the outbox record was written in the same transaction as the transition, the event is sent under the canonical envelope with runId, tenantId, fromState, toState and cause, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-012, ADR-0012).
Traces: services RunControlService
Traces: aggregates RunControlService/Run, RunControlService/Step, RunControlService/Attempt
Traces: apis RunControlService/GetRun, RunControlService/ListRuns, RunControlService/RunControl.WatchRun
Traces: events RunControlService/RunStateChanged
Traces: descriptors run-view-v1, run-list-v1, run-ref-v1, run-event-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-03, FUX-04
Traces: scenarios SCN-RunControlService-002, SCN-RunControlService-003, SCN-RunControlService-010, SCN-RunControlService-012
Relates: screens run-detail, my-work, project-overview
Relates: nfrs NFR-05
Depends on AIFACTORY-US-005
Sources model/services/control-plane/RunControlService.yaml, model/descriptors.yaml, docs/planes/control/services/run-control-service/index.md#apis

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-861 GetRun and ListRuns read model with tenant row filter, paging and freshness implementation GetRun returns run-view-v1 with steps, attempts, current lease and pending approvals; ListRuns returns run-list-v1 paged with state and freshness; both apply the tenant row filter so a run of another tenant is answered 404, the same answer as for an id that does not exist (ADR-0015), and a missing scope is 403. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-862 WatchRun server stream fed from committed transitions implementation RunControl.WatchRun streams one run-event-v1 per run and approval state change in commit order and ends at the terminal state or when the caller cancels; callers without workload identity are refused with Unauthenticated, an invalid request with InvalidArgument, and an unknown run and a run of another tenant alike with NotFound, the gRPC status NOT_FOUND (ADR-0015). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-863 Contract of the run read model, the run list, the run reference and the stream item contract The OpenAPI document and the proto of the service expose run-view-v1 and run-list-v1 matching docs/reference/schemas/resolved-execution-snapshot.schema.json, run-event-v1 matching docs/reference/schemas/run-event-envelope.schema.json and run-ref-v1 matching docs/reference/schemas/run-submission.schema.json, as model/descriptors.yaml records them. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-864 Verify follow-a-run through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-002\|Scenario=SCN-RunControlService-003\|Scenario=SCN-RunControlService-010\|Scenario=SCN-RunControlService-012)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-161, including the publication and payload of RunStateChanged; consumer handling is proven by the consumers' own suites. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-162: Cancel a run by recording intent, interrupting the lease holder and observing the settled state

behavior · persona requester As a requester, I want to cancel a run I no longer need and see when it has really stopped, so that the Factory stops spending on it without pretending that effects already made never happened.

A caller whose scope carries factory.runs.write cancels a run of the caller's tenant and project through RunControlService/CancelRun. The service records a CancellationIntent with the actor and the reason first. A run still Accepted moves to Cancelled at once, and so does a Held run, which has no open attempt (trigger CancelRequestedWhileHeld); a Running or WaitingForApproval run moves to Cancelling. After the Cancelling transition and its RunStateChanged outbox record have committed, the service calls AgentRuntimeService/Runtime.Interrupt for the open attempt of the run; AgentRuntimeService stops the attempt at its next safe point and reports the observed termination through RunControlService/RunControl.ReportAttemptOutcome. The Interrupt call is fail-fast: when AgentRuntimeService is unavailable the call fails, the committed Cancelling state stands, and the open attempt is settled later by the lease holder's report or by the LeaseExpiryReconciler. The response is 202: the resulting state is observed, not assumed. A pending approval request of the run moves from Pending to Cancelled through the modelled trigger RunCancelled, no later than the run reaching Cancelled. When every attempt has settled the run moves from Cancelling to Cancelled and RunControlService/RunCancelled is published with reason and requestedBy; AgentRuntimeService, WorkflowEngineService, CostUsageService and GenerationRecipeService, which returns the recipe of a cancelled run to Approved (D123), consume it in their own PBIs, and AuditService consumes it as the cancellation evidence, with the recorded intent and the settled attempts. Completed external effects stay visible and are never rolled back silently. The inbox handler GateRunsByProject takes the same path when ProjectService announces through ProjectService/ProjectStateChanged that a project is Suspended, Closing or Closed: every open run of the project records a CancellationIntent with the handler as actor and follows the cancel transitions, idempotently on the event id; the handler also keeps the announced state of the project in the table GateStates, which the submission gate of AIFACTORY-US-005 reads. A project becomes Suspended through SuspendProject of ProjectService, so this path is reachable on a deployed instance (D117). For a Closing project the handler also announces, once no non-terminal run of the project remains, that its runs are settled (AIFACTORY-US-548), the handler CancelRunOnBudgetExceeded records the same kind of intent for a run whose usage took its budget over the limit (AIFACTORY-US-547), and a parent run that ends Failed or Cancelled records it for each of its open child runs (AIFACTORY-US-656, D119). A caller without the scope is refused with 403; a run of another tenant is answered 404, the same answer as for a run that does not exist (ADR-0015); a run already Completed, Failed or Cancelled is refused with 409; a repeated cancel of a run already Cancelling returns 202 and records no second intent. Every transition and its outbox record are committed in one transaction.

Acceptance criteria

AC1 Given a Running run of my tenant and project with an open attempt, when RunControlService/CancelRun is called with a reason, then the response is 202, a CancellationIntent is recorded, the RunControlService/Run moves to Cancelling with RunControlService/RunStateChanged written in the same transaction, and after that transaction commits AgentRuntimeService/Runtime.Interrupt is called for the open attempt; an unavailable AgentRuntimeService fails the call fast and leaves the Cancelling state standing (SCN-RunControlService-004).
AC2 Given a run still Accepted, a Held run, or one that is WaitingForApproval, when RunControlService/CancelRun is called, then the Accepted run and the Held run move to Cancelled at once, the Held run through the trigger CancelRequestedWhileHeld, the waiting run moves to Cancelling, and its pending RunControlService/ApprovalRequest moves from Pending to Cancelled through the trigger RunCancelled no later than the run reaching Cancelled (SCN-RunControlService-004).
AC3 Given a caller without factory.runs.write, a run of another tenant, or a run already Completed, Failed or Cancelled, when RunControlService/CancelRun is called, then the response is 403, 404 or 409 respectively, the 404 being the same answer as for a run that does not exist, with a problem response and no state change (SCN-RunControlService-004, ADR-0015).
AC4 Given a Cancelling run whose last open attempt settles, or an Accepted or Held run cancelled at once, when the run moves to Cancelled and the outbox dispatcher publishes RunControlService/RunCancelled on run-control.runs.v1.run-cancelled, then the outbox record was written in the same transaction as the transition, the event is sent under the canonical envelope with runId, tenantId, reason and requestedBy from the intent, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-015, ADR-0012).
AC5 Given ProjectService/ProjectStateChanged announces a project moved to Suspended, Closing or Closed, when the GateRunsByProject inbox handler runs, then every open run of that project records a CancellationIntent with the handler as actor and follows the cancel transitions of AC1 and AC2, and a redelivery of the same event id records no second intent (SCN-RunControlService-015).
Traces: services RunControlService, AgentRuntimeService, ProjectService
Traces: aggregates RunControlService/Run, RunControlService/ApprovalRequest
Traces: apis RunControlService/CancelRun, AgentRuntimeService/Runtime.Interrupt
Traces: events RunControlService/RunStateChanged, RunControlService/RunCancelled, ProjectService/ProjectStateChanged
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-10, FUX-07
Traces: scenarios SCN-RunControlService-004, SCN-RunControlService-015
Relates: screens run-detail
Relates: nfrs NFR-05
Depends on AIFACTORY-US-005, feature:agent-runtime-service, feature:project-service
Sources model/services/control-plane/RunControlService.yaml, model/services/agent-mesh/AgentRuntimeService.yaml, docs/planes/control/services/run-control-service/index.md#failure-and-recovery

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-865 CancelRun with cancellation intent, the Cancelling transitions and the Interrupt call to the lease holder implementation CancelRun records the CancellationIntent, applies CancelRequestedBeforeDispatch, CancelRequestedWhileHeld or CancelRequested in the same transaction as the RunStateChanged outbox record, moves the pending approval to Cancelled through RunCancelled, and after the commit of a Cancelling transition calls AgentRuntimeService/Runtime.Interrupt for the open attempt through the gRPC client with fail-fast handling (a failed call is logged, the Cancelling state stands and the attempt settles by report or lease expiry); 403, 404 and 409 refusals are covered by acceptance tests. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-866 Settle a Cancelling run to Cancelled and publish RunCancelled implementation When the last open attempt of a Cancelling run settles, the run moves to Cancelled through AllAttemptsSettled and RunCancelled is written to the outbox with reason and requestedBy; a run cancelled at once from Accepted or Held writes RunCancelled in the transaction of that transition; completed external effects stay recorded. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-887 GateRunsByProject inbox handler cancelling the open runs of a suspended, closing or closed project implementation The handler of ProjectStateChanged keeps the announced state of the project in GateStates for the submission gate of SubmitRun, records a CancellationIntent with the handler as actor for every open run of a suspended, closing or closed project, applies the cancel transitions through the same path as CancelRun, and is idempotent on the event id. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-867 CLI command cancel with the observed final state implementation The client sends the cancel request, prints the 202 acknowledgement and, when asked to wait, follows the run until it reports Cancelled, Completed or Failed. ConnectSoft.AI.SoftwareFactory.Cli
AIFACTORY-TASK-868 Verify cancel-a-run through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-004\|Scenario=SCN-RunControlService-015)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-162 with a fake Runtime.Interrupt endpoint recording the call, including the publication and payload of RunCancelled; consumer handling is proven by the consumers' own suites. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-163: Dispatch each step as an attempt under an exclusive lease with a fencing token

behavior When WorkflowEngineService announces a step through WorkflowEngineService/StepReady, the inbox handler CreateStepAndDispatch creates the Step idempotently on the step run id. StepReady carries actionClass, the side-effect class of the step in the pinned workflow definition, and idempotent, true when that step declares an idempotency key; CreateStepAndDispatch copies them to sideEffectClass and idempotent of the Step; the retry rule and the Uncertain rule of AIFACTORY-US-164 read them from the Step. StepReady also carries the stepRunId of the engine and the optional failurePolicy the step declares in the pinned workflow definition (Retry, Compensate, Escalate or Fail), and the handler keeps both on the Step (D119): stepRunId is the id that StepSettled names (AIFACTORY-US-658), and failurePolicy settles whether a terminal failure of the step opens an operator decision (AIFACTORY-US-657). StepReady carries as well taskContractRef and taskContractHash, the reference and the hash of the task contract of the step (agent-task-contract-v1) that WorkflowEngineService stored in ArtifactService before it announced the step (D121). Both members are optional on StepReady: an agent or tool step carries them, and a Human step, which has no task contract, carries neither. The handler keeps them on the Step, and every attempt of the step, the first one and a retried one alike, takes them from the step, so that a retried attempt carries the same reference and the same hash; the Step of a Human step keeps neither, and its approval is opened in place of an attempt (D123). Once CostUsageService confirms the run budget through CostUsageService/BudgetReserved, after every recorded attempt outcome, and when EnvironmentService/EnvironmentStateChanged arrives, which is how a run released from Held is evaluated again, the RunDispatcher worker selects the next Pending step of a Running run that has no open attempt, creates the Attempt with the next number and its Lease with a new fencing token, moves the step to Dispatched and the run from Accepted to Running on the first dispatch, and writes RunControlService/AttemptDispatched in the same transaction; the worker is idempotent on step id plus attempt number, so a duplicate trigger dispatches nothing twice. A child run has no reservation of its own, so the RunDispatcher awaits no BudgetReserved for a run that carries parentRunId (AIFACTORY-US-656, D121): the first attempt of a child run is dispatched when StepReady announces its first step, the dispatcher selecting the child run that is still Accepted and whose first step is ready (D123). The fencing token of a new lease is strictly greater than any earlier token of the attempt, as the Lease invariant requires; as a design choice of this PBI it is also greater than every earlier token of the step, so the holder of a retried attempt's predecessor is fenced out. The dispatcher creates the lease in state Unclaimed with an empty holder and with expiresAt at its creation plus RunControl:Leases:ClaimTimeoutSeconds, and AttemptDispatched already carries leaseId and fencingToken. The task of the attempt is handed over by reference (D119): the Attempt keeps the reference and the hash of its task contract (agent-task-contract-v1) in taskContractRef and taskContractHash, and the Run keeps the reference and the hash of its resolved execution snapshot (resolved-execution-snapshot-v1) in snapshotRef and snapshotHash beside snapshotId. Each document has one writer (D121): WorkflowEngineService writes the task contract before it announces the step, and the reference and the hash the attempt keeps are the ones StepReady carried and the Step keeps; RunControlService writes the snapshot once, when it accepts the run (AIFACTORY-US-005), and writes no task contract itself. The task contract describes the step and no attempt: attemptId, the stepId of RunControlService and snapshotId are no required members of agent-task-contract-v1, and an attempt receives its ids and the reference of the snapshot in the answer of AcquireLease (D123). A runtime worker with a workload identity claims the lease on start through RunControlService/RunControl.AcquireLease and names in acquire-lease-v1 the leaseId it saw in AttemptDispatched: the first call on the Unclaimed lease records the caller as holder, moves the lease to Claimed and replaces the claim-timeout expiresAt with the lease duration from RunControl:Leases:DurationSeconds, and the reply is lease-v1 with the lease id, that expiry, the fencing token and the four values of the handoff, taskContractRef, taskContractHash, snapshotRef and snapshotHash. The reply carries references and hashes and never the content; the runtime reads both documents from ArtifactService by reference and verifies each hash in its own PBI. The reply also carries, in the required member attemptToken of lease-v1, a short-lived token bound to the run, the step and the attempt, which names them in the claims run_id, step_id and attempt_id beside the tenant claim tid; it is the proof that a model call of the attempt on the OpenAI-compatible route of ModelGatewayHost belongs to the run (D125). RunControlService issues no token itself, because the Authorization Server is the sole token issuer (ADR-0008, ADR-0015): when it grants the lease it obtains the token through a token exchange with its own workload identity, the token lives no longer than the lease, a renewal returns a token that lives no longer than the renewed lease, and the token is never written to a log, an event or a table. When the token exchange is not answered the claim fails fast, the lease stays Unclaimed and nothing is written, and the worker repeats the claim. A later call by the same holder renews the lease by extending its expiry and returns the same references and hashes; a call by any other caller is refused. A claim that names a lease other than the current lease of the attempt, a superseded one, as after the earlier lease expired unclaimed and the attempt was dispatched again, is refused with FailedPrecondition, the gRPC status FAILED_PRECONDITION (D121), and changes nothing, so a worker that acts on an old AttemptDispatched never takes over the newer lease. A lease nobody claimed by its claim timeout is Expired by the LeaseExpiryReconciler and the attempt is dispatched again without counting against maxAttempts (AIFACTORY-US-164). Exactly one current lease exists per attempt, only one non-terminal attempt exists per step, and no step is dispatched while the run is WaitingForApproval, Cancelling or Held (the environment gate of AIFACTORY-US-168); for that gate the dispatcher reads the announced state of the target environment from the table GateStates and calls no other service. A caller without workload identity, an invalid request, an unknown or settled attempt, or a lease claimed by another holder is refused with the matching gRPC status and nothing is written; an attempt of another tenant is answered like an unknown one, with the status NOT_FOUND (ADR-0015).

Acceptance criteria

AC1 Given a Running run with no open attempt and a Pending RunControlService/Step that CreateStepAndDispatch created with sideEffectClass copied from actionClass and idempotent copied from idempotent of WorkflowEngineService/StepReady, when the RunDispatcher runs after CostUsageService/BudgetReserved, after an RunControlService/AttemptOutcomeRecorded or after an EnvironmentService/EnvironmentStateChanged that released the run, then an RunControlService/Attempt, which takes from its step the taskContractRef and the taskContractHash that the step keeps from WorkflowEngineService/StepReady (D121, D123), and its RunControlService/Lease are created, the lease in state Unclaimed with an empty holder, expiresAt at its creation plus RunControl:Leases:ClaimTimeoutSeconds and a fencing token strictly greater than any earlier token of the attempt (Lease invariant) and, by the design choice of this PBI, than any earlier token of the step, the step is Dispatched, and RunControlService/AttemptDispatched is written in the same transaction with runId, stepId, attemptId, leaseId, fencingToken, agentId and skillId (SCN-RunControlService-016).
AC2 Given the attempt was dispatched, when the outbox dispatcher publishes RunControlService/AttemptDispatched on run-control.attempts.v1.attempt-dispatched, then the event is sent under the canonical envelope with the payload of AC1, a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id, and a duplicate dispatch trigger for the same step and attempt number creates no second attempt (SCN-RunControlService-016, ADR-0012).
AC3 Given a runtime worker with a workload identity and a dispatched attempt whose RunControlService/Lease is Unclaimed (claim) or Claimed by that worker (renewal), when RunControlService/RunControl.AcquireLease is called with acquire-lease-v1 naming the leaseId the worker saw in RunControlService/AttemptDispatched, then the first call records the worker's identity as holder, moves the lease to Claimed and replaces the claim-timeout expiresAt with the lease duration from RunControl:Leases:DurationSeconds, the reply carries lease-v1 with the leaseId, that expiresAt, the fencingToken, the taskContractRef and taskContractHash the RunControlService/Attempt keeps for its agent-task-contract-v1, which are the ones WorkflowEngineService/StepReady carried for the step and the RunControlService/Step keeps, so that a retried attempt of the step gets the same reference and hash (D123), and the snapshotRef and snapshotHash the RunControlService/Run keeps for the resolved-execution-snapshot-v1 that RunControlService stored when it accepted the run, and the attemptToken that the Authorization Server issued through the token exchange of RunControlService, bound to the run, the step and the attempt by the claims run_id, step_id, attempt_id and tid and living no longer than the lease (D125), and a later call by the same holder renews the lease by extending expiresAt and returns the same references and hashes with a token that lives no longer than the renewed lease (SCN-RunControlService-007, D119, D121, ADR-0015).
AC4 Given a caller without workload identity, an invalid request, an unknown or settled attempt, an attempt of another tenant, a lease that another holder has claimed, or a claim that names a superseded lease, one that is no longer the current lease of the attempt, when RunControlService/RunControl.AcquireLease is called, then the call is refused with the matching gRPC status, the status NOT_FOUND for the unknown and for the foreign attempt alike and the status FAILED_PRECONDITION for the superseded lease (D121), and no lease is written or changed, the current lease of the attempt staying Unclaimed or with its holder (SCN-RunControlService-007, ADR-0015).
AC5 Given a run that is WaitingForApproval, Cancelling or Held, or a step with an open attempt, when the RunDispatcher evaluates it, then no attempt is dispatched for that run or step, as the invariants of RunControlService/Step and the environment gate of RunControlService/Run require, the gate being read from GateStates without a call to another service; given a WorkflowEngineService/StepReady of a Human step, which carries neither taskContractRef nor taskContractHash, then the RunControlService/Step is created and keeps neither, and its approval is opened in place of an attempt; and given an Accepted child run, one that carries parentRunId, when WorkflowEngineService/StepReady announces its first step, then its first attempt is dispatched without a CostUsageService/BudgetReserved for it (SCN-RunControlService-016, ADR-0014, D123).
Traces: services RunControlService, WorkflowEngineService, CostUsageService, AgentRuntimeService, EnvironmentService
Traces: aggregates RunControlService/Run, RunControlService/Step, RunControlService/Attempt, RunControlService/Lease
Traces: apis RunControlService/RunControl.AcquireLease
Traces: events RunControlService/AttemptDispatched, RunControlService/AttemptOutcomeRecorded, RunControlService/RunStateChanged, WorkflowEngineService/StepReady, CostUsageService/BudgetReserved, EnvironmentService/EnvironmentStateChanged
Traces: descriptors acquire-lease-v1, lease-v1, agent-task-contract-v1, resolved-execution-snapshot-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015, ADR-0024
Traces: requirements FR-03, XL-06
Traces: scenarios SCN-RunControlService-007, SCN-RunControlService-016
Relates: nfrs NFR-05
Depends on AIFACTORY-US-005, feature:workflow-engine-service, feature:cost-usage-service-complete, feature:agent-runtime-service, feature:environment-service
Sources model/services/control-plane/RunControlService.yaml, model/services/agent-mesh/AgentRuntimeService.yaml, docs/planes/control/services/run-control-service/index.md#workers-and-jobs

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-869 RunDispatcher worker and the StepReady and BudgetReserved inbox handlers implementation CreateStepAndDispatch creates steps idempotently on the step run id, each with sideEffectClass copied from actionClass and idempotent copied from idempotent of the StepReady event and with the stepRunId and the failurePolicy the event carries kept on the step (D119); the handler also keeps on the step the optional taskContractRef and taskContractHash of StepReady, mapped to the nullable columns TaskContractRef and TaskContractHash of the table Steps, which an agent or tool step carries and a Human step does not (D123); ReleaseFirstStepOnReservation, the AttemptOutcomeRecorded trigger and the EnvironmentStateChanged trigger run the RunDispatcher, and for a child run, one that carries parentRunId and awaits no BudgetReserved, the StepReady of its first step runs it while the child is still Accepted (D123); the dispatcher creates the attempt with the taskContractRef and taskContractHash it takes from the step, the same for the first attempt and for every later attempt of the step, RunControlService writing no task contract itself (D121), and its lease, Unclaimed with an empty holder, expiresAt at its creation plus RunControl:Leases:ClaimTimeoutSeconds and the next fencing token, moves the step to Dispatched and the run to Running, and writes AttemptDispatched in the same transaction, idempotent on step id plus attempt number. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-870 AcquireLease for the claim and the renewal of the lease created at dispatch implementation RunControl.AcquireLease takes acquire-lease-v1 with the leaseId the caller saw in AttemptDispatched and claims the Unclaimed lease created at dispatch, so the first call records the caller as holder, moves the lease to Claimed and replaces the claim-timeout expiresAt with the lease duration from RunControl:Leases:DurationSeconds; a later call by the same holder renews the lease with a fresh expiry; every reply is lease-v1 with the fencing token and with taskContractRef and taskContractHash of the attempt and snapshotRef and snapshotHash of the run, references and hashes without the content (D119), and with the attemptToken that a typed client of the token exchange of the Authorization Server obtains with the workload identity of the service when the lease is granted or renewed, bound to the run, the step and the attempt by the claims run_id, step_id, attempt_id and tid and expiring no later than the lease, never logged, published or stored, while an unanswered exchange fails the claim fast and leaves the lease Unclaimed (D125); a claim that names a lease other than the current lease of the attempt, a superseded one, is refused with FailedPrecondition, the gRPC status FAILED_PRECONDITION (D121), and changes nothing; callers without workload identity, invalid requests, unknown or settled attempts and any caller other than the holder of a Claimed lease are refused with Unauthenticated, InvalidArgument, NotFound or FailedPrecondition, and an attempt of another tenant is answered NotFound, the gRPC status NOT_FOUND, like an unknown one (ADR-0015). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-871 Acceptance fixture of a fake runtime worker that claims attempts and renews leases fixture A test double subscribes to AttemptDispatched, claims the lease named by the event through AcquireLease with its workload identity, renews it and exposes the leases it holds, so the dispatch and lease scenarios run without AgentRuntimeService. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-872 Verify dispatch-attempts-under-lease through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-007\|Scenario=SCN-RunControlService-016)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-163, including the publication and payload of AttemptDispatched, the references and hashes of the task contract and of the snapshot in lease-v1 on the claim and on the renewal, the attemptToken with its four claims and an expiry no later than the lease, obtained from a fake token exchange endpoint, and the claim that fails without a write while that endpoint does not answer (D125), the task contract reference and hash being the ones the StepReady of the step carried and the step keeps, also on a retried attempt (D123), the step of a Human step that keeps neither and gets no attempt, the first attempt of an Accepted child run dispatched on the StepReady of its first step without BudgetReserved, and the claim that names a superseded lease refused with FAILED_PRECONDITION without a change; consumer handling is proven by the suite of AgentRuntimeService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-164: Record the outcome of an attempt exactly once, reported by the lease holder or reconciled on expiry

behavior The holder of the current lease reports the single outcome of its attempt through RunControlService/RunControl.ReportAttemptOutcome with attempt-outcome-v1: the outcome (Succeeded, Failed, TimedOut, Cancelled or Uncertain), the failure class and the receipt hash. The service accepts the report only from the holder of the current lease and only once: the attempt is settled with outcome, outcomeHash, failureClass and settledAt, the lease moves to Released, and RunControlService/AttemptOutcomeRecorded with any RunStateChanged is written in the same transaction; the reply is ack-v1 with the accepted outcome. Refusals follow one mapping: a caller without workload identity is refused with Unauthenticated; an invalid request with InvalidArgument; a report under a stale fencing token, because a newer lease exists, with FailedPrecondition and the report is audited; a second report for an attempt already settled is answered with ack-v1 carrying the conflict outcome and is audited. Every refusal changes nothing. When a Claimed lease passes its expiry plus RunControl:Leases:RenewalGraceSeconds without an outcome, the LeaseExpiryReconciler moves the lease to Expired, marks the attempt TimedOut, or Uncertain when the sideEffectClass of the Step is shared-write or above, publishes RunControlService/LeaseExpired with the holder, and re-dispatches a TimedOut attempt whose number is below maxAttempts with a new fencing token; an Uncertain attempt of a side-effecting step is never retried automatically, whether or not the step declares an idempotency key: an Uncertain outcome on a step of class shared-write or above, reported by the holder or set by the reconciler, opens an operator decision for the Operator of the project, whatever failure policy the step declares, and the run waits for the choice (AIFACTORY-US-657, D119). An Uncertain outcome that the holder reports may mark the tool invocation whose effect is uncertain, in the optional member uncertainToolInvocationId of attempt-outcome-v1, and the operator decision it opens keeps that id (AIFACTORY-US-726, D123). An Unclaimed lease is a different case, because no worker ever started the attempt: its expiresAt was set at creation plus RunControl:Leases:ClaimTimeoutSeconds, and the renewal grace applies to Claimed leases only, so when that expiresAt passes without a claim the reconciler moves the lease to Expired without waiting for the grace, publishes RunControlService/LeaseExpired with an empty holder, which is an optional field of the payload, and the attempt is dispatched again under a new fencing token, without counting against maxAttempts. A Failed attempt is retried below maxAttempts only when its failureClass is transient and the sideEffectClass of the Step is read or sandbox, or shared-write when the Step is idempotent, that is when the step of the pinned workflow definition declares an idempotency key (ADR-0014): the RunDispatcher then dispatches the next attempt of the step with a new lease and fencing token. Both rules read sideEffectClass and idempotent from the Step, where CreateStepAndDispatch copied them from actionClass and idempotent of the StepReady event of WorkflowEngineService (AIFACTORY-US-163). Any other Failed attempt, and a TimedOut or retriable Failed attempt that reached maxAttempts, fails the step terminally: the step is settled Failed and announced by StepSettled, and RunControlService does not fail the run for it (D121). WorkflowEngineService applies the failure policy of the step and reports WorkflowRunFailed, and only on that report does the run fail (AIFACTORY-US-167): under Fail at once, under Compensate once the compensation finished, and under Retry the same way as under Fail, because a step settled Failed under the failure policy Retry has its attempts spent and RunControlService has already made every retry the rule allows (D123). When the failurePolicy the Step keeps from StepReady is Escalate, the step is not settled and the operator decision of AIFACTORY-US-657 is opened instead (D119); a retry the Operator chooses there dispatches one more attempt beyond maxAttempts (D121). A step that reaches its final state is announced once by StepSettled, which is what tells WorkflowEngineService that the step is settled; an attempt outcome alone settles no step for the engine (AIFACTORY-US-658). WorkflowEngineService, CostUsageService and SecretBrokerService consume the events in their own PBIs.

Acceptance criteria

AC1 Given the holder of the current lease of an open attempt, when RunControlService/RunControl.ReportAttemptOutcome is called with attempt-outcome-v1 carrying the outcome, failure class and receipt hash, then the reply is ack-v1 accepted, the RunControlService/Attempt is settled with outcome, outcomeHash, failureClass and settledAt, the RunControlService/Lease moves to Released and RunControlService/AttemptOutcomeRecorded is written in the same transaction (SCN-RunControlService-008).
AC2 Given a report to RunControlService/RunControl.ReportAttemptOutcome from a caller without workload identity, with an invalid request, under a stale fencing token, or for an attempt already settled, when it arrives, then the call is refused with Unauthenticated, InvalidArgument or FailedPrecondition respectively, the settled attempt is answered with ack-v1 carrying the conflict outcome, the stale and the second report are audited, and no state changes in any of these cases (SCN-RunControlService-008, ADR-0014).
AC3 Given the attempt was settled, when the outbox dispatcher publishes RunControlService/AttemptOutcomeRecorded on run-control.attempts.v1.attempt-outcome-recorded, then the outbox record was written in the same transaction as the settlement, the event is sent under the canonical envelope with runId, stepId, attemptId, outcome, failureClass and outcomeHash, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-017, ADR-0012).
AC4 Given a Claimed RunControlService/Lease whose expiresAt plus RunControl:Leases:RenewalGraceSeconds has passed without an outcome, when the LeaseExpiryReconciler runs, then the lease moves to Expired, the attempt is settled TimedOut, or Uncertain when the sideEffectClass of its step is shared-write or above, RunControlService/LeaseExpired is written in the same transaction and published with attemptId, leaseId and holder, a TimedOut attempt whose number is below maxAttempts is re-dispatched with a new fencing token, and an outcome later reported under the expired token is refused with FailedPrecondition and audited; given instead an Unclaimed lease whose expiresAt, its creation plus RunControl:Leases:ClaimTimeoutSeconds, has passed without a claim, then the lease moves to Expired without the renewal grace, which applies to Claimed leases only, RunControlService/LeaseExpired is written in the same transaction and published with attemptId, leaseId and an empty holder, and the attempt is dispatched again under a new fencing token without counting against maxAttempts (SCN-RunControlService-018, XL-06).
AC5 Given a settled attempt whose number is below the maxAttempts of its RunControlService/Step, when the RunDispatcher evaluates the step after RunControlService/AttemptOutcomeRecorded, then a Failed attempt is retried with a new attempt, lease and fencing token only when its failureClass is transient and the sideEffectClass of the step is read or sandbox, or shared-write with idempotent true, both read from the step, any other Failed attempt fails the step terminally without a retry, which settles the step Failed and leaves the run Running until the engine reports its failure (AIFACTORY-US-167, D121), and an Uncertain attempt on a step whose sideEffectClass is shared-write or above is never retried automatically, idempotency key or not, and is surfaced to the Operator of the project through the operator decision that AIFACTORY-US-657 opens, as the invariants of RunControlService/Attempt require (SCN-RunControlService-017, SCN-RunControlService-018, FUX-07).
Traces: services RunControlService, AgentRuntimeService
Traces: aggregates RunControlService/Attempt, RunControlService/Lease, RunControlService/Step
Traces: apis RunControlService/RunControl.ReportAttemptOutcome
Traces: events RunControlService/AttemptOutcomeRecorded, RunControlService/LeaseExpired, RunControlService/AttemptDispatched, RunControlService/RunStateChanged
Traces: descriptors attempt-outcome-v1, ack-v1
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-10, FUX-07, XL-06
Traces: scenarios SCN-RunControlService-008, SCN-RunControlService-017, SCN-RunControlService-018
Relates: nfrs NFR-05, NFR-06
Depends on AIFACTORY-US-163
Sources model/services/control-plane/RunControlService.yaml, docs/planes/control/services/run-control-service/index.md#failure-and-recovery

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-873 ReportAttemptOutcome guarded by the current lease and the fencing token implementation RunControl.ReportAttemptOutcome settles the attempt once from the current lease holder, moves the lease to Released and writes AttemptOutcomeRecorded and RunStateChanged in one transaction; a missing workload identity is Unauthenticated, an invalid request InvalidArgument, a stale fencing token FailedPrecondition and audited, and a second report for a settled attempt is answered with ack-v1 carrying the conflict outcome and audited; every refusal changes nothing. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-874 LeaseExpiryReconciler with TimedOut and Uncertain settlement and guarded retry implementation The scheduled worker moves Claimed leases past expiresAt plus the renewal grace to Expired, settles the attempt TimedOut or Uncertain by the sideEffectClass of its step, publishes LeaseExpired, re-dispatches TimedOut attempts below maxAttempts with a new fencing token, never retries an Uncertain attempt of a shared-write step, with or without an idempotency key, and hands it to the operator decision of AIFACTORY-TASK-2646 (D119), and hands an attempt at maxAttempts to the terminal step failure of AIFACTORY-TASK-888, which settles the step and does not fail the run (D121); it moves an Unclaimed lease past its expiresAt (creation plus RunControl:Leases:ClaimTimeoutSeconds) to Expired without the renewal grace, which applies to Claimed leases only, publishes LeaseExpired for it with an empty holder, the optional field of the payload, and dispatches the attempt again under a new fencing token without counting against maxAttempts; it is idempotent on leaseId plus fencingToken. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-875 Metrics and operator surfacing of expired leases and Uncertain attempts implementation connectsoft.factory.leases.expired and connectsoft.factory.attempts.duration are emitted with their modelled dimensions, and Uncertain attempts that will not retry are listed for operators through the run read model. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-889 Retry rule of a Failed attempt in the RunDispatcher by failure class and side-effect class implementation After AttemptOutcomeRecorded the RunDispatcher dispatches the next attempt of a step whose last attempt Failed below maxAttempts only when the failureClass is transient and the sideEffectClass of the Step is read or sandbox, or shared-write with idempotent true; both values are read from the Step and from nowhere else; every other Failed attempt goes to the terminal step failure of AIFACTORY-TASK-888 without a retry, where the step is settled Failed and the run is left to the report of the engine (D121); the decision is idempotent on step id plus attempt number and unit tests cover each failure class against each side-effect class with and without the idempotent flag. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-876 Verify record-attempt-outcome-once through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-008\|Scenario=SCN-RunControlService-017\|Scenario=SCN-RunControlService-018)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-164, including the refusal mapping of ReportAttemptOutcome, the retry rule of a Failed attempt (a transient failure retried on a read or sandbox step and on a shared-write step with an idempotency key, never on a shared-write step without one), the Unclaimed lease expired after the claim timeout without the renewal grace and dispatched again without counting against maxAttempts, and the publication and payload of AttemptOutcomeRecorded and LeaseExpired, the latter with the holder of a Claimed lease and with an empty holder for an Unclaimed one; consumer handling is proven by the consumers' own suites. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-165: Open a human approval bound to the exact proposal before a gated action

behavior A runtime worker holding the current lease of an attempt whose step is gated asks for a human decision through RunControlService/RunControl.RequestApproval with approval-request-v1: the action class of the gated action, the hash of the exact artifact, diff, manifest or tool arguments, and the proposal kind. Every approval request names its subject as a kind and an id from the closed list run, provisioning-run, evaluation-run, asset-release and promotion, and runId is optional, present when the subject is a run or belongs to one (D129, D131, D135); a lease holder names the subject kind run with the id of its run, and its workload token holds the scope factory.approvals.request. A request for a subject that a service owns, made with the service identity of that service and moving no run, is delivered by AIFACTORY-US-1492. The service takes the approver rule the request supplies and resolves it through PolicyEngineService only when the request carries none (D141), sets the expiry from RunControl:Approvals:DefaultExpiryHours unless policy shortens it, opens the Pending ApprovalRequest, moves the run from Running to WaitingForApproval and writes RunControlService/ApprovalRequested and RunControlService/RunStateChanged in the same transaction; the reply is approval-ref-v1 with the approval id and the expiry, and the approval-expiry reminder of the run grain is scheduled. The request carries onExpiry (fail or re-request) and reRequestsLeft, copied when the request is made from the ApprovalRule of ApprovalPolicyService that belongs to the policy revision the run pinned, reRequestsLeft from the maxReRequests of that rule. The request carries as well quorum, the number of eligible approvals it needs, copied from the quorum of the same rule (D127). While no approval rule is supplied, which is the case until ApprovalPolicyService arrives with the approval rules in I4 (D126, D127), the request carries onExpiry fail, reRequestsLeft 0 and quorum 1, so no re-request happens and one eligible approval decides it. The request is also accepted from a run that is WaitingForApproval when no request of the step is Pending, which is the case after the earlier request went Stale: the new Pending request is opened for the current proposal hash, ApprovalRequested is written and the run stays WaitingForApproval. After a ChangesRequested decision the run is Running again, so the lease holder regenerates the proposal and asks through the first path. A Human step announced by WorkflowEngineService opens its approval the same way from the inbox handler. Two Human steps are never announced and so open no approval here (D133): from I4 a Human step whose policy guard the policy answers with allow is recorded Skipped by the engine, which emits no StepReady for it, and a Human step that waits on the promotion gate of EnvironmentPromotionService advances on the ApprovalDecided of that promotion, whose request EnvironmentPromotionService makes for its own subject (AIFACTORY-US-1492). A run-level spend approval is not requested through this method: it is opened by the inbox handler of SpendApprovalRequired of CostUsageService, names no step, no lease and no fencing token, and its expiry always fails the run (AIFACTORY-US-546). An operator decision is not requested through it either: RunControlService opens that request itself, with the kind operator-decision, when a step fails terminally under the failure policy Escalate or an Uncertain outcome is recorded on a step of class shared-write or above, and an expired one is never requested again and fails the run (AIFACTORY-US-657, D119). A repeated request from the same attempt for the same proposal hash returns the existing reference. A caller without workload identity, an invalid request, a stale fencing token, an unavailable PolicyEngineService, or a request for another proposal hash while a request of the step is still Pending is refused with the matching gRPC status and no request is opened. The event is published under the canonical envelope; StudioBff, StudioNotificationService and AuditService consume it in their own PBIs. The ApprovalExpiryReconciler moves a request past its expiry to Expired and writes ApprovalDecided with state Expired. What follows is settled by the onExpiry and reRequestsLeft the expired request carries: when onExpiry is fail, or is re-request and reRequestsLeft is 0, the run moves from WaitingForApproval to Failed with failure class policy-denied; when onExpiry is re-request and reRequestsLeft is above 0, a new Pending request is opened for the same proposal hash with the same onExpiry and with reRequestsLeft counted down by one, and the run stays WaitingForApproval; a request made while no approval rule was supplied carries fail and fails the run the same way. A Pending request whose run fails for any other reason moves from Pending to Cancelled through the trigger RunFailed in the transaction of the run transition, and ApprovalDecided with state Cancelled is written with it; this covers the request of a WaitingForApproval run that is failed by a retired environment (AIFACTORY-US-168).

Acceptance criteria

AC1 Given a lease holder of an attempt on a step with gate approval in a Running run, when RunControlService/RunControl.RequestApproval is called with approval-request-v1, then the reply carries approval-ref-v1 with approvalId and expiresAt, a Pending RunControlService/ApprovalRequest bound to the proposalHash exists with the subjectKind run, the subjectId and the runId of the run (D129, D135), its approverRule and policyRevision and with onExpiry, reRequestsLeft and quorum copied from the ApprovalRule when the request is made, fail, 0 and 1 while no approval rule is supplied (D127), and the run is WaitingForApproval with RunControlService/RunStateChanged and RunControlService/ApprovalRequested written in the same transaction; given a run already WaitingForApproval with no Pending request of the step, as after a Stale request, the call opens the new Pending request for the current proposalHash, writes RunControlService/ApprovalRequested and leaves the run WaitingForApproval (SCN-RunControlService-009).
AC2 Given a caller without workload identity, an invalid request, a stale fencing token, an unavailable PolicyEngineService, or a different proposalHash while a request of the step is still Pending, when RunControlService/RunControl.RequestApproval is called, then the call is refused with the matching gRPC status, no RunControlService/ApprovalRequest is opened and the run keeps its state (SCN-RunControlService-009).
AC3 Given the request was opened, when the outbox dispatcher publishes RunControlService/ApprovalRequested on run-control.approvals.v1.approval-requested, then the outbox record was written in the same transaction as the request, the event is sent under the canonical envelope with approvalId, runId, the subjectKind run and the subjectId (D129, D135), stepId, tenantId, actionClass, proposalHash, approverRule and expiresAt and without the optional kind, whose absence means approval (D119), and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-019, ADR-0012, FUX-05).
AC4 Given a Pending RunControlService/ApprovalRequest whose expiresAt has passed, when the ApprovalExpiryReconciler runs, then the request becomes Expired and RunControlService/ApprovalDecided with state Expired is written in the same transaction; by the onExpiry and reRequestsLeft the expired request carries, the run either moves from WaitingForApproval to Failed with RunControlService/RunStateChanged and failure class policy-denied, when onExpiry is fail or reRequestsLeft is 0, or a new Pending request is opened for the same proposalHash with reRequestsLeft counted down by one and the run stays WaitingForApproval, when onExpiry is re-request and reRequestsLeft is above 0; a request made while no approval rule was supplied carries fail and fails the run the same way; a second run of the reconciler for the same approvalId and version changes nothing (SCN-RunControlService-020, FR-06).
AC5 Given a Pending RunControlService/ApprovalRequest of a run that fails for a reason other than a decision on or the expiry of that request, as when the target environment of the WaitingForApproval run is retired, when the RunControlService/Run moves to Failed, then the request moves from Pending to Cancelled through the trigger RunFailed in the transaction of the run transition and RunControlService/ApprovalDecided with state Cancelled is written with it, so that no Pending request outlives its failed run (SCN-RunControlService-020, FR-06).
Traces: services RunControlService, PolicyEngineService, ApprovalPolicyService, AgentRuntimeService
Traces: aggregates RunControlService/ApprovalRequest, RunControlService/Run, RunControlService/Step
Traces: apis RunControlService/RunControl.RequestApproval
Traces: events RunControlService/ApprovalRequested, RunControlService/ApprovalDecided, RunControlService/RunStateChanged
Traces: descriptors approval-request-v1, approval-ref-v1
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-06, FUX-05
Traces: scenarios SCN-RunControlService-009, SCN-RunControlService-019, SCN-RunControlService-020
Relates: screens approvals-list
Relates: nfrs NFR-04
Depends on AIFACTORY-US-163, feature:policy-engine-service
Sources model/services/control-plane/RunControlService.yaml, docs/planes/control/services/run-control-service/index.md#messages

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-877 RequestApproval with approver rule resolution, expiry and the WaitingForApproval transition implementation RunControl.RequestApproval, under the scope factory.approvals.request, opens the Pending ApprovalRequest bound to the proposal hash with the subject kind run and the run id as subject id for a lease holder (D129, D135), resolves the approver rule through PolicyEngineService, sets the expiry, copies onExpiry, reRequestsLeft and quorum from the ApprovalRule onto the request when it is made (fail, 0 and 1 while no approval rule is supplied, D127), moves a Running run to WaitingForApproval with ApprovalRequested and RunStateChanged in one transaction, accepts the request from a WaitingForApproval run when no request of the step is Pending (after Stale) and then writes ApprovalRequested without a run transition, schedules the approval-expiry reminder and returns approval-ref-v1; repeats for the same proposal hash return the existing reference and another proposal hash while a request of the step is Pending is refused. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-878 ApprovalExpiryReconciler moving overdue requests to Expired and failing or re-requesting by the onExpiry and reRequestsLeft fields of the request implementation The scheduled worker moves Pending requests past expiresAt to Expired and writes ApprovalDecided with state Expired, takes onExpiry (fail or re-request) and reRequestsLeft from the expired request, where they were copied from the ApprovalRule when the request was made, and either fails the run through ApprovalRejectedOrExpired with RunFailed carrying failure class policy-denied and RunStateChanged, when onExpiry is fail or reRequestsLeft is 0, or opens a new Pending request for the same proposal hash with the same onExpiry and reRequestsLeft counted down by one and leaves the run WaitingForApproval, when onExpiry is re-request and reRequestsLeft is above 0; a request made while no approval rule was supplied carries fail and fails the run the same way; it is idempotent on approvalId plus version. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-1492 Pending approval request Cancelled through RunFailed when its run fails implementation Every transition of a run to Failed moves the Pending ApprovalRequest of that run, when one exists, to Cancelled through the trigger RunFailed in the transaction of the run transition and writes ApprovalDecided with state Cancelled; the failure of a WaitingForApproval run of a retired environment (AIFACTORY-TASK-891) takes this path; a request that already left Pending is not touched, and a redelivery of the cause changes nothing. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-879 Verify request-an-approval through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-009\|Scenario=SCN-RunControlService-019\|Scenario=SCN-RunControlService-020)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-165, including the request accepted from WaitingForApproval after a Stale request, onExpiry and reRequestsLeft copied to the request when it is made (fail and 0 without a rule), the outcomes at expiry by those fields (fail with failure class policy-denied, re-request while reRequestsLeft is above 0 with the count going down and then fail, fail for a request made without a rule), the Pending request Cancelled through RunFailed when its run fails, and the publication and payload of ApprovalRequested and ApprovalDecided; consumer handling is proven by the consumers' own suites. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-166: Decide an approval on the exact proposal version with authority, expiry and conflict handling

behavior · persona reviewer As a reviewer, I want to see the approvals waiting for my roles and decide each on exactly the proposal I was shown, so that no stale, duplicate or unauthorized decision ever advances a run.

A caller whose scope carries factory.approvals.read lists the pending and decided approval requests visible to the caller's roles through RunControlService/ListApprovals and receives approval-list-v1 with action class, proposal hash, expiry, state and version; requests of other tenants never appear. A caller whose scope carries factory.approvals.decide, whose roles satisfy the approver rule of the request and who is not the requester of the run (D18 separation of duties) decides through RunControlService/DecideApproval with approval-decision-v1, quoting the request version and the proposal hash shown. The service records approve, reject or request changes on that exact version with decidedBy, decidedAt and the reason, and writes RunControlService/ApprovalDecided with the RunStateChanged of the run transition in the same transaction: an approval moves the request to Approved and the run from WaitingForApproval back to Running; a rejection moves the request to Rejected and terminates the run as Failed with failure class policy-denied; a request-changes decision moves the request to ChangesRequested (trigger RequestChanges) and returns the run from WaitingForApproval to Running (trigger ChangesRequested), and the reviewer's reason travels in ApprovalDecided as feedback for the lease holder, who regenerates the proposal and requests a new approval. An approve decision of an eligible approver on a request of kind approval is recorded as a vote in the table ApprovalVotes, one row per approver with the vote, the reason and the time (D127): the request stays Pending until its approve votes reach its quorum and is Approved by the vote that reaches it (trigger QuorumReached), one reject decision of an eligible approver ends it Rejected at once (trigger EligibleApproverRejected), and a second vote of the same approver on the same request is answered 409 and changes nothing. A vote that leaves the request Pending moves no run and publishes no ApprovalDecided. The quorum comes from the approval rule of the request, so a quorum above one arrives with the approval rules of ApprovalPolicyService in I4; until then every request carries the quorum one and the first eligible approve decision approves it, through the same vote path. A decision on a request whose subject is not a run moves no run and is announced to the service that owns the subject (AIFACTORY-US-1492). A spend approval, the run-level request of action class spend that is opened when the estimate of a run is above the spend threshold, is listed and decided through the same routes under the same authority rules; what its decision does to the run, back to Accepted on approval, Failed on rejection and a request-changes decision answered 409 (D119), is delivered by AIFACTORY-US-546. An operator decision, the request of kind operator-decision that RunControlService opens for the Operator of the project when a step fails under the failure policy Escalate or an Uncertain outcome is recorded on a step of class shared-write or above, is listed and decided through the same routes with one of the choices retry, compensate and fail; an approve, reject or request-changes decision on it, and a choice on a request of kind approval, are answered 409 and leave the request Pending, and what each choice does to the step and to the run is delivered by AIFACTORY-US-657 (D119). The finding that an operator decision on an uncertain tool effect carries, with the 400 for a decision without one and the 503 when ToolAdapterService does not answer the call that records it, is delivered by AIFACTORY-US-726 (D123). The separation of duties covers the requests of kind approval, a proposal and a spend, and not the operator decision: the member who requested a run may take the operator decision of that run, under the approver rule of the request like any other caller, and the act is audited (D121). A missing scope, an unsatisfied approver rule or the requester deciding a request of kind approval is refused with 403; an unknown request and a request of another tenant are both answered 404, so the response never confirms that a foreign request exists (ADR-0015); a version mismatch, a second decision or a Stale request is 409 and audited; a request past its expiry is 410. A request that was Cancelled with its run, through the trigger RunCancelled when the run was cancelled or through the trigger RunFailed when the run failed (AIFACTORY-US-165), is no longer Pending: a decision on it is refused with 409 like one on a decided request and never advances the run. A pending request becomes Stale when the proposal version it was bound to changes, announced by PipelineExecutionService/ChangeProposalStateChanged, BlueprintService/BlueprintValidated or GenerationRecipeService/RecipeApproved; after Stale the run stays WaitingForApproval until the lease holder requests a new approval for the current proposal hash through RunControlService/RunControl.RequestApproval, which is accepted from WaitingForApproval when no request of the step is Pending (AIFACTORY-US-165). AgentRuntimeService, WorkflowEngineService, AgentRegistryService and AuditService consume the decision in their own PBIs.

Acceptance criteria

AC1 Given a caller with factory.approvals.read and pending and decided requests across tenants, when RunControlService/ListApprovals is called, then the response is 200 with approval-list-v1 holding only the requests of the caller's tenant visible to the caller's roles, each with actionClass, proposalHash, expiresAt, state and version (SCN-RunControlService-006).
AC2 Given a reviewer whose roles satisfy the approverRule of a Pending RunControlService/ApprovalRequest and who did not request the run, when RunControlService/DecideApproval is called with approval-decision-v1 quoting the current version and the proposalHash, then the response is 200, the request is Approved, Rejected or ChangesRequested with decidedBy, decidedAt and the reason, and RunControlService/ApprovalDecided and RunControlService/RunStateChanged are written in the same transaction, moving the run to Running on approval, to Failed with failure class policy-denied on rejection, and back to Running through the trigger ChangesRequested on a request-changes decision, whose reason RunControlService/ApprovalDecided carries as feedback for the lease holder; the approve decision is recorded as a vote in ApprovalVotes, and given a request whose quorum is two, then the first eligible approve vote leaves it Pending without RunControlService/ApprovalDecided, the second vote of another eligible approver approves it through QuorumReached, one reject vote of an eligible approver rejects it through EligibleApproverRejected, and a second vote of the same approver answers 409 (SCN-RunControlService-005, FR-06, D127).
AC3 Given a caller without factory.approvals.decide, with roles that do not satisfy the rule, or who is the requester of the run and decides a request of kind approval, a proposal or a spend, when RunControlService/DecideApproval is called, then the response is 403 with a problem response, the request stays Pending and the run does not advance; the requester is refused for these requests only and may take the operator decision of the same run (SCN-RunControlService-005, ADR-0014, D121).
AC4 Given an approvalId that is unknown or exists only in another tenant, a quoted version that no longer matches because a second decision was submitted concurrently, a request already decided, Cancelled with its cancelled or failed run through the trigger RunCancelled or RunFailed, or made Stale by PipelineExecutionService/ChangeProposalStateChanged, BlueprintService/BlueprintValidated or GenerationRecipeService/RecipeApproved, a decision that does not fit the kind of the request, that is request-changes on a spend approval, a choice of retry, compensate or fail on a request of kind approval, or approve, reject or request-changes on an operator decision (D119), or a request past expiresAt, when RunControlService/DecideApproval is called, then the response is 404 for the unknown or foreign request, 409 for the conflict and for the decision that does not fit, which leaves the request Pending, and 410 for the expired request, the refused duplicate, stale or expired decision is audited and nothing advances (SCN-RunControlService-005, XL-05, ADR-0015).
AC5 Given the request left Pending through Approve, Reject, RequestChanges or ProposalChanged, when the outbox dispatcher publishes RunControlService/ApprovalDecided on run-control.approvals.v1.approval-decided, then the outbox record was written in the same transaction as the decision and the run transition it causes, the event is sent under the canonical envelope with approvalId, runId, tenantId, state, decidedBy, reason, projectId, proposalKind and proposalHash, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-020, ADR-0012).
Traces: services RunControlService, PolicyEngineService, PipelineExecutionService, BlueprintService, GenerationRecipeService
Traces: aggregates RunControlService/ApprovalRequest, RunControlService/Run
Traces: apis RunControlService/DecideApproval, RunControlService/ListApprovals
Traces: events RunControlService/ApprovalDecided, RunControlService/RunStateChanged, PipelineExecutionService/ChangeProposalStateChanged, BlueprintService/BlueprintValidated, GenerationRecipeService/RecipeApproved
Traces: descriptors approval-decision-v1, approval-list-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-06, FUX-05, XL-05
Traces: scenarios SCN-RunControlService-005, SCN-RunControlService-006, SCN-RunControlService-020
Relates: screens approvals-list, approval-card
Relates: nfrs NFR-04
Depends on AIFACTORY-US-165, feature:pipeline-execution-service-thin, feature:blueprint-service, feature:generation-recipe-service
Sources model/services/control-plane/RunControlService.yaml, docs/planes/control/services/run-control-service/index.md#security

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-880 DecideApproval with eligibility, separation of duties, version check and the 403, 404, 409 and 410 refusals implementation DecideApproval records one decision (approve, reject or request changes) on the exact proposal hash and request version, checks the scope, the approver rule through PolicyEngineService and the requester exclusion, which covers the requests of kind approval, a proposal and a spend, and not an operator decision (D121), moves the request to Approved, Rejected or ChangesRequested and the run to Running, to Failed with failure class policy-denied or back to Running through ChangesRequested, writes ApprovalDecided with the reason and RunStateChanged in one transaction, and refuses with 403, 404 (an unknown request or one of another tenant), 409 or 410 as modelled, auditing every refused duplicate; a decision that does not fit the kind of the request, request-changes on a spend approval, a choice on a request of kind approval or approve, reject or request-changes on an operator decision, is answered 409 and leaves the request Pending (D119), while the effect of a choice on an operator decision is delivered by AIFACTORY-TASK-2647; every approve and reject decision on a request of kind approval is written as a vote to the table ApprovalVotes, keyed by approval id and voter, the request is Approved when its approve votes reach its quorum and Rejected by one reject vote of an eligible approver, a vote that leaves it Pending moves no run and writes no ApprovalDecided, and a second vote of the same approver is answered 409 (D127); unit tests cover the quorum one and the quorum two. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-881 ListApprovals read model filtered by tenant and the caller's roles implementation ListApprovals returns approval-list-v1 with the pending and decided requests of the caller's tenant visible to the caller's roles, each with action class, proposal hash, expiry, state and version. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-882 MarkApprovalStale inbox handlers for changed proposal versions implementation The handlers of ChangeProposalStateChanged, BlueprintValidated and RecipeApproved move a pending request bound to the superseded version to Stale and write ApprovalDecided with state Stale, idempotently on the event id; the run stays WaitingForApproval until the lease holder requests a new approval for the current proposal hash, which RequestApproval accepts from WaitingForApproval (AIFACTORY-TASK-877). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-883 Verify decide-an-approval through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-005\|Scenario=SCN-RunControlService-006\|Scenario=SCN-RunControlService-020)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-166, including the request-changes decision, the votes of a request with the quorum one and, with a fake of the approval rule port that states a rule, the quorum two, the rejection by one vote and the second vote of the same approver answered 409, the 409 of a decision that does not fit the kind of the request, the Stale path fed by the three producer events and the publication and payload of ApprovalDecided; consumer handling is proven by the consumers' own suites. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-167: Settle a run as Completed or Failed from the report of the engine or a denied budget, leave a failed step to the failure policy the engine applies, and announce the terminal state

behavior A run reaches its terminal state only through the single transition authority of RunControlService. When WorkflowEngineService reports WorkflowEngineService/WorkflowRunCompleted, the inbox handler CompleteRun moves the run from Running to Completed with the output reference and writes RunControlService/RunCompleted in the same transaction, with the hash of the outcome in outcomeHash and that reference in the optional outputRef, the reference of the outcome of the run in ArtifactService through which a parent reads the result of its child run (D121). When the engine reports WorkflowEngineService/WorkflowRunFailed, CostUsageService denies the budget reservation through CostUsageService/BudgetReservationDenied, or an approval is rejected or expires, the run moves to Failed and RunControlService/RunFailed is written with a failure class (transient, permanent, policy-denied, validation, cancellation or uncertain) and the reason. A run failed by a rejected or expired approval carries the failure class policy-denied, and so does a run failed by a denied budget or by a retired environment. A denied budget fails a run that is still Accepted and a run that is Held when the denial arrives: the handler FailRunOnBudgetDenial moves it from Accepted or from Held to Failed through the trigger BudgetDenied with failure class policy-denied, and RunFailed and RunStateChanged are written in the transaction of the transition. A step fails terminally in two cases: a Failed attempt that is not retriable, because its failureClass is not transient or the sideEffectClass of the Step is neither read nor sandbox nor shared-write with idempotent true, and a TimedOut or retriable Failed attempt whose number reached the maxAttempts of its step; the Step is then settled Failed and StepSettled announces it with the failure class of its last attempt (AIFACTORY-US-658). RunControlService does not fail the run for it: its own transition of a run from Running to Failed through StepFailedTerminally is withdrawn (D121). WorkflowEngineService applies the failure policy of the step and reports WorkflowRunFailed, under Fail at once, under Compensate once the compensation finished and under Retry like under Fail, since a step settled Failed under Retry has its attempts spent (D123), and the handler FailRun then moves the run from Running to Failed through the trigger WorkflowRunFailed with the failure class the engine reported; until that report the run stays Running. The run failures that no failure policy of a step decides stay with RunControlService (D121): a denied budget, a rejected or expired approval, a retired environment and the operator decision. The stop on BudgetExceeded is no run failure: the run it names follows the cancellation path and ends Cancelled with the reason budget-exceeded (AIFACTORY-US-547, D123). Nor is the stop of a run whose pinned closure holds a revoked release: under the in-flight policy drain or revoke the run ends Cancelled with the reason release-revoked, a run that waits for an approval being cancelled at once under drain and a run already Cancelling keeping the reason it has, and only the parent of such a child treats it as failed with the failure class policy-denied (D123, D125, AIFACTORY-US-736 of feature agent-registry-service-complete). One failure policy keeps the step open (D119): when the failurePolicy the Step keeps from StepReady is Escalate, the terminal failure does not settle the step; an operator decision is opened for the Operator of the project and the run waits in WaitingForApproval, and it reaches Failed only when the Operator chooses fail or the request expires, through OperatorChoseFailOrDecisionExpired (AIFACTORY-US-657). A step that reaches its final state is announced by StepSettled in the transaction of that move (AIFACTORY-US-658). A rejected or expired spend approval fails a run that is Held as it fails a waiting one (AIFACTORY-US-546), and a parent run that ends Failed cancels its open child runs (AIFACTORY-US-656). A non-terminal run whose target environment is retired also ends Failed, with failure class policy-denied, which AIFACTORY-US-168 delivers. Every transition appends an entry to the append-only RecoveryJournal and is announced by RunStateChanged; publication is never proof of a transition. A completed or failed run refuses any further transition. The terminal events are published under the canonical envelope; CostUsageService, ArtifactService, GenerationRecipeService and the services of later increments consume them in their own PBIs. Duplicate engine reports are handled idempotently through the inbox and change nothing.

Acceptance criteria

AC1 Given a Running run whose workflow run the engine reports complete through WorkflowEngineService/WorkflowRunCompleted, when the CompleteRun handler runs, then the RunControlService/Run is Completed, a RecoveryJournal entry and RunControlService/RunStateChanged are written, and RunControlService/RunCompleted is written in the same transaction with runId, tenantId, outcomeHash and, when the report names an output reference, that reference in the optional outputRef (SCN-RunControlService-013, ADR-0014, D121).
AC2 Given the run was completed, when the outbox dispatcher publishes RunControlService/RunCompleted on run-control.runs.v1.run-completed, then the event is sent under the canonical envelope with runId, tenantId, outcomeHash and the optional outputRef, which is absent when the run completed without an output reference (D121), a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id, and a duplicate engine report changes nothing (SCN-RunControlService-013, ADR-0012).
AC3 Given a Running run whose engine reports WorkflowEngineService/WorkflowRunFailed, as it does after it applied the failure policy of a step that was settled Failed (D121), or an Accepted or Held run whose budget reservation CostUsageService denies through CostUsageService/BudgetReservationDenied, when the FailRun or FailRunOnBudgetDenial handler runs, then the RunControlService/Run is Failed, the reported run leaving Running through the trigger WorkflowRunFailed and the denied run leaving Accepted or Held through the trigger BudgetDenied, and RunControlService/RunFailed with failureClass (the class the engine reported, policy-denied for the denied budget) and reason plus RunControlService/RunStateChanged are written in the transaction of the transition (SCN-RunControlService-014).
AC4 Given the run failed, when the outbox dispatcher publishes RunControlService/RunFailed on run-control.runs.v1.run-failed, then the event is sent under the canonical envelope with runId, tenantId, failureClass and reason, the failureClass being policy-denied for a run failed by a denied budget, by a rejected or expired approval or by a retired environment, a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id, and a Completed or Failed run refuses every later transition (SCN-RunControlService-014, FR-10).
AC5 Given an attempt settled Failed that is not retriable, because its failureClass is not transient or the sideEffectClass of its step is neither read nor sandbox nor shared-write with idempotent true, or an attempt settled TimedOut or retriable Failed whose number equals the maxAttempts of its RunControlService/Step, when the LeaseExpiryReconciler or the RunDispatcher evaluates the step after RunControlService/AttemptOutcomeRecorded and the failurePolicy of the step is not Escalate, then the step is settled Failed, the RunControlService/Run stays Running and neither RunControlService/RunFailed nor RunControlService/RunStateChanged is written by this evaluation, the transition through StepFailedTerminally being withdrawn; when WorkflowEngineService/WorkflowRunFailed then arrives, the FailRun handler moves the run from Running to Failed through the trigger WorkflowRunFailed and writes RunControlService/RunFailed with the reported failureClass and reason plus RunControlService/RunStateChanged in one transaction; under the failure policy Escalate the step is not settled either and the run waits for the operator decision (SCN-RunControlService-014, SCN-RunControlService-017, D119, D121).
Traces: services RunControlService, WorkflowEngineService, CostUsageService
Traces: aggregates RunControlService/Run, RunControlService/Step, RunControlService/Attempt
Traces: events RunControlService/RunCompleted, RunControlService/RunFailed, RunControlService/RunStateChanged, RunControlService/AttemptOutcomeRecorded, WorkflowEngineService/WorkflowRunCompleted, WorkflowEngineService/WorkflowRunFailed, CostUsageService/BudgetReservationDenied
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-03, FR-10, FUX-07
Traces: scenarios SCN-RunControlService-013, SCN-RunControlService-014, SCN-RunControlService-017
Relates: screens run-detail
Relates: nfrs NFR-05, NFR-06
Depends on AIFACTORY-US-163, AIFACTORY-US-164, feature:workflow-engine-service, feature:cost-usage-service-complete
Sources model/services/control-plane/RunControlService.yaml, docs/planes/control/services/run-control-service/index.md#messages

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-884 CompleteRun, FailRun and FailRunOnBudgetDenial inbox handlers implementation The handlers of WorkflowRunCompleted, WorkflowRunFailed and BudgetReservationDenied move the run to Completed or Failed with the reported failure class, FailRun moving a Running run to Failed through the trigger WorkflowRunFailed, which after a failed step is the only path that fails the run (D121), and FailRunOnBudgetDenial moving an Accepted run and a Held run to Failed through BudgetDenied with failure class policy-denied, write RunCompleted, with the output reference of the report in the optional outputRef beside outcomeHash (D121), or RunFailed and RunStateChanged in one transaction, and are idempotent on the event id so a duplicate report changes nothing. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-888 Terminal step failure of a non-retriable or exhausted attempt that settles the step and leaves the run to the report of the engine implementation When the LeaseExpiryReconciler or the RunDispatcher finds a Failed attempt that is not retriable (a failureClass other than transient, or a Step whose sideEffectClass is neither read nor sandbox nor shared-write with idempotent true) or a TimedOut or retriable Failed attempt whose number equals the maxAttempts of its step, and the failurePolicy of the step is not Escalate, the step is settled Failed and the StepSettled record of AIFACTORY-TASK-2651 is written in the same transaction, and nothing else changes, so the run stays Running, no RunFailed and no RunStateChanged are written, and the state table of the run holds no transition StepFailedTerminally (D121); the run fails when WorkflowRunFailed of the engine arrives (AIFACTORY-TASK-884); under the failure policy Escalate the step is not settled and the evaluation hands the step to the operator decision of AIFACTORY-TASK-2646 (D119); the evaluation is idempotent on step id plus attempt number. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-885 Append-only RecoveryJournal migration migration The RecoveryJournal table is created with its sequence, fromState, toState, cause and occurredAt columns and the tenant filter; UPDATE and DELETE are denied to the service principal so the journal is append-only, and every run transition appends an entry in the same transaction. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-886 Verify settle-a-run through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-013\|Scenario=SCN-RunControlService-014\|Scenario=SCN-RunControlService-017)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-167, including the budget denial of an Accepted run and of a Held run, the terminal failure of a non-retriable or exhausted attempt that settles the step and leaves the run Running, the run failed only when WorkflowRunFailed is delivered after it, with the failure class of that report, the same failure under the failure policy Escalate that settles no step and fails no run, the failure class policy-denied on RunFailed for a denied budget and the publication and payload of RunCompleted, with and without outputRef, and of RunFailed; consumer handling is proven by the consumers' own suites. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-168: Hold the runs of a frozen environment, release them when it is active again and fail them when it is retired

behavior EnvironmentService announces every state change of an environment through EnvironmentService/EnvironmentStateChanged with environmentId, tenantId, fromState, toState and the class of the environment. The inbox handler GateRunsByEnvironment of RunControlService applies it to the runs of that tenant whose environmentId names the environment, idempotently on the event id, and keeps the last announced state of the environment and, beside it, the announced class in the table GateStates, which the dispatch gate and the submission gate read without a call to EnvironmentService; the class is what the production gate of RunControlService/SubmitRun compares with the state of the pinned release (AIFACTORY-US-005, D117). A run without an environmentId is never gated by an environment. When the environment becomes Frozen, a run that is still Accepted moves to Held through the trigger EnvironmentFrozen. A Running run is not interrupted: from the freeze on the RunDispatcher dispatches no new attempt for it, its open attempt settles first by the report of the lease holder or by lease reconciliation, and then the run moves to Held through EnvironmentFrozenAfterOpenAttemptSettled; a Running run without an open attempt is held at once. A run that is WaitingForApproval or Cancelling keeps its state, and when it is Running again while the environment is still Frozen it is held by the same rule. When the environment is Active again, a Held run that never had an attempt dispatched returns to Accepted (EnvironmentActiveBeforeFirstDispatch) and any other Held run returns to Running (EnvironmentActiveAgain), and the RunDispatcher, which EnvironmentService/EnvironmentStateChanged also triggers, evaluates the run again so the next Pending step is dispatched under AIFACTORY-US-163. A Held run can carry a run-level spend approval, because that approval is opened at once and can be decided while the run is Held (AIFACTORY-US-546, D119): when the environment is Active again such a run leaves Held for WaitingForApproval through EnvironmentActiveWithSpendApprovalPending while the request is still Pending, and for Accepted through EnvironmentActiveBeforeFirstDispatch when the request is already Approved, where its first attempt is dispatched once BudgetReserved has arrived; a request that ended Rejected or Expired has already failed the Held run. When the environment is Retired, every non-terminal run that targets it fails with failure class policy-denied. An Accepted run and a Held run move to Failed at once through EnvironmentRetired. A Running run is not interrupted: no new attempt is dispatched for it, and once its open attempt settled it moves to Failed through EnvironmentRetiredAfterOpenAttemptSettled; a Running run without an open attempt fails at once. A WaitingForApproval run moves to Failed through EnvironmentRetired once its open attempt settled, and its Pending approval request moves from Pending to Cancelled through the trigger RunFailed in the same transaction, with RunControlService/ApprovalDecided carrying state Cancelled (AIFACTORY-US-165). A Cancelling run has no transition for a retired environment in the state table and settles to Cancelled (AIFACTORY-US-162). With each of these failures RunControlService/RunFailed is written with failure class policy-denied and a reason that names the retired environment; from then on RunControlService/SubmitRun refuses a run that targets the environment with 409 (AIFACTORY-US-005). A Held run can be cancelled and moves to Cancelled at once (AIFACTORY-US-162), and a Held run whose budget reservation is denied meanwhile fails through BudgetDenied (AIFACTORY-US-167). Every transition appends a RecoveryJournal entry and writes RunControlService/RunStateChanged in the same transaction. A run without an environmentId, a run of another environment and a run in a terminal state are never touched.

Acceptance criteria

AC1 Given an Accepted run whose environmentId names an environment, when EnvironmentService/EnvironmentStateChanged announces that the environment moved to Frozen and the GateRunsByEnvironment handler runs, then the announced state and the announced class of the environment are kept in GateStates, the RunControlService/Run moves from Accepted to Held through EnvironmentFrozen, a RecoveryJournal entry and RunControlService/RunStateChanged with fromState, toState and cause are written in the same transaction, and no attempt is dispatched for the run while it is Held, also when its budget reservation is confirmed meanwhile (SCN-RunControlService-012).
AC2 Given a Running run of the frozen environment with an open attempt, when the freeze is applied, then the open attempt is not interrupted, the RunDispatcher dispatches no new RunControlService/Attempt for the run, and once the open attempt settles the run moves from Running to Held through EnvironmentFrozenAfterOpenAttemptSettled with RunControlService/RunStateChanged written in the same transaction; a Running run without an open attempt is held at once (SCN-RunControlService-012, ADR-0014).
AC3 Given Held runs of an environment, when EnvironmentService/EnvironmentStateChanged announces that it is Active again, then a run that never had an attempt dispatched returns to Accepted through EnvironmentActiveBeforeFirstDispatch, unless its run-level spend approval is still Pending, in which case it moves to WaitingForApproval through EnvironmentActiveWithSpendApprovalPending (D119), and every other run returns to Running through EnvironmentActiveAgain, each with RunControlService/RunStateChanged written in the same transaction, and the RunDispatcher, which the same EnvironmentService/EnvironmentStateChanged triggers, evaluates the released runs so the next Pending step is dispatched (SCN-RunControlService-012).
AC4 Given non-terminal runs that target an environment, when EnvironmentService/EnvironmentStateChanged announces that it is Retired, then every Accepted run and every Held run moves to Failed at once through EnvironmentRetired, a Running run gets no new RunControlService/Attempt and moves to Failed through EnvironmentRetiredAfterOpenAttemptSettled once its open attempt settled, or at once when it has none, a WaitingForApproval run moves to Failed through EnvironmentRetired once its open attempt settled and its Pending RunControlService/ApprovalRequest moves to Cancelled through the trigger RunFailed in the same transaction with RunControlService/ApprovalDecided carrying state Cancelled, RunControlService/RunFailed with failureClass policy-denied and a reason that names the retired environment plus RunControlService/RunStateChanged are written in the transaction of each transition, and a later RunControlService/SubmitRun that targets the environment is answered 409 with a problem body that names the gate (SCN-RunControlService-014).
AC5 Given a redelivery of the same event id, a run without an environmentId, a run of another environment or tenant, or a run already Completed, Failed or Cancelled, when the GateRunsByEnvironment handler runs on EnvironmentService/EnvironmentStateChanged, then nothing changes for that run and no second transition or event is written (ADR-0012).
Traces: services RunControlService, EnvironmentService
Traces: aggregates RunControlService/Run, RunControlService/Attempt, RunControlService/ApprovalRequest
Traces: apis RunControlService/SubmitRun
Traces: events EnvironmentService/EnvironmentStateChanged, RunControlService/RunStateChanged, RunControlService/RunFailed, RunControlService/ApprovalDecided
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-03, FR-10
Traces: scenarios SCN-RunControlService-012, SCN-RunControlService-014
Relates: screens run-detail
Relates: nfrs NFR-05
Depends on AIFACTORY-US-163, AIFACTORY-US-165, AIFACTORY-US-167, feature:environment-service
Sources model/services/control-plane/RunControlService.yaml, model/services/control-plane/EnvironmentService.yaml, docs/planes/control/services/run-control-service/index.md#messages

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-890 GateRunsByEnvironment inbox handler applying the frozen, active and retired states of an environment to the runs that target it implementation The handler of EnvironmentStateChanged keeps the last announced state of the environment and, beside it, the class the event announces in GateStates, where the production gate of SubmitRun reads it, moves the Accepted runs of a frozen environment to Held through EnvironmentFrozen, returns Held runs to Accepted or Running when the environment is active again, a Held run whose run-level spend approval is still Pending going to WaitingForApproval through EnvironmentActiveWithSpendApprovalPending instead (D119), fails the Accepted and the Held runs of a retired environment at once through EnvironmentRetired with RunFailed carrying failure class policy-denied and a reason that names the environment, leaves its Running and WaitingForApproval runs to the settlement path of AIFACTORY-TASK-891, writes RunStateChanged and the RecoveryJournal entry in the transaction of each transition, never touches a run without environmentId, and is idempotent on the event id. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-891 Environment gate in the RunDispatcher, in the attempt settlement path and in SubmitRun implementation The RunDispatcher reads the announced state of the target environment from GateStates, calls no other service for the gate, and dispatches no attempt for a Held run or for a Running run whose target environment is frozen or retired; when the open attempt of such a Running run settles, the run moves to Held through EnvironmentFrozenAfterOpenAttemptSettled, or to Failed through EnvironmentRetiredAfterOpenAttemptSettled with RunFailed carrying failure class policy-denied, in the transaction of the settlement, and a Running run without an open attempt takes the same transition at once; a WaitingForApproval run of a retired environment moves to Failed through EnvironmentRetired once its open attempt settled, and its Pending approval request is Cancelled through RunFailed in the same transaction (AIFACTORY-TASK-1492); the dispatcher is also triggered by EnvironmentStateChanged, so a released run is evaluated again and its next Pending step is dispatched; SubmitRun reads the same GateStates row for the 409 refusal of a retired target environment (AIFACTORY-TASK-194). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-892 Verify hold-runs-by-environment through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-RunControlService-012\|Scenario=SCN-RunControlService-014)"; a zero-match filter counts as a failure; the tests deliver EnvironmentStateChanged to the inbox and prove AC1 to AC5 of AIFACTORY-US-168, that is the hold of Accepted and Running runs, the release with the dispatcher triggered by the same event, the release of a Held run with a Pending spend approval to WaitingForApproval and of one with an Approved spend approval to Accepted, the failure with failure class policy-denied of the Accepted, Held, Running and WaitingForApproval runs of a retired environment, the Pending approval request of that WaitingForApproval run Cancelled through RunFailed, the gate state read from GateStates, the run without environmentId that is never gated and the idempotent redelivery; the publication of the event is proven by the suite of EnvironmentService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-546: Hold an accepted run above the spend threshold for a run-level spend approval and dispatch it only after the approval and the reservation

behavior · persona reviewer As a reviewer, I want a run whose estimated cost is above the spend threshold of its project to wait for my decision before anything is reserved or dispatched, so that no run commits more than that threshold without a person other than its requester agreeing to the exact amount.

When the estimate of an accepted run is above the spend threshold of its budget, CostUsageService reserves nothing and announces it through CostUsageService/SpendApprovalRequired with runId, tenantId, projectId, estimatedUsd and thresholdUsd (AIFACTORY-US-453 of feature cost-usage-service-complete). This PBI is the run control side of that decision of the owner (D117). A child run never comes here: it gets no reservation of its own, CostUsageService announces no SpendApprovalRequired for it, and no spend approval is opened for a child (AIFACTORY-US-656, D121). The inbox handler OpenSpendApproval opens a run-level ApprovalRequest for the run: it carries the action class spend and the estimate of the event in estimatedUsd, its proposal is of kind spend and its proposalHash binds the decision to exactly that estimate, and it names the run and no step, no lease and no fencing token, because an Accepted run has none. It is opened by the handler and not by a lease holder, so RunControl.RequestApproval takes no part in it. The approver rule is resolved through PolicyEngineService for the policy revision the run pinned and the expiry is set from RunControl:Approvals:DefaultExpiryHours unless policy shortens it, as for every approval request (AIFACTORY-US-165). In the same transaction the run moves from Accepted to WaitingForApproval through the trigger SpendApprovalRequired, an entry is appended to the RecoveryJournal and the ApprovalRequested and RunStateChanged outbox records are written; ApprovalRequested carries no stepId for this request. The handler is idempotent on the event id, and a run has at most one spend approval, so a second announcement for a run that already has one opens nothing. An unavailable PolicyEngineService fails the handler, nothing is written and the inbox retries the event. When the announcement arrives for a run that is Held by a frozen environment, the approval is opened at once in the same way and the run stays Held (D119): the Pending request and ApprovalRequested are written, and no run transition and no RunStateChanged are. The request can be decided while the run is Held. An approval leaves the run Held and writes ApprovalDecided, on which CostUsageService reserves the approved estimate at once, whatever the state of the run; the run proceeds when the approval is granted and the environment is active, so on the release it returns to Accepted, where its first attempt is dispatched once BudgetReserved has arrived, while a release that finds the request still Pending moves the run to WaitingForApproval through EnvironmentActiveWithSpendApprovalPending (AIFACTORY-US-168). A rejection or an expiry fails the Held run from Held through ApprovalRejectedOrExpired, exactly as it fails a waiting one. The request appears in RunControlService/ListApprovals like any other and is decided through RunControlService/DecideApproval under the rules of AIFACTORY-US-166: the scope factory.approvals.decide, the approver rule, the quoted version and proposal hash, and the separation of duties, under which the requester of the run is never eligible to decide and is refused with 403 (D18). An approval moves the request to Approved and returns the run from WaitingForApproval to Accepted through the trigger SpendApprovalGranted; ApprovalDecided is written with the proposal kind spend and the approved estimatedUsd, together with RunStateChanged, in the transaction of the decision. Nothing is dispatched on the approval alone: CostUsageService consumes ApprovalDecided, reserves the approved estimate and publishes CostUsageService/BudgetReserved, on which the RunDispatcher dispatches the first attempt as for any Accepted run (AIFACTORY-US-163); when the budget no longer covers the estimate it publishes BudgetReservationDenied instead and the run fails through BudgetDenied (AIFACTORY-US-167). A rejection moves the request to Rejected, and the ApprovalExpiryReconciler moves a request past its expiry to Expired; either fails the run from WaitingForApproval through ApprovalRejectedOrExpired with the failure class policy-denied, with RunFailed, RunStateChanged and ApprovalDecided written in one transaction. An expired spend approval is never requested again, whatever the on-expiry rule of an approval policy says for the approvals of steps. A request-changes decision does not apply to a spend approval, since no lease holder exists who could change the proposal: DecideApproval answers such a decision with 409 (D119), the request stays Pending and the run keeps its state, WaitingForApproval or Held. A run that waits for its spend approval can be cancelled (AIFACTORY-US-162), and its Pending request is then Cancelled through the trigger RunCancelled; the request of a run that fails for another reason is Cancelled through the trigger RunFailed (AIFACTORY-US-165). CostUsageService reads those outcomes from ApprovalDecided and has nothing to release, because nothing was reserved. The events are published under the canonical envelope; StudioBff, StudioNotificationService, AuditService and CostUsageService consume them in their own PBIs.

Acceptance criteria

AC1 Given an Accepted run and CostUsageService/SpendApprovalRequired for it with estimatedUsd and thresholdUsd, when the OpenSpendApproval inbox handler runs, then a Pending RunControlService/ApprovalRequest exists for the run with the action class spend, the estimatedUsd of the event, an approver rule and an expiry and with no step, lease or fencing token, the RunControlService/Run is WaitingForApproval through the trigger SpendApprovalRequired, RunControlService/ApprovalRequested without a stepId and RunControlService/RunStateChanged are written in the same transaction, no attempt is dispatched, and a redelivery of the event or a second announcement for the same run opens no second request; given instead a run that is Held by a frozen environment, then the same Pending request and RunControlService/ApprovalRequested are written at once and the run stays Held, without a transition and without RunControlService/RunStateChanged (SCN-RunControlService-019, D119).
AC2 Given that Pending spend approval and a reviewer whose roles satisfy its approver rule and who did not request the run, when RunControlService/DecideApproval approves it on the quoted version and proposal hash, then the request is Approved, the run returns from WaitingForApproval to Accepted through the trigger SpendApprovalGranted, RunControlService/ApprovalDecided with the proposal kind spend and the approved estimatedUsd and RunControlService/RunStateChanged are written in the same transaction, and the first attempt of the run is dispatched only after CostUsageService/BudgetReserved arrived for it; given the run is Held when the approval is decided, then the request is Approved and RunControlService/ApprovalDecided is written the same way, the run stays Held and nothing is dispatched, and its first attempt is dispatched only once the environment is active again and CostUsageService/BudgetReserved has arrived (SCN-RunControlService-005, FR-06, D119).
AC3 Given the Pending spend approval, when RunControlService/DecideApproval rejects it, or its expiresAt passes and the ApprovalExpiryReconciler runs, then the request is Rejected or Expired, the run moves from WaitingForApproval, or from Held when it was Held at that moment (D119), to Failed through ApprovalRejectedOrExpired, RunControlService/RunFailed with the failure class policy-denied, RunControlService/RunStateChanged and RunControlService/ApprovalDecided are written in one transaction, no new request is opened for the expired one, and no attempt of the run was ever dispatched (SCN-RunControlService-020).
AC4 Given the Pending spend approval, when the requester of the run calls RunControlService/DecideApproval, then the response is 403 with a problem response; when an eligible reviewer submits a request-changes decision, then the response is 409 with a problem response (D119); in both cases the request stays Pending, the run keeps its state, WaitingForApproval or Held, and nothing is published (SCN-RunControlService-005).
AC5 Given the spend approval was opened or decided, when the outbox dispatcher publishes RunControlService/ApprovalRequested on run-control.approvals.v1.approval-requested or RunControlService/ApprovalDecided on run-control.approvals.v1.approval-decided, then the event is sent under the canonical envelope, the first with approvalId, runId, tenantId, the actionClass spend, proposalHash, approverRule and expiresAt and without a stepId, the second with approvalId, runId, tenantId, state, projectId, the proposalKind spend, proposalHash and estimatedUsd, and a transport outage is retried with backoff so that neither is lost or duplicated under its event id (SCN-RunControlService-019, SCN-RunControlService-020, ADR-0012).
Traces: services RunControlService, CostUsageService, PolicyEngineService
Traces: aggregates RunControlService/Run, RunControlService/ApprovalRequest
Traces: apis RunControlService/DecideApproval, RunControlService/ListApprovals
Traces: events CostUsageService/SpendApprovalRequired, CostUsageService/BudgetReserved, RunControlService/ApprovalRequested, RunControlService/ApprovalDecided, RunControlService/RunStateChanged, RunControlService/RunFailed
Traces: descriptors approval-request-v1, approval-decision-v1
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-06, FUX-05
Traces: scenarios SCN-RunControlService-005, SCN-RunControlService-019, SCN-RunControlService-020
Relates: screens approvals-list, approval-card
Relates: nfrs NFR-04
Depends on AIFACTORY-US-163, AIFACTORY-US-165, AIFACTORY-US-166, AIFACTORY-US-167, feature:cost-usage-service-complete, feature:policy-engine-service
Sources model/services/control-plane/RunControlService.yaml, model/services/control-plane/CostUsageService.yaml, docs/reference/schemas/approval.schema.json

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-2191 Contract of the run-level spend approval in the approval request, the approval events and the approval list contract The contract documents of the service state, against the ApprovalRequest definition of docs/reference/schemas/approval.schema.json, that a spend approval carries the action class spend and a proposal of kind spend with estimatedUsd and carries no stepId, leaseId or fencingToken; that ApprovalRequested omits stepId for it and ApprovalDecided carries the proposal kind spend and estimatedUsd; and that a request-changes decision on it is answered 409 (D119). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2192 OpenSpendApproval inbox handler opening the run-level request and moving the run to WaitingForApproval implementation The handler of SpendApprovalRequired opens one Pending ApprovalRequest per run with the action class spend, the estimatedUsd of the event, the proposal hash of the spend proposal, the approver rule resolved through PolicyEngineService for the pinned policy revision and the expiry from RunControl:Approvals:DefaultExpiryHours unless policy shortens it, with no step, lease or fencing token; it moves the Accepted run to WaitingForApproval through SpendApprovalRequired and writes the RecoveryJournal entry, ApprovalRequested without stepId and RunStateChanged in one transaction; for a run that is Held it opens the same request at once, writes ApprovalRequested and leaves the run Held without a transition (D119); it is idempotent on the event id, opens nothing for a run that already has a spend approval, and fails without a write when PolicyEngineService is unavailable, so the inbox retries. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2193 Outcomes of a spend approval in DecideApproval and in the ApprovalExpiryReconciler implementation For a request of action class spend DecideApproval applies SpendApprovalGranted on approval, returning the run from WaitingForApproval to Accepted and writing ApprovalDecided with the proposal kind spend and estimatedUsd together with RunStateChanged, applies ApprovalRejectedOrExpired on rejection with RunFailed carrying the failure class policy-denied, answers a request-changes decision with 409 and leaves the request Pending (D119), and refuses the requester of the run with 403; when the run is Held at the decision, an approval writes ApprovalDecided and leaves the run Held, and a rejection fails it from Held through ApprovalRejectedOrExpired (D119); the ApprovalExpiryReconciler fails the run of an expired spend approval the same way, from WaitingForApproval or from Held, and never re-requests it; the RunDispatcher dispatches nothing for the run before BudgetReserved arrives. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2194 Acceptance fixture publishing SpendApprovalRequired and BudgetReserved for a run fixture The acceptance suite publishes SpendApprovalRequired with a stated estimate and threshold for an accepted run and, after an approval, BudgetReserved or BudgetReservationDenied for it, through a fixture publisher, so that the waiting, the approval, the rejection, the expiry with a controlled clock and the dispatch after the reservation are reproducible without CostUsageService. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2195 Verify approve-run-spend through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Area=SpendApproval&(Scenario=SCN-RunControlService-005\|Scenario=SCN-RunControlService-019\|Scenario=SCN-RunControlService-020)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-546 with the fixture of AIFACTORY-TASK-2194, including the request opened once for a redelivered and for a second announcement, the run that is not dispatched between the approval and BudgetReserved, the rejection, the expiry without a re-request, the refused requester, the request-changes decision answered 409, the request opened at once for a Held run that stays Held, its approval and its rejection while Held, and the payload of ApprovalRequested without stepId and of ApprovalDecided with estimatedUsd; the reservation on the approval is proven by the suite of CostUsageService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-547: Cancel every run whose observed usage takes its budget over the limit or further above it, with the reason budget-exceeded

behavior A run reserves before it spends, and CostUsageService also watches what it really spends: when recorded or settled usage takes committedUsd of a budget above limitUsd, it publishes CostUsageService/BudgetExceeded with budgetId, tenantId, projectId, committedUsd, limitUsd and, when the usage names one, the runId of the run whose usage crossed the limit (AIFACTORY-US-549 of feature cost-usage-service-complete). The owner decided that such an overrun stops the running run and is not only recorded (D117); this PBI is the run control side. The inbox handler CancelRunOnBudgetExceeded records a CancellationIntent for the run the event names, with the reason budget-exceeded and the actor system, and from there the run follows the cancellation path of RunControlService/CancelRun without any other difference (AIFACTORY-US-162). Only the run an event names is stopped (D119): the other running runs of the same budget are not cancelled by it, those that stay within their estimate keep their attempts and finish, and CostUsageService denies new reservations of the budget. The overrun does not end with the first run (D121): every run whose observed usage raises the committed amount of a budget that is already above its limit is named by a BudgetExceeded of its own and stopped the same way, the run that crossed the limit and every later one, each once. The run an event names may be a child run, whose usage counts on the reservation of its parent run: only the child is stopped, no intent is recorded for its parent, and the parent then sees a failed child, which AgentRuntimeService and WorkflowEngineService handle in their own PBIs (AIFACTORY-US-656, D121). CostUsageService publishes the event once per run, so later usage of a run that is already being cancelled announces nothing further. A run that is still Accepted or is Held moves to Cancelled at once. A Running or WaitingForApproval run moves to Cancelling, its RunStateChanged outbox record is written in the transaction of the transition, and after the commit AgentRuntimeService/Runtime.Interrupt is called for the open attempt, fail-fast, so that the attempt stops at its next safe point and reports its observed termination. A Pending approval request of the run is Cancelled through the trigger RunCancelled. When every attempt has settled the run moves to Cancelled and RunControlService/RunCancelled is published with the reason budget-exceeded and the requestedBy system from the intent. The run ends Cancelled and never Failed (D117, D123): a stop on BudgetExceeded is no run failure, no RunFailed is written for it, and only the parent of a stopped child run treats that child as failed, its delegation being set to failed and its step run of kind SubWorkflow to Failed with the failure class policy-denied and the reason budget-exceeded, because the model has no failure class of its own for a budget, and the failure policy of that step applying, in the PBIs of AgentRuntimeService and WorkflowEngineService. Completed external effects stay visible and are never rolled back silently, and the usage the run already caused stays recorded in CostUsageService, which settles the reservation on RunCancelled in its own PBI. The handler is idempotent on the event id. An event that names no run changes no run: it is the event CostUsageService publishes for a project when usage recorded outside a run, such as a model call outside a run, took the committed amount above the limit, and the handler ignores it, stops no run and records no cancellation intent; it is consumed and acknowledged, since the denial of the next reservation is the work of CostUsageService and the refusal of later calls outside a run is the work of ModelRouterService (D125). An event for a run that is already Completed, Failed or Cancelled changes nothing, which is the ordinary case of an overrun found at settlement, and an event for a run that is already Cancelling records no second intent. The run is looked up under the tenant of the event, so a run of another tenant is never touched. Nothing in this PBI denies a later run: a new run of the project is accepted as before, and CostUsageService denies its reservation while the budget is over its limit, which fails that run as policy-denied through BudgetDenied (AIFACTORY-US-167).

Acceptance criteria

AC1 Given a Running run with an open attempt and CostUsageService/BudgetExceeded naming its runId, when the CancelRunOnBudgetExceeded inbox handler runs, then a CancellationIntent with the reason budget-exceeded and the actor system is recorded, the RunControlService/Run moves to Cancelling with RunControlService/RunStateChanged written in the same transaction, and after that transaction commits AgentRuntimeService/Runtime.Interrupt is called for the open attempt; an unavailable AgentRuntimeService fails the call fast and leaves the Cancelling state standing (SCN-RunControlService-015).
AC2 Given the event names a run that is still Accepted, a Held run, or a run that is WaitingForApproval, when the handler runs, then the Accepted run and the Held run move to Cancelled at once, the waiting run moves to Cancelling, and its Pending RunControlService/ApprovalRequest moves to Cancelled through the trigger RunCancelled no later than the run reaching Cancelled (SCN-RunControlService-015).
AC3 Given the run of AC1 whose last open attempt settles, or a run cancelled at once, when the run moves to Cancelled and the outbox dispatcher publishes RunControlService/RunCancelled on run-control.runs.v1.run-cancelled, then the outbox record was written in the transaction of the transition, the event is sent under the canonical envelope with runId, tenantId, the reason budget-exceeded and the requestedBy system, the run is Cancelled and never Failed, so that no RunFailed record is written for it (D123), and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-015, ADR-0012).
AC4 Given a redelivery of the same event id, an event without a runId, an event for a run that is already Completed, Failed or Cancelled, or an event for a run that is already Cancelling, when the CancelRunOnBudgetExceeded handler runs on CostUsageService/BudgetExceeded, then no run changes state, no second CancellationIntent is recorded and nothing is published; and given a second Running run of the same project and budget that the event does not name, when the handler cancels the named run, then that second run keeps its state and its open attempt and finishes (ADR-0012, FR-10, D119).
AC5 Given two Running runs of the same budget and a first CostUsageService/BudgetExceeded that named and stopped one of them, when a later CostUsageService/BudgetExceeded names the second run, because its observed usage raised the committed amount of the budget further above its limit, then the handler records a CancellationIntent with the reason budget-exceeded and the actor system for that run as well and it follows the same cancel transitions, each named run being stopped once, while a third run of the budget that stays within its estimate is named by no event, keeps its state and finishes; and given an event that names a child run, then only that child is stopped, RunControlService/RunCancelled of the child carries the reason budget-exceeded, and no intent is recorded for its parent run (SCN-RunControlService-015, D121).
Traces: services RunControlService, CostUsageService, AgentRuntimeService
Traces: aggregates RunControlService/Run, RunControlService/ApprovalRequest
Traces: apis AgentRuntimeService/Runtime.Interrupt
Traces: events CostUsageService/BudgetExceeded, RunControlService/RunStateChanged, RunControlService/RunCancelled
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-10, FUX-07
Traces: scenarios SCN-RunControlService-015
Relates: screens run-detail
Relates: nfrs NFR-05
Depends on AIFACTORY-US-162, feature:cost-usage-service-complete, feature:agent-runtime-service
Sources model/services/control-plane/RunControlService.yaml, model/services/control-plane/CostUsageService.yaml, docs/planes/control/services/run-control-service/index.md#failure-and-recovery

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-2196 CancelRunOnBudgetExceeded inbox handler recording the cancellation intent of the run a budget overrun names implementation The handler of BudgetExceeded looks the run of the event up under the tenant of the event, records a CancellationIntent with the reason budget-exceeded and the actor system and applies the cancel transitions through the same path as CancelRun (AIFACTORY-TASK-865 and AIFACTORY-TASK-866), so that RunCancelled carries that reason and actor; it is idempotent on the event id, changes nothing for an event without runId, for a terminal run and for a run that is already Cancelling, and acknowledges the event in each of these cases; it touches no run other than the one the event names, so the other running runs of the budget finish (D119); every event is handled alike, so a second run that a later event names is stopped the same way and each named run is stopped once, and a named child run is stopped without an intent for its parent run (D121). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2197 Verify cancel-run-on-budget-exceeded through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Area=BudgetExceeded&Scenario=SCN-RunControlService-015"; a zero-match filter counts as a failure; the tests deliver BudgetExceeded to the inbox through a fixture publisher and prove AC1 to AC5 of AIFACTORY-US-547 with the fake Runtime.Interrupt endpoint that records the call, including the intent with the reason budget-exceeded and the actor system, the runs cancelled at once, the Pending approval request Cancelled, the payload of RunCancelled, the event without runId, the terminal run, the run already Cancelling, the redelivery, a second running run of the same budget that is left running while no event names it and is stopped once a later event names it, and a named child run stopped without an intent for its parent; the publication of BudgetExceeded is proven by the suite of CostUsageService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-548: Announce once that no non-terminal run of a closing project remains, so that its off-boarding can complete

behavior Closing a project is a procedure, and ProjectService keeps a project in Closing until two confirmations arrived: RunControlService confirms that no non-terminal run of the project remains, and ArtifactService confirms the export or the deletion of its data (D117; AIFACTORY-US-449 of feature project-service). This PBI delivers the first confirmation, the event RunControlService/ProjectRunsSettled. When ProjectService announces through ProjectService/ProjectStateChanged that a project moved to Closing, the inbox handler GateRunsByProject keeps the announced state in the table GateStates, from which SubmitRun refuses every new run of the project with 409 (AIFACTORY-US-005), and records a cancellation intent for every open run of the project (AIFACTORY-US-162). From then on no run of the project can appear and every run that exists moves towards a terminal state. ProjectRunsSettled is written at the moment the service can state that no run of the project is in a state other than Completed, Failed or Cancelled. That is the transaction of the handler when no such run is left once the handler has applied the intents, which includes a project that never had a run and a project whose open runs were all Accepted or Held and are cancelled at once, and otherwise the transaction in which the last open run of the project reaches its terminal state, which for a run that was Running or WaitingForApproval is Cancelled once its attempts settled. The check and the outbox record belong to that transaction, so the event is never written while a run of the project is still Accepted, Held, Running, WaitingForApproval or Cancelling, and it is not lost when the service stops between the last transition and the publication. The event is published once for the closing of a project: a redelivery of the announcement is absorbed by the inbox, and a terminal transition that is not the last one of the project writes nothing. It is published on run-control.runs.v1.project-runs-settled under the canonical envelope with projectId and tenantId; ProjectService records it in its off-boarding journal through its handler RecordRunsSettled in its own PBI. Nothing is announced for a project that is Suspended, whose open runs are cancelled the same way but which is not being off-boarded, nor for an Active project, and the announcement of Closed, which follows the confirmations, causes no further event. A run of another tenant or of another project never counts, because the check is made under the tenant of the event and for the projectId it names.

Acceptance criteria

AC1 Given a project without a non-terminal run, when ProjectService/ProjectStateChanged announces that it moved to Closing and the GateRunsByProject inbox handler runs, then the announced state is kept in GateStates and the RunControlService/ProjectRunsSettled outbox record with projectId and tenantId is written in the transaction of the handler (SCN-RunControlService-021).
AC2 Given a project with an Accepted run, a Running run with an open attempt and a run that is WaitingForApproval, when ProjectService/ProjectStateChanged announces Closing, then each of them records a cancellation intent and no RunControlService/ProjectRunsSettled record is written while one of them is not yet Completed, Failed or Cancelled; when the last of them reaches Cancelled after its attempts settled, then the record is written in the transaction of that transition, beside the RunControlService/RunCancelled record of that run; and given a project whose only open runs are Accepted or Held, then they are cancelled at once and the record is written in the transaction of the handler (SCN-RunControlService-021).
AC3 Given the record was written, when the outbox dispatcher publishes RunControlService/ProjectRunsSettled on run-control.runs.v1.project-runs-settled, then the event is sent under the canonical envelope with projectId and tenantId, a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id, and a redelivery of the Closing announcement publishes no second event for the project (SCN-RunControlService-021, ADR-0012).
AC4 Given a project that ProjectService/ProjectStateChanged announced as Suspended, an Active project, or a second project of the tenant that is not closing, when the open runs of that project are cancelled or its last run reaches a terminal state, then no RunControlService/ProjectRunsSettled is written for it; and a run of another tenant or of another project neither delays nor triggers the event of the closing project (SCN-RunControlService-021, XL-13).
Traces: services RunControlService, ProjectService
Traces: aggregates RunControlService/Run
Traces: events RunControlService/ProjectRunsSettled, RunControlService/RunCancelled, ProjectService/ProjectStateChanged
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-10, XL-13
Traces: scenarios SCN-RunControlService-021
Relates: nfrs NFR-05
Depends on AIFACTORY-US-162, AIFACTORY-US-167, feature:project-service
Sources model/services/control-plane/RunControlService.yaml, model/services/control-plane/ProjectService.yaml, model/scenarios/RunControlService.yaml

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-2198 ProjectRunsSettled written by GateRunsByProject and by the last terminal transition of a run of a closing project implementation GateRunsByProject writes the ProjectRunsSettled outbox record in its transaction when the project it keeps as Closing in GateStates has, after the cancellation intents were applied, no run outside Completed, Failed and Cancelled; every transition of a run to a terminal state checks, in its own transaction, whether the project of the run is kept as Closing and whether another non-terminal run of it remains, and writes the record when none does; the record is written once per closing of a project, carries projectId and tenantId, and is never written for a project kept as Suspended or Active. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2199 Verify announce-settled-runs-of-a-closing-project through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-RunControlService-021"; a zero-match filter counts as a failure; the tests deliver ProjectStateChanged to the inbox and prove AC1 to AC4 of AIFACTORY-US-548, that is the event of a project without runs, the event withheld until the last of three open runs is terminal, the event written by the handler for a project whose open runs are all cancelled at once, the single event after a redelivered announcement, no event for a Suspended project, and the publication and payload of ProjectRunsSettled; consumer handling is proven by the suite of ProjectService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-656: Submit a child run under its parent run, on the reservation of that parent, and cancel the open child runs of a parent that ends Failed or Cancelled

behavior A run can start another run: a supervisor attempt delegates a part of its work, and a workflow step of kind SubWorkflow runs a whole workflow as one step. The owner decided that both go through one internal operation of RunControlService, so that a child run is a run like any other and its state has the same single authority (D119). RunControlService/RunControl.SubmitChildRun is an internal gRPC method, called under a workload identity by AgentRuntimeService for a delegation and by WorkflowEngineService for a step of kind SubWorkflow; it has no gateway route. The request is run-submission-v1, the descriptor of the public submission, with the parent fields that it carries for a child submission only: parentRunId always, and parentAttemptId for a delegation, where it names the attempt of the parent that delegates. The child of a step of kind SubWorkflow has the run as parent and no parent attempt, so its submission carries no parentAttemptId (D121). The reply is run-accepted-v1. The child is written as a Run in state Accepted with parentRunId and, for a delegation, parentAttemptId, and inherits from its parent the tenant, the project, the policy revision and the pinned release closure: none of them is read from the body and nothing is resolved or evaluated anew for them, so a child always runs under what its parent was accepted with. As for every run, the resolved execution snapshot of the child is stored in ArtifactService before the child is accepted, and no child is accepted without a stored snapshot (AIFACTORY-US-005, D121). The RunAccepted outbox record is written in the same transaction and carries the parentRunId of the child, the optional field by which CostUsageService recognises a child run. A child run gets no reservation of its own (D121): CostUsageService reserves nothing for it, its usage is recorded with the child run id and counts on the reservation of the parent run, and RunControlService therefore waits for no BudgetReserved and opens no spend approval for a child, whose first attempt is dispatched when StepReady announces its first step (AIFACTORY-US-163, D123). In everything else the child is dispatched, held, approved and settled exactly as the other PBIs of this feature state for any run. When the usage of a child takes the budget over its limit, BudgetExceeded names the child, only the child is stopped, and its parent then sees a failed child (AIFACTORY-US-547). The release ids of the snapshot of the child are written to RunPinnedReleases under the child run id when the child is accepted, so that a revocation selects the child in its own right (D125); a child stopped by the in-flight policy drain or revoke ends Cancelled with the reason release-revoked, and its parent treats it as failed like a child stopped for a budget overrun, which AgentRuntimeService and WorkflowEngineService apply in their own PBIs (D125). The method is idempotent on the idempotency key of the submission, which is the step run id for a step of kind SubWorkflow and the delegation id for a delegation (D121): the same key with the same input returns the same child run and writes nothing, and the same key with another input is refused as a conflict. The outcome of a child reaches its caller through the terminal run events that exist, RunCompleted, RunFailed and RunCancelled, which carry the runId of the child, RunCompleted with the optional outputRef through which the parent reads the result of the child (AIFACTORY-US-167, D121); no further event announces it, and AgentRuntimeService and WorkflowEngineService match that id to the delegation or to the step run in their own PBIs. A parent never leaves a child behind: when a parent run ends Failed or Cancelled, a CancellationIntent with a reason that names the parent run and with the actor system is recorded for each of its open child runs, once per child, and each child follows the cancellation path of CancelRun (AIFACTORY-US-162). An Accepted or a Held child is Cancelled at once, a Running or a WaitingForApproval child moves to Cancelling and its open attempt is interrupted, and a child that has children of its own passes the cancellation on in the same way when it ends Cancelled. A parent that ends Completed cancels nothing. A caller without a workload identity is refused with Unauthenticated, an invalid request, one without parentRunId among them, with InvalidArgument, and a parent run that does not exist in the tenant of the caller with NOT_FOUND, which is also the answer for a parent of another tenant (ADR-0015). A submission under a parent that is already Completed, Failed, Cancelled or Cancelling is refused with FailedPrecondition, the gRPC status FAILED_PRECONDITION, and no child run exists (D121), because such a child could no longer be cancelled with its parent; as a design choice of this PBI a submission that names a parentAttemptId which is not an attempt of that parent is refused the same way. No refusal writes a run or publishes anything. A parent is named only here: the public SubmitRun answers 400 for a body that carries parentRunId or parentAttemptId (AIFACTORY-US-005, D121).

Acceptance criteria

AC1 Given a Running parent run with an open attempt and a caller with the workload identity of AgentRuntimeService, when RunControlService/RunControl.SubmitChildRun is called with run-submission-v1 naming parentRunId and parentAttemptId, then the reply is run-accepted-v1, a RunControlService/Run exists in state Accepted with parentRunId and parentAttemptId and with the tenant, the project, the policy revision and the pinned release closure of the parent, whatever the body states for them, its resolved execution snapshot was stored before it was committed, and the RunControlService/RunAccepted outbox record, which carries the parentRunId of the child, is written in the same transaction; a call by WorkflowEngineService for a step of kind SubWorkflow that names the parent run and no parent attempt is accepted the same way; no reservation is awaited for the child, so its first attempt is dispatched when StepReady announces its first step, without BudgetReserved (D123), and no spend approval is opened for it (SCN-RunControlService-022, ADR-0014, D121).
AC2 Given a child submission repeated with the same idempotency key, which is the step run id for a step of kind SubWorkflow and the delegation id for a delegation (D121), and the same input, when it arrives at RunControlService/RunControl.SubmitChildRun, then the same child run is returned and no second RunControlService/Run and no second RunControlService/RunAccepted record exist; given the same key with another input, then the call is refused as a conflict and nothing is written (SCN-RunControlService-022).
AC3 Given a caller without a workload identity, a request without parentRunId, a parent run that is unknown or belongs to another tenant, a parent run that is already Completed, Failed, Cancelled or Cancelling, or a parentAttemptId that is not an attempt of the parent, when RunControlService/RunControl.SubmitChildRun is called, then the call is refused with Unauthenticated, InvalidArgument, NOT_FOUND for the unknown and the foreign parent alike, or the status FAILED_PRECONDITION for the last two, the answer for a terminal or Cancelling parent being decided (D121), no run is written and nothing is published (SCN-RunControlService-022, ADR-0015).
AC4 Given a parent run with an Accepted child, a Running child with an open attempt and a Completed child, when the parent moves to Failed or to Cancelled, then a CancellationIntent with a reason that names the parent run and with the actor system is recorded once for each of the two open children, the Accepted child is Cancelled at once, the Running child moves to Cancelling with RunControlService/RunStateChanged and its open attempt is interrupted through AgentRuntimeService/Runtime.Interrupt after the commit, the Completed child is not touched, and a repeated evaluation records no second intent; given a parent that ends Completed, then no intent is recorded for any child (ADR-0014).
AC5 Given a child run that reaches a terminal state, when the outbox dispatcher publishes RunControlService/RunCompleted, RunControlService/RunFailed or RunControlService/RunCancelled for it, then the event is sent under the canonical envelope with the runId of the child, the RunControlService/RunCompleted of a child carrying the optional outputRef through which the parent reads its result (D121), the RunControlService/RunCancelled of a child cancelled with its parent carrying the reason that names the parent run and the requestedBy system, no further event announces the outcome of a child, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (ADR-0012).
Traces: services RunControlService, AgentRuntimeService, WorkflowEngineService
Traces: aggregates RunControlService/Run
Traces: apis RunControlService/RunControl.SubmitChildRun, AgentRuntimeService/Runtime.Interrupt
Traces: events RunControlService/RunAccepted, RunControlService/RunStateChanged, RunControlService/RunCompleted, RunControlService/RunFailed, RunControlService/RunCancelled
Traces: descriptors run-submission-v1, run-accepted-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-03, FR-10
Traces: scenarios SCN-RunControlService-022
Relates: nfrs NFR-05
Depends on AIFACTORY-US-005, AIFACTORY-US-162, AIFACTORY-US-167, feature:agent-runtime-service, feature:workflow-engine-service
Sources model/services/control-plane/RunControlService.yaml, model/services/agent-mesh/AgentRuntimeService.yaml, model/services/agent-mesh/WorkflowEngineService.yaml, docs/reference/schemas/run-submission.schema.json

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-2641 Contract of SubmitChildRun and of the parent fields of a child submission contract The proto of the service holds RunControl.SubmitChildRun with the request run-submission-v1 and the reply run-accepted-v1, as model/services/control-plane/RunControlService.yaml names them; the contract document states, against docs/reference/schemas/run-submission.schema.json, that parentRunId and parentAttemptId are carried by a child submission only and that the public SubmitRun answers 400 for a body that carries either, that parentAttemptId is absent for the child of a step of kind SubWorkflow, that the idempotency key is the step run id for a step of kind SubWorkflow and the delegation id for a delegation, that RunAccepted carries the optional parentRunId for a child run (D121), that the tenant, the project, the policy revision and the release closure are inherited from the parent and ignored when sent, and which gRPC status answers each refusal, Unauthenticated, InvalidArgument, NOT_FOUND or FAILED_PRECONDITION, the last for a parent run that is terminal or Cancelling (D121). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2642 SubmitChildRun accepting a child run under its parent with inherited scope, policy revision and release closure implementation RunControl.SubmitChildRun checks the workload identity, looks the parent run up under the tenant of the caller, refuses a terminal or Cancelling parent with FAILED_PRECONDITION (D121) and a parentAttemptId that is not an attempt of the parent the same way, has the snapshot of the child stored before the commit and its pinnedReleases written to RunPinnedReleases under the child run id (AIFACTORY-TASK-2931, D125), and writes the child Run in state Accepted with ParentRunId and, for a delegation, ParentAttemptId, which are mapped with the index IX_Runs_Parent of the model, and with the tenant, the project, the policy revision and the release closure copied from the parent, together with its RunAccepted outbox record, which carries parentRunId, in one transaction; the same idempotency key, the step run id or the delegation id, with the same input returns the existing child and another input under that key is refused; every refusal writes nothing; a run with a parent waits for no BudgetReserved and gets no spend approval, so the RunDispatcher dispatches its first attempt when StepReady announces its first step (D121, D123). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2643 Cancellation of the open child runs of a parent that ends Failed or Cancelled implementation When a run that has child runs moves to Failed or to Cancelled, a CancellationIntent with a reason that names the parent run and with the actor system is recorded once for each child that is not Completed, Failed or Cancelled, and the cancel transitions are applied through the same path as CancelRun (AIFACTORY-TASK-865 and AIFACTORY-TASK-866), so that RunCancelled of the child carries that reason and actor; a child that is already Cancelling gets no second intent, a parent that ends Completed cancels nothing, and a child with children of its own passes the cancellation on when it ends Cancelled. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2644 Acceptance fixture of a parent run with child runs submitted under two workload identities fixture The acceptance suite submits child runs through a test client of RunControl.SubmitChildRun under the workload identity of AgentRuntimeService, with a parent attempt held by the fake runtime worker of AIFACTORY-TASK-871, and under the workload identity of WorkflowEngineService, without a parent attempt, and drives the parent to Completed, to Failed and to Cancelled, so that the inheritance, the idempotent repeat and the cancellation of open children are reproducible without either caller. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2645 Verify submit-a-child-run through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-RunControlService-022"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-656 with the fixture of AIFACTORY-TASK-2644 and the fake Runtime.Interrupt endpoint that records the call, including the inherited tenant, project, policy revision and release closure against a body that states others, the child without a parent attempt, the snapshot of the child stored before it is accepted, RunAccepted with parentRunId, the first attempt of a child dispatched without BudgetReserved, the idempotent repeat under the step run id and under the delegation id and the conflict, each refusal with its status, FAILED_PRECONDITION under a terminal and under a Cancelling parent among them, the open children cancelled with a Failed and with a Cancelled parent, the Completed child left alone, no intent after a Completed parent, and the payload of the terminal events of a child; the handling of those events is proven by the suites of AgentRuntimeService and WorkflowEngineService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-657: Open an operator decision when a step fails under Escalate or an outcome is Uncertain on a write step, and apply the choice retry, compensate or fail

behavior · persona operator As an operator, I want a step that failed under the failure policy Escalate, or whose outcome is uncertain on a step that writes, to wait for my choice of retry, compensate or fail, so that no side-effecting step is repeated or abandoned by a rule where a person has to judge what really happened.

Two situations must not be settled by a rule: a step whose workflow definition says that its failure is escalated, and an attempt whose outcome is Uncertain on a step that writes, where an external effect may or may not have happened. The owner decided that in both the run waits for a person, the Operator of the project (D119). The model builds this with the approval machinery and not as an aggregate of its own: an operator decision is an ApprovalRequest of kind operator-decision. RunControlService opens it itself, never a lease holder, in two cases. The first is a step that fails terminally by the rules of AIFACTORY-US-164 and AIFACTORY-US-167 while the failurePolicy the Step keeps from StepReady is Escalate. The second is an Uncertain outcome recorded on a step whose sideEffectClass is shared-write or above, whatever failure policy the step declares, whether the holder reported it through RunControlService/RunControl.ReportAttemptOutcome or the LeaseExpiryReconciler set it for an expired lease. The request names the run and the step and no lease and no fencing token, its proposalHash is the outcomeHash of the attempt that caused it, its approverRule names the Operator of the project, and its expiry is set as for every approval request (AIFACTORY-US-165). In the transaction that records the cause the run moves from Running to WaitingForApproval through OperatorDecisionRequested, and the ApprovalRequested outbox record, which carries the kind operator-decision, and RunStateChanged are written. The step is not settled while the request is Pending, so no StepSettled is written and WorkflowEngineService keeps its step run open. A step has at most one Pending operator decision, and a redelivered or repeated cause opens no second one. The request is listed by RunControlService/ListApprovals and decided through RunControlService/DecideApproval under the scope, the version and the proposal hash rules of AIFACTORY-US-166, by a caller whose roles satisfy its approver rule. The requester of the run is not excluded here: the separation of duties covers the requests of kind approval, a proposal and a spend, and the member who requested the run may take the operator decision of that run, an act that is audited (D121). The decision is one of three choices, and the request keeps it in choice. The choice retry ends the request Approved, returns the run to Running through OperatorChoseRetryOrCompensate and dispatches a new attempt of the step under a new lease and fencing token. That attempt is one more beyond the declared maximum of the step: it is dispatched also when the attempts of the step reached maxAttempts, and when it fails terminally or ends Uncertain in its turn a new operator decision is opened, on which retry may be chosen again (D121). The choice compensate ends the request Approved and returns the run to Running through the same trigger, and it settles the step Failed, so StepSettled is written; WorkflowEngineService reads the choice on ApprovalDecided and runs the compensation in its own PBI. The choice fail ends the request Rejected, settles the step Failed and fails the run through OperatorChoseFailOrDecisionExpired with the failure class of the attempt that caused the request, which is uncertain for an Uncertain outcome (D121). An operator decision that passes its expiry is never requested again, whatever an approval policy says about re-requests: the ApprovalExpiryReconciler ends it Expired, the step is settled Failed and the run fails through the same trigger, with the failure class policy-denied (D121). These are run failures that RunControlService decides itself, without a report of the engine (AIFACTORY-US-167). ApprovalDecided is written in the transaction of each of these outcomes with the proposalKind operator-decision, the choice and the stepRunId of the engine, together with RunStateChanged, with StepSettled where the step is settled and with RunFailed where the run fails. An approve, reject or request-changes decision on an operator decision is answered 409 and leaves it Pending, and so is a choice on a request of kind approval. A caller whose roles do not satisfy the approver rule is refused with 403, and an unknown request and one of another tenant are both answered 404 (ADR-0015). A run that waits for an operator decision can be cancelled (AIFACTORY-US-162), which moves the Pending request to Cancelled through the trigger RunCancelled. An operator decision that an Uncertain outcome of a tool call caused carries one thing more, the finding of the Operator on the effect, applied or not-applied, which limits the choice retry and is recorded in ToolAdapterService; the outcome marks the tool invocation whose effect is uncertain and the request keeps its id (D123); that part is delivered by AIFACTORY-US-726 (D121).

Acceptance criteria

AC1 Given a Running run and a RunControlService/Step whose failurePolicy, kept from WorkflowEngineService/StepReady, is Escalate, when its last attempt settles Failed and the failure is terminal, because it is not retriable or the attempt number reached maxAttempts, then a Pending RunControlService/ApprovalRequest of kind operator-decision exists that names the run and the step and no lease or fencing token, with a proposalHash equal to the outcomeHash of that attempt, an approver rule that names the Operator of the project and an expiry, the RunControlService/Run is WaitingForApproval through OperatorDecisionRequested, RunControlService/ApprovalRequested with the kind operator-decision and RunControlService/RunStateChanged are written in the same transaction, and neither RunControlService/StepSettled nor RunControlService/RunFailed is written (SCN-RunControlService-019, ADR-0014).
AC2 Given a step whose sideEffectClass is shared-write or above and whose failurePolicy is not Escalate, when an Uncertain outcome is reported for its attempt through RunControlService/RunControl.ReportAttemptOutcome, or the LeaseExpiryReconciler settles the attempt of its expired Claimed lease as Uncertain, then the same Pending request of kind operator-decision is opened in the transaction that writes RunControlService/AttemptOutcomeRecorded, the run waits in WaitingForApproval, no new RunControlService/Attempt is dispatched for the step, and a repeated report or a second run of the reconciler opens no second request (SCN-RunControlService-008, FUX-07).
AC3 Given the Pending operator decision and a caller whose roles satisfy its approver rule, who may be the member who requested the run, the act being audited (D121), when RunControlService/DecideApproval records the choice retry on the quoted version and proposal hash, then the request is Approved and keeps the choice retry, the run returns to Running through OperatorChoseRetryOrCompensate, a new attempt of the step is dispatched with RunControlService/AttemptDispatched, one more beyond maxAttempts when the declared maximum was reached, retry being open again on the operator decision that a later failure of the step opens (D121), and RunControlService/ApprovalDecided with the proposalKind operator-decision, the choice and the stepRunId plus RunControlService/RunStateChanged are written in the transaction of the decision; when the choice is compensate, then the request is Approved and the run returns to Running the same way, the step is settled Failed with RunControlService/StepSettled written in that transaction, and no new attempt of the step is dispatched (SCN-RunControlService-005, SCN-RunControlService-020).
AC4 Given the Pending operator decision, when RunControlService/DecideApproval records the choice fail, then the request is Rejected and keeps the choice fail, the step is settled Failed and the run moves from WaitingForApproval to Failed through OperatorChoseFailOrDecisionExpired, with RunControlService/ApprovalDecided, RunControlService/StepSettled, RunControlService/RunFailed and RunControlService/RunStateChanged written in one transaction, RunControlService/RunFailed carrying the failureClass of the attempt that caused the request, which is uncertain for an Uncertain outcome; when instead its expiresAt passes and the ApprovalExpiryReconciler runs, then the request is Expired, the step and the run end the same way except that RunControlService/RunFailed carries the failureClass policy-denied, and no new request is opened whatever onExpiry and reRequestsLeft an approval policy would give (SCN-RunControlService-020, FR-06, D121).
AC5 Given the Pending operator decision, when RunControlService/DecideApproval is called with an approve, a reject or a request-changes decision, then the response is 409; given a Pending request of kind approval, when a choice of retry, compensate or fail is submitted for it, then the response is 409; given a caller whose roles do not satisfy the approver rule of the operator decision, then the response is 403, and given an approvalId that is unknown or exists only in another tenant, then it is 404; in each case the request stays Pending, the run does not advance and nothing is published (SCN-RunControlService-005, ADR-0015).
Traces: services RunControlService, WorkflowEngineService
Traces: aggregates RunControlService/ApprovalRequest, RunControlService/Run, RunControlService/Step, RunControlService/Attempt
Traces: apis RunControlService/DecideApproval, RunControlService/ListApprovals, RunControlService/RunControl.ReportAttemptOutcome
Traces: events RunControlService/ApprovalRequested, RunControlService/ApprovalDecided, RunControlService/RunStateChanged, RunControlService/RunFailed, RunControlService/StepSettled, RunControlService/AttemptDispatched, RunControlService/AttemptOutcomeRecorded, WorkflowEngineService/StepReady
Traces: descriptors approval-decision-v1, attempt-outcome-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-06, FR-10, FUX-05, FUX-07
Traces: scenarios SCN-RunControlService-005, SCN-RunControlService-008, SCN-RunControlService-019, SCN-RunControlService-020
Relates: screens approvals-list, approval-card, run-detail
Relates: nfrs NFR-04
Depends on AIFACTORY-US-164, AIFACTORY-US-165, AIFACTORY-US-166, AIFACTORY-US-167, AIFACTORY-US-658, feature:workflow-engine-service
Sources model/services/control-plane/RunControlService.yaml, model/services/agent-mesh/WorkflowEngineService.yaml, docs/reference/schemas/approval.schema.json

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-2646 Operator decision opened by the outcome path for a terminal failure under Escalate and for an Uncertain outcome on a write step implementation The settlement of an attempt, in ReportAttemptOutcome, in the LeaseExpiryReconciler and in the terminal failure evaluation of AIFACTORY-TASK-888, opens one Pending ApprovalRequest of kind operator-decision per step when the step fails terminally under the failure policy Escalate or when the outcome is Uncertain on a step of class shared-write or above; the request names the run and the step, carries the outcome hash of the attempt as proposal hash and an approver rule that names the Operator of the project, and has no lease or fencing token; the run moves to WaitingForApproval through OperatorDecisionRequested and ApprovalRequested with the kind operator-decision, RunStateChanged and the RecoveryJournal entry are written in the transaction of the cause; the step is left unsettled and a repeated cause opens no second request; the tool invocation that an Uncertain outcome marks is kept on the request by AIFACTORY-TASK-3511 (D123). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2647 Choices retry, compensate and fail of an operator decision in DecideApproval and its expiry in the ApprovalExpiryReconciler implementation For a request of kind operator-decision DecideApproval accepts only the choices retry, compensate and fail from a caller whose roles satisfy its approver rule, the requester of the run not being excluded and the act being audited (D121), and keeps the choice on the request: retry ends it Approved, returns the run to Running through OperatorChoseRetryOrCompensate and has the RunDispatcher dispatch a new attempt of the step, one more beyond maxAttempts when the declared maximum was reached, and may be chosen again on a later operator decision of the step (D121); compensate ends it Approved, returns the run to Running and settles the step Failed; fail ends it Rejected, settles the step Failed and fails the run through OperatorChoseFailOrDecisionExpired with the failure class of the attempt that caused the request (D121); each writes ApprovalDecided with the proposal kind operator-decision, the choice and the stepRunId, RunStateChanged, StepSettled where the step is settled (AIFACTORY-TASK-2651) and RunFailed where the run fails, in one transaction; the ApprovalExpiryReconciler ends an expired operator decision Expired, settles the step Failed, fails the run through the same trigger with the failure class policy-denied (D121) and never re-requests it; the finding of a decision on an uncertain tool effect is delivered by AIFACTORY-TASK-2933 and AIFACTORY-TASK-2934. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2648 Contract of the operator decision in the approval request, the approval decision and the two approval events contract The contract documents of the service state, against docs/reference/schemas/approval.schema.json, that an ApprovalRequest carries kind approval or operator-decision and, for an operator decision, the choice retry, compensate or fail; that approval-decision-v1 takes approve, reject and request-changes for a request of kind approval and the three choices for one of kind operator-decision, every other pairing being answered 409; that ApprovalRequested carries the optional kind, whose absence means approval; and that ApprovalDecided carries the proposal kind operator-decision, the choice and the stepRunId for an operator decision; the finding of an operator decision on an uncertain tool effect is fixed by AIFACTORY-TASK-2932 (D121). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2649 Acceptance fixture publishing StepReady with a failure policy and driving an attempt to a terminal failure or an Uncertain outcome fixture The acceptance suite publishes StepReady through a fixture publisher with a stated failurePolicy, actionClass and idempotent flag and with the taskContractRef and taskContractHash the event carries (D121), and the fake runtime worker of AIFACTORY-TASK-871 reports a non-retriable Failed outcome, an Uncertain outcome or lets its lease expire under a controlled clock, so that the opening of an operator decision under Escalate and on a write step, each choice and the expiry are reproducible without WorkflowEngineService. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2650 Verify decide-a-failed-or-uncertain-step-as-operator through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Area=OperatorDecision&(Scenario=SCN-RunControlService-005\|Scenario=SCN-RunControlService-008\|Scenario=SCN-RunControlService-019\|Scenario=SCN-RunControlService-020)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-657 with the fixture of AIFACTORY-TASK-2649, including the request opened under Escalate and for an Uncertain outcome reported by the holder and set by the reconciler, the step left unsettled while the request is Pending, the new attempt after retry, dispatched beyond maxAttempts and chosen a second time on the next operator decision of the step, the settled step after compensate, the failed run after fail, with the failure class of the causing attempt, and after the expiry without a re-request, with the failure class policy-denied, the operator decision taken by the member who requested the run, the 409 of a decision that does not fit the kind of the request, the 403 and the 404, and the payload of ApprovalRequested with its kind and of ApprovalDecided with the choice and the stepRunId; the compensation is proven by the suite of WorkflowEngineService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-658: Announce once that a step reached its final state, with the step run of the engine, the step id, the number of attempts, the failure class and the output reference

behavior WorkflowEngineService plans a run and RunControlService executes its steps, so the engine has to learn when a step is over. An attempt outcome does not tell it: a failed attempt may be retried, an exhausted one fails the step, an Uncertain one waits for a person, and the rule that tells these apart lives in RunControlService alone. The owner decided that RunControlService says it explicitly (D119): the event RunControlService/StepSettled announces that a step reached its final state, and the engine treats no step as settled before it and holds no retry rule of its own. The event is written once per step, in the transaction that moves the Step to its final state, Succeeded, Failed, Skipped or Cancelled, whichever path makes that move: the report of a Succeeded attempt by the lease holder through RunControl.ReportAttemptOutcome, the terminal failure of a non-retriable or exhausted attempt found after a report or by the LeaseExpiryReconciler (AIFACTORY-US-167), or the choice compensate or fail of the Operator and the expiry of an operator decision (AIFACTORY-US-657). A step that ends Skipped or Cancelled is announced the same way with that final state. The payload carries runId, tenantId, the stepRunId, the finalState, the attemptCount and, when the final state is Failed, the failureClass of the last attempt of the step. It carries as well the stepId of RunControlService and, for a step whose settling attempt reported an output, the outputRef of that attempt, the reference of the output in ArtifactService; outputRef is absent when the step settled without an output (D121). stepRunId is the step run id of the engine: CreateStepAndDispatch keeps it on the Step when it creates the step from StepReady (AIFACTORY-US-163), and it is the key by which the engine finds its step run; the stepId beside it is the id under which RunControlService and the attempt events name the same step. A step that is not final writes nothing. An attempt that Failed and is retried, an attempt whose lease expired unclaimed and is dispatched again, and a step whose terminal failure or Uncertain outcome opened an operator decision that is still Pending all leave the step open, and the choice retry keeps it open with a new attempt. A step is settled once: its final state refuses every later transition, a duplicate report or a redelivered cause writes no second event, and the outbox record belongs to the transaction of the transition, so that neither an event without a settled step nor a settled step without an event can exist. The event is published on run-control.steps.v1.step-settled under the canonical envelope by the dispatcher of the run-control messaging profile; WorkflowEngineService records the final state of the step run and applies the failure policy of a step settled Failed through its handler ApplyStepSettled in its own PBI. For a failed step the event is all that RunControlService does: it settles the step and does not fail the run, the engine applies the failure policy, Fail or Compensate, and reports WorkflowRunFailed, and the run fails on that report (AIFACTORY-US-167, D121). AttemptOutcomeRecorded keeps being published for every attempt and keeps its other consumers; it no longer tells the engine that a step is over.

Acceptance criteria

AC1 Given a Running run and a dispatched RunControlService/Step that keeps the stepRunId of WorkflowEngineService/StepReady, when its attempt is reported Succeeded through RunControlService/RunControl.ReportAttemptOutcome, then the step is Succeeded and the RunControlService/StepSettled outbox record with runId, tenantId, that stepRunId, the stepId of the step, the finalState Succeeded, the attemptCount, no failureClass and, when the settling attempt reported an output with its outcome, the outputRef of that attempt is written in the transaction of the report, beside RunControlService/AttemptOutcomeRecorded (SCN-RunControlService-023, ADR-0014, D121).
AC2 Given a step under a failure policy other than Escalate whose second attempt fails terminally, when the step moves to Failed, then RunControlService/StepSettled with the finalState Failed, the attemptCount 2 and the failureClass of that last attempt is written in the transaction that fails the step, and that transaction does not fail the run, which stays Running until the engine reports its failure (D121); given a Failed attempt that is retried, an attempt whose lease expired unclaimed and is dispatched again, or a step with a Pending operator decision, then no RunControlService/StepSettled is written while the step is open (SCN-RunControlService-023, FR-10).
AC3 Given a settled step, when the outcome report is repeated, its cause is redelivered or the evaluation runs again, then no second RunControlService/StepSettled is written and the step keeps its final state; given a Pending operator decision on a step, when the Operator chooses compensate or fail or the request expires, then the step is settled Failed and exactly one RunControlService/StepSettled is written in the transaction of that decision, and when the Operator chooses retry, then none is written (SCN-RunControlService-023).
AC4 Given the record was written, when the outbox dispatcher of the run-control messaging profile publishes RunControlService/StepSettled on run-control.steps.v1.step-settled, then the event is sent under the canonical envelope with runId, tenantId, stepRunId, stepId, finalState, attemptCount, for a Failed step failureClass and, for a step whose settling attempt reported an output, outputRef (D121), and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-023, ADR-0012).
Traces: services RunControlService, WorkflowEngineService
Traces: aggregates RunControlService/Step, RunControlService/Attempt
Traces: apis RunControlService/RunControl.ReportAttemptOutcome
Traces: events RunControlService/StepSettled, RunControlService/AttemptOutcomeRecorded, WorkflowEngineService/StepReady
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-03, FR-10
Traces: scenarios SCN-RunControlService-023
Relates: nfrs NFR-05
Depends on AIFACTORY-US-163, AIFACTORY-US-164, AIFACTORY-US-167, feature:workflow-engine-service
Sources model/services/control-plane/RunControlService.yaml, model/services/agent-mesh/WorkflowEngineService.yaml, model/scenarios/RunControlService.yaml

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-2651 StepSettled written in the transaction that moves a step to its final state implementation Every path that moves a Step to Succeeded, Failed, Skipped or Cancelled, that is the report of a Succeeded attempt, the terminal failure of AIFACTORY-TASK-888, the choice compensate or fail and the expiry of an operator decision (AIFACTORY-TASK-2647) and the settlement of a step as Skipped or Cancelled, writes the StepSettled outbox record in the same transaction with runId, tenantId, the stepRunId the step keeps from StepReady, the stepId of the step, the final state, the number of attempts, for a Failed step the failure class of its last attempt and, for a step whose settling attempt reported an output, the outputRef of that attempt (D121); a step that stays open writes none, and a final step refuses a later transition, so the record is written once per step. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2652 Contract of the StepSettled event against the canonical envelope contract A golden file fixes StepSettled on the topic run-control.steps.v1.step-settled under the canonical envelope with the payload fields runId, tenantId, stepRunId, stepId, finalState, attemptCount and the optional failureClass and outputRef of model/services/control-plane/RunControlService.yaml (D121), and a contract test fails when the serialised event differs from the file, lacks the stepId or carries a failureClass for a final state other than Failed. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2653 Verify announce-a-settled-step through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-RunControlService-023"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-658 with the fixtures of AIFACTORY-TASK-871 and AIFACTORY-TASK-2649, including the event of a Succeeded step with its stepId and with the outputRef of the settling attempt, of a step Failed after two attempts with its failure class while the run stays Running, no event for a retried attempt, for a lease dispatched again and for a step with a Pending operator decision, one event after the choice compensate and after the choice fail, none after retry, no second event for a repeated report, and the publication and payload of StepSettled; the handling of the event is proven by the suite of WorkflowEngineService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-726: Take the finding of the Operator on an uncertain tool effect with the operator decision, refuse a retry unless the effect was not applied, and record the finding in ToolAdapterService

behavior · persona operator As an operator, I want to state, when I decide a step whose tool call ended with an uncertain effect, whether that effect was applied, and to be refused a retry unless it was not, so that a write is never repeated on top of an effect that already happened, and the tool service learns my finding without my acting in it.

An Uncertain outcome on a step that writes often comes from a tool call whose effect could not be confirmed: ToolAdapterService settled the invocation Uncertain, the attempt ended Uncertain, waited for nothing, and RunControlService opened an operator decision for the Operator of the project (AIFACTORY-US-657). The outcome of such an attempt marks the tool invocation whose effect is uncertain, in the optional member uncertainToolInvocationId of attempt-outcome-v1, and the ApprovalRequest that RunControlService opens for it keeps that id in its own uncertainToolInvocationId; a request that keeps the id is the operator decision on an uncertain tool effect, and every other request keeps none (D123). The owner decided that the Operator judges the effect in one place, the operator decision of the run, and that no operator acts in the tool service directly (D121). The decision of an operator decision that an Uncertain outcome of a tool call caused therefore carries, in approval-decision-v1, a finding beside the choice: applied, when the Operator found from the tool receipt and the external evidence that the effect happened, or not-applied, when it did not. RunControlService/DecideApproval keeps the finding on the ApprovalRequest, in the field finding beside choice, and ApprovalDecided carries it. A decision on such a request always states a finding: a decision that carries none, whatever its choice, retry, compensate or fail, is answered 400, the request stays Pending and nothing is recorded (D123). The finding limits the choice. The choice retry is accepted only with the finding not-applied, because a retry repeats the write: a retry with the finding applied is answered 409, the request stays Pending and the run keeps waiting. The choices compensate and fail are taken with either finding, and every accepted choice does to the step and to the run what AIFACTORY-US-657 states. As part of the same act RunControlService records the finding in ToolAdapterService through ToolAdapterService/ConfirmToolEffect, the internal operation that takes the finding for one invocation under the scope factory.tools.operate, and it calls it for the invocation the request keeps in uncertainToolInvocationId; there the finding applied moves the Uncertain invocation to Observed and the finding not-applied moves it to Failed, in the PBIs of ToolAdapterService. RunControlService sends the call with a workload token that holds factory.tools.operate and names the Operator who decided as the user it acts for (D123). ToolAdapterService is a required fail-fast dependency of this act: when it does not answer the call, DecideApproval answers 503 and records nothing, so the request stays Pending with no choice and no finding, the run keeps waiting, no event is published and the Operator takes the decision again; the decision and the finding therefore never diverge (D123). ToolAdapterService answers a repeated ConfirmToolEffect with the same finding as it answered the first, so a decision taken again with the same finding after an unanswered call is recorded; when it answers 409, because the invocation already keeps another finding, DecideApproval answers 409 and records nothing, and the request stays Pending (D125). A refused decision records nothing there. An operator decision whose request keeps no invocation id, as the one opened for a terminal failure under the failure policy Escalate or for an Uncertain outcome that marks no tool invocation, takes no finding: its choices are not limited, ApprovalDecided carries none and ToolAdapterService is not called, and a decision on it that supplies a finding is answered 400 and records nothing (D125). The authority, the version and the proposal hash rules of the decision are those of AIFACTORY-US-166 and AIFACTORY-US-657, the requester of the run being allowed to take the operator decision.

Acceptance criteria

AC1 Given a Running run and a RunControlService/Step whose sideEffectClass is shared-write or above, when RunControlService/RunControl.ReportAttemptOutcome records an Uncertain outcome whose attempt-outcome-v1 marks a tool invocation in uncertainToolInvocationId, then the Pending RunControlService/ApprovalRequest of kind operator-decision that is opened for the outcome keeps that id in uncertainToolInvocationId; given an Uncertain outcome that marks no tool invocation, or a terminal failure under the failure policy Escalate, then the request that is opened keeps none (SCN-RunControlService-008, D123).
AC2 Given a Pending RunControlService/ApprovalRequest of kind operator-decision that keeps an uncertainToolInvocationId and a caller whose roles satisfy its approver rule, when RunControlService/DecideApproval records the choice retry with the finding not-applied on the quoted version and proposal hash, then the response is 200, the request is Approved and keeps the choice retry and the finding not-applied, the run returns to Running with a new attempt of the step, the finding is recorded through ToolAdapterService/ConfirmToolEffect for the invocation the request keeps, under a workload token that holds factory.tools.operate and names the Operator who decided as the user it acts for (D123), and RunControlService/ApprovalDecided with the choice, the stepRunId and the finding plus RunControlService/RunStateChanged are written in the transaction of the decision; when the choice compensate or the choice fail is recorded with the finding applied or with the finding not-applied, then the choice is applied as for any operator decision, the request keeps the finding, it is recorded through ToolAdapterService/ConfirmToolEffect the same way, and RunControlService/ApprovalDecided carries it beside the choice (SCN-RunControlService-005, SCN-RunControlService-020, D121).
AC3 Given the same Pending request, when a decision that carries no finding is submitted to RunControlService/DecideApproval with the choice retry, compensate or fail, then the response is 400 with a problem response (D123); when the choice retry is submitted with the finding applied, then the response is 409; and when a decision that would be accepted is submitted while ToolAdapterService does not answer the call of ToolAdapterService/ConfirmToolEffect, then the response is 503 (D123); in each case the request stays Pending without a choice and without a finding, the run stays WaitingForApproval, no attempt is dispatched and nothing is published, ToolAdapterService/ConfirmToolEffect is not called for the 400 and for the 409, and the decision that was answered 503 is accepted when the Operator takes it again once ToolAdapterService answers (SCN-RunControlService-005, FUX-07).
AC4 Given a Pending operator decision that keeps no uncertainToolInvocationId, as the one a terminal failure under the failure policy Escalate caused, when RunControlService/DecideApproval records the choice retry without a finding, then the choice is accepted, RunControlService/ApprovalDecided carries no finding and ToolAdapterService/ConfirmToolEffect is not called, while the same decision supplying a finding is answered 400 and records nothing (D125); and given a decision on an uncertain tool effect taken again with the same finding after an unanswered call, then it is recorded, while a ToolAdapterService/ConfirmToolEffect answered 409 because the invocation keeps another finding makes RunControlService/DecideApproval answer 409 with the request left Pending (SCN-RunControlService-005, SCN-RunControlService-020, FR-06, D125).
AC5 Given the decision of AC1 or AC3 was recorded, when the outbox dispatcher publishes RunControlService/ApprovalDecided on run-control.approvals.v1.approval-decided, then the event is sent under the canonical envelope with approvalId, runId, tenantId, state, projectId, the proposalKind operator-decision, proposalHash, the choice, the stepRunId and the finding, applied or not-applied, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-RunControlService-020, ADR-0012).
Traces: services RunControlService, ToolAdapterService
Traces: aggregates RunControlService/ApprovalRequest, RunControlService/Run, RunControlService/Step
Traces: apis RunControlService/DecideApproval, RunControlService/RunControl.ReportAttemptOutcome, ToolAdapterService/ConfirmToolEffect
Traces: events RunControlService/ApprovalDecided, RunControlService/RunStateChanged
Traces: descriptors approval-decision-v1, attempt-outcome-v1
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-06, FUX-05, FUX-07
Traces: scenarios SCN-RunControlService-005, SCN-RunControlService-008, SCN-RunControlService-020
Relates: screens approval-card
Relates: nfrs NFR-04
Depends on AIFACTORY-US-166, AIFACTORY-US-657, feature:tool-adapter-service
Sources model/services/control-plane/RunControlService.yaml, model/services/tools-mcp/ToolAdapterService.yaml, docs/reference/schemas/approval.schema.json, docs/reference/schemas/attempt-outcome.schema.json

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-2932 Contract of the finding in the approval decision, the approval request and ApprovalDecided contract The contract documents of the service state, against docs/reference/schemas/approval.schema.json, that approval-decision-v1 carries the finding applied or not-applied for an operator decision that an Uncertain outcome of a tool call caused, that the ApprovalRequest keeps it in the field finding, that ApprovalDecided carries the optional finding for such a decision and none otherwise, and that the choice retry with the finding applied is answered 409 (D121); they state as well, against docs/reference/schemas/attempt-outcome.schema.json, that attempt-outcome-v1 carries the optional uncertainToolInvocationId on an Uncertain outcome, that the ApprovalRequest keeps it in the field uncertainToolInvocationId, that DecideApproval answers 400 for a decision without a finding on a request that keeps it, and that it answers 503 and records nothing when ToolAdapterService does not answer the call of ConfirmToolEffect (D123); a golden file fixes ApprovalDecided with and without the finding. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2933 Finding of the Operator in DecideApproval with the retry rule and the Finding column of ApprovalRequests implementation For an operator decision on an uncertain tool effect, the request that keeps an uncertainToolInvocationId, DecideApproval takes the finding of approval-decision-v1 beside the choice, answers a decision without a finding with 400 whatever its choice (D123), answers the choice retry with the finding applied with 409 and leaves the request Pending in both cases, keeps the finding of an accepted decision on the ApprovalRequest, mapped to the nullable column Finding of the table ApprovalRequests, and writes it to ApprovalDecided in the transaction of the decision; an operator decision whose request keeps no invocation id takes no finding, its choices are not limited, and a decision on it that supplies a finding is answered 400 (D125); unit tests cover each choice against each finding and against a missing one. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-3511 Invocation id of the uncertain tool effect kept on the operator decision, with the UncertainToolInvocationId column of ApprovalRequests implementation ReportAttemptOutcome reads the optional uncertainToolInvocationId of attempt-outcome-v1 on an Uncertain outcome, and the operator decision that the outcome opens (AIFACTORY-TASK-2646) keeps it on the ApprovalRequest in the field uncertainToolInvocationId, mapped to the nullable column UncertainToolInvocationId of the table ApprovalRequests (D123); a request opened for an Uncertain outcome that marks no invocation, for a terminal failure under the failure policy Escalate, or for any approval keeps none; the id is the one the call of ConfirmToolEffect of AIFACTORY-TASK-2934 names; unit tests cover the outcome with and without the marker. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2934 Typed client of ConfirmToolEffect recording the finding in ToolAdapterService as part of the decision implementation DecideApproval records the finding of an accepted decision through a typed client of ConfirmToolEffect of ToolAdapterService, called with a workload token that holds factory.tools.operate and names the Operator who decided as the user it acts for (D123), and with the finding applied or not-applied and the reason of the decision for the invocation the request keeps in uncertainToolInvocationId (AIFACTORY-TASK-3511); a refused decision and an operator decision whose request keeps no invocation id call nothing; the call belongs to the act of the decision and is answered before the decision is committed: when ToolAdapterService does not answer it, DecideApproval answers 503 with a problem response and records nothing, so the request stays Pending with no choice and no finding, no event is written and the same decision can be taken again (D123); the repeat relies on the natural idempotency of ConfirmToolEffect, which answers a repeated call with the same finding as the first, and a 409 of ConfirmToolEffect, the answer for another finding than the one the invocation keeps, is returned by DecideApproval as 409 with nothing recorded (D125). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2935 Acceptance fixture of a fake ConfirmToolEffect endpoint and of an operator decision on an uncertain tool effect fixture The acceptance suite hosts a fake ConfirmToolEffect endpoint that records the invocation id, the finding and the token of every call, with the user the token names, and that can be stopped so that it does not answer; on the fixture of AIFACTORY-TASK-2649 the fake runtime worker reports an Uncertain outcome whose attempt-outcome-v1 marks one tool invocation in uncertainToolInvocationId, and one that marks none, so that the id kept on the request, the finding kept on the request, the refused decisions, the 503 of an unanswered call and the call to ToolAdapterService are reproducible without ToolAdapterService. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-2936 Verify record-operator-finding-on-uncertain-tool-effect through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Area=OperatorFinding&(Scenario=SCN-RunControlService-005\|Scenario=SCN-RunControlService-008\|Scenario=SCN-RunControlService-020)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-726 with the fixture of AIFACTORY-TASK-2935, including the invocation id that an Uncertain outcome marks kept on the operator decision and none kept for an outcome without the marker, the retry accepted with the finding not-applied, the retry answered 409 with the finding applied, the 400 for retry, compensate and fail without a finding, compensate and fail with either finding, the finding kept on the request and carried by ApprovalDecided, the call to ConfirmToolEffect recorded once for an accepted decision, for the invocation the request keeps and under a workload token that names the Operator who decided, and never for a refused one, the 503 with the fake stopped, after which the request is still Pending with no choice and no finding and no event was written, the same decision accepted once the fake answers again, the 409 returned when the fake answers 409 for another finding, and the operator decision under Escalate that takes no finding and answers 400 when one is supplied (D125); the settlement of the invocation is proven by the suite of ToolAdapterService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-1491: Answer the input or the escalation choice a waiting attempt asks for through the public route of Run Control and pass it to the attempt

behavior · persona requester As a member of the project who follows a run, I want to give the missing input or the escalation choice an agent asks for while its attempt waits, so that the attempt resumes on my answer without anyone calling the runtime directly.

An attempt can stop inside its loop for a person in two cases that are no approval: the task lacks an input the requester holds, or the agent lays options before a human with a recommendation; AgentRuntimeService then keeps a decision request of kind Input or Escalation and the attempt waits in WaitingForDecision (AIFACTORY-US-466 of feature agent-runtime-service). The owner decided how a person answers it (D137): RunControlService has a public route, RunControlService/AnswerDecision, on POST /factory/runs/{runId}/decisions/{decisionId}/answer, which the Studio and the CLI call behind ControlPlaneApi with the user token of a member whose scope carries factory.runs.operate. The body names the answer, the input or the chosen option. The service looks the run up under the tenant row filter and the projects in the trusted scope, takes its open attempt and passes the answer to it through AgentRuntimeService/Runtime.AnswerDecision with the attempt, the decision and the answer, under its workload identity; the reply of the runtime decides the answer of the route. When the runtime acknowledges, the route answers 200; the decision request is answered and the attempt returns to Running in AgentRuntimeService, which announces it in its own PBI. RunControlService writes no state of its own for the answer and publishes no event, and the run keeps its state, because an attempt that waits for a decision of this kind keeps its lease and its run stays Running. The route is naturally idempotent: the same answer repeated for a decision already answered is acknowledged by the runtime without a second effect and answered 200 again, while another answer for a decision already answered is answered 409. An approval is never answered here: the decision of an approval goes through RunControlService/DecideApproval and reaches the attempt through ApprovalDecided (D137), so a decision of kind Approval named on this route is answered 409 and nothing is passed on. A run outside the trusted scope, a run of another tenant, a run without an open attempt and a decision the open attempt did not ask for are answered 404, the run of another tenant with the same answer as a run that does not exist (ADR-0015). A body without an answer is answered 400, and a caller without factory.runs.operate is answered 403. AgentRuntimeService is a required fail-fast dependency of the route: when it does not answer, the route answers 503 with a problem response that names it and records nothing, and the member answers again. No refusal changes state anywhere.

Acceptance criteria

AC1 Given a Running run in the trusted scope whose open attempt waits for a decision of kind Input or Escalation and a member whose scope carries factory.runs.operate, when RunControlService/AnswerDecision is called with the answer, then AgentRuntimeService/Runtime.AnswerDecision is called once with the attempt, the decision and the answer under the workload identity of RunControlService, the response is 200 once the runtime acknowledged, the RunControlService/Run keeps its state and RunControlService writes no row and no outbox record for the answer (SCN-RunControlService-024, D137).
AC2 Given the same answer repeated for the decision of AC1, when RunControlService/AnswerDecision is called again, then the response is 200 and the runtime applies nothing a second time; given another answer for that decision, then the response is 409; given a decision of kind Approval named on the route, then the response is 409, AgentRuntimeService/Runtime.AnswerDecision is not called, and the approval stays to be decided through RunControlService/DecideApproval (SCN-RunControlService-024, D137).
AC3 Given a run of another tenant, a run outside the projects of the trusted scope, a run without an open attempt or a decision the open attempt did not ask for, when RunControlService/AnswerDecision is called, then the response is 404, the foreign run answered like a run that does not exist; given a body without an answer, then 400, and given a caller without factory.runs.operate, then 403; in each case nothing is passed to the attempt and no state changes (SCN-RunControlService-024, ADR-0015).
AC4 Given AgentRuntimeService does not answer, when RunControlService/AnswerDecision is called with an answer that would be accepted, then the response is 503 with a problem response that names AgentRuntimeService, nothing is recorded, the attempt still waits, and the same call is accepted once AgentRuntimeService answers again (SCN-RunControlService-024, FUX-07).
Traces: services RunControlService, AgentRuntimeService
Traces: aggregates RunControlService/Run, RunControlService/Attempt
Traces: apis RunControlService/AnswerDecision, RunControlService/DecideApproval, AgentRuntimeService/Runtime.AnswerDecision
Traces: adrs ADR-0014, ADR-0015
Traces: requirements FR-06, FUX-07
Traces: scenarios SCN-RunControlService-024
Relates: screens run-detail
Relates: nfrs NFR-04
Depends on AIFACTORY-US-163, feature:agent-runtime-service, feature:control-plane-api
Sources model/services/control-plane/RunControlService.yaml, model/services/agent-mesh/AgentRuntimeService.yaml, docs/planes/control/services/run-control-service/index.md#apis

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-6061 AnswerDecision route passing the answer to the waiting attempt through Runtime.AnswerDecision implementation POST /factory/runs/{runId}/decisions/{decisionId}/answer is hosted under the user scope factory.runs.operate with the side-effect class shared-write and natural idempotency and is listed in the service descriptor file with the codes 200, 400, 403, 404, 409 and 503; it looks the run up under the tenant row filter and the projects of the trusted scope, takes its open attempt and calls Runtime.AnswerDecision of AgentRuntimeService through a typed gRPC client under the workload identity of the service with the attempt, the decision and the answer; it maps the acknowledgement to 200, NOT_FOUND to 404, FAILED_PRECONDITION for another answer of an answered decision to 409, a decision of kind Approval to 409 without a call, an unanswered call to 503 with a problem response naming AgentRuntimeService, and writes no row and no outbox record in any case (D137). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6062 CLI command answer for a waiting decision of a run implementation The client sends the answer of a member to the route AnswerDecision for a run id and a decision id, prints 200 as answered and prints the problem response of a 404, 409 or 503 unchanged. ConnectSoft.AI.SoftwareFactory.Cli
AIFACTORY-TASK-6063 Acceptance fixture of a fake Runtime.AnswerDecision endpoint fixture The acceptance suite hosts a fake Runtime.AnswerDecision endpoint that records the attempt, the decision, the answer and the caller identity of every call, acknowledges a first answer and the same answer repeated, answers NOT_FOUND for a decision it does not hold and FAILED_PRECONDITION for another answer of an answered decision, and can be stopped so that it does not answer. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6064 Verify answer-a-decision-an-attempt-asks-for through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-RunControlService-024"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1491 with the fixture of AIFACTORY-TASK-6063: the answer passed once with the workload identity of the service and answered 200 while the run keeps its state and nothing is written, the repeated answer answered 200, another answer and a decision of kind Approval answered 409, the 404 for a foreign run, a run outside the scope, a run without an open attempt and an unknown decision, the 400 and the 403, and the 503 with the fake stopped after which nothing is recorded and the same call is accepted once the fake answers; the resumption of the attempt is proven by the suite of AgentRuntimeService. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-1492: Open and decide the approval of a provisioning run, an evaluation run, an asset release or a promotion that the service owning it requests, and announce the decision to that service

behavior · persona reviewer As a reviewer, I want the approvals of a baseline provisioning, a waiver of a failed gate, the publication of a release and a production promotion to come to me in the same list and under the same rules as the approvals of a run, so that a decision on a subject that belongs to no run is still bound to the exact proposal and reaches the service that owns it.

Not every approval belongs to a run. The owner decided that an approval request names its subject as a kind and an id from a closed list of five, run, provisioning-run, evaluation-run, asset-release and promotion, that runId is optional and present only when the subject is a run or belongs to one, and that the service that owns the subject receives the decision (D129, D131, D135). A service requests the approval of a subject it owns with its own service identity through RunControlService/RunControl.RequestApproval, whose workload token holds the scope factory.approvals.request: EnvironmentPromotionService for a promotion, IaCProvisioningService for a provisioning run, QualityScoreService for the waiver of a failed gate of an evaluation run and AgentRegistryService, the publisher of an asset release, for that release. The request approval-request-v1 names the subject kind and id, the project of the subject where the subject has one, an asset release having none, the requester the service names, the action class, the proposal kind and the hash of the exact proposal. The service opens a Pending ApprovalRequest that keeps subjectKind, subjectId, projectId, the requesting service in requestingService and the named requester in requestedBy, takes the approver rule the requester supplies in the request and resolves it through PolicyEngineService for the active policy revision only when the request carries none (D141): for the publication of an asset release AgentRegistryService names the role by asset kind, a Reviewer who is not the submitter or a Platform admin for a ModelProfile release (D119, AIFACTORY-US-416 of feature agent-registry-service-publication), and RunControlService enforces that rule when the decision is taken; it copies onExpiry, reRequestsLeft and quorum from the approval rule when one is supplied, fail, 0 and 1 until the approval rules arrive in I4 (D127), sets the expiry as for every request, and writes RunControlService/ApprovalRequested with subjectKind, subjectId and requestingService and without runId and stepId when the subject belongs to no run; the reply is approval-ref-v1. Such a request names no step, no lease and no fencing token and moves no run state, and asks for none of them. As a design choice of this PBI a repeated request of the same service for the same subject and the same proposal hash while a request of that subject is Pending returns the existing reference, and a request for another proposal hash of a subject that has a Pending request is refused, as for a step. The request is listed by RunControlService/ListApprovals and decided through RunControlService/DecideApproval under the rules of AIFACTORY-US-166: the scope factory.approvals.decide, the approver rule, the quoted version and proposal hash, the votes up to the quorum, and the separation of duties, under which the requester the service named is never eligible to decide (D18, D135). The decision moves no run: RunControlService/ApprovalDecided is written with subjectKind, subjectId, projectId, the state and the proposal of the request and without runId when the subject belongs to no run, so that the owning service, which consumes the event, completes or refuses its own act in its own PBI. A request-changes decision returns the request ChangesRequested to its owner the same way, which opens a new request for a new proposal. The ApprovalExpiryReconciler moves such a request past its expiry to Expired and announces it through ApprovalDecided with state Expired, and it fails no run (D129, D135). A caller without workload identity is refused with Unauthenticated and a caller without the scope with PermissionDenied; a subject kind outside the closed list, a request without a subject id and a request of a service that names the subject kind run without being the lease holder of that run are refused with InvalidArgument; no refusal opens a request.

Acceptance criteria

AC1 Given a caller with the service identity of IaCProvisioningService whose workload token holds factory.approvals.request, when RunControlService/RunControl.RequestApproval is called with approval-request-v1 naming the subject kind provisioning-run, its id, its project, the requester, the action class and the proposal hash and no approver rule, then the reply carries approval-ref-v1, a Pending RunControlService/ApprovalRequest keeps subjectKind, subjectId, projectId, requestingService, requestedBy, the approver rule resolved through PolicyEngineService/Policy.Decide, the quorum 1 and no runId, step, lease or fencing token, no RunControlService/Run changes state, and RunControlService/ApprovalRequested is written in the same transaction; the subject kinds evaluation-run, asset-release and promotion are accepted the same way from QualityScoreService, AgentRegistryService and EnvironmentPromotionService; given the request of AgentRegistryService for an asset release that names the approver rule of its asset kind, Reviewer or, for a ModelProfile release, Platform admin, then the request keeps that rule and PolicyEngineService is not asked (SCN-RunControlService-009, D119, D129, D131, D135, D141).
AC2 Given a caller without workload identity, a caller without factory.approvals.request, a subject kind outside the closed list, a request without a subject id, a request of a service that names the subject kind run, or another proposal hash for a subject with a Pending request, when RunControlService/RunControl.RequestApproval is called, then the call is refused with Unauthenticated, PermissionDenied, InvalidArgument or FailedPrecondition respectively and no request is opened; given the same request repeated for the same subject and proposal hash, then the existing approval-ref-v1 is returned (SCN-RunControlService-009).
AC3 Given the Pending request of AC1 and a reviewer whose roles satisfy its approver rule and who is not the requester the service named, when RunControlService/DecideApproval approves, rejects or requests changes on the quoted version and proposal hash, then the request is Approved, Rejected or ChangesRequested, no run moves, and RunControlService/ApprovalDecided with subjectKind, subjectId, projectId, state and proposalHash and without runId is written in the transaction of the decision; given the requester the service named, then the response is 403 and the request stays Pending (SCN-RunControlService-005, D18, D135).
AC4 Given the Pending request of AC1 whose expiresAt has passed, when the ApprovalExpiryReconciler runs, then the request is Expired, RunControlService/ApprovalDecided with state Expired and the subject is written, and no RunControlService/Run is failed (SCN-RunControlService-020, D129).
AC5 Given a request of a subject a service owns was opened or decided, when the outbox dispatcher publishes RunControlService/ApprovalRequested or RunControlService/ApprovalDecided, then each event is sent under the canonical envelope with subjectKind, subjectId and, for RunControlService/ApprovalRequested, requestingService, without runId and stepId for a subject that belongs to no run, and a transport outage is retried with backoff so neither is lost nor duplicated under its event id (SCN-RunControlService-019, SCN-RunControlService-020, ADR-0012).
Traces: services RunControlService, PolicyEngineService
Traces: aggregates RunControlService/ApprovalRequest, RunControlService/Run
Traces: apis RunControlService/RunControl.RequestApproval, RunControlService/DecideApproval, RunControlService/ListApprovals, PolicyEngineService/Policy.Decide
Traces: events RunControlService/ApprovalRequested, RunControlService/ApprovalDecided
Traces: descriptors approval-request-v1, approval-ref-v1, approval-decision-v1
Traces: adrs ADR-0012, ADR-0014
Traces: requirements FR-06, FUX-05
Traces: scenarios SCN-RunControlService-005, SCN-RunControlService-009, SCN-RunControlService-019, SCN-RunControlService-020
Relates: screens approvals-list, approval-card
Relates: nfrs NFR-04
Depends on AIFACTORY-US-165, AIFACTORY-US-166, feature:policy-engine-service
Sources model/services/control-plane/RunControlService.yaml, docs/reference/schemas/approval.schema.json

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-6065 Contract of the subject of an approval request and of the two approval events contract The contract documents of the service state, against docs/reference/schemas/approval.schema.json, that approval-request-v1 names its subject as a kind and an id from the closed list run, provisioning-run, evaluation-run, asset-release and promotion, that runId is optional and present when the subject is a run or belongs to one, that requestingService names a service that requested the approval of a subject it owns, and that ApprovalRequested and ApprovalDecided carry subjectKind and subjectId and the optional runId (D129, D131, D135); a golden file fixes both events for a run and for a provisioning run. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6066 RequestApproval from the service that owns the subject, with the subject columns of ApprovalRequests implementation RunControl.RequestApproval admits a workload token with factory.approvals.request from the service that owns the subject and opens a Pending ApprovalRequest with SubjectKind, SubjectId, ProjectId, RequestingService and RequestedBy, mapped to the columns of the table ApprovalRequests with the index IX_Approvals_Tenant_Subject, an empty RunId and StepId for a subject that belongs to no run, the approver rule the request supplies, resolved through PolicyEngineService only when the request carries none (D141), and onExpiry, reRequestsLeft and quorum from the rule (fail, 0 and 1 without one, D127); it moves no run, writes ApprovalRequested with the subject in the same transaction, returns the existing reference for a repeat with the same subject and proposal hash and refuses the cases of AC2 of AIFACTORY-US-1492 without a write. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6067 Decision and expiry of a request whose subject is not a run implementation DecideApproval and the ApprovalExpiryReconciler apply to a request whose subject is not a run the rules of a request of kind approval, the votes and the separation of duties against the requester the service named included, move no run and write ApprovalDecided with subjectKind, subjectId, projectId, the state and the proposal and without runId in the transaction of the decision or the expiry; an expired request of such a subject fails no run (D129, D135). ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6068 Verify approve-a-subject-a-service-owns through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Area=SubjectApproval&(Scenario=SCN-RunControlService-005\|Scenario=SCN-RunControlService-009\|Scenario=SCN-RunControlService-019\|Scenario=SCN-RunControlService-020)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-1492 with test clients under the workload identities of IaCProvisioningService, QualityScoreService, AgentRegistryService and EnvironmentPromotionService: the request of each of the four subject kinds opened without a run, step or lease, the approver rule an asset-release request supplies kept without a call to PolicyEngineService and the rule of a request without one resolved through it (D141), each refusal, the repeat that returns the existing reference, the decision that moves no run and the 403 of the named requester, the expiry that fails no run, and the payload of both events with the subject and without runId; the handling of the decision is proven by the suites of the owning services. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-1606: Bind an opened change proposal to the run that produced it and record its merged commit on that binding

behavior · persona reviewer As a reviewer, I want the change proposal that a run opened in the system of record bound to that run with its merged commit, so that an approval of the run names the exact proposal version and the pipeline run of the merged commit finds the run it belongs to.

PipelineExecutionService announces through PipelineExecutionService/ChangeProposalOpened that the change set of a Factory run became a branch and a pull request in the system of record, with proposalId, tenantId, projectId, runId, repository and the optional pullRequestRef; in I1 the thin delivery publishes it when Pipelines.RecordProposal records the pull request that the tool propose-change-set opened (AIFACTORY-US-1278 of feature pipeline-execution-service-thin), and from I2 also for OpenChangeProposal. The inbox handler BindChangeProposal of RunControlService consumes it, idempotent on the event id, finds the Run that runId names under the tenant of the event and records the binding of the proposal to that run, with the proposal id, the repository and the pull request reference, so that an approval of the run can name the exact proposal version, which is the purpose the model gives the handler (D140). The same handler consumes PipelineExecutionService/ChangeProposalMerged, which carries proposalId, tenantId, projectId, runId and mergedCommit and which the complete delivery of PipelineExecutionService publishes from I2 (AIFACTORY-US-1282 of feature pipeline-execution-service-complete), and records the merged commit on the binding of that proposal. The handler SettleWaitingStep of I2 does not read the binding: it looks up the step that waits for a pipeline run directly by the runId and stepRunId that PipelineRunCompleted echoes from the requesting step (AIFACTORY-US-1608 of feature pipeline-execution-service-complete, D141). The binding is a record beside the run and not a transition of it: the handler moves no state of the run, writes no entry of the RecoveryJournal and publishes nothing, so it is written for a run in any state, a Completed run included, because a pull request is often merged after the run that proposed it has completed. A merge announced before the opening, as when the two events arrive out of order, writes the binding with the merged commit, and the opening completes it. An event that names a run of another tenant, which the tenant row filter does not find, or a run that does not exist writes nothing and completes; a redelivered event changes nothing. When run-control-sql does not answer, the handler fails, nothing is written and the inbox retries the event. The binding is the entity ChangeProposalBinding of the Run, kept in the table ChangeProposalBindings of run-control-sql, keyed by tenant, run and proposal with the repository, the pull request reference and the merged commit (model/services/control-plane/RunControlService.yaml, D141). Before I2 no ChangeProposalMerged is published, so the merge path is proven with fixture events of the pinned contracts package.

Acceptance criteria

AC1 Given a RunControlService/Run of tenant A, when PipelineExecutionService/ChangeProposalOpened names its runId with a proposalId, a repository and a pullRequestRef and BindChangeProposal handles it, then the binding of that proposal with its repository and pull request reference is recorded for the run, the run keeps its state and version, and no RunControlService/RunStateChanged and no other event is written; a redelivery under the same event id changes nothing (model/services/control-plane/RunControlService.yaml, ADR-0014).
AC2 Given the binding of AC1 and the run already Completed, when PipelineExecutionService/ChangeProposalMerged names the same proposal with its mergedCommit, then the merged commit is recorded on the binding and nothing is published; given instead the merge arrives before the opening, then the binding is written with the merged commit and the later PipelineExecutionService/ChangeProposalOpened completes it with the repository and the pull request reference (model/services/devops-gitops/PipelineExecutionService.yaml).
AC3 Given an event that names a run of tenant B while its tenant is A, or a run that does not exist, or run-control-sql stopped while the event arrives, when PipelineExecutionService/ChangeProposalOpened or PipelineExecutionService/ChangeProposalMerged is handled, then the first two write nothing and complete, and the last is retried by the inbox until the store answers and is applied once under its event id (ADR-0012, ADR-0015).
Traces: services RunControlService, PipelineExecutionService
Traces: aggregates RunControlService/Run
Traces: events PipelineExecutionService/ChangeProposalOpened, PipelineExecutionService/ChangeProposalMerged, RunControlService/RunStateChanged
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-07, XL-10
Depends on AIFACTORY-US-005, feature:pipeline-execution-service-thin, feature:contracts-package
Sources model/services/control-plane/RunControlService.yaml, model/services/devops-gitops/PipelineExecutionService.yaml

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-6501 BindChangeProposal inbox handler of ChangeProposalOpened and ChangeProposalMerged with the proposal bindings of a run implementation The inbox of the run-control profile registers ChangeProposalOpened and ChangeProposalMerged with idempotency on the event id; BindChangeProposal finds the run under the tenant of the event and writes the binding, keyed by tenant, run id and proposal id with the repository, the pull request reference and the merged commit, in the table ChangeProposalBindings of run-control-sql with its index IX_ChangeProposalBindings_Tenant_MergedCommit, its migration and mapping test (D141), upserting so that a merge before the opening and the opening after it leave one complete row; it moves no run state, writes no RecoveryJournal entry and publishes nothing, writes the binding for a run in any state, and writes nothing for a run of another tenant or an unknown run. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6502 Fixture change proposal events and runs of the binding suite fixture The acceptance suite holds fixture events ChangeProposalOpened and ChangeProposalMerged of two tenants, built from the event types of the pinned contracts package under the run event envelope, a Running and a Completed run to bind, a merge that arrives before its opening, and a switch that stops run-control-sql while an event arrives. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6503 Verify bind-a-change-proposal-to-its-run through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Feature=BindChangeProposal"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1606 with the fixture of AIFACTORY-TASK-6502: the binding of an opened proposal without a run transition or event, the merged commit recorded on the binding of a Completed run, the merge before the opening, the foreign and the unknown run, the redelivery and the retry while the store is stopped. ConnectSoft.AI.SoftwareFactory.RunControlService

AIFACTORY-US-1607: Take the source scope a project announces for every run accepted from then on and leave the snapshot of an accepted run unchanged

behavior · persona workspace-admin As a workspace administrator, I want a change of the source scope of my project to apply to the runs accepted after it, so that new runs touch only the sources I allow while a running run keeps the sources it was accepted with.

ProjectService announces through ProjectService/SourceScopeChanged the set of sources a run of a project may touch, with projectId, tenantId and entries, each entry rendered as its kind, repository, documents or tool, and its reference, when a Workspace admin replaces the scope through SetSourceScope (AIFACTORY-US-448 of feature project-service). The inbox handler RefreshSourceScope of RunControlService consumes it, idempotent on the event id, and keeps the announced entries as the last announced source scope of that project in its tenant, replacing the entries kept before; an announcement older than the one kept, by the time of its envelope, changes nothing. From then on SubmitRun and RunControl.SubmitChildRun build the member sources of the resolved execution snapshot of a run they accept (resolved-execution-snapshot-v1) within the entries of that scope, before the snapshot is stored through ArtifactService/Artifacts.Store (AIFACTORY-TASK-2931), so a run accepted after the change touches only the sources of the new scope. A run accepted before the change keeps the snapshot it pinned: the snapshot is stored once, when the run is accepted, and never rewritten, so a Running run, its open attempt and a retried attempt read through RunControl.AcquireLease the reference and the hash of the snapshot that held at its acceptance (ADR-0014). The handler moves no run, writes no entry of the RecoveryJournal and publishes nothing, and, as for the gates, SubmitRun reads the kept scope and calls no other service for it. The scope of one tenant is never read for a run of another tenant (ADR-0015); a redelivered event changes nothing, and when run-control-sql does not answer the handler fails, nothing is written and the inbox retries the event. The announced scope is the entity AnnouncedSourceScope of the Run, kept in the table AnnouncedSourceScopes of run-control-sql, keyed by tenant and project with the entries and the time of the announcement (model/services/control-plane/RunControlService.yaml, D141); how a project for which no scope was announced yet is treated is, as for a subject without a row in GateStates, not stated by the model, and this PBI does not decide it.

Acceptance criteria

AC1 Given a project of tenant A with a kept source scope of two entries, when ProjectService/SourceScopeChanged announces three entries for it and RefreshSourceScope handles it, then the kept scope of the project is the three entries, no RunControlService/Run changes state and nothing is published; given the same event redelivered or an announcement older than the kept one, then nothing changes (model/services/control-plane/RunControlService.yaml).
AC2 Given the changed scope, when RunControlService/SubmitRun or RunControlService/RunControl.SubmitChildRun accepts a run of that project, then the sources of the resolved-execution-snapshot-v1 stored through ArtifactService/Artifacts.Store lie within the three entries; given a run of the project accepted before the change and still Running, then its snapshotRef and snapshotHash are unchanged and RunControlService/RunControl.AcquireLease returns them for a later and for a retried attempt (ADR-0014).
AC3 Given a scope announced for a project of tenant B and a run of tenant A submitted through RunControlService/SubmitRun, or run-control-sql stopped while ProjectService/SourceScopeChanged arrives, when the run is accepted or the event is handled, then the snapshot of the run of tenant A never names an entry of tenant B, and the stopped store leads to a retry by the inbox until it answers and to one application under the event id (ADR-0012, ADR-0015).
Traces: services RunControlService, ProjectService, ArtifactService
Traces: aggregates RunControlService/Run
Traces: apis RunControlService/SubmitRun, RunControlService/RunControl.SubmitChildRun, RunControlService/RunControl.AcquireLease, ArtifactService/Artifacts.Store
Traces: events ProjectService/SourceScopeChanged
Traces: descriptors resolved-execution-snapshot-v1
Traces: adrs ADR-0012, ADR-0014, ADR-0015
Traces: requirements FR-03, XL-03
Depends on AIFACTORY-US-005, feature:project-service, feature:artifact-service
Sources model/services/control-plane/RunControlService.yaml, model/services/control-plane/ProjectService.yaml, docs/reference/schemas/resolved-execution-snapshot.schema.json

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-6504 RefreshSourceScope inbox handler and the sources of the snapshot of a newly accepted run implementation The inbox of the run-control profile registers SourceScopeChanged with idempotency on the event id; RefreshSourceScope keeps the announced entries of a project, keyed by tenant and project with the time of the announcement, in the table AnnouncedSourceScopes of run-control-sql with its migration and mapping test (D141), replaces them only with a newer announcement, moves no run and publishes nothing; the snapshot builder of AIFACTORY-TASK-2931 takes the member sources of resolved-execution-snapshot-v1 within the kept entries of the project of the run for SubmitRun and SubmitChildRun, reads them under the tenant row filter without a call to another service, and never rewrites the snapshot of a run already accepted. ConnectSoft.AI.SoftwareFactory.RunControlService
AIFACTORY-TASK-6505 Verify take-the-announced-source-scope-for-new-runs through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Feature=RefreshSourceScope"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1607 with fixture SourceScopeChanged events of two tenants, the fake Artifacts.Store endpoint of AIFACTORY-TASK-2931 that records the stored snapshot, a run accepted before and one accepted after the change, an older and a redelivered announcement and a switch that stops run-control-sql. ConnectSoft.AI.SoftwareFactory.RunControlService