AIFACTORY-FEAT-012: WorkflowEngineService: complete delivery¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Agent Mesh (PF-02) · Increment I1 Generate a microservice · State elaborated · Route backend-developer · Build order 58 of 185 (Factory code)
WorkflowEngineService is the service of bounded context BC-38 (Workflow Definitions & Engine). Loads Published workflow definitions (DAGs of agent, tool, human, condition, timer, sub-workflow and compensation steps) and drives a workflow run — computing ready steps for Run Control to dispatch, evaluating conditions on recorded outputs, firing durable timers and running compensation — without ever owning run authority (BC-38). Its own repository is ConnectSoft.AI.SoftwareFactory.WorkflowEngineService.
| Delivers | WorkflowEngineService (complete) |
| Stubs | — |
| Realizes | adrs: ADR-0014, ADR-0021 |
| Requirements | FR-02, FR-03, FR-06, FR-10, FUX-04, FUX-05, FUX-11, NFR-05, NFR-06, NFR-11, XL-05, XL-06, XL-09 |
| Depends on | — |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml |
Scenarios to trace (17, catalog): SCN-WorkflowEngineService-001 ✓, SCN-WorkflowEngineService-002 ✓, SCN-WorkflowEngineService-003 ✓, SCN-WorkflowEngineService-004 ✓, SCN-WorkflowEngineService-005 ✓, SCN-WorkflowEngineService-006 ✓, SCN-WorkflowEngineService-007 ✓, SCN-WorkflowEngineService-008 ✓, SCN-WorkflowEngineService-009 ✓, SCN-WorkflowEngineService-010 ✓, SCN-WorkflowEngineService-011 ✓, SCN-WorkflowEngineService-012 ✓, SCN-WorkflowEngineService-013 ✓, SCN-WorkflowEngineService-014 ✓, SCN-WorkflowEngineService-015 ✓, SCN-WorkflowEngineService-016 ✓, SCN-WorkflowEngineService-017 ✓
AIFACTORY-US-006: A workflow run is driven from its pinned definition to one terminal report and survives a restart¶
behavior · persona operator
As an operator, I want a run to be planned step by step from recorded outputs by an engine that holds no run authority, and to continue after a process restart, so that a crash never loses work or runs a step twice.
This is the integration PBI of the complete delivery of WorkflowEngineService, the engine of bounded context BC-38. The engine owns no run authority (ADR-0014): RunControlService accepts the run, creates steps, attempts and leases and records every outcome; the engine keeps a projection of the run in workflow-engine-sql, computes from the pinned definition and the recorded outputs which steps are ready, announces them through WorkflowEngineService/StepReady and reports the terminal result it decided exactly once through WorkflowEngineService/WorkflowRunCompleted, WorkflowEngineService/WorkflowRunFailed or WorkflowEngineService/WorkflowRunCancelled; for a run that Run Control failed itself it writes no WorkflowRunFailed and closes its projection on RunFailed. It dispatches nothing itself. Three rules of decisions D121 and D123 shape the whole. Before the engine announces an agent or tool step it stores the task contract of the step in ArtifactService, once for the step and for no attempt, StepReady carries the reference and the hash of that contract and the step run keeps both (D125); a human step has no task contract and its StepReady carries neither (AIFACTORY-US-746), and a human step that waits on the promotion gate of EnvironmentPromotionService is announced by no StepReady and advances on the decision of that gate (AIFACTORY-US-1496, D133). The engine advances a run on StepSettled alone, and AttemptOutcomeRecorded only updates its projection (AIFACTORY-US-500). And after a failed step Run Control only settles the step: the engine applies the declared failure policy, Fail or Compensate, a step settled Failed under Retry with its attempts spent being failed like one under Fail, and its WorkflowRunFailed alone fails the run in Run Control (AIFACTORY-US-501). The repository ConnectSoft.AI.SoftwareFactory.WorkflowEngineService is scaffolded from the pinned package of the agent host template, the template the model binds the service to, as the host kind workflow-engine (AIFACTORY-US-496); the Factory service profile of the base template reaches the host through the base-template submodule of that package, so the Orleans outbox and inbox, the trusted scope chain, the tenant row filter, token validation, health, readiness and telemetry come from pinned packages and are not written again. Every integration event is the run event envelope of the pinned contracts package, written to the Orleans outbox of the messaging profile workflow-engine in the transaction of the command that changed the aggregate (ADR-0012), so publication is never proof of a transition. The behaviour is delivered by the other PBIs of this feature: the skeleton of aggregates, store, messaging and workers (AIFACTORY-US-497), definitions validated and loaded (AIFACTORY-US-498), a run opened and its ready steps announced (AIFACTORY-US-499), the task contract of each agent and tool step stored before its announcement (AIFACTORY-US-746), the run advanced on settled steps (AIFACTORY-US-500), failed or compensated by the declared failure policy (AIFACTORY-US-501), durable timers and stuck-run recovery (AIFACTORY-US-502), decisions applied and the projection closed on cancellation (AIFACTORY-US-503), the choice of the Operator followed on an escalated or uncertain step and the projection closed when Run Control fails the run (AIFACTORY-US-646), a step of kind SubWorkflow executed as a child run (AIFACTORY-US-647), the run graph read (AIFACTORY-US-504), the members of a readiness bundle satisfied by kind (AIFACTORY-US-505), and the next workflow submitted by a readiness correlation (AIFACTORY-US-009). The scaffold, the hosted surface, the messaging registration and the worker hosting are proven by AIFACTORY-US-496 and AIFACTORY-US-497 and are not proven again here. This PBI proves what only the whole shows, on runs driven against the fake Run Control of AIFACTORY-TASK-2002. One run of a chain definition goes from RunAccepted to a single terminal event, each step reaching its final state when the fake settled it through StepSettled, the one event on which the engine advances, whether the recorded outcome of its attempt arrived before or after (AIFACTORY-US-500). One run of generate-product, the chain definition that holds a bounded loop, a fan-out with its join and a human step, ends with exactly one terminal event. The restart is proven: WorkflowRunGrain serialises Advance for one run and commits step-run rows with the outbox in one transaction; when the process dies between applying the settlement of a step and emitting the StepReady it causes, the re-activated grain re-applies what Run Control recorded idempotently, an outcome on the outcome sequence and a StepSettled on its event id, and re-emits StepReady with the same stepRunId and the task contract reference the step run keeps, and Run Control deduplicates on stepRunId, so no step runs twice and no output is duplicated. And the engine respects the run states that only Run Control holds. A run may be Held while its environment is frozen and WaitingForApproval for a step approval, a spend approval or an operator decision; a StepReady of such a run creates a Pending step that waits for the dispatcher of Run Control, and the engine never moves the run: while no outcome arrives it fails no step run and writes no terminal event, and all its reconciler does is to re-emit StepReady with the same stepRunId (AIFACTORY-US-502). A ChangesRequested decision returns the run to Running in Run Control with the reason of the reviewer. What Run Control does with StepReady is its own delivery: the copy of actionClass and idempotent to sideEffectClass and idempotent of its Step, the dispatch of the attempt and the opening of the approval of a human step are proven by AIFACTORY-US-163 of feature run-control-service, and the same feature proves that the Step keeps the stepRunId and the failurePolicy the event carries, that the reference and the hash of the task contract are kept on the step and copied to each attempt of it, that StepSettled is published once and that the operator decision is opened. This PBI proves the publisher side, the payload the engine writes, and a consumer contract test in the repository of Run Control pins that payload. The route through the Agent Mesh gateway arrives with feature agent-runtime-worker in a later increment; until then the three REST operations are hosted and proven on the service itself. A step of kind SubWorkflow is executed as a child run that the engine submits through the internal operation RunControlService/RunControl.SubmitChildRun, and the result of the child returns to the parent step run with the terminal run events of Run Control (AIFACTORY-US-647); none of the six chain definitions uses the kind, so no run of this suite holds such a step.
Acceptance criteria
| AC1 | Given a seeded Draft release of a chain definition and a RunControlService/RunAccepted that pins it, when the fake Run Control records a Succeeded outcome through RunControlService/AttemptOutcomeRecorded and settles the step through RunControlService/StepSettled for every announced step, then WorkflowEngineService/WorkflowRunStarted is written once, every step is announced exactly once through WorkflowEngineService/StepReady in dependency order and only after RunControlService/StepSettled settled the steps it depends on, each announcement of an agent or tool step with the reference and the hash of a task contract the engine stored before it, and WorkflowEngineService/WorkflowRunCompleted is written exactly once, each in the transaction of the transition it announces (ADR-0014, ADR-0012). |
| AC2 | Given the engine process is killed between applying a RunControlService/StepSettled and emitting the WorkflowEngineService/StepReady it causes, when the grain of the WorkflowEngineService/WorkflowRun is activated again, then the settlement is re-applied idempotently on its event id and a redelivered RunControlService/AttemptOutcomeRecorded on the outcome sequence, StepReady is emitted with the same stepRunId, the task contract reference the step run keeps and the same hash, the run completes with exactly one step run per step and iteration and no step output is duplicated (XL-06, FR-10, docs/planes/agent-runtime/services/workflow-engine-service/index.md). |
| AC3 | Given a run whose announced steps the fake Run Control keeps without an attempt for longer than the step timeout of the definition, as Run Control does for a run that is Held while its environment is frozen or WaitingForApproval for a step approval or a spend approval, when the engine, its timers and its reconciler run, then every ready step was announced once through WorkflowEngineService/StepReady by the Advance, a re-emission carries the same stepRunId and leaves one step in the fake, no step run is Failed, neither WorkflowEngineService/WorkflowRunFailed nor another terminal event is written, and once the fake records the outcomes the run continues to its single terminal event (model/services/control-plane/RunControlService.yaml, ADR-0014). |
| AC4 | Given the seeded Draft release of the chain definition generate-product with the bounded loop of generate-product/check-build, the fan-out generate-product/fan-out-fill with its join generate-product/join-fill and the human step generate-product/approve-proposal, when the fake Run Control records and settles Succeeded outcomes, one build result that makes the condition false and an Approved RunControlService/ApprovalDecided for the human step, then the run returns once to the fan-out with the declared feedback, passes the join and the human step, and ends with exactly one terminal event, WorkflowEngineService/WorkflowRunCompleted, with one step run per step and iteration (ADR-0027, FR-03). |
| AC5 | Given any run driven by the engine in this suite, when the store of the engine and the fake Run Control are inspected, then the engine wrote only its projection in workflow-engine-sql and, in the artifact store, the run variables and the task contract of each announced agent or tool step, called no operation of Run Control other than RunControlService/SubmitRun for a readiness correlation, no run of the suite holding a step of kind SubWorkflow, the only step for which the engine calls RunControlService/RunControl.SubmitChildRun, every WorkflowEngineService/StepReady carries actionClass, idempotent and failurePolicy equal to the values the step declares in the pinned definition, the StepReady of an agent or tool step carries taskContractRef and taskContractHash of a contract the artifact store holds and the StepReady of a human step carries neither, the fake received one step per stepRunId, and exactly one of WorkflowEngineService/WorkflowRunCompleted, WorkflowEngineService/WorkflowRunFailed and WorkflowEngineService/WorkflowRunCancelled exists per workflow run; the copy of these values to the RunControlService/Step is proven by feature run-control-service, by AIFACTORY-US-163 for sideEffectClass and idempotent, and not here (ADR-0014, ADR-0024). |
| Traces: services | WorkflowEngineService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, RunControlService/Step |
| Traces: apis | RunControlService/SubmitRun, RunControlService/RunControl.SubmitChildRun |
| Traces: events | WorkflowEngineService/WorkflowRunStarted, WorkflowEngineService/StepReady, WorkflowEngineService/WorkflowRunCompleted, WorkflowEngineService/WorkflowRunFailed, WorkflowEngineService/WorkflowRunCancelled, RunControlService/RunAccepted, RunControlService/AttemptOutcomeRecorded, RunControlService/StepSettled, RunControlService/ApprovalDecided |
| Traces: workflows | generate-product |
| Traces: steps | generate-product/fan-out-fill, generate-product/join-fill, generate-product/check-build, generate-product/approve-proposal |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0024, ADR-0027 |
| Traces: requirements | FR-03, FR-10, XL-06 |
| Relates: nfrs | NFR-05, NFR-06 |
| Depends on | AIFACTORY-US-496, AIFACTORY-US-497, AIFACTORY-US-498, AIFACTORY-US-499, AIFACTORY-US-500, AIFACTORY-US-501, AIFACTORY-US-502, AIFACTORY-US-503, AIFACTORY-US-504, AIFACTORY-US-505, AIFACTORY-US-009, AIFACTORY-US-646, AIFACTORY-US-647, AIFACTORY-US-746, AIFACTORY-US-1496, feature:run-control-service, feature:agent-registry-service-thin, feature:contracts-package |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, model/workflows/generate-product.yaml, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md, docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/services/workflow-engine-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-198 Run Control side of the engine contract: consumer contract test of StepReady and of the terminal reports |
contract | A consumer contract test in the RunControlService repository pins the event types StepReady, WorkflowRunCompleted and WorkflowRunFailed of the contracts package as the engine publishes them (stepRunId, stepId, kind, actionClass, idempotent, inputRef, failurePolicy and the optional taskContractRef and taskContractHash, present for an agent or tool step and absent for a Human step; outputRef; failedStepId, failureClass and compensated), so a change of a payload fails the build of the consumer; it serves AC5 of AIFACTORY-US-006, the payload from which Run Control copies actionClass, idempotent, stepRunId and failurePolicy to its Step, keeps the reference and the hash of the task contract on the step and copies them to each attempt of it, and it pins WorkflowRunFailed as the report that alone fails a run after a failed step; Run Control stays the single writer of steps, attempts, leases and outcomes, whose behaviour is delivered by AIFACTORY-US-163, AIFACTORY-US-164 and AIFACTORY-US-167 of feature run-control-service and is not built here. | ConnectSoft.AI.SoftwareFactory.RunControlService |
AIFACTORY-TASK-199 Restart test of a workflow run between an applied outcome and its StepReady |
verification | An acceptance test kills the engine process between applying a StepSettled and emitting the StepReady it causes, starts it again and proves AC2 of AIFACTORY-US-006: the settlement is re-applied idempotently on its event id and a redelivered outcome on the outcome sequence, StepReady is emitted with the same stepRunId, the task contract reference the step run keeps and the same hash, the fake Run Control creates no second step, and the run completes with one step run and one output per step and iteration. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-660 Verify durable-steps by running the declared command |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=durable-steps"; a zero-match filter counts as a failure; the tests prove AC1, AC3, AC4 and AC5 of AIFACTORY-US-006 against the fake Run Control of AIFACTORY-TASK-2002, and AIFACTORY-TASK-199 proves AC2: one run of a seeded chain definition to its single terminal report, a run the fake keeps without attempts beyond the step timeout in which nothing fails and no terminal event is written, one run of generate-product through its loop, its fan-out and its human step, and the inspection of the engine store and of what the fake received; the behaviour of each handler is proven by the suites of AIFACTORY-US-498 to AIFACTORY-US-505, AIFACTORY-US-646, AIFACTORY-US-647, AIFACTORY-US-746 and AIFACTORY-US-009, and the hosted surface and the messaging registration by AIFACTORY-US-496 and AIFACTORY-US-497. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-496: WorkflowEngineService host: scaffolded from the pinned agent host template as the workflow-engine silo kind, with its REST and gRPC surface hosted under the Factory scope chain¶
enabling · persona operator
As an operator, I want the engine repository generated from the published agent template package as the workflow-engine host kind, not assembled by hand, so that scope, tenancy, messaging and telemetry behave in the engine as in every other Factory service and a template fix reaches it by a new pin.
The model binds WorkflowEngineService to the agent host template connectsoft-ai-agent (model/services/agent-mesh/WorkflowEngineService.yaml, ADR-0028 amendment on the Factory templates), and feature component-agent-template gave that template the choice agent-host-kind with the value workflow-engine: a host that carries the silo-kind configuration of the one Orleans cluster of the Agent Mesh (ADR-0024, ADR-0028 decision 4), defaults to the Orleans messaging profile and has no agent execution path, exemplar or execution port of its own. The repository ConnectSoft.AI.SoftwareFactory.WorkflowEngineService is created and scaffolded in I0 by AIFACTORY-TASK-037 of feature repository-and-pipeline-conventions (D140), and this PBI starts from that scaffold; the repository checklist of that feature has a row for this service that states the command dotnet new connectsoft-ai-agent -n ConnectSoft.AI.SoftwareFactory.WorkflowEngineService --agent-host-kind workflow-engine, run from the package that feature component-agent-template-release publishes and pinned by version. The command carries the host kind and no Factory profile switch of its own. The agent template composes on the base template through its submodule, so the Factory service profile of the base template and its closed findings (features component-base-template-factory-profile and component-base-template-findings) arrive with the pinned agent template package, as they do for AgentRuntimeService (AIFACTORY-US-460): the Orleans outbox and inbox with the run event envelope, the tenant taken from the token and the project admitted only when the token grants it, token validation with the seven Factory roles mapped to endpoint policies, the REST and gRPC surface described by a service descriptor, health, readiness and telemetry. Nothing of that is forked or written again in the service; a defect found in it is fixed in the template and consumed by a new pin, and the repository records the template version it was generated from. On the scaffold this PBI hosts the surface of the model without its behaviour: the three REST routes with their scopes (factory.assets.write for validation, factory.assets.read for the version list, factory.runs.read for the run graph), problem responses for every modelled refusal code, the two internal gRPC methods of the service WorkflowEngine with the workload identity check, and the OpenAPI and proto documents. A caller without a valid token is answered 401, a caller without the scope 403, and a resource of another tenant 404, over gRPC NOT_FOUND, exactly as an identifier that does not exist (ADR-0015); a missing scope inside the tenant stays 403. Every query runs under the tenant row filter. The secret messaging-transport is read from the key vault by reference, and the eight metrics of the model are registered under the observability conventions. The route through the Agent Mesh gateway belongs to feature agent-runtime-worker of a later increment; until then the surface is reached on the service itself.
Acceptance criteria
| AC1 | Given the repository created and scaffolded in I0 by AIFACTORY-TASK-037 of feature repository-and-pipeline-conventions, whose first commit is the output of dotnet new connectsoft-ai-agent -n ConnectSoft.AI.SoftwareFactory.WorkflowEngineService --agent-host-kind workflow-engine from the agent host template package published by feature component-agent-template-release and pinned by version, when the repository pipeline runs, then the solution builds, the template tests pass with complete reports, a container image is published, the repository records the template version, and the generated settings carry the silo kind of the workflow engine in the Agent Mesh cluster and the messaging service name workflow-engine (ADR-0028, ADR-0024, model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC2 | Given the hosted surface, when the contract test compares it with the service descriptor and the model, then exactly WorkflowEngineService/ValidateWorkflowDefinition with factory.assets.write, WorkflowEngineService/ListWorkflowDefinitionVersions with factory.assets.read, WorkflowEngineService/GetWorkflowRunGraph with factory.runs.read, WorkflowEngineService/WorkflowEngine.PlanNextSteps and WorkflowEngineService/WorkflowEngine.GetRunGraph are hosted with their side-effect class and response codes, and the test fails when an operation outside model/services/agent-mesh/WorkflowEngineService.yaml appears. |
| AC3 | Given a REST call without a valid token, with a token that lacks the scope of the route, or that addresses a run or a definition of another tenant, when it reaches the host, then it is answered 401, 403 or 404 respectively with a problem response, the 404 being the same answer as for an identifier that does not exist; given a call to WorkflowEngineService/WorkflowEngine.GetRunGraph without a workload identity or with a token that lacks the service scope, then it is refused with UNAUTHENTICATED or PERMISSION_DENIED before any handler runs and nothing is read (ADR-0015). |
| AC4 | Given the service starts, when liveness and readiness are probed, then liveness answers and readiness is unready until workflow-engine-sql is migrated; the secret messaging-transport is resolved by reference and never from a settings file, and the eight metrics of model/services/agent-mesh/WorkflowEngineService.yaml are registered with their modelled instruments and dimensions. |
| Traces: services | WorkflowEngineService |
| Traces: apis | WorkflowEngineService/ValidateWorkflowDefinition, WorkflowEngineService/ListWorkflowDefinitionVersions, WorkflowEngineService/GetWorkflowRunGraph, WorkflowEngineService/WorkflowEngine.PlanNextSteps, WorkflowEngineService/WorkflowEngine.GetRunGraph |
| Traces: adrs | ADR-0015, ADR-0024, ADR-0028 |
| Depends on | feature:repository-and-pipeline-conventions, feature:component-agent-template, feature:component-agent-template-factory-access, feature:component-agent-template-release, feature:component-base-template-factory-profile, feature:component-base-template-findings |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/template-parameters/connectsoft-ai-agent.yaml, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md, docs/planes/agent-runtime/services/workflow-engine-service/index.md#security |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-040 API hosting skeleton: REST and gRPC hosts, scope enforcement, problem responses and contract documents for the 3 REST and 2 gRPC operations |
implementation | The three REST routes (ValidateWorkflowDefinition, GetWorkflowRunGraph, ListWorkflowDefinitionVersions) and the two gRPC methods (WorkflowEngine.PlanNextSteps, WorkflowEngine.GetRunGraph) are hosted with token validation, scope enforcement per route, problem responses for every modelled refusal code, the workload identity and service scope checks on the gRPC methods, and the OpenAPI and proto documents; the behaviour of each operation is delivered by AIFACTORY-US-498, AIFACTORY-US-499 and AIFACTORY-US-504, not here. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-043 Security, tenancy and telemetry of WorkflowEngineService |
implementation | The scopes factory.runs.read, factory.assets.read and factory.assets.write and the roles of the model are enforced through the endpoint policies of the profile, every query is tenant-filtered, the secret messaging-transport comes from the key vault by reference, traces and health checks follow the observability conventions, and the eight metrics of model/services/agent-mesh/WorkflowEngineService.yaml are registered with their dimensions, among them the gauge connectsoft.factory.workflows.inbox_lag, the age in seconds of the oldest unprocessed inbox message of the service, measured by the inbox of the messaging profile (D140); each other metric moves in the behaviour PBI that owns it. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1991 Verify the scaffolded host and its hosted surface |
verification | Run dotnet build -c Release and dotnet test -c Release --no-build --filter "Category=Contract\|Category=HostSurface"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-496: the surface equals model/services/agent-mesh/WorkflowEngineService.yaml, the 401, 403 and 404 answers and the gRPC refusals hold before any handler runs, readiness follows the store and the eight metrics are registered. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-497: WorkflowEngineService skeleton: the three aggregates, the nine tables of workflow-engine-sql, the outbox and inbox of the workflow-engine profile and the hosting of five workers and two grains¶
enabling
This PBI lays what every behaviour of the engine stands on and adds no behaviour of its own. The domain model holds the three aggregates of model/services/agent-mesh/WorkflowEngineService.yaml with their invariants: WorkflowDefinitionVersion with its step and edge definitions, immutable once loaded; WorkflowRun with its step runs, timer entries and compensation entries and the seventeen transitions of WorkflowRunState (Pending to Running on RunAccepted, Running to WaitingForTimer and back, Running to WaitingForHuman and back, Running to Compensating both for a step that failed under the policy Compensate and for the choice compensate of an operator decision, Compensating to Failed, Running to Completed, Running to Failed both for a step that failed under the policy Fail and, through StepFailedWithRetryPolicyAttemptsSpent, for a step settled Failed under the policy Retry with its attempts spent (decision D123), Running, WaitingForTimer and WaitingForHuman to Cancelled on RunCancelled, and the same three states to Failed on RunFailed); a step run keeps in childRunId the child run of a step of kind SubWorkflow, in attemptCount and failureClass the number of attempts and the failure class that StepSettled carries for it (decision D121) and in taskContractRef the reference of the task contract the engine stored for the step, empty for a step without one, such as a human step (decision D123); and ReadinessCorrelation with its members and the five transitions of ReadinessState (Open to Satisfied, Satisfied to Submitted, Open and Satisfied to Superseded, Open to Expired). A transition the model does not list is refused by the aggregate. The store workflow-engine-sql gets the nine tables of the model with a tenant column on every row, the tenant row filter of the profile and the modelled indexes, among them the unique UX_ReadinessCorrelations_Key on tenant and idempotency key, the unique UX_StepRuns_Run_Step_Iteration and IX_Timers_Due. The messaging skeleton wires the Orleans outbox and inbox of the messaging profile whose service name is workflow-engine, which belong to this service alone (ADR-0028): the outbox writer enlists in the transaction of the command, the dispatcher publishes the twelve events of the model under the run event envelope of the pinned contracts package, and the event types and topic constants are taken from that package and declared nowhere in the service. The model lists twenty-four consume rows over twenty-one event types, of which ApprovalDecided, RunFailed and RunCancelled are each handled by two handlers. The inbox of this feature registers twenty-three of them, idempotently on the event id, among them the rows by which the engine follows Run Control: StepSettled for the final state of a step, RunFailed and RunCancelled to close the projection, and RunCompleted, RunFailed and RunCancelled to return the result of a child run. The engine pins the contracts package at a version generated from the model that carries StepSettled with its stepId and outputRef, failurePolicy and the optional taskContractRef and taskContractHash of StepReady, choice and stepRunId of ApprovalDecided and the optional outputRef of RunCompleted (decisions D119, D121 and D123). The twenty-fourth row is not registered: the handler ExecuteDeletionOrder of DeletionOrderIssued is built and proven in I4 together with RetentionResidencyService, and this feature registers no handler for deletion orders. Some producers arrive later than this service: AgentRegistryService publishes AssetReleasePublished only with its complete delivery, and ReleaseService is delivered in a later increment. Their handlers are registered now and exercised on fixture events built from the contracts package. The five workers of the model (ReadinessCorrelator, ReadinessExpiryReconciler, TimerScheduler, StuckRunReconciler, DefinitionCacheWarmer) run inside the deployable of the service (ADR-0028) and are not written as five private loops. The host registers the worker engine ConnectSoft.Workers.Runtime at the version feature component-workers-runtime-library published, the way AgentRuntimeService hosts its jobs (AIFACTORY-US-462), with drain wired to the stop of the host, the idempotency and failed-work stores on workflow-engine-sql and the singleton lease, so that two replicas never run the same job instance. The tables of the engine are created by a migration of this repository from the schema script of the engine in its scoped shape, in which a processed-message row and a failed-work row carry tenant and project, with the run where the work has one, as the Factory profile of the worker template defines it (feature component-worker-template-factory-profile); they stand beside the nine tables of the model. Each worker is registered with the trigger, the retry policy, the idempotency key and the partitioning the model states. ReadinessCorrelator runs on the signal events of its modelled trigger with five exponential attempts from 30 seconds, the key correlationId plus slot plus refHash for members and the idempotency key of the correlation for the submission, and two parallel runs partitioned by project. ReadinessExpiryReconciler runs every six hours with three exponential attempts from 60 seconds, the key correlationId plus version and one run at a time. TimerScheduler runs on the reminder of a timer entry with five exponential attempts from 10 seconds, the key timerId and one run at a time per run. StuckRunReconciler runs every five minutes with three fixed attempts 60 seconds apart, the key stepRunId plus lastOutcomeSequence and two parallel runs partitioned by tenant. DefinitionCacheWarmer runs on AssetReleasePublished with five exponential attempts from 30 seconds, the key releaseId and two parallel runs partitioned by tenant. A trigger repeated under the same key has no second effect, and work that exhausts its modelled retries leaves one open failed-work record with its scope and no partial write. Run state never moves into the worker engine: it lives in the two grains (WorkflowRunGrain keyed by run id, ReadinessCorrelationGrain keyed by tenant, project, operation, workflow release and bundle fingerprint, both not reentrant), and their reminders, timer and stuck-detection of the first and expiry of the second, are grain reminders (ADR-0024) and no periodic jobs of the worker engine; the reminder of a timer entry hands the fired timer to TimerScheduler. The eight options of the model are bound with their defaults. The model lists no operation of WorkflowEngineService that serves the job catalog or the failed work, so this delivery exposes none; which surface an operator uses to list and replay the failed work of this service is reported to the owner. What each handler, worker and grain does is delivered by AIFACTORY-US-498 to AIFACTORY-US-505, AIFACTORY-US-646, AIFACTORY-US-647, AIFACTORY-US-746 and AIFACTORY-US-009.
Acceptance criteria
| AC1 | Given the aggregates WorkflowEngineService/WorkflowRun, WorkflowEngineService/ReadinessCorrelation and WorkflowEngineService/WorkflowDefinitionVersion, when the unit suite runs, then every one of the seventeen transitions of WorkflowRunState and the five of ReadinessState listed in model/services/agent-mesh/WorkflowEngineService.yaml is accepted, every pair of states the model does not list is refused, a loaded definition version cannot be changed and each modelled invariant has a test. |
| AC2 | Given an empty isolated test database, when the migrations of workflow-engine-sql run, then the nine tables of the model exist with their tenant column, keys and indexes as docs/planes/agent-runtime/services/workflow-engine-service/ddl.md lists them, the mapping tests round-trip every aggregate under the tenant row filter, a row of another tenant is never read (ADR-0015), and no shared database is used. |
| AC3 | Given a command that changes an aggregate of WorkflowEngineService, when it commits, then the state change and its outbox record are written in one transaction and a command that rolls back leaves neither; when the dispatcher of the workflow-engine profile publishes the record, then it is sent under run-event-envelope-v1 with the event type and the topic constant of the pinned contracts package, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (ADR-0012, ADR-0014). |
| AC4 | Given each of the twenty-three consume rows of model/services/agent-mesh/WorkflowEngineService.yaml that this feature registers, every row but DeletionOrderIssued, when the same event id is delivered twice, then each handler of the event runs once; the handlers of events whose producers arrive in later increments are registered and proven on fixture events of the pinned contracts package, without a dependency on those producers; and no handler is registered for DeletionOrderIssued, whose handler is built and proven in I4 with RetentionResidencyService. |
| AC5 | Given the five workers and the two grains of model/services/agent-mesh/WorkflowEngineService.yaml and the worker engine registered in the host, when the host starts on an isolated database, then the tables of the worker engine exist in workflow-engine-sql in the scoped shape, each worker is in the catalog with its modelled trigger, retry policy, partition key and idempotency key, a duplicate trigger has no second effect, work that exhausts its modelled retries leaves one open failed-work record that carries its tenant and project and no partial write, both grains are not reentrant and register their reminders, and the eight options are bound with their modelled defaults (ADR-0024, ADR-0028). |
| Traces: services | WorkflowEngineService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, WorkflowEngineService/ReadinessCorrelation, WorkflowEngineService/WorkflowDefinitionVersion |
| Traces: descriptors | run-event-envelope-v1 |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015, ADR-0024, ADR-0028 |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-496, feature:contracts-package, feature:component-workers-runtime-library, feature:component-worker-template-factory-profile |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, docs/planes/agent-runtime/services/workflow-engine-service/ddl.md, docs/planes/agent-runtime/services/workflow-engine-service/index.md#domain-model, docs/planes/agent-runtime/services/workflow-engine-service/index.md#workers-and-jobs, docs/planes/agent-runtime/workers.md, docs/architecture/adr/0012-messaging-port-and-transport-profiles.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-038 Domain model: ReadinessCorrelation, WorkflowDefinitionVersion, WorkflowRun |
implementation | The three aggregates with their entities, invariants and state transitions as modelled; unit tests cover every invariant, each of the seventeen transitions of WorkflowRunState and the five of ReadinessState, and the refusal of every transition the model does not list. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-039 Persistence in workflow-engine-sql: ReadinessCorrelations, ReadinessMembers, WorkflowDefinitions, StepDefinitions, EdgeDefinitions, WorkflowRuns, StepRuns, Timers and CompensationEntries |
migration | Mappings and migrations for the nine modelled tables with tenant filters and the modelled indexes, the unique UX_ReadinessCorrelations_Key and UX_StepRuns_Run_Step_Iteration among them, and with the nullable ChildRunId, the AttemptCount with the default 0, the nullable FailureClass, the nullable TaskContractRef, TaskContractHash and FailureReason of StepRuns (D125), and the nullable PolicyGuardJson and WaitsOn of StepDefinitions (D133); mapping tests run against an isolated test database, never a shared one. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-041 Messaging skeleton: outbox and inbox wiring of the workflow-engine profile for 12 published events and 23 of the 24 consume rows |
implementation | The outbox writer enlists in the transaction of the command and the dispatcher of the workflow-engine messaging profile publishes the 12 modelled events (WorkflowDefinitionLoaded, WorkflowRunStarted, StepReady, StepRunStateChanged, TimerFired, CompensationStarted, WorkflowRunCompleted, WorkflowRunFailed, WorkflowRunCancelled, ReadinessSatisfied, ReadinessRunSubmitted, ReadinessSuperseded) under the run event envelope with the event types and topic constants of the pinned contracts package; the inbox registers 23 of the 24 consume rows of the model with idempotency on the event id, the rows StepSettled, RunFailed, RunCompleted and the second handler of RunCancelled among them, at a contracts package version that carries StepSettled with stepId and outputRef, failurePolicy and the optional taskContractRef and taskContractHash of StepReady, choice and stepRunId of ApprovalDecided and outputRef of RunCompleted; no handler is registered for DeletionOrderIssued, whose handler ExecuteDeletionOrder is built and proven in I4 with RetentionResidencyService; what each handler does is delivered by the behaviour PBIs of this feature. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-042 Worker and grain hosting skeleton: the five workers of the model on the Workers.Runtime engine, WorkflowRunGrain and ReadinessCorrelationGrain |
implementation | The host registers the worker engine ConnectSoft.Workers.Runtime at the pinned version with drain, the singleton lease and the idempotency and failed-work stores on workflow-engine-sql, whose engine tables a migration creates in the scoped shape with tenant and project; the five workers are in its catalog with their modelled triggers, retry policies, partitioning and idempotency keys, a duplicate trigger has no second effect, and work that exhausts its retries leaves one open failed-work record with its scope and no partial write; the two grains are hosted with their keys, their reminders as grain reminders and reentrancy off; the eight options of the model are bound with their defaults; what each worker and grain does is delivered by AIFACTORY-US-498, AIFACTORY-US-500, AIFACTORY-US-502, AIFACTORY-US-505 and AIFACTORY-US-009. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1992 Verify the domain, store, messaging and worker skeleton |
verification | Run dotnet test -c Release --filter "Category=Unit\|Category=Mapping\|Category=Messaging\|Category=Workers"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-497 against model/services/agent-mesh/WorkflowEngineService.yaml: every transition and its refusal, the nine tables under the tenant filter on an isolated database, the outbox record in the transaction of its command, the inbox idempotent on the event id for the twenty-three registered rows and without a handler for DeletionOrderIssued, each worker in the catalog of the worker engine with its trigger and idempotency key, and the case of a worker whose retries are exhausted, which leaves one failed-work record with tenant and project and no partial write. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-498: Validate a workflow definition, refuse one without declared failure policies, and load a release by alias into the engine¶
behavior · persona asset-maintainer
As an asset maintainer, I want a workflow definition checked by the engine that will execute it, before any run can pin it, so that no run starts on a graph the engine cannot drive or on a step whose failure handling nobody declared.
A caller whose scope carries factory.assets.write posts a workflow definition to WorkflowEngineService/ValidateWorkflowDefinition (POST /runtime/workflow-definitions/validate, class sandbox). The request schema the model names is asset-manifests-v1: the body is the WorkflowDefinition of docs/reference/schemas/asset-manifests.schema.json, the wire shape that the descriptor workflow-definition-v1 stands for and that the contracts package carries. A step of it has the fields id, kind, agent, skill, tool, gate, sideEffectClass, dependsOn, retry, compensation, timeout, when and onFalse with to, maxIterations and feedback. The engine checks the document and writes nothing. The graph of dependsOn must be acyclic except for loops declared with a bounded iteration count, an onFalse branch bounded by its maxIterations, or by Workflows:Engine:DefaultMaxLoopIterations when the definition declares none. Every agent, tool and human step must declare its action class in sideEffectClass and its failure policy in failurePolicy, one of Retry, Compensate, Escalate and Fail, and the engine refuses a definition without it. Retry is accepted only on a step whose sideEffectClass is read or sandbox, or shared-write when the step declares idempotencyKey. A human step is an approval gate and declares its approval binding, a condition reads only recorded step outputs and run variables, the schema references of inputs and outputs resolve, and a fan-out stays within Workflows:Engine:MaxFanout. A step of kind sub-workflow is executed as a child run and must name its child workflow; that rule and the wire property workflow it reads are added to this validator with the execution of the kind (AIFACTORY-US-647). The rule on failure policies is a decision of the owner, and its source is the model file schema model/schemas/workflow.schema.json, in which the fields are failure_policy, idempotency_key and approval and which the definitions under model/workflows follow. The published wire schema does not carry them yet: a step of WorkflowDefinition declares neither a failure policy nor an idempotency key nor the approval binding nor the expression of a condition, and the activation block knows no member of kind signal. The contract task of this PBI adds them to docs/reference/schemas/asset-manifests.schema.json in the camel case of the wire (failurePolicy, idempotencyKey, approval and condition on a step; the member kind signal with ref, versionRef and hashRef in the activation block), the contracts package regenerates its types from the published schema (feature contracts-package), and the service takes them by a new pin. The validator reads the wire fields and nothing else; the model file schema stays the source of the rule and of the fixtures. A valid document is answered 200; a body that is not a WorkflowDefinition of that schema or that fails one of these rules is answered 400 with a problem response that names each finding by step and rule; a caller without the scope is answered 403. The engine copies the declared failure policy as it stands and derives none. The same validation runs when a release is loaded. The command LoadWorkflowDefinition streams the release by its id through AgentRegistryService/AssetPublication.LoadRelease, fetches the definition content its manifest release-manifest-v1 references through ArtifactService/Artifacts.Fetch, validates the graph, has every declared action class decided through PolicyEngineService/Policy.Decide, and writes the WorkflowDefinitionVersion with its StepDefinition and EdgeDefinition rows, graphHash, stepCount, schema references and maxLoopIterations together with the WorkflowEngineService/WorkflowDefinitionLoaded outbox record in one transaction. An action class that policy denies, a decision with the effect Deny or Prohibited, rejects the release like a validation failure. The model says that declared action classes are validated against policy at load and names no method; Policy.Decide is the one of the three gRPC methods of PolicyEngineService that evaluates an action, and that reading is reported to the owner. Loading is idempotent on releaseId: a version is immutable once loaded, and a changed graph has a new release id. In this increment AgentRegistryService is delivered thin: it serves seeded Draft releases and publishes no event, so AssetReleasePublished, the trigger of the DefinitionCacheWarmer, does not arrive before feature agent-registry-service-publication in I2 (D140). Until then a release is loaded by alias and on first use: the warmer resolves the alias of a workflow definition to an immutable release id through AgentRegistryService/AssetPublication.ResolveRelease and loads it, and StartWorkflowRun loads the release a RunAccepted pins when it is not cached. The alias of a definition is the one its activation block declares in submit.release_alias (submit.releaseAlias on the wire, latest in the chain definitions that declare an activation block), and the option Workflows:Readiness:StarterReleaseAlias is the value used when the block declares none. Which definitions the warmer resolves at start, so that a readiness signal finds the activation block of a definition no run has pinned yet, is an input the model does not state while the registry publishes no event and offers no list operation, and before its first load the warmer holds no activation block to read an alias from; the acceptance suite proves the path on a fixture list of the six chain definitions with the option value, and the point is reported to the owner. The manifest of a seeded release states Draft and the engine applies no waiver of its own: whether a Draft release may run is decided by the policy rule of the ADR-0021 amendment where policy is evaluated for the submission. A release that fails validation, whose manifest hash cannot be verified, or one of whose action classes policy denies is not loaded: no WorkflowDefinitionLoaded is written, connectsoft.factory.workflows.definition_rejections is incremented with the reason, and a run that pins that release fails at acceptance with an explicit reason, as the failure modes of the model state (AIFACTORY-US-501). AgentRegistryService, PolicyEngineService and ArtifactService are fail-fast dependencies: when one is unavailable the load fails, nothing is written and the worker retries with its exponential policy of five attempts. A caller whose scope carries factory.assets.read lists the loaded versions of a definition through WorkflowEngineService/ListWorkflowDefinitionVersions with version, releaseId, graphHash and loadedAt; a definition with no loaded version the tenant of the caller can read is answered 404, the same answer as for a definition of another tenant (ADR-0015), and a missing scope 403.
Acceptance criteria
| AC1 | Given a caller with factory.assets.write and a WorkflowDefinition document of docs/reference/schemas/asset-manifests.schema.json, the shape of workflow-definition-v1, whose dependsOn graph is acyclic except for loops bounded by onFalse.maxIterations, whose every agent, tool and human step declares sideEffectClass and failurePolicy, whose Retry steps are of class read or sandbox or of class shared-write with idempotencyKey, whose human steps declare their approval binding and whose schema references resolve, when WorkflowEngineService/ValidateWorkflowDefinition is called, then the response is 200, the declared failure policies are taken as they stand and nothing is written or published (SCN-WorkflowEngineService-001). |
| AC2 | Given a body that is not a WorkflowDefinition of docs/reference/schemas/asset-manifests.schema.json, or a document with a cycle outside a bounded loop, an agent, tool or human step without sideEffectClass or failurePolicy, Retry on a shared-write step without idempotencyKey or on a step of a class above shared-write, a human step without approval, an unresolved schema reference or a fan-out above Workflows:Engine:MaxFanout, or a caller without factory.assets.write, when WorkflowEngineService/ValidateWorkflowDefinition is called, then the response is 400 with a problem response that names each finding by step and rule, or 403 for the missing scope, connectsoft.factory.workflows.definition_rejections counts the rejected definition with its reason, and no state changes (SCN-WorkflowEngineService-001). |
| AC3 | Given the seeded Draft release of a workflow definition, when LoadWorkflowDefinition runs for the release id that AgentRegistryService/AssetPublication.ResolveRelease returned for its alias, or that an accepted run pins, then the release is streamed through AgentRegistryService/AssetPublication.LoadRelease, the content its manifest release-manifest-v1 references is fetched through ArtifactService/Artifacts.Fetch, the graph passes the validation of AC1, every declared action class is decided through PolicyEngineService/Policy.Decide, and the WorkflowEngineService/WorkflowDefinitionVersion with its step and edge rows and the WorkflowEngineService/WorkflowDefinitionLoaded outbox record are written in one transaction; a second load of the same releaseId changes nothing (SCN-WorkflowEngineService-006, FR-02). |
| AC4 | Given a version was loaded, when the outbox dispatcher publishes WorkflowEngineService/WorkflowDefinitionLoaded on workflows.definitions.v1.workflow-definition-loaded, then the event is sent under the canonical envelope with releaseId, tenantId, definitionId, version and graphHash, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id; given a release that fails load validation, whose manifest hash cannot be verified, or for one of whose action classes PolicyEngineService/Policy.Decide answers with the effect Deny or Prohibited, then no version and no event is written and connectsoft.factory.workflows.definition_rejections counts the rejection with its reason; given AgentRegistryService, PolicyEngineService or ArtifactService unavailable, then the load fails, nothing is written and the worker retries (SCN-WorkflowEngineService-006, ADR-0012, ADR-0021). |
| AC5 | Given a caller with factory.assets.read and a definition with loaded versions the tenant of the caller can read, when WorkflowEngineService/ListWorkflowDefinitionVersions is called, then the response is 200 with every loaded version and its version, releaseId, graphHash and loadedAt; a caller without the scope is answered 403, and a definitionId that is unknown or loaded only for another tenant is answered 404, the same answer in both cases, with a problem response and no state change (SCN-WorkflowEngineService-003, ADR-0015). |
| Traces: services | WorkflowEngineService, AgentRegistryService, PolicyEngineService, ArtifactService |
| Traces: aggregates | WorkflowEngineService/WorkflowDefinitionVersion |
| Traces: apis | WorkflowEngineService/ValidateWorkflowDefinition, WorkflowEngineService/ListWorkflowDefinitionVersions, AgentRegistryService/AssetPublication.ResolveRelease, AgentRegistryService/AssetPublication.LoadRelease, ArtifactService/Artifacts.Fetch, PolicyEngineService/Policy.Decide |
| Traces: events | WorkflowEngineService/WorkflowDefinitionLoaded |
| Traces: descriptors | workflow-definition-v1, release-manifest-v1 |
| Traces: adrs | ADR-0012, ADR-0015, ADR-0021, ADR-0024 |
| Traces: requirements | FR-02 |
| Traces: scenarios | SCN-WorkflowEngineService-001, SCN-WorkflowEngineService-003, SCN-WorkflowEngineService-006 |
| Relates: screens | workflow-definitions |
| Depends on | AIFACTORY-US-496, AIFACTORY-US-497, feature:agent-registry-service-thin, feature:policy-engine-service, feature:artifact-service, feature:contracts-package |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/schemas/workflow.schema.json, docs/reference/schemas/asset-manifests.schema.json, docs/architecture/adr/0021-quality-gates.md, docs/planes/agent-runtime/services/workflow-engine-service/index.md#apis |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2019 Add the failure policy, the idempotency key, the approval binding, the condition and the signal member to WorkflowDefinition of the published asset manifest schema |
contract | docs/reference/schemas/asset-manifests.schema.json declares on a step of WorkflowDefinition the properties failurePolicy (Retry, Compensate, Escalate, Fail), idempotencyKey, condition and approval with actionClass, proposalKind, proposalFrom, approverRule and staleOn, and in the activation block the member kind signal with ref, versionRef and hashRef, each mirroring the property of the same meaning in model/schemas/workflow.schema.json; the six chain definitions of model/workflows can be written as documents that are valid against it, the reference page of the schema shows the new properties, and python -I tools/docs_model validate passes. |
ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-2020 Publish the contracts package version that carries the extended WorkflowDefinition types |
contract | The contracts package regenerates its WorkflowDefinition types from docs/reference/schemas/asset-manifests.schema.json as AIFACTORY-TASK-2019 changed it, stamps them with the hash of the schema and is published to the ConnectSoft feed at a new version, the way feature contracts-package builds and publishes it; the repository ConnectSoft.AI.SoftwareFactory.WorkflowEngineService pins that version. |
ConnectSoft.AI.SoftwareFactory.Contracts |
AIFACTORY-TASK-1993 Definition validator and the ValidateWorkflowDefinition and ListWorkflowDefinitionVersions operations |
implementation | One validator, built on the WorkflowDefinition types of the contracts package version of AIFACTORY-TASK-2020, checks acyclicity of dependsOn with loops bounded by onFalse.maxIterations, the declared sideEffectClass and failurePolicy of every agent, tool and human step (a definition without them is refused and no policy is derived), Retry only on read or sandbox steps or on shared-write steps with idempotencyKey, the approval binding of human steps, conditions over recorded outputs only, schema references and the fan-out limit, with one unit test per rule; ValidateWorkflowDefinition answers 200, 400 with the findings for a body that is not a WorkflowDefinition of docs/reference/schemas/asset-manifests.schema.json or that breaks a rule, or 403, and writes nothing; ListWorkflowDefinitionVersions returns the loaded versions with version, releaseId, graphHash and loadedAt under the tenant row filter, 403 without the scope and 404 for an unknown or foreign definition. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1994 LoadWorkflowDefinition and the DefinitionCacheWarmer with alias resolution and loading on first use |
implementation | LoadWorkflowDefinition streams a release through AssetPublication.LoadRelease, fetches its content through Artifacts.Fetch, runs the validator of AIFACTORY-TASK-1993, has every declared action class decided through Policy.Decide and writes the WorkflowDefinitionVersion with its step and edge rows and the WorkflowDefinitionLoaded outbox record in one transaction, idempotent on releaseId; the DefinitionCacheWarmer handles AssetReleasePublished of kind Workflow and, while the registry publishes no event, resolves at start the alias of each definition it is given through AssetPublication.ResolveRelease, the alias being the one the activation block declares and the option StarterReleaseAlias when it declares none or is not loaded yet; a release that fails validation, whose manifest hash cannot be verified or whose action class is answered Deny or Prohibited writes nothing and increments definition_rejections with the reason; an unavailable dependency fails the load and is retried with the exponential policy of five attempts. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1995 Acceptance fixture of a fake registry with the six chain definitions and one invalid variant per validation rule |
fixture | A test double of AgentRegistryService serves the six chain definitions of model/workflows, written as WorkflowDefinition documents of docs/reference/schemas/asset-manifests.schema.json, as seeded Draft releases with their aliases through ResolveRelease and LoadRelease; a set of invalid documents holds one variant per rule of the validator (a cycle, a missing failurePolicy, a missing sideEffectClass, Retry on a shared-write step without idempotencyKey, a human step without approval, an unresolved schema reference, a fan-out above the limit, a body that is no WorkflowDefinition); a fake policy engine answers Allow or a scripted Deny and a fake artifact store serves the content, so the scenarios run without the registry, the policy engine and the artifact store. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1996 Verify validate-and-load-workflow-definitions through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-WorkflowEngineService-001\|Scenario=SCN-WorkflowEngineService-003\|Scenario=SCN-WorkflowEngineService-006)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-498, including the refusal of a definition without failurePolicy, the load by alias and on first use, the idempotent second load, the release rejected by validation or by a denied action class that writes no event, and the publication and payload of WorkflowDefinitionLoaded; consumer handling is proven by the consumers' own suites. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-499: Open the projection of an accepted run and announce its ready steps with the declared action class and idempotency, without dispatching anything¶
behavior
When RunControlService accepts a run it publishes RunControlService/RunAccepted with runId, tenantId, projectId, workflowId, releaseId, policyRevision, modelProfileId and requestedBy. The inbox handler StartWorkflowRun of the engine opens the projection of that run, idempotently on the event id and on runId: it takes the WorkflowDefinitionVersion of the pinned releaseId from its store, loading the release on first use (AIFACTORY-US-498), creates the WorkflowRun with the same runId, the definitionReleaseId and the graphHash in state Pending, computes the first ready steps, moves the run from Pending to Running through the modelled trigger RunAccepted, and writes WorkflowEngineService/WorkflowRunStarted and one WorkflowEngineService/StepReady per ready step in the same transaction. A step is ready when every step it depends on is Succeeded or Skipped and its when guard holds over the workflow inputs and the recorded outputs; a step whose guard does not hold is recorded Skipped and its dependants treat it as satisfied (ADR-0027). For each ready step the engine creates a StepRun, and StepReady carries runId, tenantId, stepRunId, stepId, kind, actionClass, idempotent, inputRef, failurePolicy, taskContractRef and taskContractHash: actionClass is the side-effect class the step declares in the pinned definition, idempotent is true when that step declares an idempotency key, inputRef refers to the input of the step built from its input mapping, and failurePolicy is the failure policy the step declares, which Run Control keeps on its step and reads when the step fails terminally, so that a step under Escalate opens an operator decision. The engine copies the three values from the definition and derives none. taskContractRef and taskContractHash name the task contract of an agent or tool step, which the engine stored in ArtifactService before the announcement; no agent or tool step is announced without them, and how the contract is built and stored is delivered by AIFACTORY-US-746 (decision D121). Both members are optional on StepReady: a human step has no task contract, so its StepReady carries neither (decision D123). The three copied values hold for a human step as well: its actionClass is the side-effect class the step declares, and the action class of its approval binding is read by Run Control for the approval request. A step of kind SubWorkflow is announced by no StepReady, because the engine submits its child run itself (AIFACTORY-US-647). The engine dispatches nothing: RunControlService consumes StepReady in its handler CreateStepAndDispatch, creates the Step idempotently on stepRunId, copies actionClass to sideEffectClass and idempotent to idempotent of the Step, keeps stepRunId and failurePolicy on it, keeps the reference and the hash of the task contract on the step and copies them to each attempt of it, and dispatches the attempt of an Agent or Tool step or opens the approval of a Human step, in its own PBIs. The gates of dispatch are also Run Control's: its dispatcher releases the first attempt once the budget is reserved, a run that waits for a spend approval has no reservation yet, and a Held run gets no attempt, so a StepReady of such a run creates a Pending step that waits. The engine consumes no budget event and never moves a run: the trigger of its own transition is RunAccepted alone. An internal caller may also ask instead of listening, and RunControlService does so when a run is accepted, as a fail-fast dependency of its own: WorkflowEngineService/WorkflowEngine.PlanNextSteps takes the run and the last applied outcome sequence and returns the steps that are ready to dispatch with their input mappings and action classes, in the reply the model describes by agent-task-contract-v1. The method is of class read and idempotent: it computes from the committed projection, writes nothing and returns the same steps for the same sequence. The caller presents a workload token, and the engine reads the tenant and project the caller forwards only when the token holds the service scope that names the engine (ADR-0015). A call without a workload identity is refused with UNAUTHENTICATED, a token without the service scope with PERMISSION_DENIED, an invalid request with INVALID_ARGUMENT, and a run that does not exist in the forwarded tenant, a run of another tenant included, with NOT_FOUND. When the pinned release cannot be loaded because AgentRegistryService is unavailable, the handler fails, no WorkflowRun is written and the inbox redelivers RunAccepted.
Acceptance criteria
| AC1 | Given a loaded WorkflowEngineService/WorkflowDefinitionVersion and a RunControlService/RunAccepted that pins its releaseId, when the handler StartWorkflowRun runs, then a WorkflowEngineService/WorkflowRun with the same runId, the definitionReleaseId and the graphHash exists in state Running, a step run exists for every step whose dependencies are satisfied, and WorkflowEngineService/WorkflowRunStarted and one WorkflowEngineService/StepReady per ready step are written in the same transaction; a redelivery of the same event id, or a second RunAccepted for the same runId, changes nothing (SCN-WorkflowEngineService-007, ADR-0014). |
| AC2 | Given the run was opened, when the outbox dispatcher publishes WorkflowEngineService/WorkflowRunStarted on workflows.runs.v1.workflow-run-started and WorkflowEngineService/StepReady on workflows.runs.v1.step-ready, then both are sent under the canonical envelope, the first with runId, tenantId, definitionReleaseId and graphHash, the second with runId, tenantId, stepRunId, stepId, kind, actionClass, idempotent, inputRef, failurePolicy and, for an agent or tool step, taskContractRef and taskContractHash, where actionClass equals the side-effect class the step declares, idempotent is true exactly when the step declares an idempotency key, idempotencyKey of its step in the loaded definition, failurePolicy equals the failure policy the step declares, taskContractRef and taskContractHash name the task contract stored before the announcement and are absent on the StepReady of a human step, and a transport outage is retried with backoff so neither event is lost or duplicated under its event id (SCN-WorkflowEngineService-007, SCN-WorkflowEngineService-008, ADR-0012). |
| AC3 | Given WorkflowEngineService/StepReady was published for an Agent, a Tool and a Human step, when the stores are inspected, then the engine created no step, attempt, lease or approval of Run Control, the payload is the one from which CreateStepAndDispatch creates the RunControlService/Step with sideEffectClass, idempotent, stepRunId and failurePolicy copied, and a step whose when guard does not hold was announced by no StepReady but recorded Skipped with WorkflowEngineService/StepRunStateChanged, its dependants treating it as satisfied (SCN-WorkflowEngineService-008, ADR-0027, model/services/control-plane/RunControlService.yaml). |
| AC4 | Given a caller with a workload identity whose token holds the service scope and a run of the tenant it forwards, when WorkflowEngineService/WorkflowEngine.PlanNextSteps is called with the run and the last applied outcome sequence, then the reply names the steps that are ready to dispatch with their input mappings and action classes as agent-task-contract-v1 describes them, nothing is written, and a repeated call with the same sequence returns the same steps (SCN-WorkflowEngineService-004). |
| AC5 | Given a caller without a workload identity, a token without the service scope, a request that names no run, or a run that is unknown or belongs to another tenant, when WorkflowEngineService/WorkflowEngine.PlanNextSteps is called, then the call is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT or NOT_FOUND respectively, the foreign run answered exactly as the unknown one, and no state changes; given AgentRegistryService is unavailable when StartWorkflowRun must load the pinned release through AgentRegistryService/AssetPublication.LoadRelease, then the handler fails, no workflow run is written and the inbox redelivers the event (SCN-WorkflowEngineService-004, SCN-WorkflowEngineService-007, ADR-0015). |
| Traces: services | WorkflowEngineService, RunControlService, AgentRegistryService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, WorkflowEngineService/WorkflowDefinitionVersion, RunControlService/Step |
| Traces: apis | WorkflowEngineService/WorkflowEngine.PlanNextSteps, AgentRegistryService/AssetPublication.LoadRelease |
| Traces: events | WorkflowEngineService/WorkflowRunStarted, WorkflowEngineService/StepReady, WorkflowEngineService/StepRunStateChanged, RunControlService/RunAccepted |
| Traces: descriptors | agent-task-contract-v1, run-event-envelope-v1 |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015, ADR-0024, ADR-0027 |
| Traces: requirements | FR-03, FUX-04 |
| Traces: scenarios | SCN-WorkflowEngineService-004, SCN-WorkflowEngineService-007, SCN-WorkflowEngineService-008 |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-496, AIFACTORY-US-497, AIFACTORY-US-498, feature:run-control-service, feature:agent-registry-service-thin |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, docs/reference/schemas/run-event-envelope.schema.json, docs/planes/agent-runtime/services/workflow-engine-service/index.md#messages |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-197 Plan next steps: dependencies, conditions, when guards, bounded loops |
implementation | One planner computes the ready set of a run from the pinned graph and the recorded outputs for agent, tool, human, condition, timer, parallel, sub-workflow and compensation steps as WorkflowDefinition of docs/reference/schemas/asset-manifests.schema.json defines them: dependencies, when guards with a Skipped step treated as satisfied, conditions and bounded loops; it is the single planner behind StartWorkflowRun, Advance and PlanNextSteps, with unit tests per step kind. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1997 StartWorkflowRun inbox handler with WorkflowRunStarted and the StepReady payload copied from the definition |
implementation | The handler of RunAccepted opens the WorkflowRun for the pinned release, loading the release on first use, moves it from Pending to Running, creates the step runs of the first ready steps and writes WorkflowRunStarted and one StepReady per ready step in the same transaction, idempotent on the event id and on runId; StepReady carries actionClass, idempotent and failurePolicy copied from the step of the pinned definition, never derived, inputRef, and, for an agent or tool step, taskContractRef and taskContractHash of the task contract that AIFACTORY-TASK-3011 stored before the announcement, the StepReady of a human step carrying neither; no StepReady is written for a step of kind SubWorkflow, which AIFACTORY-US-647 executes; an unavailable registry fails the handler and nothing is written. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1998 WorkflowEngine.PlanNextSteps answered from the committed projection |
implementation | WorkflowEngine.PlanNextSteps returns, for a run and the last applied outcome sequence, the steps ready to dispatch with their input mappings and action classes, writes nothing and returns the same steps for the same sequence; callers without workload identity are refused with Unauthenticated, tokens without the service scope with PermissionDenied, invalid requests with InvalidArgument, and an unknown run and a run of another tenant alike with NotFound, the gRPC status NOT_FOUND (ADR-0015). | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-1999 Verify open-a-workflow-run-and-announce-ready-steps through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-WorkflowEngineService-004\|Scenario=SCN-WorkflowEngineService-007\|Scenario=SCN-WorkflowEngineService-008)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-499, including the idempotent redelivery of RunAccepted, actionClass, idempotent and failurePolicy of StepReady equal to the declared values of each chain definition, taskContractRef and taskContractHash present on the StepReady of every agent and tool step and absent on that of a human step, the Skipped step of a guard that does not hold, the refusal mapping of PlanNextSteps and the publication and payload of WorkflowRunStarted and StepReady; consumer handling is proven by the suite of RunControlService. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-746: Store the task contract of a step in the artifact store before the step is announced ready, and carry its reference and hash on StepReady¶
behavior
An agent or tool step leaves the engine with its task handoff already written (decisions D121 and D123). Whenever the engine finds an agent or tool step ready that it announces, in the handler StartWorkflowRun after RunControlService/RunAccepted, in the Advance after a StepSettled, after a fired timer or after a decision, it first builds the task contract of the step, a document of agent-task-contract-v1: the run, the agent the step names, the objective, the inputs by reference as the input mapping of the step maps them from the workflow inputs and the recorded outputs, the tools the step allows with their side-effect class and approval mode, the model profile the run was accepted with, the expected output with its descriptor and validation, and the limits. Every value comes from the pinned definition, from RunAccepted and from recorded outputs, and the engine invents none. The contract describes the step and no attempt (decision D123): the engine writes it once for the step, every attempt of the step, a retried one included, reads the same contract, and the engine writes neither attemptId nor the stepId of RunControlService nor snapshotId into it. None of the three is a required member of the schema, because Run Control creates the step and the attempt only after StepReady; the runtime takes the ids of the attempt and the reference of its snapshot from the answer of RunControlService/RunControl.AcquireLease. The contract of a step run is built deterministically from those sources and the step run id, so the same step run always yields the same content and the same hash. The engine stores the document through ArtifactService/Artifacts.Store under its workload identity, within the tenant and the project of the run, and takes the reference and the content hash of the reply. Only then does it create the StepRun, which keeps the reference in taskContractRef and the hash in taskContractHash (D125), and write WorkflowEngineService/StepReady, which carries the reference in taskContractRef and the hash in taskContractHash beside the fields AIFACTORY-US-499 specifies, in the transaction of the announcement. No agent or tool step is announced ready without a stored contract, and a compensation step, which Run Control dispatches like any other step (AIFACTORY-US-501), gets its contract on the same path, as the owner decided (D125). The contract names the kind of its step in stepKind and names the agent only for an agent step, so the contract of a tool step or of a compensation step may name no agent (D125). A human step has no task contract (decision D123): the engine stores none for it, both members are optional on StepReady and absent on the StepReady of a human step, and its step run keeps no reference (AIFACTORY-US-503). The engine writes the contract and no other service does: RunControlService keeps the reference and the hash on its step, copies them to each attempt of the step and returns them in the answer of RunControlService/RunControl.AcquireLease, and AgentRuntimeService reads the content by that reference and verifies it against the hash before the attempt starts; both are proven by their own features. ArtifactService is a fail-fast dependency of the announcement. When Artifacts.Store fails or does not answer, the handler fails, StepReady is not published for the step, nothing is committed and the inbox redelivers the event that made the step ready, so the engine repeats the write before it announces the step. The write is idempotent on tenant and hash: a repeat after a lost answer, a redelivered event and a restarted Advance all yield the same reference and the same hash and no second contract. Once the announcement is committed the step run holds the reference, so a re-emitted StepReady, written by the re-activated grain after a crash or by the StuckRunReconciler (AIFACTORY-US-502), carries that same reference and the same hash the step run keeps, and no contract is written again (decisions D123 and D125). The delay is the only effect: no step run fails and no terminal event is written because the store is away. A step of kind SubWorkflow is announced by no StepReady and gets no task contract on this path (AIFACTORY-US-647), and neither does a step the engine evaluates itself, a condition, a timer, a fan-out or a join.
Acceptance criteria
| AC1 | Given a Running WorkflowEngineService/WorkflowRun and a step of kind Agent or Tool that becomes ready, when the engine announces it, then ArtifactService/Artifacts.Store was called before the announcement, under the workload identity of the engine and within the tenant and the project of the run, with a document of agent-task-contract-v1 whose objective, inputs, allowed tools, model profile, expected output and limits equal what the pinned definition and RunControlService/RunAccepted state for the step and which is valid against docs/reference/schemas/agent-task-contract.schema.json without attemptId, without the stepId of RunControlService and without snapshotId, and WorkflowEngineService/StepReady is written in the transaction that creates the step run with the reference of the reply in taskContractRef and the hash of the stored content in taskContractHash, the step run keeping the same reference in taskContractRef and the same hash in taskContractHash (D125, ADR-0014). |
| AC2 | Given WorkflowEngineService/StepReady events published for the first steps of a run, for steps announced after a settled step, a fired timer and a decision, and for a compensation step, when the outbox and the artifact store are inspected, then every StepReady of an agent, tool or compensation step carries taskContractRef and taskContractHash, the content the artifact store holds under each reference hashes to its taskContractHash, the StepReady of a human step carries neither member, the artifact store holds no task contract for the human step and its step run keeps no reference, and a step of kind SubWorkflow and a step the engine evaluates itself have neither a StepReady nor a task contract (model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC3 | Given ArtifactService/Artifacts.Store fails or does not answer when an agent or tool step becomes ready, when the engine handles the event that made the step ready, then the handler fails, no WorkflowEngineService/StepReady is published for the step, nothing is committed, no step run is Failed and no terminal event is written, the inbox redelivers the event, and once the store answers the contract is stored and the step is announced once, as the failure mode of model/services/agent-mesh/WorkflowEngineService.yaml states. |
| AC4 | Given the write of the contract of one step run is repeated before its announcement is committed, after a lost answer of ArtifactService/Artifacts.Store, after a redelivery of the event that made the step ready or after a restart of the engine between the write and its commit, when WorkflowEngineService/StepReady is written, then the artifact store holds one task contract for the step run and the fake Run Control holds one step; given a committed announcement and a re-emission by the re-activated grain or by the StuckRunReconciler, then the re-emitted StepReady carries the same stepRunId and the taskContractRef and taskContractHash the step run keeps, and no second contract is written; and given the fake Run Control dispatches a second attempt of the step, then the engine writes no contract for it and the artifact store still holds one task contract for the step run (XL-06). |
| Traces: services | WorkflowEngineService, ArtifactService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun |
| Traces: apis | ArtifactService/Artifacts.Store, RunControlService/RunControl.AcquireLease |
| Traces: events | WorkflowEngineService/StepReady, RunControlService/RunAccepted |
| Traces: descriptors | agent-task-contract-v1, run-event-envelope-v1 |
| Traces: adrs | ADR-0012, ADR-0014 |
| Traces: requirements | FR-03, XL-06 |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-499, AIFACTORY-US-500, feature:artifact-service, feature:run-control-service, feature:contracts-package |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, model/services/knowledge/ArtifactService.yaml, docs/reference/schemas/agent-task-contract.schema.json, docs/planes/agent-runtime/services/workflow-engine-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-3011 Task contract builder and the store-then-announce path of every StepReady |
implementation | One builder produces the agent-task-contract-v1 document of a step run from the pinned definition, RunAccepted, the recorded outputs and the step run id, deterministically, so the same step run yields the same content and hash; the document describes the step and no attempt and carries no attemptId, no stepId of RunControlService and no snapshotId; the single path that writes StepReady, used by StartWorkflowRun, the Advance, the TimerScheduler, the decision handler, the compensation plan and the StuckRunReconciler, calls Artifacts.Store for an agent, tool or compensation step with that document under the workload identity of the engine and within the tenant and project of the run before it creates the step run, keeps the reference of the reply in TaskContractRef and the hash of the stored content in TaskContractHash of the step run (D125) and writes taskContractRef and taskContractHash on StepReady in the transaction of the announcement; the document names its stepKind and names the agent only for an agent step (D125); a failed or unanswered Artifacts.Store fails the handler without a write and without a StepReady, and a write repeated before the commit yields the same reference and hash; a re-emission takes the reference and the hash from the step run and stores nothing again; for a human step the path stores no contract, leaves TaskContractRef empty and writes StepReady without the two members; no contract is written for a step of kind SubWorkflow or for a step the engine evaluates itself; unit tests cover the builder per announced step kind, the human step announced without a contract and the refusal to announce an agent or tool step without a stored contract. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-3012 Fake of the artifact store that holds task contracts and can fail, lose its answer or be stopped |
fixture | The acceptance suite holds a fake of Artifacts.Store that keeps every stored document by tenant and content hash, returns the same reference for the same tenant and hash, exposes what it holds, and on request fails a write, stores a document but loses the answer, or is stopped and started again, so that the store-then-announce path, its repetition and its delay are reachable without ArtifactService. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-3013 Verify store-the-task-contract-before-a-step-is-announced through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=store-the-task-contract-before-a-step-is-announced"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-746 against the fake Run Control of AIFACTORY-TASK-2002 and the fake artifact store of AIFACTORY-TASK-3012: the contract stored before each StepReady of an agent, tool or compensation step with the members the engine fills equal to the pinned definition and without attemptId, the stepId of Run Control and snapshotId, taskContractRef and taskContractHash on each such StepReady, the same reference kept on the step run and the stored content hashing to the hash, the StepReady of a human step without the two members and with no contract in the store, no contract and no StepReady for a sub-workflow step and for a step the engine evaluates, the stopped store after which nothing is announced, nothing fails and the step is announced once when the store is back, one contract after a lost answer, a redelivery and a restart, the reference of the step run and the same hash on a re-emission of the grain and of the reconciler, and no second contract for a second attempt of the step. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-500: Advance a workflow run on each settled step through conditions, guards, fan-out, join and bounded loops, keep recorded outcomes on the projection, and report completion once¶
behavior
RunControlService records the outcome of an attempt exactly once and publishes RunControlService/AttemptOutcomeRecorded with runId, stepId, attemptId, outcome, failureClass and outcomeHash. That event only updates the projection of the engine (decision D121): the inbox handler AdvanceOnOutcome hands it to the WorkflowRunGrain of the run, which records the attempt and its outcome on the step run once, idempotently on the outcome sequence the projection keeps in lastOutcomeSequence, so an outcome delivered twice or re-applied after a restart changes nothing; the handler computes nothing and announces nothing, whatever the outcome is. An attempt outcome settles no step. The engine advances a run on RunControlService/StepSettled alone. Run Control publishes that event once per step, and it names the step run by the stepRunId the engine announced and carries beside it the stepId of Run Control, the final state (Succeeded, Failed, Skipped or Cancelled), the number of attempts, for a failed step the failure class and, for a step whose settling attempt reported an output, the outputRef of that attempt. The inbox handler ApplyStepSettled hands it to the same grain, which serialises Advance and applies it once, idempotently on the event id: the step run moves to the final state the event carries, keeps attemptCount and failureClass from it and takes its outputRef from it, and for a step settled Succeeded the engine evaluates what the graph allows next over recorded outputs only. Run Control writes the outcome that settles a step and its StepSettled in one transaction, and the two may reach the engine in either order; the order changes nothing, because the settlement alone advances the run. A Succeeded outcome whose StepSettled has not arrived moves no step run to a final state and announces nothing, and a StepSettled with the final state Succeeded whose outcome has not arrived yet advances the run without waiting for it. A condition step is evaluated by the engine itself in the expression language Workflows:Engine:ConditionLanguage; it reads only recorded step outputs and run variables, never calls a model or a service and never retries. When a condition is false and declares onFalse, the run returns to the named earlier step with the declared feedback and a new iteration of that step runs, at most maxIterations times, or Workflows:Engine:DefaultMaxLoopIterations times when the definition declares none; after the limit the run fails with the evidence (AIFACTORY-US-501). A missing or unreadable recorded output fails the condition with evidence and is never read as true. A step whose when guard does not hold is recorded Skipped and its dependants treat it as satisfied. A parallel step fans out into at most Workflows:Engine:MaxFanout branches; only inside a fan-out may more than one step run of a step be active, and the join waits for every branch that ran, or for the declared quorum, and treats a skipped branch as satisfied. One step run exists per step and iteration (UX_StepRuns_Run_Step_Iteration). Every move of a step run between two states of StepRunState is announced by WorkflowEngineService/StepRunStateChanged with fromState and toState, and every newly ready step by WorkflowEngineService/StepReady, all written with the step-run rows in the transaction of the Advance. Run variables and step outputs are held by reference in ArtifactService, a fail-fast dependency: when it is unavailable the Advance fails, nothing is written and the inbox redelivers the event. When every terminal step of the graph is Succeeded or Skipped, the run moves from Running to Completed through TerminalStepsSucceeded and WorkflowEngineService/WorkflowRunCompleted is written in the same transaction with the outputRef of the run; RunControlService records Completed in its handler CompleteRun and carries that reference as the outputRef of its RunCompleted (decision D121), and the engine never changes the authoritative state itself. A Failed, TimedOut, Cancelled or Uncertain outcome is not a success and announces no next step; it is recorded on the step run, which stays open until StepSettled names it. Whether a new attempt of the same step follows is decided by Run Control alone, and the engine announces nothing for a retry and holds no retry rule of its own. What the engine does when a step is settled Failed or Cancelled is specified in AIFACTORY-US-501, and what it does while Run Control keeps an operator decision open for a step that failed under Escalate or ended Uncertain in AIFACTORY-US-646. An outcome or a StepSettled for a run the projection does not hold, or for a run that is already Completed, Failed or Cancelled, changes nothing. AttemptOutcomeRecorded carries the stepId of Run Control and no step run id, and StepSettled carries both ids (decision D121), so from the settlement of a step on the engine knows which step run that stepId names. Two points stay unstated by the model and are reported to the owner: how an outcome that arrives before the settlement of its step, as the Failed outcome of an attempt that Run Control retries, is matched to its step run, and which value orders outcomes in lastOutcomeSequence, since the event carries no outcome sequence.
Acceptance criteria
| AC1 | Given a Running WorkflowEngineService/WorkflowRun and, for one of its step runs, a RunControlService/StepSettled that names its stepRunId with the final state Succeeded, the stepId of Run Control, an attempt count and an outputRef, when the handler ApplyStepSettled runs, then the step run is Succeeded and keeps the attemptCount and the outputRef of the event, a step run exists for every step whose dependencies are now satisfied, and WorkflowEngineService/StepRunStateChanged and one WorkflowEngineService/StepReady per newly ready step are written, each in the transaction of the change it announces, whether the RunControlService/AttemptOutcomeRecorded of the settling attempt arrived before, arrives after or has not arrived; an AttemptOutcomeRecorded handled by AdvanceOnOutcome records the attempt and its outcome on the projection, advances lastOutcomeSequence, moves no step run to a final state and announces nothing, also when its outcome is Succeeded and its StepSettled has not arrived; and the same outcome or the same StepSettled delivered twice changes nothing and writes no second event (SCN-WorkflowEngineService-009, XL-06). |
| AC2 | Given a condition such as vision-to-blueprint/check-valid, architecture-decomposition/check-valid, module-blueprints/check-valid, database-blueprint/check-valid or generate-product/check-build whose expression is false over the recorded output while its onFalse limit is not reached, when the engine evaluates it, then no model and no service is called, the run returns to the named earlier step with the declared feedback in a new iteration, and after the last allowed iteration the run fails with the evidence as AIFACTORY-US-501 specifies; a missing or unreadable recorded output fails the condition and is never read as true (ADR-0026). |
| AC3 | Given a fan-out such as module-blueprints/fan-out-contexts or generate-product/fan-out-fill within Workflows:Engine:MaxFanout, when its branches end, then the join module-blueprints/join or generate-product/join-fill waits for every branch that ran, or for the declared quorum, treats a branch skipped by its when guard as satisfied, and exactly one step run exists per step and iteration (ADR-0027, model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC4 | Given a step run moved between two states of StepRunState, when the outbox dispatcher publishes WorkflowEngineService/StepRunStateChanged on workflows.runs.v1.step-run-state-changed, then the outbox record was written in the same transaction as the step-run row, the event is sent under the canonical envelope with runId, tenantId, stepRunId, stepId, fromState and toState, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id; given ArtifactService is unavailable when the Advance needs a recorded output, then the Advance fails, nothing is written and the event is redelivered (SCN-WorkflowEngineService-009, ADR-0012). |
| AC5 | Given a Running WorkflowEngineService/WorkflowRun whose last terminal step is settled Succeeded, when the Advance is applied, then the run moves to Completed through TerminalStepsSucceeded and WorkflowEngineService/WorkflowRunCompleted is written in the same transaction and published on workflows.runs.v1.workflow-run-completed under the canonical envelope with runId, tenantId and outputRef; a transport outage is retried with backoff without loss or duplicate, a later outcome for the Completed run changes nothing, and the engine wrote no state of the run in Run Control (SCN-WorkflowEngineService-012, ADR-0014). |
| Traces: services | WorkflowEngineService, RunControlService, ArtifactService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun |
| Traces: events | WorkflowEngineService/StepRunStateChanged, WorkflowEngineService/StepReady, WorkflowEngineService/WorkflowRunCompleted, RunControlService/AttemptOutcomeRecorded, RunControlService/StepSettled |
| Traces: steps | vision-to-blueprint/check-valid, architecture-decomposition/check-valid, module-blueprints/fan-out-contexts, module-blueprints/check-valid, module-blueprints/join, database-blueprint/check-valid, generate-product/fan-out-fill, generate-product/join-fill, generate-product/check-build |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0024, ADR-0026, ADR-0027 |
| Traces: requirements | FR-03, FR-10, XL-06 |
| Traces: scenarios | SCN-WorkflowEngineService-009, SCN-WorkflowEngineService-012 |
| Relates: screens | workflow-run-graph |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-499, feature:run-control-service, feature:artifact-service |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/schemas/workflow.schema.json, model/workflows/module-blueprints.yaml, model/workflows/generate-product.yaml, docs/planes/agent-runtime/services/workflow-engine-service/index.md#domain-model |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2000 AdvanceOnOutcome and ApplyStepSettled inbox handlers and the Advance of the WorkflowRunGrain, advanced on StepSettled alone and idempotent on the outcome sequence and the event id |
implementation | The handlers of AttemptOutcomeRecorded and of StepSettled hand their events to the WorkflowRunGrain: an outcome is applied once on lastOutcomeSequence and only updates the projection, the attempt and its outcome recorded, with nothing computed and nothing announced; StepSettled is applied once on the event id, moves the step run it names by stepRunId to the final state it carries and writes attemptCount, failureClass and outputRef of the event on the step run with the stepId of Run Control; the run is advanced on StepSettled alone, an outcome moves no step run to a final state, and the two events of one step are handled in either order with the same result; once a step is settled Succeeded, whether its outcome was applied or not, the grain computes the next ready steps through the planner of AIFACTORY-TASK-197 and commits the step-run rows with StepRunStateChanged, StepReady and, at the last terminal step, WorkflowRunCompleted in one transaction; an event applied twice, an event for an unknown run and an event for a terminal run change nothing; an unavailable ArtifactService fails the Advance without a write; what follows a step settled Failed or Cancelled is delivered by AIFACTORY-US-501. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2001 Condition evaluation with bounded onFalse loops, when guards, fan-out and join |
implementation | Conditions and guards are evaluated in the configured expression language over recorded outputs and run variables only, with no model or service call and no retry; onFalse returns to the named step with the feedback in a new iteration up to maxIterations or DefaultMaxLoopIterations; a guard that does not hold records the step Skipped; a fan-out creates at most MaxFanout branches and the join waits for every branch that ran or for the declared quorum and treats a skipped branch as satisfied; unit tests cover each rule on the nine steps this PBI traces. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2002 Acceptance fixture of a fake Run Control that consumes StepReady and records outcomes |
fixture | A test double subscribes to StepReady, creates one step per stepRunId, keeps the failurePolicy the event carries and exposes what it received, so the engine scenarios run without RunControlService and a duplicated StepReady is visible as a single step; on request it publishes RunAccepted, RunCancelled, RunFailed and RunCompleted; for each attempt it publishes AttemptOutcomeRecorded with a scripted outcome (Succeeded, Failed, TimedOut, Cancelled or Uncertain) and failure class, and when its script lets a step reach its final state it publishes StepSettled with the stepRunId, its own stepId, the final state, the attempt count, for a failed step the failure class and, for a step whose settling attempt reported an output, the outputRef, before or after the outcome or without it as the script says; it holds the task contract reference and hash of every StepReady it received and exposes them; on WorkflowRunFailed it publishes RunFailed for the run, as Run Control does when the engine reports the end of a failed step; for a step that fails under Escalate, and for an Uncertain outcome on a step of class shared-write or above, it publishes no StepSettled and plays the operator decision instead, an ApprovalDecided with proposalKind operator-decision, the stepRunId, the choice retry, compensate or fail and, for an Uncertain outcome, the finding applied or not-applied, or its expiry, each followed by what Run Control does after it (a new attempt, StepSettled with the final state Failed, RunFailed); it answers RunControl.SubmitChildRun with one child run per idempotency key and exposes the submissions it received; it can keep every announced step of a run without an attempt, as Run Control does for a run that is Held or WaitingForApproval; for human steps it publishes ApprovalDecided in each state (Approved, Rejected, ChangesRequested, Expired followed by a re-request, Expired without one, Stale and Cancelled). | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2003 Verify advance-a-workflow-run-on-recorded-outcomes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-WorkflowEngineService-009\|Scenario=SCN-WorkflowEngineService-012)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-500 on the chain definitions, including the outcome and the StepSettled of a step in both orders, the StepSettled without its outcome that advances the run, the Succeeded outcome without its StepSettled that announces nothing, the attempt count and the output reference of StepSettled kept on the step run, the outcome applied twice, the bounded loop of each check step, the fan-out and join with a skipped branch, the single WorkflowRunCompleted and the publication and payload of StepRunStateChanged and WorkflowRunCompleted; consumer handling is proven by the consumers' own suites. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-501: Fail a workflow run or compensate its succeeded steps by the failure policy each step declares, close a cancelled step run, and report the failure once¶
behavior
Every agent, tool and human step of a loaded definition declares its failure policy, Retry, Compensate, Escalate or Fail, and the engine applies the declared value: it copies failurePolicy of the step of the loaded WorkflowDefinition into failurePolicy of the StepDefinition and derives none. RunControlService/AttemptOutcomeRecorded reports one of five outcomes, and four of them are not a success. Each of the four is first the business of Run Control, as the invariants of its Attempt state (model/services/control-plane/RunControlService.yaml). A Failed attempt is retried below maxAttempts of the Step only when its failure class is transient and the side-effect class of the step is read or sandbox, or shared-write when the step declares an idempotency key (ADR-0014), both read from the Step that Run Control created from StepReady; any other Failed attempt fails the step terminally. A TimedOut attempt follows the lease reconciliation rule: the LeaseExpiryReconciler of Run Control settles the attempt of an expired lease as TimedOut and schedules the retry or the terminal failure, so a TimedOut outcome may be followed by a new attempt of the same step. An Uncertain outcome means that an external effect may have happened; it is never retried automatically when the side-effect class of the step is shared-write or above, and on such a step Run Control opens an operator decision for the Operator of the project. A Cancelled outcome is the settlement of an attempt of a run that Run Control is cancelling. The engine evaluates no retry rule and counts no attempts against a limit: StepReady carries no maxAttempts and maxAttempts of the Step comes from RunControl:Retries:DefaultMaxAttempts unless the release overrides it, so a second copy of the retry rule in the engine could disagree with the authority and is not built. The engine announces nothing for a retry and no longer guesses whether a Failed or TimedOut outcome was the last one. It learns that from RunControlService/StepSettled: Run Control publishes the event once per step, in the transaction that moves its Step to the final state, with the stepRunId the engine announced, the final state, the number of attempts and, for a failed step, the failure class of its last attempt; the step run keeps the two in attemptCount and failureClass (decision D121). A Failed or TimedOut outcome without a StepSettled changes the state of no step run, whether Run Control follows it with a new attempt or not. The inbox handler ApplyStepSettled applies the event once, idempotently on the event id, and for a step settled Failed the engine acts by the declared policy. Who ends the run is decided (decision D121): after a failed step Run Control only settles the step and fails nothing, its own transition of a run from Running to Failed on a terminally failed step is withdrawn, and the run stays Running there until the engine, which applies the failure policy of the step, Fail or Compensate, reports WorkflowRunFailed. With policy Fail the step run moves to Failed, the run moves from Running to Failed through StepFailedWithFailPolicy, and WorkflowEngineService/WorkflowRunFailed is written in the same transaction with failedStepId, the failureClass of StepSettled and compensated false. A step settled Failed under the policy Retry has its attempts spent, since Run Control settles it only when it retries no more, and the engine fails the run for it like for a step under Fail (decision D123): the step run moves to Failed, the run moves from Running to Failed through StepFailedWithRetryPolicyAttemptsSpent, and the same WorkflowRunFailed is written in the same transaction. That report alone fails the run: Run Control records Failed when it hears it and publishes RunControlService/RunFailed, which finds the projection of the engine already Failed and changes nothing. A step settled Cancelled moves its step run to Cancelled with WorkflowEngineService/StepRunStateChanged, announces no step and writes no WorkflowRunFailed, whatever policy the step declares: the run is closed by RunCancelled (AIFACTORY-US-503), so the single terminal report of a cancelled run is WorkflowRunCancelled. Two failures are not settled by rule and not by the engine. A step that fails terminally under the policy Escalate, and a step of class shared-write or above whose outcome is Uncertain whatever policy it declares, go to the operator decision of Run Control, and the step stays unsettled while the request is pending. The engine records the outcome on the step run, which stays open: it does not retry the step, announces nothing, fails nothing and writes no terminal event, and when Run Control later settles such a step Failed the engine applies no declared policy to it, because what follows is the choice of the Operator (AIFACTORY-US-646). A condition that is still false after its last allowed iteration fails the run through StepFailedWithFailPolicy with the recorded evidence. A run whose pinned release the engine refused at load fails at acceptance with an explicit reason, as the failure modes of the model state, and no step of it is ever announced. With policy Compensate a step settled Failed moves the run from Running to Compensating through StepFailedWithCompensatePolicy: the engine plans one CompensationEntry for each succeeded step that declares a compensation, in reverse completion order, writes WorkflowEngineService/CompensationStarted with the number of entries, and announces each compensation step through StepReady with the action class of the compensation step itself, so Run Control dispatches it like any other step. A step without a compensation leaves its sandbox output in place as evidence. The owner kept this rule (D133): a run compensates only when the failing step declares the failure policy Compensate or the Operator chooses compensate, a definition that promises a compensation declares Compensate on the steps that follow an effect, and the compensation of a step runs only after its effect is confirmed, which is why only succeeded steps get an entry; the engine derives no compensation from a definition otherwise. Each compensation step gets its task contract like an agent or tool step before it is announced (D125, AIFACTORY-US-746). While the compensation runs, the run stays Running in Run Control, which waits for the report of the engine and dispatches the compensation steps. When every entry is Compensated the run moves from Compensating to Failed through CompensationCompletedOrFailed and WorkflowRunFailed carries compensated true. When a compensation step fails, its entry is Failed, the run ends Failed with the failure class compensation and compensated false, the partially compensated effects stay listed on the run, no further automatic retry happens and an operator follow-up is raised, never silently. connectsoft.factory.workflows.compensations counts each compensation with its outcome. RunControlService records Failed with the reported class in its handler FailRun; StudioBff, DashboardService and AuditService consume the events in their own PBIs and increments. The same compensation plan is started by the choice compensate of an operator decision (AIFACTORY-US-646). The run failures that no failure policy of a step decides stay with Run Control, a denied budget, a rejected or expired approval, a retired environment and the operator decision among them; for those Run Control fails the run itself and the engine closes its projection on RunFailed (AIFACTORY-US-646). A RunFailed that reaches a run in Compensating finds no modelled transition and changes nothing. The following are not stated by the model and are reported to the owner. Whether the explicit reason of a refused release reaches Run Control through WorkflowRunFailed or through the answer of PlanNextSteps at acceptance is open, since a workflow run row needs a loaded definition. The failure class compensation is not among the six classes RunFailed of Run Control lists, and the failure class of a condition that is exhausted is not stated. The definitions declare compensation as free text while a CompensationEntry keeps a compensationStepId, so how the text resolves to a step is open; the fixture of this PBI names the id of a step of kind compensation. failurePolicy of a StepDefinition is not nullable although condition, timer, parallel and compensation steps declare none. And where the operator follow-up of a failed compensation is raised is not modelled.
Acceptance criteria
| AC1 | Given a Running WorkflowEngineService/WorkflowRun, a step for which the WorkflowEngineService/WorkflowDefinitionVersion stores the declared failure policy Fail or Retry, and a RunControlService/StepSettled that names its step run with the final state Failed, an attempt count and a failure class, when the handler ApplyStepSettled runs, then the step run is Failed and keeps the attemptCount and the failureClass of the event, the run moves to Failed, through StepFailedWithFailPolicy for the policy Fail and through StepFailedWithRetryPolicyAttemptsSpent for the policy Retry, whose attempts are spent when the step is settled, and WorkflowEngineService/WorkflowRunFailed with failedStepId, the failureClass of the event and compensated false is written in the same transaction, the one report on which Run Control, which only settled the step, fails the run; a Failed or TimedOut outcome in RunControlService/AttemptOutcomeRecorded that no StepSettled follows, whether the fake dispatches a new attempt or not, changes the state of no step run and announces nothing, the engine counts no attempts against a limit, and no default policy is applied to an agent, tool or human step (SCN-WorkflowEngineService-013, ADR-0014). |
| AC2 | Given a Running run and a RunControlService/StepSettled with the final state Cancelled for one of its open step runs, when the handler ApplyStepSettled runs, then the step run moves to Cancelled with WorkflowEngineService/StepRunStateChanged, no WorkflowEngineService/StepReady and no WorkflowEngineService/WorkflowRunFailed is written under any declared policy, and the run stays open until the cancellation of the run closes it; given an outcome Uncertain in RunControlService/AttemptOutcomeRecorded, then it is recorded on the step run, the step run stays open until a StepSettled names it, the engine does not retry the step, nothing is announced and no terminal event is written, since the invariants of the attempt in model/services/control-plane/RunControlService.yaml send an Uncertain outcome on a step of class shared-write or above to the operator decision of Run Control (ADR-0014). |
| AC3 | Given the run failed, when the outbox dispatcher publishes WorkflowEngineService/WorkflowRunFailed on workflows.runs.v1.workflow-run-failed, then the event is sent under the canonical envelope with runId, tenantId, failedStepId, failureClass and compensated, a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id, exactly one terminal event exists for the workflow run, and a later outcome for the Failed run, and the RunControlService/RunFailed that Run Control publishes when it hears the report, change nothing; given a condition that is still false after its last allowed iteration, then the event names the condition step with the recorded evidence and nothing is announced afterwards; given an accepted run whose pinned release the engine refused at load, then no step of it is announced and the run fails at acceptance with an explicit reason, as the failure modes of model/services/agent-mesh/WorkflowEngineService.yaml state (SCN-WorkflowEngineService-013, ADR-0012). |
| AC4 | Given the fixture definition of AIFACTORY-TASK-2006, a Running run with succeeded steps of which one declares a compensation step and one does not, and a step that declares the failure policy Compensate and that RunControlService/StepSettled names with the final state Failed, when the handler ApplyStepSettled runs, then the run moves to Compensating through StepFailedWithCompensatePolicy, one compensation entry is planned per succeeded step with a compensation step in reverse completion order, WorkflowEngineService/CompensationStarted with runId, tenantId and entries is written in the same transaction and published on workflows.runs.v1.compensation-started under the canonical envelope, retried on a transport outage without loss or duplicate, and each compensation step is announced through WorkflowEngineService/StepReady with its own action class (SCN-WorkflowEngineService-011). |
| AC5 | Given a Compensating WorkflowEngineService/WorkflowRun, when every compensation entry is Compensated, then the run moves to Failed through CompensationCompletedOrFailed and WorkflowEngineService/WorkflowRunFailed carries compensated true; when a compensation step fails, then its entry is Failed, the run ends Failed with the failure class compensation and compensated false, the partially compensated effects stay listed on the run, no further automatic retry happens, and connectsoft.factory.workflows.compensations counts the compensation with its outcome (SCN-WorkflowEngineService-011, SCN-WorkflowEngineService-013, FR-10). |
| Traces: services | WorkflowEngineService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, WorkflowEngineService/WorkflowDefinitionVersion |
| Traces: events | WorkflowEngineService/WorkflowRunFailed, WorkflowEngineService/CompensationStarted, WorkflowEngineService/StepReady, WorkflowEngineService/StepRunStateChanged, RunControlService/AttemptOutcomeRecorded, RunControlService/StepSettled, RunControlService/RunFailed |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0024 |
| Traces: requirements | FR-10, XL-06 |
| Traces: scenarios | SCN-WorkflowEngineService-011, SCN-WorkflowEngineService-013 |
| Relates: screens | workflow-run-graph, run-detail |
| Relates: nfrs | NFR-05, NFR-06 |
| Depends on | AIFACTORY-US-500, feature:run-control-service |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, model/schemas/workflow.schema.json, model/workflows/generate-product.yaml, docs/planes/agent-runtime/services/workflow-engine-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2004 Apply the declared failure policy of a step that StepSettled names Failed, leave an unsettled step open, and report WorkflowRunFailed once |
implementation | The Advance holds no retry rule and no attempt limit of its own and treats no step as settled on an outcome. When ApplyStepSettled names a step run with the final state Failed, it writes attemptCount and failureClass of the event on the step run, reads failurePolicy from the StepDefinition, where it was copied from the declared failurePolicy, and moves the step run to Failed and the run to Failed, for Fail through StepFailedWithFailPolicy and for Retry, whose attempts are spent when the step is settled, through StepFailedWithRetryPolicyAttemptsSpent, and writes WorkflowRunFailed with failedStepId, the failureClass of StepSettled and compensated false in the same transaction; that report is the one on which Run Control fails the run, since it only settled the step, and the RunFailed that follows it finds the projection Failed and changes nothing; a Failed or TimedOut outcome without StepSettled changes the state of no step run; a step settled Cancelled moves the step run to Cancelled with StepRunStateChanged and writes neither StepReady nor WorkflowRunFailed; an outcome Uncertain is recorded, is not retried and leaves the step run open; to a step settled Failed whose policy is Escalate, or whose last outcome on a step of class shared-write or above was Uncertain, no declared policy is applied, since AIFACTORY-US-646 follows the choice of the Operator; a condition false after its last iteration fails the run with its evidence, and a run whose pinned release the engine refused at load fails at acceptance with an explicit reason and gets no StepReady; at most one terminal event is written per workflow run. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2005 Compensation plan in reverse completion order with CompensationStarted and the terminal report |
implementation | For policy Compensate a step settled Failed moves the run to Compensating, one CompensationEntry is written per succeeded step that declares a compensation step in reverse completion order, CompensationStarted is written with the number of entries and each compensation step is announced through StepReady with its own action class; when all entries are Compensated the run moves to Failed with compensated true, and when a compensation step fails the entry is Failed and the run ends Failed with the failure class compensation, compensated false, the partially compensated effects listed and no further automatic retry; connectsoft.factory.workflows.compensations is emitted with tenant, definition and outcome. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2006 Fixture definitions that declare Fail, Retry and Compensate with a compensation step |
fixture | The fixture set of AIFACTORY-TASK-1995 gains definitions of its own in the wire shape of WorkflowDefinition: a step with failurePolicy Fail and one with failurePolicy Retry; a step with failurePolicy Compensate that runs after two succeeded steps, of which one names in compensation the id of a step of kind compensation and one declares none; and a compensation step scripted to fail, so both ends of the compensation path are reachable in the acceptance suite; no definition of model/workflows is changed, since the six chain definitions plan no compensation entry of an earlier step. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2007 Verify fail-or-compensate-a-workflow-run through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-WorkflowEngineService-011\|Scenario=SCN-WorkflowEngineService-013)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-501 with the scripts of the fake Run Control of AIFACTORY-TASK-2002: a StepSettled with the final state Failed under Fail and under Retry, each failing the run through its own transition, after a Failed and after a TimedOut last outcome, with the attempt count and the failure class of the event kept on the step run, the RunFailed the fake publishes after WorkflowRunFailed that changes nothing, a Failed and a TimedOut outcome without StepSettled that change nothing, with and without a new attempt, the step settled Cancelled that moves the step run to Cancelled without WorkflowRunFailed, the Uncertain outcome that leaves the step run open and is not retried, the policy read from the definition and never derived, the compensation in reverse order with compensated true, the failed compensation with compensated false and the publication and payload of CompensationStarted and WorkflowRunFailed; consumer handling is proven by the consumers' own suites. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-502: Fire durable timers once, make a late timer a no-op, and recover a lost reminder or a lost announcement by reconciliation¶
behavior · persona operator
As an operator, I want a waiting run to continue when its timer is due, and a run whose reminder or announcement was lost to be found and continued, so that no run stays stuck silently and no timer moves a run twice.
A timer step and the timeout of a step are durable timers of the WorkflowRun. When the Advance reaches a timer step, the engine writes a TimerEntry with timerId, stepRunId and dueAt, moves the run from Running to WaitingForTimer through TimerStepReached, and the WorkflowRunGrain registers one Orleans reminder per pending entry. A step timeout is a TimerEntry of the same kind, due after the timeout the step declares or after Workflows:Engine:DefaultStepTimeoutMinutes when it declares none. The TimerScheduler applies a fired reminder to the run, idempotently on timerId and partitioned by run: the entry of a timer step is marked fired, a run in WaitingForTimer returns to Running through TimerFired, the next ready steps are computed, and WorkflowEngineService/TimerFired with runId, tenantId, timerId and stepRunId is written in the same transaction as the StepReady records of those steps. A fired timer that finds its step already advanced is a no-op: nothing is written and no event is published, so a late or duplicated reminder never moves a run twice. Reminders can be lost. The StuckRunReconciler runs on its schedule, partitioned by tenant and idempotent on stepRunId plus lastOutcomeSequence: it fires a due timer whose reminder did not fire, and for a Running workflow run older than the step timeout without progress it re-emits StepReady with the same stepRunId and, for an agent or tool step, the task contract reference the step run keeps in taskContractRef and the hash of the same contract, without writing the contract again (decision D123, AIFACTORY-US-746), for every active step run that has no recorded attempt outcome, which Run Control deduplicates; the re-emitted StepReady of a human step carries no reference, as its first one did; a step run of kind SubWorkflow, which no StepReady announced, is left to its child run (AIFACTORY-US-647). That is all it does. Neither the reconciler nor a step timeout fails a step run on its own. A step timeout that fires while its step run is still open marks its entry fired and leaves the step run to that rule; it advances nothing, so it writes no TimerFired, the event of a timer that advanced the run, and it moves neither the step run nor the run. The reason is the authority of Run Control. The engine does not hold the states Held and WaitingForApproval, which only RunControlService keeps: a step that waits there because the environment of the run is frozen, or because a step approval or a spend approval is pending, has no attempt for longer than any timeout the definitions declare. A re-emitted StepReady changes nothing in Run Control, while a step run failed by the engine would end in WorkflowRunFailed for a run that Run Control keeps waiting. The model describes the reconciler as finding active step runs with no corresponding open attempt or approval in Run Control and as re-emitting StepReady or failing the step per policy, but it gives the engine no read of the attempts, the approvals or the run state of Run Control. When the reconciler may fail a step, what a fired step timeout does beyond the rule above, and how either tells a wait in Run Control from a lost dispatch are therefore not stated and are reported to the owner. The same re-emission recovers a crash between applying the settlement of a step and emitting the StepReady it causes. connectsoft.factory.workflows.timers_pending moves with every entry created and fired, and connectsoft.factory.workflows.step_latency records the time from StepReady to a terminal step state. When the run is cancelled its pending timers are skipped (AIFACTORY-US-503).
Acceptance criteria
| AC1 | Given a Running WorkflowEngineService/WorkflowRun whose Advance reaches a timer step, when it is applied, then a timer entry with timerId, stepRunId and dueAt is written, the run moves to WaitingForTimer through TimerStepReached and one reminder is registered; when the reminder fires at dueAt, then the entry is marked fired, the run returns to Running through TimerFired, and WorkflowEngineService/TimerFired and the WorkflowEngineService/StepReady of every newly ready step are written in the same transaction (SCN-WorkflowEngineService-010). |
| AC2 | Given a timer fired, when the outbox dispatcher publishes WorkflowEngineService/TimerFired on workflows.runs.v1.timer-fired, then the outbox record was written in the same transaction as the fired entry, the event is sent under the canonical envelope with runId, tenantId, timerId and stepRunId, a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id, and connectsoft.factory.workflows.timers_pending went down by one (SCN-WorkflowEngineService-010, ADR-0012). |
| AC3 | Given a timer whose step already advanced, the same timerId delivered twice, or a timer of a run that is Completed, Failed or Cancelled, when it fires, then nothing is written, no WorkflowEngineService/TimerFired is published and the run does not move, as the invariant of the WorkflowEngineService/WorkflowRun on fired timers requires (SCN-WorkflowEngineService-010, model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC4 | Given a due timer entry whose reminder was lost, when the StuckRunReconciler runs, then it fires the timer with the effect of AC1 exactly once; given a Running workflow run without progress beyond the step timeout, then for every active step run that has no recorded attempt outcome the reconciler re-emits WorkflowEngineService/StepReady with the same stepRunId and the task contract reference the step run keeps, which creates no second step in the fake Run Control and no second contract in the artifact store, it changes the state of no step run and fails none, and a second pass for the same stepRunId and lastOutcomeSequence changes nothing (SCN-WorkflowEngineService-010, FR-10, docs/planes/agent-runtime/services/workflow-engine-service/index.md). |
| AC5 | Given a run whose announced step the fake Run Control keeps without an attempt beyond the step timeout, as Run Control does for a run that is Held or WaitingForApproval, when the step timeout fires and the StuckRunReconciler runs, then the timeout entry is marked fired, no step run is Failed, no WorkflowEngineService/TimerFired and no terminal event is written, the WorkflowEngineService/WorkflowRun does not move, and when the fake later records the outcome the run advances as if nothing had fired (model/services/control-plane/RunControlService.yaml, ADR-0024). |
| Traces: services | WorkflowEngineService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun |
| Traces: events | WorkflowEngineService/TimerFired, WorkflowEngineService/StepReady |
| Traces: adrs | ADR-0012, ADR-0024 |
| Traces: requirements | FR-10 |
| Traces: scenarios | SCN-WorkflowEngineService-010 |
| Relates: nfrs | NFR-06 |
| Depends on | AIFACTORY-US-500 |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, docs/architecture/adr/0024-durable-execution-substrate.md, docs/planes/agent-runtime/services/workflow-engine-service/index.md#workers-and-jobs, docs/planes/agent-runtime/services/workflow-engine-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2008 Timer entries, grain reminders and the TimerScheduler with TimerFired |
implementation | A timer step and a step timeout write a TimerEntry and register one reminder of the WorkflowRunGrain; the TimerScheduler applies a fired reminder idempotently on timerId: for a timer step it marks the entry fired, returns a WaitingForTimer run to Running, computes the next ready steps and writes TimerFired with their StepReady records in one transaction; a step timeout that fires on a step run that is still open marks its entry fired, fails nothing and writes no event; a timer whose step advanced, a duplicate and a timer of a terminal run are no-ops; connectsoft.factory.workflows.timers_pending and step_latency are emitted with their modelled dimensions. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2009 StuckRunReconciler firing due timers and re-emitting StepReady for step runs without a recorded outcome |
implementation | The scheduled worker, partitioned by tenant with the fixed retry policy of three attempts, fires every due TimerEntry of a timer step that is not fired, and for Running workflow runs without progress beyond the step timeout re-emits StepReady with the same stepRunId, the task contract reference kept in TaskContractRef of the step run and the hash of the same contract, through the announcing path of AIFACTORY-TASK-3011, which stores nothing again on a re-emission and sends the StepReady of a human step without a reference, for every active step run that has no recorded attempt outcome, a step run of kind SubWorkflow excepted, which no StepReady announced; it changes the state of no step run and fails none, since the model gives the engine no read of the run state of Run Control; it is idempotent on stepRunId plus lastOutcomeSequence, so a second pass changes nothing. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2010 Verify fire-durable-timers-and-recover-stuck-runs through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-WorkflowEngineService-010"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-502 with a controllable clock, including the timer fired once, the late and the duplicated timer as no-ops, the lost reminder fired by the reconciler, the re-emitted StepReady that carries the task contract reference of its step run and that the fake Run Control of AIFACTORY-TASK-2002 receives as one step, the case of a run the fake keeps Held beyond the step timeout, in which no step run is Failed and no WorkflowRunFailed is written, and the publication and payload of TimerFired; consumer handling is proven by the suite of DashboardService in its increment. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-503: Wait for the decision of a human step, apply it as Run Control recorded it, and close the projection when the run is cancelled¶
behavior · persona reviewer
As a reviewer, I want my recorded decision on a human step to move the workflow exactly once, and a cancelled run to stop planning, so that the engine never advances on a decision I did not take and never announces a step of a run that was cancelled.
Until I4 a human step is always a decision (D133). When the Advance reaches a human step, the engine announces it through StepReady with kind Human and moves the run from Running to WaitingForHuman through HumanStepReached. A human step has no task contract: the engine stores none for it, its StepReady carries neither taskContractRef nor taskContractHash, and its step run keeps no reference (decision D123). actionClass of that StepReady is, as for every step (AIFACTORY-US-499), the side-effect class the step declares in the pinned definition; the action class of the approval binding of the step is read by Run Control for the ApprovalRequest, which RunControlService opens for the exact proposal in its handler CreateStepAndDispatch. The engine opens no approval and decides nothing. The owner decided that from I4 a human step may carry a policy guard, as generate-product/approve-proposal and project-bootstrap/approve-shared-write declare it: the engine then asks PolicyEngineService before it announces the step, and an allow records the step run Skipped, emits no StepReady, opens no approval and advances the run, while any other answer announces the step as here (D133). This I1 delivery keeps the guard of a loaded definition in PolicyGuardJson of its step definition and evaluates none, so the approval of every human step runs. A human step that waits on the promotion gate of EnvironmentPromotionService opens no approval and is announced by no StepReady; that wait is delivered by AIFACTORY-US-1496. RunControlService publishes every decision as RunControlService/ApprovalDecided with approvalId, the subject as subjectKind and subjectId, runId when the subject is a run or belongs to one (D135), state, decidedBy, reason and the proposal hash, and the state is one of Approved, Rejected, ChangesRequested, Expired, Stale and Cancelled. The inbox handler AdvanceOnDecision applies a decision to the human step run it belongs to, idempotently on the event id. An Approved decision completes the human step as Succeeded, returns the run from WaitingForHuman to Running through DecisionApplied, and the next ready steps are announced. A Rejected decision fails the human step, and the failure policy the step declares decides what follows (AIFACTORY-US-501): with policy Fail the engine reports WorkflowRunFailed with the failure class policy-denied, the class a run failed by a rejected or expired approval carries (ADR-0014), and Run Control, which failed the run through its own transition, takes the report as a duplicate that changes nothing; the RunFailed it publishes for that run finds the projection already Failed, and a Rejected decision that arrives after RunFailed closed the projection changes nothing either (AIFACTORY-US-646). A ChangesRequested decision returns the run to Running in Run Control with the reason of the reviewer; the engine applies it by leaving WaitingForHuman through DecisionApplied without completing the step as Succeeded and without failing the run. An Expired, a Stale and a Cancelled decision change nothing in the engine: the step run keeps waiting and the run stays WaitingForHuman. Run Control settles an expired request by the on-expiry rule of the approval policy in the pinned policy revision of the run, fail or re-request up to its limit, and publishes ApprovalDecided with the state Expired in both cases; after a re-request the run stays WaitingForApproval with a new Pending request, so an engine that failed the step on Expired would report WorkflowRunFailed for a run that Run Control keeps waiting. A Stale request is followed by a new request for the current proposal hash, and a Cancelled request belongs to a run that was cancelled or failed; the model says that an expired or stale decision never advances the run. When an expiry was final, Run Control fails the run itself with the failure class policy-denied and publishes RunFailed; the engine hears it and closes its projection as Failed without a WorkflowRunFailed of its own (AIFACTORY-US-646). A decision whose proposalKind is operator-decision is no decision on a human step: it names its step run in stepRunId, carries the choice of the Operator, and the same handler applies it as AIFACTORY-US-646 specifies. Any other decision that belongs to no human step run of the projection changes nothing in the engine. That is the case for the approval a lease holder requested for a gated action inside an agent or tool step, where the reason of the reviewer goes back to the lease holder and the run is WaitingForApproval only in Run Control, and for the run-level spend approval that precedes the budget reservation, and it is the case for a decision whose subject is no run, such as a provisioning run, an evaluation run or an asset release, which belongs to the service that owns the subject (D135); a decision of subject kind promotion is applied only to a human step that waits on that gate (AIFACTORY-US-1496). The same event feeds the readiness correlation in a separate handler (AIFACTORY-US-505). The engine also follows cancellation without owning it. RunControlService records the intent, settles the attempts and publishes RunControlService/RunCancelled; the handler CancelWorkflowRun then closes the projection: the run moves to Cancelled from Running, WaitingForTimer or WaitingForHuman, every open step run moves to Cancelled, pending timers are skipped so that a reminder that still fires is a no-op, and WorkflowEngineService/WorkflowRunCancelled is written in the same transaction with the number of step runs that were open. A step run that a StepSettled with the final state Cancelled already closed (AIFACTORY-US-501) is not counted again. A cancellation for a run that is already Completed, Failed or Cancelled, or that the projection does not hold, changes nothing; a Compensating run has no cancel transition in the model and finishes its compensation. When the cancelled run is the child run of a step of kind SubWorkflow, a second handler of the same event returns the cancellation to the parent step run (AIFACTORY-US-647). Three points are not stated by the model and are reported to the owner: ApprovalDecided names a step run in stepRunId only for an operator decision, and nothing tells the engine the approvalId of the request Run Control opened for a human step, so the match of a decision on a human step to its step run has no modelled key; which step becomes ready after a ChangesRequested decision on a human step, where no lease holder regenerates the proposal; and StepReady does not carry the approval binding, while a human step declares both a side-effect class and the action class of its binding, which the schema lets differ.
Acceptance criteria
| AC1 | Given a Running WorkflowEngineService/WorkflowRun whose Advance reaches a human step with its approval binding, such as generate-product/approve-proposal, when it is applied, then WorkflowEngineService/StepReady with kind Human, actionClass equal to the side-effect class the step declares and neither taskContractRef nor taskContractHash is written, no task contract is stored for the step, the run moves to WaitingForHuman through HumanStepReached in the same transaction, and the engine opens no RunControlService/ApprovalRequest of its own; the action class of the binding is read by Run Control for that request; the step is announced the same way although it declares a policy guard from I4, which this delivery keeps and does not evaluate (D133, FR-06, model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC2 | Given a WaitingForHuman run and a RunControlService/ApprovalDecided for its human step, when the handler AdvanceOnDecision runs, then an Approved decision completes the step run as Succeeded, returns the run to Running through DecisionApplied and announces the next ready steps; a Rejected decision fails the step run and the declared failure policy applies, policy Fail giving WorkflowEngineService/WorkflowRunFailed with the failure class policy-denied; a ChangesRequested decision leaves WaitingForHuman through DecisionApplied with the step run not Succeeded and the run not failed; and a redelivery of the same event id changes nothing (ADR-0014, FUX-05, XL-05). |
| AC3 | Given a WaitingForHuman run and a RunControlService/ApprovalDecided with the state Expired, Stale or Cancelled for its human step, when the handler runs, then the step run keeps waiting, the run stays WaitingForHuman, nothing is announced and no WorkflowEngineService/WorkflowRunFailed is written, because Run Control may re-request an expired approval by its on-expiry rule and an expired or stale decision never advances the run, and an Approved decision that follows a re-request completes the step; given a decision of any state that belongs to no human step run of the projection and is no operator decision, as for an approval requested by a lease holder or a spend approval of the run, then nothing changes in the engine and nothing is published (model/services/control-plane/RunControlService.yaml). |
| AC4 | Given a WorkflowEngineService/WorkflowRun that is Running, WaitingForTimer or WaitingForHuman and a RunControlService/RunCancelled for its runId, when the handler CancelWorkflowRun runs, then the run moves to Cancelled, every open step run moves to Cancelled, pending timers are skipped, and WorkflowEngineService/WorkflowRunCancelled with runId, tenantId and openStepRuns is written in the same transaction (SCN-WorkflowEngineService-014, FR-10). |
| AC5 | Given the projection was closed, when the outbox dispatcher publishes WorkflowEngineService/WorkflowRunCancelled on workflows.runs.v1.workflow-run-cancelled, then the event is sent under the canonical envelope with runId, tenantId and openStepRuns, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id; a cancellation for a run already Completed, Failed or Cancelled or unknown to the projection, a redelivery, and a timer that fires after the cancellation change nothing (SCN-WorkflowEngineService-014, ADR-0012). |
| Traces: services | WorkflowEngineService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, RunControlService/ApprovalRequest |
| Traces: events | WorkflowEngineService/WorkflowRunCancelled, WorkflowEngineService/StepReady, WorkflowEngineService/WorkflowRunFailed, RunControlService/ApprovalDecided, RunControlService/RunCancelled |
| Traces: steps | generate-product/approve-proposal |
| Traces: adrs | ADR-0012, ADR-0014 |
| Traces: requirements | FR-06, FR-10, FUX-05, XL-05 |
| Traces: scenarios | SCN-WorkflowEngineService-014 |
| Relates: screens | workflow-run-graph |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-500, AIFACTORY-US-501, AIFACTORY-US-502, feature:run-control-service |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, model/workflows/generate-product.yaml, docs/planes/agent-runtime/services/workflow-engine-service/index.md#messages |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2011 Human step announcement and the AdvanceOnDecision inbox handler |
implementation | A human step is announced through StepReady with kind Human and, as actionClass, the side-effect class the step declares, without a task contract and so without taskContractRef and taskContractHash, and moves the run to WaitingForHuman; the handler of ApprovalDecided applies a decision to the human step run it belongs to, idempotently on the event id: Approved completes the step and announces the next ready steps, Rejected fails the step under its declared failure policy, ChangesRequested leaves WaitingForHuman without completing or failing, an Expired decision, whether Run Control re-requests the approval or not, a Stale and a Cancelled decision leave the step run waiting and change nothing, and a decision that belongs to no human step run of the projection and is no operator decision changes nothing, a decision whose subject is no run among them (D135); a decision whose proposalKind is operator-decision is applied by the same handler as AIFACTORY-US-646 delivers it; a human step whose definition declares a policy guard is announced like any human step, the guard being kept in PolicyGuardJson and not evaluated before I4 (D133). | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2012 CancelWorkflowRun inbox handler closing the projection with WorkflowRunCancelled |
implementation | The handler of RunCancelled moves a Running, WaitingForTimer or WaitingForHuman workflow run to Cancelled, moves every open step run to Cancelled, skips pending timers and writes WorkflowRunCancelled with the number of open step runs in the same transaction, idempotently on the event id; a cancellation for a terminal or unknown run changes nothing, and connectsoft.factory.workflows.runs_active moves with the state. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2013 Verify apply-decisions-and-close-on-cancellation through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-WorkflowEngineService-014"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-503 against the fake Run Control of AIFACTORY-TASK-2002, including the Approved, Rejected and ChangesRequested decisions on a human step, the Expired decision followed by a re-request and a later Approved decision, the Expired decision without a re-request, the Stale and the Cancelled decision, each of which leaves the step run waiting and writes no WorkflowRunFailed, the decision of a lease-holder approval and of a spend approval that change nothing, the cancellation from each of the three states, the late timer as a no-op and the publication and payload of WorkflowRunCancelled; consumer handling is proven by the suite of DashboardService in its increment. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-1496: Let a human step wait on the approval of the promotion gate of EnvironmentPromotionService instead of opening an approval of its own¶
behavior · persona reviewer
As a reviewer, I want to give one production approval, on the promotion gate, and have the workflow follow it, so that a production release is never approved twice, once in the workflow and once on the promotion, and never on two different decisions.
The owner decided that a person gives one production approval: the gate of EnvironmentPromotionService, bound to the release id and the environment, and that the workflow step approve-production of the release and delivery-run definitions waits on that approval (D133). Such a human step declares in its definition that it waits on the promotion gate, which the engine keeps in WaitsOn of its step definition. When the Advance reaches it, the engine opens no approval, emits no StepReady for it, so Run Control opens no approval request for it either, records the step run as waiting and moves the run from Running to WaitingForHuman through HumanStepReached. The approval of the gate is requested by EnvironmentPromotionService with its own service identity for the promotion the earlier step of the run requested, and RunControlService announces the decision as RunControlService/ApprovalDecided with the subject kind promotion and the promotion as subject (D135). The inbox handler AdvanceOnDecision applies such a decision to the step that waits on it, idempotently on the event id: an Approved gate completes the step run as Succeeded, returns the run to Running through DecisionApplied and the next ready steps are announced; a Rejected or an Expired gate fails the step run with the failure class policy-denied and the failure policy the step declares applies, Fail for both definitions, so WorkflowEngineService/WorkflowRunFailed is written (AIFACTORY-US-501). As a design choice of this PBI a ChangesRequested, a Stale and a Cancelled decision of the gate leave the step waiting, as for every human step. The decision is matched to the waiting step by the run of the event and by the promotion id that the output of the step which requested the promotion recorded, request-promotion of delivery-run among them, compared with the subject id of the event; the model binds the gate to the release id and the environment and does not name that key, which is reported to the owner. A decision of subject kind promotion that matches no waiting step changes nothing. A step whose when guard does not hold, approve-production of delivery-run for a target that is not production, is recorded Skipped as every guarded step and waits on nothing. Cancellation and the stuck-run recovery treat the waiting step as any open human step run (AIFACTORY-US-503, AIFACTORY-US-502); no StepReady is ever re-emitted for it.
Acceptance criteria
| AC1 | Given a Running WorkflowEngineService/WorkflowRun of a fixture definition shaped like release/approve-production, whose Advance reaches a human step that waits on the promotion gate, when it is applied, then no WorkflowEngineService/StepReady is written for the step, no task contract is stored, the step run is recorded as waiting with its WaitsOn, and the run moves to WaitingForHuman through HumanStepReached in the same transaction (D133, model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC2 | Given that wait and a RunControlService/ApprovalDecided with the subject kind promotion, the run of the step and the promotion id the earlier step recorded, when AdvanceOnDecision runs, then an Approved gate completes the step run as Succeeded, returns the run to Running through DecisionApplied and announces the next ready steps; a Rejected or an Expired gate fails the step run with the failure class policy-denied and, under the failure policy Fail, WorkflowEngineService/WorkflowRunFailed is written; a redelivery of the same event id changes nothing (D133, ADR-0014). |
| AC3 | Given that wait, when a RunControlService/ApprovalDecided of the gate with the state ChangesRequested, Stale or Cancelled, or one of subject kind promotion for another promotion, arrives, then the step run keeps waiting, the run stays WaitingForHuman and nothing is published; given a decision of subject kind run for the run, which belongs to no step that waits on the gate, then the step is not completed by it (D135, model/services/control-plane/RunControlService.yaml). |
| AC4 | Given a fixture definition shaped like delivery-run/approve-production whose when guard does not hold, when the Advance reaches the step, then it is recorded Skipped with WorkflowEngineService/StepRunStateChanged and its dependants treat it as satisfied; given a waiting run that RunControlService/RunCancelled names, then the step run is cancelled as AIFACTORY-US-503 states, and the stuck-run recovery re-emits no WorkflowEngineService/StepReady for the waiting step (FR-06, model/workflows/delivery-run.yaml). |
| Traces: services | WorkflowEngineService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun |
| Traces: events | RunControlService/ApprovalDecided, RunControlService/RunCancelled, WorkflowEngineService/StepReady, WorkflowEngineService/StepRunStateChanged, WorkflowEngineService/WorkflowRunFailed |
| Traces: steps | release/approve-production, delivery-run/approve-production |
| Traces: adrs | ADR-0014 |
| Traces: requirements | FR-06 |
| Depends on | AIFACTORY-US-501, AIFACTORY-US-502, AIFACTORY-US-503, feature:run-control-service |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/workflows/release.yaml, model/workflows/delivery-run.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-6080 Wait of a human step on the promotion gate in the planner and in AdvanceOnDecision |
implementation | The planner records a human step whose definition declares waits_on the promotion gate as a waiting step run, keeps the gate in the WaitsOn column of the step definition, writes no StepReady and stores no task contract for it and moves the run to WaitingForHuman; AdvanceOnDecision matches an ApprovalDecided of subject kind promotion to that step by the run and by the promotion id the output of the requesting step recorded, completes it on Approved, fails it with the failure class policy-denied on Rejected and Expired under its declared failure policy, leaves it waiting on ChangesRequested, Stale and Cancelled and on a decision for another promotion, idempotently on the event id; the StuckRunReconciler re-emits nothing for such a step (D133). | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-6081 Fixture definitions of a human step that waits on the promotion gate |
fixture | The acceptance suite holds two fixture definitions in the shape of WorkflowDefinition, one with a human step that waits on the promotion gate after a step whose output records a promotion id, as release/approve-production, and one where that step carries a when guard on a production target, as delivery-run/approve-production, with decision scripts of the fake Run Control that publish ApprovalDecided of subject kind promotion in each state, for the matching promotion and for another one. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-6082 Verify wait-on-the-promotion-gate through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=PromotionGateWait"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1496 with the fixtures of AIFACTORY-TASK-6081: no StepReady for the waiting step, the Approved gate that completes it, the Rejected and the Expired gate that fail the run, the states and the other promotion that leave it waiting, the redelivery, the Skipped step of a target that is not production, the cancellation and the stuck-run recovery that re-emits nothing. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-646: Leave an escalated or uncertain step to the decision of the Operator, compensate when the Operator chooses it, and close the projection of a run that Run Control failed¶
behavior · persona operator
As an operator, I want the engine to wait for my choice on a step that failed under Escalate or ended Uncertain on a write step, and then to do exactly what I chose, so that no write is repeated blindly and no run is failed or compensated without my decision.
A step that fails terminally under the failure policy Escalate is not settled by rule, and neither is a step of class shared-write or above whose attempt ends Uncertain, whatever policy that step declares. For both RunControlService opens an operator decision for the Operator of the project: an ApprovalRequest of kind operator-decision that Run Control opens itself, that names the run and the step and that is decided with one of the choices retry, compensate and fail. While the request is pending the run waits in WaitingForApproval in Run Control and the step is not settled, so no RunControlService/StepSettled arrives for it (model/services/control-plane/RunControlService.yaml). Run Control knows the policy because StepReady carries failurePolicy (AIFACTORY-US-499). Opening, deciding and expiring the request is the delivery of feature run-control-service; this PBI delivers the side of the engine, which decides nothing here and follows the decision it hears. While the decision is pending the engine keeps the Failed, TimedOut or Uncertain outcome on the step run, which stays open (AIFACTORY-US-501). It does not retry the step, re-emits no StepReady for it, since its reconciler re-emits only for step runs without a recorded outcome (AIFACTORY-US-502), starts no compensation, fails no step run and writes no terminal event, and the WorkflowRun keeps its state: the engine has no state of its own for this wait, as it has none for Held. After the choice retry Run Control dispatches a new attempt of the same step; its outcome and its StepSettled arrive as any other and are applied as AIFACTORY-US-500 and AIFACTORY-US-501 specify, and the RunControlService/ApprovalDecided that carries the choice retry changes nothing in the engine by itself. After the choice compensate Run Control publishes ApprovalDecided with proposalKind operator-decision, choice compensate and the stepRunId of the step the decision is about, and settles the step Failed. The inbox handler AdvanceOnDecision reads the choice, idempotently on the event id, and starts the compensation of the run for that step run: the run moves from Running to Compensating through OperatorChoseCompensate, one CompensationEntry is planned for each succeeded step that declares a compensation step, in reverse completion order, WorkflowEngineService/CompensationStarted is written in the same transaction, each compensation step is announced through StepReady, and the compensation ends as AIFACTORY-US-501 specifies, with WorkflowEngineService/WorkflowRunFailed as the terminal report of the engine. The StepSettled with the final state Failed that Run Control publishes for the same step moves the step run to Failed, whichever of the two events arrives first, and the engine applies no declared policy to it: after the choice compensate or fail the engine follows what it hears from Run Control and applies no policy of its own to that step (decision D121). For an operator decision that an Uncertain outcome of a tool call caused, ApprovalDecided also carries the finding the Operator recorded on the effect, applied or not-applied, which Run Control records in ToolAdapterService; the engine reads the choice and never the finding. The engine reads the rule on the declared policy from its projection alone: for a step whose policy is Escalate, and for a step of class shared-write or above whose last recorded outcome is Uncertain, what follows a failure is the choice of the Operator and never the policy the step declares, so a step that declares Fail and reached the decision by an Uncertain outcome is compensated when the Operator chose compensate. After the choice fail, and when the request expires, which Run Control never requests again, Run Control settles the step Failed, fails the run itself and publishes RunControlService/RunFailed. Following that event is the second delivery of this PBI, and it holds for every run that Run Control fails itself, the run failures that no failure policy of a step decides (decision D121): after the choice fail or an expired operator decision, after a rejected or finally expired approval, after a denied budget and after a retired environment. After a step that failed under the policy Fail or Compensate Run Control fails nothing itself and waits for the WorkflowRunFailed of the engine (AIFACTORY-US-501). The inbox handler CloseFailedWorkflowRun closes the projection, idempotently on the event id: a WorkflowRun that is Running, WaitingForTimer or WaitingForHuman moves to Failed through the modelled trigger RunFailed, its open step runs move to Cancelled with WorkflowEngineService/StepRunStateChanged (decision D121), the state the cancellation of a run gives them (AIFACTORY-US-503), pending timers are skipped so that a reminder that still fires is a no-op, and connectsoft.factory.workflows.runs_active moves with the state. The engine writes no WorkflowRunFailed for such a run, because it publishes that event only for a failure it decided itself and Run Control already holds the terminal state; StudioBff and DashboardService learn the end of the run from RunFailed itself, which both consume. A RunFailed for a run that is already Completed, Failed or Cancelled, as after the WorkflowRunFailed of the engine that Run Control recorded, for a run the projection does not hold, and for a run that is Compensating, for which the model lists no such transition, changes nothing. When the failed run is the child run of a step of kind SubWorkflow, a second handler of the same event returns the failure to the parent step run (AIFACTORY-US-647). One point is not stated by the model and is reported to the owner: an Uncertain outcome on a step of class read or sandbox opens no operator decision; the engine leaves its step run open until StepSettled names it, and what Run Control does with such an outcome is its own rule.
Acceptance criteria
| AC1 | Given a Running WorkflowEngineService/WorkflowRun, a step that declares the failure policy Escalate whose last RunControlService/AttemptOutcomeRecorded is Failed or TimedOut, or a step of class shared-write or above whose outcome is Uncertain, and no RunControlService/StepSettled for its step run, as while Run Control keeps an operator decision pending, when the engine, its timers and its reconciler run beyond the step timeout, then the step run stays open with the outcome recorded, no WorkflowEngineService/StepReady is emitted or re-emitted for it, no compensation starts, neither WorkflowEngineService/WorkflowRunFailed nor another terminal event is written and the run does not move (model/services/control-plane/RunControlService.yaml, ADR-0014). |
| AC2 | Given that wait and a RunControlService/ApprovalDecided with proposalKind operator-decision and the choice retry, when the fake Run Control dispatches a new attempt of the same step and its Succeeded outcome and its RunControlService/StepSettled with the final state Succeeded arrive, then the decision changed nothing in the engine by itself, the step run is Succeeded with both attempts recorded, and the next ready steps are announced through WorkflowEngineService/StepReady as for any other step (XL-06). |
| AC3 | Given that wait, succeeded steps of which one declares a compensation step, and a RunControlService/ApprovalDecided with proposalKind operator-decision, the choice compensate and the stepRunId of the step, when the handler AdvanceOnDecision runs, then the run moves from Running to Compensating through OperatorChoseCompensate, one compensation entry is planned per succeeded step with a compensation step in reverse completion order, WorkflowEngineService/CompensationStarted is written in the same transaction and each compensation step is announced through WorkflowEngineService/StepReady with its own action class; the RunControlService/StepSettled with the final state Failed for the same step run, before or after the decision, moves the step run to Failed and applies no declared policy, also for a step that declares Fail and reached the decision by an Uncertain outcome; and a redelivery of the decision changes nothing. |
| AC4 | Given that wait and a RunControlService/RunFailed for the run, as Run Control publishes it after the choice fail and after an operator decision that expired, when the handler CloseFailedWorkflowRun runs, then the run moves to Failed through the trigger RunFailed, its open step runs move to Cancelled with WorkflowEngineService/StepRunStateChanged, pending timers are skipped, and neither WorkflowEngineService/WorkflowRunFailed nor another terminal event of the engine is written; the RunControlService/ApprovalDecided that carries the choice fail or the state Expired of the operator decision, and the RunControlService/StepSettled with the final state Failed, change nothing more, in whatever order the three events arrive (model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC5 | Given a WorkflowEngineService/WorkflowRun that is Running, WaitingForTimer or WaitingForHuman and a RunControlService/RunFailed for its runId, as after a rejected approval or a denied budget, when the handler CloseFailedWorkflowRun runs, then the projection is closed as Failed from each of the three states with pending timers skipped and no WorkflowEngineService/WorkflowRunFailed written; a RunFailed for a run that is already Completed, Failed or Cancelled, as after the engine's own WorkflowRunFailed, for a run that is Compensating or for a run unknown to the projection, a redelivery of the same event id, and a timer that fires after the closure change nothing (FR-10, ADR-0014). |
| Traces: services | WorkflowEngineService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, RunControlService/ApprovalRequest, RunControlService/Step |
| Traces: events | WorkflowEngineService/CompensationStarted, WorkflowEngineService/StepReady, WorkflowEngineService/StepRunStateChanged, WorkflowEngineService/WorkflowRunFailed, RunControlService/AttemptOutcomeRecorded, RunControlService/StepSettled, RunControlService/ApprovalDecided, RunControlService/RunFailed |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0024 |
| Traces: requirements | FR-10, XL-06 |
| Relates: screens | workflow-run-graph, run-detail |
| Relates: nfrs | NFR-05, NFR-06 |
| Depends on | AIFACTORY-US-500, AIFACTORY-US-501, AIFACTORY-US-502, AIFACTORY-US-503, feature:run-control-service, feature:contracts-package |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md, docs/planes/agent-runtime/services/workflow-engine-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2601 Hold an escalated or uncertain step open and start the compensation on the choice compensate of an operator decision |
implementation | A step run whose step declares Escalate, or whose step is of class shared-write or above with a last outcome Uncertain, stays open after a failed or Uncertain outcome: the engine retries nothing, re-emits no StepReady, starts no compensation and writes no terminal event; the handler of ApprovalDecided recognises an operator decision by proposalKind operator-decision and matches it to its step run by stepRunId, idempotently on the event id; the choice compensate moves the run from Running to Compensating through OperatorChoseCompensate, plans the compensation entries with the plan of AIFACTORY-TASK-2005 and writes CompensationStarted in the same transaction; the choices retry and fail and an expired request change nothing by themselves, and the finding an operator decision on an uncertain tool effect carries is not read; the StepSettled with the final state Failed of such a step moves the step run to Failed with its attemptCount and failureClass and applies no declared policy, in either order with the decision, so that after the choice compensate or fail the engine does only what it hears from Run Control. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2602 CloseFailedWorkflowRun inbox handler closing the projection on RunFailed without a terminal event of the engine |
implementation | The handler of RunFailed moves a Running, WaitingForTimer or WaitingForHuman workflow run to Failed through the trigger RunFailed, moves every open step run to Cancelled with StepRunStateChanged, skips pending timers and writes no WorkflowRunFailed, idempotently on the event id; a RunFailed for a Completed, Failed, Cancelled, Compensating or unknown run changes nothing, a timer that fires after the closure is a no-op, and connectsoft.factory.workflows.runs_active moves with the state. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2603 Fixture definitions that declare Escalate and a write step without an idempotency key |
fixture | The fixture set of AIFACTORY-TASK-1995 gains definitions in the wire shape of WorkflowDefinition: a step with failurePolicy Escalate that runs after two succeeded steps of which one names a compensation step, and a step of class shared-write with failurePolicy Fail and no idempotencyKey; with the operator decision scripts of the fake Run Control of AIFACTORY-TASK-2002 they reach the pending decision, each of the three choices and the expiry; no definition of model/workflows is changed, since none of them declares Escalate. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2604 Engine side of the settlement contract as a consumer contract test of StepSettled, RunFailed and the operator decision fields of ApprovalDecided |
contract | A consumer contract test in the WorkflowEngineService repository pins the event types of the contracts package as Run Control publishes them and the engine reads them: StepSettled with runId, tenantId, stepRunId, stepId, finalState, attemptCount, failureClass and outputRef; RunFailed with runId, tenantId, failureClass and reason; and ApprovalDecided with proposalKind, choice, stepRunId and finding beside its other fields; a change of one of these payloads fails the build of the consumer; publishing the events is the delivery of feature run-control-service and is not built here. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2605 Verify follow-operator-decisions-and-close-failed-runs through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=follow-operator-decisions-and-close-failed-runs"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-646 against the fake Run Control of AIFACTORY-TASK-2002: the step under Escalate and the Uncertain write step that stay open beyond the step timeout with nothing announced and nothing failed, the choice retry followed by a new attempt, the choice compensate with the compensation in reverse order and the step settled Failed without a declared policy applied, in both orders of the two events, the choice fail and the expired decision closed by RunFailed without WorkflowRunFailed, the closure on RunFailed from each of the three states, and the RunFailed for a terminal, a Compensating and an unknown run that changes nothing. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-647: Execute a step of kind SubWorkflow as a child run submitted through Run Control and return its result to the parent step run¶
behavior
A step of kind SubWorkflow runs another workflow as a child run. The engine does not announce such a step through StepReady, since no agent, tool or human acts in it and Run Control creates no step and no attempt for it; the engine executes the step itself by asking Run Control, the single writer of runs (ADR-0014), for a child run. On the wire the kind is sub-workflow, and the step names its child workflow in workflow, as model/schemas/workflow.schema.json declares it and model/workflows/sprint-execution.yaml uses it in its step run-item. The published wire schema does not carry that property yet: the contract task of this PBI adds workflow to a step of WorkflowDefinition in docs/reference/schemas/asset-manifests.schema.json, the contracts package regenerates its types (feature contracts-package) and the service takes them by a new pin, the path the properties of AIFACTORY-US-498 took. The validator of AIFACTORY-US-498 gains one rule: a step of kind sub-workflow that names no child workflow is refused with a finding by step and rule, through WorkflowEngineService/ValidateWorkflowDefinition and at load. When the planner finds a step of kind SubWorkflow ready, the Advance creates its StepRun and calls RunControlService/RunControl.SubmitChildRun, the internal gRPC operation of Run Control, with a body of run-submission-v1 that carries the workflowId of the child, the input built from the input mapping of the step, the modelProfileId the parent run was accepted with and parentRunId, the run the step belongs to. The run is the parent: no attempt exists for a step the engine executes, so the body carries no parentAttemptId. The engine chooses neither tenant nor project nor a policy revision nor a release: the child inherits the tenant, the project, the policy revision and the pinned release closure of the parent, all applied by Run Control. The child gets no reservation of its own: its usage is recorded with the child run id and counts on the reservation of the parent run, and the submission waits for no budget reservation and opens no spend approval (decision D121). The call presents the workload identity of the engine. The same idempotency key with the same input returns the same child run, and the key of a step of kind SubWorkflow is the stepRunId (decision D121), which is unique per step and iteration, so a redelivered event or a restarted Advance never creates a second child. The reply run-accepted-v1 names the child run; the engine keeps its id in childRunId of the step run and moves the step run to Running with WorkflowEngineService/StepRunStateChanged, in the transaction of the Advance. Run Control is a fail-fast dependency of this call: when it fails the Advance fails, nothing is written and the inbox redelivers the event that made the step ready, which repeats the call under the same key. One answer is no failure of the dependency: under a parent run that is terminal or Cancelling Run Control answers FAILED_PRECONDITION and no child run exists (decision D121). On that answer the engine keeps no childRunId on the step run and does not repeat the call (decision D123); it announces nothing, and the step run is closed with its projection when RunFailed or RunCancelled of the parent arrives (AIFACTORY-US-646, AIFACTORY-US-503). The child run is a run like any other: Run Control publishes RunAccepted for it and the engine opens a second projection and drives it from its own pinned definition (AIFACTORY-US-499). Its result returns to the parent through the terminal run events of Run Control and through nothing else. The inbox handler ReturnChildRunResult consumes RunControlService/RunCompleted, RunControlService/RunFailed and RunControlService/RunCancelled, idempotently on the event id, looks for the step run whose childRunId equals the runId of the event and ignores every run that is no child. For RunCompleted the parent step run moves to Succeeded with the result of the child, which the engine reads through the optional outputRef of the event, the reference of the outcome of the child run in ArtifactService, and keeps as the outputRef of the step run (decision D121), and the Advance of the parent computes what is ready next. For RunFailed the parent step run moves to Failed with the failureClass of the child, and the failure policy the step declares decides what follows, as for any other step settled Failed (AIFACTORY-US-501). For RunCancelled the parent step run moves to Cancelled, and nothing is announced and nothing fails. Two cancellations are a failure for the parent (decisions D121, D123 and D125): a child run stopped for an overrun of its budget arrives as RunCancelled with the reason budget-exceeded, and a child run stopped because a release of its pinned closure was revoked arrives with the reason release-revoked; each is a failed child, so the parent step run of kind SubWorkflow is set to Failed with the failure class policy-denied and keeps the reason of the stop, budget-exceeded or release-revoked, in failureReason, because the model has no failure class of its own for either, and the failure policy of the step applies as after RunFailed; every other step run keeps no failure reason; under the policy Fail the WorkflowRunFailed of the parent carries the failure class policy-denied. The stopped child itself ends Cancelled and not Failed, and only its parent treats it as failed. The same events also close the projection of the child itself, through CloseFailedWorkflowRun and CancelWorkflowRun, the other handlers of these events (AIFACTORY-US-646, AIFACTORY-US-503). In the other direction Run Control cancels the open child runs of a parent that ends Failed or Cancelled, and the engine follows each of those cancellations as it follows any other. While the child runs the parent step run waits: a step timeout fails nothing and the StuckRunReconciler re-emits no StepReady for a step run of kind SubWorkflow (AIFACTORY-US-502). None of the six chain definitions uses the kind, so the acceptance suite proves it on fixture definitions. Three points are not stated by the model and are reported to the owner. run-submission-v1 requires modelProfileId, which RunAccepted of the parent carries and for which the WorkflowRun has no field. The rule on failure policies binds agent, tool and human steps only, while the handler applies the failure policy of the step and run-item of model/workflows/sprint-execution.yaml declares none; the fixtures of this PBI declare one, and what applies to a step of kind sub-workflow without one is open. And a StepRun has a field for the failure class and none for a reason, and WorkflowRunFailed carries no reason, so where the reason budget-exceeded of the parent step run is kept is not stated; the suite asserts the state Failed, the failure class policy-denied and the policy applied.
Acceptance criteria
| AC1 | Given a WorkflowDefinition document of docs/reference/schemas/asset-manifests.schema.json, the shape of workflow-definition-v1, with a step of kind sub-workflow, when WorkflowEngineService/ValidateWorkflowDefinition is called or the release is loaded, then a step that names its child workflow in workflow passes, and a step that names none is refused, 400 with a problem response that names the finding by step and rule at validation and a release that is not loaded at load, with connectsoft.factory.workflows.definition_rejections counting the rejection (model/schemas/workflow.schema.json). |
| AC2 | Given a Running WorkflowEngineService/WorkflowRun whose Advance finds a step of kind SubWorkflow ready, when it is applied, then no WorkflowEngineService/StepReady is written for the step, RunControlService/RunControl.SubmitChildRun is called once with a run-submission-v1 body that carries the workflowId of the child, the input of the step and parentRunId equal to the run, without parentAttemptId and under the idempotency key stepRunId, the id of the child run named by the reply run-accepted-v1 is kept in childRunId of the step run, and the step run moves to Running with WorkflowEngineService/StepRunStateChanged in the same transaction (ADR-0014). |
| AC3 | Given RunControlService/RunControl.SubmitChildRun fails, or the event that made the step ready is delivered again, or the engine restarts between the call and its commit, when the Advance runs again, then the call is repeated under the same idempotency key, the fake Run Control holds exactly one child run for the step run, a failed call leaves no step run and no event written, and the RunControlService/RunAccepted of the child opens a second projection that the engine drives like any other run; given the call answers FAILED_PRECONDITION because the parent run is terminal or Cancelling, then no child run exists, no childRunId is kept, the call is not repeated and nothing is announced (FR-03). |
| AC4 | Given a step run with a childRunId and a RunControlService/RunCompleted whose runId equals it, when the handler ReturnChildRunResult runs, then the parent step run is Succeeded with the result of the child, its outputRef equal to the outputRef of the event, the next ready steps of the parent are announced through WorkflowEngineService/StepReady, and a RunCompleted of a run that is the child of no step run, and a redelivery of the same event id, change nothing. |
| AC5 | Given a step run with a childRunId and a RunControlService/RunFailed that names the child, when the handler ReturnChildRunResult runs, then the parent step run is Failed with the failureClass of the child and the declared failure policy of the step applies as for a step settled Failed, policy Fail giving WorkflowEngineService/WorkflowRunFailed of the parent with the step as failedStepId; given a RunControlService/RunCancelled that names the child, then the parent step run is Cancelled, nothing is announced and no WorkflowRunFailed is written, unless its reason is budget-exceeded, the stop of a child for an overrun of its budget, or release-revoked, the stop of a child whose pinned release was revoked (D125), for which the parent step run is Failed with the failure class policy-denied and that reason in failureReason and the declared failure policy applies as after RunFailed, policy Fail giving WorkflowEngineService/WorkflowRunFailed of the parent with the failure class policy-denied; and in every case the projection of the child itself is closed by the other handler of the event. |
| Traces: services | WorkflowEngineService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, WorkflowEngineService/WorkflowDefinitionVersion, RunControlService/Run |
| Traces: apis | RunControlService/RunControl.SubmitChildRun, WorkflowEngineService/ValidateWorkflowDefinition |
| Traces: events | WorkflowEngineService/StepReady, WorkflowEngineService/StepRunStateChanged, WorkflowEngineService/WorkflowRunFailed, RunControlService/RunAccepted, RunControlService/RunCompleted, RunControlService/RunFailed, RunControlService/RunCancelled |
| Traces: descriptors | run-submission-v1, run-accepted-v1, workflow-definition-v1 |
| Traces: adrs | ADR-0014, ADR-0024 |
| Traces: requirements | FR-03 |
| Relates: screens | workflow-run-graph |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-498, AIFACTORY-US-500, AIFACTORY-US-501, AIFACTORY-US-503, AIFACTORY-US-646, feature:run-control-service, feature:contracts-package |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/services/control-plane/RunControlService.yaml, model/schemas/workflow.schema.json, model/workflows/sprint-execution.yaml, docs/reference/schemas/asset-manifests.schema.json, docs/reference/schemas/run-submission.schema.json, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2606 Add the child workflow reference of a sub-workflow step to WorkflowDefinition of the published asset manifest schema |
contract | docs/reference/schemas/asset-manifests.schema.json declares on a step of WorkflowDefinition the property workflow, the id of the child workflow of a step of kind sub-workflow, mirroring workflow of model/schemas/workflow.schema.json; model/workflows/sprint-execution.yaml with its step run-item can be written as a document that is valid against it, the reference page of the schema shows the property, and python -I tools/docs_model validate passes. |
ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-2607 Publish the contracts package version that carries the child workflow reference and the parent fields of the run submission |
contract | The contracts package regenerates its WorkflowDefinition types from docs/reference/schemas/asset-manifests.schema.json as AIFACTORY-TASK-2606 changed it and its run submission type from docs/reference/schemas/run-submission.schema.json with parentRunId and parentAttemptId, carries the event type RunCompleted with its optional outputRef as the model states it, stamps them with the hash of each schema and is published to the ConnectSoft feed at a new version, the way feature contracts-package builds and publishes it; the repository ConnectSoft.AI.SoftwareFactory.WorkflowEngineService pins that version. |
ConnectSoft.AI.SoftwareFactory.Contracts |
AIFACTORY-TASK-2608 Validation rule of a sub-workflow step and the submission of its child run through RunControl.SubmitChildRun |
implementation | The validator of AIFACTORY-TASK-1993 refuses a step of kind sub-workflow that names no child workflow, with a unit test of the rule; when the planner of AIFACTORY-TASK-197 returns a ready step of kind SubWorkflow, the Advance creates its step run, writes no StepReady and calls RunControl.SubmitChildRun with the workload identity of the engine, a run-submission-v1 body with the workflowId of the child, the input of the step, the modelProfileId of the parent and parentRunId, no parentAttemptId, and the stepRunId as idempotency key; it keeps the child run id of the reply in ChildRunId and moves the step run to Running with StepRunStateChanged in one transaction; a failed call fails the Advance without a write, the answer FAILED_PRECONDITION of a parent run that is terminal or Cancelling keeps no child run id and is not repeated, the engine sets no budget and asks for no reservation for the child, and the StuckRunReconciler re-emits nothing for a step run of kind SubWorkflow. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2609 ReturnChildRunResult inbox handler for RunCompleted, RunFailed and RunCancelled of a child run |
implementation | The handler matches the runId of RunCompleted, RunFailed and RunCancelled to the step run that keeps it in ChildRunId, idempotently on the event id, and ignores a run that is no child; RunCompleted moves the parent step run to Succeeded with the result of the child, read through the outputRef of the event and kept as the outputRef of the step run, and lets the Advance of the parent announce the next ready steps; RunFailed moves it to Failed with the failureClass of the child and applies the declared failure policy of the step through the path of AIFACTORY-TASK-2004 and AIFACTORY-TASK-2005; RunCancelled moves it to Cancelled and announces nothing, except with the reason budget-exceeded or release-revoked (D125), for which the child is a failed child, the parent step run moves to Failed with the failure class policy-denied and keeps that reason in FailureReason, and the same policy path applies; the handlers that close the projection of the child itself run beside it on the same events. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2610 Fixture definitions of a parent workflow with a sub-workflow step and of its child workflow |
fixture | The fixture set of AIFACTORY-TASK-1995 gains, in the wire shape of WorkflowDefinition, a parent definition whose step of kind sub-workflow names a child definition in workflow and declares a failure policy, once Fail and once Compensate, the child definition itself with two agent steps, and one invalid variant whose sub-workflow step names no child; the fake registry serves the releases of both, and the fake Run Control of AIFACTORY-TASK-2002 answers SubmitChildRun, on request with FAILED_PRECONDITION as for a parent run that is terminal or Cancelling, and publishes RunAccepted with the parent run id and the terminal run events of the child, RunCompleted with an outputRef and RunCancelled without a reason and with the reasons budget-exceeded and release-revoked among them; no definition of model/workflows is changed. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2611 Verify execute-a-sub-workflow-step-as-a-child-run through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=execute-a-sub-workflow-step-as-a-child-run"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-647 against the fake Run Control of AIFACTORY-TASK-2002: the sub-workflow step without a child workflow refused at validation and at load, the single call of SubmitChildRun with parentRunId and the stepRunId as key and no StepReady for the step, the repeated call after a failure and after a redelivery that leaves one child run, the FAILED_PRECONDITION of a terminal and of a Cancelling parent run after which no child run id is kept and the call is not repeated, the second projection opened for the child, the parent step run Succeeded on RunCompleted with the outputRef of the event and the next steps announced, Failed on RunFailed with the declared policy applied, Cancelled on RunCancelled, Failed with the failure class policy-denied, the reason kept in failureReason and the declared policy applied on a RunCancelled with the reason budget-exceeded and on one with the reason release-revoked, and the terminal event of a run that is no child ignored. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-504: Read the engine projection of a run as a graph through REST for the Studio and through gRPC for internal callers¶
behavior · persona requester
As a requester, I want to see the graph of my run with the state of every step run, its timers and its compensation entries, so that I can tell where the workflow stands and why it waits, without reading two services.
The engine projection of a run is readable in two ways and neither writes anything. A caller whose scope carries factory.runs.read reads the graph of one run through WorkflowEngineService/GetWorkflowRunGraph (GET /runtime/workflow-runs/{runId}), the read model of the workflow run graph screen of the Studio: the state of the workflow run with definitionReleaseId, graphHash and lastOutcomeSequence, every step run with its step id, kind, state, iteration, attempt ids, the number of attempts and, for a failed step, the failure class it keeps from StepSettled (decision D121), output reference, approval id, the id of the child run of a step of kind SubWorkflow and start and end times, the timer entries with their due time and whether they fired, and the compensation entries with their order and state. The steps and edges of the pinned definition come from the cached WorkflowDefinitionVersion, so the caller sees the whole graph and not only the steps that already ran. The read runs under the tenant row filter: a run of another tenant is answered 404, the same answer as for a runId that does not exist, and the response never confirms that it exists (ADR-0015); a caller without the scope is answered 403. The response states the projection of the engine, which is not the authority: the authoritative state of the run, its steps and its attempts is read from RunControlService (ADR-0014), and the two agree once the outcomes Run Control recorded are applied, which lastOutcomeSequence lets a reader judge. Internal callers read the same projection through the gRPC method WorkflowEngineService/WorkflowEngine.GetRunGraph, of class read: RunControlService for its projections, and the Studio backend-for-frontend once it is delivered in a later increment. The caller presents a workload token, and the engine reads the tenant and project the caller forwards only when the token holds the service scope that names the engine. A call without a workload identity is refused with UNAUTHENTICATED, a token without the service scope with PERMISSION_DENIED, a request without a run id with INVALID_ARGUMENT, and a run that does not exist in the forwarded tenant, a run of another tenant included, with NOT_FOUND. When workflow-engine-sql is unavailable the read fails and no partial graph is returned. The model names no descriptor for the REST response and names run-event-envelope-v1 as the request and the reply descriptor of the gRPC method; the shape of the graph is therefore fixed by the OpenAPI and proto documents of the service and that naming is reported to the owner. The route of the REST operation through the Agent Mesh gateway arrives with feature agent-runtime-worker; in this increment the operation is hosted and proven on the service.
Acceptance criteria
| AC1 | Given a caller with factory.runs.read and a WorkflowEngineService/WorkflowRun of the caller's tenant with step runs, a pending timer and compensation entries, when WorkflowEngineService/GetWorkflowRunGraph is called, then the response is 200 with the state of the run, definitionReleaseId, graphHash and lastOutcomeSequence, the steps and edges of the pinned WorkflowEngineService/WorkflowDefinitionVersion, every step run with its state, iteration and attempt ids, the timer entries and the compensation entries, and nothing is written (SCN-WorkflowEngineService-002, FUX-04). |
| AC2 | Given a caller without factory.runs.read, or a runId that exists only in another tenant or not at all, when WorkflowEngineService/GetWorkflowRunGraph is called, then the response is 403 for the missing scope and 404 for the foreign run, the same answer as for a runId that does not exist, each with a problem response and no state change (SCN-WorkflowEngineService-002, ADR-0015). |
| AC3 | Given a caller with a workload identity whose token holds the service scope and a run of the tenant it forwards, when WorkflowEngineService/WorkflowEngine.GetRunGraph is called, then the reply carries the current projection of the engine with the same content as the REST read, nothing is written, and two calls with no Advance between them return the same projection (SCN-WorkflowEngineService-005, ADR-0014). |
| AC4 | Given a caller without a workload identity, a token without the service scope, a request without a run id, or a run that is unknown or belongs to another tenant, when WorkflowEngineService/WorkflowEngine.GetRunGraph is called, then the call is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT or NOT_FOUND respectively, the foreign run answered exactly as the unknown one, and no state changes (SCN-WorkflowEngineService-005, ADR-0015). |
| Traces: services | WorkflowEngineService, RunControlService |
| Traces: aggregates | WorkflowEngineService/WorkflowRun, WorkflowEngineService/WorkflowDefinitionVersion |
| Traces: apis | WorkflowEngineService/GetWorkflowRunGraph, WorkflowEngineService/WorkflowEngine.GetRunGraph |
| Traces: descriptors | run-event-envelope-v1 |
| Traces: adrs | ADR-0014, ADR-0015 |
| Traces: requirements | FUX-04, FUX-11 |
| Traces: scenarios | SCN-WorkflowEngineService-002, SCN-WorkflowEngineService-005 |
| Relates: screens | workflow-run-graph |
| Depends on | AIFACTORY-US-496, AIFACTORY-US-499 |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/ux/screens.yaml, docs/planes/agent-runtime/services/workflow-engine-service/index.md#apis |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2014 GetWorkflowRunGraph read model under the tenant row filter |
implementation | GetWorkflowRunGraph returns the state of the workflow run with definitionReleaseId, graphHash and lastOutcomeSequence, the steps and edges of the pinned definition, every step run, the timer entries and the compensation entries; it applies the tenant row filter so a run of another tenant is answered 404, the same answer as for an id that does not exist (ADR-0015), a missing scope is 403, and nothing is written. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2015 WorkflowEngine.GetRunGraph over the same projection for internal callers |
implementation | WorkflowEngine.GetRunGraph returns the same projection as the REST read for a run of the forwarded tenant and writes nothing; callers without workload identity are refused with Unauthenticated, tokens without the service scope with PermissionDenied, requests without a run id with InvalidArgument, and an unknown run and a run of another tenant alike with NotFound, the gRPC status NOT_FOUND (ADR-0015). | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2016 Contract of the run graph in the OpenAPI and proto documents |
contract | The OpenAPI document and the proto of the service describe one shape of the run graph for GetWorkflowRunGraph and WorkflowEngine.GetRunGraph, built from the fields of WorkflowRun, StepRun, TimerEntry, CompensationEntry, StepDefinition and EdgeDefinition in model/services/agent-mesh/WorkflowEngineService.yaml; a contract test fails when the hosted reply and the documents differ. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2017 Verify read-the-workflow-run-graph through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-WorkflowEngineService-002\|Scenario=SCN-WorkflowEngineService-005)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-504, including the 404 for a run of another tenant, the 403 for a missing scope, the gRPC refusal mapping with NOT_FOUND for the unknown and the foreign run alike, and the equality of the REST and the gRPC projection. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-505: Satisfy the members of a readiness bundle by kind, blueprint, signal, artifact and approval, and complete a bundle of several members only when the last required member is satisfied and consistent¶
behavior
A readiness-activated workflow declares in its activation block the member slots it needs, and each slot names its kind, its descriptor and the event with the predicate that satisfies it (ADR-0026 with its amendment). This PBI delivers the member side of the ReadinessCorrelation: how a signal is matched to a slot of each kind, how a bundle of several members fills, and when it is complete. The submission that follows a complete bundle, the superseding and the expiry are AIFACTORY-US-009. The inbox hands each signal to the ReadinessCorrelator, which matches it against the activation members of the loaded definitions within the trusted tenant and project of the signal, and the ReadinessCorrelationGrain serialises the member updates of one correlation, idempotently on correlationId, slot and refHash. The first matching signal for a project and operation opens the correlation in state Open with every slot of the block as a ReadinessMember with its slot, kind, descriptor and required, on the workflow release the alias of the definition resolved to; the resolution is a seam this PBI calls and AIFACTORY-US-009 delivers and proves against the registry. A member reference always carries the id, the version and the hash of the triggering object as the event gives them, and the correlation never resolves latest for a member. A blueprint member is satisfied by BlueprintService/BlueprintValidated only when passed is true and kind equals the blueprint kind of the slot, and only by a version of project scope; it takes blueprintId, version and documentHash. A signal member is satisfied by the event its slot names when the predicate of the slot holds, and takes the id, the version and the hash from the payload fields the member names in ref, version_ref and hash_ref: the slot context-map of module-blueprints takes contextMapId and blueprintVersion of BlueprintService/ContextMapDerived when contexts is above zero, the slot domain-model of database-blueprint takes domainModelId of BlueprintService/DomainModelDerived when aggregates is above zero, and the slot recipe of generate-product takes recipeId and version of GenerationRecipeService/RecipeApproved when approvedBy is set. A part for which the member names no field stays empty on the member. An artifact member is satisfied by ArtifactService/ArtifactStored whose descriptor equals the descriptor of the slot, and takes artifactId, versionInChain and contentHash. An approval member is satisfied by RunControlService/ApprovalDecided only when the predicate of the slot holds, the state Approved with the proposal descriptor of the slot, with proposalHash as its hash, and only when the memo that was approved names the same versions and hashes as the other members of the bundle; a memo that names other versions satisfies nothing. A signal whose predicate does not hold, a version that is not of project scope and a signal of another tenant or project satisfy no member and write nothing. A bundle of several members stays Open until its last required member is satisfied: the bundle of module-blueprints needs the validated Service blueprint and the context map, the bundle of database-blueprint the validated Service blueprint and the domain model, in whatever order they arrive. When the last required member is satisfied and the members are consistent with each other, the correlation moves from Open to Satisfied through LastRequiredMemberSatisfied: the bundle fingerprint is computed as sha256 over the sorted list of slot, id, version and hash of every satisfied required and optional member, the bundle document readiness-bundle-v1 with one reference per slot is stored through ArtifactService/Artifacts.Store, and WorkflowEngineService/ReadinessSatisfied is written in the same transaction. When ArtifactService is unavailable the correlation stays Open with its members, nothing is announced and the work is retried, a branch AIFACTORY-US-009 proves on the first chain. No chain definition of this increment declares an artifact or an approval member; both kinds are proven on a fixture definition that carries the member slots of model/workflows/ux-blueprint.yaml, and events whose producers arrive in later increments are fixture events built from the contracts package. On the wire the activation block of WorkflowDefinition gains the member kind signal with its reference fields through the contract task of AIFACTORY-US-498. Four points are not stated by the model and are reported to the owner. A signal member that names no version_ref or no hash_ref, as the slots context-map and domain-model do, has no hash or no version, so what makes a later signal a newer version of such a member is open. Which payload fields are the id and the version of an approval member is open; this PBI keeps approvalId as the id and leaves the version empty. Through which operation the engine reads the versions and hashes the memo names is open: ApprovalDecided carries proposalReference and the model names no read for it, so this PBI fetches the memo through ArtifactService/Artifacts.Fetch. And the entity ReadinessMember describes its kind as blueprint, artifact or approval, without the kind signal that the definitions and the consume rows of the model use.
Acceptance criteria
| AC1 | Given the loaded definition module-blueprints, whose activation names the required slots service-blueprint and context-map, when a BlueprintService/BlueprintValidated with kind Service and passed true for a version of project scope is handled, then the WorkflowEngineService/ReadinessCorrelation is Open with one satisfied member and no WorkflowEngineService/ReadinessSatisfied is written; when the BlueprintService/ContextMapDerived of the same project then arrives, then the correlation moves to Satisfied through LastRequiredMemberSatisfied, bundleFingerprint is the sha256 over the sorted list of slot, id, version and hash of both members, and WorkflowEngineService/ReadinessSatisfied carries that fingerprint and the bundleRef of the readiness-bundle-v1 document stored through ArtifactService/Artifacts.Store; the same holds for database-blueprint with BlueprintService/DomainModelDerived, and in either order of arrival (SCN-WorkflowEngineService-015, ADR-0026). |
| AC2 | Given a member of kind signal, when its event arrives and the predicate of the slot holds, then the member takes its id, version and hash from the payload fields the slot names: contextMapId and blueprintVersion of BlueprintService/ContextMapDerived for the slot context-map, domainModelId of BlueprintService/DomainModelDerived for the slot domain-model, and recipeId and version of GenerationRecipeService/RecipeApproved for the slot recipe of generate-product, whose bundle of one member is Satisfied by that event alone; a part the slot names no field for stays empty, no latest version is resolved, and an event whose predicate does not hold, with contexts zero, aggregates zero or no approvedBy, satisfies nothing (model/workflows/module-blueprints.yaml, model/workflows/generate-product.yaml). |
| AC3 | Given a fixture definition with the artifact slots of model/workflows/ux-blueprint.yaml, when an ArtifactService/ArtifactStored arrives whose descriptor equals the descriptor of a slot, then that member is satisfied with artifactId, versionInChain and contentHash of the event and names the event that satisfied it; an ArtifactService/ArtifactStored with another descriptor, without a descriptor, or of another tenant or project satisfies no member and writes nothing (ADR-0026, ADR-0015). |
| AC4 | Given the same fixture definition with its approval slot and its blueprint and artifact members satisfied, when a RunControlService/ApprovalDecided arrives with the state Approved and the proposal descriptor of the slot, and the memo it approves, read through ArtifactService/Artifacts.Fetch, names the versions and hashes the other members hold, then the approval member is satisfied with proposalHash and the WorkflowEngineService/ReadinessCorrelation moves to Satisfied; a decision in any other state, with another proposal descriptor, or whose memo names other versions or hashes than the members hold satisfies nothing, and the correlation stays Open (model/services/agent-mesh/WorkflowEngineService.yaml, ADR-0026). |
| AC5 | Given a member signal delivered twice, a BlueprintService/BlueprintValidated with passed false, of another kind or for a version that is not of project scope, or a signal of another tenant or project, when it is handled, then a repeated member is a no-op on correlationId, slot and refHash, and a signal that does not match satisfies no member and writes nothing (SCN-WorkflowEngineService-015, ADR-0015). |
| Traces: services | WorkflowEngineService, BlueprintService, GenerationRecipeService, ArtifactService, RunControlService |
| Traces: aggregates | WorkflowEngineService/ReadinessCorrelation |
| Traces: apis | ArtifactService/Artifacts.Store, ArtifactService/Artifacts.Fetch |
| Traces: events | WorkflowEngineService/ReadinessSatisfied, BlueprintService/BlueprintValidated, BlueprintService/ContextMapDerived, BlueprintService/DomainModelDerived, GenerationRecipeService/RecipeApproved, ArtifactService/ArtifactStored, RunControlService/ApprovalDecided |
| Traces: workflows | module-blueprints, database-blueprint, generate-product |
| Traces: descriptors | readiness-bundle-v1 |
| Traces: adrs | ADR-0015, ADR-0026 |
| Traces: requirements | FR-03 |
| Traces: scenarios | SCN-WorkflowEngineService-015 |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-497, AIFACTORY-US-498, feature:blueprint-service, feature:generation-recipe-service, feature:artifact-service, feature:run-control-service |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/schemas/workflow.schema.json, model/workflows/module-blueprints.yaml, model/workflows/database-blueprint.yaml, model/workflows/generate-product.yaml, model/workflows/ux-blueprint.yaml, docs/architecture/adr/0026-requirements-to-ux-handoff.md, docs/reference/schemas/readiness-bundle.schema.json |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2021 Member matching by kind and the member side of the ReadinessCorrelationGrain up to the Satisfied bundle |
implementation | The ReadinessCorrelator matches each signal against the activation members of the loaded definitions by the predicate of the slot within the tenant and project of the signal: blueprint members by kind, passed and project scope, signal members with id, version and hash from the payload fields the slot names, artifact members by descriptor, approval members by the state Approved, the proposal descriptor and a memo that names the versions and hashes of the other members; the ReadinessCorrelationGrain opens the correlation on the first matching signal with every slot of the block, for the workflow release the resolver seam returns, serialises member updates idempotently on correlationId, slot and refHash, and on the last required member computes the fingerprint over the sorted slot, id, version and hash list, stores the readiness-bundle-v1 document through Artifacts.Store and moves the correlation to Satisfied with ReadinessSatisfied in the same transaction; an unavailable ArtifactService leaves it Open. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2022 Fixture definitions and fixture events for the four member kinds |
fixture | The fixture set of AIFACTORY-TASK-1995 serves the activation blocks of module-blueprints, database-blueprint and generate-product, and gains one fixture definition with the blueprint, approval and artifact slots of model/workflows/ux-blueprint.yaml; fixture events built from the contracts package cover BlueprintValidated, ContextMapDerived, DomainModelDerived, RecipeApproved, ArtifactStored and ApprovalDecided, each in a variant whose predicate holds and one whose predicate does not, and a memo fixture exists in a variant that names the versions and hashes of the members and one that names others; the fake registry answers the resolver seam with a fixed release id. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2023 Verify satisfy-readiness-members-by-kind through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=satisfy-readiness-members-by-kind"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-505 and the given of SCN-WorkflowEngineService-015: the bundles of two members that are Open after the first and Satisfied after the last in either order, the fingerprint over both members, the signal members filled from the payload fields their slots name, the artifact member matched by descriptor, the approval member satisfied only by an Approved decision whose memo names the versions of the members, the signals that satisfy nothing and the repeated member; the branch of an unavailable artifact store is proven by AIFACTORY-TASK-208 of AIFACTORY-US-009. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-009: A validated blueprint starts the next workflow by itself: a readiness correlation joins the signals and submits exactly one run through Run Control¶
behavior · persona requester
As a requester, I want the architecture decomposition to start when my Vision Blueprint is validated, without anyone submitting it, so that the chain moves from one workflow to the next on evidence, not on a person remembering.
Every workflow definition that a person does not submit is readiness-activated (ADR-0026 with its amendment): its activation block names the member slots the workflow needs, each satisfied by an event with a predicate. How a signal satisfies a member of each kind, and when a bundle of several members is complete, is delivered by AIFACTORY-US-505; this PBI delivers the life of the correlation around it and proves the whole on the first chain, whose bundle has the one slot vision-blueprint. The ReadinessCorrelator consumes the signals through the inbox and handles each within the trusted tenant and project of the signal. The first signal for a project and operation opens a ReadinessCorrelation in state Open: the alias of the definition is resolved once through AgentRegistryService/AssetPublication.ResolveRelease to an immutable workflowReleaseId, which is persisted, is part of the key and never changes afterwards, so a moved alias never changes an open correlation. The alias is the one the activation block declares in submit.release_alias, latest in model/workflows/architecture-decomposition.yaml, and the option Workflows:Readiness:StarterReleaseAlias is the value used when the block declares none. In this increment the release is a seeded Draft release of the thin registry. AgentRegistryService is a fail-fast dependency: when ResolveRelease is unavailable, or the alias resolves to no release, no correlation opens, nothing is written and the signal is redelivered. When the last required member is satisfied and the members are consistent with each other, the correlation moves from Open to Satisfied: the bundle fingerprint is computed as sha256 over the sorted list of slot, id, version and hash of every satisfied member, the bundle document readiness-bundle-v1 is stored through ArtifactService/Artifacts.Store, and WorkflowEngineService/ReadinessSatisfied is written in the same transaction with bundleFingerprint and bundleRef. ArtifactService is fail-fast as well: when it is unavailable the correlation does not move to Satisfied and no ReadinessSatisfied is written. The engine then submits the run and never creates run state itself: the canonical submission body run-submission-v1 is stored once through the same operation (submissionRef), and RunControlService/SubmitRun is called with the workload identity of the engine as requester, the releaseId and never an alias, and an Idempotency-Key equal to {scope}:{operation}:{workflowReleaseId}:{bundleFingerprint}. On 202 the correlation records the runId, moves from Satisfied to Submitted, and WorkflowEngineService/ReadinessRunSubmitted is written in the same transaction; the accepted run then starts like any other (AIFACTORY-US-499). A repeated signal returns the existing run under the same key and body. A timeout or an uncertain answer is retried by the correlator, exponentially with five attempts, with the same key and the stored canonical body, so Run Control returns the existing run or accepts once. When the five attempts are exhausted, for a submission or for an unavailable dependency, the correlation keeps the state it had, nothing partial is written and the failed work is recorded once with its scope, as the worker hosting of AIFACTORY-US-497 defines it; a Satisfied correlation stays Satisfied, and the same key submits it once when the work runs again. A 409 for the key with a different input hash marks the correlation Superseded with reason conflict, names the existing run for an operator and never submits a second run. A newer version of any member opens a new correlation with a new fingerprint and supersedes the Open or Satisfied one, with supersededBy and WorkflowEngineService/ReadinessSuperseded; a Submitted correlation is never mutated, and a signal that arrives for it is a no-op. Beyond Workflows:Readiness:MaxOpenPerProject open correlations per project and operation the oldest is superseded. The ReadinessExpiryReconciler on its schedule, and the expiry reminder of the grain, move an Open correlation older than Workflows:Readiness:CorrelationExpiryDays to Expired and write ReadinessSuperseded with the reason expired, so a stale half-satisfied bundle never submits a run later. connectsoft.factory.workflows.readiness_open and connectsoft.factory.workflows.readiness_submissions follow each change. AuditService takes ReadinessRunSubmitted as the evidence of a run the engine requested, and StudioBff and DashboardService consume the three events, each in its own PBI and increment. The first chain proves the path: a validated Vision Blueprint submits exactly one run of architecture-decomposition, and the first step architecture-decomposition/identify-contexts becomes ready when the accepted run opens. The steps of that definition and the releases of its agents are specified by its own feature: AIFACTORY-TASK-170 of feature wf-architecture-decomposition authors the agent releases, and the engine neither loads nor executes an agent release. Two points are not stated by the model and are reported to the owner: a 409 of SubmitRun that names a gate of Run Control (a suspended, closing or closed project, a deprecated release, a retired environment), a 400 and a 403 are refusals no transition of ReadinessState covers; and where the notice to an operator about a conflict is raised is not modelled.
Acceptance criteria
| AC1 | Given the loaded definition architecture-decomposition, whose activation names the one slot vision-blueprint and the alias latest in submit.release_alias, and a BlueprintService/BlueprintValidated with kind Vision and passed true for a version of project scope, when the ReadinessCorrelator handles it, then a WorkflowEngineService/ReadinessCorrelation opens with that alias resolved once to workflowReleaseId through AgentRegistryService/AssetPublication.ResolveRelease, moves to Satisfied with the bundle fingerprint and the readiness-bundle-v1 document stored through ArtifactService/Artifacts.Store, and WorkflowEngineService/ReadinessSatisfied with correlationId, tenantId, projectId, operation, workflowReleaseId, bundleFingerprint and bundleRef is written in the same transaction and published on workflows.readiness.v1.readiness-satisfied under the canonical envelope (SCN-WorkflowEngineService-015, ADR-0026). |
| AC2 | Given a Satisfied correlation, when the correlator calls RunControlService/SubmitRun with the workload identity of the engine, the stored canonical body run-submission-v1 that carries the releaseId, and the Idempotency-Key built from scope, operation, workflow release and bundle fingerprint, and Run Control answers 202, then the correlation is Submitted with submittedRunId, WorkflowEngineService/ReadinessRunSubmitted with correlationId, tenantId, projectId, operation, runId and idempotencyKey is written in the same transaction and published on workflows.readiness.v1.readiness-run-submitted, exactly one run of architecture-decomposition exists, and the RunControlService/RunAccepted of that run opens the workflow run whose first step architecture-decomposition/identify-contexts becomes ready (SCN-WorkflowEngineService-016, FR-03, ADR-0014). |
| AC3 | Given the same signal delivered twice, a timeout or an uncertain answer of RunControlService/SubmitRun, or a signal that arrives after the correlation is Submitted, when it is handled, then the member update is idempotent on correlationId, slot and refHash, the retry sends the same Idempotency-Key and the stored canonical body so the same run is returned, the Submitted correlation is not mutated, connectsoft.factory.workflows.readiness_submissions counts the outcome, and no second run exists (SCN-WorkflowEngineService-016, model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC4 | Given an Open or a Satisfied correlation and a newer version of one of its members, a 409 of RunControlService/SubmitRun for the key with a different input hash, or more open correlations of the project and operation than Workflows:Readiness:MaxOpenPerProject, when it is handled, then a new correlation with a new fingerprint opens in the first case, the affected correlation is Superseded with supersededBy where a newer one replaced it and with its reason, WorkflowEngineService/ReadinessSuperseded with correlationId, tenantId, projectId, supersededBy and reason is written in the same transaction and published on workflows.readiness.v1.readiness-superseded, and the conflict names the existing run and submits no second run (SCN-WorkflowEngineService-017). |
| AC5 | Given an Open correlation older than Workflows:Readiness:CorrelationExpiryDays, when the expiry reminder or the ReadinessExpiryReconciler runs, then it moves to Expired and WorkflowEngineService/ReadinessSuperseded with the reason expired is written in the same transaction, and a second pass for the same correlationId and version changes nothing; given AgentRegistryService/AssetPublication.ResolveRelease unavailable, or answering that the alias resolves to no release, when a correlation would open, then no correlation opens, nothing is written and the signal is redelivered; given ArtifactService unavailable at ArtifactService/Artifacts.Store, then the correlation does not move to Satisfied and no WorkflowEngineService/ReadinessSatisfied is written; after the five attempts of the ReadinessCorrelator the correlation keeps its state, nothing partial is written and one failed-work record with its scope exists; and for each of the three readiness events a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-WorkflowEngineService-017, SCN-WorkflowEngineService-015, ADR-0012). |
| Traces: services | WorkflowEngineService, RunControlService, BlueprintService, AgentRegistryService, ArtifactService |
| Traces: aggregates | WorkflowEngineService/ReadinessCorrelation |
| Traces: apis | RunControlService/SubmitRun, AgentRegistryService/AssetPublication.ResolveRelease, ArtifactService/Artifacts.Store |
| Traces: events | WorkflowEngineService/ReadinessSatisfied, WorkflowEngineService/ReadinessRunSubmitted, WorkflowEngineService/ReadinessSuperseded, BlueprintService/BlueprintValidated, RunControlService/RunAccepted |
| Traces: steps | architecture-decomposition/identify-contexts |
| Traces: workflows | architecture-decomposition |
| Traces: descriptors | readiness-bundle-v1, run-submission-v1 |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0026 |
| Traces: requirements | FR-03, FUX-11 |
| Traces: scenarios | SCN-WorkflowEngineService-015, SCN-WorkflowEngineService-016, SCN-WorkflowEngineService-017 |
| Relates: nfrs | NFR-05 |
| Depends on | AIFACTORY-US-497, AIFACTORY-US-498, AIFACTORY-US-499, AIFACTORY-US-505, feature:run-control-service, feature:blueprint-service, feature:agent-registry-service-thin, feature:artifact-service |
| Sources | model/services/agent-mesh/WorkflowEngineService.yaml, model/workflows/architecture-decomposition.yaml, docs/architecture/adr/0026-requirements-to-ux-handoff.md, docs/reference/schemas/readiness-bundle.schema.json, docs/planes/agent-runtime/services/workflow-engine-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-206 ReadinessCorrelator: the alias resolved once when a correlation opens and the single submission through SubmitRun |
implementation | The resolver behind the member side of AIFACTORY-TASK-2021 resolves the alias the activation block declares in submit.release_alias, or the option StarterReleaseAlias when the block declares none, once through AssetPublication.ResolveRelease when a correlation opens and persists workflowReleaseId; an unavailable registry or an alias that resolves to no release opens no correlation, writes nothing and lets the signal be redelivered; for a Satisfied correlation the correlator stores the canonical submission body once through Artifacts.Store, submits one run through SubmitRun with the idempotency key of the model, the releaseId and the workload identity of the engine, and on 202 records the runId and writes ReadinessRunSubmitted in the same transaction; a timeout is retried with the same key and body, a 409 with a different input hash supersedes the correlation with reason conflict, and exhausted retries leave the correlation in its state with one failed-work record. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-207 Fixture release of architecture-decomposition in the fake registry |
fixture | The fake registry of AIFACTORY-TASK-1995 serves the Draft release of architecture-decomposition with its alias and its closure entries as references, so the correlation of the first chain resolves the alias and the accepted run loads the definition; no agent release is authored here, AIFACTORY-TASK-170 of feature wf-architecture-decomposition authors the releases of enterprise-architect and domain-modeler. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-2018 Supersede by a newer member version, the limit of open correlations and the ReadinessExpiryReconciler |
implementation | A newer version of a member opens a new correlation with a new fingerprint and supersedes the Open or Satisfied one with supersededBy and ReadinessSuperseded; beyond MaxOpenPerProject the oldest open correlation of the project and operation is superseded; the expiry reminder of the grain and the scheduled ReadinessExpiryReconciler move Open correlations older than CorrelationExpiryDays to Expired with ReadinessSuperseded and the reason expired, idempotent on correlationId plus version; a Submitted correlation is never mutated; readiness_open and readiness_submissions are emitted with their modelled dimensions. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-208 Supersede, duplicate and dependency failure tests |
verification | Acceptance tests cover a duplicate event, a newer member version, a signal after Submitted, an uncertain answer of SubmitRun and a 409 with a different input hash, and prove AC3 and AC4 of AIFACTORY-US-009 against the fake Run Control of AIFACTORY-TASK-2002: one run per key, the superseded correlation with its reason, and no second run; three further cases prove the dependency branches of AC5: ResolveRelease unavailable or an alias that resolves to no release, where no correlation opens and the signal is redelivered; ArtifactService unavailable, where the correlation is not Satisfied and no ReadinessSatisfied is written; and the five attempts of the ReadinessCorrelator exhausted, where the correlation keeps its state and one failed-work record with its scope exists. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-661 Verify readiness-starts-next by running the declared command |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-WorkflowEngineService-015\|Scenario=SCN-WorkflowEngineService-016\|Scenario=SCN-WorkflowEngineService-017)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-009, including the single run of architecture-decomposition submitted for a validated Vision Blueprint, the expiry of an Open correlation and the publication and payload of ReadinessSatisfied, ReadinessRunSubmitted and ReadinessSuperseded; consumer handling is proven by the consumers' own suites. |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |