AIFACTORY-FEAT-119: Workflow definition: Behaviour simulation¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Factory assets: workflow definitions, agents and skills · Increment I4 Trust and accountability · State elaborated · Route solution-architect · Build order 138 of 185 (Factory code)
The workflow definition behaviour-simulation as a published asset: Runs the acceptance scenarios of a generated product end to end in the sandbox with agents, scores the results into evaluation records and stores the validated results. It has 4 steps, names the agents e2e-testing, qa-engineer and the tools shell-sandbox, and is executed by WorkflowEngineService under Run Control.
| Delivers | — |
| Stubs | — |
| Realizes | workflows: behaviour-simulation adrs: ADR-0014, ADR-0021, ADR-0027 |
| Requirements | FR-08, FUX-10, XL-08, XL-09 |
| Depends on | AIFACTORY-FEAT-189 AgentRegistryService: the publication path of releases from I2 |
| Sources | model/workflows/behaviour-simulation.yaml |
Scenarios to trace (4, catalog): SCN-behaviour-simulation-001 ✓, SCN-behaviour-simulation-002 ✓, SCN-behaviour-simulation-003 ✓, SCN-behaviour-simulation-004 ✓
AIFACTORY-US-091: The Behaviour simulation workflow runs from its published definition¶
behavior · persona requester
As a requester, I want the behaviour simulation workflow to run from its published definition with the agents, tools and gates it declares, so that simulation-results are produced the same way on every run, with evidence.
The integration PBI of the definition behaviour-simulation. The definition is readiness-activated with four required members of one project: the validated Test, Service and Module Blueprints (BlueprintValidated with passed true and the matching kind) and the generation run (GenerationCompleted of GenerationRecipeService with at least one module). When every member is satisfied the ReadinessCorrelator of WorkflowEngineService submits exactly one run through SubmitRun of RunControlService with its service token naming the project and the bundle idempotency key; a newer member version supersedes the open correlation. On RunAccepted the engine loads the Published release the run pinned through AgentRegistryService and marks the four steps ready in order: the tool step prepare-sandbox on shell-sandbox, run-scenarios on the e2e-testing release, then score and store-results on the qa-engineer release. Every step declares failure_policy Retry, applied by Run Control only to a Failed attempt of failure class transient, the steps being of read and sandbox class; the simulation never touches a shared or production system. No step declares a compensation, so a failed run leaves its sandbox outputs as evidence (ADR-0014). The run reports exactly one outcome within its budget. Before any run, the definition release and the agent releases of e2e-testing and qa-engineer, with any skill, tool or model profile release they need that is not yet Published, are submitted through SubmitRelease and published through PublishRelease of AgentRegistryService (feature agent-registry-service-publication); each publication is approved by a Reviewer who is not the submitter, a Platform admin for a model profile release. In I4 the evaluation gate of the complete delivery applies: where an evaluation suite exists for the asset kind, a release is published only with a passed evaluation record (ADR-0021), and an agent release published before I4 without one is resubmitted through the gate as a new version; the runs pin the Published release, never a Draft one (D140). The step behaviour is specified in AIFACTORY-US-1236 and AIFACTORY-US-1237; this PBI proves the chain on the recorded fixture.
Acceptance criteria
| AC1 | Given the Published release of behaviour-simulation, submitted through AgentRegistryService/SubmitRelease and published through AgentRegistryService/PublishRelease after an approval by a Reviewer who is not the submitter and with a passed evaluation record where an evaluation suite exists for its asset kind (ADR-0021, D140), carries the activation block, the 4 steps, input mappings, retry policies, timeouts and the failure_policy Retry of every step exactly as in model/workflows/behaviour-simulation.yaml, when the last required member of an open WorkflowEngineService/ReadinessCorrelation of the project is satisfied, then one run is submitted through RunControlService/SubmitRun with the service token of the engine naming the project and the bundle idempotency key after WorkflowEngineService/ReadinessSatisfied, and WorkflowEngineService/ReadinessRunSubmitted names the returned run; a repeated submission with the same key and body returns the same run; the same key with a different body is answered 409 by RunControlService/SubmitRun and the correlation becomes Superseded with reason conflict, publishing WorkflowEngineService/ReadinessSuperseded and no run; and a newer member version publishes WorkflowEngineService/ReadinessSuperseded instead of a second run (ADR-0014). |
| AC2 | Given RunControlService/RunAccepted for that release with the inputs test-blueprint, generated-outputs, service-blueprint and module-blueprints, when the engine opens its projection, then WorkflowEngineService/WorkflowRunStarted is emitted and behaviour-simulation/prepare-sandbox, behaviour-simulation/run-scenarios, behaviour-simulation/score and behaviour-simulation/store-results become ready in that order on the tool shell-sandbox and the pinned Published releases of e2e-testing and qa-engineer, and no Draft release is loaded (D140). |
| AC3 | Given every step succeeds, when the run completes, then simulation-results (artifact-v1) is recorded as the run output under the run envelope, WorkflowEngineService/WorkflowRunCompleted is emitted once and RunControlService/RunCompleted follows exactly once (ADR-0014). |
| AC4 | Given a step is settled Failed with its retry budget spent or with a failure class other than transient, when the engine applies the compensation policy of model/workflows/behaviour-simulation.yaml, then WorkflowEngineService/WorkflowRunFailed names the step and the failure class, the sandbox outputs stay in place as evidence and RunControlService/RunFailed is published exactly once (recovery, not replay, ADR-0014). |
| AC5 | Given the recorded fixture inputs of model/workflows/behaviour-simulation.yaml, when the fixture run executes in the pipeline, then its outputs match the recorded ones field by field, every step output validates against its descriptor and the run stays within the declared budget (ADR-0021). |
| Traces: workflows | behaviour-simulation |
| Traces: steps | behaviour-simulation/prepare-sandbox, behaviour-simulation/run-scenarios, behaviour-simulation/score, behaviour-simulation/store-results |
| Traces: agents | e2e-testing, qa-engineer |
| Traces: tools | shell-sandbox |
| Traces: services | WorkflowEngineService, RunControlService, AgentRegistryService |
| Traces: aggregates | WorkflowEngineService/ReadinessCorrelation |
| Traces: apis | RunControlService/SubmitRun, AgentRegistryService/SubmitRelease, AgentRegistryService/PublishRelease |
| Traces: events | AgentRegistryService/AssetReleasePublished, WorkflowEngineService/ReadinessSatisfied, WorkflowEngineService/ReadinessRunSubmitted, WorkflowEngineService/ReadinessSuperseded, RunControlService/RunAccepted, RunControlService/RunCompleted, RunControlService/RunFailed, WorkflowEngineService/WorkflowRunStarted, WorkflowEngineService/WorkflowRunCompleted, WorkflowEngineService/WorkflowRunFailed |
| Traces: descriptors | workflow-definition-v1, readiness-bundle-v1, blueprint-test-v1, blueprint-service-v1, blueprint-module-v1, change-set-v1, artifact-v1 |
| Traces: adrs | ADR-0014, ADR-0021, ADR-0027 |
| Traces: requirements | FR-08, FUX-10, XL-08, XL-09 |
| Depends on | AIFACTORY-US-1236, AIFACTORY-US-1237, feature:workflow-engine-service, feature:run-control-service, feature:agent-registry-service-complete, feature:agent-registry-service-publication |
| Sources | model/workflows/behaviour-simulation.yaml, model/agents/qa/e2e-testing.yaml, model/agents/qa/qa-engineer.yaml, docs/planes/agent-runtime/workflow-definitions/behaviour-simulation.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-401 Author, submit and publish the release of the workflow definition behaviour-simulation with its agent releases |
implementation | The release of behaviour-simulation (workflow-definition-v1) authored with its activation block of four required members, its 4 steps, input mappings, retry policies, timeouts and compensation policy as in model/workflows/behaviour-simulation.yaml, the failure_policy Retry of every step copied as it stands, and accepted by ValidateWorkflowDefinition of WorkflowEngineService; it is submitted through SubmitRelease of AgentRegistryService together with the agent releases e2e-testing and qa-engineer of AIFACTORY-TASK-402 and any skill, tool or model profile release of their closure that is not yet Published, each publication is approved, through the approval that RunControlService opens with the subject kind asset-release, by a Reviewer who is not the submitter (a Platform admin for a model profile release), and each release is published through PublishRelease with the alias latest moved to it (feature agent-registry-service-publication, D140); the evaluation gate of the complete delivery applies, so a release of an asset kind with an evaluation suite is published only with a passed evaluation record, and the Published release ids are recorded for the fixture run, which pins them. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-402 Author the agent releases e2e-testing and qa-engineer with their prompts and skills |
implementation | Draft releases of e2e-testing (the three bound skills of model/agents/qa/e2e-testing.yaml, model profile code-agent, with the browser-session and shell-sandbox bindings) and qa-engineer (the ten bound skills of model/agents/qa/qa-engineer.yaml, model profile code-agent) with their instructions; each is evaluated with the suite qa-core-v1 and carries a passed evaluation record before AIFACTORY-TASK-401 submits it for publication, and a release of either agent published before I4 without an evaluation record is resubmitted through the evaluation gate as a new version (D140). | ConnectSoft.AI.SoftwareFactory.AgentTemplate |
AIFACTORY-TASK-403 Fixture run of behaviour-simulation against recorded inputs |
verification | The definition runs end to end on a fixture project in the acceptance environment from a satisfied readiness bundle to RunCompleted on the Published releases of AIFACTORY-TASK-401; every step output validates against its descriptor and simulation-results matches the recorded one field by field (AC1, AC2, AC3 and AC5 of AIFACTORY-US-091). | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-722 Verify behaviour-simulation-runs-from-its-definition by running the declared command |
verification | Run dotnet test -c Release --filter Category=WorkflowFixture; it passes and proves AC1 to AC5 of AIFACTORY-US-091, including the same-key replay, the 409 conflict with the correlation Superseded with reason conflict and no run, and the failure path of AC4 (WorkflowRunFailed then RunFailed exactly once); the output names the Published release ids of behaviour-simulation, e2e-testing and qa-engineer that the run pinned, as AgentRegistryService/GetRelease reports them with their approval by a Reviewer who is not the submitter and their evaluation record (D140). |
ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-1236: The generated product is started in the sandbox and the end-to-end role drives its acceptance scenarios¶
behavior · persona requester
As a requester, I want the acceptance scenarios of my generated product driven end to end through its UI and APIs in an isolated sandbox, so that the behaviour of the product is observed with a trace per step before anything reaches a shared environment.
The first two steps of behaviour-simulation. When the engine marks prepare-sandbox ready it resolves generated-outputs from the run input and emits StepReady with actionClass sandbox; Run Control dispatches the attempt and the tool shell-sandbox of ToolAdapterService builds and starts the generated product in the sandbox pool with seeded data, its commands audited with commandLine, exitCode, durationMs and sandbox as the tool card declares. The step returns environment (execution-scope-v1) naming the sandbox the product runs in. It declares failure_policy Retry with a fixed budget of 2 attempts and backoff 10s and a timeout PT30M. When run-scenarios becomes ready it receives automation from input.generated-outputs, environment from steps.prepare-sandbox.environment and test-blueprint; the e2e-testing release, sandbox class, with the skills e2e-testing-journey-suites, e2e-testing-browser-sessions-in-sandbox-pools and e2e-testing-typed-results and the tools browser-session and build-and-test, drives every acceptance scenario of the Test Blueprint through the UI and the APIs of the running product; every scenario step leaves a trace. It writes scenario-results with the verdict and trace of each scenario. A failing scenario is a result, not a step failure. The step declares failure_policy Retry, exponential, 3 attempts, timeout PT2H. Neither step touches a shared system.
Acceptance criteria
| AC1 | Given a run accepted with generated-outputs (change-set-v1), when behaviour-simulation/prepare-sandbox runs shell-sandbox, then the product is built and started in the sandbox pool with seeded data, the output environment (execution-scope-v1) is recorded once under the run envelope and every command is audited with the fields of model/tools/shell-sandbox.yaml (SCN-behaviour-simulation-001). |
| AC2 | Given an attempt of behaviour-simulation/prepare-sandbox settles Failed with failure class transient, when Run Control evaluates the step, then it is retried under the fixed budget of 2 attempts with backoff 10s; given the budget is spent, the attempt exceeds PT30M or fails with any other failure class, then WorkflowEngineService/WorkflowRunFailed names the step and no shared system is changed (SCN-behaviour-simulation-001, ADR-0014). |
| AC3 | Given steps.prepare-sandbox.environment is recorded, when e2e-testing executes behaviour-simulation/run-scenarios with automation, environment and test-blueprint, then every acceptance scenario of the Test Blueprint is driven through the UI or the APIs of the sandboxed product, the output scenario-results (artifact-v1) is recorded once with a verdict and a trace per scenario, and a failed scenario is recorded as a result while the step succeeds (SCN-behaviour-simulation-002). |
| AC4 | Given an attempt of behaviour-simulation/run-scenarios exceeds its timeout PT2H, settles Failed with a failure class other than transient, or settles Failed with failure class transient until the exponential budget of 3 attempts is spent, when the engine applies the policy of model/workflows/behaviour-simulation.yaml, then WorkflowEngineService/WorkflowRunFailed names the step and the sandbox output stays in place as evidence (SCN-behaviour-simulation-002, ADR-0014). |
| Traces: workflows | behaviour-simulation |
| Traces: steps | behaviour-simulation/prepare-sandbox, behaviour-simulation/run-scenarios |
| Traces: agents | e2e-testing |
| Traces: tools | shell-sandbox |
| Traces: services | WorkflowEngineService, ToolAdapterService |
| Traces: events | WorkflowEngineService/StepReady, WorkflowEngineService/WorkflowRunFailed |
| Traces: descriptors | change-set-v1, execution-scope-v1, blueprint-test-v1, artifact-v1 |
| Traces: scenarios | SCN-behaviour-simulation-001, SCN-behaviour-simulation-002 |
| Traces: adrs | ADR-0014 |
| Traces: requirements | FR-08, FUX-10, XL-08 |
| Depends on | feature:workflow-engine-service, feature:run-control-service, feature:tool-adapter-service |
| Sources | model/workflows/behaviour-simulation.yaml, model/tools/shell-sandbox.yaml, model/agents/qa/e2e-testing.yaml, model/scenarios/behaviour-simulation.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4936 Recorded fixture product with acceptance scenarios and seeded data |
fixture | A fixture generated output that builds and starts in the sandbox, its seeded data, a Test Blueprint with three acceptance scenarios of which one is expected to fail, the expected environment (execution-scope-v1) and the expected scenario-results with a verdict and a trace per scenario, as the oracle of SCN-behaviour-simulation-001 and SCN-behaviour-simulation-002. | ConnectSoft.AI.SoftwareFactory.AgentTemplate |
AIFACTORY-TASK-4937 Bind prepare-sandbox to shell-sandbox and map the run-scenarios inputs in the Draft release |
implementation | The tool step prepare-sandbox of the Draft release calls shell-sandbox with generated-outputs and returns environment as execution-scope-v1, announced with actionClass sandbox and the fixed retry of model/workflows/behaviour-simulation.yaml; the agent step run-scenarios maps automation, environment and test-blueprint as declared there. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-4938 Prove the sandbox preparation and the scenario run on the fixture |
verification | A fixture run proves SCN-behaviour-simulation-001 and SCN-behaviour-simulation-002 (AC1 to AC4 of AIFACTORY-US-1236); the expected failing scenario is recorded as a result while run-scenarios succeeds, and the retry exhaustion of prepare-sandbox ends in WorkflowRunFailed naming the step. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-US-1237: The QA role scores the scenario results into evaluation records and stores them with lineage¶
behavior · persona requester
As a requester, I want the scenario results of a simulation scored for task success, regressions and flakiness and stored with their lineage, so that the quality of a generated product is recorded as evidence that later gates and reviews can read.
The last two steps of behaviour-simulation. When run-scenarios succeeded the engine marks score ready and resolves service-blueprint, module-blueprints, test-blueprint and scenario-results from steps.run-scenarios.scenario-results. The qa-engineer release runs in read class with the skills qa-engineer-detect-regression, qa-engineer-analyze-coverage and qa-engineer-compute-confidence-score: it scores task success, regressions against the quality constraints of the Service Blueprint and flakiness of each scenario into evaluation-records. When store-results becomes ready it receives service-blueprint, module-blueprints and evaluation-records from steps.score.evaluation-records; the qa-engineer release, sandbox class, with the skill qa-engineer-tag-output-with-trace, writes the validated results and returns simulation-results as its step output. AgentRuntimeService records that output through Artifacts.Store of ArtifactService with the producer run, attempt and agent, ArtifactService being the one writer of artifacts. Three points are open and reported to the owner. First, the step summary asks to store the results and the evaluation records with lineage to the generated outputs, but the mappings of store-results carry no generated-outputs input, so a lineage edge to the generated outputs would need a run-input edge the step does not receive; whether the edge goes to the generated outputs or to the evaluation records the step receives, and whether the evaluation records are stored by this step as well, is an owner decision. Second, the definition declares emits: ArtifactValidated, EvaluationCompleted: ArtifactValidated is the validator verdict of ArtifactService behind the validated results and EvaluationCompleted is an event of QualityScoreService (quality-score-service-complete, I4), but no step of the definition calls an operation that produces either. Third, how the evaluation records reach QualityScoreService is not modelled. Both steps declare failure_policy Retry, exponential, 3 attempts, backoff 30s, timeout PT30M; the run then completes with simulation-results as its output.
Acceptance criteria
| AC1 | Given steps.run-scenarios.scenario-results is recorded, when qa-engineer executes behaviour-simulation/score with the Service, Module and Test Blueprints, then the output evaluation-records (artifact-v1) is recorded once under the run envelope with task success, regressions and flakiness per scenario, each citing the scenario result it scores (SCN-behaviour-simulation-003). |
| AC2 | Given steps.score.evaluation-records is recorded, when qa-engineer executes behaviour-simulation/store-results, then AgentRuntimeService records simulation-results (artifact-v1) through ArtifactService/Artifacts.Store with the producer run, attempt and agent and exactly one new ArtifactService/ArtifactStored is emitted for it; the lineage edge it carries (to the generated outputs, as the step summary says, or to the evaluation records the step receives) and whether the evaluation records are stored as well are asserted only once the owner decides them (SCN-behaviour-simulation-004). |
| AC3 | Given the results are stored, when the step completes, then the run output simulation-results is set, WorkflowEngineService/WorkflowRunCompleted is emitted once and RunControlService/RunCompleted follows exactly once (SCN-behaviour-simulation-004, ADR-0014). |
| AC4 | Given an attempt of behaviour-simulation/score or behaviour-simulation/store-results exceeds its timeout PT30M, settles Failed with a failure class other than transient, or settles Failed with failure class transient until the exponential budget of 3 attempts is spent, when the engine applies the policy of model/workflows/behaviour-simulation.yaml, then WorkflowEngineService/WorkflowRunFailed names the step, the earlier outputs stay in place and no artifact is written twice on recovery, ArtifactService/Artifacts.Store being idempotent on tenant and hash (SCN-behaviour-simulation-003, SCN-behaviour-simulation-004, ADR-0014). |
| Traces: workflows | behaviour-simulation |
| Traces: steps | behaviour-simulation/score, behaviour-simulation/store-results |
| Traces: agents | qa-engineer |
| Traces: services | WorkflowEngineService, AgentRuntimeService, ArtifactService |
| Traces: aggregates | ArtifactService/Artifact |
| Traces: apis | ArtifactService/Artifacts.Store |
| Traces: events | ArtifactService/ArtifactStored, WorkflowEngineService/WorkflowRunCompleted, WorkflowEngineService/WorkflowRunFailed, RunControlService/RunCompleted |
| Traces: descriptors | blueprint-service-v1, blueprint-module-v1, blueprint-test-v1, artifact-v1 |
| Traces: scenarios | SCN-behaviour-simulation-003, SCN-behaviour-simulation-004 |
| Traces: adrs | ADR-0014, ADR-0021 |
| Traces: requirements | FR-08, XL-09 |
| Depends on | AIFACTORY-US-1236, feature:workflow-engine-service, feature:run-control-service, feature:artifact-service, feature:agent-runtime-service |
| Sources | model/workflows/behaviour-simulation.yaml, model/agents/qa/qa-engineer.yaml, model/services/knowledge/ArtifactService.yaml, model/scenarios/behaviour-simulation.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4939 Fixture expected evaluation records and stored simulation results |
fixture | The expected evaluation-records of the fixture scenario results of AIFACTORY-TASK-4936 (task success, one regression, flakiness per scenario) and the expected simulation-results artifact with its producer run, with the expectation of exactly one new ArtifactStored for it, as the oracle of SCN-behaviour-simulation-003 and SCN-behaviour-simulation-004; the expected lineage edge and any stored evaluation records are added once the owner decides them. | ConnectSoft.AI.SoftwareFactory.AgentTemplate |
AIFACTORY-TASK-4940 Map the score and store-results steps and the run output in the Draft release |
implementation | The steps score and store-results of the Draft release authored by AIFACTORY-TASK-401 map their inputs exactly as declared in model/workflows/behaviour-simulation.yaml, and the recorded output simulation-results is set as the run output simulation-results when the run completes. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |
AIFACTORY-TASK-4941 Prove the scoring, the stored results and the run completion on the fixture |
verification | A fixture run proves SCN-behaviour-simulation-003 and SCN-behaviour-simulation-004 (AC1 to AC4 of AIFACTORY-US-1237); exactly one new ArtifactStored for simulation-results, WorkflowRunCompleted and RunCompleted exactly once, and the timeout path ending Failed without a second write; the lineage edge is checked once the owner decides it. | ConnectSoft.AI.SoftwareFactory.WorkflowEngineService |