AIFACTORY-FEAT-142: Demonstrate I4: Trust and accountability¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Outcome verification · Increment I4 Trust and accountability · State elaborated · Route qa-engineer · Build order 142 of 185 (Factory code)
The outcome of increment I4 is demonstrated end to end and the evidence is recorded: A release publishes only with evaluation evidence, approvals follow the rules of the project, and a run recovers after a worker is killed. Demonstrations: A workflow or agent release without evaluation evidence cannot be published. An approval is routed to the reviewers the rule names, and the requester can never decide it. A killed worker is replaced and the run resumes from its journal. A breached service level objective raises its alert and is visible on the operations views.
| Delivers | — |
| Stubs | — |
| Realizes | — |
| Requirements | — |
| Depends on | — |
| Sources | model/backlog/increments.yaml |
AIFACTORY-US-050: A run survives the loss of the worker that executes it¶
evidence · persona operator
As an operator, I want a run to continue after the worker that executes an attempt is killed, so that a failure of infrastructure costs time and never correctness.
This PBI proves the demonstration of increment I4 that a killed worker is replaced and the run resumes from its journal. A scripted drill on the acceptance environment starts a run of the first-outcome chain and, while an attempt holds its lease, kills the replica of AgentRuntimeService that executes it. The platform does what ADR-0024 decides. The lease the attempt acquired through RunControl.AcquireLease is not renewed and expires, which LeaseExpired of RunControlService records; the attempt is reconciled before any retry, and the replica range of the service brings a new replica up. The retry is dispatched as a new attempt, which AttemptDispatched records, and the new replica starts it from the last compatible checkpoint of the attempt, which AttemptStarted and AttemptCheckpointed carry and ListCheckpoints lists; a checkpoint whose adapter version is not compatible is not resumed, and Run Control starts the attempt afresh instead (ADR-0024 decision 2); the run journal of RunControlService is the restore anchor (ADR-0024 decision 5), so no step that already succeeded runs again. A second drill kills the replica between a tool effect and its receipt. ToolAdapterService cannot know whether the effect happened, records it as uncertain, which ToolEffectUncertain carries, and the attempt outcome is Uncertain. No retry happens on its own: an operator decision through DecideApproval must carry a finding, recorded in ToolAdapterService through ConfirmToolEffect, and the choice retry is accepted only with the finding not-applied. Each drill record names the killed replica, the expired lease and the resumed or held attempt.
Acceptance criteria
| AC1 | Given an attempt in progress that holds its lease, when the replica of AgentRuntimeService that executes it is killed, then the lease expires and the attempt is reconciled before any retry, and a new replica comes up within the replica range of the service (RunControlService/RunControl.AcquireLease, RunControlService/LeaseExpired, ADR-0024). |
| AC2 | Given the reconciled attempt, when the step is retried, then a new attempt is dispatched and resumes from the last compatible checkpoint, or starts afresh when no checkpoint is compatible, the run continues from its journal, and no step that already succeeded runs again (RunControlService/AttemptDispatched, AgentRuntimeService/AttemptStarted, AgentRuntimeService/AttemptCheckpointed, AgentRuntimeService/ListCheckpoints, ADR-0024 decisions 2 and 5). |
| AC3 | Given a tool effect that may have happened, when the replica died before the receipt, then the effect is recorded as uncertain and no retry happens on its own; an operator decision without a finding is refused, and the choice retry is accepted only with the finding not-applied (ToolAdapterService/ToolEffectUncertain, ToolAdapterService/ConfirmToolEffect, RunControlService/DecideApproval, RunControlService/RunControl.ReportAttemptOutcome). |
| AC4 | Given each drill ended, when its record is stored, then it names the killed replica, the expired lease and the resumed or held attempt with their identifiers, and the record is part of the evidence of I4 (ADR-0014, ADR-0024). |
| Traces: services | RunControlService, AgentRuntimeService, ToolAdapterService |
| Traces: events | RunControlService/LeaseExpired, RunControlService/AttemptDispatched, AgentRuntimeService/AttemptCheckpointed, AgentRuntimeService/AttemptStarted, ToolAdapterService/ToolEffectUncertain |
| Traces: apis | RunControlService/RunControl.AcquireLease, RunControlService/RunControl.ReportAttemptOutcome, RunControlService/DecideApproval, AgentRuntimeService/ListCheckpoints, ToolAdapterService/ConfirmToolEffect |
| Traces: adrs | ADR-0024, ADR-0014 |
| Depends on | feature:run-control-service, feature:agent-runtime-service, feature:tool-adapter-service |
| Sources | model/backlog/increments.yaml, docs/architecture/adr/0024-durable-execution-substrate.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-564 Recovery drill in the acceptance environment |
implementation | A scripted drill kills the replica of AgentRuntimeService that holds an attempt and records the expired lease, the reconciliation, the new replica, the new attempt and the checkpoint it resumed from. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |
AIFACTORY-TASK-565 Uncertain-effect path of the drill |
implementation | A second drill kills the replica between a tool effect and its receipt and proves that no retry happens before an operator decision with the finding not-applied is recorded. | ConnectSoft.AI.SoftwareFactory.ToolAdapterService |
AIFACTORY-TASK-779 Verify run-recovers-after-worker-kill by running the declared command |
verification | Run dotnet test -c Release --filter Category=Acceptance with the recovery drill; a zero-match filter counts as a failure; it passes and proves AC1, AC2 and AC4 of AIFACTORY-US-050 (RunControlService/LeaseExpired, AgentRuntimeService/ListCheckpoints). |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeService |
AIFACTORY-TASK-5417 Verify the uncertain-effect drill in the tool adapter acceptance suite |
verification | Run dotnet test -c Release --filter Category=Acceptance in ConnectSoft.AI.SoftwareFactory.ToolAdapterService with the uncertain-effect drill of AIFACTORY-TASK-565; a zero-match filter counts as a failure; it passes and proves AC3 of AIFACTORY-US-050 (ToolAdapterService/ToolEffectUncertain, ToolAdapterService/ConfirmToolEffect). |
ConnectSoft.AI.SoftwareFactory.ToolAdapterService |
AIFACTORY-US-1348: Trust proof: a workflow or agent release without evaluation evidence is not published, and an approval is routed to the reviewers the rule names and never decided by its requester¶
evidence · persona observer
As an observer, I want to see the publication gate refuse a release without evaluation evidence and an approval reach only the reviewers its rule names, so that what the Factory runs and what it changes is decided by the evidence and the people the rules require.
This PBI proves two demonstrations of increment I4, after an I4 precondition register is checked; while an entry is open, the demonstration that needs it is not run and the blocker is recorded as the evidence. The register has one open entry: on a deployed installation neither the approval rule source port nor the membership port of ApprovalPolicyService has a production binding until the owner names the operations behind them (AIFACTORY-US-923, AIFACTORY-US-925); in that state no approval rule is in force and every routing answers FAILED_PRECONDITION. Two former entries are decided and in the model and are checked as aligned. The subject of a request is a kind and an id from the closed list of ApprovalSubjectKind: Run, ProvisioningRun, EvaluationRun, AssetRelease and Promotion, and the service that owns the subject receives the decision (D135, which closes D129 and D131); the publication approval names the subject kind AssetRelease and the release id, and its decision returns to AgentRegistryService. RunControlService calls ApprovalPolicy.Route when a request is made and ApprovalPolicy.ValidateDecision at decision time from I4 (D126, D127). The first demonstration is that a workflow or agent release without evaluation evidence cannot be published, which ADR-0021 decision 2 requires. An asset maintainer submits a candidate release through SubmitRelease of AgentRegistryService for an asset kind that has an evaluation suite, so the submission itself triggers the evaluation (D117); the gate applies only to such asset kinds. Publication requested through PublishRelease before EvaluationCompleted is answered 412 naming the failing gate and nothing is published. The evaluation completes with a passed verdict, a Reviewer who did not submit the release grants the publication approval of subject kind AssetRelease, and the release is published, which AssetReleasePublished records. The second demonstration is that approvals follow the rules of the project. The rule in force has a quorum of one, because RunControlService holds one decider per request and answers an approve under a quorum above one with 409 (D127). ApprovalPolicyService routes the change-set approval of a run through ApprovalPolicy.Route, which ApprovalRouted records with the eligible approvers. The requester of the run is answered 403 through DecideApproval, whatever role the requester holds; a reviewer the rule names decides, validated through ApprovalPolicy.ValidateDecision.
Acceptance criteria
| AC1 | Given the I4 precondition register, when it is checked before the run, then the production bindings of the approval rule source port and the membership port of ApprovalPolicyService (AIFACTORY-US-923, AIFACTORY-US-925) have their state, and while either is open the demonstration that needs it is not run and the blocker is recorded; the closed list of approval subjects with AssetRelease for a publication (D135) and the dependency of RunControlService on ApprovalPolicy.Route and ApprovalPolicy.ValidateDecision (D126) are recorded as aligned (model/services/governance/ApprovalPolicyService.yaml, model/services/control-plane/RunControlService.yaml). |
| AC2 | Given a candidate agent release submitted through SubmitRelease for an asset kind that has an evaluation suite, when its publication is requested before EvaluationCompleted, then the answer is 412 naming the failing gate, the release keeps its state and no AssetReleasePublished is recorded (AgentRegistryService/SubmitRelease, AgentRegistryService/PublishRelease, ADR-0021 decision 2, FR-02). |
| AC3 | Given the evaluation the submission started, when it completes with a passed verdict and a Reviewer who did not submit the release grants the publication approval, whose request names the subject kind AssetRelease and the release id, then the decision returns to AgentRegistryService, the publication requested again is accepted, AssetReleasePublished is recorded and the evaluation record is referenced from its manifest (RunControlService/ApprovalRequested, QualityScoreService/EvaluationCompleted, AgentRegistryService/AssetReleasePublished, ADR-0021 decision 3, D135). |
| AC4 | Given a rule in force for the class shared-write in the project with a quorum of one, when the change-set approval of a run is routed, then the eligible approvers are those the rule names, without the requester, and the routing is recorded (ApprovalPolicyService/ApprovalPolicy.Route, ApprovalPolicyService/ApprovalRouted, FR-06). |
| AC5 | Given that request, when the requester of the run decides it, then the decision is answered 403 whatever role the requester holds and the request keeps its state, and when a reviewer the rule names approves it, then the decision is validated with the quorum met and recorded (RunControlService/DecideApproval, ApprovalPolicyService/ApprovalPolicy.ValidateDecision, ADR-0015). |
| Traces: services | AgentRegistryService, QualityScoreService, ApprovalPolicyService, RunControlService |
| Traces: apis | AgentRegistryService/SubmitRelease, AgentRegistryService/PublishRelease, ApprovalPolicyService/ApprovalPolicy.Route, ApprovalPolicyService/ApprovalPolicy.ValidateDecision, RunControlService/DecideApproval |
| Traces: events | QualityScoreService/EvaluationCompleted, AgentRegistryService/AssetReleasePublished, ApprovalPolicyService/ApprovalRouted, RunControlService/ApprovalRequested |
| Traces: adrs | ADR-0015, ADR-0021 |
| Traces: requirements | FR-02, FR-06 |
| Relates: screens | release-evidence, approval-policy |
| Depends on | feature:agent-registry-service-publication, feature:agent-registry-service-complete, feature:quality-score-service-complete, feature:approval-policy-service, feature:run-control-service |
| Sources | model/backlog/increments.yaml, docs/architecture/adr/0021-quality-gates.md, model/services/governance/ApprovalPolicyService.yaml, model/services/control-plane/AgentRegistryService.yaml, model/services/control-plane/RunControlService.yaml, model/backlog/pbis/governance/approval-policy-service.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-5418 Precondition register of the trust proof |
documentation | A precondition register on the evidence page that lists the production bindings of the approval rule source port and the membership port of ApprovalPolicyService (AIFACTORY-US-923, AIFACTORY-US-925), the closed list of approval subjects with AssetRelease for a publication (D135) and the dependency of RunControlService on ApprovalPolicy.Route and ApprovalPolicy.ValidateDecision (D126), each with the model file that closes it and its state, aligned or blocking. |
ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-5411 Refused and granted publication of one agent release |
operations | One recorded sequence on the acceptance environment: the submission through SubmitRelease of a candidate agent release of an asset kind with an evaluation suite, its publication requested before EvaluationCompleted and answered 412 with the failing gate, the evaluation started by the submission with its passed verdict, the publication approval of subject kind AssetRelease by a Reviewer who did not submit the release and the published release with the evaluation record in its manifest. |
ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-5412 Routing and refused requester decision of one run approval |
operations | One recorded routing of the change-set approval of a run under the rule in force for the class shared-write with a quorum of one and its eligible approvers, the decision of the requester answered 403, and the validated decision of a reviewer the rule names. | ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-5413 Verify the publication gate and approval rule proof against its criteria |
verification | The register, recorded refusals, routing and decisions are compared with AC1 to AC5 of AIFACTORY-US-1348 (AgentRegistryService/SubmitRelease, AgentRegistryService/PublishRelease, ApprovalPolicyService/ApprovalRouted, RunControlService/DecideApproval, ADR-0021); a publication without a passed verdict, a decision by the requester, or an open precondition presented as passed fails the check. | ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-US-1349: Trust proof: a breached service level objective raises its alert, which the operations views show until the objective recovers¶
evidence · persona operator
As an operator, I want a breached objective of a Factory service to raise its alert and appear on the operations views, so that a degradation is seen and acted on before a person reports it.
This PBI proves the demonstration of increment I4 that a breached service level objective raises its alert and is visible on the operations views. An operator sets, through SetSlo of SloService, an objective on a Factory service of the first outcome on the acceptance environment that the planned load of the drill cannot meet, and an alert rule for it through SetAlertRule of AlertRuleService. The drill applies the load. When the error budget of the objective is spent, SloService records the breach, which SloBreached carries, and AlertRuleService raises the alert of the rule, which AlertFired carries. The alert appears on the screen alerts and the breached objective on status-objectives of the Operations & usage work area, read through the gateway ObservabilityHost from ListAlerts of AlertRuleService and ListSlos of SloService, and the project overview on the screen overview from GetOverview of DashboardService, the overview kind DashboardService serves in I4 (D131); the freshness of each source is shown and a degraded source is never hidden. When the load stops and the objective recovers, SloRecovered and AlertResolved are recorded and the views show the recovery. The record names the objective, the alert, the times the views showed each state and the identifiers of the events.
Acceptance criteria
| AC1 | Given an objective and its alert rule set on a Factory service of the acceptance environment, when the drill load spends its error budget, then the breach and the alert are recorded with their identifiers (SloService/SetSlo, AlertRuleService/SetAlertRule, SloService/SloBreached, AlertRuleService/AlertFired, NFR-10). |
| AC2 | Given the fired alert, when an operator opens the operations views read through ObservabilityHost, then the alert is listed on alerts, the breached objective on status-objectives and the project overview on overview, each with the freshness of its source (AlertRuleService/ListAlerts, SloService/ListSlos, DashboardService/GetOverview, D131). |
| AC3 | Given the load stops, when the objective recovers, then the recovery and the resolved alert are recorded and the views show them, and the record of the drill is part of the evidence of I4 (SloService/SloRecovered, AlertRuleService/AlertResolved). |
| Traces: services | SloService, AlertRuleService, DashboardService, ObservabilityHost |
| Traces: apis | SloService/SetSlo, AlertRuleService/SetAlertRule, AlertRuleService/ListAlerts, SloService/ListSlos, DashboardService/GetOverview |
| Traces: events | SloService/SloBreached, SloService/SloRecovered, AlertRuleService/AlertFired, AlertRuleService/AlertResolved |
| Traces: adrs | ADR-0018 |
| Relates: screens | alerts, status-objectives, overview |
| Relates: nfrs | NFR-10 |
| Depends on | feature:slo-service, feature:alert-rule-service, feature:dashboard-service, feature:observability-host, feature:studio-operations-usage |
| Sources | model/backlog/increments.yaml, model/services/observability/SloService.yaml, model/services/observability/AlertRuleService.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-5414 Breach drill of one objective with its alert and views |
operations | One recorded drill on the acceptance environment: the objective and the alert rule set for a Factory service, the load applied, the breach and the fired alert with their identifiers, screenshots of alerts, status-objectives and the project overview on overview with the freshness of each source, and the recovery with the resolved alert. |
ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-5415 Verify the breach proof against its criteria |
verification | The recorded drill is compared with AC1 to AC3 of AIFACTORY-US-1349 (SloService/SloBreached, AlertRuleService/AlertFired, AlertRuleService/ListAlerts, SloService/ListSlos, DashboardService/GetOverview, SloService/SloRecovered); a breach without an alert, an alert missing from alerts or an objective missing from status-objectives fails the check. |
ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-US-051: Trust and accountability are demonstrated with evidence¶
evidence · persona observer
As an observer, I want to see that a release needs evidence, that approvals follow the rules, that a run recovers and that a breach raises its alert, so that the fourth outcome is accepted on facts.
This is the integration PBI of the feature and the closing proof of increment I4. The outcome of I4, as model/backlog/increments.yaml states it, is that a release publishes only with evaluation evidence, that approvals follow the rules of the project and that a run recovers after a worker is killed; the fourth demonstration adds that a breached service level objective raises its alert and is visible on the operations views. The increment is finished when this proof passes, not when its services are deployed. An observer replays the four demonstrations in one recorded acceptance run: the publication gate and the approval routing of AIFACTORY-US-1348, the two recovery drills of AIFACTORY-US-050 and the breach drill of AIFACTORY-US-1349. The evidence pack holds the refusals with their gates, the routing and the decisions, the drill records with the killed replicas, expired leases and resumed attempts, and the alert with its place on the operations views. The done_when of the epics governance, observability and data-event reads this proof as the outcome verification of I4. A demonstration that fails is recorded with its failing step and the increment stays open; nothing is waived.
Acceptance criteria
| AC1 | Given a release without evaluation evidence, when its publication is requested in the recorded acceptance run, then the refusal of AgentRegistryService is in the evidence, as AIFACTORY-US-1348 proves it (AgentRegistryService, QualityScoreService, ADR-0021). |
| AC2 | Given an approval request, when it is routed and the requester tries to decide it, then the routing and the refusal are in the evidence, as AIFACTORY-US-1348 proves it (ApprovalPolicyService). |
| AC3 | Given the recovery drills, when they end, then their records are in the evidence, as AIFACTORY-US-050 proves it (AgentRuntimeService, ADR-0024). |
| AC4 | Given a breached objective, when its alert fires, then the alert and its place on the operations views are in the evidence, as AIFACTORY-US-1349 proves it, and the pack reviewed against the outcome of I4 in model/backlog/increments.yaml has every demonstration, its recorded failure or the blocker of its open precondition (SloService, AlertRuleService, DashboardService, ADR-0029). |
| Traces: services | AgentRegistryService, QualityScoreService, ApprovalPolicyService, AgentRuntimeService, SloService, AlertRuleService, DashboardService |
| Traces: adrs | ADR-0021, ADR-0024, ADR-0029 |
| Depends on | AIFACTORY-US-050, AIFACTORY-US-1348, AIFACTORY-US-1349 |
| Sources | model/backlog/increments.yaml, model/backlog/epics.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-566 Run the four demonstrations and record the evidence |
documentation | An evidence page with the refusal and its gate, the routing and the decisions, the drill records and the alert with its place on the operations views, from one recorded acceptance run of AIFACTORY-US-1348, AIFACTORY-US-050 and AIFACTORY-US-1349. | ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-508 Evidence pack for increment I4 |
documentation | A recorded run of every demonstration below with its run identifiers, artifacts and logs: A workflow or agent release without evaluation evidence cannot be published. An approval is routed to the reviewers the rule names, and the requester can never decide it. A killed worker is replaced and the run resumes from its journal. A breached service level objective raises its alert and is visible on the operations views. | ConnectSoft.AI.SoftwareFactory.Documentation |
AIFACTORY-TASK-780 Verify trust-demonstrated against the recorded acceptance run |
verification | The evidence pack is compared with AC1 to AC4 of AIFACTORY-US-051 and with the outcome of I4 in model/backlog/increments.yaml; every demonstration has its evidence, and python -I tools/docs_model validate passes with the evidence page in place. |
ConnectSoft.AI.SoftwareFactory.Documentation |