Skip to content

Control Plane — Workflows

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Rewritten 2026-10-09 (design plan P1)

The June-2026 page described a WorkflowInstance state machine with AssigningTask, Correcting, Compensating and Escalated states, MassTransit sagas and a replay service. Those are superseded: the run vocabulary is fixed by ADR-0014, workflow definitions and their DAG engine belong to BC-38 in the Agent Mesh, and recovery replaces replay. The old page is archived under catalogs 2026-06.

The Control Plane owns the lifecycles whose state must be true before anything else may act on it. Each is a state machine on one aggregate of one module; execution of the work itself happens in the Agent Mesh under a lease.

Main lifecycle

flowchart LR
    Onboard["Project onboarding<br/>ProjectService"] --> Publish["Asset publication<br/>AgentRegistryService"]
    Publish --> Submit["Run submission<br/>RunControlService"]
    Submit --> Reserve["Budget reservation<br/>CostUsageService"]
    Reserve --> Dispatch["Lease and dispatch<br/>RunControlService"]
    Dispatch --> Gate{"Gated action?"}
    Gate -- yes --> Approve["Approval request and decision"]
    Approve --> Dispatch
    Gate -- no --> Outcome["Attempt outcome"]
    Outcome --> Terminal["Completed / Failed / Cancelled"]
    Terminal --> Settle["Settlement and reconciliation<br/>CostUsageService"]
    Settle --> Invoice["Period meters and invoices<br/>FactoryCommercialService"]
    Onboard -. later .-> Offboard["Off-boarding and export"]
Hold "Alt" / "Option" to enable pan & zoom

Run lifecycle (BC-03)

stateDiagram-v2
    [*] --> Accepted: submission with trusted scope, pinned release, snapshot
    Accepted --> Running: BudgetReserved, first step dispatched
    Accepted --> Failed: BudgetReservationDenied
    Running --> WaitingForApproval: gated action proposed
    WaitingForApproval --> Running: approved, exact version
    WaitingForApproval --> Failed: rejected or expired
    Running --> Cancelling: cancel requested
    WaitingForApproval --> Cancelling: cancel requested
    Cancelling --> Cancelled: interruption observed
    Running --> Completed: last step succeeded
    Running --> Failed: step failed, no retry allowed
    Completed --> [*]
    Failed --> [*]
    Cancelled --> [*]
Hold "Alt" / "Option" to enable pan & zoom
State Meaning Who moves it
Accepted The run exists, is pinned and journaled; nothing has executed RunControlService on submission
Running At least one step is dispatched under a lease dispatcher after BudgetReserved
WaitingForApproval A gated action waits for an exact-version decision runtime through RequestApproval
Cancelling Interruption requested from the lease holder, not yet observed requester, Operator or policy
Completed, Failed, Cancelled Terminal; failure class recorded (policy-denied, budget, attempt, cancellation) RunControlService

Steps follow Pending → Dispatched → Running → Succeeded | Failed | Skipped | Cancelled; attempts end as Succeeded, Failed, TimedOut, Cancelled or Uncertain. An Uncertain attempt is never retried automatically because an external effect may have happened; it is surfaced to an Operator with the receipt that caused the doubt.

Submission and dispatch

sequenceDiagram
    participant C as Client or Studio BFF
    participant P as ProjectService
    participant AR as AgentRegistryService
    participant RC as RunControlService
    participant U as CostUsageService
    participant W as AgentRuntimeWorker
    C->>RC: POST /factory/runs with Idempotency-Key
    RC->>P: ResolveScope
    RC->>AR: ResolveRelease alias or id
    RC->>RC: policy revision, snapshot, Run Accepted, commit with outbox
    RC-->>C: 202 with run location and pinned ids
    RC-)U: RunAccepted
    U-)RC: BudgetReserved
    RC->>RC: step Dispatched, Lease with fencing token, commit with outbox
    RC-)W: AttemptDispatched
    W->>AR: LoadRelease
    W->>RC: ReportAttemptOutcome with lease token
Hold "Alt" / "Option" to enable pan & zoom

Idempotency is scoped to tenant, project and operation: same key and same canonical payload returns the existing run; same key and a different payload returns 409. Dispatch asks Execution Capacity (BC-26) for a slot and leaves the step Pending when none is available.

Approval gates (D21)

Action classes shared-write, external, production, publication and spend above the threshold always need one approval per action, bound to the exact proposal version; read and sandbox actions run autonomously.

sequenceDiagram
    participant W as AgentRuntimeWorker
    participant PE as PolicyEngineService
    participant RC as RunControlService
    participant S as Studio approvals inbox
    participant R as Reviewer or Workspace admin
    W->>PE: decision for action class on target environment
    PE-->>W: approval required, approver rule, expiry
    W->>RC: RequestApproval with proposalHash, releaseId, snapshotId, policyRevision
    RC->>RC: ApprovalRequest Pending, Run WaitingForApproval, commit with outbox
    RC-)S: ApprovalRequested
    R->>RC: decision approve with request version
    RC->>PE: approver eligible, not the requester, not expired, inputs unchanged
    RC->>RC: ApprovalDecided, Run Running, commit with outbox
    RC-)W: ApprovalDecided
Hold "Alt" / "Option" to enable pan & zoom

A regenerated proposal makes the earlier request Stale; an expired request fails the run with class policy-denied; a Platform admin break-glass decision is allowed only with an audited reason. Policy (who may approve what, expiry, separation of duties) is evaluated by BC-20 at request and decision time; the Control Plane records instances.

Publication (BC-04)

Draft → InReview → Published → Deprecated. A candidate is submitted from a commit, its closure is resolved and hashed, an evaluation record is produced by Observability (BC-22), a publication approval is decided by a Reviewer who is not the author, and the release becomes loadable. Deprecation and revocation carry an in-flight policy that Run Control and the runtime honour. Details and the sequence diagram are on the AgentRegistryService page.

Spend (BC-05)

Reserve before dispatch, settle at terminal state, reconcile later. A reservation above the approval threshold is an approval of class spend bound to the estimate; a reservation beyond the budget denies the run. Receipts from model and tool calls create usage records with the pricing revision in force; unknown usage stays Uncertain until a provider statement confirms it. See CostUsageService.

Cancellation

Cancellation records intent (Cancelling), requests interruption from the current lease holder and waits for the observation. Completed external effects stay visible in the run's receipts and artifacts; nothing is rolled back silently. A run that cannot be interrupted within the lease's grace period is reconciled by the lease reconciler and the attempt ends Uncertain.

Failure and recovery

Failure class Example Handling
Transient model gateway timeout, tool adapter unavailable runtime retries within the attempt's budget; a new attempt needs a new lease
Validation agent output fails its contract runtime retries with validator feedback up to the workflow's limit, then step Failed
Policy approval rejected or expired, budget denied run Failed with class policy-denied or budget
Infrastructure host crash, lease expiry reconcilers move expired leases; grains re-activate from persisted rows; the journal is the restore anchor (D30 restore order)

There is no replay: ADR-0014 promises recovery of truth from the journal, not deterministic re-execution. Re-running work against a newer release is a new run that references the old one.

Project off-boarding (XL-13, XL-14)

Active | Suspended → Closing → Closed: open runs cancelled, delegations revoked, knowledge indexes scheduled for deletion, artifacts exported or deleted per the tenant's choice within the data governance windows, then Closed. The procedure is journaled on the project and resumes after interruption.