RunControlService

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Bounded context	BC-03 Run Control
Platform / plane	Control Plane / Control
Deployment granularity	module of host · host ControlPlaneApi
Status	target · ConnectSoft.Factory.RunControl
Built from	connectsoft-microservice · libraries: connectsoft-extensions-saas-abstractions, connectsoft-extensions-saas-asp-net-core, connectsoft-extensions-messaging-model, connectsoft-extensions-messaging-model-orleans, connectsoft-extensions-persistence-model-nhibernate
Decisions	ADR-0012, ADR-0014, ADR-0015
Traceability	FR-03 FR-06 FR-10 FUX-04 FUX-05 FUX-07 NFR-04 NFR-05 NFR-06 XL-03 XL-05 XL-06

Purpose and responsibilities

Run Control is the one writer of operational truth for work in the Factory. It accepts a run request from a trusted scope, pins it to an immutable asset release, a policy revision and a resolved execution snapshot, and then owns every transition of the run, its steps and its attempts until a terminal state. It issues leases to runtime workers, accepts exactly one outcome per attempt from the current lease holder, records approval requests bound to the exact proposal version and their decisions, and keeps an append-only recovery journal.

It never executes work itself: execution is the runtime worker's job (BC-06), policy evaluation is Governance's (BC-18/BC-20), budget is Usage Accounting's (BC-05). The Studio and every projection read run state from here or from events this service committed, never from a worker's memory or a toast.

The service absorbs the June-2026 catalog rows WorkflowOrchestrator, TaskAssignmentService, ProcessStateService and AgentTaskService, and the approval instance half of ApprovalService (approval policy stays in BC-20). WorkflowReplayService is not revived: ADR-0014 promises recovery, not replay.

Context and boundaries

flowchart LR
    S["RunControlService"]
    StudioBff["StudioBff"] -->|sync-http| S
    AgentRuntimeService["AgentRuntimeService"] -->|grpc| S
    AgentCardService["AgentCardService"] -->|grpc| S
    S -->|grpc| WorkflowEngineService["WorkflowEngineService"]
    S -->|grpc| AgentRegistryService["AgentRegistryService"]
    S -->|grpc| PolicyEngineService["PolicyEngineService"]
    S -->|event| CostUsageService["CostUsageService"]
    S -->|event| AgentRuntimeService["AgentRuntimeService"]
    S -->|sync-http| ext_identity_platform["ext-identity-platform"]
    S -->|event| AuditService["AuditService"]
    S -. "RunAccepted" .-> E_RunAccepted(("event"))
    S -. "RunStateChanged" .-> E_RunStateChanged(("event"))
    S -. "RunCompleted" .-> E_RunCompleted(("event"))
    S -. "RunFailed" .-> E_RunFailed(("event"))
    S -. "RunCancelled" .-> E_RunCancelled(("event"))
    S -. "AttemptDispatched" .-> E_AttemptDispatched(("event"))
    S -. "AttemptOutcomeRecorded" .-> E_AttemptOutcomeRecorded(("event"))
    S -. "LeaseExpired" .-> E_LeaseExpired(("event"))
    S -. "ApprovalRequested" .-> E_ApprovalRequested(("event"))
    S -. "ApprovalDecided" .-> E_ApprovalDecided(("event"))

Hold "Alt" / "Option" to enable pan & zoom

Boundaries that hold regardless of deployment shape:

• Scope (tenant, project, requester) is taken from the authenticated context under canonical ADR-0100/0101 and ADR-0015; prompt text and caller headers never supply it.
• A run pins releaseId, policyRevision, modelProfileId and snapshotId at acceptance; nothing changes them mid-flight. A moved alias affects only future submissions.
• Who may decide an approval is fixed by ADR-0015 point 6 (Reviewer or Workspace admin, never the requester of that run; Platform admin break-glass with an audited reason); which action classes need a decision is fixed by D21 and evaluated by BC-20 at request time.

Domain model

Run — One accepted unit of work pinned to an immutable asset release and policy revision; the only object whose state the Studio and workers read as truth. (catalog entry)

Field	Type	Required	Description
runId	guid	yes	Run identity (ADR-0014 envelope field).
tenantId	string	yes	Trusted tenant from the authenticated context (ADR-0015).
projectId	string	yes	Trusted project scope.
environmentId	string	no	Environment within the project, when the workflow targets one.
idempotencyKey	string	yes	Client key; same key + same input returns the same run.
workflowId	string	yes	Published workflow definition id (BC-38).
releaseId	string	yes	Pinned asset release (BC-04).
policyRevision	string	yes	Policy revision evaluated at acceptance (BC-18).
modelProfileId	string	yes	Approved model profile requested at submission (BC-07).
state	enum:RunState	yes
inputHash	hash	yes	Hash of the canonical submission payload.
snapshotId	guid	yes	Resolved execution snapshot (immutable after acceptance).
budgetReservationId	guid	no	Reservation from Usage Accounting (BC-05).
requestedBy	string	yes	Subject id of the requester (never a reviewer of this run; D18).
version	long	yes	Optimistic concurrency token.

Invariants:

• Same idempotencyKey with the same inputHash returns the existing run; a different inputHash is a conflict (409).
• A run pins releaseId, policyRevision and snapshotId at acceptance; alias moves never change them.
• Every state transition is committed in the same transaction as its outbox record; publication alone is never proof of a transition.
• Cancellation records intent first; completed external effects stay visible and are never rolled back silently.
• The requester of a run is never eligible to decide an approval of that run (D18 separation of duties).

stateDiagram-v2
    Accepted --> Running : FirstAttemptDispatched
    Running --> WaitingForApproval : ApprovalRequested
    WaitingForApproval --> Running : ApprovalGranted
    WaitingForApproval --> Failed : ApprovalRejectedOrExpired
    Accepted --> Cancelled : CancelRequestedBeforeDispatch
    Running --> Cancelling : CancelRequested
    WaitingForApproval --> Cancelling : CancelRequested
    Cancelling --> Cancelled : AllAttemptsSettled
    Running --> Completed : LastStepSucceeded
    Running --> Failed : StepFailedTerminally

Hold "Alt" / "Option" to enable pan & zoom

Events: RunAccepted, RunStateChanged, RunCompleted, RunFailed, RunCancelled · Tables: Runs, CancellationIntents, RecoveryJournal · Repository: RunRepository

Step — An ordered unit of a run bound to one agent/skill binding of the pinned release. (catalog entry)

Field	Type	Required	Description
stepId	guid	yes
runId	ref:Run	yes
tenantId	string	yes
ordinal	int	yes
agentId	string	yes	Agent definition id from the release.
skillId	string	no
state	enum:StepState	yes
maxAttempts	int	yes
gate	string	no	none

Invariants:

• Only one non-terminal attempt may exist per step.
• A step cannot be dispatched while the run is WaitingForApproval or Cancelling.

Events: — · Tables: Steps · Repository: StepRepository

Attempt — One execution try of a step; the outcome is reported exactly once by the current lease holder. (catalog entry)

Field	Type	Required	Description
attemptId	guid	yes
stepId	ref:Step	yes
tenantId	string	yes
number	int	yes
leaseId	guid	no
outcome	enum:AttemptOutcome	no
outcomeHash	hash	no	Hash of the outcome receipt.
failureClass	string	no	transient
dispatchedAt	datetime	yes
settledAt	datetime	no

Invariants:

• An outcome is accepted only from the holder of the current lease and only once; a stale fencing token is rejected and audited.
• An Uncertain outcome never retries automatically when the step's side-effect class is shared-write or above.

Events: AttemptDispatched, AttemptOutcomeRecorded · Tables: Attempts · Repository: AttemptRepository

Lease — Time-boxed exclusive right of one runtime worker to execute an attempt, with a fencing token. (catalog entry)

Field	Type	Required	Description
leaseId	guid	yes
attemptId	ref:Attempt	yes
tenantId	string	yes
holder	string	yes	Runtime worker identity (BC-06 / BC-26).
expiresAt	datetime	yes
fencingToken	long	yes

Invariants:

• Exactly one current lease per attempt; an expired lease is reconciled before any retry is dispatched.
• fencingToken strictly increases per attempt.

Events: LeaseExpired · Tables: Leases · Repository: LeaseRepository

ApprovalRequest — A human decision bound to the exact proposed artifact or action version of a run (approvals page; policy from BC-20). (catalog entry)

Field	Type	Required	Description
approvalId	guid	yes
runId	ref:Run	yes
stepId	ref:Step	yes
tenantId	string	yes
actionClass	string	yes	Side-effect class of the gated action (D21).
proposalHash	hash	yes	Hash of the exact artifact, diff, manifest or tool arguments.
policyRevision	string	yes
approverRule	string	yes	Eligibility rule resolved from BC-20 (role, scope, separation of duties).
expiresAt	datetime	yes
state	enum:ApprovalState	yes
decidedBy	string	no
decidedAt	datetime	no
reason	string	no
version	long	yes

Invariants:

• A decision authorizes exactly the referenced proposalHash; a regenerated proposal makes the pending request Stale.
• One valid decision per request; a second decision is a conflict (409).
• A wrong-tenant, expired, duplicate or stale decision never advances the run.

stateDiagram-v2
    Pending --> Approved : Approve
    Pending --> Rejected : Reject
    Pending --> Expired : ExpiryReminder
    Pending --> Stale : ProposalChanged
    Pending --> Cancelled : RunCancelled

Hold "Alt" / "Option" to enable pan & zoom

Events: ApprovalRequested, ApprovalDecided · Tables: ApprovalRequests · Repository: ApprovalRequestRepository

erDiagram
    Run {
        guid runId PK
        string tenantId
        string projectId
        string environmentId
        string idempotencyKey
        string workflowId
        string releaseId
        string policyRevision
        string modelProfileId
        enum_RunState state
        hash inputHash
        guid snapshotId
        guid budgetReservationId
        string requestedBy
        long version
    }
    Step {
        guid stepId PK
        ref_Run runId
        string tenantId
        int ordinal
        string agentId
        string skillId
        enum_StepState state
        int maxAttempts
        string gate
    }
    Attempt {
        guid attemptId PK
        ref_Step stepId
        string tenantId
        int number
        guid leaseId
        enum_AttemptOutcome outcome
        hash outcomeHash
        string failureClass
        datetime dispatchedAt
        datetime settledAt
    }
    Lease {
        guid leaseId PK
        ref_Attempt attemptId
        string tenantId
        string holder
        datetime expiresAt
        long fencingToken
    }
    ApprovalRequest {
        guid approvalId PK
        ref_Run runId
        ref_Step stepId
        string tenantId
        string actionClass
        hash proposalHash
        string policyRevision
        string approverRule
        datetime expiresAt
        enum_ApprovalState state
        string decidedBy
        datetime decidedAt
        string reason
        long version
    }
    CancellationIntent {
        string requestedBy

        string reason

        datetime requestedAt

    }
    RecoveryJournalEntry {
        long sequence

        string fromState

        string toState

        string cause

        datetime occurredAt

    }
    ReleaseRef {
        string releaseId

        hash hash

    }
    PolicyRef {
        string policyId

        string revision

    }
    Run ||--o{ CancellationIntent : contains
    Run ||--o{ RecoveryJournalEntry : contains
    Run ||--o{ Step : "runId"
    Step ||--o{ Attempt : "stepId"
    Attempt ||--o{ Lease : "attemptId"
    Run ||--o{ ApprovalRequest : "runId"
    Step ||--o{ ApprovalRequest : "stepId"

Hold "Alt" / "Option" to enable pan & zoom

The run state machine is the ADR-0014 vocabulary. Rejected and Expired are approval outcomes: they terminate the run as Failed with failure class policy-denied or cancellation, so that every run has one terminal vocabulary. Cancelling is a real state because interruption is requested from a lease holder and observed, never assumed; an attempt whose side effect may already have happened settles as Uncertain and is surfaced to an operator instead of being retried.

Entities and value objects

CancellationIntent (entity, in Run) — Recorded request to cancel with reason and actor.

Field	Type	Required	Description
requestedBy	string	yes
reason	string	yes
requestedAt	datetime	yes

RecoveryJournalEntry (entity, in Run) — Append-only journal of transitions used for restart recovery (no replay promise; ADR-0014).

Field	Type	Required	Description
sequence	long	yes
fromState	string	yes
toState	string	yes
cause	string	yes
occurredAt	datetime	yes

ReleaseRef (value-object) — Release id + content hash.

Field	Type	Required	Description
releaseId	string	yes
hash	hash	yes

PolicyRef (value-object) — Policy revision used at acceptance.

Field	Type	Required	Description
policyId	string	yes
revision	string	yes

Data model and storage

Store	Role	Engine / profile	Tables
Control plane relational store	primary	azure-sql / nhibernate-sqlserver	Runs, CancellationIntents, RecoveryJournal, Steps, Attempts, Leases, ApprovalRequests
Control plane relational store	outbox	azure-sql / nhibernate-sqlserver	—
Cache and coordination	cache	redis / azure-cache	—

Table	Owner	Columns	Indexes
dbo.Runs	Run	17	2
dbo.CancellationIntents	CancellationIntent	5	0
dbo.RecoveryJournal	RecoveryJournalEntry	7	0
dbo.Steps	Step	9	1
dbo.Attempts	Attempt	10	2
dbo.Leases	Lease	6	2
dbo.ApprovalRequests	ApprovalRequest	14	2

Full DDL: appendix.

Every table carries TenantId and is reached only through the NHibernate tenant filter (SharedDb mode of canonical ADR-0100; DatabasePerTenant and the silo modes drop the filter but keep the column). RecoveryJournal is append-only at the database-permission level and is the first table in the restore order (D30). The outbox table of the selected messaging profile lives in the same database so that a transition and its event commit together (ADR-0012 §4).

APIs

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
POST	/factory/runs	SubmitRun	factory.runs.write	shared-write	key-header	GW-03	Accept a run; 202 with the run location; idempotent on Idempotency-Key.
GET	/factory/runs/{runId}	GetRun	factory.runs.read	read	natural	GW-03	Run with steps, attempts, current lease and pending approvals.
GET	/factory/runs	ListRuns	factory.runs.read	read	natural	GW-03	Runs in the trusted scope with state and freshness; paged.
POST	/factory/runs/{runId}/cancel	CancelRun	factory.runs.write	shared-write	natural	GW-03	Record cancellation intent and request interruption; the resulting state is observed, not assumed.
POST	/factory/approvals/{approvalId}/decision	DecideApproval	factory.approvals.decide	shared-write	natural	GW-03	Approve, reject or request changes on the exact proposal version; requires the request version (optimistic concurrency).
GET	/factory/approvals	ListApprovals	factory.approvals.read	read	natural	GW-03	Pending and decided approval requests visible to the caller's roles.

gRPC service	Method	Streaming	Visibility	Side effects	Summary
RunControl	AcquireLease	none	internal	shared-write	Runtime worker acquires or renews the lease of an attempt; returns the fencing token.
RunControl	ReportAttemptOutcome	none	internal	shared-write	Lease holder reports the single outcome of an attempt with its receipt hash.
RunControl	RequestApproval	none	internal	shared-write	Worker asks for a human decision on an exact proposal before a gated action.
RunControl	WatchRun	server	internal	read	Server stream of run and approval state changes for the BFF and workers.

Submission is idempotent on the Idempotency-Key header within scope + operation: the same key with the same canonical payload returns the existing run (202 with the same location); the same key with a different payload returns 409. A decision carries the approval request version; a mismatch returns 409, an expired or stale request returns 410. Example submission:

{
  "workflowId": "repository-assessment",
  "releaseAlias": "latest",
  "modelProfileId": "code-agent",
  "input": { "repository": "ConnectSoft.CaseManagement", "revision": "a1b2c3d" },
  "limits": { "maxCostUsd": 12, "maxMinutes": 45 }
}

The response names the resolved release, the policy revision and the snapshot id, so the client can see what was pinned before the first attempt runs.

Messages

Event	Topic	Scope / v	Aggregate	Consumers	Summary
RunAccepted	run-control.runs.v1.run-accepted	integration / 1	Run	WorkflowEngineService, CostUsageService, StudioBff, DashboardService	A run was accepted and pinned to a release, policy revision and snapshot.
RunStateChanged	run-control.runs.v1.run-state-changed	integration / 1	Run	StudioBff, DashboardService	A run moved between states (Accepted, Running, WaitingForApproval, Cancelling, Completed, Failed, Cancelled).
RunCompleted	run-control.runs.v1.run-completed	integration / 1	Run	GenerationRecipeService, FeedbackService, DashboardService, MemoryPromotionService, CostUsageService, ArtifactService, StudioBff, QualityScoreService	Terminal success of a run with the hash of its result reference.
RunFailed	run-control.runs.v1.run-failed	integration / 1	Run	GenerationRecipeService, CostUsageService, StudioBff, DashboardService	Terminal failure with a failure class (transient, permanent, policy-denied, validation, cancellation, uncertain).
RunCancelled	run-control.runs.v1.run-cancelled	integration / 1	Run	SecretBrokerService, WorkflowEngineService, AgentRuntimeService, CostUsageService, StudioBff	All attempts settled after a cancellation intent.
AttemptDispatched	run-control.attempts.v1.attempt-dispatched	integration / 1	Attempt	AgentRuntimeService, DashboardService	An attempt was dispatched to a lease holder with its fencing token.
AttemptOutcomeRecorded	run-control.attempts.v1.attempt-outcome-recorded	integration / 1	Attempt	ExecutionCapacityService, WorkflowEngineService, CostUsageService, QualityScoreService, DashboardService	The outcome of an attempt was recorded exactly once.
LeaseExpired	run-control.leases.v1.lease-expired	integration / 1	Lease	SecretBrokerService, AgentRuntimeService, DashboardService	A lease passed its expiry without an outcome; the attempt is reconciled before any retry.
ApprovalRequested	run-control.approvals.v1.approval-requested	integration / 1	ApprovalRequest	StudioBff, StudioNotificationService, AuditService	A human decision is required on an exact proposal version.
ApprovalDecided	run-control.approvals.v1.approval-decided	integration / 1	ApprovalRequest	LicenseService, MarketplaceCatalogService, FleetIncidentService, FleetRolloutService, IaCProvisioningService, EnvironmentPromotionService, QualityScoreService, MemoryPromotionService, WorkflowEngineService, AgentRegistryService, AgentRuntimeService, StudioBff, AuditService	A decision was recorded (approved, rejected, expired, stale or cancelled).

RunAccepted payload:

Field	Type	Required	Description
runId	guid	yes
tenantId	string	yes
projectId	string	yes
workflowId	string	yes
releaseId	string	yes
policyRevision	string	yes
modelProfileId	string	yes
requestedBy	string	yes

RunStateChanged payload:

Field	Type	Required	Description
runId	guid	yes
tenantId	string	yes
fromState	enum:RunState	yes
toState	enum:RunState	yes
cause	string	yes

RunCompleted payload:

Field	Type	Required	Description
runId	guid	yes
tenantId	string	yes
outcomeHash	hash	yes

RunFailed payload:

Field	Type	Required	Description
runId	guid	yes
tenantId	string	yes
failureClass	string	yes
reason	string	yes

RunCancelled payload:

Field	Type	Required	Description
runId	guid	yes
tenantId	string	yes
reason	string	yes
requestedBy	string	yes

AttemptDispatched payload:

Field	Type	Required	Description
runId	guid	yes
stepId	guid	yes
attemptId	guid	yes
leaseId	guid	yes
fencingToken	long	yes
agentId	string	yes
skillId	string	no

AttemptOutcomeRecorded payload:

Field	Type	Required	Description
runId	guid	yes
stepId	guid	yes
attemptId	guid	yes
outcome	enum:AttemptOutcome	yes
failureClass	string	no
outcomeHash	hash	no

LeaseExpired payload:

Field	Type	Required	Description
attemptId	guid	yes
leaseId	guid	yes
holder	string	yes

ApprovalRequested payload:

Field	Type	Required	Description
approvalId	guid	yes
runId	guid	yes
stepId	guid	yes
tenantId	string	yes
actionClass	string	yes
proposalHash	hash	yes
approverRule	string	yes
expiresAt	datetime	yes

ApprovalDecided payload:

Field	Type	Required	Description
approvalId	guid	yes
runId	guid	yes
tenantId	string	yes
state	enum:ApprovalState	yes
decidedBy	string	no
reason	string	no

Command	Handled by	Result events	Summary
SubmitRun	Run	RunAccepted	Accept a run (idempotent on scope + idempotency key).
CancelRun	Run	RunStateChanged, RunCancelled	Record cancellation intent; interruption is requested from the lease holder.
DecideApproval	ApprovalRequest	ApprovalDecided, RunStateChanged	Record one decision bound to the proposal hash and request version.
ReportAttemptOutcome	Attempt	AttemptOutcomeRecorded, RunStateChanged	Accept the single outcome from the current lease holder.

Event	Producer	Handler	Purpose
StepReady	WorkflowEngineService	CreateStepAndDispatch	Create the step (idempotent on stepRunId) and dispatch an attempt or open the approval of a Human step.
WorkflowRunCompleted	WorkflowEngineService	CompleteRun	Record the run as Completed with the output reference.
WorkflowRunFailed	WorkflowEngineService	FailRun	Record the run as Failed with the failure class reported by the engine.
BudgetReserved	CostUsageService	ReleaseFirstStepOnReservation	Dispatch the first attempt once the run budget is reserved.
BudgetReservationDenied	CostUsageService	FailRunOnBudgetDenial	Terminate the run as Failed (policy-denied) when no budget can be reserved.

Events carry the ADR-0014 envelope (tenant, project, run, step, attempt, correlation, causation, producer, schema version). Consumers deduplicate on event id within the retention window of their inbox; ordering is guaranteed per run only, by the outbox dispatcher's ordering key runId.

Workers and jobs

Worker	Trigger	Output events	Retry	Idempotency key	Summary
LeaseExpiryReconciler	schedule */1 * * * *	LeaseExpired, AttemptOutcomeRecorded, AttemptDispatched	exponential	leaseId + fencingToken	Expires overdue leases, marks attempts TimedOut (or Uncertain when a side effect may have happened) and schedules the retry or terminal failure.
ApprovalExpiryReconciler	schedule */5 * * * *	ApprovalDecided, RunStateChanged	fixed	approvalId + version	Moves pending approvals past expiresAt to Expired and fails or re-requests per policy.
RunDispatcher	event	AttemptDispatched, RunStateChanged	exponential	stepId + attempt number	Selects the next Pending step of Running runs and dispatches an attempt with a new lease when capacity (BC-26) allows.

The reconcilers are the only components allowed to move an attempt without a lease holder's report, and they do so only after the lease expired plus the grace period. The dispatcher asks Execution Capacity (BC-26) before dispatching; a denied slot leaves the step Pending and the run Running.

Actors and grains

Grain	Key	State	Reminders	Reentrant	Summary
RunGrain	guid = runId	state, version, openAttemptId	approval-expiry, lease-expiry	no	Serialises all transitions of one run; holds the in-memory copy of state and version and enlists the outbox writer in the same NHibernate transaction.

RunGrain is an Orleans grain keyed by run id that serialises transitions and enlists the outbox writer in the same NHibernate unit of work; grain state is a cache of the persisted row, never the authority. The grain key includes the tenant id through the SaaS grain identity formatter.

Sagas and long-running flows

None modelled

No sagas are modelled for this item yet.

No saga: the run itself is the long-running object, and its journal replaces a process-manager state.

sequenceDiagram
    participant C as Studio / client
    participant RC as RunControlService
    participant U as CostUsageService BC-05
    participant W as AgentRuntimeService BC-06
    participant P as PolicyEngineService BC-18/20
    C->>RC: POST /factory/runs with Idempotency-Key
    RC->>RC: resolve release, policy revision, snapshot, commit Run Accepted with outbox
    RC-->>C: 202 run location
    RC-)U: RunAccepted
    U-)RC: BudgetReserved
    RC->>RC: dispatch first step, Lease, commit with outbox
    RC-)W: AttemptDispatched with fencing token
    W->>P: policy decision for a gated action
    W->>RC: RequestApproval proposalHash
    RC->>RC: Run → WaitingForApproval, commit with outbox
    RC-)C: ApprovalRequested
    C->>RC: POST /factory/approvals/{id}/decision with version
    RC->>P: validate approver, scope, expiry, separation of duties
    RC->>RC: ApprovalDecided, Run → Running, commit with outbox
    W->>RC: ReportAttemptOutcome outcomeHash
    RC->>RC: AttemptOutcomeRecorded, next step or Completed

Hold "Alt" / "Option" to enable pan & zoom

Options and configuration

Section	Key	Type	Default	Reloadable	Secret	Summary
RunControl:Leases	DurationSeconds	int	120	yes	no	Lease TTL before reconciliation.
RunControl:Leases	RenewalGraceSeconds	int	30	yes	no	Grace after expiry before the reconciler acts.
RunControl:Retries	DefaultMaxAttempts	int	3	yes	no	Attempts per step unless the release overrides.
RunControl:Approvals	DefaultExpiryHours	int	72	yes	no	Default approval expiry; platform policy may shorten it.
RunControl:Journal	RetentionDays	int	365	no	no	Recovery journal retention (D31 default).

Metrics and observability

Metric	Instrument	Unit	Dimensions	Summary
connectsoft.factory.runs.accepted	counter	{run}	tenant, project, workflow	Runs accepted.
connectsoft.factory.runs.active	updown	{run}	tenant, state	Runs by non-terminal state.
connectsoft.factory.attempts.duration	histogram	s	tenant, agent, outcome	Dispatch-to-settle duration.
connectsoft.factory.leases.expired	counter	{lease}	tenant	Leases reconciled by expiry.
connectsoft.factory.approvals.pending_age	histogram	s	tenant, actionClass	Age of pending approvals at decision or expiry.
connectsoft.factory.outbox.lag	histogram	s	host	Time between commit and publication by the selected messaging profile.

Spans follow the OpenTelemetry GenAI conventions where a model call is involved (D44); Run Control itself emits spans per command and per transition with run.id, step.id, attempt.id and tenant.id attributes. No prompt or artifact content is logged (D31).

Security

Authentication	oauth2-jwt
Scopes	factory.runs.read (Read runs in scope.); factory.runs.write (Submit and cancel runs.); factory.approvals.read (Read approval requests.); factory.approvals.decide (Decide approvals (Reviewer, Workspace admin; Platform admin break-glass).)
Roles	Requester, Reviewer, Operator, WorkspaceAdmin, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	internal
Max side-effect class	shared-write
Secrets	control-sql (key-vault, rotate 90 d)

Dependencies

Outbound

Target	Kind	Purpose	Failure mode	Required
WorkflowEngineService	grpc	PlanNextSteps when a run is accepted and GetRunGraph for projections (BC-38).	fail-fast	yes
AgentRegistryService	grpc	Resolve the release alias to an immutable release at submission (BC-04).	fail-fast	yes
PolicyEngineService	grpc	Evaluate policy revision and approval eligibility at acceptance and decision time (BC-18/BC-20).	fail-fast	yes
CostUsageService	event	Budget reservation and settlement (BC-05).	queue	yes
AgentRuntimeService	event	Dispatch attempts and receive outcomes (BC-06).	queue	yes
ext-identity-platform	sync-http	Token validation and role claims (ADR-0101).	fail-fast	yes
AuditService	event	Approval and cancellation evidence (BC-19).	queue	yes

Inbound

Target	Kind	Purpose	Failure mode	Required
StudioBff	sync-http	Submit, cancel, read runs; decide approvals.	—	yes
AgentRuntimeService	grpc	Lease acquisition, outcome reporting, approval requests.	—	yes
AgentCardService	grpc	A2A task lifecycle mapped onto runs (D44).	—	yes

Failure and recovery

Scenario	Detection	Recovery	Data impact	RTO (min)
Lease holder crashes mid-attempt	ExpiresAt passed without an outcome	LeaseExpiryReconciler marks the attempt TimedOut (or Uncertain for side-effecting steps) and re-dispatches with a new fencing token	none (outcomes from stale tokens are rejected)	2
Outbox publication lag or transport outage	connectsoft.factory.outbox.lag above threshold	dispatcher retry of the selected messaging profile; transitions are already committed	delayed consumers only	5
Control store restore after loss	restore procedure (D30)	restore order run journal → approvals → releases → outbox/inbox; runs with open leases are reconciled as Uncertain and surfaced to operators	up to RPO 15 min of transitions	240
Duplicate decision submitted concurrently	version mismatch on ApprovalRequest	second decision rejected with 409 and audited	none	0

NFR targets

Target	Class value (pilot-interactive)	Override
latency_p95_ms	2000	500
availability_pct	99.5	—
throughput_rps	50	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	low	—

Deployment

Granularity	module of host · host ControlPlaneApi
Topologies	saas-default, hybrid, on-prem
Runtime	aks; replicas 2–6
Ports	https 7401, grpc 7402, silo 11401, gateway 30401

Run Control ships as a module of the Control plane API host (ControlPlaneApi) in every topology. Extraction into its own host needs a documented trigger (throughput isolation, separate trust boundary or an external consumer of the gRPC surface) and the migration steps in the ownership map.

Current implementation

Nothing of this service exists. ConnectSoft.Extensions.AI.Factory provides an in-process lifecycle (FactoryAgentBase: accept → prompt → run → validate → retry → complete or escalate) with stages Accepted, OutputValidated, Completed, Failed, Escalated and no Cancelled stage; AgentTraceContext carries no tenant, project or run identity; lifecycle events are published through a MassTransit publisher without a transactional outbox. Those facts are recorded in the verification record of 2026-10-08 and in ADR-0014.

Later capabilities

Deterministic replay of a run against a newer release (explicitly not promised by ADR-0014; a separate decision on evidence); bulk approval decisions with per-item outcomes; run templates with pre-approved action classes for trusted workflows; A2A task lifecycle mapping for partner agents (D44, through AgentCardService).

Decisions and traceability

Family	Ids
FR	FR-03, FR-06, FR-10
FUX	FUX-04, FUX-05, FUX-07
NFR	NFR-04, NFR-05, NFR-06
XL	XL-03, XL-05, XL-06

Related

• Approvals · Registries and asset publication · Runtime control · Tenancy and scope

• Agent & Orchestration runtime · FinOps

• Control Plane microservices · bounded contexts · APIs · events · data model

• Services catalog · Ownership map