ExecutionCapacityService

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Bounded context	BC-26 Execution Capacity
Platform / plane	Runtime Cloud / Execution
Deployment granularity	module of host · host RuntimeCloudHost
Status	target · ConnectSoft.Factory.RuntimeCloud.Capacity
Built from	connectsoft-microservice · libraries: connectsoft-extensions-saas-abstractions, connectsoft-extensions-messaging-model, connectsoft-extensions-messaging-model-orleans, connectsoft-extensions-persistence-model-nhibernate
Decisions	ADR-0017, ADR-0024
Traceability	FR-09 FUX-08 NFR-06 NFR-07 NFR-09 XL-14

Purpose and responsibilities

ExecutionCapacityService makes capacity a first-class, governed resource. Worker pools have a kind (agent runtime, sandbox, build runner, browser session), a security boundary (shared, tenant-dedicated, customer-hosted), a region bounded by the tenant's residency rule, size bounds and an autoscaling rule applied through provisioning. Quotas per tenant and project (concurrent runs, concurrent sandboxes, CPU and memory per attempt, execution minutes per day) come from the edition (D34) and may only be tightened. Capacity leases are granted per attempt, sandbox, build or browser session within quota and pool bounds, renewed by heartbeat and released on completion; a denied lease names the limiting quota so the requester sees why a run waits (FUX-08). The PoolGrain serialises grants per pool so accounting is exact.

It absorbs AgentPoolManager and RuntimeScalingService; model-serving compute pools are the Model & Compute platform's (BC-41).

Context and boundaries

flowchart LR
    S["ExecutionCapacityService"]
    AgentRuntimeWorker["AgentRuntimeWorker"] -->|grpc| S
    ToolAdapterService["ToolAdapterService"] -->|grpc| S
    PipelineExecutionService["PipelineExecutionService"] -->|grpc| S
    StudioBff["StudioBff"] -->|sync-http| S
    S -->|event| IaCProvisioningService["IaCProvisioningService"]
    S -->|grpc| FactoryCommercialService["FactoryCommercialService"]
    S -->|grpc| RetentionResidencyService["RetentionResidencyService"]
    S -. "WorkerPoolStateChanged" .-> E_WorkerPoolStateChanged(("event"))
    S -. "QuotaChanged" .-> E_QuotaChanged(("event"))
    S -. "QuotaExceeded" .-> E_QuotaExceeded(("event"))
    S -. "CapacityLeaseGranted" .-> E_CapacityLeaseGranted(("event"))
    S -. "CapacityLeaseReleased" .-> E_CapacityLeaseReleased(("event"))

Hold "Alt" / "Option" to enable pan & zoom

Run Control's attempt lease and the tool executor's sandbox lease are about who may act; the capacity lease is about where and with what resources. Sandboxes with internet egress and browser sessions run only in pools whose boundary allows it (original design §35–38).

Domain model

WorkerPool — A pool of execution capacity of one kind with its security boundary (shared, tenant-dedicated, customer-hosted), region, node profile, size bounds and autoscaling rule, and the current utilisation. (catalog entry)

Field	Type	Required	Description
poolId	string	yes
tenantId	string	yes	Platform tenant for shared pools; a tenant for dedicated ones.
kind	enum:PoolKind	yes
securityBoundary	string	yes	shared
region	string	yes
nodeProfile	string	yes
minSize	int	yes
maxSize	int	yes
currentSize	int	yes
utilisation	decimal	yes
state	enum:PoolState	yes

Invariants:

• Sandboxes with network egress to the internet and browser sessions run only in pools whose security boundary allows it (sandbox controls, original design §35–38).
• A tenant's residency rule bounds the regions of the pools it may use.

Events: WorkerPoolStateChanged · Tables: WorkerPools · Repository: WorkerPoolRepository

Quota — Limits per tenant or project: concurrent runs, concurrent sandboxes, CPU and memory per attempt, execution minutes per day, derived from the edition (D34) and tightened by Workspace admins. (catalog entry)

Field	Type	Required	Description
quotaId	guid	yes
tenantId	string	yes
projectId	string	no
concurrentRuns	int	yes
concurrentSandboxes	int	yes
cpuPerAttempt	decimal	yes
memoryGbPerAttempt	decimal	yes
executionMinutesPerDay	int	yes
source	string	yes	edition

Invariants:

• A project quota may only tighten its tenant's; a tenant quota may only tighten its edition's.

Events: QuotaChanged, QuotaExceeded · Tables: Quotas · Repository: QuotaRepository

CapacityLease — A grant of capacity in a pool to one attempt, sandbox, build or browser session for a bounded time, renewed by heartbeat and released on completion; the unit placement and quotas are enforced on. (catalog entry)

Field	Type	Required	Description
leaseId	guid	yes
tenantId	string	yes
projectId	string	yes
runId	guid	no
attemptId	guid	no
holderKind	string	yes	attempt
holderId	string	yes
poolId	string	yes
cpu	decimal	yes
memoryGb	decimal	yes
state	enum:LeaseState	yes
grantedAt	datetime	no
expiresAt	datetime	no
releasedAt	datetime	no

Invariants:

• A lease is granted only within the holder's quota and the pool's bounds; otherwise Denied with the limiting quota named (the run waits, visible to the requester).
• An expired lease frees capacity; the holder's work is handled by its own lease semantics (attempt lease in Run Control, sandbox lease in the tool executor).

stateDiagram-v2
    Requested --> Granted : CapacityAvailable
    Requested --> Denied : QuotaOrPoolExhausted
    Granted --> Active : HolderStarted
    Active --> Released : HolderCompleted
    Active --> Expired : HeartbeatMissed
    Granted --> Expired : NotStarted

Hold "Alt" / "Option" to enable pan & zoom

Events: CapacityLeaseGranted, CapacityLeaseReleased · Tables: CapacityLeases · Repository: CapacityLeaseRepository

erDiagram
    WorkerPool {
        string poolId PK
        string tenantId
        enum_PoolKind kind
        string securityBoundary
        string region
        string nodeProfile
        int minSize
        int maxSize
        int currentSize
        decimal utilisation
        enum_PoolState state
    }
    Quota {
        guid quotaId PK
        string tenantId
        string projectId
        int concurrentRuns
        int concurrentSandboxes
        decimal cpuPerAttempt
        decimal memoryGbPerAttempt
        int executionMinutesPerDay
        string source
    }
    CapacityLease {
        guid leaseId PK
        string tenantId
        string projectId
        guid runId
        guid attemptId
        string holderKind
        string holderId
        string poolId
        decimal cpu
        decimal memoryGb
        enum_LeaseState state
        datetime grantedAt
        datetime expiresAt
        datetime releasedAt
    }

Hold "Alt" / "Option" to enable pan & zoom

Entities and value objects

None modelled

No entities or value objects are modelled for this item yet.

Data model and storage

Store	Role	Engine / profile	Tables
Runtime Cloud store	primary	azure-sql / nhibernate-sqlserver	WorkerPools, Quotas, CapacityLeases

Table	Owner	Columns	Indexes
dbo.WorkerPools	WorkerPool	11	1
dbo.Quotas	Quota	9	1
dbo.CapacityLeases	CapacityLease	14	3

Full DDL: appendix.

APIs

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
GET	/factory/capacity/pools	ListWorkerPools	factory.capacity.read	read	natural	GW-03	Pools with kind, boundary, region, size and utilisation.
PUT	/factory/capacity/pools/{poolId}	SetWorkerPool	factory.capacity.admin	production	natural	GW-03	Create or change a pool's bounds and autoscaling (Platform admin; applied through provisioning).
GET	/factory/capacity/quotas	GetQuotas	factory.capacity.read	read	natural	GW-03	Effective quotas for the tenant and its projects with the edition source.
PUT	/factory/capacity/quotas/{scope}	SetQuota	factory.capacity.admin	shared-write	natural	GW-03	Tighten a project or tenant quota (Workspace admin) or set edition defaults (Platform admin).
GET	/factory/capacity/leases	ListCapacityLeases	factory.capacity.read	read	natural	GW-03	Active and recent leases by holder, pool and state (Operator view of who is using what).

gRPC service	Method	Streaming	Visibility	Side effects	Summary
Capacity	AcquireLease	none	internal	shared-write	Request capacity for an attempt, sandbox, build or browser session; returns the pool and lease or the limiting quota.
Capacity	RenewLease	none	internal	shared-write	Heartbeat renewal.
Capacity	ReleaseLease	none	internal	shared-write	Release on completion.

Messages

Event	Topic	Scope / v	Aggregate	Consumers	Summary
WorkerPoolStateChanged	capacity.pools.v1.worker-pool-state-changed	integration / 1	WorkerPool	StudioBff, SloService, AlertRuleService	A pool changed state or size.
QuotaChanged	capacity.quotas.v1.quota-changed	integration / 1	Quota	RunControlService, StudioBff, AuditService	An effective quota changed.
QuotaExceeded	capacity.quotas.v1.quota-exceeded	integration / 1	Quota	RunControlService, StudioBff, DashboardService, CostUsageService	A lease was denied by a quota; the holder waits.
CapacityLeaseGranted	capacity.leases.v1.capacity-lease-granted	integration / 1	CapacityLease	DashboardService, CostUsageService	Capacity was granted in a pool.
CapacityLeaseReleased	capacity.leases.v1.capacity-lease-released	integration / 1	CapacityLease	DashboardService, CostUsageService	Capacity was released or expired.

WorkerPoolStateChanged payload:

Field	Type	Required	Description
poolId	string	yes
tenantId	string	yes
kind	enum:PoolKind	yes
fromState	enum:PoolState	yes
toState	enum:PoolState	yes
currentSize	int	yes

QuotaChanged payload:

Field	Type	Required	Description
quotaId	guid	yes
tenantId	string	yes
projectId	string	no
concurrentRuns	int	yes
concurrentSandboxes	int	yes

QuotaExceeded payload:

Field	Type	Required	Description
tenantId	string	yes
projectId	string	yes
holderKind	string	yes
holderId	string	yes
limit	string	yes

CapacityLeaseGranted payload:

Field	Type	Required	Description
leaseId	guid	yes
tenantId	string	yes
holderKind	string	yes
holderId	string	yes
poolId	string	yes
expiresAt	datetime	yes

CapacityLeaseReleased payload:

Field	Type	Required	Description
leaseId	guid	yes
tenantId	string	yes
holderKind	string	yes
holderId	string	yes
poolId	string	yes
state	enum:LeaseState	yes
executionSeconds	long	yes

Command	Handled by	Result events	Summary
SetWorkerPool	WorkerPool	WorkerPoolStateChanged
SetQuota	Quota	QuotaChanged

Event	Producer	Handler	Purpose
AttemptOutcomeRecorded	RunControlService	ReleaseAttemptLease	Release the attempt's lease when its outcome is recorded (belt and braces with the worker's release).
SandboxReleased	ToolAdapterService	ReleaseSandboxLease	Release sandbox capacity.
ProvisioningApplied	IaCProvisioningService	PoolProvisioned	Pool size changes are applied by provisioning.

Workers and jobs

Worker	Trigger	Output events	Retry	Idempotency key	Summary
Autoscaler	schedule */2 * * * *	WorkerPoolStateChanged	fixed	poolId + window	Adjusts pool sizes within bounds from utilisation and queued leases; size changes go through provisioning.
LeaseExpirer	schedule */1 * * * *	CapacityLeaseReleased	none	leaseId + expiry	Expires leases whose heartbeat stopped and frees their capacity.

Actors and grains

Grain	Key	State	Reminders	Reentrant	Summary
PoolGrain	string = poolId	currentSize, grantedCpu, grantedMemoryGb, queuedRequests	queue-drain, lease-expiry	no	Serialises lease grants per pool so capacity accounting is exact under concurrency; drains the queue as capacity frees.

Sagas and long-running flows

None modelled

No sagas are modelled for this item yet.

sequenceDiagram
    participant W as AgentRuntimeWorker
    participant C as ExecutionCapacityService (PoolGrain)
    participant Q as Quota
    participant P as IaCProvisioningService
    W->>C: AcquireLease (attempt, cpu 2, memory 4 GB)
    C->>Q: within tenant and project quota?
    alt capacity available
        C-->>W: lease (pool, expiresAt)
        loop every 60 s
            W->>C: RenewLease
        end
        W->>C: ReleaseLease
    else pool exhausted
        C->>C: queue, Autoscaler raises pool size within bounds
        C-)P: pool size change (provisioning)
        C-->>W: granted when capacity frees, or Denied with the limiting quota
    end

Hold "Alt" / "Option" to enable pan & zoom

Options and configuration

Section	Key	Type	Default	Reloadable	Secret	Summary
RuntimeCloud:Capacity	LeaseTtlSeconds	int	120	yes	no	Lease time-to-live between heartbeats.
RuntimeCloud:Capacity	EditionDefaults	json	{ starter: { concurrentRuns: 5, concurrentSandboxes: 5, executionMinutesPerDay: 600 }, team: { concurrentRuns: 25, concurrentSandboxes: 25, executionMinutesPerDay: 6000 } }	yes	no	Quota defaults per edition (D34).

Metrics and observability

Metric	Instrument	Unit	Dimensions	Summary
connectsoft.factory.capacity.pool_utilisation	gauge	1	poolId, kind	Pool utilisation.
connectsoft.factory.capacity.leases_active	updown	{lease}	tenant, holderKind	Active leases.
connectsoft.factory.capacity.lease_wait	histogram	s	tenant, holderKind	Time from request to grant.
connectsoft.factory.capacity.quota_denials	counter	{denial}	tenant, limit	Leases denied by quota.

Security

Authentication	client-credentials
Scopes	factory.capacity.read (Read pools, quotas and leases.); factory.capacity.admin (Pools (Platform admin) and quotas (Workspace admin tightening, Platform admin defaults).); factory.capacity.lease (Acquire, renew and release (runtime workers and the tool executor).)
Roles	Operator, WorkspaceAdmin, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	internal
Max side-effect class	production
Secrets	—

Dependencies

Outbound

Target	Kind	Purpose	Failure mode	Required
IaCProvisioningService	event	Pool size changes applied through provisioning.	queue	yes
FactoryCommercialService	grpc	Edition of the tenant for quota defaults.	degrade	yes
RetentionResidencyService	grpc	Residency bounds on pool regions.	fail-fast	yes

Inbound

Target	Kind	Purpose	Failure mode	Required
AgentRuntimeWorker	grpc	Attempt leases.	—	yes
ToolAdapterService	grpc	Sandbox and browser-session leases.	—	yes
PipelineExecutionService	grpc	Build-runner leases for Factory-hosted runners (later).	—	yes
StudioBff	sync-http	Capacity views.	—	yes

Failure and recovery

Scenario	Detection	Recovery	Data impact	RTO (min)
Pool exhausted	queued leases growing	autoscale within bounds; beyond bounds the request waits and the requester sees why (FUX-08); alert on sustained queueing	delayed runs	10

NFR targets

Target	Class value (pilot-interactive)	Override
latency_p95_ms	2000	—
availability_pct	99.5	—
throughput_rps	50	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	low	—

Deployment

Granularity	module of host · host RuntimeCloudHost
Topologies	saas-default, hybrid, on-prem, hybrid-compute
Runtime	—; replicas —–—
Ports	—

Current implementation

Nothing: Orleans silos and AKS node pools are sized by hand per deployment; no quota or lease exists.

Later capabilities

Spot and burst pools with cost-aware placement; customer-hosted pools enrolled through an agent for hybrid; per-pool GPU profiles for local model workloads shared with BC-41.

Decisions and traceability

Family	Ids
FR	FR-09
FUX	FUX-08
NFR	NFR-06, NFR-07, NFR-09
XL	XL-14

Related

• Overview · AgentRuntimeWorker · ToolAdapterService · ComputePoolService · FactoryCommercialService

• Runtime Cloud microservices · bounded contexts · APIs · events · data model

• Services catalog · Ownership map