AIFACTORY-FEAT-047: ExecutionCapacityService: complete delivery¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Runtime Cloud (PF-09) · Increment I2 Deliver it · State elaborated · Route backend-developer · Build order 98 of 185 (Factory code)
ExecutionCapacityService is the service of bounded context BC-26 (Execution Capacity). Worker pools (agent runtime, sandbox, build runner, browser) with security boundary, size and autoscaling; quotas per tenant and project tied to the edition; capacity leases per attempt, sandbox, build or session, granted within quota, renewed and released, so placement is explicit and no tenant starves another (BC-26; pool manager and scaling rows merged). This feature completes the service after its thin delivery in I1: every aggregate, operation, event and worker of the model. Repository ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService.
| Delivers | ExecutionCapacityService (complete) |
| Stubs | FactoryCommercialService as static, completed in I7RetentionResidencyService as static, completed in I4 |
| Realizes | adrs: ADR-0017, ADR-0024 |
| Requirements | FR-09, FUX-08, NFR-06, NFR-07, NFR-09, XL-14 |
| Depends on | AIFACTORY-FEAT-046 ExecutionCapacityService: thin delivery for the first outcome |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml |
Scenarios to trace (13, catalog): SCN-ExecutionCapacityService-001 ✓, SCN-ExecutionCapacityService-002 ✓, SCN-ExecutionCapacityService-003 ✓, SCN-ExecutionCapacityService-004 ✓, SCN-ExecutionCapacityService-005 ✓, SCN-ExecutionCapacityService-006 ✓, SCN-ExecutionCapacityService-007 ✓, SCN-ExecutionCapacityService-008 ✓, SCN-ExecutionCapacityService-009 ✓, SCN-ExecutionCapacityService-010 ✓, SCN-ExecutionCapacityService-011 ✓, SCN-ExecutionCapacityService-012 ✓, SCN-ExecutionCapacityService-013 ✓
AIFACTORY-US-028: Capacity is leased from pools of every kind under quotas, so no tenant starves another¶
behavior · persona platform-admin
As a platform administrator, I want worker pools with their own security boundary and quotas per tenant and project, so that one tenant cannot exhaust the capacity of another and every lease is accounted for.
This is the integration PBI of the complete delivery. The thin delivery of I1 (feature execution-capacity-service-thin) leases sandboxes with the workspace of the run from one statically declared pool, without quotas; this feature adds everything else the model of ExecutionCapacityService holds and proves on a deployed instance that it works together. Pools of the four kinds, agent runtime, sandbox, build runner and browser session, each with its security boundary, region, node profile, bounds and autoscaling, are read and set through the REST operations behind the platform gateway RuntimeCloudHost (AIFACTORY-US-1041) and change state and size through provisioning and the Autoscaler, while an exhausted pool answers a lease Denied with the reason pool-exhausted and the pool named exactly as in I1 and the holder retries (D129, AIFACTORY-US-1042). Quotas per tenant and project, derived from the edition (D34) and tightened by Workspace admins, are read and set (AIFACTORY-US-1043) and enforced at lease time, so a tenant at its limit waits and QuotaExceeded names the limit while another tenant is still served (AIFACTORY-US-1044); RunControlService then dispatches no further step of the project whose attempt lease a quota denied, without counting an attempt, and dispatches the waiting steps again when a quota changes, through its handlers HoldDispatchOnQuota and ResumeDispatchOnQuota, which this feature delivers in the repository of RunControlService (AIFACTORY-US-1609, AIFACTORY-US-1610, D140). Leases are granted for attempts, builds and browser sessions as well as sandboxes, placed by kind, boundary and region, and the lease of an attempt is also released when RunControlService records its outcome (AIFACTORY-US-1045); operators see who uses what through ListCapacityLeases (AIFACTORY-US-1046). The REST operations are reached only through RuntimeCloudHost, which establishes the scope once from the token (ADR-0028); the service answers a missing scope with 403 and a resource of another tenant with 404 (ADR-0015). The static stubs of the thin delivery for IaCProvisioningService are retired; FactoryCommercialService stays a static stub until I7 and RetentionResidencyService until I4, as the feature declares. When the store execution-capacity-sql is lost, readiness fails, the store is restored from backup within the RPO of 15 minutes, desired state is read again, observed state is re-inventoried and leases expire and are granted again (NFR-06). No deletion-order handler is built or registered here; that is I4 with RetentionResidencyService.
Acceptance criteria
| AC1 | Given the complete delivery deployed with its REST operations behind RuntimeCloudHost, pools of the four kinds and two tenants, when a Workspace admin of the first tenant sets its concurrentSandboxes quota through RuntimeCloudHost with ExecutionCapacityService/SetQuota to the number of its open sandbox leases and both tenants then request a sandbox through the internal gRPC method ExecutionCapacityService/Capacity.AcquireLease, which does not pass through the gateway, then the first is denied with ExecutionCapacityService/QuotaExceeded naming the limit while the second is granted with ExecutionCapacityService/CapacityLeaseGranted; and when the Workspace admin raises the quota again within the edition through the same operation, ExecutionCapacityService/QuotaChanged is published and the retried request of the first tenant is granted, which is the end-to-end proof on the deployed instance of the rules of AIFACTORY-US-1043 and AIFACTORY-US-1044 (FUX-08, NFR-09). |
| AC2 | Given the chain of the first outcome that ran against the thin delivery, when it runs against the complete delivery, then ExecutionCapacityService/Capacity.AcquireLease, ExecutionCapacityService/Capacity.RenewLease and ExecutionCapacityService/Capacity.ReleaseLease answer with the same messages and the same workspaceRef behaviour, the static stub of IaCProvisioningService is gone, and the stubs of FactoryCommercialService and RetentionResidencyService remain as model/backlog/features/runtime-cloud.yaml declares. |
| AC3 | Given callers through RuntimeCloudHost without factory.capacity.read or factory.capacity.admin, and callers addressing a dedicated pool or a project quota of another tenant, when they reach ExecutionCapacityService/ListWorkerPools, ExecutionCapacityService/SetWorkerPool, ExecutionCapacityService/GetQuotas, ExecutionCapacityService/SetQuota or ExecutionCapacityService/ListCapacityLeases, then the answer is 403 for the missing scope and 404 for the foreign resource, each with a problem response and no state change (ADR-0015, ADR-0028). |
| AC4 | Given open leases on the deployed instance and the store execution-capacity-sql lost, when readiness is read and the store is restored from backup in a recovery drill, then readiness fails until the store is back, the restore lies within the RPO of 15 minutes, every ExecutionCapacityService/WorkerPool is read again from its desired state, and every ExecutionCapacityService/CapacityLease open at the loss expires and its holder is granted a new lease on its next request (NFR-06, docs/planes/execution/environments/services/execution-capacity-service/index.md#failure-and-recovery). |
| AC5 | Given leases, denials and quota changes of two tenants and two projects, when the metrics, logs and activities are read, then the five metrics of model/services/runtime-cloud/ExecutionCapacityService.yaml are emitted with their dimensions and with the tenant and the project by default, connectsoft.factory.capacity.quota_denials counts each ExecutionCapacityService/QuotaExceeded, the run and the attempt appear on activities and log records only, and no stored content reaches telemetry (ADR-0019). |
| Traces: services | ExecutionCapacityService, AgentRuntimeService, ToolAdapterService, RuntimeCloudHost |
| Traces: aggregates | ExecutionCapacityService/WorkerPool, ExecutionCapacityService/Quota, ExecutionCapacityService/CapacityLease |
| Traces: apis | ExecutionCapacityService/ListWorkerPools, ExecutionCapacityService/SetWorkerPool, ExecutionCapacityService/GetQuotas, ExecutionCapacityService/SetQuota, ExecutionCapacityService/ListCapacityLeases, ExecutionCapacityService/Capacity.AcquireLease, ExecutionCapacityService/Capacity.RenewLease, ExecutionCapacityService/Capacity.ReleaseLease |
| Traces: events | ExecutionCapacityService/WorkerPoolStateChanged, ExecutionCapacityService/QuotaChanged, ExecutionCapacityService/QuotaExceeded, ExecutionCapacityService/CapacityLeaseGranted, ExecutionCapacityService/CapacityLeaseReleased |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015, ADR-0017, ADR-0019, ADR-0024, ADR-0028 |
| Traces: requirements | FUX-08, NFR-06, NFR-09 |
| Relates: screens | capacity |
| Relates: nfrs | NFR-06, NFR-07, NFR-09 |
| Depends on | AIFACTORY-US-1041, AIFACTORY-US-1042, AIFACTORY-US-1043, AIFACTORY-US-1044, AIFACTORY-US-1045, AIFACTORY-US-1046, AIFACTORY-US-1609, AIFACTORY-US-1610, feature:execution-capacity-service-thin, feature:runtime-cloud-host, feature:ia-cprovisioning-service, feature:run-control-service |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/backlog/features/runtime-cloud.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-283 API surface: 5 REST and 3 gRPC operations (ListWorkerPools, SetWorkerPool, GetQuotas, SetQuota, ListCapacityLeases, Capacity.AcquireLease and 2 more) |
implementation | The five REST operations are hosted for the gateway route GW-03 of RuntimeCloudHost with scope enforcement per route, the roles PlatformAdmin and WorkspaceAdmin where the model names them, problem responses for 403, 404 and 422 and natural idempotency of the two PUT operations, beside the three gRPC methods of the thin delivery; the OpenAPI document and the proto match model/services/runtime-cloud/ExecutionCapacityService.yaml; the behaviour of each operation is delivered by AIFACTORY-US-1041 to AIFACTORY-US-1046. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-286 Security, tenancy and telemetry of ExecutionCapacityService |
implementation | Scopes and roles of the model are enforced, every query is tenant-filtered with shared pools visible to every tenant and dedicated pools to their own, a foreign resource answers 404 and a missing scope 403, secrets come from secret references, the five metrics carry their modelled dimensions and the tenant and project by default, and traces, logs and health checks follow the observability conventions without stored content. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-287 Retire the thin scope and the stubs of ExecutionCapacityService |
implementation | Callers run against the complete service; the static declaration of the one pool and the static stub of IaCProvisioningService of the thin delivery are deleted from the code, the FAILED_PRECONDITION answer for the holder kinds attempt, build and browser is gone, and the stubs of FactoryCommercialService and RetentionResidencyService that this feature still declares stay behind their client ports. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4195 Recovery drill of the store execution-capacity-sql from backup |
operations | A recovery drill on the deployed instance takes the store execution-capacity-sql away and restores it from backup; the drill record shows readiness failing until the restore, a restore point within the RPO of 15 minutes, the pools read again from their desired state and the leases open at the loss expired and granted again on the next request of their holders, as AC4 of AIFACTORY-US-028 states. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-781 Verify capacity-pools-and-quotas by running the declared command |
verification | Run dotnet test -c Release --filter Category=Acceptance in the pipeline of the repository; a run that discovers no test counts as a failure; it passes and proves AC1, AC2, AC3 and AC5 of AIFACTORY-US-028: the quota set through RuntimeCloudHost, the internal lease call denied and granted again after QuotaChanged, the unchanged lease contract of the thin delivery, the refusals through RuntimeCloudHost and the telemetry conformance over two tenants and two projects; AC4, the restored store, is proven by the drill record of AIFACTORY-TASK-4195. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1041: List worker pools and create or change the bounds and autoscaling of a pool as a Platform admin¶
behavior · persona platform-admin
As a platform administrator, I want to see every pool with its kind, boundary, region, size and utilisation and to set the bounds of a pool, so that capacity of each kind exists where tenants need it and never beyond the bounds I set.
A caller whose scope carries factory.capacity.read lists pools through ExecutionCapacityService/ListWorkerPools, reached through the route GW-03 of RuntimeCloudHost: each pool with its kind, security boundary, region, node profile, minimum, maximum and current size, utilisation and state. Under the tenant row filter the caller sees the shared pools of the platform tenant and the dedicated pools of its own tenant; a dedicated pool of another tenant is never listed. A Platform admin whose scope carries factory.capacity.admin creates or changes a pool through ExecutionCapacityService/SetWorkerPool with its kind, boundary, region, node profile, bounds and autoscaling rule. A shared pool belongs to the platform tenant; the tenant of a tenant-dedicated or customer-hosted pool is the tenant that the trusted scope of the Platform admin names, so a Platform admin gives a tenant a dedicated pool by acting in the scope of that tenant. A pool id that exists nowhere is created: the pool is written in state Provisioning and answered 201. A change of the bounds of a Ready pool moves it to Scaling and is answered 200. WorkerPoolStateChanged is written to the outbox in the same transaction as the change (AIFACTORY-US-1042). The side-effect class of the operation is production, so each create or change also writes audit evidence in the same transaction: the Factory sink of the pinned audit library (feature component-audit-factory-evidence) writes an audit-evidence-v1 record to the outbox with the event type AuditEvidenceSubmitted on the topic audit.evidence.v1.evidence-submitted, with the tenant and the actor from the trusted scope and the pool id, never the body beyond its references. The operation is naturally idempotent: the same body again is answered 200 and writes nothing, no evidence included. It is refused with 422 when the minimum exceeds the maximum, when the region lies outside the residency bounds of the tenant of the pool, which RetentionResidencyService answers through its static stub until I4 and which fails fast when it cannot be read, or when the boundary does not allow the kind, since sandboxes with egress to the internet and browser sessions run only in pools whose boundary allows it. A caller without the scope, or a Workspace admin, is refused with 403; a pool id that exists under another tenant, a dedicated pool of that tenant, is answered 404 and is never overwritten (ADR-0015). The modelled codes of SetWorkerPool and of SetQuota do not include 404; the answer follows ADR-0015 and owner decision (2), and the missing code is reported to the owner. The size is applied through provisioning: the pool reaches it when ProvisioningApplied is consumed (AIFACTORY-US-1042). How the desired size reaches IaCProvisioningService is not modelled, because that service consumes no event of this one; the gap is reported and this PBI does not decide it.
Acceptance criteria
| AC1 | Given shared pools, a dedicated pool of the caller tenant and a dedicated pool of another tenant, when a caller with factory.capacity.read calls ExecutionCapacityService/ListWorkerPools, then the response is 200 with the shared pools and the own dedicated pool, each with kind, boundary, region, size, utilisation and state, and the pool of the other tenant never appears; a caller without the scope is answered 403 and nothing is written (SCN-ExecutionCapacityService-001, ADR-0015). |
| AC2 | Given a Platform admin with factory.capacity.admin acting in the trusted scope of a tenant, when ExecutionCapacityService/SetWorkerPool creates a tenant-dedicated pool under a pool id that exists nowhere and then changes the bounds of a Ready pool, then the first is answered 201 with the ExecutionCapacityService/WorkerPool in state Provisioning under the tenant of the scope and the second 200 with the pool in state Scaling, and each writes ExecutionCapacityService/WorkerPoolStateChanged and an AuditEvidenceSubmitted record of audit-evidence-v1 to the outbox in the same transaction; the same body repeated is answered 200 and writes nothing (SCN-ExecutionCapacityService-002). |
| AC3 | Given a body whose minimum exceeds its maximum, a region outside the residency bounds of the tenant, or a boundary that does not allow the kind, when ExecutionCapacityService/SetWorkerPool is called, then the response is 422 with a problem response naming the rule and nothing is written; given the residency stub cannot be read, then the call fails fast and nothing is written (SCN-ExecutionCapacityService-002, model/services/runtime-cloud/ExecutionCapacityService.yaml). |
| AC4 | Given a caller without factory.capacity.admin, a Workspace admin, and a Platform admin acting in the scope of one tenant who addresses a pool id that exists as a dedicated pool of another tenant, when ExecutionCapacityService/SetWorkerPool is called, then the response is 403, 403 and 404 respectively, each with a problem response, and the pool of the other tenant is not changed and nothing is written (SCN-ExecutionCapacityService-002, ADR-0015). |
| Traces: services | ExecutionCapacityService, RuntimeCloudHost |
| Traces: aggregates | ExecutionCapacityService/WorkerPool |
| Traces: apis | ExecutionCapacityService/ListWorkerPools, ExecutionCapacityService/SetWorkerPool |
| Traces: events | ExecutionCapacityService/WorkerPoolStateChanged |
| Traces: descriptors | audit-evidence-v1 |
| Traces: adrs | ADR-0014, ADR-0015, ADR-0017, ADR-0019, ADR-0028 |
| Traces: scenarios | SCN-ExecutionCapacityService-001, SCN-ExecutionCapacityService-002 |
| Relates: screens | capacity |
| Depends on | feature:runtime-cloud-host, feature:component-audit-factory-evidence |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#apis |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4184 ListWorkerPools and SetWorkerPool with the residency and boundary rules |
implementation | ListWorkerPools returns the shared pools and the dedicated pools of the caller tenant under the row filter; SetWorkerPool writes a shared pool under the platform tenant and a dedicated or customer-hosted pool under the tenant of the trusted scope of the Platform admin, creates a pool id that exists nowhere in state Provisioning with 201 or changes the bounds of a Ready pool to Scaling with 200, writes WorkerPoolStateChanged and, through the Factory sink of the pinned audit library, an AuditEvidenceSubmitted record on audit.evidence.v1.evidence-submitted in the same transaction, answers a repeated body 200 without writing, refuses with 422 an inverted bound, a region outside the residency bounds read through the static RetentionResidencyService stub with fail-fast handling, and a boundary that does not allow the kind, answers 403 without the scope or for a Workspace admin and 404 for a pool id that exists under another tenant. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-281 Domain model: WorkerPool, Quota, CapacityLease |
implementation | Aggregates with their invariants and state transitions as modelled: the WorkerPool with its bounds, boundary and residency invariants and its states, the Quota that only tightens its tenant or its edition, and the CapacityLease of every holder kind with the six modelled transitions and the RunWorkspace kept from the thin delivery; unit tests cover every invariant and every transition. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4185 Verify reading and setting worker pools through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ExecutionCapacityService-001\|Scenario=SCN-ExecutionCapacityService-002)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1041: the row-filtered list, the create with 201 under the tenant of the scope and the change with 200 with their WorkerPoolStateChanged and AuditEvidenceSubmitted outbox records, the idempotent repeat, the three 422 rules, the fail-fast residency stub and the 403 and 404 refusals. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1042: Move pools through provisioning, autoscaling and draining and announce each state or size change¶
behavior · persona operator
As an operator, I want pools to grow and shrink within their bounds from demand and every change of state or size to be announced, so that queued work is served as soon as capacity can be added and alerts and health follow the real pool.
A pool changes state or size through three paths. The inbox handler PoolProvisioned consumes ProvisioningApplied of IaCProvisioningService, idempotent on the event id: the pool the provisioning run applied takes its new current size and moves from Provisioning or Scaling to Ready, in the transaction of the handler. The payload of that event names no pool id, only outputs; the pool is identified from the outputs, and the missing member is reported to the owner. The worker Autoscaler runs every two minutes, partitioned by pool with at most four in parallel, with two fixed retries thirty seconds apart and the idempotency key pool id plus window: from utilisation and the requests queued in PoolGrain it asks for a new size within the minimum and maximum of the pool and moves a Ready pool to Scaling; a size outside the bounds is never asked for. A pool in state Draining or Unavailable grants no new lease, and the leases running in it end normally; which command moves a pool to Draining or Unavailable is not modelled and is reported. Capacity.AcquireLease is a unary method, so a grant made later has no path back to a caller that is no longer waiting on the call. An exhausted pool therefore answers exactly as in I1: the lease moves from Requested to Denied (trigger QuotaOrPoolExhausted), the reply carries the reason pool-exhausted and names the pool as the limit, no capacity is granted, and the holder asks again with backoff (D129), ToolAdapterService as its failure mode for an exhausted sandbox capacity states. The denied request is recorded in queuedRequests of PoolGrain only as demand: the Autoscaler reads it to raise the size within bounds, and connectsoft.factory.capacity.lease_wait measures, for a holder with a queued request, the time from its first denied request to the grant of its retry. The retry of the holder is granted once capacity frees. The reminder queue-drain of PoolGrain fires every five seconds while requests are queued and only reserves freed capacity for the oldest queued holder, so that its retry is granted before a later request of another holder takes the capacity; the reminder grants nothing by itself and sends nothing to a holder. As decided for I2 (D129), the denied request is recorded as demand, freed capacity is reserved for the oldest waiting holder, and a reservation not claimed within the lease time to live passes to the next; a queued holder is matched by its holder kind and holder id, its entry leaves queuedRequests when its retry is granted, and a reservation that its holder does not claim by a retry within RuntimeCloud:Capacity:LeaseTtlSeconds passes to the next queued holder. Beyond the bounds the holder keeps being answered Denied with the pool named, so the requester sees why (FUX-08). The model states a five-second reminder while the amendment of ADR-0024 records that Orleans reminders are documented for periods of minutes or longer; this PBI implements the period of the model and the conflict is reported. Every change of state or size writes WorkerPoolStateChanged with poolId, tenantId, kind, fromState, toState and currentSize to the outbox in the transaction of the change; StudioBff, AlertRuleService and SloService consume it in their own PBIs, the alert on sustained queueing among them.
Acceptance criteria
| AC1 | Given a pool in state Provisioning or Scaling, when IaCProvisioningService/ProvisioningApplied for it is consumed, then the ExecutionCapacityService/WorkerPool takes its new current size and moves to Ready with ExecutionCapacityService/WorkerPoolStateChanged written in the same transaction, and a redelivery of the same event id changes nothing (SCN-ExecutionCapacityService-009). |
| AC2 | Given a Ready pool with queued requests and utilisation above its rule, when the Autoscaler runs, then a new size within the minimum and maximum is asked for, the pool moves to Scaling with ExecutionCapacityService/WorkerPoolStateChanged, a second trigger in the same window has no second effect, and a pool at its maximum is asked for no size beyond it while its queued holders keep being answered Denied with the pool named (FUX-08, ADR-0024). |
| AC3 | Given a pool in state Draining, set through a test fixture until the command that drains a pool is decided, with running leases, when a lease of its kind is requested through ExecutionCapacityService/Capacity.AcquireLease, then no lease is granted from that pool and the running leases renew and release normally until they end (model/services/runtime-cloud/ExecutionCapacityService.yaml). |
| AC4 | Given the only pool of the kind exhausted, when a holder requests a lease through ExecutionCapacityService/Capacity.AcquireLease, then the ExecutionCapacityService/CapacityLease moves from Requested to Denied by QuotaOrPoolExhausted, the reply carries the reason pool-exhausted and names the pool exactly as in I1, no capacity is granted and the request is recorded in queuedRequests of PoolGrain; given a lease of the pool is then released and queue-drain reserves the freed capacity for that oldest queued holder, then a request of another holder is still answered Denied, the retry of the queued holder is granted with ExecutionCapacityService/CapacityLeaseGranted, and connectsoft.factory.capacity.lease_wait records the time from its first denied request to that grant (FUX-08, ADR-0024). |
| AC5 | Given a pool changed state or size, when the outbox dispatcher publishes ExecutionCapacityService/WorkerPoolStateChanged on capacity.pools.v1.worker-pool-state-changed, then the event is sent under the run event envelope with poolId, tenantId, kind, fromState, toState and currentSize, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-ExecutionCapacityService-009, ADR-0012). |
| Traces: services | ExecutionCapacityService, IaCProvisioningService |
| Traces: aggregates | ExecutionCapacityService/WorkerPool, ExecutionCapacityService/CapacityLease |
| Traces: apis | ExecutionCapacityService/Capacity.AcquireLease |
| Traces: events | ExecutionCapacityService/WorkerPoolStateChanged, ExecutionCapacityService/CapacityLeaseGranted, IaCProvisioningService/ProvisioningApplied |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0024 |
| Traces: requirements | FUX-08, NFR-09 |
| Traces: scenarios | SCN-ExecutionCapacityService-009 |
| Depends on | AIFACTORY-US-1041, feature:ia-cprovisioning-service |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/services/runtime-cloud/IaCProvisioningService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#workers-and-jobs, docs/architecture/adr/0024-durable-execution-substrate.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-336 Worker pools with autoscaling |
implementation | Pools of each kind are declared with their boundary and size; the Autoscaler follows utilisation and the queue of PoolGrain within the declared bounds, every two minutes partitioned by pool with at most four in parallel, two fixed retries and the idempotency key pool id plus window, and writes WorkerPoolStateChanged with every state or size change; a pool in Draining or Unavailable grants no new lease. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-285 Workers and grains: Autoscaler hosting, the PoolProvisioned handler and the queue-drain reminder of PoolGrain |
implementation | The Autoscaler is hosted with its modelled trigger, retry policy and idempotency key so a duplicate trigger has no second effect; the handler of ProvisioningApplied sets the size and the Ready state of the pool idempotently on the event id; AcquireLease answers a request on an exhausted pool Denied by QuotaOrPoolExhausted with the reason pool-exhausted and the pool named, as in I1, and records it in queuedRequests of PoolGrain as demand for the Autoscaler and for connectsoft.factory.capacity.lease_wait; the queue-drain reminder, at the period the model states, only reserves freed capacity for the oldest queued holder, matched by holder kind and holder id, so that its retry is granted first, grants nothing by itself, and passes a reservation not claimed within RuntimeCloud:Capacity:LeaseTtlSeconds to the next queued holder. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4186 Verify pool state, provisioning and autoscaling through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ExecutionCapacityService-009"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-1042 with a fixture ProvisioningApplied event and a test clock: the Ready transition, the redelivered event, the Autoscaler within bounds and its duplicate window, the pool at its maximum whose holders are answered Denied, the pool set to Draining by a test fixture, the Denied reply with the reason pool-exhausted and the pool named on an exhausted pool in I2, the reservation of queue-drain that lets the retry of the oldest queued holder be granted before another holder, lease_wait from the first denied request to that grant, and the publication and payload of WorkerPoolStateChanged with a stopped transport. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1043: Read the effective quotas of a tenant and its projects and tighten a quota as a Workspace admin¶
behavior · persona workspace-admin
As a workspace administrator, I want to see the quotas my tenant and projects run under and to tighten them, so that one project cannot take the whole share of my tenant and every change is recorded as evidence.
A caller whose scope carries factory.capacity.read reads through ExecutionCapacityService/GetQuotas the effective quotas of its tenant and of each of its projects: concurrent runs, concurrent sandboxes, cpu and memory per attempt and execution minutes per day, each with its source, edition or override. The defaults of the edition come from RuntimeCloud:Capacity:EditionDefaults (D34) for the edition of the tenant, which FactoryCommercialService answers through its static stub until I7 and which degrades when it cannot be read, as the model states for that dependency. A Workspace admin whose scope carries factory.capacity.admin tightens the quota of the tenant or of one of its projects through ExecutionCapacityService/SetQuota: the Quota row of that scope is written with the source override and QuotaChanged with quotaId, tenantId, projectId, concurrentRuns and concurrentSandboxes is written to the outbox in the same transaction. A project quota may only tighten the quota of its tenant and a tenant quota only the defaults of its edition; a looser value, a negative value or an unknown scope is refused with 422. The operation is naturally idempotent: the same body again is answered 200 and writes nothing. A caller without the scope is refused with 403, and so is a Workspace admin who addresses the defaults of an edition, which only a Platform admin sets; a project of another tenant is answered 404 (ADR-0015). The model keeps the defaults of an edition in the option RuntimeCloud:Capacity:EditionDefaults and has no row for an edition in Quotas, so where a value a Platform admin sets through this operation is kept is reported to the owner and this PBI does not decide it. RunControlService consumes QuotaChanged to resume the steps that wait for capacity (ResumeDispatchOnQuota, AIFACTORY-US-1610 of this feature), StudioBff refreshes its quota view and AuditService records the change as evidence, each in its own PBI.
Acceptance criteria
| AC1 | Given a tenant on an edition with one project override, when a caller with factory.capacity.read calls ExecutionCapacityService/GetQuotas, then the response is 200 with the effective quota of the tenant and of each project, each limit with its source edition or override, the defaults read from RuntimeCloud:Capacity:EditionDefaults for the edition of the tenant; a caller without the scope is answered 403 and nothing is written (SCN-ExecutionCapacityService-003). |
| AC2 | Given a Workspace admin with factory.capacity.admin, when ExecutionCapacityService/SetQuota tightens the quota of a project of its tenant, then the response is 200, the ExecutionCapacityService/Quota of that project is written with the source override, and ExecutionCapacityService/QuotaChanged is written to the outbox in the same transaction; the same body repeated is answered 200 and writes nothing (SCN-ExecutionCapacityService-004). |
| AC3 | Given a project quota looser than its tenant quota, a tenant quota looser than its edition, a negative value, a caller without the scope, a Workspace admin addressing the defaults of an edition and a project of another tenant, when ExecutionCapacityService/SetQuota is called, then the response is 422, 422, 422, 403, 403 and 404 respectively, each with a problem response and no state change (SCN-ExecutionCapacityService-004, ADR-0015). |
| AC4 | Given a quota was changed, when the outbox dispatcher publishes ExecutionCapacityService/QuotaChanged on capacity.quotas.v1.quota-changed, then the event is sent under the run event envelope with quotaId, tenantId, projectId where the quota is of a project, concurrentRuns and concurrentSandboxes, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-ExecutionCapacityService-010, ADR-0012). |
| Traces: services | ExecutionCapacityService, RuntimeCloudHost |
| Traces: aggregates | ExecutionCapacityService/Quota |
| Traces: apis | ExecutionCapacityService/GetQuotas, ExecutionCapacityService/SetQuota |
| Traces: events | ExecutionCapacityService/QuotaChanged |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015 |
| Traces: requirements | FUX-08 |
| Traces: scenarios | SCN-ExecutionCapacityService-003, SCN-ExecutionCapacityService-004, SCN-ExecutionCapacityService-010 |
| Relates: screens | capacity |
| Depends on | feature:runtime-cloud-host |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#apis |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4187 GetQuotas and SetQuota with the edition source and the tightening invariant |
implementation | GetQuotas returns the effective quota of the tenant and its projects with the source of each limit, the edition read through the static FactoryCommercialService stub and its defaults from RuntimeCloud:Capacity:EditionDefaults; SetQuota writes the override of a tenant or project scope with QuotaChanged in the same transaction, refuses a looser or negative value and an unknown scope with 422, a missing scope and a Workspace admin on edition defaults with 403 and a project of another tenant with 404, and answers a repeated body 200 without writing. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-282 Persistence in execution-capacity-sql: WorkerPools, Quotas, CapacityLeases |
migration | Mappings and migrations of the complete delivery on top of the thin tables: the Quotas table with UX_Quotas_Scope over tenant and project, the columns of the complete WorkerPools and CapacityLeases and the tenant filters; the retention of 90 days of CapacityLeases is applied by the query filter of ListCapacityLeases, and no purge of rows is built here because the model declares no retention worker for CapacityLeases or RunWorkspaces and no deletion-order handler belongs to this increment; mapping tests run against an isolated test database, never a shared one. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4188 Verify reading and setting quotas through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ExecutionCapacityService-003\|Scenario=SCN-ExecutionCapacityService-004\|Scenario=SCN-ExecutionCapacityService-010)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1043: the effective quotas with their sources, the tightened override with its QuotaChanged outbox record, the idempotent repeat, the 422, 403 and 404 refusals and the publication and payload of QuotaChanged with a stopped transport. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1044: Deny a lease that would exceed a quota of its tenant or project and announce the limit that denied it¶
behavior · persona workspace-admin
As a workspace administrator, I want a lease that would exceed a quota to be denied with the limit named, so that my tenant waits within its share, no other tenant is starved and the denial is visible and counted.
Every request of ExecutionCapacityService/Capacity.AcquireLease is checked against the effective quota of its project and of its tenant before the capacity of a pool, the tighter limit winning: concurrent runs, concurrent sandboxes, counted as the open sandbox leases, cpu and memory per attempt against the request of an attempt lease, and execution minutes per day. Concurrent runs and execution minutes are counted by the reading of D129: concurrent runs are counted as the distinct runs of the scope with an open lease, and execution minutes per day are counted against the executionSeconds of the leases of the scope that ended that day plus the seconds held so far by its open leases. A request within quota proceeds to the pool (AIFACTORY-US-1045). A request that would exceed a limit moves its lease from Requested to Denied (trigger QuotaOrPoolExhausted), the reply names the limiting quota, no capacity is granted, and QuotaExceeded with tenantId, projectId, holderKind, holderId and limit is written to the outbox in the same transaction; connectsoft.factory.capacity.quota_denials counts it with the tenant and the limit. The holder waits and the requester sees why (FUX-08): RunControlService keeps the next steps of the project Pending without counting an attempt (HoldDispatchOnQuota, AIFACTORY-US-1609 of this feature), CostUsageService records the denial, StudioBff refreshes its quota view, each in its own PBI, and when a quota is changed RunControlService evaluates the waiting steps again on QuotaChanged (ResumeDispatchOnQuota, AIFACTORY-US-1610 of this feature). Because quotas are checked per tenant and project before the pool, a tenant at its limit never takes capacity that another tenant within its quota could use, which is how no tenant starves another (NFR-09). A denial is not a refusal of the call: the caller had the scope and a valid request; the refusals of the method stay those of the thin delivery.
Acceptance criteria
| AC1 | Given a project at its concurrentSandboxes limit, when its next sandbox is requested through ExecutionCapacityService/Capacity.AcquireLease, then the ExecutionCapacityService/CapacityLease moves from Requested to Denied, the reply names the limiting quota, no capacity is granted, and ExecutionCapacityService/QuotaExceeded is written to the outbox in the same transaction and counted in connectsoft.factory.capacity.quota_denials (SCN-ExecutionCapacityService-011). |
| AC2 | Given a tenant quota tighter than a project override, an attempt lease asking for more cpu than cpuPerAttempt, and a tenant that used its execution minutes of the day, when each requests a lease through ExecutionCapacityService/Capacity.AcquireLease, then each is denied by the tighter or exhausted limit, which the reply and ExecutionCapacityService/QuotaExceeded name (SCN-ExecutionCapacityService-011, model/services/runtime-cloud/ExecutionCapacityService.yaml). |
| AC3 | Given one tenant at its quota and another within it, both asking the same pool, when their requests reach ExecutionCapacityService/Capacity.AcquireLease, then the first is denied by its quota before the pool is asked and the second is granted from the pool, so the denied tenant never takes the capacity of the other (NFR-09). |
| AC4 | Given a lease was denied by a quota, when the outbox dispatcher publishes ExecutionCapacityService/QuotaExceeded on capacity.quotas.v1.quota-exceeded, then the event is sent under the run event envelope with tenantId, projectId, holderKind, holderId and limit, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-ExecutionCapacityService-011, ADR-0012). |
| Traces: services | ExecutionCapacityService |
| Traces: aggregates | ExecutionCapacityService/Quota, ExecutionCapacityService/CapacityLease |
| Traces: apis | ExecutionCapacityService/Capacity.AcquireLease |
| Traces: events | ExecutionCapacityService/QuotaExceeded |
| Traces: adrs | ADR-0012, ADR-0014 |
| Traces: requirements | FUX-08, NFR-09 |
| Traces: scenarios | SCN-ExecutionCapacityService-011 |
| Depends on | AIFACTORY-US-1043 |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#messages |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-337 Quotas per tenant and project enforced at lease time |
implementation | AcquireLease checks the effective project and tenant quota before the pool, the tighter limit winning, over concurrent runs, concurrent sandboxes, cpu and memory per attempt and execution minutes per day, counting concurrent runs and execution minutes by the reading of D129 that the description of AIFACTORY-US-1044 states; a denial moves the lease to Denied by QuotaOrPoolExhausted, names the limiting quota in the reply, writes QuotaExceeded in the same transaction and increments connectsoft.factory.capacity.quota_denials with tenant and limit. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4189 Verify quota enforcement through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ExecutionCapacityService-011"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1044: the denial at the concurrent sandbox limit, the tighter tenant limit, the cpu per attempt and the daily minutes, the two tenants on one pool where only the one at its quota waits, and the publication and payload of QuotaExceeded with a stopped transport. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1045: Place attempt, build, sandbox and browser leases in pools of their kind and release the lease of an attempt on its recorded outcome¶
behavior · persona operator
As an operator, I want every kind of work to be leased from a pool of its kind, within the boundary and region it may use, so that placement is explicit for agents, builds, sandboxes and browsers alike and no lease outlives its work.
The complete delivery serves the four holder kinds through the same three methods of the thin delivery. AgentRuntimeWorker acquires an attempt lease for an attempt of a run, ToolAdapterService a sandbox or a browser lease and, in a later increment, PipelineExecutionService a build lease, each with a workload identity whose token carries factory.capacity.lease. The model names AgentRuntimeWorker as the inbound caller of attempt leases, but the model of AgentRuntimeWorker declares no outbound call to this service and its feature arrives in I3; this delivery therefore proves the attempt lease with a fake caller and the gap is reported. Within quota (AIFACTORY-US-1044), a lease is placed in a pool of the requested kind that is Ready or Scaling, whose region the residency bounds of the tenant allow, read through the static stub of RetentionResidencyService until I4 with fail-fast handling, and whose boundary allows the holder: a sandbox with egress to the internet and a browser session only in a pool whose boundary allows it, and never in a dedicated pool of another tenant. A lease that names a run receives the workspace of the run as in the thin delivery (D121). The transition from Granted to Active (trigger HolderStarted) is signalled for a sandbox by the start of its sandbox, as in the thin delivery. For the holder kinds attempt, build and browser the service starts nothing on the pool, and the model names no signal; by the reading of D129, the first Capacity.RenewLease of a Granted lease by its holder is its HolderStarted and moves it to Active, so that HeartbeatMissed, Released and executionSeconds are reached for every holder kind, and the choice is reported. A Granted lease that is never renewed expires by NotStarted. Renewal, release and expiry otherwise work as in the thin delivery for every kind and every pool: LeaseExpirer and the reminder lease-expiry of PoolGrain expire leases whose heartbeat stopped in each pool. The inbox handler ReleaseAttemptLease consumes AttemptOutcomeRecorded of RunControlService and releases the lease of that attempt, belt and braces with the release of the worker. The payload of that event carries no tenantId, so the handler takes the tenant from the run event envelope of the event and finds the lease by attemptId within that tenant under the tenant row filter. It is idempotent on the event id and on the lease state, so a lease already released by its holder is not released twice and one CapacityLeaseReleased exists per lease. The messaging profile execution-capacity now publishes its five events and registers its six consumed events. The refusals of the methods are those of the thin delivery, without the FAILED_PRECONDITION for the holder kinds the thin delivery did not serve. A sandbox lease whose egress allow list is not empty, or a browser lease, for which no pool of its kind visible to the tenant has a security boundary that allows it is still refused with FAILED_PRECONDITION and nothing written, because no freed capacity would ever serve it. A request for a kind that no Ready or Scaling pool serves is answered exactly as an exhausted pool (AIFACTORY-US-1042): the lease moves from Requested to Denied by QuotaOrPoolExhausted, the reply carries the reason pool-exhausted and names the kind as the limit, the request is recorded as demand, and the retry of the holder is granted once a pool of that kind is Ready.
Acceptance criteria
| AC1 | Given pools of the four kinds in two regions and a tenant whose residency allows one region, when an attempt, a build, a sandbox and a browser lease are requested through ExecutionCapacityService/Capacity.AcquireLease, then each ExecutionCapacityService/CapacityLease is granted from a pool of its kind in the allowed region with ExecutionCapacityService/CapacityLeaseGranted, never from a dedicated pool of another tenant (ADR-0015, ADR-0017). |
| AC2 | Given a browser lease and a sandbox lease with egress to the internet, and pools whose boundary does or does not allow them, when they are requested through ExecutionCapacityService/Capacity.AcquireLease, then they are placed only in the pools whose boundary allows them, and the residency stub that cannot be read fails the request fast with nothing written (model/services/runtime-cloud/ExecutionCapacityService.yaml). |
| AC3 | Given an Active attempt lease, when RunControlService/AttemptOutcomeRecorded of that attempt is consumed, then the lease is released with ExecutionCapacityService/CapacityLeaseReleased; given the worker had already released it through ExecutionCapacityService/Capacity.ReleaseLease, or the event is redelivered, then nothing is written a second time (model/services/control-plane/RunControlService.yaml). |
| AC4 | Given leases of every kind in several pools whose holders stop renewing, when LeaseExpirer runs, then each moves to Expired in its own pool with ExecutionCapacityService/CapacityLeaseReleased and its capacity returns to that pool, and a lease of any kind that names a run leaves the workspace of the run in place (ADR-0024, D121). |
| AC5 | Given an attempt lease granted through ExecutionCapacityService/Capacity.AcquireLease, when its holder calls ExecutionCapacityService/Capacity.RenewLease for the first time, then the ExecutionCapacityService/CapacityLease moves from Granted to Active by HolderStarted; when the holder then calls ExecutionCapacityService/Capacity.ReleaseLease, then the lease moves to Released and ExecutionCapacityService/CapacityLeaseReleased carries the state Released and executionSeconds; given a request for a kind that no Ready or Scaling pool serves, then the lease is Denied with the kind named, and given a browser lease for which no visible pool has a boundary that allows it, then the call is refused with FAILED_PRECONDITION and nothing is written. |
| Traces: services | ExecutionCapacityService, AgentRuntimeService, AgentRuntimeWorker, ToolAdapterService, RunControlService |
| Traces: aggregates | ExecutionCapacityService/CapacityLease, ExecutionCapacityService/WorkerPool |
| Traces: apis | ExecutionCapacityService/Capacity.AcquireLease, ExecutionCapacityService/Capacity.RenewLease, ExecutionCapacityService/Capacity.ReleaseLease |
| Traces: events | ExecutionCapacityService/CapacityLeaseGranted, ExecutionCapacityService/CapacityLeaseReleased, RunControlService/AttemptOutcomeRecorded |
| Traces: adrs | ADR-0015, ADR-0017, ADR-0024 |
| Traces: requirements | NFR-09 |
| Depends on | AIFACTORY-US-1042, AIFACTORY-US-1044, feature:run-control-service |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/services/control-plane/RunControlService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#sagas-and-long-running-flows |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-338 Lease placement, renewal and expiry for every holder kind and every pool |
implementation | AcquireLease places attempt, build, sandbox and browser leases in a Ready or Scaling pool of the requested kind, in a region the residency stub allows and a boundary that allows the holder, never in a dedicated pool of another tenant; a sandbox lease with a non-empty egress allow list or a browser lease that no visible pool boundary allows is refused with FAILED_PRECONDITION, and a kind that no Ready or Scaling pool serves is answered Denied with the kind named as on an exhausted pool; the first RenewLease of a Granted attempt, build or browser lease moves it to Active by HolderStarted, the reading of D129 that AIFACTORY-US-1045 states, so that an attempt lease reaches Released with executionSeconds through ReleaseLease; leases are renewed by their holders and expired by LeaseExpirer and the lease-expiry reminder of each PoolGrain when they are not, and every lease that names a run keeps receiving the one workspace of the run. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-284 Messaging: publishes 5 events, handles 6 |
implementation | Events are published through the outbox of the profile execution-capacity with the run envelope (WorkerPoolStateChanged, QuotaChanged, QuotaExceeded, CapacityLeaseGranted, CapacityLeaseReleased); the six consumed events, AttemptOutcomeRecorded, RunCompleted, RunFailed and RunCancelled of RunControlService, SandboxReleased of ToolAdapterService and ProvisioningApplied of IaCProvisioningService, are handled idempotently through its inbox and inbox_lag reports the oldest unprocessed message. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4190 ReleaseAttemptLease inbox handler on AttemptOutcomeRecorded |
implementation | The handler of AttemptOutcomeRecorded takes the tenant from the run event envelope of the event, finds the open lease by attemptId within that tenant under the tenant row filter and releases it through the release command of the thin delivery, idempotent on the event id and on the lease state, so a lease the worker already released or an event redelivered writes nothing a second time. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4191 Verify the leases of every holder kind through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=LeasesOfEveryKind"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-1045 with pool fixtures of four kinds in two regions and a fake attempt-lease caller: the placement by kind, region and boundary, the fail-fast residency stub, the attempt lease released on a fixture RunControlService/AttemptOutcomeRecorded found by attemptId in the tenant of its envelope and not twice, the expiry in every pool that leaves the workspace in place, an attempt lease moved from Granted to Active by its first renewal and to Released with executionSeconds, the Denied answer for a kind no pool serves and the FAILED_PRECONDITION refusal of a browser lease that no boundary allows. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1046: List the active and recent capacity leases of a tenant by holder, pool and state¶
behavior · persona operator
As an operator, I want to see who holds capacity in which pool and in which state, so that I can tell why a run waits and which holder keeps a pool busy.
A caller whose scope carries factory.capacity.read, reached through the route GW-03 of RuntimeCloudHost, lists capacity leases through ExecutionCapacityService/ListCapacityLeases: the active leases and the recent ones within the retention of 90 days of CapacityLeases, which the query applies as a filter because the model declares no worker that purges the rows (reported), each with its holder kind and holder id, pool, state, cpu and memory, grant, expiry and release times, filtered by holder, pool and state. Under the tenant row filter only the leases of the caller tenant are listed; a lease of another tenant never appears, also when it holds capacity in a shared pool the caller can see. A denied lease is listed with the state Denied, so an operator sees which holder waits; the limit that denied it is named by QuotaExceeded and by the reply to the holder (AIFACTORY-US-1044). A caller without the scope is refused with 403. Nothing is written by the read, and the answer holds references and numbers only, never the content of a workspace (ADR-0019). The Studio view of capacity and quotas that reads this operation is built by StudioBff in its own increment.
Acceptance criteria
| AC1 | Given leases of the caller tenant in several pools and states and leases of another tenant in a shared pool, when a caller with factory.capacity.read calls ExecutionCapacityService/ListCapacityLeases filtered by pool and state, then the response is 200 with the matching leases of the caller tenant only, each with holder kind, holder id, pool, state, cpu, memory and its times, and the leases of the other tenant never appear (SCN-ExecutionCapacityService-005, ADR-0015). |
| AC2 | Given a caller without factory.capacity.read, when ExecutionCapacityService/ListCapacityLeases is called, then the response is 403 with a problem response and nothing is written; given a denied lease of the tenant, then it is listed with the state Denied (SCN-ExecutionCapacityService-005). |
| AC3 | Given leases released more than 90 days ago and active leases, when ExecutionCapacityService/ListCapacityLeases is called without a state filter, then the active and the recent leases are listed and none beyond the retention of CapacityLeases, and no record of the answer or of its telemetry holds workspace content (ADR-0019, model/services/runtime-cloud/ExecutionCapacityService.yaml). |
| Traces: services | ExecutionCapacityService, RuntimeCloudHost |
| Traces: aggregates | ExecutionCapacityService/CapacityLease |
| Traces: apis | ExecutionCapacityService/ListCapacityLeases |
| Traces: adrs | ADR-0015, ADR-0019 |
| Traces: scenarios | SCN-ExecutionCapacityService-005 |
| Relates: screens | capacity |
| Depends on | AIFACTORY-US-1045, feature:runtime-cloud-host |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#apis |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4192 ListCapacityLeases with filters by holder, pool and state under the tenant row filter |
implementation | ListCapacityLeases returns the active and recent leases of the caller tenant, filtering out in the query every lease released, expired or denied more than 90 days ago, the retention window of CapacityLeases, whether or not its row still exists, filtered by holder, pool and state, with holder kind, holder id, pool, state, cpu, memory and the grant, expiry and release times, Denied leases included, answers 403 without the scope and writes nothing. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4193 Verify the operator view of leases through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ExecutionCapacityService-005"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1046: the filtered list of the caller tenant only, the 403 refusal, the denied lease in the list and the retention boundary. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1609: RunControlService dispatches no further attempt of a project while a quota denies the attempt leases of that project, and counts no attempt¶
behavior · persona workspace-admin
As a workspace administrator, I want the steps of my project to wait Pending while a quota denies the capacity of its attempts, so that no attempt is spent on capacity my project does not have and my runs go on when capacity frees.
The inbox handler HoldDispatchOnQuota of RunControlService takes ExecutionCapacityService/QuotaExceeded, which carries tenantId, projectId, holderKind, holderId and limit. RunControlService itself is delivered in I1 by feature run-control-service, but the producer of quotas arrives with this feature in I2, so the handler is delivered here and its tasks are written in the repository ConnectSoft.AI.SoftwareFactory.RunControlService (D140). The handler registers the event in the inbox of the messaging profile run-control, idempotent on the event id. The purpose the model gives the handler is to keep the step Pending while a quota denies its lease, with no attempt counted. The model keeps no record of a waiting step and names no key for it, and holderId of an attempt lease is not stated, so this PBI reads the hold per project and keeps it, like the other announced states the dispatcher reads, in GateStates (D114): for holderKind attempt the handler writes the row of the tenant with the subject kind quota, the project of the event as the subject, the state exceeded and the limit of the event in Class, a reading reported to the owner because the model describes Class as the class of an environment. While the row of a project is exceeded, the RunDispatcher creates no attempt and no lease for a Pending step of a Running run of that project and does not dispatch the first attempt of an Accepted child run of that project: the step stays Pending, no attempt is counted against maxAttempts, and no AttemptDispatched and no RunStateChanged are written. The attempt whose capacity lease was denied is neither settled nor counted again by the handler; it keeps the rules of its attempt and its lease in RunControlService (D113, D114). A denial of a sandbox, build or browser lease changes nothing here, because its holder waits on its own lease. A concurrency limit frees when a lease of the project is released and the model announces no quota change for it, so this PBI also lifts the hold when the outcome of an attempt of a run of the project is recorded: the RunDispatcher, which AttemptOutcomeRecorded already triggers, removes the row of that project before it evaluates, and a next attempt that the quota still denies writes a new QuotaExceeded and sets the hold again; this reading is reported to the owner. A quota change lifts the hold through ResumeDispatchOnQuota (AIFACTORY-US-1610). The row belongs to one tenant, so a denial for tenant B holds no project of tenant A, also under the same project id (ADR-0015). When run-control-sql does not answer, the inbox keeps the event and retries it, and the event is applied once.
Acceptance criteria
| AC1 | Given a Running RunControlService/Run of tenant A in project P with a Pending RunControlService/Step and no open attempt, when ExecutionCapacityService/QuotaExceeded names tenant A, project P, the holder kind attempt and the limit concurrentRuns and HoldDispatchOnQuota handles it, then the hold of project P is kept with the limit, the RunDispatcher creates no RunControlService/Attempt and no lease for that step, the step stays Pending, no attempt is counted against maxAttempts and no RunControlService/AttemptDispatched is written; a redelivery under the same event id changes nothing (model/services/control-plane/RunControlService.yaml, FUX-08). |
| AC2 | Given project P is held, when RunControlService/AttemptOutcomeRecorded is written for an attempt of a run of project P, then the RunDispatcher removes the hold of project P and dispatches the Pending step with RunControlService/AttemptDispatched, and when the next attempt lease is denied again a new ExecutionCapacityService/QuotaExceeded sets the hold again. |
| AC3 | Given ExecutionCapacityService/QuotaExceeded with the holder kind sandbox, build or browser, an event of tenant B for a project id that tenant A also uses, or run-control-sql stopped while the event arrives, when HoldDispatchOnQuota handles it, then the first two hold no step of tenant A and change no dispatch of tenant A, and the last is retried by the inbox until the store answers and is applied once under its event id (ADR-0012, ADR-0015, NFR-09). |
| Traces: services | RunControlService, ExecutionCapacityService |
| Traces: aggregates | RunControlService/Run, RunControlService/Step, RunControlService/Attempt |
| Traces: events | ExecutionCapacityService/QuotaExceeded, RunControlService/AttemptDispatched, RunControlService/AttemptOutcomeRecorded |
| Traces: adrs | ADR-0012, ADR-0015 |
| Traces: requirements | FUX-08, NFR-09 |
| Depends on | AIFACTORY-US-1044, feature:run-control-service, feature:contracts-package |
| Sources | model/services/control-plane/RunControlService.yaml, model/services/runtime-cloud/ExecutionCapacityService.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-6509 HoldDispatchOnQuota of RunControlService for QuotaExceeded |
implementation | The inbox of the RunControlService messaging profile registers QuotaExceeded with idempotency on the event id; HoldDispatchOnQuota writes for the holder kind attempt the GateStates row of the tenant with the subject kind quota, the project of the event as the subject, the state exceeded and the limit in Class, upserted under the tenant row filter, and changes nothing for the holder kinds sandbox, build and browser. | ConnectSoft.AI.SoftwareFactory.RunControlService |
AIFACTORY-TASK-6510 RunDispatcher honours and lifts the quota hold of a project |
implementation | The RunDispatcher reads the quota hold of the project of each run it evaluates and, while it is exceeded, creates no attempt and no lease for a Pending step of a Running run and no first attempt of an Accepted child run of that project, counts no attempt and writes no AttemptDispatched or RunStateChanged; when AttemptOutcomeRecorded of an attempt of a run of the project triggers it, it removes the hold row of that project before it evaluates. | ConnectSoft.AI.SoftwareFactory.RunControlService |
AIFACTORY-TASK-6511 Fixture quota events and held projects of the quota dispatch suite |
fixture | The acceptance suite of RunControlService holds fixture events QuotaExceeded for the holder kinds attempt, sandbox, build and browser and QuotaChanged with and without a project, for two tenants that share a project id, built from the event types of the pinned contracts package under the run event envelope, Running runs with a Pending step and no open attempt, an Accepted child run whose first step is ready, and a switch that stops run-control-sql while an event arrives. | ConnectSoft.AI.SoftwareFactory.RunControlService |
AIFACTORY-TASK-6512 Verify hold-dispatch-while-a-quota-denies-attempts through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=HoldDispatchOnQuota" in the repository of RunControlService; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1609 with the fixture of AIFACTORY-TASK-6511: the Pending step with no attempt counted while the project is held, the hold lifted on a recorded outcome and set again by a new denial, the holder kinds and the tenant that change nothing, and the retry while the store is stopped with one application under the event id. |
ConnectSoft.AI.SoftwareFactory.RunControlService |
AIFACTORY-US-1610: RunControlService dispatches the waiting steps of a held project again when a quota of its scope changes¶
behavior · persona workspace-admin
As a workspace administrator, I want the steps that wait for capacity dispatched again as soon as I change a quota, so that a raised quota is used at once and my runs never wait on a limit that no longer applies.
The inbox handler ResumeDispatchOnQuota of RunControlService takes ExecutionCapacityService/QuotaChanged, which carries quotaId, tenantId, the optional projectId, concurrentRuns and concurrentSandboxes. Like HoldDispatchOnQuota (AIFACTORY-US-1609), it is delivered with this feature and its tasks are written in the repository ConnectSoft.AI.SoftwareFactory.RunControlService (D140). The handler registers the event in the inbox of the messaging profile run-control, idempotent on the event id. The purpose the model gives the handler is to evaluate again the steps that wait for capacity after a quota change. When the event names a project, the handler removes the quota hold of that project from GateStates; when it names none, the quota is the one of the tenant and the handler removes the quota holds of every project of the tenant. It then asks the RunDispatcher to evaluate the Running runs with a Pending step and no open attempt and the Accepted child runs whose first step is ready in that scope, as GateRunsByEnvironment asks it for the runs of an environment that is active again (D117); the trigger list of the RunDispatcher in the model does not name QuotaChanged, and this call is reported as the reading. The dispatcher dispatches as capacity allows, with AttemptDispatched and a new lease for each attempt, and RunStateChanged when the first attempt moves an Accepted run to Running. RunControlService does not compare the limits of the event with any count: ExecutionCapacityService stays the judge at lease time (AIFACTORY-US-1044), so a limit that still denies the next attempt lease writes a new QuotaExceeded and sets the hold again. A change of a quota of another tenant lifts no hold of this tenant (ADR-0015), and a change for a scope without a hold dispatches nothing the dispatcher would not dispatch anyway. When run-control-sql does not answer, the inbox keeps the event and retries it, and the event is applied once.
Acceptance criteria
| AC1 | Given project P of tenant A is held by a quota and a Running RunControlService/Run of project P has a Pending RunControlService/Step with no open attempt, when ExecutionCapacityService/QuotaChanged names tenant A and project P and ResumeDispatchOnQuota handles it, then the hold of project P is removed and the RunDispatcher dispatches the step with RunControlService/AttemptDispatched and a new lease; a redelivery under the same event id changes nothing (model/services/control-plane/RunControlService.yaml, FUX-08). |
| AC2 | Given projects P and Q of tenant A are held and an Accepted child RunControlService/Run of project Q has its first step ready, when ExecutionCapacityService/QuotaChanged names tenant A and no project, then both holds are removed, the steps of both projects are dispatched, and the first attempt of the child run moves it to Running with RunControlService/RunStateChanged. |
| AC3 | Given ExecutionCapacityService/QuotaChanged of tenant B for a project id that tenant A also holds, or run-control-sql stopped while the event arrives, when ResumeDispatchOnQuota handles it, then the first lifts no hold of tenant A and dispatches no step of tenant A, and the last is retried by the inbox until the store answers and is applied once under its event id (ADR-0012, ADR-0015). |
| Traces: services | RunControlService, ExecutionCapacityService |
| Traces: aggregates | RunControlService/Run, RunControlService/Step |
| Traces: events | ExecutionCapacityService/QuotaChanged, RunControlService/AttemptDispatched, RunControlService/RunStateChanged |
| Traces: adrs | ADR-0012, ADR-0015 |
| Traces: requirements | FUX-08 |
| Depends on | AIFACTORY-US-1043, AIFACTORY-US-1609, feature:run-control-service, feature:contracts-package |
| Sources | model/services/control-plane/RunControlService.yaml, model/services/runtime-cloud/ExecutionCapacityService.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-6513 ResumeDispatchOnQuota of RunControlService for QuotaChanged |
implementation | The inbox of the RunControlService messaging profile registers QuotaChanged with idempotency on the event id; ResumeDispatchOnQuota removes the quota hold of the project the event names, or of every project of the tenant when it names none, under the tenant row filter, and asks the RunDispatcher to evaluate the Running runs with a Pending step and no open attempt and the Accepted child runs whose first step is ready in that scope, without comparing the limits of the event with any count. | ConnectSoft.AI.SoftwareFactory.RunControlService |
AIFACTORY-TASK-6515 Verify resume-dispatch-on-a-quota-change through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=ResumeDispatchOnQuota" in the repository of RunControlService; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1610 with the fixture of AIFACTORY-TASK-6511: the step of the held project dispatched after a quota change of the project, both projects and the child run dispatched after a quota change of the tenant, the change of another tenant that lifts nothing, and the retry while the store is stopped with one application under the event id. |
ConnectSoft.AI.SoftwareFactory.RunControlService |