AIFACTORY-FEAT-046: ExecutionCapacityService: thin delivery for the first outcome¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Runtime Cloud (PF-09) · Increment I1 Generate a microservice · State elaborated · Route backend-developer · Build order 74 of 185 (Factory code)
ExecutionCapacityService is the service of bounded context BC-26 (Execution Capacity). Worker pools (agent runtime, sandbox, build runner, browser) with security boundary, size and autoscaling; quotas per tenant and project tied to the edition; capacity leases per attempt, sandbox, build or session, granted within quota, renewed and released, so placement is explicit and no tenant starves another (BC-26; pool manager and scaling rows merged). In this increment it is delivered thin: Sandbox and workspace leases from one worker pool; no quotas. Its own repository is ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService.
| Delivers | ExecutionCapacityService (thin): Sandbox and workspace leases from one worker pool; no quotas. |
| Stubs | IaCProvisioningService as static, completed in I2FactoryCommercialService as static, completed in I7RetentionResidencyService as static, completed in I4 |
| Realizes | adrs: ADR-0017, ADR-0024 |
| Requirements | FR-09, FUX-08, NFR-06, NFR-07, NFR-09, XL-14 |
| Depends on | — |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml |
Scenarios to trace (13, catalog): SCN-ExecutionCapacityService-001 ✓, SCN-ExecutionCapacityService-002 ✓, SCN-ExecutionCapacityService-003 ✓, SCN-ExecutionCapacityService-004 ✓, SCN-ExecutionCapacityService-005 ✓, SCN-ExecutionCapacityService-006 ✓, SCN-ExecutionCapacityService-007 ✓, SCN-ExecutionCapacityService-008 ✓, SCN-ExecutionCapacityService-009 ✓, SCN-ExecutionCapacityService-010 ✓, SCN-ExecutionCapacityService-011 ✓, SCN-ExecutionCapacityService-012 ✓, SCN-ExecutionCapacityService-013 ✓
AIFACTORY-US-068: ExecutionCapacityService leases sandboxes with the workspace of the run from one pool for the first outcome¶
behavior · persona operator
As an operator, I want ExecutionCapacityService to lease the sandboxes of the first-outcome chain from one worker pool and to own the workspace of each run, so that every tool that writes or builds runs in a sandbox placed by one owner and the files of a run survive from one step to the next.
This is the integration PBI of the thin delivery. ExecutionCapacityService is the service of bounded context BC-26. In its complete form it manages worker pools of four kinds, quotas per tenant and project and capacity leases for attempts, sandboxes, builds and browser sessions; that form arrives with feature execution-capacity-service-complete in I2. The first outcome needs exactly this (the thin scope of model/backlog/features/runtime-cloud.yaml): sandbox and workspace leases from one worker pool, with no quota. ToolAdapterService is the one caller of the thin delivery: its broker stage acquires a sandbox lease through ExecutionCapacityService/Capacity.AcquireLease with the tenant, project, run and attempt of the call, renews it through ExecutionCapacityService/Capacity.RenewLease and ends it through SandboxReleased, which this service consumes, or through ExecutionCapacityService/Capacity.ReleaseLease. The workspace of a run belongs to this service (D121): it is created with the first lease of the run, kept between leases, handed to the holder through the workspaceRef of every lease that names the run and removed when RunControlService announces that the run ended. The repository read tools run inside the sandbox of the tool executor with the workspace mounted read-only (D125), so this service adds no operation that reads a workspace. The feature is delivered by five PBIs: the grant of a lease with the workspace of the run (AIFACTORY-US-1036), renewal and expiry (AIFACTORY-US-1037), release (AIFACTORY-US-1038), the one sandbox pool and its sandboxes (AIFACTORY-US-1039) and the removal of the workspace on a terminal run event (AIFACTORY-US-1040). Nothing else is served in I1: the five REST operations arrive with the complete delivery behind RuntimeCloudHost in I2, no quota exists, so neither QuotaChanged nor QuotaExceeded is written, the pool is declared statically, so no WorkerPoolStateChanged is written, attempt, build and browser leases are not served, and the workers Autoscaler and the handlers of AttemptOutcomeRecorded and ProvisioningApplied arrive in I2. IaCProvisioningService, FactoryCommercialService and RetentionResidencyService are static stubs of this increment. The handler of a deletion order is built in I4 with RetentionResidencyService and none is registered here. The repository ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService is created and scaffolded in I0 by AIFACTORY-TASK-095 of feature repository-and-pipeline-conventions (D140), by the steps of its repository checklist, and this feature starts from that scaffold; it was scaffolded once from the pinned base template package with UseFactoryProfile (features component-base-template-factory-profile and component-base-template-findings): the scaffold brings token validation, the scope chain with its 401, 403 and 404 answers, health and readiness, the Orleans messaging profile execution-capacity whose outbox writes the run event envelope of the contracts package in the transaction of each command (ADR-0012, ADR-0014), and the telemetry wiring. The internal gRPC methods admit a workload token by the service scope factory.capacity.lease; a workload token carries no tenant claim, so the tenant, the project and the run of a lease are those of the Factory scope the caller carries on the call, and a lease of another tenant is answered NOT_FOUND (ADR-0015). Metrics carry the tenant and the project by default, the run and the attempt go on activities and log records only, and no workspace content, path listing or command text reaches telemetry (ADR-0019).
Acceptance criteria
| AC1 | Given the repository ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService created and scaffolded in I0 by AIFACTORY-TASK-095 of feature repository-and-pipeline-conventions, whose first commit is the unchanged output of dotnet new connectsoft-base -n ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService --UseFactoryProfile true from the pinned template package, when its pipeline runs with the work of this feature, then no second scaffold was run, the solution builds, the unit and acceptance tests pass with complete reports and a container image is published (ADR-0028, model/services/runtime-cloud/ExecutionCapacityService.yaml). |
| AC2 | Given the deployed service with its one sandbox pool and a fake ToolAdapterService caller with a workload identity, when two attempts of one run each acquire a lease through ExecutionCapacityService/Capacity.AcquireLease, renew it through ExecutionCapacityService/Capacity.RenewLease and end it, the first by SandboxReleased and the second through ExecutionCapacityService/Capacity.ReleaseLease, and the run then ends with RunControlService/RunCompleted, then a file the first sandbox wrote is read by the second, both leases carry the same workspaceRef, ExecutionCapacityService/CapacityLeaseGranted and ExecutionCapacityService/CapacityLeaseReleased are published twice each, and the workspace is removed once, as AIFACTORY-US-1036 to AIFACTORY-US-1040 prove one by one (D121). |
| AC3 | Given the service descriptor and the proto of the thin delivery, when the contract test compares them with model/services/runtime-cloud/ExecutionCapacityService.yaml and the scope of model/backlog/features/runtime-cloud.yaml, then the three gRPC methods of Capacity are served, none of the five REST operations is routed, the outbox declares ExecutionCapacityService/CapacityLeaseGranted and ExecutionCapacityService/CapacityLeaseReleased only, the inbox registers SandboxReleased and the three terminal run events only, and the test fails when an operation or an event outside the thin scope appears. |
| AC4 | Given calls without workload identity, with a workload token that lacks factory.capacity.lease, and with a lease of another tenant, when they reach any of the three methods of ExecutionCapacityService/Capacity.AcquireLease, ExecutionCapacityService/Capacity.RenewLease and ExecutionCapacityService/Capacity.ReleaseLease, then they are refused with UNAUTHENTICATED, PERMISSION_DENIED and NOT_FOUND, the last the same answer as for a lease that does not exist, and nothing is written; given the service starts, then liveness and readiness answer, readiness only once the store execution-capacity-sql and the messaging profile are reachable (ADR-0015). |
| AC5 | Given leases of two tenants and two projects, when the metrics, logs and activities of the service are read, then connectsoft.factory.capacity.leases_active, connectsoft.factory.capacity.lease_wait, connectsoft.factory.capacity.pool_utilisation and connectsoft.factory.capacity.inbox_lag are emitted with the dimensions of model/services/runtime-cloud/ExecutionCapacityService.yaml, the tenant and the project on the lease metrics by default, the run and the attempt on activities and log records only, and no record holds a workspace path listing, file content or command text (ADR-0019). |
| Traces: services | ExecutionCapacityService, ToolAdapterService, RunControlService |
| Traces: aggregates | ExecutionCapacityService/CapacityLease, ExecutionCapacityService/WorkerPool |
| Traces: apis | ExecutionCapacityService/Capacity.AcquireLease, ExecutionCapacityService/Capacity.RenewLease, ExecutionCapacityService/Capacity.ReleaseLease |
| Traces: events | ExecutionCapacityService/CapacityLeaseGranted, ExecutionCapacityService/CapacityLeaseReleased, RunControlService/RunCompleted |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015, ADR-0017, ADR-0019, ADR-0024, ADR-0028 |
| Traces: requirements | NFR-09 |
| Relates: nfrs | NFR-06, NFR-09 |
| Depends on | AIFACTORY-US-1036, AIFACTORY-US-1037, AIFACTORY-US-1038, AIFACTORY-US-1039, AIFACTORY-US-1040, feature:repository-and-pipeline-conventions, feature:component-base-template-factory-profile, feature:component-base-template-findings, feature:component-observability-factory-conventions, feature:contracts-package, feature:trusted-scope-chain, feature:run-control-service |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/backlog/features/runtime-cloud.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#apis, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-096 Thin host wiring: the three gRPC methods, the messaging profile, the thin tables, tenancy and telemetry |
implementation | The host serves the three gRPC methods of Capacity with workload identity checks and the scope factory.capacity.lease and routes no REST operation; the messaging profile execution-capacity enlists its outbox in the transaction of each command for CapacityLeaseGranted and CapacityLeaseReleased under the run event envelope of the contracts package and registers in its inbox SandboxReleased of ToolAdapterService and RunCompleted, RunFailed and RunCancelled of RunControlService, idempotent on the event id; the migrations of execution-capacity-sql create WorkerPools, CapacityLeases and RunWorkspaces with the tenant row filter and the modelled indexes; the four metrics of the thin scope are registered through the helper of the pinned observability library, the deployment holds the replica range 2 to 6 and the secret reference messaging-transport; the behaviour of each operation is delivered by AIFACTORY-US-1036 to AIFACTORY-US-1040. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-097 Contract of the thin surface and contract tests of the capacity lease messages and the two published events |
contract | The proto fixes the request of AcquireLease with tenant, project, run, attempt, holder kind, holder id, cpu, memory, mount mode and egress allow list, the reply with lease id, state, pool id, expiry, workspaceRef and, for a denial, the limit that denied it, the renew and release requests by lease id and the acknowledgement of a release; the model names acquire-lease-v1, lease-v1 and ack-v1 for these methods, which docs/reference/schemas/attempt-outcome.schema.json defines as the attempt lease of RunControlService, so the gap is reported to the owner and the golden files make the contract checkable without deciding it; the contract tests compare the descriptor with model/services/runtime-cloud/ExecutionCapacityService.yaml in both directions and the payloads of CapacityLeaseGranted and CapacityLeaseReleased with the model, so the complete delivery replaces the thin one without a contract change. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-782 Verify execution-capacity-service-thin-contract by running the declared command |
verification | Run dotnet build -c Release and dotnet test -c Release --no-build in the pipeline of the repository; a run that discovers no test counts as a failure; the runs pass and prove AC1 to AC5 of AIFACTORY-US-068, that is the scaffold, the two-attempt chain over one workspace with a fake ToolAdapterService caller and a fake RunControlService publisher (SCN-ExecutionCapacityService-006, SCN-ExecutionCapacityService-008), the surface limited to the thin scope, the refusals of the three methods and the telemetry conformance over two tenants and two projects. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1036: Grant a sandbox lease from the one pool and hand over the workspace of the run, created with its first lease¶
behavior · persona operator
As an operator, I want every sandbox of the chain to be granted from the pool through one lease that also names the workspace of its run, so that placement is explicit, the pool is never oversubscribed and a later step finds the files an earlier step wrote.
ToolAdapterService, with a workload identity whose token carries factory.capacity.lease, calls ExecutionCapacityService/Capacity.AcquireLease for a sandbox with the tenant, the project, the run and the attempt of the tool call, the holder kind sandbox and the invocation as holder id, the cpu and memory of the call, the mount mode and the egress allow list. The tenant, project and run are those of the Factory scope the call carries; a workload token has no tenant claim. The request is serialised per pool by PoolGrain (ADR-0024). Grain serialisation alone does not guarantee exact accounting: the amendment of ADR-0024 records that Orleans allows occasional duplicate activations with the default grain directory and offers no lease or fencing feature. The grant therefore commits with a concurrency check on the granted cpu and memory of the pool in execution-capacity-sql, and a grant whose check fails is evaluated again, so a duplicate activation of PoolGrain cannot oversubscribe the pool. When the one sandbox pool has the capacity, a CapacityLease is written in state Granted from Requested (trigger CapacityAvailable) with grantedAt and expiresAt at the grant plus RuntimeCloud:Capacity:LeaseTtlSeconds, and CapacityLeaseGranted is written to the outbox in the same transaction. A lease that names a run receives the workspace of that run. The first lease of the run creates the workspace storage of the run on the pool, isolated per run and tenant, and the RunWorkspace with createdWithLeaseId whose workspaceRef resolves to that storage, in the same transaction as the grant. Every later lease of the same run, of any step and attempt, receives the same workspaceRef, so a run has one workspace (D121). The model does not say which medium holds the workspace, a volume or a share on the pool; the medium is an owner input, an entry of the I1 precondition register of AIFACTORY-US-1339 that the owner supplies before the I1 run (D140). The reply carries the lease with its id, state, pool, expiry and workspaceRef. When the pool has no free capacity the lease moves from Requested to Denied (trigger QuotaOrPoolExhausted) and the reply names the pool as the limit; no quota exists in I1, so no QuotaExceeded is written, and ToolAdapterService asks again with backoff until the deadline of its call. As a design choice of this PBI, a repeated request for the same holder kind and holder id while its lease is Granted or Active returns that lease and grants nothing twice. Refusals change nothing: no workload identity is UNAUTHENTICATED, a token without the scope is PERMISSION_DENIED, a request without tenant, project or run, with an unknown holder kind or with cpu or memory outside the bounds of the pool is INVALID_ARGUMENT, a holder kind the thin scope does not serve, attempt, build or browser, is FAILED_PRECONDITION, because the one pool is a sandbox pool and the other kinds arrive with the complete delivery. The invariant of WorkerPool lets sandboxes with network egress to the internet run only in pools whose security boundary allows it, so a sandbox lease whose egress allow list is not empty is also refused with FAILED_PRECONDITION when the security boundary of the one pool does not allow internet egress. Whether the boundary of the pool allows internet egress is part of its declared configuration, an owner input, because the three boundary values of the model do not say it; like the other Sandbox pool values it is an entry of the I1 precondition register of AIFACTORY-US-1339 that the owner supplies before the I1 run (D140). The model names acquire-lease-v1 and lease-v1 for this method, the attempt lease descriptors of RunControlService; the proto of AIFACTORY-TASK-097 makes the contract checkable and the gap is reported.
Acceptance criteria
| AC1 | Given a caller with a workload identity and factory.capacity.lease, the one sandbox pool with free capacity and a run without a workspace, when ExecutionCapacityService/Capacity.AcquireLease is called for a sandbox of that run, then a ExecutionCapacityService/CapacityLease is written in state Granted with grantedAt, expiresAt from RuntimeCloud:Capacity:LeaseTtlSeconds and the pool, the workspace storage of the run exists on the pool, isolated per run and tenant, and the RunWorkspace of the run is created with this lease as createdWithLeaseId in the same transaction with a workspaceRef that resolves to that storage, the reply carries the lease with its workspaceRef, and the ExecutionCapacityService/CapacityLeaseGranted outbox record is written in that transaction (SCN-ExecutionCapacityService-006, D121). |
| AC2 | Given a run whose workspace exists and whose first lease is Released or Expired, when ExecutionCapacityService/Capacity.AcquireLease is called for a later attempt of the same run, then the new lease carries the same workspaceRef, no second RunWorkspace is created, and a lease of another run never receives that reference; given the same holder kind and holder id repeated while its lease is Granted or Active, then the same lease is returned and nothing is granted or written twice (SCN-ExecutionCapacityService-006). |
| AC3 | Given the one pool has no free cpu or memory for the request, when ExecutionCapacityService/Capacity.AcquireLease is called, then the lease moves from Requested to Denied, the reply names the pool as the limit, no capacity is granted, no ExecutionCapacityService/CapacityLeaseGranted is written and connectsoft.factory.capacity.lease_wait is not recorded for it; given concurrent requests for the last capacity of the pool, also through two activations of PoolGrain for the same pool, then exactly one is granted, because the grant commits with a concurrency check on the granted capacity of the pool in execution-capacity-sql beside the serialisation of PoolGrain (SCN-ExecutionCapacityService-006, the amendment of ADR-0024 on duplicate activations). |
| AC4 | Given a caller without workload identity, a token without factory.capacity.lease, a request without tenant, project or run or with an unknown holder kind, a request for the holder kind attempt, build or browser, and a sandbox request whose egress allow list is not empty while the security boundary of the one pool does not allow internet egress, when ExecutionCapacityService/Capacity.AcquireLease is called, then the call is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, FAILED_PRECONDITION and FAILED_PRECONDITION respectively, and no lease, workspace or outbox record is written (SCN-ExecutionCapacityService-006, ADR-0015, ExecutionCapacityService/WorkerPool). |
| AC5 | Given the lease was granted, when the outbox dispatcher of the profile execution-capacity publishes ExecutionCapacityService/CapacityLeaseGranted on capacity.leases.v1.capacity-lease-granted, then the event is sent under the run event envelope with leaseId, tenantId, holderKind, holderId, poolId and expiresAt, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-ExecutionCapacityService-012, ADR-0012). |
| Traces: services | ExecutionCapacityService, ToolAdapterService |
| Traces: aggregates | ExecutionCapacityService/CapacityLease, ExecutionCapacityService/WorkerPool |
| Traces: apis | ExecutionCapacityService/Capacity.AcquireLease |
| Traces: events | ExecutionCapacityService/CapacityLeaseGranted |
| Traces: descriptors | acquire-lease-v1, lease-v1 |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015, ADR-0024 |
| Traces: requirements | NFR-09 |
| Traces: scenarios | SCN-ExecutionCapacityService-006, SCN-ExecutionCapacityService-012 |
| Depends on | AIFACTORY-US-1039, feature:component-base-template-factory-profile, feature:contracts-package |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/services/tools-mcp/ToolAdapterService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#actors-and-grains |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4171 AcquireLease through PoolGrain with the workspace of the run and the CapacityLeaseGranted outbox record |
implementation | AcquireLease validates the request, resolves the tenant, project and run from the Factory scope of the call, asks PoolGrain of the one sandbox pool for the cpu and memory, commits the grant with a concurrency check on the granted capacity of the pool in execution-capacity-sql so that a duplicate activation of PoolGrain cannot oversubscribe the pool (the amendment of ADR-0024 on duplicate activations), writes the CapacityLease in state Granted with grantedAt and expiresAt from RuntimeCloud:Capacity:LeaseTtlSeconds or in state Denied with the pool named as the limit, creates the RunWorkspace of the run with the first lease through the workspace storage of AIFACTORY-TASK-4194 and hands every later lease of the run the same workspaceRef, writes CapacityLeaseGranted in the same transaction, returns the open lease of the same holder kind and holder id on a repeat, refuses the holder kinds attempt, build and browser with FAILED_PRECONDITION, refuses with FAILED_PRECONDITION and without writing a lease, workspace or outbox record a sandbox lease whose egress allow list is not empty when the security boundary of the one pool does not allow internet egress, and records connectsoft.factory.capacity.lease_wait and connectsoft.factory.capacity.leases_active. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4172 Migrations of CapacityLeases and RunWorkspaces with the one-workspace-per-run key |
migration | The migrations create CapacityLeases with IX_Leases_Tenant_State, IX_Leases_Pool_State and IX_Leases_Expiry and RunWorkspaces keyed by RunId with the foreign key CreatedWithLeaseId and IX_RunWorkspaces_Tenant_Open, as docs/planes/execution/environments/services/execution-capacity-service/ddl.md lists them; both are tenant-filtered and the mapping tests run against an isolated test database, never a shared one. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4194 Workspace storage of a run on the pool, created with the first lease and resolved by workspaceRef |
implementation | The first lease of a run creates the workspace storage of the run on the one sandbox pool, isolated per run and tenant so that no other run and no other tenant can reach it, and the workspaceRef of the RunWorkspace resolves to that storage; a later lease of the run resolves to the same storage and creates none; a grant that rolls back leaves no storage behind; the storage medium, a volume or a share on the pool, is taken from the configuration of the deployment as an owner input, an entry of the I1 precondition register of AIFACTORY-US-1339 that the owner supplies before the I1 run (D140). | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4173 Verify the grant of a sandbox lease and the workspace of the run through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ExecutionCapacityService-006\|Scenario=SCN-ExecutionCapacityService-012)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-1036: the grant with the workspace created by the first lease, the workspace storage of the run that exists on the pool after the first grant and that the workspaceRef of a later lease of the run resolves to unchanged, the same workspaceRef for a later attempt of the run and never for another run, the repeat that grants nothing twice, the denial on an exhausted pool, the concurrent grant of the last capacity through two activations of PoolGrain with exactly one grant, the five refusals including the non-empty egress allow list on a pool whose boundary does not allow internet egress, and the publication and payload of CapacityLeaseGranted with a stopped transport. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1037: Keep a sandbox lease alive by heartbeat and expire it when the heartbeat stops, freeing capacity and keeping the workspace¶
behavior · persona operator
As an operator, I want a lease to stay granted only while its holder renews it and to expire on its own when the holder disappears, so that a crashed holder never keeps capacity of the pool and never takes the workspace of its run with it.
The holder of a lease, ToolAdapterService with a workload identity whose token carries factory.capacity.lease, renews it through ExecutionCapacityService/Capacity.RenewLease every RuntimeCloud:Capacity:LeaseRenewSeconds, half the time-to-live. A renewal by the holder of a Granted or Active lease of its tenant extends expiresAt to the renewal plus RuntimeCloud:Capacity:LeaseTtlSeconds and answers the lease with its unchanged id, pool and workspaceRef; a repeated renewal only extends the expiry again. A lease moves from Granted to Active (trigger HolderStarted) when its sandbox has started in the pool with the workspace mounted (AIFACTORY-US-1039). A lease whose heartbeat stopped is expired by the worker LeaseExpirer, which runs every minute partitioned by pool with at most four in parallel, idempotent on the lease id and the expiry and without retry, and by the reminder lease-expiry of PoolGrain, which fires at the earliest expiry: an Active lease past its expiry moves to Expired (trigger HeartbeatMissed) and a Granted lease whose sandbox never started moves to Expired (trigger NotStarted). Expiry frees the cpu and memory in PoolGrain, ends the sandbox, sets releasedAt and writes CapacityLeaseReleased with state Expired in the same transaction; it never removes the workspace of the run, which a later lease of the run receives again (D121). The work of the holder is handled by its own lease semantics, the sandbox lease in ToolAdapterService. A renewal is refused and changes nothing when the caller has no workload identity (UNAUTHENTICATED), lacks the scope (PERMISSION_DENIED), sends no lease id (INVALID_ARGUMENT), names an unknown lease or a lease of another tenant (NOT_FOUND, the same answer for both) or names a lease already Released, Expired or Denied (FAILED_PRECONDITION); the holder then acquires a new lease of the same run.
Acceptance criteria
| AC1 | Given a Granted or Active ExecutionCapacityService/CapacityLease of the caller tenant, when its holder calls ExecutionCapacityService/Capacity.RenewLease, then expiresAt is extended to the renewal plus RuntimeCloud:Capacity:LeaseTtlSeconds, the reply carries the same lease id, pool and workspaceRef, and no capacity is granted twice; given the sandbox of a Granted lease has started with the workspace mounted, then the lease moves to Active (SCN-ExecutionCapacityService-007). |
| AC2 | Given a caller without workload identity, a token without factory.capacity.lease, a request without lease id, an unknown lease, a lease of another tenant and a lease already Released, Expired or Denied, when ExecutionCapacityService/Capacity.RenewLease is called, then the call is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, NOT_FOUND for both the unknown and the foreign lease, and FAILED_PRECONDITION, and nothing changes (SCN-ExecutionCapacityService-007, ADR-0015). |
| AC3 | Given an Active lease whose holder stopped renewing and a Granted lease whose sandbox never started, when LeaseExpirer or the reminder lease-expiry of PoolGrain runs after their expiry, then the first moves to Expired by HeartbeatMissed and the second by NotStarted, their capacity returns to the pool, their sandboxes end, releasedAt is set and ExecutionCapacityService/CapacityLeaseReleased with state Expired is written in the same transaction; a second run for the same lease and expiry changes nothing (model/services/runtime-cloud/ExecutionCapacityService.yaml, ADR-0024). |
| AC4 | Given a run whose only lease expired, when a later attempt of the run calls ExecutionCapacityService/Capacity.AcquireLease, then the workspace of the run still exists and the new lease carries the same workspaceRef, because an expired lease frees capacity and never removes the workspace (SCN-ExecutionCapacityService-007, D121). |
| Traces: services | ExecutionCapacityService, ToolAdapterService |
| Traces: aggregates | ExecutionCapacityService/CapacityLease |
| Traces: apis | ExecutionCapacityService/Capacity.RenewLease, ExecutionCapacityService/Capacity.AcquireLease |
| Traces: events | ExecutionCapacityService/CapacityLeaseReleased |
| Traces: descriptors | lease-v1 |
| Traces: adrs | ADR-0015, ADR-0024 |
| Traces: requirements | NFR-09 |
| Traces: scenarios | SCN-ExecutionCapacityService-007 |
| Depends on | AIFACTORY-US-1036 |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#workers-and-jobs |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4174 RenewLease by heartbeat and the HolderStarted transition |
implementation | RenewLease extends expiresAt of a Granted or Active lease of the caller tenant by RuntimeCloud:Capacity:LeaseTtlSeconds and returns the same lease id, pool and workspaceRef; the start of the sandbox with the workspace mounted moves the lease from Granted to Active; the refusals UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, NOT_FOUND for an unknown and a foreign lease alike, and FAILED_PRECONDITION for a lease no longer open, change nothing. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4175 LeaseExpirer and the lease-expiry reminder of PoolGrain for the one pool |
implementation | LeaseExpirer runs every minute partitioned by pool with at most four in parallel, without retry and idempotent on lease id plus expiry; it and the lease-expiry reminder of PoolGrain move an Active lease past its expiry to Expired by HeartbeatMissed and a Granted lease never started to Expired by NotStarted, free the capacity, end the sandbox, set releasedAt and write CapacityLeaseReleased with state Expired and executionSeconds in one transaction, and never remove the workspace of the run. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4176 Verify renewal and expiry through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ExecutionCapacityService-007"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1037 with a test clock: the extended expiry, the Active transition at sandbox start, the five refusals, the HeartbeatMissed and NotStarted expiries with the CapacityLeaseReleased outbox record, the idempotent second expiry run and the workspace that survives an expired lease. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1038: Release a sandbox lease on completion or on SandboxReleased, freeing its capacity once and leaving the workspace of the run¶
behavior · persona operator
As an operator, I want capacity to return to the pool the moment a sandbox is done, whichever path reports it, so that the pool serves the next call at once and every lease ends exactly once with the seconds it was held.
A lease ends on one of two paths that are one command. The holder, with a workload identity whose token carries factory.capacity.lease, calls ExecutionCapacityService/Capacity.ReleaseLease with the lease id on completion; and the inbox handler ReleaseSandboxLease consumes SandboxReleased of ToolAdapterService, which its SandboxReaper writes when a sandbox lease is released or reaped, idempotent on the event id. Either path moves an Active lease to Released (trigger HolderCompleted), frees its cpu and memory in PoolGrain, ends the sandbox, sets releasedAt and writes CapacityLeaseReleased in the same transaction with the state Released and executionSeconds, the seconds between grantedAt and releasedAt. The answer of the method is ack-v1. The release is idempotent on the lease: a second release by the other path or a repeated call finds the lease already Released or Expired, answers the acknowledgement and writes nothing, so capacity is freed once and one CapacityLeaseReleased exists per lease. A release never removes the workspace of the run: it stays for the next lease of the run until RunControlService announces that the run ended (AIFACTORY-US-1040, D121). The model has no transition from Granted to Released; how a release of a lease whose sandbox never started is recorded is an open point that this PBI does not decide, and such a lease expires by NotStarted (AIFACTORY-US-1037). A call is refused and changes nothing when the caller has no workload identity (UNAUTHENTICATED), lacks the scope (PERMISSION_DENIED), sends no lease id (INVALID_ARGUMENT) or names an unknown lease or a lease of another tenant (NOT_FOUND, the same answer for both, ADR-0015). The consumers CostUsageService and DashboardService record the usage and the projection in their own PBIs.
Acceptance criteria
| AC1 | Given an Active ExecutionCapacityService/CapacityLease of the caller tenant, when its holder calls ExecutionCapacityService/Capacity.ReleaseLease, then the lease moves to Released, its capacity returns to PoolGrain, its sandbox ends, releasedAt is set, the reply is ack-v1, and the ExecutionCapacityService/CapacityLeaseReleased outbox record with state Released and executionSeconds is written in the same transaction (SCN-ExecutionCapacityService-008). |
| AC2 | Given an Active lease, when SandboxReleased of ToolAdapterService names it and the handler ReleaseSandboxLease runs, then the lease is released exactly as in AC1; given the same lease then released again through ExecutionCapacityService/Capacity.ReleaseLease, or the same event redelivered, then the answer is the acknowledgement, nothing is written and capacity is freed once (SCN-ExecutionCapacityService-008). |
| AC3 | Given a caller without workload identity, a token without factory.capacity.lease, a request without lease id, an unknown lease and a lease of another tenant, when ExecutionCapacityService/Capacity.ReleaseLease is called, then the call is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT and NOT_FOUND for both the unknown and the foreign lease, and nothing changes; given any release, then the workspace of the run is not removed (SCN-ExecutionCapacityService-008, ADR-0015, D121). |
| AC4 | Given a lease released or expired, when the outbox dispatcher publishes ExecutionCapacityService/CapacityLeaseReleased on capacity.leases.v1.capacity-lease-released, then the event is sent under the run event envelope with leaseId, tenantId, holderKind, holderId, poolId, state and executionSeconds, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-ExecutionCapacityService-013, ADR-0012). |
| Traces: services | ExecutionCapacityService, ToolAdapterService |
| Traces: aggregates | ExecutionCapacityService/CapacityLease |
| Traces: apis | ExecutionCapacityService/Capacity.ReleaseLease |
| Traces: events | ExecutionCapacityService/CapacityLeaseReleased |
| Traces: descriptors | lease-v1, ack-v1 |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015 |
| Traces: requirements | NFR-09 |
| Traces: scenarios | SCN-ExecutionCapacityService-008, SCN-ExecutionCapacityService-013 |
| Depends on | AIFACTORY-US-1036 |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/services/tools-mcp/ToolAdapterService.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#messages |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4177 ReleaseLease and the ReleaseSandboxLease inbox handler as one idempotent release command |
implementation | ReleaseLease and the handler of SandboxReleased share one release command that moves an Active lease to Released by HolderCompleted, frees the capacity in PoolGrain, ends the sandbox, sets releasedAt and writes CapacityLeaseReleased with state Released and executionSeconds in one transaction, answers ack-v1, acknowledges a lease already Released or Expired without writing, is idempotent on the event id in the inbox, never removes the workspace of the run and decrements connectsoft.factory.capacity.leases_active. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4178 Verify the release paths through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ExecutionCapacityService-008\|Scenario=SCN-ExecutionCapacityService-013)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1038: the release by call and by a SandboxReleased fixture event, the second release and the redelivered event that change nothing, the four refusals, the workspace left in place and the publication and payload of CapacityLeaseReleased with a stopped transport. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1039: Serve the chain from one statically declared sandbox pool that starts each sandbox with the workspace mounted and ends it at release¶
behavior · persona platform-admin
As a platform administrator, I want the first outcome to run its sandboxes in one declared pool whose boundary and region I know, so that every sandbox of the chain runs where the deployment placed it and nothing reaches the host or another run.
In I1 the service holds one WorkerPool of kind Sandbox. IaCProvisioningService is a static stub until I2, so the pool is not provisioned through it: the deployment declares the pool, and at start the service writes or confirms the one row of WorkerPools with its pool id, the platform tenant of a shared pool, the security boundary, whether that boundary allows internet egress, region, node profile and size bounds from the configuration of the deployment, in state Ready. The values of the pool, its boundary and its egress permission, region, node profile and size, are inputs of the owner and are not invented here; they are entries of the I1 precondition register of AIFACTORY-US-1339 that the owner supplies before the I1 run (D140). RetentionResidencyService is a static stub until I4 that allows the region of the one pool, and FactoryCommercialService is a static stub until I7 that is not called, because no quota exists in I1. No size change, autoscaling or state change happens in this increment, so WorkerPoolStateChanged is not written; it arrives with the complete delivery. PoolGrain, keyed by the pool id, holds the current size and the granted cpu and memory and serialises every grant (ADR-0024). For each granted sandbox lease the service starts the sandbox on the pool with the limits, the egress allow list and the mount mode that the lease carries. The workspace is mounted for ToolAdapterService by the lease: the started sandbox mounts the workspace that the workspaceRef of the lease hands to ToolAdapterService, with the mount mode of the lease, read-only for the repository read tools of the tool executor (D121, D125). The start of the sandbox moves the lease to Active. At release or expiry the sandbox ends and the workspace stays. The build tooling of the worker image and the enforcement of the egress allow list, with the proof that a destination outside the list is unreachable from a sandbox, are AIFACTORY-TASK-219 of feature tool-adapter-service in this repository and are not planned again here; this PBI proves only that the sandbox is started with the configuration the lease carries. connectsoft.factory.capacity.pool_utilisation reports the share of the pool granted, with the pool id and the kind. The service adds no operation that reads a workspace: every read of a workspace happens inside a sandbox.
Acceptance criteria
| AC1 | Given the configuration of the deployment that declares the one sandbox pool, when the service starts, then one ExecutionCapacityService/WorkerPool of kind Sandbox exists in state Ready with the declared boundary, region, node profile and bounds, a restart writes no second pool, and no WorkerPoolStateChanged is written in I1 (model/services/runtime-cloud/ExecutionCapacityService.yaml, model/backlog/features/runtime-cloud.yaml). |
| AC2 | Given a lease granted through ExecutionCapacityService/Capacity.AcquireLease with limits, an egress allow list and a mount mode, when its sandbox starts on the pool, then the sandbox is started with the limits, the egress allow list and the mount mode that the lease carries, it mounts the workspace that the workspaceRef of the lease hands to ToolAdapterService, read-only when the lease names read-only as for a read tool (D121, D125), and the ExecutionCapacityService/CapacityLease moves to Active; that a destination outside the list is unreachable is proven by AIFACTORY-TASK-219 of feature tool-adapter-service. |
| AC3 | Given two runs with one sandbox each on the pool, when the sandbox of the first lists its mount, then it sees the workspace of its own run only; given its lease is released or expired, then its sandbox ends and the workspace of its run is still present for the next lease of the run (ExecutionCapacityService/CapacityLease, D121). |
| AC4 | Given sandbox leases granted and released, when connectsoft.factory.capacity.pool_utilisation is read, then it reports the share of the one ExecutionCapacityService/WorkerPool granted with its pool id and kind, and the region of the pool is the one the static residency stub allows (ADR-0017, ADR-0024). |
| Traces: services | ExecutionCapacityService |
| Traces: aggregates | ExecutionCapacityService/WorkerPool, ExecutionCapacityService/CapacityLease |
| Traces: apis | ExecutionCapacityService/Capacity.AcquireLease |
| Traces: adrs | ADR-0017, ADR-0024 |
| Depends on | feature:component-base-template-factory-profile |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/backlog/features/runtime-cloud.yaml, docs/planes/execution/environments/services/execution-capacity-service/index.md#data-model-and-storage |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4179 Static declaration of the one sandbox pool, the WorkerPools migration and the static stubs of the increment |
implementation | The migration creates WorkerPools with IX_Pools_Kind_Region; at start the service writes or confirms the one Sandbox pool in state Ready from the configuration of the deployment, whose boundary, egress permission of the boundary, region, node profile and bounds are inputs of the owner; the static stubs of IaCProvisioningService, RetentionResidencyService and FactoryCommercialService are bound behind their client ports so the complete delivery replaces them without a contract change; PoolGrain loads the pool with its size and granted capacity. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4180 Sandbox start with the workspace mounted and sandbox end at release or expiry |
implementation | For a granted sandbox lease the service starts the sandbox on the pool with the limits, the egress allow list and the mount mode that the lease carries, the sandbox mounts for ToolAdapterService the workspace that the workspaceRef of the lease hands over, the lease moves to Active by HolderStarted when the sandbox runs, the sandbox ends at release or expiry and the workspace is never removed there; pool_utilisation follows the granted share of the pool; the enforcement of the egress allow list stays with AIFACTORY-TASK-219. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4181 Verify the one pool and its sandboxes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=SandboxPool"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1039 with a sandbox fixture: the one pool declared once from configuration, a sandbox started with the limits, egress list and mount mode that its lease carries and mounting the workspace the lease hands over, read-only for a read-only lease, the isolation of two runs, the workspace that outlives its sandbox and the utilisation metric. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-US-1040: Remove the workspace of a run once when RunControlService announces that the run completed, failed or was cancelled¶
behavior · persona operator
As an operator, I want the workspace of a run to disappear when the run ends and never before, so that no files of a finished run stay on the pool and no running step loses its files.
ExecutionCapacityService owns the workspace of a run and removes it on exactly one signal: a terminal run event of RunControlService (D121). The inbox handler RemoveRunWorkspace consumes RunCompleted, RunFailed and RunCancelled from the run-control topics, each with runId and tenantId. For the RunWorkspace of that run in that tenant it removes the workspace storage of the run on the pool, which the first lease created (AIFACTORY-US-1036), and sets removedAt, idempotent on the run id: a redelivered event, or a second terminal event of the same run, finds removedAt set and does nothing, and a run that never took a lease has no workspace and the event changes nothing. The release or the expiry of a lease never removes a workspace (AIFACTORY-US-1037, AIFACTORY-US-1038). Leases of the run still open when the run ends are not released by this handler; they end through their holder or their expiry. The model does not say what Capacity.AcquireLease answers for a run whose workspace was removed; this PBI does not decide it and the open point is reported. The removal writes no outbox record, because the model declares no event for it; connectsoft.factory.capacity.inbox_lag reports the age of the oldest unprocessed message of the inbox. When the transport delivers late the removal waits in the inbox and is applied once (ADR-0012). The handler of a deletion order of RetentionResidencyService is built in I4 with that service and none is registered here.
Acceptance criteria
| AC1 | Given a run with a workspace and no open lease, when RunControlService/RunCompleted, RunControlService/RunFailed or RunControlService/RunCancelled of that run is consumed, then the content of the workspace is removed and removedAt of its RunWorkspace is set once, under the tenant of the event (model/services/runtime-cloud/ExecutionCapacityService.yaml, D121). |
| AC2 | Given the same terminal event redelivered, a second terminal event of the same run, or a terminal event of a run that never took a lease, when the handler runs, then nothing is removed or written a second time and the event is acknowledged in the inbox under its event id (RunControlService/RunCompleted, ADR-0012). |
| AC3 | Given a run whose leases were released and expired while it still runs, when no terminal event of the run was consumed, then its workspace is present and a later lease of the run receives it through ExecutionCapacityService/CapacityLease; given a terminal event whose tenantId is not the tenant of the workspace of that run id, then nothing is removed, because the handler acts only on the RunWorkspace row of the tenant of the event (ADR-0015, D121). |
| Traces: services | ExecutionCapacityService, RunControlService |
| Traces: aggregates | ExecutionCapacityService/CapacityLease |
| Traces: events | RunControlService/RunCompleted, RunControlService/RunFailed, RunControlService/RunCancelled |
| Traces: adrs | ADR-0012, ADR-0015 |
| Traces: requirements | XL-14 |
| Depends on | AIFACTORY-US-1036, feature:run-control-service |
| Sources | model/services/runtime-cloud/ExecutionCapacityService.yaml, model/services/control-plane/RunControlService.yaml |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4182 RemoveRunWorkspace inbox handler for RunCompleted, RunFailed and RunCancelled |
implementation | The handler of the three terminal run events removes the workspace storage of the run that AIFACTORY-TASK-4194 created, in the tenant of the event, and sets removedAt of its RunWorkspace, idempotent on the run id and on the event id, does nothing for a run without a workspace, releases no lease, writes no outbox record and leaves inbox_lag to report a late delivery. | ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |
AIFACTORY-TASK-4183 Verify the removal of the workspace on a terminal run event through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=RunWorkspaceRemoval"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1040 with fixture events of RunControlService/RunCompleted, RunControlService/RunFailed and RunControlService/RunCancelled: one removal per run, the redelivered and the second terminal event that change nothing, the run without a workspace, and the workspace kept while the run still runs. |
ConnectSoft.AI.SoftwareFactory.ExecutionCapacityService |