AIFACTORY-FEAT-013: AgentRuntimeWorker: gateway delivery¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Agent Mesh (PF-02) · Increment I3 Studio · State elaborated · Route backend-developer · Build order 110 of 185 (Factory code)
AgentRuntimeWorker is the platform API gateway of bounded context BC-06 (Execution). The internal gateway of the Agent Mesh platform (ADR-0028): fronts AgentRuntimeService and WorkflowEngineService, which deploy alone as silo kinds of one Orleans cluster (ADR-0024); routes internal gRPC and owns no run state. Its own repository is ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker.
| Delivers | AgentRuntimeWorker (complete) |
| Stubs | — |
| Realizes | adrs: ADR-0001, ADR-0012, ADR-0014, ADR-0016, ADR-0017, ADR-0022 |
| Requirements | FR-03, FR-10, FUX-04, FUX-07, NFR-05, NFR-06, NFR-08, NFR-13, XL-06, XL-16 |
| Depends on | — |
| Sources | model/services/agent-mesh/AgentRuntimeWorker.yaml |
Scenarios to trace (4, catalog): SCN-AgentRuntimeWorker-001 ✓, SCN-AgentRuntimeWorker-002 ✓, SCN-AgentRuntimeWorker-003 ✓, SCN-AgentRuntimeWorker-004 ✓
AIFACTORY-US-080: AgentRuntimeWorker routes its platform's calls under a trusted scope¶
enabling · persona operator
As an operator, I want one gateway in front of AgentRuntimeService and WorkflowEngineService that validates tokens and scopes, so that no service of the platform is reachable except through its gateway and every call arrives with a scope the service can trust.
This is the integration PBI of the Agent Mesh gateway. ADR-0028 decision 4 makes AgentRuntimeWorker the internal gateway of the platform: it fronts the two services the model lists under deployment.fronts, AgentRuntimeService and WorkflowEngineService, which deploy alone as two silo kinds of one Orleans cluster (ADR-0024), routes their REST operations and their internal gRPC methods, validates tokens and scopes and owns no run state. It keeps the identifier of the former runtime worker so that references stay valid, and it is a gateway and no worker: it executes no attempt and hosts no silo, no grain and no job. The repository ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker is scaffolded from the pinned API gateway template package with UseFactoryProfile and the gateway role Platform (feature component-api-gateway-template), which fixes persistence, messaging and the actor model to none: the gateway has no relational store, no outbox and no inbox, consumes no event (ADR-0028 amendment on gateways and events), holds no secret, and its only store is the shared cache. The scope of a call is established once from the validated token with the scope accessor of the pinned ConnectSoft.Extensions.AI.Factory.Abstractions package; the gateway copies neither the accessor nor the token contract. Work never enters the Agent Mesh through this gateway: an attempt arrives as AttemptDispatched in the inbox of AgentRuntimeService. What passes the gateway is the synchronous traffic into the platform: interrupt requests and timeline reads of the Control plane over gRPC, and attempt timeline and workflow run graph reads of the Studio BFF. The feature is delivered by four PBIs beside this one: the fronts file and the routes generated from it (AIFACTORY-US-472), the read and sandbox routes under token, tenant and scope (AIFACTORY-US-473), the internal gRPC routes with the pass-through of the two shared-write methods (AIFACTORY-US-474), and health, modules, limits and telemetry with the three failure modes of a gateway (AIFACTORY-US-475). A call without a token, with an expired token or with a user token issued for another gateway is answered 401 and a missing scope 403, and no fronted service is called. Every route of the two services addresses its resource by an identifier only, an attempt, a release, a run or a definition, so the gateway, which holds no state, forwards under the tenant of the token and the scope of the operation; the fronted service answers 403 for a missing project role and 404 for a resource of another tenant (ADR-0015 amendment of D113), and the gateway returns that answer unchanged. Both services were delivered in I1 and are reached on their internal addresses until this gateway exists. This feature delivers the routes and proves them with its own suite against fakes of the two services; it changes no caller. The address a caller uses for a service of the Agent Mesh is a deployment value of the feature that delivers that caller, in its own repository, and the model does not name the callers with one voice: the gateway lists ControlPlaneApi and StudioBff as its inbound callers, while the two services list RunControlService and StudioBff. Which feature moves the address of each caller to the gateway, and which closes the internal addresses of the two services to callers outside the platform, is therefore not claimed here and is reported to the owner. The gateway invents no response.
Acceptance criteria
| AC1 | Given the repository ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker scaffolded with dotnet new connectsoft-apigateway -n ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker --UseFactoryProfile true --GatewayRole Platform from the pinned template package, when its pipeline runs, then the solution builds, the tests pass with complete reports and a container image is published (ADR-0028, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| AC2 | Given a valid token for a route of AgentRuntimeService and for a route of WorkflowEngineService, when the call passes the gateway, then the scope is established once from the token through the scope accessor of the pinned abstractions package, the call is forwarded with tenant, project and correlation id, the fronted service answers and the response returns unchanged, as AIFACTORY-US-473 and AIFACTORY-US-474 prove per route class (ADR-0015). |
| AC3 | Given the generated gateway, when its architecture test runs, then the test fails on a persistence model, an aggregate, a migration, a message consumer, a grain or a silo, the gateway binds no relational store, no outbox and no inbox, and no event of the model lists AgentRuntimeWorker as a consumer; the silo kinds of the cluster stay in the two fronted services (ADR-0028, ADR-0024, ADR-0012, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| AC4 | Given the gateway deployed with two replicas, when AgentRuntimeWorker/RuntimeHealth is probed and one replica is stopped, then readiness reflects the identity platform and the two fronted services, the other replica keeps serving and nothing is replayed, because the gateway holds no state (ADR-0017, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| Traces: services | AgentRuntimeWorker, AgentRuntimeService, WorkflowEngineService |
| Traces: apis | AgentRuntimeWorker/RuntimeHealth |
| Traces: adrs | ADR-0012, ADR-0015, ADR-0017, ADR-0024, ADR-0028 |
| Depends on | AIFACTORY-US-472, AIFACTORY-US-473, AIFACTORY-US-474, AIFACTORY-US-475, feature:component-api-gateway-template, feature:component-extensions-ai, feature:agent-runtime-service, feature:workflow-engine-service |
| Sources | model/services/agent-mesh/AgentRuntimeWorker.yaml, model/gateways.yaml, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md, docs/planes/agent-runtime/services/agent-runtime-worker/index.md#purpose-and-responsibilities |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-356 Create the repository ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker and scaffold it from the ConnectSoft.ApiGatewayTemplate template with the Factory profile |
implementation | The repository exists, the output of dotnet new connectsoft-apigateway -n ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker --UseFactoryProfile true --GatewayRole Platform from the pinned template package is committed, the solution builds and the template tests, the architecture test of the Platform role included, pass in the repository pipeline, which publishes the container image. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1886 Deployment values of the gateway: replica range, ports, probes and the address of each fronted service |
operations | The deployment manifest of the gateway states the replica range 2 to 20, the ports https 7500 and grpc 7502, the liveness and readiness probes on the health operation and one base address per fronted service from configuration, as model/services/agent-mesh/AgentRuntimeWorker.yaml records them, and sets Runtime:Host:FrontedServices to the two service ids of deployment.fronts; the same image runs in the saas-default, hybrid, on-prem and hybrid-compute topologies with the identity authority of the profile (ADR-0017). | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-659 Verify agent-runtime-worker-routes-its-platform by running the declared command |
verification | Run dotnet build -c Release and dotnet test -c Release --no-build in the pipeline of the repository; a run that discovers no test counts as a failure; the runs pass and prove AC1 to AC4 of AIFACTORY-US-080, that is the scaffold, the scope established once and forwarded, the architecture test of the Platform role and, for AC4, the acceptance test of feature TwoReplicas (dotnet test -c Release --filter "Category=Acceptance&Feature=TwoReplicas"), which hosts two instances of the gateway behind one test balancer with the fakes of the two fronted services, probes AgentRuntimeWorker/RuntimeHealth on both, stops one instance and asserts that the other keeps serving and that no call is replayed. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-US-472: Agent Mesh fronts file: named clients and the six REST and five gRPC routes of the two fronted services, generated at build¶
enabling · persona platform-admin
As a platform admin, I want the route table of the Agent Mesh gateway generated from the list of services it fronts, so that every operation of the platform has exactly one route with its scope and class and no forwarding code is written by hand.
The API gateway template is a custom gateway with controllers, use cases and named clients and is no reverse proxy, so a route exists only when it is declared. Under UseFactoryProfile the generated solution holds a build-time source generator that reads a fronts file on every build and emits one named HTTP client and one gRPC client per fronted service, the REST and gRPC routes that forward to them and the route table (feature component-api-gateway-template). This PBI writes the fronts file of the Agent Mesh. It lists the two services of deployment.fronts in model/services/agent-mesh/AgentRuntimeWorker.yaml, each with its base addresses from configuration and its operations in the shape of the apis section of its service file: name, method and path or gRPC method, scope, side-effect class and idempotency. For AgentRuntimeService these are the three REST reads GetAttemptTimeline, ListCheckpoints and GetLoadedRelease and the three methods of the gRPC service Runtime, Interrupt, AnswerDecision and WatchAttempt. For WorkflowEngineService they are the REST operations ValidateWorkflowDefinition, GetWorkflowRunGraph and ListWorkflowDefinitionVersions and the two methods of the gRPC service WorkflowEngine, PlanNextSteps and GetRunGraph. That is six REST routes and five gRPC routes; WatchAttempt is a server stream, and its route forwards the stream and its cancellation without buffering. Each route carries the scope and the side-effect class of its operation, which is what the route PBIs of this feature enforce. A test lists the emitted route table against the file and against the two service files of the model, so an operation of the model without a route, a route without an operation, a service named twice or a route named twice fails the build with the entry named. A path that no route matches is answered 404. The gateway adds no operation of its own beyond RuntimeHealth and RuntimeModules, transforms no payload and composes no response from the two services: the response of a fronted service returns unchanged. The fronts file alone decides which routes are hosted, because the routes are generated from it at build. The option Runtime:Host:FrontedServices of the model is bound at startup and compared with the services of the fronts file: a value that names no service of the file, or a service of the file that the value omits, fails startup with the name, so the option can neither add nor remove a route; it is not reloadable, so a change of the list is a new build and a new deployment. The modelled default of the option holds the names attempt-execution and workflow-engine, which match neither service id of deployment.fronts; that mismatch is reported to the owner as a model correction, and until the model is corrected the deployment values set the option to the two service ids.
Acceptance criteria
| AC1 | Given the fronts file of the Agent Mesh, when the gateway builds, then one named HTTP client and one gRPC client exist for AgentRuntimeService and for WorkflowEngineService and one route per REST operation and gRPC method of the two services, six REST and five gRPC routes, each with the scope, side-effect class and idempotency of its operation, and the route table test passes against model/services/agent-mesh/AgentRuntimeWorker.yaml and the two service files it fronts (ADR-0028). |
| AC2 | Given an operation removed from the file, a route named twice or a service named twice, when the gateway builds, then the build fails with the entry named; given a value of Runtime:Host:FrontedServices that differs from the services of the fronts file, when the gateway starts, then startup fails with the differing name; given a path no route matches, when it is called, then the answer is 404 and no fronted service is called (ADR-0028, docs/architecture/catalogs/gateways.md). |
| AC3 | Given a fake of each fronted service, when one REST route and one gRPC route per service are called with a valid token, then each call reaches the fake of its service with the authorization header, the tenant, the correlation id and the trace context, and the response of the fake returns unchanged (ADR-0015, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| Traces: services | AgentRuntimeWorker, AgentRuntimeService, WorkflowEngineService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Depends on | feature:component-api-gateway-template, feature:agent-runtime-service, feature:workflow-engine-service |
| Sources | model/services/agent-mesh/AgentRuntimeWorker.yaml, model/services/agent-mesh/AgentRuntimeService.yaml, model/services/agent-mesh/WorkflowEngineService.yaml, model/gateways.yaml, docs/architecture/catalogs/gateways.md, docs/planes/agent-runtime/services/agent-runtime-worker/index.md#context-and-boundaries |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-357 Fronts file with the REST and gRPC operations of AgentRuntimeService and WorkflowEngineService |
implementation | The fronts file lists the two services of deployment.fronts with their base addresses from configuration and every REST operation and gRPC method of their service files with name, path or method, scope, side-effect class and idempotency; the build emits the named clients, the six REST and five gRPC routes and the route table; at startup the option Runtime:Host:FrontedServices is compared with the services of the fronts file and a difference in either direction fails startup with the name; a call without a valid token or scope is refused at the gateway by the generated policies. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1887 Fakes of the two fronted services and the token matrix of the acceptance suite |
fixture | The acceptance suite of the gateway holds one fake of AgentRuntimeService and one of WorkflowEngineService over REST and gRPC, each recording the headers and metadata it received and answering a configured status or stream, a switch that makes a fake stop answering, and tokens for the matrix of role, tenant and project, including an expired token, an identity token, a user token of another audience and a workload token with and without a service scope. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1888 Verify the route table against the fronts file and the model |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=RouteTable"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-472: one route per operation of model/services/agent-mesh/AgentRuntimeService.yaml and model/services/agent-mesh/WorkflowEngineService.yaml, the failing builds, the failing startup on an option value that differs from the fronts file, the 404 of an unknown path and one forwarded call per service and protocol. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-US-473: A read or sandbox route of the Agent Mesh forwards a call only under a validated token, the tenant of the token and the scope of its operation¶
behavior · persona requester
As a requester, I want my reads of an attempt and of a workflow run to reach the service only with the tenant and the scope my token grants, so that I see what I am entitled to and nobody reads across a tenant through the gateway.
A caller reaches a read or a sandbox operation of the Agent Mesh through the gateway: the Studio BFF reads the timeline and the checkpoints of an attempt, a loaded release, the graph of a workflow run and the loaded versions of a definition on behalf of a requester, an operator or an observer, and an asset maintainer has a workflow definition validated. A user presents an access token of the authorization server whose one audience is this gateway (ADR-0015 amendment on the audience of a user token); a client that also calls the gateway of the Control Plane requests a second token. The gateway validates issuer, signature, lifetime and audience through the pinned resource-server library and requires the scope the operation names: factory.runs.read for GetAttemptTimeline, ListCheckpoints and GetWorkflowRunGraph, factory.assets.read for GetLoadedRelease and ListWorkflowDefinitionVersions, and factory.assets.write for the sandbox operation ValidateWorkflowDefinition. It takes the tenant from the tid claim, overwrites any tenant header of the caller and establishes the scope once with the scope accessor of the pinned ConnectSoft.Extensions.AI.Factory.Abstractions package. No route of the two services names a project or a tenant in its path: each addresses an attempt, a release, a run or a definition by its identifier, and the sandbox route addresses nothing. A gateway without business state cannot know the project or the tenant of such a resource, so it forwards the call under the tenant of the token and the scope of the operation, with the authorization header, which carries the user token unchanged because no token is exchanged between a gateway and the service it fronts, the tenant in X-Tenant-Id, the correlation id and the trace context. The fronted service validates the token again, accepting the audience of its own gateway, applies its tenant row filter and answers 403 for a caller without a role in the project of the resource and 404 for a resource of another tenant, the same answer as for an unknown identifier (ADR-0015 amendment of D113); the gateway returns that answer unchanged. For a read route the response is served by the fronted service with no side effect. The sandbox route, ValidateWorkflowDefinition, has a workflow DSL document validated and reported on; it changes no shared state, and the gateway adds no gate of its own to the class. The gateway returns the response unchanged with the correlation id and keeps nothing of the call. A call without a token, with an expired token, with an identity token or with a user token issued for another gateway is answered 401 and a token without the scope of the operation 403; in each of these cases no fronted service is called, and the refusal is counted for the route and logged with the correlation id and without the token. When a fronted service does not answer, its routes answer 503 with a retry hint that names the service and never its address, and the routes of the other service keep serving.
Acceptance criteria
| AC1 | Given a user token whose audience is this gateway and whose scope carries the scope of a read operation, when the route of AgentRuntimeService/GetAttemptTimeline, AgentRuntimeService/ListCheckpoints, AgentRuntimeService/GetLoadedRelease, WorkflowEngineService/GetWorkflowRunGraph or WorkflowEngineService/ListWorkflowDefinitionVersions is called, then the gateway forwards the call with the authorization header, the tenant of the tid claim, the correlation id and the trace context, the fronted service answers, the response returns unchanged with the correlation id and nothing is written by the gateway (SCN-AgentRuntimeWorker-003, ADR-0015). |
| AC2 | Given the token matrix of the acceptance suite, when the read routes are called, then a call without a token, with an expired token, with an identity token or with a user token issued for another gateway is answered 401 and a token without the scope 403, and in none of these cases a fronted service is called; a call the gateway admits is forwarded, the 403 of the fake for a missing project role and its 404 for a resource of another tenant return unchanged, and a tenant header of the caller that names another tenant is overwritten with the tid claim before the call is forwarded (SCN-AgentRuntimeWorker-003, ADR-0015). |
| AC3 | Given a token with factory.assets.write, when the sandbox route of WorkflowEngineService/ValidateWorkflowDefinition is called, then the call is forwarded under the same token, tenant and scope rules, the gateway adds no gate of its own to the class and the answer of the service returns unchanged; the refusals apply as on a read route and no fronted service is called when the gateway refuses (SCN-AgentRuntimeWorker-004, model/services/agent-mesh/WorkflowEngineService.yaml). |
| AC4 | Given AgentRuntimeService does not answer, when one of its read routes is called, then the gateway answers 503 with a problem response that carries a retry hint and names the service and never its address, invents no response, and the routes of WorkflowEngineService keep serving (SCN-AgentRuntimeWorker-003, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| Traces: services | AgentRuntimeWorker, AgentRuntimeService, WorkflowEngineService |
| Traces: apis | AgentRuntimeService/GetAttemptTimeline, AgentRuntimeService/ListCheckpoints, AgentRuntimeService/GetLoadedRelease, WorkflowEngineService/ValidateWorkflowDefinition, WorkflowEngineService/GetWorkflowRunGraph, WorkflowEngineService/ListWorkflowDefinitionVersions |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FUX-04 |
| Traces: scenarios | SCN-AgentRuntimeWorker-003, SCN-AgentRuntimeWorker-004 |
| Relates: screens | run-timeline, workflow-run-graph |
| Depends on | AIFACTORY-US-472, feature:component-api-gateway-template, feature:component-extensions-ai, feature:agent-runtime-service, feature:workflow-engine-service |
| Sources | model/services/agent-mesh/AgentRuntimeWorker.yaml, model/services/agent-mesh/WorkflowEngineService.yaml, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md, docs/planes/agent-runtime/services/agent-runtime-worker/index.md#purpose-and-responsibilities |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-1889 Issuer, audience, scope and forwarded headers of the read and sandbox routes |
implementation | The gateway is configured with the authority of ConnectSoftIdentityPlatform:Authority and its own resource as the one audience of a user token; each of the five read routes and the sandbox route requires the scope of its operation and is marked as addressed by a resource id only, so the gateway checks token and scope and leaves the project role and the tenant of the resource to the fronted service; the tenant header is overwritten with the tid claim, the scope is established through the scope accessor of the pinned abstractions package and forwarded in the canonical tenant header with the correlation id and the trace context; refusals are counted per route and logged with the correlation id and without the token. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1890 Verify the read and sandbox routes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-AgentRuntimeWorker-003\|Scenario=SCN-AgentRuntimeWorker-004)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-473 over the token matrix: each read route forwarded with its headers, the 401 and 403 of the gateway without a call to a fake, the 403 and 404 of a fake returned unchanged, the overwritten tenant header, the sandbox route and a fake that stops answering. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-US-474: An internal gRPC route admits only a workload token with the service scope of its operation and passes the two shared-write methods through once¶
behavior
The internal surface of the Agent Mesh is five gRPC methods, and their callers are services. Run Control asks the lease holder to stop an attempt through Runtime.Interrupt; an answer of a human for a waiting attempt is delivered through Runtime.AnswerDecision, which RunControlService calls with its workload identity when a member answers on its public route AnswerDecision behind ControlPlaneApi (D137); the phases of an attempt are watched as a stream through Runtime.WatchAttempt; Run Control plans the next steps of a run through WorkflowEngine.PlanNextSteps and reads the projection of a run through WorkflowEngine.GetRunGraph. The model names ControlPlaneApi as the inbound caller of the gateway over gRPC. A caller presents a workload token, which carries service scopes, no audience of a user API and no tenant claim (ADR-0015 amendment of D114). The gateway validates the token through the pinned resource-server library and admits it only on a route whose operation names a service scope the token holds: a call without a token or with an expired one is answered UNAUTHENTICATED, and a workload token without that scope, or a user token, which holds no service scope, PERMISSION_DENIED, in each case before any fronted service is called. Because a workload token has no tenant claim, the overwrite of the tenant header does not apply to it: for a token that holds the service scope the gateway forwards the tenant-id and the project metadata of the caller unchanged, together with the authorization metadata, the correlation id and the trace context, and the fronted service reads the forwarded tenant and project only under that scope. The gateway knows neither the tenant nor the project of an attempt or a run; an attempt or a run outside the forwarded tenant is answered NOT_FOUND by the fronted service, exactly as an unknown one, and that status returns unchanged, as does every other status of the service. Runtime.Interrupt and Runtime.AnswerDecision are of class shared-write. The gateway keeps nothing between calls and applies no idempotency rule of its own: it forwards each call once, never retries a shared-write call on its own, and returns the ack-v1 or the refusal of AgentRuntimeService unchanged. The service applies an interrupt and an answer once under its own rule (feature agent-runtime-service), so a caller that repeats a call after a timeout gets the answer of the service and no second effect. Runtime.WatchAttempt is a server stream: the route forwards each message as it arrives, in order and without buffering the stream, ends when the service ends it and cancels the upstream call when the subscriber cancels. WorkflowEngine.PlanNextSteps and WorkflowEngine.GetRunGraph are of class read and are forwarded under the same token and metadata rules. When AgentRuntimeService or WorkflowEngineService does not answer, the routes of that service answer UNAVAILABLE with a retry hint that names the service, the routes of the other keep serving, and the gateway invents no reply. The derivation of scenarios reads only the REST operations of the fronted services, so the route class shared-write of the two gRPC methods has no scenario of its own; the criteria of this PBI cite the operations and the model, and the missing scenario is reported.
Acceptance criteria
| AC1 | Given a workload token that holds the service scope of the operation, when AgentRuntimeService/Runtime.Interrupt, AgentRuntimeService/Runtime.AnswerDecision, WorkflowEngineService/WorkflowEngine.PlanNextSteps or WorkflowEngineService/WorkflowEngine.GetRunGraph is called through the gateway, then the call reaches the fake of its service once with the authorization metadata, the tenant-id and the project metadata of the caller unchanged, the correlation id and the trace context, and the reply returns unchanged (ADR-0015, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| AC2 | Given a call without a token, with an expired token, with a workload token that lacks the service scope of the operation, or with a user token, when an internal gRPC route is called, then the answers are UNAUTHENTICATED for the first two and PERMISSION_DENIED for the other two, and no fronted service is called (ADR-0015). |
| AC3 | Given the fake of AgentRuntimeService answers ack-v1, FAILED_PRECONDITION or NOT_FOUND, when AgentRuntimeService/Runtime.Interrupt is called twice with the same request, then each call is forwarded once, the gateway retries none and keeps nothing between the calls, and each answer returns unchanged, the NOT_FOUND of a foreign attempt exactly as that of an unknown one (ADR-0028, ADR-0015). |
| AC4 | Given a subscriber of AgentRuntimeService/Runtime.WatchAttempt through the gateway, when the fake streams three messages and the subscriber then cancels, then the three messages arrive in order as they are sent and the upstream call to the fake is cancelled (model/services/agent-mesh/AgentRuntimeService.yaml). |
| AC5 | Given AgentRuntimeService does not answer, when its gRPC routes are called, then the answer is UNAVAILABLE with a retry hint that names the service, no reply is invented, and the routes of WorkflowEngineService/WorkflowEngine.GetRunGraph and WorkflowEngineService/WorkflowEngine.PlanNextSteps keep serving (FR-10, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| Traces: services | AgentRuntimeWorker, AgentRuntimeService, WorkflowEngineService |
| Traces: apis | AgentRuntimeService/Runtime.Interrupt, AgentRuntimeService/Runtime.AnswerDecision, AgentRuntimeService/Runtime.WatchAttempt, WorkflowEngineService/WorkflowEngine.PlanNextSteps, WorkflowEngineService/WorkflowEngine.GetRunGraph |
| Traces: descriptors | ack-v1 |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FR-10 |
| Depends on | AIFACTORY-US-472, feature:component-api-gateway-template, feature:component-extensions-ai, feature:agent-runtime-service, feature:workflow-engine-service |
| Sources | model/services/agent-mesh/AgentRuntimeWorker.yaml, model/services/agent-mesh/AgentRuntimeService.yaml, model/services/agent-mesh/WorkflowEngineService.yaml, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md, docs/planes/agent-runtime/services/agent-runtime-worker/index.md#context-and-boundaries |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-1891 Service scope, metadata forwarding and stream pass-through of the five internal gRPC routes |
implementation | Each of the five gRPC routes names the service scope a workload token must hold; a token without it, and a user token, is answered PERMISSION_DENIED and a missing or expired token UNAUTHENTICATED before any call; for an admitted token the tenant-id and the project metadata of the caller are forwarded unchanged with the authorization metadata, the correlation id and the trace context (ADR-0015 amendment of D114); every call is forwarded once without a retry of the gateway, the status of the service returns unchanged, Runtime.WatchAttempt is forwarded message by message with its cancellation, and an unavailable service is answered UNAVAILABLE with a retry hint that names it. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1892 Verify the internal gRPC routes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=InternalGrpcRoutes"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-474 for AgentRuntimeService/Runtime.Interrupt, AgentRuntimeService/Runtime.AnswerDecision, AgentRuntimeService/Runtime.WatchAttempt, WorkflowEngineService/WorkflowEngine.PlanNextSteps and WorkflowEngineService/WorkflowEngine.GetRunGraph: the forwarded metadata of a workload token with the service scope, the refusals without a call to a fake, the single forward of a repeated shared-write call with each answer unchanged, the ordered stream with its cancellation and a fake that stops answering. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-US-475: Health, modules, limits and telemetry of the Agent Mesh gateway while the identity platform, a fronted service or a replica fails¶
behavior · persona operator
As an operator, I want to see whether the gateway, the identity platform and each of the two fronted services answer, and which services the gateway fronts, so that I can tell a failing service from a failing gateway and act on the right one.
An operator, the Studio and the probes of the cluster read the state of the gateway through its two own operations. RuntimeHealth reports liveness and readiness of the gateway with the aggregated health of AgentRuntimeService and WorkflowEngineService and of the identity platform. Liveness needs no token; the detail is served to a caller whose scope carries factory.health.read. The identity platform is the one required dependency of the gateway, and the health check is the detection signal the model names for its failure. While a cached signing key still validates tokens, the gateway keeps validating tokens that were already issued, stays in rotation and reports degraded, naming the identity platform; it leaves rotation only when no valid key remains, as the amendment of ADR-0015 on the forwarded user token states: then it refuses calls, fails closed, and health answers 503 with a problem response that names the identity platform. A fronted service is an optional dependency of the degrade kind: when it does not answer, health reports it as unavailable in the detail, its routes answer 503 with a retry hint, and the routes of the other service keep serving. RuntimeModules serves a caller whose scope carries factory.meta.read. The model describes its answer as the runtime adapter version, the checkpoint format version, the fronted services and the supported tool schema versions, used for release compatibility checks. The fronted services are known to the gateway from its fronts file and its build, and the answer lists them with their versions and repositories. The adapter version, the checkpoint format version and the tool schema versions are facts of AgentRuntimeService, and the model names no operation of that service from which a gateway without state could read them. The gateway may not invent them: until the owner names their source the answer states the three values as not available, and the gap is reported. A call that exceeds the rate limit configured for its route is refused with 429 through the pinned rate-limiting library, and a limit is configured per route in the configuration of the gateway. The gateway emits connectsoft.factory.runtime.host.requests with the dimensions service, route and status and connectsoft.factory.runtime.host.request_duration with service and route, the latency including the fronted service, through the single registration of feature component-observability-factory-conventions, and carries the trace context to the service; its logs never contain a token, a prompt, an artifact or a payload. A replica that crashes is detected by the readiness probe and restarts; because the gateway holds no state, nothing is replayed and callers retry with their own idempotency key. The roles of the gateway are Observer, Operator and Platform admin, and they apply to its two own operations: the health detail and the modules answer need the scope of the operation and one of the three roles, and a token that holds the scope and none of them is answered 403. The routes of the fronted services are admitted at the gateway by token and scope alone, the role in the project being checked by the service.
Acceptance criteria
| AC1 | Given the gateway with the identity platform and both fronted services reachable, when AgentRuntimeWorker/RuntimeHealth is called, then liveness answers 200 without a token and a caller with factory.health.read and one of the roles Observer, Operator or Platform admin receives 200 with the state of AgentRuntimeService, of WorkflowEngineService and of the identity platform (SCN-AgentRuntimeWorker-001, NFR-17). |
| AC2 | Given the identity platform is unreachable and a signing key is cached, when calls arrive, then tokens that were already issued are validated with the cached key, the gateway stays in rotation and AgentRuntimeWorker/RuntimeHealth reports degraded and names the identity platform; given no valid key remains, when calls arrive, then they are refused, AgentRuntimeWorker/RuntimeHealth answers 503 with a problem response that names the identity platform, the replica leaves rotation and no state changes (SCN-AgentRuntimeWorker-001, ADR-0015). |
| AC3 | Given one fronted service does not answer, when AgentRuntimeWorker/RuntimeHealth and the routes are called, then health reports that service as unavailable in its detail, its routes answer 503 with a retry hint that names the service and the routes of the other service keep serving (SCN-AgentRuntimeWorker-001, ADR-0028). |
| AC4 | Given a caller with factory.meta.read, when AgentRuntimeWorker/RuntimeModules is called, then the response is 200 with AgentRuntimeService and WorkflowEngineService, each with its version and repository, as model/services/agent-mesh/AgentRuntimeWorker.yaml lists them under deployment.fronts, and with the adapter version, the checkpoint format version and the tool schema versions stated as not available and never invented; a call without a token is answered 401, a token without the scope 403, and a token that holds factory.meta.read and none of the roles Observer, Operator and Platform admin 403, as is a token with factory.health.read and none of the three roles on the detail of AgentRuntimeWorker/RuntimeHealth; nothing is written (SCN-AgentRuntimeWorker-002, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| AC5 | Given routed calls and a call above the rate limit of its route, when the metrics and logs are read, then connectsoft.factory.runtime.host.requests counts every call by service, route and status, the call above the limit was refused with 429, connectsoft.factory.runtime.host.request_duration holds the latency by service and route, and no log record holds a token or a payload (NFR-08, XL-16, model/services/agent-mesh/AgentRuntimeWorker.yaml). |
| Traces: services | AgentRuntimeWorker |
| Traces: apis | AgentRuntimeWorker/RuntimeHealth, AgentRuntimeWorker/RuntimeModules |
| Traces: adrs | ADR-0015, ADR-0017, ADR-0028 |
| Traces: requirements | NFR-08, NFR-17, XL-16 |
| Traces: scenarios | SCN-AgentRuntimeWorker-001, SCN-AgentRuntimeWorker-002 |
| Depends on | AIFACTORY-US-472, feature:component-api-gateway-template, feature:component-observability-factory-conventions |
| Sources | model/services/agent-mesh/AgentRuntimeWorker.yaml, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md, docs/planes/agent-runtime/services/agent-runtime-worker/index.md#metrics-and-observability, docs/planes/agent-runtime/services/agent-runtime-worker/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-358 Health, modules and telemetry of the gateway |
implementation | The health operation serves liveness without a token and the detail under factory.health.read with the aggregated state of the two fronted services and of the identity platform, reports degraded and stays in rotation while a cached signing key validates tokens, and answers 503 naming the identity platform when no valid key remains; the modules operation lists the two fronted services with version and repository under factory.meta.read and states the adapter version, the checkpoint format version and the tool schema versions as not available; the health detail and the modules operation require one of the roles Observer, Operator and Platform admin beside their scope and answer 403 to a token that holds the scope and none of them; every routed call records connectsoft.factory.runtime.host.requests and connectsoft.factory.runtime.host.request_duration with their modelled dimensions and carries the trace context to the service. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1893 Rate limits per route from the configuration of the gateway |
implementation | Each of the eleven routes and the two own operations takes its rate limit from the configuration of the gateway through the pinned rate-limiting library and answers 429 above it; the refusal is counted in connectsoft.factory.runtime.host.requests with its status and reaches no fronted service. | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1894 Runbook of the three failure modes of the gateway |
documentation | The repository documentation holds one runbook entry per failure mode of model/services/agent-mesh/AgentRuntimeWorker.yaml (identity platform unreachable, fronted service unavailable, gateway replica crash) with its detection signal, the expected answers of the gateway and the recovery step (NFR-17). | ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |
AIFACTORY-TASK-1895 Verify health, modules, limits and telemetry through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-AgentRuntimeWorker-001\|Scenario=SCN-AgentRuntimeWorker-002)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-475 with the identity platform stopped with and without a cached key, one fake stopped, the modules answer under and without factory.meta.read, the 403 of the modules operation and of the health detail for a token that holds the scope and none of the roles Observer, Operator and Platform admin, and a call above its limit. |
ConnectSoft.AI.SoftwareFactory.AgentRuntimeWorker |