AIFACTORY-FEAT-014: ModelGatewayHost: gateway delivery¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Model & Compute Platform (PF-03) · Increment I1 Generate a microservice · State elaborated · Route backend-developer · Build order 59 of 185 (Factory code)
ModelGatewayHost is the platform API gateway of bounded context BC-07 (Model Access). The API gateway of the Model & Compute platform (ADR-0028): routes public REST and internal gRPC calls to the 2 services it fronts, validates tokens and scopes, and owns no business state. ModelRouterService behind it stays the only caller of model providers (GW-04). Its own repository is ConnectSoft.AI.SoftwareFactory.ModelGatewayHost.
| Delivers | ModelGatewayHost (complete) |
| Stubs | — |
| Realizes | adrs: ADR-0002, ADR-0012, ADR-0013, ADR-0017, ADR-0021 |
| Requirements | FR-04, FUX-08, FUX-10, NFR-07, NFR-12, NFR-13, NFR-18, XL-04, XL-16 |
| Depends on | — |
| Sources | model/services/model-compute/ModelGatewayHost.yaml |
Scenarios to trace (8, catalog): SCN-ModelGatewayHost-001 ✓, SCN-ModelGatewayHost-002 ✓, SCN-ModelGatewayHost-008 ✓, SCN-ModelGatewayHost-003 ✓, SCN-ModelGatewayHost-004 ✓, SCN-ModelGatewayHost-005 ✓, SCN-ModelGatewayHost-006 ✓, SCN-ModelGatewayHost-007 ✓
AIFACTORY-US-057: ModelGatewayHost routes the calls of its platform under a trusted scope¶
enabling · persona platform-admin
As a platform admin, I want one gateway in front of ModelRouterService and ComputePoolService that validates tokens and scopes, so that no service of the platform is reachable except through its gateway and every call arrives with a scope the service can trust.
This is the integration PBI of the gateway of the Model and Compute platform. ADR-0028 re-scopes the former composition host as the API gateway of its platform: ModelGatewayHost routes public REST and internal gRPC calls to the two services the model lists under deployment.fronts, ModelRouterService and ComputePoolService, validates tokens and scopes and owns no business state, no relational store and no secret. ModelRouterService stays the only service that calls model providers, and this gateway is the only route to it (ADR-0028 decision 3, gateway role GW-04). The repository ConnectSoft.AI.SoftwareFactory.ModelGatewayHost is created and scaffolded in I0 by AIFACTORY-TASK-044 of feature repository-and-pipeline-conventions (D140), and this feature starts from that scaffold: it is scaffolded once from the pinned API gateway template package with UseFactoryProfile and the gateway role Platform (feature component-api-gateway-template), which fixes persistence, messaging and the actor model to none. The gateway therefore has no outbox and no inbox, consumes no event (ADR-0028 amendment on gateways and events) and its only store is the shared cache. The scope of a call is established once from the validated token with the scope accessor of the pinned ConnectSoft.Extensions.AI.Factory.Abstractions package (feature component-extensions-ai); the gateway copies neither the accessor nor the token contract. A user token has one audience, this gateway, and the gateway forwards it to the fronted service unchanged and exchanges no token, because the fronted service accepts the audience of its own gateway (ADR-0015 amendment D117). A workload token carries service scopes and no tenant claim and is admitted only on an internal gRPC route whose operation names a service scope the token holds (ADR-0015 amendment D114). The feature is delivered by six further PBIs: the fronts file and the routes generated from it (AIFACTORY-US-561), token, tenant and scope on every read and sandbox route (AIFACTORY-US-562), the pass-through of a shared-write call with its idempotency key (AIFACTORY-US-563), the publication, production and spend routes whose gate stays in the fronted service, with the relay of a model stream (AIFACTORY-US-564), health, adapter metadata, limits and telemetry with the four failure modes of this gateway (AIFACTORY-US-565), and the OpenAI-compatible chat completions route, the one operation in which the gateway translates a wire protocol into model-invocation-v1 and calls ModelRouterService with the forwarded token of the caller, the token of an attempt for a run-bound call and a user token for a call outside a run (AIFACTORY-US-756, decisions D121 and D125). A call without a token, with an expired token or with a user token issued for another gateway is answered 401 and a missing scope 403, and no fronted service is called. No route of the two fronted services names a project or a tenant in its path: each addresses a profile, a deployment, a run, a pool or a placement by its identifier, so the gateway forwards under the tenant of the token and the scope of the operation, the fronted service answers 403 for a missing right and 404 for a resource of another tenant (ADR-0015 amendment D113), and the gateway returns that answer unchanged. ModelRouterService is delivered in the same increment. ComputePoolService arrives in I8: until then its routes are generated from its fronts entry and proven against a fake, and on a deployed instance a route of the service that is not deployed answers 503 with a problem response that names the service. The gateway invents no response.
Acceptance criteria
| AC1 | Given the repository ConnectSoft.AI.SoftwareFactory.ModelGatewayHost created and scaffolded in I0 by AIFACTORY-TASK-044 of feature repository-and-pipeline-conventions, whose first commit is the unchanged output of dotnet new connectsoft-apigateway -n ConnectSoft.AI.SoftwareFactory.ModelGatewayHost --UseFactoryProfile true --GatewayRole Platform from the pinned template package, when its pipeline runs with the work of this feature, then no second scaffold was run, the solution builds, the tests pass with complete reports and a container image is published (ADR-0028, model/services/model-compute/ModelGatewayHost.yaml). |
| AC2 | Given a valid token for a route of ModelRouterService, when the call passes the gateway, then the scope is established once from the token through the scope accessor of the pinned abstractions package, the call is forwarded with the token unchanged, the tenant, the correlation id and the trace context, the fronted service answers and the response returns unchanged, as AIFACTORY-US-562 to AIFACTORY-US-564 prove per route class (ADR-0015, FR-04). |
| AC3 | Given the generated gateway, when its architecture test runs, then the test fails on a persistence model, an aggregate, a migration or a message consumer, the gateway binds no relational store, no outbox and no inbox, its configuration holds no secret and no provider credential, and no event of the model lists ModelGatewayHost as a consumer (ADR-0028, model/services/model-compute/ModelGatewayHost.yaml). |
| AC4 | Given the gateway deployed with two replicas, when ModelGatewayHost/GatewayHealth is probed and one replica is stopped, then health reports the identity platform and the fronted services, a replica stays in rotation while it can validate tokens, the other replica keeps serving and nothing is replayed, because the gateway holds no state (ADR-0017, model/services/model-compute/ModelGatewayHost.yaml). |
| Traces: services | ModelGatewayHost, ModelRouterService |
| Traces: apis | ModelGatewayHost/GatewayHealth |
| Traces: adrs | ADR-0015, ADR-0017, ADR-0028 |
| Traces: requirements | FR-04 |
| Depends on | AIFACTORY-US-561, AIFACTORY-US-562, AIFACTORY-US-563, AIFACTORY-US-564, AIFACTORY-US-565, AIFACTORY-US-756, feature:repository-and-pipeline-conventions, feature:component-api-gateway-template, feature:component-extensions-ai |
| Sources | model/services/model-compute/ModelGatewayHost.yaml, model/gateways.yaml, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md, docs/planes/model-compute/services/model-gateway-host/index.md#purpose-and-responsibilities |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2261 Deployment values of the gateway: replica range, ports, probes and the address of each fronted service |
operations | The deployment manifest of the gateway states the replica range 2 to 12, the ports https 7600 and grpc 7602, the liveness and readiness probes on the health operation and one base address per fronted service from configuration, as model/services/model-compute/ModelGatewayHost.yaml records them; it states ModelGateway:Host:FrontedServices with the two service identifiers of the fronts file, which are also the modelled default of the option (D121), and holds no secret; the same image runs in the saas-default, hybrid, on-prem and hybrid-compute topologies with the identity authority of the profile (ADR-0017). | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-766 Verify model-gateway-host-routes-its-platform by running the declared command |
verification | Run dotnet build -c Release and dotnet test -c Release --no-build in the pipeline of the repository; a run that discovers no test counts as a failure; the runs pass and prove AC1 to AC4 of AIFACTORY-US-057, that is the repository with its scaffold of the Platform role, the scope established once and forwarded, the architecture test of the Platform role and, for AC4, the acceptance test of feature TwoReplicas (dotnet test -c Release --filter "Category=Acceptance&Feature=TwoReplicas"), which hosts two instances of the gateway behind one test balancer with the fakes of the fronted services, probes ModelGatewayHost/GatewayHealth on both, stops one instance and asserts that the other keeps serving and that no call is replayed. |
ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-US-561: Model gateway fronts file: named clients and REST and gRPC routes of ModelRouterService and ComputePoolService, generated at build¶
enabling · persona platform-admin
As a platform admin, I want the route table of the model gateway generated from the list of services it fronts, so that every operation of the platform has exactly one route with its scope and class and no forwarding code is written by hand.
The API gateway template is a custom gateway with controllers, use cases and named clients and is no reverse proxy, so a route exists only when it is declared. Under UseFactoryProfile the generated solution holds a build-time source generator that reads a fronts file on every build and emits one named HTTP client and one gRPC client per fronted service, the REST and gRPC routes that forward to them and the route table (feature component-api-gateway-template). This PBI writes the fronts file of the Model and Compute platform. It lists the two services of deployment.fronts in model/services/model-compute/ModelGatewayHost.yaml, each with its base addresses from configuration and its operations in the shape of the apis section of its service file: name, method and path or gRPC method with its streaming mode, scope, side-effect class and idempotency. For ModelRouterService these are the ten public REST operations, ListModelProfiles, GetModelProfile, ListModelDeployments, RegisterDeployment, EvaluateDeployment, ApproveDeployment, AttestDeployment, which decision D121 added for the approval of a deployment by attestation, PreviewRoutePlan, ListRunReceipts and ListProjectReceipts, which decision D123 added for the receipts of a project that name no run, and the four internal gRPC methods of ModelGateway: ResolveRoutePlan, Invoke with server streaming, Embed and Rerank. For ComputePoolService these are its five REST operations and the gRPC methods ComputePools.Place and ComputePools.ReportUtilization, the second with client streaming. The REST operations are the public surface that the typed model gateway client and, later, StudioBff call; the gRPC methods are the internal surface the agent hosts call through the Factory model client of the AI libraries and the Knowledge services call for embeddings and reranking. Each route carries the scope and the side-effect class of its operation, which is what the route class PBIs of this feature enforce; an internal gRPC route names the service scope a workload token must hold. A test lists the emitted route table against the file and against the two service files of the model, so an operation of the model without a route, a route without an operation, a service named twice or a route named twice fails the build with the entry named. A path that no route matches is answered 404. The gateway adds no operation of its own beyond GatewayHealth, GatewayAdapters and the OpenAI-compatible route CreateChatCompletion of AIFACTORY-US-756, transforms no payload on a route of the fronts file and composes no response from two services. The option ModelGateway:Host:FrontedServices is bound at startup and compared with the service names of the fronts file: a value that names no service of the file, or a service of the file that the value omits, fails startup with the name, so the option can neither add nor remove a route; it is not reloadable. The fronts file names its services by their identifiers, ModelRouterService and ComputePoolService, as deployment.fronts lists them, and the modelled default of the option names the same two identifiers (decision D121), so a gateway that leaves the option unset starts. The two switches ModelGateway:Protocols:ExposeOpenAiCompatible and ModelGateway:Protocols:ExposeAnthropicCompatible are bound and validated at startup, and neither is reloadable. ExposeOpenAiCompatible, true by default, guards the one public operation the gateway serves itself, CreateChatCompletion, POST /v1/chat/completions, which is no entry of the fronts file: the gateway serves it itself, translates the call into model-invocation-v1 and calls ModelGateway.Invoke of ModelRouterService, and AIFACTORY-US-756 delivers it in this increment (decision D121). ExposeAnthropicCompatible is false by default; its route is a later feature and the model declares no operation for it, so this delivery hosts no Anthropic-compatible route, and a start with that switch true fails with the switch named, because no route stands behind it. The gateway clients of feature typed-gateway-clients are generated from a contract-first OpenAPI document of the client repository until a gateway publishes its own (decision D117): the pipeline of this repository publishes the OpenAPI document of the ten public REST operations of ModelRouterService and of the public operation CreateChatCompletion of the gateway itself as a build output, checked against the fronts file and against the two service files of the model. The five REST operations of ComputePoolService are not in the document of this increment. Feature typed-gateway-clients generates the model gateway client from the operations of the services of increment I1 or earlier that the gateway fronts, and ComputePoolService arrives in I8, so a document that held its operations would put operations of a later increment into the I1 client; they join the published document with feature compute-pool-service. Their routes are generated from the fronts file all the same and proven against the fake, as the route class PBIs of this feature need.
Acceptance criteria
| AC1 | Given the fronts file of the Model and Compute platform, when the gateway builds, then one named HTTP client and one gRPC client exist per service of deployment.fronts and one route per REST operation and gRPC method of those services, each with the scope, side-effect class, idempotency and streaming mode of the operation, and the route table test passes against model/services/model-compute/ModelGatewayHost.yaml, model/services/model-compute/ModelRouterService.yaml and model/services/model-compute/ComputePoolService.yaml (ADR-0028). |
| AC2 | Given an operation removed from the file, a route named twice or a service named twice, when the gateway builds, then the build fails with the entry named; given ModelGateway:Host:FrontedServices left at its modelled default of model/services/model-compute/ModelGatewayHost.yaml, which names the two service identifiers of the fronts file (D121), or set to those two names, when the gateway starts, then startup succeeds; given a value of the option that differs from the services of the fronts file, when the gateway starts, then startup fails with the differing name; given ModelGateway:Protocols:ExposeAnthropicCompatible true, when the gateway starts, then startup fails with the switch named, because this delivery holds no Anthropic-compatible route; given a path no route matches, when it is called, then the answer is 404 and no fronted service is called (ADR-0028, docs/architecture/catalogs/gateways.md). |
| AC3 | Given a fake of each fronted service, when one REST route and one gRPC route per service are called with a valid token, then each call reaches the fake of its service with the authorization header, which carries the token of the caller unchanged because the gateway exchanges no token, the tenant, the correlation id and the trace context, and the response of the fake returns unchanged (ADR-0015, FR-04). |
| AC4 | Given the built gateway, when its pipeline publishes the OpenAPI document of the public REST routes, then the document holds exactly the ten REST operations of ModelRouterService of the fronts file and the public operation CreateChatCompletion of the gateway itself with their methods, paths, scopes and response codes, holds no operation of ComputePoolService, no internal operation of the gateway and no internal gRPC method, and a contract check fails the build when it differs from model/services/model-compute/ModelRouterService.yaml or from model/services/model-compute/ModelGatewayHost.yaml; the routes of the five operations of ComputePoolService exist in the route table of AC1 and are absent from the document until feature compute-pool-service adds them (docs/architecture/api-governance.md). |
| Traces: services | ModelGatewayHost, ModelRouterService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FR-04 |
| Depends on | feature:component-api-gateway-template |
| Sources | model/services/model-compute/ModelGatewayHost.yaml, model/services/model-compute/ModelRouterService.yaml, model/services/model-compute/ComputePoolService.yaml, model/gateways.yaml, docs/architecture/catalogs/gateways.md, docs/planes/model-compute/services/model-gateway-host/index.md#context-and-boundaries |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-045 Fronts file with the REST and gRPC operations of ModelRouterService and ComputePoolService |
implementation | The fronts file lists the two services of deployment.fronts with their base addresses from configuration and every REST operation and gRPC method of their service files with name, path or method, streaming mode, scope, side-effect class and idempotency; the build emits the named clients, the routes and the route table; at startup the option ModelGateway:Host:FrontedServices is compared with the service names of the fronts file and a difference in either direction fails startup with the name; the two protocol switches are bound and validated, ExposeOpenAiCompatible guarding the route of AIFACTORY-US-756 and a start with ExposeAnthropicCompatible true failing with the switch named, because its route is a later feature (D121); a call without a valid token or scope is refused at the gateway by the generated policies. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2262 Fakes of the two fronted services and the token matrix of the acceptance suite |
fixture | The acceptance suite of the gateway holds one fake per fronted service that records the headers and metadata it received and answers a configured status, a fake of the server stream of ModelGateway.Invoke that emits a scripted number of messages and then a receipt or an error status, a switch that makes a fake stop answering, and tokens for the matrix of role and tenant, including an expired token, an identity token, a user token of another audience and a workload token with and without the service scope of a gRPC route. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2263 OpenAPI document of the public REST routes of the gateway as a published build output |
contract | The pipeline publishes one OpenAPI document with the ten public REST operations of ModelRouterService of the fronts file and the public operation CreateChatCompletion of the gateway itself, their methods, paths, scopes, idempotency headers and response codes, and no internal gRPC method; a contract check compares the document with model/services/model-compute/ModelRouterService.yaml and, for CreateChatCompletion, with model/services/model-compute/ModelGatewayHost.yaml, and fails the build on a missing operation, an operation of another service or a differing method, path, scope or response code, so the document can take the place of the contract-first document of the client repository (decision D117); the five REST operations of ComputePoolService stay out of the document until feature compute-pool-service adds them in I8, while their routes stay in the route table. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2264 Verify the route table against the fronts file and the model |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=RouteTable"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-561: one route per operation of model/services/model-compute/ModelRouterService.yaml and model/services/model-compute/ComputePoolService.yaml, the failing builds, the start with the option left at its modelled default and with the option set to the names of the fronts file, the failing startup on a differing value and with ExposeAnthropicCompatible true, the 404 of an unknown path, one forwarded call per service and protocol and the contract check of the published OpenAPI document, which is limited to the ten operations of model/services/model-compute/ModelRouterService.yaml and the operation CreateChatCompletion of model/services/model-compute/ModelGatewayHost.yaml and fails when the document holds an operation of ComputePoolService. |
ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-US-562: A read or sandbox route of the model gateway forwards a call only under a validated token, the tenant of the token and the scope of its operation¶
behavior · persona requester
As a requester, I want my read calls to reach the model services only with the tenant and scope my token grants, so that I see the profiles, deployments and receipts I am entitled to and nobody reads across a tenant through the gateway.
A caller reaches a read or sandbox operation of the Model and Compute platform through the gateway: a requester reads the model profiles before submitting a run, the model receipts of a run and the receipts of a project that name no run, a Platform admin or an Observer lists the model deployments, a Platform admin previews a route plan, and a Platform admin starts the capability evaluation of a deployment, which is the sandbox route of this platform. A user presents an access token of the authorization server whose one audience is this gateway (ADR-0015 amendment D114); a client that also calls the Control Plane gateway requests a second token for it. The gateway validates issuer, signature, lifetime and audience through the pinned resource-server library and requires the scope the operation names: factory.models.read for ListModelProfiles, GetModelProfile, PreviewRoutePlan and ListProjectReceipts, factory.models.admin for ListModelDeployments and EvaluateDeployment, factory.runs.read for ListRunReceipts, and factory.compute.read for the two read operations of ComputePoolService. It takes the tenant from the tid claim and overwrites any tenant header of the caller. Every public REST operation of the two services names a factory. scope and none names a service scope, so no read or sandbox REST route admits a workload token today, the state feature typed-gateway-clients records as well; a workload token on such a route is answered 403. The scope is established once with the scope accessor of the pinned abstractions package and forwarded: the authorization header with the user token itself, the tenant in X-Tenant-Id, the correlation id and the trace context. The gateway exchanges no token; ModelRouterService validates the forwarded token again, accepts the audience of its own gateway, which is this one, and applies its own tenant filter and role policies (ADR-0015 amendment D117). One read route names a project in its path: ListProjectReceipts, GET /factory/projects/{projectId}/model-receipts, the read of the receipts of a project that name no run, which are the receipts of the calls outside a run on the OpenAI-compatible route (AIFACTORY-US-756, decision D123). For that route the gateway checks the projectId of the path against the role claims of the validated token, answers 403 for a project the token does not grant and calls no fronted service in that case, and forwards the call for a granted project; the check needs no business state, because the token carries the roles. No other read or sandbox route names a project or a tenant in its path: a profile, a deployment, a run and a placement are addressed by identifier, and a gateway without business state cannot know the tenant or the project of such a resource. It forwards such a call under the tenant of the token and the scope of the operation; the fronted service answers 403 for a missing role, such as an Observer who starts an evaluation or a caller without a role in the project of a run, and 404 for a profile, a deployment or a run of another tenant (ADR-0015 amendment D113), and the gateway returns that answer unchanged. For a read route the response is served by the fronted service with no side effect. For the sandbox route EvaluateDeployment the gateway forwards the Idempotency-Key header unchanged, adds no gate of its own to the class and returns the 202 of the service; the service changes no shared state by that call, and the capability evidence arrives later inside ModelRouterService. The gateway returns every response unchanged with the correlation id and keeps nothing of the call. A call without a token, with an expired token, with an identity token or with a user token issued for another gateway is answered 401 and a token without the scope of the operation 403; in these cases no fronted service is called, and the refusal is counted for the route and logged with the correlation id and without the token. When the fronted service does not answer, the route answers 503 with a retry hint and the routes of the other service keep serving. The read routes of ComputePoolService are proven against the fake generated from its fronts entry, because the service arrives in I8.
Acceptance criteria
| AC1 | Given a user token whose audience is this gateway, with factory.models.read, when the read route of ListModelProfiles is called, then the gateway forwards the call with the user token unchanged in the authorization header, without a token exchange, the tenant of the tid claim, the correlation id and the trace context, ModelRouterService answers, the response returns unchanged with the correlation id and nothing is written by the gateway (SCN-ModelGatewayHost-005, ADR-0015, FR-04). |
| AC2 | Given the token matrix of the acceptance suite, when the read routes of ModelRouterService are called, then a call without a token, with an expired token, with an identity token or with a user token issued for another gateway is answered 401, a token without the scope of the operation and a workload token are answered 403, and in none of these cases a fronted service is called; each refusal is counted for its route and logged with the correlation id and without the token (SCN-ModelGatewayHost-005, ADR-0015). |
| AC3 | Given a valid user token and a tenant header of the caller that names another tenant, when a read route is called, then the header is overwritten with the tid claim before the call is forwarded; given a route addressed by a profileId, a deploymentId or a runId, when the fronted service answers 403 for a missing role or 404 for a resource of another tenant (ADR-0015 amendment D113), then the gateway returns that answer unchanged and adds no check of its own; given the read route of ListProjectReceipts, whose path names a project, when it is called with a token whose role claims do not grant that project, then the gateway answers 403 and calls no fronted service, and for a project the token grants it forwards the call and returns the answer of ModelRouterService unchanged (SCN-ModelGatewayHost-005, ADR-0015). |
| AC4 | Given a user token with factory.models.admin and an Idempotency-Key, when the sandbox route of EvaluateDeployment is called, then the call is forwarded under the same token, tenant and scope rules with the header unchanged, the gateway adds no gate of its own to the class and the 202 of ModelRouterService returns unchanged; the refusals apply as on a read route and no fronted service is called when the gateway refuses (SCN-ModelGatewayHost-006, model/services/model-compute/ModelRouterService.yaml). |
| AC5 | Given a fronted service that does not answer, among them ComputePoolService before its delivery in I8, when one of its read routes is called, then the gateway answers 503 with a problem response that carries a retry hint and names the service and never its address, invents no response, and the read routes of the other service keep serving (SCN-ModelGatewayHost-005, model/services/model-compute/ModelGatewayHost.yaml). |
| Traces: services | ModelGatewayHost, ModelRouterService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FR-04, FUX-08 |
| Traces: scenarios | SCN-ModelGatewayHost-005, SCN-ModelGatewayHost-006 |
| Depends on | AIFACTORY-US-561, feature:component-api-gateway-template, feature:component-extensions-ai |
| Sources | model/services/model-compute/ModelGatewayHost.yaml, model/services/model-compute/ModelRouterService.yaml, docs/architecture/adr/0015-factory-tenancy-under-canonical-adr-0100.md, docs/planes/model-compute/services/model-gateway-host/index.md#purpose-and-responsibilities |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2265 Issuer, audience and scope configuration of the read and sandbox routes |
implementation | The gateway is configured with the authority of ConnectSoftIdentityPlatform:Authority and its own resource as the one audience of a user token; each read and sandbox route requires the scope of its operation and is marked as addressed by a resource id only, so the gateway checks token and scope and leaves the role and the tenant of the resource to the fronted service, except the one route whose path names a project, ListProjectReceipts, for which the gateway checks the projectId of the path against the role claims of the token and answers 403 for a project the token does not grant, without a call of the fronted service (D123); the tenant header is overwritten with the tid claim; a workload token is refused with 403 on a REST route, because no REST operation names a service scope; the scope is established through the scope accessor of the pinned abstractions package and forwarded in the canonical tenant header; the user token is forwarded unchanged in the authorization header and no token is exchanged (ADR-0015 amendment D117); the Idempotency-Key header of EvaluateDeployment is forwarded unchanged; refusals are counted per route and logged with the correlation id and without the token. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2266 Verify the read and sandbox routes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ModelGatewayHost-005\|Scenario=SCN-ModelGatewayHost-006)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-562 over the token matrix, the token received by the fake equal to the user token of the caller, a route addressed by a resource id whose 403 and 404 come from the fake and return unchanged, the route of ListProjectReceipts called with a token that grants the project of the path, which is forwarded, and with one that does not, which is answered 403 without a call of the fake, the overwritten tenant header, a workload token refused on a REST route, the sandbox route with its header and its 202 unchanged, and a fake that stops answering. |
ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-US-563: A shared-write route of the model gateway passes the idempotency key through and the gateway keeps nothing between calls¶
behavior · persona platform-admin
As a platform admin, I want a write I repeat after a failure to be applied once, so that a lost answer or a restarted gateway never registers a second deployment or resolves a second route plan.
A shared-write operation changes state that others read. On this platform these are RegisterDeployment of ModelRouterService, a public REST operation a Platform admin calls with factory.models.admin, the internal gRPC method ModelGateway.ResolveRoutePlan of ModelRouterService, which AgentRuntimeService calls once per run and profile, and, from I8, RegisterComputePool and WarmModel and the gRPC methods ComputePools.Place and ComputePools.ReportUtilization of ComputePoolService. The caller sends a REST call through the gateway under the token, tenant and scope rules of AIFACTORY-US-562. RegisterDeployment declares idempotency key-header: it is called with an Idempotency-Key, the gateway forwards the header unchanged and ModelRouterService applies the request once under its own rule, where the same key with the same body returns the same deployment and the same key with a different body is a conflict, and that 409 returns to the caller unchanged. An internal gRPC route is called by a workload: the token carries service scopes, no audience of a user API and no tenant claim, and the route names the service scope it must hold. For a workload token that holds the scope, the gateway forwards the tenant-id and the project metadata of the caller unchanged, because the fronted service reads a forwarded tenant and project only under that scope (ADR-0015 amendment D114); for a workload token without it, and for a user token, which holds no service scope, the answer is PERMISSION_DENIED and no fronted service is called. ResolveRoutePlan is idempotent per run and profile inside ModelRouterService, so a repeated call returns the plan that was resolved first. The gateway is no party to idempotency. It stores neither a key nor a result, keeps no session and no pending call, has no outbox and writes no event, so a gateway replica that crashes during a call loses nothing: the replica restarts, nothing is replayed, and the caller retries with the same key, or for the same run and profile, and receives from the service the result of the first application (failure mode of a gateway replica crash in the model). When the fronted service does not answer, the route answers 503, over gRPC UNAVAILABLE, with the service named and a retry hint, the routes of the other service keep serving, and the caller retries. A refusal by the gateway reaches no fronted service, so it applies nothing. A missing key, an invalid body, the 403 of the service for a caller who is not Platform admin, a NOT_FOUND for a profile the tenant cannot read and every domain refusal are answers of the fronted service and return unchanged with the correlation id. The gateway records request count and latency for the route with the status that was returned. The shared-write routes of ComputePoolService are proven against the fake generated from its fronts entry until the service arrives in I8.
Acceptance criteria
| AC1 | Given a valid user token with factory.models.admin and the shared-write route of RegisterDeployment, when the call arrives with an Idempotency-Key, then the gateway forwards the request with the header unchanged, the fronted service applies it once and the response returns unchanged with the correlation id (SCN-ModelGatewayHost-007, ADR-0015). |
| AC2 | Given the same call repeated with the same key, and then with the same key and a different body, when both pass the gateway, then the first returns the result of the first application and the second returns the 409 of the fronted service unchanged; the gateway has stored neither the key nor a result (SCN-ModelGatewayHost-007, ADR-0028). |
| AC3 | Given a workload token that holds the service scope the route of ModelGateway.ResolveRoutePlan names, when the method is called with tenant-id and project metadata, then the gateway forwards the call with that metadata unchanged and returns the reply unchanged, and a second call for the same run and profile returns the plan of the first as the fronted service answers it; given a workload token without the scope, or a user token, which holds no service scope, then the answer is PERMISSION_DENIED and no fronted service is called (SCN-ModelGatewayHost-007, ADR-0015). |
| AC4 | Given a gateway replica stopped while a shared-write call is in flight, when the caller retries with the same key through another replica, then the fronted service returns the result of the one application, nothing is replayed by the gateway and no second state change exists (SCN-ModelGatewayHost-007, model/services/model-compute/ModelGatewayHost.yaml). |
| AC5 | Given a call without a token, a token that lacks the scope, or a fronted service that does not answer, when a shared-write route is called, then the answer is 401, 403 or 503 respectively, over gRPC UNAUTHENTICATED, PERMISSION_DENIED or UNAVAILABLE; a call the gateway refuses reaches no fronted service; the 403 of the fronted service for a missing role and its 404 for a resource of another tenant (ADR-0015 amendment D113) return unchanged; the 503 names the service and carries a retry hint while the routes of the other service keep serving (SCN-ModelGatewayHost-007, ADR-0015). |
| Traces: services | ModelGatewayHost, ModelRouterService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FR-04 |
| Traces: scenarios | SCN-ModelGatewayHost-007 |
| Depends on | AIFACTORY-US-561, AIFACTORY-US-562, feature:component-api-gateway-template |
| Sources | model/services/model-compute/ModelGatewayHost.yaml, model/services/model-compute/ModelRouterService.yaml, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md, docs/architecture/adr/0014-run-and-lifecycle-envelopes.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2267 Forwarded headers and metadata of a write route and a stateless call path |
implementation | A shared-write REST route forwards the Idempotency-Key header, the body and the established scope unchanged and returns the status and the body of the fronted service unchanged; an internal gRPC route names the service scope a workload token must hold, answers PERMISSION_DENIED without it and forwards the tenant-id and project metadata of the caller unchanged with it (ADR-0015 amendment D114); no key, body or result is written to the shared cache or to any store; request count and latency are recorded for the route with the returned status. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2268 Verify the shared-write routes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ModelGatewayHost-007"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-563 with a fake that applies a key once, the gRPC route of ModelGateway.ResolveRoutePlan under a workload token with and without the service scope and under a user token of the token matrix (PERMISSION_DENIED and no call to the fake), a replica stopped during a call, a route addressed by a resource id whose 403 and 404 come from the fake and return unchanged, and a fake that stops answering. |
ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-US-564: A publication, production or spend route is forwarded under its scope, the gate stays in the fronted service and a model stream passes through unspliced¶
behavior · persona platform-admin
As a platform admin, I want a gated action and a model call to be decided by the service that owns their state, with the gateway only checking who calls and what may be called, so that an approval, a budget or a provider is never decided twice or by a component that holds no state.
Three side-effect classes of this platform carry a gate beyond the scope. The publication routes are two operations of ModelRouterService: ApproveDeployment, which moves an Evaluated deployment to Approved after license and evidence checks, and AttestDeployment, by which a Platform admin approves a Candidate or an Evaluated deployment by attestation with a required reason (decisions D121 and D123). The production route is DrainComputePool of ComputePoolService, a production action on serving capacity. The spend routes are the internal gRPC methods ModelGateway.Invoke, ModelGateway.Embed and ModelGateway.Rerank of ModelRouterService, each of which calls a model provider; spend is the highest side-effect class the model gives this gateway. The one spend operation the gateway serves itself, the OpenAI-compatible route CreateChatCompletion, ends in the same ModelGateway.Invoke and is delivered by AIFACTORY-US-756. ADR-0028 leaves a platform gateway routing and the validation of tokens and scopes and no business state, and names a gateway that accumulates logic beyond routing and validation as a reason to revisit the decision. The gateway therefore treats these routes as it treats every route: it validates the token, takes the tenant from a user token, requires the scope the operation names, factory.models.admin for the approval and for the attestation and factory.compute.admin for the drain, or the service scope a workload token must hold on a gRPC route, and forwards the call with the established scope. The gate itself is decided where the state is. ModelRouterService approves a deployment only in state Evaluated, with license metadata that permits the use and capability evidence that meets the thresholds of every profile that lists the deployment, and answers 409 or 412 otherwise; it approves by attestation only a Candidate or an Evaluated deployment (decision D123) whose request states a reason, and answers 400 without a reason and 409 for a deployment in another state or with a license that forbids the use; the gateway evaluates no approval and no attestation, holds no approval reference and returns those answers unchanged. For a model call the gateway resolves no profile, selects no provider, holds no provider credential, asks for no policy decision, reads no budget and writes no receipt: the policy decision, the brokered credential, the adapter call, the receipt and the usage belong to ModelRouterService. The gateway writes no audit evidence; ModelRouterService writes its events in the transaction of its command and AuditService records evidence from them. ModelGateway.Invoke is a server stream. The gateway relays each message in the order the service sends it, buffers no whole output, propagates a cancellation of the caller to the service and ends the stream with the final message of the service, which carries the receipt; an error status of the service ends the stream with that status unchanged. When a gateway replica crashes during a stream, the caller sees the stream break (failure mode of a gateway crash mid-stream in the model). The gateway does not reconnect, resume or splice a stream and replays nothing; ModelRouterService observes the broken call, records the partial output status in the receipt of the invocation, and the caller reads the run state and the receipt and never joins the output of two calls. A caller that repeats the call with the same idempotency key receives the existing receipt from the service and no second provider call. A call without a token is answered 401, over gRPC UNAUTHENTICATED, and a token without the scope 403, over gRPC PERMISSION_DENIED, and no fronted service is called. A user token on ModelGateway.Invoke, ModelGateway.Embed or ModelGateway.Rerank holds no service scope and is answered PERMISSION_DENIED in the same way; a user reaches a chat model only through the route of AIFACTORY-US-756, under the scope factory.models.invoke. Every one of these routes names only a resource id or a message of the call: the fronted service answers 403 for a missing role and 404, over gRPC NOT_FOUND, for a resource of another tenant (ADR-0015 amendment D113), and the gateway returns that answer unchanged. The production route is proven against the fake of ComputePoolService generated from its fronts entry, because the service arrives in I8.
Acceptance criteria
| AC1 | Given a user token with factory.models.admin, when the publication route of ApproveDeployment is called against the fake of ModelRouterService that answers 200, 409 or 412, or the publication route of AttestDeployment against the fake that answers 200, 400 or 409, then the gateway forwards the call, for the attestation with its body unchanged, under the established scope and returns each answer unchanged with the correlation id; it evaluates no approval and no attestation, holds no approval reference and writes nothing (SCN-ModelGatewayHost-004, ADR-0028, model/services/model-compute/ModelRouterService.yaml). |
| AC2 | Given a user token with factory.compute.admin, when the production route of DrainComputePool is called against the fake of ComputePoolService generated from the fronts entry, then the gateway forwards the call under the tenant of the token and the scope of the operation, because the route names only the poolId, and returns the answer of the fake unchanged; the gateway evaluates no approval and writes no audit evidence (SCN-ModelGatewayHost-003, ADR-0028). |
| AC3 | Given a workload token that holds the service scope of the spend routes, when ModelGateway.Embed and the server stream of ModelGateway.Invoke are called, then the unary reply returns unchanged and every message of the stream reaches the caller in the order the fake of ModelRouterService sent it, ending with the message that carries the receipt; a cancellation of the caller reaches the fake, and an error status of the fake ends the stream with that status unchanged; the gateway selected no provider, held no credential and read no budget (ADR-0028, model/services/model-compute/ModelGatewayHost.yaml). |
| AC4 | Given a gateway replica stopped while a stream of ModelGateway.Invoke is open, when the caller observes the break and repeats the call with the same idempotency key through another replica, then the first stream ended with an error and was neither resumed nor spliced, the second call returns what the fronted service answers for that key, and the gateway replayed nothing, as the failure mode of model/services/model-compute/ModelGatewayHost.yaml states. |
| AC5 | Given a call without a token or with an expired token, or a token that lacks the scope of the operation, when a publication, production or spend route is called, then the gateway answers 401 or 403, over gRPC UNAUTHENTICATED or PERMISSION_DENIED, and no fronted service is called; given a user token, which holds no service scope, on the spend route of ModelGateway.Invoke, ModelGateway.Embed or ModelGateway.Rerank, then the answer is PERMISSION_DENIED and no fronted service is called; given a route that names only a resource id, such as the deploymentId of an approval or the poolId of a drain, when the fronted service answers 403 for a missing role or 404 for a resource of another tenant (ADR-0015 amendment D113), then the gateway returns that answer unchanged (SCN-ModelGatewayHost-003, SCN-ModelGatewayHost-004, ADR-0015). |
| Traces: services | ModelGatewayHost, ModelRouterService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FR-04, XL-04 |
| Traces: scenarios | SCN-ModelGatewayHost-003, SCN-ModelGatewayHost-004 |
| Depends on | AIFACTORY-US-561, AIFACTORY-US-562, feature:component-api-gateway-template |
| Sources | model/services/model-compute/ModelGatewayHost.yaml, model/services/model-compute/ModelRouterService.yaml, model/services/model-compute/ComputePoolService.yaml, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md, docs/planes/model-compute/model-gateway.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-2269 Fakes of the publication, production and spend routes and the route class register of the gateway |
fixture | The acceptance suite holds a fake of ApproveDeployment that answers 200, 409 and 412, a fake of AttestDeployment that records the body it received and answers 200, 400 and 409, a fake of ComputePoolService generated from the production entry of the fronts file, and the stream fake of ModelGateway.Invoke of AIFACTORY-TASK-2262 with scripts for a complete stream, an error status and a stream that stays open; a register test lists every route of class publication, production and spend from the route table and fails when such a route has no scope or when the gateway holds a handler, a cache entry or a store for an approval, a budget, a provider credential or a receipt. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2270 Relay of the server stream of ModelGateway.Invoke with cancellation and status pass-through |
implementation | The generated gRPC route of ModelGateway.Invoke relays each message of the fronted service to the caller in order without buffering the whole output, propagates the cancellation and the deadline of the caller to the fronted service, ends the stream with the final message or the error status of the service unchanged, and holds no state of an open stream that another replica could resume; request count and latency of the route are recorded when the stream ends, with the status that ended it. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2271 Verify the publication, production and spend routes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ModelGatewayHost-003\|Scenario=SCN-ModelGatewayHost-004)" and dotnet test -c Release --filter "Category=Acceptance&Feature=ModelStreamRelay"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-564: each class forwarded under its scope, the answers 409 and 412 of the approval and 400 and 409 of the attestation returned unchanged, the ordered relay and the cancellation of a stream, a replica stopped during an open stream, the refusals of the gateway without a call to a fronted service, among them a user token of the token matrix on ModelGateway.Invoke, ModelGateway.Embed and ModelGateway.Rerank answered PERMISSION_DENIED, and the register of gated routes. |
ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-US-565: Health, adapter metadata, limits and telemetry of the model gateway while the identity platform, a fronted service or a replica fails¶
behavior · persona operator
As an operator, I want to see whether the model gateway, the identity platform and each fronted service answer, and which provider adapters stand behind the gateway, so that I can tell a failing service from a failing gateway and act on the right one.
An operator and the probes of the cluster read the state of the gateway through its two own operations. ModelGatewayHost/GatewayHealth reports liveness and readiness of the gateway with the aggregated health of the two services it fronts and of the identity platform. Liveness needs no token; the detail is served to a caller whose scope carries factory.health.read. The identity platform is the one required dependency of the gateway, and the health check is the detection signal the model names for its failure. What the gateway does when it is unreachable is decided (D117, ADR-0015 amendment). While a cached signing key still validates tokens, the gateway stays in rotation: readiness stays ready, tokens that were already issued keep passing, and from the first failed probe GatewayHealth answers 200 with the health degraded and a detail that names the identity platform as unreachable, so the operator sees the failure in health while calls still pass. The gateway leaves rotation only when no valid key remains, because none was cached or the cached keys expired: readiness turns unready, GatewayHealth answers 503 with a problem response that names the identity platform, and calls are refused, so the gateway fails closed. Liveness keeps answering 200 without a token in both phases. A fronted service is an optional dependency: when it does not answer, health reports it as unavailable in the detail, its routes answer 503 with a retry hint, and the routes of the other service keep serving. Until ComputePoolService is delivered in I8, the detail reports it as unavailable and the answer stays 200. ModelGatewayHost/GatewayAdapters returns to a caller whose scope carries factory.meta.read the provider adapters of the services behind this gateway with their package versions and supported protocols: OpenAI-compatible, Anthropic-compatible, embeddings, reranking, streaming, structured outputs and tool calls. The gateway loads no adapter and holds no state: the adapters are loaded by ModelRouterService from the pinned AI library, as its option ModelGateway:Adapters:Enabled names them. The model gives ModelRouterService no operation that returns that list, so the source of the answer is an open point reported to the owner. Until it is decided, the operation is hosted with its scope, its refusals and its response shape and reads the list through one metadata port of the gateway, which the acceptance suite fills with the fake of ModelRouterService; a deployed instance whose port has no source answers 503 with a problem response that says so and invents no list. A call that exceeds the rate limit configured for its route is refused with 429 through the pinned rate-limiting library, from the configuration of the gateway. The gateway emits connectsoft.factory.model_gateway.host.requests with the dimensions protocol and status and connectsoft.factory.model_gateway.host.request_duration with service and route, the latency including the fronted service, and carries the trace context to the service; the duration is the series against which the objective of the model for the access overhead of gateway and router, 150 ms at p95 without provider time, is measured once the system is deployed, and no value is claimed here. Logs, activities and metrics never contain a token, a prompt, a completion, a tool output or a payload (ADR-0019). A replica that crashes is detected by the readiness probe and restarts; because the gateway holds no state, nothing is replayed, and an open model stream of that replica breaks as AIFACTORY-US-564 states. The roles of the gateway are Observer, Operator and Platform admin.
Acceptance criteria
| AC1 | Given the gateway with the identity platform and ModelRouterService reachable, when ModelGatewayHost/GatewayHealth is called, then liveness answers 200 without a token and a caller with factory.health.read receives 200 with the state of each of the two fronted services and of the identity platform, ComputePoolService being reported as unavailable in the detail until its delivery in I8 without turning the answer into 503 (SCN-ModelGatewayHost-001, NFR-18). |
| AC2 | Given the identity platform is unreachable and a cached signing key still validates tokens, when calls arrive and ModelGatewayHost/GatewayHealth is probed, then tokens that were already issued are validated with the cached key, readiness stays ready so the replica stays in rotation, and from the first failed probe health answers 200 with the health degraded and a detail that names the identity platform as unreachable, while liveness keeps answering 200; given no valid key remains, because none was cached or the cached keys expired, when calls arrive, then they are refused, readiness is unready so the replica leaves rotation, ModelGatewayHost/GatewayHealth answers 503 with a problem response that names the identity platform and no state changes (SCN-ModelGatewayHost-001, model/services/model-compute/ModelGatewayHost.yaml). |
| AC3 | Given ModelRouterService does not answer, when ModelGatewayHost/GatewayHealth and the routes are called, then health reports that service as unavailable in its detail, its routes answer 503 with a retry hint that names the service, over gRPC UNAVAILABLE, and the gateway invents no response (SCN-ModelGatewayHost-001, ADR-0028). |
| AC4 | Given a caller with factory.meta.read and a metadata source that answers, when ModelGatewayHost/GatewayAdapters is called, then the response is 200 with each provider adapter, its package version and its supported protocols as the source reported them; a call without a token is answered 401 and a token without the scope 403; when the source does not answer or no source is configured, the answer is 503 with a problem response that names it; nothing is written and no list is invented (SCN-ModelGatewayHost-002). |
| AC5 | Given routed calls and a call above the rate limit of its route, when the metrics and logs are read, then connectsoft.factory.model_gateway.host.requests counts every call by protocol and status, the call above the limit was refused with 429, connectsoft.factory.model_gateway.host.request_duration holds the latency by service and route, and no log record, activity or metric holds a token, a prompt, a completion or a payload (ADR-0019, model/services/model-compute/ModelGatewayHost.yaml). |
| Traces: services | ModelGatewayHost |
| Traces: apis | ModelGatewayHost/GatewayHealth, ModelGatewayHost/GatewayAdapters |
| Traces: adrs | ADR-0015, ADR-0017, ADR-0019, ADR-0028 |
| Traces: requirements | NFR-07, NFR-13, NFR-18 |
| Traces: scenarios | SCN-ModelGatewayHost-001, SCN-ModelGatewayHost-002 |
| Depends on | AIFACTORY-US-561, feature:component-api-gateway-template, feature:component-observability-factory-conventions |
| Sources | model/services/model-compute/ModelGatewayHost.yaml, docs/planes/model-compute/services/model-gateway-host/index.md#metrics-and-observability, docs/planes/model-compute/services/model-gateway-host/index.md#failure-and-recovery, docs/planes/model-compute/providers-and-profiles.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-046 Health, adapter metadata and telemetry of the gateway |
implementation | The health operation serves liveness without a token and the detail under factory.health.read with the aggregated state of the two fronted services and of the identity platform; while the identity platform is unreachable and a cached signing key still validates tokens, readiness stays ready and the operation answers 200 with the health degraded and the identity platform named in the detail from the first failed probe; when no valid key remains, readiness turns unready and the operation answers 503 with a problem response that names the identity platform (D117); liveness stays at 200 in both phases; the adapters operation serves, under factory.meta.read, the adapter list its metadata port returns and answers 503 when the port has no source or the source does not answer; every routed call records connectsoft.factory.model_gateway.host.requests and connectsoft.factory.model_gateway.host.request_duration with their modelled dimensions and carries the trace context to the service. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2272 Rate limits per route from the configuration of the gateway |
implementation | Each route takes its rate limit from the configuration of the gateway through the pinned rate-limiting library and answers 429 above it, over gRPC RESOURCE_EXHAUSTED; a refusal by the limit calls no fronted service and is counted under its status; the limit values are configuration of the deployment and none is fixed in code. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2273 Runbook of the four failure modes of the gateway |
documentation | The repository documentation holds one runbook entry per failure mode of model/services/model-compute/ModelGatewayHost.yaml (identity platform unreachable, fronted service unavailable, gateway replica crash, gateway crash mid-stream) with its detection signal, the expected answers of the gateway and the recovery step; the entry of the identity platform states its two phases, degraded and in rotation while a cached signing key validates tokens, and out of rotation and failing closed when no valid key remains (D117); the entry of the crash mid-stream states that no stream is resumed or spliced and that the caller reads the receipt (NFR-18). | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-2274 Verify health, adapter metadata, limits and telemetry through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ModelGatewayHost-001\|Scenario=SCN-ModelGatewayHost-002)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-565 with the identity platform stopped while a cached key validates tokens (readiness ready, health 200 and degraded), without a cached key and after the cached key expired (readiness unready, health 503), one fake stopped, the adapters operation with a source, without one and without its scope, a call above its limit, and a scan of the captured logs and activities for token and payload content. |
ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-US-756: The OpenAI-compatible chat completions route turns a run-bound call under the token of an attempt or a call outside a run under a user token into one model invocation of ModelRouterService¶
behavior · persona requester
As a requester, I want to call a model profile from a client that speaks the OpenAI chat completions protocol, inside a run and outside one, so that a tool inside my tenant reaches the models of the Factory under the same policy, credential and receipt rules as an agent and never holds a provider key.
An in-tenant client that speaks the OpenAI chat completions protocol reaches a model through CreateChatCompletion, POST /v1/chat/completions on gateway role GW-01, the one public operation this gateway serves itself; decision D121 delivers it in I1. The route is hosted while ModelGateway:Protocols:ExposeOpenAiCompatible is true, its modelled default; with the switch false no route is hosted and the path is answered 404 like every path without a route. The switch is not reloadable. The Anthropic-compatible switch is false by default, its route is a later feature, and nothing of it is built here (AIFACTORY-US-561). The route admits two kinds of token and no other (decision D125). The gateway validates issuer, signature, lifetime and audience as on every route (AIFACTORY-US-562); a call without a token, with an expired token, with an identity token or with a user token issued for another gateway is answered 401, a user token without factory.models.invoke, the invoke scope of the model services, and a workload token that is not the token of an attempt 403, and in these cases no fronted service is called. The route serves two kinds of call in I1 (decision D123), and the kind of token tells them apart. A run-bound call presents the token of an attempt: the short-lived token that RunControlService hands out in the attemptToken member of the lease answer, which the Authorization Server issues by a token exchange and which names the run, the step and the attempt in the claims run_id, step_id and attempt_id beside the tenant claim tid. The gateway takes the three from those claims and copies them into the invocation, requires no OpenAI-Project header and forwards the token of the attempt in place of a user token; ModelRouterService takes the project, the classification and the pinned profile version from the route plan of the run, never from a header or the body of the caller, the usage of the call counts on the reservation of the run, and its receipt is listed with the receipts of the run by ListRunReceipts of ModelRouterService. The gateway holds no state and needs no dependency on RunControlService for this, because the claims carry the run and the route plan carries the rest. A call outside a run presents a user access token whose one audience is this gateway and whose scope carries factory.models.invoke. It names its project in the OpenAI-Project request header of the protocol: a call without the header is answered 400 with the header named, and the gateway checks the project against the role claims of the validated token and answers 403 for a project the token does not grant; in both cases no fronted service is called. Its classification is the value of ModelGateway:Protocols:OpenAiCompatibleClassification, one of the classification values of model-invocation-v1; the option has no default value and is not reloadable, and a start with the route enabled and the option unset fails with the key named. The invocation of such a call names no run and no attempt, which are optional members of model-invocation-v1 since decision D123. ModelGateway.Invoke resolves the candidates for the call and persists no route plan, the usage is recorded without a run and metered on the budget of the project without a reservation, and the receipt is read back through ListProjectReceipts of ModelRouterService, GET /factory/projects/{projectId}/model-receipts, which this gateway fronts as a read route with the same check of the project (AIFACTORY-US-562). This route is the one place where the gateway translates a payload, and it translates the wire protocol only. In both kinds of call the model member of the request is a model profile id, such as code-agent, and never a provider model name: the gateway copies it into the profile of the invocation and resolves nothing, and ModelRouterService resolves it to the version pinned by the snapshot of the run for a run-bound call and to the active version of the profile otherwise. From the body the gateway builds one model-invocation-v1: a new invocation id, the profile, the trusted scope with the tenant of the validated token and, for a call outside a run, the project of the header, the hash of the canonical request, a new idempotency key for every call, because the operation declares no idempotency and a repeated HTTP call is a new invocation, for a call outside a run the classification of the option, the capabilities the body asks for (tools, structured output, streaming), the budget with the output token limit of the body, and the stream switch. The messages and the tool definitions travel in the request message of the proto of ModelRouterService and are stored nowhere by the gateway. It then calls ModelGateway.Invoke of ModelRouterService through its generated gRPC client with the token of the caller forwarded unchanged and exchanges no token (ADR-0015 amendment D117); ModelRouterService validates the token again, requires factory.models.invoke of a user token, admits the token of an attempt as a run-bound call and runs its pipeline. Everything that decides the call stays there (ADR-0028): the gateway resolves no profile, selects no deployment, asks for no policy decision, holds no provider credential, reads no budget, makes no reservation and writes no receipt. Every call of either kind produces a receipt like any other invocation, written by ModelRouterService with the deployment that answered and the usage status. The reply is translated back. Without streaming the gateway collects the messages of the server stream and answers 200 with one chat completion whose id is the invocation id of the receipt, whose model is the profile id the caller sent and whose usage holds the token counts of the receipt when the provider reported them and is absent, never zero, when the usage is Unknown. With stream true it answers 200 with an event stream, relays every message of the router as one chunk in the order received, buffers no whole output, propagates a cancellation of the caller to the router and ends with the terminal marker of the protocol after the message that carries the receipt. A refusal is translated into the error body of the protocol with the correlation id, and since decision D123 the operation lists every code the translation answers: 200, 400, 403, 404, 429, 503 and 504; 401 is the answer of every route of this gateway. A body the gateway cannot translate is answered 400 with the member named and calls no fronted service: a member the translation does not carry is refused and never dropped silently, because the protocols row of docs/planes/model-compute/model-gateway.md promises no complete wire compatibility. INVALID_ARGUMENT of the router becomes 400 and PERMISSION_DENIED, the denial of a policy decision among them, 403. NOT_FOUND becomes 404, the answer when no profile answers to the model name. A rate or budget refusal is answered 429: a call above the rate limit of the route is refused by the gateway itself (AIFACTORY-US-565) and reaches no fronted service, and RESOURCE_EXHAUSTED of the router is translated into the same code. The router answers RESOURCE_EXHAUSTED to a call outside a run of a project for which it keeps a budget mark, set from BudgetExceeded of CostUsageService, and refuses no run-bound call for that mark (decision D125); the gateway reads no budget and only translates the answer. UNAVAILABLE becomes 503 with a retry hint that names the service, and DEADLINE_EXCEEDED 504. When the stream of the router ends with an error after output was emitted, or a gateway replica stops while the stream is open, the event stream of the caller breaks; the gateway resumes and splices nothing, and ModelRouterService records the partial output in the receipt (failure mode of a gateway crash mid-stream in the model, AIFACTORY-US-564). The gateway records connectsoft.factory.model_gateway.host.requests with the protocol of the route and the returned status and connectsoft.factory.model_gateway.host.request_duration for the route, and no log record, activity or metric holds a token, a message, a completion or a tool output (ADR-0019). The route is proven against the stream fake of ModelGateway.Invoke of the acceptance suite; what ModelRouterService does with the forwarded token in either kind of call is proven by feature model-router-service.
Acceptance criteria
| AC1 | Given ModelGateway:Protocols:ExposeOpenAiCompatible true, a user token whose audience is this gateway with factory.models.invoke that carries no run, an OpenAI-Project header that names a project the role claims of the token grant, and a body that names a model profile id as model, when ModelGatewayHost/CreateChatCompletion is called without streaming, then the fake of ModelRouterService receives exactly one call of ModelGateway.Invoke whose model-invocation-v1 holds a new invocation id, the profile id as profile, the tenant of the token, the project of the header, the request hash, a new idempotency key, the classification of ModelGateway:Protocols:OpenAiCompatibleClassification and no run and no attempt, and whose authorization metadata carries the user token unchanged, without a token exchange; the answer is 200 with one chat completion whose id is the invocation id, whose model is the profile id and whose usage holds the token counts of the receipt, or no usage when the receipt states Unknown; the gateway wrote nothing (SCN-ModelGatewayHost-008, ADR-0015). |
| AC2 | Given the token of an attempt whose claims run_id, step_id and attempt_id name a run, a step and an attempt, as the attemptToken member of the lease answer of docs/reference/schemas/attempt-outcome.schema.json carries it, when ModelGatewayHost/CreateChatCompletion is called, with or without an OpenAI-Project header, then the model-invocation-v1 the fake receives carries that run, step and attempt from the claims, takes its project from no header and its classification not from ModelGateway:Protocols:OpenAiCompatibleClassification, and the token of the attempt is forwarded unchanged in place of a user token; given stream true in either kind of call, then the answer is 200 with an event stream in which every message of the fake reaches the caller as one chunk in the order sent and which ends with the terminal marker of the protocol after the message that carries the receipt; a cancellation of the caller reaches the fake; an error status of the fake after output was emitted, and a gateway replica stopped while the stream is open, break the stream of the caller, and nothing is resumed or spliced (SCN-ModelGatewayHost-008, model/services/model-compute/ModelGatewayHost.yaml). |
| AC3 | Given the token matrix of the acceptance suite, when ModelGatewayHost/CreateChatCompletion is called, then a call without a token, with an expired token, with an identity token or with a user token issued for another gateway is answered 401, a user token without factory.models.invoke and a workload token that is not the token of an attempt 403, a call outside a run without the OpenAI-Project header 400 with the header named, a call outside a run whose header names a project the role claims of the token do not grant 403, and a body the gateway cannot translate 400 with the member named, and in none of these cases the fake is called; given ModelGateway:Protocols:ExposeOpenAiCompatible false, then no route is hosted and the path is answered 404; given the route enabled and ModelGateway:Protocols:OpenAiCompatibleClassification unset, then the start fails with the key named (SCN-ModelGatewayHost-008, ADR-0015). |
| AC4 | Given a fake of ModelRouterService that ends ModelGateway.Invoke with INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND, RESOURCE_EXHAUSTED, DEADLINE_EXCEEDED or UNAVAILABLE, or that does not answer, when ModelGatewayHost/CreateChatCompletion is called, then the answer is 400, 403, 404, 429, 504 or 503 respectively in the error body of the protocol with the correlation id, each a code model/services/model-compute/ModelGatewayHost.yaml lists for the operation, the 503 carrying a retry hint that names the service and never its address; the gateway invents no completion; given the fake answers RESOURCE_EXHAUSTED for a call outside a run of a project under a budget mark and serves a run-bound call of the same project, then the first is answered 429 and the second 200; a call above the rate limit of the route is refused with 429 and reaches no fronted service (SCN-ModelGatewayHost-008, ADR-0028). |
| AC5 | Given calls of the route of both kinds that completed, were refused and broke mid-stream, when the metrics, logs and stores are read, then connectsoft.factory.model_gateway.host.requests counts each call with the protocol of the route and the returned status, connectsoft.factory.model_gateway.host.request_duration holds the latency of the route, no log record, activity or metric holds a token, a message, a completion or a tool output, and the gateway holds no profile, no deployment, no provider credential, no budget, no reservation, no idempotency key and no receipt of any call (ADR-0019, ADR-0028, model/services/model-compute/ModelGatewayHost.yaml). |
| Traces: services | ModelGatewayHost, ModelRouterService |
| Traces: apis | ModelGatewayHost/CreateChatCompletion |
| Traces: descriptors | model-invocation-v1 |
| Traces: adrs | ADR-0015, ADR-0019, ADR-0028 |
| Traces: requirements | FR-04, XL-04 |
| Traces: scenarios | SCN-ModelGatewayHost-008 |
| Depends on | AIFACTORY-US-561, AIFACTORY-US-562, AIFACTORY-US-564, AIFACTORY-US-565, feature:component-api-gateway-template, feature:contracts-package |
| Sources | model/services/model-compute/ModelGatewayHost.yaml, model/services/model-compute/ModelRouterService.yaml, docs/reference/schemas/model-invocation.schema.json, docs/reference/schemas/attempt-outcome.schema.json, docs/planes/model-compute/model-gateway.md, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-3051 Translation contract of the OpenAI-compatible chat completions route |
contract | The repository documentation holds one table that maps the members of a chat completions request the route carries (model as the model profile id, messages, stream, tools with the tool choice, the output token limit and the response format) and the OpenAI-Project request header onto the fields of model-invocation-v1 of docs/reference/schemas/model-invocation.schema.json and onto the request message of ModelGateway.Invoke, and names the fields the gateway assigns itself for each kind of call (D123): the invocation id, the idempotency key, the request hash and the tenant of the token in both; for a run-bound call under the token of an attempt the run, the step and the attempt from its claims run_id, step_id and attempt_id, no OpenAI-Project header, and no project and no classification, which ModelRouterService takes from the route plan of the run (D125); for a call outside a run under a user token the project of the OpenAI-Project header, checked against the role claims of the token, the classification from ModelGateway:Protocols:OpenAiCompatibleClassification and no run and no attempt; it states that every other member is refused with 400 and the member named, fixes the shape of the completion, of a stream chunk and of the error body, and maps every gRPC status of ModelGateway.Invoke onto its HTTP code: INVALID_ARGUMENT 400, UNAUTHENTICATED 401, PERMISSION_DENIED 403, NOT_FOUND 404, RESOURCE_EXHAUSTED 429, UNAVAILABLE 503 and DEADLINE_EXCEEDED 504; golden files of a request of each kind, a completion, a chunk sequence and each error body are checked in the pipeline, and the OpenAPI document of AIFACTORY-TASK-2263 lists the route with its scope, the OpenAI-Project header and the codes 200, 400, 403, 404, 429, 503 and 504 of model/services/model-compute/ModelGatewayHost.yaml. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-3052 Stream scripts and tokens of the acceptance suite for the chat completions route |
fixture | The stream fake of ModelGateway.Invoke of AIFACTORY-TASK-2262 gains scripts for a complete stream that ends with a receipt of reported usage, one that ends with a receipt of usage Unknown, one per error status of the contract of AIFACTORY-TASK-3051 before any output, RESOURCE_EXHAUSTED among them, one that ends with an error after output and one that stays open; the fake records the invocation and the authorization metadata it received; the token matrix gains a user token with factory.models.invoke that carries no run and whose role claims grant one project and not another, the token of an attempt with the claims run_id, step_id, attempt_id and tid as the Authorization Server issues it by token exchange, a workload token that is not the token of an attempt, and a user token without the scope; a script answers RESOURCE_EXHAUSTED for a call outside a run of a marked project and serves a run-bound call of the same project. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-3053 CreateChatCompletion behind the protocol switch with the translation to ModelGateway.Invoke and back |
implementation | The route POST /v1/chat/completions is hosted only while ModelGateway:Protocols:ExposeOpenAiCompatible is true, admits the token of an attempt or a user token of the audience of the gateway with factory.models.invoke, refuses a workload token that is not the token of an attempt with 403, and tells the two kinds of call apart by the kind of token (D123, D125): for the token of an attempt it copies the run, the step and the attempt of the claims run_id, step_id and attempt_id into the invocation, requires no OpenAI-Project header, forwards that token in place of a user token and takes neither the project from a header nor the classification from the option; for a user token it requires the OpenAI-Project header (400 with the header named), checks its project against the role claims of the token (403), states the classification of ModelGateway:Protocols:OpenAiCompatibleClassification and names no run and no attempt; it refuses a body the contract of AIFACTORY-TASK-3051 does not carry with 400 and the member named, builds the model-invocation-v1 with the model name as the model profile id, the tenant of the token, a new invocation id and idempotency key and the request hash, calls ModelGateway.Invoke through the generated gRPC client with the token forwarded unchanged, answers one completion or relays the messages as an event stream that ends with the terminal marker of the protocol, propagates the cancellation of the caller, translates every refusal by the mapping of the contract and resumes or splices no broken stream; a start with the route enabled and ModelGateway:Protocols:OpenAiCompatibleClassification unset fails with the key named; the route is rate limited like every route and records the two metrics of the gateway with its protocol and the returned status; nothing of a call is written to the shared cache or to any store. | ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |
AIFACTORY-TASK-3054 Verify the OpenAI-compatible chat completions route through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ModelGatewayHost-008"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-756 with the scripts of AIFACTORY-TASK-3052: a call outside a run that completes without streaming with reported and with Unknown usage, the invocation with the project of the header, the configured classification and no run and no attempt and the unchanged user token received by the fake, the same call without the OpenAI-Project header (400) and with a project the role claims do not grant (403), neither reaching the fake, a run-bound call under the token of an attempt whose invocation carries the run, the step and the attempt of its claims, with and without the header, a call outside a run of a marked project answered 429 beside a run-bound call of the same project answered 200, a workload token that is not the token of an attempt (403), an event stream in order with its terminal marker, a cancellation of the caller, an error after output and a replica stopped during an open stream, the token matrix with and without factory.models.invoke and with a workload token, a body with a member the translation does not carry, the switch false, the start without the classification key, each refusal of the fake mapped to its code, RESOURCE_EXHAUSTED to 429 among them, a call above the rate limit, the golden files of the contract and a scan of the captured logs, activities and metrics for token and message content. |
ConnectSoft.AI.SoftwareFactory.ModelGatewayHost |