AIFACTORY-FEAT-066: ObservabilityHost: gateway delivery¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Observability (PF-13) · Increment I3 Studio · State elaborated · Route backend-developer · Build order 111 of 185 (Factory code)
ObservabilityHost is the platform API gateway of bounded context BC-21 (Operations). The API gateway of the Observability platform (ADR-0028): routes public REST and internal gRPC calls to the 5 services it fronts, validates tokens and scopes, and owns no business state. Its own repository is ConnectSoft.AI.SoftwareFactory.ObservabilityHost.
| Delivers | ObservabilityHost (complete) |
| Stubs | — |
| Realizes | adrs: ADR-0012, ADR-0014, ADR-0016, ADR-0018, ADR-0021 |
| Requirements | FR-10, FUX-04, FUX-10, NFR-10, NFR-11, NFR-17, XL-08, XL-16 |
| Depends on | — |
| Sources | model/services/observability/ObservabilityHost.yaml |
Scenarios to trace (6, catalog): SCN-ObservabilityHost-001 ✓, SCN-ObservabilityHost-002 ✓, SCN-ObservabilityHost-003 ✓, SCN-ObservabilityHost-004 ✓, SCN-ObservabilityHost-005 ✓, SCN-ObservabilityHost-006 ✓
AIFACTORY-US-084: ObservabilityHost routes the calls of its platform under a trusted scope¶
enabling · persona operator
As an operator, I want one gateway in front of the five services of the Observability platform that validates tokens and scopes, so that no service of the platform is reachable from outside the cluster except through its gateway and every call arrives with a scope the service can trust.
This is the integration PBI of the gateway of the Observability platform. ADR-0028 makes ObservabilityHost the API gateway of its platform: it routes public REST and internal gRPC calls to the five services the model lists under deployment.fronts, AlertRuleService, DashboardService, FeedbackService, QualityScoreService and SloService, validates tokens and scopes and owns no business state, no relational store and no secret. The repository ConnectSoft.AI.SoftwareFactory.ObservabilityHost is created by the first task of this feature with the steps of the repository checklist of feature repository-and-pipeline-conventions and is scaffolded once from the pinned API gateway template package with UseFactoryProfile and the gateway role Platform (feature component-api-gateway-template), which fixes persistence, messaging and the actor model to none. The gateway has no outbox and no inbox, consumes and publishes no event (D129), and its only store is the shared cache; the model of the gateway lists no event and no consumed event, so nothing is to be corrected there. The scope of a call is established once from the validated token with the scope accessor of the pinned ConnectSoft.Extensions.AI.Factory.Abstractions package (feature component-extensions-ai, feature trusted-scope-chain). A user token has one audience, this gateway, and is forwarded to the fronted service unchanged with no token exchange (ADR-0015 amendments D114 and D117); the tenant comes from the tid claim and a project header a caller sends is dropped. A call without a token or with an expired token is answered 401, a missing scope 403, and no fronted service is called; a resource of another tenant is answered 404 by the fronted service (ADR-0015 amendment D113) and the gateway returns that answer unchanged. The feature is delivered by five further PBIs: the fronts file and the routes generated from it (AIFACTORY-US-1104), the read routes (AIFACTORY-US-1105), the shared-write routes (AIFACTORY-US-1106), the sandbox and publication routes of QualityScoreService (AIFACTORY-US-1107), and health, backends, limits and telemetry with the three failure modes of the gateway (AIFACTORY-US-1108). The gateway is delivered in I3 together with StudioBff, its first caller. In this increment no fronted service hosts a public route yet: QualityScoreService is delivered thin in I1 and hosts no operation before its complete delivery in I4, and the other four services arrive in I4. The fronts file holds the routes of all five services from the start, and they begin to serve as the services appear in I4 without a change of the gateway. One rule decides the answer of a route whose operation is not served: a route of a service that is deployed but does not host the operation returns the not-found answer of that service unchanged, so in I3 the routes of the thin QualityScoreService return its unknown-route 404 (AIFACTORY-US-1089); only a service that is not deployed or does not answer gives 503 with a retry hint that names the service. Every route class is proven against fakes generated from the fronts file. The gateway invents no response. The model of the gateway declares the ports silo 12100 and gateway 31100, which belong to an actor host, while the Platform gateway role fixes the actor model to none; the deployment leaves them out, and their removal from model/services/observability/ObservabilityHost.yaml is a correction reported with this feature.
Acceptance criteria
| AC1 | Given the repository ConnectSoft.AI.SoftwareFactory.ObservabilityHost created by the steps of the repository checklist, whose first commit is the unchanged output of dotnet new connectsoft-apigateway -n ConnectSoft.AI.SoftwareFactory.ObservabilityHost --UseFactoryProfile true --GatewayRole Platform, when its pipeline runs with the work of this feature, then the solution builds, the tests pass and a container image is published (ADR-0028, model/services/observability/ObservabilityHost.yaml). |
| AC2 | Given a valid token for a route of each fronted service, when the call passes the gateway, then the scope is established once from the token, the call is forwarded with the token unchanged, the tenant, the correlation id and the trace context, and the response of the fronted service returns unchanged, as AIFACTORY-US-1105 to AIFACTORY-US-1107 prove per route class (ADR-0015). |
| AC3 | Given the generated gateway, when its architecture test runs, then the test fails on a persistence model, an aggregate, a migration or a message consumer, the gateway binds no relational store, no outbox and no inbox, its configuration holds no secret, and no event of the model lists ObservabilityHost as a publisher or a consumer (ADR-0028, model/services/observability/ObservabilityHost.yaml). |
| AC4 | Given the gateway deployed with two replicas, when ObservabilityHost/ObservabilityHealth is probed and one replica is stopped, then health reports the identity platform and the five fronted services, the other replica keeps serving and nothing is replayed, because the gateway holds no state (ADR-0016, model/services/observability/ObservabilityHost.yaml). |
| Traces: services | ObservabilityHost, AlertRuleService, DashboardService, FeedbackService, QualityScoreService, SloService |
| Traces: apis | ObservabilityHost/ObservabilityHealth |
| Traces: adrs | ADR-0015, ADR-0016, ADR-0018, ADR-0028 |
| Traces: requirements | FR-10, FUX-04, FUX-10, NFR-10, NFR-17 |
| Depends on | AIFACTORY-US-1104, AIFACTORY-US-1105, AIFACTORY-US-1106, AIFACTORY-US-1107, AIFACTORY-US-1108, feature:repository-and-pipeline-conventions, feature:component-api-gateway-template, feature:component-extensions-ai, feature:component-observability-factory-conventions, feature:trusted-scope-chain, feature:quality-score-service-thin |
| Sources | model/services/observability/ObservabilityHost.yaml, model/gateways.yaml, docs/architecture/catalogs/gateways.md, docs/planes/trust-operations/observability/services/observability-host/index.md#purpose-and-responsibilities |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-469 Create the repository ConnectSoft.AI.SoftwareFactory.ObservabilityHost and scaffold it from the API gateway template with the Factory profile and the Platform gateway role |
implementation | The repository exists, created by the steps of the repository checklist of feature repository-and-pipeline-conventions; its first commit is the unchanged output of dotnet new connectsoft-apigateway -n ConnectSoft.AI.SoftwareFactory.ObservabilityHost --UseFactoryProfile true --GatewayRole Platform from the pinned template package, whose version the repository records; the solution builds and the template tests, the architecture test of the Platform role included, pass in the repository pipeline, which publishes the container image. |
ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4395 Deployment values of the gateway: replica range, ports, probes and the address of each fronted service |
operations | The deployment manifest states the replica range 2 to 8, the ports https 8100 and grpc 8102 and leaves out the silo port 12100 and the gateway port 31100 that the model also declares, because the Platform gateway role has no actor model, a correction of the model reported with this feature; it states the liveness and readiness probes on the health operation and one base address per fronted service from configuration, as model/services/observability/ObservabilityHost.yaml records them; it sets ConnectSoftIdentityPlatform:Authority to the authority of the profile, holds no secret and binds the shared cache as its only store; the same image runs in the saas-default, hybrid, on-prem and hybrid-compute topologies. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-771 Verify observability-host-routes-its-platform by running the declared command |
verification | Run dotnet build -c Release and dotnet test -c Release --no-build in the pipeline of the repository; a run that discovers no test counts as a failure; the runs pass and prove AC1 to AC4 of AIFACTORY-US-084, AC4 through the acceptance test of feature TwoReplicas (dotnet test -c Release --filter "Category=Acceptance&Feature=TwoReplicas"), which hosts two instances behind one test balancer with the fakes of the fronted services, probes ObservabilityHost/ObservabilityHealth on both, stops one and asserts that the other keeps serving and nothing is replayed. |
ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-US-1104: Observability fronts file: named clients and REST and gRPC routes of the five fronted services, generated at build¶
enabling · persona platform-admin
As a platform admin, I want the route table of the observability gateway generated from the list of services it fronts, so that every operation of the platform has exactly one route with its scope and class and no forwarding code is written by hand.
The API gateway template is a custom gateway with controllers and named clients, not a reverse proxy, so a route exists only when it is declared. Under UseFactoryProfile a build-time generator reads a fronts file and emits one named HTTP client and one gRPC client per fronted service, the REST and gRPC routes that forward to them and the route table (feature component-api-gateway-template). This PBI writes the fronts file of the Observability platform from the apis sections of the five service files. For AlertRuleService these are ListAlerts and ListAlertRules (read) and AcknowledgeAlert, ResolveAlert and SetAlertRule (shared-write). For DashboardService these are ListRunProjections, GetRunTimeline, GetOverview and GetSubjectOverview (read), RebuildRunProjection (shared-write) and the gRPC server stream Operations.Subscribe. For FeedbackService these are ListFeedback, ListDatasets and ListLearningProposals (read), RecordFeedback, SnapshotDataset and DecideLearningProposal (shared-write) and the gRPC method Learning.GetDatasetSnapshot. For QualityScoreService these are ListEvaluationSuites, GetEvaluation and ListReleaseEvaluations (read), StartEvaluation (sandbox), WaiveEvaluation (publication) and the gRPC methods Evaluation.GetVerdict and Evaluation.RunSuite. For SloService these are ListSlos and ListServiceHealth (read) and SetSlo (shared-write). That is twenty-four REST routes and four gRPC routes; Operations.Subscribe is a server stream and the other three are unary. Each route carries the scope, side-effect class and idempotency of its operation, which the route class PBIs of this feature enforce. No operation of the five service files names a project in its path; each route is forwarded under the tenant of the token and the fronted service applies its project rule from the role claims of the forwarded token. A test lists the emitted route table against the file and against the five service files, so an operation without a route, a route without an operation or a name used twice fails the build with the entry named. A path no route matches is answered 404. The gateway adds no operation of its own beyond ObservabilityHealth and ObservabilityBackends, transforms no payload and composes no response from two services. The option Observability:Host:FrontedServices is compared at startup with the service names of the fronts file, and a difference fails startup with the name. Its default in the model, [dashboards, slos, alerts, evaluation, feedback], names short names and not the service identifiers of deployment.fronts, which the governance gateway rule of D127 requires, so a correction of the default to the five identifiers is reported with this feature, and until it is made the deployment manifest sets the option to the five identifiers. The pipeline publishes the OpenAPI document of the twenty-four public REST operations, checked against the fronts file and the service files. The typed client of this gateway belongs to this feature (D140): one library in the repository ConnectSoft.AI.SoftwareFactory.Clients, generated with dotnet new connectsoft-api-library, the generation switch and UseFactoryProfile in the user-token layout from that published document, by the rules of feature typed-gateway-clients (AIFACTORY-US-381 to AIFACTORY-US-383), which it applies and does not repeat. It holds the twenty-four public REST operations and no internal gRPC method, because the routes exist from I3 and begin to serve as the fronted services appear in I4 without a change of the gateway or of the client; until then a call returns the answer of the route, the unknown-route 404 of the thin QualityScoreService or the 503 of a service that is not deployed, as a typed error. It is checked against the model by the contract check of that feature, takes the shared types from the pinned contracts package, attaches the token issued for this gateway only to requests to the base address of this gateway, is versioned with the contracts package, published to the ConnectSoft feed and recorded on the Runbook page with the consumer that pins it. No workload layout is generated, because no public REST operation of this gateway admits a workload token. The caller of the package is StudioBff, which holds a user token and pins it in its own repository for the operations, SLO, alert, evaluation and feedback views and the search of alerts (AIFACTORY-US-1363, AIFACTORY-US-037); no caller writes a client of this gateway by hand.
Acceptance criteria
| AC1 | Given the fronts file of the Observability platform, when the gateway builds, then one named HTTP client and one gRPC client exist per service of deployment.fronts and one route per operation, twenty-four REST routes and four gRPC routes, each with the scope, side-effect class, idempotency and streaming mode of the operation, and the route table test passes against model/services/observability/ObservabilityHost.yaml and the five service files under model/services/observability (ADR-0028). |
| AC2 | Given an operation removed from the file or a name used twice, when the gateway builds, then the build fails with the entry named; given Observability:Host:FrontedServices set to the five service identifiers, when the gateway starts, then startup succeeds, and a value that names no service of the file or omits one fails startup with the name; given a path no route matches, then the answer is 404 and no fronted service is called (docs/architecture/catalogs/gateways.md). |
| AC3 | Given a fake of each fronted service, when one route per service is called with a valid token, then the call reaches the fake with the token of the caller unchanged, the tenant of the token, the correlation id and the trace context, a project header of the caller has been dropped, and the response of the fake returns unchanged; the published OpenAPI document holds exactly the twenty-four REST operations and no gRPC method (ADR-0015, docs/architecture/api-governance.md). |
| AC4 | Given the published OpenAPI document and the pinned API library template, when the client of this gateway is generated, built and published in the repository ConnectSoft.AI.SoftwareFactory.Clients, then it holds the twenty-four public REST operations in the user-token layout and no internal gRPC method, the contract check against the five service files under model/services/observability passes, the token issued for ObservabilityHost is attached only to requests to its base address, a 404 or a 503 of a route surfaces as a typed error, and the package is on the ConnectSoft feed with a version tied to the contracts package and recorded on the Runbook page with StudioBff as its consumer (model/backlog/pbis/engineering-readiness/typed-gateway-clients.yaml, ADR-0015). |
| Traces: services | ObservabilityHost, AlertRuleService, DashboardService, FeedbackService, QualityScoreService, SloService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FR-10 |
| Depends on | feature:component-api-gateway-template, feature:quality-score-service-thin, feature:typed-gateway-clients, feature:component-api-library-template-factory-clients |
| Sources | model/services/observability/ObservabilityHost.yaml, model/backlog/pbis/engineering-readiness/typed-gateway-clients.yaml, model/services/observability/AlertRuleService.yaml, model/services/observability/DashboardService.yaml, model/services/observability/FeedbackService.yaml, model/services/observability/QualityScoreService.yaml, model/services/observability/SloService.yaml, docs/architecture/catalogs/gateways.md, docs/architecture/api-governance.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-470 Fronts file with the REST and gRPC operations of AlertRuleService, DashboardService, FeedbackService, QualityScoreService and SloService |
implementation | The fronts file lists the five services of deployment.fronts with their base addresses from configuration and every REST operation and gRPC method of their service files with name, path or method, streaming mode, scope, side-effect class and idempotency; the build emits the named clients, the twenty-four REST routes, the four gRPC routes and the route table; at startup Observability:Host:FrontedServices is compared with the service names of the file and a difference fails startup with the name; a project header of a caller is dropped on every route; a call without a valid token or scope is refused by the generated policies. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4396 Fakes of the five fronted services and the token matrix of the acceptance suite |
fixture | The acceptance suite holds one fake per fronted service, generated from its fronts entry, that records the headers and metadata it received, answers a configured status and serves the server stream of Operations.Subscribe, a switch that makes a fake stop answering, also in the middle of a stream, and tokens for the matrix of role, tenant and project, including an expired token, a user token of another audience and a workload token with and without the service scope of a gRPC route. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4397 OpenAPI document of the public REST routes of the gateway as a published build output |
contract | The pipeline publishes one OpenAPI document with the twenty-four public REST operations of the fronts file, their methods, paths, scopes, idempotency headers and response codes and no gRPC method; a contract check compares it with the five service files under model/services/observability and fails the build on a difference; the document is the input of the client of AIFACTORY-TASK-6603, and no client package is built in this repository. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-6603 Typed client of ObservabilityHost in the user-token layout, generated from the published document and published to the ConnectSoft feed |
implementation | One client library in the repository ConnectSoft.AI.SoftwareFactory.Clients, generated with dotnet new connectsoft-api-library, the generation switch and UseFactoryProfile in the JWT bearer layout of the template from the OpenAPI document of AIFACTORY-TASK-4397, holds the twenty-four public REST operations of AlertRuleService, DashboardService, FeedbackService, QualityScoreService and SloService and no internal gRPC method, takes the shared types from the pinned contracts package and declares no handler, header name or claim name of its own; the contract check of AIFACTORY-TASK-1556 covers its document; a stub of the gateway in the fixture of the client solution answers 401, 403, 404, 503 and a redirect to a second host, a 404 and a 503 surface as typed errors, and the token issued for ObservabilityHost is attached only to requests to its base address; the package is versioned with the contracts package, published to the ConnectSoft feed by the generated pipeline after its tests pass and restored by the probe consumer from the feed alone; the Runbook page records its version with StudioBff as the consumer that pins it. |
ConnectSoft.AI.SoftwareFactory.Clients |
AIFACTORY-TASK-4398 Verify the route table against the fronts file and the model |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=RouteTable"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1104: one route per operation of the five service files under model/services/observability, the failing builds and startups, the 404 of an unknown path, one forwarded call per service with the dropped project header, and the contract check of the OpenAPI document; AC4 is proven in the pipeline of ConnectSoft.AI.SoftwareFactory.Clients by the build, the tests and the probe stage of AIFACTORY-TASK-6603. |
ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-US-1105: A read route of the observability gateway forwards a call only under a validated token, the tenant of the token and the scope of its operation¶
behavior · persona observer
As an observer, I want my read calls to reach the alert, dashboard, feedback, evaluation and objective services only with the tenant and the scope my token grants, so that I see the runs, alerts, objectives, evaluations and feedback I am entitled to and nobody reads across a tenant through the gateway.
A person reaches the read operations of the Observability platform through StudioBff and the gateway: run projections and timelines, overviews, alerts and alert rules, objectives and service health, evaluation suites, evaluations and the evaluations of a release, feedback, datasets and learning proposals. The gateway validates issuer, signature, lifetime and audience of the user token and requires the scope the operation names: factory.runs.read for ListRunProjections and GetRunTimeline, factory.operations.read for GetOverview, GetSubjectOverview, ListAlerts, ListAlertRules, ListSlos and ListServiceHealth, factory.evaluation.read for ListEvaluationSuites, GetEvaluation, ListReleaseEvaluations, ListDatasets and ListLearningProposals, and factory.feedback.read for ListFeedback. It takes the tenant from the tid claim, overwrites any tenant header of the caller, drops a project header of the caller, and forwards the user token unchanged with the correlation id and the trace context (ADR-0015 amendment D117). Every public REST operation names a factory. scope, so a workload token on a REST route is answered 403. The fronted service answers 403 for a missing role and 404 for a resource of another tenant (ADR-0015 amendment D113), and the gateway returns that answer unchanged. The internal gRPC surface has three read methods, Operations.Subscribe of DashboardService, a server stream that StudioRealtimeGateway holds open, Evaluation.GetVerdict of QualityScoreService, a read for internal callers that the Published gate of AgentRegistryService does not call, since the gate checks the evidence the registry recorded from EvaluationCompleted (D141), and Learning.GetDatasetSnapshot of FeedbackService. A workload token carries service scopes and no tenant claim; the gateway forwards the tenant-id and project metadata of a workload whose token holds the service scope that names the fronted service (D114), and answers PERMISSION_DENIED to a workload without it and to a user token. The stream is passed through as messages arrive, nothing is buffered or cached, and when the service fails in the middle of a stream the stream ends with UNAVAILABLE and nothing is replayed. Each read is served by the fronted service with no side effect, the gateway keeps nothing of the call and records connectsoft.factory.observability.host.requests and connectsoft.factory.observability.host.request_duration for the route. A call without a token, with an expired token or with a user token issued for another gateway is answered 401 and a token without the scope 403, and no fronted service is called. In this increment the read routes follow the rule of AIFACTORY-US-084: the routes of the thin QualityScoreService, which is deployed and hosts no operation before I4, return its unknown-route 404 unchanged (AIFACTORY-US-1089), and the routes of the four services not yet deployed answer 503 with a retry hint that names the service.
Acceptance criteria
| AC1 | Given a user token whose audience is this gateway with factory.evaluation.read, when the read route of GetEvaluation of QualityScoreService is called, then the gateway forwards the call with the user token unchanged, the tenant of the tid claim, the correlation id and the trace context, the fake of QualityScoreService answers, and the response returns unchanged with the correlation id; the 404 of the service for an evaluation of another tenant returns unchanged (SCN-ObservabilityHost-004, model/services/observability/QualityScoreService.yaml). |
| AC2 | Given the token matrix of the acceptance suite, when the read routes of the fronted services are called, then a call without a token, with an expired token or with a user token for another gateway is answered 401, a token without the scope and a workload token on a REST route are answered 403, a tenant header of the caller is overwritten and a project header dropped, and in no refusal a fronted service is called (SCN-ObservabilityHost-004, ADR-0015). |
| AC3 | Given a workload token that holds the service scope naming the fronted service, when Evaluation.GetVerdict of QualityScoreService and the server stream of Operations.Subscribe of DashboardService are called with tenant-id metadata, then the metadata is forwarded unchanged and the stream is passed through without buffering; given a workload token without the scope or a user token, then the answer is PERMISSION_DENIED and no service is called; given the fake stopping in the middle of the stream, then the stream ends with UNAVAILABLE and is not replayed (SCN-ObservabilityHost-004, model/services/observability/QualityScoreService.yaml). |
| AC4 | Given a fronted service that does not answer or is not deployed, when its read routes are called, then the answer is 503, over gRPC UNAVAILABLE, with a retry hint that names the service and never its address, no response is invented, and the read routes of the other services keep serving; given the thin QualityScoreService of I3, deployed and hosting no operation, when the read route of ListEvaluationSuites is called, then its unknown-route 404 returns unchanged and is not turned into 503 (SCN-ObservabilityHost-004, model/services/observability/ObservabilityHost.yaml). |
| Traces: services | ObservabilityHost, AlertRuleService, DashboardService, FeedbackService, QualityScoreService, SloService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | FUX-04, FUX-10, NFR-11 |
| Traces: scenarios | SCN-ObservabilityHost-004 |
| Relates: screens | evaluation-runs, release-evidence, status-objectives, alerts, feedback |
| Depends on | AIFACTORY-US-1104, feature:quality-score-service-thin, feature:trusted-scope-chain |
| Sources | model/services/observability/ObservabilityHost.yaml, docs/planes/trust-operations/observability/services/observability-host/index.md#apis |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4399 Read routes with token, tenant and scope checks and the pass-through of the server stream |
implementation | The fourteen REST read routes require the scope of their operation, take the tenant from the tid claim, drop a project header of the caller, forward the user token unchanged and refuse a workload token with 403; the three gRPC read routes forward the tenant-id and project metadata of a workload whose token holds the service scope naming the fronted service and answer PERMISSION_DENIED otherwise; Operations.Subscribe is passed through without buffering; an absent or failing service answers 503 or UNAVAILABLE with a retry hint naming it, while the not-found answer of a deployed service that does not host the operation returns unchanged. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4400 Verify the read route class through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ObservabilityHost-004"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1105 against the fakes of the five services: the forwarded read and its unchanged 404, the token matrix refusals without a call, the gRPC metadata and stream pass-through with PERMISSION_DENIED and the interrupted stream, the 503 of an absent service while the others serve, and the unknown-route 404 of the thin QualityScoreService returned unchanged. |
ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-US-1106: A shared-write route of the observability gateway passes the call and its idempotency key through unchanged¶
behavior · persona operator
As an operator, I want my acknowledgements, resolutions, rule and objective changes, rebuilds, feedback, snapshots and proposal decisions to reach their service once, so that a retry after a lost answer never applies a change twice and the gateway never decides what the change means.
The platform has eight shared-write operations: AcknowledgeAlert, ResolveAlert and SetAlertRule of AlertRuleService, RebuildRunProjection of DashboardService, RecordFeedback, SnapshotDataset and DecideLearningProposal of FeedbackService, and SetSlo of SloService. The gateway requires the scope the operation names (factory.operations.operate for acknowledging and resolving alerts, factory.operations.admin for alert rules, objectives and rebuilds, factory.feedback.write for feedback, factory.evaluation.run for snapshots and factory.assets.write for proposal decisions), establishes the tenant from the token and forwards the user token unchanged. Two of the operations, RecordFeedback and SnapshotDataset, take an Idempotency-Key header; the gateway forwards the header unchanged, keeps no record of it and is no party to idempotency: the fronted service applies the call once and answers a repeat with the same result. The other six are idempotent by their natural key, the alert, the rule name, the run, the proposal or the objective, and the gateway adds no key. A conflict answer of the service, such as 409 for a decided proposal or a conflicting objective, and its 404 for a resource of another tenant (ADR-0015 amendment D113) return unchanged. The gateway never retries a shared-write call itself: when the fronted service does not answer, the route answers 503 with a retry hint naming the service, and the caller repeats the call with the same key or the same natural target. A call without a token or with an expired token is answered 401 and a token without the scope 403, and no fronted service is called.
Acceptance criteria
| AC1 | Given a user token with factory.feedback.write and an Idempotency-Key, when the route of RecordFeedback is called twice with the same key and body, then both calls reach the fake of FeedbackService with the key and the token unchanged, the gateway stores nothing, and each answer of the fake returns unchanged (SCN-ObservabilityHost-006, ADR-0015). |
| AC2 | Given the routes of AcknowledgeAlert, SetSlo and DecideLearningProposal, when they are called with a valid token and scope, then the call is forwarded without an added key and a 409 or a 404 of the fronted service returns unchanged; given a token without the scope of the operation, then the answer is 403 and no service is called (SCN-ObservabilityHost-006, ADR-0015). |
| AC3 | Given a fronted service that does not answer, when a shared-write route is called, then the gateway answers 503 with a retry hint naming the service, does not retry the call itself and invents no response (SCN-ObservabilityHost-006, model/services/observability/ObservabilityHost.yaml). |
| Traces: services | ObservabilityHost, AlertRuleService, DashboardService, FeedbackService, SloService |
| Traces: adrs | ADR-0015, ADR-0028 |
| Traces: requirements | NFR-10 |
| Traces: scenarios | SCN-ObservabilityHost-006 |
| Relates: screens | alerts, status-objectives, feedback, datasets, learning-proposals |
| Depends on | AIFACTORY-US-1104 |
| Sources | model/services/observability/ObservabilityHost.yaml, docs/architecture/api-governance.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4401 Shared-write routes with pass-through of the idempotency key |
implementation | The eight shared-write routes require the scope of their operation, forward the user token and an Idempotency-Key header unchanged, add no key to a naturally idempotent call, store nothing of the call, never retry it and answer 503 with a retry hint naming the service when it does not answer; conflict and not-found answers of the service return unchanged. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4402 Verify the shared-write route class through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-ObservabilityHost-006"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1106: the key passed through on both calls, the unchanged 409 and 404, the 403 without a call, and the 503 without a retry by the gateway. |
ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-US-1107: The sandbox and publication routes of QualityScoreService pass through the gateway while the service keeps its gate¶
behavior · persona asset-maintainer
As an asset maintainer, I want to start evaluations and have a reviewer waive a failed verdict through the gateway, so that evaluations and waivers reach the evaluation service with my scope while the approval and the evidence stay with that service.
Three operations of QualityScoreService carry a side-effect class above shared-write. StartEvaluation (sandbox, scope factory.evaluation.run, Idempotency-Key, answers 202, 403 and 409) starts an evaluation whose tasks run as sandbox-class Factory runs in their isolated workspaces; Evaluation.RunSuite (sandbox, internal gRPC) lets an internal caller start a suite with release-manifest-v1 and receive ack-v1; WaiveEvaluation (publication, scope factory.evaluation.waive, answers 200, 403, 404, 409 and 412) records the waiver of a failed verdict by a Reviewer who is not the author, under an approval of class publication bound to the evaluation. The gateway admits each call only with the scope of the operation and forwards it with the user token unchanged, or with the tenant-id metadata of a workload whose token holds the service scope naming QualityScoreService (D114) for Evaluation.RunSuite. The gateway holds no approval reference, checks no approval and is no party to idempotency: the approval gate, the author rule and the evidence are decided by QualityScoreService, and its 412 for a missing approval, its 409 for a decided or running evaluation and its 404 for an evaluation of another tenant (ADR-0015 amendment D113) return unchanged. The gateway never retries such a call; when the service does not answer, the route answers 503 or UNAVAILABLE with a retry hint naming the service. A call without a token or with an expired token is answered 401, a token without the scope 403, a user token on the gRPC route PERMISSION_DENIED, and no service is called. Until the complete delivery of QualityScoreService in I4, the thin delivery is deployed and hosts none of the three operations, so by the rule of AIFACTORY-US-084 the routes return its unknown-route 404 (AIFACTORY-US-1089) unchanged and never 503; only when QualityScoreService is not deployed or does not answer is the answer 503 or UNAVAILABLE.
Acceptance criteria
| AC1 | Given a user token with factory.evaluation.run and an Idempotency-Key, when the route of StartEvaluation of QualityScoreService is called, then the call is forwarded with the token and the key unchanged, the 202 or the 409 of the service returns unchanged and the gateway stores nothing; given a workload token holding the service scope naming QualityScoreService, when Evaluation.RunSuite is called, then it is forwarded with its tenant metadata and the ack-v1 reply returns unchanged; a user token on that route gets PERMISSION_DENIED (SCN-ObservabilityHost-005, model/services/observability/QualityScoreService.yaml). |
| AC2 | Given a user token with factory.evaluation.waive, when the route of WaiveEvaluation of QualityScoreService is called, then the gateway forwards it without an approval reference of its own, and the 200, 412, 409 or 404 of the service returns unchanged; given a token without the scope, then the answer is 403 and no service is called (SCN-ObservabilityHost-003, ADR-0015). |
| AC3 | Given QualityScoreService not answering, when a sandbox or publication route is called, then the answer is 503 or UNAVAILABLE with a retry hint naming the service, the gateway does not retry and invents no response; given the thin QualityScoreService of I3, deployed and hosting none of the three operations, then its unknown-route 404 returns unchanged and is not turned into 503 (SCN-ObservabilityHost-003, SCN-ObservabilityHost-005, model/services/observability/ObservabilityHost.yaml). |
| Traces: services | ObservabilityHost, QualityScoreService |
| Traces: descriptors | release-manifest-v1, ack-v1 |
| Traces: adrs | ADR-0015, ADR-0021, ADR-0028 |
| Traces: requirements | FR-10 |
| Traces: scenarios | SCN-ObservabilityHost-003, SCN-ObservabilityHost-005 |
| Relates: screens | evaluation-runs, release-evidence |
| Depends on | AIFACTORY-US-1104, feature:quality-score-service-thin |
| Sources | model/services/observability/ObservabilityHost.yaml, model/services/observability/QualityScoreService.yaml, docs/architecture/quality-gates.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4403 Sandbox and publication routes of QualityScoreService without a gate in the gateway |
implementation | The routes of StartEvaluation, Evaluation.RunSuite and WaiveEvaluation require the scope of their operation, forward the user token, the Idempotency-Key and the tenant metadata of an admitted workload unchanged, hold no approval reference, never retry and answer 503 or UNAVAILABLE with a retry hint naming the service when it does not answer; the 202, 409, 412 and 404 answers of the service, the unknown-route 404 of the thin delivery included, return unchanged. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4404 Verify the sandbox and publication route classes through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ObservabilityHost-003\|Scenario=SCN-ObservabilityHost-005)"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1107 against the fake of QualityScoreService: the forwarded start with its key, the gRPC start with PERMISSION_DENIED for a user token, the waiver forwarded without an approval reference with the unchanged 412, 409 and 404, the 403, the 503 without a retry, and the unknown-route 404 of the thin QualityScoreService returned unchanged. |
ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-US-1108: Health, backends metadata, rate limits and telemetry of the observability gateway with its three failure modes¶
behavior · persona operator
As an operator, I want the gateway to report its own health and that of the services behind it, show which telemetry backends are configured, and limit the rate of calls, so that I can tell at a glance whether the platform is reachable and a noisy caller cannot starve the others.
ObservabilityHealth (GET /observability/health, scope factory.health.read, internal, gateway role GW-01) reports liveness and readiness of the gateway with the aggregated health of the five fronted services and of the identity platform. It answers 200 while the gateway can validate tokens, also with a cached signing key while the identity platform is unreachable, in which case it reports degraded, and 503 with readiness false when no valid key remains (D117). A fronted service that does not answer is reported as degraded and does not take the gateway out of rotation. ObservabilityBackends (GET /observability/meta/backends, scope factory.meta.read, internal, GW-03) returns the configured OpenTelemetry backends for traces, metrics and logs and the GenAI semantic-convention version in use, from configuration and without a call to a fronted service; it answers 200, and fails fast with a problem response when the token cannot be validated: 401 for a token the gateway cannot validate, and 503 when no valid signing key remains (D117). The model lists only 200 for this operation, so the omission of 401 and 503 from its response codes is reported with this feature. Both are read-only and keep no state. Every route belongs to a rate-limit class of ADR-0018 through the pinned rate-limiting library; a call beyond the limit is answered 429, over gRPC RESOURCE_EXHAUSTED, with a retry hint, and is counted. The gateway records connectsoft.factory.observability.host.requests by service, route and status and connectsoft.factory.observability.host.request_duration by service and route, and passes the trace context to the fronted service, following the observability conventions of the Factory. The three failure modes of the model hold: when the identity platform is unreachable the gateway fails closed and cached signing keys validate already-issued tokens until they expire; when a fronted service is unavailable its routes answer 503 with a retry hint and the other routes keep serving; when a replica crashes it restarts and nothing is replayed, because the gateway holds no state.
Acceptance criteria
| AC1 | Given a caller with factory.health.read, when ObservabilityHost/ObservabilityHealth is called with all dependencies up, then the answer is 200 with the five fronted services and the identity platform; with the identity platform unreachable and a cached key valid, then 200 degraded; with no valid key left, then 503 with readiness false; with one fronted service down, then 200 with that service degraded (SCN-ObservabilityHost-001, ADR-0016). |
| AC2 | Given a caller with factory.meta.read, when ObservabilityHost/ObservabilityBackends is called, then the answer is 200 with the configured trace, metric and log backends and the GenAI semantic-convention version and no fronted service is called; given a token the gateway cannot validate, then the call fails fast with 401 and a problem response, and given no valid signing key left, then with 503 (D117), in both cases with no state change (SCN-ObservabilityHost-002, model/services/observability/ObservabilityHost.yaml). |
| AC3 | Given a route whose rate-limit class is exceeded, when further calls arrive, then they are answered 429 or RESOURCE_EXHAUSTED with a retry hint and counted in connectsoft.factory.observability.host.requests with their status; every forwarded call carries the trace context and is measured in the request duration (ADR-0018, model/services/observability/ObservabilityHost.yaml). |
| Traces: services | ObservabilityHost |
| Traces: apis | ObservabilityHost/ObservabilityHealth, ObservabilityHost/ObservabilityBackends |
| Traces: adrs | ADR-0016, ADR-0018 |
| Traces: requirements | NFR-10, NFR-17, XL-16 |
| Traces: scenarios | SCN-ObservabilityHost-001, SCN-ObservabilityHost-002 |
| Depends on | AIFACTORY-US-1104, feature:component-observability-factory-conventions |
| Sources | model/services/observability/ObservabilityHost.yaml, docs/planes/trust-operations/observability/services/observability-host/index.md#failure-and-recovery, docs/planes/trust-operations/observability/services/observability-host/index.md#metrics-and-observability |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-471 Health, backends metadata and telemetry of the gateway |
implementation | ObservabilityHealth aggregates the identity platform and the five fronted services and answers 200, 200 degraded or 503 by the rules of D117; ObservabilityBackends returns the configured OpenTelemetry backends and the GenAI semantic-convention version from configuration; the two modelled metrics are recorded per route and the trace context is passed to the fronted service. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4405 Rate-limit classes of the routes and the alerts of the three failure modes |
operations | Each route of the fronts file is assigned its rate-limit class of ADR-0018 in configuration, with 429 and RESOURCE_EXHAUSTED carrying a retry hint; the operations configuration of the gateway holds alerts on token validation failures, the error rate per fronted service and the readiness probe, the detections the model names for its three failure modes. | ConnectSoft.AI.SoftwareFactory.ObservabilityHost |
AIFACTORY-TASK-4406 Verify health, backends, limits and telemetry through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-ObservabilityHost-001\|Scenario=SCN-ObservabilityHost-002)"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1108: the four health answers, the backends answer and its fail-fast with 401 and 503, the 429 with its count, and the trace context and duration of a forwarded call. |
ConnectSoft.AI.SoftwareFactory.ObservabilityHost |