Skip to content

AIFACTORY-FEAT-071: SloService: complete delivery

Generated from the architecture model

This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.

A design artefact, not a tracker

This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).

Epic Observability (PF-13) · Increment I4 Trust and accountability · State elaborated · Route backend-developer · Build order 134 of 185 (Factory code)

SloService is the service of bounded context BC-21 (Operations). SLO definitions per tenant, workflow, agent and platform (success rate, latency, cost per task, availability, human intervention), their measurements and error budgets, and the service health rollup from every host's probes; breaches feed alert rules and the fleet (BC-21, D30). Its own repository is ConnectSoft.AI.SoftwareFactory.SloService.

Delivers SloService (complete)
Stubs —
Realizes adrs: ADR-0018
Requirements FR-10, FUX-04, NFR-06, NFR-07, NFR-17, NFR-18, XL-11, XL-16
Depends on —
Sources model/services/observability/SloService.yaml

Scenarios to trace (7, catalog): SCN-SloService-001 ✓, SCN-SloService-002 ✓, SCN-SloService-003 ✓, SCN-SloService-004 ✓, SCN-SloService-005 ✓, SCN-SloService-006 ✓, SCN-SloService-007 ✓

AIFACTORY-US-044: A breached objective raises one alert that I can acknowledge

behavior · persona operator As an operator, I want objectives for the Factory and an alert when one is breached, so that I learn of a problem from the Factory and not from a user.

This is the integration PBI of the feature and the end-to-end proof that a breach reaches an operator. SloService is the service of bounded context BC-21 (Operations) that turns the NFR baseline of ADR-0018 (D30) and the operations metrics into objectives with targets, windows, burn thresholds and error budgets, for the platform, a tenant, a project, a workflow, an agent or a generated product, and keeps the health rollup of every host. It owns SloDefinition with its SloMeasurement rows and ServiceHealth in slo-sql. Callers reach its three REST operations through the route GW-03 of the gateway ObservabilityHost, which forwards the user token (D117), under factory.operations.read and factory.operations.admin; every query runs under the tenant row filter (ADR-0015). The behaviour is delivered by the behaviour PBIs of this feature: objectives are set and listed (AIFACTORY-US-1129), evaluated into breaches and recoveries (AIFACTORY-US-1130), the health of every host is rolled up (AIFACTORY-US-1131) and the retention and deletion orders are applied (AIFACTORY-US-1132). This PBI proves the chain with feature alert-rule-service: an objective defined through SetSlo publishes SloDefined and its measurement starts, as ListSlos shows; the Workspace admin sets an active tenant rule of source kind Slo through SetAlertRule, since Slo is not among the platform rules that are always on; a breach publishes SloBreached, which AlertRuleService turns into exactly one alert under that rule however often the breach is observed; the operator acknowledges the alert through AcknowledgeAlert and it stays acknowledged; and the recovery publishes SloRecovered, which resolves the alert. Of the nine consumed events of the model, IncidentResolved of FleetIncidentService and ProductOnboarded and ProductObjectiveDefined of FleetInventoryService come from producers of I7; this feature registers no handler for them, and which feature registers them is reported to the owner. Telemetry follows the conventions of feature component-observability-factory-conventions, taken by the version pin of its packages and never forked. Repository: ConnectSoft.AI.SoftwareFactory.SloService, created by the repository checklist of feature repository-and-pipeline-conventions from the base template with the Factory profile.

Acceptance criteria

AC1 Given an objective for the availability of a project, when it is defined through SloService/SetSlo, then SloService/SloDefined is published and the SloService/SloDefinition is measured from the next evaluation on, as SloService/ListSlos shows with its first measurement (model/services/observability/SloService.yaml).
AC2 Given an active tenant rule of source kind Slo set through AlertRuleService/SetAlertRule and the measurement breaches the objective, when the evaluator runs, then SloService/SloBreached is published and AlertRuleService fires one AlertRuleService/AlertInstance with AlertRuleService/AlertFired, however often the breach is observed in later evaluations (ADR-0018).
AC3 Given the fired alert, when the operator acknowledges it through AlertRuleService/AcknowledgeAlert, then it stays Acknowledged in AlertRuleService/ListAlerts until it resolves, and a further breach of the same objective fires no second alert (FR-10).
AC4 Given the measurement recovers, when the evaluator runs, then SloService/SloRecovered is published and the alert is resolved with AlertRuleService/AlertResolved (FUX-04).
AC5 Given the repository ConnectSoft.AI.SoftwareFactory.SloService, whose first commit is the unchanged output of dotnet new connectsoft-base -n ConnectSoft.AI.SoftwareFactory.SloService --UseFactoryProfile true from the pinned template package, when its pipeline runs with the work of this feature, then no second scaffold was run, the solution builds, the tests pass, a container image is published, each of the seven scenarios of SloService has a passing test that names its id, and the telemetry conformance test passes with the four metrics of model/services/observability/SloService.yaml (ADR-0028).
Traces: services SloService, AlertRuleService, ObservabilityHost
Traces: aggregates SloService/SloDefinition, AlertRuleService/AlertRule, AlertRuleService/AlertInstance
Traces: apis SloService/ListSlos, SloService/SetSlo, AlertRuleService/ListAlerts, AlertRuleService/AcknowledgeAlert, AlertRuleService/SetAlertRule
Traces: events SloService/SloDefined, SloService/SloBreached, SloService/SloRecovered, AlertRuleService/AlertFired, AlertRuleService/AlertResolved
Traces: adrs ADR-0015, ADR-0018, ADR-0028
Traces: requirements FR-10, FUX-04, XL-11, XL-16
Relates: screens status-objectives, alerts
Relates: nfrs NFR-06, NFR-07, NFR-17, NFR-18
Depends on AIFACTORY-US-1129, AIFACTORY-US-1130, AIFACTORY-US-1131, AIFACTORY-US-1132, feature:alert-rule-service, feature:observability-host, feature:repository-and-pipeline-conventions, feature:component-base-template-factory-profile, feature:contracts-package, feature:component-observability-factory-conventions
Sources model/services/observability/SloService.yaml, model/services/observability/AlertRuleService.yaml, model/scenarios/SloService.yaml, docs/planes/trust-operations/observability/services/slo-service/index.md#purpose-and-responsibilities, docs/architecture/adr/0018-nfr-baseline.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-774 Verify slo-breach-raises-alert by running the declared command verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario~SCN-SloService" in the pipeline of ConnectSoft.AI.SoftwareFactory.SloService and the end-to-end suite of AC1 to AC4 against SloService and AlertRuleService on one messaging transport; a filter that discovers no test counts as a failure; they pass and prove AC1 to AC5 of AIFACTORY-US-044, with a test for each of SCN-SloService-001 to SCN-SloService-007. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-500 Create the repository ConnectSoft.AI.SoftwareFactory.SloService and scaffold it from the ConnectSoft.BaseTemplate template with the Factory profile implementation The repository exists by the repository checklist of feature repository-and-pipeline-conventions, the output of dotnet new connectsoft-base -n ConnectSoft.AI.SoftwareFactory.SloService --UseFactoryProfile true from the pinned template package is committed unchanged with the messaging service name slo, the repository records the template version, the solution builds and the template tests pass in the repository pipeline, which publishes the container image. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-501 Domain model: SloDefinition, ServiceHealth implementation The aggregates hold the fields and invariants of the model: a platform objective cannot be loosened per tenant, a measurement is computed and never entered, error budget consumption is monotonic within the window, and health is observed and never declared; the transitions of SloState that docs/planes/trust-operations/observability/services/slo-service/index.md draws are unit-tested one by one. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-502 Persistence in slo-sql: SloDefinitions, SloMeasurements, ServiceHealth implementation Mappings and migrations for the three modelled tables with the tenant filter and the unique index of an objective per tenant, subject kind, subject and kind; mapping tests run against an isolated test database, never a shared one. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-503 API hosting skeleton: the 3 REST operations with scope enforcement and problem responses implementation ListSlos, SetSlo and ListServiceHealth are hosted with JWT validation of the token the gateway forwards, scope enforcement per route, problem responses for every modelled refusal code and the OpenAPI document; the behaviour of each operation is delivered by AIFACTORY-US-1129 and AIFACTORY-US-1131, not here. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-504 Messaging skeleton: outbox and inbox of the slo profile for 4 published and 6 consumed events implementation The outbox writer enlists in the transaction of each change and the dispatcher publishes SloDefined, SloBreached, SloRecovered and ServiceHealthChanged under the canonical envelope; the inbox registers OperationsDigestProduced, WorkerPoolStateChanged, ObservedStateChanged and the three orders of RetentionResidencyService with idempotency on the event id; IncidentResolved, ProductOnboarded and ProductObjectiveDefined are not registered; what each handler does is delivered by the behaviour PBIs of this feature. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-505 Worker hosting skeleton: triggers, partitioning and retry policies of SloEvaluator and HealthCollector implementation The two workers are hosted with their modelled schedules (every five minutes, every minute), retry policies, partitioning (by tenant, by host) and idempotency keys (sloId plus the five-minute evaluation slot, read from the modelled sloId plus window, and hostId plus minute), and a duplicate trigger has no second effect; what each worker does is delivered by AIFACTORY-US-1130 and AIFACTORY-US-1131. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-506 Security, tenancy and telemetry of SloService implementation The two scopes and four roles of the model are enforced, every query is tenant-filtered, the messaging transport secret comes from a secret reference, the host makes the one call of the registration of the pinned observability packages, creates the four modelled metrics through its instrument helper, among them the gauge connectsoft.factory.slo.inbox_lag, the age in seconds of the oldest unprocessed inbox message of the service, measured by the inbox of the messaging profile (D140), and references the conformance test package from its architecture tests. ConnectSoft.AI.SoftwareFactory.SloService

AIFACTORY-US-1129: Define or tighten an objective for a subject and list the objectives in scope with their state and error budget

behavior · persona operator As an operator, I want to set objectives for my tenant, projects, workflows and agents, never looser than the platform baseline, and see each with its current state and budget, so that the Factory is held to targets I chose and I see at once which objective is at risk.

An Operator or a Platform admin whose scope carries factory.operations.admin defines or changes an objective of the trusted tenant through SloService/SetSlo, a PUT on /factory/slos/{subjectKind}/{subjectId}/{kind}, where the subject kind is platform, tenant, project, workflow, agent or product and the kind is one of ObjectiveKind. The body carries the target, the window, a rolling 30 days or the calendar month, the fast and slow burn thresholds and the source, a metric name or the projection. The objective is unique per tenant, subject kind, subject and kind, so a PUT never reaches an objective of another tenant. A new objective is written active, a known one is changed, and either writes SloDefined with sloId, tenantId, subjectKind, subjectId, kind and target in the same transaction and answers 200; the measurement starts with the next evaluation. Platform objectives are the NFR baseline of Observability:Slo:PlatformBaseline (D30) and a tenant objective may only be stricter: a target looser than the baseline of its kind, such as an availability below 0.995 or an interactive latency above 2 seconds at p95, is refused with 409 and a problem response that names the baseline value, and nothing is written. The operation is naturally idempotent: a repeat of the same body changes nothing and publishes nothing. A caller without the scope is refused with 403, and so is a caller with the scope but without the Platform admin role who addresses the subject kind platform, because the platform objectives are the baseline of Observability:Slo:PlatformBaseline that belongs to the Platform admin. SloService has no lookup of projects, workflows, agents or products, and SetSlo lists only 200, 403 and 409, so it cannot tell an unknown subject from a subject of another tenant. Until the owner decides how the subject is verified and whether it is answered 404, a subject id that names nothing of the trusted tenant is written as an objective of the trusted tenant: every measurement reads only data of the trusted tenant, so such an objective reads and changes nothing of another tenant and gets no measurement while no data of that subject exists in the tenant, and the subject kind tenant accepts only the id of the trusted tenant as its subject, refused with 403 otherwise. A caller whose scope carries factory.operations.read lists the objectives of the trusted tenant through SloService/ListSlos, each with its target and window and with the state, the value, the remaining error budget and the burn rate of its latest measurement, read under the tenant row filter; nothing is written. StudioBff and AuditService consume SloDefined in their own PBIs.

Acceptance criteria

AC1 Given a caller with factory.operations.admin and an availability target of 0.999 for a project of the trusted tenant, when SloService/SetSlo is called, then the response is 200, one active SloService/SloDefinition exists for the tenant, subject and kind with the target, window, burn thresholds and source of the body, and the SloService/SloDefined outbox record is written in the same transaction; a repeat of the same body changes nothing and writes no record, and a tighter target changes the row and writes a second record (SCN-SloService-002).
AC2 Given an availability target of 0.99, below the platform baseline of 0.995, or an interactive latency target above 2 seconds, when SloService/SetSlo is called, then the response is 409 with a problem response that names the baseline value and nothing is written or published; a caller without factory.operations.admin is refused with 403 and no state changes (SCN-SloService-002, ADR-0018).
AC3 Given objectives of two tenants with measurements, when SloService/ListSlos is called with factory.operations.read, then the response is 200 with the objectives of the trusted tenant only, each with target, window and the state, value, remaining error budget and burn rate of its latest measurement, and nothing is written; a caller without the scope is refused with 403 (SCN-SloService-001, ADR-0015).
AC4 Given an objective was defined or changed, when the outbox dispatcher publishes SloService/SloDefined on operations.slos.v1.slo-defined, then the event is sent under the canonical envelope with sloId, tenantId, subjectKind, subjectId, kind and target, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-SloService-004, ADR-0012).
AC5 Given an Operator with factory.operations.admin but without the Platform admin role, when SloService/SetSlo addresses the subject kind platform, or the subject kind tenant with the id of another tenant, then the response is 403 with a problem response and nothing is written or published, while a Platform admin is accepted on the subject kind platform; given a project id that names no project of the trusted tenant, then the objective is written under the trusted tenant only, no row of another tenant is read or changed, and no measurement is written for it while the tenant holds no data of that subject (SCN-SloService-002, ADR-0015).
Traces: services SloService
Traces: aggregates SloService/SloDefinition
Traces: apis SloService/SetSlo, SloService/ListSlos
Traces: events SloService/SloDefined
Traces: adrs ADR-0012, ADR-0015, ADR-0018
Traces: requirements FR-10, FUX-04
Traces: scenarios SCN-SloService-001, SCN-SloService-002, SCN-SloService-004
Relates: screens status-objectives
Relates: nfrs NFR-06
Sources model/services/observability/SloService.yaml, docs/planes/trust-operations/observability/services/slo-service/index.md#apis

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-4501 SetSlo with the tightening-only rule and ListSlos with the latest measurement implementation SetSlo writes or changes the objective of the trusted tenant addressed by the path with its SloDefined outbox record in one transaction, refuses a target looser than the platform baseline of its kind with 409 and a problem response that names the baseline value, changes and publishes nothing on a repeat, answers 403 without the scope, 403 for the subject kind platform without the Platform admin role and 403 for the subject kind tenant with an id other than the trusted tenant, and writes an objective for an unverified project, workflow, agent or product id under the trusted tenant only; ListSlos returns the objectives of the tenant with the state, value, remaining budget and burn rate of the latest measurement. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-4502 Verify objectives-set-and-listed through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-SloService-001\|Scenario=SCN-SloService-002\|Scenario=SCN-SloService-004)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-1129 over a two-tenant fixture, including the 409 below the baseline, the 403 on the subject kind platform for an Operator, the objective for a foreign project id that reads nothing of the other tenant, the repeat that publishes nothing and the payload of SloDefined; consumer handling is proven by the suites of StudioBff and AuditService. ConnectSoft.AI.SoftwareFactory.SloService

AIFACTORY-US-1130: Evaluate every active objective, keep its error budget and announce a breach, a fast burn and a recovery once

behavior · persona operator As an operator, I want every objective measured on a schedule with its error budget and burn rate, so that a breach or a fast burn is announced once, early enough to act, and its recovery closes it.

The SloEvaluator worker runs every five minutes, partitioned by tenant with at most two in parallel, and evaluates every active objective: an objective sourced from the projection takes its value from the overview digests of DashboardService, which the handler EvaluateSlos follows through OperationsDigestProduced, and an objective sourced from a metric queries the telemetry backend (ext-otel-backend, sync HTTP, required, degrade). Each evaluation writes one SloMeasurement with measuredAt, the value, the remaining error budget, the burn rate and the state; a measurement is computed and never entered, and error budget consumption is monotonic within the window. The state follows the transitions of docs/planes/trust-operations/observability/services/slo-service/index.md: Meeting to BurningFast when the burn rate exceeds the fast threshold, BurningFast to Breached when the budget is exhausted, Meeting to Breached when the value misses the target at the end of the window, Breached to Recovered when the value meets the target again, BurningFast back to Meeting when the burn rate is normal, Recovered to Meeting with the next window, and Meeting to Suspended and back when the source is unavailable and returns. Entering BurningFast or Breached writes SloBreached with sloId, tenantId, subjectKind, subjectId, kind, value and state in the transaction of the measurement; leaving them for Recovered or Meeting writes SloRecovered with sloId, tenantId and kind; an evaluation that keeps the state publishes nothing. The worker is idempotent on the modelled key sloId + window, and the window of that key is read as the five-minute evaluation slot of the schedule, not the objective window of 30 days: one measurement is written per objective per slot, a duplicate trigger in the same slot writes nothing, and the next slot writes the next measurement; the reading is reported to the owner. When the telemetry backend fails, the objectives sourced from metrics that are in Meeting go Suspended with a note and those sourced from the projection continue, which is the modelled failure mode; an objective sourced from a metric that is in BurningFast or Breached keeps its state, gets no measurement and publishes nothing while the backend fails, because the drawn transitions lead into Suspended from Meeting only, which is reported to the owner. A failed pass is attempted three times in all with the modelled fixed backoff of 60 seconds. The digest carries only tenantId, window and computedAt, and the model gives SloService no read of the overviews of DashboardService, so where the projection-sourced values are read is reported to the owner and this PBI reads them through a projection source port that the acceptance suite binds to a fake. connectsoft.factory.slo.state and connectsoft.factory.slo.error_budget_remaining follow every measurement. AlertRuleService and StudioBff consume SloBreached and SloRecovered in their own PBIs, and FleetInventoryService and FleetRolloutService in I7; DashboardService declares the handler ProjectSloDefinition for SloBreached, which feature dashboard-service does not register while the owner decides the projections the model does not hold.

Acceptance criteria

AC1 Given an active objective sourced from the projection whose burn rate rises above the fast threshold after DashboardService/OperationsDigestProduced, when the SloEvaluator runs, then a SloMeasurement in state BurningFast is written for the SloService/SloDefinition with value, remaining budget and burn rate, and the SloService/SloBreached outbox record with state BurningFast is written in the same transaction; a run in the next five-minute slot that keeps the state writes its own measurement and publishes nothing, and a duplicate trigger in the same slot writes no second measurement and publishes nothing (SCN-SloService-005).
AC2 Given that objective exhausts its budget and later meets its target again, when the SloEvaluator runs, then it moves to Breached with a second SloService/SloBreached whose state is Breached, then to Recovered with SloService/SloRecovered, and to Meeting with the next window; the remaining budget never grows within a window (SCN-SloService-006, ADR-0018).
AC3 Given the fake telemetry backend fails, when the SloEvaluator runs, then every objective sourced from a metric that is in Meeting goes Suspended with a note and no breach or recovery is published for it, an objective sourced from a metric that is in BurningFast keeps its state without a measurement, the objectives sourced from the projection are evaluated as before, and once the backend answers the suspended objectives return to Meeting (model/services/observability/SloService.yaml).
AC4 Given a breach or a recovery, when the outbox dispatcher publishes SloService/SloBreached on operations.slos.v1.slo-breached or SloService/SloRecovered on operations.slos.v1.slo-recovered, then each is sent under the canonical envelope with its payload, sloId, tenantId, subjectKind, subjectId, kind, value and state for the breach and sloId, tenantId and kind for the recovery, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-SloService-005, SCN-SloService-006, ADR-0012).
Traces: services SloService, DashboardService
Traces: aggregates SloService/SloDefinition
Traces: events SloService/SloBreached, SloService/SloRecovered, DashboardService/OperationsDigestProduced
Traces: adrs ADR-0012, ADR-0018
Traces: requirements FR-10, XL-11
Traces: scenarios SCN-SloService-005, SCN-SloService-006
Relates: screens status-objectives
Relates: nfrs NFR-06, NFR-07
Depends on AIFACTORY-US-1129, feature:dashboard-service
Sources model/services/observability/SloService.yaml, docs/planes/trust-operations/observability/services/slo-service/index.md#sagas-and-long-running-flows, docs/architecture/adr/0018-nfr-baseline.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-546 Objectives, measurement and error budget implementation The SloEvaluator evaluates every active objective every five minutes per tenant from a projection source port and from a typed client of the telemetry backend, writes one SloMeasurement per evaluation with value, remaining budget, burn rate and state, applies the drawn SloState transitions, writes SloBreached on entering BurningFast or Breached and SloRecovered on leaving them in the transaction of the measurement, suspends metric-sourced objectives in Meeting while the backend fails and leaves those in BurningFast or Breached unmeasured, is idempotent on sloId plus the five-minute evaluation slot so that a duplicate trigger in the slot writes nothing, attempts a failed pass three times in all and sets the two objective gauges. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-4503 EvaluateSlos inbox handler on the overview digest implementation The handler of OperationsDigestProduced marks the projection-sourced objectives of the tenant and window of the digest for the next evaluation, idempotent on the event id, so that a digest leads to one evaluation of each affected objective. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-4504 Fake projection source and fake telemetry backend for the evaluation suite fixture A fake projection source answers configured values per tenant, subject and window, a fake telemetry backend answers configured series and can fail, and the fixture drives an objective through Meeting, BurningFast, Breached, Recovered, Meeting and Suspended. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-4505 Verify objectives-evaluated-into-breaches-and-recoveries through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-SloService-005\|Scenario=SCN-SloService-006)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1130 against the fakes of AIFACTORY-TASK-4504, including the evaluation that keeps its state and publishes nothing, the duplicate trigger in the same slot that writes nothing, the monotonic budget, the BurningFast objective left unmeasured during the outage, the suspension and the payloads of SloBreached and SloRecovered; consumer handling is proven by the suites of the consumers. ConnectSoft.AI.SoftwareFactory.SloService

AIFACTORY-US-1131: Roll up the health of every host from its probes, list it for the status page and announce every change

behavior · persona operator As an operator, I want the health of every host from its own probes, with its dependencies and ready replicas, so that the status page shows what is really up, and a host that stops reporting is never shown as healthy.

The HealthCollector worker runs every minute, partitioned by host with at most eight in parallel, and collects the liveness, readiness and dependency probes of every host from the HealthChecks aggregator and the hosts themselves through the engine package of feature component-healthchecks-aggregator-engine-library, taken at the version its B-F06 page records and never forked. It rolls them up into the ServiceHealth of the host: Healthy when every probe passes, Degraded when the host is live but not ready or a dependency is degraded, Unhealthy when liveness fails, and Unknown when no probe of the host is fresher than Observability:Slo:HealthStalenessSeconds, 180 seconds by default, because health is observed and never declared. It keeps the state per dependency, replicasReady and observedAt, and when the state changes it writes ServiceHealthChanged with hostId, tenantId, fromState and toState in the same transaction; an unchanged rollup publishes nothing, and the worker is idempotent on the host and the minute and attempts a failed collection twice in all with the modelled fixed backoff of 15 seconds. Shared hosts are kept under the platform tenant. The handler UpdateServiceHealth, idempotent on the event id, keeps one ServiceHealth row per pool and per environment under the tenant of the event, because the model names no host for them: a pool is the host pool:{poolId} and an environment the host environment:{environmentId}. For a pool it maps toState of WorkerPoolStateChanged of ExecutionCapacityService: Ready to Healthy, Provisioning, Scaling and Draining to Degraded, and Unavailable to Unhealthy, with currentSize as replicasReady. For an environment it maps ObservedStateChanged of RuntimeEnvironmentService: stale true to Unknown; otherwise Healthy when every workload in changes has as many ready replicas as desired, Degraded when one has fewer but more than none, and Unhealthy when one has none of a desired number above zero. The observedAt of the row is the occurredAt of the event. A change of the mapped state writes ServiceHealthChanged with hostId, tenantId, fromState and toState in the same transaction, and an unchanged state publishes nothing. This keeps the invariant that health is observed and never declared, since the pool state and the observed state are themselves observations of their services, not declarations; the mapping is not in the model and is reported to the owner. A caller whose scope carries factory.operations.read lists the rollup through SloService/ListServiceHealth for the status page of the Studio: the hosts of the trusted tenant and the shared hosts, each with state, dependencies, ready replicas and observedAt; a caller without the scope is refused with 403 and nothing is written. connectsoft.factory.health.hosts counts the hosts by state. StudioBff and AlertRuleService consume ServiceHealthChanged in their own PBIs; DashboardService declares the handler ProjectServiceHealth, which feature dashboard-service does not register while the owner decides the projections the model does not hold.

Acceptance criteria

AC1 Given a host whose probes all pass and then whose readiness fails while it stays live, when the HealthCollector runs, then the SloService/ServiceHealth of the host moves from Healthy to Degraded with its dependencies, ready replicas and observedAt, and the SloService/ServiceHealthChanged outbox record with fromState Healthy and toState Degraded is written in the same transaction; a run with an unchanged rollup publishes nothing (SCN-SloService-007).
AC2 Given a host whose last probe is older than 180 seconds, when the HealthCollector runs, then its state is Unknown and never Healthy, and a change into Unknown publishes SloService/ServiceHealthChanged; given a host whose liveness fails, then its state is Unhealthy (model/services/observability/SloService.yaml).
AC3 Given ExecutionCapacityService/WorkerPoolStateChanged of pool p1 from Ready to Draining, when UpdateServiceHealth handles it, then the SloService/ServiceHealth of host pool:p1 under the tenant of the event moves from Healthy to Degraded with currentSize as replicasReady and SloService/ServiceHealthChanged with fromState Healthy and toState Degraded is written, and a change to Unavailable moves it to Unhealthy; given RuntimeEnvironmentService/ObservedStateChanged of environment e1 with stale true, then host environment:e1 is Unknown, and with a workload of no ready replica out of two desired it is Unhealthy; a redelivered event id and an unchanged mapped state change and publish nothing (model/services/runtime-cloud/ExecutionCapacityService.yaml).
AC4 Given hosts of two tenants and shared hosts of the platform tenant, when SloService/ListServiceHealth is called with factory.operations.read, then the response is 200 with the hosts of the trusted tenant and the shared hosts, each with state, dependencies, ready replicas and observedAt, and nothing is written; a caller without the scope is refused with 403 (SCN-SloService-003, ADR-0015).
AC5 Given a rollup changed, when the outbox dispatcher publishes SloService/ServiceHealthChanged on operations.health.v1.service-health-changed, then the event is sent under the canonical envelope with hostId, tenantId, fromState and toState, and a transport outage is retried with backoff so the event is neither lost nor duplicated under its event id (SCN-SloService-007, ADR-0012).
Traces: services SloService, ExecutionCapacityService, RuntimeEnvironmentService
Traces: aggregates SloService/ServiceHealth
Traces: apis SloService/ListServiceHealth
Traces: events SloService/ServiceHealthChanged, ExecutionCapacityService/WorkerPoolStateChanged, RuntimeEnvironmentService/ObservedStateChanged
Traces: adrs ADR-0012, ADR-0015, ADR-0018
Traces: requirements FUX-04, XL-16
Traces: scenarios SCN-SloService-003, SCN-SloService-007
Relates: screens status-objectives
Relates: nfrs NFR-17
Depends on feature:component-healthchecks-aggregator-engine-library, feature:execution-capacity-service-complete, feature:runtime-environment-service
Sources model/services/observability/SloService.yaml, model/services/runtime-cloud/ExecutionCapacityService.yaml, docs/planes/trust-operations/observability/services/slo-service/index.md#workers-and-jobs

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-548 Service health from the probes of every host implementation The HealthCollector polls the targets of every host through the pinned HealthChecks aggregator engine package every minute, rolls liveness, readiness and dependency probes up into Healthy, Degraded, Unhealthy or Unknown with the staleness of Observability:Slo:HealthStalenessSeconds, keeps dependencies, ready replicas and observedAt, writes ServiceHealthChanged on a change in the same transaction, is idempotent on hostId plus minute and moves connectsoft.factory.health.hosts. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-4506 UpdateServiceHealth handlers and ListServiceHealth implementation The handler of WorkerPoolStateChanged and ObservedStateChanged writes the ServiceHealth of host pool:{poolId} or environment:{environmentId} under the tenant of the event, maps Ready to Healthy, Provisioning, Scaling and Draining to Degraded and Unavailable to Unhealthy for a pool, and stale to Unknown and the ready against desired replicas of the changed workloads to Healthy, Degraded or Unhealthy for an environment, writes ServiceHealthChanged with fromState and toState on a change of the mapped state in the same transaction and nothing otherwise, and is idempotent on the event id; ListServiceHealth returns the hosts of the trusted tenant and the shared hosts of the platform tenant with state, dependencies, ready replicas and observedAt, and answers 403 without the scope. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-4507 Verify service-health-rolled-up-per-host through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-SloService-003\|Scenario=SCN-SloService-007)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-1131 with probe targets served by test hosts that can fail readiness, liveness and stop answering, including the stale host shown Unknown, the unchanged rollup that publishes nothing, the pool and environment states mapped onto the hosts pool:{poolId} and environment:{environmentId} with the redelivered event that changes nothing, and the payload of ServiceHealthChanged. ConnectSoft.AI.SoftwareFactory.SloService

AIFACTORY-US-1132: Apply retention rules, legal holds and deletion orders to the objectives, measurements and health of a tenant

behavior · persona workspace-admin As a workspace admin, I want the objectives and measurements of my tenant kept no longer than my retention rule and deleted on order, so that the operations data of a deleted project or tenant does not stay behind in the objective store.

Three inbox handlers apply the orders of RetentionResidencyService to slo-sql, each idempotent on the event id. ApplyRetentionRule handles RetentionRuleChanged and applies the effective retention period of the rule to the records of its store class; the model keeps measurements 13 months, a tenant rule may only shorten that, and the retention job of the service deletes the measurements older than the effective period. ApplyLegalHold handles LegalHoldChanged and suspends deletion and expiry for the held scope while the hold is active. ExecuteDeletionOrder handles DeletionOrderIssued: an order of tenant scope deletes every row of the tenant in SloDefinitions, SloMeasurements and ServiceHealth; an order of project scope deletes the objectives whose subject kind is project and whose subject is that project, with their measurements; a run purge finds no rows, because the store holds no run, and its receipt says so; a held scope is not deleted and the receipt names the hold. Platform objectives and shared hosts under the platform tenant are never deleted by a tenant order. The handler acknowledges its task through RetentionResidencyService/AcknowledgeOrderTask with the workload token of the service, which holds factory.data.acknowledge, and a receipt that names the order, the service, the tables and the row counts; a redelivered order deletes nothing more and acknowledges with the same receipt, and an answer 403 or 404 or a failure before the acknowledgement leaves the task unacknowledged, is logged with the order id and is retried by the inbox.

Acceptance criteria

AC1 Given RetentionResidencyService/DeletionOrderIssued of project scope, when ExecuteDeletionOrder handles it, then the SloService/SloDefinition rows whose subject is that project are deleted with their measurements, objectives of other subjects and of other tenants are untouched, and the task is acknowledged through RetentionResidencyService/AcknowledgeOrderTask with a receipt that names the tables and the row counts (ADR-0019, XL-14).
AC2 Given an order of tenant scope and a run purge, when they are handled, then every row of the tenant in the three tables is deleted while the platform objectives and the shared hosts stay, and the run purge deletes nothing and says so in its receipt; a redelivered order deletes nothing more, and an acknowledgement answered 403 or 404 leaves the task unacknowledged for the inbox to retry (model/services/data-event/RetentionResidencyService.yaml).
AC3 Given RetentionResidencyService/LegalHoldChanged that holds a tenant and RetentionResidencyService/RetentionRuleChanged that shortens the retention of the store class of slo-sql, when the retention job runs, then no measurement of the held tenant is deleted, and after the hold is lifted the measurements older than the effective period are deleted (ADR-0019).
Traces: services SloService, RetentionResidencyService
Traces: aggregates SloService/SloDefinition, SloService/ServiceHealth
Traces: apis RetentionResidencyService/AcknowledgeOrderTask
Traces: events RetentionResidencyService/RetentionRuleChanged, RetentionResidencyService/LegalHoldChanged, RetentionResidencyService/DeletionOrderIssued
Traces: adrs ADR-0019
Traces: requirements XL-14
Depends on AIFACTORY-US-1129, feature:retention-residency-service
Sources model/services/observability/SloService.yaml, model/services/data-event/RetentionResidencyService.yaml, docs/architecture/adr/0019-data-governance.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-4508 ApplyRetentionRule, ApplyLegalHold and ExecuteDeletionOrder over the three objective tables implementation The three handlers apply the effective retention period through a retention job of the service, suspend and resume expiry and deletion for a held scope, delete the rows of a tenant order and the project objectives with their measurements of a project order without touching platform objectives or shared hosts, answer a run purge with a receipt that names no rows, acknowledge through a typed client of AcknowledgeOrderTask with the workload token that holds factory.data.acknowledge, and are idempotent on the order id. ConnectSoft.AI.SoftwareFactory.SloService
AIFACTORY-TASK-4509 Verify objectives-follow-retention-and-deletion-orders through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Feature=InboxHandlers"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1132 over a two-tenant fixture with a fake of AcknowledgeOrderTask that records the receipt and can answer 403 and 404, including the redelivered order and the hold that blocks expiry. ConnectSoft.AI.SoftwareFactory.SloService