Control Plane — Observability¶
Target
This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.
Correlation¶
Every command and every event of the Control Plane carries the ADR-0014 envelope: tenant, project, run, step, attempt, correlation id, causation id, producer and schema version. The run id is the trace anchor: a Studio run view, an audit export and an Observability trace tree all start from it. The Control Plane emits spans per command (factory.command) and per transition (factory.transition) with tenant.id, project.id, run.id, step.id, attempt.id, module and route attributes. Model and tool calls are not made here; their spans follow the OpenTelemetry GenAI conventions in the Model & Compute and Tools & MCP platforms and join the same trace.
Metrics¶
| Metric | Instrument | Why it matters |
|---|---|---|
connectsoft.factory.host.request_duration |
histogram | the 2 s p95 target for interactive commands (D30) |
connectsoft.factory.host.outbox_backlog |
gauge | transitions committed but not yet dispatched; the first sign of messaging trouble |
connectsoft.factory.runs.* (accepted, active, terminal by class) |
counter, updown | throughput and failure classes per tenant |
connectsoft.factory.approvals.* (pending, decision latency, expired) |
updown, histogram | human gate health; expired approvals are a product problem, not a user problem |
connectsoft.factory.releases.published, releases.blocked |
counter | publication flow and gate failures by reason |
connectsoft.factory.usage.cost_usd, usage.uncertain_records, usage.reservations_denied |
counter, updown | spend, reconciliation debt, budget pressure |
connectsoft.factory.projects.active, projects.delegations_active |
updown | scope growth and delegation hygiene |
connectsoft.factory.environments.frozen |
updown | change freezes in force |
connectsoft.factory.commercial.subscriptions, commercial.invoiced_usd |
updown, counter | commercial state |
connectsoft.factory.a2a.tasks_received |
counter | partner traffic |
The full list with dimensions is on each service page and in the generated metrics catalog.
Logs¶
Structured logs carry the envelope ids and the module. The Control Plane never logs prompt text, model responses, artifact content, payloads of submissions or decision reasons in full (D31); it logs hashes and ids. Log retention follows traces (30 days); audit evidence lives in BC-19 for 7 years.
Health¶
Liveness, readiness and startup probes are described on the deployment page; the HealthChecks aggregator exposes the host and its dependencies (SQL, blob, cache, Authorization Server, transport) to the Studio status page.
Service-level objectives (pilot class, D30)¶
| Objective | Target | Measured by |
|---|---|---|
| Interactive command acknowledgement | p95 under 2 s | host.request_duration on submit, decide, publish, freeze |
| Control plane availability | 99.5 % monthly | readiness over gateway health checks |
| Event dispatch latency | p95 under 5 s from commit to transport | outbox backlog age |
| Approval surfacing | under 10 s from request to Studio inbox | ApprovalRequested to BFF read |
| Recovery | RPO 15 minutes, RTO 4 hours | restore drills in the restore order |
Dashboards and alerts¶
| Dashboard | Panels |
|---|---|
| Run health | accepted, running, waiting, terminal by class; lease expiries; uncertain attempts |
| Approvals | pending by age, decision latency, expired, stale; approvers' load |
| Publication | candidates by state, gate failures by reason, evaluation latency |
| Spend | reserved versus observed versus reconciled per tenant and project, denied reservations, uncertain records |
| Platform | request latency by module, outbox backlog, grain activations, SQL and cache health |
Alerts fire on outbox backlog age, uncertain attempts, approval expiry rate, denied reservations spike, reconciliation debt age and readiness failures. Alert payloads carry ids only.
Evidence for tenants¶
A tenant can export, per project, the run journal, approval decisions, usage records and the audit trail in open formats (D31); the export is itself an audited action and is produced by the Observability platform from the events this plane committed.