Skip to content

Observability — overview

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Rewritten 2026-10-09 (design plan P8)

This page replaces the June-2026 overview with its telemetry, trace, log, metric and dashboard services. The 2026 design keeps traces, logs and raw metrics in OpenTelemetry backends and makes the Factory own what a backend cannot know — attribution, projections, objectives, alerts, evaluation evidence and the learning loop — as five modules of one host. The old page is archived under catalogs 2026-06.

Observability is where the Factory proves what it did, measures whether it was good, and learns without anything learning in place. Every run leaves a timeline; every release carries evaluation evidence; every objective has an error budget; and every improvement is a proposal a human reviews.
Platform PF-13 Observability (Trust & Operations)
Bounded contexts BC-21, BC-22
Services 15 (1 hosts, 8 modules, 4 external, 2 later) — 6 modelled
Gateways —
Status target

Purpose

  • Projections, never truth. The run timeline (original handoff §51) joins Run Control's journal with model, tool, context and usage receipts; it is rebuilt from them whenever it disagrees (D30 restore order puts this store last).
  • Objectives with budgets. The pilot NFR numbers (D30) are the platform baseline that tenants may only tighten; breaches are events.
  • One place for operator attention. Alerts from objectives, health, uncertain tool effects, findings and metric conditions, deduplicated and routed through the Studio and connectors.
  • Evaluation is a core capability (§47–49, D32). Suites per target, verdicts against thresholds and a baseline, waivers only with an approval of class publication, drift detection, immutable evidence read at the Published gate (ADR-0021).
  • Governed learning (§32). Feedback, datasets of historical tasks and learning proposals — accepted proposals become releases, nothing else changes an asset.
  • OpenTelemetry GenAI conventions. Spans, metrics and evaluation attributes follow the GenAI semantic conventions so any backend can read them (technology radar, D44).

Bounded contexts and services

Context Owns Services (granularity)
BC-21 Operations Observability Run timelines, overviews, objectives, health, alerts DashboardService (module; agent telemetry row merged) · SloService (module; runtime health row merged) · AlertRuleService (module)
BC-22 Evaluation, Feedback & Learning Suites, evaluation runs, verdicts, evidence, drift, feedback, datasets, proposals QualityScoreService (module; knowledge quality row merged) · FeedbackService (module)
Host Composition, inbox, backend connection ObservabilityHost (standalone)
External canonical Trace, log and metric storage and query OpenTelemetry backend (Azure Monitor default; Jaeger, Seq on-prem) — LogQuery, MetricAggregation, TelemetryCorrelation, Trace rows
Later Incidents, template analytics IncidentService, TemplateAnalyticsService
Service Context Granularity Status
AgentTelemetryService BC-21 module of host unmodelled
AlertRuleService BC-21 module of host target
DashboardService BC-21 module of host target
FeedbackService BC-22 module of host target
IncidentService BC-21 later stage unmodelled
KnowledgeQualityService BC-22 module of host unmodelled
LogQueryService BC-21 external / canonical unmodelled
MetricAggregationService BC-21 external / canonical unmodelled
ObservabilityHost BC-21 standalone host target
QualityScoreService BC-22 module of host target
RuntimeHealthService BC-21 module of host unmodelled
SloService BC-21 module of host target
TelemetryCorrelationService BC-21 external / canonical unmodelled
TemplateAnalyticsService BC-21 later stage unmodelled
TraceService BC-21 external / canonical unmodelled

Context

flowchart LR
    subgraph P["Observability"]
        ObservabilityHost["ObservabilityHost"]
    end
    P --> ConnectorRegistryService["ConnectorRegistryService"]
    P --> StudioNotificationService["StudioNotificationService"]
    P --> ext_otel_backend["ext-otel-backend"]
    P --> RunControlService["RunControlService"]
    P --> ArtifactService["ArtifactService"]
    P --> AgentRegistryService["AgentRegistryService"]
    P --> ModelRouterService["ModelRouterService"]
    P --> MemoryPromotionService["MemoryPromotionService"]
    P --> ControlPlaneApi["ControlPlaneApi"]
    P --> ModelGatewayHost["ModelGatewayHost"]
    P --> GovernanceHost["GovernanceHost"]
    P --> KnowledgeHost["KnowledgeHost"]
    P --> ext_identity_platform["ext-identity-platform"]
Hold "Alt" / "Option" to enable pan & zoom

Containers

See microservices for the container view of hosts and modules.

What a run leaves behind

Signal Where it lives Who owns it
Spans, logs, raw metrics with GenAI attributes (gen_ai.*, connectsoft.factory.*) OpenTelemetry backend the installation's backend
Run journal, approvals, usage states Control plane stores Run Control, Approval, Cost & Usage
Model, tool, context receipts Their producers' stores and artifacts Model router, Tool executor, Context builder
Timeline, overview, objectives, alerts observability-sql this platform (rebuildable)
Evaluation evidence, dataset snapshots, proposals observability-sql + artifact-blob this platform (immutable evidence)

Metrics the platform computes (§51)

Task success rate · agent success rate · cost per task · latency per task · tokens per task · tool failure rate · model fallback rate · human intervention rate · sandbox failure rate · context retrieval quality · evaluation score — per tenant, project, agent, workflow, tool and model, per window.

Metric Service Instrument Dimensions
connectsoft.factory.alerts.fired AlertRuleService counter tenant, sourceKind, severity
connectsoft.factory.alerts.open AlertRuleService updown tenant, severity
connectsoft.factory.alerts.time_to_acknowledge AlertRuleService histogram tenant, severity
connectsoft.factory.operations.projection_lag DashboardService gauge tenant
connectsoft.factory.operations.task_success_rate DashboardService gauge tenant, window
connectsoft.factory.operations.human_intervention_rate DashboardService gauge tenant, window
connectsoft.factory.operations.cost_per_task_usd DashboardService gauge tenant, window
connectsoft.factory.learning.feedback FeedbackService counter tenant, kind
connectsoft.factory.learning.proposals FeedbackService updown tenant, kind, state
connectsoft.factory.learning.datasets FeedbackService updown tenant, target
connectsoft.factory.observability.host.inbox_lag ObservabilityHost gauge module
connectsoft.factory.observability.host.projection_lag ObservabilityHost gauge projection
connectsoft.factory.evaluation.runs QualityScoreService counter tenant, target, verdict
connectsoft.factory.evaluation.duration QualityScoreService histogram target
connectsoft.factory.evaluation.cost_usd QualityScoreService counter tenant, target
connectsoft.factory.evaluation.drift QualityScoreService counter tenant, metric
connectsoft.factory.evaluation.waivers QualityScoreService counter tenant
connectsoft.factory.slo.state SloService gauge tenant, subjectKind, kind, state
connectsoft.factory.slo.error_budget_remaining SloService gauge tenant, subjectKind, kind
connectsoft.factory.health.hosts SloService updown state