Agent Mesh — Observability¶
Target
This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.
Trace trees¶
A run is one trace anchored on run.id. The runtime contributes an attempt span per attempt with child spans per phase and per call. Model calls carry the OpenTelemetry GenAI attributes (gen_ai.system, gen_ai.request.model as the resolved deployment, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons) with the Factory's model.profile, receipt.id and policy.decision.id; tool calls carry tool.id, tool.side_effect_class, receipt.id and the fencing token. Content is never on spans; the artifact id of a prompt or response is, so an authorized reviewer can open it under the tenant's retention. Workflow runs add a workflow span with step child spans so that a run's wall-clock time is attributable to steps, waits and decisions.
Metrics¶
| Metric | Why it matters |
|---|---|
connectsoft.factory.runtime.attempts_active by state |
backlog and waiting attempts |
connectsoft.factory.runtime.attempt_duration by outcome |
the dominant latency of a run |
connectsoft.factory.runtime.calls by kind and status |
model and tool reliability |
connectsoft.factory.runtime.corrections |
validation quality; feeds the evaluation gates (D32) |
connectsoft.factory.runtime.uncertain_outcomes |
the number an operator must look at |
connectsoft.factory.runtime.release_load_rejected |
publication or adapter compatibility problems |
connectsoft.factory.workflows.step_latency by step kind |
where runs wait |
connectsoft.factory.workflows.timers_pending, compensations |
engine health |
connectsoft.factory.runtime.host.attempt_slots, inbox_lag, control_plane_rtt |
capacity and connectivity |
Logs¶
Structured, keyed on the envelope ids, with phase and checkpoint sequence; no prompt, response, payload or artifact content (D31). Retention follows traces (30 days).
Health¶
/runtime/health reports store, silo membership, Control plane reachability, model gateway and tool executor reachability and free attempt slots; readiness fails when the inbox lag exceeds its threshold so that a replica stops receiving dispatches it cannot process.
Objectives (pilot class, D30)¶
| Objective | Target |
|---|---|
Claim latency (dispatch to AttemptStarted) |
p95 under 5 s |
| Checkpoint write | p95 under 500 ms |
| Lease renewal success | above 99.9 % per hour |
| Uncertain outcomes | reviewed within the operator SLO; counted, never hidden |
| Worker availability | 99.5 % monthly with at least two replicas |
Dashboards and alerts¶
Dashboards: attempt throughput and outcomes, waiting attempts by reason (tool, decision, pause), corrections per agent, release loads and rejections, workflow step latency and compensations, host capacity. Alerts: uncertain outcomes above zero per tenant, inbox lag, lease renewal failures, release load rejections, stuck workflow runs, attempt slot exhaustion.
Evaluation feed¶
AttemptOutcomeReported with corrections, call counts and receipts, together with Run Control's authoritative outcome, feed the evaluation and learning services (BC-22) that compute the quality gates per profile and agent release.