Tools & MCP — Observability¶
Target
This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.
Spans and receipts¶
Each invocation is a span joined to the run trace (run.id, attempt.id) with child spans per pipeline stage and the attributes tool.id, tool.version, tool.kind, tool.side_effect_class (declared and effective), policy.decision.id, approval.id, sandbox.kind, receipt.id and invocation.state. Arguments and results appear only as hashes; the receipt artifact holds the sanitized record. MCP calls add mcp.server.id, mcp.transport and the protocol version.
Metrics¶
| Metric | Why it matters |
|---|---|
connectsoft.factory.tools.invocations by effective class and state |
what agents do and how often it is denied, uncertain or failed |
connectsoft.factory.tools.latency by tool and kind |
pipeline overhead versus execution time |
connectsoft.factory.tools.denials by stage |
policy and schema problems by cause |
connectsoft.factory.tools.uncertain |
the operator queue |
connectsoft.factory.tools.approvals_waiting |
gated calls waiting on humans |
connectsoft.factory.tools.sandboxes_open, host.sandbox_capacity |
capacity pressure and leaks |
connectsoft.factory.tools.mcp_servers, mcp_handshakes, mcp_unbound_tools |
registry health |
Logs¶
Structured, keyed on invocation, run and attempt ids; command lines of shell tools are logged as on the receipt; no arguments or results content (D31).
Health¶
/tools/health reports the store, policy engine, secret broker, sandbox capacity and the handshake state of Active MCP servers; readiness fails when no sandbox capacity is available for a mandatory kind.
Objectives (pilot class, D30)¶
| Objective | Target |
|---|---|
| Pipeline overhead (excluding execution and approval wait) | p95 under 300 ms |
| Receipt written | for 100 % of invocations past validation |
| Uncertain effects | reviewed within the operator SLO; never auto-retried |
| Sandbox lease | p95 under 5 s for warm pools |
| MCP handshake detection | within 5 minutes |
Dashboards and alerts¶
Dashboards: invocations by class and outcome, denials by stage, approvals waiting, uncertain queue, sandbox capacity and leases, MCP server health and unbound tools. Alerts: uncertain above zero per tenant, denial spikes, sandbox capacity exhausted, leases older than their timeout, MCP server degraded, unbound-tool growth after a schema change.