Model & Compute — Observability¶
Target
This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.
Spans¶
Every invocation is a span following the OpenTelemetry GenAI semantic conventions (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons) with the Factory's model.profile, model.deployment, route_plan.id, receipt.id, policy.decision.id and fallback.used, joined to the run trace keyed on run.id. Embeddings and reranks carry the same shape with their modality. Placement decisions are spans on the compute side with the chosen pool and exclusion count.
Metrics¶
| Metric | Why it matters |
|---|---|
connectsoft.factory.model_gateway.invocations by outcome |
success, partial, failed, denied, cancelled per profile and provider |
connectsoft.factory.model_gateway.latency |
time to first token and completion per profile |
connectsoft.factory.model_gateway.tokens |
input and output tokens; feeds FinOps |
connectsoft.factory.model_gateway.fallbacks |
every fallback is an exception to look at |
connectsoft.factory.model_gateway.denials |
policy denials per reason |
connectsoft.factory.model_gateway.usage_unknown |
reconciliation debt |
connectsoft.factory.model_gateway.provider_errors |
adapter errors by class |
connectsoft.factory.model_gateway.host.provider_health |
provider up or down per region |
connectsoft.factory.compute_pools.utilization, placements, cold_starts |
pool health and placement outcomes |
Logs¶
Structured, keyed on envelope ids, deployment and receipt ids; no prompt or response content (D31). Retention follows traces (30 days).
Health¶
/model-gateway/health reports adapters, store, policy and usage-accounting reachability and per-provider health; readiness fails when no Approved deployment is healthy for a mandatory profile.
Objectives (pilot class, D30)¶
| Objective | Target |
|---|---|
| Gateway overhead (excluding provider time) | p95 under 150 ms |
| Receipt written | for 100 % of invocations that reached an adapter |
| Usage reconciliation | Unknown receipts confirmed within 24 hours |
| Provider health detection | within 2 minutes |
| Availability | 99.5 % monthly with two replicas |
Dashboards and alerts¶
Dashboards: invocations and outcomes by profile and provider, latency, tokens and cost by tenant, fallbacks and denials, provider health, reconciliation debt, pool utilization and placements. Alerts: provider unhealthy for a mandatory profile, fallback rate above threshold, usage-unknown backlog age, denial spikes, pool utilization reports stale, placement PolicyDenied rate.
Evaluation feed¶
Receipts and outcomes feed the evaluation and learning services (BC-22) that keep capability evidence current and detect drift per deployment.