Skip to content

Model gateway

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

The model gateway (packet role GW-04, re-validated) hides physical providers behind logical profiles and makes every invocation attributable, policy-checked and budgeted. It is a library pipeline inside the runtime worker first; a separate host is justified only by central credential isolation, independent capacity or multiple consumers.

Contract

Concern Required behaviour
Input Trusted tenant/project/run/step/attempt scope; immutable release; versioned model profile; policy decision; request hash and idempotency key; classification; invocation budget
Profile Modality; tool and structured-output support; context and output limits; permitted regions and deployments; latency class; embedding compatibility; allowed provider and model revisions
Resolution Pin the profile and route policy for the run; record the actual provider, model and version per invocation, including an approved fallback; never silently change privacy, tool or output guarantees
Authorization Caller workload identity, user delegation and provider credential are distinct; secrets resolved only for an authorized connection and never placed in prompts, events or manifests
Budget Reserve before dispatch; settle observed usage; reconcile unknown or partial usage; timeout or cancellation does not prove no billable work happened; record pricing revision and estimated vs confirmed cost
Retry and fallback Classify transient, policy, validation and provider failures; bound retries by deadline, budget and side-effect rules; a partially emitted stream is never spliced with another model's output
Streaming Cancellation propagation, partial-output status, backpressure, final usage receipt; if reconnect is unsupported, say so and let the client query run state
Protocols OpenAI-compatible and Anthropic-compatible protocols, internal model API, embeddings, reranking, structured outputs, tool calls; no promise of complete wire compatibility merely because adapters use IChatClient
Response Typed output or stream plus invocation id, actual route, policy receipt, usage status, validation status and references to redacted diagnostics; Run Control decides workflow transitions
Operations Provider and capability health, latency, errors, tokens, cost reconciliation, policy denial and fallback metrics; no raw prompt or response logging by default 
sequenceDiagram
    participant W as Runtime worker
    participant G as Model gateway (pipeline)
    participant P as Policy (Trust & Ops)
    participant U as Usage Accounting (Control)
    participant A as Provider adapter (IChatClient)
    W->>G: Invoke(profile, scope, request hash, budget)
    G->>P: Decision for profile, data class, region, tools
    P-->>G: Allowed route plan (+ permitted fallback)
    G->>U: Reserve
    G->>A: Call resolved provider/model/deployment
    A-->>G: Result or stream (+ reported usage)
    G->>U: Settle (observed / estimated / unknown)
    G-->>W: Receipt: actual model, usage status, policy receipt, output
Hold "Alt" / "Option" to enable pan & zoom

Current implementation

  • ConnectSoft.Extensions.AI registers keyed IChatClients for OpenAI, Azure OpenAI, Ollama and Azure AI Inference and embedding generators; the agents framework resolves them per agent. No profile, policy, reservation, receipt, fallback-on-failure or streaming contract exists.
  • Model invocations happen inside FactoryAgentBase/AIAgent.RunAsync; two template entry paths bypass even that.
  • The prototype's provider dropdowns are Claude-branded fixtures; no Anthropic adapter exists. One is required for the first solution pack (D16, 2026-10-09).