Skip to content

Model & Compute — Deployment

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Runtime model

Concern Choice
Deployment unit ModelGatewayHost, one image from the microservice template; modules routing and compute pools
Hosting AKS, 2 to 12 replicas, scaled on request rate and gateway overhead
Data model-sql (profiles, deployments, evidence, route plans, receipts, pools, placements); cache for profiles, route plans and health
Messaging MessagingModel outbox and inbox; Orleans profile by default; PoolGrain for placement serialisation
Identity workload identity; provider credentials brokered from Key Vault through the Secret Broker
Ingress gRPC from runtime workers and Knowledge; GW-01 for the OpenAI-compatible and Anthropic-compatible protocols; GW-03 for Studio reads
Egress provider endpoints per region, self-hosted pool endpoints, Governance, Control plane; everything else denied

Topology by profile

Profile Gateway placement Providers and pools
saas-default ConnectSoft AKS in the tenant's region regional provider deployments; ConnectSoft pools
hybrid ConnectSoft cloud; runtime workers in the customer network call it over a private link as saas-default; customer pools reachable over the link when registered
on-prem customer cluster customer provider accounts and pools; Ollama and self-hosted deployments typical
hybrid-compute ConnectSoft cloud tenant-private pools for pii-local; regional providers for the rest 
flowchart LR
    subgraph CS["ConnectSoft cloud, tenant region"]
        MG["ModelGatewayHost"]
        SQL["('model-sql')"]
        Prov["Provider deployments (regional)"]
    end
    subgraph T["Tenant-private capacity (hybrid-compute)"]
        Pool["Compute pool: self-hosted models"]
    end
    MG --> SQL
    MG --> Prov
    MG -- "private link, brokered credential" --> Pool
Hold "Alt" / "Option" to enable pan & zoom

Scaling

Signal Action
gateway overhead above 150 ms p95 add replica
provider rate limits per-tenant and per-profile rate limiting at the gateway (ConnectSoft.Extensions.RateLimiting); queueing within the attempt budget
receipt write latency partition receipts by tenant and month
pool utilization above target WarmKeeper and placement spread; later-stage scheduler provisions pools

Configuration and secrets

Options sections ModelGateway:Host, ModelGateway:Adapters, ModelGateway:Protocols, ModelAccess:Routing, ModelAccess:Invocation, ModelAccess:Privacy, ComputePools:Placement, ComputePools:Warm. Provider and pool credentials are brokered per call from Key Vault; nothing is configured as a plain secret on the host.

Health and rollout

/model-gateway/health with per-provider health; a replica drains in-flight streams before stopping; adapter package upgrades are rollouts whose capability evidence is re-run for affected deployments before approval.

NFR class and recovery

pilot-interactive (D30). model-sql is restored after the control and runtime stores; receipts referenced by audit evidence are kept 7 years, the rest 1 year by default.