ModelGatewayHost¶

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.


Bounded context	BC-07 Model Access
Platform / plane	Model & Compute Platform / Model & Compute
Deployment granularity	standalone host
Status	target · `ConnectSoft.Factory.ModelAccess`
Built from	connectsoft-microservice · libraries: connectsoft-extensions-ai, connectsoft-extensions-saas-abstractions, connectsoft-extensions-saas-asp-net-core, connectsoft-extensions-messaging-model, connectsoft-extensions-messaging-model-orleans, connectsoft-extensions-persistence-model-nhibernate, connectsoft-extensions-observability, connectsoft-extensions-rate-limiting
Decisions	ADR-0002, ADR-0012, ADR-0013, ADR-0017, ADR-0021
Traceability	`FR-04` `FUX-08` `FUX-10` `NFR-07` `NFR-12` `NFR-13` `NFR-18` `XL-04` `XL-16`

Purpose and responsibilities¶

The model gateway design says "in-process first, separate host only on evidence". ModelGatewayHost is that separate host, and the evidence is already in the design: provider credentials must be isolated from every runtime worker (a worker in a customer's network in the hybrid profile must never hold ConnectSoft's provider keys), Knowledge needs embeddings and reranking without a runtime worker, and partner-facing OpenAI-compatible and Anthropic-compatible wire protocols need one endpoint behind GW-01. The host composes ModelRouterService (BC-07) and ComputePoolService (BC-41), loads the provider adapters from ConnectSoft.Extensions.AI (OpenAI, Azure OpenAI, Azure AI Inference, Ollama and the required Anthropic adapter, D16), and exposes the gateway protocols.

Context and boundaries¶

flowchart LR
    S["ModelGatewayHost"]
    AgentRuntimeWorker["AgentRuntimeWorker"] -->|grpc| S
    KnowledgeHost["KnowledgeHost"] -->|grpc| S
    StudioBff["StudioBff"] -->|sync-http| S
    S -->|sync-http| ext_model_providers["ext-model-providers"]
    S -->|grpc| GovernanceHost["GovernanceHost"]
    S -->|grpc| ControlPlaneApi["ControlPlaneApi"]
    S -->|sync-http| ext_identity_platform["ext-identity-platform"]

Hold "Alt" / "Option" to enable pan & zoom

flowchart LR
    subgraph Host["ModelGatewayHost (2 to 12 replicas)"]
        direction TB
        Proto["Protocols: gRPC ModelGateway, OpenAI-compatible, Anthropic-compatible"]
        Router["ModelRouterService<br/>route plans, receipts"]
        Pools["ComputePoolService<br/>placement, warm state"]
        Adapters["Adapters: openai, azure-openai, azure-ai-inference, ollama, anthropic"]
    end
    RT["AgentRuntimeWorker"] -- gRPC --> Proto
    KH["KnowledgeHost"] -- "Embed, Rerank" --> Proto
    GW["GW-01 (in-tenant clients)"] --> Proto
    Proto --> Router --> Adapters
    Router --> Pools
    Adapters --> Prov["Provider endpoints and self-hosted pools"]
    Router --> SQL["('model-sql')"]
    Router -- "decisions, credentials" --> Gov["GovernanceHost"]
    Router -- "RecordUsage" --> CP["ControlPlaneApi"]

Hold "Alt" / "Option" to enable pan & zoom

Domain model¶

None modelled

No aggregates are modelled for this item yet.

No entities modelled.

The host has no aggregates; see the two module pages.

Entities and value objects¶

None modelled

No entities or value objects are modelled for this item yet.

Data model and storage¶

Store	Role	Engine / profile	Tables
Model access store	primary	azure-sql / nhibernate-sqlserver	—
Cache and coordination	cache	redis / azure-cache	—

None modelled

No tables are modelled for this item yet.

APIs¶

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
`GET`	`/model-gateway/health`	GatewayHealth	`factory.health.read`	read	natural	GW-01	Liveness, readiness and per-provider health of this host (adapters, store, policy, usage accounting reachability).
`GET`	`/model-gateway/meta/adapters`	GatewayAdapters	`factory.meta.read`	read	natural	GW-03	Provider adapters loaded in this host with their package versions and supported protocols (OpenAI-compatible, Anthropic-compatible, embeddings, reranking, streaming, structured outputs, tool calls).

None modelled

No gRPC methods are modelled for this item yet.

Messages¶

None modelled

No events are modelled for this item yet.

None modelled

No commands are modelled for this item yet.

None modelled

No consumed events are modelled for this item yet.

Workers and jobs¶

None modelled

No workers are modelled for this item yet.

Actors and grains¶

None modelled

No grains are modelled for this item yet.

Sagas and long-running flows¶

None modelled

No sagas are modelled for this item yet.

Options and configuration¶

Section	Key	Type	Default	Reloadable	Secret	Summary
`ModelGateway:Host`	`EnabledModules`	string[]	`[routing, compute-pools]`	no	no	Modules composed into this host.
`ModelGateway:Adapters`	`Enabled`	string[]	`[openai, azure-openai, azure-ai-inference, ollama, anthropic]`	no	no	Keyed IChatClient and embedding adapters loaded at start; anthropic is required for the first solution pack (D16).
`ModelGateway:Protocols`	`ExposeOpenAiCompatible`	bool	`true`	no	no	Serve the OpenAI-compatible wire protocol to in-tenant clients through GW-01.
`ModelGateway:Protocols`	`ExposeAnthropicCompatible`	bool	`true`	no	no	Serve the Anthropic-compatible wire protocol.
`OrleansMessaging`	`ServiceName`	string	`model-gateway`	no	no	Messaging profile service name.

Metrics and observability¶

Metric	Instrument	Unit	Dimensions	Summary
`connectsoft.factory.model_gateway.host.requests`	counter	{request}	protocol, status	Requests by protocol.
`connectsoft.factory.model_gateway.host.provider_health`	gauge	1	provider, region	1 healthy, 0 unhealthy per provider adapter.

Security¶


Authentication	client-credentials
Scopes	`factory.health.read` (Health probes.); `factory.meta.read` (Adapter metadata.)
Roles	Operator, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	confidential
Max side-effect class	spend
Secrets	`model-sql` (key-vault, rotate 90 d); `provider-credentials` (key-vault, rotate 90 d); `messaging-transport` (key-vault, rotate 90 d)

Dependencies¶

Outbound

Target	Kind	Purpose	Failure mode	Required
ext-model-providers	sync-http	Provider endpoints behind the adapters.	degrade	yes
GovernanceHost	grpc	Policy decisions per invocation and secret brokering for provider credentials.	fail-fast	yes
ControlPlaneApi	grpc	Usage recording and spend reservations (Usage Accounting).	fail-fast	yes
ext-identity-platform	sync-http	Workload identity tokens and caller validation.	fail-fast	yes

Inbound

Target	Kind	Purpose	Failure mode	Required
AgentRuntimeWorker	grpc	Model invocations of attempts.	—	yes
KnowledgeHost	grpc	Embeddings and reranking for ingestion and retrieval.	—	yes
StudioBff	sync-http	Profile, deployment and pool administration views.	—	yes

Failure and recovery¶

Scenario	Detection	Recovery	Data impact	RTO (min)
Provider outage	adapter errors and health probe	route plan falls back only within equivalent constraints; otherwise invocations fail with class transient and the runtime retries within budget	none	30
Policy engine unreachable	decision call failures	fail closed; no invocation without a decision	none	15
Host crash mid-stream	client stream break	partial output status reported in the receipt; the client queries run state; no splicing of outputs	partial output flagged	5

NFR targets¶

Target	Class value (pilot-interactive)	Override
latency_p95_ms	2000	—
availability_pct	99.5	—
throughput_rps	50	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	low	—

Deployment¶


Granularity	standalone host
Topologies	saas-default, hybrid, on-prem, hybrid-compute
Runtime	aks; replicas 2–12
Ports	https 7600, grpc 7602, silo 11600, gateway 30600

In hybrid-compute the host stays in ConnectSoft's cloud and reaches tenant-private pools over a private link; in on-prem it runs in the customer's cluster with their provider accounts.

Current implementation¶

ConnectSoft.Extensions.AI registers keyed IChatClient and embedding generators for OpenAI, Azure OpenAI, Ollama and Azure AI Inference; no gateway host, profile, receipt, route plan or Anthropic adapter exists.

Later capabilities¶

Prompt-cache sharing across attempts of one run; regional host pairs for the ResidencySilo mode; a request-level cost governor.

Decisions and traceability¶

Family	Ids
FR	FR-04
FUX	FUX-08, FUX-10
NFR	NFR-07, NFR-12, NFR-13, NFR-18
XL	XL-04, XL-16

Model gateway · ModelRouterService · ComputePoolService · Deployment
Model & Compute Platform microservices · bounded contexts · APIs · events · data model
Services catalog · Ownership map

ModelGatewayHost¶

Purpose and responsibilities¶

Context and boundaries¶

Domain model¶

Entities and value objects¶

Data model and storage¶

APIs¶

Messages¶

Workers and jobs¶

Actors and grains¶

Sagas and long-running flows¶

Options and configuration¶

Metrics and observability¶

Security¶

Dependencies¶

Failure and recovery¶

NFR targets¶

Deployment¶

Current implementation¶

Later capabilities¶

Decisions and traceability¶

Related¶