Skip to content

Control Plane — Deployment

The Control Plane runs as a fleet of independently deployable containerized .NET 10 / ASP.NET Core services on Azure. Infrastructure is provisioned with Pulumi (infrastructure-as-code) and delivered through the factory's own DevOps & GitOps pipelines. Each microservice is a separately versioned, separately scaled deployment unit with its own database, aligned to the database-per-service storage model.

Target Architecture — Final-State Design

The runtime targets Azure Kubernetes Service (AKS) for the core, high-traffic, stateful-saga services and Azure Container Apps (ACA) for elastic, event-driven workers. All infrastructure (clusters, namespaces, Service Bus, SQL/PostgreSQL, Redis, Key Vault, identity) is declared in Pulumi stacks per environment.

Runtime Model

Concern Choice
Orchestration AKS (core services & sagas) + Azure Container Apps (workers, bursty consumers).
Packaging OCI containers built from ConnectSoft.MicroserviceTemplate / ConnectSoft.WorkerTemplate.
Messaging Azure Service Bus (MassTransit transport).
Data Azure SQL / Azure Database for PostgreSQL (per service), Azure Cache for Redis.
Identity OpenIddict authorization server (ConnectSoft.AuthorizationServerTemplate); workload identity via Azure AD-managed identities.
Ingress ConnectSoft.ApiGatewayTemplate fronts public APIs; internal/gRPC traffic stays in-cluster.
IaC Pulumi (clusters, namespaces, data stores, Service Bus, Key Vault, DNS, autoscaling).
Secrets Azure Key Vault via CSI driver + workload identity.

Deployment Topology

flowchart TB
    Gateway["API Gateway<br/>(ConnectSoft.ApiGatewayTemplate)"] --> CoreNs

    subgraph AKS["AKS Cluster"]
        subgraph CoreNs["Namespace: control-plane-core"]
            WfOrch["WorkflowOrchestrator (sagas)"]
            TaskSvc["TaskAssignmentService"]
            Policy["PolicyEngineService"]
            Tenant["TenantService"]
            Identity["IdentityService / AuthorizationService"]
        end
        subgraph SvcNs["Namespace: control-plane-services"]
            Project["Project / Blueprint / Agent services"]
            Approval["ApprovalService / AuditService"]
            Cost["CostUsageService"]
            Integration["IntegrationService"]
        end
    end

    subgraph ACA["Azure Container Apps"]
        Workers["Workers<br/>(timeout, assignment, projection,<br/>quota, rollup, audit-export, validation,<br/>dependency, replay)"]
    end

    CoreNs --> ServiceBus["Azure Service Bus"]
    SvcNs --> ServiceBus
    Workers --> ServiceBus
    CoreNs --> SQL["Azure SQL (per service)"]
    SvcNs --> PG["PostgreSQL (per service)"]
    CoreNs --> Redis["Azure Cache for Redis"]
    CoreNs --> KeyVault["Azure Key Vault"]
    SvcNs --> KeyVault
    Workers --> KeyVault
Hold "Alt" / "Option" to enable pan & zoom

Scaling

Workload class Scaling signal Strategy
API services (Project, Blueprint, Tenant, etc.) CPU + request rate Horizontal Pod Autoscaler (AKS); min 2 replicas for HA.
Saga core (WorkflowOrchestrator) Saga throughput + queue depth Scale by partitioned correlation; replicas process disjoint correlation sets.
Workers (event-driven) Service Bus queue depth (KEDA) Scale-to-many on backlog, scale-to-zero when idle (ACA).
Read models (ProcessStateService) Read QPS Scale out replicas; Redis hot cache absorbs spikes.
Hot-path gRPC (Policy, Quota, TaskAssignment) RPS + latency SLO Over-provision; circuit breakers protect dependents.

Tenant-aware concurrency limits (via AgentPoolManager leases and QuotaService) prevent any single tenant from starving the platform.

Configuration

  • Layered config: appsettings.json → environment overrides → Key Vault secrets → runtime feature flags (FeatureFlagService). Bound through ConnectSoft.Extensions.Options.
  • Per-environment Pulumi stacks: dev, test, staging, prod each declare their own clusters, data stores, and scaling envelopes.
  • Tenant isolation config: editions select shared vs dedicated data isolation; dedicated tenants get isolated databases provisioned by Pulumi.
  • Messaging config: topics/subscriptions and dead-letter policies are declared as code; subscription filters use cs-* application properties (see Events).

Secrets (Key Vault)

  • Connection strings, OpenIddict signing/encryption keys, and integration credentials live in Azure Key Vault, mounted via the Secrets Store CSI driver using workload identity (no secrets in images or env files).
  • IntegrationConnection aggregates store only a configRef pointing to Key Vault; secrets are never persisted in service databases (see Aggregate Roots).
  • Keys and credentials are rotated on a schedule; rotation emits CredentialRotated and is audited.

Health Checks & Readiness

Every service exposes the standard probes (via ConnectSoft.Extensions.Diagnostics.HealthChecks), aggregated by ConnectSoft.HealthChecksAggregatorTemplate:

Probe Path Checks
Liveness /health/live Process responsive.
Readiness /health/ready DB reachable, Service Bus connected, Redis reachable, migrations applied.
Startup /health/startup Migrations + warm caches before accepting traffic.

Probes gate rollouts: a service is not added to the load balancer until readiness passes, and rolling updates halt on failing health.

Deployment Pipeline & Resilience

  • GitOps delivery: container images and Pulumi stacks are promoted through environments by the DevOps & GitOps platform; the Control Plane's own Release workflow drives prod promotion behind the approval gate.
  • Zero-downtime rollouts: rolling updates with surge + health gating; database migrations are backward-compatible (expand/contract).
  • Resilience: retries, circuit breakers, and timeouts on inter-service calls; Service Bus dead-letter queues preserve the full envelope for replay; multi-zone deployment for HA.
  • Observability: OpenTelemetry traces (keyed on traceId), metrics, and Serilog logs flow to the Observability & Feedback platform.