Skip to content

Storage

Target Architecture — Final-State Design

This page maps every class of Integration Platform data to its physical store, owning service, access pattern, and retention. The guiding rule is single ownership: each store is owned by exactly one service, and credentials never exist as plaintext anywhere outside Azure Key Vault.

The platform uses three physical store families: a relational store (Azure SQL / PostgreSQL via NHibernate) for connection, provider, run, and delivery metadata; Azure Key Vault for all secret material; and Azure Blob for raw webhook payloads and large request/response bodies. Azure Service Bus is the transport (not a store of record), and Redis provides hot caches for provider lookups and rate-limit buckets.

Storage Map

Data Store Owner Service Access Pattern Retention Notes
Integration connections Azure SQL / PostgreSQL Owning integration service (e.g. GitHubIntegrationService) Read-heavy point lookups by connectionId / tenantId Lifetime + 90 days Tenant-partitioned; configuration as JSON column
Providers & API clients Azure SQL / PostgreSQL VendorApiClientRegistryService Read-heavy catalogue reads; cached in Redis Lifetime global rows shared read-only across tenants
Credential metadata Azure SQL / PostgreSQL VendorApiClientRegistryService Point lookups + rotation writes Lifetime + audit window References only — keyVaultRef, version, fingerprint; no secret bytes
Credential secrets Azure Key Vault VendorApiClientRegistryService (custodian) Resolve at call time; rotate by version Per Key Vault rotation policy Never plaintext in any DB, log, or event; tenant-namespaced
Integration runs Azure SQL / PostgreSQL Owning integration service Append-heavy writes; query by runId / connectionId 180 days hot, then archive One row per run + child RunAttempt rows
Integration failures Azure SQL / PostgreSQL Owning integration service / WebhookGatewayService Write on failure; sweep by IntegrationRetryWorker 180 days Drives retry and escalation analytics
Webhook subscriptions Azure SQL / PostgreSQL WebhookGatewayService Read on every matching event; write on subscribe Lifetime signingSecretRef points to Key Vault
Webhook delivery metadata Azure SQL / PostgreSQL WebhookGatewayService Append + status updates; query by deliveryId 90 days hot, then archive payloadRef points to Blob
Raw webhook payloads Azure Blob WebhookGatewayService Write once on ingress; read on replay 30 days, then tiered/purged Container partitioned by tenantId; short-lived SAS
Request / response bodies Azure Blob Owning integration service Write on run; read for diagnostics/replay 30 days, then tiered/purged Large model/cloud payloads kept off the relational store
Provider lookups & rate-limit buckets Redis VendorApiClientRegistryService Hot read/write; TTL-based Ephemeral (cache) Rebuildable from SQL; never a source of record
Events (transport) Azure Service Bus MassTransit (all services) Publish/subscribe; dead-letter on poison Per subscription TTL Carries canonical envelope; not a store of record

Ownership & Isolation Rules

  • One owner per store. Only the owning service writes to a given table or container; other services read via APIs or react to events. No shared databases.
  • Secrets are custodied, not owned by the domain. IntegrationCredential rows hold only references; the VendorApiClientRegistryService is the Key Vault custodian and the only component with rotation rights.
  • Plaintext-free guarantee. Secret material is resolved transiently at call time (via ConnectSoft.Extensions.Http.OAuth2 / Key Vault SDK), used in-memory, and never persisted, logged, or placed in an event payload. Logs and events carry only credentialRef and SecretFingerprint.
  • Tenant partitioning. tenantId is the partition key in SQL, the namespace in Key Vault, and the container prefix in Blob.
  • Caches are rebuildable. Redis holds derived data (provider capabilities, rate-limit counters) that can always be reconstructed from the relational store.

Backup & Durability

  • Relational stores use geo-redundant backups with point-in-time restore; migrations are managed via the DatabaseModel.Migrations project per service.
  • Key Vault uses soft-delete and purge protection; rotation retains prior versions until verification succeeds.
  • Blob containers use lifecycle management to tier hot → cool → archive and to enforce purge of expired payloads.