Skip to content

Storage

Target Architecture — Final-State Design

This page describes the final-state storage topology of the Observability & Feedback Platform. Each data class has a single owning service and a deliberate store chosen for its access pattern, retention, and cost profile.

The platform spans three families of store: telemetry stores (Application Insights, Log Analytics) optimised for high-volume ingest and time-series query; transactional stores (Azure SQL / PostgreSQL via NHibernate) for the aggregates that need ACID guarantees and lifecycle; and Azure Blob for exports, archives, and attachments. The choice per data class is driven by access pattern, not convenience — high-cardinality telemetry never lands in relational tables, and transactional aggregates never live only in telemetry stores.

Storage Map

Data Store Owner Service Access Pattern Retention Notes
Distributed traces / spans Application Insights TraceService Write-heavy ingest; query by traceId/dimensions 30–90 days tiered OTLP ingest; correlated projection cached.
Structured logs Log Analytics LogQueryService Write-heavy ingest; KQL search 30–90 days hot + Blob archive Serilog sink; tenant-scoped queries only.
Metric series Application Insights (metrics) MetricAggregationService Append points; time-range/group query 13 months Series metadata mirrored in Azure SQL.
Dashboard definitions Azure SQL / PostgreSQL DashboardService Transactional CRUD Lifecycle NHibernate; projected to workbooks/Grafana.
Alert rules Azure SQL / PostgreSQL AlertRuleService Transactional CRUD; rule scan Lifecycle NHibernate.
Incidents Azure SQL / PostgreSQL IncidentService Transactional; lifecycle + event log 3+ years NHibernate; append-only incident events.
Feedback items Azure SQL / PostgreSQL + Azure Blob FeedbackService Transactional metadata + blob attachments Indefinite Improvement-loop memory; attachments in Blob.
Quality scores Azure SQL / PostgreSQL QualityScoreService Transactional; recompute upsert 3+ years NHibernate; deterministic recompute.
Cost signals Azure SQL / PostgreSQL + Azure Blob CostTelemetryService Transactional metadata + detailed exports 7 years Financial retention; exports in Blob.
SLO definitions Azure SQL / PostgreSQL SloService Transactional CRUD; budget burn read 13+ months NHibernate; SLI computed from MetricSeries.
Telemetry correlation index Application Insights + Azure SQL TelemetryCorrelationService Upsert by traceId; link lookups Aligned to trace retention Anchor in App Insights; link index in SQL.
Exports / archives / attachments Azure Blob Feedback / Cost / Trace export Bulk write; cold read Per data-class policy Original traceId + dimensions preserved.

Storage Topology Diagram

flowchart TB
    subgraph Telemetry["Telemetry stores"]
        AI[("Application Insights<br/>traces, metrics")]
        LA[("Log Analytics<br/>logs")]
    end

    subgraph Transactional["Transactional stores (Azure SQL / PostgreSQL)"]
        SQLInc[("Incidents")]
        SQLFb[("Feedback")]
        SQLQ[("Quality")]
        SQLCost[("Cost")]
        SQLDash[("Dashboards / Alerts / SLO")]
        SQLCorr[("Correlation index")]
    end

    subgraph Archive["Object store"]
        Blob[("Azure Blob<br/>exports, archives, attachments")]
    end

    Runtime["Runtime &amp; Agents"] -->|OTLP| AI
    Runtime -->|Serilog| LA
    AI --> SQLCorr
    LA -->|archive| Blob
    AI -->|important traces| Blob
    SQLFb --> Blob
    SQLCost --> Blob
Hold "Alt" / "Option" to enable pan & zoom

Storage Principles

  • Right store for the access pattern. Telemetry (high cardinality, time-series, write-heavy) lives in App Insights / Log Analytics; aggregates needing transactions and lifecycle live in Azure SQL / PostgreSQL; bulk/cold data lives in Blob.
  • Single owner. Each data class is owned and written by exactly one service (see Microservices); other services read via APIs or events, never the store directly.
  • tenantId is a partition boundary. Every store partitions or filters on tenantId; see Data Model — Multi-Tenancy.
  • Trace-preserving archive. When data ages out of hot stores it is archived to Blob with the original traceId and required dimensions intact, so historical lineage and replay remain possible.
  • Persistence via NHibernate. Transactional stores follow the ConnectSoft microservice template's PersistenceModel.NHibernate layering with DatabaseModel.Migrations for schema evolution.
  • Cost-aware retention. Retention tiers (hot → archive) are configured per data class and per tenant to balance debuggability, compliance, and cost.