Skip to content

Data Model

Target Architecture — Final-State Design

This page describes the final-state logical data model of the Observability & Feedback Platform. Physical stores are described in Storage; aggregate detail is in Aggregate Roots.

The platform's data model is organised around one universal correlation key — traceId — and a fixed set of required telemetry dimensions that stamp every record. This is what makes any signal sliceable by factory concept (tenant, project, module, agent, skill, artifact, workflow) and traceable end to end.

Logical Model

The core entities and their relationships: a TraceRecord is the anchor; MetricSeries, Incident, FeedbackItem, and QualityScore all attribute back to it (directly or via the required dimensions). A TelemetryCorrelation is the explicit join.

erDiagram
    TraceRecord ||--o{ Span : contains
    TraceRecord ||--o| TelemetryCorrelation : "anchors"
    TelemetryCorrelation ||--o{ MetricSeries : links
    TelemetryCorrelation ||--o{ Incident : links
    TelemetryCorrelation ||--o{ FeedbackItem : links
    Incident ||--o{ IncidentEvent : logs
    Incident ||--o| FeedbackItem : "resolves into"
    FeedbackItem ||--o| QualityScore : "contributes to"
    MetricSeries ||--o{ MetricPoint : contains
    QualityScore ||--o{ ArtifactScore : "scores"

    TraceRecord {
        string traceId PK
        string tenantId
        string projectId
        string moduleId
        string status
        int durationMs
        datetime startedAt
    }
    MetricSeries {
        string seriesId PK
        string tenantId
        string metricName
        string groupKey
        string unit
        string aggregation
    }
    Incident {
        string incidentId PK
        string tenantId
        string projectId
        string traceId FK
        string severity
        string status
        datetime openedAt
        datetime resolvedAt
    }
    FeedbackItem {
        string feedbackItemId PK
        string tenantId
        string projectId
        string artifactId
        string traceId FK
        string source
        string category
        string sentiment
    }
    QualityScore {
        string qualityScoreId PK
        string tenantId
        string projectId
        decimal overall
        datetime computedAt
    }
Hold "Alt" / "Option" to enable pan & zoom

Required Telemetry Dimensions

Every telemetry record, signal, incident, feedback item, and score carries the following dimensions. They are the join keys of the whole platform and are mandatory on all ingestion paths.

Dimension Type Meaning Source
traceId string End-to-end trace spanning prompt-to-runtime. Edge / propagated
executionId string A single agent task or request execution. Agent Mesh / runtime
tenantId string Owning tenant; isolation boundary. Identity
projectId string Factory project. Control Plane
moduleId string Module (service, UI, worker, library). Generation metadata
agentId string Agent that produced the work, if any. Agent Mesh
skillId string Skill invoked, if any. Agent Mesh
artifactId string Artifact the signal concerns. Knowledge Platform
workflowId string Orchestration workflow. Control Plane
environment string development / staging / production. Runtime & Cloud
version string Artifact / deployment version. DevOps / GitOps

These dimensions are emitted as OpenTelemetry resource/span attributes and Serilog enrichers, mirrored into the event envelope payload.dimensions, and indexed in relational stores for query. See Observability for how they are enforced and Metadata Schema for the cross-cutting definition.

Retention

Data class Store Retention Rationale
Raw traces / spans Application Insights 30–90 days (tiered) High volume; recent trace debugging.
Sampled / important traces App Insights + Blob export 1 year Lineage for incidents and audits.
Logs Log Analytics 30–90 days hot, archive to Blob Cost-controlled; compliance archive.
Metric series App Insights / metric store 13 months Year-over-year SLO and trend analysis.
Incidents Azure SQL / PostgreSQL 3+ years Post-mortems, governance, audit.
Feedback items Azure SQL + Blob Indefinite (lifecycle policy) Durable improvement-loop memory.
Quality scores Azure SQL 3+ years Quality trend tracking.
Cost signals Azure SQL + Blob 7 years Financial/audit retention.

Retention is configured per tenant where contractual requirements differ; archive tiers move cold data to Azure Blob with the original traceId and dimensions preserved for replay.

Multi-Tenancy

  • tenantId everywhere. Every aggregate, series, index, dashboard, alert, and export carries tenantId. It is a partition key and a mandatory query filter.
  • Isolation at query time. LogQueryService and MetricAggregationService inject the caller's tenantId into every Log Analytics/metric query; cross-tenant queries are impossible through the API.
  • Per-tenant partitioning. App Insights and Log Analytics use tenant-aware partitioning / workspace strategies; relational stores partition transactional tables by tenantId.
  • Per-tenant retention & cost. Retention windows and cost attribution are tenant-scoped, so each tenant's data lifecycle and economics are independent.
  • No cross-tenant correlation. TelemetryCorrelation enforces that all linked refs share the same tenantId; a traceId never spans tenants.