Skip to content

Observability by plane

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Merged page (2026-10-09)

Seven per-platform observability.md pages are consolidated here, one tab each. Everything below is Target. Verified facts: OpenTelemetry, Serilog, a Factory meter and Audit.NET integrations exist as libraries; no run-attributed tracing, terminal-state coverage or export verification exists for the Factory; TraceService, MetricAggregationService and similar rows are projections over external telemetry backends (ownership map). Raw prompts and retrieval queries must not be logged by default.

Consolidated from platforms/integration-platform/observability.md (2026 target design). Technology names inside are profile statements, not requirements.

Target Architecture — Final-State Design

This page describes the final-state observability model of the Integration Platform. Every service uses ConnectSoft.Extensions.Observability (Serilog + OpenTelemetry + Application Insights). Because this platform spans the boundary to external systems, its telemetry emphasises external API health, latency, error classification, and credential lifecycle — the signals that predict and explain vendor-side failures.

Observability here answers three questions continuously: Is the factory's connectivity to each vendor healthy? Why did a specific external call fail? Is any tenant or provider approaching a rate or cost limit? All signals carry the canonical envelope identity so they correlate with the Knowledge Platform and feed the Observability & Feedback Platform.

Logs

  • Structured Serilog logs with the canonical metadata enriched on every line: tenantId, projectId, traceId, correlationId, connectionId, providerId, runId / deliveryId.
  • Secret-safe. Logs carry credentialRef and SecretFingerprint only — never secret material, tokens, or full vendor payloads (payloads are referenced by Blob payloadRef).
  • Outbound call logs record provider, operation, attempt, HTTP status class, and latency; inbound webhook logs record direction, signature status, and normalisation outcome.
  • Log levels escalate failures by classification: Transient/RateLimited at warning, Auth/Validation/Poison at error.

Metrics

Metric Type Purpose
integration.run.count Counter Outbound operations by provider/operation/outcome
integration.run.latency Histogram Vendor call latency distribution
integration.run.failures Counter Failures by category (Transient/RateLimited/Auth/Validation/Poison)
webhook.delivery.count Counter Inbound/outbound deliveries by status
webhook.delivery.attempts Histogram Retry attempts per delivery
webhook.deadletter.count Counter Dead-lettered deliveries
external.health.status Gauge Per-connection health (0=Unhealthy … 3=Healthy)
external.health.latency Histogram Probe latency per provider
integration.ratelimit.remaining Gauge Remaining budget per provider/tenant
credential.rotation.count Counter Rotations by outcome
credential.expiry.days Gauge Days until credential expiry
circuit.state Gauge Open/half-open/closed circuits per provider

Traces

  • OpenTelemetry spans wrap every outbound vendor call and inbound webhook processing stage; traceId/correlationId propagate from the event envelope into each span and onward into vendor request headers where supported.
  • A single trace links: agent decision → ExecuteIntegrationRun → vendor HTTP span → IntegrationRunCompleted → downstream factory consumer.
  • Inbound traces link: vendor webhook → signature verify span → normalise span → WebhookDelivered → Knowledge Platform ingestion.
  • Spans are exported to Application Insights with provider, operation, attempt, and outcome attributes (never secrets or full payloads).

Dashboards

  • Vendor Health — health status, latency, and error rate per provider/connection; open circuits highlighted.
  • Integration Throughput — run and delivery volume, success rate, and latency percentiles by provider and tenant.
  • Failure Analytics — failures by category and provider over time; retry success rate; dead-letter trends.
  • Credential Lifecycle — upcoming expirations, rotation success/failure, and revocations.
  • Rate & Cost — remaining rate-limit budget and per-tenant usage against quotas.

Alerts

Alert Condition Action
Vendor outage external.health.status Unhealthy for N intervals Open circuit; page on-call; notify dependent platforms
Elevated failure rate integration.run.failures rate > threshold per provider Investigate; possible vendor incident
Dead-letter surge webhook.deadletter.count spike Inspect poison payloads; pause/replay after fix
Auth failures Auth-category failures > threshold Trigger credential rotation; review credential validity
Credential expiring credential.expiry.days below warning window Rotate ahead of expiry
Rate-limit exhaustion integration.ratelimit.remaining near zero Backpressure; review tenant quota

Alerts and incidents are forwarded to the Observability & Feedback Platform, where they become runtime memory attributable back to the producing connection and trace.

Required Dimensions

Every log, metric, and trace MUST carry: tenantId, traceId, correlationId, providerId, and (where applicable) connectionId, runId / deliveryId, operation, outcome, and failure category. These dimensions are what make external behaviour sliceable by tenant, provider, and operation, and are mandatory per the factory metadata schema.

External API Health Monitoring

The ExternalApiHealthWorker continuously probes vendor endpoints and writes per-connection health that gates routing:

  • Probing — lightweight, provider-appropriate health checks at a configurable interval per connection.
  • Health write — latest-wins external.health.status with latency; sustained failure opens a circuit so dependent services fail fast.
  • Recovery — half-open probing restores routing once the vendor recovers; recovery emits an event so paused work resumes.
  • Correlation — health degradation links to the IntegrationFailure records observed in the same window for root-cause analysis.

Consolidated from platforms/marketplace/observability.md (2026 target design). Technology names inside are profile statements, not requirements.

Target Architecture — Final-State Design

This page describes the final-state observability of the Marketplace Platform. All services emit structured logs via ConnectSoft.Extensions.Logging.Serilog, metrics and traces via ConnectSoft.Extensions.Observability/Telemetry (OpenTelemetry), and feed the Observability & Feedback platform. Every signal carries the canonical event envelope identity dimensions.

The Marketplace is instrumented so that any publish, search, compatibility evaluation, or installation can be traced end to end and correlated with the originating agent task or user action. Telemetry is the feedback loop that keeps the asset ecosystem healthy and governable at multi-tenant scale.

Required dimensions

Every log line, metric, and span carries these dimensions so signals correlate across the factory:

Dimension Source Purpose
tenantId envelope / token Tenant isolation and per-tenant SLOs
traceId envelope End-to-end trace stitching
correlationId envelope Workflow/saga correlation
assetId, assetType domain Per-asset and per-type analytics
version domain Per-version health
publisherId domain Publisher quality and reputation
installationId domain Installation lifecycle tracing
requestedBy (task-/user) envelope Attribution of autonomous vs human actions
service, environment runtime Service- and environment-scoped views

Logs

  • Structured Serilog with the envelope identity enriched into the log context; no secret material is ever logged (see Security).
  • Key logged events: submission received, quality verdict, signing, version released, compatibility verdict, dependency resolution result, license grant/deny, install applied/rolled back, dead-letter.
  • Logs are shipped to the central log store and correlated with traces by traceId.

Metrics

Metric Type Dimensions Use
marketplace_catalog_search_latency_ms histogram tenantId, assetType Search responsiveness
marketplace_publish_duration_ms histogram tenantId, publisherId, assetType Publishing pipeline health
marketplace_quality_scan_result_total counter result(pass/fail), assetType Quality gate effectiveness
marketplace_compatibility_eval_total counter verdict Compatibility outcomes
marketplace_install_total counter result(installed/failed/rolledback), assetType Installation success rate
marketplace_install_duration_ms histogram tenantId, assetType Install latency
marketplace_dependency_conflicts_total counter assetType Dependency health
marketplace_license_grant_total counter result(granted/denied), pricingModel Commerce health
marketplace_active_licenses gauge tenantId, assetId Entitlement footprint
marketplace_dead_letter_total counter topic, eventType Poison-message detection
marketplace_review_submitted_total counter rating Feedback volume

Traces

  • OpenTelemetry spans cover the full publishing and installation sagas: a single trace links submit → quality scan → policy gate → sign → release → index, and request → compatibility → resolve → license → apply → installed.
  • gRPC and Service Bus hops propagate traceparent and cs-correlation-id so cross-service spans stitch into one trace.
  • Traces link to the emitted events by eventId/causationId, enabling navigation from a span to the canonical event and back.

Dashboards

  • Marketplace Health — search latency, publish duration, install success rate, dead-letter trend.
  • Publishing Pipeline — submissions, quality pass/fail by publisher and asset type, time-to-release.
  • Installation Funnel — Requested → Evaluating → Resolving → LicenseCheck → Applying → Installed conversion, with failure/rollback breakdown.
  • Commerce — license grants/denials, active licenses, revenue-relevant pricing-model mix (via billing).
  • Publisher Quality — per-publisher quality, ratings, reputation, incident history.
  • Per-Tenant — install volume, active assets, and SLO compliance per tenant.

Dashboards ship as Dashboard Packs (one of the nine asset types), so the Marketplace dogfoods its own distribution model.

Alerts

Alert Condition Action
Install success rate drop success rate < SLO over 15m Page on-call; inspect installation saga
Dead-letter growth marketplace_dead_letter_total rising Investigate poison messages; replay from envelope
Quality scan failure spike fail ratio > threshold per publisher Review publisher submissions; consider trust-tier review
Compatibility incompatible spike verdict=Incompatible surge for an asset Flag version; notify publisher
Search latency SLO breach p95 latency > target Scale search; check index health
License denial spike denials surge Check billing integration; entitlement issues
Signature/hash verification failure any verification failure Security incident; quarantine package and publisher

Feedback loop

All marketplace events are ingested by the Observability & Feedback and Knowledge platforms. Adoption, quality, and incident signals feed catalog ranking, publisher trust tiers, and governance decisions — closing the loop from "what was published/installed" back to "what should be promoted, deprecated, or suspended."

Continue to Deployment, Workflows, and Security.

Consolidated from platforms/devops-gitops/observability.md (2026 target design). Technology names inside are profile statements, not requirements.

The DevOps / GitOps Platform is instrumented end to end so that every commit, build, release, and reconciliation is observable. Telemetry is emitted via OpenTelemetry and Serilog (ConnectSoft.Extensions.Observability, ConnectSoft.Extensions.Telemetry) and flows into the Observability & Feedback Platform.

Target Architecture — Final-State Design

All signals carry the required dimensions below so they can be correlated across services and with the broader factory lifecycle via a single traceId.

Required Dimensions

Every log line, metric, span, and event carries these dimensions:

Dimension Source Purpose
traceId Envelope / span context End-to-end correlation from artifact to runtime
tenantId Tenant context Tenant-scoped filtering and isolation
projectId Operation scope Per-project views
moduleId Operation scope Per-module delivery health
environment Release/promotion context dev / test / staging / prod segmentation
correlationId Envelope Workflow/saga correlation

Logs

  • Structured JSON logs via Serilog, enriched with the required dimensions and actor identity.
  • Pipeline and build logs are streamed to Blob and tailed live over gRPC; structured summaries are logged per stage.
  • Log levels are governed centrally; sensitive values are masked (see Security).

Metrics

Metric Type Dimensions Use
devops.commits.total counter tenant, project, module Commit throughput
devops.pullrequests.open gauge tenant, project Review backlog
devops.pipeline.run.duration histogram tenant, module, status Build performance
devops.build.success.rate gauge tenant, module Build health
devops.release.lead.time histogram tenant, module, environment Commit-to-deploy lead time (DORA)
devops.deployment.frequency counter tenant, environment Deployment frequency (DORA)
devops.change.failure.rate gauge tenant, environment Change failure rate (DORA)
devops.mttr histogram tenant, environment Mean time to restore (DORA)
devops.gitops.drift gauge tenant, environment Drift incidents
devops.iac.apply.duration histogram tenant, environment Pulumi apply time

Traces

  • Distributed traces span the full delivery flow; each service and worker is a span linked by traceId/correlationId.
  • gRPC streams (run logs, sync state) propagate context so live tails are correlated to their parent trace.
  • Traces tie back to the originating artifact and forward to runtime signals, enabling prompt-to-production trace stitching.

Dashboards

  • Delivery Health — DORA metrics (lead time, deployment frequency, change failure rate, MTTR) per tenant/module/environment.
  • Pipeline Overview — run durations, success rates, queue times, flaky-test signals.
  • Release & Promotion — releases by state, approval latency, promotion success, rollback counts.
  • GitOps Status — sync state and drift per environment, reconciliation latency.
  • Infrastructure — Pulumi apply durations, resource change counts, drift detections.

Alerts

Alert Condition Routing
Build success rate drop devops.build.success.rate < threshold over window DevOps Engineer agent + incident
Release stuck ReleasePlan in PendingApproval/Promoting beyond SLA Release Manager agent + Studio banner
Deployment failure DeploymentPromoted failed / rollback executed Deployment Orchestrator agent + incident
Persistent drift GitOps sync Degraded beyond reconcile window Environment Manager agent + incident
Dead-letter growth DLQ depth rising on any topic On-call + replay tooling
IaC apply failure Pulumi apply error Cloud Provisioner agent + incident

Feedback Loop

Pillar Alignment

  • Observability — comprehensive logs, metrics, and traces with mandatory correlation dimensions.
  • Traceability — traceId correlation spans artifact → commit → build → release → runtime.
  • Autonomy — alerts route to the responsible agent first, escalating to humans only when needed.

Consolidated from platforms/runtime-cloud/observability.md (2026 target design). Technology names inside are profile statements, not requirements.

Target Architecture — Final-State Design

Observability is built on Serilog + OpenTelemetry exporting to Application Insights, the same stack the factory uses everywhere. The Runtime & Cloud Platform is both observed (its own services) and the conduit through which generated runtimes become observable to the rest of the factory.

The platform's job is to make running SaaS legible. It instruments its own services and standardizes telemetry from every generated workload, then forwards the operationally significant signals to the Observability & Feedback Platform where they drive SLOs, incidents, and the factory's learning loop.

Logs

  • Structured logging via Serilog with the canonical envelope context (tenantId, projectId, moduleId, traceId, correlationId) bound to every log entry.
  • Platform service logs and generated-workload logs flow to Application Insights; deployment step logs are also persisted to Azure Blob for long-term audit.
  • Logs are tenant-scoped and queryable by traceId for end-to-end correlation.

Metrics

  • OpenTelemetry metrics exported to Application Insights, covering both platform operations and generated-runtime SLOs.
Metric Description
runtime.deployment.duration Time from Requested to Completed/RolledBack
runtime.deployment.rollback.rate Fraction of deployments that roll back
runtime.health.status Healthy/Degraded/Unhealthy per service
runtime.scaling.replicas Current vs desired replicas per service
runtime.scaling.actions Scale-up/down actions per policy
runtime.drift.open Open drift findings by severity
runtime.drift.remediation.time Time to remediate a finding
runtime.secret.rotation.age Age since last rotation per binding
Generated SaaS SLIs Latency, throughput, error rate, saturation per generated component

Traces

  • Distributed tracing via OpenTelemetry; traceId propagates from the triggering command through every service, worker, and emitted event.
  • A single trace spans provision → configure → deploy → health gate → promote → scale → drift, linking back through the DevOps delivery trace to the originating business intent.

Dashboards

  • Runtime Center tiles (see UI) render live service inventory, health, deployments, scaling, and drift.
  • Application Insights workbooks provide per-environment and per-tenant operational dashboards.
  • Cross-platform SLO dashboards live in Observability & Feedback, fed by this platform's signals.

Alerts

Alert Condition Routing
Deployment failure RuntimeDeploymentRolledBack or Failed state On-call + Runtime Center incident
Service unhealthy HealthCheckCompleted = Unhealthy beyond threshold On-call + Observability incident
Scaling violation ScalingPolicyViolated (can't meet target) On-call + capacity review
High-severity drift RuntimeDriftDetected severity = High in prod Governance + on-call
Secret rotation overdue runtime.secret.rotation.age exceeds policy Security + Governance

Required Dimensions

Every log, metric, trace, and alert carries the cross-cutting dimensions so signals are filterable and correlatable:

  • tenantId, projectId, moduleId, environmentId, serviceId
  • traceId, correlationId
  • stage (dev/test/staging/prod), region, computeTarget
  • version (image/deployment), deploymentId

Feeding Observability & Feedback

flowchart LR
    Gen["Generated SaaS Runtimes"] -->|"logs, metrics, traces"| RC["Runtime & Cloud Platform"]
    RC -->|"HealthCheckCompleted, RuntimeDriftDetected, ScalingPolicyApplied, RuntimeDeploymentCompleted"| Obs["Observability & Feedback"]
    Obs -->|"SLOs, alert rules, scaling targets"| RC
    Obs -->|"runtime signals + incidents"| Knowledge["Knowledge Platform"]
Hold "Alt" / "Option" to enable pan & zoom

Generated runtimes emit standardized telemetry because they are produced from factory templates that wire in ConnectSoft.Extensions.Observability / Extensions.Telemetry / Extensions.Logging.Serilog and ConnectSoft.Extensions.Diagnostics.HealthChecks. The Runtime & Cloud Platform normalizes and forwards the operationally significant events as integration events, closing the loop: runtime behaviour becomes feedback that the factory attributes back to the context and decisions that produced each component.

Consolidated from platforms/governance-security-compliance/observability.md (2026 target design). Technology names inside are profile statements, not requirements.

Target Architecture — Final-State Design

This page describes the final-state observability of the Governance, Security & Compliance Platform. Telemetry uses Serilog + OpenTelemetry exported to Application Insights via ConnectSoft.Extensions.Observability / ConnectSoft.Extensions.Telemetry / ConnectSoft.Extensions.Logging.Serilog, and the platform's own canonical events feed Observability & Feedback. Distinct from audit: audit is the legal record of trust decisions; observability is the operational view of platform health.

Telemetry Pillars

flowchart LR
    Services["Governance services + workers"] --> Logs["Structured logs (Serilog)"]
    Services --> Metrics["Metrics (OpenTelemetry)"]
    Services --> Traces["Distributed traces (OpenTelemetry)"]
    Logs --> AppInsights["Application Insights"]
    Metrics --> AppInsights
    Traces --> AppInsights
    Services -->|canonical events| ObsPlatform["Observability &amp; Feedback Platform"]
    AppInsights --> Dashboards["Dashboards"]
    AppInsights --> Alerts["Alerts"]
Hold "Alt" / "Option" to enable pan & zoom

Logs

  • Structured & contextual — Serilog emits structured JSON with the Metadata Schema fields enriched into every log entry (tenantId, traceId, correlationId, projectId, moduleId).
  • Decision logs — each PolicyDecision logs the matched policy/version, effect, and risk at Information; denies and gates at Warning.
  • Security logs — finding ingestion, secret leak detection, and rotation events log with severity.
  • No sensitive payloads — logs carry identifiers and digests, never secret values or restricted-classified content (redaction enforced by classification).
  • Separation from audit — operational logs are mutable/expiring; the immutable record is the audit trail.

Metrics

Metric Type Purpose
governance.policy.evaluations counter PDP call volume (by decision: allow/deny/gate).
governance.policy.evaluation.latency histogram Inline PDP latency (p50/p95/p99) — guards the request-path budget.
governance.approval.pending gauge Open approval requests (by tier/age).
governance.approval.timeouts counter Auto-rejections/escalations from timeout.
governance.approval.decision.latency histogram Time from request to human decision.
governance.audit.entries counter Audit write throughput.
governance.audit.export.lag gauge Backlog between recorded and exported entries.
governance.security.findings.open gauge Open findings by severity.
governance.risk.score histogram Distribution of computed risk scores by band.
governance.classification.events counter Classifications assigned/changed.
governance.compliance.reports counter Reports generated (by framework/status).
governance.worker.dlq gauge Dead-lettered messages per worker.

Traces

  • End-to-end spans — the inline PDP call, supplier gRPC calls (isolation/classification/risk), approval lifecycle, and audit writes are all spans on the originating action's traceId, so a deployment promotion can be traced from request → evaluation → approval → audit.
  • Cross-platform stitching — traceId/correlationId propagate from the Control Plane / DevOps & GitOps caller into governance spans and out via emitted events, giving a single thread across platforms.
  • Worker spans — each worker creates a span linked to the consumed event's eventId for replay debugging.

Dashboards

Dashboard Audience Shows
PDP Health Platform operators Evaluation volume, latency percentiles, allow/deny/gate mix, error rate.
Approval Operations Compliance/release managers Pending queue depth and age, timeout/escalation rate, decision latency.
Security Posture Security engineers Open findings by severity, leaked-secret rate, risk band distribution.
Audit & Compliance Compliance officers Audit throughput, export lag, chain-integrity status, reports generated.
Worker Health Platform operators Throughput, retry/DLQ counts, lag per worker.

Alerts

Alert Condition Severity
PDP latency breach p99 evaluation latency above budget for 5 min High
PDP error spike Evaluation error rate above threshold High
Approval queue aging Open requests exceeding SLA age Medium
Excessive timeouts Timeout/escalation rate spike Medium
Critical finding Any Critical SecurityFinding raised High
Leaked secret SecurityFindingRaised (leak category) Critical
Audit export lag Export lag above threshold High
Chain integrity Hash-chain verification failure Critical
Worker DLQ growth DLQ depth rising for any worker Medium

Required Dimensions

Every metric, log, and trace carries — at minimum — the following dimensions so signals can be sliced and correlated, per the Metadata Schema:

tenantId, traceId, correlationId, projectId, moduleId, service, environment, plus domain dimensions: governanceDomain (one of the ten), decision, severity, riskBand, framework, and worker.

How Observability Ties Back

  • Traceability — shared traceId makes any governance decision queryable end to end.
  • Autonomy — PDP latency and approval-queue metrics confirm the factory keeps moving autonomously and surfaces only true exceptions to humans.
  • Governance & compliance — export lag and chain-integrity alerts guarantee the evidence record stays complete and provable.
  • Multi-tenant scale — every signal is tenant-dimensioned, so per-tenant health and load are visible.

Consolidated from platforms/observability-feedback/observability.md (2026 target design). Technology names inside are profile statements, not requirements.

Target Architecture — Final-State Design

This page describes how the Observability & Feedback Platform observes itself. It dogfoods the same Serilog + OpenTelemetry + Application Insights stack it provides to the rest of the factory — if the platform that watches everything is not itself observable, the trust loop has a blind spot.

The platform applies its own discipline to itself. Every service and worker is instrumented with the shared ConnectSoft observability libraries, emits the required telemetry dimensions, and is monitored by the same dashboards, alerts, and SLOs it offers to generated SaaS products. This is consistent with the factory's observability-driven design principle.

Implemented

The instrumentation substrate is already realised in the codebase: ConnectSoft.Extensions.Observability, ConnectSoft.Extensions.Telemetry, ConnectSoft.Extensions.Logging.Serilog, and ConnectSoft.Extensions.Diagnostics.Metrics, exporting OTEL traces/metrics/logs to Application Insights / Log Analytics, with optional Grafana/Prometheus dashboards. See Platform Foundations — Observability and Factory Runtime — Observability.

Logs

  • Structured Serilog via ConnectSoft.Extensions.Logging.Serilog, sinking to Log Analytics.
  • Every log line is enriched with the required dimensions (traceId, executionId, tenantId, projectId, moduleId, agentId, skillId, artifactId, workflowId, environment, version).
  • Redaction enrichers scrub secrets and PII before egress (see Security).
  • The platform's own logs are queried through the same LogQueryService it exposes to others.

Metrics

  • OpenTelemetry metrics via ConnectSoft.Extensions.Diagnostics.Metrics, exported to Application Insights.
  • Per-service operational metrics: ingestion throughput and lag (TraceIngestionWorker, MetricAggregationWorker), Service Bus subscription backlog, aggregation window latency, alert evaluation duration, incident open/resolve rates, feedback creation rate, cost-recompute duration.
  • All metrics are dimensioned by the required dimensions plus service and worker, so the platform's own health is sliceable the same way factory telemetry is.

Traces

  • OpenTelemetry traces via ConnectSoft.Extensions.Observability, exported to Application Insights.
  • Inbound traceId/correlationId from the event envelope are continued into worker spans, so processing of a runtime signal is itself part of the original trace — the platform's own work is traceable end to end.
  • Span attributes carry the required dimensions, enabling the platform's own TelemetryCorrelationService to correlate its self-telemetry.

Dashboards

  • Self-monitoring dashboards (App Insights workbooks, optional Grafana) cover ingestion pipeline health, event-flow latency across the workers, store health (App Insights, Log Analytics, Azure SQL, Blob), and the end-to-end loop latency from TraceRecorded to FeedbackItemCreated.
  • These dashboards are defined through the platform's own DashboardService.

Alerts

  • Alert rules (via the platform's own AlertRuleService) watch for: ingestion lag exceeding threshold, dead-letter accumulation, aggregation falling behind, loop latency regressions, and store unavailability.
  • Critical platform alerts page the platform on-call and open incidents through the platform's own IncidentService — the platform reacts to its own problems with its own machinery.

Required Dimensions

The platform both enforces and consumes the required telemetry dimensions:

Dimension Enforced as
traceId OTEL trace context; envelope field; mandatory on ingest.
executionId Span attribute + Serilog enricher.
tenantId Span/log enricher; isolation boundary; rejected if absent.
projectId Span/log enricher; query filter.
moduleId Span/log enricher.
agentId Span attribute (nullable for non-agent runtime).
skillId Span attribute (nullable).
artifactId Span attribute; feedback attribution key.
workflowId Span attribute; ties to Control Plane workflow.
environment OTEL resource attribute.
version OTEL resource attribute.

Telemetry missing traceId or tenantId is rejected at ingestion; the remaining dimensions are validated and back-filled where derivable, and gaps are themselves reported as a data-quality metric.

Consolidated from platforms/generated-saas-platform/observability.md (2026 target design). Technology names inside are profile statements, not requirements.

Target Architecture — Final-State Design

This page describes the observability architecture generated into every SaaS Product. Instrumentation is generated from the standard stack — Serilog for structured logs, OpenTelemetry for metrics and traces, and Azure Application Insights as the backend — and dashboards/alerts are generated as artifacts. The product's signals feed the factory's Observability & Feedback platform.

Every generated product is observable by construction. Each service and worker emits structured logs, metrics, and distributed traces correlated by the canonical event envelope traceId/correlationId. Generated dashboards and alerts make the running product legible to operators, and the same signals flow back to the factory to close the feedback loop.

Logs

  • Structured logging via Serilog with a consistent JSON schema; no unstructured string logs.
  • Log context carries the required dimensions on every entry (tenantId, traceId, correlationId, service, operation).
  • Sinks — Application Insights / Azure Monitor in production; console in development. PII is redacted at the sink per data-classification rules.
  • Correlation — log scopes enrich entries with the ambient traceId so logs join traces and metrics.

Metrics

Metric category Examples
Request request rate, latency (p50/p95/p99), error rate per route + tenant
Messaging events published/consumed, consumer lag, retry count, dead-letter count
Worker messages processed, processing duration, failures per worker
Domain tenants provisioned, subscriptions activated, notifications sent, reports generated
Resource CPU, memory, connection-pool usage, cache hit ratio
Business active tenants, active users, edition distribution, metered usage

Metrics are emitted via OpenTelemetry instruments and tagged with the required dimensions below.

Traces

  • Distributed tracing via OpenTelemetry spans across the gateway, services, message consumers, and external calls.
  • Trace propagation — traceId/correlationId flow from the inbound request through synchronous hops and into published events, so an async side effect (e.g. a notification) shares the originating trace.
  • Span attributes include tenantId, eventType, operation, and outcome.

Required dimensions

Every signal (log, metric, trace) carries these dimensions so the factory can correlate and slice consistently:

Dimension Source Purpose
tenantId token / envelope Per-tenant slicing and isolation analysis
traceId envelope / OTEL End-to-end correlation
correlationId envelope Workflow/saga correlation
service / moduleId runtime Service attribution
eventType envelope Event-level analysis
environment runtime Env separation (dev/staging/prod)
version build Release correlation

Dashboards

Generated dashboards (from observability dashboard templates) give operators a ready-made view per product:

  • Product Health — request rate, error rate, latency, dependency health, by service and tenant.
  • Messaging & Workers — topic throughput, consumer lag, retries, dead-letter trends.
  • Tenant Activity — active tenants/users, onboarding funnel, edition distribution.
  • SaaS Spine — subscriptions, billing health, feature-flag rollout, notification delivery, report generation.
  • Security — auth failures, authorization denials, rate-limit hits, audit volume.

These dashboards are the product-side counterpart to the factory's Observability & Feedback dashboards; product signals are exported upstream.

Alerts

Alert Condition Action
High error rate error rate > threshold over window, per service/tenant Page on-call; link to trace
Latency SLO breach p99 latency > SLO Notify; autoscale evaluation
Consumer lag Service Bus lag > threshold Scale workers; investigate poison
Dead-letter spike DLQ growth > threshold Operator replay workflow
Provisioning failure TenantProvisioningTimedOut / failure rate Notify onboarding owner
Auth failure spike auth failures > baseline Security review
Resource saturation CPU/memory/connection pool > threshold Autoscale; capacity review

How observability contributes to the pillars

  • Traceability — traceId/correlationId on every signal make any request reconstructable end to end and back to factory intent.
  • Reusability — instrumentation, dimensions, dashboards, and alerts are generated identically across products.
  • Autonomy — agents generate the observability artifacts; SRE-style agents can act on alerts.
  • Governance — security metrics and audit volume are observable; PII redaction enforces classification.
  • Observability — this is the observability pillar realized in the generated product and fed back to the factory.
  • Multi-tenant scale — tenantId on every signal enables per-tenant SLOs, cost attribution, and noisy-neighbor detection.