Data & Event plane¶
Target
This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.
Different classes of data are separated conceptually and owned by the plane whose state they hold. No single database is responsible for every concern. Events are the integration style between planes; the port is ConnectSoft.Extensions.MessagingModel and the transport is a deployment profile (ADR-0012).
Store classes¶
| Class | Holds | Authoritative owner | Notes |
|---|---|---|---|
| Operational relational store | Runs, steps, attempts, leases, approvals, releases, projects, policies, usage | Control plane modules | NHibernate (ADR-0006); transition + outbox in one transaction; tenant predicate on every shared-DB query (canonical ADR-0100) |
| Worker durable store | Framework checkpoints, execution/tool receipt ledger | Runtime adapter | Separate schema ownership from Control plane tables even on one server |
| Cache / coordination | Ephemeral locks, projections, session state | Owning module | Redis where selected; never authoritative |
| Artifact and file storage | Immutable content, logs, generated files | Knowledge & Memory (BC-12) | Content-addressed references; scoped read/upload |
| Vector store | Embeddings | Knowledge & Memory (BC-11) | Derived, rebuildable; profile per ADR-0013 |
| Graph store | Relationships, temporal facts, provenance | Knowledge & Memory | Separate domains; federated query |
| Event bus | Domain and integration events | Messaging profile | Outbox/inbox semantics per profile |
| Audit store | Immutable audit history | Trust & Operations (BC-19) | Append-only, tamper-evident, retention per policy |
| Analytics warehouse | Usage, cost, BI | Trust & Operations (FinOps) | Projection, never a source of truth |
Event architecture¶
- Envelope: versioned; required event id, schema version, occurred-at, trusted tenant and project, run/step/attempt where applicable, correlation, causation, producer, type (ADR-0014). Trace id is diagnostic, not authority.
- Event families: agent and version published; run started/completed/failed/cancelled; tool called/failed; approval requested/granted/rejected/expired; skill published; knowledge ingested/updated/deleted; model failed/evaluated; budget exceeded; deployment started/completed.
- Delivery: consumers deduplicate on event id within a stated window; idempotency per trusted scope and operation; no exactly-once external effect is promised; uncertain outcomes are recorded and reconciled.
- Replay: not promised; an append-oriented transition journal is required for recovery; event sourcing is a separate decision.
Current implementation¶
- Port:
IEventBuswithSendCommand/PublishEvent,ICommand,IEvent, handlers (1.0.16). Adapters: Orleans (outbox dispatcher and inbox grains, NHibernate outbox writer, SQL schema, 33 test files; approved by DerechHaLev 2026-07-20), MassTransit (RabbitMQ/ASB/SQL Server/PostgreSQL transports, options only), NServiceBus (uses its own outbox). - Factory events:
AgentEventEnvelope(event id, time, agent id, trace id, blueprint id, origin, workflow step) published through the MassTransit publisher only; no tenant/run identity; noCancelledstage. - Stores in templates: SQL Server/MySQL/PostgreSQL migrations, Hangfire storage, Orleans AdoNet/Azure Tables, Audit.NET SQL, file-system storage abstraction, in-memory vector store. None is deployed for the Factory.
Existing pages that belong to this plane¶
- Event-Driven Architecture, Data Architecture
- Catalogs: events, storage, aggregate roots (a Plane column replaces Platform)
- Reference: event envelope, metadata schema (revised to ADR-0014)
- Per-plane
events.md,storage.md,aggregate-roots.md,data-model.mdpages stay with their owning plane
Later capabilities¶
Analytics warehouse, graph store selection, multi-region data placement, formal retention and residency profiles (Q-09, Q-10).
Platform design pages (2026-10-09, design plan P9)¶
- Data & Event narrative: overview · workflows · UI surfaces · security · observability · deployment · later capabilities
- Service design pages: DataEventFoundations (host) · EnvelopeRegistryService · StoreProfileService · RetentionResidencyService
Modernization (2026-10-09, design plan M1)¶
Horizon now to next, per the technology radar: CloudEvents as an optional envelope profile over the MessagingModel port; topic and schema registry with compatibility; retention and residency as enforceable orders. Exists in ConnectSoft today: Port and three adapters exist; no registry. Each platform page of this plane states its horizon in its overview and later-capabilities pages.
Platforms in this plane¶
| Platform | Bounded contexts | Services | Modelled | Pages |
|---|---|---|---|---|
| Data & Event Platform (PF-11) | BC-42 | 4 | 4 | overview · services · APIs · events |
Bounded contexts¶
| Context | Platform | Authoritative state |
|---|---|---|
| BC-42 Data & Event Foundations | Data & Event Platform | Store profiles, envelope and topic registry, outbox/inbox schemas, retention and residency rules |
Services¶
| Service | Platform | Context | Granularity | Status | Page |
|---|---|---|---|---|---|
| DataEventFoundations | Data & Event Platform | BC-42 | standalone host | target | design |
| EnvelopeRegistryService | Data & Event Platform | BC-42 | module of host | target | design |
| StoreProfileService | Data & Event Platform | BC-42 | module of host | target | design |
| RetentionResidencyService | Data & Event Platform | BC-42 | module of host | target | design |
Events crossing the plane boundary¶
| Direction | Event | Producer | Consumer | Topic |
|---|---|---|---|---|
| out | TopicRegistered | EnvelopeRegistryService | StudioBff | data-foundations.topics.v1.topic-registered |
| out | TopicRegistered | EnvelopeRegistryService | AuditService | data-foundations.topics.v1.topic-registered |
| out | SchemaVersionPublished | EnvelopeRegistryService | StudioBff | data-foundations.topics.v1.schema-version-published |
| out | SchemaVersionPublished | EnvelopeRegistryService | AuditService | data-foundations.topics.v1.schema-version-published |
| out | TopicDeprecated | EnvelopeRegistryService | StudioBff | data-foundations.topics.v1.topic-deprecated |
| out | TopicDeprecated | EnvelopeRegistryService | AuditService | data-foundations.topics.v1.topic-deprecated |
| out | EnvelopeProfileActivated | EnvelopeRegistryService | StudioBff | data-foundations.envelopes.v1.envelope-profile-activated |
| out | EnvelopeProfileActivated | EnvelopeRegistryService | AuditService | data-foundations.envelopes.v1.envelope-profile-activated |
| out | StoreProfileActivated | StoreProfileService | StudioBff | data-foundations.stores.v1.store-profile-activated |
| out | StoreProfileActivated | StoreProfileService | AuditService | data-foundations.stores.v1.store-profile-activated |
| out | StoreBindingChanged | StoreProfileService | StudioBff | data-foundations.stores.v1.store-binding-changed |
| out | StoreBindingChanged | StoreProfileService | AuditService | data-foundations.stores.v1.store-binding-changed |
| out | RetentionRuleChanged | RetentionResidencyService | StudioBff | data-foundations.retention.v1.retention-rule-changed |
| out | RetentionRuleChanged | RetentionResidencyService | AuditService | data-foundations.retention.v1.retention-rule-changed |
| out | RetentionRuleChanged | RetentionResidencyService | KnowledgeHost | data-foundations.retention.v1.retention-rule-changed |
| out | RetentionRuleChanged | RetentionResidencyService | ControlPlaneApi | data-foundations.retention.v1.retention-rule-changed |
| out | RetentionRuleChanged | RetentionResidencyService | ObservabilityHost | data-foundations.retention.v1.retention-rule-changed |
| out | ResidencyRuleChanged | RetentionResidencyService | StudioBff | data-foundations.residency.v1.residency-rule-changed |
| out | ResidencyRuleChanged | RetentionResidencyService | AuditService | data-foundations.residency.v1.residency-rule-changed |
| out | ResidencyRuleChanged | RetentionResidencyService | ModelRouterService | data-foundations.residency.v1.residency-rule-changed |
| out | DeletionOrderIssued | RetentionResidencyService | ControlPlaneApi | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | AgentRuntimeWorker | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | ModelGatewayHost | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | KnowledgeHost | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | ToolExecutorHost | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | IntegrationHost | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | GovernanceHost | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | ObservabilityHost | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderIssued | RetentionResidencyService | StudioBff | data-foundations.orders.v1.deletion-order-issued |
| out | DeletionOrderCompleted | RetentionResidencyService | FleetIncidentService | data-foundations.orders.v1.deletion-order-completed |
| out | DeletionOrderCompleted | RetentionResidencyService | AuditService | data-foundations.orders.v1.deletion-order-completed |
| out | DeletionOrderCompleted | RetentionResidencyService | StudioBff | data-foundations.orders.v1.deletion-order-completed |
| out | DeletionOrderCompleted | RetentionResidencyService | FactoryCommercialService | data-foundations.orders.v1.deletion-order-completed |
| out | ExportCompleted | RetentionResidencyService | FleetIncidentService | data-foundations.orders.v1.export-completed |
| out | ExportCompleted | RetentionResidencyService | AuditService | data-foundations.orders.v1.export-completed |
| out | ExportCompleted | RetentionResidencyService | StudioBff | data-foundations.orders.v1.export-completed |
| out | ExportCompleted | RetentionResidencyService | StudioNotificationService | data-foundations.orders.v1.export-completed |
| out | LegalHoldChanged | RetentionResidencyService | AuditService | data-foundations.orders.v1.legal-hold-changed |
| out | LegalHoldChanged | RetentionResidencyService | KnowledgeHost | data-foundations.orders.v1.legal-hold-changed |
| out | LegalHoldChanged | RetentionResidencyService | ControlPlaneApi | data-foundations.orders.v1.legal-hold-changed |
| out | LegalHoldChanged | RetentionResidencyService | ObservabilityHost | data-foundations.orders.v1.legal-hold-changed |
| out | LegalHoldChanged | RetentionResidencyService | StudioBff | data-foundations.orders.v1.legal-hold-changed |
Gateways¶
No gateway role is anchored in this plane.
Model coverage¶
| Count | |
|---|---|
| Services in the registry | 4 |
| Modelled (YAML exists) | 4 |
| With a design page | 4 |
| Unmodelled | 0 |