Storage¶

Target Architecture — Final-State Design

The storage decomposition below is the planned data layer of the ConnectSoft.Factory.Templates.* services. Relational metadata uses ConnectSoft.Extensions.PersistenceModel.NHibernate over Azure SQL / PostgreSQL; payloads, packages, and sources use Azure Blob, Azure Artifacts, and Git respectively; hot data is cached in Redis. Infrastructure is provisioned with Pulumi.

The platform deliberately uses purpose-fit stores: structured catalog metadata in a relational database, large generator/scaffold payloads in blob storage, distributable packages in the NuGet feed, immutable template sources in Git, and hot lookups in a cache. This separation keeps the catalog query-fast while letting payloads and packages scale independently, and it makes every artifact independently versioned and traceable.

Storage map¶

Data	Store	Owner Service	Access Pattern	Retention	Notes
Template & library registry metadata (identities, versions, schemas, rules, executions, publications, upgrade plans)	Azure SQL / PostgreSQL	TemplateRegistryService, TemplateVersionService, LibraryRegistryService, LibraryVersionService, et al.	Transactional read/write via NHibernate; tenant-scoped queries	Indefinite (system of record); soft-delete via status	Relational, multi-tenant by `tenantId`; primary catalog
Template version payloads (packaged generator content)	Azure Blob Storage	TemplateVersionService	Write-once on version create; read on execute	Indefinite for `Published`; `Draft` payloads GC'd after TTL	Immutable per version; content-addressed; referenced by `payloadBlobRef`
Scaffold outputs (generated solution trees)	Azure Blob Storage	ScaffoldEngineService	Write-once on generation; read for diff/re-scaffold	Per-tenant policy (e.g. 90 days hot, then archive tier)	Immutable; `contentHash` verifiable; also committed to Git
Reusable packages (NuGet binaries)	Azure Artifacts (NuGet)	PackagePublisherService	Push on publish; restore on build	Indefinite for `Published`; per-feed retention for prerelease	Standard NuGet immutability per version; consumed by DevOps builds
Template sources	Git	TemplatePublishingService, ScaffoldEngineService	Read on execute; tag on publish; commit scaffold output	Indefinite; full history	`sourceRef`/`gitCommitRef`; tags mark published versions
Catalog & compatibility cache (template/library lookups, resolved dependency closures, compatibility matrices)	Redis	TemplateRegistryService, LibraryRegistryService, CompatibilityMatrixService, DependencyAnalyzerService	Read-through cache; short TTL; invalidated on publish	Ephemeral (TTL-bound)	Accelerates pre-generation checks; authoritative copy remains in Azure SQL
Search index (capabilities, tags)	Knowledge Platform index / ElasticSearch	LibraryIndexingWorker	Write on index; query on search	Rebuildable from registry	Powers capability-based `POST /libraries/search`; replicated, not source of truth
Usage analytics (adoption, reuse metrics)	Azure SQL + Redis (counters)	TemplateAnalyticsService	Append + aggregate	Rolling window per policy	Feeds Marketplace ranking and Observability dashboards

Topology¶

flowchart TB
    subgraph Relational["Azure SQL / PostgreSQL"]
        Meta["Registry metadata<br/>(NHibernate)"]
    end
    subgraph Blob["Azure Blob Storage"]
        Payload["Template payloads"]
        Output["Scaffold outputs"]
    end
    Artifacts["Azure Artifacts (NuGet)"]
    GitStore["Git source registry"]
    Cache["Redis cache"]
    Index["Search index"]

    TRS["TemplateRegistryService"] --> Meta
    TRS --> Cache
    TVS["TemplateVersionService"] --> Meta
    TVS --> Payload
    SES["ScaffoldEngineService"] --> Output
    SES --> GitStore
    LRS["LibraryRegistryService"] --> Meta
    LRS --> Cache
    LRS --> Index
    PPS["PackagePublisherService"] --> Artifacts
    CMS["CompatibilityMatrixService"] --> Meta
    CMS --> Cache
    DAS["DependencyAnalyzerService"] --> Cache

Hold "Alt" / "Option" to enable pan & zoom

Design rationale¶

Relational catalog (Azure SQL / PostgreSQL) — the registry metadata is highly relational (templates → versions → schemas → rules; packages → versions → capabilities → dependencies) and transactional, fitting NHibernate over a relational engine. PostgreSQL is the open option; Azure SQL the managed default. Every table carries tenantId for multi-tenant isolation.
Blob for payloads — generator payloads and scaffold outputs are large, immutable, and content-addressed; blob storage is cheaper and scales independently of the catalog. Lifecycle policies move cold scaffold outputs to archive tiers.
Azure Artifacts for packages — reusable libraries are distributed as NuGet packages, so the natural store is the same feed DevOps restores from, giving immutability and standard tooling.
Git for sources — template sources live in Git for full history, diffing, and review; published versions are tagged, and scaffold outputs are committed to product repositories for the GitOps pipeline.
Redis for hot paths — catalog discovery and pre-generation compatibility checks are read-heavy and latency-sensitive; Redis caches resolved closures and matrices with short TTLs, invalidated on publish, while Azure SQL stays authoritative.

Multi-tenancy, retention, and governance¶

Tenant isolation — all relational access is filtered by tenantId via ConnectSoft.Extensions.Saas.NHibernate; blob and artifact paths are tenant-prefixed; cache keys are tenant-scoped.
Immutability — published versions, payloads, packages, and scaffold outputs are immutable, guaranteeing reproducible generation and trustworthy provenance for artifact metadata.
Retention — system-of-record metadata is retained indefinitely (soft-deleted via status); draft payloads and scaffold outputs follow tenant lifecycle policies; analytics use rolling windows.
Backup & DR — Azure SQL/PostgreSQL geo-redundant backups, blob geo-replication, and Git remotes provide durability; the cache and search index are rebuildable from the authoritative stores.
Provisioning — all storage resources are declared as Pulumi infrastructure-as-code and deployed through the DevOps & GitOps platform.