KnowledgeIngestionService

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Bounded context	BC-10 Sources & Ingestion
Platform / plane	Knowledge Platform / Knowledge & Memory
Deployment granularity	module of host · host KnowledgeHost
Status	target · ConnectSoft.Factory.Knowledge.Ingestion
Built from	connectsoft-microservice · libraries: connectsoft-extensions-data-ingestion, connectsoft-extensions-data-storage-abstractions, connectsoft-extensions-saas-abstractions, connectsoft-extensions-messaging-model, connectsoft-extensions-persistence-model-nhibernate, connectsoft-extensions-compliance
Decisions	ADR-0013, ADR-0019
Traceability	FR-07 FUX-09 NFR-03 NFR-16 XL-07 XL-14

Purpose and responsibilities

Knowledge contains the organization's source-of-truth information (original handoff §20): Git repositories, Azure DevOps, GitHub, SharePoint, Confluence, documents, SQL databases, APIs, tickets, architecture documentation, ADRs, runbooks, logs, schemas and MCP resources. KnowledgeIngestionService registers those sources with their classification, ACL mapping and sync policy, takes immutable snapshots at a version, and runs the pipeline: snapshot → parse → normalize → metadata → chunk → full-text and vector indexing (through retrieval) → entity and relationship extraction → ACL propagation → publish. Nothing is retrievable before ACL propagation succeeded.

It absorbs the June-2026 rows CodebaseKnowledgeService, DocumentationKnowledgeService, LibraryKnowledgeService and TemplateKnowledgeService: they are extractors for source types, not services.

Context and boundaries

flowchart LR
    S["KnowledgeIngestionService"]
    VectorMemoryService["VectorMemoryService"] -->|event| S
    RepositoryIntelligenceService["RepositoryIntelligenceService"] -->|event| S
    StudioBff["StudioBff"] -->|sync-http| S
    S -->|grpc| IntegrationService["IntegrationService"]
    S -->|grpc| SecretBrokerService["SecretBrokerService"]
    S -->|grpc| ArtifactService["ArtifactService"]
    S -->|grpc| PolicyEngineService["PolicyEngineService"]
    S -->|grpc| ProjectService["ProjectService"]
    S -. "SourceRegistered" .-> E_SourceRegistered(("event"))
    S -. "SourceStateChanged" .-> E_SourceStateChanged(("event"))
    S -. "SourceDeleted" .-> E_SourceDeleted(("event"))
    S -. "SourceSnapshotTaken" .-> E_SourceSnapshotTaken(("event"))
    S -. "ChunksPublished" .-> E_ChunksPublished(("event"))
    S -. "SnapshotFailed" .-> E_SnapshotFailed(("event"))

Hold "Alt" / "Option" to enable pan & zoom

Boundaries: credentials stay with the Integration platform and the Secret Broker; embeddings and graph extraction run in retrieval with model profiles; artifacts hold the normalized content; deletion lineage (D31) reaches every derived item within 30 days.

Domain model

KnowledgeSource — A registered source with its connection, extractor, ACL mapping, sync policy and classification; owned by a project or shared at tenant level. (catalog entry)

Field	Type	Required	Description
sourceId	guid	yes
tenantId	string	yes
projectId	string	no	Null for a tenant-wide source.
kind	enum:SourceKind	yes
reference	string	yes	Repository URL, site, database or API reference.
connectionId	guid	no	Connection in BC-16 when credentials are needed.
extractor	string	yes	code
classification	string	yes	public
aclMapping	json	yes	How source permissions map to Factory scopes.
syncPolicy	json	yes	interval, webhook, change detection, re-index triggers
state	enum:SourceState	yes
lastSnapshotId	guid	no
version	long	yes

Invariants:

• A source belongs to exactly one tenant and is readable only within the project scope that registered it or tenant-wide when declared so.
• Credentials are never stored here; a source references a connection (BC-16) and the Secret Broker issues short-lived access.
• Deleting a source produces deletion lineage for every derived chunk, vector, graph fact and context snapshot that referenced it (D31).

stateDiagram-v2
    Registered --> Syncing : FirstSync
    Syncing --> Healthy : SnapshotPublished
    Healthy --> Syncing : ChangeDetected
    Healthy --> Stale : StalenessWindowExceeded
    Stale --> Syncing : Resync
    Syncing --> Failing : SyncFailed
    Failing --> Syncing : Retry
    Healthy --> Disabled : Disable
    Disabled --> Deleting : Delete
    Healthy --> Deleting : Delete

Hold "Alt" / "Option" to enable pan & zoom

Events: SourceRegistered, SourceStateChanged, SourceDeleted · Tables: KnowledgeSources, Collections · Repository: KnowledgeSourceRepository

SourceSnapshot — An immutable capture of a source at a version (commit, document revision, export time) that flows through the pipeline; superseded by the next snapshot, never mutated. (catalog entry)

Field	Type	Required	Description
snapshotId	guid	yes
sourceId	guid	yes
tenantId	string	yes
sourceVersion	string	yes	Commit id, revision or export timestamp.
state	enum:SnapshotState	yes
documentCount	int	yes
chunkCount	int	yes
contentRef	string	yes	Blob prefix of the normalized content.
takenAt	datetime	yes
publishedAt	datetime	no

Invariants:

• A snapshot's documents and chunks are immutable; a change in the source produces a new snapshot and supersedes the old one after publication.
• Chunks carry the source version, classification, ACL scope and trust tier SourceFact; indexes are published only after ACL propagation succeeded.

stateDiagram-v2
    Taken --> Parsed : ParseCompleted
    Parsed --> Chunked : ChunkCompleted
    Chunked --> Indexed : IndexesWritten
    Indexed --> Published : AclPropagated
    Published --> Superseded : NextSnapshotPublished
    Taken --> Failed : StageFailed
    Parsed --> Failed : StageFailed
    Chunked --> Failed : StageFailed

Hold "Alt" / "Option" to enable pan & zoom

Events: SourceSnapshotTaken, ChunksPublished, SnapshotFailed · Tables: SourceSnapshots, Documents, Chunks · Repository: SourceSnapshotRepository

erDiagram
    KnowledgeSource {
        guid sourceId PK
        string tenantId
        string projectId
        enum_SourceKind kind
        string reference
        guid connectionId
        string extractor
        string classification
        json aclMapping
        json syncPolicy
        enum_SourceState state
        guid lastSnapshotId
        long version
    }
    SourceSnapshot {
        guid snapshotId PK
        guid sourceId
        string tenantId
        string sourceVersion
        enum_SnapshotState state
        int documentCount
        int chunkCount
        string contentRef
        datetime takenAt
        datetime publishedAt
    }
    Collection {
        guid collectionId

        string name

        string projectId

        list_guid sourceIds

        string embeddingProfileVersion

    }
    Document {
        guid documentId

        string path

        string mediaType

        hash contentHash

        json metadata

        json aclScope

        string classification

    }
    Chunk {
        guid chunkId

        guid documentId

        int ordinal

        hash contentHash

        int tokenCount

        enum_TrustTier trustTier

        json metadata

    }
    KnowledgeSource ||--o{ Collection : contains
    SourceSnapshot ||--o{ Document : contains
    SourceSnapshot ||--o{ Chunk : contains

Hold "Alt" / "Option" to enable pan & zoom

stateDiagram-v2
    [*] --> Taken: snapshot captured at a source version
    Taken --> Parsed
    Parsed --> Chunked
    Chunked --> Indexed: indexes written by retrieval
    Indexed --> Published: ACL propagated
    Published --> Superseded: next snapshot published
    Taken --> Failed
    Parsed --> Failed
    Chunked --> Failed
    Superseded --> [*]
    Failed --> [*]

Hold "Alt" / "Option" to enable pan & zoom

Entities and value objects

Collection (entity, in KnowledgeSource) — A named grouping of sources a project retrieves from (knowledge collection definition declared in the Control plane).

Field	Type	Required	Description
collectionId	guid	yes
name	string	yes
projectId	string	yes
sourceIds	list:guid	yes
embeddingProfileVersion	string	yes	Pins the index to an embedding profile version (BC-07).

Document (entity, in SourceSnapshot) — One normalized document of a snapshot.

Field	Type	Required	Description
documentId	guid	yes
path	string	yes
mediaType	string	yes
contentHash	hash	yes
metadata	json	yes
aclScope	json	yes
classification	string	yes

Chunk (entity, in SourceSnapshot) — A retrieval unit with position, hash and provenance.

Field	Type	Required	Description
chunkId	guid	yes
documentId	guid	yes
ordinal	int	yes
contentHash	hash	yes
tokenCount	int	yes
trustTier	enum:TrustTier	yes
metadata	json	yes

Data model and storage

Store	Role	Engine / profile	Tables
Knowledge metadata store	primary	azure-sql / nhibernate-sqlserver	KnowledgeSources, Collections, SourceSnapshots, Documents, Chunks
Artifact content store	blob	blob / azure-blob	—

Table	Owner	Columns	Indexes
dbo.KnowledgeSources	KnowledgeSource	13	1
dbo.Collections	Collection	6	1
dbo.SourceSnapshots	SourceSnapshot	10	1
dbo.Documents	Document	9	1
dbo.Chunks	Chunk	8	1

Full DDL: appendix.

APIs

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
POST	/factory/knowledge/sources	RegisterSource	factory.knowledge.admin	shared-write	key-header	GW-03	Register a source with kind, reference, extractor, classification, ACL mapping and sync policy (Workspace admin).
GET	/factory/knowledge/sources	ListSources	factory.knowledge.read	read	natural	GW-03	Sources in scope with state, last snapshot and staleness.
POST	/factory/knowledge/sources/{sourceId}/sync	SyncSource	factory.knowledge.admin	shared-write	key-header	GW-03	Take a new snapshot now.
DELETE	/factory/knowledge/sources/{sourceId}	DeleteSource	factory.knowledge.admin	production	natural	GW-03	Delete a source and every derived item with deletion lineage (D31).
GET	/factory/knowledge/snapshots/{snapshotId}	GetSnapshot	factory.knowledge.read	read	natural	GW-03	Pipeline state, counts and stage timings of a snapshot.
PUT	/factory/projects/{projectId}/knowledge/collections/{name}	SetCollection	factory.knowledge.admin	shared-write	natural	GW-03	Define a collection of sources with its embedding profile version.

gRPC service	Method	Streaming	Visibility	Side effects	Summary
Ingestion	GetSourceScope	none	internal	read	Sources and collections a scope may retrieve from, with ACL scopes; used by retrieval and context assembly.

Messages

Event	Topic	Scope / v	Aggregate	Consumers	Summary
SourceRegistered	ingestion.sources.v1.source-registered	integration / 1	KnowledgeSource	StudioBff, AuditService, RepositoryIntelligenceService	A source exists.
SourceStateChanged	ingestion.sources.v1.source-state-changed	integration / 1	KnowledgeSource	StudioBff, DashboardService	Syncing, healthy, stale, failing, disabled or deleting.
SourceDeleted	ingestion.sources.v1.source-deleted	integration / 1	KnowledgeSource	VectorMemoryService, RepositoryIntelligenceService, ContextBuilderService, MemoryPromotionService, AuditService	A source and its derived items were deleted; carries the deletion lineage reference.
SourceSnapshotTaken	ingestion.snapshots.v1.source-snapshot-taken	integration / 1	SourceSnapshot	RepositoryIntelligenceService, DashboardService	An immutable snapshot of a source version was captured.
ChunksPublished	ingestion.snapshots.v1.chunks-published	integration / 1	SourceSnapshot	TemplateRegistryService, VectorMemoryService, DashboardService	Chunks with ACL and provenance are ready for indexing.
SnapshotFailed	ingestion.snapshots.v1.snapshot-failed	integration / 1	SourceSnapshot	StudioBff, DashboardService	A pipeline stage failed; the previous snapshot stays published.

SourceRegistered payload:

Field	Type	Required	Description
sourceId	guid	yes
tenantId	string	yes
projectId	string	no
kind	enum:SourceKind	yes
classification	string	yes

SourceStateChanged payload:

Field	Type	Required	Description
sourceId	guid	yes
tenantId	string	yes
fromState	enum:SourceState	yes
toState	enum:SourceState	yes

SourceDeleted payload:

Field	Type	Required	Description
sourceId	guid	yes
tenantId	string	yes
lineageRef	string	yes

SourceSnapshotTaken payload:

Field	Type	Required	Description
snapshotId	guid	yes
sourceId	guid	yes
tenantId	string	yes
sourceVersion	string	yes
kind	enum:SourceKind	yes

ChunksPublished payload:

Field	Type	Required	Description
snapshotId	guid	yes
sourceId	guid	yes
tenantId	string	yes
chunkCount	int	yes
embeddingProfileVersion	string	yes

SnapshotFailed payload:

Field	Type	Required	Description
snapshotId	guid	yes
sourceId	guid	yes
tenantId	string	yes
stage	string	yes
reason	string	yes

Command	Handled by	Result events	Summary
RegisterSource	KnowledgeSource	SourceRegistered
SyncSource	SourceSnapshot	SourceSnapshotTaken
DeleteSource	KnowledgeSource	SourceDeleted

Event	Producer	Handler	Purpose
SourceScopeChanged	ProjectService	ReconcileProjectSources	Register or disable sources when a project's source scope changes.
ProjectStateChanged	ProjectService	OffboardProjectSources	Delete or export project sources on Closing (XL-13, XL-14).

Workers and jobs

Worker	Trigger	Output events	Retry	Idempotency key	Summary
SourceSyncScheduler	schedule */5 * * * *	SourceSnapshotTaken, SourceStateChanged	exponential	sourceId + sourceVersion	Takes snapshots per sync policy or on webhook change notifications; detects staleness.
IngestionPipeline	event	ChunksPublished, SnapshotFailed	exponential	snapshotId + stage	Runs parse → normalize → metadata → chunk → index publish → ACL propagation for a snapshot; each stage is journaled and resumable.
DeletionPropagator	event	—	exponential	lineageRef + target	Executes deletion lineage across chunks, vectors, graph facts, context snapshots and memory proposals within the 30-day window (D31).

Actors and grains

None modelled

No grains are modelled for this item yet.

Sagas and long-running flows

None modelled

No sagas are modelled for this item yet.

sequenceDiagram
    participant S as Source (Git, SharePoint, …)
    participant I as KnowledgeIngestionService
    participant A as ArtifactService
    participant R as VectorMemoryService
    participant RI as RepositoryIntelligenceService
    I->>S: fetch at version through the connection
    I->>A: store normalized content by hash
    I->>I: parse, normalize, metadata, chunk, ACL
    I-)R: ChunksPublished
    I-)RI: SourceSnapshotTaken (repositories)
    R-)I: IndexRevisionReady
    I->>I: Published, supersede the previous snapshot

Hold "Alt" / "Option" to enable pan & zoom

Options and configuration

Section	Key	Type	Default	Reloadable	Secret	Summary
Knowledge:Ingestion	DefaultChunkTokens	int	512	yes	no	Target chunk size; extractors may override per media type.
Knowledge:Ingestion	StalenessHours	int	72	yes	no	A source without a fresh snapshot is Stale.
Knowledge:Ingestion	MaxSnapshotSizeMb	int	2048	yes	no	Snapshot size limit per source version.

Metrics and observability

Metric	Instrument	Unit	Dimensions	Summary
connectsoft.factory.knowledge.sources	updown	{source}	tenant, kind, state	Sources by state.
connectsoft.factory.knowledge.snapshots	counter	{snapshot}	tenant, kind, outcome	Snapshots taken and their outcome.
connectsoft.factory.knowledge.ingestion_stage_duration	histogram	s	stage	Pipeline stage durations.
connectsoft.factory.knowledge.deletion_pending	updown	{item}	tenant	Deletion lineage items not yet executed.

Security

Authentication	client-credentials
Scopes	factory.knowledge.read (Read sources and snapshots.); factory.knowledge.admin (Register, sync, delete sources; define collections.)
Roles	Requester, AssetMaintainer, Operator, WorkspaceAdmin, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	confidential
Max side-effect class	production
Secrets	—

Dependencies

Outbound

Target	Kind	Purpose	Failure mode	Required
IntegrationService	grpc	Connections and connectors to pull source content.	degrade	yes
SecretBrokerService	grpc	Short-lived source credentials.	fail-fast	yes
ArtifactService	grpc	Store normalized content and chunk payloads by hash.	fail-fast	yes
PolicyEngineService	grpc	Classification and ACL mapping validation.	fail-fast	yes
ProjectService	grpc	Project source scope.	fail-fast	yes

Inbound

Target	Kind	Purpose	Failure mode	Required
VectorMemoryService	event	Chunks to index.	—	yes
RepositoryIntelligenceService	event	Repository snapshots to index.	—	yes
StudioBff	sync-http	Source and collection administration.	—	yes

Failure and recovery

Scenario	Detection	Recovery	Data impact	RTO (min)
Source unreachable	sync failures	source Failing with retries; previous snapshot stays published; staleness surfaces in the Studio	none	60
ACL propagation fails	stage failure before Published	snapshot not published; nothing retrievable from it	none	30
Deletion propagation interrupted	deletion_pending above zero past the window	propagator resumes idempotently; export and deletion evidence recorded	none	1440

NFR targets

Target	Class value (pilot-batch)	Override
latency_p95_ms	0	—
availability_pct	99.5	—
throughput_rps	10	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	medium	—

Deployment

Granularity	module of host · host KnowledgeHost
Topologies	saas-default, hybrid, on-prem, hybrid-compute
Runtime	—; replicas —–—
Ports	—

Current implementation

ConnectSoft.Extensions.DataIngestion wraps Microsoft's data ingestion primitives (chunking, embedding, upsert) with configuration and metrics; no source registry, snapshot, ACL propagation, staleness or deletion lineage exists. ConnectSoft.MediaIngestion is a candidate transcript connector.

Later capabilities

Freshness scoring and drift detection against code and runtime reality; external knowledge connectors for approved standards and vendor documentation; webhook-driven change detection for every source kind.

Decisions and traceability

Family	Ids
FR	FR-07
FUX	FUX-09
NFR	NFR-03, NFR-16
XL	XL-07, XL-14

Related

• Knowledge workflows · Governance · VectorMemoryService · Data governance

• Knowledge Platform microservices · bounded contexts · APIs · events · data model

• Services catalog · Ownership map