AIFACTORY-FEAT-019: KnowledgeIngestionService: thin delivery for the first outcome¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Knowledge Platform (PF-04) · Increment I1 Generate a microservice · State elaborated · Route backend-developer · Build order 63 of 185 (Factory code)
KnowledgeIngestionService is the service of bounded context BC-10 (Sources & Ingestion). Owns knowledge sources and collections, immutable source snapshots, the ingestion pipeline (parse, normalize, metadata, chunk, publish to indexes with ACL propagation), incremental sync, staleness and deletion lineage; code, documentation, library and template sources are source types, not services (BC-10). In this increment it is delivered thin: No ingestion; the calls the retrieval path requires answer empty. Its own repository is ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService.
| Delivers | KnowledgeIngestionService (thin): No ingestion; the calls the retrieval path requires answer empty. |
| Stubs | — |
| Realizes | adrs: ADR-0013, ADR-0019 |
| Requirements | FR-07, FUX-09, NFR-03, NFR-16, XL-07, XL-14 |
| Depends on | — |
| Sources | model/services/knowledge/KnowledgeIngestionService.yaml |
Scenarios to trace (13, catalog): SCN-KnowledgeIngestionService-001, SCN-KnowledgeIngestionService-002, SCN-KnowledgeIngestionService-003, SCN-KnowledgeIngestionService-004, SCN-KnowledgeIngestionService-005, SCN-KnowledgeIngestionService-006, SCN-KnowledgeIngestionService-007 ✓, SCN-KnowledgeIngestionService-008, SCN-KnowledgeIngestionService-009, SCN-KnowledgeIngestionService-010, SCN-KnowledgeIngestionService-011, SCN-KnowledgeIngestionService-012, SCN-KnowledgeIngestionService-013
AIFACTORY-US-061: KnowledgeIngestionService answers the chain with its thin contract¶
behavior · persona requester
As a requester, I want KnowledgeIngestionService to answer the retrieval path of the first outcome with a well-formed, empty source scope, so that my run can call its knowledge tools before any source is ingested, and the Sources and Ingestion context has one owner behind a contract that does not change later.
This is the integration PBI of the thin delivery. KnowledgeIngestionService owns the knowledge sources and collections, the immutable source snapshots and the ingestion pipeline of bounded context BC-10. In this increment nothing is ingested: no source is registered, no snapshot is taken, no chunk is published and nothing is embedded. The first outcome needs one thing of this service: the retrieval path must be able to ask which sources and collections a scope may retrieve from and receive a well-formed answer. That path starts at the tools knowledge-search and graph-query, which an agent calls through ToolAdapterService and which VectorMemoryService serves; VectorMemoryService and RepositoryIntelligenceService are the two services whose model lists a required, fail-fast gRPC dependency on this service for the source scope. The thin scope is therefore exactly this: the service hosts the one internal gRPC method Ingestion.GetSourceScope and answers it with an empty source scope (AIFACTORY-US-792), on the repository and the surface of AIFACTORY-US-791. Nothing else is served. The six REST operations of the model, RegisterSource, ListSources, SyncSource, DeleteSource, GetSnapshot and SetCollection, the six events of the two aggregates, the workers SourceSyncScheduler, IngestionPipeline and DeletionPropagator, the nine consumed events and the five tables arrive with feature knowledge-ingestion-service-complete in I5. Because no chunk is published, nothing is embedded in this increment; when the complete delivery embeds, it does so through ModelGateway.Embed of ModelRouterService with the model profile embedding, whose Draft release is seeded in I1 with the other profiles. The thin service calls no dependency: none of IntegrationService, SecretBrokerService, ArtifactService, PolicyEngineService and ProjectService is called, because no source exists whose content, credential, classification or scope would have to be resolved. The repository is scaffolded from the pinned base template package with UseFactoryProfile, so token validation, the scope chain with its 401, 403 and 404 answers, health, readiness and the telemetry wiring come from the profile and are not written again. The model names execution-scope-v1 as the request and as the reply descriptor of the method. The proto request message carries the execution scope, and the reply message carries the same scope with a sourceScope that names no repository and no document collection; a contract test with golden files compares both messages with docs/reference/schemas/execution-scope.schema.json, so callers run against this service without a fake and the complete delivery fills the same reply without a change of the messages. Two points are reported to the owner. The summary of the method promises the ACL scopes of the sources, and the schema has no member for them and allows no additional property; an empty answer carries none, so the conflict waits for the complete delivery. And the Studio screen knowledge-sources of feature studio-intelligence in I3 names RegisterSource and ListSources, which the thin delivery does not serve and the complete delivery brings only in I5. A caller without a workload identity is refused before any handler runs. What this PBI proves beyond the suites of the two other PBIs is the call on a deployed instance: the typed client that VectorMemoryService builds from the golden proto calls the deployed thin service over the network with a workload token of the identity authority of the acceptance environment and receives the empty source scope. Feature vector-memory-service-thin depends on this feature and proves its own retrieval call against the same deployed service, so this feature does not depend on it in turn. Repository: ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService.
Acceptance criteria
| AC1 | Given the repository ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService scaffolded with dotnet new connectsoft-base -n ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService --UseFactoryProfile true from the pinned template package, when its pipeline runs, then the solution builds, the unit and acceptance tests pass with complete reports and a container image is published (ADR-0028, model/services/knowledge/KnowledgeIngestionService.yaml). |
| AC2 | Given the published image of the thin service deployed in the acceptance environment and the typed client that VectorMemoryService builds from the golden proto of AIFACTORY-TASK-066, with a workload token that the identity authority of that environment issued with the service scope and that forwards a tenant and a project, when the client calls KnowledgeIngestionService/Ingestion.GetSourceScope on the deployed instance over the network, then the reply is the execution-scope-v1 of the request, valid against docs/reference/schemas/execution-scope.schema.json, with a sourceScope whose repositories and document collections are present and empty, which is the answer the retrieval call of the thin VectorMemoryService needs to return an empty, well-formed result; that call against this deployed service is proven by feature vector-memory-service-thin, which depends on this one. |
| AC3 | Given the service descriptor of the thin delivery, when the contract test compares it with the hosted surface, then exactly KnowledgeIngestionService/Ingestion.GetSourceScope is hosted, no REST operation answers, the OrleansMessaging section declares no published event and no subscription, no worker is hosted, no table of model/services/knowledge/KnowledgeIngestionService.yaml exists in knowledge-ingestion-sql, and the test fails when an operation outside the thin scope appears. |
| AC4 | Given a call without a workload identity or with a token that lacks the service scope, when it reaches KnowledgeIngestionService/Ingestion.GetSourceScope, then it is refused before any handler runs and nothing is read (ADR-0015); given the service starts, when liveness and readiness are probed, then liveness answers and readiness is unready until the store is migrated, the silo has started and the identity authority is reachable. |
| Traces: services | KnowledgeIngestionService |
| Traces: apis | KnowledgeIngestionService/Ingestion.GetSourceScope |
| Traces: descriptors | execution-scope-v1 |
| Traces: adrs | ADR-0015, ADR-0028 |
| Depends on | AIFACTORY-US-791, AIFACTORY-US-792, feature:component-base-template-factory-profile, feature:component-base-template-findings |
| Sources | model/services/knowledge/KnowledgeIngestionService.yaml, model/backlog/features/knowledge.yaml, docs/reference/schemas/execution-scope.schema.json, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-066 Contract of the thin surface: service descriptor and proto of Ingestion.GetSourceScope against execution-scope-v1 |
contract | The service descriptor file of the repository lists exactly Ingestion.GetSourceScope, internal, without streaming and of class read. The proto fixes the request message, which carries the execution scope, and the reply message, which carries the same scope with a sourceScope whose repositories and documentCollections are empty; a contract test with golden files compares both messages with docs/reference/schemas/execution-scope.schema.json, and the contract test of the Factory profile compares the hosted endpoints with the service descriptor file in both directions, so callers run against this service without a fake and the complete delivery adds operations and fills the reply without changing the two messages. | ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService |
AIFACTORY-TASK-763 Verify knowledge-ingestion-service-thin-contract by running the declared command |
verification | Run dotnet build -c Release and dotnet test -c Release --no-build in the pipeline of the repository, and dotnet test -c Release --filter "Category=Acceptance&Feature=ThinSourceScopeDeployed" against the instance deployed in the acceptance environment, where the typed client built from the golden proto of AIFACTORY-TASK-066 calls Ingestion.GetSourceScope over the network with a workload token of the identity authority of that environment; a run that discovers no test and a zero-match filter count as failures; the runs pass and prove AC1 to AC4 of AIFACTORY-US-061, that is the scaffold and the published image, the empty source scope of SCN-KnowledgeIngestionService-007 answered by the deployed instance to the client VectorMemoryService builds, the thin surface with no REST operation, no event, no worker and no table of the model, the golden files of the request and reply messages, readiness and the refusal of a caller without workload identity. |
ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService |
AIFACTORY-US-791: KnowledgeIngestionService repository: scaffolded from the base template with the Factory profile, hosting one gRPC method and nothing else of the model¶
enabling
KnowledgeIngestionService has its own repository, ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService, its own deployable and its own store (ADR-0028). The repository is created and scaffolded in I0 by AIFACTORY-TASK-064 of feature repository-and-pipeline-conventions (D140), by the steps of its repository checklist, and this feature starts from that scaffold; its first commit is the unchanged output of dotnet new connectsoft-base -n ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService --UseFactoryProfile true, generated from the template package at the version the gate of feature component-base-template-factory-profile published, with the messaging service name knowledge-ingestion, the default of OrleansMessaging:ServiceName in the model. On that scaffold the thin delivery adds one thing and leaves out the rest on purpose. The service descriptor file lists exactly one operation, the internal gRPC method Ingestion.GetSourceScope, without streaming and of class read, behind the workload policy of the service scope that names KnowledgeIngestionService; the contract test of the profile compares the hosted endpoints with that file in both directions, so no REST route of the model answers. The store knowledge-ingestion-sql holds only what the scaffold of the profile creates for the messaging profile; the tables KnowledgeSources, Collections, SourceSnapshots, Documents and Chunks are created by the complete delivery. The OrleansMessaging section declares no published event and no subscription, so nothing is written to the outbox and no event of another service reaches a handler, and no worker is hosted. The options DefaultChunkTokens, StalenessHours and MaxSnapshotSizeMb of the section Knowledge:Ingestion are not bound, because nothing reads them; OrleansMessaging:ServiceName is bound. The one secret of the model, messaging-transport, reaches the service as a secret reference of the key vault, and the service is unready while it cannot be resolved. Telemetry is the wiring of the profile: the activity and the log record of a handled call carry tenant and project, and run, step and attempt when the forwarded scope carries a run. The five instruments of the model, the source and snapshot counts, the stage duration, the pending deletions and the index lag, measure an ingestion that does not exist, so the thin delivery creates none of them and no series reports a zero that would read as a healthy pipeline. Readiness is unready until the store is migrated, the silo has started and the identity authority is reachable.
Acceptance criteria
| AC1 | Given the repository created and scaffolded in I0 by AIFACTORY-TASK-064 of feature repository-and-pipeline-conventions, whose first commit is the unchanged output of dotnet new connectsoft-base -n ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService --UseFactoryProfile true from the template package at its published version with the messaging service name knowledge-ingestion, when the pipeline of the repository runs with the work of this feature, then the solution builds, the generated tests pass with complete reports and a container image is published (ADR-0028). |
| AC2 | Given the service descriptor file and the hosted surface, when the contract test of the profile runs, then exactly KnowledgeIngestionService/Ingestion.GetSourceScope is hosted as an internal gRPC method of class read behind the workload policy, a call to the route of any REST operation of model/services/knowledge/KnowledgeIngestionService.yaml finds no endpoint, and the test fails on an endpoint the file does not name. |
| AC3 | Given the migrated knowledge-ingestion-sql and the configuration of the service, when they are inspected, then none of the tables KnowledgeSources, Collections, SourceSnapshots, Documents and Chunks exists, the OrleansMessaging section declares no published event and no subscription, no worker is hosted and no option of Knowledge:Ingestion is bound (model/backlog/features/knowledge.yaml). |
| AC4 | Given a start without a resolvable secret reference for messaging-transport, when readiness is probed, then the service is unready and names the reference; and given a handled call, when the collected telemetry is read, then its activity and its log record carry tenant and project and none of the five instruments of model/services/knowledge/KnowledgeIngestionService.yaml exists (ADR-0019). |
| Traces: services | KnowledgeIngestionService |
| Traces: apis | KnowledgeIngestionService/Ingestion.GetSourceScope |
| Traces: adrs | ADR-0019, ADR-0028 |
| Depends on | feature:component-base-template-factory-profile, feature:component-base-template-findings, feature:repository-and-pipeline-conventions, feature:component-observability-factory-conventions |
| Sources | model/services/knowledge/KnowledgeIngestionService.yaml, model/backlog/features/knowledge.yaml, docs/planes/knowledge-memory/services/knowledge-ingestion-service/index.md#apis, docs/architecture/adr/0028-one-repository-and-deployable-per-service.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-3180 Thin host of KnowledgeIngestionService with one gRPC method, no messaging declaration, no worker and no ingestion instrument |
implementation | The host maps Ingestion.GetSourceScope behind the workload policy of the service scope and no REST route; the OrleansMessaging section holds the service name knowledge-ingestion and declares no published event and no subscription; no worker is registered; the migrations create no table of model/services/knowledge/KnowledgeIngestionService.yaml; the options of Knowledge:Ingestion are not bound; messaging-transport is read as a secret reference and a missing reference keeps the service unready; the enrichment registration of the pinned observability libraries is called once and none of the five instruments of the model is created. | ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService |
AIFACTORY-TASK-3181 Verify the repository and the thin surface of KnowledgeIngestionService |
verification | Run dotnet build -c Release and dotnet test -c Release --filter "Category=Acceptance&Feature=ThinSurface"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-791: the unchanged scaffold, the surface compared with the service descriptor in both directions, the absent tables, subscriptions, workers and instruments of model/services/knowledge/KnowledgeIngestionService.yaml, the unready start without the secret reference and the tenant and project on the activity and the log record of a call. |
ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService |
AIFACTORY-US-792: Answer the retrieval path with a well-formed, empty source scope for any scope of the tenant of the caller¶
behavior · persona requester
As a requester, I want a knowledge search of my run to learn that no source is registered yet, as an empty answer and not as a failure, so that an agent continues with what it has and never mistakes an empty knowledge base for a broken one.
The retrieval path asks this service which sources and collections a scope may retrieve from. The callers the model lists are VectorMemoryService, for the source scope and the ACL of a caller, and RepositoryIntelligenceService, for the source scope of a snapshot; each calls the internal gRPC method Ingestion.GetSourceScope, without streaming and of class read, with a workload token. Such a token carries service scopes and no tenant claim, so the service reads the tenant and the project the caller forwards only when the token holds the service scope that names KnowledgeIngestionService (ADR-0015). The request is an execution-scope-v1: tenant, project, caller, caller kind and policy context. The tenant and the project of the answer are those of the trusted scope: a request whose body names another tenant or project than the forwarded scope is refused with INVALID_ARGUMENT, so no caller can ask for the sources of another tenant, and because the thin delivery addresses nothing by identifier it has no not-found answer. The reply is the execution-scope-v1 of the request with a sourceScope whose repositories and documentCollections are present and empty. That is the honest answer of this increment: no source is registered and no collection is defined, so a scope may retrieve from nothing. It is never an error and never an absent member, so that VectorMemoryService returns an empty, well-formed search result instead of failing the tool call, and an agent learns that nothing was found, not that knowledge is broken (XL-07). The member toolIds of sourceScope is no statement of this service and is returned as the request carried it. The method writes nothing and publishes nothing, and the same request receives the same reply. A call without a workload identity is refused with UNAUTHENTICATED, a token without the service scope with PERMISSION_DENIED, and a request that lacks a member the schema requires, tenant, project, caller, caller kind or policy context, with INVALID_ARGUMENT. The method reads no store and calls no dependency, so no outage of another service can fail it. The one failure it has is the one of the profile: while the signing keys of the identity authority cannot be loaded, the service refuses calls and reports the authority in its readiness, and the callers, for which the dependency is fail-fast, fail their own call; when the keys load again the method answers without a restart.
Acceptance criteria
| AC1 | Given a workload token that holds the service scope and forwards a tenant and a project, and a valid execution-scope-v1 request for that scope, when KnowledgeIngestionService/Ingestion.GetSourceScope is called, then the reply is the execution-scope-v1 of the request with a sourceScope whose repositories and documentCollections are present and empty, and nothing is written or published (SCN-KnowledgeIngestionService-007). |
| AC2 | Given scopes of two tenants and of several projects, when each is asked through KnowledgeIngestionService/Ingestion.GetSourceScope, then each reply names its own tenant and project and the same empty source scope, and the same request asked twice receives the same reply, because the thin scope of model/backlog/features/knowledge.yaml ingests nothing (SCN-KnowledgeIngestionService-007, XL-07). |
| AC3 | Given a caller without a workload identity, a token without the service scope, a request without a member the schema requires, or a request whose body names another tenant than the forwarded scope, when KnowledgeIngestionService/Ingestion.GetSourceScope is called, then it is refused with UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT and INVALID_ARGUMENT respectively, and no source scope is returned (SCN-KnowledgeIngestionService-007, ADR-0015). |
| AC4 | Given the signing keys of the identity authority cannot be loaded, when KnowledgeIngestionService/Ingestion.GetSourceScope is called, then the call is refused and readiness names the authority, and once the keys load again the method answers without a restart (SCN-KnowledgeIngestionService-007). |
| Traces: services | KnowledgeIngestionService |
| Traces: apis | KnowledgeIngestionService/Ingestion.GetSourceScope |
| Traces: descriptors | execution-scope-v1 |
| Traces: adrs | ADR-0015 |
| Traces: requirements | XL-07 |
| Traces: scenarios | SCN-KnowledgeIngestionService-007 |
| Depends on | AIFACTORY-US-791, feature:component-base-template-factory-profile |
| Sources | model/services/knowledge/KnowledgeIngestionService.yaml, model/services/knowledge/VectorMemoryService.yaml, model/services/knowledge/RepositoryIntelligenceService.yaml, docs/reference/schemas/execution-scope.schema.json |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-065 Implement the thin scope of KnowledgeIngestionService for the first outcome |
implementation | Ingestion.GetSourceScope takes the request message of AIFACTORY-TASK-066, reads the forwarded tenant and project only under the service scope, refuses a request that lacks a required member of execution-scope-v1 or whose body names another tenant or project than the forwarded scope with INVALID_ARGUMENT, and returns the scope of the request with a sourceScope whose repositories and documentCollections are present and empty and whose toolIds are as received; it answers UNAUTHENTICATED and PERMISSION_DENIED as the PBI states, reads no store, calls no dependency and writes nothing. No ingestion is built; the contracts match the model, so the complete delivery replaces the thin one without a contract change. | ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService |
AIFACTORY-TASK-3182 Scope requests of two tenants and workload tokens with and without the service scope |
fixture | The acceptance suite holds execution-scope-v1 requests of two tenants and several projects valid against docs/reference/schemas/execution-scope.schema.json, one request per missing required member, one whose body names another tenant than the forwarded scope, workload tokens with and without the service scope naming KnowledgeIngestionService, and an identity authority double whose signing keys can be withheld and restored. | ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService |
AIFACTORY-TASK-3183 Verify answer-an-empty-source-scope-to-the-retrieval-path through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-KnowledgeIngestionService-007"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-792 over the requests of two tenants, the repeated request, every refusal and the withheld and restored signing keys. |
ConnectSoft.AI.SoftwareFactory.KnowledgeIngestionService |