Skip to content

AIFACTORY-FEAT-069: FeedbackService: complete delivery

Generated from the architecture model

This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.

A design artefact, not a tracker

This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).

Epic Observability (PF-13) · Increment I4 Trust and accountability · State elaborated · Route backend-developer · Build order 132 of 185 (Factory code)

FeedbackService is the service of bounded context BC-22 (Evaluation & Learning). Human feedback on runs and artifacts, evaluation datasets built from historical tasks and completed runs, and learning proposals (prompt, skill and context-strategy improvements, threshold changes, dataset additions) raised from feedback, drift and reinforced memory for Asset maintainers to review; nothing changes an asset without a release (BC-22, original design §32). Its own repository is ConnectSoft.AI.SoftwareFactory.FeedbackService.

Delivers FeedbackService (complete)
Stubs —
Realizes adrs: ADR-0021
Requirements FR-11, FUX-09, FUX-10, NFR-11, NFR-16, XL-08
Depends on —
Sources model/services/observability/FeedbackService.yaml

Scenarios to trace (11, catalog): SCN-FeedbackService-001 ✓, SCN-FeedbackService-002 ✓, SCN-FeedbackService-003 ✓, SCN-FeedbackService-004 ✓, SCN-FeedbackService-005 ✓, SCN-FeedbackService-006 ✓, SCN-FeedbackService-007 ✓, SCN-FeedbackService-008 ✓, SCN-FeedbackService-009 ✓, SCN-FeedbackService-010 ✓, SCN-FeedbackService-011 ✓

AIFACTORY-US-046: Feedback becomes datasets and proposals that a person decides

behavior · persona asset-maintainer As an asset maintainer, I want the feedback of people and the failures of runs collected into datasets and improvement proposals, so that the next release of an agent is better for a reason I can inspect.

This is the integration PBI of the complete delivery of FeedbackService, the service of bounded context BC-22 that closes the learning loop of model/services/observability/FeedbackService.yaml, in which nothing changes an asset without a release. A requester, reviewer or operator records feedback on a run, an attempt, an artifact, a memory or an evaluation, and the feedback is listed by subject and kind (AIFACTORY-US-1098). Completed runs, validator outcomes and merged changes are harvested as dataset tasks, and an Asset maintainer or Platform admin snapshots a dataset into a new immutable version stored as an artifact and announced to the suites (AIFACTORY-US-1099), which QualityScoreService resolves to its snapshot artifact through an internal method (AIFACTORY-US-1100). The ProposalGenerator worker clusters feedback, drift and failed evaluations per asset and raises learning proposals with evidence and an expected effect, which maintainers list (AIFACTORY-US-1101), and an Asset maintainer accepts a proposal, for which AgentRegistryService opens a Draft release in BC-04 that FeedbackService records from LearningReleaseOpened (D131), or rejects it with a reason (AIFACTORY-US-1102). Retention, deletion and legal hold orders are applied to the records of the service (AIFACTORY-US-1103). Every query runs under the tenant row filter of the service and a resource of another tenant is answered 404, the same answer as for one that does not exist (ADR-0015 amendment D113); tasks derived from tenant runs stay tenant-scoped (D32). Two consumed events of the model come from services delivered after this increment, SkillCandidateProposed of MemoryPromotionService (I5) and IncidentResolved of FleetIncidentService (I7); separately, the optional consent check of MemoryPromotionService for shared datasets also arrives with that service. Both handlers and the consent client are built against the contracts package here and receive their first events when those services arrive, and until then no dataset is shared across tenants. Repository: ConnectSoft.AI.SoftwareFactory.FeedbackService, created by the first task of this feature with the steps of the repository checklist.

Acceptance criteria

AC1 Given a run or an artifact, when a person records feedback on it, then FeedbackService/FeedbackRecorded is published with the tenant and the subject, as AIFACTORY-US-1098 proves (ADR-0021).
AC2 Given a dataset, when a snapshot is taken through FeedbackService/SnapshotDataset, then the snapshot is immutable, stored as an artifact and published as a version through FeedbackService/DatasetPublished, as AIFACTORY-US-1099 proves (ADR-0021).
AC3 Given failed evaluations or detected drift with enough evidence, when the generator runs, then a learning proposal is raised with its evidence through FeedbackService/LearningProposalRaised, as AIFACTORY-US-1101 proves (ADR-0021).
AC4 Given a learning proposal in review, when a maintainer accepts it through FeedbackService/DecideLearningProposal, then FeedbackService/LearningProposalDecided is published and AgentRegistryService opens the release candidate; a rejection is recorded with its reason, as AIFACTORY-US-1102 proves (ADR-0021).
Traces: services FeedbackService, QualityScoreService, AgentRegistryService
Traces: aggregates FeedbackService/HumanFeedback, FeedbackService/Dataset, FeedbackService/LearningProposal
Traces: apis FeedbackService/RecordFeedback, FeedbackService/ListFeedback, FeedbackService/ListDatasets, FeedbackService/SnapshotDataset, FeedbackService/ListLearningProposals, FeedbackService/DecideLearningProposal, FeedbackService/Learning.GetDatasetSnapshot
Traces: events FeedbackService/FeedbackRecorded, FeedbackService/DatasetPublished, FeedbackService/LearningProposalRaised, FeedbackService/LearningProposalDecided
Traces: adrs ADR-0015, ADR-0021
Traces: requirements FR-11, FUX-09, FUX-10, NFR-11, NFR-16, XL-08
Relates: screens feedback, datasets, learning-proposals
Depends on AIFACTORY-US-1098, AIFACTORY-US-1099, AIFACTORY-US-1100, AIFACTORY-US-1101, AIFACTORY-US-1102, AIFACTORY-US-1103, feature:repository-and-pipeline-conventions, feature:component-base-template-factory-profile
Sources model/services/observability/FeedbackService.yaml, docs/architecture/adr/0021-quality-gates.md, docs/planes/trust-operations/observability/services/feedback-service/index.md#purpose-and-responsibilities

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-486 Create the repository ConnectSoft.AI.SoftwareFactory.FeedbackService and scaffold it from the base template with the Factory profile implementation The repository exists, created by the steps of the repository checklist of feature repository-and-pipeline-conventions; its first commit is the unchanged output of dotnet new connectsoft-base -n ConnectSoft.AI.SoftwareFactory.FeedbackService --UseFactoryProfile true from the pinned template package, whose version the repository records; the solution builds and the template tests pass in the repository pipeline, which publishes the container image. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-487 Domain model: HumanFeedback, Dataset, LearningProposal implementation Aggregates with their invariants and the four transitions of ProposalState as modelled; unit tests cover every invariant and every transition, and a transition the model does not list is refused. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-488 Persistence in feedback-sql: HumanFeedback, Datasets, DatasetTasks, LearningProposals implementation Mappings and migrations for the modelled tables with tenant filters and the indexes IX_Feedback_Subject, UX_Datasets_Name_Version, IX_Proposals_Tenant_State and IX_Proposals_Target; mapping tests run against an isolated test database, never a shared one. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-489 API hosting: 6 REST and 1 gRPC operations (RecordFeedback, ListFeedback, ListDatasets, SnapshotDataset, ListLearningProposals, DecideLearningProposal, Learning.GetDatasetSnapshot) implementation The six REST routes and the gRPC method are hosted with JWT validation, scope enforcement per route, workload identity checks on the gRPC method, problem responses for every modelled refusal code and the Idempotency-Key filter of RecordFeedback and SnapshotDataset; the behaviour of each operation is delivered by AIFACTORY-US-1098 to AIFACTORY-US-1102. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-490 Messaging: publishes 4 events, handles 11 implementation Events are published through the outbox of the feedback profile under the run envelope (FeedbackRecorded, DatasetPublished, LearningProposalRaised, LearningProposalDecided); the inbox registers the eleven consumed events idempotently on the event id; what each handler does is delivered by the behaviour PBIs of this feature. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-491 Workers: ProposalGenerator, DatasetSnapshotter implementation Each worker runs with its modelled trigger, retry policy, idempotency key and concurrency partitioned by tenant; a duplicate trigger has no second effect; what each worker does is delivered by AIFACTORY-US-1099 and AIFACTORY-US-1101. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-492 Security, tenancy and telemetry of FeedbackService implementation Scopes and roles of the model are enforced, every query is tenant-filtered and a resource of another tenant is answered 404 (ADR-0015 amendment D113), secrets come from secret references, and traces, metrics and health checks follow the observability conventions with the four modelled metrics, among them the gauge connectsoft.factory.learning.inbox_lag, the age in seconds of the oldest unprocessed inbox message of the service, measured by the inbox of the messaging profile (D140). ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-770 Verify feedback-to-learning-proposal by running the declared command verification Run dotnet test -c Release --filter Category=Acceptance; a zero-match filter counts as a failure; it passes and proves AC1 to AC4 of AIFACTORY-US-046 end to end: feedback recorded and published, a dataset snapshot published as a version, a proposal raised from evaluation and drift evidence, and a proposal accepted and rejected. ConnectSoft.AI.SoftwareFactory.FeedbackService

AIFACTORY-US-1098: Record feedback on a run, attempt, artifact, memory or evaluation and list it by subject and kind

behavior · persona reviewer As a reviewer, I want to rate, correct, comment on or escalate what a run produced, and see the feedback others gave on the same subject, so that what people think of a result is kept as evidence for datasets and proposals without changing the result itself.

A requester, reviewer or operator whose scope carries factory.feedback.write records feedback through RecordFeedback with an Idempotency-Key. The body names the subject kind (run, attempt, artifact, memory or evaluation) and the subject reference, the feedback kind (Rating, Correction, Comment or Escalation), a rating where the kind is Rating, the comment text and optionally a corrected artifact. The tenant and the project come from the trusted scope and givenBy from the token, never from the body (ADR-0015). The comment text is stored as an artifact through ArtifactService, a required dependency that fails fast, and only its reference is kept; feedback references its subject by id and never alters it. The service writes the HumanFeedback row with givenAt and the FeedbackRecorded outbox record with feedbackId, tenantId, subjectKind, subjectRef, kind and rating in one transaction and answers 201 with the feedback location. A repeated call under the same key and body returns the same feedback and changes nothing; a caller without the scope is refused with 403. The model lists only 201 and 403 for this operation, so the answer to a body that fails validation and to a key reused with a different body is an open point reported with this feature; whatever the code, neither writes a row or publishes. When ArtifactService is unreachable the call fails fast and nothing is written, and the caller repeats it under the same key. A caller whose scope carries factory.feedback.read lists feedback through ListFeedback, filtered by subject and kind, under the tenant row filter and the projects in scope; a caller without the scope is refused with 403. The resolved incidents of FleetIncidentService become feedback on the releases involved through the handler HarvestIncident, built here against the contracts package, which receives its first event when that service arrives in I7. The counter connectsoft.factory.learning.feedback moves with each recorded feedback.

Acceptance criteria

AC1 Given a caller with factory.feedback.write and a valid body naming a run, when FeedbackService/RecordFeedback is called with an Idempotency-Key, then the response is 201, the FeedbackService/HumanFeedback row holds tenant and project from the trusted scope, givenBy from the token and the reference of the comment stored through ArtifactService, and the FeedbackService/FeedbackRecorded outbox record is written in the same transaction; the same key and body return the same feedback and change nothing (SCN-FeedbackService-001, ADR-0015).
AC2 Given a caller without factory.feedback.write, a body that fails validation, or a key reused with a different body, when FeedbackService/RecordFeedback is called, then it is refused with a problem response, 403 for the missing scope, and no row is written and nothing is published; given ArtifactService unreachable, then the call fails fast and nothing is written (SCN-FeedbackService-001).
AC3 Given feedback on several subjects in two tenants, when FeedbackService/ListFeedback is called with factory.feedback.read and a subject and kind filter, then the response is 200 with the matching feedback of the caller tenant and projects only; a caller without the scope is refused with 403 and nothing is written (SCN-FeedbackService-002, ADR-0015).
AC4 Given the record was written, when the dispatcher of the feedback profile publishes FeedbackService/FeedbackRecorded on evaluation.feedback.v1.feedback-recorded, then it is sent under the canonical envelope with feedbackId, tenantId, subjectKind, subjectRef, kind and rating, and an outage is retried so it is neither lost nor duplicated under its event id (SCN-FeedbackService-008, ADR-0012).
Traces: services FeedbackService, ArtifactService
Traces: aggregates FeedbackService/HumanFeedback
Traces: apis FeedbackService/RecordFeedback, FeedbackService/ListFeedback
Traces: events FeedbackService/FeedbackRecorded
Traces: adrs ADR-0012, ADR-0015, ADR-0021
Traces: requirements FUX-09, NFR-16
Traces: scenarios SCN-FeedbackService-001, SCN-FeedbackService-002, SCN-FeedbackService-008
Relates: screens feedback
Depends on feature:artifact-service
Sources model/services/observability/FeedbackService.yaml, docs/planes/trust-operations/observability/services/feedback-service/index.md#apis

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-552 Feedback on runs, attempts, artifacts, memories and evaluations implementation RecordFeedback takes tenant, project and author from the trusted scope, stores the comment through ArtifactService, writes HumanFeedback with its FeedbackRecorded outbox record in one transaction and answers 201, repeats under the same key without a second row and fails fast without a write when ArtifactService is unreachable; ListFeedback returns feedback by subject and kind under the tenant row filter; HarvestIncident is registered against the contracts package; the feedback counter moves with each record. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4383 Verify recording and listing feedback through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-FeedbackService-001\|Scenario=SCN-FeedbackService-002\|Scenario=SCN-FeedbackService-008)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1098 with a fake ArtifactService: the recorded feedback and its repeat, the refusals without a write, the fail-fast store, the listing by subject and kind across two tenants, and the publication and payload of FeedbackRecorded through an outage. ConnectSoft.AI.SoftwareFactory.FeedbackService

AIFACTORY-US-1099: Harvest dataset tasks from runs, validations and merged changes and snapshot a dataset into an immutable published version

behavior · persona asset-maintainer As an asset maintainer, I want datasets of historical tasks built from what the Factory actually ran, and a new immutable version when I snapshot one, so that suites evaluate releases on real tasks of my tenant and every evaluation names the exact data it used.

Datasets are built from completed runs, validator outcomes, merged changes and curated additions. RunCompleted of RunControlService (handler HarvestTask) makes a completed run a dataset task candidate with its source run and the references of its input and outcome; ArtifactValidated of ArtifactService (handler HarvestValidation) enriches a candidate with the validator verdict; ChangeProposalMerged of PipelineExecutionService (handler HarvestTask) harvests a merged change as an accepted task outcome. Each handler is idempotent on the event id and keeps the task in the tenant of the run (D32). A caller whose scope carries factory.evaluation.read lists the datasets of the tenant through ListDatasets with their versions, task counts and classification; a caller without the scope is refused with 403. An Asset maintainer or Platform admin whose scope carries factory.evaluation.run snapshots a dataset through SnapshotDataset with an Idempotency-Key; the service answers 202 and the DatasetSnapshotter worker, triggered by the operation, builds the snapshot from the tasks and the curated additions of the dataset, stores it through ArtifactService as an immutable artifact, and writes a new Datasets row, with a new DatasetId, the same tenant and name, the version of the addressed row plus one, the task count, snapshotRef and publishedAt, and the DatasetTasks rows of that version under the new DatasetId, together with the DatasetPublished outbox record with datasetId, tenantId, name, version and taskCount, in one transaction. Earlier rows are never updated, so the snapshot reference and the tasks of an earlier version stay as they were, and the path {datasetId} names any version of the dataset whose next version is to be written. SnapshotRef is non-null and DatasetTasks has a foreign key to Datasets, so no row can hold harvested tasks before the first snapshot of a dataset: until the owner decides, harvested candidates are held in a candidate table of the service keyed by tenant, dataset name and task id, outside the tables of the model, and are copied into DatasetTasks of the version the snapshot writes. The non-null SnapshotRef, the per-version identity of datasetId, the pin of EvaluationSuites on DatasetId alone, which then pins one version, and how the first version of a new dataset is created when no row exists for the path to address are open points reported with this feature. The idempotency key of the worker is datasetId plus the version, so a repeated call or a duplicate trigger publishes one version; its retry policy is exponential with three attempts, and when they are exhausted no version is written and nothing is published. A dataset of another tenant and an unknown dataset are answered 404, the same answer for both (ADR-0015 amendment D113); a caller without the scope is refused with 403. A snapshot is never changed after it is written, and a suite pins a snapshot version. Sharing a dataset across tenants needs anonymisation and consent, checked through MemoryPromotionService, which arrives in I5; until then every dataset stays in its tenant. The gauge connectsoft.factory.learning.datasets moves with each version.

Acceptance criteria

AC1 Given RunControlService/RunCompleted, ArtifactService/ArtifactValidated and PipelineExecutionService/ChangeProposalMerged of one tenant, when the harvest handlers run, then dataset task candidates exist in that tenant with their source run, input, outcome and validator verdict, and a redelivered event adds nothing (model/services/observability/FeedbackService.yaml, ADR-0015).
AC2 Given a caller with factory.evaluation.run and a dataset of the caller tenant, when FeedbackService/SnapshotDataset is called with an Idempotency-Key, then the response is 202, the snapshot is stored once through ArtifactService as an immutable artifact, a new FeedbackService/Dataset row with a new datasetId, the same tenant and name, the version plus one, its task count, snapshotRef and tasks is written while every earlier row stays unchanged, and the FeedbackService/DatasetPublished outbox record is written in the same transaction; a repeated call publishes no second version (SCN-FeedbackService-004, ADR-0021).
AC3 Given a dataset of another tenant or an unknown id, or a caller without the scope, when FeedbackService/SnapshotDataset is called, then it is answered 404 or 403 and nothing is written; given ArtifactService fails on all three attempts of the worker, then no version is written and nothing is published (SCN-FeedbackService-004, ADR-0015).
AC4 Given a caller with factory.evaluation.read, when FeedbackService/ListDatasets is called, then the response is 200 with the datasets of the caller tenant, their versions, task counts and classification; without the scope it is 403; when the dispatcher publishes FeedbackService/DatasetPublished on evaluation.datasets.v1.dataset-published, then it is sent under the canonical envelope with datasetId, tenantId, name, version and taskCount, and an outage is retried so it is neither lost nor duplicated (SCN-FeedbackService-003, SCN-FeedbackService-009, ADR-0012).
Traces: services FeedbackService, ArtifactService, RunControlService, PipelineExecutionService
Traces: aggregates FeedbackService/Dataset
Traces: apis FeedbackService/ListDatasets, FeedbackService/SnapshotDataset
Traces: events FeedbackService/DatasetPublished, RunControlService/RunCompleted, ArtifactService/ArtifactValidated, PipelineExecutionService/ChangeProposalMerged
Traces: adrs ADR-0012, ADR-0015, ADR-0021
Traces: requirements NFR-16, XL-08
Traces: scenarios SCN-FeedbackService-003, SCN-FeedbackService-004, SCN-FeedbackService-009
Relates: screens datasets
Depends on feature:artifact-service, feature:run-control-service, feature:pipeline-execution-service-complete
Sources model/services/observability/FeedbackService.yaml, docs/planes/trust-operations/observability/services/feedback-service/index.md#workers-and-jobs

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-553 Datasets and snapshots implementation HarvestTask and HarvestValidation turn RunCompleted, ArtifactValidated and ChangeProposalMerged into tenant-scoped dataset task candidates held in the candidate table, idempotently; ListDatasets returns the datasets of the tenant; SnapshotDataset answers 202 and triggers DatasetSnapshotter, which stores the snapshot through ArtifactService and writes a new Datasets row with a new DatasetId and the version plus one, its DatasetTasks copied from the candidates and its DatasetPublished outbox record in one transaction, never updating an earlier row, keyed by datasetId and version, with the exponential retry policy and nothing written when it is exhausted. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4384 Harvest and snapshot fixtures fixture The acceptance suite holds RunCompleted, ArtifactValidated and ChangeProposalMerged fixtures of two tenants from the contracts package, datasets with curated additions in both tenants, a fake ArtifactService that can fail a given number of times, and a transport switch for the outbox dispatcher. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4385 Verify harvest, listing and snapshot through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-FeedbackService-003\|Scenario=SCN-FeedbackService-004\|Scenario=SCN-FeedbackService-009)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1099: the harvested candidates and their redelivery, the snapshot published once as a new row with its version while the earlier row stays unchanged, the 404 for another tenant, the 403, the exhausted retry without a version, the listing and the publication and payload of DatasetPublished through an outage. ConnectSoft.AI.SoftwareFactory.FeedbackService

AIFACTORY-US-1100: Resolve a dataset version to its snapshot artifact for an evaluation suite over the internal gRPC method

behavior · persona asset-maintainer As an asset maintainer, I want the evaluation service to read the exact snapshot my suite pins, so that every evaluation runs on the data the suite names and on nothing else.

QualityScoreService starts an evaluation only after it has resolved the dataset version its suite pins to the snapshot artifact, through the internal gRPC method Learning.GetDatasetSnapshot of FeedbackService. The caller presents a workload identity that holds the service scope that names FeedbackService (D114), and forwards the tenant of the evaluation as metadata. The model declares artifact-v1 as the request, and artifact-v1 requires artifactId, kind, contentHash, contentRef, classification, validation and createdAt and has no field for a dataset or a version, so a caller cannot build a valid request that names the dataset version before it has resolved the snapshot; this is an open point reported with this feature. Until the owner decides, the request carries the per-version datasetId of AIFACTORY-US-1099 in the gRPC metadata dataset-id and its version in dataset-version, the service checks that both name the same Datasets row and does not read the body. The service reads that row under the forwarded tenant and replies with artifact-v1 carrying the snapshotRef of the immutable snapshot. The method is of class read: it writes nothing, publishes nothing and answers the same for the same request. A version that does not exist, a dataset of another tenant and a version without a snapshot are answered with the status NOT_FOUND, the same answer in each case, so the reply never confirms that a dataset of another tenant exists (ADR-0015 amendment D113); a caller without a workload identity is refused with UNAUTHENTICATED, a workload without the service scope with PERMISSION_DENIED, and a request that fails validation with INVALID_ARGUMENT. QualityScoreService treats this dependency as required and fail-fast, so a NOT_FOUND or an outage of FeedbackService fails the start of an evaluation or, during a run, fails the run with the reason snapshot missing, in the PBIs of that service.

Acceptance criteria

AC1 Given a workload identity with the service scope, the tenant metadata of the evaluation and the dataset-id and dataset-version metadata naming a published dataset version of that tenant, when FeedbackService/Learning.GetDatasetSnapshot is called, then the reply carries artifact-v1 with the snapshot reference of that version, nothing is written and a repeated call answers the same (SCN-FeedbackService-007).
AC2 Given an unknown version, a dataset of another tenant or a version without a snapshot, when FeedbackService/Learning.GetDatasetSnapshot is called, then each is answered NOT_FOUND alike; given no workload identity, a workload without the service scope or an invalid request, then the answer is UNAUTHENTICATED, PERMISSION_DENIED or INVALID_ARGUMENT, and no state changes (SCN-FeedbackService-007, ADR-0015).
Traces: services FeedbackService, QualityScoreService
Traces: aggregates FeedbackService/Dataset
Traces: apis FeedbackService/Learning.GetDatasetSnapshot
Traces: descriptors artifact-v1
Traces: adrs ADR-0015, ADR-0021
Traces: scenarios SCN-FeedbackService-007
Depends on AIFACTORY-US-1099
Sources model/services/observability/FeedbackService.yaml, docs/planes/trust-operations/observability/services/feedback-service/index.md#apis

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-4386 Learning.GetDatasetSnapshot under the forwarded tenant implementation The gRPC method reads the Datasets row named by the dataset-id and dataset-version metadata under the tenant a workload with the service scope forwards, refuses metadata that is missing or names two different rows with INVALID_ARGUMENT, and replies with artifact-v1 carrying its snapshotRef; unknown versions, other tenants and versions without a snapshot answer NOT_FOUND alike; UNAUTHENTICATED, PERMISSION_DENIED and INVALID_ARGUMENT refuse the call before any read. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4387 Contract of the dataset snapshot method for QualityScoreService contract The proto of the service exposes Learning.GetDatasetSnapshot with artifact-v1 as request and reply as model/descriptors.yaml records it, published with the contracts so that QualityScoreService calls it through a generated client; the contract documents that artifact-v1 has no field for a dataset or a version, that the dataset-id and dataset-version metadata carry them until the owner decides the request descriptor, and that the body is not read. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4388 Verify the dataset snapshot method through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-FeedbackService-007"; a zero-match filter counts as a failure; the tests prove AC1 and AC2 of AIFACTORY-US-1100: the resolved snapshot and its repeat, NOT_FOUND alike for the three cases, and the three refusals without a state change. ConnectSoft.AI.SoftwareFactory.FeedbackService

AIFACTORY-US-1101: Raise learning proposals with evidence from feedback, drift and failed evaluations, and list them for maintainers

behavior · persona asset-maintainer As an asset maintainer, I want improvement proposals for my assets drafted from the evidence the Factory collected, with the effect they are expected to have, so that I review a concrete change with its reasons instead of searching through feedback and drift reports myself.

Evidence arrives from three sources the service handles now. EvaluationCompleted of QualityScoreService (handler RaiseFromEvaluation) adds failed gates and regressions; DriftDetected of QualityScoreService (handler RaiseFromDrift) adds drift of a release; recorded feedback adds what people said. The thin QualityScoreService of I1 publishes no event, and the complete delivery that publishes both events, feature quality-score-service-complete, itself depends on this feature, so RaiseFromEvaluation and RaiseFromDrift are built against the contracts package and proven with fixtures of both events (AIFACTORY-TASK-4389); their first real events arrive when quality-score-service-complete is deployed in the same increment. The dependency on feature quality-score-service-thin names the service that owns both events, not a publisher of them. A fourth source, SkillCandidateProposed of MemoryPromotionService (handler RaiseSkillProposal), is built against the contracts package and receives its first events when that service arrives in I5. The ProposalGenerator worker runs on its modelled schedule, clusters the new evidence per target asset, and raises a proposal only for a cluster with at least Observability:Learning:MinEvidenceItems items (default 3), because a proposal without evidence is not raised. It drafts the proposal text and diff through ModelRouterService with the model profile Observability:Learning:ProposalProfile (default structured), stores the draft through ArtifactService as summaryRef, and writes the LearningProposal in state Proposed with its kind (PromptImprovement, SkillImprovement, ContextStrategy, ThresholdChange, DatasetAddition or ProfileChange), target asset and version, the evidence references and the expected effect, together with the LearningProposalRaised outbox record with proposalId, tenantId, kind, targetAssetId and targetVersion, in one transaction. A Proposed proposal for the same target and kind is moved to Superseded (trigger NewerProposal) in the same transaction. The idempotency key of the worker is targetAssetId plus the evidence hash, so the same evidence never raises a second proposal; its retry policy is fixed with two attempts. When the drafting model is unavailable the evidence cluster waits for the next run and no proposal is raised without a draft. A proposal never changes an asset. StudioBff, StudioNotificationService and AuditService consume the event in their own PBIs. A caller whose scope carries factory.evaluation.read lists proposals through ListLearningProposals by state, kind and target with their evidence and expected effect, under the tenant row filter; without the scope the answer is 403. Which project the drafting usage is metered on is an open point reported with this feature, because a proposal belongs to a tenant and names no project while a model call outside a run names its project. The gauge connectsoft.factory.learning.proposals moves with each state change.

Acceptance criteria

AC1 Given at least three evidence items for one asset from QualityScoreService/EvaluationCompleted, QualityScoreService/DriftDetected and recorded feedback, when ProposalGenerator runs, then one FeedbackService/LearningProposal in Proposed holds the kind, target, evidence references, expected effect and the draft stored through ArtifactService, and the FeedbackService/LearningProposalRaised outbox record is written in the same transaction; an earlier Proposed proposal for the same target and kind is Superseded in that transaction (SCN-FeedbackService-010, ADR-0021).
AC2 Given fewer than three evidence items, or the same evidence a second time, when ProposalGenerator runs, then no proposal is raised; given the drafting model unavailable, then the cluster waits, no proposal is raised without a draft and the next run raises it once the model answers (SCN-FeedbackService-010, model/services/observability/FeedbackService.yaml).
AC3 Given the record was written, when the dispatcher publishes FeedbackService/LearningProposalRaised on evaluation.learning.v1.learning-proposal-raised, then it is sent under the canonical envelope with proposalId, tenantId, kind, targetAssetId and targetVersion, and an outage is retried so it is neither lost nor duplicated under its event id (SCN-FeedbackService-010, ADR-0012).
AC4 Given proposals in several states and two tenants, when FeedbackService/ListLearningProposals is called with factory.evaluation.read and a state, kind or target filter, then the response is 200 with the matching proposals of the caller tenant and their evidence and expected effect; without the scope it is 403 and nothing is written (SCN-FeedbackService-005, ADR-0015).
Traces: services FeedbackService, QualityScoreService, ModelRouterService, ArtifactService
Traces: aggregates FeedbackService/LearningProposal
Traces: apis FeedbackService/ListLearningProposals
Traces: events FeedbackService/LearningProposalRaised, QualityScoreService/EvaluationCompleted, QualityScoreService/DriftDetected
Traces: adrs ADR-0012, ADR-0015, ADR-0021
Traces: requirements FR-11, FUX-09, NFR-11
Traces: scenarios SCN-FeedbackService-005, SCN-FeedbackService-010
Relates: screens learning-proposals
Depends on AIFACTORY-US-1098, feature:contracts-package, feature:quality-score-service-thin, feature:model-router-service, feature:artifact-service
Sources model/services/observability/FeedbackService.yaml, model/services/observability/QualityScoreService.yaml, docs/planes/trust-operations/observability/services/feedback-service/index.md#workers-and-jobs

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-554 Learning proposals raised from evidence implementation RaiseFromEvaluation and RaiseFromDrift collect evidence per asset idempotently; ProposalGenerator clusters it, raises a proposal only from MinEvidenceItems items, drafts it through ModelRouterService with ProposalProfile, stores the draft through ArtifactService and writes the Proposed proposal with its LearningProposalRaised outbox record and the Superseded predecessor in one transaction, keyed by targetAssetId and evidence hash; the cluster waits when drafting fails; RaiseSkillProposal is registered against the contracts package; ListLearningProposals filters by state, kind and target. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4389 Evidence and drafting fixtures for the proposal generator fixture The acceptance suite holds EvaluationCompleted and DriftDetected fixtures from the contracts package, recorded feedback, evidence clusters of two, three and more items for one asset, a fake drafting model that can be made unavailable, a fake ArtifactService and a clock and trigger the test controls. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4390 Verify raising and listing proposals through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-FeedbackService-005\|Scenario=SCN-FeedbackService-010)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1101: the raised proposal with its draft and superseded predecessor, no proposal below the evidence minimum or for repeated evidence, the waiting cluster while the model is down, the publication and payload of LearningProposalRaised through an outage, and the listing with its 403. ConnectSoft.AI.SoftwareFactory.FeedbackService

AIFACTORY-US-1102: An asset maintainer accepts a learning proposal, which opens a Draft release, or rejects it with a reason

behavior · persona asset-maintainer As an asset maintainer, I want to accept a proposal into a Draft release of my asset or reject it with my reason, so that every change a proposal suggests goes through evaluation and publication like any other change, and a rejection is remembered.

An Asset maintainer whose scope carries factory.assets.write decides a proposal in review through DecideLearningProposal, naming accept, or reject with a reason. Accepting moves the LearningProposal from InReview to Accepted (trigger ReleaseCandidateOpened): the change becomes a Draft release candidate of the target asset in BC-04, which then goes through evaluation and publication like any change, so a proposal never changes an asset itself. Rejecting moves it from InReview to Rejected (trigger Rejected) with the reason and reviewedBy. In both cases the LearningProposalDecided outbox record with proposalId, tenantId and state, and for an accepted proposal targetAssetId, targetVersion and summaryRef without resultingReleaseId, is written in the transaction of the decision, and the answer is 200. AgentRegistryService, StudioBff and AuditService consume the event in their own PBIs. The operation is idempotent by its natural key: the same decision repeated by the same caller on a proposal already in that state answers 200 with the recorded state and changes nothing, so a caller that retries after a lost answer, as the gateway tells callers to do, learns that its decision was applied. A different decision on an Accepted or Rejected proposal, and any decision on a proposal that is Proposed or Superseded, is answered 409 and the proposal keeps its state. A proposal of another tenant and an unknown id are answered 404 alike (ADR-0015 amendment D113); a caller without the scope is refused with 403. FeedbackService opens no release itself and makes no outbound call to AgentRegistryService: AgentRegistryService, the owner of releases, opens the Draft release when it consumes LearningProposalDecided with the state Accepted and announces its release id with the proposal id through LearningReleaseOpened. The handler RecordResultingRelease of FeedbackService records that release id as resultingReleaseId of the proposal, idempotent on the proposal id, so a redelivered announcement for a proposal that already keeps that release id changes nothing (D131). The model also lists the transition from Proposed to InReview with the trigger MaintainerPicksUp, and no operation of the model raises it; which operation raises it is an open point reported with this feature, and this PBI delivers the transition in the domain model and the decision on an InReview proposal.

Acceptance criteria

AC1 Given a proposal in InReview of the caller tenant and a caller with factory.assets.write, when FeedbackService/DecideLearningProposal accepts it, then the FeedbackService/LearningProposal is Accepted with reviewedBy, the FeedbackService/LearningProposalDecided outbox record with the state Accepted and the target asset and without resultingReleaseId is written in the transaction of the decision, and FeedbackService opens no release itself; the response is 200 (SCN-FeedbackService-006, ADR-0021).
AC2 Given a proposal in InReview, when FeedbackService/DecideLearningProposal rejects it with a reason, then it is Rejected with the reason and reviewedBy and FeedbackService/LearningProposalDecided is written with the state Rejected; given the same decision repeated by the same caller on a proposal already in that state, then the answer is 200 with the recorded state, no second event is written and nothing changes; given a different decision on an Accepted or Rejected proposal, or a proposal that is Proposed or Superseded, then the answer is 409 and nothing changes (SCN-FeedbackService-006, SCN-FeedbackService-011).
AC3 Given a proposal of another tenant or an unknown id, or a caller without the scope, when FeedbackService/DecideLearningProposal is called, then it is answered 404 or 403 and nothing is written (SCN-FeedbackService-006, ADR-0015).
AC4 Given the record was written, when the dispatcher publishes FeedbackService/LearningProposalDecided on evaluation.learning.v1.learning-proposal-decided, then it is sent under the canonical envelope with proposalId, tenantId, state and, for an accepted proposal, targetAssetId, targetVersion and summaryRef without resultingReleaseId, and an outage is retried so it is neither lost nor duplicated (SCN-FeedbackService-011, ADR-0012).
AC5 Given an accepted proposal, when AgentRegistryService/LearningReleaseOpened naming it is handled by RecordResultingRelease, then the FeedbackService/LearningProposal keeps the announced release id as resultingReleaseId; a redelivered announcement for a proposal that already keeps that release id changes nothing (D131, model/services/observability/FeedbackService.yaml).
Traces: services FeedbackService, AgentRegistryService
Traces: aggregates FeedbackService/LearningProposal
Traces: apis FeedbackService/DecideLearningProposal
Traces: events FeedbackService/LearningProposalDecided, AgentRegistryService/LearningReleaseOpened
Traces: adrs ADR-0012, ADR-0015, ADR-0021
Traces: requirements FR-11, FUX-09
Traces: scenarios SCN-FeedbackService-006, SCN-FeedbackService-011
Relates: screens learning-proposals
Depends on AIFACTORY-US-1101, feature:agent-registry-service-complete
Sources model/services/observability/FeedbackService.yaml, model/services/control-plane/AgentRegistryService.yaml

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-4391 DecideLearningProposal with the guarded transitions and the RecordResultingRelease handler implementation DecideLearningProposal accepts or rejects an InReview proposal of the caller tenant, records reviewedBy and the reason, writes LearningProposalDecided in the transaction of the decision with the target asset and without resultingReleaseId, answers 200 with the recorded state and writes nothing for the same decision repeated by the same caller, answers 409 for a different decision on an Accepted or Rejected proposal and for a Proposed or Superseded proposal, 404 for another tenant or an unknown id and 403 without the scope; the handler RecordResultingRelease of LearningReleaseOpened records the announced release id as resultingReleaseId, idempotent on the proposal id, and the service makes no outbound call to AgentRegistryService. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4392 Verify the decision of a proposal through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-FeedbackService-006\|Scenario=SCN-FeedbackService-011)"; a zero-match filter counts as a failure; the tests prove AC1 to AC5 of AIFACTORY-US-1102: acceptance and rejection with their events, the 200 without a second event for the repeated same decision, the 409 for a different decision on a decided proposal and for a Proposed or Superseded proposal, the 404 and 403, the publication and payload of LearningProposalDecided through an outage, and the resulting release recorded once from LearningReleaseOpened across a redelivery. ConnectSoft.AI.SoftwareFactory.FeedbackService

AIFACTORY-US-1103: Retention rules, deletion orders and legal holds are applied to feedback, datasets and proposals

behavior · persona workspace-admin As a workspace admin, I want the retention, deletion and hold orders of my tenant applied to feedback, dataset tasks and proposals, so that what people said and what was harvested from runs is kept as long as the rules say and no longer, except under a legal hold.

FeedbackService owns HumanFeedback, Datasets, DatasetTasks and LearningProposals in feedback-sql, and references texts, snapshots and drafts kept by ArtifactService. Three consumed events of RetentionResidencyService govern the lifecycle of its rows. RetentionRuleChanged (handler ApplyRetentionRule) applies the effective retention period of the rule to the rows of its store class. DeletionOrderIssued (handler ExecuteDeletionOrder) deletes the ordered scope from the stores the service owns and acknowledges its task of the order through AcknowledgeOrderTask of RetentionResidencyService under the scope factory.data.acknowledge, once per order id, with a receipt that names the order, the tenant, the deleted rows and what a legal hold kept. A published dataset snapshot is immutable, so a deletion order that reaches tasks of a published version removes the rows of the service and leaves the artifact to the order of ArtifactService. LegalHoldChanged (handler ApplyLegalHold) suspends deletion and retention expiry for the held scope and resumes them when the hold is lifted. Each handler is idempotent on the event id and touches only rows of the tenant of the event; an event without a tenant is dead-lettered and changes nothing; a failed acknowledgement is retried by the inbox and the deletion is not repeated.

Acceptance criteria

AC1 Given RetentionResidencyService/RetentionRuleChanged for the store class of the service, when ApplyRetentionRule handles it, then rows older than the effective period are removed by the retention sweep and younger rows are kept; a redelivered event changes nothing (ADR-0019, model/services/observability/FeedbackService.yaml).
AC2 Given RetentionResidencyService/DeletionOrderIssued for a scope of one tenant while RetentionResidencyService/LegalHoldChanged holds a part of it, when the handlers run, then the ordered rows are deleted except the held ones, no row of another tenant is touched, and the task is acknowledged once through RetentionResidencyService/AcknowledgeOrderTask with a receipt; a failed acknowledgement is retried without a second deletion (ADR-0015, ADR-0019).
AC3 Given a hold lifted by RetentionResidencyService/LegalHoldChanged, when the next sweep runs, then expiry resumes for the scope it held; given an event without a tenant, then it is dead-lettered and nothing changes (ADR-0019).
Traces: services FeedbackService, RetentionResidencyService
Traces: apis RetentionResidencyService/AcknowledgeOrderTask
Traces: events RetentionResidencyService/RetentionRuleChanged, RetentionResidencyService/DeletionOrderIssued, RetentionResidencyService/LegalHoldChanged
Traces: adrs ADR-0015, ADR-0019
Traces: requirements NFR-03
Relates: screens retention-residency
Depends on feature:retention-residency-service
Sources model/services/observability/FeedbackService.yaml, model/services/data-event/RetentionResidencyService.yaml, docs/architecture/adr/0019-data-governance.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-4393 Handlers ApplyRetentionRule, ExecuteDeletionOrder and ApplyLegalHold over feedback-sql implementation The three handlers apply the effective retention to the rows of the store class, delete the ordered scope except held rows and acknowledge the order task once through AcknowledgeOrderTask with its receipt, suspend and resume expiry for a held scope, are idempotent on the event id, touch only rows of the tenant of the event and dead-letter an event without a tenant. ConnectSoft.AI.SoftwareFactory.FeedbackService
AIFACTORY-TASK-4394 Verify retention, deletion and hold through the acceptance suite verification Run dotnet test -c Release --filter "Category=Acceptance&Feature=RetentionOrders"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1103 with a fake RetentionResidencyService/AcknowledgeOrderTask endpoint: the sweep under a rule, the deletion under a hold with its receipt, the retried acknowledgement, the lifted hold and the event without a tenant. ConnectSoft.AI.SoftwareFactory.FeedbackService