FeedbackService

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Bounded context	BC-22 Evaluation & Learning
Platform / plane	Observability / Trust & Operations
Deployment granularity	module of host · host ObservabilityHost
Status	target · ConnectSoft.Factory.Observability.Learning
Built from	connectsoft-microservice · libraries: connectsoft-extensions-ai, connectsoft-extensions-saas-abstractions, connectsoft-extensions-messaging-model, connectsoft-extensions-persistence-model-nhibernate
Decisions	ADR-0021
Traceability	FR-11 FUX-10 FUX-09 NFR-11 NFR-16 XL-08

Purpose and responsibilities

FeedbackService closes the learning loop of the original design (§32) without letting anything learn in place. It records human feedback (ratings, corrections with a corrected artifact, comments, escalations) on runs, attempts, artifacts, memories and evaluations; it builds datasets of historical tasks from completed runs, feedback and curated additions, versioned as immutable snapshots that suites pin; and it raises learning proposals — prompt, skill and context-strategy improvements, threshold changes, dataset additions, profile changes — each with its evidence (feedback clusters, drift, failed evaluations, reinforced procedural memory from MemoryPromotionService) and an expected effect, for an Asset maintainer to accept (which opens a Draft release in BC-04 that goes through evaluation and publication like any change) or reject.

Context and boundaries

flowchart LR
    S["FeedbackService"]
    QualityScoreService["QualityScoreService"] -->|grpc| S
    StudioBff["StudioBff"] -->|sync-http| S
    S -->|grpc| ArtifactService["ArtifactService"]
    S -->|grpc| AgentRegistryService["AgentRegistryService"]
    S -->|grpc| ModelRouterService["ModelRouterService"]
    S -->|grpc| MemoryPromotionService["MemoryPromotionService"]
    S -. "FeedbackRecorded" .-> E_FeedbackRecorded(("event"))
    S -. "DatasetPublished" .-> E_DatasetPublished(("event"))
    S -. "LearningProposalRaised" .-> E_LearningProposalRaised(("event"))
    S -. "LearningProposalDecided" .-> E_LearningProposalDecided(("event"))

Hold "Alt" / "Option" to enable pan & zoom

Datasets derived from a tenant's runs are that tenant's (D32): sharing across tenants needs anonymisation and consent, checked through MemoryPromotionService's consent model. A proposal without evidence is not raised; a proposal never changes anything by itself.

Domain model

HumanFeedback — Feedback from a requester, reviewer or operator on a run, attempt, artifact or memory, with the subject reference and an optional corrected artifact. (catalog entry)

Field	Type	Required	Description
feedbackId	guid	yes
tenantId	string	yes
projectId	string	yes
subjectKind	string	yes	run
subjectRef	string	yes
kind	enum:FeedbackKind	yes
rating	int	no
commentRef	string	no	Artifact reference of the text.
correctedArtifactRef	string	no
givenBy	string	yes
givenAt	datetime	yes

Invariants:

• Feedback references its subject by id and keeps text as artifacts; it never alters the subject.

Events: FeedbackRecorded · Tables: HumanFeedback · Repository: HumanFeedbackRepository

Dataset — A versioned evaluation dataset of historical tasks (inputs, expected outcomes, scoring rules) with snapshots stored as artifacts; built from completed runs, feedback and curated additions. (catalog entry)

Field	Type	Required	Description
datasetId	guid	yes
tenantId	string	yes
name	string	yes
target	string	yes	Evaluation target kind the dataset serves.
version	int	yes
taskCount	int	yes
snapshotRef	string	yes	Artifact reference of the snapshot.
classification	string	yes
publishedAt	datetime	no

Invariants:

• Tasks derived from tenant runs stay tenant-scoped (D32); sharing across tenants needs anonymisation and consent.
• A snapshot is immutable; suites pin a snapshot version.

Events: DatasetPublished · Tables: Datasets, DatasetTasks · Repository: DatasetRepository

LearningProposal — A reviewable suggestion to change an asset or a strategy, with the evidence (feedback, drift, evaluations, reinforced memory) and the target asset; accepted proposals become release candidates in BC-04. (catalog entry)

Field	Type	Required	Description
proposalId	guid	yes
tenantId	string	yes
kind	enum:ProposalKind	yes
targetAssetId	string	yes
targetVersion	string	yes
summaryRef	string	yes	Artifact reference of the proposal text and diff.
evidenceRefs	list:string	yes
expectedEffect	json	no	Metric, from, to, confidence
state	enum:ProposalState	yes
reviewedBy	string	no
resultingReleaseId	string	no
raisedAt	datetime	yes

Invariants:

• A proposal never changes an asset; acceptance opens a Draft release that goes through evaluation and publication like any change.
• Proposals cite evidence; a proposal without evidence is not raised.

stateDiagram-v2
    Proposed --> InReview : MaintainerPicksUp
    InReview --> Accepted : ReleaseCandidateOpened
    InReview --> Rejected : Rejected
    Proposed --> Superseded : NewerProposal

Hold "Alt" / "Option" to enable pan & zoom

Events: LearningProposalRaised, LearningProposalDecided · Tables: LearningProposals · Repository: LearningProposalRepository

erDiagram
    HumanFeedback {
        guid feedbackId PK
        string tenantId
        string projectId
        string subjectKind
        string subjectRef
        enum_FeedbackKind kind
        int rating
        string commentRef
        string correctedArtifactRef
        string givenBy
        datetime givenAt
    }
    Dataset {
        guid datasetId PK
        string tenantId
        string name
        string target
        int version
        int taskCount
        string snapshotRef
        string classification
        datetime publishedAt
    }
    LearningProposal {
        guid proposalId PK
        string tenantId
        enum_ProposalKind kind
        string targetAssetId
        string targetVersion
        string summaryRef
        list_string evidenceRefs
        json expectedEffect
        enum_ProposalState state
        string reviewedBy
        string resultingReleaseId
        datetime raisedAt
    }

Hold "Alt" / "Option" to enable pan & zoom

Entities and value objects

None modelled

No entities or value objects are modelled for this item yet.

Data model and storage

Store	Role	Engine / profile	Tables
Operations projection store	primary	azure-sql / nhibernate-sqlserver	HumanFeedback, Datasets, DatasetTasks, LearningProposals
Artifact content store	blob	blob / azure-blob	—

Table	Owner	Columns	Indexes
dbo.HumanFeedback	HumanFeedback	11	1
dbo.Datasets	Dataset	9	1
dbo.DatasetTasks	Dataset	7	0
dbo.LearningProposals	LearningProposal	12	2

Full DDL: appendix.

APIs

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
POST	/factory/feedback	RecordFeedback	factory.feedback.write	shared-write	key-header	GW-03	Record feedback on a run, attempt, artifact, memory or evaluation.
GET	/factory/feedback	ListFeedback	factory.feedback.read	read	natural	GW-03	Feedback by subject and kind.
GET	/factory/evaluation/datasets	ListDatasets	factory.evaluation.read	read	natural	GW-03	Datasets with versions, task counts and classification.
POST	/factory/evaluation/datasets/{datasetId}/snapshot	SnapshotDataset	factory.evaluation.run	shared-write	key-header	GW-03	Publish a new immutable version of a dataset.
GET	/factory/learning/proposals	ListLearningProposals	factory.evaluation.read	read	natural	GW-03	Proposals by state, kind and target with evidence and expected effect.
POST	/factory/learning/proposals/{proposalId}/decide	DecideLearningProposal	factory.assets.write	shared-write	natural	GW-03	Accept (opens a Draft release in BC-04) or reject with a reason (Asset maintainer).

gRPC service	Method	Streaming	Visibility	Side effects	Summary
Learning	GetDatasetSnapshot	none	internal	read	Resolve a dataset version to its snapshot artifact for a suite.

Messages

Event	Topic	Scope / v	Aggregate	Consumers	Summary
FeedbackRecorded	evaluation.feedback.v1.feedback-recorded	integration / 1	HumanFeedback	DashboardService, StudioBff	Feedback exists on a subject.
DatasetPublished	evaluation.datasets.v1.dataset-published	integration / 1	Dataset	QualityScoreService, StudioBff	A dataset version is available to suites.
LearningProposalRaised	evaluation.learning.v1.learning-proposal-raised	integration / 1	LearningProposal	StudioBff, StudioNotificationService, AuditService	A proposal with evidence awaits a maintainer.
LearningProposalDecided	evaluation.learning.v1.learning-proposal-decided	integration / 1	LearningProposal	StudioBff, AuditService, AgentRegistryService	A proposal was accepted (release candidate opened) or rejected.

FeedbackRecorded payload:

Field	Type	Required	Description
feedbackId	guid	yes
tenantId	string	yes
subjectKind	string	yes
subjectRef	string	yes
kind	enum:FeedbackKind	yes
rating	int	no

DatasetPublished payload:

Field	Type	Required	Description
datasetId	guid	yes
tenantId	string	yes
name	string	yes
version	int	yes
taskCount	int	yes

LearningProposalRaised payload:

Field	Type	Required	Description
proposalId	guid	yes
tenantId	string	yes
kind	enum:ProposalKind	yes
targetAssetId	string	yes
targetVersion	string	yes

LearningProposalDecided payload:

Field	Type	Required	Description
proposalId	guid	yes
tenantId	string	yes
state	enum:ProposalState	yes
resultingReleaseId	string	no

Command	Handled by	Result events	Summary
RecordFeedback	HumanFeedback	FeedbackRecorded
SnapshotDataset	Dataset	DatasetPublished
DecideLearningProposal	LearningProposal	LearningProposalDecided

Event	Producer	Handler	Purpose
RunCompleted	RunControlService	HarvestTask	Completed runs become dataset task candidates (learning loop §32).
EvaluationCompleted	QualityScoreService	RaiseFromEvaluation	Failed gates and regressions raise proposals with evidence.
DriftDetected	QualityScoreService	RaiseFromDrift	Drift raises re-evaluation and improvement proposals.
SkillCandidateProposed	MemoryPromotionService	RaiseSkillProposal	Reinforced procedural memory becomes a skill-improvement proposal.
ArtifactValidated	ArtifactService	HarvestValidation	Validator outcomes enrich dataset tasks.

Workers and jobs

Worker	Trigger	Output events	Retry	Idempotency key	Summary
ProposalGenerator	schedule 0 6 * * *	LearningProposalRaised	fixed	targetAssetId + evidenceHash	Clusters feedback, drift and failed evaluations per asset and drafts proposals with evidence and an expected effect (structured profile); never applies anything.
DatasetSnapshotter	manual on SnapshotDataset	DatasetPublished	exponential	datasetId + version	Builds dataset snapshots from harvested tasks and curated additions and stores them as artifacts.

Actors and grains

None modelled

No grains are modelled for this item yet.

Sagas and long-running flows

None modelled

No sagas are modelled for this item yet.

flowchart LR
    R[RunCompleted] --> H[HarvestTask → dataset candidates]
    FB[Human feedback] --> C[cluster per asset]
    DR[DriftDetected] --> C
    EV[EvaluationCompleted · Failed] --> C
    SM[SkillCandidateProposed] --> C
    C --> PG[ProposalGenerator · structured profile]
    PG --> LP[LearningProposal · Proposed]
    LP --> M{Asset maintainer}
    M -- accept --> DRAFT[Draft release in BC-04 → evaluation → publication]
    M -- reject --> X[Rejected with reason]

Hold "Alt" / "Option" to enable pan & zoom

stateDiagram-v2
    [*] --> Proposed
    Proposed --> InReview: MaintainerPicksUp
    InReview --> Accepted: ReleaseCandidateOpened
    InReview --> Rejected: Rejected
    Proposed --> Superseded: NewerProposal

Hold "Alt" / "Option" to enable pan & zoom

Options and configuration

Section	Key	Type	Default	Reloadable	Secret	Summary
Observability:Learning	ProposalProfile	string	structured	yes	no	Model profile used to draft proposals.
Observability:Learning	MinEvidenceItems	int	3	yes	no	Evidence items before a proposal is raised.

Metrics and observability

Metric	Instrument	Unit	Dimensions	Summary
connectsoft.factory.learning.feedback	counter	{feedback}	tenant, kind	Feedback recorded.
connectsoft.factory.learning.proposals	updown	{proposal}	tenant, kind, state	Proposals by state.
connectsoft.factory.learning.datasets	updown	{dataset}	tenant, target	Datasets and versions.

Security

Authentication	client-credentials
Scopes	factory.feedback.write (Record feedback.); factory.feedback.read (Read feedback.); factory.evaluation.read (Read datasets and proposals.); factory.evaluation.run (Snapshot datasets.); factory.assets.write (Decide proposals (Asset maintainer).)
Roles	Requester, Reviewer, AssetMaintainer, Operator, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	internal
Max side-effect class	shared-write
Secrets	—

Dependencies

Outbound

Target	Kind	Purpose	Failure mode	Required
ArtifactService	grpc	Feedback text, snapshots and proposal artifacts.	fail-fast	yes
AgentRegistryService	grpc	Open Draft releases for accepted proposals.	degrade	yes
ModelRouterService	grpc	Proposal drafting through the structured profile.	degrade	no
MemoryPromotionService	grpc	Consent and scope checks for shared datasets.	degrade	no

Inbound

Target	Kind	Purpose	Failure mode	Required
QualityScoreService	grpc	Dataset snapshots.	—	yes
StudioBff	sync-http	Feedback, datasets and proposals views.	—	yes

Failure and recovery

Scenario	Detection	Recovery	Data impact	RTO (min)
Proposal drafting model unavailable	drafting failures	evidence clusters wait; no proposal is raised without a draft	none	60

NFR targets

Target	Class value (pilot-batch)	Override
latency_p95_ms	0	—
availability_pct	99.5	—
throughput_rps	10	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	medium	—

Deployment

Granularity	module of host · host ObservabilityHost
Topologies	saas-default, hybrid, on-prem, hybrid-compute
Runtime	—; replicas —–—
Ports	—

Current implementation

Nothing: no feedback capture, dataset or proposal exists; the agent specs describe learning in prose.

Later capabilities

Active-learning selection of tasks that most reduce evaluation uncertainty; feedback capture from the VS Code extension and CLI; maintainer-approved automatic re-evaluation when a proposal's release is opened.

Decisions and traceability

Family	Ids
FR	FR-11
FUX	FUX-10, FUX-09
NFR	NFR-11, NFR-16
XL	XL-08

Related

• Observability overview · QualityScoreService · MemoryPromotionService · AgentRegistryService

• Observability microservices · bounded contexts · APIs · events · data model

• Services catalog · Ownership map