QualityScoreService

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Bounded context	BC-22 Evaluation & Learning
Platform / plane	Observability / Trust & Operations
Deployment granularity	module of host · host ObservabilityHost
Status	target · ConnectSoft.Factory.Observability.Evaluation
Built from	connectsoft-microservice · libraries: connectsoft-extensions-ai, connectsoft-extensions-saas-abstractions, connectsoft-extensions-messaging-model, connectsoft-extensions-messaging-model-orleans, connectsoft-extensions-persistence-model-nhibernate
Decisions	ADR-0021, ADR-0014
Traceability	FR-02 FR-04 FR-11 FUX-10 FUX-02 NFR-11 NFR-13 XL-08 XL-15

Purpose and responsibilities

QualityScoreService is the original design's §47–49 made executable: suites per target kind (model, agent, skill, tool, workflow, context strategy, memory strategy, policy) with the quality-gate metrics (task success, tool selection, argument validity, structured-output success, multi-turn consistency, instruction compliance, patch compilation, test success, human intervention, regression, latency, cost), evaluation runs of a candidate release against a dataset snapshot — and against its baseline release — executed as ordinary Factory runs under the evaluation project with sandbox-class tools, a verdict that is Passed only when every threshold holds and the baseline rule holds (no lower task success, no higher tool-error rate; D32), a waiver path that needs an approval of class publication, drift detection when production outcomes of a release fall below its evaluated scores, and an immutable evidence record that AgentRegistryService reads at the Published gate (ADR-0021).

It absorbs KnowledgeQualityService (context and retrieval quality are metrics of the context-strategy and memory-strategy targets) and the June-2026 QualityScoreService.

Context and boundaries

flowchart LR
    S["QualityScoreService"]
    AgentRegistryService["AgentRegistryService"] -->|grpc| S
    ModelRouterService["ModelRouterService"] -->|event| S
    StudioBff["StudioBff"] -->|sync-http| S
    S -->|grpc| RunControlService["RunControlService"]
    S -->|grpc| AgentRegistryService["AgentRegistryService"]
    S -->|grpc| FeedbackService["FeedbackService"]
    S -->|grpc| ArtifactService["ArtifactService"]
    S -->|grpc| ModelRouterService["ModelRouterService"]
    S -. "EvaluationSuiteActivated" .-> E_EvaluationSuiteActivated(("event"))
    S -. "EvaluationStarted" .-> E_EvaluationStarted(("event"))
    S -. "EvaluationCompleted" .-> E_EvaluationCompleted(("event"))
    S -. "EvaluationFailed" .-> E_EvaluationFailed(("event"))
    S -. "DriftDetected" .-> E_DriftDetected(("event"))

Hold "Alt" / "Option" to enable pan & zoom

Thresholds live in the suite asset, set per model profile by Platform admin — never by the author of the thing evaluated. The seeds of D32 (tool selection ≥ 97 %, valid arguments ≥ 99 %, structured output ≥ 99.5 %, multi-turn ≥ 95 %, instruction compliance ≥ 95 %, patch compile ≥ 90 %) are the defaults until a suite overrides them. ModelRouterService consumes the same EvaluationCompleted events for its capability registry: a model deployment's scorecard is evidence from this service, not a vendor claim.

Domain model

EvaluationSuite — Runtime read model of a Published evaluation-suite asset — target kind, dataset reference, metrics with thresholds per model profile, baseline rule (no regression on task success, no rise in tool-error rate) and cost and latency budgets. (catalog entry)

Field	Type	Required	Description
suiteId	string	yes
version	string	yes
releaseId	string	yes
tenantId	string	yes
target	enum:EvaluationTarget	yes
datasetId	guid	yes	Dataset of historical tasks (FeedbackService).
metrics	json	yes	Metric ids with thresholds, for example tool-selection ≥ 0.97, valid-arguments ≥ 0.99, structured-output ≥ 0.995, multi-turn ≥ 0.95, instruction-compliance ≥ 0.95, patch-compile ≥ 0.90
baselineRule	json	yes	Compared release must not lower task success or raise tool-error rate (D32)
budgets	json	yes	max cost and duration per run
active	bool	yes

Invariants:

• Thresholds are set per profile by Platform admin through the suite asset (D32) and never by the evaluated asset's author.
• A suite pins its dataset version; a changed dataset is a new suite version.

Events: EvaluationSuiteActivated · Tables: EvaluationSuites · Repository: EvaluationSuiteRepository

EvaluationRun — One execution of a suite against one target release (and optionally a baseline release) over a dataset snapshot — per-metric results, cost, latency, verdict, waiver and the evidence record artifact. (catalog entry)

Field	Type	Required	Description
evaluationId	guid	yes
tenantId	string	yes
suiteId	string	yes
suiteVersion	string	yes
targetReleaseId	string	yes
baselineReleaseId	string	no
datasetSnapshotRef	string	yes
state	enum:EvaluationState	yes
results	json	yes	Per-metric values for target and baseline.
verdict	enum:Verdict	no
waiverApprovalId	guid	no
costUsd	decimal	no
durationMs	long	no
evidenceRef	string	no	Evaluation record artifact read by Asset Publication (ADR-0021).
startedAt	datetime	yes
completedAt	datetime	no

Invariants:

• Evaluations run as Factory runs under a dedicated project with sandbox-class tools only; their model calls carry receipts and are attributed to the evaluation.
• A verdict is Passed only when every threshold is met and the baseline rule holds; a waiver needs an approval of class publication bound to the evaluation id and is recorded as evidence.
• The evidence record is immutable and references the dataset snapshot, the releases and every receipt.

stateDiagram-v2
    Queued --> Running : Started
    Running --> Scoring : TasksCompleted
    Scoring --> Completed : VerdictComputed
    Running --> Failed : InfrastructureFailure
    Queued --> Cancelled : Cancel
    Running --> Cancelled : Cancel

Hold "Alt" / "Option" to enable pan & zoom

Events: EvaluationStarted, EvaluationCompleted, EvaluationFailed, DriftDetected · Tables: EvaluationRuns, EvaluationResults · Repository: EvaluationRunRepository

erDiagram
    EvaluationSuite {
        string suiteId PK
        string version
        string releaseId
        string tenantId
        enum_EvaluationTarget target
        guid datasetId
        json metrics
        json baselineRule
        json budgets
        bool active
    }
    EvaluationRun {
        guid evaluationId PK
        string tenantId
        string suiteId
        string suiteVersion
        string targetReleaseId
        string baselineReleaseId
        string datasetSnapshotRef
        enum_EvaluationState state
        json results
        enum_Verdict verdict
        guid waiverApprovalId
        decimal costUsd
        long durationMs
        string evidenceRef
        datetime startedAt
        datetime completedAt
    }

Hold "Alt" / "Option" to enable pan & zoom

Entities and value objects

None modelled

No entities or value objects are modelled for this item yet.

Data model and storage

Store	Role	Engine / profile	Tables
Operations projection store	primary	azure-sql / nhibernate-sqlserver	EvaluationSuites, EvaluationRuns, EvaluationResults
Artifact content store	blob	blob / azure-blob	—

Table	Owner	Columns	Indexes
dbo.EvaluationSuites	EvaluationSuite	10	0
dbo.EvaluationRuns	EvaluationRun	16	1
dbo.EvaluationResults	EvaluationRun	6	0

Full DDL: appendix.

APIs

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
GET	/factory/evaluation/suites	ListEvaluationSuites	factory.evaluation.read	read	natural	GW-03	Active suites per target with thresholds, datasets and baseline rules.
POST	/factory/evaluation/runs	StartEvaluation	factory.evaluation.run	sandbox	key-header	GW-03	Evaluate a target release (and baseline) with a suite; runs as sandbox-class Factory runs under the evaluation project.
GET	/factory/evaluation/runs/{evaluationId}	GetEvaluation	factory.evaluation.read	read	natural	GW-03	State, per-metric results for target and baseline, verdict, cost, duration, evidence reference.
POST	/factory/evaluation/runs/{evaluationId}/waive	WaiveEvaluation	factory.evaluation.waive	publication	natural	GW-03	Record a waiver (approval-gated, class publication) with a reason; the verdict becomes Waived and evidence says so.
GET	/factory/asset-releases/{releaseId}/evaluations	ListReleaseEvaluations	factory.evaluation.read	read	natural	GW-03	Evaluations of a release with verdicts (the evidence Asset Publication reads).

gRPC service	Method	Streaming	Visibility	Side effects	Summary
Evaluation	GetVerdict	none	internal	read	The latest verdict and evidence reference for a release and suite; used by Asset Publication at the Published gate (ADR-0021).
Evaluation	RunSuite	none	internal	sandbox	Start an evaluation (publication, model deployments, regression).

Messages

Event	Topic	Scope / v	Aggregate	Consumers	Summary
EvaluationSuiteActivated	evaluation.suites.v1.evaluation-suite-activated	integration / 1	EvaluationSuite	StudioBff, AuditService	A suite version is active.
EvaluationStarted	evaluation.runs.v1.evaluation-started	integration / 1	EvaluationRun	StudioBff, DashboardService	An evaluation run started.
EvaluationCompleted	evaluation.runs.v1.evaluation-completed	integration / 1	EvaluationRun	AgentRegistryService, ModelRouterService, FeedbackService, StudioBff, AuditService, DashboardService	A verdict and evidence record exist.
EvaluationFailed	evaluation.runs.v1.evaluation-failed	integration / 1	EvaluationRun	StudioBff, DashboardService	The evaluation could not complete (infrastructure).
DriftDetected	evaluation.runs.v1.drift-detected	integration / 1	EvaluationRun	StudioBff, AlertRuleService, FeedbackService, DashboardService	Production outcomes of a release drifted below its evaluated scores.

EvaluationSuiteActivated payload:

Field	Type	Required	Description
suiteId	string	yes
version	string	yes
tenantId	string	yes
target	enum:EvaluationTarget	yes

EvaluationStarted payload:

Field	Type	Required	Description
evaluationId	guid	yes
tenantId	string	yes
suiteId	string	yes
targetReleaseId	string	yes
baselineReleaseId	string	no

EvaluationCompleted payload:

Field	Type	Required	Description
evaluationId	guid	yes
tenantId	string	yes
suiteId	string	yes
targetReleaseId	string	yes
verdict	enum:Verdict	yes
evidenceRef	string	yes
kind	string	yes	capability-suite

EvaluationFailed payload:

Field	Type	Required	Description
evaluationId	guid	yes
tenantId	string	yes
reason	string	yes

DriftDetected payload:

Field	Type	Required	Description
tenantId	string	yes
targetReleaseId	string	yes
metric	string	yes
evaluated	decimal	yes
observed	decimal	yes

Command	Handled by	Result events	Summary
StartEvaluation	EvaluationRun	EvaluationStarted
WaiveEvaluation	EvaluationRun	EvaluationCompleted

Event	Producer	Handler	Purpose
AssetReleaseSubmitted	AgentRegistryService	RunReleaseGate	Evaluate every candidate release with the suite of its kind (publication gate).
RunCompleted	RunControlService	ScoreEvaluationTask	Score tasks of evaluation runs and feed production outcome statistics.
AttemptOutcomeRecorded	RunControlService	CollectOutcome	Outcome statistics per release for drift.
AttemptOutcomeReported	AgentRuntimeService	CollectRuntimeStats	Corrections, call counts.
ModelInvoked	ModelRouterService	CollectModelStats	Structured-output and tool-call reliability per deployment.
ToolInvoked	ToolAdapterService	CollectToolStats	Tool selection and argument validity.
ContextAssembled	ContextBuilderService	CollectContextStats	Context retrieval quality signals.
RetrievalExecuted	VectorMemoryService	CollectRetrievalStats	Retrieval quality.
ArtifactValidated	ArtifactService	CollectValidationStats	Validator outcomes (patch compile, tests).
ApprovalDecided	RunControlService	ApplyWaiver	Record waivers once approved.

Workers and jobs

Worker	Trigger	Output events	Retry	Idempotency key	Summary
SuiteRunner	event	EvaluationStarted, EvaluationCompleted, EvaluationFailed	exponential	evaluationId + taskId	Executes the tasks of a suite as Factory runs under the evaluation project, collects per-task metrics and computes the verdict against thresholds and the baseline.
DriftMonitor	schedule 0 */6 * * *	DriftDetected	fixed	targetReleaseId + window	Compares production outcome statistics per release with its evaluated scores and raises drift.

Actors and grains

None modelled

No grains are modelled for this item yet.

Sagas and long-running flows

None modelled

No sagas are modelled for this item yet.

sequenceDiagram
    participant AR as AgentRegistryService
    participant Q as QualityScoreService
    participant RC as RunControlService
    participant F as FeedbackService
    participant AS as ArtifactService
    AR-)Q: AssetReleaseSubmitted (candidate InReview)
    Q->>F: GetDatasetSnapshot (suite's pinned dataset)
    loop every task in the snapshot
        Q->>RC: submit run (evaluation project, sandbox class, target release)
        RC-)Q: RunCompleted (+ receipts)
    end
    Q->>Q: score metrics, compare with baseline release, compute verdict
    Q->>AS: store evidence record (immutable)
    Q-)AR: EvaluationCompleted (verdict, evidenceRef)
    AR->>Q: GetVerdict at the Published gate

Hold "Alt" / "Option" to enable pan & zoom

stateDiagram-v2
    [*] --> Queued
    Queued --> Running: Started
    Running --> Scoring: TasksCompleted
    Scoring --> Completed: VerdictComputed
    Running --> Failed: InfrastructureFailure
    Queued --> Cancelled: Cancel
    Running --> Cancelled: Cancel
    Completed --> [*]

Hold "Alt" / "Option" to enable pan & zoom

Options and configuration

Section	Key	Type	Default	Reloadable	Secret	Summary
Observability:Evaluation	EvaluationProjectId	string	factory-evaluation	no	no	Project under which evaluation runs execute with sandbox-class tools.
Observability:Evaluation	DefaultThresholds	json	{ tool-selection: 0.97, valid-arguments: 0.99, structured-output: 0.995, multi-turn: 0.95, instruction-compliance: 0.95, patch-compile: 0.90 }	yes	no	Seed thresholds (D32) until suites override them per profile.
Observability:Evaluation	DriftTolerance	decimal	0.05	yes	no	Allowed drop between evaluated and observed before DriftDetected.

Metrics and observability

Metric	Instrument	Unit	Dimensions	Summary
connectsoft.factory.evaluation.runs	counter	{evaluation}	tenant, target, verdict	Evaluations by verdict.
connectsoft.factory.evaluation.duration	histogram	s	target	Evaluation duration.
connectsoft.factory.evaluation.cost_usd	counter	USD	tenant, target	Evaluation spend.
connectsoft.factory.evaluation.drift	counter	{release}	tenant, metric	Drift detections.
connectsoft.factory.evaluation.waivers	counter	{evaluation}	tenant	Waived verdicts.

Security

Authentication	client-credentials
Scopes	factory.evaluation.read (Read suites, runs and verdicts.); factory.evaluation.run (Start evaluations (Asset maintainer, Platform admin).); factory.evaluation.waive (Waive (Reviewer approval required).)
Roles	AssetMaintainer, Reviewer, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	internal
Max side-effect class	publication
Secrets	—

Dependencies

Outbound

Target	Kind	Purpose	Failure mode	Required
RunControlService	grpc	Submit evaluation tasks as runs; request waiver approvals.	fail-fast	yes
AgentRegistryService	grpc	Release manifests of targets and baselines.	fail-fast	yes
FeedbackService	grpc	Dataset snapshots.	fail-fast	yes
ArtifactService	grpc	Evidence records.	fail-fast	yes
ModelRouterService	grpc	Capability suites invoke deployments directly through the gateway.	degrade	yes

Inbound

Target	Kind	Purpose	Failure mode	Required
AgentRegistryService	grpc	GetVerdict at the Published gate.	—	yes
ModelRouterService	event	Capability evidence for deployments.	—	yes
StudioBff	sync-http	Evaluation views and waivers.	—	yes

Failure and recovery

Scenario	Detection	Recovery	Data impact	RTO (min)
Evaluation budget exceeded	cost or duration above the suite budget	evaluation Failed with reason; the candidate stays InReview	none	0
Dataset snapshot missing	snapshot reference unresolved	evaluation Failed; maintainer re-snapshots the dataset	none	0

NFR targets

Target	Class value (pilot-batch)	Override
latency_p95_ms	0	—
availability_pct	99.5	—
throughput_rps	10	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	medium	—

Deployment

Granularity	module of host · host ObservabilityHost
Topologies	saas-default, hybrid, on-prem, hybrid-compute
Runtime	—; replicas —–—
Ports	—

Current implementation

Nothing executable: the agent specs name evaluation criteria in prose and the original handoff lists the metrics and thresholds. ConnectSoft.Extensions.AI exposes the model invocation surface a suite would call.

Later capabilities

Replay and simulation suites (original design §50) over recorded timelines; adversarial suites for prompt-injection and exfiltration resistance (governance §44); cross-tenant benchmark sharing with anonymised datasets and consent; evaluation of policies by simulated decision sets.

Decisions and traceability

Family	Ids
FR	FR-02, FR-04, FR-11
FUX	FUX-10, FUX-02
NFR	NFR-11, NFR-13
XL	XL-08, XL-15

Related

• Observability overview · Quality gates (ADR-0021) · AgentRegistryService · ModelRouterService · FeedbackService · Evaluation

• Observability microservices · bounded contexts · APIs · events · data model

• Services catalog · Ownership map