AIFACTORY-FEAT-070: QualityScoreService: complete delivery¶
Generated from the architecture model
This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.
A design artefact, not a tracker
This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).
Epic Observability (PF-13) · Increment I4 Trust and accountability · State elaborated · Route backend-developer · Build order 133 of 185 (Factory code)
QualityScoreService is the service of bounded context BC-22 (Evaluation & Learning). Evaluation as a core capability: suites per target (models, agents, skills, tools, workflows, context and memory strategies, policies) with the quality-gate metrics, evaluation runs over datasets of historical tasks, verdicts with baseline comparison, waivers, drift monitoring and the evidence records publication reads (BC-22, D32, original design §47–49). This feature completes the service after its thin delivery in I1: every aggregate, operation, event and worker of the model. Repository ConnectSoft.AI.SoftwareFactory.QualityScoreService.
| Delivers | QualityScoreService (complete) |
| Stubs | — |
| Realizes | adrs: ADR-0021, ADR-0014 |
| Requirements | FR-02, FR-04, FR-11, FUX-02, FUX-10, NFR-11, NFR-13, XL-08, XL-15 |
| Depends on | AIFACTORY-FEAT-065 QualityScoreService: thin delivery for the first outcome |
| Sources | model/services/observability/QualityScoreService.yaml |
Scenarios to trace (12, catalog): SCN-QualityScoreService-001 ✓, SCN-QualityScoreService-002 ✓, SCN-QualityScoreService-003 ✓, SCN-QualityScoreService-004 ✓, SCN-QualityScoreService-005 ✓, SCN-QualityScoreService-006 ✓, SCN-QualityScoreService-007 ✓, SCN-QualityScoreService-008 ✓, SCN-QualityScoreService-009 ✓, SCN-QualityScoreService-010 ✓, SCN-QualityScoreService-011 ✓, SCN-QualityScoreService-012 ✓
AIFACTORY-US-041: A release is published only with evaluation evidence¶
behavior · persona asset-maintainer
As an asset maintainer, I want every release of an asset kind that has an active suite (D117) evaluated by that suite before it can be published, so that nothing runs in a project that was not measured, unless a named person waived it.
This is the integration PBI of the complete delivery of QualityScoreService (BC-22), which follows the thin delivery of I1 and adds every aggregate, operation, event and worker of model/services/observability/QualityScoreService.yaml. Evaluation suites are the runtime read model of the suites in force, listed through ListEvaluationSuites and announced by EvaluationSuiteActivated (AIFACTORY-US-1090). An Asset maintainer or a Platform admin starts an evaluation through StartEvaluation, an internal caller through Evaluation.RunSuite, and the evaluation is announced by EvaluationStarted (AIFACTORY-US-1091). The SuiteRunner worker scores the tasks of the evaluation as sandbox-class Factory runs under the evaluation project, computes the verdict against the thresholds of the suite and the baseline rule of D32, stores the immutable evidence record and publishes EvaluationCompleted, and GetEvaluation shows the result (AIFACTORY-US-1092); an evaluation that cannot complete ends Failed with EvaluationFailed (AIFACTORY-US-1093). Every release candidate that AgentRegistryService submits is evaluated by the suite of its kind, and AgentRegistryService records the verdict and evidence of EvaluationCompleted through its handler RecordEvaluationEvidence and checks that record at the Published gate without a synchronous call (D141), so a release of an asset kind that has an active suite (D117) is not published without a passing or waived verdict (AIFACTORY-US-1094); a release of a kind without a suite is not gated by evaluation, as Publication:Gates:RequireEvaluation of AgentRegistryService states. A Platform admin evaluates a model deployment against its capability suite, whose tasks invoke the deployment directly through ModelRouterService (AIFACTORY-US-1109). From the policy revision that activates the gate, the Draft allowance of the ADR-0021 amendment (D83) is withdrawn. A Reviewer who is not the author may waive a failed verdict with a reason through WaiveEvaluation, under an approval of class publication bound to the evaluation (AIFACTORY-US-1095). The DriftMonitor compares production outcomes per release, stored since the thin delivery of I1, with the evaluated scores and publishes DriftDetected (AIFACTORY-US-1096). The retention, deletion and legal hold orders of RetentionResidencyService are applied to the records of the service (AIFACTORY-US-1097). Every query runs under the tenant row filter of the service, and a resource of another tenant is answered 404, the same answer as for one that does not exist (ADR-0015 amendment D113). The thin scope and its records are kept: the complete delivery reads the records the thin delivery stored and retires nothing a caller relied on.
Acceptance criteria
| AC1 | Given a submitted release of an agent, skill, workflow or policy with an active suite of its kind, when its evaluation runs, then QualityScoreService/EvaluationCompleted carries the verdict and the reference to the immutable evidence record, as AIFACTORY-US-1092 and AIFACTORY-US-1094 prove (ADR-0021). |
| AC2 | Given a release of an asset kind that has an active suite (D117) without a passing or waived verdict, when publication is requested through AgentRegistryService/PublishRelease, then AgentRegistryService/PublishRelease refuses it on the evidence recorded from QualityScoreService/EvaluationCompleted by RecordEvaluationEvidence, makes no synchronous call to QualityScoreService and names the missing evidence (D141); from the policy revision that activates the gate, the Draft allowance of the ADR-0021 amendment no longer applies (ADR-0021). |
| AC3 | Given a waiver with its reason and an approval of class publication bound to the evaluation, when it is recorded through QualityScoreService/WaiveEvaluation, then the verdict is Waived, the release may be published and the waiver is part of its evidence, as AIFACTORY-US-1095 proves (ADR-0021). |
| AC4 | Given the outcomes of a published release fall below its evaluated scores by more than the drift tolerance, when the drift monitor runs, then QualityScoreService/DriftDetected is published, as AIFACTORY-US-1096 proves (model/services/observability/QualityScoreService.yaml). |
| Traces: services | QualityScoreService, AgentRegistryService, FeedbackService, PolicyEngineService |
| Traces: aggregates | QualityScoreService/EvaluationSuite, QualityScoreService/EvaluationRun |
| Traces: apis | QualityScoreService/StartEvaluation, QualityScoreService/GetEvaluation, QualityScoreService/WaiveEvaluation, QualityScoreService/ListReleaseEvaluations, QualityScoreService/Evaluation.GetVerdict, QualityScoreService/Evaluation.RunSuite, AgentRegistryService/PublishRelease |
| Traces: events | QualityScoreService/EvaluationCompleted, QualityScoreService/EvaluationFailed, QualityScoreService/DriftDetected |
| Traces: adrs | ADR-0014, ADR-0015, ADR-0021 |
| Traces: requirements | FR-02, FR-04, FR-11, FUX-02, FUX-10, NFR-11, NFR-13, XL-08, XL-15 |
| Relates: screens | evaluation-runs, release-evidence |
| Depends on | AIFACTORY-US-1090, AIFACTORY-US-1091, AIFACTORY-US-1092, AIFACTORY-US-1093, AIFACTORY-US-1094, AIFACTORY-US-1095, AIFACTORY-US-1096, AIFACTORY-US-1097, AIFACTORY-US-1109, feature:quality-score-service-thin, feature:agent-registry-service-complete, feature:feedback-service, feature:policy-engine-service |
| Sources | model/services/observability/QualityScoreService.yaml, docs/architecture/adr/0021-quality-gates.md, docs/architecture/quality-gates.md, docs/planes/trust-operations/evaluation.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-493 Domain model: EvaluationSuite, EvaluationRun |
implementation | Aggregates with their invariants and the six transitions of EvaluationState as modelled; unit tests cover every invariant and every transition, and a transition the model does not list is refused. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-494 Persistence in quality-score-sql: EvaluationSuites, EvaluationRuns, EvaluationResults |
implementation | Mappings and migrations for the modelled tables with tenant filters and the index IX_Evaluations_Target, added beside the table of accepted events of the thin delivery without moving its records; mapping tests run against an isolated test database, never a shared one. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-495 API hosting: 5 REST and 2 gRPC operations (ListEvaluationSuites, StartEvaluation, GetEvaluation, WaiveEvaluation, ListReleaseEvaluations, Evaluation.GetVerdict, Evaluation.RunSuite) |
implementation | The five REST routes and the two gRPC methods are hosted with JWT validation, scope enforcement per route, workload identity checks on the gRPC methods, problem responses for every modelled refusal code and the Idempotency-Key filter of StartEvaluation; the behaviour of each operation is delivered by AIFACTORY-US-1090 to AIFACTORY-US-1095. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-496 Messaging: publishes 5 events, handles 16 |
implementation | Events are published through the outbox of the quality-score profile under the run envelope (EvaluationSuiteActivated, EvaluationStarted, EvaluationCompleted, EvaluationFailed, DriftDetected); the inbox registers the sixteen consumed events, the ten of the thin delivery among them, idempotently on the event id; what each new handler does is delivered by the behaviour PBIs of this feature. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-497 Workers: SuiteRunner, DriftMonitor |
implementation | Each worker runs with its modelled trigger, retry policy, idempotency key and concurrency partitioned by tenant; a duplicate trigger has no second effect; what each worker does is delivered by AIFACTORY-US-1091, AIFACTORY-US-1092 and AIFACTORY-US-1096. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-498 Security, tenancy and telemetry of QualityScoreService |
implementation | Scopes and roles of the model are enforced, every query is tenant-filtered and a resource of another tenant is answered 404 (ADR-0015 amendment D113), secrets come from secret references, and traces, metrics and health checks follow the observability conventions with the six modelled metrics. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-499 Replace the thin scope of QualityScoreService by the complete delivery |
implementation | The descriptor test of the thin surface is replaced by the descriptor test of the complete surface, the records of accepted events stored since I1 stay where they are and are read by the drift monitor, and the contract stub of FeedbackService is deleted because the service calls the real FeedbackService. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-772 Verify release-needs-evaluation-evidence by running the declared command |
verification | Run dotnet test -c Release --filter Category=Acceptance in the pipeline of the repository; a zero-match filter counts as a failure; it passes and proves AC1 to AC4 of AIFACTORY-US-041 end to end: a submitted release evaluated with its evidence, publication refused without a verdict, a waived verdict that allows publication, and drift published for a release whose outcomes fell. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1090: Evaluation suites are activated as pinned versions, announced and listed with their thresholds, datasets and baseline rules¶
behavior · persona platform-admin
As a platform admin, I want the suites in force per target listed with the thresholds I set per profile, their pinned dataset and their baseline rule, so that an asset maintainer sees what a release must prove before it is evaluated, and no author sets the bar for their own asset.
An EvaluationSuite is the runtime read model of a Published evaluation-suite asset: its target kind (Model, Agent, Skill, Tool, Workflow, ContextStrategy, MemoryStrategy or Policy), the dataset it pins by version, the metrics with their thresholds per model profile, the baseline rule (no regression on task success, no rise in tool-error rate) and the cost and duration budgets of one run. Thresholds are set per profile by a Platform admin through the suite asset and never by the author of the evaluated asset (D32); until a suite overrides them, the seeds of Observability:Evaluation:DefaultThresholds apply. A suite pins its dataset version, so a changed dataset is a new suite version. When a suite version becomes active the service writes its row in EvaluationSuites with Active set, deactivates the earlier version of the same suite, and writes the EvaluationSuiteActivated outbox record with suiteId, version, tenantId and target in the same transaction; StudioBff and AuditService consume it in their own PBIs. The model gives no input for this activation: AssetKind of AgentRegistryService has no evaluation-suite kind, QualityScoreService consumes no AssetReleasePublished, and no operation creates a suite. This PBI therefore delivers the activation transaction behind one activation port and proves it from a test fixture; which input drives the port is the open point reported with this feature. DatasetPublished of FeedbackService (handler RefreshSuiteDataset) records that a dataset version is available, so a new suite version may pin it; it never changes the dataset of an active version. A caller whose scope carries factory.evaluation.read lists the active suites through ListEvaluationSuites, filtered by target, under the tenant row filter; a caller without the scope is refused with 403 and nothing is written.
Acceptance criteria
| AC1 | Given a suite version fed to the activation port, when it is activated, then its QualityScoreService/EvaluationSuite row is active with target, pinned dataset, thresholds per profile, baseline rule and budgets, the earlier version of the same suite is no longer active, and the QualityScoreService/EvaluationSuiteActivated outbox record is written in the same transaction (SCN-QualityScoreService-008, ADR-0021). |
| AC2 | Given the record was written, when the dispatcher of the quality-score profile publishes QualityScoreService/EvaluationSuiteActivated on evaluation.suites.v1.evaluation-suite-activated, then it is sent under the canonical envelope with suiteId, version, tenantId and target, and a transport outage is retried with backoff so it is neither lost nor duplicated under its event id (SCN-QualityScoreService-008, ADR-0012). |
| AC3 | Given a caller with factory.evaluation.read, when QualityScoreService/ListEvaluationSuites is called with a target filter, then the response is 200 with the active suites of that target in the caller tenant, each with thresholds per profile, pinned dataset, baseline rule and budgets; a caller without the scope is refused with 403 and nothing is written (SCN-QualityScoreService-001, ADR-0015). |
| AC4 | Given FeedbackService/DatasetPublished for a dataset an active suite pins at an older version, when RefreshSuiteDataset handles it, then the new version is recorded as available and the active suite keeps its pinned version; given a suite without its own thresholds, then the seeds of Observability:Evaluation:DefaultThresholds in model/services/observability/QualityScoreService.yaml apply. |
| Traces: services | QualityScoreService, FeedbackService |
| Traces: aggregates | QualityScoreService/EvaluationSuite |
| Traces: apis | QualityScoreService/ListEvaluationSuites |
| Traces: events | QualityScoreService/EvaluationSuiteActivated, FeedbackService/DatasetPublished |
| Traces: adrs | ADR-0012, ADR-0015, ADR-0021 |
| Traces: requirements | FUX-10, NFR-11 |
| Traces: scenarios | SCN-QualityScoreService-001, SCN-QualityScoreService-008 |
| Relates: screens | evaluation-runs |
| Depends on | feature:quality-score-service-thin, feature:feedback-service |
| Sources | model/services/observability/QualityScoreService.yaml, model/services/control-plane/AgentRegistryService.yaml, docs/architecture/adr/0021-quality-gates.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-537 Suites per target with the gate metrics behind the activation port |
implementation | The activation port writes an EvaluationSuites row per suite version with target, pinned dataset, thresholds per profile, baseline rule and budgets, deactivates the earlier version and writes EvaluationSuiteActivated to the outbox in the same transaction; DefaultThresholds seed a suite without its own thresholds; RefreshSuiteDataset records an available dataset version without changing an active suite; ListEvaluationSuites returns the active suites by target under the tenant row filter. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4365 Suite fixtures for the activation port and the dataset refresh |
fixture | The acceptance suite holds suite versions for an agent, a skill, a workflow and a policy target, two versions of one suite, a suite without thresholds, a DatasetPublished fixture from the contracts package and a transport switch for the outbox dispatcher. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4366 Verify suite activation and listing through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-QualityScoreService-001\|Scenario=SCN-QualityScoreService-008)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1090: activation with its outbox record, the publication and payload of EvaluationSuiteActivated through an outage, the listing by target with its 403, the dataset refresh and the seeded thresholds. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1091: Start an evaluation of a target release against a baseline with an active suite, from a person or an internal caller¶
behavior · persona asset-maintainer
As an asset maintainer, I want to evaluate a release of my asset, and optionally a baseline release, with an active suite, so that I know before submission whether the candidate meets the thresholds and does not regress against the release in force.
An Asset maintainer or a Platform admin whose scope carries factory.evaluation.run starts an evaluation through StartEvaluation with an Idempotency-Key, naming an active suite, a target release and optionally a baseline release; an internal caller with a workload identity starts one through the gRPC method Evaluation.RunSuite with release-manifest-v1 and receives ack-v1, for a publication gate, a model deployment or a regression. The tenant and the project come from the trusted scope, never from the body (ADR-0015). The service resolves the target and the baseline releases through AgentRegistryService and the pinned dataset version of the suite to its snapshot artifact through Learning.GetDatasetSnapshot of FeedbackService; both dependencies are required and fail fast, so when either is unreachable the call fails, nothing is written and the caller repeats it under the same key. The service then writes the EvaluationRun in state Queued with suiteId, suiteVersion, targetReleaseId, baselineReleaseId and datasetSnapshotRef, together with the EvaluationStarted outbox record, the result event of the command, in one transaction, and answers 202 with the evaluation location. A repeated call under the same key and body returns the same evaluation and changes nothing; the same key with a different body, a suite that is not active and a release that AgentRegistryService does not resolve for the tenant are refused with 409 and a problem body that names the cause; a caller without the scope is refused with 403; none of these writes a row or publishes. Over gRPC the service reads the forwarded tenant only when the workload token holds the service scope that names QualityScoreService (ADR-0015 amendment D114): a caller without a workload identity is refused with UNAUTHENTICATED, a workload token without that scope with PERMISSION_DENIED before any read or write, and an invalid request with INVALID_ARGUMENT, with no state change. A RunSuite request is deduplicated on the releaseId of the manifest, the suite, which is the evaluation.suiteId of the manifest or else the active suite of the asset kind, and the kind of the evaluation: a retried request while an evaluation with that key is Queued, Running or Scoring queues no second evaluation and answers ack-v1 again. The kind of an evaluation, one of the values capability-suite, release-gate, regression and drift that EvaluationCompleted carries, is taken from the path that starts it and kept on the run: RunReleaseGate starts a release-gate evaluation, a start for a suite whose target is Model starts a capability-suite evaluation, and every other start through StartEvaluation or RunSuite starts a regression evaluation; no start path of the model gives drift (D131). EvaluationRun and the table EvaluationRuns have no field for the kind, a gap of the model reported with this feature as a correction; the migration of this feature adds the column that keeps it. The SuiteRunner worker picks up a Queued evaluation, submits the tasks of the dataset snapshot as sandbox-class Factory runs under the project Observability:Evaluation:EvaluationProjectId, and moves the run from Queued to Running (trigger Started); its idempotency key is evaluationId plus taskId, so a duplicate trigger submits no task twice. SuiteRunner submits each task through SubmitRun of RunControlService with a service token that holds factory.runs.submit and names the evaluation project, and each run records QualityScoreService as its submitting service (D131). The model calls of the tasks carry the attempt token of their lease as the proof that they belong to a run (ADR-0015 amendment D125) and their receipts are attributed to the evaluation.
Acceptance criteria
| AC1 | Given a caller with factory.evaluation.run, an active suite and a target release that resolves, when QualityScoreService/StartEvaluation is called with an Idempotency-Key, then the response is 202, the QualityScoreService/EvaluationRun is Queued with suite, suite version, target, baseline and dataset snapshot reference, and the QualityScoreService/EvaluationStarted outbox record is written in the same transaction; the same key and body return the same evaluation and change nothing (SCN-QualityScoreService-002, ADR-0014). |
| AC2 | Given the same key with a different body, a suite that is not active, a release that does not resolve for the tenant, or a caller without the scope, when QualityScoreService/StartEvaluation is called, then it is refused with 409 naming the cause or with 403, no row is written and nothing is published; given AgentRegistryService or FeedbackService/Learning.GetDatasetSnapshot unreachable, then the call fails fast, nothing is written and the repeat under the same key is accepted once both answer (SCN-QualityScoreService-002, ADR-0015). |
| AC3 | Given an internal caller whose workload token holds the service scope naming QualityScoreService and release-manifest-v1, when QualityScoreService/Evaluation.RunSuite is called, then the reply carries ack-v1, one evaluation is Queued for that release with its kind kept on the run, and a retried request with the same release, suite and kind while that evaluation is Queued, Running or Scoring queues no second evaluation; a caller without a workload identity is refused with UNAUTHENTICATED, a workload token without the service scope naming QualityScoreService with PERMISSION_DENIED before any read or write, and an invalid request with INVALID_ARGUMENT, with no state change (SCN-QualityScoreService-007, ADR-0015). |
| AC4 | Given the record was written, when the dispatcher publishes QualityScoreService/EvaluationStarted on evaluation.runs.v1.evaluation-started, then it is sent under the canonical envelope with evaluationId, tenantId, suiteId, targetReleaseId and baselineReleaseId and an outage is retried so it is neither lost nor duplicated; when SuiteRunner picks the evaluation up, its tasks are submitted once each through RunControlService/SubmitRun with a service token that holds factory.runs.submit and names the evaluation project, each run records QualityScoreService as its submitting service, and the run moves from Queued to Running (SCN-QualityScoreService-009, ADR-0012). |
| Traces: services | QualityScoreService, AgentRegistryService, FeedbackService, RunControlService |
| Traces: aggregates | QualityScoreService/EvaluationRun |
| Traces: apis | QualityScoreService/StartEvaluation, QualityScoreService/Evaluation.RunSuite, FeedbackService/Learning.GetDatasetSnapshot, RunControlService/SubmitRun |
| Traces: events | QualityScoreService/EvaluationStarted |
| Traces: descriptors | release-manifest-v1, ack-v1 |
| Traces: adrs | ADR-0012, ADR-0014, ADR-0015, ADR-0021 |
| Traces: requirements | FR-04, XL-08 |
| Traces: scenarios | SCN-QualityScoreService-002, SCN-QualityScoreService-007, SCN-QualityScoreService-009 |
| Relates: screens | evaluation-runs |
| Depends on | AIFACTORY-US-1090, feature:agent-registry-service-complete, feature:feedback-service, feature:run-control-service |
| Sources | model/services/observability/QualityScoreService.yaml, model/services/observability/FeedbackService.yaml, docs/planes/trust-operations/observability/services/quality-score-service/index.md#apis |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4367 StartEvaluation and Evaluation.RunSuite with idempotency, scope and dependency checks |
implementation | Both operations take tenant and project from the trusted scope, resolve the releases through AgentRegistryService and the dataset snapshot through Learning.GetDatasetSnapshot, fail fast without a write when either is unreachable, write the EvaluationRun in Queued with its kind and its EvaluationStarted outbox record in one transaction and answer 202 or ack-v1; RunSuite deduplicates on release, suite and kind while the evaluation is Queued, Running or Scoring; the idempotent repeat, the 409 for a reused key with a different body, an inactive suite and an unresolved release, the 403, and the gRPC refusals UNAUTHENTICATED, PERMISSION_DENIED for a workload token without the service scope naming QualityScoreService before any read or write, and INVALID_ARGUMENT are covered by acceptance tests; the migration adds the column of the kind to EvaluationRuns. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4368 SuiteRunner submission of the tasks of an evaluation under the evaluation project |
implementation | SuiteRunner picks up Queued evaluations per tenant partition, submits each task of the dataset snapshot once as a sandbox-class Factory run under Observability:Evaluation:EvaluationProjectId through SubmitRun of RunControlService with a service token that holds factory.runs.submit and names the evaluation project, so each run records QualityScoreService as its submitting service, keyed by evaluationId and taskId, and moves the run from Queued to Running; RunControlService is a fake in the acceptance suite. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4369 Verify the start of an evaluation through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-QualityScoreService-002\|Scenario=SCN-QualityScoreService-007\|Scenario=SCN-QualityScoreService-009)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1091 with fakes of AgentRegistryService, FeedbackService and RunControlService: the accepted start and its repeat, every refusal, the fail-fast dependencies, the gRPC start with its UNAUTHENTICATED, PERMISSION_DENIED and INVALID_ARGUMENT refusals and the retry deduplicated on release, suite and kind, the kind kept on the run, the publication of EvaluationStarted and the tasks submitted once through SubmitRun with the service token that names the evaluation project. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1092: Score the tasks of an evaluation, compute the verdict against thresholds and baseline, and record immutable evidence¶
behavior · persona asset-maintainer
As an asset maintainer, I want the verdict of an evaluation computed from the tasks it ran, compared with the thresholds of my profile and with the release in force, so that I can read per metric why a release passed or failed, and publication can rely on evidence nobody can alter.
The tasks of an evaluation run as Factory runs. When one ends, RunCompleted of RunControlService (handler ScoreEvaluationTask) scores it: the per-metric values of the task for the target, and for the baseline where one is named, are written as an EvaluationResults row keyed by evaluationId and taskId with the runId that executed it and whether it passed, once per task, idempotent on the event id. The signals the thin delivery already stores, model and tool receipts, context and retrieval statistics and validator outcomes of the attempts of those runs, feed the metrics of the suite. When every task of the snapshot has a result, SuiteRunner moves the run from Running to Scoring (trigger TasksCompleted), computes the verdict and moves it from Scoring to Completed (trigger VerdictComputed). The verdict is Passed only when every threshold of the suite for the profile is met and the baseline rule holds, that is task success does not fall and the tool-error rate does not rise against the baseline release (D32); otherwise it is Failed. The model also lists the verdict Inconclusive and gives no rule that produces it, so this delivery never produces it and the open point is reported with this feature. The service stores the evidence record through ArtifactService: it is immutable and references the dataset snapshot, the target and baseline releases and every receipt of the evaluation; its reference is kept in evidenceRef, with the cost and the duration of the run. The EvaluationCompleted outbox record with evaluationId, tenantId, suiteId, targetReleaseId, verdict, evidenceRef and kind, the kind kept on the run when it started (AIFACTORY-US-1091), is written in the transaction that completes the run. AgentRegistryService, ModelRouterService, FeedbackService, StudioBff and AuditService consume it in their own PBIs. ArtifactService is a required dependency that fails fast: when the evidence cannot be stored, the run stays in Scoring and SuiteRunner retries with its exponential policy of three attempts; when all three fail, the run stays in Scoring with no verdict and no event, the failure is counted and logged with the evaluation id, and the next trigger of SuiteRunner retries it. EvaluationState has no transition from Scoring to Failed, so a run whose evidence can never be stored has no terminal state; this is an open point reported with this feature. A caller whose scope carries factory.evaluation.read reads an evaluation through GetEvaluation: its state, the per-metric results for target and baseline, the verdict, the cost, the duration and the evidence reference. An evaluation of another tenant is answered 404, the same answer as for an id that does not exist (ADR-0015 amendment D113); a missing scope is 403; nothing is written by the read. The metrics connectsoft.factory.evaluation.runs, connectsoft.factory.evaluation.duration and connectsoft.factory.evaluation.cost_usd move with each completed run.
Acceptance criteria
| AC1 | Given an evaluation in Running whose task runs end, when RunControlService/RunCompleted arrives for each, then one EvaluationResults row per task holds the metrics for target and baseline and whether it passed, a redelivered event writes nothing, and the QualityScoreService/EvaluationRun moves to Scoring when the last task has a result (SCN-QualityScoreService-010, model/services/observability/QualityScoreService.yaml). |
| AC2 | Given results that meet every threshold of the profile and a baseline rule that holds, when the verdict is computed, then the verdict is Passed; given one metric below its threshold, or a fall of task success or a rise of the tool-error rate against the baseline, then it is Failed; in both cases the evidence record is stored once through ArtifactService and referenced in evidenceRef, the run is Completed and the QualityScoreService/EvaluationCompleted outbox record is written in the same transaction (SCN-QualityScoreService-010, ADR-0021). |
| AC3 | Given the record was written, when the dispatcher publishes QualityScoreService/EvaluationCompleted on evaluation.runs.v1.evaluation-completed, then it is sent under the canonical envelope with evaluationId, tenantId, suiteId, targetReleaseId, verdict, evidenceRef and kind, and an outage is retried so it is neither lost nor duplicated; given ArtifactService unreachable, then the run stays in Scoring and nothing is published until the evidence is stored; given ArtifactService fails on all three attempts of SuiteRunner, then the run stays in Scoring with no verdict and no event, the failure is counted and logged, and the next trigger of SuiteRunner retries it (SCN-QualityScoreService-010, ADR-0012). |
| AC4 | Given a caller with factory.evaluation.read and an evaluation of the caller tenant, when QualityScoreService/GetEvaluation is called, then the response is 200 with state, per-metric results for target and baseline, verdict, cost, duration and evidence reference; an evaluation of another tenant and an unknown id are both answered 404, a missing scope 403, and nothing is written (SCN-QualityScoreService-003, ADR-0015). |
| Traces: services | QualityScoreService, RunControlService, ArtifactService |
| Traces: aggregates | QualityScoreService/EvaluationRun |
| Traces: apis | QualityScoreService/GetEvaluation |
| Traces: events | QualityScoreService/EvaluationCompleted, RunControlService/RunCompleted |
| Traces: adrs | ADR-0012, ADR-0015, ADR-0021 |
| Traces: requirements | FUX-10, NFR-11 |
| Traces: scenarios | SCN-QualityScoreService-003, SCN-QualityScoreService-010 |
| Relates: screens | evaluation-runs, release-evidence |
| Depends on | AIFACTORY-US-1091, feature:run-control-service, feature:artifact-service |
| Sources | model/services/observability/QualityScoreService.yaml, docs/architecture/quality-gates.md, docs/planes/trust-operations/evaluation.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4370 ScoreEvaluationTask and the verdict of SuiteRunner against thresholds and the baseline rule |
implementation | ScoreEvaluationTask writes one EvaluationResults row per task of an evaluation, idempotent on the event id; SuiteRunner moves the run Running to Scoring when every task has a result, computes Passed or Failed from the thresholds per profile and the baseline rule of D32, stores the immutable evidence record through ArtifactService with the snapshot, both releases and every receipt, and completes the run with its EvaluationCompleted outbox record, carrying the kind kept on the run, in one transaction; the run stays in Scoring while the store fails, and after the third failed attempt it stays in Scoring with no verdict and no event, the failure is counted and logged with the evaluation id, and the next SuiteRunner trigger retries it; Inconclusive is never produced. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4371 GetEvaluation read model with results, verdict, cost, duration and evidence |
implementation | GetEvaluation returns state, per-metric results for target and baseline, verdict, cost, duration and evidence reference under the tenant row filter; an evaluation of another tenant and an unknown id are both 404, a missing scope 403. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4372 Verify scoring, verdict, evidence and the read model through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-QualityScoreService-003\|Scenario=SCN-QualityScoreService-010)"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1092 with a fake ArtifactService: results per task and their redelivery, Passed, Failed on a threshold and Failed on the baseline rule, the evidence stored once, the publication and payload of EvaluationCompleted, the run held in Scoring while the store fails, the run left in Scoring without a verdict or event after three failed attempts and retried at the next trigger, and the read with its 404 and 403. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1093: An evaluation that exceeds its budget, misses its dataset snapshot or loses its infrastructure ends Failed with a reason¶
behavior · persona asset-maintainer
As an asset maintainer, I want an evaluation that cannot complete to end Failed with the reason named, so that my candidate is not published on a missing verdict and I know whether to re-snapshot the dataset or retry.
Three failure modes of model/services/observability/QualityScoreService.yaml end an evaluation without a verdict. Evaluation budget exceeded: SuiteRunner tracks the cost and the duration of a Running evaluation against the budgets of its suite, and when either is above the budget it stops submitting tasks and moves the run from Running to Failed with the reason budget exceeded; a release candidate under evaluation stays InReview. Dataset snapshot missing: when the snapshot reference of the run no longer resolves through FeedbackService while it runs, the run ends Failed with the reason snapshot missing, and the maintainer re-snapshots the dataset and starts a new evaluation. Infrastructure failure: when SuiteRunner exhausts its retry policy (exponential, three attempts) on a task submission or on the dependencies of the run, the run moves from Running to Failed with the reason of the last failure. In each case the EvaluationFailed outbox record with evaluationId, tenantId and reason is written in the transaction that fails the run, completedAt is set, no verdict and no evidence record are written, and StudioBff consumes the event in its own PBI. A Failed run is terminal: a late RunCompleted of one of its tasks writes no result and moves nothing. The model lists two further transitions, Queued to Cancelled and Running to Cancelled with the trigger Cancel, and no operation, event or worker of the model raises that trigger; how an evaluation is cancelled is an open point reported with this feature, and this PBI delivers the two transitions in the domain model only, refusing any other path to Cancelled.
Acceptance criteria
| AC1 | Given a Running evaluation whose cost or duration rises above the budget of its suite, when SuiteRunner checks it, then no further task is submitted, the QualityScoreService/EvaluationRun is Failed with the reason budget exceeded and no verdict, and the QualityScoreService/EvaluationFailed outbox record is written in the same transaction (SCN-QualityScoreService-011, model/services/observability/QualityScoreService.yaml). |
| AC2 | Given a Running evaluation whose dataset snapshot no longer resolves, or whose task submission fails after the third attempt of the retry policy, when SuiteRunner handles it, then the run is Failed with the reason snapshot missing or the reason of the last failure, and the QualityScoreService/EvaluationFailed record is written with it; a later RunControlService/RunCompleted of one of its tasks writes no result (SCN-QualityScoreService-011). |
| AC3 | Given the record was written, when the dispatcher publishes QualityScoreService/EvaluationFailed on evaluation.runs.v1.evaluation-failed, then it is sent under the canonical envelope with evaluationId, tenantId and reason, and an outage is retried so it is neither lost nor duplicated under its event id (SCN-QualityScoreService-011, ADR-0012). |
| AC4 | Given the domain model, when a transition to Cancelled is requested from Queued or Running, then it is applied, and from any other state it is refused; no operation of the service raises it in this delivery (model/services/observability/QualityScoreService.yaml). |
| Traces: services | QualityScoreService |
| Traces: aggregates | QualityScoreService/EvaluationRun |
| Traces: events | QualityScoreService/EvaluationFailed, RunControlService/RunCompleted |
| Traces: adrs | ADR-0012, ADR-0021 |
| Traces: requirements | NFR-12 |
| Traces: scenarios | SCN-QualityScoreService-011 |
| Depends on | AIFACTORY-US-1091, AIFACTORY-US-1092, feature:run-control-service |
| Sources | model/services/observability/QualityScoreService.yaml, docs/planes/trust-operations/observability/services/quality-score-service/index.md#failure-and-recovery |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4373 Budget, snapshot and retry-exhaustion failures of SuiteRunner |
implementation | SuiteRunner fails a Running evaluation when its cost or duration exceeds the suite budget, when its snapshot no longer resolves or when its retry policy is exhausted, each time with the reason, completedAt set, no verdict and the EvaluationFailed outbox record in the same transaction; a result for a task of a Failed run is ignored; the Cancel transitions exist in the domain model only. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4374 Verify the failure branches of an evaluation through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-QualityScoreService-011"; a zero-match filter counts as a failure; the tests prove AC1 to AC4 of AIFACTORY-US-1093: the budget breach, the missing snapshot, the exhausted retry policy, the late task result ignored, the publication and payload of EvaluationFailed through an outage, and the guarded Cancel transitions. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1094: Every submitted release is evaluated by the suite of its kind and its verdict is recorded for the Published gate¶
behavior · persona reviewer
As a reviewer, I want the verdict and evidence of a candidate release in front of me and enforced at publication, so that no release of an asset kind that has an active suite (D117) is published without a passing or waived evaluation, and the Draft allowance ends when the gate exists.
AgentRegistryService publishes AssetReleaseSubmitted when a release candidate enters InReview; the handler RunReleaseGate of QualityScoreService evaluates every candidate with the active suite of its kind, with the current Published release of the same asset as the baseline where one exists, as an evaluation of kind release-gate started by the same path as AIFACTORY-US-1091, once per release and suite, idempotent on the event id. A candidate whose kind has no active suite is not evaluated, as SubmitRelease of AgentRegistryService states. EvaluationCompleted reaches AgentRegistryService, whose handler RecordEvaluationEvidence attaches the verdict and evidence to the release. At the Published gate AgentRegistryService checks the evidence it recorded itself and makes no synchronous call to QualityScoreService (D141); for a release of an asset kind that has an active suite (D117), AgentRegistryService/PublishRelease refuses it when the latest recorded verdict is neither Passed nor Waived and names the missing evidence, which feature agent-registry-service-complete proves on its side. The internal gRPC method Evaluation.GetVerdict stays a read for other callers: a workload caller sends release-manifest-v1 and receives the latest verdict and evidence reference for the release and suite in release-manifest-v1 (D141). A release with no evaluation in the tenant of the caller, and a release of another tenant, are answered NOT_FOUND, never confirming that a foreign release exists (ADR-0015 amendment D113); the service reads the forwarded tenant only when the workload token holds the service scope that names QualityScoreService (ADR-0015 amendment D114), so a caller without a workload identity is refused with UNAUTHENTICATED and a workload token without that scope with PERMISSION_DENIED before any read; the read writes nothing. The request and reply descriptor of Evaluation.GetVerdict and Evaluation.RunSuite, release-manifest-v1, requires publishedAt, publishedBy and state, which a candidate in InReview at the Published gate does not have, and its evaluation verdict enum (passed, failed, waived) has no Inconclusive; this is an open point reported with this feature, and until the owner decides the service reads only releaseId, assetKind and evaluation.suiteId of the request and refuses a request without releaseId with INVALID_ARGUMENT. A person whose scope carries factory.evaluation.read lists the evaluations of a release with their verdicts through ListReleaseEvaluations, the evidence Asset Publication reads and the release evidence view shows; a release with no evaluation in the caller tenant and a release of another tenant are answered 404, a missing scope 403. With the gate in place the platform policy rule of D83 is withdrawn: the policy revision that activates the gate removes the Draft allowance of the ADR-0021 amendment, and from that revision a release is published only with its evaluation evidence or a recorded waiver.
Acceptance criteria
| AC1 | Given AgentRegistryService/AssetReleaseSubmitted for a candidate whose kind has an active suite, when RunReleaseGate handles it, then one evaluation of kind release-gate is Queued for the candidate with the current Published release as baseline, and a redelivered event starts none; given a kind without an active suite, then no evaluation starts (model/services/observability/QualityScoreService.yaml, ADR-0021). |
| AC2 | Given a completed evaluation of a release, when a workload caller calls QualityScoreService/Evaluation.GetVerdict with its workload identity, then the reply carries release-manifest-v1 with the latest verdict and evidence reference for that release and suite; a release without evaluation in the tenant and a release of another tenant are answered NOT_FOUND, a caller without identity UNAUTHENTICATED, a workload token without the service scope naming QualityScoreService PERMISSION_DENIED before any read, a request without releaseId INVALID_ARGUMENT, and nothing is written (SCN-QualityScoreService-006, ADR-0015). |
| AC3 | Given a release of an asset kind that has an active suite (D117) whose latest verdict is Failed, when AgentRegistryService/PublishRelease is called, then AgentRegistryService/PublishRelease refuses it on the evidence recorded from QualityScoreService/EvaluationCompleted by RecordEvaluationEvidence, makes no synchronous call to QualityScoreService and names the missing evidence (D141); given the policy revision that activates the gate, then the rule of D83 is removed and a Draft release no longer runs under the allowance of the ADR-0021 amendment (ADR-0021). |
| AC4 | Given a caller with factory.evaluation.read, when QualityScoreService/ListReleaseEvaluations is called for a release of the caller tenant, then the response is 200 with its evaluations and verdicts; a release with no evaluation in the tenant and a release of another tenant are both answered 404, a missing scope 403, and nothing is written (SCN-QualityScoreService-005, ADR-0015). |
| Traces: services | QualityScoreService, AgentRegistryService, PolicyEngineService |
| Traces: apis | QualityScoreService/Evaluation.GetVerdict, QualityScoreService/ListReleaseEvaluations, AgentRegistryService/PublishRelease |
| Traces: events | AgentRegistryService/AssetReleaseSubmitted, QualityScoreService/EvaluationCompleted |
| Traces: descriptors | release-manifest-v1 |
| Traces: adrs | ADR-0015, ADR-0021 |
| Traces: requirements | FR-02, FUX-02, XL-08 |
| Traces: scenarios | SCN-QualityScoreService-005, SCN-QualityScoreService-006 |
| Relates: screens | release-evidence, release-detail |
| Depends on | AIFACTORY-US-1091, AIFACTORY-US-1092, feature:agent-registry-service-complete, feature:policy-engine-service |
| Sources | model/services/observability/QualityScoreService.yaml, model/services/control-plane/AgentRegistryService.yaml, docs/architecture/adr/0021-quality-gates.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-538 Verify the release gate between submission and publication against QualityScoreService |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=ReleaseGateAgainstQualityScore" in the pipeline of ConnectSoft.AI.SoftwareFactory.AgentRegistryService against a deployed QualityScoreService; a zero-match filter counts as a failure; the tests prove AC3 of AIFACTORY-US-1094 on the side of publication: a release of an asset kind with an active suite whose latest verdict is Failed is refused by AgentRegistryService/PublishRelease on the evidence that RecordEvaluationEvidence recorded from QualityScoreService/EvaluationCompleted, with no synchronous call to QualityScoreService, and the refusal names the missing evidence (D141), a release without any recorded verdict is refused the same way, and a Waived verdict lets the release pass with the waiver in its evidence; the removal of the rule of D83 in the second sentence of AC3 is proven by the tests of AIFACTORY-TASK-512 in ConnectSoft.AI.SoftwareFactory.PolicyEngineService. |
ConnectSoft.AI.SoftwareFactory.AgentRegistryService |
AIFACTORY-TASK-512 Withdraw the waiver rule for Draft releases |
implementation | The platform policy rule of decision D83 is removed in the policy revision that activates the evaluation gate (ADR-0021); from that revision a release is published only with its evaluation evidence or a recorded waiver. | ConnectSoft.AI.SoftwareFactory.PolicyEngineService |
AIFACTORY-TASK-4375 RunReleaseGate, Evaluation.GetVerdict and ListReleaseEvaluations |
implementation | RunReleaseGate queues one release-gate evaluation per submitted candidate and suite with the current Published release as baseline, idempotent on the event id, and none for a kind without an active suite; GetVerdict reads releaseId, assetKind and evaluation.suiteId of the request and returns the latest verdict and evidence reference in release-manifest-v1 to a workload whose token holds the service scope naming QualityScoreService, NOT_FOUND for no evaluation or another tenant, UNAUTHENTICATED without a workload identity, PERMISSION_DENIED for a workload token without that scope before any read, INVALID_ARGUMENT without releaseId; ListReleaseEvaluations returns the evaluations of a release under the tenant row filter, 404 for none or another tenant, 403 without the scope. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4376 Verify the release gate and the verdict reads through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&(Scenario=SCN-QualityScoreService-005\|Scenario=SCN-QualityScoreService-006)"; a zero-match filter counts as a failure; the tests prove AC1, AC2 and AC4 of AIFACTORY-US-1094: the gate evaluation queued once per candidate, none without a suite, the verdict read with its NOT_FOUND, UNAUTHENTICATED, PERMISSION_DENIED and INVALID_ARGUMENT, and the release listing with its 404 and 403; AC3 is proven by AIFACTORY-TASK-538 and AIFACTORY-TASK-512. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1095: Waive a failed evaluation with a reason under an approval of class publication bound to the evaluation¶
behavior · persona reviewer
As a reviewer, I want to waive a failed verdict only with a reason and an approval bound to that evaluation, so that a release may be published without passing its suite only when a named person accepted that, and the evidence says so.
A caller whose scope carries factory.evaluation.waive asks to waive a completed evaluation through WaiveEvaluation, with a reason. A waiver is an approval-gated action of class publication: a Reviewer who is not the author of the evaluated release approves it, and the approval is bound to the evaluation id (ADR-0021, D32). The operation follows one flow. The first call with a reason on a Completed evaluation whose verdict is Failed asks RunControlService, its outbound dependency for waiver approvals, for one approval request bound to the evaluation id and carrying the reason, and answers 412 with a problem response that names the pending approval; the request is idempotent on the evaluation id while it is pending, so a repeated call answers 412 again and raises no second approval request, and the verdict is unchanged. The waiver is recorded by whichever comes first: the handler ApplyWaiver of ApprovalDecided of RunControlService when the decision reports the bound approval Approved, or a later call of WaiveEvaluation that finds the bound approval Approved in RunControlService. Either sets the verdict to Waived with waiverApprovalId, extends the evidence record through ArtifactService with the reason, the approver and the approval, and writes the EvaluationCompleted outbox record with the verdict Waived in the same transaction, the result event of the command, so AgentRegistryService attaches the waiver to the release; the other path then finds the evaluation Waived by that approval and changes nothing. A call on an evaluation already Waived by that approval answers 200 with the recorded state and changes nothing. A rejected, expired, stale or cancelled approval leaves the verdict Failed, and a later call raises a new approval request and answers 412. The other answers follow the model: 404 for an evaluation of another tenant or an unknown id (ADR-0015 amendment D113); 403 for a missing scope; 409 only for an evaluation that is not Completed or whose verdict is Passed. The approval is requested through RequestApproval of RunControlService with the subject kind evaluation-run and the evaluation id, under the service identity of QualityScoreService with the scope factory.approvals.request (D131, D135). The service calls no Policy.Decide: the eligibility of the approver is decided by Run Control and the approval policy, and policy-decision-v1 already has the kinds approval-eligibility and publication. The counter connectsoft.factory.evaluation.waivers moves with each recorded waiver.
Acceptance criteria
| AC1 | Given a Completed evaluation with the verdict Failed and an approval of class publication bound to it that RunControlService/ApprovalDecided reports Approved, when ApplyWaiver handles the decision, then the QualityScoreService/EvaluationRun has the verdict Waived with waiverApprovalId, the evidence names the reason, the approver and the approval, and the QualityScoreService/EvaluationCompleted outbox record with the verdict Waived is written in the same transaction (SCN-QualityScoreService-004, ADR-0021). |
| AC2 | Given a caller with factory.evaluation.waive and a Completed evaluation with the verdict Failed, when QualityScoreService/WaiveEvaluation is called with a reason, then one approval request with the subject evaluation-run and the evaluation id is raised through RunControlService/RunControl.RequestApproval under the service identity of QualityScoreService with factory.approvals.request and the answer is 412, and a repeated call while it is pending answers 412 and raises no second approval request; when a later call finds the bound approval Approved, then it records the waiver as AC1 states and answers 200, and a call on an evaluation already Waived by that approval answers 200 with the recorded state and changes nothing; it answers 409 for an evaluation not Completed or whose verdict is Passed, 404 for an evaluation of another tenant or an unknown id and 403 without the scope, each with a problem response and no state change (SCN-QualityScoreService-004, ADR-0015). |
| AC3 | Given an approval for the waiver that RunControlService/ApprovalDecided reports Rejected, Expired, Stale or Cancelled, when ApplyWaiver handles it, then the verdict stays Failed, nothing is published, a redelivered decision changes nothing and a later call of QualityScoreService/WaiveEvaluation raises a new approval request and answers 412; given the waiver already recorded by a call, when the Approved decision arrives, then ApplyWaiver changes nothing; given a recorded waiver, then connectsoft.factory.evaluation.waivers is incremented once (SCN-QualityScoreService-004, model/services/observability/QualityScoreService.yaml). |
| Traces: services | QualityScoreService, RunControlService |
| Traces: aggregates | QualityScoreService/EvaluationRun |
| Traces: apis | QualityScoreService/WaiveEvaluation, RunControlService/RunControl.RequestApproval |
| Traces: events | RunControlService/ApprovalDecided, QualityScoreService/EvaluationCompleted |
| Traces: descriptors | approval-request-v1 |
| Traces: adrs | ADR-0015, ADR-0021 |
| Traces: requirements | FUX-10, XL-08 |
| Traces: scenarios | SCN-QualityScoreService-004 |
| Relates: screens | evaluation-runs, approvals-list |
| Depends on | AIFACTORY-US-1092, feature:run-control-service |
| Sources | model/services/observability/QualityScoreService.yaml, docs/reference/schemas/approval.schema.json, docs/architecture/adr/0021-quality-gates.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4377 WaiveEvaluation, the approval request through RunControl.RequestApproval and the ApplyWaiver handler |
implementation | WaiveEvaluation checks scope, tenant and the state of the evaluation, raises one approval request of class publication with the subject evaluation-run and the evaluation id through RunControl.RequestApproval of RunControlService, under the service identity of QualityScoreService with the scope factory.approvals.request, idempotent on the evaluation id while it is pending, and answers 412 while it is pending; a later call that finds the bound approval Approved, or ApplyWaiver on an Approved ApprovalDecided, whichever comes first, records the verdict Waived with waiverApprovalId and the reason, extends the evidence through ArtifactService and writes EvaluationCompleted with the verdict Waived in one transaction, and the other path changes nothing; a call on an evaluation Waived by that approval answers 200 with the recorded state; 409 only for an evaluation not Completed or Passed; rejected, expired, stale and cancelled decisions and redeliveries leave the verdict, and a later call raises a new request; RunControlService is a fake in the acceptance suite. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4378 Verify the waiver through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-QualityScoreService-004"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1095: the approved waiver with its evidence and its EvaluationCompleted recorded once by the handler or by a later call, the 412 with one approval request across repeated calls, the 200 on an evaluation already Waived, the 409, 404 and 403, the decisions that leave the verdict and the new request after them, the redelivery and the waiver counter. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1096: Detect when production outcomes of a release drift below its evaluated scores and publish the drift¶
behavior · persona asset-maintainer
As an asset maintainer, I want to learn when a published release performs worse in production than its evaluation promised, so that I re-evaluate or improve it before the drop spreads, with the metric and both values in hand.
The DriftMonitor worker runs on its modelled schedule and compares, per release with an evaluation and with production outcomes in the window, the outcome statistics with the evaluated scores. The outcomes are the records the service has stored since the thin delivery of I1: attempt starts with their release, attempt outcomes recorded by Run Control (handler CollectOutcome of AttemptOutcomeRecorded) and the reliability signals of model and tool calls. When an observed value of a metric falls below the evaluated value of the last Passed or Waived evaluation of the release by more than Observability:Evaluation:DriftTolerance (default 0.05), the worker writes the DriftDetected outbox record with tenantId, targetReleaseId, metric, evaluated and observed; the event belongs to the aggregate EvaluationRun and moves no state of it. StudioBff, AlertRuleService and FeedbackService consume it in their own PBIs. The idempotency key of the worker is targetReleaseId plus the window, so a repeated trigger for the same window publishes nothing twice; its retry policy is fixed with two attempts, and when both fail the window is checked again at the next trigger and nothing is published for it in between. A release without an evaluation, or without outcomes in the window, is skipped. The work is partitioned by tenant and reads only the records of that tenant. The counter connectsoft.factory.evaluation.drift moves with each detection.
Acceptance criteria
| AC1 | Given a release with a Passed or Waived evaluation and stored outcomes in the window whose observed value of a metric falls below the evaluated value by more than Observability:Evaluation:DriftTolerance, when DriftMonitor runs, then the QualityScoreService/DriftDetected outbox record is written with tenant, release, metric, evaluated and observed values, and connectsoft.factory.evaluation.drift is incremented (SCN-QualityScoreService-012, model/services/observability/QualityScoreService.yaml). |
| AC2 | Given the same release and window, when DriftMonitor is triggered again, then nothing is published a second time; given a drop within the tolerance, a release without an evaluation or a release without outcomes in the window, then nothing is published; given two tenants, then the worker of one tenant reads only its records (SCN-QualityScoreService-012, ADR-0015). |
| AC3 | Given the record was written, when the dispatcher publishes QualityScoreService/DriftDetected on evaluation.runs.v1.drift-detected, then it is sent under the canonical envelope with tenantId, targetReleaseId, metric, evaluated and observed, and an outage is retried so it is neither lost nor duplicated; given both attempts of the worker fail, then nothing is published for that window until the next trigger (SCN-QualityScoreService-012, ADR-0012). |
| Traces: services | QualityScoreService, RunControlService |
| Traces: aggregates | QualityScoreService/EvaluationRun |
| Traces: events | QualityScoreService/DriftDetected, RunControlService/AttemptOutcomeRecorded |
| Traces: adrs | ADR-0012, ADR-0015, ADR-0021 |
| Traces: requirements | NFR-11, XL-08 |
| Traces: scenarios | SCN-QualityScoreService-012 |
| Depends on | AIFACTORY-US-1092, feature:quality-score-service-thin, feature:run-control-service |
| Sources | model/services/observability/QualityScoreService.yaml, docs/planes/trust-operations/evaluation.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-539 Drift monitor over production outcomes |
implementation | DriftMonitor compares, per tenant partition and release, the stored outcomes of the window with the evaluated values of the last Passed or Waived evaluation, writes DriftDetected to the outbox above Observability:Evaluation:DriftTolerance, is idempotent on targetReleaseId and window, skips releases without evaluation or outcomes, and increments connectsoft.factory.evaluation.drift. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4379 Outcome fixtures over a window for releases with and without drift |
fixture | The acceptance suite holds stored outcome records of two tenants for a release that drifted beyond the tolerance, one within it, one without an evaluation and one without outcomes in the window, and a clock and trigger the test controls. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4380 Verify drift detection through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Scenario=SCN-QualityScoreService-012"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1096: the detection with its payload and counter, no repeat for the same window, the skipped releases, the tenant partition, the publication through an outage and the exhausted retry. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1097: Retention rules, deletion orders and legal holds are applied to the evaluation records and accepted events of the service¶
behavior · persona workspace-admin
As a workspace admin, I want the retention, deletion and hold orders of my tenant applied to evaluation records and stored outcomes, so that evaluation data is kept as long as the rules say and no longer, except where a legal hold keeps it.
QualityScoreService owns its records in quality-score-sql: suites, evaluation runs, results and the accepted events stored since the thin delivery. Three consumed events of RetentionResidencyService govern their lifecycle. RetentionRuleChanged (handler ApplyRetentionRule) applies the effective retention period of the rule to the records of its store class; the model sets seven years for evidence references and two years for results on EvaluationRuns, and the evidence record itself stays with ArtifactService. DeletionOrderIssued (handler ExecuteDeletionOrder) deletes the ordered scope from the stores the service owns and acknowledges its task of the order through AcknowledgeOrderTask of RetentionResidencyService, under the scope factory.data.acknowledge, once per order id, with a receipt that names the order, the tenant, what was deleted and what a legal hold kept. LegalHoldChanged (handler ApplyLegalHold) suspends deletion and retention expiry for the held scope, and resumes them when the hold is lifted. Each handler is idempotent on the event id and touches only rows of the tenant of the event; an event without a tenant is refused to the dead-letter path of the inbox and changes nothing. A failed acknowledgement is retried by the inbox and the deletion is not repeated. The thin delivery of I1 applies no retention period, so this PBI is the first to apply the rules of ADR-0019 to the records of the service, those stored since I1 included.
Acceptance criteria
| AC1 | Given RetentionResidencyService/RetentionRuleChanged for the store class of the service, when ApplyRetentionRule handles it, then the records of that class older than the effective period are removed by the retention sweep and the records under it are kept; a redelivered event changes nothing (ADR-0019, model/services/observability/QualityScoreService.yaml). |
| AC2 | Given RetentionResidencyService/DeletionOrderIssued for a scope of one tenant while RetentionResidencyService/LegalHoldChanged holds a part of it, when the handlers run, then the ordered records are deleted except the held ones, no row of another tenant is touched, and the task is acknowledged once through RetentionResidencyService/AcknowledgeOrderTask with a receipt naming the order, the tenant, the deleted records and what the hold kept; a failed acknowledgement is retried without a second deletion (ADR-0015, ADR-0019). |
| AC3 | Given a hold is lifted by RetentionResidencyService/LegalHoldChanged, when the next retention sweep runs, then expiry resumes for the scope it held; given an event without a tenant, then it is dead-lettered and nothing changes (ADR-0019). |
| Traces: services | QualityScoreService, RetentionResidencyService |
| Traces: apis | RetentionResidencyService/AcknowledgeOrderTask |
| Traces: events | RetentionResidencyService/RetentionRuleChanged, RetentionResidencyService/DeletionOrderIssued, RetentionResidencyService/LegalHoldChanged |
| Traces: adrs | ADR-0015, ADR-0019 |
| Traces: requirements | NFR-03 |
| Relates: screens | retention-residency |
| Depends on | feature:retention-residency-service |
| Sources | model/services/observability/QualityScoreService.yaml, model/services/data-event/RetentionResidencyService.yaml, docs/architecture/adr/0019-data-governance.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4381 Handlers ApplyRetentionRule, ExecuteDeletionOrder and ApplyLegalHold over quality-score-sql |
implementation | The three handlers apply the effective retention to the records of the store class, delete the ordered scope except held records and acknowledge the order task once through AcknowledgeOrderTask with its receipt, suspend and resume expiry for a held scope, are idempotent on the event id, touch only rows of the tenant of the event and dead-letter an event without a tenant. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4382 Verify retention, deletion and hold through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=RetentionOrders"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1097 with a fake RetentionResidencyService/AcknowledgeOrderTask endpoint: the sweep under a rule, the deletion under a hold with its receipt, the retried acknowledgement, the lifted hold and the event without a tenant. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-US-1109: Evaluate a model deployment against its capability suite by invoking the deployment directly, and degrade while the router is unavailable¶
behavior · persona platform-admin
As a platform admin, I want a model deployment evaluated against its capability suite before I approve it, with a score per capability, so that a profile routes only to deployments whose tool calling, structured output and other capabilities were measured.
A Platform admin asks ModelRouterService to evaluate a deployment through EvaluateDeployment, and ModelRouterService starts the evaluation through the internal gRPC method Evaluation.RunSuite with its workload identity, as AIFACTORY-US-1091 describes for every internal start. The suite of a deployment is an active suite whose target is Model, so the evaluation is of kind capability-suite and keeps that kind on the run. Unlike the tasks of the other kinds, which run as sandbox-class Factory runs under the evaluation project, the tasks of a capability suite invoke the deployment under evaluation directly through ModelRouterService, the outbound dependency the model lists as required with the failure mode degrade: SuiteRunner calls ModelGateway.Invoke of ModelRouterService once per task of the dataset snapshot, keyed by evaluationId plus taskId as for every task, and scores each answer against the capability metrics of the suite, with the seeds of Observability:Evaluation:DefaultThresholds where the suite sets no threshold (D32). A capability call belongs to no run, so it presents no attempt token: the attempt token of a lease is the proof only for a call that belongs to a run (ADR-0015 amendment D125). Its receipt names no run and no attempt and names the evaluation project Observability:Evaluation:EvaluationProjectId as its project. Which token a direct suite call presents to the model gateway is an open point reported with this feature; the call is made behind a model port whose credential the owner chooses, and the port is a fake in the acceptance suite. When every task has a result, the verdict, the evidence record and the EvaluationCompleted outbox record with the kind capability-suite follow AIFACTORY-US-1092, and ModelRouterService imports the evidence through its handler ImportCapabilityEvidence in its own PBIs. A call that the deployment answers with a failed or denied outcome is a scored result of the task, not an infrastructure failure. Degrade: while ModelRouterService is unavailable, a Queued capability-suite evaluation is not started and stays Queued, SuiteRunner tries it again at its next trigger, and evaluations of the other kinds, the reads and the release gate keep serving; a Running capability-suite evaluation whose calls still fail after the third attempt of the retry policy ends Failed with the reason of the last failure, as AIFACTORY-US-1093 states. No score is ever computed for a call that did not return.
Acceptance criteria
| AC1 | Given an evaluation started through QualityScoreService/Evaluation.RunSuite for an active suite whose target is Model, when SuiteRunner runs it, then the run keeps the kind capability-suite and each task invokes the deployment once through ModelRouterService/ModelGateway.Invoke, outside a run and without an attempt token, keyed by evaluation and task, and a duplicate trigger invokes nothing twice (model/services/observability/QualityScoreService.yaml, ADR-0015). |
| AC2 | Given every task of a capability suite scored, including tasks the deployment answered with a failed or denied outcome, when the verdict is computed, then the evidence record is stored once and the QualityScoreService/EvaluationCompleted outbox record carries the verdict, the evidence reference and the kind capability-suite in the transaction that completes the run (ADR-0021, model/services/observability/QualityScoreService.yaml). |
| AC3 | Given ModelRouterService unavailable, when SuiteRunner picks up a Queued capability-suite evaluation, then it stays Queued with no task invoked and no event, and an evaluation of another kind and QualityScoreService/GetEvaluation keep serving; given a Running capability-suite evaluation whose calls fail after the third attempt, then it is Failed with the reason and the QualityScoreService/EvaluationFailed record, and no score exists for a call that did not return (ADR-0012, model/services/observability/QualityScoreService.yaml). |
| Traces: services | QualityScoreService, ModelRouterService |
| Traces: aggregates | QualityScoreService/EvaluationRun |
| Traces: apis | QualityScoreService/Evaluation.RunSuite, QualityScoreService/GetEvaluation, ModelRouterService/EvaluateDeployment, ModelRouterService/ModelGateway.Invoke |
| Traces: events | QualityScoreService/EvaluationCompleted, QualityScoreService/EvaluationFailed |
| Traces: adrs | ADR-0012, ADR-0015, ADR-0021 |
| Traces: requirements | FR-04, NFR-11 |
| Depends on | AIFACTORY-US-1091, AIFACTORY-US-1092, AIFACTORY-US-1093, feature:model-router-service |
| Sources | model/services/observability/QualityScoreService.yaml, model/services/model-compute/ModelRouterService.yaml, docs/architecture/adr/0021-quality-gates.md |
Tasks
| Task | Kind | Expected output | Repository |
|---|---|---|---|
AIFACTORY-TASK-4407 Capability-suite tasks of SuiteRunner through the model port with the degrade branch |
implementation | For an evaluation of kind capability-suite, SuiteRunner invokes the deployment once per task through a model port bound to ModelGateway.Invoke of ModelRouterService, outside a run and without an attempt token, keyed by evaluationId and taskId, scores each answer, failed and denied outcomes included, against the capability metrics of the suite, leaves a Queued evaluation Queued while ModelRouterService is unavailable and tries it at the next trigger, and fails a Running evaluation after the third failed attempt; the credential of the port is configuration until the owner decides it. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4408 Capability suite fixtures and a fake model gateway |
fixture | The acceptance suite holds an active suite with the target Model and its dataset snapshot, a fake of the model port that answers completed, failed and denied outcomes, can be made unavailable and records whether an attempt token was presented, and a trigger the test controls. | ConnectSoft.AI.SoftwareFactory.QualityScoreService |
AIFACTORY-TASK-4409 Verify the capability-suite evaluation through the acceptance suite |
verification | Run dotnet test -c Release --filter "Category=Acceptance&Feature=CapabilitySuite"; a zero-match filter counts as a failure; the tests prove AC1 to AC3 of AIFACTORY-US-1109 for QualityScoreService/Evaluation.RunSuite and ModelRouterService/ModelGateway.Invoke: one call per task without an attempt token, the kind capability-suite on QualityScoreService/EvaluationCompleted, the Queued evaluation kept while the router is unavailable with other evaluations serving, and the Failed run after the third attempt. |
ConnectSoft.AI.SoftwareFactory.QualityScoreService |