Skip to content

Evaluation, regression and replay

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Nothing is promoted on the strength of a fluent output or a validator pass. Evaluation attaches evidence to an exact candidate (a release, a model profile, a skill version, a context strategy) and publication is blocked while the evidence fails.

Targets and metrics

Target Example metrics
Models (before a profile assignment) Tool selection accuracy, valid tool arguments, structured-output and JSON-schema success, instruction compliance, multi-turn consistency, patch compilation, effective context, latency, cost
Agents and agent versions Task success rate, human intervention rate, rework, cost per accepted outcome, regression rate
Skills Task success on the skill's historical tasks, tool error rate, cost
Tools and MCP servers Argument validity, failure rate, side-effect correctness
Workflows Completion, gate outcomes, duration
Context and memory strategies Retrieval quality, citation correctness, unsupported-claim rate
Policies Decision correctness on recorded cases, bypass attempts

Thresholds are chosen empirically from a baseline, recorded with sample size and dataset version, and owned by someone; copied numbers are not targets (quality attributes guidance, packet NFR-11).

Gates

  1. Model gate — a model enters a profile only after capability evaluation; profile constraints are declared, not assumed.
  2. Regression gate — a change to model, prompt, skill, context strategy, routing or agent implementation is run against historical tasks; promotion requires no regression within the accepted tolerance and records the comparison (success, cost, tool errors).
  3. Release gate — a release manifest references its evaluation suite and the evidence of the last run; missing or failed evidence blocks publication (registries).

Replay and simulation

Replaying incidents or historical task sets against old and new agents, or against two models, requires run traceability: pinned snapshots, persisted inputs, tool results and decisions. ADR-0014 does not promise replay yet; this page records it as a capability that depends on the run journal (capability dependencies).

Datasets and learning loop

Evaluation datasets are versioned, scoped (tenant data never silently becomes a shared dataset, Q-13) and grow from completed runs through review. Learning proposals (skill improvements, candidate facts, memory promotion) enter the same review before any asset changes (Knowledge & Memory).

Current implementation

  • Microsoft.Extensions.AI.Evaluation evaluators are registered through ConnectSoft.Extensions.AI; unit tests exist for the lifecycle library (7) and the agents framework adapter (29) as last reported.
  • No datasets, suites, gates, comparison harness or replay exist. The Workflow Studio prototype shows an evaluation-lab pattern with fixture data.
  • Canonical guidance: ConnectSoft.Documentation cross-cutting-concerns/ai-machine-learning/ai-evaluation.md (linked, not copied).