Skip to content

Quality gates and release authority

Decision

This page records a decision and its status. A Proposed decision is not binding; an Accepted decision is binding on the Factory until amended or superseded.

Decided on 2026-10-09 (ADR-0021; D32). Three rules:

  1. Model profiles, providers, SDK adapters and fallback chains are published assets. They move Draft → InReview → Published like agents and skills, are approved by Platform admin with capability-registry evidence attached, and a Workspace admin may only restrict which profiles a project may use, never add one.
  2. Every release of an agent, skill or workflow passes an evaluation gate. The maintainer proposes; a Reviewer approves to Published; thresholds are set per model profile by Platform admin; a candidate may not lower task success or raise the tool-error rate compared with the current Published release.
  3. Learning is tenant-scoped. Episodic memory and evaluation datasets stay in the tenant; promotion to shared memory needs a reviewed proposal and tenant consent.

Release pipeline for an asset

stateDiagram-v2
    [*] --> Draft : authored in Git
    Draft --> InReview : release candidate with closure resolved
    InReview --> InReview : evaluation run, findings
    InReview --> Published : gates passed and Reviewer approval (exact manifest hash)
    InReview --> Draft : rejected or changed
    Published --> Deprecated : revoked or superseded
    Deprecated --> [*]
Hold "Alt" / "Option" to enable pan & zoom

Gate checks at InReview → Published: dependency closure complete (every skill, tool schema, policy revision and profile referenced is Published); evaluation suite executed against the candidate on the declared profile; thresholds met; no regression against the current release on the same suite; approval recorded by an eligible Reviewer (ADR-0015 separation of duties applies: not the author).

Threshold seeds per profile

Seeds from the original target architecture, marked empirical there; Platform admin fixes the actual values per profile when the capability registry has evidence.

Metric Seed threshold Measured on
Tool selection accuracy ≥ 97 % Tool-calling evaluation cases
Valid tool arguments ≥ 99 % Schema validation of emitted calls
Structured-output validity ≥ 99.5 % JSON-schema compliance of outputs
Multi-turn consistency ≥ 95 % Conversation state cases
Instruction compliance ≥ 95 % Instruction-following cases
Patch compile rate (engineering agents) ≥ 90 % Generated change sets built in a sandbox
Task success (per suite) no lower than the current release Suite-level score
Tool-error rate no higher than the current release Executor receipts during evaluation

Evaluation records

An evaluation run produces an immutable record: suite id and version, candidate manifest hash, profile and resolved provider/model, per-case results with inputs by reference, scores, cost and latency, the comparison against the current release, and the gate verdict. Records are artifacts (BC-12) referenced from the release manifest (BC-04) and visible in the Studio quality views (FUX-10).

Capability registry evidence for profiles

A profile reaches Published only with a capability scorecard: tool calling, parallel tool calls, strict JSON and JSON-schema compliance, instruction following, agent task success, patch success, declared versus reliable context, file editing, shell tool use, multi-turn consistency, planning, error recovery, latency class, cost basis, privacy class, provider type and license metadata (commercial use, redistribution, fine-tuning, customer deployment). Fallback chains are part of the profile release and must satisfy the same privacy class.

Learning governance

Source Allowed use Gate
Run outcomes and feedback inside a tenant Episodic memory, tenant evaluation datasets, skill-improvement candidates for that tenant Tenant policy
Promotion to the tenant's shared memory Reusable facts and procedures Reviewed proposal (Asset maintainer proposes, Reviewer approves)
Promotion to ConnectSoft shared memory or shared suites Cross-tenant learning Reviewed proposal and explicit tenant consent; anonymization recorded

Current implementation

Microsoft.Extensions.AI.Evaluation (with Quality, NLP and Safety previews) is referenced by ConnectSoft.Extensions.AI; no suite, record, gate or capability scorecard exists. AgentKit's reviewPending flags belong to the earlier, non-Factory experiments (D19).