ADR-0021: Release authority, evaluation gates and learning governance¶
Decision
This page records a decision and its status. A Proposed decision is not binding; an Accepted decision is binding on the Factory until amended or superseded.
- Status: Accepted 2026-10-09 by the product owner (decision D32; closes Q-11, Q-12, Q-13)
- Date: 2026-10-09
- Deciders: Dmitry Khaymov (product owner)
- Tags: evaluation, release, model-profiles, memory
Context¶
Three questions were open: who approves models and adapters (Q-11), what quality a release must prove and who may publish (Q-12), and whether tenant feedback may feed learning (Q-13). D21 already requires approval for publication.
Decision¶
- Profiles are assets. Model profiles, provider connections, SDK adapters and fallback chains follow Draft → InReview → Published, approved by Platform admin with a capability scorecard attached; a Workspace admin may only restrict profiles per project.
- Evaluation gate per release. An agent, skill or workflow release candidate is evaluated on its declared profile; thresholds are set per profile by Platform admin from the seeds (tool selection ≥ 97 %, valid arguments ≥ 99 %, structured output ≥ 99.5 %, multi-turn ≥ 95 %, instruction compliance ≥ 95 %, patch compile ≥ 90 %); a release may not lower task success or raise the tool-error rate against the current Published release; the maintainer proposes and a Reviewer (not the author) approves.
- Evaluation records are artifacts referenced from the release manifest and shown in the Studio.
- Learning is tenant-scoped. Promotion to shared memory needs a reviewed proposal; promotion across tenants needs explicit tenant consent and recorded anonymization.
Consequences¶
- Positive: "published" means evidenced; profile changes cannot silently degrade quality or privacy; the Studio can show why a release is allowed.
- Negative: evaluation suites and the capability registry must exist before anything is Published; the first workflow runs on InReview assets under a documented exception.
Alternatives considered¶
| Alternative | Why not chosen |
|---|---|
| Platform admin judgement without numeric gates | No regression protection; quality claims unverifiable |
| Publish freely, evaluate later | Contradicts D21 and makes releases non-reproducible |