Skip to content

Testing & Quality Strategy

This page defines how quality is guaranteed across everything the factory produces and everything the factory itself runs. Testing is not a phase — it is a continuous, agent-driven, gated activity. The Testing and Quality Assurance Workflows page describes the orchestration; this page defines the strategy and targets.

Target Architecture — Final-State Design

Every generated module ships with its tests. The factory's QA agent cluster generates, runs, and validates tests as part of generation, and quality gates block promotion when targets are not met. Generated services inherit a ready-made test harness from the templates' testing.md (for example the microservice template testing).

Test pyramid

flowchart TB
    e2e["E2E Tests<br/>(few, high value)"]
    contract["Contract Tests<br/>(API + event boundaries)"]
    integration["Integration Tests<br/>(service + store + bus)"]
    unit["Unit Tests<br/>(many, fast)"]
    subgraph crosscut [Cross-cutting]
        load["Load & Performance"]
        chaos["Chaos / Resilience"]
        security["Security / SAST / DAST"]
    end

    unit --> integration
    integration --> contract
    contract --> e2e
    crosscut -.-> e2e
Hold "Alt" / "Option" to enable pan & zoom
Level Scope Owned / generated by Gate
Unit Pure domain logic, handlers, value objects Test Generator Agent Coverage + must pass
Integration Service + NHibernate store + Service Bus (Testcontainers) Test Automation Engineer Agent Must pass
Contract API and event-envelope compatibility across bounded contexts Contract Testing Agent No breaking change without version bump
E2E Vision-to-running-SaaS critical paths E2E Testing Agent Must pass on critical journeys
Load / performance NFR validation against quality attribute targets Load & Performance Testing Agent Meet p95/p99 + throughput SLOs
Chaos / resilience Fault injection, failover, recovery Resiliency & Chaos Engineer Agent Recover within RTO/RPO
Security SAST/DAST, secret scanning, dependency CVEs Security Penetration Testing Agent, Vulnerability Management Agent Zero high/critical shipped

Contract testing

Because every message uses the canonical Event Envelope and every integration event is registered in the Event Catalog, contracts are explicit and machine-checkable:

  • API contracts — provider and consumer tests verify request/response schemas; OpenAPI/gRPC proto definitions are the source of truth (see API Governance).
  • Event contracts — consumers verify they tolerate the published payload schema and that producers do not break eventType semantics. Breaking changes require a new versioned type, never an in-place change.
  • Contract drift fails the build before it can reach a shared environment.

Quality gates in CI/CD

flowchart LR
    gen["Generate / Change"] --> build["Build"]
    build --> unitg["Unit + Coverage Gate"]
    unitg --> intg["Integration Gate"]
    intg --> contractg["Contract Gate"]
    contractg --> qualg["Static Analysis + Lint Gate"]
    qualg --> secg["Security Scan Gate"]
    secg --> e2eg["E2E Gate"]
    e2eg --> approve["Governance Approval"]
    approve --> promote["Promote Environment"]
    unitg -.->|fail| block["Block + Feedback"]
    contractg -.->|fail| block
    secg -.->|fail| block
Hold "Alt" / "Option" to enable pan & zoom

Gates are enforced by the DevOps / GitOps Platform pipelines and Governance / Security / Compliance Platform approval policies. A failed gate stops promotion and emits a feedback signal.

Coverage & quality targets

Metric Target
Unit test coverage (domain/application) >= 80% line, >= 70% branch
Critical-path E2E 100% of defined critical journeys pass
Mutation score (key domains) Tracked and improving over time
Cyclomatic complexity Within template thresholds; violations flagged
Static analysis / lint Zero errors; warnings tracked
Security findings Zero high/critical at promotion
Flaky-test rate Quarantined and tracked to zero

Quality of generated artifacts

The Code Quality Agent, Code Reviewer Agent, and Test Coverage Validator Agent review generated code and tests against the same standards a human reviewer would, before a pull request is opened by the Pull Request Creator Agent. This closes the loop between generation and quality.

Pillar alignment

  • Traceability — every test run, result, and gate decision is correlated by traceId and linked to the moduleId/artifactId under test.
  • Reusability — test harnesses, fixtures, and Testcontainers setups come from templates, reused across every generated module.
  • Autonomy — tests are generated, executed, and validated by the QA agent cluster without manual authoring.
  • Governance — gates are policy-enforced; exceptions require recorded approval and, when they change a target, an ADR.
  • Observability — test and gate outcomes feed the Observability & Feedback Platform.
  • Multi-tenant scale — isolation and per-tenant behavior are explicitly tested.