Skip to content

AIFACTORY-FEAT-177: Compliance library: the Factory data classes and redaction profiles

Generated from the architecture model

This page is rendered from model/**/*.yaml by tools/docs_model during mkdocs build. Edit the YAML, not this page; hand-written design lives on the linked service, platform and plane pages.

A design artefact, not a tracker

This backlog is generated from model/backlog/ and validated against the architecture model. No work item exists in any tracker, and no item carries a date, an estimate or an assignee (ADR-0029).

Epic Ecosystem component extensions · Increment I1 Generate a microservice · State elaborated · Route backend-developer · Phase before development · Build order 23 of 185 (Templates and libraries)

Before development

Template or library work: every task of this feature changes a template or a shared library the Factory builds on, and it lands before the Factory's own repositories of I1 are developed. The mark travels with every task (tag phase:before-development).

ADR-0019 fixes four classification classes, pii and classification markers on model fields, redaction in every projection and notification, and no raw prompt, response or tool-output logging. The Compliance library (classification, redaction; D31) carries the mechanism but not the Factory's data classes: prompts, completions, tool outputs, secret references, customer source code and tenant personal data. The library gains those classes and the redaction profiles the telemetry, audit and artifact projections apply, so that redaction is one implementation rather than one per service. Component: Compliance (extend).

Delivers —
Stubs —
Realizes components: cmp-compliance
adrs: ADR-0019
Requirements —
Depends on —
Sources model/components.yaml, docs/architecture/adr/0019-data-governance.md

AIFACTORY-US-264: Compliance library: six Factory data classes join the taxonomy and map to the four classification classes and to the pii and classification field markers of the model

enabling · persona platform-admin As a platform admin, I want the data kinds of the Factory to be classes of the one taxonomy every service already uses, so that a field is classified once by a marker and no service keeps its own list of what is sensitive.

The README of the Compliance library describes a shared taxonomy with marker attributes such as [EmailData], [PhoneData] and [TenantIdData], custom redactors, the registration AddConnectSoftCompliance and the profiles default, audit and strict that configuration selects. The Overview and Features pages of its documentation site still carry the text of the library template the repository was scaffolded from and do not describe the library. ADR-0019 decision 2 fixes four classification classes, public, internal, confidential and restricted, and has model fields carry pii and classification; docs/architecture/data-governance.md places source code, prompts and outputs in confidential and credentials, secret references and personal fields in restricted. The README and the source of the library already hold the class of a secret value, [SecretData] with the generic secret redactor, and personal data attributes such as [EmailData], [PhoneData], [PersonNameData], [PostalAddressData] and [GovernmentIdData]. The taxonomy has no class for a prompt, a completion, a tool output, a secret reference or source code, and none that gathers the personal attributes under the name the Factory uses. This PBI adds six data classes to ConnectSoftTaxonomy, each with its marker attribute: prompt, completion, tool output, secret reference, source code and tenant personal data. Each class declares the classification class it belongs to. One table in the library maps the taxonomy and the two field markers of the model, and it states the relation of the new classes to what exists. The existing personal data attributes are members of tenant personal data and keep their own redactors. [SecretData] stays the class of a secret value and takes the fixed mask, while secret reference is the new class of a value that only points at a secret and passes as the reference. A member may carry one existing attribute together with the Factory class that attribute is a member of, and no other pair. A field marked pii is tenant personal data, and a field with a classification takes the default treatment of that class unless a marker attribute names a narrower data class. The registration AddConnectSoftCompliance(configuration, environment) takes no list of types today, so it gains an option that names the assemblies to scan. A validation at registration runs over the types of those assemblies and refuses a type that marks one member with two data classes of the Factory, with an existing attribute and a Factory class it is no member of, or with a data class weaker than the classification of the member, and names the member; with no assembly named nothing is scanned and the registration behaves as it does today. Nothing is redacted by this PBI. It gives the redaction profiles of the next PBI the classes they act on, and it replaces the template text of the two documentation pages with the description of the library, so that a consumer can read which classes exist.

Acceptance criteria

AC1 Given the library with the work of this PBI, when a type marks members with the six marker attributes, then the taxonomy resolves each member to its data class, and each data class reports its classification class as docs/architecture/data-governance.md assigns it: prompt, completion, tool output and source code are confidential, and secret reference and tenant personal data are restricted (ADR-0019 decision 2).
AC2 Given a model field marked pii, a model field with a classification and no marker attribute, and members marked [EmailData] and [SecretData], when the mapping table resolves them, then the first is tenant personal data, the second takes the default treatment of its classification class, the [EmailData] member is a member of tenant personal data that keeps its own redactor, and the [SecretData] member stays a secret value and is not a secret reference; a test covers each of the four classification classes and each existing attribute the table lists (ADR-0019 decision 2).
AC3 Given the option of the registration names an assembly with a type that marks one member with two data classes of the Factory, with an existing attribute and a Factory class it is no member of, or with a data class weaker than the classification of the member, when AddConnectSoftCompliance validates the types of the named assemblies, then registration is refused and the member is named; a member that carries [EmailData] together with the tenant personal data marker registers, and with no assembly named nothing is scanned (ADR-0019 decision 2, model/components.yaml).
Traces: adrs ADR-0019
Relates: nfrs NFR-03
Sources model/components.yaml, docs/architecture/adr/0019-data-governance.md, docs/architecture/data-governance.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1203 Six Factory data classes, their marker attributes and the mapping table with the existing attributes contract The taxonomy declares prompt, completion, tool output, secret reference, source code and tenant personal data as data classes with one marker attribute each and the classification class of each; one table maps the pii and classification field markers of the model to the taxonomy, lists the existing personal data attributes as members of tenant personal data that keep their own redactors, keeps [SecretData] as the class of a secret value with the fixed mask beside the new secret reference, and names the one pair of markers a member may carry; unit tests cover every class and every row of the table. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-637 Factory data classes resolved per member and validated at registration, before the redaction profiles implementation A resolver returns the data class and the classification class of a member from its marker attributes and model markers by the table of AIFACTORY-TASK-1203; the registration gains an option that names the assemblies to scan, and its validation refuses a member with two Factory data classes, with an existing attribute and a Factory class it is no member of, or with a class weaker than its classification, and names the member; the redaction profiles the migrated title named are AIFACTORY-TASK-1206. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-1204 Overview and Features pages describe the library and its data classes documentation The Overview and Features pages of the documentation of the library no longer carry the text of the library template; they describe the taxonomy, the marker attributes with the six data classes of the Factory and their classification classes, the redactors and the registration, in agreement with the README of the repository. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-1205 Verify the data classes, the mapping table and the registration refusal verification Run dotnet test -c Release --no-build on the library solution; the tests of the six classes, of the mapping table over the four classification classes and the existing attributes, of the three refused conflicts, of the admitted pair and of a registration that names no assembly pass and prove AC1 to AC3 of AIFACTORY-US-264 (ADR-0019, docs/architecture/data-governance.md). ConnectSoft.Extensions.Compliance

AIFACTORY-US-265: Compliance library: one redaction profile per projection replaces Factory content by its reference, and a secret never leaves a service as a value

enabling · persona platform-admin As a platform admin, I want one redaction implementation with a profile for every place data leaves a service, so that telemetry, audit evidence, notifications and events are redacted the same way by every service.

ADR-0019 decision 2 requires redaction in every projection and notification, and decision 3 forbids raw prompt, response and tool-output logging. The Compliance library selects one of three profiles, default, audit and strict, by configuration, and its README names idempotent redactors as a design goal. A Factory service projects the same record to several places with different rules, so one profile per process is not enough. This PBI adds a redaction profile per projection on top of the data classes of the earlier PBI: log, audit, notification, envelope and artifact projection. In the log, audit, notification and envelope profiles a prompt, a completion, a tool output and source code are replaced by their reference, the hash and the size of the value and the artifact identifier when there is one. In the artifact projection profile they pass only when the caller states the explicit scope docs/architecture/data-governance.md requires for showing confidential content, and are a reference otherwise. A secret is never a value in any profile: a member marked as a secret reference, such as a secret broker reference, stays the reference, and a member marked [SecretData], such as a model provider key, is replaced by a fixed mask. Tenant personal data is redacted in the log, notification and envelope profiles, each existing personal data attribute by its own redactor; the audit profile keeps the actor identifier the evidence schema requires and redacts every other personal field. The notification profile removes every restricted field and lists it, so that a restricted field never reaches a notification. A field with a marker and no rule in the selected profile is redacted in full, so that a new class fails closed. Redacting a redacted value changes nothing. The log profile is also the implementation of the allow decision of the content guard that the observability feature ships without one: registering the Compliance library registers it, and it allows a field only when its class is public or internal.

Acceptance criteria

AC1 Given a record with a prompt, a completion, a tool output and source code, when it is projected with the log, audit, notification and envelope profiles, then each of the four fields is replaced by its hash and size, with the artifact identifier when there is one, and the artifact projection profile passes them only with the explicit scope stated and returns the reference otherwise (ADR-0019 decisions 2 and 3, docs/architecture/data-governance.md).
AC2 Given a record with a secret broker reference marked as a secret reference and a model provider key marked [SecretData], when it is projected with the log, audit and envelope profiles, then the reference is kept as the reference, the key is replaced by the fixed mask and the value of the key appears in no projection (ADR-0019 decision 2).
AC3 Given a record with a restricted field and tenant personal data, when it is projected with the notification profile, then every restricted field is removed and listed and no personal value remains; when it is projected with the audit profile, then the actor identifier is kept and every other personal field is redacted (ADR-0019 decision 2, docs/reference/schemas/audit-evidence.schema.json).
AC4 Given a host that registers the Compliance library beside the single registration of the observability library, when a prompt and a field classified internal are written to a log record, then the content guard replaces the prompt by its reference and lets the internal field pass, because the log profile answers the allow decision from the class of the field (ADR-0019 decision 3).
AC5 Given a field with a marker attribute and no rule in the selected profile, when it is projected, then it is redacted in full; given a value that was already redacted, when it is redacted again, then it is unchanged (ADR-0019 decision 2, model/components.yaml).
Traces: descriptors run-event-envelope-v1
Traces: adrs ADR-0019
Relates: nfrs NFR-03, NFR-10
Depends on AIFACTORY-US-264, feature:component-observability-factory-conventions
Sources model/components.yaml, docs/architecture/adr/0019-data-governance.md, docs/architecture/data-governance.md, docs/reference/schemas/audit-evidence.schema.json

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1206 Redaction profiles for the log, audit, notification, envelope and artifact projections implementation Five named profiles apply the rules of the data classes per projection: content classes become a reference, a secret value becomes the fixed mask, personal data is redacted except the actor identifier in the audit profile, the notification profile removes and lists restricted fields, the artifact projection profile passes content only with the explicit scope, and a marked field without a rule is redacted in full; the profile is chosen per call and no longer only per process. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-1207 Log profile as the allow decision of the telemetry content guard implementation The registration of the library registers an implementation of the allow decision interface of the pinned ConnectSoft.Extensions.Telemetry package that answers from the classification class of the field and allows only public and internal fields; the library defines no content guard of its own. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-638 A secret reference never leaves as a value fixture Redaction tests across the log, audit and envelope projections prove that a secret broker reference and a model provider key are replaced by their reference or mask in every projection, that a restricted field never reaches a notification, that a marked field without a rule is redacted in full and that redaction is idempotent. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-1208 Verify the five profiles and the content guard decision verification Run dotnet test -c Release --no-build on the library solution; the profile tests over the five projections, the secret tests, the notification test, the fail-closed and idempotency tests and the content guard test pass and prove AC1 to AC5 of AIFACTORY-US-265 (ADR-0019). ConnectSoft.Extensions.Compliance

AIFACTORY-US-266: Compliance library: the package with the Factory data classes and the redaction profiles is built, tested and published to the ConnectSoft feed with a version its consumers pin

evidence · persona platform-admin As a platform admin, I want the Compliance package with the Factory classes on the feed under a version a service can pin, so that the audit sink, the telemetry guard and every service redact through one published implementation.

This PBI is the publication proof of the feature. The README of the library names the package in the ConnectSoft feed and the Azure DevOps pipeline of the repository, and the epic is done only when each library publishes before the Factory code that consumes it is built (model/backlog/epics.yaml). The pipeline builds the library, runs its tests, with the table-driven and snapshot tests the README describes extended to the six data classes and the five profiles, packs and pushes the package to the ConnectSoft feed, the Azure Artifacts feed the ConnectSoft libraries already publish to. The Telemetry package with the allow decision interface is restored from that feed at its published version. Publication is reached only when the tests pass and refuses a version that is already on the feed. A probe then proves the package from the side of a consumer: a clean host that takes every ConnectSoft package from the feed registers AddConnectSoftCompliance beside the single registration of the observability library, projects one record with a planted prompt, a planted provider key and a planted personal field through the five profiles and writes it to a log record, and no planted value is found in any output. The published version is recorded on the Runbook page of the library, where the audit library and the services read their pin. This is an evidence PBI (D113): a library feature has no scenario, so the probe names what it proves through its ADRs, the requirement on release gates and its sources.

Acceptance criteria

AC1 Given the pipeline of the library repository after the work of the feature, when it runs, then the package is built, tested with the snapshot tests of the six data classes and the five profiles, packed and pushed to the ConnectSoft feed, and the published version is recorded on the Runbook page (FR-11, ADR-0028, model/backlog/epics.yaml).
AC2 Given a clean probe host that restores every ConnectSoft package from the feed and holds no project reference, when it projects a record with a planted prompt, a planted provider key and a planted personal field through the five profiles and writes it to a log record, then a search of every output finds no planted value (ADR-0019 decisions 2 and 3).
AC3 Given a failing test, a changed snapshot or a version that is already on the feed, when the pipeline reaches the publication stage, then the stage is refused with the reason and nothing is pushed; a run with a planted changed snapshot proves the refusal (FR-11, docs/architecture/ecosystem-composition.md).
Traces: adrs ADR-0019, ADR-0028
Traces: requirements FR-11
Depends on AIFACTORY-US-264, AIFACTORY-US-265
Sources model/components.yaml, model/backlog/epics.yaml, docs/architecture/ecosystem-composition.md, docs/architecture/catalogs/libraries.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1209 Version guard and snapshot gate in the compliance library pipeline operations The pipeline of the repository restores the Telemetry package from the ConnectSoft feed, publishes the package only after the tests and the snapshot comparison of the solution pass and refuses a version that is already on the feed with the reason. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-1210 Feed probe host for the published compliance package fixture A probe project that restores every ConnectSoft package from the feed and holds no project reference registers the library beside the single registration of the observability library, projects one record with three planted values through the five profiles and a log record, and searches every output for the planted values; the probe runs as the last stage of the pipeline. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-1211 Published version and the profiles recorded on the library pages documentation The Runbook page of the library names the published version and the version of the Telemetry package it pins, and the Use Cases page describes the five profiles with the projection each belongs to and the fail-closed rule. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-1212 Verify the published compliance package by the probe and by a planted changed snapshot verification One run of the pipeline publishes the package and the probe finds no planted value; one run with a planted changed snapshot stops before publication; both runs prove AC1 to AC3 of AIFACTORY-US-266 (FR-11, ADR-0019). ConnectSoft.Extensions.Compliance

AIFACTORY-US-117: Compliance library: the Factory data classes and redaction profiles is delivered and proven by its consumers

enabling · persona platform-admin As a platform admin, I want Compliance to carry the Factory data classes and redaction profiles, so that it serves data classification and redaction (D31) without later rework.

This is the integration PBI of the feature. With the six data classes, the five redaction profiles and the publication proof of the earlier PBIs, redaction in the Factory is one implementation and not one per service. A service marks a field once, with a marker attribute or with the pii and classification markers of the model, pins the published ConnectSoft.Extensions.Compliance package and registers AddConnectSoftCompliance. Every projection of the field then follows the profile of its destination: a prompt, a completion, a tool output and source code leave the service as a reference, a secret never leaves as a value, a restricted field never reaches a notification, and a field the selected profile has no rule for is redacted in full. The telemetry content guard of the observability libraries and the sink of the audit library take their decisions from this package and hold no rules of their own. The implementation lives in the tasks of the earlier PBIs; this PBI keeps the migrated verification of the feature. The feature is done when the package is on the ConnectSoft feed with the version the Runbook page names, the probe host finds no planted value in any projection, and the audit library of the same increment restores the package from the feed.

Acceptance criteria

AC1 Given the library repository ConnectSoft.Extensions.Compliance with the work of this feature, when its pipeline runs, then the package builds, its tests pass and it publishes to the ConnectSoft feed with a verified version that the consuming repositories pin (ADR-0019, model/components.yaml).
AC2 Given the work of AIFACTORY-TASK-637 (factory data classes resolved and validated) and of AIFACTORY-TASK-1206 (redaction profiles) is done, then prompt, completion, tool output, secret reference, source code and tenant personal data are data classes of the taxonomy with redaction profiles for logs, audit, notifications, envelopes and artifact projections, and the pii and classification field markers and the existing attributes map to them (ADR-0019 decision 2, docs/architecture/data-governance.md).
AC3 Given the work of AIFACTORY-TASK-638 (a secret reference never leaves as a value) is done, then redaction tests across the log, audit and envelope projections prove that a secret broker reference and a model provider key are replaced by their reference or mask in every projection, and that a restricted field never reaches a notification (ADR-0019 decisions 2 and 3).
Traces: services ArtifactService, AuditService, ContextBuilderService
Traces: adrs ADR-0019
Depends on AIFACTORY-US-264, AIFACTORY-US-265, AIFACTORY-US-266
Sources model/components.yaml, docs/architecture/adr/0019-data-governance.md

Tasks

Task Kind Expected output Repository
AIFACTORY-TASK-1213 Getting Started page for classifying and projecting a Factory record documentation The Getting Started page of the library holds one worked case of a Factory service: a record marked with the marker attributes and the model markers, the registration, and the same record projected to a log, an audit record, a notification and an event, with what each profile leaves. ConnectSoft.Extensions.Compliance
AIFACTORY-TASK-678 Verify compliance-factory-classifications-delivered by running the declared command verification Run dotnet build -c Release and dotnet test -c Release --no-build on the library solution and the probe host of AIFACTORY-TASK-1210 against the published version; they pass and prove AC1 to AC3 of AIFACTORY-US-117. ConnectSoft.Extensions.Compliance