Skip to content

Workers

Target Architecture — Final-State Design

This page describes the final-state background workers of the Governance, Security & Compliance Platform. Workers are MassTransit consumers / scheduled jobs built on ConnectSoft.WorkerTemplate, run on .NET 10, and consume the canonical Event Envelope. Every worker is idempotent on eventId per the Metadata Schema.

The platform runs eight workers. They keep the policy and trust layer reactive (evaluating, scanning, scoring, classifying) and durable (timeouts, exports, report generation) without blocking the inline request path.

Worker Catalogue

Worker Trigger Purpose Input Output Retry Idempotency
PolicyEvaluationWorker Event: sensitive-action signals (e.g. DeploymentRequested, AgentTaskAssigned) Asynchronous (re)evaluation of policy where inline gating is not required; bulk re-evaluation on policy version change. Action context envelope PolicyDecisionRecorded Exponential backoff, 5 attempts, then dead-letter sha256(eventId + ":PolicyEvaluationWorker"); decision keyed by (action, policyVersion)
ApprovalTimeoutWorker Schedule (1 min tick) + per-request timer Detect expired ApprovalRequests; escalate or auto-reject per policy. Open approvals past expiresAt ApprovalRejected (auto) or escalation ApprovalRequested At-least-once; safe re-run Guarded by ApprovalRequest.status transition; no-op if already decided
AuditExportWorker Schedule (hourly) + AuditEntryRecorded batches Export immutable audit entries to Blob for long-term, queryable evidence; verify hash chain. New AuditEntry rows since last watermark Blob NDJSON export; AuditEntryRecorded (export marker) 3 attempts then alert Watermark on auditEntryId; export is append-only and re-runnable
ComplianceReportWorker Event: compliance.report.requested (from POST /compliance/reports) Assemble report by querying decisions, findings, and audit; render evidence bundle to Blob. ComplianceReport (Generating) ComplianceReportGenerated; Blob evidence bundle 3 attempts then mark Failed Keyed on complianceReportId; regenerates deterministically
SecurityScanIngestionWorker Event: external scan results / POST /security-findings overflow Normalise, deduplicate, and persist findings from scanners; enrich with classification/risk. Raw scan result envelope SecurityFindingRaised Exponential backoff, 5 attempts Dedup on dedupKey (rule + resource + location)
RiskScoreWorker Event: SecurityFindingRaised, DataClassified, behavioural signals Recompute composite RiskScores for affected actions/artifacts/tenants. Finding/classification/signal envelope RiskScored Exponential backoff, 5 attempts Keyed on (subjectType, subjectId); latest-wins with version
DataClassificationWorker Event: ArtifactCreated, ArtifactUpdated (from factory) Classify new/changed artifacts and data; assign labels that drive handling policy. Artifact metadata envelope DataClassified Exponential backoff, 5 attempts Keyed on (artifactId, contentHash); reclassify only on content change
SecretScanWorker Event: ArtifactCreated, commit/push signals; Schedule (daily sweep) Scan artifacts/config/commits for leaked secrets; reconcile SecretReferences and rotation due dates. Artifact/commit envelope SecurityFindingRaised (leak); rotation reminders 3 attempts then alert Dedup on dedupKey; sweep watermark per resource

Event-Driven Flow

flowchart TB
    subgraph Inbound["Inbound signals"]
        Artifact["ArtifactCreated / ArtifactUpdated"]
        Scan["External scan results"]
        Action["Sensitive action signals"]
        ReportReq["compliance.report.requested"]
        Tick["Scheduler ticks"]
    end

    Artifact --> ClassW["DataClassificationWorker"]
    Artifact --> SecretW["SecretScanWorker"]
    Scan --> IngestW["SecurityScanIngestionWorker"]
    Action --> EvalW["PolicyEvaluationWorker"]
    ReportReq --> ReportW["ComplianceReportWorker"]
    Tick --> TimeoutW["ApprovalTimeoutWorker"]
    Tick --> ExportW["AuditExportWorker"]

    ClassW -->|DataClassified| RiskW["RiskScoreWorker"]
    IngestW -->|SecurityFindingRaised| RiskW
    SecretW -->|SecurityFindingRaised| RiskW
    RiskW -->|RiskScored| EvalW
    EvalW -->|PolicyDecisionRecorded| Audit["AuditService"]
    TimeoutW -->|ApprovalRejected| Audit
    ExportW -->|audit export| Blob["Blob evidence"]
    ReportW -->|ComplianceReportGenerated| Blob
Hold "Alt" / "Option" to enable pan & zoom

Reliability Notes

  • Idempotency — every worker deduplicates on eventId and additionally on a domain key (see table) so retries and duplicate Service Bus deliveries are safe, per the Metadata Schema idempotency-key rule.
  • Poison handling — unprocessable messages move to a dead-letter subqueue with the full envelope preserved for replay.
  • Tenant guard — every consumer asserts tenantId before touching a store; cross-tenant work is impossible by construction.
  • Backpressure — high-volume workers (SecurityScanIngestionWorker, DataClassificationWorker) use prefetch limits and concurrency caps tuned per environment in Deployment.
  • Trace propagation — traceId/correlationId flow from the inbound envelope into worker spans and the emitted events so the full chain is queryable in Observability.