Skip to content

Control Plane — Workers

Workers are the autonomous, event- and schedule-driven components of the Control Plane. They keep workflows moving without human intervention, enforce limits, build read projections, and export data. Every worker consumes the canonical event envelope, runs under a tenant context, and is idempotent — it derives an idempotency key from the triggering event so retries and duplicate deliveries are safe (see the Metadata Schema).

Target Architecture — Final-State Design

Workers are built on ConnectSoft.WorkerTemplate with MassTransit consumers and (for scheduled jobs) the Hangfire scheduler extension. Each worker is independently deployable and scales on its queue depth or schedule.

Idempotency & Retry Model

  • Idempotency key: sha256(eventId + ":" + handlerName + ":" + targetAggregateId). The result is stored so a replay returns the original outcome instead of re-executing side effects.
  • Retry: transient failures use MassTransit's exponential back-off (e.g. 5 attempts, 2s→2m). Exhausted messages move to a dead-letter subqueue with the full envelope preserved for replay.
  • Tenant guard: every handler asserts tenantId against the operation scope before touching a store.
  • Trace propagation: traceId/correlationId flow into OpenTelemetry spans and Serilog context.

Worker Catalogue

Worker Trigger Purpose Input Output Retry Idempotency
WorkflowTimeoutWorker Scheduled (Hangfire) + saga timeout Detect workflow steps/instances that exceed their deadline and raise timeout transitions or escalation. Due timeouts from the saga store / scheduled scan WorkflowStepTimedOut, escalation command, or compensation trigger Exponential, 5 attempts Key on (workflowInstanceId, stepId, deadline); timing out an already-advanced step is a no-op.
TaskAssignmentWorker Event: WorkflowStepReady Acquire an agent lease from AgentPoolManager and place an agent task. Ready step + required role/skill AgentTaskAssigned; lease acquired Exponential; requeue on no-capacity with back-off Key on (workflowInstanceId, stepId); re-delivery does not create a second task for the same step.
ProcessStateProjectionWorker Event: all Workflow*/AgentTask* events Build/refresh the read-optimized process-state projection in ProcessStateService. Workflow & task events Updated projection rows; ProcessStateProjected Exponential, 8 attempts Key on eventId; projection upserts are last-writer-wins by occurredAt.
QuotaEnforcementWorker Event: UsageRecorded + scheduled scan Compare consumption against edition quota; raise breaches and signal Tenant & Edition. UsageRecord, quota balances QuotaExceeded, throttle/suspend signal Exponential, 5 attempts Key on (tenantId, quotaKey, period, usageRecordId); double-count safe.
UsageRollupWorker Scheduled (hourly/daily) Aggregate raw UsageRecords into period rollups for billing and reporting. Raw usage records for a window UsageRolledUp, rollup rows Idempotent re-run for a window Key on (tenantId, period); rollups are computed deterministically and overwrite the window.
AuditExportWorker Scheduled (daily) + on-demand Export immutable audit entries to long-term/compliance storage (Blob/SIEM). Audit entries since last watermark Export file + AuditExported Exponential; resumes from watermark Key on (exportBatchId); watermark ensures no gaps/overlaps.
BlueprintValidationWorker Event: BlueprintCreated/BlueprintVersionSubmitted Run asynchronous, expensive validation (schema, naming, dependency, policy) via BlueprintValidatorService. BlueprintVersion BlueprintValidated / BlueprintValidationFailed Exponential, 3 attempts Key on (blueprintVersionId, ruleSetVersion); same version validated once per ruleset.
DependencyResolutionWorker Event: DependencyDeclared/ModuleRegistered Recompute the module dependency graph and detect cycles/missing nodes. Module + dependency edges DependencyResolved/DependencyCycleDetected; updated graph Exponential, 5 attempts Key on (projectId, graphRevision); graph rebuild is deterministic.
WorkflowReplayWorker Command: ReplayWorkflow Deterministically replay an instance's event history to rebuild state or re-derive an outcome. Instance id + event history Rebuilt projection / shadow instance; WorkflowReplayCompleted Manual retry (operator-initiated) Key on (workflowInstanceId, replayId); replays write to a shadow stream, never mutating the source.

Worker Trigger Flow

flowchart LR
    Bus["Azure Service Bus<br/>(canonical envelope)"] --> StepReady["WorkflowStepReady"]
    StepReady --> TaskAssignmentWorker
    TaskAssignmentWorker -->|lease| Pool["AgentPoolManager"]
    Bus --> UsageRecorded["UsageRecorded"]
    UsageRecorded --> QuotaEnforcementWorker
    UsageRecorded --> UsageRollupWorker
    Bus --> WfEvents["Workflow & Task events"]
    WfEvents --> ProcessStateProjectionWorker
    Scheduler["Hangfire scheduler"] --> WorkflowTimeoutWorker
    Scheduler --> AuditExportWorker
    Scheduler --> UsageRollupWorker
    Bus --> BpCreated["BlueprintCreated"]
    BpCreated --> BlueprintValidationWorker
    Bus --> DepDeclared["DependencyDeclared"]
    DepDeclared --> DependencyResolutionWorker
    ReplayCmd["ReplayWorkflow command"] --> WorkflowReplayWorker
Hold "Alt" / "Option" to enable pan & zoom

Design Principles

  • Event-first, schedule-as-safety-net: workers react to events for low latency; scheduled scans catch missed timeouts and reconcile drift.
  • No side effects without a recorded outcome: every worker persists its result keyed by idempotency key before emitting downstream events.
  • Compensation over rollback: failed multi-step operations trigger compensating actions in the workflow, not distributed transactions.
  • Tenant-scoped concurrency: AgentPoolManager leases and quota checks bound concurrency per tenant so one tenant cannot starve others.