Skip to content

Generated SaaS Workers

Target Architecture — Final-State Design

This page documents the common workers generated into every SaaS Product. Each is a stamped instance of ConnectSoft.WorkerTemplate, hosted as a .NET worker process, consuming the canonical event envelope from Azure Service Bus via MassTransit. Products add domain-specific workers; the ones below form the reusable baseline.

Workers are the asynchronous backbone of a generated product. They decouple slow, fan-out, or external-facing work from request handling, and they are the mechanism by which domain events become side effects — notifications sent, reports produced, external systems synced, audit exported, and outbound events reliably published. Every worker deduplicates on eventId, asserts tenantId, and moves poison messages to a dead-letter subqueue with the full envelope preserved for replay.

Common generated workers

Worker Trigger Purpose Input Output Retry Idempotency
NotificationDispatchWorker NotificationRequested / * domain events with notification rules Render and deliver multi-channel notifications (email, SMS, in-app, webhook) Enveloped event + notification template NotificationSent event; delivery receipt Exponential backoff, max 5; DLQ on exhaustion Dedup on eventId + channel; delivery key per recipient
AuditExportWorker Schedule (e.g. hourly) + AuditEntryRecorded batch Export immutable audit entries to long-term Blob/archive store Audit entries since watermark Archived audit batch in Blob; AuditBatchExported Backoff, max 3; resumes from watermark Watermark cursor; batch hash dedup
ReportGenerationWorker ReportRequested / report run command Generate report artifacts (PDF/CSV/XLSX) from definitions ReportDefinition + parameters Report artifact in Blob; ReportGenerated event Backoff, max 3; DLQ on exhaustion Dedup on reportRunId
IntegrationSyncWorker Schedule + IntegrationSyncRequested Pull/push data to/from external systems via connectors IntegrationConnection + sync window External sync result; IntegrationSucceeded/IntegrationFailed Backoff, max 5; circuit breaker per connection Dedup on (connectionId, syncWindow); external cursor
OutboxWorker Polling outbox table / IMessageScheduler tick Publish persisted outbox messages to Service Bus, guaranteeing at-least-once delivery Unpublished outbox rows Published enveloped events; outbox rows marked sent Continuous retry until published Outbox row id; broker dedup on eventId

Implementation Notes

The OutboxWorker is the linchpin of reliable messaging: services write domain events into an outbox table in the same database transaction as the state change, and the OutboxWorker publishes them asynchronously. This avoids the dual-write problem and means a service never loses an event even if Service Bus is briefly unavailable. MassTransit's transactional outbox support backs this pattern.

Event-flow diagram

flowchart TB
    DomainSvc["Domain / Spine Services"] -->|"write outbox row<br/>(same transaction)"| Outbox[("Outbox Table")]
    Outbox --> OutboxWorker["OutboxWorker"]
    OutboxWorker -->|"publish enveloped events"| Bus["Azure Service Bus"]

    Bus -->|"NotificationRequested"| NotificationDispatchWorker["NotificationDispatchWorker"]
    Bus -->|"ReportRequested"| ReportGenerationWorker["ReportGenerationWorker"]
    Bus -->|"IntegrationSyncRequested"| IntegrationSyncWorker["IntegrationSyncWorker"]
    Bus -->|"AuditEntryRecorded"| AuditExportWorker["AuditExportWorker"]

    NotificationDispatchWorker -->|"NotificationSent"| Bus
    ReportGenerationWorker -->|"ReportGenerated"| Bus
    IntegrationSyncWorker -->|"IntegrationSucceeded / IntegrationFailed"| Bus
    AuditExportWorker -->|"AuditBatchExported"| Blob[("Blob Archive")]

    NotificationDispatchWorker --> Channels["Email / SMS / In-App / Webhook"]
    ReportGenerationWorker --> ReportBlob[("Report Blob")]
    IntegrationSyncWorker --> External["External Systems"]

    Bus -.->|"poison messages"| DLQ[("Dead-Letter Subqueue")]
Hold "Alt" / "Option" to enable pan & zoom

Worker execution rules

  • Idempotency first. Every consumer checks an idempotency store keyed by eventId + handler name before acting, so redelivery (at-least-once) never double-applies an effect.
  • Tenant guard. Each handler asserts tenantId from the envelope against the operation scope before touching any store.
  • Bounded retry. Transient failures retry with exponential backoff up to a per-worker maximum; exhausted messages dead-letter with the full envelope for later replay.
  • Trace propagation. traceId/correlationId flow from the triggering event into the worker's OpenTelemetry span and Serilog context, so async side effects remain correlated to the originating request.
  • Scheduling. Time-based workers (AuditExport, IntegrationSync) use durable schedules; they resume from a persisted watermark rather than reprocessing from the beginning.

How workers contribute to the pillars

  • Traceability — every worker carries the originating traceId forward, so a notification or report links back to the action that triggered it.
  • Reusability — all workers are stamped from ConnectSoft.WorkerTemplate with identical retry, idempotency, and DLQ wiring.
  • Autonomy — the Worker Generator Agent produces each worker from blueprint rules describing the events it consumes and effects it produces.
  • Governance — audit export and dead-lettering create durable, inspectable records of asynchronous work.
  • Observability — workers emit per-message metrics (processed, retried, dead-lettered) and durations tagged by tenant and event type.
  • Multi-tenant scale — workers scale horizontally on queue depth; tenant scoping keeps one tenant's backlog from starving another (via partitioning/competing consumers).