FleetIncidentService¶

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.


Bounded context	BC-37 Fleet Incidents & Off-boarding
Platform / plane	Product Fleet Operations / Execution
Deployment granularity	module of host · host `FleetOperationsHost`
Status	target · `ConnectSoft.Factory.Fleet.Incidents`
Built from	connectsoft-microservice · libraries: connectsoft-extensions-saas-abstractions, connectsoft-extensions-messaging-model, connectsoft-extensions-persistence-model-nhibernate, connectsoft-extensions-scheduler-model-hang-fire
Decisions	ADR-0019
Traceability	`FR-09` `FR-12` `FUX-04` `FUX-11` `NFR-17` `NFR-18` `XL-13` `XL-16`

Purpose and responsibilities¶

FleetIncidentService is the operations desk for operated products. Incidents open from critical alerts, halted rollouts or an Operator, carry a severity with acknowledgement targets per edition (D33), page the on-call responder through a paging connector, notify affected customers through the owner's chosen connectors with receipts, link the alerts, rollouts and Factory runs involved, and close with a post-incident review for Sev1 and Sev2. A mitigation that changes a product goes through a rollout or a promotion; an incident never applies changes itself. Off-boarding — a customer tenant leaving a product, or a product leaving operation — is a saga over the Data & Event platform's governance orders: approval, export in open formats, handover acknowledged by the customer (or a recorded waiver), then deletion within the retention window, with evidence at every step (D31).

Context and boundaries¶

flowchart LR
    S["FleetIncidentService"]
    StudioBff["StudioBff"] -->|sync-http| S
    S -->|grpc| ConnectorRegistryService["ConnectorRegistryService"]
    S -->|grpc| RunControlService["RunControlService"]
    S -->|grpc| RetentionResidencyService["RetentionResidencyService"]
    S -->|grpc| FleetInventoryService["FleetInventoryService"]
    S -->|grpc| ArtifactService["ArtifactService"]
    S -. "IncidentOpened" .-> E_IncidentOpened(("event"))
    S -. "IncidentStateChanged" .-> E_IncidentStateChanged(("event"))
    S -. "IncidentResolved" .-> E_IncidentResolved(("event"))
    S -. "OnCallChanged" .-> E_OnCallChanged(("event"))
    S -. "OffboardingRequested" .-> E_OffboardingRequested(("event"))
    S -. "OffboardingCompleted" .-> E_OffboardingCompleted(("event"))

Hold "Alt" / "Option" to enable pan & zoom

The Observability platform's alerts are the Factory's own operator attention; this service is the product-level incident desk on top of them, and the later IncidentService of BC-21 would be its sibling for the Factory itself.

Domain model¶

ProductIncident — An incident on one product (and the affected instances): severity, state, timeline entries, linked alerts, rollouts and Factory runs, the responder, customer notifications sent, and the post-incident review. (catalog entry)

Field	Type	Required	Description
`incidentId`	`guid`	yes
`tenantId`	`string`	yes
`productId`	`string`	yes
`instanceIds`	`list:guid`	yes
`severity`	`enum:IncidentSeverity`	yes
`state`	`enum:IncidentState`	yes
`title`	`string`	yes
`linkedAlertIds`	`list:guid`	yes
`linkedRolloutId`	`guid`	no
`linkedRunIds`	`list:guid`	yes
`responder`	`string`	no
`timeline`	`json`	yes
`customerNotifications`	`json`	yes
`reviewRef`	`string`	no	Post-incident review artifact.
`openedAt`	`datetime`	yes
`resolvedAt`	`datetime`	no

Invariants:

Sev1 and Sev2 incidents page the on-call of the product's edition and notify affected customers within the edition's target.
A mitigation that changes a product goes through a rollout or a promotion; an incident never applies changes itself.
Resolution requires a review for Sev1 and Sev2 before the incident is Reviewed.

stateDiagram-v2
    Open --> Acknowledged : ResponderAcknowledges
    Acknowledged --> Mitigated : MitigationApplied
    Mitigated --> Resolved : ObjectivesRecovered
    Resolved --> Reviewed : ReviewRecorded

Hold "Alt" / "Option" to enable pan & zoom

Events: IncidentOpened, IncidentStateChanged, IncidentResolved · Tables: ProductIncidents, IncidentTimeline · Repository: ProductIncidentRepository

OnCallRecord — Who is on call for which products and edition in which window, with the paging connector used. (catalog entry)

Field	Type	Required
`recordId`	`guid`	yes
`tenantId`	`string`	yes
`productIds`	`list:string`	yes
`edition`	`string`	yes
`responder`	`string`	yes
`from`	`datetime`	yes
`to`	`datetime`	yes
`pagingConnectionId`	`guid`	yes

Invariants:

Every ManagedPlus product has an on-call responder at all times; a gap is an alert.

Events: OnCallChanged · Tables: OnCallRecords · Repository: OnCallRecordRepository

Offboarding — Off-boarding of a customer tenant from a product or of a whole product: approval, export order (data handover in open formats), handover acknowledgement by the customer, deletion order within the retention window, and completion evidence — orchestrated over the Data & Event platform's governance orders. (catalog entry)

Field	Type	Required	Description
`offboardingId`	`guid`	yes
`tenantId`	`string`	yes
`productId`	`string`	yes
`customerTenantId`	`string`	no	Empty for a whole-product off-boarding.
`approvalId`	`guid`	no
`exportOrderId`	`guid`	no
`handoverAcknowledgedBy`	`string`	no
`deletionOrderId`	`guid`	no
`state`	`enum:OffboardingState`	yes
`requestedAt`	`datetime`	yes
`completedAt`	`datetime`	no

Invariants:

Deletion starts only after the export was handed over and acknowledged, or the customer waived the export in writing (recorded as evidence).
Off-boarding a whole product needs an approval of class production; a customer tenant off-boarding needs the customer's request and the owner's approval.

stateDiagram-v2
    Requested --> Approved : ApprovalGranted
    Approved --> Exporting : ExportOrdered
    Exporting --> HandedOver : HandoverAcknowledged
    HandedOver --> Deleting : DeletionOrdered
    Deleting --> Completed : DeletionCompleted
    Requested --> Cancelled : Cancel

Hold "Alt" / "Option" to enable pan & zoom

Events: OffboardingRequested, OffboardingCompleted · Tables: Offboardings · Repository: OffboardingRepository

erDiagram
    ProductIncident {
        guid incidentId PK
        string tenantId
        string productId
        list_guid instanceIds
        enum_IncidentSeverity severity
        enum_IncidentState state
        string title
        list_guid linkedAlertIds
        guid linkedRolloutId
        list_guid linkedRunIds
        string responder
        json timeline
        json customerNotifications
        string reviewRef
        datetime openedAt
        datetime resolvedAt
    }
    OnCallRecord {
        guid recordId PK
        string tenantId
        list_string productIds
        string edition
        string responder
        datetime from
        datetime to
        guid pagingConnectionId
    }
    Offboarding {
        guid offboardingId PK
        string tenantId
        string productId
        string customerTenantId
        guid approvalId
        guid exportOrderId
        string handoverAcknowledgedBy
        guid deletionOrderId
        enum_OffboardingState state
        datetime requestedAt
        datetime completedAt
    }

Hold "Alt" / "Option" to enable pan & zoom

Entities and value objects¶

None modelled

No entities or value objects are modelled for this item yet.

Data model and storage¶

Store	Role	Engine / profile	Tables
Fleet operations store	primary	azure-sql / nhibernate-sqlserver	ProductIncidents, IncidentTimeline, OnCallRecords, Offboardings

Table	Owner	Columns	Indexes
`dbo.ProductIncidents`	ProductIncident	15	1
`dbo.IncidentTimeline`	ProductIncident	7	0
`dbo.OnCallRecords`	OnCallRecord	8	1
`dbo.Offboardings`	Offboarding	11	1

Full DDL: appendix.

APIs¶

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
`GET`	`/factory/fleet/incidents`	ListIncidents	`factory.fleet.read`	read	natural	GW-03	Incidents by product, severity and state.
`GET`	`/factory/fleet/incidents/{incidentId}`	GetIncident	`factory.fleet.read`	read	natural	GW-03	One incident with timeline, links, notifications and review.
`POST`	`/factory/fleet/incidents`	OpenIncident	`factory.fleet.operate`	shared-write	key-header	GW-03	Open an incident manually (Operator) or from an alert rule.
`POST`	`/factory/fleet/incidents/{incidentId}/transition`	TransitionIncident	`factory.fleet.operate`	shared-write	natural	GW-03	Acknowledge, mitigate (with the rollout or promotion reference), resolve, record the review.
`GET`	`/factory/fleet/on-call`	ListOnCall	`factory.fleet.read`	read	natural	GW-03	On-call records by product and window.
`PUT`	`/factory/fleet/on-call/{recordId}`	SetOnCall	`factory.fleet.admin`	shared-write	natural	GW-03	Create or change an on-call record.
`POST`	`/factory/fleet/offboardings`	RequestOffboarding	`factory.fleet.admin`	production	key-header	GW-03	Request off-boarding of a customer tenant or a product (approval, export, handover, deletion).
`POST`	`/factory/fleet/offboardings/{offboardingId}/acknowledge-handover`	AcknowledgeHandover	`factory.fleet.admin`	shared-write	natural	GW-03	The customer (or the owner on their written waiver) acknowledges the export; deletion may start.
`GET`	`/factory/fleet/offboardings/{offboardingId}`	GetOffboarding	`factory.fleet.read`	read	natural	GW-03	State with export, handover and deletion order references.

None modelled

No gRPC methods are modelled for this item yet.

Messages¶

Event	Topic	Scope / v	Aggregate	Consumers	Summary
IncidentOpened	`fleet-operations.incidents.v1.incident-opened`	integration / 1	ProductIncident	StudioBff, StudioNotificationService, AuditService, DashboardService, AlertRuleService	An incident was opened; on-call paged for Sev1 and Sev2.
IncidentStateChanged	`fleet-operations.incidents.v1.incident-state-changed`	integration / 1	ProductIncident	StudioBff, AuditService, DashboardService	Acknowledged, mitigated, resolved or reviewed.
IncidentResolved	`fleet-operations.incidents.v1.incident-resolved`	integration / 1	ProductIncident	StudioBff, AuditService, SloService, FeedbackService	The incident is resolved; customers notified.
OnCallChanged	`fleet-operations.on-call.v1.on-call-changed`	integration / 1	OnCallRecord	StudioBff, AuditService	On-call coverage changed.
OffboardingRequested	`fleet-operations.offboarding.v1.offboarding-requested`	integration / 1	Offboarding	StudioBff, AuditService, FactoryCommercialService	An off-boarding was requested.
OffboardingCompleted	`fleet-operations.offboarding.v1.offboarding-completed`	integration / 1	Offboarding	FleetInventoryService, StudioBff, AuditService, FactoryCommercialService	Export handed over and deletion completed with evidence.

IncidentOpened payload:

Field	Type	Required
`incidentId`	`guid`	yes
`tenantId`	`string`	yes
`productId`	`string`	yes
`severity`	`enum:IncidentSeverity`	yes
`title`	`string`	yes

IncidentStateChanged payload:

Field	Type	Required
`incidentId`	`guid`	yes
`tenantId`	`string`	yes
`productId`	`string`	yes
`fromState`	`enum:IncidentState`	yes
`toState`	`enum:IncidentState`	yes

IncidentResolved payload:

Field	Type	Required
`incidentId`	`guid`	yes
`tenantId`	`string`	yes
`productId`	`string`	yes
`severity`	`enum:IncidentSeverity`	yes
`durationMinutes`	`int`	yes

OnCallChanged payload:

Field	Type	Required
`recordId`	`guid`	yes
`tenantId`	`string`	yes
`responder`	`string`	yes
`from`	`datetime`	yes
`to`	`datetime`	yes

OffboardingRequested payload:

Field	Type	Required
`offboardingId`	`guid`	yes
`tenantId`	`string`	yes
`productId`	`string`	yes
`customerTenantId`	`string`	no

OffboardingCompleted payload:

Field	Type	Required
`offboardingId`	`guid`	yes
`tenantId`	`string`	yes
`productId`	`string`	yes
`customerTenantId`	`string`	no
`deletionOrderId`	`guid`	yes

Command	Handled by	Result events
`OpenIncident`	ProductIncident	IncidentOpened
`TransitionIncident`	ProductIncident	IncidentStateChanged, IncidentResolved
`SetOnCall`	OnCallRecord	OnCallChanged
`RequestOffboarding`	Offboarding	OffboardingRequested
`AcknowledgeHandover`	Offboarding	—

Event	Producer	Handler	Purpose
AlertFired	AlertRuleService	`OpenFromAlert`	Critical alerts on operated products open incidents by rule.
RolloutHalted	FleetRolloutService	`OpenFromRollout`	A halted rollout opens an incident linked to it.
ProductInstanceStateChanged	FleetInventoryService	`TrackAffected`	Affected instances and recovery.
ApprovalDecided	RunControlService	`ApplyOffboardingApproval`	Off-boarding approvals.
ExportCompleted	RetentionResidencyService	`ExportReady`	The export bundle is ready for handover.
DeletionOrderCompleted	RetentionResidencyService	`DeletionDone`	Off-boarding completes.

Workers and jobs¶

Worker	Trigger	Output events	Retry	Idempotency key	Summary
Pager	event	—	exponential	`incidentId + escalationLevel`	Pages the on-call responder for Sev1 and Sev2 through the paging connector and escalates after the edition's acknowledgement target.
CustomerNotifier	event	—	exponential	`incidentId + state + customerTenantId`	Sends customer notifications for opened, mitigated and resolved incidents through the product owner's chosen connectors, recording receipts.
OnCallGapChecker	schedule `0 * * * *`	—	none	`hour`	Alerts when a ManagedPlus product has no on-call coverage in the coming window.

Actors and grains¶

None modelled

No grains are modelled for this item yet.

Sagas and long-running flows¶

OffboardingSaga — correlation offboardingId; starts on FleetIncidentService.OffboardingRequested

Step	Trigger	Do	Compensation
1	OffboardingRequested	Request the approval (production class for a whole product; owner approval for a customer tenant).	Cancel.
2	ApprovalDecided (Approved)	Issue an export order through RetentionResidencyService for the scope.	Cancel; evidence.
3	ExportCompleted	Deliver the bundle to the customer through a connector; wait for the handover acknowledgement or a recorded waiver.	Keep Exporting; remind; escalate.
4	HandoverAcknowledged	Issue the deletion order (30-day window) and suspend the instances.	none (irreversible from here)
5	DeletionOrderCompleted	Mark Completed; close the subscription; evidence.	none

stateDiagram-v2
    [*] --> Requested
    Requested --> Approved: ApprovalGranted
    Approved --> Exporting: ExportOrdered
    Exporting --> HandedOver: HandoverAcknowledged (or recorded waiver)
    HandedOver --> Deleting: DeletionOrdered (30-day window)
    Deleting --> Completed: DeletionCompleted
    Requested --> Cancelled: Cancel
    Completed --> [*]

Hold "Alt" / "Option" to enable pan & zoom

Options and configuration¶

Section	Key	Type	Default	Reloadable	Secret	Summary
`Fleet:Incidents`	`ResponseTargets`	json	`{ Managed: { Sev1AckMinutes: 30, Sev2AckMinutes: 120 }, ManagedPlus: { Sev1AckMinutes: 15, Sev2AckMinutes: 60 } }`	yes	no	Acknowledgement targets per edition.
`Fleet:Offboarding`	`HandoverReminderDays`	int	`7`	yes	no	Days between handover reminders.

Metrics and observability¶

Metric	Instrument	Unit	Dimensions	Summary
`connectsoft.factory.fleet.incidents_open`	updown	{incident}	tenant, productId, severity	Open incidents.
`connectsoft.factory.fleet.time_to_acknowledge`	histogram	min	edition, severity	Time to acknowledgement.
`connectsoft.factory.fleet.time_to_resolve`	histogram	min	edition, severity	Time to resolution.
`connectsoft.factory.fleet.offboardings`	counter	{offboarding}	tenant, state	Off-boardings by state.

Security¶


Authentication	client-credentials
Scopes	`factory.fleet.read` (Read incidents, on-call and off-boardings.); `factory.fleet.operate` (Open and transition incidents (Operator).); `factory.fleet.admin` (On-call and off-boarding (Workspace admin of the owner, Platform admin).)
Roles	Operator, WorkspaceAdmin, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	confidential
Max side-effect class	production
Secrets	—

Dependencies¶

Outbound

Target	Kind	Purpose	Failure mode	Required
ConnectorRegistryService	grpc	Paging and customer notifications.	degrade	yes
RunControlService	grpc	Off-boarding approvals.	fail-fast	yes
RetentionResidencyService	grpc	Export and deletion orders.	fail-fast	yes
FleetInventoryService	grpc	Instances and editions.	fail-fast	yes
ArtifactService	grpc	Post-incident reviews.	degrade	no

Inbound

Target	Kind	Purpose	Failure mode	Required
StudioBff	sync-http	Incident, on-call and off-boarding views.	—	yes

Failure and recovery¶

Scenario	Detection	Recovery	Data impact	RTO (min)
Paging connector down	delivery failures	Studio notification always delivered; escalation to the next responder; alert	delayed response	15

NFR targets¶

Target	Class value (pilot-batch)	Override
latency_p95_ms	0	—
availability_pct	99.5	—
throughput_rps	10	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	medium	—

Deployment¶


Granularity	module of host · host `FleetOperationsHost`
Topologies	saas-default, hybrid
Runtime	—; replicas —–—
Ports	—

Current implementation¶

Nothing: incidents are handled in chat and Azure DevOps work items; no on-call record, customer notification or off-boarding flow exists.

Later capabilities¶

Customer-facing incident communication pages; on-call schedules generated per edition; automated runbooks proposed from reviews as skills.

Decisions and traceability¶

Family	Ids
FR	FR-09, FR-12
FUX	FUX-04, FUX-11
NFR	NFR-17, NFR-18
XL	XL-13, XL-16

Overview · AlertRuleService · RetentionResidencyService · ConnectorRegistryService
Product Fleet Operations microservices · bounded contexts · APIs · events · data model
Services catalog · Ownership map

FleetIncidentService¶

Purpose and responsibilities¶

Context and boundaries¶

Domain model¶

Entities and value objects¶

Data model and storage¶

APIs¶

Messages¶

Workers and jobs¶

Actors and grains¶

Sagas and long-running flows¶

Options and configuration¶

Metrics and observability¶

Security¶

Dependencies¶

Failure and recovery¶

NFR targets¶

Deployment¶

Current implementation¶

Later capabilities¶

Decisions and traceability¶

Related¶