FleetRolloutService

Target

This page describes intended behaviour or structure. No implementation is claimed unless a section is explicitly marked Current implementation or Evidence.

Bounded context	BC-36 Fleet Upgrades & Rollouts
Platform / plane	Product Fleet Operations / Execution
Deployment granularity	module of host · host FleetOperationsHost
Status	target · ConnectSoft.Factory.Fleet.Rollouts
Built from	connectsoft-microservice · libraries: connectsoft-extensions-saas-abstractions, connectsoft-extensions-messaging-model, connectsoft-extensions-messaging-model-orleans, connectsoft-extensions-persistence-model-nhibernate, connectsoft-extensions-scheduler-model-hang-fire
Decisions	ADR-0015, ADR-0021
Traceability	FR-08 FR-09 FUX-02 FUX-05 NFR-08 NFR-09 XL-04 XL-10

Purpose and responsibilities

FleetRolloutService moves many instances safely. A rollout has a kind (release upgrade, template upgrade, library upgrade, re-scaffold, configuration change), a target, ordered waves (instance selection, size, bake minutes), guards (product objectives that halt the rollout on breach) and, when any wave touches production, a production-class approval. Each wave requests promotions through EnvironmentPromotionService with the same gates as any promotion, then bakes: the guards are watched for the bake time; a breach or failed promotion halts the rollout, rolls the wave back and opens an incident; an Operator resumes or rolls back fully. A re-scaffold rollout begins earlier: when a template release is published, the ReScaffoldRequester submits a Factory run that regenerates the product with it; only after that release passed evaluation and is ready does the rollout move it.

Context and boundaries

flowchart LR
    S["FleetRolloutService"]
    StudioBff["StudioBff"] -->|sync-http| S
    S -->|grpc| EnvironmentPromotionService["EnvironmentPromotionService"]
    S -->|grpc| RunControlService["RunControlService"]
    S -->|grpc| FleetInventoryService["FleetInventoryService"]
    S -->|event| SloService["SloService"]
    S -. "RolloutPlanned" .-> E_RolloutPlanned(("event"))
    S -. "RolloutWaveCompleted" .-> E_RolloutWaveCompleted(("event"))
    S -. "RolloutHalted" .-> E_RolloutHalted(("event"))
    S -. "RolloutCompleted" .-> E_RolloutCompleted(("event"))

Hold "Alt" / "Option" to enable pan & zoom

ManagedPlus customers opt in before early waves; the first wave never includes their production instances otherwise. The rollout never bypasses promotion gates; it is a scheduler of promotions with guards, not a second deployment path.

Domain model

Rollout — A plan to move a product's instances to a target (release, template or library version, re-scaffolded release) in ordered waves with sizes, bake times, objective guards and a halt policy; production waves need a production-class approval; a failed wave halts the rollout and rolls back the wave. (catalog entry)

Field	Type	Required	Description
rolloutId	guid	yes
tenantId	string	yes
productId	string	yes
kind	enum:RolloutKind	yes
targetReleaseId	guid	no
targetAssetReleaseId	string	no	Template or library release for upgrades and re-scaffolds.
sourceRunId	guid	no	The Factory run that produced the re-scaffolded release.
waves	json	yes	Ordered waves: instance selection, size, bake minutes
guards	json	yes	Objectives that halt the rollout on breach.
approvalId	guid	no
state	enum:RolloutState	yes
currentWave	int	yes
requestedBy	string	yes
startedAt	datetime	no
completedAt	datetime	no

Invariants:

• A wave is a set of promotions through EnvironmentPromotionService with the same gates; the rollout never bypasses them.
• The first wave never includes production instances of customers on ManagedPlus unless they opted in; bake time runs against the product's objectives and halts on breach.
• A re-scaffold rollout starts from a Factory run that regenerated the product with the new template and passed evaluation (ADR-0021); the rollout moves the resulting release.

stateDiagram-v2
    Planned --> AwaitingApproval : ProductionWaves
    Planned --> Rolling : StartNoApproval
    AwaitingApproval --> Rolling : ApprovalGranted
    AwaitingApproval --> Cancelled : ApprovalRejectedOrExpired
    Rolling --> Baking : WavePromoted
    Baking --> Rolling : BakePassedNextWave
    Baking --> Completed : BakePassedLastWave
    Baking --> Halted : GuardBreached
    Rolling --> Halted : PromotionFailed
    Halted --> RolledBack : WaveRolledBack
    Halted --> Rolling : OperatorResumes
    Planned --> Cancelled : Cancel

Hold "Alt" / "Option" to enable pan & zoom

Events: RolloutPlanned, RolloutWaveCompleted, RolloutHalted, RolloutCompleted · Tables: Rollouts, RolloutWaves · Repository: RolloutRepository

erDiagram
    Rollout {
        guid rolloutId PK
        string tenantId
        string productId
        enum_RolloutKind kind
        guid targetReleaseId
        string targetAssetReleaseId
        guid sourceRunId
        json waves
        json guards
        guid approvalId
        enum_RolloutState state
        int currentWave
        string requestedBy
        datetime startedAt
        datetime completedAt
    }
    RolloutWave {
        int index

        list_guid instanceIds

        list_guid promotionIds

        int bakeMinutes

        enum_WaveState state

        datetime startedAt

        datetime completedAt

    }
    Rollout ||--o{ RolloutWave : contains

Hold "Alt" / "Option" to enable pan & zoom

Entities and value objects

RolloutWave (entity, in Rollout) — One wave with its instances and promotions.

Field	Type	Required	Description
index	int	yes
instanceIds	list:guid	yes
promotionIds	list:guid	yes
bakeMinutes	int	yes
state	enum:WaveState	yes
startedAt	datetime	no
completedAt	datetime	no

Data model and storage

Store	Role	Engine / profile	Tables
Fleet operations store	primary	azure-sql / nhibernate-sqlserver	Rollouts, RolloutWaves

Table	Owner	Columns	Indexes
dbo.Rollouts	Rollout	15	1
dbo.RolloutWaves	RolloutWave	9	0

Full DDL: appendix.

APIs

Method	Path	Operation	Scope	Side effects	Idempotency	Gateway	Summary
GET	/factory/fleet/rollouts	ListRollouts	factory.fleet.read	read	natural	GW-03	Rollouts by product, kind and state with wave progress.
GET	/factory/fleet/rollouts/{rolloutId}	GetRollout	factory.fleet.read	read	natural	GW-03	One rollout with waves, promotions, guards, approval and evidence.
POST	/factory/fleet/rollouts	PlanRollout	factory.fleet.operate	production	key-header	GW-03	Plan a rollout (waves, bake, guards) of a release, template or library upgrade or a re-scaffold; production waves are production class.
POST	/factory/fleet/rollouts/{rolloutId}/resume	ResumeRollout	factory.fleet.operate	production	natural	GW-03	Resume a halted rollout after the Operator judged the breach (production class).
POST	/factory/fleet/rollouts/{rolloutId}/rollback	RollbackRollout	factory.fleet.operate	production	natural	GW-03	Roll back the current wave (and optionally completed waves) to the previous release.
POST	/factory/fleet/rollouts/{rolloutId}/cancel	CancelRollout	factory.fleet.operate	shared-write	natural	GW-03	Cancel a planned rollout.

None modelled

No gRPC methods are modelled for this item yet.

Messages

Event	Topic	Scope / v	Aggregate	Consumers	Summary
RolloutPlanned	fleet-rollouts.rollouts.v1.rollout-planned	integration / 1	Rollout	StudioBff, AuditService, StudioNotificationService	A rollout was planned (and may await approval).
RolloutWaveCompleted	fleet-rollouts.rollouts.v1.rollout-wave-completed	integration / 1	Rollout	StudioBff, DashboardService, FleetInventoryService	A wave promoted and baked successfully.
RolloutHalted	fleet-rollouts.rollouts.v1.rollout-halted	integration / 1	Rollout	FleetIncidentService, AlertRuleService, StudioBff, AuditService, StudioNotificationService	A guard breached or a promotion failed; the wave is being rolled back.
RolloutCompleted	fleet-rollouts.rollouts.v1.rollout-completed	integration / 1	Rollout	FleetInventoryService, StudioBff, AuditService, DashboardService	Every wave completed (or the rollout was rolled back or cancelled).

RolloutPlanned payload:

Field	Type	Required	Description
rolloutId	guid	yes
tenantId	string	yes
productId	string	yes
kind	enum:RolloutKind	yes
waves	int	yes
instances	int	yes

RolloutWaveCompleted payload:

Field	Type	Required	Description
rolloutId	guid	yes
tenantId	string	yes
productId	string	yes
wave	int	yes
instances	int	yes

RolloutHalted payload:

Field	Type	Required	Description
rolloutId	guid	yes
tenantId	string	yes
productId	string	yes
wave	int	yes
reason	string	yes

RolloutCompleted payload:

Field	Type	Required	Description
rolloutId	guid	yes
tenantId	string	yes
productId	string	yes
state	enum:RolloutState	yes
targetReleaseId	guid	no

Command	Handled by	Result events	Summary
PlanRollout	Rollout	RolloutPlanned
ResumeRollout	Rollout	—
RollbackRollout	Rollout	RolloutCompleted
CancelRollout	Rollout	RolloutCompleted

Event	Producer	Handler	Purpose
ApprovalDecided	RunControlService	ApplyApproval	Production rollouts wait for their approval.
PromotionCompleted	EnvironmentPromotionService	TrackWavePromotion	Wave promotions complete.
PromotionFailed	EnvironmentPromotionService	HaltOnFailure	A failed promotion halts the wave.
SloBreached	SloService	HaltOnGuard	A guard objective breached during bake halts the rollout.
ReleaseReady	ReleaseService	OfferRollout	A new product release can be rolled out to instances.
AssetReleasePublished	AgentRegistryService	OfferUpgrade	A published template or library release offers an upgrade or re-scaffold rollout.
ProductInstanceStateChanged	FleetInventoryService	TrackInstances	Instance set and states for wave selection.

Workers and jobs

Worker	Trigger	Output events	Retry	Idempotency key	Summary
WaveDriver	event	RolloutWaveCompleted, RolloutHalted, RolloutCompleted	fixed	rolloutId + wave	Requests the promotions of the current wave, waits for their completion, runs the bake timer against the guards, and advances, halts or completes.
ReScaffoldRequester	event	—	exponential	productId + targetAssetReleaseId	For template upgrades, submits a Factory run that regenerates the product with the new template release (through Run Control) and waits for its release to be ready before planning the rollout.

Actors and grains

None modelled

No grains are modelled for this item yet.

Sagas and long-running flows

RolloutSaga — correlation rolloutId; starts on FleetRolloutService.RolloutPlanned

Step	Trigger	Do	Compensation
1	RolloutPlanned	Validate waves against instance states and edition opt-ins; request the production-class approval when any wave is production.	Cancel.
2	Approved	Request the promotions of wave 1 through EnvironmentPromotionService.	Halt; roll back completed promotions of the wave.
3	WavePromoted	Start the bake timer; watch the guard objectives and alerts.	On breach: halt, roll back the wave, open an incident.
4	BakePassed	Mark the wave Passed; request the next wave or complete.	none

sequenceDiagram
    participant AR as AgentRegistryService
    participant RO as FleetRolloutService
    participant RC as RunControlService
    participant RS as ReleaseService
    participant E as EnvironmentPromotionService
    participant S as SloService
    AR-)RO: AssetReleasePublished (template 3.2.0)
    RO->>RC: submit re-scaffold run for product P (sandbox → proposal → pipeline)
    RS-)RO: ReleaseReady (P 4.0.0)
    RO->>RO: plan waves 10 % / 40 % / 50 %, guards: availability, error rate
    RO->>RC: approval (production class) for production waves
    RC-)RO: ApprovalDecided
    loop each wave
        RO->>E: RequestPromotion × instances
        E-)RO: PromotionCompleted × instances
        RO->>RO: bake (60–240 min)
        S-)RO: SloBreached? → halt, roll back wave, incident
    end
    RO-)AR: RolloutCompleted

Hold "Alt" / "Option" to enable pan & zoom

Options and configuration

Section	Key	Type	Default	Reloadable	Secret	Summary
Fleet:Rollouts	DefaultWaves	json	[ { share: 0.1, bakeMinutes: 60 }, { share: 0.4, bakeMinutes: 120 }, { share: 0.5, bakeMinutes: 240 } ]	yes	no	Default wave plan.
Fleet:Rollouts	ManagedPlusOptInRequired	bool	true	no	no	ManagedPlus customers opt in before early waves.

Metrics and observability

Metric	Instrument	Unit	Dimensions	Summary
connectsoft.factory.fleet.rollouts	counter	{rollout}	tenant, kind, state	Rollouts by outcome.
connectsoft.factory.fleet.rollout_duration	histogram	h	kind	Plan to completion.
connectsoft.factory.fleet.rollout_halts	counter	{halt}	tenant, reason	Halts by reason.

Security

Authentication	client-credentials
Scopes	factory.fleet.read (Read rollouts.); factory.fleet.operate (Plan, resume, roll back, cancel (Operator; production waves approved by a Reviewer).)
Roles	Operator, Reviewer, PlatformAdmin, Observer
Tenant isolation	row-filter
Data classification	confidential
Max side-effect class	production
Secrets	—

Dependencies

Outbound

Target	Kind	Purpose	Failure mode	Required
EnvironmentPromotionService	grpc	Promotions per wave; rollbacks.	fail-fast	yes
RunControlService	grpc	Approvals; re-scaffold runs.	fail-fast	yes
FleetInventoryService	grpc	Instances and editions.	fail-fast	yes
SloService	event	Guard objectives.	queue	yes

Inbound

Target	Kind	Purpose	Failure mode	Required
StudioBff	sync-http	Rollout views and operator actions.	—	yes

Failure and recovery

Scenario	Detection	Recovery	Data impact	RTO (min)
Guard breached during bake	SloBreached on a guard	halt; roll back the wave; incident; Operator decides to resume or roll back fully	brief exposure on the wave's instances	30

NFR targets

Target	Class value (pilot-batch)	Override
latency_p95_ms	0	—
availability_pct	99.5	—
throughput_rps	10	—
rpo_minutes	15	—
rto_minutes	240	—
cost_class	medium	—

Deployment

Granularity	module of host · host FleetOperationsHost
Topologies	saas-default, hybrid
Runtime	—; replicas —–—
Ports	—

Current implementation

Nothing: upgrades are per-repository pull requests and pipeline runs.

Later capabilities

Canary analysis against per-instance objectives; customer-scheduled maintenance windows per instance; dependency-ordered rollouts across products that integrate with each other.

Decisions and traceability

Family	Ids
FR	FR-08, FR-09
FUX	FUX-02, FUX-05
NFR	NFR-08, NFR-09
XL	XL-04, XL-10

Related

• Overview · EnvironmentPromotionService · AgentRegistryService · Quality gates (ADR-0021)

• Product Fleet Operations microservices · bounded contexts · APIs · events · data model

• Services catalog · Ownership map