Fly.io CI Deployment Control

Overview

Build CI as a controlled release system rather than a single deploy command. Separate pull-request validation from authenticated deployment, bind production to an immutable revision, and retain enough evidence to identify and reverse the release.

Prerequisites

• Named app, organization, environment, release owner, and rollback owner
• Protected CI environment with an approved secret manager
• Health endpoint or other deployment check and a tested prior release

Instructions

Step 1: Separate validation from deployment

Run secret-free configuration, container, and contract checks on pull requests. Permit authenticated Fly.io access only in a protected post-merge or manually approved environment.

Step 2: Issue the narrowest token

Use an app-scoped deploy token for one app, an organization token only for approved multi-app workflows, or a read-only token for observation. Set an explicit expiry and store only the secret reference.

Step 3: Pin the release inputs

Bind source commit, image digest, flyctl setup action or binary version, target app, configuration hash, and deployment strategy before execution.

Step 4: Validate the candidate

Check the rendered app configuration, image startup contract, health checks, release command, volume constraints, and rollback target without changing production.

Step 5: Deploy through one serialized lane

Use environment concurrency so two production releases cannot race. Capture the release identifier and wait for health checks rather than treating command exit alone as success.

Step 6: Verify and close

Probe the approved endpoint, inspect aggregate health and Machine state, reconcile the running image, and either record success or invoke the tested rollback.

Authentication

Expose the token to the deployment step as FLY_API_TOKEN or FLY_ACCESS_TOKEN only. The provider recommends scoped tokens created with fly tokens create; do not use the deprecated hidden fly auth token output in CI. Mask the value and revoke it after suspected exposure.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• CI trust-boundary and approval map
• Pinned workflow with validation, deploy, health, and rollback stages
• Release receipt containing revision, image, app, strategy, checks, and token reference

Return…

Fly.io Failure Triage

Overview

Classify the failing layer before changing anything. Separate local build failures, release orchestration, Machine lifecycle, Fly Proxy routing, private DNS, volume placement, and application behavior so remediation does not destroy the evidence.

Prerequisites

• App and environment identifiers plus incident start time
• Last known healthy release, image, configuration, and region placement
• Read-only access to health, Machine, release, volume, and log evidence

Instructions

Step 1: Freeze the incident window

Record the symptom, first observation, affected regions, recent release or secret changes, and the last known healthy revision.

Step 2: Validate configuration and release identity

Compare the deployed image and rendered configuration with the intended release. Check process groups, ports, services, health paths, signals, and timeouts.

Step 3: Inspect health and Machine lifecycle

Distinguish a failed health check from a Machine that is stopped, suspended, replacing, or repeatedly restarting. Preserve instance-version information.

Step 4: Check routing and networking

Verify public allocation, Fly Proxy service configuration, listening address, target port, and private .internal DNS semantics. Remember that stopped Machines are absent from AAAA responses.

Step 5: Check regional storage dependencies

Confirm that any attached volume exists in the same region and is mounted at the expected path. Treat a single volume as a single-host durability boundary.

Step 6: Choose the smallest recovery

Prefer rollback, configuration correction, or one bounded Machine action over fleet-wide restart. Reconcile health and image state after recovery.

Authentication

Use a read-only organization token for observation where possible. A deploy-capable identity can change code and therefore reach runtime secrets; keep diagnostic access separate from mutation authority and never paste tokens into incident records.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Layered incident timeline and affected-resource map
• Redacted evidence matrix for config, release, health, Machine, network, and volume state
• Bounded recovery or escalation plan with rollback and verification criteria

Return the target organization, app, environment, region set, Machine or database identifiers, source-c…

Fly.io Application Release Lifecycle

Overview

Treat the app release as a state transition with explicit configuration, image, capacity, health, and rollback boundaries. Fly Launch manages Machines from fly.toml, but production success still requires reconciliation against the running fleet.

Prerequisites

• Source revision, image build contract, app name, environment, and primary region
• Reviewed fly.toml with process groups, services, health checks, and shutdown settings
• Approved deploy strategy, capacity envelope, secret plan, and rollback target

Instructions

Step 1: Inventory the current app

Record current release, running image, Machine counts and regions, health, IP allocation, secrets digests, and attached storage.

Step 2: Render the desired state

Review build, process groups, services, concurrency, autostart or autostop, VM resources, regions, mounts, release command, and deployment policy.

Step 3: Stage secrets separately

Use Fly secrets rather than plaintext [env] values. Decide whether a secret change should restart Machines immediately or be staged for the release.

Step 4: Select the deployment strategy

Use rolling by default. Use canary only when temporary extra capacity is valid and no Machine volume blocks it; use blue-green only with health checks and compatible stateless capacity.

Step 5: Deploy and wait

Bind the release to an immutable image, observe replacement, and wait for health and Machine convergence. Do not infer success from image push alone.

Step 6: Reconcile or roll back

Compare desired and actual image, count, regions, health, and service behavior. Roll back if acceptance criteria or error budgets fail.

Authentication

Use an app-scoped deploy token for a single app and a short-lived organization token only when the operation legitimately spans apps. Fly App secrets are encrypted at rest but become environment variables inside Machines; anyone with deploy access can deploy code that reads them.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Preflight snapshot and desired-state review
• Release plan with strategy, capacity, health, and rollback gates
• Post-release reconciliation receipt or rollback record

Return the target organization, app, environment, region set, Machine or database identifiers, source-contract fingerprint, evid…

Fly.io Managed Data and Private Networking

Overview

Choose the data service before writing commands: Managed Postgres is a separate fully managed service, while Fly Volumes are region-bound local NVMe storage attached to Machines. Private 6PN networking connects apps in the same organization but does not make either storage model globally consistent.

Prerequisites

• Workload, data classification, residency, availability, recovery, and latency requirements
• Approved regions and network boundary
• Restore-test owner and application cutover plan

Instructions

Step 1: Select the persistence model

Use Managed Postgres when the workload needs the provider-managed database service. Use a Fly Volume only for software that owns replication and recovery or for region-local state.

Step 2: Place data intentionally

Choose a supported Managed Postgres region close to the app or create volumes in the same region as their Machines. Query current region availability instead of hardcoding a global count.

Step 3: Define private connectivity

Use organization 6PN addresses and the documented .internal DNS name appropriate to the service, region, process group, or Machine. Document cross-organization exceptions separately.

Step 4: Protect the data

For Managed Postgres, capture plan, backup, failover, pooling, and support boundaries. For volumes, set snapshot retention and add independent backup or replication when recovery objectives exceed snapshot coverage.

Step 5: Test failure and restore

Prove application reconnect behavior, credential rotation, snapshot or managed restore procedure, regional loss response, and data reconciliation in a non-production target.

Step 6: Cut over with rollback

Freeze writes if required, migrate through an approved method, validate counts and application behavior, then retain the old data path until rollback expiry.

Authentication

Keep database credentials in Fly App secrets or an approved external secret manager, not in fly.toml or receipts. Use a scoped Fly.io identity for resource operations and a separate least-privilege database role for application access.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Persistence decision record distinguishing Managed Postgres from Machine volumes
• Region, network, backup, restore, and cutover plan
• Recovery exercise and post-cu…

Fly.io Cost and Capacity Review

Overview

Optimize from measured usage and the current pricing page, not copied monthly constants. Separate running and stopped Machine charges, persistent volume and snapshot charges, network and IP costs, and Managed Postgres resources that live outside application deletion.

Prerequisites

• Billing window, currency, organization, apps, and cost owner
• Machine state and resource history plus traffic, latency, and availability objectives
• Inventory of volumes, snapshots, IPs, transfer, builders, and Managed Postgres clusters

Instructions

Step 1: Reconcile the bill to resources

Map each charge class to current Machines, volumes, snapshots, addresses, transfer, builders, and databases. Include unattached and stopped-resource dependencies.

Step 2: Measure utilization and state

Compare CPU, memory, concurrency, request, latency, restart, and running-time signals with the selected VM size and Machine count.

Step 3: Evaluate autostop safely

For eligible service apps, model auto_stop_machines as stop or suspend with auto_start_machines and a deliberate minimum-running setting. Include cold-start and state-safety costs.

Step 4: Right-size and place

Change one dimension at a time: VM preset, additional memory, count, region placement, or database plan. Preserve capacity and recovery headroom.

Step 5: Remove orphaned cost with approval

Identify unattached volumes, obsolete snapshots beyond policy, unused addresses, old builders, and databases no longer referenced. Deletion requires owner confirmation and recovery evidence.

Step 6: Verify savings and service health

Compare the next billing interval and performance signals with the baseline; reverse changes that violate latency, availability, or recovery objectives.

Authentication

Prefer read-only organization tokens for inventory and billing analysis. Cost review does not authorize deleting Machines, volumes, addresses, snapshots, or Managed Postgres clusters; obtain resource-owner approval and preserve recovery evidence first.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Cost allocation by resource class and owner
• Ranked optimization plan with forecast, risk, and rollback
• Post-change cost and service-objective comparison

Return the target organization, app, environment…

Fly.io Redacted Support Evidence

Overview

Create a minimal evidence package that preserves diagnostic value without copying tokens, secret values, connection strings, full customer payloads, or unbounded logs. Every artifact needs source, timestamp, scope, redaction rule, and collection status.

Prerequisites

• Incident ID, owner, app, environment, regions, and time window
• Approved secure destination with retention and access controls
• Redaction rules for identifiers, logs, environment, network, and customer content

Instructions

Step 1: Define the evidence manifest

List requested artifacts, collection commands or APIs, owners, time bounds, redaction transformations, and expected hashes before gathering data.

Step 2: Capture release and configuration state

Record flyctl version, release or image identity, configuration hash, process groups, services, health definitions, and recent change references without including secrets.

Step 3: Capture Machine and health state

Collect aggregate status, Machine IDs or approved pseudonyms, regions, lifecycle and instance-version state, checks, restarts, and relevant exit metadata.

Step 4: Capture bounded logs and platform context

Use the smallest incident window, remove sensitive fields, note truncation, and record provider status separately from application evidence.

Step 5: Capture storage and network topology

Record volume IDs or pseudonyms, regions, attachments, snapshot state, public allocations, and private DNS expectations; exclude credentials and packet contents.

Step 6: Seal and review

Hash the sanitized artifacts, have the incident owner review the manifest, and publish only to the approved destination with expiry metadata.

Authentication

Collect with a read-only token where possible. Never place FLY_API_TOKEN, app secret values, database URLs, WireGuard private keys, registry credentials, or raw environment dumps in the bundle. Treat deploy access as sensitive because deployed code can read runtime secrets.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Evidence manifest with scope, source, timestamps, hashes, and redaction rules
• Sanitized release, configuration, Machine, health, log, volume, and network artifacts
• Collection failures, gaps, escalation question, retention owner, and deletion date

Return the target organization, app, e…

Fly.io Progressive Deployment Design

Overview

Select a provider-supported deployment strategy based on state, volume attachment, temporary capacity, and risk. Do not implement a custom traffic switch when Fly Launch already provides rolling, canary, blue-green, and immediate strategies with defined constraints.

Prerequisites

• Immutable candidate and previous image references
• Healthy baseline Machines and at least one meaningful service health check
• State and volume inventory plus temporary capacity and cost approval

Instructions

Step 1: Classify the workload

Identify stateless and stateful process groups, attached volumes, singleton regions, release commands, connection draining, and external dependencies.

Step 2: Choose a supported strategy

Use rolling for the general case. Canary creates one new Machine then rolls forward; blue-green creates replacement capacity in each region. Immediate is an emergency option with explicit downtime risk.

Step 3: Validate strategy constraints

Reject canary or blue-green for Machines with attached volumes. Ensure blue-green has health checks and enough quota and capacity; account for max-per-region behavior.

Step 4: Define acceptance gates

Set health, error, latency, saturation, business, and data-reconciliation thresholds with an observation window and named approver.

Step 5: Execute one release owner

Pin image and config, serialize the release, observe Machine replacement and health, and prevent unrelated scaling or secret changes during the window.

Step 6: Promote or reverse

Confirm every region and process group runs the intended image. If a gate fails, stop progression and restore the recorded prior release or configuration.

Authentication

Use a scoped deploy token whose app and expiry match the rollout. Keep observation separate with read-only access where practical. Never embed the token in fly.toml, image layers, release metadata, or health-check output.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Strategy decision with workload and volume constraints
• Capacity, health, observation, promotion, and rollback gates
• Per-region release and image reconciliation receipt

Return the target organization, app, environment, region set, Machine or database identifiers, source-contract fingerprint, evidence, unresolved risks, rollbac…

First Fly.io App Launch

Overview

Create the smallest useful first deployment while keeping generated configuration reviewable. Start with one environment, one explicit region, a health endpoint, and no persistent data; add global placement or storage only after the base release is observable and repeatable.

Prerequisites

• Application source, runtime port, startup command, and local health proof
• Organization, app naming policy, region choice, and cost owner
• Interactive identity for setup or a scoped token for approved automation

Instructions

Step 1: Inspect the application

Identify build method, Dockerfile status, process command, listening address, internal port, health route, environment, and secret inputs.

Step 2: Generate without deploying

Use the provider launch flow with deployment disabled so fly.toml and any Dockerfile changes can be reviewed before resource creation.

Step 3: Review the config

Confirm app name, primary region, build source, process groups, service ports, health checks, VM size, autostart or autostop, and shutdown behavior.

Step 4: Set secrets safely

Store runtime credentials through Fly App secrets. Leave non-sensitive configuration in reviewed environment fields and exclude local secret files from source control.

Step 5: Perform the approved first deploy

Bind the source revision and image, use rolling behavior, and observe build, Machine creation, and health rather than assuming the public URL proves complete success.

Step 6: Verify and document

Check the endpoint, health status, Machine image and region, logs, restart behavior, and billable resources. Record teardown or ownership before handoff.

Authentication

Interactive onboarding can use fly auth login. Automation should use the narrowest expiring token exposed as FLY_API_TOKEN or FLY_ACCESS_TOKEN. Never copy the personal login token into CI.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Reviewed Docker and fly.toml changes
• First-release plan with app, region, image, health, cost, and teardown boundaries
• Post-deploy receipt or clean rollback and resource-removal plan

Return the target organization, app, environment, region set, Machine or database identifiers, source-contract fingerprint, evidence, unresolved risks, rollback state, and final deci…

Fly.io CLI and Least-Privilege Token Setup

Overview

Separate interactive login from automation credentials. Current Fly.io guidance provides scoped deploy, organization, read-only, SSH, Machine-exec, and WireGuard token types; choose the smallest authority and lifespan that satisfies the workflow.

Prerequisites

• Named human or workload identity and business owner
• Target organization, app, commands, environment, and expiry requirement
• Approved installation source and secret storage location

Instructions

Step 1: Select the identity mode

Use browser login for a human workstation. For automation, map required actions to app deploy, organization deploy, read-only, SSH, Machine-exec, or WireGuard scope.

Step 2: Install from an official channel

Follow the provider installation method for the operating system, then record fly version and the provider-maintained release tag. Pin CI setup where reproducibility matters.

Step 3: Create a scoped expiring token

Prefer fly tokens create deploy for one app and fly tokens create readonly for observation. Use organization scope only for justified multi-app operations.

Step 4: Store and inject once

Place the token in the approved secret manager and expose it only as FLY_API_TOKEN or FLY_ACCESS_TOKEN to the intended process.

Step 5: Validate the smallest read

Confirm effective identity and scope through a non-mutating app or Machine read. Distinguish an empty scoped listing from a failed credential.

Step 6: Plan rotation and revocation

Record token ID, owner, scope, expiry, dependents, overlap procedure, emergency revocation, and post-rotation verification.

Authentication

Machines API requests send Authorization: Bearer <token> to the public https://api.machines.dev endpoint outside the private network. The hidden deprecated fly auth token output is an all-powerful short-lived login token and must not be used as the routine CI credential.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• flyctl installation and version receipt
• Token inventory containing only identifier, owner, scope, expiry, and secret reference
• Read-only scope proof plus rotation and revocation runbook

Return the target organization, app, environment, region set, Machine or database ide…

Fly.io Local-to-Remote Development Loop

Overview

Keep local unit and container checks separate from provider integration checks. Use synthetic fixtures by default, validate configuration before deployment, and treat fly proxy or WireGuard access to a remote service as production-adjacent even when initiated from a laptop.

Prerequisites

• Local build and test commands plus expected runtime port
• Development-only app or isolated process group and disposable test data
• Approved remote-service access path with data handling and teardown rules

Instructions

Step 1: Define the parity contract

Record runtime image, architecture, process command, port, environment names, health route, filesystem expectations, and provider-only dependencies.

Step 2: Build and test locally

Use deterministic dependencies and synthetic fixtures. Exercise startup, health, shutdown, migration, and failure paths without Fly.io credentials.

Step 3: Validate configuration separately

Review fly.toml against the app contract, including process groups, services, checks, signals, resources, and mounts. Keep secrets out of the file.

Step 4: Connect to remote services narrowly

If remote access is necessary, target a development service through an approved proxy or WireGuard path, bind locally, limit time, and never copy remote data into fixtures.

Step 5: Deploy to a disposable boundary

Use a dedicated development app or temporary Machine, immutable image, modest resource size, and clear expiry. Do not reuse the production app as a test target.

Step 6: Reconcile and tear down

Compare local and remote health, logs, architecture, and timing; capture differences, then stop or remove approved disposable resources.

Authentication

Use an interactive operator session or a development-app token with no production scope. Database credentials remain in the secret manager and are injected only for the bounded session. Close proxies and revoke temporary access after use.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Local/remote parity matrix and fixture inventory
• Reviewed development configuration and access plan
• Disposable integration receipt with results, differences, costs, and teardown state

Return the target organization, app, environment, region set, Machine or database identifiers, source-contract fingerprint, …

Fly.io Performance and Placement Tuning

Overview

Treat performance as an end-to-end path through Anycast routing, Fly Proxy, Machine placement, process concurrency, VM resources, autostart, and data locality. Optimize from percentiles and saturation evidence rather than assuming more regions or larger VMs always help.

Prerequisites

• Latency, throughput, error, availability, and cold-start objectives
• Per-region request, CPU, memory, concurrency, restart, and dependency latency evidence
• Current image, VM sizes, Machine counts, service concurrency, autostop, and database placement

Instructions

Step 1: Establish a representative baseline

Measure regional latency percentiles, errors, throughput, concurrency, CPU, memory, restarts, health transitions, cold starts, and dependency timing.

Step 2: Locate the bottleneck

Separate network distance, queueing, application work, CPU throttling, memory pressure, startup, connection setup, database latency, and capacity placement.

Step 3: Tune concurrency with capacity

Align service soft and hard limits with measured per-Machine capacity. Preserve headroom so Fly Proxy can route around unhealthy or saturated Machines.

Step 4: Tune lifecycle behavior

Compare always-running, stopped, and suspended behavior. Autostart can reduce idle cost but adds startup delay; set minimum running capacity from the service objective.

Step 5: Align placement with state

Add or remove regions based on user and dependency latency, available capacity, residency, and data topology. Keep region-bound volumes and database writes explicit.

Step 6: Change one variable and verify

Canary a single resource, count, concurrency, lifecycle, or placement adjustment; compare the same observation window and roll back on regression.

Authentication

Use read-only access for metrics, health, state, and placement analysis. Resource or scaling changes require a deploy-capable scoped identity and explicit approval. Performance evidence must not include tokens, request bodies, or sensitive labels.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Regional performance baseline and bottleneck hypothesis
• Ranked experiment plan with one variable, expected effect, risk, and rollback per test
• Before/after service-objective and resource reconciliation

Return the target organization, app, environment, region set…

Fly.io Production Readiness Gate

Overview

Make go-live a signed evidence boundary. The gate must cover the actual workload: stateless Machines, region-bound volumes, Managed Postgres, release commands, health semantics, autostart, networking, observability, billing, and tested rollback.

Prerequisites

• Business, technical, security, data, support, and rollback owners
• Immutable candidate image and reviewed configuration
• Load, failure, restore, deployment, and rollback evidence from a representative environment

Instructions

Step 1: Confirm ownership and scope

Record app, organization, environment, domains, regions, process groups, data stores, objectives, on-call route, and accepted risks.

Step 2: Review identity and secrets

Use scoped expiring tokens, protect production environments, inspect secret names and digests without values, and prove rotation and revocation.

Step 3: Review runtime and deployment

Validate image, architecture, ports, processes, signals, release command, health checks, strategy, unavailable capacity, and volume compatibility.

Step 4: Review resilience and data

Confirm Machine redundancy where required, regional placement, dependency failure behavior, Managed Postgres or volume backup boundaries, restore tests, and reconciliation.

Step 5: Review observability and cost

Prove logs, metrics, health, alerts, provider-status escalation, resource inventory, cost owner, and cleanup of obsolete billable resources.

Step 6: Exercise launch and rollback

Run the approved release path, observe acceptance gates, restore the prior image or config in a drill, and close only after actual fleet reconciliation.

Authentication

Production deploy uses the narrowest app or organization token in a protected environment. Read-only monitoring should not share deploy authority. Remember that anyone with deploy access can deploy code that reads runtime App secrets.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Signed production readiness decision with owners and exceptions
• Evidence index for identity, config, health, load, data recovery, observability, cost, and support
• Launch, rollback, and post-launch verification plan

Return the target organization, app, environment, region set, Machine or database identifiers, source-contract fingerprint, evidence, unresolved risks, rollback state, and final d…

Fly.io Machines API Rate Control

Overview

Replace the old invented per-organization table with Fly.io documented limits: Machine actions are constrained per action and per Machine or app identifier, reads have a separate ceiling, and app deletion has its own minute-level limit. Safe workers serialize conflicting operations and reconcile after ambiguous outcomes.

Prerequisites

• Exact API operations, app and Machine identifiers, request volume, and completion objective
• Durable operation IDs, desired-state fingerprints, retry counts, and acknowledgement state
• Current rate-limit contract and an error budget for delayed or failed operations

Instructions

Step 1: Classify each request

Separate create, update, start, stop, suspend, delete, wait, list, and get operations; identify whether the provider scopes the action to a Machine ID or app ID.

Step 2: Encode the documented ceilings

For Machine actions, budget one request per second per action and identifier with a short burst up to three. For get-Machine, budget five per second with a burst up to ten. Treat app deletion as at most 100 per minute.

Step 3: Serialize conflicts

Permit parallel work across independent identifiers only. Never overlap updates, starts, stops, leases, or deletion for the same Machine without a reconciled state boundary.

Step 4: Make intent replay-safe

Persist desired-state hash, operation, target, current Machine instance version, attempt, and acknowledgement before sending a mutation.

Step 5: Back off on throttling

Honor provider retry timing when present; otherwise use bounded exponential backoff with jitter. Apply a retry budget and circuit breaker rather than an unbounded loop.

Step 6: Reconcile ambiguous results

After timeout or transport failure, read current Machine and instance state before retrying. Mark complete only when observed state matches intent.

Authentication

Send the scoped token as a bearer header only to the documented public or private Machines API base URL. Do not widen token scope to solve throttling, and redact headers and Machine configuration secrets from retry telemetry.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Per-operation and per-identifier rate-control policy
• Durable queue state with attempts, acknowledgements, and reconciliation outcomes
• Throttle, retry, circuit-breaker, and manual-dispositi…

Fly.io Multi-Region Application Architecture

Overview

Compose Fly.io primitives without implying global state that the platform does not provide. Anycast and Fly Proxy route toward Machines, 6PN connects services inside an organization, Machines and volumes are regional, and Managed Postgres has its own supported-region and recovery contract.

Prerequisites

• Users, traffic, latency, availability, residency, recovery, and consistency requirements
• Service and process-group boundaries plus external dependencies
• Current provider regions, capacity, pricing, and Managed Postgres availability

Instructions

Step 1: Partition stateless and stateful work

Separate request-serving, workers, scheduled tasks, caches, durable databases, and region-local files. Name the system of record for every data class.

Step 2: Design routing and placement

Choose primary and secondary regions from observed users, dependency location, residency, and capacity. Document Fly Proxy, public IP, Flycast, or dynamic routing behavior.

Step 3: Design the Machine fleet

Set process groups, VM resources, Machine counts, health checks, deployment strategy, autostart, shutdown, and failure-domain expectations per region.

Step 4: Design data explicitly

Use Managed Postgres where its service contract fits. Use volumes only with software-managed replication and recovery or for disposable regional state; never present one volume as multi-region storage.

Step 5: Design private and external connectivity

Map 6PN names, organization boundaries, WireGuard peers, outbound requirements, certificates, DNS, and third-party trust zones.

Step 6: Prove resilience and operations

Exercise regional and dependency failure, deploy rollback, database restore, volume restore, secret rotation, monitoring, cost controls, and ownership handoff.

Authentication

Use separate scoped identities for deployment, observation, database access, and external integrations. Protect 6PN and WireGuard credentials, App secrets, and database URLs; private networking narrows reachability but does not replace application authentication.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Architecture diagram and authority map
• Region, routing, Machine, data, identity, observability, cost, and recovery decisions
• Failure-mode exercises with measured recovery and unresolved risks

Return…

Typed Fly.io Machines API Adapter

Overview

Fly.io publishes a REST API and OpenAPI reference; this workflow does not claim a provider-maintained JavaScript SDK. Generate or handwrite only a thin local adapter, keep provider fields visible, and centralize authentication, timeouts, limits, state waits, and redaction.

Prerequisites

• Chosen operations and current OpenAPI or endpoint documentation
• Runtime and HTTP client policy, schema generator decision, and error model
• Synthetic fixtures plus an authorized non-production app for contract verification

Instructions

Step 1: Freeze the contract

Record base URL, operation paths, request and response schemas, state enums, response codes, and retrieval fingerprint. Preserve unknown fields for forward compatibility.

Step 2: Define a thin transport

Centralize bearer auth, content type, request ID, timeout, retry eligibility, response parsing, and error redaction. Do not hide target app or Machine identifiers.

Step 3: Model desired and observed state

Represent Machine ID, instance version, configuration, lifecycle state, and target state separately. Require callers to reconcile before replay.

Step 4: Use the provider wait operation

After a mutation, wait for started, stopped, suspended, or destroyed with a bounded timeout and the required instance version where applicable.

Step 5: Apply rate and conflict policy

Serialize conflicting actions per identifier, use documented per-action limits, retry only eligible failures, and refresh after 409, 408, or ambiguous transport results.

Step 6: Test the adapter

Use fixture tests for every response and error class, then run an authorized read and one reversible non-production lifecycle operation with cleanup.

Authentication

Inject FLY_API_TOKEN from a secret manager and send it only as the bearer header to the documented Machines API host. Keep token scope outside the client configuration file and scrub headers and Machine config secrets from exceptions.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Contract fingerprint and supported-operation matrix
• Typed adapter with transport, state, rate, conflict, and redaction policies
• Fixture and non-production contract-test receipt with cleanup state

Return the target organization, app, environment, region…

Fly.io Identity, Secrets, and Network Baseline

Overview

Protect the control plane, deployed code, runtime secrets, images, network paths, and support evidence as one system. Scoped tokens reduce control-plane authority, but deploy access remains highly sensitive because new code can read secrets injected into Machines.

Prerequisites

• Human and workload identity inventory with owners and expiry policy
• Apps, process groups, images, domains, certificates, private peers, and data flows
• Incident, rotation, vulnerability, and access-review procedures

Instructions

Step 1: Separate identity classes

Distinguish interactive operators, app deploy automation, organization automation, read-only monitoring, SSH, Machine-exec, WireGuard, database, and external-service identities.

Step 2: Apply least privilege and expiry

Use app deploy tokens for one app, read-only organization tokens for observation, and short-lived command or SSH tokens for bounded access. Review and revoke by token ID.

Step 3: Protect runtime secrets

Store app values through Fly secrets, inspect only names and digests, stage changes when appropriate, and prevent deployed code, logs, crashes, and support bundles from exfiltrating values.

Step 4: Control code and images

Require reviewed immutable images, dependency and vulnerability gates, provenance, protected production environments, release approval, and rollback. Treat image registry and build credentials separately.

Step 5: Protect network paths

Map public services, certificates, 6PN, Flycast, WireGuard peers, outbound destinations, and application authentication. Private reachability does not imply trusted requests.

Step 6: Prove operations

Exercise token rotation, emergency revocation, secret update, deploy rollback, access review, log redaction, and incident escalation with timestamped receipts.

Authentication

Current guidance deprecates routine use of the all-powerful fly auth token output. Store scoped tokens in an approved secret manager and expose them only to the intended process. Never place tokens, secret values, WireGuard keys, or database URLs in source or evidence.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Identity and authority matrix with scope, owner, expiry, and review date
• Secret, image, network, logging, and evidence control assessment
• Prioritized remedi…

Fly.io Runtime and Platform Change Control

Overview

Replace the obsolete Apps v1-to-v2 tutorial with ongoing change control for the current Machines platform. Treat flyctl, fly.toml, images, architecture, Machine resources, regions, volumes, and Managed Postgres as separate migration surfaces with distinct rollback and data risks.

Prerequisites

• Current and target versions, configuration, image, architecture, resources, regions, and data topology
• Provider documentation and release notes retrieved at the change boundary
• Backup, export, restore test, acceptance gates, owner, approver, and rollback window

Instructions

Step 1: Classify the change surface

Separate CLI behavior, config schema, API contract, image or runtime, CPU architecture, VM size, region placement, volume state, and Managed Postgres changes.

Step 2: Freeze current state and contract

Record flyctl version, image, Machine instance versions, configuration hash, health, regions, volumes and snapshots, database plan and version, and dependency compatibility.

Step 3: Build a staging rehearsal

Use production-like configuration with synthetic or protected copied data. Exercise application startup, release command, health, connectivity, schema compatibility, and restore.

Step 4: Plan data movement explicitly

For a region-bound volume, create and verify the supported copy or snapshot path before Machine movement. For Managed Postgres, follow its current service features and support path; do not apply unmanaged Postgres commands.

Step 5: Canary the approved change

Change one process group, Machine, or non-production database boundary first. Observe health, errors, latency, data counts, and dependent services for the agreed window.

Step 6: Complete or reverse

Roll through remaining resources only after acceptance. Reconcile every image, instance version, region, volume attachment, and data count; retain rollback assets until expiry.

Authentication

Use a scoped deploy identity for app or Machine changes and a separate least-privilege database identity for data verification. Upgrade work does not authorize exposing snapshots, connection strings, or customer data to local or unapproved systems.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Current-to-target contract and compatibility matrix
• Staging, backup, restore, canary, rollout, reconcili…

Fly.io Machine and Health Change Feed

Overview

The general app platform does not document a customer-configurable webhook subscription and signing surface, so this route uses supported change detection through Machines reads and state waits, health status, deployment evidence, and documented log shipping. Extension-provider machine-event webhooks are a separate partner-specific contract and are not assumed here.

Prerequisites

• Selected signal types, apps, regions, process groups, and service objectives
• Durable last-seen state, deduplication key, overlap policy, and sink acknowledgement
• Read-only token and redaction rules for Machine, health, release, and log fields

Instructions

Step 1: Choose supported signals

Use Machine list or get for inventory, the Machine wait endpoint for a known transition, health-check status for routing readiness, release evidence for deployments, and documented log shipping for application events.

Step 2: Snapshot the starting boundary

Record active Machine ID and instance version, lifecycle state, image, checks, regions, and the observation-window checkpoint before polling.

Step 3: Normalize change identity

Create a deterministic key from app, Machine, instance version, signal type, observed transition, and source timestamp. Preserve provider request or release lineage.

Step 4: Deduplicate before publishing

Compare with last-seen state and the sink acknowledgement store. Publish only changed, validated, redacted records and advance the checkpoint after acknowledgement.

Step 5: Detect gaps and stale monitors

Alert on missed polls, wait timeouts, health age, log-export lag, state discontinuities, and unknown instance replacement. Re-read authoritative state before replay.

Step 6: Reconcile downstream action

Operational events may recommend but must not automatically restart, scale, or delete resources without a separately approved control policy.

Authentication

Use a read-only organization token for monitoring where possible and send bearer credentials only to documented Fly.io endpoints. Keep log-export credentials separate. Do not invent a shared webhook secret or signature header for general Fly Apps.

Tool Discipline

Use Read and Grep to inspect application configuration, deployment evidence, provider documentation, fixtures, logs, schemas, and existing tests before proposing a change. Use Write or Edit only for an approved plan, configuration, implementation, test, or redacted receipt. Do not create, deploy, scale, restart, stop, suspend, destroy, rotate, revoke, expose, or migrate live Fly.io resources without explicit operator approval.

Output

• Signal catalog and checkpoint policy
• Normalized redacted changes with deterministic IDs and source lineage
• Lag, gap, dedupl…