Fly.io CI Integration

Overview

Set up CI/CD for Fly.io edge deployments: run unit tests on every PR, deploy to staging on pull requests, and promote to production on merge to main. Fly.io uses Machines API for app management and deploy tokens for scoped CI authentication. CI pipelines build Docker images, deploy via flyctl, and run post-deploy health checks against the edge endpoints.

Prerequisites

• Protected CI environments with app-scoped tokens available only to trusted jobs.
• Synthetic test traffic, reviewed deployment policy, health thresholds, and a named rollback owner.

Instructions

1. Run unit, config, and container checks with no platform credentials on pull requests.
2. Restrict authenticated staging deployment to protected branches and redacted logs.
3. Use a canary health check and require explicit approval before production promotion.
4. Stop on unexpected region, image, configuration, or health result and retain the rollback receipt.

Output

Emit a CI receipt with commit SHA, image digest, checks run, protected-environment approval, aggregate health result, and rollback status. Exclude tokens, env values, and request data.

Examples

A pull request builds and tests the image without secrets. A protected merge job deploys a staging canary with synthetic traffic; an unexpected region or health failure blocks promotion and triggers a return to the prior release.

GitHub Actions Workflow

# .github/workflows/fly-ci.yml
name: Fly.io CI
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npm test -- --reporter=verbose

  deploy:
    if: github.ref == 'refs/heads/main'
    needs: unit-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: fly deploy --ha=false
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
      - name: Health check
        run: |
          sleep 10
          curl -sf https://my-app.fly.dev/health || exit 1

Mock-Based Unit Tests

// tests/fly-service.test.ts
import { describe, it, expect, vi } from 'vitest';
import { scaleApp } from '../src/fly-service';

vi.mock('../src/fly-client', () => ({
  FlyClient: vi.fn().mockImplementation(() => ({
    listMachines: vi.fn().mockResolvedValue([
      { id: 'mch_abc', state: 'started', region: 'iad', config: { size: 'shared-cpu-1x' } },
      { id: 'mch_def', state: 'started', region: 'lhr', config: { size: …

Fly.io Common Errors

Overview

Quick reference for the most common Fly.io deployment and runtime errors with solutions.

Prerequisites

• An authorized operator, opaque correlation ID, redacted logs/metrics, and a known service owner.
• A safe staging/read-only reproduction path; do not use destructive lifecycle actions to diagnose a production issue.

Instructions

1. Classify the failure as build, deploy, health, networking, storage, access, rate-limit, or platform availability.
2. Reproduce with the smallest safe probe, then inspect configuration, secret scope, release state, machine health, and region policy.
3. Apply the least disruptive reversible correction and verify recovery plus a safe failure path.
4. Escalate possible secret exposure, data loss, or cross-region integrity issues immediately.

Output

Return a diagnostic receipt with category, opaque correlation ID, reproduction result, corrective action, verification, owner, and follow-up. Exclude tokens, log bodies, configuration secrets, and user data.

Error Handling

• Do not solve permission problems with broader tokens; route them to the authorized owner.
• Quarantine failed deployment or storage operations for review and use bounded retry/backoff.
• Roll back before replaying stateful work after an integrity or health failure.

Examples

Use a synthetic health failure, inspect only redacted release/machine status, restore the prior configuration, and verify readiness. If the issue is a token mismatch, pause automation until the scoped credential is corrected and a read-only check succeeds.

Error Reference

Health Check Failed

Error: health checks for machine e784... failed

Causes: App not listening on correct port, slow startup, missing dependencies.

Fix:

# Check logs for startup errors
fly logs -a my-app

# Verify internal_port matches your app
grep internal_port fly.toml

# SSH in and test manually
fly ssh console -C "curl localhost:3000/health"

# Increase health check grace period

# fly.toml — give app more time to start
[http_service.checks]
  grace_period = "30s"
  interval = "15s"
  timeout = "5s"

Deployment Failed — Image Build

Error: failed to build: exit code 1

Fix:

# Test Docker build locally first
docker build -t test .
docker run -p 3000:3000 test

# Check Dockerfile — common issues:
# - Missing EXPOSE directive
# - Wrong WORKDIR
# - npm install before COPY (layer caching)

Machine Won't Start

Error: machine e784... failed to start

…

Fly.io Core Workflow A: Deploy & Scale

Overview

The primary Fly.io workflow: configure fly.toml, deploy apps, manage secrets, scale across regions, and control machine lifecycle.

Prerequisites

• A reviewed image and configuration, app-scoped deployment identity, health criteria, launch owner, and rollback operator.
• Staging synthetic traffic plus an approved region/data policy.

Output

Record a workflow receipt with release/image reference, app/region scope, health/canary result, scale decision, approver, and rollback outcome. Exclude secrets, config values, request bodies, and user data.

Examples

Deploy a fictitious staging app to one approved region, validate a generic health response under synthetic load, then simulate a failed check and restore the prior release. Expand to another region only after the owner accepts the canary receipt.

Instructions

Step 1: Configure fly.toml

# fly.toml — app configuration
app = "my-app"
primary_region = "iad"

[build]
  dockerfile = "Dockerfile"

[env]
  NODE_ENV = "production"
  PORT = "3000"

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = "stop"    # Stop idle machines
  auto_start_machines = true     # Start on request
  min_machines_running = 1       # Always keep 1 warm

[http_service.concurrency]
  type = "requests"
  hard_limit = 250
  soft_limit = 200

[[vm]]
  cpu_kind = "shared"
  cpus = 1
  memory = "512mb"

Step 2: Deploy and Manage Secrets

# Set secrets (encrypted, injected as env vars)
fly secrets set DATABASE_URL="postgres://..." API_KEY="sk_..."

# List secrets (values hidden)
fly secrets list

# Deploy
fly deploy

# Check deployment status
fly status
fly releases

Step 3: Scale Across Regions

# Add machines in new regions
fly scale count 2 --region iad    # 2 machines in Virginia
fly scale count 1 --region lhr    # 1 machine in London
fly scale count 1 --region nrt    # 1 machine in Tokyo

# Adjust VM size
fly scale vm shared-cpu-2x --memory 1024

# Check current scale
fly scale show

Step 4: Manage App Lifecycle

# Restart all machines
fly apps restart

# Suspend an app (stop billing)
fly apps suspend my-app

# Resume
fly apps resume my-app

# Destroy (irreversible)
fly apps destroy my-app --yes

fly.toml Key Settings

Fly.io Core Workflow B: Postgres, Volumes & Networking

Overview

Set up Fly Postgres, persistent Fly Volumes, and private networking between apps. Fly Postgres runs as a regular Fly app with automated replication. Volumes provide persistent NVMe storage attached to specific machines.

Prerequisites

• A data owner, documented locality/retention/backup/recovery requirements, and a staging environment with synthetic data.
• Scoped database and deployment identities, private-network policy, and a tested restore/rollback procedure.

Output

Maintain a storage/network receipt with resource references, region, encryption/access controls, backup/restore verification, owner, and recovery result. Do not include connection strings, records, or keys.

Examples

Create a disposable staging database and volume with fictional data, confirm an unauthorized app cannot reach it, and test a backup/restore without copying credentials to logs. Tear down only the validated disposable resources through the approved process.

Instructions

Step 1: Create Fly Postgres

# Create a Postgres cluster
fly postgres create --name my-db --region iad --vm-size shared-cpu-1x --volume-size 10

# Attach to your app (sets DATABASE_URL secret automatically)
fly postgres attach my-db -a my-app

# Connect directly
fly postgres connect -a my-db
# psql> SELECT version();

# Proxy to local machine for dev tools
fly proxy 5432 -a my-db
# Now connect with the secret-bearing URL supplied by the local proxy
psql "$DATABASE_URL"

Step 2: Create Persistent Volumes

# Create a volume (same region as your machine)
fly volumes create data --size 10 --region iad -a my-app

# List volumes
fly volumes list -a my-app

# Mount in fly.toml

# fly.toml
[mounts]
  source = "data"
  destination = "/data"

# Deploy to pick up mount
fly deploy

# Verify mount inside machine
fly ssh console -C "df -h /data"

Step 3: Private Networking (6PN)

# Apps in the same org can reach each other via .internal DNS
# my-app can reach my-db at: my-db.internal:5432

# Internal DNS format: <app-name>.internal
# Machine-specific: <machine-id>.vm.<app-name>.internal

# Example: connect from app code
DATABASE_URL=${FLY_DATABASE_URL}

// Access internal services (no public internet)
const dbUrl = process.env.DATABASE_URL;
const apiUrl = `http://my-api.internal:3000/health`;  // Internal HTTP

Step 4: Postgres Backups and Failover

# List backups
fly postgres barman list-backups -a my-db

# Create manual backup
fly postgres barman backup -a my-db

# …

Fly.io Cost Tuning

Overview

Fly.io charges per-second for running machines plus storage. Key levers: auto-stop idle machines, suspend instead of stop, right-size VMs, and clean up unused volumes.

Prerequisites

• Current contract/billing data reviewed by the account owner, aggregate resource usage, and a named cost owner.
• Availability, latency, durability, and retention requirements that constrain autoscaling and volume decisions.
• Synthetic staging load and a rollback plan for each cost-control change.

Output

Produce a cost-control receipt with measurement window, aggregate usage, approved setting change, expected effect, owner, verification date, and rollback result. Never include tokens, customer data, or internal billing details.

Error Handling

• Do not remove volumes or capacity without confirmed retention, backup, and recovery requirements.
• Revert a cost change that breaches availability, latency, or data-durability thresholds.
• Quarantine unexpected resource changes for review rather than applying bulk cleanup.

Examples

Apply a suspend policy to a fictional staging app, compare aggregate idle cost and cold-start health, and roll back when a synthetic health check breaches the agreed latency threshold. Do not delete any volume as part of an exploratory test.

Pricing Quick Reference

Instructions

Strategy 1: Auto-Stop Idle Machines

# fly.toml — stop machines when no traffic
[http_service]
  auto_stop_machines = "stop"     # Full stop (cheapest, ~5s cold start)
  auto_start_machines = true
  min_machines_running = 0        # Allow all machines to stop
  # Use min_machines_running = 1 only for production apps

Strategy 2: Suspend for Faster Resume

# Suspend keeps memory state — resumes in ~100ms but costs ~$0.50/month
[http_service]
  auto_stop_machines = "suspend"

Strategy 3: Audit and Clean Up

# List all apps and their machine counts
fly apps list

# Find idle/stopped machines
fly machine list -a my-app --json | jq '.[] | select(.state != "started") | {id, state, region}'

# Destroy unused apps
fly apps destroy old-app --yes

# List and…

Fly.io Debug Bundle

Overview

Collect machine state, app health, volume status, deploy history, network connectivity, and platform diagnostics into a single archive for Fly.io support tickets. This bundle captures everything needed to troubleshoot stuck deployments, machine boot failures, volume corruption, and edge networking problems.

Prerequisites

• An incident owner, secure evidence location, retention deadline, and redaction rules for logs, configuration, tokens, and user data.
• An opaque correlation ID and a safe health/read-only probe before collecting broad runtime evidence.

Instructions

1. Capture version, release, aggregate health, opaque machine identifiers, and relevant configuration references.
2. Review all logs and generated files for tokens, credentials, request bodies, and personal data before archiving.
3. Encrypt and restrict the resulting evidence to incident responders, then retire it according to the retention decision.

Output

Create a redacted bundle index with correlation ID, artifact list, access owner, retention date, reproduction result, and next action. Sensitive originals belong only in the approved incident store.

Error Handling

• Stop collection and rotate credentials if a secret or sensitive data is found in the bundle.
• Record missing diagnostics rather than expanding access or collection without approval.
• Escalate possible exposure before continuing normal troubleshooting.

Examples

For a synthetic machine boot failure, retain release ID, opaque machine ID, and aggregate health result. Verify the archive contains no token or request body, grant access only to the incident owner, and delete it when its retention period ends.

Debug Collection Script

#!/bin/bash
set -euo pipefail
APP="${1:?Usage: fly-debug.sh <app-name>}"
BUNDLE="debug-flyio-${APP}-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BUNDLE"

# Environment check
echo "=== Fly.io Debug Bundle: $APP ===" | tee "$BUNDLE/summary.txt"
echo "Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$BUNDLE/summary.txt"
echo "FLY_API_TOKEN: ${FLY_API_TOKEN:+[SET]}" >> "$BUNDLE/summary.txt"
echo "flyctl: $(fly version 2>/dev/null || echo 'not found')" >> "$BUNDLE/summary.txt"

# API connectivity
HTTP=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  https://api.machines.dev/v1/apps 2>/dev/null || echo "000")
echo "Machines API: HTTP $HTTP" >> "$BUNDLE/summary.txt"

# App status and machine state
fly status -a "$APP" > "$BUNDLE/status.txt" 2>&1 || true
fly machine list -a "$APP" --json…

Fly.io Deploy Integration

Overview

Deploy edge applications on Fly.io with Docker containers and the fly.toml configuration file. This skill covers building production images optimized for Fly's micro-VM architecture, configuring fly.toml for services, health checks, and multi-region placement, verifying API connectivity from edge locations, and executing rolling updates with automatic rollback. Fly.io deploys as Firecracker micro-VMs, so containers start in under a second and scale to zero when idle.

Prerequisites

• A deployment owner, scoped CI token, reviewed image, environment config, health criteria, and rollback operator.
• Staging with synthetic traffic and explicit data/region requirements before any multi-region promotion.

Instructions

1. Build reproducibly and run as a non-root user; inject secrets only through the platform.
2. Deploy a small canary, observe redacted health and saturation metrics, and verify graceful failure behavior.
3. Promote region by region only when the canary succeeds; stop and roll back on health, configuration, or access failures.
4. Retain the release reference and recovery evidence for the approved change window.

Output

Record image/release identifier, environment, regions, health criteria, canary metrics, approval, and rollback result. Exclude tokens, env values, request bodies, and user data.

Examples

Deploy a staging image to one region with synthetic traffic, simulate a failed health check, and confirm the release stops before traffic expands. Restore the previous release and verify the readiness endpoint remains generic and redacted.

Docker Configuration

FROM node:20-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY tsconfig.json ./
COPY src/ ./src/
RUN npm run build

FROM node:20-slim
RUN addgroup --system app && adduser --system --ingroup app app
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package*.json ./
USER app
EXPOSE 8080
CMD ["node", "dist/index.js"]

Fly.io Configuration

# fly.toml
app = "my-integration"
primary_region = "iad"

[build]
  dockerfile = "Dockerfile"

[env]
  LOG_LEVEL = "info"
  PORT = "8080"

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true

[[http_service.checks]]
  interval = "30s"
  timeout = "5s"
  grace_period = "10s"
  method = "GET"
  path = "/health"

Environment Variables

export FLY_API_TOKEN="fo1_xxxxxxxxxxxx"
fly secrets set FLYIO_APP_NAME="my-integration"
fly …

Fly.io Hello World

Overview

Deploy a minimal app to Fly.io using fly launch. Fly.io runs Docker containers on Firecracker microVMs across 30+ regions worldwide. Two paths: flyctl CLI (simple) or Machines API (programmatic).

Prerequisites

• A disposable staging app name, app-scoped token from the secret manager, Docker, and a known teardown owner.

Examples

Deploy the example to a disposable staging app using a scoped token, send only a synthetic health request, verify the response contains no secrets, and destroy or stop the test app through the approved cleanup path after validating rollback behavior.

Instructions

Step 1: Launch with flyctl

# Create a new directory with a Dockerfile
mkdir fly-hello && cd fly-hello

cat > Dockerfile << 'EOF'
FROM node:20-alpine
WORKDIR /app
COPY server.js .
EXPOSE 3000
CMD ["node", "server.js"]
EOF

cat > server.js << 'EOF'
const http = require('http');
const server = http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify({
    message: 'Hello from Fly.io!',
    region: process.env.FLY_REGION,
    app: process.env.FLY_APP_NAME,
  }));
});
server.listen(3000, () => console.log('Listening on :3000'));
EOF

# Launch — creates app, generates fly.toml, deploys
fly launch --name hello-fly --region iad --now

Step 2: Verify Deployment

# Check status
fly status

# Open in browser
fly open

# View logs
fly logs

# Test with cURL
curl https://hello-fly.fly.dev/
# {"message":"Hello from Fly.io!","region":"iad","app":"hello-fly"}

Step 3: Deploy via Machines API

const FLY_API = 'https://api.machines.dev';
const headers = {
  'Authorization': `Bearer ${process.env.FLY_API_TOKEN}`,
  'Content-Type': 'application/json',
};

// Create an app
const app = await fetch(`${FLY_API}/v1/apps`, {
  method: 'POST',
  headers,
  body: JSON.stringify({
    app_name: 'hello-api',
    org_slug: 'personal',
  }),
}).then(r => r.json());

// Create a machine in the app
const machine = await fetch(`${FLY_API}/v1/apps/hello-api/machines`, {
  method: 'POST',
  headers,
  body: JSON.stringify({
    region: 'iad',
    config: {
      image: 'nginx:alpine',
      services: [{
        ports: [{ port: 443, handlers: ['tls', 'http'] }],
        protocol: 'tcp',
        internal_port: 80,
      }],
      guest: { cpu_kind: 'shared', cpus: 1, memory_mb: 256 },
    },
  }),
}).then(r => r.json());

console.log(`Machine ${machine.id} created in ${machin…

Fly.io Install & Auth

Overview

Install flyctl CLI and configure authentication for Fly.io edge compute platform. Two auth methods: interactive login (opens browser) and API tokens (CI/CD and Machines API). The Machines API base URL is https://api.machines.dev.

Output

Record the authentication method, secret-manager reference, minimum scope, validation time, owner, and rotation/revocation procedure. Never put a token, header, machine response, or organization data in the receipt.

Examples

Use an app-scoped staging deploy token injected by the CI environment, verify a read-only status check, then revoke it and confirm it can no longer access the app. Keep only redacted evidence of the scope and result.

Prerequisites

• Fly.io account at fly.io
• macOS, Linux, or WSL2

Instructions

Step 1: Install flyctl

# macOS / Linux
curl -L https://fly.io/install.sh | sh

# Or via Homebrew
brew install flyctl

# Verify
fly version

Step 2: Authenticate

# Interactive login (opens browser)
fly auth login

# Or with token (CI/CD)
fly auth token  # Get current token
export FLY_API_TOKEN="fo1_your_token_here"

# Verify auth
fly auth whoami

Step 3: Create API Token for Machines API

# Create deploy token (scoped to an app)
fly tokens create deploy -a my-app

# Create org-level token
fly tokens create org

# Use with Machines API
curl -s -H "Authorization: Bearer $FLY_API_TOKEN" \
  https://api.machines.dev/v1/apps | jq '.[].name'

Step 4: Verify Machines API Access

const FLY_API = 'https://api.machines.dev';

async function verifyFlyAccess() {
  const res = await fetch(`${FLY_API}/v1/apps`, {
    headers: { 'Authorization': `Bearer ${process.env.FLY_API_TOKEN}` },
  });
  const apps = await res.json();
  console.log(`Connected. Found ${apps.length} apps.`);
  apps.forEach((app: any) => console.log(`  ${app.name} (${app.organization.slug})`));
}

Token Types

Error Handling

Fly.io Local Dev Loop

Overview

Fast local development workflow for Fly.io apps: build and test Docker containers locally, proxy remote Fly services (Postgres, Redis) to localhost, and use fly deploy for integration testing.

Prerequisites

• Local Docker tooling, a non-production environment, synthetic fixtures, and a directory excluded from source control for runtime secrets.
• Approved read-only/sandbox access for any remote proxy; development must not proxy production data by default.

Output

Create a local validation receipt with image/config version, fixture set, health result, and redacted failures. Never commit connection strings, proxy credentials, database contents, or user data.

Error Handling

• Stop a proxy or local test that targets production or exposes credentials; notify the environment owner if access was attempted.
• Treat configuration or schema mismatches as review items and clean up temporary containers/fixtures through approved processes.
• Revoke test credentials if they were disclosed in a terminal capture or artifact.

Examples

Run a container locally with a fictional database URL and verify the health endpoint. Use an isolated staging proxy only for a read-only synthetic fixture, then stop the proxy and confirm no remote credential or data was written to the repository.

Instructions

Step 1: Local Docker Testing

# Build and run locally — same Dockerfile used by Fly
docker build -t my-app .
docker run -p 3000:3000 \
  -e NODE_ENV=development \
  -e DATABASE_URL="postgres://localhost:5432/dev" \
  my-app

# Test
curl http://localhost:3000/health

Step 2: Proxy Remote Fly Services

# Proxy Fly Postgres to localhost:5432
fly proxy 5432 -a my-db &

# Now use local tools against remote Fly Postgres
psql "$DATABASE_URL"
npx prisma studio  # Prisma GUI works against proxied DB

# Proxy Redis
fly proxy 6379 -a my-redis &
redis-cli -h localhost -p 6379

Step 3: Development fly.toml

# fly.dev.toml — dev overrides (not committed)
app = "my-app-dev"
primary_region = "iad"

[env]
  NODE_ENV = "development"
  LOG_LEVEL = "debug"

[http_service]
  internal_port = 3000
  auto_stop_machines = "off"  # Keep running for debugging
  min_machines_running = 1

[[vm]]
  cpu_kind = "shared"
  cpus = 1
  memory = "256mb"  # Smaller for dev

Step 4: Fast Deploy Cycle

# Deploy to dev app
fly deploy -a my-app-dev --config fly.dev.toml

# Watch logs while testing
fly logs -a my-app-dev --no-tail &

# SSH in for debugging
fly ssh console -a my-app-dev

# Quick restart after config change
fly apps restart m…

Fly.io Performance Tuning

Overview

Optimize Fly.io performance: eliminate cold starts, right-size VMs, leverage multi-region for low latency, and tune concurrency settings.

Prerequisites

• A redacted baseline for latency, error rate, saturation, cost, and region-level health.
• A staging application, synthetic load, named change owner, and a tested rollback mechanism.

Output

Publish a tuning receipt with baseline/post-change aggregate metrics, VM/concurrency settings, regions affected, canary result, owner, and rollback outcome. Exclude runtime secrets, request bodies, and user data.

Error Handling

• Stop rollout on health, latency, saturation, or cost threshold breaches and revert the canary.
• Reduce concurrency or capacity changes before retrying a failed region promotion.
• Keep diagnostics redacted and route deployment incidents to the on-call owner.

Examples

Run a synthetic load test against a staging region, adjust one VM setting, and compare aggregate p95 latency and error rate. Simulate a failed health check and confirm the release rolls back before traffic expands.

Instructions

Step 1: Eliminate Cold Starts

# fly.toml — suspend instead of stop for faster resume (~100ms vs ~5s)
[http_service]
  auto_stop_machines = "suspend"   # Suspend to RAM, not full stop
  auto_start_machines = true
  min_machines_running = 1          # Always-warm in primary region

# For latency-critical: keep machines running in all regions
# min_machines_running applies globally

Step 2: Right-Size VMs

# Check current allocation
fly scale show -a my-app

# Start small, scale up based on metrics
fly scale vm shared-cpu-1x --memory 256    # Start here
fly scale vm shared-cpu-1x --memory 512    # If memory-constrained
fly scale vm shared-cpu-2x --memory 1024   # If CPU-bound
fly scale vm performance-2x --memory 4096  # For compute-heavy workloads

Step 3: Multi-Region Latency Optimization

# Deploy close to your users
fly scale count 1 --region iad    # US East
fly scale count 1 --region lhr    # Europe
fly scale count 1 --region nrt    # Asia Pacific

# Fly automatically routes to nearest region via Anycast
# Verify: curl with timing
curl -w "DNS: %{time_namelookup}s…

Fly.io Production Checklist

Overview

Fly.io runs applications on edge infrastructure across 30+ regions with Machines, Volumes, and managed Postgres. A production deployment requires multi-region redundancy, proper secret management, health checks, and rollback procedures. Misconfigured auto-scaling means cold starts; missing volume backups mean data loss. This checklist ensures your Fly.io app is production-hardened.

Prerequisites

• A launch owner, approver, on-call contact, recovery owner, and completed staging evidence using synthetic traffic.
• Documented data locality, backup/restore, access, retention, and escalation requirements.

Instructions

1. Complete every applicable checklist item with evidence or an explicit owner decision.
2. Confirm secrets, identities, health checks, backup/restore, monitoring, region policy, and rollback before launch.
3. Run a canary, observe aggregate health/cost/error signals, and stop promotion when any defined threshold is breached.
4. Record approval, exceptions, and recovery verification in the release receipt.

Output

Produce a go-live receipt with controls completed, evidence references, canary metrics, regions, approver, rollback owner, exceptions, and follow-up date. Do not include secrets or user data.

Examples

Deploy a synthetic workload to one staging region, revoke a test deployment token, and simulate a health failure. Promote only after the rollback succeeds and the designated approver records the canary evidence.

Authentication & Secrets

• [ ] FLY_API_TOKEN stored in CI secrets (never in fly.toml or source)
• [ ] All app secrets set via fly secrets (not [env] block)
• [ ] Deploy tokens scoped per app (not org-wide personal tokens)
• [ ] Key rotation scheduled (quarterly, or after team changes)
• [ ] No hardcoded secrets in Dockerfile or codebase

API Integration

• [ ] Production base URL: app deployed to https://<app>.fly.dev
• [ ] force_https = true in fly.toml http_service
• [ ] Custom domain with TLS certificate active and auto-renewing
• [ ] min_machines_running = 1 to avoid cold starts
• [ ] Machines deployed in 2+ regions for redundancy
• [ ] Concurrency limits tuned (soft_limit/hard_limit per workload)
• [ ] Volumes backed up if using persistent storage

Error Handling & Resilience

• [ ] Health check endpoint configured with appropriate grace period
• [ ] Graceful shutdown handles SIGTERM within 10s window
• [ ] Auto-stop/auto-start configured for cost optimization
• [ ] Postgres standby replica provisioned for database apps
• [ ] Rollback procedure tested: …

Fly.io Rate Limits

Overview

The Fly.io Machines API rate-limits per organization, with write operations (create, delete, update) throttled much more aggressively than reads. Deploying fleets of edge machines across multiple regions can easily trigger 429s, especially during rolling deployments or auto-scaling events. The API returns a Retry-After header on rate-limited responses, and organizations running 50+ machines should implement client-side token bucket limiting to avoid cascading failures during high-churn operations.

Prerequisites

• Current platform limits confirmed for the organization plus approved concurrency, retry bounds, and a queue owner.
• Redacted telemetry for request category, queue age, throttle count, and machine lifecycle—not tokens or payloads.
• A staging fleet/synthetic workload for testing pauses, retries, and cancellation.

Instructions

1. Honor explicit throttling guidance and use bounded concurrency with jittered backoff.
2. Attach idempotency/operation tracking to lifecycle changes so a retry cannot duplicate a create, stop, or delete action.
3. Queue exhausted operations for reviewed handling, alert on backlog growth, and reduce demand before resuming.

Output

Publish a rate-control receipt with policy version, concurrency, retry bounds, throttle count, queue outcome, owner, and manual disposition. Do not expose app names if they are sensitive, tokens, or request bodies.

Examples

Apply a small synthetic scale change, simulate a 429, and confirm the worker waits and then performs the operation once. A repeated failure must enter the review queue rather than trigger a fleet-wide replay.

Rate Limit Reference

Rate Limiter Implementation

class FlyRateLimiter {
  private tokens: number;
  private lastRefill: number;
  private readonly max: number;
  private readonly refillRate: number;
  private queue: Array<{ resolve: () => void }> = [];

  constructor(maxPerMinute: number) {
    this.max = maxPerMinute;
    this.tokens = maxPerMinute;
    this.lastRefill = Date.now();
    this.refillRate = maxPerMinute / 60_000;
  }

  async acquire(): Promise<void> {
    this.refill();
    if (this.tokens >= 1) { this.tokens -= 1; retur…

Fly.io Reference Architecture

Overview

Production architecture for Fly.io: multi-region web tier, Postgres with read replicas, Redis for caching, background workers, and private networking.

Prerequisites

• A documented data-flow inventory, trust boundaries, ownership, region/retention choices, and disaster-recovery objectives.
• Separate scoped identities for deployment, runtime, database, worker, and observability systems.

Instructions

1. Place public ingress, private services, storage, workers, and observability behind explicit network and identity boundaries.
2. Define data locality, replication, backup, access, and recovery behavior before creating additional regions or consumers.
3. Use staged deployment, health checks, redacted telemetry, and an independently tested rollback per service.
4. Validate architecture changes with synthetic traffic and ensure a failure in one region cannot leak secrets or corrupt cross-region state.

Output

Maintain an architecture decision record with components, trust boundaries, data locations, identities, health/rollback controls, owners, and recovery evidence. Do not include secrets or customer data.

Error Handling

• Isolate an unhealthy region or consumer and preserve a safe primary path while recovery proceeds.
• Quarantine unexpected cross-region writes or permission failures for review.
• Restore the previous routing/configuration before replaying queued work.

Examples

Deploy a fictional workload to a staging primary and replica region, deny the worker access to public ingress secrets, and simulate a regional health failure. Verify traffic stays on the healthy route and rollback does not replay writes.

Architecture

           ┌─────────── Fly.io Anycast DNS ──────────┐
           │                                          │
    ┌──────▼──────┐  ┌──────────────┐  ┌─────────────▼───┐
    │  Web (iad)  │  │  Web (lhr)   │  │   Web (nrt)     │
    │  shared-1x  │  │  shared-1x   │  │   shared-1x     │
    └──────┬──────┘  └──────┬───────┘  └────────┬────────┘
           │                │                    │
    ───────┴────────────────┴────────────────────┴─── .internal DNS
           │                │                    │
    ┌──────▼──────┐  ┌──────▼───────┐  ┌────────▼────────┐
    │ Postgres    │  │ Postgres     │  │   Redis          │
    │ Primary     │  │ Replica      │  │   (upstash.io)   │
    │ (iad)       │  │ (lhr)        │  │                  │
    └─────────────┘  └──────────────┘  └──────────────────┘
           │
    ┌──────▼──────┐
    │  Worker     │
    │  (iad)      │
    │  shared-1x  │
    └─────────────┘

Setup Commands

# 1. Web app — multi-region
fly launch --name my-web --region iad
fly scale count 1 --region lhr
fly scale c…

Fly.io SDK Patterns

Overview

Production-ready patterns for the Fly.io Machines REST API at https://api.machines.dev. Fly.io exposes both GraphQL (organization queries) and REST (machine lifecycle) APIs. The Machines REST API is the primary integration surface for creating, starting, stopping, and destroying VMs across 30+ global regions. A structured client ensures consistent auth, typed machine states, and reliable wait-for-state polling.

Prerequisites

• An app-scoped token held in a secret manager, approved app/region policy, and synthetic staging app.
• Idempotent lifecycle design, rate controls, redacted diagnostics, and a rollback owner.

Instructions

1. Validate app, region, operation, and request schema before a lifecycle call.
2. Track opaque operation IDs, use bounded retries, and protect create/stop/delete from duplicate execution.
3. Route unexpected state, permission, or target results to reviewed handling and preserve the prior configuration.

Output

Produce a client-validation receipt with API/contract version, fixture result, operation ID, idempotency outcome, owner, and redacted failure reference. Never log tokens, machine config secrets, or user data.

Examples

Create a disposable staging machine from a synthetic configuration, retry the request under the same operation ID, and verify only one machine results. Simulate an invalid region and ensure the client rejects it before a provider call.

Singleton Client

const FLY_API = 'https://api.machines.dev';
let _client: FlyClient | null = null;
export function getClient(appName: string): FlyClient {
  if (!_client) {
    const token = process.env.FLY_API_TOKEN;
    if (!token) throw new Error('FLY_API_TOKEN must be set');
    _client = new FlyClient(appName, token);
  }
  return _client;
}
class FlyClient {
  private h: Record<string, string>;
  constructor(private app: string, token: string) {
    this.h = { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' };
  }
  async listMachines(): Promise<FlyMachine[]> {
    const r = await fetch(`${FLY_API}/v1/apps/${this.app}/machines`, { headers: this.h });
    if (!r.ok) throw new FlyError(r.status, await r.text()); return r.json();
  }
  async createMachine(config: MachineConfig, region: string): Promise<FlyMachine> {
    const r = await fetch(`${FLY_API}/v1/apps/${this.app}/machines`, {
      method: 'POST', headers: this.h, body: JSON.stringify({ region, config }) });
    if (!r.ok) throw new FlyError(r.status, await r.text()); return r.json();
  }
  async waitForState(id: string, state: string, timeout = 30): Promise<void> {
    const r = await fetch(`${FLY_API}/v1/apps/${this.app}/machines/${id}/wait?state=${state}&timeout=${timeout}`,
      { he…

Fly.io Security Basics

Overview

Fly.io deploys applications to edge locations worldwide using Firecracker microVMs. Security concerns center on deploy token scoping (org-wide vs per-app), secrets management (encrypted at rest, injected as env vars), private networking via WireGuard mesh (6PN), and TLS certificate management. A leaked deploy token can push arbitrary code to production machines across all regions.

Prerequisites

• A named security owner, app/organization access inventory, secret-manager integration, and recurring access-review cadence.
• Approved network, region, TLS, logging, and incident/revocation policies plus synthetic staging fixtures.

Instructions

1. Use app-scoped deploy tokens and separate identities per environment; never place tokens in code, tickets, terminal captures, or debug bundles.
2. Restrict private services and secrets to the minimum set of machines and roles, with explicit network boundaries and access review.
3. Verify incoming signed events before processing, log opaque IDs only, and make downstream actions idempotent.
4. Monitor for unauthorized deployment, secret, region, or certificate changes and rotate/revoke credentials immediately after suspected exposure.

Output

Maintain a security receipt with identity scope, secret reference, policy version, access-review date, verification/rotation result, owner, and redacted incident state. Never include tokens, configuration secrets, or user data.

Examples

Create a disposable staging app using a scoped token, attempt an unauthorized app operation, and verify it is denied. Rotate the token, confirm the old credential fails, and retain only the redacted policy and control outcome.

API Key Management

function validateFlyToken(): void {
  const token = process.env.FLY_API_TOKEN;
  if (!token) {
    throw new Error("Missing FLY_API_TOKEN — use `fly tokens create deploy -a <app>`");
  }
  // Never log tokens; log only token type for debugging
  const isDeployToken = token.startsWith("FlyV1");
  console.log("Fly.io token loaded, type:", isDeployToken ? "deploy" : "personal");
}

Webhook Signature Verification

import crypto from "crypto";
import { Request, Response, NextFunction } from "express";

function verifyFlyWebhook(req: Request, res: Response, next: NextFunction): void {
  const signature = req.headers["x-fly-signature"] as string;
  const secret = process.env.FLY_WEBHOOK_SECRET!;
  const expected = crypto.createHmac("sha256", secret).update(req.body).digest("hex");
  if (!signature || !crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    res.status(401).send("Invalid signature");
…

Fly.io Upgrade & Migration

Overview

Guide for Fly.io platform migrations: Apps v1 (Nomad) to v2 (Machines), flyctl CLI upgrades, Postgres major version upgrades, and region migrations.

Prerequisites

• Current platform documentation and an inventory of applications, machines, regions, volumes, databases, identities, and dependent consumers.
• A staging environment, synthetic traffic/data, tested backup/restore, rollback owner, and explicit acceptance/reconciliation criteria.

Output

Produce a migration receipt with versions reviewed, affected resources, staging/canary results, backup/restore evidence, reconciliation outcome, approver, and rollback state. Keep tokens, connection strings, and user data out of the receipt.

Error Handling

• Stop promotion on health, schema, region, permission, or reconciliation mismatches and restore the prior configuration.
• Quarantine failed migrations by opaque resource ID; do not bulk replay stateful workloads to diagnose failures.
• Escalate potential data loss or credential exposure and retain only approved incident evidence.

Examples

Migrate a disposable staging app using synthetic traffic, exercise a backup/restore of fictional data, and simulate a failed health check. Verify rollback returns routing and data access to the known-good release before considering a production canary.

Instructions

Apps v1 to v2 Migration

# Check current platform version
fly status -a my-app  # Look for "Platform: machines" vs "nomad"

# Migrate to Apps v2 (Machines)
fly migrate-to-v2 -a my-app

# Verify
fly status -a my-app
fly machine list -a my-app

flyctl CLI Upgrade

# Check current version
fly version

# Upgrade
fly version update

# Or reinstall
curl -L https://fly.io/install.sh | sh

Postgres Major Version Upgrade

# Check current version
fly postgres connect -a my-db -c "SELECT version();"

# Create new cluster with target version
fly postgres create --name my-db-v16 --region iad --image-ref flyio/postgres-flex:16

# Migrate data
fly postgres import pg_dump_url -a my-db-v16

# Update app to point to new cluster
fly postgres detach my-db -a my-app
fly postgres attach my-db-v16 -a my-app
fly deploy -a my-app  # Picks up new DATABASE_URL

Region Migration

# Add machines in new region
fly scale count 1 --region fra -a my-app

# Verify new region is healthy
fly status -a my-app

# Remove machines from old region
fly scale count 0 --region iad -a my-app

# For volumes: create new volume, migrate data, destroy old
fly volumes create data --size 10 --region fra -a my-app

Migration Checklist

• [ ] Current state documented (…

Fly.io Events & Monitoring

Overview

Fly.io does not have traditional webhooks. Instead, monitor machine state changes via the Machines API, process structured logs via fly logs, and use health check endpoints for automated responses.

Prerequisites

• A scoped monitoring identity, approved app/region policy, event ledger, and an incident owner.
• Redaction controls for logs and synthetic fixtures for state-change, health, and duplicate-event testing.

Output

Return an event-processing receipt with opaque machine/event ID, monitor version, idempotency result, aggregate health state, notification destination, and redacted error category. Do not include tokens, log bodies, or user data.

Error Handling

• Reject unknown app/region events and quarantine unexpected schemas or notification destinations.
• Bound polling/retry, deduplicate state transitions, and pause automated actions on access or health anomalies.
• Preserve only redacted incident evidence and use the rollback path before replaying actions.

Examples

Send a synthetic machine state transition twice. The monitor records the first once, marks the second duplicate, and rejects an event from an unapproved app without sending its body to any notification channel.

Instructions

Step 1: Poll Machine State Changes

// Monitor machine state transitions via Machines API
async function watchMachines(appName: string, callback: (event: MachineEvent) => void) {
  const client = new FlyClient(appName, process.env.FLY_API_TOKEN!);
  const stateCache = new Map<string, string>();

  setInterval(async () => {
    const machines = await client.listMachines();
    for (const m of machines) {
      const prev = stateCache.get(m.id);
      if (prev && prev !== m.state) {
        callback({
          machineId: m.id,
          region: m.region,
          previousState: prev,
          currentState: m.state,
          timestamp: new Date(),
        });
      }
      stateCache.set(m.id, m.state);
    }
  }, 10_000);  // Check every 10 seconds
}

interface MachineEvent {
  machineId: string;
  region: string;
  previousState: string;
  currentState: string;
  timestamp: Date;
}

Step 2: Health Check Event Handler

// Implement health check that reports machine health
// Fly.io uses this to auto-restart unhealthy machines

import express from 'express';
const app = express();

app.get('/health', async (req, res) => {
  const checks = {
    database: await checkPostgres(),
    redis: await checkRedis(),
    memory: process.memoryUsage().heapUsed < 500 * 1024 * 1024,  // < 500MB
  };

  const healthy = Object.values(checks).every(Boolean);
  res.status(healthy ? 200 : 503).json({
    status: healthy ? 'healt…