Fly.io Rate Limits

Overview

The Fly.io Machines API rate-limits per organization, with write operations (create, delete, update) throttled much more aggressively than reads. Deploying fleets of edge machines across multiple regions can easily trigger 429s, especially during rolling deployments or auto-scaling events. The API returns a Retry-After header on rate-limited responses, and organizations running 50+ machines should implement client-side token bucket limiting to avoid cascading failures during high-churn operations.

Prerequisites

• Current platform limits confirmed for the organization plus approved concurrency, retry bounds, and a queue owner.
• Redacted telemetry for request category, queue age, throttle count, and machine lifecycle—not tokens or payloads.
• A staging fleet/synthetic workload for testing pauses, retries, and cancellation.

Instructions

1. Honor explicit throttling guidance and use bounded concurrency with jittered backoff.
2. Attach idempotency/operation tracking to lifecycle changes so a retry cannot duplicate a create, stop, or delete action.
3. Queue exhausted operations for reviewed handling, alert on backlog growth, and reduce demand before resuming.

Output

Publish a rate-control receipt with policy version, concurrency, retry bounds, throttle count, queue outcome, owner, and manual disposition. Do not expose app names if they are sensitive, tokens, or request bodies.

Examples

Apply a small synthetic scale change, simulate a 429, and confirm the worker waits and then performs the operation once. A repeated failure must enter the review queue rather than trigger a fleet-wide replay.

Rate Limit Reference

Rate Limiter Implementation

class FlyRateLimiter {
  private tokens: number;
  private lastRefill: number;
  private readonly max: number;
  private readonly refillRate: number;
  private queue: Array<{ resolve: () => void }> = [];

  constructor(maxPerMinute: number) {
    this.max = maxPerMinute;
    this.tokens = maxPerMinute;
    this.lastRefill = Date.now();
    this.refillRate = maxPerMinute / 60_000;
  }

  async acquire(): Promise<void> {
    this.refill();
    if (this.tokens >= 1) { this.tokens -= 1; return; }
    retur

Fly.io Reference Architecture

Overview

Production architecture for Fly.io: multi-region web tier, Postgres with read replicas, Redis for caching, background workers, and private networking.

Prerequisites

• A documented data-flow inventory, trust boundaries, ownership, region/retention choices, and disaster-recovery objectives.
• Separate scoped identities for deployment, runtime, database, worker, and observability systems.

Instructions

1. Place public ingress, private services, storage, workers, and observability behind explicit network and identity boundaries.
2. Define data locality, replication, backup, access, and recovery behavior before creating additional regions or consumers.
3. Use staged deployment, health checks, redacted telemetry, and an independently tested rollback per service.
4. Validate architecture changes with synthetic traffic and ensure a failure in one region cannot leak secrets or corrupt cross-region state.

Output

Maintain an architecture decision record with components, trust boundaries, data locations, identities, health/rollback controls, owners, and recovery evidence. Do not include secrets or customer data.

Error Handling

• Isolate an unhealthy region or consumer and preserve a safe primary path while recovery proceeds.
• Quarantine unexpected cross-region writes or permission failures for review.
• Restore the previous routing/configuration before replaying queued work.

Examples

Deploy a fictional workload to a staging primary and replica region, deny the worker access to public ingress secrets, and simulate a regional health failure. Verify traffic stays on the healthy route and rollback does not replay writes.

Architecture

           ┌─────────── Fly.io Anycast DNS ──────────┐
           │                                          │
    ┌──────▼──────┐  ┌──────────────┐  ┌─────────────▼───┐
    │  Web (iad)  │  │  Web (lhr)   │  │   Web (nrt)     │
    │  shared-1x  │  │  shared-1x   │  │   shared-1x     │
    └──────┬──────┘  └──────┬───────┘  └────────┬────────┘
           │                │                    │
    ───────┴────────────────┴────────────────────┴─── .internal DNS
           │                │                    │
    ┌──────▼──────┐  ┌──────▼───────┐  ┌────────▼────────┐
    │ Postgres    │  │ Postgres     │  │   Redis          │
    │ Primary     │  │ Replica      │  │   (upstash.io)   │
    │ (iad)       │  │ (lhr)        │  │                  │
    └─────────────┘  └──────────────┘  └──────────────────┘
           │
    ┌──────▼──────┐
    │  Worker     │
    │  (iad)      │
    │  shared-1x  │
    └─────────────┘

Setup Commands

# 1. Web app — multi-region
fly launch --name my-web --region iad
fly scale count 1 --region lhr
fly scale count 1 --regio

Fly.io SDK Patterns

Overview

Production-ready patterns for the Fly.io Machines REST API at https://api.machines.dev. Fly.io exposes both GraphQL (organization queries) and REST (machine lifecycle) APIs. The Machines REST API is the primary integration surface for creating, starting, stopping, and destroying VMs across 30+ global regions. A structured client ensures consistent auth, typed machine states, and reliable wait-for-state polling.

Prerequisites

• An app-scoped token held in a secret manager, approved app/region policy, and synthetic staging app.
• Idempotent lifecycle design, rate controls, redacted diagnostics, and a rollback owner.

Instructions

1. Validate app, region, operation, and request schema before a lifecycle call.
2. Track opaque operation IDs, use bounded retries, and protect create/stop/delete from duplicate execution.
3. Route unexpected state, permission, or target results to reviewed handling and preserve the prior configuration.

Output

Produce a client-validation receipt with API/contract version, fixture result, operation ID, idempotency outcome, owner, and redacted failure reference. Never log tokens, machine config secrets, or user data.

Examples

Create a disposable staging machine from a synthetic configuration, retry the request under the same operation ID, and verify only one machine results. Simulate an invalid region and ensure the client rejects it before a provider call.

Singleton Client

const FLY_API = 'https://api.machines.dev';
let _client: FlyClient | null = null;
export function getClient(appName: string): FlyClient {
  if (!_client) {
    const token = process.env.FLY_API_TOKEN;
    if (!token) throw new Error('FLY_API_TOKEN must be set');
    _client = new FlyClient(appName, token);
  }
  return _client;
}
class FlyClient {
  private h: Record<string, string>;
  constructor(private app: string, token: string) {
    this.h = { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' };
  }
  async listMachines(): Promise<FlyMachine[]> {
    const r = await fetch(`${FLY_API}/v1/apps/${this.app}/machines`, { headers: this.h });
    if (!r.ok) throw new FlyError(r.status, await r.text()); return r.json();
  }
  async createMachine(config: MachineConfig, region: string): Promise<FlyMachine> {
    const r = await fetch(`${FLY_API}/v1/apps/${this.app}/machines`, {
      method: 'POST', headers: this.h, body: JSON.stringify({ region, config }) });
    if (!r.ok) throw new FlyError(r.status, await r.text()); return r.json();
  }
  async waitForState(id: string, state: string, timeout = 30): Promise<void> {
    const r = await fetch(`${FLY_API}/v1/apps/${this.app}/machines/${id}/wait?state=${state}&timeout=${timeout}`,
      { headers: this.h 

Fly.io Security Basics

Overview

Fly.io deploys applications to edge locations worldwide using Firecracker microVMs. Security concerns center on deploy token scoping (org-wide vs per-app), secrets management (encrypted at rest, injected as env vars), private networking via WireGuard mesh (6PN), and TLS certificate management. A leaked deploy token can push arbitrary code to production machines across all regions.

Prerequisites

• A named security owner, app/organization access inventory, secret-manager integration, and recurring access-review cadence.
• Approved network, region, TLS, logging, and incident/revocation policies plus synthetic staging fixtures.

Instructions

1. Use app-scoped deploy tokens and separate identities per environment; never place tokens in code, tickets, terminal captures, or debug bundles.
2. Restrict private services and secrets to the minimum set of machines and roles, with explicit network boundaries and access review.
3. Verify incoming signed events before processing, log opaque IDs only, and make downstream actions idempotent.
4. Monitor for unauthorized deployment, secret, region, or certificate changes and rotate/revoke credentials immediately after suspected exposure.

Output

Maintain a security receipt with identity scope, secret reference, policy version, access-review date, verification/rotation result, owner, and redacted incident state. Never include tokens, configuration secrets, or user data.

Examples

Create a disposable staging app using a scoped token, attempt an unauthorized app operation, and verify it is denied. Rotate the token, confirm the old credential fails, and retain only the redacted policy and control outcome.

API Key Management

function validateFlyToken(): void {
  const token = process.env.FLY_API_TOKEN;
  if (!token) {
    throw new Error("Missing FLY_API_TOKEN — use `fly tokens create deploy -a <app>`");
  }
  // Never log tokens; log only token type for debugging
  const isDeployToken = token.startsWith("FlyV1");
  console.log("Fly.io token loaded, type:", isDeployToken ? "deploy" : "personal");
}

Webhook Signature Verification

import crypto from "crypto";
import { Request, Response, NextFunction } from "express";

function verifyFlyWebhook(req: Request, res: Response, next: NextFunction): void {
  const signature = req.headers["x-fly-signature"] as string;
  const secret = process.env.FLY_WEBHOOK_SECRET!;
  const expected = crypto.createHmac("sha256", secret).update(req.body).digest("hex");
  if (!signature || !crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    res.status(401).send("Invalid signature");
    return;
  

Fly.io Upgrade & Migration

Overview

Guide for Fly.io platform migrations: Apps v1 (Nomad) to v2 (Machines), flyctl CLI upgrades, Postgres major version upgrades, and region migrations.

Prerequisites

• Current platform documentation and an inventory of applications, machines, regions, volumes, databases, identities, and dependent consumers.
• A staging environment, synthetic traffic/data, tested backup/restore, rollback owner, and explicit acceptance/reconciliation criteria.

Output

Produce a migration receipt with versions reviewed, affected resources, staging/canary results, backup/restore evidence, reconciliation outcome, approver, and rollback state. Keep tokens, connection strings, and user data out of the receipt.

Error Handling

• Stop promotion on health, schema, region, permission, or reconciliation mismatches and restore the prior configuration.
• Quarantine failed migrations by opaque resource ID; do not bulk replay stateful workloads to diagnose failures.
• Escalate potential data loss or credential exposure and retain only approved incident evidence.

Examples

Migrate a disposable staging app using synthetic traffic, exercise a backup/restore of fictional data, and simulate a failed health check. Verify rollback returns routing and data access to the known-good release before considering a production canary.

Instructions

Apps v1 to v2 Migration

# Check current platform version
fly status -a my-app  # Look for "Platform: machines" vs "nomad"

# Migrate to Apps v2 (Machines)
fly migrate-to-v2 -a my-app

# Verify
fly status -a my-app
fly machine list -a my-app

flyctl CLI Upgrade

# Check current version
fly version

# Upgrade
fly version update

# Or reinstall
curl -L https://fly.io/install.sh | sh

Postgres Major Version Upgrade

# Check current version
fly postgres connect -a my-db -c "SELECT version();"

# Create new cluster with target version
fly postgres create --name my-db-v16 --region iad --image-ref flyio/postgres-flex:16

# Migrate data
fly postgres import pg_dump_url -a my-db-v16

# Update app to point to new cluster
fly postgres detach my-db -a my-app
fly postgres attach my-db-v16 -a my-app
fly deploy -a my-app  # Picks up new DATABASE_URL

Region Migration

# Add machines in new region
fly scale count 1 --region fra -a my-app

# Verify new region is healthy
fly status -a my-app

# Remove machines from old region
fly scale count 0 --region iad -a my-app

# For volumes: create new volume, migrate data, destroy old
fly volumes create data --size 10 --region fra -a my-app

Migration Checklist

• [ ] Current state documented (fly status