Disposable Vast.ai GPU CI

Overview

Use ordinary CPU CI for deterministic tests and reserve Vast.ai for a small GPU acceptance slice. The cloud step must publish its resource ID immediately and make cleanup independent of test success.

Prerequisites

• GPU-only acceptance command and expected machine-readable result
• Scoped CI key, dedicated SSH public key, immutable image digest, and masked logs
• Offer constraints, maximum price, readiness deadline, job timeout, and cleanup escalation

Instructions

Step 1: Gate before provisioning

Run lint, unit, CPU, and image tests first. Skip the GPU job when those fail or when the change does not require hardware acceptance.

Step 2: Create a scoped run manifest

Record commit, image digest, test command, GPU policy, maximum spend, external artifact target, and unique CI run label.

Step 3: Provision one resource

Search compliant verified offers and create exactly one instance. Write the returned instance ID to CI state before any polling or test step.

Step 4: Wait and test

Use structured status, bounded polling, terminal failure branches, and a non-interactive GPU test. Keep command output and logs free of secrets.

Step 5: Publish evidence

Copy the test result, relevant redacted logs, image identity, timings, and cost estimate to durable CI artifacts.

Step 6: Destroy unconditionally

Run destruction in the CI finalizer for success, failure, cancellation, and timeout; confirm removal and alert on cleanup failure.

Authentication

Inject VAST_API_KEY from the CI secret store into the control step only. Use a scoped key without billing-write or team-write and prevent forked or untrusted PRs from receiving it.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• CI run and immutable workload manifest
• Provisioning, state, GPU assertion, timing, and cost artifacts
• Unconditional cleanup result and escalation receipt

Return CI run, commit, image digest, offer/instance IDs, test result, elapsed time, estimate, and confirmed destruction.

Examples

A protected-branch workflow runs CPU tests first, rents one verified GPU under a fixed ceiling, executes a five-minute CUDA acceptance test, uploads JSON evidence, and destroys the instance in always() cleanup.

Error Handling

Vast.ai Failure Classifier

Overview

Diagnose from provider state and error evidence instead of retrying every failure. Separate identity and permission errors, marketplace scarcity, transient startup, terminal host states, billing stops, SSH configuration, and workload exits.

Prerequisites

• Exact command, non-secret arguments, exit status, timestamp, and redacted response
• Affected account context, instance ID, image identity, and expected state
• Authority to inspect but not automatically destroy or fund resources

Instructions

Step 1: Capture structured evidence

Run the failing command with --raw where supported and record CLI version. For request diagnosis, use --explain or --curl only after ensuring generated output cannot expose the key.

Step 2: Classify control-plane failure

Treat 401 as credential failure, 403 as missing scoped permission, 429 as endpoint/identity rate limiting, and insufficient credit or spend-rate errors as billing policy—not host failure.

Step 3: Classify instance state

Loading may reflect an image pull; scheduling after a stop may wait indefinitely for the original GPU; exited is a workload/container failure; unknown or offline indicates missing host heartbeat.

Step 4: Check SSH and network facts

Wait for running, resolve the current SSH URL, verify the registered public key and selected private key, and do not disable host verification as a blanket fix.

Step 5: Choose reversible recovery

Relax offer filters explicitly, repair permissions, change host, restore credit through the approved owner, or resume from checkpoint according to the class.

Step 6: Close or escalate

Preserve identifiers and redacted evidence, confirm any replacement or cleanup, and escalate host or billing cases without speculative retries.

Authentication

Do not paste API keys into diagnostic commands. A scoped read key is usually sufficient for user, instance, logs, offers, and audit evidence; request additional authority only for the selected recovery.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Failure class and evidence timeline
• Ranked recovery action with mutation boundary
• Redacted resolution or escalation receipt

Return command, CLI version, resource ID, observed status/error class, chosen response, outcome, and remaining billing risk.

Examples

An instance stuck in scheduling after being stopped is classified as GPU reacquisition, no…

Checkpointed Vast.ai Training Run

Overview

Treat a training run as a recoverable state machine, not an SSH session. Bind code and image identity, select an offer through policy, persist checkpoints outside the disposable root disk, export final evidence, and destroy.

Prerequisites

• Immutable image and code revision with deterministic training command
• GPU, VRAM, disk, reliability, geography, and price policy
• Checkpoint destination, resume test, runtime deadline, and cleanup owner

Instructions

Step 1: Freeze the run manifest

Record code revision, image digest, dataset version, command, seed, expected checkpoint cadence, budget, and output destination.

Step 2: Select and create

Search only verified rentable offers that meet the manifest, record price components, and create one labeled instance. Persist new_contract immediately.

Step 3: Reach readiness safely

Poll structured instance state with a deadline and terminal branches. Confirm image identity, disk headroom, GPU model, and CUDA visibility.

Step 4: Run with external checkpoints

Start the workload so checkpoints are uploaded or copied to durable storage at the declared cadence. A local checkpoint alone is not recovery evidence.

Step 5: Verify completion or resume

Validate artifact checksums and run metadata. For an interruption, provision a replacement from policy and prove resume from the last durable checkpoint.

Step 6: Close the cost boundary

Copy final logs and checksums, destroy the instance, confirm removal, and reconcile actual spend against the manifest.

Authentication

Use a scoped control-plane key for search and instance operations. Give the workload only the storage credential needed for its checkpoint prefix, with no Vast.ai billing or team authority.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Frozen run and offer-selection manifest
• State, checkpoint, resume, artifact, and spend evidence
• Confirmed instance destruction and discrepancy report

Return revision, image digest, offer and instance IDs, checkpoint URI/checksum, terminal result, actual spend, and cleanup confirmation.

Examples

A fine-tuning job checkpoints every ten minutes to a run-specific object prefix; after a simulated interruption, a replacement instance resumes from the last checksum and the original contract is destroyed.

Error Handling

Vast.ai Serverless Endpoint Rollout

Overview

Replace ad hoc multi-instance orchestration with the provider Serverless control plane. Prove a template independently, establish worker and queue bounds from load evidence, then let the workergroup perform a graceful rolling update.

Prerequisites

• Latency, error-rate, queue-time, concurrency, and cost objectives
• Immutable model/template candidate and a separate canary endpoint
• Initial, minimum, maximum, cold-worker, and inactivity policy

Instructions

Step 1: Prove the candidate template

Launch the new model or environment on a non-production endpoint and verify load, readiness, response schema, and representative outputs.

Step 2: Define scaling bounds

Set min_load, min_workers, max_workers, cold_workers, inactivity_timeout, target_queue_time, and max_queue_time from explicit SLO and budget assumptions.

Step 3: Exercise convergence

During initial rollout, drive representative load up to roughly twice expected capacity and back down three times so the engine can learn GPU cost/performance.

Step 4: Establish the pre-update baseline

Record endpoint latency, queue time, error rate, active/inactive workers, model identity, and spend before changing production.

Step 5: Trigger the rolling update

Save the new template, update the workergroup reference, and monitor inactive workers updating first while active workers drain in-flight requests.

Step 6: Accept or roll back

Verify every worker is on the candidate and compare SLOs. If it fails, point the workergroup back to the last verified template and observe the reverse rollout.

Authentication

Use a scoped key with the documented misc Serverless permissions and no billing-write authority. Keep model registry credentials in approved environment variables, separate from the Vast.ai key.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Canary and production template identities
• Scaling policy plus load-test and rollout timeline
• SLO comparison, worker convergence, and rollback decision

Return endpoint/workergroup IDs, old and new template identities, scaling bounds, load profile, SLO delta, and final rollout state.

Examples

A vLLM endpoint validates a new model on a canary, applies bounded queue targets, then updates its workergroup; active requests drain while new requests move to updated workers, with the old templat…

Vast.ai GPU Cost and Leakage Control

Overview

Optimize total useful-work cost, not headline GPU price. Account for performance, reliability, storage, bandwidth, loading behavior, stopped-instance charges, interruptible semantics, and recovery overhead.

Prerequisites

• GPU/VRAM, throughput, reliability, geography, disk, and completion-time requirements
• Hourly and total budget plus checkpoint/restart cost assumptions
• Instance, volume, charge, and invoice inventory for the analysis window

Instructions

Step 1: Build a normalized offer set

Search verified rentable offers and retain GPU price, storage, bandwidth, reliability, dlperf, dlperf_usd, network, and host constraints.

Step 2: Model useful-work cost

Estimate runtime from measured throughput, then add storage, data transfer, startup, checkpoint, failure, and operator recovery costs.

Step 3: Choose rental semantics

Use on-demand when completion certainty dominates. Use bid pricing only for checkpointed work and pass an explicit bid; a bid search alone does not create an interruptible rental.

Step 4: Find leakage

Identify stopped instances still paying storage, idle active GPUs, abandoned volumes, oversized disks, duplicate canaries, and failed jobs without teardown.

Step 5: Apply bounded changes

Destroy confirmed abandoned resources, resize only through a tested replacement path, and preserve external artifacts before irreversible actions.

Step 6: Reconcile savings

Compare charges and invoices before and after using completed-work units, not just hourly rate, and record any service or reliability regression.

Authentication

Use billing-read for analysis and separate instance-write authority for approved cleanup. Never grant billing-write or credit-transfer permission to an optimizer.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Normalized offer and useful-work cost model
• Leak inventory with owner and safe disposition
• Verified savings, performance delta, and cleanup receipt

Return window, workload unit, selected offer policy, resource IDs, modeled/actual cost, savings, and unresolved billing risk.

Examples

A checkpointed batch job selects a high dlperf_usd bid offer with an explicit bid, while an idle stopped instance and orphaned volume are destroyed after artifact verification; savings are measured per completed batch.

Error Handling

Recoverable Vast.ai Data Movement

Overview

Disposable instance storage is a working tier, not the system of record. Choose the supported location syntax, verify checksums, use trusted datacenters for sensitive cloud sync, and externalize recovery artifacts before stop, expiry, or destroy.

Prerequisites

• Classified source, destination, size, checksum, encryption, and retention requirements
• Approved local, instance, volume, or saved cloud-connection identifiers
• Recovery point objective and owner for copy verification and cleanup

Instructions

Step 1: Plan the route

Choose local:, C.instance:path, V.volume:path, or saved cloud connection syntax based on the documented supported directions. Do not assume volumes can copy directly to local.

Step 2: Prepare least privilege

Scope the Vast.ai control key and cloud credential to the required operation and prefix. Prefer a trusted datacenter for sensitive Cloud Sync.

Step 3: Transfer into a staging path

Copy to a versioned temporary destination with enough disk and bandwidth budget. Never target /root or /, which can break SSH permissions and future copies.

Step 4: Verify before promotion

Compare size, count, cryptographic checksums, and a workload-level sample before renaming or consuming the staged data.

Step 5: Checkpoint externally

Write checkpoints and final artifacts to a recoverable volume or cloud target at a cadence inside the recovery objective.

Step 6: Close retention

Verify the external copy, remove temporary credentials, and destroy or retain instance/volume data according to policy; record ongoing storage cost.

Authentication

Keep storage credentials distinct from VAST_API_KEY, prefix-scoped, and short-lived. Never place either credential in copy logs, image layers, or evidence manifests.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Source/destination route and data-class decision
• Transfer, checksum, checkpoint, and recovery evidence
• Retention, credential revocation, and residual-storage receipt

Return location types and IDs, byte/file counts, checksums, checkpoint age, verification result, and remaining storage owner.

Examples

A training dataset moves from a saved cloud connection to C.123:/workspace/data, is verified in a staging directory, and checkpoints return to a run-specific cloud prefix before the instance is destroyed.

Error Handling

Redacted Vast.ai Support Manifest

Overview

A useful bundle binds exact resource IDs, CLI version, timestamps, states, and redacted errors. It does not archive the entire environment, shell history, account profile, or secret-bearing generated curl commands.

Prerequisites

• Incident window, affected instance/endpoint/workergroup IDs, and symptom
• Evidence classification, retention period, support case owner, and approved destination
• Redaction rules for API keys, URLs, emails, storage credentials, model inputs, and customer data

Instructions

Step 1: Freeze the collection plan

List each command, field, time range, and justification before reading data. Exclude environment dumps and home-directory archives.

Step 2: Collect control-plane facts

Record CLI version, redacted show user, structured instance or Serverless state, offer/template identity, and timestamps.

Step 3: Collect bounded logs

Retrieve only the relevant container, endpoint, or workergroup log window and filter known credential and payload fields.

Step 4: Capture request diagnostics safely

Use --explain or equivalent call metadata only when the output is reviewed for bearer tokens and query credentials before storage.

Step 5: Redact and inventory

Replace sensitive values consistently, retain resource IDs needed by support, and create file hashes plus a redaction attestation.

Step 6: Validate the bundle

Have a second pass search for key patterns, private keys, URLs with credentials, emails, and workload secrets before sharing.

Authentication

Prefer a read-only scoped key. Never store VAST_API_KEY, key-file contents, SSH private material, webhook secrets, cloud credentials, or Authorization headers in the bundle.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Collection scope and command manifest
• Redacted evidence files with hashes and time bounds
• Redaction attestation, retention date, and support handoff receipt

Return incident window, resource IDs, collected fields, excluded classes, file hashes, redaction result, and recipient.

Examples

A support manifest contains CLI version, one offline instance record, the last 100 redacted log lines, template hash, timestamps, and checksums; it excludes environment variables and generated Authorization headers.

Error Handling

Immutable Vast.ai Instance Deployment

Overview

Make the deployment reproducible outside the console. Bind release bytes to an immutable image and versioned template hash, launch a canary under policy, validate service and GPU health, then promote or destroy.

Prerequisites

• Release commit, image digest, template hash, ports, environment names, and startup contract
• Offer policy, health checks, artifact/checkpoint destinations, and maximum rollout time
• Last-known-good image/template plus rollback and teardown owners

Instructions

Step 1: Freeze the deployment manifest

Record all non-secret template settings, image digest, on-start behavior, exposed ports, disk, label, and required GPU/host constraints.

Step 2: Validate outside production

Build and scan the image, run local contract tests, and launch a disposable canary from the exact template hash.

Step 3: Verify readiness

Bound state polling, resolve current endpoints, check process and GPU health, and execute a representative request or batch assertion.

Step 4: Promote deliberately

Create or update the production instance only after canary acceptance. Persist resource IDs and externalize important data before traffic or work begins.

Step 5: Observe the release

Track state, logs, request/job outcome, cost, disk, and checkpoint health through the acceptance window.

Step 6: Roll back or close

On regression, route work to the last-known-good template or recreate from it; copy evidence and destroy rejected or superseded instances.

Authentication

Give deployment automation only search, template, and instance permissions it needs. Keep registry, model, and storage secrets in approved runtime variables and out of template descriptions and logs.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Versioned deployment manifest and immutable identities
• Canary and production health/acceptance evidence
• Promotion, rollback, and superseded-resource cleanup receipt

Return release, image/template identities, offer and instance IDs, health result, acceptance window, decision, and cleanup status.

Examples

A model API is launched from a pinned template hash and image digest, passes GPU and request canaries, then replaces the previous instance; the rejected candidate is destroyed and the old template remains recorded for rollback.

Error Handling

Native Vast.ai Team and Key Governance

Overview

Use the platform's Teams and permission-category model instead of inventing an application-only proxy. Separate human roles from automation keys, constrain high-risk endpoints when possible, and prove both required access and expected denial.

Prerequisites

• Team owner and inventory of members, services, resources, and environments
• Actor-by-action matrix for instance, user, billing, machine, miscellaneous, and team categories
• Joiner, mover, leaver, emergency-access, and periodic-review procedures

Instructions

Step 1: Model responsibilities

Map each actor to read and write operations. Keep billing-write, team-write, machine-write, and instance destruction separate unless a documented duty requires them.

Step 2: Choose default or custom roles

Use Owner, Manager, or Member only when the preset matches. Otherwise create a named custom role from the minimum documented permission categories.

Step 3: Constrain automation keys

Create a different named key per service and environment. Use endpoint and ID constraints where the API supports eq, gte, or lte.

Step 4: Test both directions

For every role or key, run one required action and one prohibited action. A role is not accepted until the denial is observed.

Step 5: Operate membership lifecycle

Assign roles during invitation, review movers before expanding access, remove departed members, revoke stale keys, and record effective context.

Step 6: Review and recover

Export members, roles, keys, and audit evidence on schedule; time-bound emergency elevation and verify removal afterward.

Authentication

Team roles govern member actions; scoped API keys govern programmatic access. Do not share personal full-access keys, and never give monitoring or cost-analysis jobs write authority by convenience.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Actor/action matrix and role/key design
• Positive and negative access-test evidence
• Membership, key, review, and emergency-access receipt

Return team context, role/key IDs, permission categories and constraints, tests, exceptions, reviewer, and next review date.

Examples

A deployment service gets misc, user_read, instance_read, and instance_write; a monitoring key gets only read categories; neither receives billing-write or team-write, and prohibited credit transfer is tested.

…

First Vast.ai GPU Rental With a Cost Fence

Overview

Prove the complete renter lifecycle without leaving billable storage behind. Search a verified offer, register SSH before creation, capture the returned contract ID, bound readiness polling, run one GPU check, retrieve evidence, and destroy.

Prerequisites

• Maximum hourly price, maximum elapsed time, GPU/VRAM need, disk size, and region policy
• Approved image tag and registered SSH public key
• Cleanup owner who can destroy the contract even when the workload fails

Instructions

Step 1: Verify identity and budget

Run vastai show user --raw; confirm the intended account and enough credit. Set an external deadline and total-spend ceiling before searching.

Step 2: Search a suitable offer

Use vastai search offers --raw with verified=true, rentable=true, reliability, direct-port, GPU, disk, and price constraints. Record the selected offer ID and quoted price components.

Step 3: Create exactly one instance

Create with an immutable image tag, SSH, direct networking when required, and the smallest sufficient disk. Capture new_contract as the instance ID.

Step 4: Wait with terminal branches

Poll vastai show instance ID --raw until actual_status is running. Abort and clean up on timeout, exited, unknown, or offline; never poll forever.

Step 5: Run and collect the canary

Resolve the SSH URL, execute nvidia-smi, run the bounded workload, and copy out the small result or checksum.

Step 6: Destroy and prove cleanup

Destroy the instance non-interactively and confirm it no longer appears. Destruction is irreversible but is required to stop storage charges.

Authentication

Use a scoped key that can search offers and manage instances but cannot transfer credit or administer teams. Keep SSH private keys local and never pass them through instance environment variables.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Selected offer and cost-policy decision
• Instance state timeline and GPU canary result
• Artifact checksum plus confirmed destruction receipt

Return account ID, offer ID, instance ID, price, state transitions, canary result, and cleanup confirmation with secrets removed.

Examples

An operator rents one verified RTX 4090 below the approved hourly ceiling, reaches running within the deadline, records a …

Vast.ai Workload Recovery Runbook

Overview

Protect people and credentials first, then preserve recoverable data, stop uncontrolled spend, and restore service from an external checkpoint or last-known-good template. Provider states imply different actions and must not be collapsed into retry.

Prerequisites

• Severity, start time, affected instances/endpoints, release, and user impact
• Last verified external checkpoint or Serverless template
• Incident commander, data owner, billing owner, and authority for stop/destroy/rollback

Instructions

Step 1: Stabilize identity and scope

Confirm account/team context, affected IDs, current state, balance, and whether a credential or data exposure is involved.

Step 2: Classify the provider state

Outbid/stopped may retain disk and storage charges; exited is workload failure; scheduling may await a reclaimed GPU; unknown/offline is host heartbeat loss; low credit threatens availability and data.

Step 3: Preserve recovery evidence

Copy reachable artifacts and logs, record template/image identity, and verify the newest external checkpoint before any destroy.

Step 4: Contain cost or exposure

Stop new creates, revoke compromised keys, and use stop or destroy only under the incident's data-versus-cost decision.

Step 5: Restore on a clean target

Resume from the verified checkpoint on a compliant different offer, or roll Serverless back to the last-known-good template.

Step 6: Reconcile and close

Confirm service and output, destroy superseded resources, audit charges, rotate temporary access, and document the gap and prevention action.

Authentication

Use an incident key with the minimum temporary permissions and a short revocation deadline. Keep billing-write, team administration, workload storage, and control-plane authority separated.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• State-based incident timeline and containment decision
• Checkpoint/template recovery and service-validation evidence
• Resource cleanup, charge reconciliation, access rotation, and postmortem receipt

Return severity, IDs, state class, last good checkpoint/template, containment, restored target, data gap, spend impact, and closed resources.

Examples

After a host goes offline, the team avoids blind restarts, restores the last external checkpoint on a different verified offer, validates output, and retains the original instance ID for support and billing reconciliation.

Error Handling

Vast.ai CLI Authentication Boundary

Overview

Install from the provider-supported channel, choose file-backed versus environment-backed authentication by runtime, and prove that the key can perform only the intended operations. The current CLI migrates legacy key files into the XDG configuration location.

Prerequisites

• Named workload owner and classified workstation, CI, container, or service runtime
• Permission inventory covering search, instance, user, billing, machine, and team operations
• Approved secret store plus a rotation and revocation owner

Instructions

Step 1: Select the install channel

Use the managed installer on Linux, macOS, or WSL when an isolated runtime is desired. Use the vastai PyPI package for Windows or Python SDK work; record the installed version.

Step 2: Place the credential

For an interactive workstation, let vastai set api-key write ~/.config/vastai/vast_api_key or the corresponding XDG configuration path. For CI or containers, inject VAST_API_KEY; the environment value takes precedence and avoids a key file.

Step 3: Create least privilege

Build a permission file containing only the categories and endpoint constraints the workload needs, then create a named scoped key. Do not use a full-access console key for shared automation.

Step 4: Verify positive and negative paths

Run vastai show user --raw, one required operation, and one operation expected to be denied. Preserve only status, key ID, and permission evidence.

Step 5: Plan rotation

Create and verify the replacement before deleting the old key. Revoke immediately if a key, command trace, or artifact may have exposed its value.

Authentication

Every API request requires a key. Treat VAST_API_KEY, the XDG key file, and the legacy ~/.vast_api_key location as secrets; never print key values or copy them into receipts.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Install channel and exact CLI/SDK version
• Scoped permission manifest with positive and expected-denial results
• Redacted rotation, revocation, and rollback receipt

Return runtime, install channel, key ID, granted categories, verification outcomes, and credential owner.

Examples

A CI deployer receives misc, user_read, instance_read, and instance_write without billing or team access; its key is injected as a masked environment secret and delete…

Vast.ai Local-to-GPU Canary Loop

Overview

Keep slow marketplace operations out of the inner loop. Validate code and the container locally, publish an immutable image, then use a single policy-constrained GPU rental only for the behavior that cannot be proven without CUDA.

Prerequisites

• Local unit and CPU-mode test commands with expected outputs
• Immutable image registry and a reproducible startup contract
• Canary GPU, price, reliability, timeout, and cleanup policy

Instructions

Step 1: Define the local contract

Use Read and Grep to identify entrypoints, required files, ports, and environment names. Write an explicit smoke command that works without cloud credentials.

Step 2: Run local gates

Execute unit tests, build the image, start it in CPU mode where supported, and validate its health or process exit contract.

Step 3: Publish immutable bytes

Push a digest or commit-addressed image. Reject mutable latest as canary evidence because the remote host may pull different bytes.

Step 4: Rent the smallest canary

Search verified offers within the budget, create one instance with the immutable image, and record the contract ID before polling.

Step 5: Compare GPU behavior

Run the same smoke assertion plus a CUDA-specific check. Capture structured output and the image digest, not interactive screenshots.

Step 6: Tear down on every path

Copy only the required evidence, destroy the instance, and confirm removal even when the canary fails.

Authentication

Inject the scoped Vast.ai key only into the local control process. Put workload secrets in an approved runtime secret mechanism and never bake them into the image or startup script.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Local test/build receipt and immutable image identity
• GPU canary offer, instance, state, and assertion evidence
• Confirmed teardown plus a classified local/remote delta

Return commit, image digest, offer and instance IDs, local/GPU outcomes, elapsed time, estimated cost, and cleanup status.

Examples

A model-server change passes CPU request-shape tests, is pushed by digest, then runs one CUDA inference on a verified low-cost GPU before the disposable contract is destroyed.

Error Handling

Evidence-Gated Migration to Vast.ai

Overview

Separate portability from cutover. First identify source-provider dependencies, then prove immutable image, storage, networking, secrets, GPU, and output behavior on a disposable Vast.ai canary before moving production work.

Prerequisites

• Source inventory covering images, accelerators, storage, network, identity, schedules, and cost
• Acceptance thresholds for correctness, throughput, latency, recovery, and total spend
• Versioned data/checkpoint transfer, dual-run or drain plan, and rollback owner

Instructions

Step 1: Freeze source truth

Record source release, image digest, GPU profile, command, secrets interfaces, ports, persistent data, checkpoints, SLOs, and representative outputs.

Step 2: Map Vast.ai equivalents

Choose offer or Serverless profiles, template, disk/volume/cloud-copy route, scoped keys, SSH/network mode, and lifecycle semantics.

Step 3: Prove artifact parity

Run the same image and input sample on one disposable Vast.ai target; verify CUDA, dependencies, output schema, checksums, and external checkpoint recovery.

Step 4: Compare production characteristics

Measure startup, throughput, latency, reliability, bandwidth, storage, interruption recovery, and cost per accepted unit.

Step 5: Cut over reversibly

Quiesce or dual-run according to data semantics, move only verified state, switch a bounded slice, and monitor explicit acceptance gates.

Step 6: Accept or roll back

Promote only if every gate passes. Otherwise restore source routing, reconcile writes and checkpoints, and destroy rejected Vast.ai resources.

Authentication

Translate identity to scoped Vast.ai keys and separate storage/registry credentials. Do not export source-provider credentials into images or long-lived Vast.ai environment variables.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Source-to-Vast dependency and control map
• Canary parity, recovery, performance, and cost evidence
• Cutover or rollback timeline with reconciled data and resource cleanup

Return source/target releases, immutable identities, data checkpoint, acceptance deltas, decision, rollback point, and destroyed resources.

Examples

A Runpod training job keeps its container contract, moves checkpoints to a versioned cloud prefix, proves resume on one Vast.ai canary, then shifts scheduled jobs while the source environment remains available for one rollback window.

Error Handling

Vast.ai Environment Isolation

Overview

Vast.ai does not turn naming conventions into isolation automatically. Build the boundary from team/account context, scoped keys, immutable template hashes, labels, data prefixes, budgets, and promotion evidence.

Prerequisites

• Environment owners, sensitivity, workloads, regions, budgets, and approved accounts or teams
• Separate keys, SSH identities, storage prefixes, templates, registries, and alert routes
• Promotion, break-glass, rollback, and access-review procedures

Instructions

Step 1: Define environment identity

Assign each environment an explicit team/account context, key IDs, labels, data prefix, budget, and owner. Reject implicit current context.

Step 2: Separate credentials

Create environment-specific scoped keys and dedicated SSH access. Production credentials must never be available to development jobs or forked CI.

Step 3: Version templates and images

Use immutable image digests and template hashes. Promote the same bytes by recorded identity rather than rebuilding for each environment.

Step 4: Constrain data and resources

Use distinct checkpoint/storage prefixes, resource labels, Serverless IDs, and notifications. Validate that a dev key cannot read or mutate production.

Step 5: Promote through evidence

Require dev tests, staging canary, recovery, cost, and security receipts before a production role references the candidate identity.

Step 6: Audit drift

Regularly compare members, roles, keys, templates, active resources, labels, budgets, and stale environment variables across contexts.

Authentication

Environment isolation depends on distinct scoped credentials and explicit team/account context. Never infer environment only from a filename, branch, or mutable image tag.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Environment authority and resource map
• Cross-environment positive/negative access tests
• Promotion, drift, exception, and rollback receipt

Return environment contexts, role/key IDs, immutable release identity, data prefixes, access tests, budget owners, and promotion state.

Examples

Staging and production use different team contexts and keys but promote the identical template hash; a staging key's attempted production instance read is retained as an expected denial.

Error Handling

Vast.ai Control-Plane Observability

Overview

Measure provider state and workload health separately. Alerts must identify a resource, threshold, evidence link, responder, and safe action; a dashboard without terminal-state and billing coverage is incomplete.

Prerequisites

• Instance, endpoint, workergroup, deployment, and account scopes
• Latency, error, queue, utilization, state-age, balance, and cost objectives
• Collection interval, retention, alert routing, and incident owner

Instructions

Step 1: Inventory labeled resources

Map instance labels and Serverless IDs to service, environment, release, cost center, and owner.

Step 2: Collect structured control state

Read instance actual status, timestamps, price, disk, and endpoints; collect endpoint/workergroup status, logs, and deployment versions.

Step 3: Collect workload signals

Measure request/job success, latency, queue time, GPU utilization, memory, disk, checkpoint age, and last successful artifact.

Step 4: Add billing protection

Track credit balance, instance and storage charges, active/stopped age, and orphaned resources. Alert before balance or cleanup risk becomes urgent.

Step 5: Define stateful alerts

Alert on terminal states, excessive transition age, queue/SLO breach, checkpoint staleness, low balance, and cleanup failure with deduplication.

Step 6: Test the path

Inject a canary event or threshold breach, verify delivery and ownership, then record recovery and false-positive behavior.

Authentication

Use read-only keys for collectors and distinct secrets for alert sinks. Never put a mutation-capable Vast.ai key in dashboards or telemetry processors.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Resource-to-owner inventory and telemetry schema
• Dashboard and actionable alert definitions
• Alert-path test, retention, and unresolved coverage receipt

Return resource scope, collection interval, SLOs, tested alert, responder, evidence location, and blind spots.

Examples

A dashboard separates an instance's running state from workload request success, pages on stale external checkpoints and low balance, and assigns stopped-storage leaks to the billing owner.

Error Handling

Vast.ai Useful-Work Performance Tuning

Overview

Provider benchmark fields help shortlist offers but do not replace workload measurement. Tune one bottleneck at a time and compare completed-work latency, throughput, reliability, and total cost on immutable workload bytes.

Prerequisites

• Representative input, warm-up, sample count, correctness assertion, and target SLO
• Pinned code/image/model plus candidate GPU, VRAM, host, network, and disk profiles
• Baseline throughput, utilization, memory, startup time, and useful-work cost

Instructions

Step 1: Build the baseline

Measure end-to-end completion, startup, data wait, GPU utilization, memory, errors, and cost on the current profile.

Step 2: Shortlist offers

Use verified, rentable, reliability, CUDA, network, dlperf, and dlperf_usd fields to select comparable candidates without weakening workload constraints.

Step 3: Remove startup waste

Pin a smaller prebuilt image, avoid runtime package installs, and measure pull plus readiness separately from execution.

Step 4: Tune the data path

Stage hot data on suitable local storage, increase loaders or prefetch only from evidence, and watch bandwidth and disk cost.

Step 5: Tune compute and memory

Adjust batch, precision, accumulation, kernel, and multi-GPU strategy one variable at a time while preserving output correctness.

Step 6: Choose by useful work

Select the profile that meets the SLO at the best total cost per accepted unit, then destroy all benchmark instances.

Authentication

Benchmark automation needs search and instance permissions only. Keep datasets, model tokens, and registry credentials separate and scoped to the benchmark.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Reproducible baseline and candidate benchmark manifest
• Bottleneck attribution and one-variable experiment results
• Selected profile with SLO, cost, correctness, and cleanup evidence

Return workload identity, candidates, sample counts, throughput, latency, utilization, cost per accepted unit, decision, and destroyed instance IDs.

Examples

Two RTX 4090 offers with different dlperf_usd are tested on the same digest and dataset sample; the faster headline offer is rejected when startup and bandwidth make completed-batch cost worse.

Error Handling

Vast.ai Production Go/No-Go

Overview

Production approval is an evidence bundle, not a list of optimistic assertions. Every gate needs an owner, artifact, and expiry; any failed critical gate yields NO-GO.

Prerequisites

• Release ID, image digest, dataset/model identity, and expected traffic or job profile
• Capacity, latency, reliability, recovery, security, and spend objectives
• Named incident, billing, data-recovery, and teardown owners

Instructions

Step 1: Verify account and permissions

Confirm account/team context, positive balance or approved autobilling, least-privilege keys, audit visibility, and no credential in release artifacts.

Step 2: Verify capacity policy

Demonstrate compliant offers or Serverless worker capacity across required GPU, VRAM, geography, reliability, and price constraints.

Step 3: Verify immutable execution

Pin image/template/model identity and prove startup, health, output contract, and workload-specific acceptance on a canary.

Step 4: Verify recovery

Restore from an external checkpoint or roll back a Serverless template. Prove that stop, outbid, offline, expiry, and zero-balance paths have owners.

Step 5: Verify operations

Show bounded retries, terminal-state handling, logs/metrics, signed webhook or polling coverage, cost alarms, and escalation contacts.

Step 6: Issue the decision

Record PASS, FAIL, owner, evidence URI, and expiry for each gate. Launch only on GO; retain the exact rollback and cleanup commands.

Authentication

Production keys must be named and scoped by function. Separate billing, team administration, deployment, workload storage, and monitoring authority.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Per-gate PASS/FAIL matrix with evidence and expiry
• GO or NO-GO decision with accepted residual risks
• Rollback, incident, billing, and teardown owner receipt

Return release ID, account context, immutable identities, gate results, decision, approvers, expiry, and rollback target.

Examples

A Serverless model release receives GO only after canary output parity, bounded scale testing, signed webhook acceptance, cost thresholds, and reverse rolling-update evidence are attached.

Error Handling

Bounded Vast.ai API Traffic

Overview

Prefer the CLI's built-in 429 retry for ordinary commands and add one orchestration-level call budget around it. For direct REST clients, implement capped backoff because the API does not return standard rate-limit headers.

Prerequisites

• Endpoint and HTTP-method inventory with expected call volume
• Single owner for retry count, polling interval, deadline, and concurrency
• Metrics for calls, 429s, latency, cached hits, and abandoned work

Instructions

Step 1: Map the identity boundary

Group calls by endpoint, method, bearer token, session user, query key, and client IP because each can contribute to the enforced identity.

Step 2: Choose one retry layer

For CLI commands, configure --retry and do not wrap them in another exponential retry. For REST, retry only 429 with a capped backoff and total deadline.

Step 3: Replace tight polling

Cache stable offer and account data, poll only resources still in transitional states, stop on terminal states, and progressively lengthen the interval.

Step 4: Flatten bursts

Queue work, spread scheduled jobs, and batch operations where the documented endpoint supports it. Bound concurrency per endpoint.

Step 5: Honor the call budget

Stop when attempts, elapsed time, or request count reaches the approved ceiling; surface a retryable operational result instead of looping.

Step 6: Escalate sustained pressure

Report endpoint, identity shape, measured call rate, 429 rate, and business need to support when production volume requires a higher limit.

Authentication

Use the narrowest key for each worker and never rotate keys merely to evade a limit. Redact tokens and query credentials from request and retry logs.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Endpoint/identity traffic inventory
• Retry, polling, caching, concurrency, and deadline policy
• Measured before/after call volume and 429 receipt

Return endpoints, identity partition, client type, retry owner, call ceiling, observed 429s, elapsed time, and final disposition.

Examples

A readiness watcher uses one CLI command with --retry 3, polls only while state is transitional, lengthens the interval, and exits immediately on running, exited, unknown, or offline.

Error Handling

Governed Vast.ai GPU Workload Architecture

Overview

Center the architecture on an immutable run or release manifest and a lifecycle ledger. Search and planning are read-only; paid resource creation crosses an approval boundary; execution writes recoverable state externally; teardown closes both cost and evidence.

Prerequisites

• Batch, training, interactive, or Serverless workload inventory with SLOs
• Data, model, image, credential, region, reliability, and spend policies
• Owners for approval, execution, recovery, billing, security, and platform incidents

Instructions

Step 1: Define the immutable intent

Create a signed or versioned manifest containing workload bytes, image/template/model identity, GPU policy, data/checkpoint routes, SLOs, budget, and expiry.

Step 2: Separate planner and mutator

Let a read-scoped planner evaluate offers or Serverless profiles. Require explicit approval before a narrowly scoped mutator creates, updates, transfers credit, or destroys.

Step 3: Choose the executor

Use an instance lifecycle for bounded jobs or dedicated services; use Serverless endpoint/workergroup control for managed inference scaling and rolling updates.

Step 4: Externalize durable state

Keep datasets, checkpoints, artifacts, event ledgers, and evidence outside disposable root disks with checksums and recovery objectives.

Step 5: Observe and reconcile

Combine provider states, signed notifications, bounded polling, workload SLOs, balance, charges, and resource inventory; reconcile events against periodic reads.

Step 6: Close every lifecycle

Accept output, copy evidence, destroy disposable resources, revoke temporary access, reconcile charges, and leave an auditable handoff for retained resources.

Authentication

Use native Teams roles and distinct scoped keys for planning, mutation, monitoring, and administration. Workload storage and registry credentials must never inherit control-plane authority.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Trust-boundary and component decision record
• Immutable manifest, lifecycle ledger, recovery, and observability contracts
• Threat, failure, cost, rollback, and teardown evidence plan

Return workload classes, chosen executors, authority boundaries, immutable artifacts, recovery targets, SLOs, budgets, event reconciliation, and lifecycle owners.

Examples

A planner selects verified offers but cannot rent; an approved mutator creates from a signed run manifest; the training…

Owned Vast.ai Python SDK Lifecycles

Overview

Use the highest-level client that preserves the required control. Keep resource identity and cleanup ownership explicit, normalize provider response variants at one boundary, and never hide a billable instance behind an unbounded retry.

Prerequisites

• Chosen high-level, synchronous, asynchronous, or Serverless client surface
• Typed internal model for offers, instances, terminal states, and provider errors
• Idempotency, timeout, test-double, and cleanup design

Instructions

Step 1: Select one client boundary

Use VastAI for broad CLI-equivalent operations, SyncClient for typed synchronous instance control, AsyncClient inside an async context, or Serverless for endpoint inference.

Step 2: Normalize responses once

Map provider dictionaries and error shapes into a small internal result type. Preserve offer ID, contract ID, status, price, and raw error code for diagnosis.

Step 3: Separate plan from mutation

Search and score offers without creating resources. Require an approved plan object before calling create, update, destroy, or credit operations.

Step 4: Own the lifecycle

Persist the returned instance ID immediately, apply a monotonic deadline, classify terminal failure states, and place destroy or handoff in an explicit finalizer.

Step 5: Test failure boundaries

Cover 401, 403, 429, malformed responses, create-without-ID, readiness timeout, and cleanup failure using local fakes.

Step 6: Expose a redacted receipt

Return normalized decisions and state transitions; never return the API key or full environment.

Authentication

Construct clients from VAST_API_KEY or the approved local configuration. Do not pass keys as literals, serialize client objects, or let provider credentials cross into workload payloads.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Client-selection decision and typed adapter contract
• Bounded lifecycle implementation with local failure tests
• Redacted mutation and cleanup receipt

Return client type, package version, plan identity, provider resource IDs, terminal outcome, and cleanup owner.

Examples

An async job runner uses AsyncClient as a context manager, records instance.id before waiting, cancels on its deadline, and destroys the instance in a tested finalizer.

Error Handling

Vast.ai Renter Security Boundary

Overview

A Vast.ai renter controls credentials and workload configuration but does not own the physical host. Apply least privilege, prefer verified datacenters for sensitive data, encrypt critical material, and assume destroyed local storage is not a substitute for external recovery.

Prerequisites

• Data classification and decision on whether shared marketplace hardware is permitted
• Scoped API-key and dedicated SSH-key owners
• Approved image provenance, secret injection, network exposure, and checkpoint controls

Instructions

Step 1: Constrain the control plane

Create named scoped keys with only needed permission categories and endpoint constraints. Separate human, CI, monitoring, and deployment identities.

Step 2: Establish SSH trust

Register a dedicated public key before creation, protect the private key, resolve the current connection endpoint, and validate host identity according to policy.

Step 3: Verify workload provenance

Use an immutable image digest, scan it, document its entrypoint, and keep registry, model, and storage credentials out of image layers and startup text.

Step 4: Limit host and network exposure

Prefer verified datacenters for sensitive workloads, expose only required ports, and avoid placing secrets or regulated plaintext on untrusted hosts.

Step 5: Protect recoverability

Encrypt critical data, scope storage credentials to a run prefix, and checkpoint externally before stop, expiry, host failure, or destroy.

Step 6: Rotate and close

Revoke temporary keys, remove obsolete SSH keys and environment variables, verify resource destruction, and retain a redacted audit receipt.

Authentication

API keys are password-equivalent and do not expire by default. Environment injection is appropriate for CI, but logs, shell traces, generated curl commands, and debug bundles must remain secret-free.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Threat and data-placement decision
• Credential, SSH, image, host, network, and storage controls
• Positive/negative access tests plus teardown receipt

Return workload class, selected host trust tier, key IDs/scopes, image digest, exposed ports, checkpoint destination, and cleanup outcome.

Examples

A sensitive training canary uses a verified datacenter, immutable image digest, constrained instance key, dedicated SSH key, encrypted dataset, prefix-scoped checkpoint credential, and confirmed credential revocation after teardown…

Reversible Vast.ai Client and Template Upgrade

Overview

Treat the provider client and workload template as separate compatibility boundaries. Pin and test the managed CLI or PyPI package, preserve the vastai_sdk compatibility shim during migration, and canary template changes before a rolling update.

Prerequisites

• Current CLI/SDK version, install channel, import usage, commands, templates, and dependent automation
• Target version and release evidence plus contract tests for critical commands and response adapters
• Rollback client version, prior template hash, and an acceptance deadline

Instructions

Step 1: Freeze the current contract

Record versions, install location, key precedence, imports, command help, structured outputs, and immutable template/model identities.

Step 2: Upgrade in isolation

For the managed CLI, use its version-aware update mechanism and retain the prior version. For Python, pin the target vastai package in a disposable environment.

Step 3: Test client compatibility

Run auth, offer search, instance read, response normalization, and expected-denial tests without creating paid resources unless the plan requires a canary.

Step 4: Migrate SDK imports deliberately

Move from vastai_sdk to vastai while the documented compatibility shim remains; test high-level and sync/async client boundaries actually used.

Step 5: Canary workload changes

Create a separate endpoint or disposable instance from the target template, prove output and recovery, then trigger a controlled rolling update if Serverless.

Step 6: Accept or revert

Compare contract and SLO evidence. Restore the prior client pin or template reference on failure and verify the rollback path.

Authentication

Confirm the upgraded client still reads the intended XDG or environment credential and preserves least privilege. Never test upgrades with an unscoped production key by default.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Before/after client and template contract inventory
• Compatibility, canary, SLO, and expected-denial results
• Acceptance or rollback receipt with retained prior versions

Return install channel, old/new versions, import boundary, old/new template hashes, tests, decision, and rollback verification.

Examples

A service moves from the vastai_sdk shim to vastai in staging, pins the package, validates its response adapter, canaries a new templat…

Signed Vast.ai Notification Receiver

Overview

Vast.ai notifications are signed, at-least-once deliveries. Verify the exact raw body before JSON parsing, reject stale timestamps, deduplicate by stable event ID, enqueue durably, and return success only after acceptance.

Prerequisites

• Full subscription keys such as client:low_credit or client:outbid
• Public HTTPS endpoint without redirects or private-address resolution
• Webhook secret store, five-minute replay window, durable queue, and event ledger

Instructions

Step 1: Create a bounded subscription

Discover valid notification types and subscribe only the required full client or host keys. Respect the four-webhook-per-user limit and retain context-specific webhooks when short slugs overlap.

Step 2: Store the one-time secret

Capture the signing secret at create or rotate time because list and update responses do not return it. Test retrieval and rotation ownership.

Step 3: Verify raw delivery

Require POST, read exact body bytes, combine integer X-Vast-Timestamp, a period, and raw body, then compare the HMAC-SHA256 against X-Vast-Signature-256 in constant time.

Step 4: Reject replay and duplicates

Reject missing or malformed headers and timestamps older than 300 seconds. Deduplicate retries using X-Vast-Event-Id or payload event_id.

Step 5: Acknowledge after durable enqueue

Persist the verified event and return 2xx quickly. Process side effects asynchronously and idempotently; the delivery timeout is ten seconds.

Step 6: Test and operate failure paths

Use the provider test delivery, verify rotation, and monitor permanent 3xx/most-4xx failures versus retryable 408, 429, 5xx, timeout, or connection errors.

Authentication

Webhook configuration uses the scoped Vast.ai API key; delivery verification uses the distinct webhook secret. Never log either secret or parse/re-serialize JSON before signature verification.

Tool Discipline

Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.

Output

• Subscription keys, webhook ID, and secret-rotation ownership
• Signature, replay, deduplication, enqueue, and idempotency tests
• Delivery health and retry/permanent-failure receipt

Return webhook ID, context keys, test event ID, signature/replay/dedupe results, acknowledgment latency, queue record, and rotation date.

Examples

A client:low_credit delivery is verified from raw bytes, rejected if older than five minutes, …