CoreWeave CI Integration

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Set up CI/CD for CoreWeave GPU cloud workloads: run unit tests with mocked Kubernetes clients on every PR, deploy inference containers to CoreWeave namespaces on merge to main, and validate GPU resource requests against quota. CoreWeave uses standard Kubernetes APIs with GPU-specific scheduling, so CI pipelines authenticate via kubeconfig and manage deployments through kubectl.

GitHub Actions Workflow

Prerequisites

• A repository with protected branches and a trusted runner policy.
• Mocked tests that need no cluster credential, plus a dedicated low-privilege staging identity for optional integration tests.
• Secret-manager-backed CI secrets available only to trusted, non-fork workflows.

Instructions

1. Run deterministic unit and manifest validation on every pull request without credentials.
2. Run live integration checks only after merge or from an approved protected workflow.
3. Bound test resources, collect redacted pod events on failure, and revoke the staging identity if exposure is suspected.
4. Keep deploy approval and production rollout separate from the test job.

# .github/workflows/coreweave-ci.yml
name: CoreWeave CI
on:
  pull_request:
    paths: ['src/**', 'k8s/**', 'Dockerfile']
  push:
    branches: [main]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npm test -- --reporter=verbose

  deploy:
    if: github.ref == 'refs/heads/main'
    needs: unit-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and push container
        run: |
          echo "${{ secrets.GHCR_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
          docker build -t ghcr.io/${{ github.repository }}/inference:${{ github.sha }} .
          docker push ghcr.io/${{ github.repository }}/inference:${{ github.sha }}
      - name: Deploy to CoreWeave
        env:
          KUBECONFIG_DATA: ${{ secrets.COREWEAVE_KUBECONFIG }}
        run: |
          echo "$KUBECONFIG_DATA" | base64 -d > /tmp/kubeconfig
          export KUBECONFIG=/tmp/kubeconfig
          kubectl set image deployment/inference \
            inference=ghcr.io/${{ github.repository }}/inference:${{ github.sha }}
          kubectl rollout status deployment/inference --timeout=300s

Mock-Based Unit Tests

// tests/coreweave-service.test.ts
import { describe, it, expect, vi } from 'vitest'…

CoreWeave Common Errors

Overview

Use this triage guide to classify common GPU, Kubernetes, storage, and connectivity failures before changing capacity or credentials. Capture the smallest redacted evidence set and use a reversible fix in the affected namespace.

Prerequisites

• Read-only access to the affected namespace, pod events, quota, and node labels.
• The workload name, expected GPU class, and a named service or platform owner.

Instructions

1. Identify the pod, Job, or Service and collect its status plus recent events.
2. Match the symptom to the table below, then validate the proposed cause with the listed read-only command before applying a fix.
3. Make the smallest namespace-scoped change, verify recovery, and record the redacted event/command outcome in the incident or change record.

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Error Reference

1. Pod Stuck Pending -- No GPU Available

kubectl describe pod <pod-name> | grep -A5 Events
# "0/N nodes are available: insufficient nvidia.com/gpu"

Fix: Check GPU availability: kubectl get nodes -l gpu.nvidia.com/class=A100_PCIE_80GB. Try a different GPU type or region.

2. CUDA Out of Memory

torch.cuda.OutOfMemoryError: CUDA out of memory

Fix: Reduce batch size, enable gradient checkpointing, or use a larger GPU (A100-80GB instead of 40GB).

3. Image Pull BackOff

Fix: Create an imagePullSecret:

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=$GH_USER \
  --docker-password=$GH_TOKEN

4. NCCL Timeout (Multi-GPU)

NCCL error: unhandled system error

Fix: Ensure all GPUs are on the same node (NVLink). For multi-node, use InfiniBand-connected nodes.

5. PVC Not Mounting

Fix: Check storage class availability: kubectl get sc. Use CoreWeave storage classes like shared-hdd-ord1 or shared-ssd-ord1.

6. Node Affinity Mismatch

Fix: List valid GPU class labels:

kubectl get nodes -o json | jq -r '.items[].metadata.labels["gpu.nvidia.com/class"]' | sort -u

7. Service Not Reachable

Fix: Check Service and Endpoints:

kubectl get svc,endpoints <service-name>

Output

• A classified failure with supporting redacted events and a bounded recovery action.
• A verified recovery resu…

CoreWeave Core Workflow: KServe Inference

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Deploy production inference services on CoreWeave using KServe InferenceService with GPU scheduling, autoscaling, and scale-to-zero. CKS natively integrates with KServe for serverless GPU inference.

Prerequisites

• Completed coreweave-install-auth setup
• KServe available on your CKS cluster
• Model stored in S3, GCS, or HuggingFace

Instructions

Step 1: Deploy an InferenceService

# inference-service.yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-inference
  annotations:
    autoscaling.knative.dev/class: "kpa.autoscaling.knative.dev"
    autoscaling.knative.dev/metric: "concurrency"
    autoscaling.knative.dev/target: "1"
    autoscaling.knative.dev/minScale: "1"
    autoscaling.knative.dev/maxScale: "5"
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 5
    containers:
      - name: kserve-container
        image: vllm/vllm-openai:latest
        args:
          - "--model"
          - "meta-llama/Llama-3.1-8B-Instruct"
          - "--port"
          - "8080"
        ports:
          - containerPort: 8080
            protocol: TCP
        resources:
          limits:
            nvidia.com/gpu: "1"
            memory: 48Gi
            cpu: "8"
          requests:
            nvidia.com/gpu: "1"
            memory: 32Gi
            cpu: "4"
        env:
          - name: HUGGING_FACE_HUB_TOKEN
            valueFrom:
              secretKeyRef:
                name: hf-token
                key: token
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: gpu.nvidia.com/class
                  operator: In
                  values: ["A100_PCIE_80GB"]

kubectl apply -f inference-service.yaml
kubectl get inferenceservice llama-inference -w

Step 2: Scale-to-Zero Configuration

# For dev/staging -- scale down to zero when idle
metadata:
  annotations:
    autoscaling.knative.dev/minScale: "0"    # Scale to zero
    autoscaling.knative.dev/maxScale: "3"
    autoscaling.knative.dev/scaleDownDelay: "5m"

Step 3: Test the Endpoint

# Get inference URL
INFERENCE_URL=$(kubectl get inferenceservice llama-inference \
  -o jsonpath='{.status.url}')

curl -X POST "${INFERENCE_URL}/v1/chat/completions" \
  -H "C…

CoreWeave Core Workflow: GPU Training

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Run distributed GPU training on CoreWeave: single-node multi-GPU and multi-node training with PyTorch DDP, Slurm-on-Kubernetes, and shared storage.

Prerequisites

• CKS cluster with multi-GPU node pools (8xA100 or 8xH100)
• Shared storage (CoreWeave PVC or NFS)
• Training container with PyTorch and NCCL

Instructions

Step 1: Single-Node Multi-GPU Training

# training-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: llm-finetune
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: trainer
          image: ghcr.io/myorg/trainer:latest
          command: ["torchrun"]
          args:
            - "--nproc_per_node=8"
            - "train.py"
            - "--model_name=meta-llama/Llama-3.1-8B"
            - "--batch_size=4"
            - "--epochs=3"
          resources:
            limits:
              nvidia.com/gpu: "8"
              memory: 512Gi
              cpu: "64"
          volumeMounts:
            - name: data
              mountPath: /data
            - name: checkpoints
              mountPath: /checkpoints
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: training-data
        - name: checkpoints
          persistentVolumeClaim:
            claimName: model-checkpoints
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: gpu.nvidia.com/class
                    operator: In
                    values: ["A100_NVLINK_A100_SXM4_80GB"]

Step 2: Persistent Storage for Training Data

# storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-data
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 500Gi
  storageClassName: shared-hdd-ord1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-checkpoints
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 200Gi
  storageClassName: shared-ssd-ord1

Step 3: Monitor Training Progress

# Watch training logs
kubectl logs -f job/llm-finetune

# Check GPU utilization
kubectl exec -it $(kubectl get pod -l job-name=llm-finetune -o name) -- nvidia-smi

# Check training metrics
kubectl exec -it $(kubectl get pod -l job-name=llm-finetune -o name) -- \
  cat /checkpoints/training_log.json | tail -5

Error Handling

CoreWeave Cost Tuning

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Reduce GPU spend by matching a workload's memory, throughput, availability, and latency requirements to the smallest approved capacity. Cost changes must preserve the service SLO and retain a measured rollback path; approximate public prices are planning inputs, not a billing source of truth.

Prerequisites

• Read-only access to workload utilization, namespace quota, and billing allocation data.
• A named service owner and target SLO for the workload being resized.
• Approval for any change that can reduce production capacity or alter availability.

GPU Pricing Reference (approximate)

Cost Optimization Strategies

Scale-to-Zero for Dev/Staging

autoscaling.knative.dev/minScale: "0"
autoscaling.knative.dev/scaleDownDelay: "5m"

Right-Size GPU Selection

def recommend_gpu(model_size_b: float, inference_only: bool = True) -> str:
    if model_size_b <= 7:
        return "L40" if inference_only else "A100_PCIE_80GB"
    elif model_size_b <= 13:
        return "A100_PCIE_80GB"
    elif model_size_b <= 70:
        return "A100_PCIE_80GB (4x tensor parallel)"
    else:
        return "H100_SXM5 (8x tensor parallel)"

Quantization to Use Smaller GPUs

Use AWQ or GPTQ quantization to fit larger models on smaller GPUs:

# 70B model at 4-bit fits on single A100-80GB instead of 4x
vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ --quantization awq

Instructions

1. Baseline seven days of GPU utilization, queue time, request latency, error rate, and allocated cost by namespace; do not decide from a single peak.
2. Propose one reversible change—right-size a non-production workload, set a conservative scale-down delay, or run a quantized canary.
3. Compare the canary against its SLO and budget for an agreed observation window. Promote only if quality, latency, and error rate remain within the published limit.
4. Record the instance choice, owner, forecast, and rollback trigger i…

CoreWeave Data Handling

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave GPU cloud workloads involve large-scale data artifacts: model weights (multi-GB safetensors/GGUF), training datasets (parquet, TFRecord, WebDataset), checkpoint snapshots, and inference cache volumes. Data flows through Kubernetes PersistentVolumeClaims backed by region-specific storage classes. Compliance requires encryption at rest via the storage driver, namespace-scoped RBAC for volume access, and audit logging for any data egress from GPU nodes.

Prerequisites

• Approved data classification, retention schedule, and region for the artifact.
• A namespace-scoped service account, encrypted storage class, and approved destination.
• Expected artifact size and SHA-256 from a trusted source before import.

Instructions

1. Create or select an encrypted PVC in the approved region and grant its mount only to the intended namespace service account.
2. Import artifacts using a short-lived job; verify the expected SHA-256 before any training or serving workload consumes them.
3. Export only to a reviewed destination, preserve the checksum and data-owner approval, and enforce the retention policy for checkpoints and datasets.
4. Record storage provisioning, access changes, deletion, and external egress in the audit system without copying sensitive artifact contents into logs.

Data Classification

Data Import

import { KubeConfig, BatchV1Api } from '@kubernetes/client-node';

async function importDataset(pvcName: string, sourceUrl: string, namespace: string) {
  const kc = new KubeConfig();
  kc.loadFromDefault();
  const batch = kc.makeApiClient(BatchV1Api);
  const job = {
    metadata: { name: `import-${Date.now()}`, namespace },
    spec: { template: { spec: {
      restartPolicy: 'Never',
      containers: [{ name: 'loader', image: 'python:3.11-slim',
        command: ['python3', '-c', `
import urllib.request, hashlib
dest = '/data/dataset…

CoreWeave Debug Bundle

Overview

Collect the minimum redacted cluster, workload, and rate-limit evidence needed to triage a CoreWeave incident. A debug bundle is diagnostic evidence, not an archive for credentials, prompts, model weights, datasets, kubeconfigs, or raw secret values.

Prerequisites

• Authorized read-only Kubernetes access to the affected namespace or cluster.
• An incident identifier, retention destination, and named incident owner.
• A local encrypted workspace with sufficient space; do not collect a bundle on a shared or unmanaged machine.

Instructions

1. Create the bundle for the affected scope and set restrictive file permissions.
2. Capture status, events, allocation, and bounded log tails; remove headers or values that contain credentials before packaging.
3. Inspect the archive locally for secrets and customer data, then attach it only to the approved incident record with the required retention period.
4. Delete the local copy after verified upload, according to the incident policy.

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Scope

Collect GPU node health, Kubernetes pod status, event logs, and API connectivity into a single diagnostic archive for CoreWeave support tickets. This bundle captures cluster-level resource allocation, failed pod logs, GPU device plugin state, and network reachability so support engineers can diagnose infrastructure issues without requesting additional information. Useful when GPU pods are stuck pending, inference workloads OOM, or node autoscaling behaves unexpectedly.

Debug Collection Script

#!/bin/bash
set -euo pipefail
BUNDLE="debug-coreweave-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BUNDLE"

# Environment check
echo "=== CoreWeave Debug Bundle ===" | tee "$BUNDLE/summary.txt"
echo "Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$BUNDLE/summary.txt"
echo "COREWEAVE_API_KEY: ${COREWEAVE_API_KEY:+[SET]}" >> "$BUNDLE/summary.txt"
echo "KUBECONFIG: ${KUBECONFIG:-default}" >> "$BUNDLE/summary.txt"
echo "kubectl: $(kubectl version --client --short 2>/dev/null || echo 'not found')" >> "$BUNDLE/summary.txt"

# API connectivity
HTTP=$(curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer ${COREWEAVE_API_KEY}" \
  https://api.coreweave.com/v1/namespaces 2>/dev/null || echo "000")
echo "API Status: HTTP $HTTP" >> "$BUNDLE/summary.txt"

# Cluster state
kubectl get nodes -o wide > "$BUNDLE/nodes.txt" 2>&1 || true
kubectl get pods --all-namespaces -o wide > "$BUNDLE/…

CoreWeave Deploy Integration

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Deploy GPU-accelerated inference services on CoreWeave Kubernetes (CKS). This skill covers containerizing inference workloads with NVIDIA CUDA base images, configuring GPU resource limits and node affinity for A100/H100 scheduling, setting up health checks that validate GPU availability and model loading, and executing rolling updates that respect GPU node draining. CoreWeave's scheduler requires explicit GPU resource requests to place pods on the correct hardware tier.

Docker Configuration

Prerequisites

• A reviewed image digest in an approved registry and a namespace-scoped image pull secret.
• A production deployment manifest with resource limits, health checks, SLO, and rollback revision.
• Approval for the target namespace, GPU capacity, and service exposure.

Instructions

1. Build and scan the image, then deploy the immutable digest to staging first.
2. Verify readiness, health, GPU allocation, and baseline request behavior before promotion.
3. Use a controlled rolling update with a timeout and named observer.
4. Roll back immediately if readiness, error rate, latency, or security verification fails.

FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04 AS base
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip curl && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt ./
RUN pip3 install --no-cache-dir -r requirements.txt

FROM base
RUN groupadd -r app && useradd -r -g app app
COPY --chown=app:app src/ ./src/
COPY --chown=app:app models/ ./models/
USER app
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1
CMD ["python3", "src/server.py"]

Environment Variables

export COREWEAVE_API_KEY="cw_xxxxxxxxxxxx"
export COREWEAVE_NAMESPACE="tenant-my-org"
export MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct"
export GPU_TYPE="A100_PCIE_80GB"
export GPU_COUNT="1"
export LOG_LEVEL="info"
export PORT="8080"

Health Check Endpoint

import express from 'express';
import { execSync } from 'child_process';

const app = express();

app.get('/health', async (req, res) => {
  try {
    const gpuInfo = execSync('nvidia-smi --query-gpu=name,memory.used --format=csv,noheader').toString().trim();
    const modelLoaded = globalThis.modelReady === true;
    if (!modelLoaded) throw new Error('Model not loaded');
    res.json(…

CoreWeave Enterprise RBAC

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave runs GPU workloads on Kubernetes, so RBAC maps directly to K8s namespace isolation and ResourceQuotas. Each team gets a dedicated namespace with GPU limits, storage caps, and network policies. This prevents noisy-neighbor problems where one team's training job starves another's inference service. SOC 2 and HIPAA workloads require namespace-level audit logging and team-scoped API key rotation.

Prerequisites

• A verified human or workload identity group from the organization identity provider.
• Cluster-admin approval for namespace, quota, and RoleBinding changes.
• A team owner, approved GPU quota, and data-classification decision for the namespace.

Instructions

1. Create a namespace per team and apply ResourceQuota and NetworkPolicy before granting workload permissions.
2. Bind an IdP group to the smallest suitable ClusterRole; do not bind individual users or reuse a cluster-wide edit role without a documented exception.
3. Run a SubjectAccessReview for the intended verbs and resources, then retain the redacted decision and audit entry with the access request.
4. Review bindings and service-account tokens on a regular schedule; remove access promptly when a team, project, or incident requires it.

Role Hierarchy

Permission Check

import { KubeConfig, RbacAuthorizationV1Api } from '@kubernetes/client-node';

async function checkNamespaceAccess(user: string, namespace: string, verb: string, resource: string): Promise<boolean> {
  const kc = new KubeConfig();
  kc.loadFromDefault();
  const rbac = kc.makeApiClient(RbacAuthorizationV1Api);
  const review = { apiVersion: 'authorization.k8s.io/v1', kind: 'SubjectAccessReview',
    spec: { user, resourceAttributes: { namespace, verb, resource } } };
  const result = await rbac.createSubjectAccessReview(review);
  return result.body.status?.allowed ?? false;
}

Role Assignment

CoreWeave Fabric Diagnostics

> Community-contributed. Not affiliated with, endorsed by, or sponsored by > CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Detects when a CoreWeave multi-node GPU job has silently fallen off the InfiniBand fabric onto TCP — the failure that makes distributed training run at a fraction of the hardware's speed while every GPU keeps billing at the full rate — and gives the exact fix.

Overview

On a CoreWeave multi-node job, NCCL should carry collectives over InfiniBand with GPUDirect RDMA (NET/IB). If any one of three conditions is missing, NCCL silently falls back to TCP sockets (NET/Socket): the job still runs, still converges, and raises no error — but all-reduce throughput collapses (commonly cited as 5-20x slower [[nccl]]) because it now crosses the Ethernet control plane instead of the 400 Gb/s-class fabric. You keep paying full GPU rate for a multi-node run that performs like a badly-connected one. This is the single highest-dollar invisible failure on the platform, and nothing in the default output flags it.

The diagnosis is deterministic: the bundled scripts/fabric-check.py greps the pasted NCCL_DEBUG=INFO log for the decisive Using network line (and NET/IB vs NET/Socket), parses the pod-spec's resources block for the RDMA device request, reads ibstat port state, and echoes any all_reduce_perf bus bandwidth — then emits a VERDICT with the rdma_engaged/transport call, the missing conditions, and the fix. The LLM never eyeballs which transport is in use; the script decides. Deep grounding lives in references/, loaded only when a leg of the diagnosis needs it.

Prerequisites

• An NCCL_DEBUG=INFO log from the actual run — the primary signal. Re-run the job (or one rank) with NCCL_DEBUG=INFO set and capture stderr. The decisive line is Using network IB (good) vs Using network Socket (the fallback). This is the one input the skill really needs; everything else corroborates.
• Optional, for a full diagnosis: the pod/job spec (kubectl get pod NAME -o yaml) to check the RDMA device request; ibstat output from the node for port health; and all_reduce_perf results from CoreWeave's [nccl-tests][nt] to measure bus bandwidth.
• python3 to run the deterministic checker (stdlib only).
• kubectl (read-only) if corroborating the live pod spec / node cordon state.

Authentication. Nothing secret is read. If the pod spec is pull…

CoreWeave GPU Cost Leak Hunter

> Community-contributed. Not affiliated with, endorsed by, or sponsored by > CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Audits a CoreWeave GPU cluster for real-dollar cost leaks — idle reserved capacity, GPUs on the wrong SKU, allocated-but-idle instances, and steady on-demand spend that should be committed — then emits a CFO-grokkable, dollar-ranked FinOps report.

Overview

CoreWeave ships no cost dashboard and no billing API ([usage-monitoring docs][um]). There is also no single "dollars" metric — spend is reconstructed by querying usage from CoreWeave's managed Grafana in PromQL and multiplying each resource's usage by its rate-card price. This skill does exactly that, then ranks the leaks by monthly dollar impact.

The math is deterministic: PromQL returns usage counts, and the bundled scripts/rank-and-report.py does every multiplication, sum, and ranking — the agent never eyeballs a number. Two of the four categories are billed waste (Confirmed); the other two are a right-sizing model (Estimated) and a commitment decision (At-risk), labeled so a CFO never reads a modeled number as recoverable cash. Deep domain knowledge lives in references/, loaded only when a leak needs it.

Prerequisites

• CoreWeave managed Grafana access — the Prometheus data source is reachable only to a member of the admin, metrics, or write group in the CoreWeave Cloud Console ([usage-monitoring docs][um]). This is the hard dependency; Step 1 probes it and fails fast if the group is missing.
• A Prometheus/Grafana query endpoint in $CW_PROM_URL (the Grafana data-source proxy, e.g. https://grafana.ORG.coreweave.com/api/datasources/proxy/uid/UID) and a bearer token in $CW_TOKEN for curl.
• kubeconfig for the cluster (CoreWeave-issued) so kubectl get can corroborate live GPU allocation and node labels.
• The rate card — CoreWeave publishes no price metric, so on-demand and committed rates are supplied to the ranker from references/gpu-right-sizing.md (dated snapshot of [coreweave.com/pricing][pr]) or the customer's contract.
• jq and python3 for parsing query JSON and running the ranker.

Authentication. All auth comes from the environment ($CW_PROM_URL, $CW_TOKEN, $KUBECONFIG) — no secrets are hardcoded. Grafana enforces the group membership above on every query.

Instructions

The pipeline is detect →…

CoreWeave GPU Node Forensics

> Community-contributed. Not affiliated with, endorsed by, or sponsored by > CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc. > "NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are > cited from NVIDIA's public documentation.

Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one grounded move — reschedule, reset-gpu, reboot-node, rma, watch, or app-bug-not-hardware — from an NVIDIA Xid code or a pasted dmesg / nvidia-smi blob. The classification is deterministic: a bundled script does the mapping so the agent never guesses whether a card is dead, degraded, or fine.

Overview

A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars and days of wall-clock gone — because one rank stalls the whole collective. The expensive mistakes are triage mistakes: RMAing a healthy card for an app bug, restarting a job onto a GPU whose memory error was uncontained, or manually uncordoning a node the lifecycle controller is trying to replace. This skill kills that ambiguity.

The decision logic is grounded in the NVIDIA Xid error catalog (<https://docs.nvidia.com/deploy/xid-errors/>) and CoreWeave's node-lifecycle / cordon behavior. The math-of-the-matter — which Xid means what, and how row-remapper state overrides it — lives in scripts/triage.py as a table the LLM does not get to re-litigate. Deep domain knowledge (the full code→action table, the row-remap decision, the cordon rules) loads from references/ on demand.

The headline is the Xid 94-vs-95 split. A contained memory error (94) cost you one job restart on a healthy node; an uncontained one (95) means the GPU could not isolate the fault and everything it touched is suspect. Getting that one bit wrong is the difference between a 30-second reschedule and a run that quietly trained on corrupt gradients. The script decides it; the skill never eyeballs it.

This skill is diagnostic, not destructive: it recommends the cordon / drain / reset / RMA next-step but its tools are scoped read-only (nvidia-smi -q, kubectl get, dmesg) — it never runs a reset, a reboot, or an uncordon itself.

Prerequisites

• The failure evidence. Either an Xid number, or a pasted dmesg / nvidia-smi dump. The skill works from a paste alone — no live cluster access required — which is the common case (an operator pastes what the run's logs showed).
• Optional live access for corroboration: kubectl context on the CoreWeave cluster (read-only is enough), and nvidia-smi on the node. If neither i…

CoreWeave Hello World

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Deploy your first GPU workload on CoreWeave: a simple inference service using vLLM or a batch CUDA job. CoreWeave runs Kubernetes on bare-metal GPU nodes with A100, H100, and L40 GPUs.

Prerequisites

• Completed coreweave-install-auth setup
• kubectl configured with CoreWeave kubeconfig
• Namespace with GPU quota

Instructions

Step 1: Deploy a vLLM Inference Server

# vllm-inference.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-server
  template:
    metadata:
      labels:
        app: vllm-server
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "meta-llama/Llama-3.1-8B-Instruct"
            - "--port"
            - "8000"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 48Gi
              cpu: "8"
            requests:
              nvidia.com/gpu: 1
              memory: 32Gi
              cpu: "4"
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: gpu.nvidia.com/class
                    operator: In
                    values: ["A100_PCIE_80GB"]
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-server
spec:
  selector:
    app: vllm-server
  ports:
    - port: 8000
      targetPort: 8000
  type: ClusterIP

# Create HuggingFace token secret
kubectl create secret generic hf-token --from-literal=token="${HF_TOKEN}"

# Deploy
kubectl apply -f vllm-inference.yaml
kubectl get pods -w  # Wait for Running state

# Port-forward and test
kubectl port-forward svc/vllm-server 8000:8000 &
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello!"}]}'

Step 2: Batch GPU Job

# gpu-batch-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-benchmark
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: …

CoreWeave Incident Runbook

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Respond to GPU workload incidents by stabilizing customer impact, preserving redacted evidence, and restoring a known-good state. The incident commander owns communications and escalation; responders use only the access required for triage.

Prerequisites

• An incident ID, named commander, affected namespace/service, and on-call route.
• Authorized read-only cluster access plus a documented production rollback revision.
• A redaction policy for logs, prompts, model artifacts, and credentials.

Instructions

1. Declare severity and scope, then capture pod, event, node, and service status.
2. Stabilize impact with the documented scale, failover, or rollback action before root-cause work.
3. Collect only redacted, bounded diagnostics and escalate hardware or capacity issues through CoreWeave support.
4. Verify recovery against the service SLO, update stakeholders, and create follow-up work for root cause and prevention.

Triage Steps

# 1. Check pod status
kubectl get pods -l app=inference -o wide

# 2. Check recent events
kubectl get events --sort-by=.lastTimestamp | tail -20

# 3. Check node status
kubectl get nodes -l gpu.nvidia.com/class -o wide

# 4. Check GPU health
kubectl exec -it $(kubectl get pod -l app=inference -o name | head -1) -- nvidia-smi

Common Incidents

Inference Service Down

1. Check pod status and events
2. If OOMKilled: reduce batch size or upgrade GPU
3. If ImagePullBackOff: check registry credentials
4. If Pending: check GPU quota and availability

GPU Node Failure

1. Pods will be rescheduled automatically
2. If no capacity: scale down non-critical workloads
3. Contact CoreWeave support for extended outages

Model Loading Failure

1. Check HuggingFace token secret exists
2. Verify model name spelling
3. Check PVC has sufficient storage
4. Review container logs for download errors

Rollback

kubectl rollout undo deployment/inference

Output

• A time-stamped incident record with scope, owner, stabilization action, and redacted evidence.
• A verified recovery or an explicit escalation with a safe customer-impact mitigation.

Error Handling

CoreWeave Install & Auth

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Set up access to CoreWeave Kubernetes Service (CKS). CKS runs bare-metal Kubernetes with NVIDIA GPUs -- no hypervisor overhead. Access is via standard kubeconfig with CoreWeave-issued credentials.

Prerequisites

• CoreWeave account at https://cloud.coreweave.com
• kubectl v1.28+ installed
• Kubernetes namespace provisioned by CoreWeave

Instructions

Step 1: Download Kubeconfig

1. Log in to https://cloud.coreweave.com
2. Navigate to API Access > Kubeconfig
3. Download the kubeconfig file

# Save kubeconfig
mkdir -p ~/.kube
cp ~/Downloads/coreweave-kubeconfig.yaml ~/.kube/coreweave

# Set as active context
export KUBECONFIG=~/.kube/coreweave

# Verify connection
kubectl get nodes
kubectl get namespaces

Step 2: Configure API Token

# CoreWeave API token for programmatic access
export COREWEAVE_API_TOKEN="your-api-token"

# Store securely
echo "COREWEAVE_API_TOKEN=${COREWEAVE_API_TOKEN}" >> .env
echo "KUBECONFIG=~/.kube/coreweave" >> .env

Step 3: Verify GPU Access

# List available GPU nodes
kubectl get nodes -l gpu.nvidia.com/class -o custom-columns=\
NAME:.metadata.name,GPU:.metadata.labels.gpu\.nvidia\.com/class,\
STATUS:.status.conditions[-1].type

# Check GPU allocatable resources
kubectl describe nodes | grep -A5 "Allocatable:" | grep nvidia

Step 4: Test with a Simple GPU Pod

# test-gpu.yaml
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda-test
      image: nvidia/cuda:12.2.0-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: gpu.nvidia.com/class
                operator: In
                values: ["A100_PCIE_80GB"]

kubectl apply -f test-gpu.yaml
kubectl logs gpu-test  # Should show nvidia-smi output
kubectl delete pod gpu-test

Error Handling

CoreWeave Local Dev Loop

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Local development workflow for CoreWeave: build containers, test YAML manifests with dry-run, push to registry, and deploy to CoreWeave CKS.

Prerequisites

• Completed coreweave-install-auth setup
• Docker installed locally
• Container registry access (Docker Hub, GHCR, or CoreWeave registry)

Instructions

Step 1: Project Structure

my-inference-service/
├── Dockerfile
├── src/
│   ├── server.py          # Inference server code
│   └── model_config.py    # Model configuration
├── k8s/
│   ├── deployment.yaml    # GPU deployment manifest
│   ├── service.yaml       # Service and ingress
│   └── hpa.yaml           # Horizontal pod autoscaler
├── scripts/
│   ├── build.sh           # Build and push container
│   └── deploy.sh          # Deploy to CoreWeave
├── .env.local
└── Makefile

Step 2: Build and Push Container

# Build locally
docker build -t my-inference:latest .

# Tag for registry
docker tag my-inference:latest ghcr.io/myorg/my-inference:v1.0.0

# Push
docker push ghcr.io/myorg/my-inference:v1.0.0

Step 3: Validate Manifests Before Deploy

# Dry-run against CoreWeave cluster
kubectl apply -f k8s/deployment.yaml --dry-run=server

# Diff against current state
kubectl diff -f k8s/deployment.yaml

# Check resource requests match available GPU types
kubectl get nodes -l gpu.nvidia.com/class=A100_PCIE_80GB --no-headers | wc -l

Step 4: Deploy and Watch

kubectl apply -f k8s/
kubectl rollout status deployment/my-inference
kubectl logs -f deployment/my-inference

Error Handling

Output

• A locally built, versioned image and a server-validated deployment manifest.
• A staging rollout receipt with readiness and bounded log evidence.
• A repeatable local-to-cluster loop that leaves production credentials and data out of the workspace.

Examples

Use the staging namespace for the complete loop and inspect the diff before apply:

docker build -t ghcr.io/myorg/my-inference:dev-20260826 .
kubectl -n inference-staging diff -f k8s/deployment.yaml
kubectl -n inference-staging ap…

CoreWeave Multi-Environment Setup

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave GPU cloud requires strict environment separation to control infrastructure costs and prevent resource contention. Each environment maps to an isolated Kubernetes namespace with its own GPU quota, scaling policy, and access controls. Development uses cheaper GPU tiers for iteration speed, staging mirrors production GPU types for accurate benchmarking, and production runs full-scale with no scale-to-zero to guarantee inference latency SLAs.

Environment Configuration

Prerequisites

• Separate, approved namespaces and service identities for development, staging, and production.
• Environment-specific secrets injected by a secrets manager, never committed .env files.
• A promotion owner, staging evaluation gate, and production rollback manifest.

Instructions

1. Define overlays that differ only in reviewed capacity, endpoint, and namespace values.
2. Validate required variables and secret references before applying an overlay.
3. Promote dev to staging, run the service evaluation, then use a controlled production rollout.
4. Roll back the production overlay on an SLO, quality, or security-gate failure; do not copy staging credentials into production.

const coreweaveConfig = (env: string) => ({
  development: {
    namespace: "app-dev", apiEndpoint: process.env.CW_API_ENDPOINT_DEV!,
    token: process.env.CW_TOKEN_DEV!, gpuType: "L40", scaleToZero: true, replicas: [0, 1],
  },
  staging: {
    namespace: "app-staging", apiEndpoint: process.env.CW_API_ENDPOINT_STG!,
    token: process.env.CW_TOKEN_STG!, gpuType: "A100_PCIE_40GB", scaleToZero: true, replicas: [0, 2],
  },
  production: {
    namespace: "app-prod", apiEndpoint: process.env.CW_API_ENDPOINT_PROD!,
    token: process.env.CW_TOKEN_PROD!, gpuType: "A100_PCIE_80GB", scaleToZero: false, replicas: [2, 10],
  },
}[env]);

Environment Files

# Per-env files: .env.development, .env.staging, .env.production
CW_API_ENDPOINT_{DEV|STG|PROD}=https://k8s.{ord1|ord1|las1}.coreweave.com
CW_TOKEN_{DEV|STG|PROD}=<service-account-token>
CW_NAMESPACE={app-dev|app-staging|app-prod}
CW_GPU_TYPE={L40|A100_PCIE_40GB|A100_PCIE_80GB}

Environment Validation

function validateCoreWeaveEnv(env: string): void {
  const required = ["CW_API_ENDPOINT", "CW_TOKEN", "CW_NAMESPACE", "CW_GPU_TYPE"];
  const suffix = { development: "_DEV", staging: "_STG", production: "_PROD" }[env];
  cons…

CoreWeave Observability

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave runs GPU-intensive workloads on Kubernetes where hardware failures, memory exhaustion, and underutilization directly impact cost and reliability. Observability must cover DCGM GPU metrics, Kubernetes pod health, inference latency, and job completion rates. Proactive monitoring prevents wasted spend on idle GPUs and catches OOM conditions before they cascade.

Prerequisites

• A metrics backend receiving Kubernetes and DCGM exporter metrics.
• A named dashboard and on-call owner for the namespace or service.
• Log and trace redaction rules that exclude prompts, model outputs, tokens, and credentials.

Instructions

1. Tag metrics with bounded values such as namespace, model family, and status; do not use request IDs, prompts, or user identifiers as labels.
2. Build dashboards for utilization, memory, queue depth, latency, error rate, and restart rate, then set alert thresholds from a measured baseline.
3. Route critical alerts to the responsible on-call team and link a runbook that includes a safe scale-down or rollback action.
4. Test one alert in a non-production namespace and verify that the receipt contains only operational metadata, not workload data.

Key Metrics

Instrumentation

async function trackInference(model: string, fn: () => Promise<any>) {
  const start = Date.now();
  try {
    const result = await fn();
    metrics.record('coreweave.inference.latency', Date.now() - start, { model, status: 'ok' });
    metrics.increment('coreweave.inference.completed', { model });
    return result;
  } catch (err) {
    metrics.increment('coreweave.inference.errors', { model, error: err.code });
    throw err;
  }
}

Health Check Dashboard

async function coreweaveHealth(): Promise<Record<string, string>> {
  const gpu = await queryPrometheus('avg(DCGM_FI_DEV_…

CoreWeave Performance Tuning

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Tune GPU inference or training only against measured throughput, latency, quality, availability, and cost targets. A higher utilization figure is not a success if it causes queueing, memory pressure, or a customer-facing SLO regression.

Prerequisites

• A baseline for p95/p99 latency, throughput, error rate, GPU memory, and utilization.
• A representative non-sensitive evaluation set and a named owner for the SLO.
• A staging lane and a rollback manifest for every resource or serving change.

Instructions

1. Change one variable at a time—batching, GPU class, replicas, or memory target.
2. Run the agreed load and quality evaluation in staging, then compare with baseline.
3. Promote a canary only when all SLO and quality thresholds pass for the observation window.
4. Revert to the prior manifest when latency, errors, or quality crosses the agreed limit.

GPU Selection by Workload

Inference Optimization

# Continuous batching with vLLM
containers:
  - name: vllm
    args:
      - "--model=meta-llama/Llama-3.1-8B-Instruct"
      - "--max-num-batched-tokens=8192"
      - "--max-num-seqs=256"
      - "--gpu-memory-utilization=0.90"
      - "--enable-prefix-caching"
      - "--dtype=float16"

Autoscaling Tuning

# HPA based on GPU utilization
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-server
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: DCGM_FI_DEV_GPU_UTIL
        target:
          type: AverageValue
          averageValue: "70"

Performance Benchmarks

CoreWeave Production Checklist

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Use this gate before exposing a CoreWeave inference or training workload to production. It proves that capacity, data protection, observability, and recovery are owned and tested; it is not a substitute for a service-specific risk review.

Prerequisites

• A signed deployment change, named service owner, and approved production namespace.
• Passing staging evaluation, capacity forecast, and rollback manifest.
• On-call routing, secret-manager references, and data classification for the workload.

Instructions

1. Complete each checklist item with a link to its evidence, not a verbal assertion.
2. Confirm a staged rollout, health check, and rollback command during the change window.
3. Stop the release if any security, data, capacity, or observability gate lacks an owner.
4. Record the final go/no-go decision, time, owner, and redacted verification receipt.

Inference Services

• [ ] GPU type and count validated for model size
• [ ] Autoscaling configured (KServe or HPA)
• [ ] Health and readiness probes set
• [ ] Resource requests AND limits specified
• [ ] Node affinity targeting correct GPU class
• [ ] minReplicas >= 1 for production (no cold starts)

Storage

• [ ] Model weights in PVC (not downloaded at startup)
• [ ] Checkpoints saved to persistent storage
• [ ] Storage class appropriate (SSD for inference, HDD for archival)

Security

• [ ] Secrets for model tokens and registry access
• [ ] Network policies applied
• [ ] Container images from trusted registries

Monitoring

• [ ] GPU utilization metrics collected
• [ ] Inference latency and throughput tracked
• [ ] Alert on pod restarts and OOM events
• [ ] Log aggregation configured

Rollback

kubectl rollout undo deployment/my-inference
kubectl rollout status deployment/my-inference

Output

• A production-readiness record linking capacity, storage, security, monitoring, and rollback evidence to the deployment.
• A clear go/no-go decision with a responsible owner and tested recovery command.

Error Handling

CoreWeave SDK Patterns

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave is Kubernetes-native -- use kubectl, Kubernetes Python client, or Helm for programmatic management. These patterns cover GPU-aware deployment templates, inference client wrappers, and node affinity configurations.

Instructions

GPU Affinity Helper

# coreweave_helpers.py
from dataclasses import dataclass

@dataclass
class GPUConfig:
    gpu_class: str        # A100_PCIE_80GB, H100_SXM5, L40, etc.
    gpu_count: int = 1
    memory_gb: int = 32
    cpu_cores: int = 4

GPU_CATALOG = {
    "a100-80gb": GPUConfig("A100_PCIE_80GB", memory_gb=48, cpu_cores=8),
    "h100-80gb": GPUConfig("H100_SXM5", memory_gb=64, cpu_cores=12),
    "l40":       GPUConfig("L40", memory_gb=24, cpu_cores=4),
    "a100-8x":   GPUConfig("A100_NVLINK_A100_SXM4_80GB", gpu_count=8, memory_gb=256, cpu_cores=64),
}

def gpu_affinity_block(gpu_class: str) -> dict:
    return {
        "nodeAffinity": {
            "requiredDuringSchedulingIgnoredDuringExecution": {
                "nodeSelectorTerms": [{
                    "matchExpressions": [{
                        "key": "gpu.nvidia.com/class",
                        "operator": "In",
                        "values": [gpu_class],
                    }]
                }]
            }
        }
    }

def gpu_resources(config: GPUConfig) -> dict:
    return {
        "limits": {
            "nvidia.com/gpu": str(config.gpu_count),
            "memory": f"{config.memory_gb}Gi",
            "cpu": str(config.cpu_cores),
        },
        "requests": {
            "nvidia.com/gpu": str(config.gpu_count),
            "memory": f"{config.memory_gb // 2}Gi",
            "cpu": str(config.cpu_cores // 2),
        },
    }

Inference Client Wrapper

# inference_client.py
import requests
from typing import Optional

class CoreWeaveInferenceClient:
    def __init__(self, endpoint: str, timeout: int = 30):
        self.endpoint = endpoint.rstrip("/")
        self.timeout = timeout
        self.session = requests.Session()

    def generate(self, prompt: str, max_tokens: int = 256, **kwargs) -> str:
        resp = self.session.post(
            f"{self.endpoint}/v1/completions",
            json={"prompt": prompt, "max_tokens": max_tokens, **kwargs},
            timeout=self.timeout,
        )
        resp.raise_for_status()
        return resp.json()["choices"][0]…

CoreWeave Security Basics

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave provides bare-metal GPU cloud on Kubernetes. Security concerns center on compute credential management (kubeconfig, deploy tokens), network isolation between inference workloads, secrets for model registry access (HuggingFace, container registries), and protecting sensitive training data on persistent volumes. A compromised namespace can expose GPU resources, model weights, and customer inference data.

Prerequisites

• A named namespace owner and a current data classification for the workload.
• Secrets-manager access for deployment credentials; no credentials in manifests or git.
• Authority to apply Kubernetes policies in the target namespace and an approved rollback plan.

Instructions

1. Store API, registry, and model credentials in the approved secrets manager and mount only the minimum secret into the intended workload.
2. Apply namespace-scoped RBAC, ResourceQuota, and default-deny NetworkPolicy before exposing an inference endpoint.
3. Validate images and request payloads in CI, then test webhook signatures with a known valid and invalid payload without logging raw secrets.
4. Review access events and rotate/revoke the affected secret after suspected exposure.

API Key Management

import { KubeConfig, CoreV1Api } from "@kubernetes/client-node";

function createCoreWeaveClient(): CoreV1Api {
  const apiKey = process.env.COREWEAVE_API_KEY;
  if (!apiKey) {
    throw new Error("Missing COREWEAVE_API_KEY — set via secrets manager");
  }
  const kc = new KubeConfig();
  kc.loadFromDefault();
  const api = kc.makeApiClient(CoreV1Api);
  // Never log kubeconfig or API key contents
  console.log("CoreWeave client initialized for namespace:", process.env.CW_NAMESPACE);
  return api;
}

Webhook Signature Verification

import crypto from "crypto";
import { Request, Response, NextFunction } from "express";

function verifyCoreWeaveWebhook(req: Request, res: Response, next: NextFunction): void {
  const signature = req.headers["x-coreweave-signature"] as string;
  const secret = process.env.COREWEAVE_WEBHOOK_SECRET!;
  const expected = crypto.createHmac("sha256", secret).update(req.body).digest("hex");
  const supplied = Buffer.from(signature ?? "");
  const expectedBuffer = Buffer.from(expected);
  if (supplied.length !== expectedBuffer.length || !crypto.timingSafeEqual(supplied, expectedBuffer)) {
    res.status(401).send("Invalid signature");
    return;
  }
  next();
}

Input Validation

CoreWeave Upgrade & Migration

> Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

CoreWeave is a GPU-specialized cloud provider running Kubernetes-native infrastructure. Migrations involve upgrading between GPU instance types (A100 to H100), updating CUDA driver versions, and handling Kubernetes API version changes across namespaces. Tracking API versions is critical because CoreWeave's instance type labels and resource quotas change between platform releases, and deploying to a deprecated instance class will cause scheduling failures.

Version Detection

import { KubeConfig, CoreV1Api } from "@kubernetes/client-node";

async function detectCoreWeaveVersion(): Promise<void> {
  const kc = new KubeConfig();
  kc.loadFromDefault();
  const k8sApi = kc.makeApiClient(CoreV1Api);

  // Check current namespace GPU allocations
  const pods = await k8sApi.listNamespacedPod("my-namespace");
  for (const pod of pods.body.items) {
    const gpuClass = pod.spec?.nodeSelector?.["gpu.nvidia.com/class"];
    const cudaVersion = pod.metadata?.labels?.["cuda-version"];
    console.log(`Pod ${pod.metadata?.name}: GPU=${gpuClass}, CUDA=${cudaVersion}`);
  }

  // Detect deprecated instance types
  const deprecated = ["A100_PCIE_40GB", "V100_PCIE_16GB", "RTX_A5000"];
  const activeGpus = pods.body.items
    .map((p) => p.spec?.nodeSelector?.["gpu.nvidia.com/class"])
    .filter(Boolean);
  const stale = activeGpus.filter((g) => deprecated.includes(g!));
  if (stale.length > 0) console.warn(`Deprecated GPU types in use: ${stale.join(", ")}`);
}

Migration Checklist

• [ ] Review CoreWeave release notes for deprecated instance types
• [ ] Audit all deployments for gpu.nvidia.com/class node selectors
• [ ] Verify CUDA version compatibility with target GPU (see matrix below)
• [ ] Update container base images to match new CUDA/cuDNN requirements
• [ ] Test inference latency on new GPU type in staging namespace
• [ ] Update resource requests (nvidia.com/gpu) for new instance memory
• [ ] Migrate persistent volumes if switching regions or availability zones
• [ ] Update Kubernetes API version in manifests (e.g., apps/v1 changes)
• [ ] Validate HPA scaling behavior on new instance type throughput
• [ ] Run canary deployment with traffic split before full cutover

Schema Migration

// CoreWeave instance type labels changed in 2025 platform update
// Old: gpu.nvidia.com/class: "A100_PCIE_80GB"
// New: gpu.nvidia.com/class: "H100_SXM5_80GB"

interface Deploym…