CoreWeave GPU Node Forensics

> Community-contributed. Not affiliated with, endorsed by, or sponsored by

> CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

> "NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are

> cited from NVIDIA's public documentation.

Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one

grounded move — **reschedule, reset-gpu, reboot-node, rma, watch, or

app-bug-not-hardware** — from an NVIDIA Xid code or a pasted dmesg /

nvidia-smi blob. The classification is deterministic: a bundled script does the

mapping so the agent never guesses whether a card is dead, degraded, or fine.

Overview

A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars

and days of wall-clock gone — because one rank stalls the whole collective. The

expensive mistakes are triage mistakes: RMAing a healthy card for an app bug,

restarting a job onto a GPU whose memory error was uncontained, or manually

uncordoning a node the lifecycle controller is trying to replace. This skill kills

that ambiguity.

The decision logic is grounded in the NVIDIA Xid error catalog

() and CoreWeave's node-lifecycle /

cordon behavior. The math-of-the-matter — which Xid means what, and how

row-remapper state overrides it — lives in scripts/triage.py as a table the LLM

does not get to re-litigate. Deep domain knowledge (the full code→action table,

the row-remap decision, the cordon rules) loads from references/ on demand.

The headline is the Xid 94-vs-95 split. A contained memory error (94) cost

you one job restart on a healthy node; an uncontained one (95) means the GPU

could not isolate the fault and everything it touched is suspect. Getting that one

bit wrong is the difference between a 30-second reschedule and a run that quietly

trained on corrupt gradients. The script decides it; the skill never eyeballs it.

This skill is diagnostic, not destructive: it recommends the cordon / drain /

reset / RMA next-step but its tools are scoped read-only (nvidia-smi -q,

kubectl get, dmesg) — it never runs a reset, a reboot, or an uncordon itself.

Prerequisites

• The failure evidence. Either an Xid number, or a pasted dmesg /

nvidia-smi dump. The skill works from a paste alone — no live cluster access

required — which is the common case (an operator pastes what the run's logs

showed).

• Optional live access for corroboration: kubectl context on the CoreWeave

cluster (read-only is enough), and nvidia-smi on the node. If neither is

available the skill still triages from the paste.

• python3 to run the deterministic classifier (scripts/triage.py, stdlib

only — no dependencies).

No secrets are handled. All commands are read-only queries.

Instructions

The pipeline is capture → classify → act. The classifier is authoritative for

the verdict; references/ supplies the "why" when a case needs depth.

Step 1: Capture the evidence

If the user has not already pasted it, ask for (or read) the fault signal. The two

richest sources:

dmesg -T | grep -i xid                       # the Xid line(s) with timestamps
nvidia-smi -q -d ROW_REMAPPER,ECC,PERFORMANCE # remap state, ECC counts, throttle reasons

On a CoreWeave node you can also check who owns any cordon before acting:

kubectl get node NODE -o json | jq '{unschedulable: .spec.unschedulable, taints: .spec.taints}'

Step 2: Run the deterministic triage

Feed the Xid code, or the whole blob, to the classifier. **Do not classify by

hand** — the script owns the Xid→action mapping and the row-remap override.

# From an Xid code:
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 95

# With row-remapper state (Xid 63/64 or a DBE):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 63 --pending yes
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 48 --remap-failure yes

# From a pasted dmesg / nvidia-smi blob (file or stdin):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --blob /path/to/dmesg.txt
dmesg -T | grep -i xid | python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py"

The script emits a VERDICT block plus a JSON object

({classification, severity, action, why, nextcommand, cordonrule?, unverified?}).

Add --json for machine-readable output only.

When a blob carries several Xids, the most severe one governs the move and the

rest are reported as cooccurringxids — a co-occurring app-side Xid 43 next to a

hardware Xid 79 does not soften the "reboot the node" verdict.

Step 3: Read the verdict and act on the action

Present the action and the next_command to the user in plain language. The six

actions and what each means live in

${CLAUDESKILLDIR}/references/xid-triage-table.md.

Load it when the user wants the full table or asks about an Xid not in the summary.

• reschedule — restart the failed rank; the node stays in service.
• reset-gpu — drain the GPU and reset it; re-run any job that shared it.
• reboot-node — the card is off the bus; only a bare-metal reboot returns it.
• rma — terminal hardware fault; the card must be replaced.
• watch — correctable / trending; monitor, do not act yet.
• app-bug-not-hardware — the app faulted, not the GPU. Do NOT RMA.

Step 4: The 94-vs-95 headline (load-bearing — get this right)

If the Xid is 94 or 95, state the containment explicitly, because the two look

almost identical in the logs and lead to opposite actions:

• Xid 94 (CONTAINED) → reschedule. The error was isolated to the app's

context; the GPU and node are healthy. Restart the job. Do not reset or RMA.

• Xid 95 (UNCONTAINED) → reset-gpu. The GPU could not isolate it; every

context it touched is suspect. Drain, reset, and re-run anything that shared the

card — otherwise the run may continue on corrupt state.

Step 5: Row-remap (Xid 63 / 64 / 48) — routine vs terminal

For any ECC/DBE or row-remap Xid, the nvidia-smi -q -d ROW_REMAPPER fields

override the base verdict. Pass them in (--pending, --remap-failure) and let

the script decide. The rule:

• Remapping Failure Occurred: Yes → rma (terminal — sparing failed).
• Pending: Yes (no failure) → reset-gpu (routine — applies on reset).

Full logic + how to read the histogram:

${CLAUDESKILLDIR}/references/row-remap-decision.md.

Step 6: The cordon hard rule (never violate)

Whenever the action is a hardware move (reset-gpu, reboot-node, rma), the

verdict carries a cordon_rule. Surface it verbatim:

> Never manually uncordon a CoreWeave health cordon — the node-lifecycle

> controller owns cordon/uncordon and is driving remediation. Uncordoning

> re-admits the run onto known-bad hardware and races the controller.

Determine cordon provenance before touching schedulability. Full guidance:

${CLAUDESKILLDIR}/references/cordon-rules.md.

Output

• A verdict — the deterministic `{classification, severity, action, why,

next_command}` for the governing signal, presented in plain language with the

single next command an operator runs.

• The 94-vs-95 call stated explicitly when either fires — contained →

reschedule, uncontained → reset — never conflated.

• The cordon rule attached to every hardware action, verbatim, so nobody

uncordons a node the controller is replacing.

• Co-occurring Xids listed when a blob carried several, with the most-severe

one named as governing.

• [unverified] hedges surfaced honestly — e.g. CoreWeave's exact auto-RMA

thresholds and health-cordon taint strings are not publicly published, and the

skill says so rather than inventing a number.

Error Handling

Situation	Cause	Response
Unmapped Xid	Xid not in the triage table	Script returns a conservative reset-gpu default, flags it [unverified], and points at the NVIDIA catalog. Do not RMA on an unmapped Xid.
Xid 94 vs 95 ambiguity	Both are "ECC error" in casual logs	Never infer from prose — use the exact code. 94 = contained (reschedule), 95 = uncontained (reset).
Row-remap without an Xid	Only nvidia-smi pasted	Script reads Remapping Failure Occurred / Pending and decides from those alone.
App-side Xid mistaken for hardware	Xid 13/31/43/45	Classed app-bug-not-hardware; the fix is compute-sanitizer on the job, never an RMA.
Multiple Xids in one blob	Cascade (e.g. 43 then 79)	Most-severe governs; others listed as cooccurringxids.
Empty / no signal	Blob has no Xid, remap, or thermal line	Script returns watch and asks for fresh dmesg + nvidia-smi -q evidence.
Thermal throttle, not a fault	HW/SW Thermal Slowdown : Active	Classed watch — healthy but hot; investigate cooling, not the card.
Tempted to uncordon a stalled node	Health cordon owned by the controller	Refuse. Surface the cordon rule; let remediation run.

Examples

Example 1: "Xid 94 on one rank — do I need to replace the GPU?"

VERDICT: Xid 94 — Contained ECC/memory error [severity: HIGH]
ACTION:  reschedule
WHY:     The error was CONTAINED to the faulting application's context — the GPU
         and node are healthy. Just reschedule/restart the job; no reset or RMA needed.
NEXT:    Restart or reschedule the failed rank; the node stays in service.

No replacement. The containment did its job — restart the rank and keep the node.

Example 2: "Xid 95 — same run, different node"

VERDICT: Xid 95 — Uncontained ECC/memory error [severity: CRITICAL]
ACTION:  reset-gpu
WHY:     The error was UNCONTAINED — the GPU could not isolate it, so every context
         it touched is suspect. Drain the GPU and reset it; treat all in-flight work as corrupt.
NEXT:    Cordon + drain, GPU-reset (nvidia-smi -r) or node reset; re-run any job that shared this GPU.
CORDON:  Do NOT manually uncordon a CoreWeave health cordon — the node-lifecycle
         controller owns cordon/uncordon. Let it drain and replace the node.

Opposite of Example 1 despite looking identical in the logs: drain, reset, and

re-run the shared work — do not just restart.

Example 3: "GPU fell off the bus"

python3 scripts/triage.py --xid 79 → reboot-node. The card is unreachable on

the PCIe bus and returns only after a bare-metal reboot; expect the controller to

cordon and reboot — let it, and RMA only if it recurs after the reboot.

Example 4: "Xid 63 with a remap failure"

python3 scripts/triage.py --xid 63 --remap-failure yes → rma. The remapper

tried to swap in a spare row and physically could not — terminal. Cordon, drain,

open the RMA. (The script flags that CoreWeave's exact auto-RMA threshold is

[unverified].)

Example 5: "Xid 43 killed my job"

python3 scripts/triage.py --xid 43 → app-bug-not-hardware. Software-induced

fault — the fix is in the application (run it under compute-sanitizer), not an RMA.

Pulling the card would waste healthy hardware and never fix the job.

Resources

• ${CLAUDESKILLDIR}/references/xid-triage-table.md — the full Xid→action table, the six actions, the 94-vs-95 distinction, cited to the NVIDIA Xid catalog.
• ${CLAUDESKILLDIR}/references/row-remap-decision.md — the ROW_REMAPPER routine-vs-terminal decision and how to read the histogram.
• ${CLAUDESKILLDIR}/references/cordon-rules.md — CoreWeave health-cordon ownership and the safe operator path per action.
• NVIDIA Xid Errors reference
• NVIDIA GPU Memory Error Management (row remapping)