Databricks Cluster Forensics

The operational SRE spine of the pack — what a Databricks engineer reaches for at

2 AM when the compute layer is broken or unexplained. It correlates a cluster's

live event stream across API surfaces to name the failure with its **actual error

code and its version-specific mitigation**, not "network problem, try again".

Overview

Six real compute-layer failures live in this skill; each has a deterministic

detector and an on-demand reference:

1. Cold-start long-tail (D01) — on VNet/VPC-injected workspaces a 5-minute

start randomly takes 20-35, and Databricks reports only the aggregate.

scripts/cluster-coldstart-forensics.py splits the PENDING window into stages

(provisioning / init-scripts / spark-startup) so you see which stage spiked.

1. Photon premium without the speedup (D02) — Photon silently falls back to

Spark on UDFs while the cluster still bills the ~2× Photon DBU premium for its

whole uptime. See references/photon-eligibility-and-fallback.md.

1. DBR upgrade landmines (D03/D04/D05) — 14.x moved the working dir to the

workspace filesystem (a ~500 MB cap that silently breaks large intermediate

writes); 15.1 removed DBFS-root library storage and JDK 11; 15.4 flipped a JDBC

calendar default. scripts/find-cwd-writes.py (AST) and scripts/scan-jar-jdk.sh

(bytecode target) are the pre-upgrade detectors; references/dbr-upgrade-paths.md

is the per-hop encyclopedia.

1. The launch-failure umbrella (D06) — CLOUDPROVIDERLAUNCH_FAILURE /

NPIPTUNNELSETUP_FAILURE each hide five distinct causes (subnet IP

exhaustion, DNS, NSG/security-group block, deleted VNet, cloud throttling).

references/termination-codes.md disambiguates them.

1. Spot shuffle aborts (D10) — a reclaimed spot node forces a shuffle

recompute; lose another mid-recompute and the stage exceeds

spark.stage.maxConsecutiveAttempts and the job aborts.

references/spot-vs-ondemand-decision.md is the config decision tree.

It is architecturally distinct from the v1 databricks-common-errors and

databricks-incident-runbook skills: those narrate. This one reads live cluster

events, buckets them deterministically (the arithmetic is in scripts/, never

eyeballed), fans out parallel root-cause threads via the cluster-event-investigator

subagent, and loads deep knowledge from references/ only when a symptom needs it.

Two data planes. Cluster control-plane

Databricks Cost Leak Hunter

Audits a Databricks workspace for real-dollar cost leaks — idle compute, jobs on

the wrong SKU, overprovisioned clusters, and the Photon premium paid without the

speedup — then emits a CFO-grokkable, dollar-ranked FinOps remediation report.

Overview

This skill finds where Databricks money is leaking and how much, in dollars, per

month. Confirmed-spend figures come from the customer's own system.billing.usage

joined to system.billing.list_prices — never an estimate. Two of the four

categories (overprovisioning, Photon premium) are explicitly modeled/at-risk

amounts, labeled as such so a CFO never confuses them with recoverable spend. The

skill surfaces four named leak categories, ranks them by monthly dollar impact, and

explains each root cause in FinOps language a CFO can act on without an engineer to

translate.

It is architecturally distinct from the v1 databricks-cost-tuning skill: that one

AUTHORS policy (creates cluster policies, spot configs). This one DETECTS leaks and

reports them, dollarized and ranked. The math is deterministic — bundled scripts in

scripts/ do the arithmetic so the agent never eyeballs numbers — and deep domain

knowledge lives in references/ loaded only when a leak needs it.

The skill uses two data planes. Dollar figures come from the **Databricks CLI

Statement Execution API** (databricks api post /api/2.0/sql/statements) reading

system.* — authenticated by the CLI's own DATABRICKSHOST+DATABRICKSTOKEN /

databricks auth login. The live config/event evidence that explains why a leak

exists (auto-termination setting, node type, autoscale floor, pool min_idle) comes

from the custom databricks-workspace-mcp control-plane tools — the one MCP

dependency. The SQL produces the number; the workspace MCP turns it into a verified,

single-config-change fix.

Prerequisites

Read access to the billing system tables is the hard dependency and the most common

failure. system.billing.usage requires a metastore-admin grant chain — the

skill detects a missing grant upfront and reports it, rather than failing mid-flow.

• Databricks Premium or Enterprise workspace with Unity Catalog (system tables

are UC-governed; not available on Standard).

• Metastore-admin grant chain on the billing schema, granted by a metastore

admin to the running principal:

• GRANT USE CATALOG ON CATALOG system TO
• GRANT USE SCHEMA ON SCHEMA system.billing TO

Databricks Streaming Guardian

The data-ops spine of the pack. Delta Lake, Liquid Clustering, Structured

Streaming, and DLT each ship a different set of foot-guns that fire most visibly

when production data flows through them at scale — and most of them are documented

platform decisions that surprise engineers, not bugs. This skill's job is

friction at trigger time (a hook that blocks the genuinely-irreversible op) plus

deterministic recovery when something already broke.

Overview

Twelve foot-guns, grouped by the surface that triggers them. Eleven are owned

outright (D01–D10, D12); the twelfth — D11, DLT rebuild cost — is shared with

databricks-cost-leak-hunter: this skill checks the rebuild cost as part of

pre-refresh safety, that skill owns ongoing cost optimization.

Delta write conflicts. D01 ConcurrentDeleteDeleteException — a manual

OPTIMIZE colliding with auto-compaction, which is silently enabled on any table

touched by MERGE/UPDATE/DELETE. D02 ConcurrentAppendException after moving

to Liquid Clustering — LC keeps file-set-level writer conflicts; a fan-out MERGE

breaks unless its predicate is narrowed to the clustering keys.

Streaming + checkpoint. D03 DELTAFILENOTFOUNDDETAILED — VACUUM deletes

files the checkpoint pins to. D04 silent checkpoint corruption / reset to batch 0.

D05 RocksDB state-store off-heap OOM (the heap looks fine while off-heap state pins

multi-GB). D12 DIFFERENTDELTATABLEREADBYSTREAMINGSOURCE — `CREATE OR

REPLACE` mints a new UUID and kills every active consumer.

Migration + evolution. D06 Liquid-Clustering migration's hidden full-rewrite

cost + downstream partition-predicate breakage. D07 time travel breaking silently

when VACUUM crosses the retention boundary. D10 Auto Loader

UnknownFieldException stopping the stream on every new column.

DLT. D08 the @dlt.table thread race (out-of-order registration). D09 full

refresh silently dropping data from a non-replayable source. D11 the rebuild cost

multiplier (checked before a full refresh; ongoing DLT cost is

databricks-cost-leak-hunter's job).

The hook (AP02/AP06 — this pack's only blocking hook). A PreToolUse hook

(hooks/streaming-guard-hook.py) intercepts a Bash command that runs DROP TABLE,

CREATE OR REPLACE TABLE, or VACUUM against a table and — only when it confirms

via system.streaming.query_progress that an active stream reads th

Databricks Unity Catalog Migration Pilot

Pilots a Hive Metastore → Unity Catalog migration end to end: it audits which

tables can migrate (and why the rest cannot), turns that audit into a

dependency-ordered execution plan, traces UC's two-level permission model when

access breaks, and picks an environment-isolation pattern for the

one-metastore-per-region constraint.

Overview

The September 30, 2026 forcing function. Databricks is deprecating the legacy

Hive Metastore. On the cutover date the hive_metastore catalog goes

read-only — no new writes, no schema changes — and stays available only for

read-through while you migrate. Everything that writes to HMS (jobs, DLT

pipelines, dashboards, external BI) must be pointed at a Unity Catalog table

before then, or it breaks. This is not optional and it is not fast: the blockers

below (legacy storage schemes, DBFS-root managed data, DENY-based ACLs) each

require a physical data move or a grant rewrite, so the work is measured in weeks,

not an afternoon.

> ⚠️ **Verify the exact date and scope against your workspace's current

> deprecation notice* (Databricks account console → Previews / Announcements*).

> Deprecation dates move; this skill's job is to get you ready well ahead of

> whatever the live date is — treat 2026-09-30 as the planning deadline, not an

> excuse to wait for it.

This skill does four things, in order, and each is deterministic where it can be:

1. Detect — scripts/audit-hms-readiness.py classifies every HMS table by its

storage URI into READY (cloud-native, UC-governable), BLOCKED (a named

un-migratable condition), or ORPHAN (a dangling HMS row pointing at deleted

storage). The LLM never eyeballs a URI; the script owns the verdict.

1. Plan — the migration-planner subagent turns the readiness CSV into a

per-table plan (SYNC / DEEP CLONE / rewrite / skip) **ordered by

dependencies** so a view never migrates before its base tables.

1. Trace — the uc-permission-tracer subagent walks UC's two-level access

model to answer "why can't user X read this?" in one pass — replacing ~90

minutes of doc-spelunking per ticket.

1. Isolate — /uc-env-pattern-picker recommends one of four dev/test/prod

isolation patterns under the single-metastore-per-region limit.

It is architecturally distinct from the v1 databricks-upgrade-migration and

databricks-migration-deep-dive skills: those narrate migration in prose. This

one runs a real