Databricks Bundle Medic

The deploy + infrastructure spine of the pack. Databricks Asset Bundles (DAB) is

immature tooling — the replacement for the deprecated dbx, with a moving bug list

across CLI versions — and the infrastructure operations around a deploy (encryption

keys, networking, workspace config) bite production platform teams hardest. This skill

turns the five worst deploy-time foot-guns into deterministic, reversible operations.

Overview

Five pains, all from the pack's deploy-ops research:

Asset Bundles (DAB). D4 — databricks bundle bind does NOT yet support UC catalogs

or external locations (databricks/cli#4842), so existing UC resources cannot come under

bundle management without hand-editing terraform.tfstate (hostile) or

destroy-and-recreate (impossible with dependent tables). D5 — "unexpected EOF reading

terraform.tfstate" on every redeploy after the first (databricks/cli#4986), which bricks

the bundle until you switch to DATABRICKSBUNDLEENGINE=direct. D6 — the schema

GRANT-ordering bug (databricks/cli#4573): the first deploy to a fresh workspace fails

with "User does not have CREATE TABLE on Schema", but the second succeeds because the

first applied the grants before dying.

Deploy-time infrastructure. D8 — rotating a workspace's customer-managed key (CMK)

requires terminating every cluster, instance pool, and SQL warehouse first: a hard

maintenance window. D9 — the PrivateLink trap: enabling PrivateLink for the control

plane is mistaken for "all traffic is private," but the data plane's S3 / STS / Kinesis

calls still traverse the NAT, billing NAT-processing + cross-AZ transfer until each gets

its own VPC endpoint.

The two hooks (this is the pack's only two-hook skill). A PreToolUse hook

(hooks/bundle-deploy-guard.py) runs before every databricks bundle deploy: it

validates the bundle's local terraform state parses as JSON, caches a timestamped

known-good backup as a recovery escape hatch, and warns loudly if the state is corrupt

or has shrunk (the D5 signature). A PostToolUse hook (hooks/bundle-grant-retry.py)

watches a deploy's output and — ONLY on the exact D6 "does not have … on Schema"

signature — adds context recommending one retry, with the reason. Both are advisory:

the pre-hook never blocks a deploy, and the post-hook never masks a real error — it

surfaces the diagnosis and stops recommending retries if the same failure persists.

Deterministic work lives in scripts/; deep knowledge in references/. The

import-uc-resource-to-bundle.py D4 workaround is self-deprecating — it exists only

until #4842 closes and says so. Control-plane state comes from the

databricks-workspace-mcp server (externallocationslist, storagecredentialslist)

or the CLI; advisory-mode fallback accepts pasted input.

Prerequisites

• Databricks CLI authenticated, plus terraform and jq on PATH — for the bundle,

import, and state operations.

• databricks-workspace-mcp registered (optional) — for externallocationslist

and storagecredentialslist. Absent → the CLI (`databricks external-locations

list`) or advisory mode on pasted input.

• Account-level OAuth M2M (service principal) for CMK rotation — the D8 operations

are account-scoped; a workspace PAT is insufficient. AWS/Azure/GCP CLI for the

cloud-side key + VPC-endpoint work.

• The two hooks are plugin-level — installed with the pack. The pre-hook is silent

unless a state looks corrupt; the post-hook is silent unless the exact D6 error fires.

Instructions

Pick the flow by symptom. Always name the exact error string / issue id —

unexpected EOF reading terraform.tfstate (#4986), `does not have CREATE TABLE on

Schema (#4573), the bundle bind` UC gap (#4842) — an operator greps for those.

Step 1: A bundle deploy that fails on state (D5)

"unexpected EOF reading terraform.tfstate" after the first deploy is #4986. The

bundle-deploy-guard PreToolUse hook already cached a known-good backup; restore it,

or switch engines:

DATABRICKS_BUNDLE_ENGINE=direct databricks bundle deploy -t "$TARGET"

Engine tradeoffs + the migration steps:

${CLAUDESKILLDIR}/references/bundle-engine-tradeoffs.md.

Step 2: First deploy fails "does not have CREATE TABLE on Schema" (D6)

This is the transient GRANT-ordering bug (#4573). The bundle-grant-retry PostToolUse

hook flags it; re-run the SAME deploy exactly once — the failed first pass already

applied the grants. If the identical error persists after one retry, it is NOT this

transient — treat it as a real missing privilege and stop retrying.

Step 3: A UC resource that will not bind (D4)

databricks bundle bind cannot take a UC catalog or external location yet (#4842).

Generate a review-first Terraform import plan (never hand-edit state):

python3 "${CLAUDE_SKILL_DIR}/scripts/import-uc-resource-to-bundle.py" \
  --type external_location --name raw_zone --resource-key raw_zone_loc

The three workarounds, ranked by risk:

${CLAUDESKILLDIR}/references/uc-resource-binding-workarounds.md.

Step 4: Rotate a customer-managed key (D8)

CMK rotation needs the whole workspace drained. Inventory the running compute (via

externallocationslist / the CLI), then plan the drain — dry-run by default:

databricks clusters list --output json > /tmp/inv-clusters.json   # + warehouses, pools
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --inventory inv.json          # dry-run
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --inventory inv.json --execute --manifest drain.json
# ... rotate the CMK per cloud, then:
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --resume drain.json

Per-cloud playbooks (AWS/Azure/GCP):

${CLAUDESKILLDIR}/references/cmk-rotation-by-cloud.md.

Step 5: Audit the PrivateLink cost leak (D9)

PrivateLink covers the control plane only. Audit the data-plane VPC for the S3/STS/Kinesis

endpoints and emit remediation Terraform for any that are missing:

python3 "${CLAUDE_SKILL_DIR}/scripts/audit-vpc-endpoints.py" \
  --present s3 --vpc-id vpc-abc --region us-east-1 --emit-terraform

The full per-service leak/fix map:

${CLAUDESKILLDIR}/references/cost-leak-map.md.

Output

• A hook backup + advisory — before each deploy, a validated known-good state

backup and, if the state is corrupt/shrunk, the #4986 recovery message.

• A retry recommendation — on the exact D6 signature, a one-retry recommendation

with the reason (never a masked error).

• A UC import plan — the resources: YAML + Terraform import block/command to

adopt an existing UC resource, with the self-deprecation notice.

• A drain / resume plan — the idempotent list of compute to terminate for CMK

rotation and the manifest to resume exactly what was drained.

• A VPC-endpoint verdict — SAFE or a COST LEAK finding naming the missing

S3/STS/Kinesis endpoints, with remediation Terraform.

Error Handling

Examples

Example 1: "My second bundle deploy fails with unexpected EOF reading terraform.tfstate."

D5 (#4986). The bundle-deploy-guard hook already cached a known-good state before the

deploy — restore it, then redeploy with DATABRICKSBUNDLEENGINE=direct (the direct

engine never reads the state file, so the EOF class cannot fire).

Example 2: "First deploy to a fresh workspace: User does not have CREATE TABLE on Schema."

D6 (#4573). The bundle-grant-retry hook recognises the exact signature and recommends

one retry — the failed first deploy applied the schema grants, so the second succeeds.

Example 3: "I need to bring an existing external location under my bundle."

D4 (#4842) — bundle bind can't. `import-uc-resource-to-bundle.py --type

external_location emits the resources: YAML plus a Terraform import` block to adopt

it without recreating it or hand-editing state. The script self-deprecates when #4842 closes.

Example 4: "We turned on PrivateLink but the NAT bill went up."

D9. audit-vpc-endpoints.py --present s3 finds STS and Kinesis have no Interface

endpoint, so credential-vending and log traffic still cross the NAT — it emits the

remediation Terraform for both.

Resources

• ${CLAUDESKILLDIR}/references/uc-resource-binding-workarounds.md — the D4 bundle bind gap + 3 ranked workarounds (#4842).
• ${CLAUDESKILLDIR}/references/bundle-engine-tradeoffs.md — Terraform vs direct engine, the D5 EOF bug (#4986).
• ${CLAUDESKILLDIR}/references/cmk-rotation-by-cloud.md — AWS/Azure/GCP CMK rotation playbooks + the drain requirement (D8).
• ${CLAUDESKILLDIR}/references/cost-leak-map.md — the PrivateLink S3/STS/Kinesis NAT cost leak + fix map (D9).
• ${CLAUDESKILLDIR}/scripts/import-uc-resource-to-bundle.py — self-deprecating UC-resource import-plan generator (D4).
• ${CLAUDESKILLDIR}/scripts/drain-workspace.py — idempotent drain + resume for CMK rotation (D8).
• ${CLAUDESKILLDIR}/scripts/audit-vpc-endpoints.py — VPC-endpoint audit + remediation Terraform (D9).
• ${CLAUDESKILLDIR}/hooks/bundle-deploy-guard.py — PreToolUse state backup + validation (D5).
• ${CLAUDESKILLDIR}/hooks/bundle-grant-retry.py — PostToolUse D6 retry recommendation (D6).
• Databricks Asset Bundles · Customer-managed keys · PrivateLink