Databricks Streaming Guardian

The data-ops spine of the pack. Delta Lake, Liquid Clustering, Structured

Streaming, and DLT each ship a different set of foot-guns that fire most visibly

when production data flows through them at scale — and most of them are documented

platform decisions that surprise engineers, not bugs. This skill's job is

friction at trigger time (a hook that blocks the genuinely-irreversible op) plus

deterministic recovery when something already broke.

Overview

Twelve foot-guns, grouped by the surface that triggers them. Eleven are owned

outright (D01–D10, D12); the twelfth — D11, DLT rebuild cost — is shared with

databricks-cost-leak-hunter: this skill checks the rebuild cost as part of

pre-refresh safety, that skill owns ongoing cost optimization.

Delta write conflicts. D01 ConcurrentDeleteDeleteException — a manual

OPTIMIZE colliding with auto-compaction, which is silently enabled on any table

touched by MERGE/UPDATE/DELETE. D02 ConcurrentAppendException after moving

to Liquid Clustering — LC keeps file-set-level writer conflicts; a fan-out MERGE

breaks unless its predicate is narrowed to the clustering keys.

Streaming + checkpoint. D03 DELTAFILENOTFOUNDDETAILED — VACUUM deletes

files the checkpoint pins to. D04 silent checkpoint corruption / reset to batch 0.

D05 RocksDB state-store off-heap OOM (the heap looks fine while off-heap state pins

multi-GB). D12 DIFFERENTDELTATABLEREADBYSTREAMINGSOURCE — `CREATE OR

REPLACE` mints a new UUID and kills every active consumer.

Migration + evolution. D06 Liquid-Clustering migration's hidden full-rewrite

cost + downstream partition-predicate breakage. D07 time travel breaking silently

when VACUUM crosses the retention boundary. D10 Auto Loader

UnknownFieldException stopping the stream on every new column.

DLT. D08 the @dlt.table thread race (out-of-order registration). D09 full

refresh silently dropping data from a non-replayable source. D11 the rebuild cost

multiplier (checked before a full refresh; ongoing DLT cost is

databricks-cost-leak-hunter's job).

The hook (AP02/AP06 — this pack's only blocking hook). A PreToolUse hook

(hooks/streaming-guard-hook.py) intercepts a Bash command that runs DROP TABLE,

CREATE OR REPLACE TABLE, or VACUUM against a table and — only when it confirms

via system.streaming.query_progress that an active stream reads that table —

blocks it with a message naming the consumers and the pain. It is precise by

design: it matches only real SQL-execution surfaces (never a git commit mentioning

"drop table"), and it fails open — if it cannot verify consumers, it allows

rather than false-block. Blocking is reserved for the genuinely irreversible.

Deterministic work lives in scripts/; deep knowledge in references/; the

Liquid-Clustering predicate rewrite in the merge-rewriter subagent. Two data

planes: the databricks-workspace-mcp control plane (cluster/pipeline events) and

the CLI Statement Execution API for system.* reads. Either absent → advisory mode

on pasted input.

Prerequisites

• databricks-workspace-mcp registered — for clusters_events (RocksDB OOM

correlation) and pipelines_get (DLT event log). Absent → advisory mode.

• Databricks CLI authenticated + jq, and DATABRICKSWAREHOUSEID set —

for the system.streaming.query_progress reads the hook and recovery flows use.

The hook fails open (allows) if these are absent, so it never false-blocks.

• The hook is a plugin-level PreToolUse hook — it runs on Bash commands once

the pack is installed. It is silent on everything except a confirmed-unsafe

destructive op.

Instructions

Pick the flow by symptom. **Always name the exact, searchable Databricks error

string** — ConcurrentAppendException, ConcurrentDeleteDeleteException,

DELTAFILENOTFOUNDDETAILED, DIFFERENTDELTATABLEREADBYSTREAMINGSOURCE,

UnknownFieldException — even when the user paraphrases it or gives a short form;

the full code is what an operator greps logs and docs for.

Step 1: Before a destructive op (the hook does this automatically)

Running DROP TABLE / CREATE OR REPLACE TABLE / VACUUM on a table? The hook

checks for active streaming consumers first and blocks if any exist. To check

manually, query system.streaming.query_progress for a stream whose

source_description names the table. If consumers exist: do NOT `CREATE OR

REPLACE (use ALTER/in-place — D12) and do NOT VACUUM` below the consumers'

checkpoint lag (D03/D07). See

${CLAUDESKILLDIR}/references/checkpoint-recovery.md.

Step 2: A Delta write conflict (D01, D02)

• ConcurrentDeleteDeleteException (D01) — before a manual OPTIMIZE, probe

the table for auto-compaction:

  bash "${CLAUDE_SKILL_DIR}/scripts/pre-optimize-check.sh" --table main.sales.orders

If it reports COLLISION RISK, don't run manual OPTIMIZE (or disable

auto-compaction first). Details:

${CLAUDESKILLDIR}/references/concurrency-conflicts.md.

• ConcurrentAppendException on a Liquid-Clustering table (D02) — hand the

failing MERGE to the merge-rewriter subagent; it fetches the target's

clustering keys via DESCRIBE DETAIL and narrows the ON predicate so writers

touch disjoint file sets.

Step 3: A broken streaming source (D03, D04, D12)

Map the symptom to the failure class and its exact error code, then get the

recovery tier from the decision tree:

• file-not-found → DELTAFILENOTFOUNDDETAILED (VACUUM deleted pinned files — D03)
• uuid-changed → DIFFERENTDELTATABLEREADBYSTREAMINGSOURCE (CREATE OR REPLACE minted a new UUID — D12)
• checkpoint-reset → silent batchId regression / checkpoint corruption (D04)
• transient → a restartable blip with an intact checkpoint

python3 "${CLAUDE_SKILL_DIR}/scripts/recover-streaming-source.py" \
  --failure file-not-found --time-travel yes    # or uuid-changed / checkpoint-reset / transient

It echoes the canonical error code and recommends SAFE_RESTART /

REPROCESSFROMOFFSET / RESTOREFROMTIMETRAVEL / FULLRESET_BACKFILL with the

data-loss tradeoff stated up front. Name that full code in your answer — not just

the short class. The full

three-tier reasoning is in

${CLAUDESKILLDIR}/references/checkpoint-recovery.md.

Step 4: RocksDB state-store OOM (D05)

A driver/executor OOM while the JVM heap looks healthy points at off-heap RocksDB

state. Correlate the OOM to state size with clusters_events, then bound the

memory and enable changelog checkpointing per

${CLAUDESKILLDIR}/references/rocksdb-state-store-tuning.md.

Step 5: Auto Loader schema evolution (D10)

A stream stopping with UnknownFieldException on a new column is the default

addNewColumns mode. Choose the mode deliberately (evolve-and-restart vs

rescue's silent widening) and pin types with schemaHints per

${CLAUDESKILLDIR}/references/autoloader-schema-evolution.md.

Step 6: DLT rebuild safety (D08, D09, D11)

Before a DLT full refresh, confirm every source is replayable (a Kafka topic past

retention or a truncate-and-load source loses data on refresh — D09) and that

@dlt.table registration is deterministic (the thread race — D08). Read the DLT

event log with pipelines_get; the checklist is in

${CLAUDESKILLDIR}/references/dlt-rebuild-safety.md.

Output

• A hook decision — a destructive op on a streamed-from table is blocked with

the active consumers named and the pain (D12 / D03-D07) explained; everything

else passes silently.

• A pre-OPTIMIZE verdict — SAFE or COLLISION RISK (auto-compaction on) with the

disable-or-serialize fix.

• A rewritten MERGE — the LC clustering-key-scoped predicate (from

merge-rewriter) that stops ConcurrentAppendException.

• A recovery recommendation — the recovery tier + steps + the data-loss risk,

for the specific failure class.

• A tuning / mode / refresh-safety recommendation — RocksDB bounds, Auto Loader

mode, or the DLT full-refresh checklist, from the matching reference.

Error Handling

Error	Cause	Solution
ConcurrentDeleteDeleteException	Manual OPTIMIZE races auto-compaction (D01)	Run pre-optimize-check.sh; don't manually OPTIMIZE an auto-compacted table, or disable auto-compact first.
ConcurrentAppendException on an LC table	MERGE predicate not scoped to clustering keys (D02)	Route the MERGE to merge-rewriter; narrow the ON predicate to the clustering keys.
DELTAFILENOTFOUNDDETAILED	VACUUM deleted checkpoint-pinned files (D03)	recover-streaming-source.py --failure file-not-found; restore via time travel if in retention, else reprocess/reset.
DIFFERENTDELTATABLEREADBYSTREAMINGSOURCE	CREATE OR REPLACE minted a new UUID (D12)	The old checkpoint is dead — new checkpoint + backfill; the hook prevents this going forward.
Driver OOM, heap looks fine	Off-heap RocksDB state (D05)	Bound state-store memory + changelog checkpointing; size state with a watermark.
UnknownFieldException, stream stopped	Auto Loader addNewColumns default (D10)	Restart to evolve (idempotent sink), or choose rescue/schemaHints deliberately.
Hook allowed a destructive op with a warning	Could not verify consumers (no CLI/warehouse)	Advisory — the hook fails open; verify system.streaming.query_progress manually before running it.

Examples

Example 1: "About to CREATE OR REPLACE a table other jobs stream from."

The PreToolUse hook fires, confirms 2 active consumers via

system.streaming.query_progress, and blocks with: *"CREATE OR REPLACE mints a

new UUID → both consumers die with DIFFERENTDELTATABLEREADBYSTREAMINGSOURCE;

use ALTER / in-place."*

Example 2: "My MERGE into a Liquid-Clustering table fails with ConcurrentAppendException."

The merge-rewriter subagent reads the target's clustering keys via `DESCRIBE

DETAIL and rewrites the ON` predicate to include them, so concurrent writers

touch disjoint file sets — the exception stops without serializing the jobs.

Example 3: "My stream died with DELTAFILENOT_FOUND after a VACUUM."

recover-streaming-source.py --failure file-not-found --time-travel yes →

RESTOREFROMTIME_TRAVEL (no data loss): restore the source to a pre-VACUUM version,

restart on the existing checkpoint, then align VACUUM retention with the checkpoint lag.

Example 4: "Before I OPTIMIZE this table."

pre-optimize-check.sh --table main.sales.orders reports COLLISION RISK because

delta.autoOptimize.autoCompact is on — so the skill recommends letting

auto-compaction do it, or disabling it for the maintenance window first.

Resources

• ${CLAUDESKILLDIR}/references/concurrency-conflicts.md — Delta OCC, auto-compaction collisions (D01), Liquid-Clustering writer conflicts (D02).
• ${CLAUDESKILLDIR}/references/checkpoint-recovery.md — the three-tier streaming checkpoint recovery (D03/D04/D12).
• ${CLAUDESKILLDIR}/references/rocksdb-state-store-tuning.md — bounded off-heap state + changelog checkpointing (D05).
• ${CLAUDESKILLDIR}/references/autoloader-schema-evolution.md — the schema-evolution modes + schemaHints (D10).
• ${CLAUDESKILLDIR}/references/dlt-rebuild-safety.md — DLT thread race, full-refresh data loss, tier cost (D08/D09/D11).
• ${CLAUDESKILLDIR}/scripts/pre-optimize-check.sh — auto-compaction collision probe.
• ${CLAUDESKILLDIR}/scripts/recover-streaming-source.py — 4-way recovery decision tree.
• ${CLAUDESKILLDIR}/hooks/streaming-guard-hook.py — the PreToolUse block for destructive ops on streamed-from tables.
• ${CLAUDESKILLDIR}/agents/merge-rewriter.md — rewrites a MERGE predicate for Liquid Clustering.
• Delta concurrency control · Structured Streaming production · Auto Loader schema evolution