ClickHouse Observability

Overview

Set up comprehensive monitoring for ClickHouse using built-in system tables,

Prometheus integration, Grafana dashboards, and alerting rules. The workflow

layers four signal sources: system.* tables (always available, zero

dependencies), a Prometheus scrape endpoint, application-level client

instrumentation, and alert rules that fire on the failure modes that actually

page an on-call — high error rate, latency creep, merge backlog, and resource

exhaustion.

Deep configs live in references/ so this file stays a fast, followable map.

Prerequisites

• ClickHouse instance with system.* table access
• Prometheus (or compatible: Grafana Alloy, Victoria Metrics)
• Grafana for dashboards
• AlertManager or PagerDuty for alerts

Instructions

Step 1: Query system tables for a health snapshot

Start with zero dependencies — the system.* tables already hold everything.

Run this for an instant server-health read:

SELECT
    (SELECT count() FROM system.processes) AS running_queries,
    (SELECT value FROM system.metrics WHERE metric = 'MemoryTracking') AS memory_bytes,
    (SELECT count() FROM system.merges) AS active_merges;

Query throughput, insert rates, and per-table part counts (the merge-health

signal), plus a full table of which system.* table to poll at what frequency:

system table queries & reference.

Step 2: Wire up Prometheus scraping

ClickHouse Cloud exposes a managed Prometheus endpoint (Basic auth with a Cloud

API key); self-hosted uses the built-in :9363 /metrics endpoint enabled in

config.xml. Write the scrape config to your prometheus.yml. Full Cloud +

self-hosted scrape configs and the config.xml block:

Prometheus scrape config & Grafana dashboards.

Step 3: Instrument the application client

Server metrics show what ClickHouse does; client metrics attribute latency,

error codes, and insert volume to your own code. Wrap queries in a prom-client

histogram/counter and expose /metrics. Full instrumentation + structured

logging: application-level instrumentation.

Step 4: Build Grafana dashboard panels

Panels for QPS, P50/P95/P99 latency, error rate, and insert throughput are in

the Grafana dashboards reference. Or import

the official community dashboard: https://grafana.com/grafana/dashboards/23415.

Step 5: Load alert rules

Write Prometheus alert rules for the five production failure modes (error rate,

latency, part count, memory, disk) to a rules file loaded by AlertManager. Full

rule set plus per-alert tuning notes: Prometheus alert rules.

Output

Applying this skill produces a set of monitoring config artifacts you write to

your infrastructure repo:

• prometheus.yml — scrape config targeting your ClickHouse endpoint
• clickhouse-alerts.yml — the five-rule alert group loaded by AlertManager
• A Grafana dashboard (imported ID 23415 or the custom JSON panels)
• Client instrumentation exposing clickhousequeryduration_seconds,

clickhousequeryerrorstotal, and clickhouseinsertrowstotal

• Ad-hoc system.* queries for on-demand health snapshots

Error Handling

Examples

Snapshot server health right now — run the Step 1 query against any instance

with system.* access; no exporter or scrape needed. See

system-tables.md for throughput and merge-health

variants.

Alert when merges fall behind — the ClickHouseTooManyParts rule fires when

a table exceeds 300 active parts for 10 minutes, the classic

inserts-outpacing-merges signal. Full rule + tuning guidance:

alerting.md.

Attribute slow queries to your service — wrap calls in instrumentedQuery()

so P95 latency and error codes land in Prometheus labeled by query type. See

instrumentation.md.

Resources

• Prometheus Integration
• ClickHouse Grafana Dashboard
• System Tables Reference
• Cloud Monitoring
• For incident response, see the clickhouse-incident-runbook skill.