Glean Observability

Overview

Set up comprehensive observability for Glean integrations.

Prerequisites

• Prometheus or compatible metrics backend
• OpenTelemetry SDK installed
• Grafana or similar dashboarding tool
• AlertManager configured

Metrics Collection

Key Metrics

Prometheus Metrics

import { Registry, Counter, Histogram, Gauge } from 'prom-client';

const registry = new Registry();

const requestCounter = new Counter({
  name: 'glean_requests_total',
  help: 'Total Glean API requests',
  labelNames: ['method', 'status'],
  registers: [registry],
});

const requestDuration = new Histogram({
  name: 'glean_request_duration_seconds',
  help: 'Glean request duration',
  labelNames: ['method'],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [registry],
});

const errorCounter = new Counter({
  name: 'glean_errors_total',
  help: 'Glean errors by type',
  labelNames: ['error_type'],
  registers: [registry],
});

Instrumented Client

async function instrumentedRequest<T>(
  method: string,
  operation: () => Promise<T>
): Promise<T> {
  const timer = requestDuration.startTimer({ method });

  try {
    const result = await operation();
    requestCounter.inc({ method, status: 'success' });
    return result;
  } catch (error: any) {
    requestCounter.inc({ method, status: 'error' });
    errorCounter.inc({ error_type: error.code || 'unknown' });
    throw error;
  } finally {
    timer();
  }
}

Distributed Tracing

OpenTelemetry Setup

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('glean-client');

async function tracedGleanCall<T>(
  operationName: string,
  operation: () => Promise<T>
): Promise<T> {
  return tracer.startActiveSpan(`glean.${operationName}`, async (span) => {
    try {
      const result = await operation();
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (error: any) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      span.recordException(error);
      throw error;
    } finally {
      span.end();
    }
  });
}

Logging Strategy

Structured Logging

import pino from 'pino';

const logger = pino({
  name: 'glean',
  level: process.env.LOG_LEVEL || 'info',
});

function logGleanOperation(
  operation: string,
  data: Record<string, any>,
  duration: number
) {
  logger.info({
    service: 'glean',
    operation,
    duration_ms: duration,
    ...data,
  });
}

Alert Configuration

Prometheus AlertManager Rules

# glean_alerts.yaml
groups:
  - name: glean_alerts
    rules:
      - alert: GleanHighErrorRate
        expr: |
          rate(glean_errors_total[5m]) /
          rate(glean_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Glean error rate > 5%"

      - alert: GleanHighLatency
        expr: |
          histogram_quantile(0.95,
            rate(glean_request_duration_seconds_bucket[5m])
          ) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Glean P95 latency > 2s"

      - alert: GleanDown
        expr: up{job="glean"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Glean integration is down"

Dashboard

Grafana Panel Queries

{
  "panels": [
    {
      "title": "Glean Request Rate",
      "targets": [{
        "expr": "rate(glean_requests_total[5m])"
      }]
    },
    {
      "title": "Glean Latency P50/P95/P99",
      "targets": [{
        "expr": "histogram_quantile(0.5, rate(glean_request_duration_seconds_bucket[5m]))"
      }]
    }
  ]
}

Instructions

Step 1: Set Up Metrics Collection

Implement Prometheus counters, histograms, and gauges for key operations.

Step 2: Add Distributed Tracing

Integrate OpenTelemetry for end-to-end request tracing.

Step 3: Configure Structured Logging

Set up JSON logging with consistent field names.

Step 4: Create Alert Rules

Define Prometheus alerting rules for error rates and latency.

Output

• Metrics collection enabled
• Distributed tracing configured
• Structured logging implemented
• Alert rules deployed

Error Handling

Examples

Quick Metrics Endpoint

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', registry.contentType);
  res.send(await registry.metrics());
});

Resources

• Prometheus Best Practices
• OpenTelemetry Documentation
• Glean Observability Guide

Next Steps

For incident response, see glean-incident-runbook.