Glean Performance Tuning

Overview

Glean's enterprise search API handles search queries across multiple connectors, bulk document indexing, and connector sync throughput. Search latency compounds when querying across dozens of datasources simultaneously. Large indexing jobs (10K+ documents) require careful batching to avoid rate limits and maintain connector sync schedules. Optimizing batch sizes, caching frequent search results, and tuning connector configurations reduces search P95 latency and keeps indexing pipelines within SLA windows.

Caching Strategy

const cache = new Map<string, { data: any; expiry: number }>();
const TTL = { search: 60_000, suggestions: 30_000, datasources: 600_000 };

async function cached(key: string, ttlKey: keyof typeof TTL, fn: () => Promise<any>) {
  const entry = cache.get(key);
  if (entry && entry.expiry > Date.now()) return entry.data;
  const data = await fn();
  cache.set(key, { data, expiry: Date.now() + TTL[ttlKey] });
  return data;
}
// Search results expire fast (1 min). Datasource metadata is stable (10 min).

Batch Operations

import PQueue from 'p-queue';
const BATCH_SIZE = 100;

async function indexDocsBatched(glean: any, dsName: string, docs: any[]) {
  const batches = [];
  for (let i = 0; i < docs.length; i += BATCH_SIZE) batches.push(docs.slice(i, i + BATCH_SIZE));
  const queue = new PQueue({ concurrency: 3, interval: 500 });
  await Promise.all(batches.map(batch =>
    queue.add(() => glean.indexDocuments(dsName, batch))
  ));
}

Connection Pooling

import { Agent } from 'https';
const agent = new Agent({ keepAlive: true, maxSockets: 15, maxFreeSockets: 5, timeout: 30_000 });
// High socket count for parallel indexing across multiple datasources

Rate Limit Management

async function withGleanRateLimit(fn: () => Promise<any>): Promise<any> {
  try { return await fn(); }
  catch (err: any) {
    if (err.status === 429) {
      const retryMs = parseInt(err.headers?.['retry-after'] || '5') * 1000;
      await new Promise(r => setTimeout(r, retryMs));
      return fn();
    }
    throw err;
  }
}

Monitoring

const metrics = { searches: 0, indexOps: 0, cacheHits: 0, p95LatencyMs: 0, errors: 0 };
const latencies: number[] = [];
function trackSearch(startMs: number, cached: boolean) {
  const lat = Date.now() - startMs; latencies.push(lat); metrics.searches++;
  if (cached) metrics.cacheHits++;
  latencies.sort((a, b) => a - b);
  metrics.p95LatencyMs = latencies[Math.floor(latencies.length * 0.95)] || 0;
}

Performance Checklist

• [ ] Batch indexing calls at 100 docs per request with 3 concurrent wo

Glean Rate Limits

Overview

Glean's APIs split into two tiers: the Indexing API for pushing documents into the search corpus, and the Client API for executing searches. The Indexing API handles bulk document ingestion at approximately 100 requests per minute, while search queries are capped at 60 per minute per token. Organizations indexing large knowledge bases (100K+ documents from Confluence, Notion, or internal wikis) must implement careful batching to avoid 429 responses that can stall multi-hour ingestion pipelines.

Rate Limit Reference

Rate Limiter Implementation

class GleanRateLimiter {
  private tokens: number;
  private lastRefill: number;
  private readonly max: number;
  private readonly refillRate: number;
  private queue: Array<{ resolve: () => void }> = [];

  constructor(maxPerMinute: number) {
    this.max = maxPerMinute;
    this.tokens = maxPerMinute;
    this.lastRefill = Date.now();
    this.refillRate = maxPerMinute / 60_000;
  }

  async acquire(): Promise<void> {
    this.refill();
    if (this.tokens >= 1) { this.tokens -= 1; return; }
    return new Promise(resolve => this.queue.push({ resolve }));
  }

  private refill() {
    const now = Date.now();
    this.tokens = Math.min(this.max, this.tokens + (now - this.lastRefill) * this.refillRate);
    this.lastRefill = now;
    while (this.tokens >= 1 && this.queue.length) {
      this.tokens -= 1;
      this.queue.shift()!.resolve();
    }
  }
}

const indexLimiter = new GleanRateLimiter(18);  // buffer under 20 bulk/min
const searchLimiter = new GleanRateLimiter(50);

Retry Strategy

async function gleanRetry<T>(
  limiter: GleanRateLimiter, fn: () => Promise<Response>, maxRetries = 4
): Promise<T> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    await limiter.acquire();
    const res = await fn();
    if (res.ok) return res.json();
    if (res.status === 429) {
      const retryAfter = parseInt(res.headers.get("Retry-After") || "30", 10);
      const jitter = Math.random() * 5000;
      await new Promise(r => setTimeout(r, retryAfter * 1000 + jitter));
      continue;
    }
    if (res.status >= 500 && attempt < maxRetries) {
      

Glean Reference Architecture

Overview

Enterprise search integration architecture for connecting internal knowledge systems to Glean's indexing and search platform. Designed for organizations needing unified search across Confluence, Google Drive, Notion, Slack, Jira, and custom internal tools. Key design drivers: connector reliability for continuous indexing, permission synchronization to enforce source-system ACLs, incremental vs bulk indexing tradeoffs, and low-latency search aggregation across heterogeneous document types.

Architecture Diagram

Source Systems ──→ Connector Framework ──→ Queue (SQS) ──→ Glean Indexing API
(Confluence, Drive,    (Cloud Run)              ↓            /indexing/documents
 Notion, Slack, Jira)       ↓              Permission Sync  /indexing/permissions
                      Schedule (cron) ──→  Bulk Reindexer   /indexing/datasources
                            ↓
                      Glean Search Index ──→ Client API ──→ Your Apps
                                              /search       (Slack bot, portal)
                                              /chat         (internal tools)

Service Layer

class ConnectorService {
  constructor(private glean: GleanIndexingClient, private cache: CacheLayer) {}

  async indexDocument(doc: SourceDocument): Promise<void> {
    const gleanDoc = this.transformToGleanFormat(doc);
    await this.glean.indexDocument(doc.datasource, gleanDoc);
    await this.syncPermissions(doc.id, doc.acl);
  }

  async bulkReindex(datasource: string, since?: string): Promise<IndexReport> {
    const docs = await this.fetchAllDocuments(datasource, since);
    const batches = this.chunk(docs, 100);  // Glean recommends batches of 100
    let indexed = 0;
    for (const batch of batches) {
      await this.glean.bulkIndex(datasource, batch);
      indexed += batch.length;
    }
    return { datasource, totalIndexed: indexed, timestamp: new Date().toISOString() };
  }
}

Caching Strategy

const CACHE_CONFIG = {
  searchResults:  { ttl: 30,   prefix: 'search' },   // 30s — freshness critical for search
  permissions:    { ttl: 300,  prefix: 'perm' },      // 5 min — ACL changes are infrequent
  datasources:    { ttl: 3600, prefix: 'ds' },        // 1 hr — datasource config rarely changes
  connectorState: { ttl: 60,   prefix: 'conn' },      // 1 min — sync cursor freshness
  documentMeta:   { ttl: 120,  prefix: 'docmeta' },   // 2 min — title/author for search previews
};
// Webhook-driven invalidation: source system change events flush document cache immediately

Event Pipeline

class IndexingPipeline {
  private queue = new Bull('glean-indexing', { redis: process.env.REDIS_URL });

  async onSourceChange(event: SourceChangeEvent):

Glean SDK Patterns

Overview

Production-ready patterns for the Glean enterprise search platform. Glean uses POST-based REST endpoints for both search and indexing. Search queries go to the Client API while document ingestion uses the Indexing API. A structured client centralizes token management, enforces batch pagination for bulk indexing, and provides typed responses for search results.

Singleton Client

let _client: GleanClient | null = null;
export function getClient(): GleanClient {
  if (!_client) {
    const domain = process.env.GLEAN_DOMAIN, key = process.env.GLEAN_API_KEY;
    if (!domain || !key) throw new Error('GLEAN_DOMAIN and GLEAN_API_KEY must be set');
    _client = new GleanClient(domain, key);
  }
  return _client;
}
class GleanClient {
  private base: string; private h: Record<string, string>;
  constructor(domain: string, key: string) {
    this.base = `https://${domain}/api`;
    this.h = { 'Authorization': `Bearer ${key}`, 'Content-Type': 'application/json' };
  }
  async search(query: string, opts: { pageSize?: number; datasource?: string } = {}) {
    const r = await fetch(`${this.base}/client/v1/search`, { method: 'POST',
      headers: { ...this.h, 'X-Glean-Auth-Type': 'BEARER' },
      body: JSON.stringify({ query, pageSize: opts.pageSize ?? 20,
        requestOptions: opts.datasource ? { datasourceFilter: opts.datasource } : undefined }) });
    if (!r.ok) throw new GleanError(r.status, await r.text()); return r.json() as Promise<GleanSearchResponse>;
  }
  async indexDocuments(datasource: string, docs: GleanDocument[]): Promise<void> {
    const r = await fetch(`${this.base}/index/v1/indexdocuments`, {
      method: 'POST', headers: this.h, body: JSON.stringify({ datasource, documents: docs }) });
    if (!r.ok) throw new GleanError(r.status, await r.text());
  }
  async bulkIndex(ds: string, docs: GleanDocument[], batch = 100): Promise<void> {
    for (let i = 0; i < docs.length; i += batch) await this.indexDocuments(ds, docs.slice(i, i + batch));
  }
}

Error Wrapper

export class GleanError extends Error {
  constructor(public status: number, message: string) { super(message); this.name = 'GleanError'; }
}
export async function safeCall<T>(operation: string, fn: () => Promise<T>): Promise<T> {
  try { return await fn(); }
  catch (err: any) {
    if (err instanceof GleanError && err.status === 429) { await new Promise(r => setTimeout(r, 3000)); return fn(); }
    if (err instanceof GleanError && err.status === 401) throw new GleanError(401, 'Invalid GLEAN_API_KEY');
    throw new GleanError(err.status ?? 0, `${operation} failed: ${err.message}`);
  }
}

Request Builder

class GleanSearchBuilder 

Glean Security Basics

Overview

Glean indexes and searches across an enterprise's entire knowledge base — Confluence, Google Drive, Slack, GitHub, and dozens more connectors. Security concerns center on indexing token management (write-access tokens that can push content into the search index), client token scoping (user-level search permissions), and document-level access controls. A leaked indexing token allows injecting arbitrary content into enterprise search results.

API Key Management

function createGleanClient(tokenType: "indexing" | "client"): { token: string; baseUrl: string } {
  const token = tokenType === "indexing"
    ? process.env.GLEAN_INDEXING_TOKEN
    : process.env.GLEAN_CLIENT_TOKEN;
  if (!token) {
    throw new Error(`Missing GLEAN_${tokenType.toUpperCase()}_TOKEN — store in secrets manager`);
  }
  // Indexing tokens have WRITE access — never expose in frontend code
  if (tokenType === "indexing") {
    console.log("WARNING: Indexing token loaded — backend use only");
  }
  return { token, baseUrl: `https://${process.env.GLEAN_INSTANCE}.glean.com/api` };
}

Webhook Signature Verification

import crypto from "crypto";
import { Request, Response, NextFunction } from "express";

function verifyGleanWebhook(req: Request, res: Response, next: NextFunction): void {
  const signature = req.headers["x-glean-signature"] as string;
  const secret = process.env.GLEAN_WEBHOOK_SECRET!;
  const expected = crypto.createHmac("sha256", secret).update(req.body).digest("hex");
  if (!signature || !crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    res.status(401).send("Invalid signature");
    return;
  }
  next();
}

Input Validation

import { z } from "zod";

const IndexDocumentSchema = z.object({
  datasource: z.string().min(1).max(100),
  document_id: z.string().min(1).max(500),
  title: z.string().min(1).max(500),
  body: z.string().max(1_000_000),
  allowed_users: z.array(z.string().email()).optional(),
  allowed_groups: z.array(z.string()).optional(),
  permissions_type: z.enum(["public", "restricted", "private"]).default("restricted"),
});

function validateIndexDocument(data: unknown) {
  return IndexDocumentSchema.parse(data);
}

Data Protection

const GLEAN_SENSITIVE_FIELDS = ["indexing_token", "client_token", "document_body", "user_query", "search_results"];

function redactGleanLog(record: Record<string, unknown>): Record<string, unknown> {
  const redacted = { ...record };
  for (const field of GLEAN_SENSITIVE_FIELDS) {
    if (field in redacted) redacte

Glean Upgrade & Migration

Overview

Glean is an enterprise search platform that indexes documents across SaaS tools via connectors and exposes Search and Indexing APIs. Migrations involve connector schema changes, search API response format updates, and document permission model upgrades. Tracking API versions is critical because Glean's Indexing API enforces document schema validation — adding required fields or changing permission structures in a new version will cause bulk indexing failures and stale search results if connectors are not updated in lockstep.

Version Detection

const GLEAN_BASE = "https://your-domain-be.glean.com/api";

async function detectGleanApiVersion(apiToken: string): Promise<void> {
  // Check indexing API health and version
  const indexRes = await fetch(`${GLEAN_BASE}/index/v1/status`, {
    headers: { Authorization: `Bearer ${apiToken}`, "Content-Type": "application/json" },
  });
  const indexStatus = await indexRes.json();
  console.log(`Indexing API version: ${indexRes.headers.get("x-glean-api-version") ?? "v1"}`);
  console.log(`Connector status: ${JSON.stringify(indexStatus.connectors)}`);

  // Check search API for deprecated query parameters
  const searchRes = await fetch(`${GLEAN_BASE}/client/v1/search`, {
    method: "POST",
    headers: { Authorization: `Bearer ${apiToken}`, "Content-Type": "application/json" },
    body: JSON.stringify({ query: "test", pageSize: 1 }),
  });
  const deprecationHeader = searchRes.headers.get("x-glean-deprecated-params");
  if (deprecationHeader) console.warn(`Deprecated parameters: ${deprecationHeader}`);
}

Migration Checklist

• [ ] Review Glean developer changelog for Indexing API schema changes
• [ ] Audit custom connectors for deprecated document fields
• [ ] Verify objectType definitions match current Glean schema requirements
• [ ] Check if new required fields were added to document permission model
• [ ] Test search API response parsing — results[].snippets format may change
• [ ] Update datasource configuration if connector authentication method changed
• [ ] Validate bulk indexing with a small document batch before full re-index
• [ ] Check people API for identity resolution field changes
• [ ] Update search query syntax if faceted search operators were modified
• [ ] Monitor indexing error dashboard for 48 hours post-migration

Schema Migration

// Glean document schema evolved: flat permissions → structured ACL model
interface OldGleanDocument {
  id: string;
  datasource: string;
  title: string;
  body: { mimeType: string; textContent: string };
  permissions: { allowedUsers: string[] };
  updatedAt: string;
}

Glean Webhooks & Events

Overview

Glean uses an event-driven indexing model where source system webhooks trigger incremental updates to the Glean Indexing API. Instead of emitting its own webhooks, Glean receives document changes from platforms like GitHub, Confluence, and Notion. You can also monitor internal Glean events such as document indexing completion, permission changes, connector sync status, and search anomalies through the admin API.

Webhook Registration

// Register a source system webhook that pushes to Glean Indexing API
const response = await fetch("https://yourapp.com/admin/webhooks", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://yourapp.com/webhooks/glean-indexer",
    events: ["document.indexed", "permission.changed", "connector.synced", "search.anomaly"],
    secret: process.env.GLEAN_WEBHOOK_SECRET,
  }),
});

Signature Verification

import crypto from "crypto";
import { Request, Response, NextFunction } from "express";

function verifyGleanSignature(req: Request, res: Response, next: NextFunction) {
  const signature = req.headers["x-glean-signature"] as string;
  const expected = crypto.createHmac("sha256", process.env.GLEAN_WEBHOOK_SECRET!)
    .update(req.body).digest("hex");
  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).json({ error: "Invalid signature" });
  }
  next();
}

Event Handler

import express from "express";
const app = express();

app.post("/webhooks/glean-indexer", express.raw({ type: "application/json" }), verifyGleanSignature, (req, res) => {
  const event = JSON.parse(req.body.toString());
  res.status(200).json({ received: true });

  switch (event.type) {
    case "document.indexed":
      confirmIndexStatus(event.data.datasource, event.data.doc_id); break;
    case "permission.changed":
      reindexPermissions(event.data.datasource, event.data.object_id); break;
    case "connector.synced":
      logSyncMetrics(event.data.connector_name, event.data.docs_processed); break;
    case "search.anomaly":
      alertOps(event.data.query_pattern, event.data.anomaly_type); break;
  }
});

Event Types