Langfuse Experiment Guardian

Overview

Determine whether a candidate meets a reviewed case-level release policy against an explicit baseline. An average score cannot hide missing cases, failed tasks, evaluator omissions or a critical regression. The tool compares supplied evidence; it does not run an experiment, judge model output or release anything.

Prerequisites

• The reviewed baseline run identity and review reference; never choose the newest run automatically.
• Project, dataset/version and required case manifest with content hashes and explicit critical-case classifications.
• Versioned evaluators, score ranges, threshold directions and release policy.
• Complete baseline/candidate item evidence with task status and collection completeness, mapped using the receipt contract.
• Python 3.10+ and authorized access to the supplied private evaluation data.

Offline comparison needs no authentication or model credentials. Do not start a paid evaluator or provider call to fill missing evidence during this workflow.

Instructions

Step 1: Establish the release question

Name the proposed change and exact baseline/candidate identities. Record the dataset snapshot, case content, evaluator versions and configuration that make the comparison meaningful. Same dataset name or aggregate score is insufficient. If identities or policy are missing, return inconclusive and request the gap.

Step 2: Preserve item-level evidence

Read the contract to retain each required case's identity/hash, task outcome and evaluator observations. Missing scores stay missing; do not replace them with zero, success or an average. Preserve errored, unfinished and omitted items. Mark incomplete exports explicitly.

For an existing JavaScript SDK result, the contract documents the optional scripts/project-result.mjs module. It extracts an allowlisted projection without copying inputs, outputs or evaluator comments. Do not call the experiment again to obtain a fresh result. Optional task capture must be integrated deliberately in separately authorized work, not retroactively claimed as execution evidence.

Step 3: Apply the reviewed policy

python3 scripts/compare.py private-experiment-comparison.json

Run relative to this skill's directory. Choose the policy explicitly: all-required requires passing candidate evidence for required evaluations; preserve-baseline-passes additionally preserves reviewed baseline successes while handling only explicitly reviewed baseline-failure exceptions. Follow the contract rather than inventing exception records.

An exception needs an exact case/evaluator target and review reference. Critical candidate failure is never waived by this comparison. Do not change thresholds, critical flags, hashes or expected cases simply to make the gate pass.

Step 4: Assess comparability before success

Inspect project/dataset/version, manifest hashes, evaluator definitions, duplicate cases, completion and task states before interpreting numeric scores. An empty result or non-comparable comparison is not a passing release gate.

Separate candidate regressions from pre-existing baseline failures and inconclusive evidence. The script's known input contract does not authenticate the source, establish evaluator validity or prove that the benchmark represents production traffic. Keep those limitations in the review.

Step 5: Return the release finding

Give the explicit policy result, affected case/evaluator references privately, critical blockers, incomplete evidence and smallest next check. Preserve the script's actual status and exit code. A passing comparison can support an owner decision; it is not deployment authority or a production-quality certificate.

Output

Return langfuse-experiment-comparison.json and a short supplied-data ledger: reviewed identities and policy, comparable/inconclusive state, case-level regressions, critical blockers, exceptions and unresolved evidence. Keep raw prompts, generated text, evaluator comments and secrets out of the shared report.

Examples

• Average improved, critical case failed: retain the critical blocker and reject release readiness under the reviewed policy.
• One evaluator result disappeared: report missing evaluation evidence; an omitted score cannot inherit the baseline's passing result.
• Baseline itself had a reviewed failure: handle only the exact allowed exception under the chosen policy, preserving critical-case requirements.
• Candidate used a changed case input: report non-comparability rather than treating the same display ID as an identical test.

Error Handling

Malformed or oversized receipts exit 2 with a fixed held code. Non-passing comparisons exit 1; preserve failed and inconclusive distinctions in the JSON. Do not rerun business tasks or paid evaluators automatically. Missing SDK task capture remains unknown rather than being inferred from a produced output.

Resources

• Receipt, policy and SDK projection contract.
• Experiment comparison authority.
• Evaluation specification: reviewed cases, not a completed behavioral run.