AI Development

How to Build a Small Evaluation Set for Your LLM Feature Before You Ship

Build a 30-case evaluation set, a 40-line runner and a CI check so prompt changes can't quietly break your LLM feature. Includes a runnable Node.js example with a baseline file.

By Rajib Roy Published 5 min read

Most LLM features ship on the strength of a demo. Someone tries ten inputs, nine look good, and the feature goes out. Then a prompt tweak, a model upgrade or a new type of customer message changes behavior, and nobody can say whether it got better or worse because nobody ever wrote down what "good" meant.

An evaluation set fixes that. It is a list of inputs with the answers you expect, plus a script that runs them and reports a pass rate. It does not need a platform, a dataset team or a research degree. This post builds one in about 40 lines of Node.js and shows the baseline check that stops regressions from slipping through.

What an evaluation set is (and isn't)

It is a regression test for behavior you care about. If your feature classifies support tickets, the set contains tickets and their correct categories. If it extracts fields from invoices, the set contains invoices and the fields you expect.

It is not a benchmark of the model's intelligence, and it is not proof that the feature works for every user. It only tells you whether today's version still handles the cases you decided mattered. That is already a huge improvement over memory and optimism.

Step 1: collect 30 real cases, not 1,000 invented ones

Start small and real. Pull 30 actual inputs from logs, support inboxes or your own testing, with personal data removed. Aim for a spread:

  • Typical cases that make up most of your traffic.
  • Boundary cases that sit between two categories, like a billing question that mentions a crash.
  • Known past failures, the examples that embarrassed you once. Each bug you fix should add a case.
  • Junk and edge inputs: empty messages, very long messages, another language, an attempt to confuse the model.

Store them as JSON Lines, one object per line, so a diff shows exactly which case changed. The example below has 12 cases to keep the post short, but the format is the same:

cases.jsonl

{"id":"b1","input":"I was charged twice this month","expected":"billing"}
{"id":"b2","input":"Please send me an invoice for last quarter","expected":"billing"}
{"id":"b3","input":"My card was declined but the money left my account","expected":"billing"}
{"id":"u1","input":"The app crashes when I open settings","expected":"bug"}
{"id":"u2","input":"Export button does nothing on Safari","expected":"bug"}
{"id":"u3","input":"Login page shows a blank screen","expected":"bug"}
{"id":"f1","input":"Could you add dark mode?","expected":"feature"}
{"id":"f2","input":"It would be great to export to CSV","expected":"feature"}
{"id":"o1","input":"What are your opening hours?","expected":"other"}
{"id":"o2","input":"Thanks, that fixed it!","expected":"other"}
{"id":"t1","input":"The invoice page crashes when I click download","expected":"bug"}
{"id":"t2","input":"Why does the export feature charge extra?","expected":"billing"}

Write the expected answer before you look at what the model returns. Otherwise you will quietly bend your expectations to match the output.

Step 2: the runner

The runner loads the cases, calls a classify function for each, and prints the pass rate, the failing cases and a confusion count that shows which mistakes you make. In this demo, classify is a keyword stub so the example is deterministic and free. In a real project you replace its body with your model call, ideally the same retry-and-validate wrapper you use in production, as in this reliability module.

run-evals.js

import { readFileSync, writeFileSync, existsSync } from "node:fs";

// Swap this for your real model call. A keyword stub keeps the demo deterministic and free.
export async function classify(text) {
  const t = text.toLowerCase();
  if (/(charge|invoice|card|refund|billing)/.test(t)) return "billing";
  if (/(crash|error|blank|does nothing|bug)/.test(t)) return "bug";
  if (/(add |would be great|feature)/.test(t)) return "feature";
  return "other";
}

const cases = readFileSync(new URL("./evals/cases.jsonl", import.meta.url), "utf8")
  .trim().split("\n").map((l) => JSON.parse(l));

const results = [];
for (const c of cases) {
  const got = await classify(c.input);
  results.push({ id: c.id, expected: c.expected, got, pass: got === c.expected });
}

const passed = results.filter((r) => r.pass).length;
const rate = passed / results.length;

// Confusion counts show *which* mistakes you make, not just how many.
const confusion = {};
for (const r of results.filter((r) => !r.pass)) {
  const k = `${r.expected} -> ${r.got}`;
  confusion[k] = (confusion[k] ?? 0) + 1;
}

console.log(`passed ${passed}/${results.length} (${(rate * 100).toFixed(1)}%)`);
for (const r of results.filter((r) => !r.pass)) console.log(`  FAIL ${r.id}: expected ${r.expected}, got ${r.got}`);
console.log("confusion:", confusion);

// Compare with the last accepted run so a prompt change can't quietly make things worse.
const baselineFile = new URL("./evals/baseline.json", import.meta.url);
const baseline = existsSync(baselineFile) ? JSON.parse(readFileSync(baselineFile, "utf8")) : null;
if (process.argv.includes("--accept")) {
  writeFileSync(baselineFile, JSON.stringify({ rate, failing: results.filter((r) => !r.pass).map((r) => r.id) }, null, 2));
  console.log("baseline updated");
} else if (baseline) {
  const newlyFailing = results.filter((r) => !r.pass && !baseline.failing.includes(r.id)).map((r) => r.id);
  console.log(`baseline ${(baseline.rate * 100).toFixed(1)}%  newly failing: ${newlyFailing.join(", ") || "none"}`);
  if (newlyFailing.length || rate < baseline.rate) process.exit(1);
}

Two details matter. First, the --accept flag saves the current result as a baseline file, and later runs compare against it. Second, the script exits with a non-zero code if any case that used to pass now fails, which is what lets a CI job block a bad change.

What the output looks like

The first run with node run-evals.js --accept printed:

passed 11/12 (91.7%)
  FAIL t1: expected bug, got billing
confusion: { 'bug -> billing': 1 }
baseline updated

That failing case, t1, is the "invoice page crashes" example, a boundary case where my keyword rule wrongly latches onto the word "invoice". A pass rate alone would hide that; the failing ID and the confusion count point straight at it.

To check the safety net, I then broke the stub on purpose by removing "invoice" from the billing keywords. The run reported newly failing: b2 and exited with code 1, even though the headline percentage was unchanged (one case fixed, another broken). That is the reason to track individual cases and not just an average.

Step 3: choose how to score

Exact match works for classification. For open-ended output you have a few options, in order of preference:

  1. Programmatic checks. Does the JSON validate? Does the answer contain the required order number? Is it under the length limit? These are cheap, stable and should come first.
  2. Rubric checks you can code. "Mentions the refund window", "does not promise a delivery date".
  3. Human review of a sample. For tone and helpfulness, read 20 outputs yourself before and after a change.
  4. A second model as judge. It scales, but it is itself a model with biases. If you use it, calibrate it against your own judgments on a few cases before you trust it.

Step 4: wire it into your workflow

  • Run the set on every pull request that touches a prompt, a model name or the code around them.
  • Update the baseline deliberately, in the same pull request, with a note explaining why.
  • Add a case for every production bug before you fix it.
  • Keep a few cases secret from whoever tunes the prompt. If you tune against every case, you end up measuring memorization.

Limits you should know about

  • Thirty cases can't prove much. A change from 28 to 29 passing is noise. Use the set to catch big regressions, not to rank two near-identical prompts.
  • Models are not perfectly repeatable. Even with the temperature at zero, outputs can vary. For important cases, run them a few times and look at the spread.
  • The set drifts out of date. When your users change, your cases should too.
  • It measures what you wrote down. Cases you never thought of aren't covered, which is why real production examples are the best source.

Growing the set without drowning in it

Once the habit sticks, the set grows. Keep it healthy by tagging each case with why it exists, such as "regression", "boundary" or "adversarial", so you can see at a glance what a failure means. Review it every few months and delete cases that no longer reflect real traffic. A set of 60 cases you understand beats 600 you've stopped reading. And when a case fails, resist the reflex to edit the expected answer until it passes. First ask whether the model is wrong or your label was, and write down which.

A short checklist

  • 30 real cases, expected answers written first.
  • One command that prints pass rate, failing IDs and a confusion count.
  • A baseline file and a non-zero exit code on regression.
  • A rule: every bug becomes a case.

It takes an afternoon, and it changes the question from "does it feel better?" to "what broke, and was it worth it?".

Build with me

Want an AI feature with a safety net?

I build AI tools and automation with evaluation and monitoring built in. If you'd like help setting up evaluation for something you're shipping, get in touch.

See services Get in touch

What I ran this against

  • Node.js 24.22.1 on 17 November 2026, using the built-in node:test runner. No provider API was called: the examples use stubs, so they cost nothing to run.
  • The classifier in the example is a deliberately simple keyword stub, so the run is deterministic. The numbers shown (11 of 12 passing) come from that stub, not from any real model, and say nothing about real-model accuracy.

Sources checked

  • OpenAI's published notes on its AI-text classifier (withdrawn on 20 July 2023 "due to its low rate of accuracy") are a useful example of a vendor reporting real error rates from a challenge set: openai.com, read 17 November 2026.