AI Development

Choosing an LLM for Your Feature: How to Compare Answers Side by Side Before You Write Code

A practical, blind-scoring method for choosing between AI models for a real feature, with a tested Node.js scorer, plus where side-by-side chat tools like Shortcut Chat help and where they stop.

By Rajib Roy Published 6 min read
Disclosure
This post contains affiliate links to Shortcut Chat. If you buy through them, RAJIBS may earn a commission at no extra cost to you. It doesn't change what I write: I say where a tool isn't a fit, and I say when I haven't used it myself. Full disclosure.

Sooner or later, every team building an AI feature has the same conversation. "Should we use this model or that one?" Someone pastes a prompt into two chat windows, the second answer reads better, and the decision is made. Two weeks later the feature is in production and you discover that "reads better" was not the same as "does the job".

Looking at answers side by side is a good way to start. It is a terrible way to finish. This post shows how to make the starting point much better, with a small blind-scoring method and a script, and then how to confirm the choice properly before you commit. I'll also cover where a side-by-side chat tool such as Shortcut Chat fits, and where it doesn't.

Step 1: write the job, not the prompt

Before comparing anything, write one sentence describing what the model must do and what a bad result costs. "Summarize a support thread so an agent can answer in 30 seconds" is a job. "Be helpful" is not. Then list what matters for this job, usually a short set of things like factual accuracy, following the required format, tone, length and how it fails (does it say "I don't know" or does it invent?).

Models differ in voice, speed and blind spots. The point of comparing is to find which of those differences matter to your users.

Step 2: collect 10 to 20 realistic prompts

Use real inputs, with personal data removed: actual support threads, actual documents, actual user questions. Include a few easy ones, a few hard ones and a few nasty ones, such as empty input, a very long input or a message in another language. If you plan to build an evaluation set later, these prompts are its first draft, so keep them in a file.

Step 3: run the same prompt on several models, with the same settings

This is where a side-by-side tool earns its place. Shortcut Chat, for example, advertises sending one prompt to up to three models at once and streaming each answer next to the others, with the ability to swap models in or out during a conversation. That removes the busywork of pasting into three tabs and makes differences in voice and length obvious at a glance. You can do the same with your providers' playgrounds or a small script. The method matters more than the tool.

Whichever way you do it, keep these constant: the exact prompt text, the system instructions and whether web search is on. If one model gets web access and another doesn't, you are comparing tools, not models.

Step 4: score blind

This is the step people skip, and it is the one that changes decisions. If you can see "Model A" above an answer, you will judge it with everything you already believe about Model A. Hide the names. Assign random labels per prompt, copy the answers into a sheet, score each 1 to 5 against your list from step 1, and only then reveal which was which.

The script below does the unglamorous parts: it generates a reproducible random label mapping and summarizes scores after the reveal. For each model it reports the mean score, the worst score, and the number of strict wins, where ties count for nobody.

compare.js

import { readFileSync } from "node:fs";

// Input: CSV with header  prompt_id,label,score   (score 1-5, given while you could NOT see which model wrote it)
// A separate key file maps blind labels back to model names, per prompt.
export function parseCsv(text) {
  const [head, ...rows] = text.trim().split("\n").map((l) => l.split(",").map((s) => s.trim()));
  return rows.map((r) => Object.fromEntries(head.map((h, i) => [h, r[i]])));
}

// Deterministic shuffle so the blinding is reproducible and testable (seeded LCG).
export function blindLabels(models, seed) {
  let s = seed >>> 0;
  const rnd = () => (s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32;
  const order = [...models];
  for (let i = order.length - 1; i > 0; i--) {
    const j = Math.floor(rnd() * (i + 1));
    [order[i], order[j]] = [order[j], order[i]];
  }
  return Object.fromEntries(order.map((m, i) => ["ABC"[i], m])); // label -> model
}

export function summarise(scoreRows, keyRows) {
  const keyBy = new Map(keyRows.map((k) => [`${k.prompt_id}:${k.label}`, k.model]));
  const byModel = {};
  const byPrompt = {};
  for (const r of scoreRows) {
    const model = keyBy.get(`${r.prompt_id}:${r.label}`);
    if (!model) throw new Error(`no key for ${r.prompt_id}:${r.label}`);
    const score = Number(r.score);
    (byModel[model] ??= []).push(score);
    (byPrompt[r.prompt_id] ??= []).push({ model, score });
  }
  // A "win" means strictly the best score on that prompt. Ties count for nobody.
  const wins = Object.fromEntries(Object.keys(byModel).map((m) => [m, 0]));
  for (const entries of Object.values(byPrompt)) {
    const top = Math.max(...entries.map((e) => e.score));
    const best = entries.filter((e) => e.score === top);
    if (best.length === 1) wins[best[0].model]++;
  }
  return Object.entries(byModel).map(([model, scores]) => ({
    model,
    prompts: scores.length,
    mean: +(scores.reduce((a, b) => a + b, 0) / scores.length).toFixed(2),
    worst: Math.min(...scores),   // the failure you will actually ship
    wins: wins[model],
  }));
}

if (process.argv[2] && process.argv[3]) {
  const scores = parseCsv(readFileSync(process.argv[2], "utf8"));
  const key = parseCsv(readFileSync(process.argv[3], "utf8"));
  console.table(summarise(scores, key));
}

The worst score matters more than people expect. A model that averages 4.2 but occasionally produces a 1, a confidently wrong answer or a broken format, may be worse for your feature than one that averages 3.9 and never drops below 3. You ship the failures, not the average.

compare.test.js

import test from "node:test";
import assert from "node:assert/strict";
import { parseCsv, blindLabels, summarise } from "./compare.js";

// ILLUSTRATIVE, MADE-UP numbers to exercise the code. They are not measurements of any real model.
const scores = parseCsv(`prompt_id,label,score
p1,A,4
p1,B,5
p2,A,3
p2,B,3
p3,A,2
p3,B,4`);
const key = parseCsv(`prompt_id,label,model
p1,A,model-x
p1,B,model-y
p2,A,model-y
p2,B,model-x
p3,A,model-x
p3,B,model-y`);

test("summarise maps blind labels back to models and counts strict wins only", () => {
  const out = Object.fromEntries(summarise(scores, key).map((r) => [r.model, r]));
  // model-x scored 4, 3, 2. model-y scored 5, 3, 4.
  assert.deepEqual(out["model-x"], { model: "model-x", prompts: 3, mean: 3, worst: 2, wins: 0 });
  assert.deepEqual(out["model-y"], { model: "model-y", prompts: 3, mean: 4, worst: 3, wins: 2 }); // p2 was a tie
});

test("an unknown label is an error, not a silent skip", () => {
  assert.throws(() => summarise(parseCsv("prompt_id,label,score\np9,A,3"), key), /no key for p9:A/);
});

test("blindLabels is reproducible for a seed and uses every model once", () => {
  const a = blindLabels(["m1", "m2", "m3"], 7);
  const b = blindLabels(["m1", "m2", "m3"], 7);
  assert.deepEqual(a, b);
  assert.deepEqual(Object.values(a).sort(), ["m1", "m2", "m3"]);
  assert.deepEqual(Object.keys(a), ["A", "B", "C"]);
});

Three tests pass. They check that blind labels map back to the right models and that wins exclude ties, that an unknown label is an error instead of being silently skipped, and that the label shuffle is reproducible for a given seed. Again: the scores in that file are invented to exercise the code and say nothing about any real model.

Step 5: watch for the usual traps

  • Length bias. Longer answers often feel more thorough. Research on using models as judges documents position, verbosity and self-enhancement biases, and humans have similar tendencies. Score against your criteria, and penalize padding if your users won't read it.
  • Position bias. The answer on the left gets read first. Randomize the order across prompts, which the blind labels do for you.
  • Prompt overfitting. If you tweak the prompt until Model A looks great, you've tuned for A. Use the same prompt for all, or tune each one's prompt with the same effort.
  • Tiny samples. Fifteen prompts can show a large gap, but they can't rank two close models. If the scores are within noise, choose on cost, speed and reliability.
  • Letting a model judge itself. Using one of the candidates as the judge adds bias. If you use a model judge at all, check it against your own scores first.

Step 6: where side-by-side chat stops

A chat app is a convenient microscope, but it is not your production environment. Before you commit, do these things through the API you will really call:

  1. Re-run the shortlist through the API with your real system prompt, your output schema and your settings. A chat product may add its own instructions or defaults, so the same model can behave differently through the API.
  2. Measure latency and cost from your servers. Timings in a chat window reflect someone else's setup and load. Look at your providers' pricing pages for current rates.
  3. Test structured output and tool calls. Does the JSON validate every time? The validation and retry wrapper is the place to find out.
  4. Check data handling. Read the privacy terms of each provider, and don't paste customer data into any tool you haven't reviewed.
  5. Plan for change. Models are updated and retired. Keep your prompts in version control, log the model name and keep the evaluation set so you can rerun the comparison when a new model arrives.

Where I would use a side-by-side tool, and where not

  • Good fit: early exploration, building intuition about model differences, drafting the first 15 prompts, and a quick sanity check when a new model is announced.
  • Reasonable fit: non-developers on your team, such as support or content leads, who need to judge answers without writing code.
  • Poor fit: your final decision, anything involving private data, and automated regression testing. For those you want an API, a repository and a script.

On cost: Shortcut Chat describes a monthly usage allowance shown as a percentage, with different models using different amounts, and says that you won't be hit with overage charges. That is worth understanding before you run big comparisons, since heavier models use the allowance faster. Check the current plans on the site, because I have not verified them beyond the homepage text. The programme also lists a promo code, FRIEND5, but I haven't confirmed what it takes off or whether it applies to your plan, so check at checkout.

A short checklist

  • Write the job and the criteria first.
  • 10 to 20 real prompts, saved in a file.
  • Same prompt and settings for every model.
  • Score blind, record the worst case as well as the average.
  • Confirm the shortlist through the production API for format, latency and cost.
  • Keep the prompts and scores, because you will repeat this.
Optional tool

Want to try side-by-side answers yourself?

Shortcut Chat lets you send one prompt to up to three models at once and compare the answers. Check what the current plans include before you subscribe, and treat it as a way to explore, not as a replacement for testing through the API you will actually ship with.

Look at Shortcut Chat

What I ran this against

  • The scoring script (blind labels and a summary of mean, worst case and strict wins) was run with node --test on Node.js 24.22.1 on 17 November 2026. Three tests pass.
  • The scores in the test file are made-up numbers used to check the code. They are not measurements of any real model, and I make no claims about which model is better at anything.
  • I have not used Shortcut Chat for this post. What I say about the product comes only from its public homepage and FAQ, listed under Sources, read on 17 November 2026. I haven't tested its answers, speed, accuracy or limits, and its model list changes, so check the site for what is available today. This post contains an affiliate link.

Sources checked

  • Shortcut Chat homepage and FAQ (side-by-side answers from up to three models at once, reusable shortcuts inserted by click or by typing /, web search, image generation, a monthly usage allowance shown as a percentage, cancel anytime): shortcutchat.com, read 17 November 2026. I could not open a separate pricing page (it returned 404), so this post quotes no prices; check the current plans on the site.
  • Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena": the abstract discusses position, verbosity and self-enhancement biases in model judges: arxiv.org/abs/2306.05685.