AI Development

How to Make LLM API Calls Reliable in Node.js: Timeouts, Retries and JSON Validation

A short, tested Node.js module that adds a hard timeout, backoff with jitter, Retry-After handling and schema-validated JSON to any chat-style LLM API call.

By Rajib Roy Published 5 min read

The first version of an LLM feature always works. You send a prompt, you get JSON back, you ship the demo. Then real traffic arrives and you meet the other half of the job: requests that hang for a minute, a 429 at the worst moment, and a model that wraps its JSON in a code fence or forgets a field.

None of this is exotic, and none of it needs a framework. In this post I'll build a small Node.js module that handles the three failure classes I see most often in LLM-backed features: slow or dead requests, transient errors and invalid output. It is provider-neutral, about 90 lines, and I ran it against a mock server so I can show you what it actually does.

The three failure classes

It helps to name them, because each one gets a different fix:

  • The request never finishes. A network stall or an overloaded endpoint leaves your server waiting. Fix: a hard timeout on every call.
  • The request fails in a way that is worth repeating. Rate limits (429) and server errors (5xx) are usually temporary. Fix: bounded retries with exponential backoff and jitter.
  • The request succeeds but the answer is unusable. The HTTP status is 200, yet the JSON is missing a field or has a value outside your enum. Fix: validate against a schema, and re-ask once with the error message.

What you should not retry matters just as much. A 400 or 401 will fail the same way a second time, so retrying only burns time and money.

Step 1: a hard timeout on every call

Node's built-in fetch has no timeout by default. The simplest fix is AbortSignal.timeout(ms), which is available in current Node versions. Pass it as the signal and the request is aborted when the clock runs out.

Choose the number from your user experience, not from the provider's maximum. If the user will give up after 20 seconds, a 60-second timeout only means your server works for someone who has already left. I treat a timeout as retryable, since it often means one unlucky connection.

Step 2: retry only what is worth retrying

My rule is short. Retry on network errors, timeouts, 429 and any 5xx. Everything else is a bug in my request and should surface immediately.

For the delay I use exponential backoff (500 ms, 1 s, 2 s, and so on) plus a little random jitter. Jitter matters when many requests fail together: without it they all retry at the same instant and hit the limit again. If the server sends a Retry-After header, I use that instead of my own guess. My code only reads the seconds form of that header. The header can also contain an HTTP date, so if your provider sends dates, extend the parsing.

Two limits keep this honest: a maximum attempt count (I use four), and no retries when the call has side effects. If your LLM call is followed by "send the email" or "charge the card", keep those actions outside the retry loop. Repeating the model call is cheap. Repeating a payment is not.

Step 3: validate the output, then re-ask once

Models are good at following a format and still occasionally wrong. I treat the output like any other untrusted input: parse it, then check it against a schema. I use Zod here, but any schema library works.

If validation fails, I send the model its own answer plus the list of schema issues and ask for corrected JSON, once. That one extra round fixes most formatting slips. If the second answer also fails, I throw. A loop that keeps asking forever is just a slower outage, and every round costs tokens.

I also strip a leading ```json fence before parsing, because models add one often enough that it deserves a line of code.

The module

Save this as llm.js. It uses ES modules, so set "type": "module" in your package.json and run npm i zod.

llm.js

import { z } from "zod";

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export class LlmError extends Error {
  constructor(message, { status, retryable } = {}) {
    super(message);
    this.name = "LlmError";
    this.status = status;
    this.retryable = retryable ?? false;
  }
}

// One HTTP call with a hard timeout. Assumes an OpenAI-style chat completions
// response shape: { choices: [{ message: { content } }] }
async function callOnce({ url, apiKey, body, timeoutMs }) {
  let res;
  try {
    res = await fetch(url, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${apiKey}`,
      },
      body: JSON.stringify(body),
      signal: AbortSignal.timeout(timeoutMs),
    });
  } catch (err) {
    // Timeouts and network resets are worth retrying.
    throw new LlmError(`network: ${err.name}`, { retryable: true });
  }

  if (!res.ok) {
    const retryable = res.status === 429 || res.status >= 500;
    const err = new LlmError(`http ${res.status}`, { status: res.status, retryable });
    err.retryAfterMs = Number(res.headers.get("retry-after")) * 1000 || undefined;
    throw err;
  }

  const json = await res.json();
  const text = json?.choices?.[0]?.message?.content;
  if (typeof text !== "string") throw new LlmError("unexpected response shape");
  return text;
}

export async function callWithRetry(opts, { maxAttempts = 4, baseMs = 500 } = {}) {
  for (let attempt = 1; ; attempt++) {
    try {
      return await callOnce(opts);
    } catch (err) {
      if (!(err instanceof LlmError) || !err.retryable || attempt >= maxAttempts) throw err;
      const backoff = baseMs * 2 ** (attempt - 1);
      const jitter = Math.random() * backoff * 0.25;
      await sleep(err.retryAfterMs ?? backoff + jitter);
    }
  }
}

// Ask for JSON, validate it, and re-ask once with the validation error if it fails.
export async function askForJson({ schema, prompt, ...conn }) {
  const messages = [
    { role: "system", content: "Reply with a single JSON object and nothing else." },
    { role: "user", content: prompt },
  ];

  for (let round = 1; round <= 2; round++) {
    const text = await callWithRetry({
      ...conn,
      body: { model: conn.model, messages, temperature: 0 },
    });

    let parsed;
    try {
      parsed = JSON.parse(stripFence(text));
    } catch {
      parsed = undefined;
    }
    const result = schema.safeParse(parsed);
    if (result.success) return result.data;

    messages.push({ role: "assistant", content: text });
    messages.push({
      role: "user",
      content: `That did not match the schema: ${JSON.stringify(result.error.issues)}. Reply again with corrected JSON only.`,
    });
  }
  throw new LlmError("model never returned valid JSON");
}

function stripFence(s) {
  return s.trim().replace(/^```(?:json)?\s*/i, "").replace(/\s*```$/, "");
}

export const TicketSchema = z.object({
  category: z.enum(["billing", "bug", "feature", "other"]),
  urgent: z.boolean(),
  summary: z.string().min(1).max(200),
});

A few details worth pointing out. The LlmError class carries a retryable flag, so the retry loop never has to guess from a message string. The temperature: 0 in askForJson is there because a structured-extraction task benefits from the most repeatable output the model can give, though it does not guarantee identical answers. And the schema lives next to the code that uses it, so a change to the contract is a change you can see in review.

Testing it without spending a cent

The nice thing about this design is that every failure mode can be faked. This script starts a local server that replies with a 429, then a 200 whose JSON is wrong, then a correct answer wrapped in a code fence:

mock-test.js

import http from "node:http";
import { askForJson, TicketSchema } from "./llm.js";
let calls = 0;
const server = http.createServer((req, res) => {
  calls++;
  if (calls === 1) { res.writeHead(429, { "retry-after": "0" }); return res.end(); }
  if (calls === 2) { res.writeHead(200, {"content-type":"application/json"}); return res.end(JSON.stringify({choices:[{message:{content:'{"category":"oops"}'}}]})); }
  res.writeHead(200, {"content-type":"application/json"});
  res.end(JSON.stringify({choices:[{message:{content:'```json\n{"category":"bug","urgent":true,"summary":"Login fails"}\n```'}}]}));
});
server.listen(0, async () => {
  const url = `http://127.0.0.1:${server.address().port}/v1/chat/completions`;
  const out = await askForJson({ schema: TicketSchema, prompt: "x", url, apiKey: "k", model: "m", timeoutMs: 2000 });
  console.log(out, "calls:", calls);
  server.close();
});

Run it with node mock-test.js. When I ran it, it printed the parsed ticket and calls: 3: one rate-limit retry, one schema re-ask, one success. That single run exercises the backoff path, the validation path and the fence stripping.

For a real project I'd turn this into a proper test with a test runner and one case per failure class, but even this script is more than most demos ever get.

What I'd add next

  • A request ID in every log line, so one user complaint maps to one chain of attempts.
  • A budget guard. Count tokens or calls per user per day and refuse politely past the limit, so a bug in a loop cannot become a bill.
  • A fallback model or a cached answer for features where an approximate result is better than an error page.
  • A circuit breaker. If the provider has failed ten times in a row, stop calling it for 30 seconds instead of making every user wait through four retries.

Common mistakes

  • Retrying everything. It turns a clear 401 into a confusing 8-second delay.
  • No timeout. The default is effectively "wait for as long as it takes".
  • Trusting JSON.parse as validation. It only proves the text is JSON, not that it has the fields you need.
  • Putting side effects inside the retry. Keep payments, emails and database writes outside it.
Build with me

Need an AI feature that holds up in production?

I build AI tools, automation and full-stack apps for clients. If you'd like help turning a prototype into something dependable, see what I do or send a short note.

See services Get in touch

What I ran this against

  • Node.js 24.22.1 and Zod 4.6.7, run on 17 November 2026.
  • The code was exercised against a local mock HTTP server that returns a 429, then malformed JSON, then a valid answer. I did not run it against any real provider's live API.
  • The response shape (choices[0].message.content) follows the widely copied OpenAI-style chat completions format. If your provider differs, only the small callOnce function needs to change.