AI Development

Cost Control for LLM Apps: Token Budgets, Caching and Per-User Limits

A tested Node.js module that adds a per-user daily token budget and a TTL response cache around any LLM call, plus the habits that keep an AI feature's bill predictable.

By Rajib Roy Published 5 min read

The first LLM bill that surprises you is rarely caused by a flood of customers. It is usually a bug: a retry loop, a prompt that quietly grew to ten times its size, or one user who found a button that calls the model in a loop. The fix is not to watch a dashboard more closely. It is to put limits in the code, so a mistake has a ceiling.

This post builds a small module with three protections: a per-user daily token budget, a response cache for repeated prompts, and a worst-case reservation so a request can't blow past the limit in a single call. It works around any function that calls a model, and I've included tests.

Where the money actually goes

Most providers charge by tokens, with input and output often priced differently. I deliberately don't quote prices here, because they change; check your provider's pricing page. What matters for design is what multiplies the token count:

  • Long context. Every request re-sends your system prompt, tools and history. A chat that grows for 40 turns costs far more per message than the first message did.
  • Output length. Without a cap, a model may write far more than you need.
  • Retries and loops. Each retry pays again. Agent-style loops can call the model dozens of times for one user action.
  • Repeated identical work. The same question asked by a hundred users is a hundred paid answers, unless you cache it.

Principle 1: set a hard ceiling per user

A per-user daily budget turns an unbounded cost into a bounded one. Pick a number from your product, not from your fear: what does a normal heavy user consume in a day? Set the limit above that and let the module refuse politely beyond it. For anonymous users, key the budget on something coarse like an IP address or a session, knowing that it is easy to evade.

One subtle point: check the budget before the call using a worst-case estimate (prompt tokens plus your maximum output tokens), then record the real usage the provider reports afterward. If you only check after the call, a single large request can overshoot.

Principle 2: cache identical requests

Many features ask the model the same thing repeatedly: summaries of the same article, classifications of the same template, answers to the same FAQ. A cache keyed by a hash of the model name and prompt returns those instantly and for free. Keep a short time-to-live so stale answers expire, and a size cap so memory doesn't grow without limit.

Caching has a trap: never cache anything that includes personal data across users, and never serve one user's answer to another when the prompt contained their private context. Build the key from everything that changes the answer, or don't cache that call.

Principle 3: cap the output and the context

Always pass a maximum output length. It is the cheapest control you have, and it bounds the worst-case reservation. For chat history, send a summary or the last few turns rather than everything. Trimming context is the biggest saving for long conversations, and often barely changes answer quality.

The module

This file holds the whole idea. DailyBudget tracks tokens per user per UTC day. TtlCache is a tiny cache with expiry and oldest-first eviction. guarded wraps your model function: it checks the cache, reserves the worst case against the budget, calls the model, records actual usage and stores the result.

budget.js

import { createHash } from "node:crypto";

// Rough estimate only: about 4 characters per token for English text. Use your provider's
// tokenizer or the usage numbers in API responses when you need real figures.
export const estimateTokens = (text) => Math.ceil(text.length / 4);

export class DailyBudget {
  constructor({ limitTokens, now = () => new Date() }) {
    this.limit = limitTokens;
    this.now = now;
    this.usage = new Map(); // `${userId}:${YYYY-MM-DD}` -> tokens
  }
  key(userId) { return `${userId}:${this.now().toISOString().slice(0, 10)}`; }
  remaining(userId) { return this.limit - (this.usage.get(this.key(userId)) ?? 0); }
  check(userId, wanted) {
    if (wanted > this.remaining(userId)) {
      const err = new Error("daily token budget exceeded");
      err.code = "BUDGET_EXCEEDED";
      throw err;
    }
  }
  record(userId, used) {
    const k = this.key(userId);
    this.usage.set(k, (this.usage.get(k) ?? 0) + used);
  }
}

export class TtlCache {
  constructor({ ttlMs, max = 500, now = Date.now }) { this.ttl = ttlMs; this.max = max; this.now = now; this.map = new Map(); }
  get(k) {
    const e = this.map.get(k);
    if (!e) return undefined;
    if (e.exp < this.now()) { this.map.delete(k); return undefined; }
    this.map.delete(k); this.map.set(k, e); // refresh recency
    return e.value;
  }
  set(k, value) {
    if (this.map.size >= this.max) this.map.delete(this.map.keys().next().value); // evict oldest
    this.map.set(k, { value, exp: this.now() + this.ttl });
  }
}

export const cacheKey = (model, prompt) => createHash("sha256").update(`${model}\n${prompt}`).digest("hex");

// Wrap any async (prompt) => { text, usage:{total_tokens} } function.
export function guarded(callModel, { budget, cache, model, maxOutputTokens = 500 }) {
  return async function (userId, prompt) {
    const key = cacheKey(model, prompt);
    const hit = cache.get(key);
    if (hit) return { ...hit, cached: true };           // cache hits cost nothing

    budget.check(userId, estimateTokens(prompt) + maxOutputTokens); // worst-case reserve
    const out = await callModel(prompt, { maxOutputTokens });
    budget.record(userId, out.usage.total_tokens);      // record what the provider says it used
    cache.set(key, out);
    return { ...out, cached: false };
  };
}

Three choices are worth calling out. Cache hits return before any budget check, since they cost nothing. The budget rejects with an error code, BUDGET_EXCEEDED, so your API layer can turn it into a clear message and a 429 response. And the clock is injectable, which is what makes the day-rollover test possible without waiting a day.

The tests

budget.test.js

import test from "node:test";
import assert from "node:assert/strict";
import { DailyBudget, TtlCache, guarded, estimateTokens } from "./budget.js";

const fakeModel = async (prompt) => ({ text: "ok", usage: { total_tokens: estimateTokens(prompt) + 50 } });

test("caches identical prompts and does not charge twice", async () => {
  const budget = new DailyBudget({ limitTokens: 10_000 });
  let calls = 0;
  const call = guarded(async (p, o) => { calls++; return fakeModel(p, o); }, { budget, cache: new TtlCache({ ttlMs: 1000 }), model: "m" });
  const a = await call("u1", "hello world");
  const b = await call("u1", "hello world");
  assert.equal(a.cached, false);
  assert.equal(b.cached, true);
  assert.equal(calls, 1);
});

test("refuses when the worst case would exceed the daily budget", async () => {
  const budget = new DailyBudget({ limitTokens: 600 });
  const call = guarded(fakeModel, { budget, cache: new TtlCache({ ttlMs: 1000 }), model: "m", maxOutputTokens: 500 });
  await call("u1", "short prompt one");            // reserve 504 <= 600, records ~54
  await assert.rejects(() => call("u1", "x".repeat(2000)), { code: "BUDGET_EXCEEDED" }); // 500 + 500 > remaining
  await call("u2", "short prompt one");            // other users are unaffected
});

test("budget resets on a new UTC day", async () => {
  let day = "2026-11-17T10:00:00Z";
  const budget = new DailyBudget({ limitTokens: 100, now: () => new Date(day) });
  budget.record("u1", 100);
  assert.equal(budget.remaining("u1"), 0);
  day = "2026-11-18T00:00:01Z";
  assert.equal(budget.remaining("u1"), 100);
});

test("cache entries expire", () => {
  let t = 0;
  const cache = new TtlCache({ ttlMs: 10, now: () => t });
  cache.set("k", 1);
  assert.equal(cache.get("k"), 1);
  t = 11;
  assert.equal(cache.get("k"), undefined);
});

They run with node --test budget.test.js. All four passed. The second test is the important one: the second request is refused because its worst case (a long prompt plus 500 output tokens) exceeds what is left, while a different user is unaffected.

What this does not cover

  • Persistence. The budget lives in memory, so it resets on restart and isn't shared between servers. In production, keep the counters in Redis or your database, with an atomic increment.
  • Concurrency. Two simultaneous requests can both pass the check before either records usage. For strict limits, reserve tokens atomically up front and refund the difference afterward.
  • Accurate counting. The characters-divided-by-four estimate is rough. Use your provider's tokenizer or response usage fields when billing matters.
  • Abuse beyond cost. A budget doesn't stop prompt injection or misuse. It just caps the damage.

Habits that keep the bill predictable

  1. Log usage per feature. Record tokens in and out, the model and the prompt version, so you know which feature costs what.
  2. Alert on a daily total that is a multiple of your normal. A spike is usually a bug.
  3. Use a smaller model where quality allows. Prove it with an evaluation set rather than by feel.
  4. Cap retries. Bounded attempts, as in the reliability module, keep a failure from multiplying cost.
  5. Set provider-side spending limits if your provider offers them, as a last line of defense.

Telling users about limits kindly

A limit is a product decision as much as a technical one. When the budget runs out, say so plainly: what happened, when it resets, and what they can do about it. A bare 429 error makes people think your app is broken. A message such as "You've used today's AI allowance, which resets at midnight UTC" makes it a feature. If you offer a paid tier, the limit message is also the honest place to mention it, once, without nagging.

A short checklist

  • Per-user daily budget with a worst-case reservation.
  • Cache for repeated, non-personal prompts with a TTL and a size limit.
  • Maximum output tokens on every call.
  • Trimmed history for long chats.
  • Usage logged per feature and an alert on spikes.
Build with me

Want your AI feature to have a predictable bill?

I build AI tools and automation with budgets, caching and monitoring from day one. If you'd like help putting guardrails around something you've already shipped, get in touch.

See services Get in touch

What I ran this against

  • Node.js 24.22.1 on 17 November 2026, using the built-in node:test runner. No provider API was called: the examples use stubs, so they cost nothing to run. Four tests pass: cache hits are not charged, the worst-case reservation refuses over-budget calls, the budget resets on a new UTC day, and cache entries expire.
  • The 4-characters-per-token estimate is a rough rule of thumb for English, not a measurement. Use your provider's usage numbers for real accounting. I use no prices here: check your provider's current pricing page.