Why prompt caching often saves nothing

The business case is in your AI bill is mostly the same instructions. These are the four things that decided whether caching worked.

1. Stable content first, changing content last

The cache matches the beginning of a request, exactly, in the order tools, then system prompt, then messages. Anything that differs early in the request — a timestamp, a user name, the document being processed — breaks the match for everything after it. Instructions and examples go first; the document goes last.

2. Put the marker on the last thing that doesn’t change

The cache marker says “everything up to here is reusable.” Put it on the incoming document and every call writes a new cache entry at the higher write price and never reads one back. That’s the most common reason caching appears switched on but the cache-read count stays at zero.

const body = {
  model: MODEL,
  max_tokens: 2048,
  system: [{
    type: 'text',
    text: INSTRUCTIONS,                          // identical every call
    cache_control: { type: 'ephemeral' }         // marker on the stable part
  }],
  messages: [{ role: 'user', content: documentBlocks }]   // changes every call
};

3. Parallel requests can’t share a cache nobody has written

Our order-booking tool reads each PO with several parallel calls — header plus line reads — all sending the same PDF. With the PDF marked for caching, they still all paid full price: requests that start at the same instant can’t read a cache entry none of them has finished writing.

The fix is a small warm-up call per document that sends the PDF with the cache marker, asks for almost nothing back, and exists only to write the cache. The parallel reads start once it returns, and each one reads the PDF from the cache:

await warmCache(pdfBlocks);          // one write, tiny response
await Promise.all(reads.map(r =>     // every read now hits the cache
  callModel(r.instructions, pdfBlocks)));

If the warm-up fails, the reads run uncached exactly as before; a failed warm-up never stops a read. It added one to three seconds before the reads began, and the reads then ran slightly faster, so total time was about the same.

4. Mind the minimum and the duration

• Minimum size. Content below the provider’s minimum (512 tokens on the current Claude 5.5 models) is not cached, and no error says so. One tool’s instructions were just over it; the first logged call confirmed an 866-token cache write.
• Duration. The default five-minute cache refreshes each time it’s read. That suits batches. A tool people use every twenty minutes through the day fits the one-hour cache better; a once-a-day job fits neither.
• Single-use documents. A document sent once costs more to cache than not. Cache the instructions in front of it instead.

Prove it in the logs

Every response reports how many input tokens were written to the cache and how many were read from it. Log both on every call:

const u = response.usage;
log.debug('AI usage', `cache write ${u.cache_creation_input_tokens || 0} | ` +
                      `cache read ${u.cache_read_input_tokens || 0} | ` +
                      `uncached ${u.input_tokens}`);

A working setup shows one write followed by large reads. Writes on every call, or zeros in both, mean the cache isn’t doing anything.

Before you deploy

Diff the request bodies before and after: they should be identical apart from the cache marker and any warm-up call. Run the old and new versions against the same test documents and confirm identical results. Caching should change what you pay, never what the model returns.

Related: your AI bill is mostly the same instructions, and filing inbound documents with AI.