What question did this study set out to answer?

The aim is to enhance inference efficiency in tool-augmented LLMs by implementing a persistent KV cache system.

March 1, 2026Open Access

ContextCache: Persistent KV Cache with Content-Hash Addressing for Zero-Degradation Tool Schema Caching

Key Points

The aim is to enhance inference efficiency in tool-augmented LLMs by implementing a persistent KV cache system.
Developed a persistent key-value (KV) cache using SHA-256 for content-hash addressing.
Logged performance metrics against traditional prefill practices.
Assessed tool selection accuracy across various evaluation splits.
Achieved a 6.9x reduction in time-to-first-token (from 787ms to 114ms) on cache hits.
Maintained zero quality degradation, matching full prefill accuracy in evaluations.
Demonstrated the failure of independent KV compilation for tools due to dependencies.

Abstract

ContextCache is a persistent KV cache system that accelerates tool-augmented LLM inference by caching prefilled key-value states of tool schema prefixes with SHA-256 content-hash addressing. On cache hits, only the user query requires prefilling, reducing time-to-first-token by 6.9x (787ms to 114ms) with zero quality degradation — group-cached generation matches full prefill exactly on Tool Selection Accuracy across all evaluation splits. This work also documents a negative result: per-tool independent KV compilation with NoPE/deferred-RoPE fails (TSA ~0.1) due to cross-tool attention dependencies, motivating the group caching design. The system includes disk persistence, a model-agnostic adapter layer (Qwen, Llama, Mistral), FastAPI serving, and a browser-based UI, breaking even after just 2.1 requests.

Demander à l'IA

Bookmark

View Full Paper