Nvidia KV Cache Transfer Skips 7-Second Re-Prefills

One of the quiet taxes on large language model serving is the handoff. When a system routes a conversation from a small model to a bigger one, the bigger model usually has to read the entire conversation again from scratch. Nvidia researchers have published work arguing that a closed-form linear mapping can move the existing KV cache across instead, and the reported numbers are hard to ignore.

• Transferring a 32,768-token cache took 278 milliseconds versus roughly 7 seconds for a standard re-prefill
• The method ran 2.7x to 25x faster than recomputing the conversation across the tested model pairs
• Accuracy retention landed between 73% and 98% of the receiver model's standalone prefill accuracy
• Fitting the mapper required only about 500 calibration sequences

What Is Cross-Model KV Cache Transfer?

A KV cache is the stored key and value tensors an LLM builds while reading a prompt. It is what lets a model generate token 5,000 without rereading tokens 1 through 4,999. The catch is that the cache belongs to one specific model. Swap models mid-conversation and the cache is normally discarded.

The Nvidia paper, titled "Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse" and submitted to arXiv on August 4, 2026, reports that the relationship between source and target KV pairs inside a model family is substantially linear. The authors fit a ridge regression mapper operating per attention head, and strip rotational positional embeddings from keys first so the mapping stays portable across context lengths. VentureBeat's write-up on August 21, 2026 brought the work to a wider audience.

How Well Does It Hold Up Across Model Sizes?

The hardest case in the paper is the biggest jump: Llama 3.1 8B to 70B, where the technique retained 72.8% of standalone prefill accuracy. That is the low end of the range, and it is honest about where a linear mapper starts to strain. The authors also describe a nonlinear MLP variant that closes the gap on the pairs where the linear version underperforms.

Evaluation spanned ARC-Challenge, HellaSwag, WinoGrande, MMLU, GSM8K, WikiText-2 perplexity, and CoQA for multi-turn conversations, with the researchers reporting stability across turns rather than degradation that compounds. Readers who follow our AI infrastructure coverage will recognise the theme: most of the recent wins in serving efficiency have come from removing redundant work rather than from new silicon, much like the Kog inference engine's throughput gains on existing GPUs.

Why Routing Systems Care About 278 Milliseconds

Model routing is now standard practice. Cheap model first, escalate to the expensive one when the task demands it. The escalation step is where users feel latency, and it is also where operators pay twice for the same tokens. Cutting a 7-second re-prefill to under 300 milliseconds changes the economics of escalating aggressively, which in turn makes routing a better deal for everyone.

A fair caveat: this is research, not a shipping product, and the technique is scoped to model pairs within a family where the representations are close enough for a linear map to work. That is a real constraint. But it is also exactly the case that production routing stacks hit most often, and 500 calibration sequences is a very low bar for adopting it.

Sources: arXiv — August 4, 2026; VentureBeat — August 21, 2026.