KV cache reuse gets harder across multiple replicas: a follow-up request often lands on a GPU that never prefilled the prompt, so the cache it needs isn't local. We tested whether a shared storage pool can bridge that gap. Four replicas, plain round robin, so follow-ups are deliberately spread across GPUs regardless of which one did the original prefill. Only one line of configuration changes: whether the replicas keep private cache directories or share one pool on the Solidigm D7-PS1010.
The 8.4x latency win is what you notice first; the 3.5x drop in GPU power is what scales as the fleet grows.
A shared SSD-backed KV pool served cross-replica follow-ups in 2.20 s against 18.38 s for a full recompute. Every request met the 5 s SLO at 119 W against 413 W.
What we found
Offloading KV cache to SSD is now a standard way to avoid re-prefilling long contexts. What is less settled is who gets to read it back. A single replica reloading its own cache is the easy case, and it is not the case a production fleet runs. Behind a router, a session's turns land wherever capacity happens to be, and a cache that only one GPU can see is a cache that mostly misses.
The industry answer is a cache-aware router that sends the request back to the GPU holding the KV. Red Hat reports an 87.4% hit rate for prefix-aware routing in llm-d1. Baseten and NVIDIA report a 50% TTFT reduction from cross-replica KV routing in production2.
We tested the opposite. The router here is plain round robin and is not prefix aware, so every follow-up went to a replica that had never seen the prefix. The question is how much of the benefit a shared storage tier recovers when routing gives you nothing at all, since that is also what autoscaling and failover leave behind.
What we ran
Four vLLM replicas, one per GPU, tensor-parallel size 1 each, identical engine arguments, and the same LMCache kv_both connector against the same physical drive. The workload runs in two phases: each document is sent once, round robin, to populate the pool; then the follow-up for each document is sent to a different replica than the one that prefilled it. We ran sync and dropped the page cache between every step, so every reload is a genuine device read rather than a page-cache hit.3
Prefix caching is enabled on every replica, so the GPU-resident cache does its normal job in both arms, and the difference is confined to the disk tier.4
The only variable between the two arms is the value of local_disk. The shared arm points all four replicas at one directory; the isolated arm gives each replica its own. Same drive, same filesystem, same model, same prompts, same order. One path string is the entire independent variable.
| Model | Qwen3-Coder-30B-A3B-Instruct-FP8 (30B MoE / 3B active), tensor-parallel size 1 per replica |
| Hardware | Gigabyte G293-Z23, 4x RTX PRO 6000 Blackwell Server Edition (96 GB each), AMD EPYC 9554, 384 GB DDR5 |
| Storage | Solidigm D7-PS1010 (TLC, 15.36 TB, PCIe 5.0); Solidigm D5-P5336 (QLC, 122.88 TB) for the tier comparison |
| Stack | vLLM 0.26, LMCache 0.5.3, gpu-memory-utilization 0.90, max-model-len 128,000, prefix caching on |
| KV backend | LMCache local-disk L2, local_cpu false, chunk size 256 / 1024 / 2048; PYTHONHASHSEED=0 on every replica |
| Router | Round robin across four equal replicas, deliberately not prefix aware |
| Workload | 64 unique prompts at roughly 99K tokens each, a pool of about 620 GB, or roughly 11x the KV-usable HBM on one GPU after model weights |
| Method | Two phases with sync and dropped page cache per step; cross-replica on 64 of 64 follow-ups; validity gate of zero errors and a pool of at least 600 GB |
At roughly 10 GB of KV state per 99K-token session, the 64-document pool is about 620 GB. That is 6.5 times the 96 GB of physical HBM on one GPU, and roughly 11 times what is left for KV after model weights, so the working set cannot be held on a single card. A reload is dominated by moving bytes, and a miss is dominated by GPU compute. That is the regime where cache locality is worth arguing about; at a few thousand tokens it is not.
One directory path decides between a recompute and a read.
Replica B receives a follow-up whose prefix it has never seen, and it performs three lookups in order. The GPU prefix cache misses in both arms, because B never prefilled the document and its block table holds nothing. The host-memory tier is disabled, so it returns nothing. The disk tier is the fork. In the isolated arm, B finds its own directory empty and falls through to a full prefill of roughly 99K tokens; in the shared arm, B finds the chunks replica A wrote and reads them off the drive in 2.20 s against 18.38 s.
Figure 1. Cross-replica reuse latency at p50
Time to first token on a follow-up routed to a replica that did not prefill the document. Shorter is better.
Median of 64 cross-replica follow-ups, four replicas, concurrency 1, zero errors. Preliminary results.
The tail holds, not just the median.
The isolated percentiles are 18.4, 19.5 and 19.6 s, a spread of about 1.2 s across the whole distribution. That flatness is the control working: every request is a full recompute, so there is no second mode from partial hits. Reload rate and SLO attainment at 5 are both 0% . These are not degraded numbers, they are the floor.
The shared arm returns 89% of requests warm within 3 s and clears the 5 s SLO on every one, with a p99 of 3.8 s that still sits below the isolated median of 18.4 s. A median win with a broken tail would not be worth reporting. this one holds.
Figure 2. Tail latency at p99
The 99th percentile of the same 64 cross-replica follow-ups. Shorter is better.
Shared p95 is 3.7 s and p99 is 3.8 s against 19.5 s and 19.6 s isolated. Preliminary results.
The saving compounds at the fleet level.
Wall clock for the 64-request reuse phase falls from 338 s to 38 s, a factor of 8.8, against a per-request factor of 8.4. The two ratios are not measuring the same thing: 8.4 compares medians, while 8.8 compares totals, and totals follow the mean. A handful of slow recomputes in the isolated arm pull its mean above its median and widen the wall-clock ratio. Read the 8.4 as the per-request result and the 8.8 as what the fleet observes. Errors were zero in both arms.
Figure 3. Wall clock for the 64-request reuse phase
End-to-end time to complete all 64 cross-replica follow-ups. Shorter is better.
The 8.8 total-time ratio exceeds the 8.4 median ratio because totals follow means, not medians. Preliminary results.
Prefill is the power draw, and a reload is not.
Latency is the visible number, but the resource result matters more. Mean GPU power across the reuse phase falls from 413 W to 119 W, because a 99K-token prefill is dense floating-point work on every layer, while a KV reload is a device transfer with the GPU largely idle. The power ratio understates the saving, because the shared arm also finishes sooner. Energy per follow-up is 413 W across 18.38 s, or roughly 7.6 kJ, against 119 W across 2.20 s, or roughly 0.26 kJ, about 29 times less energy to serve the same turn. The 2.20 s is not free: it is roughly 10 GB moving off the drive and across PCIe. At this context length that trade is heavily favourable, and it converts directly into workloads per GPU and into cost per token.
Figure 4. Mean GPU power during the reuse phase
Board power averaged across the reuse phase on the serving replicas. Shorter is better.
The shared arm spends NVMe read bandwidth in place of GPU prefill compute. Power falls 3.5x and energy per follow-up falls about 29x, because the shared arm also finishes 8.4x sooner. Preliminary results.
The numbers
Table 1. Headline comparison, private caches against a shared pool
Same drive, same model, same prompts, same order. Four replicas, cross-replica on 64 of 64 follow-ups, concurrency 1.
| Metric | Isolated | Shared | Delta |
|---|---|---|---|
| Reuse TTFT p50 | 18.38 s | 2.20 s | 8.4x |
| Reuse TTFT p95 | 19.5 s | 3.7 s | 5.3x |
| Reuse TTFT p99 | 19.6 s | 3.8 s | 5.2x |
| Reload rate, warm within 3 s | 0% | 89% | n/a |
| SLO attainment at 5 s | 0% | 100% | n/a |
| Wall clock, 64 requests | 338 s | 38 s | 8.8x |
| Mean GPU power | 413 W | 119 W | 3.5x |
| Errors | 0 | 0 | n/a |
Reload rate is the share of follow-ups served warm within 3 s. Delta is stated where the ratio is meaningful. Preliminary results.
Table 2. Tier comparison, TLC against QLC on the same workload
The medians match; the difference is in the tail.
| Signal | TLC · D7-PS1010 | QLC · D5-P5336 |
|---|---|---|
| Drive read during reuse | 6.07 GB/s | 5.21 GB/s |
| Rated sequential read | 14.5 GB/s | 7.0 GB/s |
| Average read I/O size | 972 KB | 123 KB |
| Queue depth p95 | 45.7 | 585 |
| r_await p50 / p95 | 2.3 / 4.3 ms | 5.0 / 10.1 ms |
| TTFT p50 / p95 | 11.4 / 15.3 s | 11.4 / 20.1 s |
| Composite temperature | 43 °C | 34 °C |
| Wear / available spare | 0% / 100% | 0% / 100% |
QLC issues smaller reads, which builds a 585-deep queue and pushes p95 to 20.1 s against 15.3 s. The QLC drive measured 5.2 GB/s against a rated 7.0 GB/s on its PCIe 4.0 link, about 74% of spec, so the tail difference is not the drive running out of headroom. Across repeated 600 GB pool builds and multi-hour sweeps both drives ran between 34 °C and 43 °C at 0% wear with 100% spare. Preliminary results.
So, should you share the pool?
Yes, for long-context serving on any fleet larger than one replica. With private caches the chance a follow-up finds its KV falls as one over the replica count, and every miss is a full recompute. A shared pool decouples the cache from the replica, so hit rate does not decay as the fleet grows.
On TLC if you are judged on the tail, on QLC if you are judged on capacity. The two tiers matched at the median. QLC ran cooler and holds eight times the capacity; TLC held p95 about 5 s lower.
No, if your contexts are short or your reuse is incidental. Below the crossover a recompute is cheaper than a 10 GB read, and you are not storage-bound at all.
The takeaway
We removed routing intelligence entirely and forced the worst case: every follow-up went to a replica that had never seen the prefix. A shared SSD-backed KV pool still returned 89% of those requests warm within 3 s, met a 5 s SLO on all of them, and did it at 3.5x less GPU power than recompute.
The result is not that a shared directory produces cache hits, which is obvious on paper. The result is that it produces 99K tokens, across four replicas writing concurrently into one pool, with zero errors and a tail that holds, on a drive that never became the bottleneck. Any one of those could have failed in practice and any one would have ended the idea.
A cache-aware router and a shared storage tier are not alternatives. Routing raises the odds the KV is already on the right GPU; a shared pool makes it recoverable when it is not, which is what autoscaling and failover guarantee will happen. The next question is cost across model tiers, where a router downgrades an easy turn to a cheaper model that holds no KV for the session and must recompute the whole context. The cheaper model is only cheaper when the cache is cold.
Caveats and dependencies
| Concurrency of the headline | The 2.20 s median is the leftmost point of the concurrency sweep. At 16 concurrent reloads the shared arm reaches roughly 11.7 s and at 32 roughly 42 s, so the headline describes a lightly loaded fleet. |
| Isolated arm under load | Thirty-two concurrent 99K-token prefills would queue heavily, so the isolated arm would also degrade, but we did not measure it. The comparison at load is therefore not defended by data. That run is the right follow-up. |
| Cold reads were forced | The page cache was dropped between every step, so every reload is a device read and the isolated arm is a strict worst case. A production fleet would serve some fraction from host memory. |
| The ceiling is in software | Throughput saturated at about four concurrent reloads inside the LMCache fetch and copy path, not at the drive. These storage numbers describe a pipeline that is not yet saturating the device, so the drive figures are a floor rather than a limit. |
| Matched parameters | Both arms share model, prompt set, prompt order, chunk size, GPU memory fraction and drive. The one deliberate difference is the cache directory. The chunk-size sweep was run separately and the headline arms are not necessarily at the best point. |
| Cross-replica key stability | Cross-replica hits require PYTHONHASHSEED=0 on every replica. This is a dependency of the deployment rather than a tuning choice, and the pool silently returns nothing without it. |
| Workload definition | Two related workloads exist in this programme: 64 documents at roughly 99K tokens, and 24 sessions over 8 shared prefixes. Every number reported here comes from the 64-document run. |
| Scope | One server, one model, one router policy, two drive tiers, one otherwise-idle machine. No multi-tenant contention was tested in this round, which is precisely where the GPU power saving would matter most. |
References
| 1. | Red Hat. "Master KV cache aware routing with llm-d for efficient AI inference," Red Hat Developer, 7 October 2025. 87.4% overall cache hit rate measured across pods. developers.redhat.com/articles/2025/10/07/master-kv-cache-aware-routing-llm-d-efficient-ai-inference |
| 2. | Baseten. "How Baseten achieved 2x faster inference with NVIDIA Dynamo." 50% TTFT and 34% TPOT reduction at an 89% hit rate across four replicas, roughly 50K-token inputs. baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo |
| 3. | LMCache. "Storage Backends: Local Disk," LMCache documentation. docs.lmcache.ai/kv_cache |
| 4. | vLLM. "Automatic Prefix Caching" and KV transfer configuration, vLLM documentation. docs.vllm.ai/en/latest/features/automatic_prefix_caching.html |
| 5. | Solidigm. "D7-PS1010 and D7-PS1030 Product Brief." Rated up to 14.5 GB/s sequential read, TLC, capacities to 15.36 TB. Solidigm. "D5-P5336 Product Specification," Rev 006, December 2024. Rated up to 7,000 MB/s sequential read, QLC, capacities to 122.88 TB. |
Disclaimer
A preliminary technical evaluation, for informational purposes only. Not a benchmark, product announcement, or performance guarantee. Measurements were made by Metrum AI in a single test configuration and are reported as observed behaviour, not as claims of throughput, latency or availability in other environments; results will vary with model, workload and deployment. The caveats above bound the conclusions. Solidigm is a trademark of Solidigm. RTX is a trademark of its respective owner. LMCache and vLLM are open-source projects. Qwen is a trademark of Alibaba Group. All other trademarks belong to their respective owners, and their use does not imply endorsement.