KV cache offload normally routes every retrieved byte through host RAM on its way to the GPU. GPUDirect Storage moves it from NVMe straight into GPU memory instead. We benchmarked both paths on a 100K-token serving workload on the Solidigm D7-PS1010.
The 2.9x speed-up is what you notice first; the host bandwidth it frees is what scales as the box gets busier.
GDS restored a 100K-token cache 2.9x faster than the conventional POSIX path while using 5x less host memory bandwidth, returning roughly 7.5 GB/s to the rest of the system.
What we found
Offloading KV cache to SSD is now a standard way to avoid re-prefilling long contexts, and the case for it is well established: a 100K-token session costs seconds of GPU time to rebuild and a few gigabytes to store. What is less settled is which path the bytes take on the way back, and whether that choice is worth configuring deliberately.
There are two. The conventional path reads from the filesystem into a host buffer, then copies that buffer into GPU memory: two hops, both through DRAM. GPUDirect Storage1 replaces them with a single peer-to-peer DMA from the NVMe device into GPU memory across PCIe, with no host bounce. Its value is not primarily latency but resource, since skipping the bounce frees host memory bandwidth and CPU cycles for everything else on the box. LMCache exposes both paths2, which makes them directly comparable. This evaluation measures both at 100K-token scale.
What we ran
Every configuration serves the same 40-request sequence: 20 unique long prompts, then 20 repeats. Each repeat is a genuine restore, since the session has been evicted from GPU memory before it is re-requested. The model is served under vLLM3 at tensor-parallel size 2, with LMCache handling KV offload to its local-disk L2 backend and cuFile driving the GDS path. Reported metrics are medians across the 20 repeat windows, with disk, DRAM, PCIe and CPU samplers clipped to each request's start and end timestamps.
| Model | Qwen3-Coder-30B-A3B-Instruct-FP8 (30B MoE / 3B active), tensor-parallel size 2 |
| Hardware | 2x RTX PRO 6000 Blackwell Server Edition |
| Storage | Solidigm D7-PS1010 (TLC, 15.36 TB), XFS, shared by both tensor-parallel ranks |
| Stack | vLLM 0.26.0, LMCache 0.5.3rc2, cuFile / GDS |
| KV backend | LMCache local-disk L2; offload chunk size 1024 (GDS) and 256 (O_DIRECT) |
| Workload | 40 requests: 20 unique prompts of 98,058-100,022 tokens, plus 20 repeats |
| Method | Medians across 20 repeat windows |
At roughly 9 GB of KV state per session, restore time is dominated by moving bytes rather than by per-request overhead. That is the regime where the data path is worth arguing about; at a few thousand tokens it is not.
Both paths read directly from the Solidigm D7-PS1010.
Both paths read directly from the Solidigm D7-PS1010 during the restore. The POSIX O_DIRECT path sustains 1.97 GB/s and GDS 4.74 GB/s, both comfortably inside the drive's rated throughput4, so the SSD is doing genuine work on every repeat rather than being served from memory.
Figure 1. Physical disk read during retrieval
Bytes read from the device inside each repeat window.
Both paths read from the same Solidigm D7-PS1010. Preliminary results.
GDS restores a 100K-token session 2.9x faster.
GDS restores the evicted 100K-token session in 1,725 ms against 4,952 ms on the O_DIRECT path, both reading the same payload from the same drive. The gap is a property of the data path itself, not of tuning around it.
Figure 2. Time to restore a 100K-token session
End-to-end, median of 20 repeat requests. Shorter is better.
Both paths read the same payload from the same drive. Preliminary results.
The saving lands in host memory bandwidth.
Latency is the visible number, but the resource result matters more. During retrieval the O_DIRECT path drives 9.41 GB/s of host DRAM traffic, while GDS drives 1.85 GB/s for the same restore. The O_DIRECT path still stages every byte through a host buffer before the copy to GPU memory; only the peer-to-peer path removes that transit.
On an idle box that bandwidth is effectively free. On a box running other work it is contended, and the 7.5 GB/s returned to the rest of the system is the substantive result rather than the 3.2 seconds of latency.
Figure 3. Host DRAM bandwidth consumed during retrieval
Total memory bandwidth inside each repeat window. Shorter is better.
Sampling is coarse relative to the window; see caveats. Preliminary results.
The host copy is the bottleneck, not the drive.
If the Solidigm D7-PS1010 were the limit, both paths would deliver bytes to the GPU at similar rates and differ only in how long they took to start. They do not. GDS lands 4.82 GB/s into GPU memory against 2.04 GB/s on the O_DIRECT path, reading the same data from the same drive. The staging copy through host memory is what caps the conventional path, not the SSD.
Figure 4. Delivery rate into GPU memory
PCIe receive, both tensor-parallel ranks summed. Longer is better.
Higher delivery rate means the link is occupied for less wall-clock time. Preliminary results.
Higher CPU rate, but 41% less CPU consumed.
While the restore is running, GDS uses more CPU per second: 3.20 cores against 1.91 on the O_DIRECT path. But it runs for far less time, so the CPU actually consumed per request is lower: 5.53 core-seconds against 9.37, a 41% reduction. GDS runs hotter for 1.73 s instead of cooler for 4.91 s, and the product is what the workload pays for.
The distinction changes how the figure is read. The per-second rate is what would govern how many concurrent streams a single box can hold; the per-request total is what shows up on the bill for a given amount of work.
Figure 5. Host CPU consumed per request
Core-seconds of host CPU per request, over the full repeat phase (87 and 33 samples at 99.9% busy). Shorter is better.
GDS draws more CPU per second (3.20 cores against 1.91) but for far less time, so it consumes 41% less host CPU per request. Preliminary results.
The numbers
Table 1. Measured medians, 20 repeat requests per configuration
Same drive, same workload, same model.
| CONFIGURATION | RESTORE | DISK READ | DRAM BW | PCIE RX | CPU / REQ |
|---|---|---|---|---|---|
| O_DIRECT | 4,952 ms | 1.97 GB/s | 9.41 GB/s | 2.04 GB/s | 9.37 core-s |
| GDS | 1,725 ms | 4.74 GB/s | 1.85 GB/s | 4.82 GB/s | 5.53 core-s |
CPU / REQ is host CPU consumed per request, in core-seconds. Cold prefill TTFT was 11,201 ms and 9,992 ms respectively; see caveats. Preliminary results.
So, should you turn it on?
Yes, for long-context workloads with real reuse where the host is doing other work. The latency gap is the easy sell, but the durable argument is the 7.5 GB/s of memory bandwidth returned to the rest of the machine.
No, if your working set fits comfortably in RAM. In that case the page cache will outrun both paths, and you are not storage-bound at all.
The takeaway
For long-context serving with genuine reuse, the path between the drive and the GPU is worth configuring deliberately. Both paths read from the same Solidigm D7-PS1010 at multiple gigabytes per second, so the drive is not the limiting factor; the cost of the conventional path is the copy that stages every byte through host memory on the way to the GPU. GPUDirect Storage removes that hop, restoring an evicted 100K-token cache 2.9x faster and returning roughly 7.5 GB/s of host memory bandwidth to the rest of the system. The latency improvement is the visible result; the reclaimed host bandwidth is the one that compounds as more work runs on the same machine.
Caveats and dependencies
| Scope | One model, one context length, one drive, one otherwise-idle machine. No multi-tenant contention was tested in this round, which is precisely where the DRAM saving would matter most. |
| Parameter asymmetry | The GDS configuration used a chunk size of 1024 against 256 on the O_DIRECT path, chosen from earlier measurement of per-file registration overhead. This is not a matched-parameter comparison and some of the gap may be attributable to it. |
| DRAM sampling | The memory-bandwidth counter samples at roughly 1.2 s against retrieval windows of 1.7-5.0 s, so those figures rest on one to three samples per window. Treat them as directional. |
| Upstream cache | 3 of 20 repeats in every configuration were served by the GPU prefix cache with no storage access. They are included in the medians and affect both configurations equally. |
| Latency capture | On the fastest GDS responses the client did not record a first-token timestamp. End-to-end latency is used throughout and is unaffected. |
| Cold prefill | GDS showed roughly 11% lower cold-prefill TTFT. Cold prefill is compute-bound and should not depend on the storage path; this is unexplained and is not claimed as a GDS benefit. |
| CPU rate | The per-second CPU figure, higher for GDS, would govern how many concurrent streams a single box can hold. This sequential test cannot measure that; a concurrency test is the right follow-up. |
References
| 1. | NVIDIA. "Magnum IO GPUDirect Storage Overview Guide," r1.18. docs.nvidia.com/gpudirect-storage/overview-guide |
| 2. | LMCache. "GDS Backend," LMCache documentation. docs.lmcache.ai/kv_cache/gds.html |
| 3. | vLLM. "Automatic Prefix Caching," vLLM documentation. docs.vllm.ai/en/latest/features/automatic_prefix_caching.html |
| 4. | Solidigm. "D7-PS1010 and D7-PS1030 Product Brief." Rated up to 14.5 GB/s sequential read, 9.3 GB/s sequential write, TLC, capacities to 15.36 TB. |
Disclaimer
A preliminary technical evaluation, for informational purposes only. Not a benchmark, product announcement, or performance guarantee. Measurements were made by Metrum AI in a single test configuration and are reported as observed behaviour, not as claims of throughput, latency or availability in other environments; results will vary with model, workload and deployment. The caveats above bound the conclusions. Solidigm is a trademark of Solidigm. RTX and GPUDirect are trademarks of their respective owners. LMCache and vLLM are open-source projects. Qwen is a trademark of Alibaba Group. All other trademarks belong to their respective owners, and their use does not imply endorsement.