KV cache compression is usually applied inside GPU memory, where it taxes every token you generate. LMCache lets you apply it at the storage boundary instead. We benchmarked TurboQuant both ways on Solidigm NVMe.
Compression made the storage tier more valuable, not less.
Aggressive KV compression was widely read as a reduction in storage demand. On a capacity-bound tier it does the opposite. It multiplies the scarce resource instead of substituting for it.
What we found
Compression and offload are usually presented as alternatives, and when Google Research's TurboQuant landed at ICLR 20261 with 3-bit KV quantization it was widely read as evidence that compression would reduce AI infrastructure's memory and storage needs. The more useful question is narrower. Where should compression happen, and what does each answer cost?
Applied inside GPU memory, a compressed cache is dequantized on every attention call, so the cost lands on every token generated. Applied at the storage boundary, it is paid once on the way to the drive and once on the way back, and the copy the GPU attends over is never quantized. LMCache makes the second placement available through a per-adapter serde4 on its L2 tier, with TurboQuant as one of the options. This evaluation measures both.
What we ran
Two rounds. Round one prefills and offloads to the Solidigm drive. Round two re-queries after eviction from HBM and from the host L1 tier, forcing a restore from flash rather than a cache hit in memory. Only high-coverage runs are counted.
| Model | Qwen3-Coder-30B-A3B-Instruct-FP8, BF16 KV cache in HBM |
| Hardware | 4x NVIDIA RTX PRO 6000 Blackwell, 96 GB each, 1 GPU serving |
| Storage | Solidigm D7-PS1010 (TLC, 15.36 TB) and D5-P5336 (QLC, 122.88 TB) |
| Stack | vLLM 0.22 (V1), LMCache 0.5.1 MP, fs L2 adapter, TurboQuant serde |
| Serde | max_workers 8, block_size 16, 25 GB host L1 |
| Workload | 48 prompts at roughly 30k tokens |
Grouped-query attention with 4 KV heads across 48 layers9 puts KV at 96 KiB per token in BF16, about 2.8 GiB for a 30k-token session. Models with full multi-head attention are several times more demanding, so the economics below are more pronounced, not less.
Compressing in GPU memory taxes every token. Compressing on the way to disk mostly does not.
In HBM, the cache is dequantized on every attention call, so the penalty applies to every token of every response and scales with compression aggressiveness. At the storage boundary the codec runs in under a millisecond, once per write and once per read, off the attention path entirely, so that mechanism is absent.
A residual remains. On the 4-bit preset, time-per-output-token rose from 0.118 s to 0.130 s, giving up 9% of decode throughput on a path that does not touch attention. Contention during offload is the likely mechanism, which makes it a tuning surface rather than a structural cost. Warm-restore latency rose 9.6%, from 21.9 s to 24.0 s, once per restore rather than per token.
Figure 1. How much decode speed each approach gives up
Percent of decode throughput lost against that configuration's own uncompressed baseline. Shorter is better.
The three upper bars are charged on every token generated. The lower bar is a one-time cost on the storage path. Each is measured against its own uncompressed baseline, so read them as costs rather than as absolute speed. Preliminary results.
For external calibration, a Red Hat and vLLM study2 across four models from 30B to 200B parameters reports 20% to 52% throughput reduction for TurboQuant in GPU memory and latency overheads of 10% to 68%. The 14% to 34% here sits at the conservative end of that range.
4-bit holds up. 3-bit falls off a cliff.
For the in-GPU placement this is already documented. The same study2 reports k8v4 even with the BF16 control, 4bit_nc down 1 to 4 points, and both k3v4_nc and 3bit_nc down by as much as 20 points on hard math and coding, with a recommendation to avoid the aggressive variants. Our own in-GPU gsm8k runs track that shape: 0.833 at k8v4 against a 0.833 control, 0.800 at 4-bit, 0.640 at 3-bit.
What remains open is whether the same holds at the storage boundary, where the error path differs: the KV is quantized once, written to flash, read back and reconstructed before the model sees it. To measure it, an independent LLM judge compared round-one responses generated from fresh KV against round-two responses generated from restored KV, scoring semantic equivalence, with exact-match retrieval alongside.
Figure 2. Accuracy of responses generated from SSD-restored KV
Semantic equivalence against responses generated from fresh KV. Compare each bar to the control.
The control scores 0.90 rather than 1.00 because the judge does not always call two responses generated from identical KV equivalent. That is the ceiling for every other bar. k8v4 and 4-bit sit at it; 3-bit is 28 points below. Preliminary results.
That control is also the resolution limit. A ten-point spread sits inside the judge's own disagreement rate, and the published in-GPU figures put 4bit_nc 1 to 4 points below its control, well inside it. The supportable statement is therefore narrower than losslessness: 3-bit degrades accuracy substantially and unambiguously, and 4-bit is not distinguishable from the uncompressed control at the resolution of this test. Moving the codec to the storage boundary adds no accuracy penalty of its own and does not rescue the aggressive presets.
On scope, this is TurboQuant's 3bit_nc preset on open-ended generation, not a general result for 3-bit KV quantization, which other schemes handle differently using per-channel calibration7. Exact-match retrieval returned 1.00 at every preset including 3-bit, so it is reported as a saturated negative control rather than as evidence of fidelity.
Bandwidth was not the constraint. Capacity was.
Peak SSD write throughput during offload measured 1.5 GB/s uncompressed and 0.2 GB/s on the 4-bit path, against roughly 3.3 GB/s of sequential write for the QLC D5-P53365 and up to 10 GB/s for the PCIe 5.0 TLC D7-PS10106.
Figure 3. Write bandwidth used, against each drive's limit
Solid blocks are bandwidth consumed. Open blocks are headroom left unused.
Even uncompressed, writes reached about 45% of the capacity-optimized drive and about 15% of the performance drive. The offload is capacity-bound, not bandwidth-bound, which determines what compression is for. Preliminary results.
Bytes and peak rate do not fall by the same factor: the 4-bit path writes about 3.8 times fewer bytes, set by the codec ratio, while its peak rate falls roughly 7.5 times because the codec is slower per byte. Only the byte reduction converts into sessions per drive. This is consistent with external work; Tutti3 finds the bottleneck in tiered KV cache is not raw SSD bandwidth, citing the same D7-PS1010 class of drive, and locates the constraint in the CPU-centric data path.
So bandwidth was not the scarce resource; capacity was, and every byte saved converts directly into session state retained rather than headroom reclaimed. At 2.8 GiB per session, a 122.88 TB D5-P5336 holds on the order of 41,000 sessions uncompressed and 157,000 at 4-bit, which follows from the codec ratio rather than from measured on-drive density. Writing 3.8 times fewer bytes also extends usable life by roughly the same factor on media specified for modest drive writes per day. QLC was never bandwidth-limited here, so compression does not move a workload onto the capacity tier; the case for that tier rests on capacity economics, which compression only strengthens.
The bill arrives on the host, not the GPU.
Host CPU during offload rose from 1.9 cores uncompressed to 4.8 cores at 4-bit, a factor of 2.5. The compression kernel itself is a GPU Triton kernel; the additional host CPU is orchestration. The two costs differ in kind: host CPU is provisioned once, at deployment time, against a known offload rate, while a per-token decode penalty is paid on every response the fleet generates and grows with traffic.
One default will make this look broken
LMCache's serde max_workers defaults to 14. At one worker, compression cannot keep pace with generation and offload coverage collapses to 16%, leaving the storage tier effectively idle. Eight workers reach 88%, sixteen reach 96%. A deployment that enables the serde, leaves the defaults alone and measures the result will conclude the approach does not work. Tutti3 identifies the same low-parallelism CPU path as the severe bottleneck in tiered KV cache.
Figure 4. Offload coverage by serde worker count
Coverage is the share of KV tokens that reached the SSD.
At the shipped default the storage tier is inert. Set max_workers to 8 or more, sized against generation rate. Preliminary results.
Why density is what matters at the storage tier
The Red Hat and vLLM study2 recommends FP8 KV cache as the best default and finds TurboQuant k8v4 offers no meaningful advantage over it. For what to keep in GPU memory, that holds: FP8 quantizes the attention computation itself on hardware-native tensor cores, so there is nothing to unpack and no decode penalty, whereas TurboQuant compresses storage only and must dequantize to BF16 before attention.
At the storage boundary neither option is on the attention path, so that advantage does not apply. What remains is a trade between how much a codec shrinks the cache and how long it takes to do it, and Table 1 sets both out. The fp8 serde shrinks the KV cache by 2.0 times and encodes in 0.024 ms. TurboQuant 4-bit shrinks it by 3.8 times and encodes in 0.524 ms. Choosing TurboQuant over fp8 therefore fits about 1.9 times more sessions on the same drive, and costs about 22 times more encode time to get there, spent in host CPU during offload rather than on every token generated. On a capacity-bound tier that extra density is precisely what is being bought, which argues for the slower codec. The reasoning comes from the codec figures rather than an end-to-end run, so the head-to-head on the storage path is still open.
The numbers
Table 1. Codec characteristics, measured at the serde
Model-independent, measured against a BF16 KV cache.
| CODEC | KEY / VALUE BITS | COMPRESSION | ENCODE MS | DECODE MS | FIDELITY |
|---|---|---|---|---|---|
| fp8 serde | FP8 / FP8 | 2.00x | 0.024 | 0.031 | 0.9996 |
| turboquant k8v4 | FP8 / 4-bit | 2.61x | 0.368 | 0.468 | 0.9973 |
| turboquant 4bit_nc | 4-bit / 4-bit | 3.82x | 0.524 | 0.583 | 0.9952 |
| turboquant k3v4_nc | 3-bit / 4-bit | 4.34x | 0.520 | 0.578 | 0.9891 |
| turboquant 3bit_nc | 3-bit / 3-bit | 5.02x | 0.523 | 0.581 | 0.9804 |
Against an FP8 resident cache the effective multiplier is roughly half these figures. Fidelity is reconstruction correlation and does not predict downstream quality: 3bit_nc reconstructs at 0.9804 yet loses 28 points in Figure 2. Preliminary results.
Table 2. Where the cost lands, at the 4-bit preset
Same preset, three placements.
| PLACEMENT | DECODE COST / TOKEN | WARM-RESTORE TTFT | HOST CPU | ACCURACY |
|---|---|---|---|---|
| No compression, offload to SSD | baseline | 21.9 s | 1.9 cores | 0.90 |
| 4-bit, compressed on the SSD | -9% | 24.0 s, +9.6% | 4.8 cores, 2.5x | 0.90 |
| 4-bit, compressed in GPU memory | -29% | not applicable | not applicable | 0.800 |
The bottom row has no SSD and no serde, so restore latency and serde CPU do not apply. Its accuracy comes from gsm8k against an FP8-in-HBM control and is not comparable to Figure 2. Preliminary results.
So, should you turn it on?
Yes, at 4-bit, with workers provisioned, for reuse-heavy long-context workloads. Agents, multi-turn coding sessions and long-context RAG share one signature: long prefixes and retrieval artifacts referenced across many turns. That is state worth keeping, and keeping it is a capacity problem. Not at 3-bit: a 28-point drop is not a tuning issue, and the extra 1.3 times compression does not pay for it at any capacity price. For first-restore latency, raw FP8 on the TLC tier is the better choice, with no serde CPU and no codec on the read path.
Table 3. Which prompts to offload, in what format, on which tier
Format follows reuse economics, tier follows latency sensitivity. The two decisions are separable.
| PROMPT PROFILE | OFFLOAD | FORMAT | SOLIDIGM TIER | WHY |
|---|---|---|---|---|
| One-off, short, latency-first | No | keep in HBM | none | No reuse, store cost wasted |
| Hot, reused, latency-sensitive | Yes | raw FP8 | D7-PS1010 TLC | Fastest restore, no serde CPU |
| High-volume reuse, capacity-bound | Yes | TurboQuant 4-bit | D5-P5336 QLC | Roughly 3.8x more sessions per drive |
| Precision-critical reasoning | Yes | FP16 | either tier | Never 3-bit, 28 points below control |
A reusable prompt left uncached is re-prefilled on every hit, spending GPU time to save storage. Reuse rate is a property of the workload, not the hardware. Preliminary results.
The takeaway
The drive was not under bandwidth pressure in any configuration tested. What the storage tier was short of was capacity, and that determines what compression is for. Applied at the storage boundary, compression stops being a substitute for capacity and becomes a multiplier on it: the full-precision cache the GPU attends over is preserved, the codec is paid once per round trip instead of once per token, and every byte saved is another session that does not have to be recomputed. The costs are host CPU, a restore latency premium, and an accuracy floor that rules out the most aggressive preset, all budgeted at deployment time rather than paid on every response.
Caveats and dependencies
| Scope | One model, one prompt length, 48 prompts, a single pass per configuration. No run-to-run variance established. |
| Density | Compression ratios are codec ceilings measured on a fixed tensor, not megabytes per token on the deployed model. Sessions per drive is inferred from them. |
| Restore latency | Of the 21.9 s, roughly 2 s is data movement. The rest is host orchestration, a known property of the current SSD tier with upstream work in flight. |
| Decode residual | The 9 percent is attributed to contention on circumstantial evidence. GPU kernel occupancy was not instrumented and is an equally plausible mechanism. |
| Concurrency | Bandwidth headroom was measured at low concurrency and shrinks as simultaneous offloads scale. |
| Drive limits | Figure 3 compares measured peaks against datasheet sequential-write figures. QLC steady-state write under continuous offload was not tested. |
| Two code paths | The in-GPU arm cannot be produced by LMCache serde and was measured separately, so cross-placement comparisons are directional. |
| fp8 serde | Characterized at codec level only. The end-to-end comparison against TurboQuant on the storage path is the open question. |
| Untested preset | k3v4_nc was not run on the storage path, and the published in-GPU figures group it with 3bit_nc rather than 4bit_nc. |
| Resident dtype | All runs keep BF16 KV in HBM. FP8-resident offload is supported but not characterized here. |
References
| 1. | A. Zandieh, M. Daliri, M. Hadian, V. Mirrokni. "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate." ICLR 2026. arXiv:2504.19874 |
| 2. | E. Kurtic, M. Goin, A. Marques, Red Hat AI. "A First Comprehensive Study of TurboQuant: Accuracy and Performance." vLLM Blog, 11 May 2026 |
| 3. | "Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving." 2026. arXiv:2605.03375 |
| 4. | LMCache. "KV Cache Compression: per-adapter serde." LMCache documentation, docs.lmcache.ai/mp/serde.html |
| 5. | Solidigm. "D5-P5336 High-Density PCIe 4.0 SSD for AI." Product documentation |
| 6. | Solidigm. "D7-PS1010 and D7-PS1030 Product Brief." Rated up to 14.5 GB/s sequential read, 10 GB/s sequential write |
| 7. | Z. Liu, J. Yuan, H. Jin, et al. "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization." NeurIPS 2024. arXiv:2401.18079 |
| 8. | "IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compression." 2026. arXiv:2603.28430 |
| 9. | Qwen. "Qwen3-Coder-30B-A3B-Instruct-FP8." Model card, huggingface.co/Qwen |
Disclaimer
A preliminary technical evaluation, for informational purposes only. Not a benchmark, product announcement, or performance guarantee. Measurements were made by Metrum AI in a single test configuration and are reported as observed behaviour, not as claims of throughput, latency, accuracy or availability in other environments; results will vary with model, workload and deployment. Accuracy figures for the two placements used different scoring methods and are not interchangeable. The caveats bound the conclusions. Solidigm, D5-P5336 and D7-PS1010 are trademarks of Solidigm. NVIDIA and RTX are trademarks of NVIDIA Corporation. TurboQuant is described in research published by Google Research. LMCache and vLLM are open-source projects. Qwen is a trademark of Alibaba Group. All other trademarks belong to their respective owners, and their use does not imply endorsement.