Streamlined Agentic Inference with KV Cache Offload On HPE ProLiant Compute DL380a Gen12 servers powered by Intel Xeon 6 processors

Executive Summary

Enterprise agentic workloads are quietly paying for the same work twice. As an autonomous agent reasons across dozens of rounds, its Key-Value (KV) cache grows until it overflows GPU memory, forcing the GPU to recompute context it has already produced rather than generate new tokens.

The HPE ProLiant Compute DL380a Gen12 with Intel Xeon 6787P processors removes that waste by putting the server's full memory hierarchy to work. The GPUs hold active context and stay fully engaged, while CPU DDR5 DRAM and NVMe SSDs persist the overflow and reload it on demand, all within one integrated server.

Metrum AI benchmarked four KV-cache configurations on real agentic traces, across two models that bracket the enterprise range: the flagship Qwen3-235B-A22B-FP8 on all four GPUs and the compact gpt-oss-20b on a single GPU. The DRAM + NVMe hierarchy is the most balanced configuration across the production range on both models, and the outright best at peak scale, sustaining the highest throughput and lowest latency at 512 users with no GPUs added or removed.

Key Results at a Glance

Agentic Workloads Invert the Inference Problem

Generative-AI value is increasingly judged by the cost-effectiveness of inference rather than the choice of model. As agentic workloads scale, the measure of success shifts from model accuracy to the efficiency of the serving stack. Every second a GPU spends reprocessing context it has already seen is a second not spent generating value, and agentic workloads make this problem acute. A conventional chat request is short and single-turn; agentic workloads invert almost every property of that model:

• Multi-round. Each session runs many rounds. Coding agents iterate over a task, calling tools, reading results, and revising. Representative coding traces show mean session lengths well into the hundreds of rounds.
• Long and growing context. Context grows monotonically. Every tool result, file read, and reasoning step is appended, so the KV cache for a single session expands turn over turn until it dominates VRAM.
• High reuse. Each new turn adds a small amount of fresh text on top of a large reused prefix, producing very high cache-hit potential and shifting the workload from compute-bound to I/O-bound.
• Host-bound phases. The GPU idles during tool execution. Retrieval, code execution, web fetches, and tokenization run on or are orchestrated by the host CPU, creating substantial GPU idle time within every round.

Recent work characterizing agentic execution from the host side finds the CPU to be a first-order bottleneck rather than a supporting actor. For retrieval-augmented pipelines, nearest-neighbor retrieval alone can consume 81 to 89% of end-to-end latency. A serving architecture tuned only for GPU FLOPs leaves the accelerator stalled while the host works. The path to higher effective utilization runs through faster context retention and retrieval, which is exactly what KV-cache offload to DRAM and NVMe provides.

The Case for Keeping Agents On-Premises

• Security and IP control. Agents require deep access to proprietary code and sensitive data, so on-premises deployment keeps intellectual property behind the corporate firewall under strict access control.
• Predictable economics. Agent workflows can generate volatile, runaway token costs because of large context windows. On-premises infrastructure converts unpredictable per-token operating expense into controlled, fixed capital expense.
• Unrestricted leverage. With per-token cost anxiety removed, engineering teams can scale large swarms of autonomous agents for continuous integration and deep codebase modernization without metering each call.

These pressures put the server's memory hierarchy at the center of the deployment decision. The question is not whether to buy a GPU, since every agent deployment needs one. It is how to allocate the rest of the server's memory, and when to activate each layer, to sustain the most concurrent agents within SLA at a given concurrency target.

When the KV Cache Outgrows VRAM, GPU-Only Collapses

GPU VRAM, the fastest memory in the server, is shared between holding the model weights (a fixed cost) and storing the KV cache for active sessions (a variable cost that grows with users and context length). On the flagship workload, Qwen3-235B-A22B-FP8, a sparse mixture-of-experts model with 22B active of 235B total, consumes roughly 235 GB of the four NVIDIA RTX PRO 6000 Blackwell GPUs' 384 GB of combined VRAM, leaving a finite KV budget (~72 GB) that fills around 32 concurrent users at the 37,000-token average session length.

Above that point the GPU-only configuration must evict sessions to admit new ones, then recompute each evicted session's entire context on its next turn before producing a single output token. The benchmark makes the consequence concrete: at 256 concurrent users, GPU-only TTFT degraded +496% from round 1 to round 25 (195.9 s to 1,168.0 s, a 6x worsening) as VRAM filled and the GPU reprocessed context it had already handled. At 512 users the collapse reaches +524%. This is not a fault of the GPU. It is the result of serving a long-context workload from a single memory layer that is too small for the working set.

Figure 1 | Qwen3-235B mean TTFT vs concurrency (log scale). GPU-only climbs into multi-minute territory once the KV budget saturates; the offload tiers stay an order of magnitude lower, and the DRAM + NVMe hierarchy is lowest at peak.

The compact gpt-oss-20b shows the same physics on a different timescale. Served on a single GPU with native 128k context, one GPU comfortably absorbs concurrency to 64 users with sub-7-second TTFT and no offload benefit, but at 128 users the KV budget saturates and GPU-only TTFT jumps to 74.6 s, then 215.8 s at 256 and 512.9 s at 512. The offload tiers hold 16 to 42 s at 128 to 256 users; the crossover where offload becomes essential is concurrency 128 for this model, versus conc 32 for Qwen3-235B. The smaller the model, the more KV headroom a GPU has, but the cliff is the same, only later.

Figure 2 | gpt-oss-20b mean TTFT vs concurrency (log scale). Flat to concurrency 64; at concurrency 128+ GPU-only collapses while the offload tiers stay an order of magnitude lower.

Reload the Cache Instead of Recomputing It

The KV cache does not have to live only in GPU VRAM. Beneath the GPU, the server's CPU DDR5 DRAM and NVMe SSD act as additional KV layers that persist computed tensors instead of discarding them. When a session resumes, LMCache reloads its cache from the nearest layer that holds it rather than recomputing it on the GPU, which removes the prefill penalty for cache hits. Because a reloaded cache contains the same tensors the GPU originally computed, offload is bit-exact: unlike KV quantization or token-eviction schemes, it carries zero accuracy trade-off. The three layers work as one system, and each maps cleanly onto how an enterprise agent uses memory.

Table 1 | The enterprise agentic memory hierarchy. The full configuration activates all three layers simultaneously, with DRAM serving the hot working set and NVMe absorbing the overflow.

Why the Host CPU Makes Offload Work

The DRAM and full-hierarchy tiers work because of the host. Two Intel Xeon 6787P processors (86 cores per socket, a 16-channel DDR5 subsystem) supply the aggregate memory bandwidth that turns CPU DDR5 into a high-performance KV layer. The 2,048 GB of installed DDR5 holds the working set, and the dual-socket topology stages KV blocks at speed across 176 PCIe Gen5 lanes to the four GPUs, which stay active throughout. Across every cell in the configuration grid, Intel Xeon 6787P CPU utilization stayed below ~4.3% across both models, confirming that the gains come from memory tiering and bandwidth, not additional CPU cores.

Match the Memory Tier to Your Concurrency

Figure 3 | Qwen3-235B generation throughput vs concurrency. Offload tiers deliver 2 to 2.7x the GPU-only throughput once the GPU saturates; the DRAM + NVMe hierarchy leads at peak. Plotted on a generation-throughput basis; compare with the sustained basis used in Figure 4.

• Light, low-concurrency work. Below the point where the GPU KV cache approaches capacity, roughly 32 users on Qwen3-235B and 64 on gpt-oss-20b, a developer running small tasks can use the GPU layer alone; activating offload adds only 3 to 4% overhead at the lowest concurrencies.
• Production agent fleets. Once concurrency fills GPU memory, activating the DRAM layer raises sustained throughput 2.7x at 256 users while the GPUs keep running. The DRAM + NVMe hierarchy matches DRAM alone here within 1% while keeping NVMe capacity in reserve, the practical production default.
• Organizational scale and persistence. When the working set exceeds DRAM, or KV cache must persist across restarts or be shared across nodes, the NVMe and full-hierarchy configurations sustain full throughput and a ~75% cache-hit rate while DRAM alone has already saturated. At 512 users the DRAM + NVMe hierarchy delivers the highest throughput and lowest TTFT of any tier on both models. In the mid-concurrency range on gpt-oss-20b (128-256 users), the CPU DRAM tier can lead on throughput; the full hierarchy's advantage is its consistency across the full range and its freedom from a saturation cliff.

Qwen3-235B configurations at a glance (256 concurrent users)

Table 2 | Memory configurations compared at 256 concurrent users (Qwen3-235B). The DRAM + NVMe hierarchy is statistically tied with the CPU DRAM configuration here while holding NVMe overflow in reserve.

At 512 Users, DRAM Saturates and the Full Hierarchy Takes Over

The DRAM layer is the right default for production concurrency, but it has a ceiling. At 512 users the Qwen3-235B working set exceeds the 1,024 GB allocated DRAM budget (half of the 2,048 GB installed, deliberately capped to leave ~1 TB of host memory free for the OS and agent orchestration). LMCache can no longer keep the working set resident, the external cache-hit rate collapses from 75.5% to 23.7%, and the DRAM tier begins recomputing context like GPU-only. Its throughput falls to 59 tok/s and TTFT rises to 1,294 s, barely better than GPU-only at the same scale. The NVMe layer has no such ceiling, and the DRAM + NVMe hierarchy combines DRAM speed with NVMe capacity, sustaining a 75.8% hit rate by spilling overflow to NVMe.

Table 3 | Qwen3-235B at 512 concurrent users. The NVMe drives peak at only ~16% utilization; the binding ceiling is PCIe Gen5 transfer, not SSD bandwidth.

Note: DRAM consumed figures reflect total host memory in use during the benchmark run, including OS, model-serving processes, and orchestration overhead. They are not equivalent to the LMCache KV allocation alone.

gpt-oss-20b reaches the same conclusion even more sharply. Its DRAM tier saturates to a 31.8% hit rate at 512 users while NVMe holds 88.4% and the full hierarchy 85.4%. The full hierarchy serves a larger share of its hits from fast DRAM, so it delivers the highest throughput of any tier despite a marginally lower aggregate hit rate than NVMe alone. It completes the 12,800-request workload in 0.91 h versus 4.22 h for GPU-only, a 4.6x speed-up, at 566.8 tok/s sustained. Because the model is small, the full hierarchy never has to read much from NVMe until 512 users, so its overflow path stays nearly free until it is genuinely needed.

Table 4 | gpt-oss-20b at 512 concurrent users. The full hierarchy is the fastest and highest-throughput of all four configurations. Throughput is reported on a sustained (wall-clock-normalized) basis, consistent with the Key Results table; the speed-up row is the wall-clock ratio.

Figure 4 | gpt-oss-20b sustained throughput vs concurrency. Flat to concurrency 64; at concurrency 128+ offload tiers deliver 3 to 5x the GPU-only throughput. Note: plotted on a sustained basis, unlike the Qwen3-235B generation-throughput chart in Figure 3.

SLA-based capacity on P99 latency (Qwen3-235B)

Table 5 | Maximum concurrent agents within each P99-TTFT target. At the 300-second target used for background fleets, the offload tier serves 128 agents vs 32 for GPU-only, a 4x capacity gain on the same server, with no added GPUs.

Benchmark Methodology

The serving configuration influences TTFT more than the storage tier does, so the configuration is locked identically across all four scenarios and the offload tier is the only variable that changes. Both models share the same server, workload generator and tier definitions.

System under test

Table 6 | System under test (shared by both models).

Table 7 | The two model runs. Both replay sammshen/lmcache-agentic-traces (SWE-bench Verified, GAIA, WildClawBench), 25 rounds/session, output capped at 512 tokens/round, concurrency 1-512, closed-loop.

Agentic workload

The workload is replayed from real agentic conversation traces rather than generated synthetically, which is the reason the results transfer to production. Sessions are capped at 25 rounds; because TTFT degradation compounds round over round, this is conservative relative to the production coding traces cited earlier, which run to hundreds of rounds. Output length was capped at 512 tokens per round across all runs, while input context accumulated turn over turn.[1]

Table 8 | Agentic workload

Note on concurrency points. The two runs used slightly different concurrency points. Qwen3-235B was swept at 1, 2, 4, 8, 16, 32, 128, 256, 512 users; gpt-oss-20b at 1, 4, 8, 16, 32, 64, 128, 256, 512. The KV Cache is cleared from all the tiers between concurrencies. Side-by-side comparisons are drawn only at concurrency points present in both runs.

Configurations under test

All four configurations run on the same server with the GPUs installed and active. They differ only in which memory layers the inference engine is allowed to use for KV cache.

Table 9 | The four memory configurations under test. Between runs the cache is cleared and the server restarted before any concurrency change; GPU clocks are locked and the box is single-tenant during steady-state windows.

Results Across Both Models and Every Configuration

DRAM + NVMe hierarchy vs GPU-only at 256 concurrent users (Qwen3-235B)

Table 10 | DRAM + NVMe hierarchy vs GPU-only at 256 concurrent users (Qwen3-235B). The full hierarchy matches the CPU DRAM configuration here while holding NVMe capacity in reserve. The 2.7x throughput headline reflects the CPU DRAM configuration (124.9 tok/s); the DRAM + NVMe hierarchy delivers 124.0 tok/s, a 2.6x gain.

Round-to-round degradation at 256 users

Table 11 | GPU-only does not merely start behind; it deteriorates as context grows. The offload tiers keep the system generating tokens rather than recomputing context, and at 256 users the full hierarchy actually finishes round 25 faster than round 1 on gpt-oss (-63%).

Where the data path binds

On Qwen3-235B at 256 users, combined PCIe traffic measured ~54.6 GB/s with the GPU cache alone (recomputation churn) against ~33 GB/s with DRAM and with the full hierarchy; activating offload cuts PCIe traffic ~39% because cache hits eliminate full prefill cycles. The binding ceiling for the offload tiers is PCIe Gen5 transfer, not raw DDR5 or SSD bandwidth.

Throughput and energy

On Qwen3-235B, generation throughput rose from 47 tok/s (GPU-only) to 124.9 (DRAM) and 124.0 (Full) at 256 users, a 2.7x and 2.6x uplift, while tokens-per-joule improved 2.7x (0.090/0.089 vs 0.033). gpt-oss-20b shows the same inversion at conc256: 0.195 tok/J GPU-only vs ~1.2 to 1.50 tok/J for the offload tiers, 6 to 8x efficiency gain once the GPU stops recomputing context. All energy figures are sourced from Intel RAPL and NVIDIA NVML package telemetry.

Deployment Guidance for Agentic Fleets

• Deploy the full server, then activate every memory layer. The path to the highest sustained throughput is not a different server or fewer GPUs. It is the complete DL380a with its whole memory hierarchy working. Activating offload raised Qwen3-235B throughput 2.7x at 256 users with the GPUs unchanged.
• Make the DRAM + NVMe hierarchy the production default. On both models it matches DRAM through the production range and is the outright best at 512 users, with the lowest TTFT, highest throughput, and best energy efficiency, and no saturation cliff. Unless a node genuinely lacks NVMe or DRAM, run all three layers.
• Treat a single memory layer as non-viable for sustained load. GPU-only degraded Qwen3-235B TTFT nearly 6x (+496%) from round 1 to round 25 at 256 users, and gpt-oss-20b collapses at concurrency 128. For any multi-round, long-context workload above its crossover point, activate DRAM and NVMe from the start.
• Match the deployment to the model. A large MoE like Qwen3-235B fills all four GPUs as one replica and crosses into offload-required territory by roughly 32 users. A compact model like gpt-oss-20b runs one replica per GPU and does not cross into offload-required territory until roughly 128 users; because each replica needs only a single GPU, one four-GPU server can host several independent replicas.

The Bottom Line

Agentic AI has changed what an inference benchmark must measure. The workloads that now drive enterprise spend are multi-round, long-context, high-reuse, and host-bound, and under that profile the efficiency of the memory hierarchy, not raw GPU compute, sets the ceiling on how many agents a node can serve within SLA.

On the same fully configured DL380a in every test, activating the CPU DRAM layer raised sustained Qwen3-235B throughput 2.7x at 256 users while lowering TTFT 63% and improving energy efficiency 2.7x. At 512 users, where DRAM saturates, the DRAM + NVMe hierarchy sustained that lead on a capacity basis, completing the same 12,800-request workload in 5.13 h versus 12.67 h for GPU-only - a 2.47x wall-clock speed-up at 512 users, the best of any tier, while extending KV capacity to organizational scale. The compact gpt-oss-20b tells the same story 4.6x faster at peak. The question is not whether to buy GPUs; every agent server needs them. It is how much of the GPU investment already in place is being spent recomputing context that the server's own DRAM and NVMe could serve in a fraction of the time.

Next steps

• Size your deployment against your own concurrency and SLA targets using these configurations and throughput figures.
• Engage your HPE and Intel account team to model sustained throughput and energy efficiency for your agent workload, at either model scale, on the DL380a Gen12.
• Validate on your traces by replaying a representative agentic workload across the GPU, DRAM, NVMe, and full-hierarchy configurations on the same server.

Appendix: Test Platform Reference

Glossary

Run Configuration

References

1. Raj et al. Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective. arXiv:2511.00739, 2025.
2. Intel. Intel Xeon 6 Agentic AI Platform Briefing. June 2026.
3. Metrum AI, HPE, and Intel. KV Cache Offload Benchmark, full-grid run, Qwen3-235B-A22B-FP8. full-grid-sliding-safe-20260604T153951Z, June 2026.

Images

1. HPE image attribution: HPE ProLiant Compute DL380a Gen12 product image sourced from Hewlett Packard Enterprise at buy.hpe.com. Hewlett Packard Enterprise, HPE, HPE ProLiant, and the HPE logo are trademarks or registered trademarks of Hewlett Packard Enterprise Development LP.
2. Intel image attribution: Intel® Xeon® 6787P processor product image sourced from Intel Corporation at intel.com. Intel, Intel Xeon, and the Intel logo are trademarks or registered trademarks of Intel Corporation in the U.S. and/or other countries.

Copyright © 2026 Metrum AI, Inc. All Rights Reserved. NVIDIA and RTX are trademarks of NVIDIA Corporation. Intel and Xeon are trademarks of Intel Corporation. HPE and ProLiant are trademarks of Hewlett Packard Enterprise. KIOXIA is a trademark of KIOXIA Corporation

Testing performed by Metrum AI in collaboration with HPE and Intel. All performance figures represent observed measurements under the described test conditions on a single HPE ProLiant Compute DL380a Gen12 configuration. Results vary by model, hardware configuration, software version, and deployment workload, and should not be interpreted as guarantees of performance on different systems. The relative comparisons between memory configurations are the transferable result.

[1] Output length was capped at 512 tokens per round across all runs. Input (context) length was not capped and accumulated across rounds, averaging roughly 37,000 tokens per session.