Roofline Benchmarking & Auto-Tuning LLM Serving: How We Pushed Pure PyTorch to 27 tok/s
Most engineers treat LLM serving as an opaque black box—guessing thread counts, setting arbitrary batch sizes, and hoping VRAM doesn't crash. Here is how empirical roofline modeling and batched paged KV-cache gather eliminated DRAM memory bandwidth waste.
1. The Fundamental Memory-Bandwidth Bottleneck
Autoregressive LLM generation is overwhelmingly memory-bandwidth bound. During single-token decoding, every forward pass reads the model's entire weight matrix from GPU DRAM into on-chip cache just to emit a single token:
$$\text{Arithmetic Intensity} \approx \frac{2 \times P \text{ FLOPs}}{2 \times P \text{ Bytes}} \approx 1 \text{ FLOP / Byte}$$
On an entry-level consumer laptop GPU like the NVIDIA GeForce RTX 3050 (192 GB/s memory bandwidth), theoretical peak decode speed for a 0.5B model in bfloat16 (1 GB of weights) is strictly capped by how fast weights stream from DRAM:
Theoretical Max Decode = 192 GB/s / 1.0 GB ≈ 192 tokens / second (unbatched upper limit)However, in naive Python serving runtimes, request concurrency often tanks generation speed rather than accelerating it. Why?
2. The Serial Iteration Flaw vs. Unified Batched Decoding
In naive continuous batching architectures, active sequences are stepped in a serial loop:
# Naive unbatched loop: Reads weights from DRAM 4 times per step!
for seq in active_sequences:
token = model.forward(seq.current_token, seq.kv_cache)When 4 requests run concurrently, the GPU reads the entire 1 GB model weights from DRAM 4 separate times every single iteration step.
In mini-infer v0.2.1, we introduced zero-copy physical block gathering:
# mini-infer v0.2.1: Unified [B, 1] forward pass
k_all, v_all, mask = block_pool.gather_batched_paged_kv(
layer_idx=layer, block_tables=tables, context_lens=lens, device=device
)
# Weights are read from DRAM exactly ONCE per iteration across all 4 sequences!
logits = model(tokens_batch, k_all, v_all, mask)3. Empirical Load Testing with servebench
We load-tested the engine using servebench 1.1.2 on an RTX 3050 6GB Laptop GPU running Qwen2.5-0.5B-Instruct across concurrency levels c = 1, 2, 4:
| Concurrency | OK / Fail | Throughput (v0.1) | Throughput (v0.2.1) | Throughput Gain | TTFT p50 | TPOT p50 | Power Avg | Tok/Joule |
|---|---|---|---|---|---|---|---|---|
| c = 1 | 12 / 0 | 9.6 tok/s | 13.6 tok/s | +41.6% | 556 ms | 70 ms | 13.4 W | 1.01 |
| c = 2 | 12 / 0 | 12.0 tok/s | 18.6 tok/s | +55.0% | 1,162 ms | 98 ms | 13.1 W | 1.41 |
| c = 4 | 12 / 0 | 11.7 tok/s | 27.0 tok/s | +130.8% (2.3×) | 2,456 ms | 126 ms | 13.2 W | 2.04 |
4. Key Systems Takeaways
- Unified Batched Gather Doubled Concurrency Speed: Concurrency 4 throughput leaped from 11.7 tok/s to 27.0 tok/s (+130.8%) because weights are streamed from DRAM once per step.
- Extreme Energy Scaling (2.04 Tokens / Joule): Generating 27 tok/s while drawing only 13.2 Watts makes consumer laptop inference over 2× more energy-efficient than idle cloud instances.
- Automated Auto-Tuning: Using
servebench tune, developers can detect saturation knees automatically and prevent latency degradation before deploying to production.