RESEARCH & HARDWARE PROFILING TEARDOWNS
Engineering Notes
Empirical investigations on LLM serving runtimes, Paged KV-cache memory allocators, continuous batching schedulers, and consumer laptop GPU capacity engineering.
Roofline Benchmarking & Auto-Tuning LLM Serving: How We Pushed Pure PyTorch to 27 tok/s
Most engineers treat LLM serving as an opaque black box—guessing thread counts, setting arbitrary batch sizes, and hoping VRAM doesn't crash. Here is how empirical roofline modeling and batched paged KV-cache gather eliminated DRAM memory bandwidth waste on our RTX 3050 6GB Laptop GPU.
c=1 throughput: 13.6 tok/s
c=4 throughput: 27.0 tok/s (+130.8%)
energy: 2.04 Tok/Joule
dram power: 13.2 W
silicon: RTX 3050 6GB Laptop GPU
Why Naive KV-Caches Waste >60% VRAM (And How Paged Memory Fixes It)
How replacing contiguous dynamic memory allocations with 16-token physical blocks and virtual table mapping brings production PagedAttention to pure PyTorch, eliminating CUDA out-of-memory crashes.
block size: 16 tokens
vram waste before: >60%
vram waste after: <3.8%
runtime: Pure PyTorch
silicon: RTX 3050 6GB Laptop GPU