AI/ML Systems & LLM Inference Engineer
I engineer from-scratch continuous batching runtimes, Paged KV-cache virtual memory allocators, and low-overhead benchmarking harnesses. Calibrated against constrained silicon (RTX 3050 6GB Laptop GPU) to achieve zero CUDA OOMs and high-density serving throughput.
| Runtime | KV Strategy | tok/s | VRAM | Status |
|---|---|---|---|---|
| mini-infer v0.2.4 | PagedAttention | 27.0 | 1.82 GB | ✓ Sigstore Verified |
| HF Transformers | Naive Eager | 11.2 | 3.40 GB | Fragmented |
| vLLM (Standard) | Fixed Prealloc | OOM | >5.5 GB | ❌ CUDA OOM |
| servebench v1.1.2 | SSE Harness | -- | <50 MB | ✓ Sigstore Verified |
High-performance, from-scratch continuous batching LLM inference runtime featuring Paged KV-Cache allocation, iteration-level scheduling, and raw Safetensors streaming deserialization.
- [Paging] Zero-fragmentation virtual memory table mapping logical token blocks to physical GPU memory slots.
- [Batching] Step-level continuous iteration merges new prompts with ongoing decodes dynamically.
- [Loader] Direct tensor streaming loader bypasses HuggingFace overhead for instant memory mapping.
Universal production load testing & capacity harness for LLM serving engines. Captures streaming SSE chunks with microsecond precision to diagnose TTFT and ITL tail latency degradation.
- [SSE Parser] High-throughput Server-Sent Events client extracting TTFT, ITL jitter, and queue delays.
- [Adapters] Multi-engine support for vLLM, SGLang, Ollama, and custom local inference servers.
- [Telemetry] Heuristic bottleneck diagnosis differentiating compute bound vs KV-cache bound clusters.
High-fidelity discrete-event simulator for LLM serving schedulers. Evaluates Sarathi Chunked Prefill, Radix Prefix Caching, and PD Disaggregation on CPU with <8% variance to real silicon.
- [Roofline] Analytical compute & DRAM bandwidth latency physics calibrated with physical silicon data.
- [Radix Cache] SGLang/vLLM-style Radix tree prefix caching with chained block hashing and LRU eviction.
- [Perfetto] Microsecond forward-pass iteration slicing and request lifecycle export for ui.perfetto.dev.
quant-from-scratch: Handcrafted GPTQ, AWQ, and SmoothQuant Kernels
Custom implementation of post-training quantization algorithms from mathematical first principles (Hessian matrix inversion, activation-aware channel scaling, and INT4 packing). Quantizes Qwen2.5-1.5B/3B and benchmarks accuracy loss vs. inference speedup against official AutoGPTQ and vLLM kernels.
Document sensitivity classifier and enterprise DLP service engineered for strictly air-gapped CPU/edge inference. Combines a LoRA fine-tune of Qwen2.5-0.5B with a deterministic Tier-1 structural secret scanner to ensure zero data leaves the machine.
- [Batching] Dynamic Tensor Chunk Batching yields a 6.7x multi-page document speedup.
- [Speculative] Speculative early-exit mechanism cuts redundant compute by 88.9%.
- [Accuracy] Achieves 100% secret recall on real-world balanced 100-document SEC/Enron benchmark.
Document search and chat application featuring hybrid retrieval and strict page-level source citations. Combines dense MiniLM embeddings with sparse BM25 using Reciprocal Rank Fusion (RRF) for high-precision recall.
- [Hybrid RRF] Dense semantic embeddings fused with sparse BM25 keyword matching on Pinecone.
- [Routing] Multi-tenant LLM intent router seamlessly falls back to cited web search.
- [Grounding] Zero hallucination citation mechanism with interactive PDF page jump links.
Why Naive KV-Caches Waste >60% VRAM (And How Paged Memory Fixes It)
Read Architecture Essay → ROOFLINE PROFILING // OCTOBER 2026Roofline Benchmarking & Auto-Tuning: Pushing Pure PyTorch to 27 tok/s
Read Architecture Essay → ARCHITECTURAL LOGS // 2026All Engineering Notes, Hardware Benchmarks & Serving Teardowns
Browse All Notes (/blog/) →Document AI & Enterprise RAG
Building end-to-end document intelligence pipelines: visual layout parsing, table extraction, multi-stage hybrid search, and citation-grounded LLM synthesis.
- Complex PDF & tabular OCR extraction
- Hybrid dense/sparse BM25 vector retrieval
- Sub-second P99 answer generation
Private Small Language Models (SLMs)
Quantizing and deploying domain-specific models (1.5B–8B parameters) entirely within air-gapped on-premise infrastructure. Zero cloud exposure with low VRAM footprint.
- FP8, AWQ, and GGUF quantization tuning
- Consumer & workstation GPU deployment
- 100% data privacy and compliance
Inference Capacity & Cost Audit
Load-testing existing vLLM, TGI, or custom serving clusters with `servebench`. Finding the exact concurrency sweet-spot to cut cloud compute bills by 30–60%.
- Time-to-First-Token & ITL jitter profiling
- KV-cache memory sizing under burst load
- Autoscaling threshold calibration