Local LLM Serving Engines
Architecture comparison of PagedAttention, GGUF CPU offloading, and RadixAttention. Throughput benchmarks and hardware sizing.
Engine Architecture Comparison
| Engine | Architecture | Throughput | Min Hardware | Best For |
|---|---|---|---|---|
| vLLM | PagedAttention | ~3,000 tok/s (batch 8) | 2x H100 | Production serving |
| Ollama | llama.cpp backend | ~120 tok/s (single) | RTX 4090 | Local development |
| SGLang | RadixAttention | ~2,500 tok/s (batch 4) | 2x H100 | High-throughput serving |
| llama.cpp | GGUF offloading | ~60 tok/s (single CPU) | Any x86 CPU | Edge / laptop |
Architecture Deep Dive
PagedAttention (vLLM)
PagedAttention allocates KV cache in fixed-size blocks managed by a block table, similar to virtual memory paging. This eliminates external fragmentation and enables continuous batching, achieving 2-4x throughput over naive implementations. Prefix caching further optimizes repeated prompt processing.
GGUF CPU Offloading (llama.cpp)
GGUF stores quantized model weights with metadata allowing CPU-to-GPU offloading. Each layer can reside on different devices. Supports INT4/INT8/FP16 quantization with minimal overhead. Enables inference on any platform from Raspberry Pi to data-center GPU clusters.
RadixAttention (SGLang)
RadixAttention uses a tree-structured KV cache for multi-step decoding. It enables efficient speculative decoding where a small draft model generates tokens that the larger target model verifies in parallel. Combined with Radix tree lookup, it achieves near-optimal prefix reuse across concurrent requests.
Hardware Sizing Table
| Model | VRAM (INT4) | GPU | CPU RAM | Disk |
|---|---|---|---|---|
| 7B (INT4) | 6 GB | RTX 4090 | 8 GB RAM | 5 GB |
| 14B (INT4) | 10 GB | RTX 4090 | 16 GB RAM | 10 GB |
| 32B (INT4) | 20 GB | RTX 4090 ×2 | 32 GB RAM | 20 GB |
| 70B (INT4) | 40 GB | 2x H100 | 64 GB RAM | 40 GB |