Production vLLM Cluster Setup: Continuous Batching, PagedAttention & Docker Runbooks

PagedAttention v2 is the memory management engine behind vLLM's high throughput. Traditional attention implementations pre-allocate KV-cache memory for the maximum sequence length — wasting memory when sequences are shorter. PagedAttention divides KV-cache into fixed-size blocks (like OS virtual memory pages) and allocates them on-demand. For Llama 70B at 128k context, pre-allocation wastes ~40% of KV-cache memory. PagedAttention reduces waste to <5%, enabling 2-3x higher batch sizes on the same GPU. Combined with FlashAttention-3 on Hopper GPUs, this delivers the highest tokens/dollar in the market.

Production vLLM deployment uses Docker containers for reproducibility and isolation. The recommended setup: # Pull the vLLM image docker pull vllm/vllm-openai:latest # Launch with GPU access and resource limits docker run -d --gpus all --shm-size=16g \ --name vllm-instance \ -p 8000:8000 \ vllm/vllm-openai:latest \ --model meta-llama/Llama-3.3-70B-Instruct \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.92 \ --max-model-len 32768 \ --enable-prefix-caching The --shm-size=16g is critical — PyTorch shared memory tensors require /dev/shm space. Insufficient shared memory causes silent failures in multi-GPU tensor parallelism.

Key vLLM engine arguments for production: --tensor-parallel-size N: Split model across N GPUs. Use 2 for 70B on H100, 4 for 70B on A100. --gpu-memory-utilization 0.92: Reserve 8% VRAM for CUDA kernels and fragmentation. Higher values risk OOM. --max-model-len 32768: Maximum sequence length. Reducing this frees KV-cache for higher batch sizes. --enable-prefix-caching: Cache common prompt prefixes across requests. Reduces TTFT by 30-50% for RAG workloads. --quantization awq: Enable INT4 quantization. Reduces VRAM 75% with <3% quality loss. --block-size 16: KV-cache block size in tokens. Smaller blocks = less waste, more fragmentation. 16 is optimal.

Production vLLM clusters require three monitoring layers: (1) GPU metrics via nvidia-smi — utilization %, memory used, temperature, power draw. Export to Prometheus via dcgm-exporter. (2) vLLM metrics — request latency, tokens/s, queue depth, cache hit rate. vLLM exposes /metrics endpoint for Prometheus scraping. (3) Application logs — request/response payloads, error rates, model loading times. Ship to Grafana Loki or ELK stack. Alert on: GPU temperature >85°C, KV-cache utilization >90%, request latency p99 >2s.