Production vLLM Deployment: PagedAttention, KV-Cache & Continuous Batching

Executive TL;DR

To run Llama 3.3 70B in production with vLLM, allocate 2x 80GB GPUs (Tensor Parallelism = 2) or a single H200 (141GB) using FP8 precision. Configure --gpu-memory-utilization 0.92 and --kv-cache-dtype fp8 to maximize concurrent batch slots. At FP8, the 70B model occupies ~71GB with an additional 8GB for KV-cache at 32k context — fitting entirely on a single H100 SXM5 (80GB) with headroom.

Architecture Overview: How PagedAttention Prevents VRAM Waste

Traditional inference frameworks allocate contiguous memory blocks for the KV-cache. If a request needs 4KB but the allocator hands out a 16MB block, the remaining 15.996MB sits unused — internal fragmentation routinely reaches 60-80%. vLLM's PagedAttention solves this by borrowing the virtual memory paging paradigm from operating systems: the KV-cache is split into fixed-size blocks (typically 16KB), and a page table maps each attention layer's KV-cache to these blocks. Only the blocks needed by active requests are allocated. Internal fragmentation drops to <4%, reclaiming gigabytes of usable VRAM.

Step-by-Step Production Deployment

Step 1: Environment Setup & Docker Launch

Launch vLLM in a container with GPU access, shared memory for tensor parallelism, and a persistent volume for HuggingFace cache:

docker run --gpus all --shm-size 16g -p 8000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface vllm/vllm-openai:latest --model meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.92 --kv-cache-dtype fp8 --port 8000

Step 2: Key Launch Flags Reference

Step 3: Python Client Test Script

Verify the deployment with the OpenAI SDK pointing to the local vLLM server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy-key")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Explain PagedAttention in 3 sentences"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

VRAM Allocation Breakdown

Production Troubleshooting & Tuning

Resolving CUDA Out of Memory (OOM)

OOM errors during peak token generation typically indicate the KV-cache has exhausted the 8% reserved buffer. Solutions: (1) Reduce --max-model-len from 32768 to 16384, halving KV-cache usage. (2) Lower --gpu-memory-utilization to 0.88 to free more headroom. (3) Enable --chunked-prefill with --max-chunk-list-len 4096 to split long prompts into smaller batches, preventing a single prefill from exhausting VRAM. (4) Quantize the model to INT4 with --quantization awq, cutting the weights footprint from 71GB to 38GB.

Mitigating Prefill Latency Spikes

Long prompts trigger latency spikes because the entire prompt is processed in a single matrix multiplication. To mitigate: (1) Enable --enable-chunked-prefill with --max-chunk-list-len 4096 to split prompts into 4K-token chunks. (2) Use --num-prefill-threads 4 to parallelize prefill computation across CPU threads. (3) Pre-fill requests asynchronously using a separate queue so they don't block decoding requests. (4) Increase --max-num-seqs to allow more concurrent sequences during the prefill phase.

Next Action Decision CTAs

Calculate exact KV-cache footprint for your model: VRAM Calculator — Compare H100 vs H200 spot rental rates across providers: GPU Comparison Matrix