Meta

Llama 3.3 70B uses a dense transformer architecture with grouped-query attention (GQA) that reduces KV-cache memory by 8x compared to multi-head attention while maintaining quality. The model was pre-trained on 15 trillion tokens of publicly available data with a carefully curated quality filter pipeline. Meta's Llama 3.1 8B achieves competitive coding performance through a supervised fine-tuning dataset of 10 billion tokens including code from GitHub, Stack Overflow, and synthetic code generation. Their model card provides detailed benchmarks, training configurations, and safety evaluations, establishing a benchmark for model transparency in the open-weights ecosystem.