Alibaba Qwen

Qwen 2.5 Coder 32B uses a dense transformer architecture with grouped-query attention and a pre-training corpus of 5.5 trillion tokens including code from 50+ programming languages. Their context extrapolation technique allows 128K context windows through rotary position interpolation. The Qwen 2.5 family introduced an architecture upgrade with multi-query attention that reduces KV-cache overhead by 90% compared to standard multi-head attention. Their code generation is trained on a curated dataset of 10 billion lines of verified code from GitHub and Hugging Face.