Can You Run Mistral NeMo 12B on NVIDIA RTX 3090?

The Mistral NeMo 12B (12.2B Dense) contains approximately 12.2 billion parameters. At FP16 (2 bytes per parameter), the raw weight matrix occupies 24.5 GB. With a 15% CUDA kernel overhead factor, effective VRAM for weights alone is 24.5 GB. The NVIDIA RTX 3090 provides 24 GB GDDR6X of memory, leaving insufficient headroom — quantization to INT8 or INT4 is mandatory for KV-cache and activation tensors.

At INT8 quantization (1 byte per parameter), the weight footprint drops to 12.5 GB, requiring a tensor parallel degree of approximately 1× across NVIDIA RTX 3090 instances. The estimated inference throughput is ~48 tok/s (FP16, batch=1), which translates to a cost-per-million-output-tokens of roughly $2.00 at current spot rates. For production deployments, vLLM or Ollama (any precision) with PagedAttention is the recommended serving stack.