EdgeDenseContext: 125K
Mistral NeMo 12B
Comprehensive deployment profile and benchmark telemetry for Mistral NeMo 12B. Self-hosting memory footprints, verified token economics, and direct API endpoints.
Verified Engineering Benchmarks
SWE-bench
49.2%
LiveCodeBench
52.1%
MATH-500
86%
MMLU
82%
VRAM Requirements & Sizing
🔗 Quick Actions
🖥️ Compatible GPUs & Self-Host Pricing
Deterministic VRAM math from entity graph. Green = fits in single GPU.
| GPU | VRAM | FP16 | FP8 | INT4 | Calculator Link |
|---|---|---|---|---|---|
| H100 SXM5 | 80 GB | ✓ fits | ✓ fits | ✓ fits | Pre-filled → |
| H200 | 141 GB | ✓ fits | ✓ fits | ✓ fits | Pre-filled → |
| B200 | 192 GB | ✓ fits | ✓ fits | ✓ fits | Pre-filled → |
| A100 80GB | 80 GB | ✓ fits | ✓ fits | ✓ fits | Pre-filled → |
| L40S | 48 GB | ✓ fits | ✓ fits | ✓ fits | Pre-filled → |
| RTX 4090 | 24 GB | ✓ fits | ✓ fits | ✓ fits | Pre-filled → |
⚡ Verified Free API Available
KV-Cache Memory Scaling
| Precision | Weights | 4K Context | 32K Context | 128K Context | Total 4K | Total 32K | Total 128K |
|---|---|---|---|---|---|---|---|
| FP16 | 24.5 GB | 0.3 GB | 1.2 GB | 4.8 GB | ~24.8 GB | ~25.7 GB | ~29.3 GB |
| FP8 | 12.5 GB | 0.15 GB | 0.6 GB | 2.4 GB | ~12.7 GB | ~13.1 GB | ~14.9 GB |
| INT4/AWQ | 7.5 GB | 0.08 GB | 0.3 GB | 1.2 GB | ~7.6 GB | ~7.8 GB | ~8.7 GB |
✓ OptimizedWeights + KV-Cache formula:
Total = Weights + 2 × Layers × Heads × Head_Dim × Context × Bytes_Per_Element + 2 GB CUDAProduction Run Commands
vLLM ServerCopy via DevTools →
python3 -m vllm.entrypoints.openai.api_server \ --model mistralai/Mistral-Nemo-Instruct-2407 \ --max-model-len 32768 \ --gpu-memory-utilization 0.95 \ --tensor-parallel-size 1 \ --fp8-mlx
Tensor parallelism: TP=1 · Max model len: 32768 · Quantization: FP8
Ollama ModelfileCopy via DevTools →
FROM mistralai/Mistral-Nemo-Instruct-2407
PARAMETER num_ctx 4096
PARAMETER num_thread 4
TEMPLATE "<|im_start|>{{user}}<|im_end|>
<|im_start|>assistant<|im_end|>
{{.}}"num_ctx tuned for 4096 tokens · Increase num_ctx for longer context (up to 128k for supported GPUs)
Host-vs-API Breakeven Analysis
Self-Host Cost
$0.34/hr
1x L40S 48GB · 520K tokens/hr
API Cost
$0.15/$M in
$0.15/$M out · Groq/DeepInfra/OpenRouter
Breakeven Point
1.1M tokens/day
Self-host cheaper above this volume
💡 Breakeven Insight
At $0.34/hr self-hosting on 1x L40S 48GB, you break even after 1.1M input tokens per day compared to $0.15/M API costs. Below that threshold, serverless APIs (Groq , DeepInfra) are strictly cheaper. Above it, self-hosting with vLLM on spot GPUs provides better unit economics at scale.
Active API Providers & Free Tiers
| Provider | Input / 1M | Output / 1M | Speed | Free Tier Status |
|---|---|---|---|---|
| DeepInfra | $0.07 | $0.09 | 120 tok/s | Paid API Only |
| Together AI | $0.10 | $0.10 | 100 tok/s | Paid API Only |
| OpenRouter | $0.07 | $0.09 | 110 tok/s | Paid API Only |
OpenAI SDK Drop-in
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.deepinfra.com/v1",
apiKey: process.env.DEEPINFRA_API_KEY,
});
const response = await client.chat.completions.create({
model: "mistral-nemo-12b",
messages: [{ role: "user", content: "Hello" }],
});