~/can-i-run/qwen-2-5-72b
Alibaba provider

Can I run Qwen 2.5 72B?

Short answer: yes, on a 2× H100 NVL (188GB) at FP16/BF16. Long answer below.

qwen2.5:72bLM StudiovLLMMLXoMLX

The math, in one paragraph.

$ ./vrambudget --explain qwen-2-5-72b

Qwen 2.5 72B has 72.7B parameters. At FP16 that's 145 GB of raw weights. Quantization shrinks that, but you also need budget for the KV cache (definition), framework overhead, and safety headroom. The rule of thumb: real usable budget on a card is roughly its nameplate VRAM minus 25%. That's how the table below was computed.

What hardware actually fits.

$ grep "fits" gpus.json
FP16/BF16
145GB
5 GPUs fit
2× H100 NVL188GBM2 Ultra 192192GBB200192GBMI300X192GB+ 1 more
Q8_0
77GB
12 GPUs fit
M3 Max 9696GBRTX Pro 600096GBM4 Max 128128GBM5 Max 128128GB+ 8 more
Q5_K_M
50GB
17 GPUs fit
M2 Max 6464GBM3 Max 6464GBM4 Pro 6464GBM5 Pro 6464GB+ 13 more
Q4_K_M
41GB
17 GPUs fit
M2 Max 6464GBM3 Max 6464GBM4 Pro 6464GBM5 Pro 6464GB+ 13 more
Q3_K_M
31GB
21 GPUs fit
RTX A600048GBRTX 6000 Ada48GBL40S48GBRadeon Pro W790048GB+ 17 more

Pick your path.

$ ls strategies/
Tightest budget

Smallest GPU that fits Qwen 2.5 72B at any quant: 2× H100 NVL at FP16/BF16.

Reference quality (FP16)

Lossless inference needs 145 GB. Pick from 5 cards.

Best quality on a 24GB card

None of the showcase quants fit on a 24GB card. Step up.

Tune the math yourself

Open the calculator pre-tuned for Qwen 2.5 72B: ↗ /calc?model=qwen-2-5-72b

See the full model page.

$ ./open
$ openQwen 2.5 72B72.7BAlibaba · ctx 128K$ rankBest GPU for Qwen 2.5 72Bsee the ranking →ranked by fit quality

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.