~/can-i-run/yi-34b
01.AI

Can I run Yi 34B?

Short answer: yes, on a M3 Max 96 (96GB) at FP16/BF16. Long answer below.

yi:34bLM StudiovLLMMLXoMLX

The math, in one paragraph.

$ ./vrambudget --explain yi-34b

Yi 34B has 34B parameters. At FP16 that's 68 GB of raw weights. Quantization shrinks that, but you also need budget for the KV cache (definition), framework overhead, and safety headroom. The rule of thumb: real usable budget on a card is roughly its nameplate VRAM minus 25%. That's how the table below was computed.

What hardware actually fits.

$ grep "fits" gpus.json
FP16/BF16
68GB
12 GPUs fit
M3 Max 9696GBRTX Pro 600096GBM4 Max 128128GBM5 Max 128128GB+ 8 more
Q8_0
36GB
21 GPUs fit
RTX A600048GBRTX 6000 Ada48GBL40S48GBRadeon Pro W790048GB+ 17 more
Q5_K_M
23GB
22 GPUs fit
RTX 509032GBRTX A600048GBRTX 6000 Ada48GBL40S48GB+ 18 more
Q4_K_M
19GB
22 GPUs fit
RTX 509032GBRTX A600048GBRTX 6000 Ada48GBL40S48GB+ 18 more
Q3_K_M
15GB
27 GPUs fit
RTX 309024GBRTX 3090 Ti24GBRTX 409024GBRX 7900 XTX24GB+ 23 more

Pick your path.

$ ls strategies/
Tightest budget

Smallest GPU that fits Yi 34B at any quant: M3 Max 96 at FP16/BF16.

Reference quality (FP16)

Lossless inference needs 68 GB. Pick from 12 cards.

Best quality on a 24GB card

Q3_K_M fits comfortably (15 GB weights).

Tune the math yourself

Open the calculator pre-tuned for Yi 34B: ↗ /calc?model=yi-34b

See the full model page.

$ ./open
$ openYi 34B34B01.AI · ctx 32K$ rankBest GPU for Yi 34Bsee the ranking →ranked by fit quality

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.