~/can-i-run/mixtral-8x7b
Mistral provider

Can I run Mixtral 8x7B?

Short answer: yes, on a M4 Max 128 (128GB) at FP16/BF16. Long answer below.

mixtral:8x7bLM StudiovLLMMLXoMLX

The math, in one paragraph.

$ ./vrambudget --explain mixtral-8x7b

Mixtral 8x7B has 46.7B parameters (MoE: 12.9B active per forward pass, but all 46.7B must fit in memory). At FP16 that's 93 GB of raw weights. Quantization shrinks that, but you also need budget for the KV cache (definition), framework overhead, and safety headroom. The rule of thumb: real usable budget on a card is roughly its nameplate VRAM minus 25%. That's how the table below was computed.

What hardware actually fits.

$ grep "fits" gpus.json
FP16/BF16
93GB
10 GPUs fit
M4 Max 128128GBM5 Max 128128GBDGX Spark128GBGaudi 3128GB+ 6 more
Q8_0
50GB
17 GPUs fit
M2 Max 6464GBM3 Max 6464GBM4 Pro 6464GBM5 Pro 6464GB+ 13 more
Q5_K_M
32GB
21 GPUs fit
RTX A600048GBRTX 6000 Ada48GBL40S48GBRadeon Pro W790048GB+ 17 more
Q4_K_M
26GB
21 GPUs fit
RTX A600048GBRTX 6000 Ada48GBL40S48GBRadeon Pro W790048GB+ 17 more
Q3_K_M
20GB
22 GPUs fit
RTX 509032GBRTX A600048GBRTX 6000 Ada48GBL40S48GB+ 18 more

Pick your path.

$ ls strategies/
Tightest budget

Smallest GPU that fits Mixtral 8x7B at any quant: M4 Max 128 at FP16/BF16.

Reference quality (FP16)

Lossless inference needs 93 GB. Pick from 10 cards.

Best quality on a 24GB card

None of the showcase quants fit on a 24GB card. Step up.

Tune the math yourself

Open the calculator pre-tuned for Mixtral 8x7B: ↗ /calc?model=mixtral-8x7b

See the full model page.

$ ./open
$ openMixtral 8x7B46.7BMistral · ctx 32K$ rankBest GPU for Mixtral 8x7Bsee the ranking →ranked by fit quality

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.