~/model/huggingfacemistralai/Mixtral-8x7B-Instruct-v0.1
Mistral provider

Mixtral 8x7B

The MoE that proved sparse models work for inference. 8 experts, 2 active per token.

Parameters
46.7B
Family
Mistral
Context
32K tokens
FP16 weights
93GB
// where you can run it
mixtral:8x7bLM StudiovLLMMLXoMLX
// hugging face stats (cached daily)
593K downloads · 5K likes · license: apache-2.0 · updated 10 months ago

What you need to run this.

$ ./vrambudget --model mixtral-8x7b --by quant
FP1693 GB
M4 Max 128128 GB · fitsM5 Max 128128 GB · fitsDGX Spark128 GB · fitsGaudi 3128 GB · fits
Q8_050 GB
M2 Max 6464 GB · fitsM3 Max 6464 GB · fitsM4 Pro 6464 GB · fitsM5 Pro 6464 GB · fits
Q6_K38 GB
RTX A600048 GB · fitsRTX 6000 Ada48 GB · fitsL40S48 GB · fitsRadeon Pro W790048 GB · fits
Q5_K_M32 GB
RTX A600048 GB · fitsRTX 6000 Ada48 GB · fitsL40S48 GB · fitsRadeon Pro W790048 GB · fits
Q4_K_M26 GB
RTX A600048 GB · fitsRTX 6000 Ada48 GB · fitsL40S48 GB · fitsRadeon Pro W790048 GB · fits

// budgets shown at ctx 8K, concurrency 1, 15% safety headroom. Tune in the calculator →

Alternatives at this size.

$ grep --params similar catalog.json
$ similarQwen 3.6 35B A3B35BAlibaba$ similarYi 34B34B01.AI$ similarQwen 2.5 32B32.5BAlibaba$ similarQwen 2.5 Coder 32B32.5BAlibaba

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.