~/best-gpu-for/llama-3-1-405b
Meta provider

Best GPU for Llama 3.1 405B

Ranked recommendations from the 42-GPU catalog for running Llama 3.1 405B (405B params). Top picks by quality, then by cost.

Top picks.

$ ./rank --by quality
Tightest budget
M3 Ultra 512
512GB · runs at FP8/INT8
405 GB weights · 28 GB headroom
Best quality
M3 Ultra 512
512GB · runs at FP8/INT8
smallest card supporting the best available quant
Apple Silicon
M3 Ultra 512
512GB · runs at FP8/INT8
smallest M-series that fits

Full ranked list.

$ ./vrambudget --rank llama-3-1-405b
GPUVRAMBest quantWeights at 15% safetyFit
appleM3 Ultra 512512GBFP8/INT8
405fitsnvidiaRTX 30708GBQ4_K_M
228overnvidiaRTX 40608GBQ4_K_M
228overnvidiaRTX 308010GBQ4_K_M
228overnvidiaRTX 3060 12GB12GBQ4_K_M
228overnvidiaRTX 3080 Ti12GBQ4_K_M
228overnvidiaRTX 407012GBQ4_K_M
228overnvidiaRTX 4070 Super12GBQ4_K_M
228overnvidiaRTX 507012GBQ4_K_M
228overintelArc B58012GBQ4_K_M
228overnvidiaRTX 4060 Ti 16GB16GBQ4_K_M
228overnvidiaRTX 4070 Ti Super16GBQ4_K_M
228overnvidiaRTX 408016GBQ4_K_M
228overnvidiaRTX 4080 Super16GBQ4_K_M
228overnvidiaRTX 5070 Ti16GBQ4_K_M
228overnvidiaRTX 508016GBQ4_K_M
228overnvidiaRTX 309024GBQ4_K_M
228overnvidiaRTX 3090 Ti24GBQ4_K_M
228over

Next steps.

$ ./next
$ checkCan I run Llama 3.1 405B?see all quants →full quant matrix$ tuneTune the mathopen calculator →/calc?model=llama-3-1-405b

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.