~/model/huggingfacemeta-llama/Llama-3.1-8B-Instruct
Meta provider

Llama 3.1 8B

Meta's small workhorse. Runs comfortably on any 8GB+ card at Q4 or any 12GB+ at FP16.

Parameters
8B
Family
Meta
Context
128K tokens
FP16 weights
16GB
// where you can run it
llama3.1:8bLM StudiovLLMMLXoMLX
// hugging face stats (cached daily)
10.9M downloads · 6K likes · license: llama3.1 · updated 1 year ago

What you need to run this.

$ ./vrambudget --model llama-3-1-8b --by quant
FP1616 GB
RTX 309024 GB · fitsRTX 3090 Ti24 GB · fitsRTX 409024 GB · fitsRX 7900 XTX24 GB · fits
Q8_08.5 GB
RTX 3060 12GB12 GB · fitsRTX 3080 Ti12 GB · fitsRTX 407012 GB · fitsRTX 4070 Super12 GB · fits
Q6_K6.6 GB
RTX 308010 GB · fitsRTX 3060 12GB12 GB · fitsRTX 3080 Ti12 GB · fitsRTX 407012 GB · fits
Q5_K_M5.5 GB
RTX 30708 GB · fitsRTX 40608 GB · fitsRTX 308010 GB · fitsRTX 3060 12GB12 GB · fits
Q4_K_M4.5 GB
RTX 30708 GB · fitsRTX 40608 GB · fitsRTX 308010 GB · fitsRTX 3060 12GB12 GB · fits

// budgets shown at ctx 8K, concurrency 1, 15% safety headroom. Tune in the calculator →

Alternatives at this size.

$ grep --params similar catalog.json
$ similarGranite 8B Code8BIBM$ similarMistral 7B v0.37.2BMistral$ similarQwen 2.5 7B7BAlibaba$ similarQwen 3.5 9B9BAlibaba

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.