~/can-i-run/starcoder2-15b
BigCode

Can I run StarCoder2 15B?

Short answer: yes, on a RTX A6000 (48GB) at FP16/BF16. Long answer below.

starcoder2:15bLM StudiovLLMMLXoMLX

The math, in one paragraph.

$ ./vrambudget --explain starcoder2-15b

StarCoder2 15B has 15B parameters. At FP16 that's 30 GB of raw weights. Quantization shrinks that, but you also need budget for the KV cache (definition), framework overhead, and safety headroom. The rule of thumb: real usable budget on a card is roughly its nameplate VRAM minus 25%. That's how the table below was computed.

What hardware actually fits.

$ grep "fits" gpus.json
FP16/BF16
30GB
21 GPUs fit
RTX A600048GBRTX 6000 Ada48GBL40S48GBRadeon Pro W790048GB+ 17 more
Q8_0
16GB
27 GPUs fit
RTX 309024GBRTX 3090 Ti24GBRTX 409024GBRX 7900 XTX24GB+ 23 more
Q5_K_M
10GB
33 GPUs fit
RTX 4060 Ti 16GB16GBRTX 4070 Ti Super16GBRTX 408016GBRTX 4080 Super16GB+ 29 more
Q4_K_M
8.4GB
39 GPUs fit
RTX 3060 12GB12GBRTX 3080 Ti12GBRTX 407012GBRTX 4070 Super12GB+ 35 more
Q3_K_M
6.5GB
40 GPUs fit
RTX 308010GBRTX 3060 12GB12GBRTX 3080 Ti12GBRTX 407012GB+ 36 more

Pick your path.

$ ls strategies/
Tightest budget

Smallest GPU that fits StarCoder2 15B at any quant: RTX A6000 at FP16/BF16.

Reference quality (FP16)

Lossless inference needs 30 GB. Pick from 21 cards.

Best quality on a 24GB card

Q8_0 fits comfortably (16 GB weights).

Tune the math yourself

Open the calculator pre-tuned for StarCoder2 15B: ↗ /calc?model=starcoder2-15b

See the full model page.

$ ./open
$ openStarCoder2 15B15BBigCode · ctx 16K$ rankBest GPU for StarCoder2 15Bsee the ranking →ranked by fit quality

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.