~/gpu/a6000 vs rtx-4090

RTX A6000 manufacturerRTX A6000vsRTX 4090 manufacturerRTX 4090

Head-to-head for local LLM inference. The honest comparison: VRAM, bandwidth, compute, and which of the 30 catalog models actually fit on each.

The specs.

$ diff specs a6000 rtx-4090
Stat
a6000
rtx-4090
Δ
VRAM
48 GB
24 GB
-50%
Memory bandwidth
768 GB/s
1,008 GB/s
+31%
FP16 compute
155 TFLOPS
330 TFLOPS
+113%
Weights budget at 8K ctx
37 GB
18 GB
-51%

Model fit difference.

$ models that change with the card
Fits on both
21of 30
Only on a6000
3
Only on rtx-4090
0

// showing 12 of 30 models; differing fits first

Model
a6000
rtx-4090
Llama 3.3 70B70.6B
fitsQ3_K_M
overQ4_K_M
Qwen 2.5 72B72.7B
fitsQ3_K_M
overQ4_K_M
Mixtral 8x7B46.7B
fitsQ5_K_M
overQ4_K_M
Llama 3.2 1B1.23B
fitsFP16/BF16
fitsFP16/BF16
Llama 3.2 3B3.21B
fitsFP16/BF16
fitsFP16/BF16
Llama 3.1 8B8B
fitsFP16/BF16
fitsFP16/BF16
Llama 3.1 405B405B
overQ4_K_M
overQ4_K_M
Qwen 2.5 7B7B
fitsFP16/BF16
fitsFP16/BF16
Qwen 2.5 32B32.5B
fitsQ8_0
fitsAWQ 4-BIT
Qwen 2.5 Coder 32B32.5B
fitsQ8_0
fitsAWQ 4-BIT
Qwen3 30B A3B30.5B
fitsQ8_0
fitsQ4_K_M
Qwen 3.5 9B9B
fitsFP16/BF16
fitsFP16/BF16

Which one wins for…

$ ./recommend --by-workload
More VRAM headroom

RTX A6000 has 24 GB more.

Faster decode (bandwidth)

RTX 4090 by +31%.

Faster prefill (compute)

RTX 4090 by +113% TFLOPS.

Catalog models that fit

RTX A6000: 24 fit · RTX 4090: 21.

Drill into either card.

$ ./vrambudget --gpu
$ openRTX A600048GBampere · 48GB$ openRTX 409024GBada · flagship

Discussion.

$ gh discussion list

// sign in with github to leave a comment. threads live in the repo's discussions tab.