llama.cpp · Q4_K_M
RTX 4090 · 24 GBFastest
Q4_K_M, Q8, FP16 · needs 8 GB · est. 122–175 tok/s
Unavailable
RTX 3080 Ti · 12 GB
Q4_K_M, Q8 · needs 8 GB · est. 111–158 tok/s
Unavailable
M1 · 16 GB unified
Q4_K_M, Q8 · needs 8 GB
Unavailable
M1 Pro · 16 GB unified
Q4_K_M, Q8 · needs 8 GB · est. 24–35 tok/s
Unavailable
16 more
llama.cpp · Q4_K_M
RTX 4090 · 24 GBFastest
Q4_K_M · needs 22 GB · est. 42–60 tok/s
Unavailable
M4 Pro · 24 GB unified
Q4_K_M · needs 22 GB · est. 11–17 tok/s
Unavailable
RTX 3090 · 24 GB
Q4_K_M · needs 22 GB · est. 39–56 tok/s
Unavailable
RX 7900 XTX · 24 GB
Q4_K_M · needs 22 GB
Unavailable
7 more
llama.cpp · Q4_K_M
RTX 4090 · 24 GBFastest
Q4_K_M, Q8 · needs 11 GB · est. 67–96 tok/s
Unavailable
RTX 3080 Ti · 12 GB
Q4_K_M · needs 11 GB · est. 60–87 tok/s
Unavailable
M1 · 16 GB unified
Q4_K_M · needs 11 GB
Unavailable
M1 Pro · 16 GB unified
Q4_K_M · needs 11 GB · est. 13–19 tok/s
Unavailable
16 more
llama.cpp · Q4_K_M
RTX 4090 · 24 GBFastest
Q4_K_M · needs 22 GB · est. 30–44 tok/s
Unavailable
M4 Pro · 24 GB unified
Q4_K_M · needs 22 GB · est. 8–12 tok/s
Unavailable
RTX 3090 · 24 GB
Q4_K_M · needs 22 GB · est. 28–41 tok/s
Unavailable
RX 7900 XTX · 24 GB
Q4_K_M · needs 22 GB
Unavailable
7 more
llama.cpp · Q4_K_M
RTX 4090 · 24 GBFastest
Q4_K_M, Q8, FP16 · needs 4 GB · est. 541–767 tok/s
Unavailable
RTX 3080 Ti · 12 GB
Q4_K_M, Q8, FP16 · needs 4 GB · est. 489–694 tok/s
Unavailable
M1 · 16 GB unified
Q4_K_M, Q8, FP16 · needs 4 GB
Unavailable
M1 Pro · 16 GB unified
Q4_K_M, Q8, FP16 · needs 4 GB · est. 107–153 tok/s
Unavailable
16 more
llama.cpp · Q4_K_M
RTX 4090 · 24 GBFastest
Q4_K_M · needs 22 GB · est. 34–49 tok/s
Unavailable
M4 Pro · 24 GB unified
Q4_K_M · needs 22 GB · est. 9–14 tok/s
Unavailable
RTX 3090 · 24 GB
Q4_K_M · needs 22 GB · est. 31–45 tok/s
Unavailable
RX 7900 XTX · 24 GB
Q4_K_M · needs 22 GB
Unavailable
7 more
llama.cpp · Q4_K_M
RTX 4090 · 24 GBFastest
Q4_K_M, Q8, FP16 · needs 8 GB · est. 543–771 tok/s
Unavailable
RTX 3080 Ti · 12 GB
Q4_K_M, Q8, FP16 · needs 8 GB · est. 491–697 tok/s
Unavailable
M1 · 16 GB unified
Q4_K_M, Q8, FP16 · needs 8 GB
Unavailable
M1 Pro · 16 GB unified
Q4_K_M, Q8, FP16 · needs 8 GB · est. 107–153 tok/s
Unavailable
16 more
Not sure what runs on your machine? See what fits.