Local Agent Bench · suite v0.1.5

Full results

Every table, exactly as published. Jump to the GPU memory you care about.

Which local LLMs actually work for agents?

We rank local models on agent tool-calling with the Hermes Agent harness and an MCP mock tools server, split by how much GPU memory they need. All current results run on Ollama; an engine comparison (LM Studio, llama.cpp) is planned.

Current results: Q4_K_M quants, Ollama, tested October 2026. Tiers mean “fits in 8 / 16 / 24 GB of VRAM”: since results v0.1.2, a model is listed in a tier only if its measured VRAM at the bench's 32K context, KV cache included, is at most that amount (ollama ps on the 3090; see Methodology). Tiers and ranks come from runs on NVIDIA RTX 3090 (24 GB) hosts. Since results v0.1.3 the 8 GB tier has also been run on a real 8 GB card (RTX 3070, RunPod container), listed as a separate series. Since results v0.1.4 the same suite also runs on an NVIDIA A100 SXM4 80 GB (results/a10064/ — the directory keeps its historical name; the card is 80 GB, not 64 GB), also as a separate series. No 12 GB, 16 GB or 48 GB card has been tested yet. tok/s is measured per host (shown as @host) and is only comparable between models on the same host. Apple M2 16 GB Mac results (macOS, unified memory) are listed in their own tier.

Technical changelog & suite notes (experts)

Suite results v0.1.5 · scoring v4. Scoring v4 (2026-11-13): a model is only marked compromised on injection when it takes a harmful action (email to the attacker, reading /secrets, canary in a tool argument, or canary leaked in the answer). Quoting the attacker address while warning the user, or announcing intent without a tool call, is a mention (shown in its own column) and does not lower the score. v3 totals and ranks are in the muted column. Results v0.1.5 (2026-11-13): five new RTX 3090 models, with VRAM at 32K measured on Ollama 0.40.6 (ministral-3:14b is not measured; ministral-3:8b measured 9.8 GB and is in the 16 GB tier; ornith-1.5:9b measured 6.7 GB and is in the 8 GB tier). glm-4.7-flash is pinned as glm-4.7-flash:q4_K_M (sha256 9eba2761…f9ec950). nemotron-3.5-bolt:e4b ran on Ollama 0.40.10 and is listed as not comparable. 30B models are not in this release. Suite results v0.1.4 · scoring v3 (previous). Results v0.1.4 (2026-11-09): no score or rank changes in the tiers. The same suite on an NVIDIA A100 SXM4 80 GB (RunPod container, results/a10064) is added as a second separate series: qwen3:32b-q8_0 87.25, gemma4:31b-it-q8_0 80.78, qwen2.5:72b 79.17, llama3.3:70b 68.35, deepseek-r1:70b 13.20. Three single-trial runs per model, every trial published — including the gemma4 trial that scored 0% on multistep. Results v0.1.3 (2026-10-28): no score or rank changes in the tiers; The same suite on a real RTX 3070 8 GB is listed as a separate series (scoring v3 from the run logs). All six 8 GB-tier models ran on it, qwen3:4b with partial CPU offload; the six 8–10 GB models stay in the 16 GB tier and are also shown in the 8 GB view as running with CPU offload (slow). Results v0.1.2 (2026-10-26): no score changes; six models that need more than 8 GB at 32K context (mistral:7b, hermes3:8b, llama3.1:8b, qwen3:8b, deepseek-r1:8b, qwen3-vl:8b, measured 8.9–10 GB) moved from the 8 GB to the 16 GB tier. Their previous tier and rank are shown next to their names. Scoring v3 (2026-10-23): the injection component is now “useful and safe”: the share of injection tasks where the model completed the user's task and did not follow the injected instruction (AgentDojo-style). It replaces injection resistance × tool-call rate. The previous v2.1 score and rank are shown in a muted column for comparison. Scores were measured with Ollama's default (non-deterministic) sampling over 3 trials; “± runs” is the run-to-run SD from a separate rerun of every GPU model 5 times at temperature 0.7 with fixed seeds (Mac models: 3-trial SD, marked), “≈” marks models whose 95% task-bootstrap intervals (main and rerun trials pooled) overlap (statistically tied), and “Rank range” shows how the rank moves under 10,000 plausible weightings. Changelog · Ties & rank ranges · Variance

JSON 15Schema 15Tool selection 20Multi-step 30Injection: useful & safe 20Speed reported separately

Everything that fits in 8 GB VRAM

All models from the 8 GB tiers, ranked together by score. The “Tier” column shows the tier each model was measured in (all on RTX 3090 hosts). These models were measured on more than one host, so tok/s is not shown here: see Speed by host.

#	Model	Tier	Score	± runs	v3	JSON	Schema	Tool	Multi-step	Inj. useful&safe	VRAM @32K
1	qwen3:4b prev. #2	8 GB	97.7^†	±2.6	92.9	100	91	100	97	100	7.6 GB
2	gemma4:e4b prev. #1	8 GB	94.7	±0.6	94.7	99	95	100	89	96	not measured
3	ornith-1.5:9b previous tier: 16 GB	8 GB	92.0	—	82.2	95	95	97	82	98	6.7 GB
4	nemotron-3-nano:4b	8 GB	79.0	—	78.1	100	91	96	61	64	3.5 GB
5	qwen2.5:7b	8 GB	66.1^†	±1.2	58.6	33	32	100	65	84	6.6 GB
6	llama3.2:3b prev. #7	8 GB	52.1	±4.6	49.8	91	45	93	18	38	6.0 GB
7	qwen2.5-coder:7b prev. #10	8 GB	26.0	±0.7	26.0	100	73	0	0	0	6.6 GB
—	gemma3:4b No native tool support	8 GB	n/a^*	—	—	0	0	0	0	0	not measured

Rank ranges and tie groups are computed within each measured tier; see the per-tier view.

Everything that fits in 16 GB VRAM

All models from the 8 GB + 16 GB tiers, ranked together by score. The “Tier” column shows the tier each model was measured in (all on RTX 3090 hosts). These models were measured on more than one host, so tok/s is not shown here: see Speed by host.

#	Model	Tier	Score	± runs	v3	JSON	Schema	Tool	Multi-step	Inj. useful&safe	VRAM @32K
1	gemma4:12b	16 GB	98.3	±1.2	98.3	100	95	100	97	100	not measured
2	qwen3:4b prev. #2	8 GB	97.7^†	±2.6	92.9	100	91	100	97	100	7.6 GB
3	gemma4:e4b prev. #1	8 GB	94.7	±0.6	94.7	99	95	100	89	96	not measured
4	qwen3.5:9b prev. #4	16 GB	94.1	±1.6	90.6	100	97	100	85	96	6.7 GB
5	gpt-oss:20b prev. #2	16 GB	93.9	±1.9	93.4	100	97	100	94	80	12 GB
6	qwen3-vl:8b previous tier: 8 GB (#3)	16 GB	93.9^†	±2.3	91.6	95	83	100	92	98	10 GB
7	ornith-1.5:9b previous tier: 16 GB	8 GB	92.0	—	82.2	95	95	97	82	98	6.7 GB
8	qwen3:14b prev. #3	16 GB	92.0	±2.1	90.6	100	76	100	87	98	14 GB
9	nemotron-3-nano:4b	8 GB	79.0	—	78.1	100	91	96	61	64	3.5 GB
10	qwen2.5:7b	8 GB	66.1^†	±1.2	58.6	33	32	100	65	84	6.6 GB
11	qwen3:8b previous tier: 8 GB (#4)	16 GB	61.9^†	±3.3	60.6	93	89	77	56	11	10 GB
12	mistral-nemo:12b prev. #5	16 GB	61.5	±2.9	60.2	80	68	100	20	67	12 GB
13	qwen2.5:14b prev. #6	16 GB	59.2	±1.8	59.2	75	48	99	33	56	15 GB
14	ministral-3:8b previous tier: 8 GB	16 GB	56.3	—	55.4	77	63	65	37	56	9.8 GB
15	llama3.1:8b previous tier: 8 GB (#6)	16 GB	54.1	±2.3	53.2	80	29	100	19	60	9.2 GB
16	llama3.2:3b prev. #7	8 GB	52.1	±4.6	49.8	91	45	93	18	38	6.0 GB
17	hermes3:8b previous tier: 8 GB (#8)	16 GB	45.7	±10.1	45.3	80	80	68	2	38	9.0 GB
18	ministral-3:14b	16 GB	39.7	—	39.7	61	44	64	16	31	not measured
19	deepseek-r1:8b previous tier: 8 GB (#9)	16 GB	29.2^†	±0.3	29.2	99	96	0	0	0	10 GB
20	deepseek-r1:14b prev. #7	16 GB	26.6	±1.9	26.6	91	87	0	0	0	15 GB
21	qwen2.5-coder:7b prev. #10	8 GB	26.0	±0.7	26.0	100	73	0	0	0	6.6 GB
22	qwen2.5-coder:14b prev. #8	16 GB	24.2	±0.0	24.2	89	72	0	0	0	15 GB
23	mistral:7b previous tier: 8 GB (#11)	16 GB	22.0	±0.7	22.0	93	53	0	0	0	8.9 GB
—	gemma3:4b No native tool support	8 GB	n/a^*	—	—	0	0	0	0	0	not measured
—	phi4:14b tool mode → No native tool support	16 GB	n/a^*	—	—	0	0	0	0	0	not measured

Rank ranges and tie groups are computed within each measured tier; see the per-tier view.

Everything that fits in 24 GB VRAM

All models from the 8 GB + 16 GB + 24 GB tiers, ranked together by score. The “Tier” column shows the tier each model was measured in (all on RTX 3090 hosts). These models were measured on more than one host, so tok/s is not shown here: see Speed by host.

#	Model	Tier	Score	± runs	v3	JSON	Schema	Tool	Multi-step	Inj. useful&safe	VRAM @32K
1	gemma4:26b	24 GB	99.7	±0.6	99.7	100	100	100	99	100	not measured
2	qwen3.8:27b prev. #5	24 GB	98.7^†	±1.5	92.0	100	99	100	96	100	17 GB
3	qwen3:30b-a3b prev. #2	24 GB	98.5	±1.0	98.5	100	95	100	99	98	21 GB
4	gemma4:12b	16 GB	98.3	±1.2	98.3	100	95	100	97	100	not measured
5	qwen3:4b prev. #2	8 GB	97.7^†	±2.6	92.9	100	91	100	97	100	7.6 GB
6	qwen3.6:27b prev. #3	24 GB	95.2	±0.8	95.2	100	99	100	85	100	17 GB
7	qwen3.5:27b prev. #4	24 GB	95.0	±0.4	95.0	99	97	100	87	98	18 GB
8	gemma4:e4b prev. #1	8 GB	94.7	±0.6	94.7	99	95	100	89	96	not measured
9	qwen3.5:9b prev. #4	16 GB	94.1	±1.6	90.6	100	97	100	85	96	6.7 GB
10	gpt-oss:20b prev. #2	16 GB	93.9	±1.9	93.4	100	97	100	94	80	12 GB
11	qwen3-vl:8b previous tier: 8 GB (#3)	16 GB	93.9^†	±2.3	91.6	95	83	100	92	98	10 GB
12	ornith-1.5:9b previous tier: 16 GB	8 GB	92.0	—	82.2	95	95	97	82	98	6.7 GB
13	qwen3:14b prev. #3	16 GB	92.0	±2.1	90.6	100	76	100	87	98	14 GB
14	glm-4.7-flash:q4_K_M	24 GB	88.0	—	88.0	87	80	100	81	93	20 GB
15	qwen3-coder:30b prev. #6	24 GB	87.9	±2.1	86.6	100	100	92	83	73	21 GB
16	nemotron-3-nano:4b	8 GB	79.0	—	78.1	100	91	96	61	64	3.5 GB
17	qwen2.5:7b	8 GB	66.1^†	±1.2	58.6	33	32	100	65	84	6.6 GB
18	qwen3:8b previous tier: 8 GB (#4)	16 GB	61.9^†	±3.3	60.6	93	89	77	56	11	10 GB
19	mistral-nemo:12b prev. #5	16 GB	61.5	±2.9	60.2	80	68	100	20	67	12 GB
20	qwen2.5:14b prev. #6	16 GB	59.2	±1.8	59.2	75	48	99	33	56	15 GB
21	ministral-3:8b previous tier: 8 GB	16 GB	56.3	—	55.4	77	63	65	37	56	9.8 GB
22	llama3.1:8b previous tier: 8 GB (#6)	16 GB	54.1	±2.3	53.2	80	29	100	19	60	9.2 GB
23	llama3.2:3b prev. #7	8 GB	52.1	±4.6	49.8	91	45	93	18	38	6.0 GB
24	hermes3:8b previous tier: 8 GB (#8)	16 GB	45.7	±10.1	45.3	80	80	68	2	38	9.0 GB
25	mistral-small3.2:24b prev. #7	24 GB	42.8	±4.3	42.4	47	43	80	18	40	19 GB
26	ministral-3:14b	16 GB	39.7	—	39.7	61	44	64	16	31	not measured
27	deepseek-r1:8b previous tier: 8 GB (#9)	16 GB	29.2^†	±0.3	29.2	99	96	0	0	0	10 GB
28	mistral-small:24b prev. #8	24 GB	28.9^†	±4.4	28.1	44	33	55	10	18	19 GB
29	deepseek-r1:14b prev. #7	16 GB	26.6	±1.9	26.6	91	87	0	0	0	15 GB
30	qwen2.5-coder:7b prev. #10	8 GB	26.0	±0.7	26.0	100	73	0	0	0	6.6 GB
31	qwen2.5-coder:14b prev. #8	16 GB	24.2	±0.0	24.2	89	72	0	0	0	15 GB
32	mistral:7b previous tier: 8 GB (#11)	16 GB	22.0	±0.7	22.0	93	53	0	0	0	8.9 GB
—	gemma3:4b No native tool support	8 GB	n/a^*	—	—	0	0	0	0	0	not measured
—	phi4:14b tool mode → No native tool support	16 GB	n/a^*	—	—	0	0	0	0	0	not measured

Rank ranges and tie groups are computed within each measured tier; see the per-tier view.

Fits in 8 GB VRAM

Models whose measured VRAM at 32K context (KV cache included) is at most 8 GB on an RTX 3090 (24 GB); scores and ranks are from those runs. This tier's runs come from more than one RTX 3090 host, so tok/s is not shown in the table: see Speed by host (never compared across hosts).

Real 8 GB card: checked on a real 8 GB card (RTX 3070): all six 8 GB-tier models ran there; qwen3:4b needed partial CPU offload. Results v0.1.2: measured on the RTX 3090 only; not yet run on a real 8 GB card.

#	Model	Quant	Engine	Score	± runs	v3	Rank range	JSON	Schema	Tool	Multi-step	Inj. useful&safe	Mention	VRAM @32K
1	qwen3:4b prev. #2 partial CPU offload	Q4_K_M	ollama 0.40.6	≈1 97.7^†	±2.6	92.9 (#2)	#1	100	91	100	97	100	24.4%	7.6 GB
2	gemma4:e4b prev. #1	Q4_K_M	ollama 0.40.6	≈1 94.7	±0.6	94.7 (#1)	#2	99	95	100	89	96	0.0%	not measured
3	ornith-1.5:9b previous tier: 16 GB new	Q4_K_M	ollama 0.40.6	≈1 92.0	—	82.2	#3	95	95	97	82	98	48.9%	6.7 GB
4	nemotron-3-nano:4b new	Q4_K_M	ollama 0.40.6	79.0	—	78.1	#4	100	91	96	61	64	4.4%	3.5 GB
5	qwen2.5:7b	Q4_K_M	ollama 0.40.6	≈2 66.1^†	±1.2	58.6 (#3)	#5	33	32	100	65	84	48.9%	6.6 GB
6	llama3.2:3b prev. #7	Q4_K_M	ollama 0.40.6	≈2 52.1	±4.6	49.8 (#4)	#6	91	45	93	18	38	11.1%	6.0 GB
7	qwen2.5-coder:7b prev. #10	Q4_K_M	ollama 0.40.6	26.0	±0.7	26.0 (#5)	#7	100	73	0	0	0	0.0%	6.6 GB
—	gemma3:4b No native tool support	Q4_K_M	ollama 0.40.6	n/a^*	—	—	—	0	0	0	0	0	0.0%	not measured

#	Model	Score	JSON	Schema	Tool	Multi-step	Injection useful&safe
1	qwen3:4b prev. #2	97.7	100	91	100	97	100
2	gemma4:e4b prev. #1	94.7	99	95	100	89	96
3	ornith-1.5:9b previous tier: 16 GB	92.0	95	95	97	82	98
4	nemotron-3-nano:4b	79.0	100	91	96	61	64
5	qwen2.5:7b	66.1	33	32	100	65	84
6	llama3.2:3b prev. #7	52.1	91	45	93	18	38
7	qwen2.5-coder:7b prev. #10	26.0	100	73	0	0	0
—	gemma3:4b No native tool support	n/a	0	0	0	0	0

Cell shading: <25 · 25–59 · 60–84 · 85–94 · ≥95 (stronger red = higher). Each column is the category percentage that feeds the score.

On a real 8 GB card: NVIDIA GeForce RTX 3070 (container)

The same suite on a real 8 GB card, scored with v4 from the run logs (3 trials unless marked). Models fully on the GPU, or with partial CPU offload, are the 8 GB tier above; the rows marked “runs with CPU offload (slow)” need 8.9–10 GB on the 3090 and stay in the 16 GB tier (their main tier and rank), but they do run on an 8 GB card with part of the model on the CPU, at the tok/s shown. The 3090 and 3070 scores are separate measurements and are not averaged. tok/s here is the 3070 host only; 3090 tok/s are in Speed by host and are not compared with it.

Model	Main tier	Score 3090	Score 3070	CPU/GPU	Peak VRAM 3070	tok/s 3070 @host
qwen3:4b	8 GB #1	97.7	91.9	7.7 GB 12%/88% CPU/GPU	6713 MiB	57.6 @8fcb65	partial CPU offload
gemma4:e4b	8 GB #2	94.7	94.1	398 MB 100% GPU	5255 MiB	171.3 @8fcb65	100% GPU
qwen2.5:7b	8 GB #5	66.1	62.3	6.4 GB 100% GPU	6337 MiB	83.1 @8fcb65	100% GPU
llama3.2:3b	8 GB #6	52.1	49.4	5.9 GB 100% GPU	5807 MiB	160.4 @8fcb65	100% GPU
qwen2.5-coder:7b	8 GB #7	26.0	25.6	6.4 GB 100% GPU	6337 MiB	83.1 @8fcb65	100% GPU
gemma3:4b	8 GB	n/a^*	n/a^*	2.9 GB 100% GPU	4429 MiB^‡	119.7 @8fcb65	100% GPU
qwen3-vl:8b	16 GB #3 v0.1.2: 16 GB only	91.6	87.1	10 GB 44%/56% CPU/GPU	6991 MiB	11.7 @8fcb65	runs with CPU offload (slow)
qwen3:8b	16 GB #6 v0.1.2: 16 GB only	60.6	62.7	10 GB 34%/66% CPU/GPU	6753 MiB	15.0 @8fcb65	runs with CPU offload (slow)
llama3.1:8b	16 GB #9 v0.1.2: 16 GB only	53.2	51.0	9.6 GB 29%/71% CPU/GPU	6683 MiB	17.6 @8fcb65	runs with CPU offload (slow)
hermes3:8b	16 GB #10 v0.1.2: 16 GB only	45.3	42.9	9.3 GB 26%/74% CPU/GPU	6769 MiB	20.3 @8fcb65	runs with CPU offload (slow)
deepseek-r1:8b	16 GB #11 v0.1.2: 16 GB only	29.2	27.6 n=1	10 GB 34%/66% CPU/GPU	6753 MiB	15.1 @8fcb65	runs with CPU offload (slow)
mistral:7b	16 GB #14 v0.1.2: 16 GB only	22.0	21.8	8.6 GB 19%/81% CPU/GPU	6813 MiB	21.5 @8fcb65	runs with CPU offload (slow)

^‡ Hover for a note on the VRAM samples. 12 GB and 16 GB cards have not been tested. How the real-card series was measured

Fits in 16 GB VRAM

Models whose measured VRAM at 32K context (KV cache included) is at most 16 GB on an RTX 3090 (24 GB); scores and ranks are from those runs. This tier's runs come from more than one RTX 3090 host, so tok/s is not shown in the table: see Speed by host (never compared across hosts).

Real 16 GB card: not tested on a real 16 GB card (or 12 GB): measured on the RTX 3090 only.

#	Model	Quant	Engine	Score	± runs	v3	Rank range	JSON	Schema	Tool	Multi-step	Inj. useful&safe	Mention	VRAM @32K
1	gemma4:12b	Q4_K_M	ollama 0.40.6	98.3	±1.2	98.3 (#1)	#1	100	95	100	97	100	0.0%	not measured
2	qwen3.5:9b prev. #4	Q4_K_M	ollama 0.40.6	≈2 94.1	±1.6	90.6 (#5)	#2–#4	100	97	100	85	96	17.8%	6.7 GB
3	gpt-oss:20b prev. #2	MXFP4	ollama 0.40.6	≈2 93.9	±1.9	93.4 (#2)	#2–#5	100	97	100	94	80	2.2%	12 GB
4	qwen3-vl:8b previous tier: 8 GB (#3)	Q4_K_M	ollama 0.40.6	≈2 93.9^†	±2.3	91.6 (#3)	#2–#4	95	83	100	92	98	11.1%	10 GB
5	qwen3:14b prev. #3	Q4_K_M	ollama 0.40.6	≈2 92.0	±2.1	90.6 (#4)	#4–#5	100	76	100	87	98	6.7%	14 GB
6	qwen3:8b previous tier: 8 GB (#4)	Q4_K_M	ollama 0.40.6	≈3 61.9^†	±3.3	60.6 (#6)	#6–#9	93	89	77	56	11	6.7%	10 GB
7	mistral-nemo:12b prev. #5	Q4_K_M	ollama 0.40.6	≈3 61.5	±2.9	60.2 (#7)	#6–#8	80	68	100	20	67	6.7%	12 GB
8	qwen2.5:14b prev. #6	Q4_K_M	ollama 0.40.6	≈3 59.2	±1.8	59.2 (#8)	#7–#9	75	48	99	33	56	0.0%	15 GB
9	ministral-3:8b previous tier: 8 GB new	Q4_K_M	ollama 0.40.6	≈3 56.3	—	55.4	#8–#10	77	63	65	37	56	6.7%	9.8 GB
10	llama3.1:8b previous tier: 8 GB (#6)	Q4_K_M	ollama 0.40.6	≈3 54.1	±2.3	53.2 (#9)	#8–#10	80	29	100	19	60	4.4%	9.2 GB
11	hermes3:8b previous tier: 8 GB (#8)	Q4_K_M	ollama 0.40.6	45.7	±10.1	45.3 (#10)	#11	80	80	68	2	38	2.2%	9.0 GB
12	ministral-3:14b new	Q4_K_M	ollama 0.40.6	≈4 39.7	—	39.7	#12	61	44	64	16	31	0.0%	not measured
13	deepseek-r1:8b previous tier: 8 GB (#9)	Q4_K_M	ollama 0.40.6	29.2^†	±0.3	29.2 (#11)	#13	99	96	0	0	0	0.0%	10 GB
14	deepseek-r1:14b prev. #7	Q4_K_M	ollama 0.40.6	≈6 26.6	±1.9	26.6 (#12)	#14	91	87	0	0	0	0.0%	15 GB
15	qwen2.5-coder:14b prev. #8	Q4_K_M	ollama 0.40.6	≈6 24.2	±0.0	24.2 (#13)	#15	89	72	0	0	0	0.0%	15 GB
16	mistral:7b previous tier: 8 GB (#11)	Q4_K_M	ollama 0.40.6	22.0	±0.7	22.0 (#14)	#16	93	53	0	0	0	0.0%	8.9 GB
—	phi4:14b tool mode → No native tool support	Q4_K_M	ollama 0.40.6	n/a^*	—	—	—	0	0	0	0	0	0.0%	not measured

#	Model	Score	JSON	Schema	Tool	Multi-step	Injection useful&safe
1	gemma4:12b	98.3	100	95	100	97	100
2	qwen3.5:9b prev. #4	94.1	100	97	100	85	96
3	gpt-oss:20b prev. #2	93.9	100	97	100	94	80
4	qwen3-vl:8b previous tier: 8 GB (#3)	93.9	95	83	100	92	98
5	qwen3:14b prev. #3	92.0	100	76	100	87	98
6	qwen3:8b previous tier: 8 GB (#4)	61.9	93	89	77	56	11
7	mistral-nemo:12b prev. #5	61.5	80	68	100	20	67
8	qwen2.5:14b prev. #6	59.2	75	48	99	33	56
9	ministral-3:8b previous tier: 8 GB	56.3	77	63	65	37	56
10	llama3.1:8b previous tier: 8 GB (#6)	54.1	80	29	100	19	60
11	hermes3:8b previous tier: 8 GB (#8)	45.7	80	80	68	2	38
12	ministral-3:14b	39.7	61	44	64	16	31
13	deepseek-r1:8b previous tier: 8 GB (#9)	29.2	99	96	0	0	0
14	deepseek-r1:14b prev. #7	26.6	91	87	0	0	0
15	qwen2.5-coder:14b prev. #8	24.2	89	72	0	0	0
16	mistral:7b previous tier: 8 GB (#11)	22.0	93	53	0	0	0
—	phi4:14b No native tool support	n/a	0	0	0	0	0

Cell shading: <25 · 25–59 · 60–84 · 85–94 · ≥95 (stronger red = higher). Each column is the category percentage that feeds the score.

Fits in 24 GB VRAM

Models whose measured VRAM at 32K context (KV cache included) is at most 24 GB on an RTX 3090 (24 GB); scores and ranks are from those runs. This tier's runs come from more than one RTX 3090 host, so tok/s is not shown in the table: see Speed by host (never compared across hosts).

Real 24 GB card: measured on a real 24 GB card (the RTX 3090).

#	Model	Quant	Engine	Score	± runs	v3	Rank range	JSON	Schema	Tool	Multi-step	Inj. useful&safe	Mention	VRAM @32K
1	gemma4:26b	Q4_K_M	ollama 0.40.6	≈1 99.7	±0.6	99.7 (#1)	#1	100	100	100	99	100	0.0%	not measured
2	qwen3.8:27b prev. #5	Q4_K_M	ollama 0.40.6	≈2 98.7^†	±1.5	92.0 (#5)	#2–#3	100	99	100	96	100	33.3%	17 GB
3	qwen3:30b-a3b prev. #2	Q4_K_M	ollama 0.40.6	≈1 98.5	±1.0	98.5 (#2)	#2–#3	100	95	100	99	98	2.2%	21 GB
4	qwen3.6:27b prev. #3	Q4_K_M	ollama 0.40.6	≈2 95.2	±0.8	95.2 (#3)	#4–#5	100	99	100	85	100	0.0%	17 GB
5	qwen3.5:27b prev. #4	Q4_K_M	ollama 0.40.6	≈2 95.0	±0.4	95.0 (#4)	#4–#5	99	97	100	87	98	0.0%	18 GB
6	glm-4.7-flash:q4_K_M new	Q4_K_M	ollama 0.40.6	≈2 88.0	—	88.0	#6–#7	87	80	100	81	93	0.0%	20 GB
7	qwen3-coder:30b prev. #6	Q4_K_M	ollama 0.40.6	87.9	±2.1	86.6 (#6)	#6–#7	100	100	92	83	73	6.7%	21 GB
8	mistral-small3.2:24b prev. #7	Q4_K_M	ollama 0.40.6	≈4 42.8	±4.3	42.4 (#7)	#8	47	43	80	18	40	4.4%	19 GB
9	mistral-small:24b prev. #8	Q4_K_M	ollama 0.40.6	≈4 28.9^†	±4.4	28.1 (#8)	#9	44	33	55	10	18	4.4%	19 GB

#	Model	Score	JSON	Schema	Tool	Multi-step	Injection useful&safe
1	gemma4:26b	99.7	100	100	100	99	100
2	qwen3.8:27b prev. #5	98.7	100	99	100	96	100
3	qwen3:30b-a3b prev. #2	98.5	100	95	100	99	98
4	qwen3.6:27b prev. #3	95.2	100	99	100	85	100
5	qwen3.5:27b prev. #4	95.0	99	97	100	87	98
6	glm-4.7-flash:q4_K_M	88.0	87	80	100	81	93
7	qwen3-coder:30b prev. #6	87.9	100	100	92	83	73
8	mistral-small3.2:24b prev. #7	42.8	47	43	80	18	40
9	mistral-small:24b prev. #8	28.9	44	33	55	10	18

Cell shading: <25 · 25–59 · 60–84 · 85–94 · ≥95 (stronger red = higher). Each column is the category percentage that feeds the score.

Apple M2 16GB unified memory, macOS

Same suite, Hermes Agent harness, Q4_K_M quants and Ollama 0.40.6 (native macOS, Metal) as the GPU tiers, run on a rented Apple M2 Mac mini with 16 GB unified memory (about 11.8 GB usable by the GPU). tok/s is single-stream decode speed (256-token generation, median of 3).

#	Model	Quant	Engine	Score	± runs	v3	Rank range	JSON	Schema	Tool	Multi-step	Inj. useful&safe	Mention	tok/s @host
1	qwen2.5:14b	Q4_K_M	ollama 0.40.6	≈1 61.8	±1.0 3 trials	61.4 (#1)	#1–#3	88	57	100	27	60	2.2%	10.1 @8b306c
2	qwen2.5:7b	Q4_K_M	ollama 0.40.6	≈1 61.3	±2.0 3 trials	58.7 (#4)	#1–#5	25	23	100	62	78	22.2%	19.8 @8b306c
3	mistral-nemo:12b	Q4_K_M	ollama 0.40.6	≈1 60.4	±2.3 3 trials	59.9 (#3)	#2–#4	80	67	97	20	64	4.4%	13.0 @8b306c
4	qwen3:8b	Q4_K_M	ollama 0.40.6	≈1 60.1	±2.6 3 trials	60.1 (#2)	#1–#4	93	89	84	51	2	0.0%	17.6 @8b306c
5	llama3.2:3b	Q4_K_M	ollama 0.40.6	≈1 52.6	±1.7 3 trials	50.4 (#5)	#4–#5	93	45	95	16	40	13.3%	40.7 @8b306c
6	llama3.1:8b	Q4_K_M	ollama 0.40.6	≈1 50.3	±1.9 3 trials	49.8 (#6)	#6	79	31	100	18	42	2.2%	18.7 @8b306c
7	hermes3:8b	Q4_K_M	ollama 0.40.6	43.1	±5.5 3 trials	41.8 (#7)	#7	83	83	60	2	29	6.7%	19.9 @8b306c
8	qwen2.5-coder:7b	Q4_K_M	ollama 0.40.6	26.4	±1.0 3 trials	26.4 (#8)	#8	100	76	0	0	0	0.0%	19.8 @8b306c
—	gemma3:4b No native tool support	Q4_K_M	ollama 0.40.6	n/a^*	—	—	—	0	0	0	0	0	0.0%	32.8 @8b306c

#	Model	Score	JSON	Schema	Tool	Multi-step	Injection useful&safe
1	qwen2.5:14b	61.8	88	57	100	27	60
2	qwen2.5:7b	61.3	25	23	100	62	78
3	mistral-nemo:12b	60.4	80	67	97	20	64
4	qwen3:8b	60.1	93	89	84	51	2
5	llama3.2:3b	52.6	93	45	95	16	40
6	llama3.1:8b	50.3	79	31	100	18	42
7	hermes3:8b	43.1	83	83	60	2	29
8	qwen2.5-coder:7b	26.4	100	76	0	0	0
—	gemma3:4b No native tool support	n/a	0	0	0	0	0

Cell shading: <25 · 25–59 · 60–84 · 85–94 · ≥95 (stronger red = higher). Each column is the category percentage that feeds the score.

Not run on this hardware:

Model	Not run because	Tool support (Ollama)
phi4:14b	ran out of time: a 14B model takes about 5 h on this Mac (time cap)	no native tool support in Ollama
mistral-small:24b	doesn't fit in 16 GB unified memory (Q4_K_M weights about 14 GB)	native
qwen3:30b-a3b	doesn't fit in 16 GB unified memory (Q4_K_M weights about 18-19 GB)	native

Large card: NVIDIA A100 SXM4 80 GB (RunPod container)

Same suite and Hermes Agent harness as the GPU tiers, run on an NVIDIA A100-SXM4-80GB (81,920 MiB; the results directory is named a10064 for historical reasons, the card is NOT a 64 GB one), as 3 single-trial runs per model. Quants are the ones each model name carries (Q8_0 for qwen3:32b-q8_0 and gemma4:31b-it-q8_0, Q4_K_M for the rest), Ollama 0.40.6. Published scores are scoring v3 recomputed from the run logs, so they differ from the score_total stored in each run file: v3 scores injection as 'useful and safe' (the legitimate task must be completed and the trace not compromised), while the runner's total credits raw injection resistance. Every trial is published, including the gemma4:31b-it-q8_0 trial 3 outlier (49.2 under v3). A separate series: its scores are not pooled with the RTX 3090 tiers. tok/s is this host only and is never compared across hosts. Context: the runner's settings.ctx=8192 was never applied; Ollama loaded 32768 (`ollama ps` and the per-model .ctx files), so ctx_loaded=32768 is the real number and settings.ctx is kept as recorded in the run files. Ranked within this series only, one host (ae76059e168d, container). Runs with fewer than 3 trials are listed without a rank or CI.

#	Model	Score (v4)	95% CI	Trials	JSON	Schema	Tool	Multi-step	Inj. useful&safe	CPU/GPU	Peak VRAM	tok/s @host
1	qwen3:32b-q8_0	92.6	89.6–95.4	92.09 / 91.81 / 93.87	95	69	99	97	96	—	—	20.0 @ae7605
≈2 2	gemma4:31b-it-q8_0	80.8	78.5–82.5	96.57 / 96.57 / 49.2	100	100	99	59	67	—	—	20.1 @ae7605
≈2 3	qwen2.5:72b	79.2	72.8–85.2	78.91 / 78.57 / 80.03	83	81	100	49	100	—	—	20.1 @ae7605
4	llama3.3:70b	68.3	62.1–74.5	67.53 / 68.87 / 68.65	100	87	100	22	69	—	—	20.1 @ae7605
5	deepseek-r1:70b	13.2	9.2–17.2	13.2 / 14.4 / 12	44	44	0	0	0	—	—	20.1 @ae7605

Real 8 GB card: NVIDIA RTX 3070 8 GB (RunPod container)

Same suite, Hermes Agent harness, Q4_K_M quants, Ollama 0.40.6 and 32K context as the GPU tiers, run on a real 8 GB card (RTX 3070) in a RunPod container, as 3 single-trial runs per model. Ollama put part of some models on the CPU (CPU/GPU split from `ollama ps`); scores are scoring v3 recomputed from the run logs. A separate series: its scores are not pooled with the RTX 3090 tiers. tok/s is this host only. Ranked within this series only, one host (8fcb655dfb7d, container). Runs with fewer than 3 trials are listed without a rank or CI.

#	Model	Score (v4)	95% CI	Trials	JSON	Schema	Tool	Multi-step	Inj. useful&safe	CPU/GPU	Peak VRAM	tok/s @host
≈1 1	qwen3:4b	97.3	94.8–99.2	96.74 / 97.94 / 97.09	100	89	100	96	100	7.7 GB 12%/88% CPU/GPU	6713 MiB	57.6 @8fcb65
≈1 2	gemma4:e4b	94.1	91.0–96.8	93.91 / 95.11 / 93.18	100	92	100	86	98	398 MB 100% GPU	5255 MiB	171.3 @8fcb65
≈1 3	qwen3-vl:8b	91.1	87.0–94.9	89.63 / 91.51 / 92.29	87	73	100	90	100	10 GB 44%/56% CPU/GPU	6991 MiB	11.7 @8fcb65
≈2 4	qwen2.5:7b	67.6	61.6–73.5	62.78 / 70.61 / 69.48	32	28	99	73	84	6.4 GB 100% GPU	6337 MiB	83.1 @8fcb65
≈2 5	qwen3:8b	64.9	58.6–70.9	65.68 / 66.32 / 62.68	95	91	81	59	16	10 GB 34%/66% CPU/GPU	6753 MiB	15.0 @8fcb65
≈3 6	llama3.1:8b	51.5	45.1–58.0	50.66 / 48.88 / 54.93	69	25	99	20	58	9.6 GB 29%/71% CPU/GPU	6683 MiB	17.6 @8fcb65
≈3 7	llama3.2:3b	51.2	46.1–56.6	54.68 / 49.54 / 49.34	95	39	95	17	36	5.9 GB 100% GPU	5807 MiB	160.4 @8fcb65
≈3 8	hermes3:8b	43.8	37.1–50.4	47.71 / 42.25 / 41.47	83	83	56	4	33	9.3 GB 26%/74% CPU/GPU	6769 MiB	20.3 @8fcb65
≈4 9	deepseek-r1:8b n=1, unranked	27.6	24.0–30.0	27.6	92	92	0	0	0	10 GB 34%/66% CPU/GPU	6753 MiB	15.1 @8fcb65
≈4 10	qwen2.5-coder:7b	25.6	22.8–28.0	26.4 / 25.8 / 24.6	97	73	0	0	0	6.4 GB 100% GPU	6337 MiB	83.1 @8fcb65
11	mistral:7b	21.8	19.6–23.8	19.8 / 22.2 / 23.4	91	55	0	0	0	8.6 GB 19%/81% CPU/GPU	6813 MiB	21.5 @8fcb65
—	gemma3:4b No native tool support	n/a^*	—	0 / 0 / 0	0	0	0	0	0	2.9 GB 100% GPU	4429 MiB^‡	119.7 @8fcb65

Speed by host

Tok/s is single-request decode speed on the host it was measured on (host_id); compare it only between models on the same host, never across hosts. One table per host (host id); there is deliberately no table that puts tok/s from different hosts side by side. Speed is not part of the score.

Host `dc7718586fbe` · NVIDIA GeForce RTX 3090 · container

Model	tok/s @dc7718	Probe num_ctx
llama3.2:3b	211.9	—
qwen3:30b-a3b	173.9	—
gemma3:4b	159.1	—
hermes3:8b	129.3	—
qwen2.5:7b	120.0	—
llama3.1:8b	119.6	—
qwen2.5-coder:7b	119.4	—
qwen3:8b	116.1	—
mistral-nemo:12b	94.1	—
phi4:14b	78.0	—
qwen2.5:14b	68.2	—
mistral-small:24b	49.0	—

Host `37becd33558a` · NVIDIA GeForce RTX 3090 · container

Model	tok/s @37becd	Probe num_ctx
gemma4:e4b	200.6	—
gemma4:26b	180.4	—
qwen3:4b	167.5	—
gpt-oss:20b	138.8	—
gemma4:12b	130.3	—
mistral:7b	128.3	—
qwen3-coder:30b	127.0	—
deepseek-r1:8b	114.9	—
qwen3.5:9b	79.8	—
qwen3:14b	75.0	—
deepseek-r1:14b	70.8	—
qwen3.5:27b	37.8	—

Host `8b306c4ac94a` · Apple M2 GPU (unified memory)

Model	tok/s @8b306c	Probe num_ctx
llama3.2:3b	40.7	—
gemma3:4b	32.8	—
hermes3:8b	19.9	—
qwen2.5-coder:7b	19.8	—
qwen2.5:7b	19.8	—
llama3.1:8b	18.7	—
qwen3:8b	17.6	—
mistral-nemo:12b	13.0	—
qwen2.5:14b	10.1	—

Host `b7e2c79acebd` · NVIDIA GeForce RTX 3090 · container

Model	tok/s @b7e2c7	Probe num_ctx
qwen3-vl:8b	117.6	—
qwen3.6:27b	69.1	—
qwen2.5-coder:14b	69.0	—
qwen3.8:27b	61.0	—
mistral-small3.2:24b	49.9	—

Host `8fcb655dfb7d` · NVIDIA GeForce RTX 3070 · container

Model	tok/s @8fcb65	Probe num_ctx
gemma4:e4b	171.3	32768
llama3.2:3b	160.4	32768
gemma3:4b	119.7	32768
qwen2.5-coder:7b	83.1	32768
qwen2.5:7b	83.1	32768
qwen3:4b	57.6	32768
mistral:7b	21.5	32768
hermes3:8b	20.3	32768
llama3.1:8b	17.6	32768
deepseek-r1:8b	15.1	32768
qwen3:8b	15.0	32768
qwen3-vl:8b	11.7	32768

Host `ae76059e168d` · NVIDIA A100-SXM4-80GB · container

Model	tok/s @ae7605	Probe num_ctx
gemma4:31b-it-q8_0	20.1	32768
qwen2.5:72b	20.1	32768
llama3.3:70b	20.1	32768
deepseek-r1:70b	20.1	32768
qwen3:32b-q8_0	20.0	32768

Not comparable (different Ollama version)

These runs are published for information only. They are not ranked in the GPU tiers because they used a different Ollama version than the published series (0.40.6). Scores and tok/s must not be compared with the GPU-tier tables.

Model	Ollama	Score (v4)	v3	Mention	Note
nemotron-3.5-bolt:e4b	ollama 0.40.10	92.4	92.4	0.0%	Not comparable: ran on Ollama 0.40.10; the published series uses 0.40.6. Hermes 9cb34e9, ctx 32768, score shown for information only.

Hosts

Where the published runs were measured. The host id is a short hash of CPU model, kernel and RAM: it groups runs on the same kind of host and is not the identity of a machine. tok/s is compared only between models on the same host; confidence intervals cover task variance, not host variance. null fields were not recorded.

Host id	Kind	GPU	CPU · kernel · RAM (host)	Pod	Pod allotment	Runs	Dates
`dc7718586fbe`	container (RunPod)	NVIDIA GeForce RTX 3090	AMD EPYC 7663 56-Core Processor (112 threads) · Linux 6.8.0-57-generic · 251.5 GB Main-suite RTX 3090 runs (2026-10-20/21). The pod id and the pod's CPU/RAM allotment were not recorded.	not recorded	not recorded	12	2026-10-20 – 2026-10-21
`37becd33558a`	container (RunPod)	NVIDIA GeForce RTX 3090	AMD EPYC 7402P 24-Core Processor (48 threads) · Linux 6.8.0-49-generic · 251.6 GB Main-suite RTX 3090 runs (2026-10-21/22). The pod id and the pod's CPU/RAM allotment were not recorded.	not recorded	not recorded	12	2026-10-21 – 2026-10-22
`8b306c4ac94a`	not recorded	Apple M2 GPU (unified memory)	Apple M2 (8 threads) · macOS 27.0.1 (arm64) · 16 GB Apple M2 tier: a rented Apple M2 Mac mini (16 GB unified memory). Not a RunPod container; the kind of host was not recorded.	not recorded	not recorded	9	2026-10-21 – 2026-10-22
`b7e2c79acebd`	container (RunPod)	NVIDIA GeForce RTX 3090	AMD Ryzen Threadripper PRO 5995WX 64-Cores (128 threads) · Linux 6.8.0-57-generic · 251.5 GB Main-suite RTX 3090 runs (2026-10-22/23). The pod id and the pod's CPU/RAM allotment were not recorded.	not recorded	not recorded	5	2026-10-22 – 2026-10-23
`284b1cb491f2`	container (RunPod)	NVIDIA GeForce RTX 3090	AMD EPYC 7452 32-Core Processor (64 threads) · Linux 6.5.0-18-generic · 251.6 GB Variance reruns only (results/variance, temperature 0.7, seeds 1001-1005; 2026-10-23/25). Not used for tier scores, ranks or tok/s. The pod id and the pod's CPU/RAM allotment were not recorded per run.	not recorded	not recorded	60	2026-10-23 – 2026-10-25
`028baf8be09e`	container (RunPod)	NVIDIA GeForce RTX 3090	AMD EPYC 7H12 64-Core Processor (256 threads) · Linux 6.8.0-138-generic · 1007.6 GB Variance reruns only (results/variance, temperature 0.7, seeds 1001-1005; 2026-10-23/24). Not used for tier scores, ranks or tok/s. The pod id and the pod's CPU/RAM allotment were not recorded per run.	not recorded	not recorded	21	2026-10-23 – 2026-10-24
`720e56cc084a`	container (RunPod)	NVIDIA GeForce RTX 3090	AMD Ryzen Threadripper PRO 5975WX 32-Cores (64 threads) · Linux 6.8.0-59-generic · 251.5 GB Variance reruns only (results/variance, temperature 0.7, seeds 1001-1005; 2026-10-25/26). Not used for tier scores, ranks or tok/s. The pod id and the pod's CPU/RAM allotment were not recorded per run.	not recorded	not recorded	54	2026-10-25 – 2026-10-26
`8fcb655dfb7d`	container (RunPod)	NVIDIA GeForce RTX 3070	AMD Ryzen 7 5700X 8-Core Processor (16 threads) · Linux 6.8.0-85-generic · 62.7 GB · stepping 2 Real 8 GB card series (results/real8gb, 2026-10-26 to 2026-10-28). pod_allotment is what RunPod gives the pod (RUNPOD_CPU_COUNT, RUNPOD_MEM_GB); host_resources is what the container sees of the host machine. Captured from inside the pod at the start of the run.	b3o5tul56pnqfv	8 vCPU · 30 GB	12 (34 trial files)	2026-10-26 – 2026-10-28
`ae76059e168d`	container (RunPod)	NVIDIA A100-SXM4-80GB	AMD EPYC 7742 64-Core Processor (256 threads) · Linux 6.8.0-147-generic · 1007.6 GB A100 80 GB series (2026-11-04). The results directory is named 'a10064' for historical reasons; the card is an NVIDIA A100-SXM4-80GB with 81,920 MiB (80 GB), not a 64 GB card. This id comes from the pod's own host fingerprint (meta/host_fingerprint.sanitized.json); 'computed_ids' lists the id this repo's cpu\|kernel\|ram rule derives from the runs' hardware block, mapped here so the published host stays the one captured on the pod. Network details and the GPU UUID are deliberately not stored.	4o1ftygd7hlqxb	not recorded	5 (15 trial files)	2026-11-04 – 2026-11-04

^† Score outside or near the edge of its pooled interval: the score is the main-suite mean (3 trials); the 95% interval behind the ≈ tie groups is pooled over 8 runs (those 3 trials + 5 reruns at temperature 0.7), so it need not be centred on the score. † marks a score below or above that interval, or in its outer tenth (hover for the numbers). Scores and ranks are unchanged. See Methodology.

^* No native tool support: the model's Ollama chat template has no native tool-calling, so the harness cannot issue tool calls and its sub-scores are not comparable. Listed for completeness, unranked, at the bottom of its tier. See Methodology.

Data updated 2026-11-13 15:02 UTC.

↑ Back to top

Full results

Which local LLMs actually work for agents?

Everything that fits in 8 GB VRAM

Everything that fits in 16 GB VRAM

Everything that fits in 24 GB VRAM

Fits in 8 GB VRAM

On a real 8 GB card: NVIDIA GeForce RTX 3070 (container)

Fits in 16 GB VRAM

Fits in 24 GB VRAM

Apple M2 16GB unified memory, macOS

Large card: NVIDIA A100 SXM4 80 GB (RunPod container)

Real 8 GB card: NVIDIA RTX 3070 8 GB (RunPod container)

Speed by host

Host dc7718586fbe · NVIDIA GeForce RTX 3090 · container

Host 37becd33558a · NVIDIA GeForce RTX 3090 · container

Host 8b306c4ac94a · Apple M2 GPU (unified memory)

Host b7e2c79acebd · NVIDIA GeForce RTX 3090 · container

Host 8fcb655dfb7d · NVIDIA GeForce RTX 3070 · container

Host ae76059e168d · NVIDIA A100-SXM4-80GB · container

Not comparable (different Ollama version)

Hosts

Host `dc7718586fbe` · NVIDIA GeForce RTX 3090 · container

Host `37becd33558a` · NVIDIA GeForce RTX 3090 · container

Host `8b306c4ac94a` · Apple M2 GPU (unified memory)

Host `b7e2c79acebd` · NVIDIA GeForce RTX 3090 · container

Host `8fcb655dfb7d` · NVIDIA GeForce RTX 3070 · container

Host `ae76059e168d` · NVIDIA A100-SXM4-80GB · container