qwen3:4b
Family: qwen3 · 4B parameters · 2 run(s)
Fits in 8 GB VRAM · ollama 0.40.6 · Q4_K_M
Measured VRAM at 32K context (KV cache included, ollama ps on the RTX 3090): 7.6 GB · previous rank in this tier: #2 (results v0.1.1, before the v0.1.2 re-tiering; score unchanged). Tier rule
97.7/100 · scoring v4
95% task-bootstrap CI (main suite, 3 trials): 95.9–99.2 · pooled over 8 runs (3 main-suite trials + 5 reruns at temperature 0.7): 86.3–93.8 · ≈ statistically tied with other models in pooled tie group 1 of this tier (the leaderboard’s ≈) · rank #1 in tier, rank range under weight sensitivity #1–#1 (5th–95th percentile; min #1, max #1) · v3: 92.9 (#2) · mention 24.4% (11/45) · how
† The published score (97.7, main-suite mean of 3 trials) lies above the pooled 8-run interval (86.3–93.8): the 5 reruns at temperature 0.7 scored lower (rerun mean 88.62). Score and rank stay those of the main suite. Why
Plain-language summary
Excellent at tool-using assistant work on this suite.
Good for
- returning clean JSON
- matching the requested format
- picking the right tool
- multi-step plans
- staying safe under prompt injection
Weak at
- —
Category breakdown
| JSON | Schema | Tool | Multi-step | Injection (useful&safe) | tok/s | p50 s/task | p95 s/task |
|---|---|---|---|---|---|---|---|
| 100 | 91 | 100 | 97 | 100 | 167.5 @37becd | 23.3 | 47.2 |
Prompt injection (15 tasks × 3 trials): useful and safe 100.0% (task completed and not compromised; counts in the score) · utility under attack 100.0% (completed the user's task) · harmful-action attack success 0.0% (a real harmful tool call or canary leak). Mentions (quoting the attacker without acting): 24.4% (11/45); they do not lower the score. v3 useful and safe: 75.6%. v2.1 used injection resistance 75.6 × tool-call rate 1. Definitions
- Run:
20261021-qwen3-4b-e68e65· 2026-10-21T16:51:01+00:00 - GPU: NVIDIA GeForce RTX 3090 (24 GB) · Linux 6.8.0-49-generic
- Host:
37becd33558a· container (RunPod) · host AMD EPYC 7402P 24-Core Processor, 251.6 GB RAM · measured 2026-10-21T16:51:01+00:00 · hosts - Harness: hermes-agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d · trials 3 · sampling: Ollama default (temperature and seed not set by harness) · context window: 32768 tokens as loaded by Ollama (the unused runner field records 8192; see Methodology)
- Score per trial (v4): 93.94 / 90.08 / 94.54 (SD 2.42, range 90.08–94.54)
- GGUF sha256:
3e4cb14174460404e7a233e531675303b2fbf7749c02f91864fe311ab6344e4f - Sponsor disclosure: none
Real 8 GB card: NVIDIA RTX 3070 8 GB (RunPod container) · ollama 0.40.6 · Q4_K_M
Separate series, not part of the tier ranking. 3 single-trial run(s) merged, scoring v4 from the run logs. CPU/GPU split (ollama ps): 7.7 GB 12%/88% CPU/GPU partial CPU offload · peak VRAM (nvidia-smi): 6713 MiB of 8192 MiB · series table · how it was measured
97.3/100 · scoring v4
95% task-bootstrap CI: 94.8–99.2 · rank #1 in this series · ≈ tie group 1, rank range under weight sensitivity #1–#1 · how
Plain-language summary
Excellent at tool-using assistant work on this suite.
Good for
- returning clean JSON
- matching the requested format
- picking the right tool
- multi-step plans
- staying safe under prompt injection
Weak at
- —
Category breakdown
| JSON | Schema | Tool | Multi-step | Injection (useful&safe) | tok/s | p50 s/task | p95 s/task |
|---|---|---|---|---|---|---|---|
| 100 | 89 | 100 | 96 | 100 | 57.6 @8fcb65 | — | — |
Prompt injection (15 tasks × 3 trials): useful and safe 100.0% (task completed and not compromised; counts in the score) · utility under attack 100.0% (completed the user's task) · harmful-action attack success 0.0% (a real harmful tool call or canary leak). Mentions (quoting the attacker without acting): 26.7% (12/45); they do not lower the score. v3 useful and safe: 73.3%. Definitions
- Run:
real8gb-qwen3-4b· 2026-10-27T05:37:11+00:00 - Hardware: Real 8 GB card: NVIDIA RTX 3070 8 GB (RunPod container) · NVIDIA GeForce RTX 3070 · Linux 6.8.0-85-generic
- Host:
8fcb655dfb7d· container (RunPod) · pod b3o5tul56pnqfv · pod allotment 8 vCPU / 30 GB · host AMD Ryzen 7 5700X 8-Core Processor, 62.7 GB RAM · measured 2026-10-27T05:37:11+00:00 · hosts - Harness: hermes-agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d · trials 3 · sampling: Ollama default (temperature and seed not set by harness) · context window: 32768 tokens as loaded by Ollama (the unused runner field records ; see Methodology)
- Score per trial (v4): 92.74 / 91.28 / 91.75 (SD 0.75, range 91.28–92.74)
- GGUF sha256:
3e4cb14174460404e7a233e531675303b2fbf7749c02f91864fe311ab6344e4f - Sponsor disclosure: none
Chat template vs upstream
Matches: the chat template in the GGUF we tested is identical to the provider's upstream template after whitespace normalisation.
- GGUF template sha256 (normalised):
240f81df50a712a8cf059ff79ab620967e0aa21e861e8ae799afa63a9b2ee62f - Upstream reference sha256 (normalised):
240f81df50a712a8cf059ff79ab620967e0aa21e861e8ae799afa63a9b2ee62f· source
Labels are only “matches / differs / no reference”, never safe/unsafe. Ollama may use a built-in Go template or renderer instead of the GGUF Jinja. Why templates matter: Chat-template backdoors. Not part of the score.