deepseek-r1:8b
Family: deepseek-r1 · 8B parameters · 2 run(s)
Why this model scores about 0 on tool selection: what it wrote instead of tool calls, and whether the template, the engine or the model is the cause. Read the analysis. Applies to all runs on this page; scores unchanged.
Fits in 16 GB VRAM · ollama 0.40.6 · Q4_K_M
Measured VRAM at 32K context (KV cache included, ollama ps on the RTX 3090): 10 GB · previous tier: 8 GB, rank #9 (results v0.1.1). Moved to the 16 GB tier in results v0.1.2 (2026-10-26) because it needs more than 8 GB at 32K context; score unchanged. Tier rule
29.2/100 · scoring v4
95% task-bootstrap CI (main suite, 3 trials): 28.2–30.0 · pooled over 8 runs (3 main-suite trials + 5 reruns at temperature 0.7): 29.2–30.0 · rank #13 in tier, rank range under weight sensitivity #13–#13 (5th–95th percentile; min #10, max #13) · v3: 29.2 (#11) · mention 0.0% (0/45) · how
† The published score (29.2, main-suite mean of 3 trials) is near the low edge of the pooled 8-run interval (29.2–30.0); rerun mean 29.88. Score and rank stay those of the main suite. Why
Plain-language summary
Doesn't work as a tool-using agent in this suite.
Good for
- returning clean JSON
- matching the requested format
Weak at
- picking the right tool
- multi-step plans
- staying safe under prompt injection
Category breakdown
| JSON | Schema | Tool | Multi-step | Injection (useful&safe) | tok/s | p50 s/task | p95 s/task |
|---|---|---|---|---|---|---|---|
| 99 | 96 | 0 | 0 | 0 | 114.9 @37becd | 36.0 | 81.2 |
Prompt injection (15 tasks × 3 trials): useful and safe 0.0% (task completed and not compromised; counts in the score) · utility under attack 0.0% (completed the user's task) · harmful-action attack success 0.0% (a real harmful tool call or canary leak). Mentions (quoting the attacker without acting): 0.0% (0/45); they do not lower the score. v3 useful and safe: 0.0%. v2.1 used injection resistance 100.0 × tool-call rate 0. Definitions
- Run:
20261021-deepseek-r1-8b-f9a788· 2026-10-21T21:40:21+00:00 - GPU: NVIDIA GeForce RTX 3090 (24 GB) · Linux 6.8.0-49-generic
- Host:
37becd33558a· container (RunPod) · host AMD EPYC 7402P 24-Core Processor, 251.6 GB RAM · measured 2026-10-21T21:40:21+00:00 · hosts - Harness: hermes-agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d · trials 3 · sampling: Ollama default (temperature and seed not set by harness) · context window: 32768 tokens as loaded by Ollama (the unused runner field records 8192; see Methodology)
- Score per trial (v4): 30 / 28.8 / 28.8 (SD 0.69, range 28.8–30)
- GGUF sha256:
e6a7edc1a4d7d9b2de136a221a57336b76316cfe53a252aeba814496c5ae439d - Sponsor disclosure: none
Real 8 GB card: NVIDIA RTX 3070 8 GB (RunPod container) · ollama 0.40.6 · Q4_K_M
Separate series, not part of the tier ranking. 1 single-trial run(s) merged, scoring v4 from the run logs. Only 1 of 3 trials: trials 2-3 were skipped because the projected pod cost exceeded the run's budget cap (trial 1 took 18,584 s). Unranked, no confidence interval. CPU/GPU split (ollama ps): 10 GB 34%/66% CPU/GPU runs with CPU offload (slow) (main tier stays 16 GB) · peak VRAM (nvidia-smi): 6753 MiB of 8192 MiB · series table · how it was measured
27.6/100 · scoring v4
95% task-bootstrap CI: 24.0–30.0 · rank #9 in this series · ≈ tie group 4, rank range under weight sensitivity #9–#9 · how
Plain-language summary
Doesn't work as a tool-using agent in this suite.
Good for
- returning clean JSON
- matching the requested format
Weak at
- picking the right tool
- multi-step plans
- staying safe under prompt injection
Category breakdown
| JSON | Schema | Tool | Multi-step | Injection (useful&safe) | tok/s | p50 s/task | p95 s/task |
|---|---|---|---|---|---|---|---|
| 92 | 92 | 0 | 0 | 0 | 15.1 @8fcb65 | — | — |
Prompt injection (15 tasks × 1 trials): useful and safe 0.0% (task completed and not compromised; counts in the score) · utility under attack 0.0% (completed the user's task) · harmful-action attack success 0.0% (a real harmful tool call or canary leak). Mentions (quoting the attacker without acting): 0.0% (0/15); they do not lower the score. v3 useful and safe: 0.0%. Definitions
- Run:
real8gb-deepseek-r1-8b· 2026-10-28T22:51:08+00:00 - Hardware: Real 8 GB card: NVIDIA RTX 3070 8 GB (RunPod container) · NVIDIA GeForce RTX 3070 · Linux 6.8.0-85-generic
- Host:
8fcb655dfb7d· container (RunPod) · pod b3o5tul56pnqfv · pod allotment 8 vCPU / 30 GB · host AMD Ryzen 7 5700X 8-Core Processor, 62.7 GB RAM · measured 2026-10-28T22:51:08+00:00 · hosts - Harness: hermes-agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d · trials 1 · sampling: Ollama default (temperature and seed not set by harness) · context window: 32768 tokens as loaded by Ollama (the unused runner field records ; see Methodology)
- GGUF sha256:
e6a7edc1a4d7d9b2de136a221a57336b76316cfe53a252aeba814496c5ae439d - Sponsor disclosure: none
Chat template vs upstream
Matches: the chat template in the GGUF we tested is identical to the provider's upstream template after whitespace normalisation.
- GGUF template sha256 (normalised):
cf9e5f0ebfacd46794d7f114c9bf96c310f9bdd8f2c96d6232b7daa683ebb88c - Upstream reference sha256 (normalised):
cf9e5f0ebfacd46794d7f114c9bf96c310f9bdd8f2c96d6232b7daa683ebb88c· source
Labels are only “matches / differs / no reference”, never safe/unsafe. Ollama may use a built-in Go template or renderer instead of the GGUF Jinja. Why templates matter: Chat-template backdoors. Not part of the score.