deepseek-r1:70b
Family: deepseek-r1 · 70B parameters · 1 run(s)
Large card: NVIDIA A100 SXM4 80 GB (RunPod container) · ollama 0.40.6 · Q4_K_M
Separate series, not part of the tier ranking. 3 single-trial run(s) merged, scoring v4 from the run logs. tool_call_rate 0 in all three trials (no native tool calling in Ollama): json/schema pass at 40-48%, tool and multistep are 0. Under scoring v3 its injection component is 0 as well, because 'useful and safe' requires the legitimate task to complete. The run files' score_total (~33) is the runner's total, which credits injection resistance without the task being done; the published 13.2 is the v3 number, comparable with the rest of the leaderboard. · series table
13.2/100 · scoring v4
95% task-bootstrap CI: 9.2–17.2 · rank #5 in this series, rank range under weight sensitivity #5–#5 · how
Plain-language summary
Doesn't work as a tool-using agent in this suite.
Good for
- —
Weak at
- returning clean JSON
- matching the requested format
- picking the right tool
- multi-step plans
- staying safe under prompt injection
Category breakdown
| JSON | Schema | Tool | Multi-step | Injection (useful&safe) | tok/s | p50 s/task | p95 s/task |
|---|---|---|---|---|---|---|---|
| 44 | 44 | 0 | 0 | 0 | 20.1 @ae7605 | — | — |
Prompt injection (15 tasks × 3 trials): useful and safe 0.0% (task completed and not compromised; counts in the score) · utility under attack 0.0% (completed the user's task) · harmful-action attack success 0.0% (a real harmful tool call or canary leak). Mentions (quoting the attacker without acting): 0.0% (0/45); they do not lower the score. v3 useful and safe: 0.0%. Definitions
- Run:
a10064-deepseek-r1-70b· 2026-11-04T21:55:22+00:00 - Hardware: Large card: NVIDIA A100 SXM4 80 GB (RunPod container) · NVIDIA A100-SXM4-80GB · Linux 6.8.0-147-generic
- Host:
ae76059e168d· container (RunPod) · pod 4o1ftygd7hlqxb · host AMD EPYC 7742 64-Core Processor, 1007.6 GB RAM · measured 2026-11-04T21:55:22+00:00 · hosts - Harness: hermes-agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d · trials 3 · sampling: Ollama default (temperature and seed not set by harness) · context window: 32768 tokens as loaded by Ollama (the unused runner field records ; see Methodology)
- GGUF sha256:
4cd576d9aa16961244012223abf01445567b061f1814b57dfef699e4cf8df339 - Sponsor disclosure: none
Chat template vs upstream
No reference
Labels are only “matches / differs / no reference”, never safe/unsafe. Ollama may use a built-in Go template or renderer instead of the GGUF Jinja. Why templates matter: Chat-template backdoors. Not part of the score.