mistral-small3.2:24b
Family: mistral-small3.2 · 24B parameters · 1 run(s)
Fits in 24 GB VRAM · ollama 0.40.6 · Q4_K_M
Measured VRAM at 32K context (KV cache included, ollama ps on the RTX 3090): 19 GB · previous rank in this tier: #7 (results v0.1.1, before the v0.1.2 re-tiering; score unchanged). Tier rule
42.8/100 · scoring v4
95% task-bootstrap CI (main suite, 3 trials): 36.6–49.2 · pooled over 8 runs (3 main-suite trials + 5 reruns at temperature 0.7): 40.1–53.5 · ≈ statistically tied with other models in pooled tie group 4 of this tier (the leaderboard’s ≈) · rank #8 in tier, rank range under weight sensitivity #8–#8 (5th–95th percentile; min #8, max #8) · v3: 42.4 (#7) · mention 4.4% (2/45) · how
Plain-language summary
Weak as an agent today on this suite.
Good for
- picking the right tool
Weak at
- returning clean JSON
- matching the requested format
- multi-step plans
- staying safe under prompt injection
Category breakdown
| JSON | Schema | Tool | Multi-step | Injection (useful&safe) | tok/s | p50 s/task | p95 s/task |
|---|---|---|---|---|---|---|---|
| 47 | 43 | 80 | 18 | 40 | 49.9 @b7e2c7 | 11.2 | 17.0 |
Prompt injection (15 tasks × 3 trials): useful and safe 40.0% (task completed and not compromised; counts in the score) · utility under attack 40.0% (completed the user's task) · harmful-action attack success 0.0% (a real harmful tool call or canary leak). Mentions (quoting the attacker without acting): 4.4% (2/45); they do not lower the score. v3 useful and safe: 37.8%. v2.1 used injection resistance 95.6 × tool-call rate 0.8308. Definitions
- Run:
20261022-mistral-small3.2-24b-a1b6e6· 2026-10-23T00:47:17+00:00 - GPU: NVIDIA GeForce RTX 3090 (24 GB) · Linux 6.8.0-57-generic
- Host:
b7e2c79acebd· container (RunPod) · host AMD Ryzen Threadripper PRO 5995WX 64-Cores, 251.5 GB RAM · measured 2026-10-23T00:47:17+00:00 · hosts - Harness: hermes-agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d · trials 3 · sampling: Ollama default (temperature and seed not set by harness) · context window: 32768 tokens as loaded by Ollama (the unused runner field records 8192; see Methodology)
- Score per trial (v4): 46.46 / 39.63 / 41.07 (SD 3.6, range 39.63–46.46)
- GGUF sha256:
41a5b0c36a28a3a0480ce2e4007d3a21e3298be70e2b9a103960581412997dca - Sponsor disclosure: none
Chat template vs upstream
No reference: GGUF has no Jinja chat template (prompt built by Ollama's Go template), so there is nothing to hash-compare.
- GGUF template sha256 (normalised):
— - Upstream reference sha256 (normalised):
—
Labels are only “matches / differs / no reference”, never safe/unsafe. Ollama may use a built-in Go template or renderer instead of the GGUF Jinja. Why templates matter: Chat-template backdoors. Not part of the score.