Tool mode study

Some models have no native tool-calling in Ollama, so the main suite can't measure them fairly. This side study asks a different question: how well do they pick tools when the tools are described in the prompt, or when a grammar forces valid tool-call JSON?

Headline: a model Ollama can't use for tools works as an agent with grammar-forced output.

phi4:14b has no native tool support in Ollama, so it scores 0 on tool use in the main suite. Loaded in llama.cpp with every turn constrained by a JSON grammar, the same Q4_K_M file scored 100 tool selection, 96 multi-step, 76.67 error recovery and 100 injection resistance (3 trials). This uses a separate runner and agent loop (llama.cpp, not Ollama with Hermes Agent), so it is not comparable with the main ranking. Details

Not part of the main ranking.

Tool mode is its own dimension, like the VRAM tiers. There is no combined score and no rank here, and these numbers are not comparable with the leaderboard: level 1 is single-turn only, and level 3 uses llama.cpp with a reference agent loop instead of Ollama with Hermes Agent.

Modes

native: tools passed through Ollama's tools parameter. In the level 2 comparison, native mode runs the same 60 single-turn items as text-only. In the model cards for levels 1 and 3, the native row is the model's main-suite result (Ollama with Hermes Agent), shown for reference.
text-only (level 1): Ollama, no tools parameter. The 10 mock tools are described in the system prompt and the model must reply with {"tool", "args"}. 60 items: 25 tool selection, first step of 25 multi-step tasks, 10 error-recovery tasks. Only the first call of each item is scored (single turn), and both models tested score 100 tool correct / 100 args valid / 97 arg match, so this level is likely saturated and does not separate models.
grammar (level 3): llama.cpp llama-server loading the exact same Q4_K_M GGUF file. Every agent turn is constrained by a JSON schema (one tool call or a final answer), compiled to a grammar. Full 100-task agent loop with the same mock tools and scoring.

Native vs emulated tool calling (level 2)

Eight models from the main leaderboard, each run twice on the same 60 items (25 tool selection, first step of 25 multi-step tasks, 10 error-recovery tasks), same Ollama 0.40.6, same Q4_K_M GGUF, same RTX 3090, 3 trials, temp 0, seed 42, ctx 8192. Native: tools passed through Ollama's tools parameter and the first returned tool call is scored. Emulated: no tools parameter; the tools are described in the system prompt and the model replies with {"tool", "args"} (the text-only mode). Single-turn only, so this is not the main suite's multi-step score and no number here feeds the ranking.

Model	Mode	Tool correct	Args valid	Arg match	Tool items	Multi-step 1st call	Recovery	Well-formed call	p50 s
gemma4:e4b Fits in 8 GB VRAM	native	95	95	93	100	96	80	95	2.4
gemma4:e4b Fits in 8 GB VRAM	emulated	100	100	97	100	100	100	100	0.5
qwen3:8b Fits in 8 GB VRAM	native	93	92	90	100	88	90	93	2.5
qwen3:8b Fits in 8 GB VRAM	emulated	92	91	90	100	88	80	92	1.9

All columns are % of item-trials. "Well-formed call": native = the reply contained a tool call; emulated = the reply contained a parseable JSON object. Runs: 20261023-L2-gemma4-e4b-65f554 20261023-L1-gemma4-e4b-05718c 20261023-L2-qwen3-8b-99c854 20261023-L1-qwen3-8b-c54073

Level 2 in progress: 6 of 8 models not yet complete.

Levels 1 and 3: models without native tool support

gemma3:4b

Tool mode	Engine	Tool correct	Args valid	Arg match	Multi-step	Recovery	Injection resist	tok/s	n
native No native tool support	ollama 0.40.6	0	—	—	0	—	n/a	—	100
text-only	ollama 0.40.6	100	100	97	—¹	—	—	69.0	60
grammar	llama.cpp 643b0cdac	no result*						—	—

* grammar: llama.cpp rejected every grammar-constrained request for this model's chat template (HTTP 400, "Failed to initialize samplers"), so no level-3 result was measured. This is likely a llama.cpp issue with this model's chat template combined with grammar-constrained sampling rather than a model failure (the text-only row shows the model can pick tools). Starting llama-server with --jinja may avoid it; that workaround is untested.

Runs: 20261023-L1-gemma3-4b-42e502 · 20261023-L3-gemma3-4b-259a7a · RTX 3090 · 3 trials, temp 0, seed 42, ctx 8192.

phi4:14b

Tool mode	Engine	Tool correct	Args valid	Arg match	Multi-step	Recovery	Injection resist	tok/s	n
native No native tool support	ollama 0.40.6	0	—	—	0	—	n/a	—	100
text-only	ollama 0.40.6	100	100	97	—¹	—	—	57.1	60
grammar	llama.cpp 643b0cdac	100	—²	—²	96	77	100	71.6	100

Runs: 20261023-L1-phi4-14b-a66198 · 20261023-L3-phi4-14b-a9438c · RTX 3090 · 3 trials, temp 0, seed 42, ctx 8192.

¹ Text-only is single-turn: it only checks the first tool call of each multi-step task (counted in "Tool correct"), so there is no full multi-step score. Text-only replies were also never strict JSON (both models wrapped answers in code fences despite the instruction), but every reply parsed leniently.

² Not measured: the level 3 (grammar) harness scores tool selection, multi-step success, recovery and injection per task, but does not record argument validity or argument match.