An individual developer ran 100 BFCL v4 multi-turn tasks five times per model and scored the resulting system state. Tasks completed correctly on all five runs numbered 60 for Gemma4 26B, 57 for Sonnet 5.5, 45 for LFM2.5 24B, 42 for Qwen3.8 27B, 36 for Haiku 5.5 and 28 for gpt-oss 20B. Qwen solved 83 at least once, illustrating the gap between capability and consistency. Unconditional permission replies triggered unsolicited work; replies limiting execution to the request kept average results within four percentage points of baseline. Local models used an RTX 3090 and four-bit quantization, while Claude ran without extended thinking. These are self-reported results on a small English task sample.
Back to Tech
Repeated local-LLM tests separate average success from consistency
This personal experiment covers 100 English tasks, not overall model quality or production reliability.