Quantization's effect on agent tool-failure recovery flips with eval design

A single-author study finds 8-bit vs 4-bit comparisons reverse depending on prompt, task set and parsing strictness.

A new paper on arXiv argues that benchmark claims about quantized LLM agents recovering from tool failures depend heavily on how the benchmark is built. Not just on the model. Yuhe Hu tested 8-bit and 4-bit versions of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on twenty deterministic tool-use tasks. Each ran through five different prompts, measuring how well each precision level handled temporary tool failures and recovered.

The direction of the result was not stable. On tasks both variants complete without faults, the 8-bit-minus-4-bit gap ranged from 0 to +20.2 percentage points for Llama. For Qwen it ranged from -50.0 to +35.0 points, depending on the prompt. Full-pipeline scores favoured 8-bit Llama under every prompt tested. But the same comparison for Qwen switched direction depending on which prompt was used.

The paper isolates the scoring method as its own variable. For one Llama prompt, scoring each quantization level only on the tasks it personally passes cleanly favours 4-bit by 17.5 points. Scoring both variants on the same task set gives no difference at all. Scoring the full pipeline favours 8-bit by 28.3 points instead. Switching to strict output parsing, which 8-bit Llama violates far more often than 4-bit under that prompt, turns the +28.3 result into -15.0. The Qwen comparison, by contrast, stays essentially unchanged.

Hu's conclusion is that a single prompt, a single screened task set and a single scoring policy cannot support a general claim about which quantization level is more robust for agentic use. The paper recommends comparing variants on matched tasks and reporting full-pipeline success rather than success on self-selected clean subsets. It also calls for stating the scoring policy explicitly and quantifying uncertainty across tasks rather than across injected fault sites. The work was accepted at the NeurIPS 2026 Workshop on Small Language Models for Agentic Systems.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.