Meta Paper: Trained 8B Model Picks Tools Better Than Llama 4

An 8-billion-parameter open model, trained for one narrow job, picked the exact right set of software tools for 41.5% of test requests, against 30.6% for Meta’s 402-billion-parameter Llama-4-Maverick in its best prompted setup. That is according to a paper by researchers at Meta and Carnegie Mellon University posted on arXiv on 22 October 2026, titled “Training Small Language Models for Transferable Tool Selection.”

The job is tool selection: when an AI agent has a catalogue of thousands of tools, it has to choose which ones a request needs. The paper says “selecting the right tool for a complex multi-step user instruction becomes a central bottleneck”, and that “Recent work instead leverages proprietary commercial LLMs like Claude or ChatGPT to drive a retriever.” Its answer: “We show that a small, open model can be trained via reinforcement learning to perform the same functionality.”

Business Pill · AI AGENTS

A short Business Engineer explainer on AI agents that do work rather than answer questions. It teaches the general idea only and says nothing about the paper in this piece.

What the Paper Found

• The headline result. On the ToolBench G3 test split of 2,124 requests, the trained Qwen3-8B model reached a 41.5% success rate. Llama-4-Maverick, prompted four different ways, reached at most 30.6%, with a Search-R1-style workflow, according to the paper’s Table 1.
• A strict scoring rule. A request counts only if the chosen tool set matches the correct set exactly. In the paper’s words, “the agent must commit to a set, and both a missing tool and an extra tool count as a failure.”
• Smaller still works. The paper says “Even the smallest 1.7B model can be trained to reach 37.6% success rate.” The 4B model reached 40.3%.
• The training recipe. The authors fine-tuned off-the-shelf Qwen3 models first on examples written by a larger teacher model, then with reinforcement learning, which rewards the model for useful search steps and exact picks. RL lifted the 8B model from 36.2% after the first stage to 41.5%, the paper says.
• Other tool catalogues. Without retraining, the same models were tested on four other benchmarks. The 8B model scored higher than every Maverick setup on three of them, ToolBench G2, Gorilla and CRAFT, and lower on the fourth, TaskBench, per the paper’s Table 2.

What Was Posted on X

Rohan Paul summarised the paper on X. Per his post, the authors trained Qwen3 models to split a request into steps, search each step with plain keyword search and pick one tool per step. He reports the trained 8B model’s 41.5% exact-pick rate on ToolBench against 30.6% for Maverick’s best prompted setup, and says it beat Maverick on 3 of 4 new tool benchmarks without retraining. This piece draws on the paper directly.

The Results in One Table

All figures are success rates from the paper, exact match on the ToolBench G3 test split (2,124 requests).

Source: Zhou et al., “Training Small Language Models for Transferable Tool Selection”, Meta and Carnegie Mellon University, arXiv 2610.31483, 22 October 2026, Table 1.

By Our Arithmetic

By our arithmetic, the trained 8B model’s 41.5% is 10.9 percentage points above Maverick’s best 30.6%, or about 36% higher. On size, the paper describes Llama-4-Maverick as a mixture-of-experts model with 402 billion total and 17 billion active parameters, “roughly two orders of magnitude larger than our largest trained model in total parameters and twice as large in active parameters.” Dividing 402 billion by 8 billion gives about 50 times, our arithmetic.

One more detail from the paper: Maverick was also the teacher. It wrote the example plans and reasoning used to train the small models in the first stage, and it is the prompted model behind all four comparison setups.

How the Method Works

The paper describes a three-step loop it calls Plan–Retrieve–Select. The model splits a request into sub-steps, runs one plain keyword search per sub-step, and picks one tool from each set of results. The training set covers 21,361 requests over a catalogue of 1,419 APIs, the paper says. It notes that “Tool descriptions cannot all fit in context, and relevance-based retrievers only rank tools without knowing which are actually needed.”

The search engine can be swapped. The models were trained with a basic keyword search called BM25; when the authors replaced it with COLT, a retriever trained on the ToolBench catalogue, the paper reports that “Swapping in COLT improves the success rate at every scale.” The training needed “no human annotation beyond the gold tool sets,” the paper says.

Where the Small Model Fell Short

On TaskBench, which needs several tools per task from different domains, Maverick with the Search-R1-style workflow scored 80.6% against 72.8% for the trained 8B model, per Table 2. The authors write: “This shows that the Plan stage requires some domain knowledge.” On ToolBench G2, a larger catalogue of 11,474 tools, the 8B model scored 25.8%, well below its in-domain result, though above Maverick’s best of 20.8%.

The paper also notes a cost trade-off in Maverick’s setups: the ReAct workflow scored higher than the paper’s own workflow for Maverick “at the cost of up to five model calls per substep instead of one.”

Connecting the Coverage

The paper is the latest in a run of research on the software around a model, not only the model itself. Our harness-training piece covered a 21 October paper from Renmin University, IQuest Research and Beihang University, in which training an agent across 18 harness configurations instead of one raised a small open model’s average score from 29.81 to 39.04 across three agent benchmarks.

Our Mubit piece covered a start-up that records the calls a company’s agents send to model providers and trains a small model for each repeated task, charging $5 per 1,000 runs. Our Nadella piece covered Microsoft’s annual letter, which says the harness is “separate from the model,” so that “models become interchangeable.” And our Google engineering-test piece reported the best agent setup on a professional engineering benchmark scoring a mean 50.6% at $17.18 per task.

What’s new: a Meta-affiliated team reports a small model trained for one step of the agent loop, tool selection, scoring above the 402-billion-parameter model that taught it.

The Structural Read

As we read it, the paper treats tool selection as a separate job inside an agent that can be given to its own small model, rather than as one more thing the largest model does. Its Plan–Retrieve–Select loop uses one search per sub-step, and the paper says the search engine can be swapped without retraining.

As we read it, the trade-off is stated in the paper itself: Maverick’s ReAct setup scored higher than Maverick on the paper’s own workflow “at the cost of up to five model calls per substep instead of one.” The paper does not give dollar costs, so any saving is not established.

As we read it, the limit is planning in unfamiliar domains: on TaskBench the prompted Maverick scored 80.6% against 72.8% for the trained 8B model, and the authors write that “the Plan stage requires some domain knowledge.”

Three Implications

FOR TEAMS BUILDING AGENTS The paper reports that one model, trained on one catalogue, scored higher than every prompted Maverick setup on three of four other benchmarks without retraining.

FOR MODEL PROVIDERS The paper’s comparison point is prompting a large model to drive a retriever, the approach it says recent work runs with Claude or ChatGPT.

FOR RESEARCHERS The training needed “no human annotation beyond the gold tool sets,” the paper says, and RL lifted the 8B model from 36.2% to 41.5%.

The Business Engineer Lens

This story maps onto the Business Engineer framework The State of The Inference Economy.

The framework states: “Inference is a revenue engine: continuous, distributed across millions of applications and agents, incurring marginal cost with every query.”

As we read it, tool selection is one of those queries, repeated at every step of an agent’s work. The paper reports a small trained model doing that step more accurately than a 402-billion-parameter prompted model on its benchmark; it does not measure cost. The framework was published in March 2026 and does not discuss this paper.

What Is Not Established

The paper compares against Llama-4-Maverick only; it does not report results for Claude, ChatGPT or other commercial models on its benchmarks. It does not give dollar costs or response times for either approach. The results are on research benchmarks, not on a company’s live tool catalogue, and the paper has not been peer reviewed in what we read. We did not contact the authors or Meta.

The Bottom Line

A Meta and Carnegie Mellon paper reports that a Qwen3-8B model trained with supervised examples and reinforcement learning chose the exact right tools for 41.5% of ToolBench G3 requests, against 30.6% for Llama-4-Maverick, a 402-billion-parameter model prompted four ways. The trained models also scored higher on three of four other benchmarks without retraining, and lower on the fourth, TaskBench.