Small language models for AI agents: what the early 2026 evidence suggests

Many calls an AI agent makes look short and routine, such as choosing a tool and filling in its arguments; one estimate puts them at 40% to 70% of model calls in three open-source agents, and we found no measurement across production systems. On that kind of work, the best small open models came close to GPT-5 and Claude Opus 4.5 in the tests below, though no test we found compares them with today's frontier models. At full load they cost far less than flagship APIs, but only about 2 to 3 times less than the cheapest API tiers at Azure GPU rates. They still fall short on long, multi-step tasks, so the setup we think the evidence points to, though no study here tests all three parts together, is a small model by default, trained on your own traces, with a large model kept for the hard steps.

The argument, and what has been tested since

In June 2025, researchers at NVIDIA and Georgia Tech posted a position paper, "Small Language Models are the Future of Agentic AI," and revised it as recently as September 2026 [1]. Their starting point is that agents ask language models to "perform a small number of specialized tasks repetitively and with little variation" [1]. For many of those calls, they argue, small models are "sufficiently powerful, inherently more suitable, and necessarily more economical" than large ones [1].

In their definition, a small model "can fit onto a common consumer electronic device" and answers one user's agent requests fast enough to be practical. As of 2025, they count most models under 10 billion parameters as small [1]. Parameters are the learned numbers inside a model, a rough measure of its size. We draw the line a little higher, at about 15 billion, which is far larger than a 300-million-parameter model but still small next to frontier models, and we also count sparse models that use only 3 to 4 billion parameters per token, such as Gemma 4 26B, even though they must still hold all their parameters in memory; several of the strongest results below come from models above 15 billion, and we say so where they do. We focus on open models, whose weights you can download and run yourself. Frontier models are the largest models from the major labs.

The authors argue that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy and compute than serving one with 70 to 175 billion [1]. After reviewing three existing agents (MetaGPT, Open Operator and Cradle), they estimate that about 40% to 70% of each agent's model calls could be handled by specialized small models [1]. Where broad conversation matters, they recommend mixing several models in one system [1].

Those percentages are the authors' estimates, not measurements. It is a position paper, built on earlier results and cost reasoning rather than new end-to-end agent experiments. It also lists alternative views itself, including that large generalists keep an advantage in general language understanding and that centralized large-model inference may stay cheaper, a cost question the authors concede is still open [1].

The past year added direct tests, though most are preprints that have not been peer reviewed, and each covers a narrow set of tasks. AgentFloor, posted in May 2026, ran 16 open models, from 0.27 to 32 billion parameters, plus GPT-5, through 30 agent tasks at six levels of difficulty [2]. For following instructions and for making a single tool call, some model under 5 billion parameters scored above 90% (Nemotron 3 Nano 4B and Gemma 4 E4B respectively, though neither did both); the study's best-case Mac costs at those two levels were $0.0002 and $0.00007 per passed task [2]. The best open model, Gemma 4 26B, matched GPT-5 overall within the study's preset margin, with both passing about 60% of tasks. Per passed task, it cost about 15 times less than GPT-5's list price on a Mac Studio counted at $0.50 an hour, or about 3 times less on a rented cloud GPU at $2.50 an hour [2].

A July 2026 study found that small models "typically fall short when swapped into a harness designed for a frontier LLM" [3]. A harness is the prompts, tools and control loop around a model. Two of the study's three small models were sparse (26 to 30 billion parameters in total, 3 to 4 billion used per token); the third was Ministral 3 8B [3]. When an automated optimizer adapted the harness to each small model, results improved on 16 of 21 task-and-model pairs. The best, Gemma 4 26B, averaged 80.2% against 89.7% for Gemini 3.1 Pro preview, about 89% of its score, at 4% of its cost at hosted API prices. Ministral 3 8B, the only one of the three under our 15-billion line, rose from 9.5% to just 25.0% [3].

The newest small models: strong on single steps, weaker on long chains

Table 1 lists recent small open models with the agent-related scores their makers report. Treat these as claims: each maker picks its own benchmark versions and settings, so compare within a row, not across rows.

Table 1. Recent small open models and maker-reported agent scores

BFCL is Berkeley's function-calling test. TAU2 tests agents on airline, retail and telecom tasks [7]. IFEval and IFBench test whether a model follows explicit, checkable instructions [22][23].

Qwen reports its new 9B model ahead of its own earlier 30B and 80B models on both BFCL-V4 and TAU2-Bench [4]. Google's numbers show that size still matters for multi-turn agent work: Gemma 4 goes from 24.5% on Tau2 at its smallest size to 76.9% at 31B [5][25].

Independent tests give a more careful picture. Table 2 shows how models scored in this one study.

Table 2. AgentFloor pass rates (%) by what the task requires [2]

GPT-5's 80% on the first row comes from one task, where a keyword check missed its paraphrased answers. Each level has five tasks, so small gaps are not meaningful, and the study cannot show equivalence at the middle levels [2].

The best small models hold up on single steps and short chains, then fall away when a task branches or pulls from several sources. On those harder levels, no model tested, GPT-5 included, cleared 60% without extra help [2]. On long-horizon planning, the authors say neither GPT-5 nor the best open model "reaches reliability that would support confident deployment" [2].

The Berkeley Function Calling Leaderboard, last updated in April 2026, shows the same split [9]. On non-live single-turn calls, Qwen3-8B scored 87.58% against 88.58% for Claude Opus 4.5, the top entry overall, and on live single-turn calls 80.53% against 79.79%. On multi-turn tasks it scored 41.75% against 68.38%, and on the web-search and memory tasks 12.00% and 14.62% against 84.50% and 73.76% [9]. The newest models in Table 1 were not yet listed when we checked.

The failures have causes you can work on. The harness study sorted them into five kinds, from malformed tool calls to losing track of long context and weak planning [3]. An ACL 2026 paper found that small models often "hallucinate plausible tool names" that are not in the tool list, which its authors attribute to naming habits learned in pretraining. Renaming tools to match those habits, with no retraining, improved accuracy by up to 17 percentage points on one subtask and, in an error analysis of one model, cut these errors by 80%; gains shrink as models get larger [10].

Training a small model on your own traces

Vendor demos report large gains from training a small model on the exact calls an agent makes, but we found no independent test on real production traces. Fine-tuning means further training a model on your own examples. Distillation is fine-tuning in which a small "student" model learns from a larger "teacher." LoRA is a cheap way to fine-tune that trains a small add-on instead of the whole model.

Google's FunctionGemma is a 270-million-parameter model built for function calling. On Google's "Mobile Actions" test, fine-tuning raised its accuracy from a 58% baseline to 85% [11]. Untuned, it scored 0% at every level of AgentFloor [2]. Google pitches it as a local "traffic controller" that handles routine commands on the device and sends harder requests to a larger model [11].

In June 2025, NVIDIA showed the trace-based version on its reference customer-support agent. Logs of the large model's answers became training data, tagged by workload, with no manual labeling. A Llama-3.2-1B model, fine-tuned with LoRA on those logs, reached 98% of the tool-calling accuracy of the original Llama-3.3-70B model, on one GPU instead of two [12]. This is NVIDIA's own demo, on synthetically generated queries scored by an LLM judge rather than on real user traffic.

One example shows specialization working on the skill you train, not beyond it. On the Berkeley leaderboard, Salesforce's xLAM-2-8b, trained for function calling, scored 70.00% on multi-turn tasks, above Claude Opus 4.5's 68.38%, but 6.50% on web search [9].

How you build the training data matters. Students usually copy the teacher's full transcripts, but gaps between the two "can cause compounding errors" [13]. In SCoRe, accepted at ICML 2026, the student attempts each task and the teacher corrects only its first mistake. The student trains on the corrected attempts, then practices from its last correct step by trial and reward (reinforcement learning). Across 12 math, question-answering and deep-search benchmarks, students of 7B and 8B trained this way averaged above a 72B teacher (52.3 vs. 51.1 on eight, 30.5 vs. 27.3 on four), though they trailed it on four of the 12 [13].

Fix the harness alongside the model. The harness study's optimizer moved difficulty into "tailored instructions, tools, and orchestration loops," including deterministic checks in the agent loop [3]. Each optimization run had a $20 budget, and the authors say per-run savings repaid the optimization cost after 13 agent runs on average; they ran three optimizations per task-and-model pair, so counting all three could take about three times as many [3]. NVIDIA's paper outlines a similar path: collect usage data securely, curate it, cluster tasks, pick a small model, fine-tune it, and keep refining. Its authors say this fine-tuning takes "only a few GPU-hours" [1].

Routing between small and large models

Routing sends each request to the model best placed to handle it, which matters because a small model rarely covers everything. Our companion article on decision models covers routers and classifiers in depth.

The early results were strong. FrugalGPT (2023) sends a query to a list of models in turn and stops at the first answer a scoring model judges reliable, which saves money when the first models are cheap [14]. It matched GPT-4 at up to 98% lower cost on the best of its three test sets and at 73% lower on another; on the third, where GPT-3 was the best single model, it matched GPT-3 at 59% lower [14]. RouteLLM (2024) trained routers on human preference data and cut costs "by over 2 times in certain cases" without hurting quality [15].

Newer work is more sober. LLMRouterBench (January 2026) re-tested routing methods on over 400,000 examples from 21 datasets and 33 models. Many methods performed about the same, and several recent ones, including the one commercial router tested, did not reliably beat the simple baseline of sending every query to the single best model [16]. The best methods saved up to 31.7% at that model's accuracy, while two-model routers such as FrugalGPT struggled to save money without losing accuracy [16]. That study did not test rule-based escalation, so treat rules such as escalating when an output fails a check as a sensible starting point rather than a proven one, and add a learned router only if it beats them on your own data.

For agents, a useful split is by role. AgentCARD (May 2026) assigned different models to planner and executor roles across API, self-hosted and mixed deployments [17]. Mixed teams matched the strongest single-model team at 1.4 to 12 times lower cost per task. The 12-times case was a math benchmark with Claude Opus 4.6 planning and a self-hosted 27-billion-parameter executor at full GPU load, and the paper elsewhere calls that pairing roughly 5 times cheaper [17]. The executor role cost a median of 6 times more per task than the planner, so a cheaper model saves the most there, but only if it can do the job: on FinanceBench, moving from a 4B to a 27B executor raised accuracy from 28.7% to 60.0% [17].

What it really costs: API prices versus self-hosting

API prices are quoted per million tokens. In English, a token is roughly three-quarters of a word by OpenAI's rule of thumb [21], and Anthropic's newer models produce more tokens for the same text [18]. Table 3 sets selected list prices we checked on November 1, 2026, next to measured self-hosting costs for an 8B model.

Table 3. API list prices and measured self-hosting cost, in dollars per million tokens

API rows are standard list prices; OpenAI's apply to prompts up to 272,000 tokens [19].

The self-hosted rows come from a June 2026 study that measured costs on H100 GPUs priced at $6.98 per GPU-hour, an Azure rate [20]. The cost of reading the input is folded into the output price. At full load, the 8B model cost 24 to 31 cents per million output tokens. At one request per second, the GPU sat mostly idle and the cost rose to $7.60, close to Claude Haiku 4.5 at the study's mix of 512 input to 256 output tokens ($7.00) and more than ten times Claude Haiku 5.5 or gpt-6-luna at that mix ($0.70) [18][19][20]. Across 1 to 10 requests per second, idle capacity raised costs by 2.5 to 24 times, which is why calculators that assume a busy GPU end up "over-selling" self-hosting for low-traffic workloads [20].

The self-hosted rows assume synthetic requests of 512 input and 256 output tokens, no prefix caching, and another tenant paying for the second GPU on a two-GPU node; a single-tenant operator should double them. At full load, time to first token runs to seconds, and under limits of 300 milliseconds to first token and 50 milliseconds per later token, the 8B model cost 34 to 37 cents [20]. By our arithmetic, at this token mix the API rows work out to $0.70 per million output tokens for gpt-6-luna and Claude Haiku 5.5, $7 for Claude Haiku 4.5, $14 for Claude Sonnet 5.5, $28 for Claude Opus 5.5 and $70 for gpt-6-astra.

GPU prices vary too. AgentCARD priced a different H100 variant at $1.87 an hour on a GPU marketplace [17], about a quarter of the Azure rate. Model shape matters as well. The cheapest setup in the cost study was a 30B mixture-of-experts model, which uses only 3B of its parameters for each token, at $0.209 per million output tokens [20]. The author's data suggest that parameters used per token, not total size, are a primary driver of cost at full load [20].

Per-token prices also hide differences between models. Anthropic notes that its newer models' tokenizer, which splits text into tokens, produces about 30% more tokens for the same text [18]. Reasoning modes add tokens too, and Qwen3.5 models think before answering by default [4]. So compare cost per completed task. On that basis, Gemma 4 26B on a Mac Studio, counted at $0.50 an hour, was about 15 times cheaper per passed task than GPT-5 at its list price, and 2.5 times faster, with both passing about 60% of tasks [2]. The best harness-adapted small agent ran at 4% of the frontier cost, at hosted API prices [3].

The practical reading: with low or spiky traffic, a cheap API tier such as gpt-6-luna can cost less than a lightly used GPU [19][20]. Self-hosting pays when volume is steady and high: by our arithmetic, at the Azure rate the 8B model beats gpt-6-luna only above roughly 12 requests per second at this token mix, though it beats premium tiers at far lower volume. Privacy and control can also justify it.

Running inside your own walls

A small model can run where your data already lives, in your own data center or on an edge device, so sensitive inputs need not leave your network.

Recent releases are built for this. Hugging Face describes Gemma 4 as coming in "sizes you can use everywhere including on-device" [5]. Mistral lists optimized Ministral 3 deployments for DGX Spark, RTX PCs and laptops, and Jetson devices [8]. NVIDIA aims Nemotron 3 Nano 4B at agents on edge platforms such as Jetson Thor, GeForce RTX and DGX Spark [7]. Google says FunctionGemma is small enough for phones and the Jetson Nano, and pitches on-device use as a privacy benefit [11].

Keeping data in one region costs extra even with APIs. Anthropic charges 1.1 times its standard price for US-only inference on its newer models [18], and OpenAI adds 10% for regional processing on models released since March 5, 2026 [19].

Holding your own copy of the weights also means the model does not change or disappear on someone else's schedule. API providers retire models on their own schedule; Anthropic's price list, for example, marks several earlier models, including Claude Opus 4 and Claude Haiku 3.5, as retired on its own API [18]. A model you host stays the same until you change it, which makes testing and audits simpler.

Check licenses early. Most models in Table 1 use Apache 2.0. Nemotron uses NVIDIA's own open model license [7], and some fine-tuned models are non-commercial: the Berkeley leaderboard lists xLAM-2-8b under CC BY-NC 4.0 [9].

What to do next: test before you switch

The evidence points toward moving routine calls to small models. It does not support swapping them in blindly. A practical path:

1. Log every model call your agents make, with inputs, outputs, tool calls and outcome. Tag calls by workload, as NVIDIA's flywheel does [12].

2. Rank call types by volume. Start with frequent single-step work, such as single tool calls, where small models already do well [2]. Extraction is a good candidate too: one preprint found fine-tuned models of 3 billion parameters or less beat untuned frontier models on relation extraction [24].

3. Build a frozen test set from real traces, with automatic checks: valid format, right tool, right arguments, correct end result.

4. Try two or three current small models, adapting the harness for each: fewer or simpler tools, tool names the model recognizes, tighter instructions, checks in the loop [3][10].

5. Fine-tune the best candidate with LoRA on cleaned traces. Consider training on the student's own attempts, corrected by your current large model [13].

6. Run it in shadow mode, where it sees live traffic but your current model's answers are the ones used. Then route: small by default, escalating when a check fails. Try simple rules first; the studies here don't test them, but the largest benchmark found lightweight routers may be enough [16].

7. Repeat when new models ship. The releases in Table 1 span less than a year.

Table 4. A starting point for model choice, by kind of step

Measure these before and after the switch:

• Task success rate against your current model, on the frozen set and in shadow traffic.

• Cost per successful task, at your real GPU utilization rather than peak throughput [20].

• End-to-end latency at the median and the 95th percentile (the slowest 1 in 20 requests).

• Escalation rate, the share of calls the small model hands to the large one.

• Share of outputs that parse and call a real tool with valid arguments.

• Regressions on the frozen set after every model, prompt or tool change.

How we think about this at OctaNeural

We see small models as the default for the routine calls inside an agent, with frontier models kept for the steps that need them. We help companies with this through custom deployments and customizations, including training small custom models on a client's own data and agent traces and deploying them on-prem, so weigh our view with that in mind. Who controls the models and data is set by each agreement and each base model's license. If you are working on this problem, bring it to us.

Sources

1. Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, Pavlo Molchanov. "Small Language Models are the Future of Agentic AI." arXiv:2506.02153, June 2025 (revised September 2026). https://arxiv.org/abs/2506.02153

2. Ranit Karmakar, Jayita Chatterjee. "AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?" arXiv:2605.00334, May 2026. https://arxiv.org/abs/2605.00334

3. Chenyang Yang, Xinran Zhao, Tongshuang Wu, Christian Kästner. "Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation." arXiv:2607.08938, July 2026. https://arxiv.org/abs/2607.08938

4. Qwen Team (Alibaba). "Qwen3.5-9B" model card. Hugging Face, March 2026 (checked November 1, 2026). https://huggingface.co/Qwen/Qwen3.5-9B

5. Hugging Face. "Welcome Gemma 4: Frontier multimodal intelligence on device." Hugging Face blog, April 2026. https://huggingface.co/blog/gemma4

6. IBM. "Granite-4.2-8B" model card. Hugging Face, August 2026. https://huggingface.co/ibm-granite/granite-4.2-8b

7. NVIDIA. "NVIDIA-Nemotron-3-Nano-4B-BF16" model card. Hugging Face, March 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

8. Mistral AI. "Introducing Mistral 3." Mistral AI news, December 2025. https://mistral.ai/news/mistral-3

9. Shishir G. Patil, Huanzhi Mao, et al. (UC Berkeley). "Berkeley Function Calling Leaderboard (BFCL) V4." Gorilla project, last updated April 2026 (data checked November 1, 2026). https://gorilla.cs.berkeley.edu/leaderboard.html

10. Jonggeun Lee, Woojung Song, Jongwook Han, Haesung Pyun, Yohan Jo. "Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models." ACL 2026 (arXiv:2510.07248), October 2025 (revised April 2026). https://arxiv.org/abs/2510.07248

11. Google. "FunctionGemma: Bringing bespoke function calling to the edge." Google blog, December 2025. https://blog.google/technology/developers/functiongemma/

12. Daniel Glogowski, Sylendran Arunagiri. "Build Efficient AI Agents Through Model Distillation With the NVIDIA Data Flywheel Blueprint." NVIDIA Technical Blog, June 2025. https://developer.nvidia.com/blog/build-efficient-ai-agents-through-model-distillation-with-nvidias-data-flywheel-blueprint

13. Yuanjie Lyu, Chengyu Wang, Jun Huang, Tong Xu. "Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs." ICML 2026 (arXiv:2509.14257), September 2025. https://arxiv.org/abs/2509.14257

14. Lingjiao Chen, Matei Zaharia, James Zou. "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance." arXiv:2305.05176, May 2023. https://arxiv.org/abs/2305.05176

15. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, Ion Stoica. "RouteLLM: Learning to Route LLMs with Preference Data." arXiv:2406.18665, June 2024 (revised February 2025). https://arxiv.org/abs/2406.18665

16. Hao Li, Yiqun Zhang, Zhaoyan Guo, et al. "LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing." arXiv:2601.07206, January 2026. https://arxiv.org/abs/2601.07206

17. Yinsicheng Jiang, Liang Cheng, Yeqi Huang, Yufan Zhao, Zhan Lu, Li Dong, Wenda Li, Edoardo Ponti, Luo Mai. "Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams." arXiv:2606.20629, May 2026. https://arxiv.org/abs/2606.20629

18. Anthropic. "Pricing." Claude Platform documentation (checked November 1, 2026). https://platform.claude.com/docs/en/about-claude/pricing

19. OpenAI. "Pricing." OpenAI API documentation (checked November 1, 2026). https://developers.openai.com/api/docs/pricing

20. Chitral Patil. "Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation." arXiv:2606.11690, June 2026. https://arxiv.org/abs/2606.11690

21. OpenAI. "Understanding and counting tokens." OpenAI Help Center, October 2026 (checked November 1, 2026). https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens

22. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, Le Hou. "Instruction-Following Evaluation for Large Language Models." arXiv:2311.07911, November 2023. https://arxiv.org/abs/2311.07911

23. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, Hannaneh Hajishirzi. "Generalizing Verifiable Instruction Following." arXiv:2507.02833, July 2025 (revised November 2025). https://arxiv.org/abs/2507.02833

24. Despina Christou, Grigorios Tsoumakas. "Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction." arXiv:2606.22606, June 2026. https://arxiv.org/abs/2606.22606

25. Google. "Gemma 4 model card." Google AI for Developers, last updated July 2026. https://ai.google.dev/gemma/docs/core/model_card_4

26. Google. "Gemma releases." Google AI for Developers (checked November 1, 2026). https://ai.google.dev/gemma/docs/releases

Sources checked November 1, 2026.