On this page
Free AI API tiers are metered in tokens and requests. Most comparisons, including our own speed tests, measure one chat completion: send a prompt, time the answer. That is the wrong unit for anyone running an agent. An agent works in rounds. Each round it calls a tool, reads the result, and sends the entire transcript back to the model. A model that needs 12 rounds pays for the system prompt and the early tool results 12 times.
A reader asked us directly whether we had token-efficiency data for free models. We didn't, so we built a test that measures it.
How we tested
On October 9, 2026, we ran a real tool-calling loop against each model through its OpenAI-compatible /chat/completions endpoint, using free-tier keys only.
- The task: a small Python order-pricing repo (8 files, held in memory). The bug report: "My subtotal was exactly $100.00 but I did not get the 10% bulk discount advertised for orders of $100 or more." The cause is one character:
if amount > BULK_THRESHOLDshould be>=. - The tools:
list_files,read_fileandsearch_code. Tool calls were actually executed and their output returned to the model, as a real agent harness would do. - Pass condition: the final answer must name
pricing.py, the functionapply_bulk_discount, and a corrected line using>=. We checked this with code, not by eye. - Counting: we summed
prompt_tokensandcompletion_tokensfrom the API's ownusagefield on every round. Temperature 0,max_tokens2048, a cap of 12 rounds, 3 runs per model.
The system prompt and three tool definitions add up to only about 270 to 530 tokens, depending on the tokenizer. Real harnesses are much heavier, which makes the round count matter even more (see the projection below).
Results: tokens per agent task
| Model (provider) | Solved | Rounds | Tokens per task | Input share |
|---|---|---|---|---|
| Kimi K3 (NVIDIA NIM) | 1 of 1 completed run | 3 | 1,911 | 93% |
| gpt-oss-120b (Groq) | 2 of 3 | 4 | 2,353 – 2,609 | 83–84% |
| Gemini 3.5 Flash-Lite (Google) | 3 of 3 | 5 – 7 | 2,721 – 4,851 | 97% |
| GLM-5.3-Flash (Gonka Broker) | 3 of 3 | 3 | 2,744 – 2,748 | 88% |
| Gemini 3.5 Flash (Google) | 2 of 2 completed runs | 4 | 2,901 | 92% |
| Nemotron 3 Super 120B :free (OpenRouter) | 3 of 3 | 3 – 5 | 2,564 – 5,296 | 77–80% |
| GLM-4-Flash (Zhipu) | 0 of 3 | 7 | 3,652 | 97% |
| gpt-oss-20b (Groq) | 3 of 3 | 11 – 12 | 12,254 – 14,435 | 87–88% |
| qwen3.8-27b (Groq) | 3 of 3 | 12 | 16,399 | 97% |
Rows are sorted by the cheapest solved run. "Completed runs" excludes runs that ended on a network or quota error before the model could answer; those are explained below.
Why the same task costs 8× more on some models
The task, repo and tools were identical for every model, and the token spend still differed 8.6×. The per-round logs show where it goes.
Every round replays everything before it. qwen3.8-27b on Groq read the repo one small step at a time: 11 tool-call rounds, each producing only about 30 output tokens. But each round re-sent the growing transcript, so its input climbed from 527 tokens in round 1 to 2,269 in round 12. Total: 15,842 input tokens against 557 output tokens. The model barely "said" anything; it paid for re-reading its own history.
Parallel tool calls are the cheapest trick. GLM-5.3-Flash on Gonka asked for five tool calls in a single round, twice, and then answered. Three rounds, about 2,745 tokens, the same result on all three runs. It is not a bigger model than qwen3.8-27b. It just made fewer round trips.
Small models wander. gpt-oss-20b solved the task every time, but took 11 to 12 rounds to do it, against 4 for its 120B sibling on the same provider. Per token, the small model is no cheaper on a free tier, because it isn't the price per token that is free-tier limited. It's the count. Using the "cheaper" small model cost about 5× more of the quota here.
Reasoning models shift the bill to output. Nemotron 3 Super spent 20 to 23% of its tokens on output, the highest share in the test, because it reasons before each tool call. It still finished in 3 to 5 rounds, so its total stayed mid-table.
Failed tasks still cost tokens
Free quotas are charged per request, not per success. Three kinds of failure showed up:
- Wrong answer, full price. GLM-4-Flash found the right file and the right function on all three runs, then returned the original buggy line as the "fix". It spent 3,652 tokens each time and solved nothing. Our grader caught it; a hurried human might not.
- Provider-side tool-call errors. One gpt-oss-120b run ended with Groq returning HTTP 400: "Parsing failed. The model generated output that could not be parsed." The 784 tokens of the first two rounds were already spent.
- Rate limits mid-task. qwen3.8-27b triggered Groq's per-minute limit (HTTP 429) two to four times per task, even with 2-second pauses between rounds. Twelve fast rounds are exactly the burst pattern a per-minute limit punishes. Our script waited and retried, which is why those runs took 82 to 154 seconds. One Gemini 3.5 Flash run hit "You exceeded your current quota" on its third round after our earlier runs had used up the day's allowance.
What this means with a real harness
Our test harness is tiny. A production coding agent's system prompt plus tool definitions is usually several thousand tokens, and that block is sent again on every round. A simple estimate is: measured tokens + rounds × extra preamble.
For illustration, assume a harness with 5,000 tokens more preamble than ours. This is arithmetic from our measured round counts, not a separate test:
| Model | Rounds | Measured | With +5,000-token preamble |
|---|---|---|---|
| GLM-5.3-Flash (Gonka) | 3 | 2,745 | ≈ 17,700 |
| gpt-oss-120b (Groq) | 4 | ≈ 2,500 | ≈ 22,500 |
| gpt-oss-20b (Groq) | 11 | 12,254 | ≈ 67,300 |
| qwen3.8-27b (Groq) | 12 | 16,399 | ≈ 76,400 |
The gap between a 3-round and a 12-round model grows from about 13,700 tokens to nearly 59,000 per task. On Gonka Broker's 1M free tokens a month, that is the difference between roughly 56 such tasks and roughly 13. On Groq, it also matters for the per-minute window: our earlier test found that Groq's free tier caps a single prompt at about 7,500 tokens, so a heavy preamble plus a 12-round transcript will hit that wall well before round 12.
Which free model to use for agent work
- Token quota is your limit (Gonka, OpenRouter credits): pick a model that finishes in few rounds. GLM-5.3-Flash on Gonka was the most consistent: three rounds and the same token count on every run.
- Groq: use gpt-oss-120b, not the smaller models. It took a third of the rounds of gpt-oss-20b and qwen3.8-27b, which also keeps you under the per-minute limit. Retry on HTTP 400 tool-parse errors; they happen.
- Gemini: 3.5 Flash and Flash-Lite were both cheap per task (about 2,700 to 2,900 tokens in 4 to 5 rounds), but the daily free quota runs out fast when you test repeatedly. Budget requests, not just tokens.
- Avoid for agent loops: GLM-4-Flash. It's free and fast, but it answered confidently and wrong on every run.
- Whatever you choose: ask the model to batch independent tool calls in one round. Parallel calls were the biggest single saving we saw.
What we couldn't test
- Mistral Small (free tier): every request on our account returned HTTP 429 "Rate limit exceeded" on October 9, even after waiting and retrying. We have no token data for it.
- DeepSeek V4 Flash on NVIDIA NIM: it now returns HTTP 410 "Gone"; NVIDIA lists its end of life as September 21, 2026.
- Kimi K3 on NVIDIA: only one of three runs finished. The other two lost the connection mid-task, which may be our network rather than NVIDIA, so treat its 1,911 tokens as one data point.
One small, single-answer task can't rank models for every workload. Longer tasks will grow the gaps, not shrink them, because each extra round replays a longer transcript. The method transfers: log usage on every round of your own agent for a day, and you'll know your real cost per task.