Every "fastest free AI API" list — including our own — measures tokens per second. That number tells you how fast a model talks. It tells you nothing about whether it can do anything.

So we changed the test. Instead of asking models to generate text, we plugged them into DeepSeek Harness — an agent runtime that can read files, run shell commands and act on the results — and gave them real work. Then we checked the filesystem afterwards to see whether the work was actually done.

The setup

Every run used the same harness, the same disposable Linux container, and the same free backend (NVIDIA's endpoint, which hosts models from DeepSeek, Moonshot, Google and NVIDIA itself under one free key). Only the model changed. Success wasn't judged by what the model said — it was judged by inspecting the files it left behind.

Three jobs, each mirroring something people genuinely delegate to agents:

• Fix a broken script. A Python file with two planted bugs: a misspelled variable (tota vs total) and a divide-by-zero when a filtered list comes back empty. The second bug is invisible unless you actually run the script — so this tests whether the model executes code and reads the traceback, or just skims and guesses.
• Tidy a folder. Eight mixed files to be moved into subfolders by type. Simple, boring, and exactly the kind of chore people hand to an agent.
• Summarise data. Three monthly CSVs; total the units per region across all of them and write the answer to a file.

Results

Dashes are jobs we haven't run for that model yet, not failures.

What surprised us

The fastest debugger was the worst at chores

Nemotron Super 120B fixed the broken script in 43 seconds — the best result of the whole test. Given the folder to tidy, it finished in 20 seconds and reported the job done. The folder was untouched: zero files moved.

That's the failure mode worth fearing. A model that errors out is annoying but honest. A model that confidently reports success while having done nothing will quietly corrupt any automation you build on it, and you won't notice until something downstream breaks.

All three "passes" were six times apart

Kimi K3, Nemotron and DeepSeek V4 all fixed both bugs correctly and explained them properly. But they took 43 s, 56 s and 251 s. Same job, same result, six-fold spread. If you're running an agent loop dozens of times a day, that difference is the whole experience.

"Flash" was the slowest one that worked

DeepSeek V4 Flash took 251 seconds — nearly six times Nemotron's 43. The name describes token generation, not how quickly the model converges on an answer. In an agent loop, wall-clock time is set by how many detours it takes, not how fast it types.

Token throughput did not predict agent speed at all

This is the finding that should change how you read benchmarks. In our plain streaming benchmark, Nemotron Super 120B looked bad — it returned empty responses and ranked among the slowest models we measured. In the agent harness it was the fastest of all four.

The two measure different things. Tokens per second measures typing speed. Agent completion time measures decision quality: how many tool calls it makes, how many wrong turns it takes, whether it re-reads a file it already understood. A verbose fast model can lose to a terse slow one.

So which free model should you point your agent at?

There isn't one answer, and that's the practical takeaway:

• Code debugging, error diagnosis → Nemotron Super 120B. Fastest and accurate on exactly this, but verify it did the work when the job involves changing files.
• Data wrangling, file operations, anything multi-step and boring → Kimi K3. The most consistent performer here, and the only model that passed every job we gave it.
• Don't reach for Gemma 4 31B in an agent loop — it never returned at all.

The broader lesson: test the model on your job, not on a leaderboard. Ranking tables measure generation speed, which turns out to be close to irrelevant for agent work. A twenty-minute experiment with your real task will tell you more than any benchmark.

How to reproduce this

The harness and workflow are open: free-ai-api-benchmark. Runs happen in throwaway containers — worth doing, since an agent that executes shell commands should not be pointed at a machine holding your SSH keys. DeepSeek Harness's own safety notice says the same.

One caveat we hit: dsh accepts any OpenAI-compatible endpoint on paper, but not in practice. NVIDIA's endpoint worked; Groq failed silently — an empty response after one second, no error, no log. If you're wiring a free model into it, verify the pairing before assuming the model is at fault.

Measured 2026-08-31. Free tiers and model catalogues change weekly in 2026 — we re-run rather than quote vendor claims.