"Did that prompt change make things better or worse?" is the question every LLM project eventually has to answer with numbers instead of vibes. The tooling for it consolidated in 2026 around three projects — and the free-vs-paid question is less obvious than it looks, because the code is free while the grading is metered.

What "Free" Actually Means for LLM Evals

Ragas, DeepEval, and Promptfoo are all permissively licensed libraries you install and run locally. No account, no key, no quota on the framework itself. But most useful eval metrics are LLM-as-judge: to score whether an answer was faithful to its context, the metric sends the answer and context to a model and asks. That's a real inference call.

So your eval bill is roughly test cases x metrics x judge calls. A 200-case suite with three LLM-judged metrics is ~600 judge calls per run — trivial once, meaningful when it's in CI on every pull request. Two ways to keep that at zero:

• Use a free judge. All three accept any OpenAI-compatible endpoint, so a local Ollama model, a free Gemini key, or a Groq free tier does the grading.
• Use deterministic metrics where they suffice. Ragas ships non-LLM metrics — BLEU, ROUGE, CHRF, Exact Match, String Presence, non-LLM string similarity — that cost nothing and run instantly. Promptfoo's contains, cost, latency, and javascript asserts are the same idea.

Ragas vs DeepEval vs Promptfoo

Ragas: Scoring a RAG Pipeline

Ragas exists for one question — is my retrieval actually helping? — and it's the only one of the three that splits the answer into retrieval-side and generation-side scores, so you know which half to fix. Context Recall and Context Precision grade the retriever; Faithfulness and Factual Correctness grade what the model did with what it got.

pip install ragas

from ragas import evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness

evaluator_llm = LangchainLLMWrapper(llm)   # any LangChain-compatible model

result = evaluate(
    dataset=evaluation_dataset,
    metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],
    llm=evaluator_llm,
)
print(result)

The quickstart uses OPENAI_API_KEY, but LangchainLLMWrapper takes any LangChain LLM — swap in ChatOllama and the run is free. The other reason teams reach for Ragas is test-set generation: point it at your document corpus and it synthesizes question/ground-truth pairs, which solves the cold-start problem of having no labelled data on day one.

DeepEval: Unit Tests for Your LLM App

DeepEval's design choice is to look exactly like pytest, so evals live next to your existing test suite and fail a build the same way. It carries the widest metric catalog of the three — RAG metrics, agentic metrics (task completion, tool correctness, plan adherence), multi-turn conversation metrics, multimodal metrics, plus hallucination, bias, toxicity, and summarization.

pip install -U deepeval

from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

QUESTION = "What is the refund window?"

def test_answer_relevancy():
    metric = AnswerRelevancyMetric(threshold=0.7)
    test_case = LLMTestCase(
        input=QUESTION,
        actual_output=my_chatbot(QUESTION),
        retrieval_context=["Refunds are accepted within 30 days of purchase."],
    )
    assert_test(test_case, [metric])

Run it with deepeval test run test_chatbot.py. Two features worth knowing: G-Eval lets you define a custom metric in a sentence of plain English, and DAG builds a decision tree of deterministic checks when you want reproducibility instead of a judge's opinion. The company behind it sells Confident AI, a hosted platform for datasets and production monitoring — entirely optional, and deepeval login is the only thing that touches it.

Promptfoo: A-B Testing Prompts and Models

Promptfoo comes at evaluation from the other direction. Instead of writing Python around your app, you declare prompts, providers, and test cases in YAML, and it runs the full matrix — every prompt against every model — then shows you a side-by-side grid.

npx promptfoo@latest init

prompts:
  - 'Convert the following English text to {{language}}: {{input}}'
providers:
  - openai:chat:gpt-5.4
  - ollama:chat:llama3.3
tests:
  - vars:
      language: French
      input: Hello world
    assert:
      - type: contains
        value: 'Bonjour le monde'
      - type: llm-rubric
        value: 'is a natural, idiomatic translation'
      - type: latency
        threshold: 2000

Then promptfoo eval and promptfoo view for the web report. Because it speaks 60+ providers and can call an arbitrary HTTP endpoint, it's the only one of the three that's stack-agnostic — your app can be Go, Rust, or a hosted agent and Promptfoo still tests it.

Its second mode is red teaming: promptfoo redteam init auto-generates adversarial inputs from 50+ attack plugins covering prompt injection, jailbreaks, PII leakage, SSRF, SQL injection, and excessive agency, with an owasp:llm preset that maps directly to the OWASP Top 10 for LLM applications. That's a security scanner most teams would otherwise buy.

One thing to know before you standardize on it: OpenAI announced its acquisition of Promptfoo on March 9, 2026, with the technology folding into OpenAI Frontier. OpenAI's own announcement states the project stays open source under its current license. MIT is irrevocable for code already published, so the existing tool can't be taken away — but "the framework I benchmark model vendors with is owned by a model vendor" is a governance fact worth putting on the table, not a bug.

Making the Judge Free

The switch to a $0 judge is a one-liner in each. DeepEval has a dedicated command:

deepeval set-ollama --model=deepseek-r1:1.5b --base-url="http://localhost:11434"

Or per-metric in Python:

from deepeval.models import OllamaModel
from deepeval.metrics import AnswerRelevancyMetric

model = OllamaModel(model="deepseek-r1:1.5b", base_url="http://localhost:11434", temperature=0)
answer_relevancy = AnswerRelevancyMetric(model=model)

Promptfoo takes a provider string (ollama:chat:llama3.3) and Ragas takes any wrapped LangChain model. One caveat that matters: a small local judge is a weaker judge. Its scores are noisier than a frontier model's, so use local grading for fast iteration and relative comparisons, and reserve a stronger judge for the numbers you'll actually report.

Which One Should You Use?

• Building RAG? Ragas. Nothing else separates retrieval failure from generation failure as cleanly, and its test-set generator gets you a benchmark before you have labelled data.
• Want evals to block a bad merge? DeepEval. It's pytest, so it drops into CI with no new concepts, and the agentic + multi-turn metrics cover things the other two don't.
• Choosing between prompts or models, or shipping to production users? Promptfoo. The YAML matrix is the fastest way to compare options, and the red-team mode is free security testing you'd otherwise pay for.

They're not mutually exclusive, and the common production setup uses two: Promptfoo for pre-commit prompt selection and red teaming, DeepEval or Ragas in CI for regression. Pair either with Langfuse for tracing, and you can run evals against real captured production traffic instead of test cases you invented.

Frequently Asked Questions

Are these LLM evaluation tools really free?

The frameworks are, unconditionally — Apache 2.0 for Ragas and DeepEval, MIT for Promptfoo, with commercial use permitted and no revenue cap. The only cost is the judge model's inference, which you can zero out with a local model or a free API tier.

Can I run LLM evals without any API key?

Yes. Run Ollama locally and point the framework at http://localhost:11434, or restrict yourself to deterministic metrics (BLEU, ROUGE, exact match, contains, latency) that need no model at all.

Is LLM-as-judge reliable enough to trust?

It correlates well with human judgment on well-scoped criteria and poorly on vague ones. Practical mitigations: set temperature=0, use a stronger judge than the model being judged, define narrow rubrics (DeepEval's G-Eval, Promptfoo's llm-rubric), and use deterministic asserts wherever a rule can express the check.

What's the difference between evaluation and observability?

Observability (Langfuse, LangSmith) records what happened in production — traces, tokens, latency, cost. Evaluation scores whether output was good against a test set. They compose: capture traces in production, export them as a dataset, and score that dataset with one of these three.

The Verdict

There's no paywall standing between you and rigorous LLM evaluation in 2026 — three mature, permissively licensed frameworks cover RAG scoring, CI regression, and prompt/model comparison, and each one runs entirely on your machine. Start with the one that matches how you already work: Python tests point to DeepEval, a RAG pipeline points to Ragas, and a config file plus a results grid points to Promptfoo. Wire in a local or free-tier judge, and the whole practice costs nothing but the time to write test cases.

Related Reads

• Langfuse: Free Open-Source LLM Observability
• Mem0 vs Zep vs Letta: Free AI Agent Memory Compared
• CrewAI vs AutoGPT vs LangGraph: Free Agent Frameworks
• Ollama: Run AI Models Locally for Free
• Browser Use: The Free AI Browser Agent Framework
• Free Content Moderation API: What Still Costs $0