Self-hosted evals for AI agents

What self-hosted evals for AI agents are

Self-hosted evals for AI agents are four things you keep in your own repository: a file of saved cases, a script that runs the agent over them, a set of checks that grade each answer, and a table of results you can query. Nothing in that list needs a vendor. The whole loop is a few hundred lines of Python and one SQLite file.

The agent worked in the demo because you picked the five inputs yourself. It broke in week two because a prompt line changed, or a model changed, or a tool description changed, and no measurement covered any of it. An eval loop turns "it feels worse now" into "pass rate went from 58 of 60 to 51 of 60 on commit 4f1c9ab".

The loop has four steps, and this guide is one section per step: collect real traces, promote the interesting ones into cases, grade every case on every change, and store the pass rate next to the commit that produced it. The same loop works whatever you run the agent on, and the self-hosted agent frameworks worth running differ mostly in how much of the trace they hand you for free.

Why the agent breaks in week two

An agent is a prompt, a model, a set of tool definitions, and whatever context gets retrieved at run time. All four can move without your application code changing, so a normal code review sees nothing to object to.

The most common cause is a prompt edit. You add one sentence to stop a rude reply. That sentence changes behaviour on inputs nobody retested, and the traces show it plainly: last week's trace for the same question contains a create_refund tool call, this week's contains none, and the reply is a polite apology instead. Nothing raised an error, so no alert fired.

The second cause is the model. Record the exact model string you sent with every run, claude-haiku-4-5-20251001 rather than a shorthand you keep in your head, because a pass rate that drops on the day you switched models is only diagnosable when the model is in the row.

The third is tools. Rewording a tool description changes when the model decides to call it. If your tools arrive over MCP servers running on a VPS, the schema lives in another process, so it can change under you with no diff in your repository at all. The fourth is retrieval: the same question hits an index that was rebuilt overnight, and the answer follows the new document.

Build the golden set from traces you already collect

Do not invent eval cases. Take them from traffic. If you already run self-hosted Langfuse tracing for your agent, every request is stored with its input, its tool calls and its output, which is exactly the raw material a case needs.

Export a window of root observations over the public API. It uses basic authentication, with your public key as the username and your secret key as the password.

export LF_HOST="https://langfuse.example.com"
curl -sS -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
  "$LF_HOST/api/public/v2/observations?limit=50&isRootObservation=true&fromStartTime=2026-07-01T00:00:00Z" \
  | jq '.data[0]'

Read one record before you write any parsing. The rows come back under data, but the field names holding the question and the reply depend on how your agent instruments its spans, so map what you actually see rather than what you expected. Then write the cases by hand, one JSON object per line, in evals/cases.jsonl:

{"id": "refund-double-charge", "tags": ["smoke"], "input": "I was charged twice for order 41822.", "must_call": ["lookup_order", "create_refund"], "must_not_include": ["I cannot help"], "rubric": "The reply confirms exactly one refund for order 41822 and states the amount."}

Five rules keep the set worth running:

• 40 to 80 cases is enough to start. Below 20, one flaky case moves the pass rate by 5 points, and a number that jumps for no reason gets ignored.
• Every production bug you fix becomes a case on the day you fix it. That habit is what makes the set grow in the right direction.
• One behaviour per case. A case that checks the refund amount and the tone together tells you nothing when it fails.
• The id never changes, because the id is how today's run compares with last month's.
• Redact before you commit. This file goes into git, so strip customer names and any order numbers you do not own.

Grade with deterministic checks first, because they are free

Anything with a right answer gets a plain assertion. No model call, no cost, no ambiguity. Deterministic checks catch the structural regressions, and those are the ones that break the systems around your agent: the JSON does not parse, the tool was never called, the forbidden phrase came back, the answer cites no source.

One function knows about your agent. Everything else in the harness is generic.

import json, os, urllib.request

def run_agent(case):
    req = urllib.request.Request(
        os.environ["AGENT_URL"],
        data=json.dumps({"input": case["input"]}).encode(),
        headers={"content-type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=120) as resp:
        return json.load(resp)


def deterministic(case, result):
    text = result.get("output", "")
    called = [c["name"] for c in result.get("tool_calls", [])]
    failures = []
    for tool in case.get("must_call", []):
        if tool not in called:
            failures.append(f"tool not called: {tool}")
    for phrase in case.get("must_not_include", []):
        if phrase.lower() in text.lower():
            failures.append(f"forbidden phrase: {phrase}")
    if len(called) > case.get("max_tool_calls", 12):
        failures.append(f"too many tool calls: {len(called)}")
    return failures

Keep the tool budget in that list. An agent that solves a case in 3 calls today and 11 tomorrow has regressed even when the final answer is right, because you pay for every call it makes.

LLM as judge, and the four ways it goes wrong

Whatever survives the assertions needs a grader that reads. An LLM judge is a second model call: it receives the question, the agent's answer and one criterion, then returns a verdict. It is the only practical way to grade "does the reply answer what the user asked".

Four rules make a judge usable:

• Binary verdict, never a 1 to 10 score. A scale returns 7 and 8 for nearly everything, so the number never moves and you learn nothing from it.
• One criterion per call. Ask about the refund amount, or about the tone, not both at once.
• Give the judge the expected answer whenever the case has one. Grading against a reference is a far easier task than grading in the abstract.
• Force the output shape and parse it strictly.

from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment


def judge_prompt(case, output):
    return (
        "You grade one answer against one criterion.\n"
        "Reply with JSON only, in this exact shape:\n"
        '{"verdict": "pass", "confidence": "high", "reason": "one short sentence"}\n'
        f"Criterion: {case['rubric']}\n"
        f"Question: {case['input']}\n"
        f"Answer: {output}\n"
        "Length is not a criterion. Judge only the criterion above."
    )


def judge(case, output, model):
    msg = client.messages.create(
        model=model,
        max_tokens=200,
        messages=[{"role": "user", "content": judge_prompt(case, output)}],
    )
    return json.loads(msg.content[0].text)

Now the failure modes. Each one has a test you can run this afternoon, and running them matters, because an unchecked judge produces numbers that look precise and mean nothing.

Length bias. Longer answers pass more often. Test it: take ten answers the judge failed, pad each with two paragraphs of confident filler that add no new fact, and judge them again. Any verdict that flips to pass is length bias, and the rubric is the thing to fix.

Self-preference. A judge often grades output from its own model family more kindly than output from another. Test it: grade the same 30 answers with judges from two different families and compare the verdicts case by case. Where they disagree, you read the case yourself.

Position bias. If you use the judge to compare two answers, A and B, swap the order and run it again. A verdict that flips on the swap means pairwise comparison is not safe for that rubric yet.

Rubric drift. Vague criteria produce agreeable judges. "Is the answer helpful" passes almost anything. "Does the answer state the refund amount in dollars" passes only what you meant. Rewrite every criterion until it names the fact being checked.

One guard covers all four. Keep 30 cases you labelled by hand, and score the judge against your labels every time you change the judge model or the judge prompt. If it disagrees with you on more than one case in ten, fix the rubric before you trust any pass rate it produces. The judge is code, so it gets versioned and reviewed like code.

Grade cheap, escalate to a frontier model

Judging every case with the most expensive model on every commit is how an eval bill outgrows the agent it is testing. Order the graders by price, and stop as soon as the answer is clear.

[
  {
    "label": "Haiku 4.5, Batch API",
    "usd_per_1000_judge_calls": "0.90"
  },
  {
    "label": "Haiku 4.5",
    "usd_per_1000_judge_calls": "1.80"
  },
  {
    "label": "Sonnet 5",
    "usd_per_1000_judge_calls": "3.60"
  },
  {
    "label": "Opus 5",
    "usd_per_1000_judge_calls": "9.00"
  }
]

Those figures assume about 1,200 input tokens and 120 output tokens per judge call, which is a realistic size for one question, one answer and one criterion. Judging 1,000 cases costs 1.80 US dollars on Claude Haiku 4.5 and 9.00 on Claude Opus 5. The gap looks trivial until you multiply it out. A 60 case set, judged on every commit, at 40 commits a week, is 2,400 judge calls a week before anyone has run the nightly job.

Two discounts apply cleanly to eval work, and they stack. Eval runs are not interactive, so the Batch API halves both input and output prices in exchange for asynchronous delivery, which is the first row of the chart. The rubric and the instructions are byte for byte identical in every call, so prompt caching fits: a cache read costs a tenth of the base input price, and a five minute cache write costs 1.25 times base input, so the cache pays for itself after a single hit. These are Anthropic list prices as of August 2026, and Sonnet 5 is on introductory pricing until 31 August 2026, so the third bar rises after that date.

The ladder, in order:

• Deterministic checks on every case. No API cost at all.
• A small model judge on the cases that got past those checks.
• A frontier judge only where the small judge says fail, or says pass with low confidence.
• Human review on a small sample, once a week.

CHEAP = "claude-haiku-4-5-20251001"
STRICT = "claude-opus-5"


def grade(case, result):
    hard = deterministic(case, result)
    if hard:
        return False, "deterministic", "; ".join(hard)
    first = judge(case, result["output"], CHEAP)
    if first["verdict"] == "pass" and first["confidence"] == "high":
        return True, CHEAP, first["reason"]
    second = judge(case, result["output"], STRICT)
    return second["verdict"] == "pass", STRICT, second["reason"]

This trades some grading accuracy for cost, so measure the trade instead of assuming it. Once a month, grade the whole set with the strict judge as well and compare the two columns. If they disagree on more than a handful of cases, your rubric is too loose for the small model, and the rubric is what you fix. Controlling what the agent itself spends is a separate job, covered in cost control for an AI agent on a VPS.

Track pass rate over time in a system you own

A pass rate you cannot join to a commit is a feeling. Store one row per case per run, with the commit and the model inside the row.

CREATE TABLE IF NOT EXISTS results (
  run_id      TEXT NOT NULL,
  ran_at      TEXT NOT NULL,
  git_sha     TEXT NOT NULL,
  agent_model TEXT NOT NULL,
  case_id     TEXT NOT NULL,
  passed      INTEGER NOT NULL,
  graded_by   TEXT NOT NULL,
  reason      TEXT
);

SELECT run_id, git_sha, agent_model,
       count(*) AS cases,
       round(100.0 * sum(passed) / count(*), 1) AS pass_pct
FROM results
GROUP BY run_id
ORDER BY ran_at DESC
LIMIT 10;

Load the schema with sqlite3 evals/results.db < evals/schema.sql, then read the trend with sqlite3 -box evals/results.db < evals/passrate.sql. A year of daily runs over 60 cases is about 22,000 rows, so the store never becomes a project of its own. Running SQLite in production on a VPS covers the settings that start to matter if this file gets shared between machines.

The runner prints the same information for a person:

run 2026-08-05T09:14:22Z  sha 4f1c9ab  model claude-sonnet-5  58/60 pass (96.7%)
FAIL refund-double-charge  deterministic: tool not called: create_refund
FAIL pto-policy-question   judge(opus): reply gives no dollar amount

Run the suite on the changes that can break an agent, which means prompt edits, model changes and tool changes rather than every commit anywhere in the repository. A pre-push hook covers the fast subset:

cat > .git/hooks/pre-push <<'EOF'
#!/bin/sh
python3 evals/run.py --set smoke || exit 1
EOF
chmod +x .git/hooks/pre-push

Full runs are slower and belong on a schedule. A nightly systemd service and timer on the VPS runs the whole set against the deployed prompt, which is what catches the changes that arrive from outside your repository, such as a hosted tool whose behaviour moved.

Human review, sampled rather than exhaustive

The judge is calibrated against human labels, so somebody has to produce them. Read a sample every week: every case the judge failed, plus ten passes picked at random. The random passes are the important half, because a judge that has quietly started passing bad answers looks perfect on any dashboard built from its own verdicts.

Fifteen cases at three minutes each is 45 minutes a week, and it returns corrections to the rubric where you and the judge disagree, plus new cases for failure types nobody had imagined. Write the human verdict into the same table with graded_by set to human, so agreement between judge and human becomes a query instead of a memory.

What breaks in the eval harness itself

anthropic.RateLimitError on the first full run. Sixty cases fanned out at once exceeds the request or token limit for your tier. Cap concurrency at four workers, and move the nightly run to the Batch API.

json.JSONDecodeError: Expecting value: line 1 column 1 (char 0) from the judge. The model replied in prose, or wrapped its JSON in a code fence. Retry once, then record the case as an error. Never let a parse failure count as a pass, because a suite that converts errors into passes climbs toward 100% while the agent gets worse.

Flaky cases. The same input passes on one run and fails on the next, because the agent samples its output. Run the flaky case three times and record the fraction rather than deleting the case. A case that passes two runs in three is a real robustness bug, and a customer will find it.

Golden set rot. Someone edits an expected answer to turn the suite green. Review diffs to evals/cases.jsonl as carefully as diffs to the agent, because that file is your written definition of correct.

A suite that never fails. A pass rate parked at 100% for a month means the set has stopped tracking the product. Pull ten recent traces, find the ones the agent handled badly, and add them.

FAQ

How many cases does an AI agent eval set need?

Start with 40 to 80 and grow the set from real failures. Below about 20 cases, one flaky result swings the pass rate by 5 points, so the number stops carrying information. Past a few hundred, every run costs real money and real time while the marginal case adds little coverage. The measure that matters is not the count: it is the share of your known production failure types that appear in the set at least once.

Can I trust an LLM judge to grade my agent?

Only after you have measured it against your own labels. Keep 30 cases you graded by hand, and score the judge against them whenever you change the judge model or the judge prompt. Judges show length bias, where padded answers pass more often, and self-preference, where output from their own model family is graded more kindly. Both are testable: pad a failed answer and re-judge it, or grade the same answers with a judge from another family. If the judge disagrees with your labels on more than one case in ten, the rubric is too vague to use.

Which model should grade the evals?

Grade cheap and escalate. Deterministic assertions cost nothing, so they run first on every case. A small model handles the clear passes. Only fails and low-confidence verdicts go to a frontier model. At list prices in August 2026, judging 1,000 cases costs about 1.80 US dollars with Claude Haiku 4.5 and about 9.00 with Claude Opus 5, and because eval runs are asynchronous, the Batch API halves either figure.

Do evals replace production monitoring?

No, because they answer different questions. An eval suite tells you whether a change you are about to ship makes a fixed set of cases better or worse. Tracing and monitoring tell you what real users are hitting right now, including inputs no case covers. They feed each other: traces supply the new cases, and the eval suite decides whether your fix actually worked.