How to Run Self-Hosted Evals for AI Agents

Wetín self-hosted evals for AI agents be

Self-hosted evals for AI agents na four things wey you keep for your own repository: file of saved cases, script wey run the agent on top dem, set of checks wey grade each answer, and table of results wey you fit query. Nothing for that list need vendor. The whole loop na few hundred lines of Python and one SQLite file.

The agent work for the demo because na you pick the five inputs yourself. E break for week two because prompt line change, or model change, or tool description change, and no measurement cover any of dem. Eval loop turn "e feel worse now" to "pass rate drop from 58 of 60 to 51 of 60 for commit 4f1c9ab".

The loop get four steps, and this guide get one section for each step: collect real traces, promote the interesting ones to cases, grade every case for every change, and store the pass rate beside the commit wey produce am. The same loop work no matter wetin you run the agent on, and the self-hosted agent frameworks wey worth running mostly differ for how much of the trace dem hand give you free.

Why agent dey break for week two

Agent na prompt, model, set of tool definitions, and any context wey system retrieve at run time. All four fit change without your application code changing, so normal code review no go see anything to object to.

The most common cause na prompt edit. You add one sentence to stop rude reply. That sentence changes behaviour for inputs wey nobody test again, and the traces show am clearly: last week trace for the same question get a create_refund tool call, this week own no get any, and the reply na polite apology instead. Nothing raise error, so no alert fire.

The second cause na model. Record the exact model string wey you send with every run, claude-haiku-4-5-20251001 instead of shorthand wey you just remember, because you fit only diagnose pass rate wey drop on the day you switch models when model dey inside the row.

The third one na tools. Rewording tool description changes when model decide to call am. If your tools dey come through MCP servers wey dey run for VPS, the schema dey for another process, so e fit change under you without any diff for your repository at all. The fourth one na retrieval: the same question hit index wey dem rebuild overnight, and the answer follow the new document.

Build the golden set from traces wey you already dey collect

No invent eval cases. Take dem from traffic. If you already dey run self-hosted Langfuse tracing for your agent, every request dey stored with its input, tool calls, and output. Na exactly this raw material case need.

Export one window of root observations through the public API. E dey use basic authentication, with your public key as username and your secret key as password.

export LF_HOST="https://langfuse.example.com"
curl -sS -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
  "$LF_HOST/api/public/v2/observations?limit=50&isRootObservation=true&fromStartTime=2026-07-01T00:00:00Z" \
  | jq '.data[0]'

Read one record before you write any parsing. The rows dey come under data, but the field names wey hold the question and reply depend on how your agent instrument its spans. So map wetin you actually see instead of wetin you expect. Then write the cases by hand, one JSON object for each line, inside evals/cases.jsonl:

{"id": "refund-double-charge", "tags": ["smoke"], "input": "I was charged twice for order 41822.", "must_call": ["lookup_order", "create_refund"], "must_not_include": ["I cannot help"], "rubric": "The reply confirms exactly one refund for order 41822 and states the amount."}

Five rules go make the set worth running:

• 40 to 80 cases dey enough to start. Below 20, one flaky case fit move the pass rate by 5 points, and people go ignore number wey dey jump without reason.
• Every production bug wey you fix become a case that same day. Na this habit dey make the set grow for the correct direction.
• One behaviour for each case. Case wey check refund amount and tone together no go tell you anything when e fail.
• The id no dey ever change, because na the id you use compare today run with last month own.
• Redact before you commit. This file dey enter git, so remove customer names and any order numbers wey no belong to you.

Faw grade with deterministic checks first, because dem free

Anything wey get correct answer, use plain assertion check. No model call, no cost, no ambiguity. Deterministic checks dey catch structural regressions, and na dem dey break systems around your agent: JSON no parse, tool never get called, forbidden phrase come back, or answer no cite any source.

Na one function sabi about your agent. Everything else for the harness generic.

import json, os, urllib.request

def run_agent(case):
    req = urllib.request.Request(
        os.environ["AGENT_URL"],
        data=json.dumps({"input": case["input"]}).encode(),
        headers={"content-type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=120) as resp:
        return json.load(resp)


def deterministic(case, result):
    text = result.get("output", "")
    called = [c["name"] for c in result.get("tool_calls", [])]
    failures = []
    for tool in case.get("must_call", []):
        if tool not in called:
            failures.append(f"tool not called: {tool}")
    for phrase in case.get("must_not_include", []):
        if phrase.lower() in text.lower():
            failures.append(f"forbidden phrase: {phrase}")
    if len(called) > case.get("max_tool_calls", 12):
        failures.append(f"too many tool calls: {len(called)}")
    return failures

Keep tool budget for that list. Agent wey solve case with 3 calls today and 11 tomorrow don regress, even when final answer correct, because you dey pay for every call e make.

LLM as judge, and the four ways e fit go wrong

Anything wey pass the assertions still need grader wey fit read. An LLM judge na second model call: e receive the question, agent answer, and one criterion, then e return verdict. Na the only practical way to grade “whether the reply answer wetin user ask”.

Four rules make judge useful:

• Binary verdict, never 1 to 10 score. Scale dey return 7 and 8 for almost everything, so the number no dey change and you learn nothing from am.
• One criterion per call. Ask about refund amount, or about tone, no be both together.
• Give judge the expected answer whenever the case get one. Grading against reference answer easier pass grading without specific reference.
• Force the output shape and parse am strictly.

from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment


def judge_prompt(case, output):
    return (
        "You grade one answer against one criterion.\n"
        "Reply with JSON only, in this exact shape:\n"
        '{"verdict": "pass", "confidence": "high", "reason": "one short sentence"}\n'
        f"Criterion: {case['rubric']}\n"
        f"Question: {case['input']}\n"
        f"Answer: {output}\n"
        "Length is not a criterion. Judge only the criterion above."
    )


def judge(case, output, model):
    msg = client.messages.create(
        model=model,
        max_tokens=200,
        messages=[{"role": "user", "content": judge_prompt(case, output)}],
    )
    return json.loads(msg.content[0].text)

Now make we look the failure modes. Each one get test wey you fit run this afternoon, and e matter to run dem, because unchecked judge dey produce numbers wey look precise but mean nothing.

Length bias. Longer answers dey pass more often. Test am: take ten answers wey judge fail, add two paragraphs of confident filler to each one without adding any new fact, then judge dem again. If any verdict change to pass, na length bias, and na rubric you need fix.

Self-preference. Judge often dey grade output from e own model family more kindly than output from another one. Test am: use judges from two different model families to grade the same 30 answers, then compare the verdicts case by case. For cases wey dem disagree, read the case yourself.

Position bias. If you use judge to compare two answers, A and B, swap their order and run am again. If verdict change after the swap, pairwise comparison no safe yet for that rubric.

Rubric drift. Vague criteria dey produce judges wey agree too easily. “Whether the answer helpful” go pass almost anything. “Whether the answer state the refund amount in dollars” go pass only the thing wey you mean. Rewrite every criterion until e name the fact wey you dey check.

One guard fit cover all four. Keep 30 cases wey you label by hand, and score judge against your labels every time you change judge model or judge prompt. If e disagree with you for more than one case out of ten, fix the rubric before you trust any pass rate wey e produce. Judge na code, so version and review am like code.

Use cheap model first, then move go frontier model

If you use the most expensive model to judge every case for every commit, eval bill fit grow pass the agent wey e dey test. Arrange the graders by price, then stop once the answer don clear.

[
  {
    "label": "Haiku 4.5, Batch API",
    "usd_per_1000_judge_calls": "0.90"
  },
  {
    "label": "Haiku 4.5",
    "usd_per_1000_judge_calls": "1.80"
  },
  {
    "label": "Sonnet 5",
    "usd_per_1000_judge_calls": "3.60"
  },
  {
    "label": "Opus 5",
    "usd_per_1000_judge_calls": "9.00"
  }
]

Those figures assume say each judge call get about 1,200 input tokens and 120 output tokens. This na realistic size for one question, one answer, and one criterion. To judge 1,000 cases, e cost 1.80 US dollars for Claude Haiku 4.5 and 9.00 for Claude Opus 5. The difference fit look small, but e go big when you multiply am. If you judge a 60 case set for every commit, with 40 commits every week, na 2,400 judge calls every week before anybody run the nightly job.

Two discounts fit apply well to eval work, and dem stack together. Eval runs no dey interactive, so Batch API cut both input and output prices by half in exchange for asynchronous delivery. This na the first row for the chart. The rubric and instructions dey exactly the same, byte for byte, for every call, so prompt caching fit work. Cache read cost one-tenth of the normal input price. A five-minute cache write cost 1.25 times the normal input price, so the cache don pay for itself after one hit. These na Anthropic list prices as of August 2026. Sonnet 5 dey use introductory pricing until 31 August 2026, so the third bar go rise after that date.

The order be this:

• Run deterministic checks for every case. E no cost any API money.
• Use small model judge for cases wey pass those checks.
• Use frontier judge only where small judge talk say fail, or talk say pass with low confidence.
• Do human review for small sample once every week.

CHEAP = "claude-haiku-4-5-20251001"
STRICT = "claude-opus-5"


def grade(case, result):
    hard = deterministic(case, result)
    if hard:
        return False, "deterministic", "; ".join(hard)
    first = judge(case, result["output"], CHEAP)
    if first["verdict"] == "pass" and first["confidence"] == "high":
        return True, CHEAP, first["reason"]
    second = judge(case, result["output"], STRICT)
    return second["verdict"] == "pass", STRICT, second["reason"]

This approach exchange some grading accuracy for lower cost, so measure the difference instead of assuming am. Once every month, use the strict judge to grade the whole set too, then compare the two columns. If dem disagree for more than a few cases, your rubric too loose for the small model. Na the rubric you need fix. Controlling wetin the agent itself spend na separate work. We cover am for cost control for an AI agent on a VPS.

Track pass rate over time for system wey you own

Pass rate wey you no fit connect to one commit na just feeling. Store one row for each case and each run, with the commit and model inside the row.

CREATE TABLE IF NOT EXISTS results (
  run_id      TEXT NOT NULL,
  ran_at      TEXT NOT NULL,
  git_sha     TEXT NOT NULL,
  agent_model TEXT NOT NULL,
  case_id     TEXT NOT NULL,
  passed      INTEGER NOT NULL,
  graded_by   TEXT NOT NULL,
  reason      TEXT
);

SELECT run_id, git_sha, agent_model,
       count(*) AS cases,
       round(100.0 * sum(passed) / count(*), 1) AS pass_pct
FROM results
GROUP BY run_id
ORDER BY ran_at DESC
LIMIT 10;

Use sqlite3 evals/results.db < evals/schema.sql load the schema, then use sqlite3 -box evals/results.db < evals/passrate.sql read the trend. One year of daily runs for 60 cases na about 22,000 rows, so the store no go turn into project on its own. Running SQLite for production on VPS explain the settings wey become important if dem share this file between machines.

The runner dey print the same information for one person:

run 2026-08-05T09:14:22Z  sha 4f1c9ab  model claude-sonnet-5  58/60 pass (96.7%)
FAIL refund-double-charge  deterministic: tool not called: create_refund
FAIL pto-policy-question   judge(opus): reply gives no dollar amount

Run the suite for changes wey fit break agent. This means prompt edits, model changes, and tool changes, instead of every commit anywhere for the repository. One pre-push hook fit handle the fast subset:

cat > .git/hooks/pre-push <<'EOF'
#!/bin/sh
python3 evals/run.py --set smoke || exit 1
EOF
chmod +x .git/hooks/pre-push

Full runs dey slower, so dem belong for schedule. One nightly systemd service and timer for the VPS go run the complete set against the deployed prompt. Na this one go catch changes wey come from outside your repository, like hosted tool wey change how e dey behave.

Human review, sample-based instead of complete

Judge dey use human labels take calibrate, so person must produce those labels. Every week, read one sample: every case wey judge fail, plus ten passes wey dem pick randomly. The random passes na the important half, because judge wey don quietly start passing bad answers go look perfect for any dashboard wey build from naim own verdicts.

Fifteen cases, three minutes each, na 45 minutes every week. E go return corrections to rubric for where you and judge no agree, plus new cases for failure types wey nobody don imagine before. Write human verdict inside the same table with graded_by set to human, so agreement between judge and human go become query instead of memory.

Wetin fit break for the eval harness itself

anthropic.RateLimitError for the first full run. Sixty cases run at once pass the request or token limit for your tier. Limit concurrency to four workers, and move the nightly run go the Batch API.

json.JSONDecodeError: Expecting value: line 1 column 1 (char 0) from the judge. The model reply with prose, or put its JSON inside a code fence. Retry am once, then record the case as an error. Never let parse failure count as pass, because a suite wey dey turn errors into passes go move toward 100% while the agent dey get worse.

Flaky cases. The same input pass for one run and fail for the next because the agent dey sample its output. Run the flaky case three times and record the fraction instead of deleting the case. A case wey pass two out of three runs na real robustness bug, and customer go find am.

Golden set rot. Somebody edit expected answer to make the suite green. Review diffs to evals/cases.jsonl as carefully as diffs to the agent, because that file na your written definition of wetin correct mean.

A suite wey never fail. If pass rate stay for 100% for one month, e mean say the set don stop tracking the product. Pull ten recent traces, find the ones wey the agent handle badly, and add dem.

FAQ

How many cases AI agent eval set need?

Start with 40 to 80, then grow the set from real failures. If cases dey below about 20, one flaky result fit change pass rate by 5 points, so the number no carry useful information again. When e pass few hundred, every run dey cost real money and real time, while each extra case dey add small coverage. The measure wey matter no be the count: na the share of known production failure types wey appear for the set at least once.

I fit trust LLM judge to grade my agent?

Only after you don measure am against your own labels. Keep 30 cases wey you grade by hand, then score the judge against dem anytime you change the judge model or judge prompt. Judges dey show length bias, where padded answers dey pass more often, and self-preference, where output from their own model family dey receive kinder grading. You fit test both: add padding to failed answer and judge am again, or use judge from another family to grade the same answers. If judge disagree with your labels for more than one case out of ten, the rubric too vague to use.

Which model suppose grade the evals?

Grade cheap ones first, then escalate. Deterministic assertions cost nothing, so dem run first for every case. Small model fit handle clear passes. Only fails and low-confidence verdicts go to frontier model. For list prices in August 2026, judging 1,000 cases cost about 1.80 US dollars with Claude Haiku 4.5 and about 9.00 with Claude Opus 5. Since eval runs dey asynchronous, Batch API cut either figure by half.

Evals fit replace production monitoring?

No, because dem answer different questions. An eval suite tell you whether a change wey you wan ship go make fixed set of cases better or worse. Tracing and monitoring tell you wetin real users dey hit now, including inputs wey no case cover. Dem dey feed each other: traces supply the new cases, while eval suite decide whether your fix really work.