How to Run Self-Hosted Evals for AI Agents

Wetín self-hosted evals for AI agents be

Self-hosted evals for AI agents na four things wey you keep for your own repository: file of saved cases, script wey run the agent on top dem, set of checks wey grade each answer, and table of results wey you fit query. No vendor dey needed for any item for that list. The whole loop na just some hundred lines of Python and one SQLite file.

The agent work for the demo because na you pick the five inputs yourself. E spoil for week two because prompt line change, or model change, or tool description change, and no measurement cover any of dem. Eval loop turn "e dey feel worse now" to "pass rate drop from 58 of 60 to 51 of 60 for commit 4f1c9ab".

The loop get four steps, and this guide get one section for each step: collect real traces, promote the interesting ones into cases, grade every case for every change, and store the pass rate beside the commit wey produce am. The same loop work for anything wey you run the agent on, and the self-hosted agent frameworks wey worth running mainly differ for how much of the trace dem hand give you free.

Why agent dey break for week two

Agent na prompt, model, set of tool definitions, and any context wey system retrieve when e dey run. All four fit change even when your application code no change, so normal code review no go see anything to object to.

The most common cause na prompt edit. You add one sentence to stop rude reply. That sentence change behaviour for inputs wey nobody test again, and traces show am clearly: last week's trace for the same question get a create_refund tool call, this week's one no get any, and reply na polite apology instead. Nothing raise error, so no alert fire.

The second cause na model. Record the exact model string wey you send with every run, claude-haiku-4-5-20251001 instead of shorthand wey you just keep for your head, because you fit only diagnose pass rate wey drop on the day you switch models when model dey inside the row.

The third one na tools. If you change the wording of tool description, e fit change when model decide to call am. If your tools dey come through MCP servers wey dey run for VPS, the schema dey for another process, so e fit change under you without any diff for your repository at all. The fourth one na retrieval: the same question hit index wey system rebuild overnight, and answer follow the new document.

Make the golden set from traces wey you already collect

No invent eval cases. Take dem from traffic. If you already dey run self-hosted Langfuse tracing for your agent, every request dey store with its input, tool calls, and output. Na exactly this raw material case need.

Export one window of root observations through the public API. E dey use basic authentication, with your public key as the username and your secret key as the password.

export LF_HOST="https://langfuse.example.com"
curl -sS -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
  "$LF_HOST/api/public/v2/observations?limit=50&isRootObservation=true&fromStartTime=2026-07-01T00:00:00Z" \
  | jq '.data[0]'

Read one record before you write any parser. The rows dey come under data, but the field names wey hold the question and reply depend on how your agent instrument its spans. So map wetin you actually see, instead of wetin you expect. Then write the cases by hand, one JSON object for each line, inside evals/cases.jsonl:

{"id": "refund-double-charge", "tags": ["smoke"], "input": "I was charged twice for order 41822.", "must_call": ["lookup_order", "create_refund"], "must_not_include": ["I cannot help"], "rubric": "The reply confirms exactly one refund for order 41822 and states the amount."}

Five rules go keep the set useful to run:

• 40 to 80 cases dey enough to start. If e below 20, one flaky case fit change the pass rate by 5 points, and people go ignore number wey dey jump without reason.
• Every production bug wey you fix become a case that same day. Na this habit dey make the set grow for the correct direction.
• One behaviour for each case. Case wey check refund amount and tone together no go tell you anything when e fail.
• The id no dey ever change, because na the id dey help compare today's run with last month's run.
• Redact before you commit. This file go enter git, so remove customer names and any order numbers wey no belong to you.

Do grade with deterministic checks first, because dem free

Anything wey get correct answer fit use plain assertion. No model call, no cost, no confusion. Deterministic checks dey catch structural regressions, and na dem dey break systems around your agent: JSON no parse, tool no ever get called, forbidden phrase come back, answer no cite any source.

Na one function sabi about your agent. Everything else for the harness dey generic.

import json, os, urllib.request

def run_agent(case):
    req = urllib.request.Request(
        os.environ["AGENT_URL"],
        data=json.dumps({"input": case["input"]}).encode(),
        headers={"content-type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=120) as resp:
        return json.load(resp)


def deterministic(case, result):
    text = result.get("output", "")
    called = [c["name"] for c in result.get("tool_calls", [])]
    failures = []
    for tool in case.get("must_call", []):
        if tool not in called:
            failures.append(f"tool not called: {tool}")
    for phrase in case.get("must_not_include", []):
        if phrase.lower() in text.lower():
            failures.append(f"forbidden phrase: {phrase}")
    if len(called) > case.get("max_tool_calls", 12):
        failures.append(f"too many tool calls: {len(called)}")
    return failures

Keep the tool budget for that list. Agent wey solve one case with 3 calls today and 11 tomorrow don regress, even when final answer correct, because you dey pay for every call e make.

LLM as judge, and the four ways e fit go wrong

Anything wey pass the assertions still need grader wey fit read. An LLM judge na second model call: e receive the question, the agent answer, and one criterion, then e return verdict. Na the only practical way to grade “whether the reply answer wetin user ask”.

Four rules make judge useful:

• Binary verdict, never 1 to 10 score. Scale dey return 7 and 8 for almost everything, so the number no dey change and you learn nothing from am.
• One criterion for each call. Ask about the refund amount, or about the tone, no be both together.
• Give judge the expected answer whenever the case get one. Grading against reference easier well-well than grading in the abstract.
• Force the output shape and parse am strictly.

from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment


def judge_prompt(case, output):
    return (
        "You grade one answer against one criterion.\n"
        "Reply with JSON only, in this exact shape:\n"
        '{"verdict": "pass", "confidence": "high", "reason": "one short sentence"}\n'
        f"Criterion: {case['rubric']}\n"
        f"Question: {case['input']}\n"
        f"Answer: {output}\n"
        "Length is not a criterion. Judge only the criterion above."
    )


def judge(case, output, model):
    msg = client.messages.create(
        model=model,
        max_tokens=200,
        messages=[{"role": "user", "content": judge_prompt(case, output)}],
    )
    return json.loads(msg.content[0].text)

Now make we look the failure modes. Each one get test wey you fit run this afternoon, and e important to run dem, because unchecked judge dey produce numbers wey look precise but mean nothing.

Length bias. Longer answers dey pass more often. Test am: take ten answers wey judge fail, add two paragraphs of confident filler to each one, without adding any new fact, then judge dem again. If any verdict change to pass, na length bias be that, and na the rubric you need fix.

Self-preference. Judge often dey grade output from im own model family more kindly than output from another one. Test am: use judges from two different families to grade the same 30 answers, then compare the verdicts case by case. For cases wey dem disagree, read the case yourself.

Position bias. If you use judge to compare two answers, A and B, change the order and run am again. If verdict change after the swap, pairwise comparison no safe yet for that rubric.

Rubric drift. Vague criteria dey produce agreeable judges. “Is the answer helpful” go pass almost anything. “Does the answer state the refund amount in dollars” go pass only the thing wey you mean. Rewrite every criterion until e name the fact wey you dey check.

One guard fit cover all four. Keep 30 cases wey you label by hand, and score the judge against your labels every time you change the judge model or judge prompt. If e disagree with you for more than one case out of ten, fix the rubric before you trust any pass rate wey e produce. Judge na code, so version and review am like code.

Use cheap model first, then escalate go frontier model

If you use the most expensive model to judge every case for every commit, eval bill fit pass the agent wey e dey test. Arrange graders from the cheapest to the most expensive, then stop once the answer don clear.

[
  {
    "label": "Haiku 4.5, Batch API",
    "usd_per_1000_judge_calls": "0.90"
  },
  {
    "label": "Haiku 4.5",
    "usd_per_1000_judge_calls": "1.80"
  },
  {
    "label": "Sonnet 5",
    "usd_per_1000_judge_calls": "3.60"
  },
  {
    "label": "Opus 5",
    "usd_per_1000_judge_calls": "9.00"
  }
]

Those figures assume around 1,200 input tokens and 120 output tokens for each judge call. This size realistic for one question, one answer, and one criterion. Judging 1,000 cases costs 1.80 US dollars with Claude Haiku 4.5 and 9.00 with Claude Opus 5. The difference fit look small until you multiply am. A 60-case set, judged for every commit at 40 commits per week, na 2,400 judge calls every week before anybody run the nightly job.

Two discounts fit apply directly to eval work, and dem stack together. Eval runs no be interactive, so Batch API cuts both input and output prices by half in exchange for asynchronous delivery. This na the first row for the chart. The rubric and instructions dey exactly the same bytes for every call, so prompt caching fit help. A cache read costs one tenth of the base input price. A five-minute cache write costs 1.25 times the base input price. This means the cache pays for itself after one hit. These na Anthropic list prices as of August 2026. Sonnet 5 dey use introductory pricing until 31 August 2026, so the third bar go increase after that date.

The ladder, in order:

• Run deterministic checks for every case. API cost no dey at all.
• Use a small model judge for cases wey pass those checks.
• Use a frontier judge only where the small judge talk say fail, or talk say pass with low confidence.
• Do human review for small sample once every week.

CHEAP = "claude-haiku-4-5-20251001"
STRICT = "claude-opus-5"


def grade(case, result):
    hard = deterministic(case, result)
    if hard:
        return False, "deterministic", "; ".join(hard)
    first = judge(case, result["output"], CHEAP)
    if first["verdict"] == "pass" and first["confidence"] == "high":
        return True, CHEAP, first["reason"]
    second = judge(case, result["output"], STRICT)
    return second["verdict"] == "pass", STRICT, second["reason"]

This approach reduce grading accuracy small to cut cost, so measure the trade-off instead of assuming say e good enough. Once every month, use the strict judge grade the whole set too, then compare the two columns. If dem disagree for more than a few cases, your rubric too loose for the small model. Na the rubric you need fix. Controlling how much the agent itself spends na separate work, and cost control for an AI agent on a VPS cover am.

Track pass rate over time inside system wey you own

Pass rate wey you no fit link to commit na just feeling. Store one row for every case and every run, with the commit and model inside the row.

CREATE TABLE IF NOT EXISTS results (
  run_id      TEXT NOT NULL,
  ran_at      TEXT NOT NULL,
  git_sha     TEXT NOT NULL,
  agent_model TEXT NOT NULL,
  case_id     TEXT NOT NULL,
  passed      INTEGER NOT NULL,
  graded_by   TEXT NOT NULL,
  reason      TEXT
);

SELECT run_id, git_sha, agent_model,
       count(*) AS cases,
       round(100.0 * sum(passed) / count(*), 1) AS pass_pct
FROM results
GROUP BY run_id
ORDER BY ran_at DESC
LIMIT 10;

Load the schema with sqlite3 evals/results.db < evals/schema.sql, then read the trend with sqlite3 -box evals/results.db < evals/passrate.sql. One year of daily runs for 60 cases na about 22,000 rows, so the store no go turn into project by itself. Running SQLite for production on VPS explain the settings wey go start matter if dem share this file between machines.

The runner dey print the same information for person:

run 2026-08-05T09:14:22Z  sha 4f1c9ab  model claude-sonnet-5  58/60 pass (96.7%)
FAIL refund-double-charge  deterministic: tool not called: create_refund
FAIL pto-policy-question   judge(opus): reply gives no dollar amount

Run the suite on changes wey fit break an agent. This means prompt edits, model changes, and tool changes, instead of every commit anywhere for the repository. A pre-push hook dey cover the fast subset:

cat > .git/hooks/pre-push <<'EOF'
#!/bin/sh
python3 evals/run.py --set smoke || exit 1
EOF
chmod +x .git/hooks/pre-push

Full runs dey slower, so schedule dem. A nightly systemd service and timer for the VPS runs the whole set against the deployed prompt. Na this one dey catch changes wey come from outside your repository, like hosted tool wey its behaviour don change.

Human review, sample, no be exhaustive

Judge dey calibrated against human labels, so person must produce those labels. Read one sample every week: every case wey judge fail, plus ten passes wey you pick randomly. The random passes na the important half, because judge wey don quietly start to pass bad answers go look perfect for any dashboard wey build from judge own verdicts.

Fifteen cases at three minutes each na 45 minutes every week, and e go return corrections to the rubric where you and judge no agree, plus new cases for failure types wey nobody don imagine before. Write the human verdict inside the same table with graded_by set to human, so agreement between judge and human go become query instead of memory.

Wetin fit break for the eval harness itself

anthropic.RateLimitError for the first full run. Sixty cases wey run at once pass the request or token limit for your tier. Limit concurrency to four workers, and move the nightly run go the Batch API.

json.JSONDecodeError: Expecting value: line 1 column 1 (char 0) from the judge. The model reply with prose, or put its JSON inside code fence. Retry am once, then record the case as error. Never allow parse failure count as pass, because suite wey turn errors into passes go climb toward 100% while the agent dey worse.

Flaky cases. The same input pass for one run and fail for the next because the agent dey sample its output. Run the flaky case three times and record the fraction instead of deleting the case. Case wey pass two runs out of three na real robustness bug, and customer go discover am.

Golden set rot. Person edit expected answer to make the suite green. Review diffs to evals/cases.jsonl as carefully as diffs to the agent, because that file na your written definition of correct.

Suite wey never fail. Pass rate wey stay for 100% for one month mean say the set don stop tracking the product. Pull ten recent traces, find the ones wey the agent handle badly, and add dem. Then break something on purpose and confirm say the run go red. This check na mutation testing dey apply to test suite and the only way to know say your set still get teeth.

FAQ

How many cases AI agent eval set need?

Start with 40 to 80, then grow the set with real failures. If cases no reach about 20, one unstable result fit change pass rate by 5 points, so the number no get enough meaning. Once e pass few hundred, every run dey cost real money and time, while each extra case dey add small coverage. The important measure no be the count: na the share of known production failure types wey appear for the set at least once.

I fit trust LLM judge to grade my agent?

Only after you don measure am against your own labels. Keep 30 cases wey you grade by hand, and score the judge against dem whenever you change the judge model or judge prompt. Judges get length bias, where answers wey dem pad dey pass more often, and self-preference, where dem grade output from their own model family more kindly. You fit test both: add extra text to one failed answer and judge am again, or use judge from another model family to grade the same answers. If judge disagree with your labels for more than one case out of ten, the rubric too vague to use.

Which model suppose grade the evals?

Grade the cheap ones first, then escalate. Deterministic assertions no cost anything, so dem run first for every case. Small model fit handle clear passes. Only fails and verdicts wey get low confidence go to frontier model. For list prices in August 2026, judging 1,000 cases cost about 1.80 US dollars with Claude Haiku 4.5 and about 9.00 with Claude Opus 5. Because eval runs dey asynchronous, Batch API cut either amount by half.

Evals fit replace production monitoring?

No, because dem answer different questions. Eval suite tell you whether a change wey you about to release make a fixed set of cases better or worse. Tracing and monitoring tell you wetin real users dey hit now, including inputs wey no case cover. Dem dey support each other: traces provide the new cases, while eval suite decide whether your fix actually work.