Self-host an AI PR review agent on a VPS

What a self-hosted PR review agent does

A self-hosted PR review agent is a small program on a server you own. It reads the diff of a pull request (PR) and sends only the changed lines to a model. What comes back is posted as inline review comments. It never checks out your branch, and it never reads a file the pull request did not touch. The only credentials it holds are one model API (application programming interface) key and one token that can comment and do nothing else.

A model can read a diff. That part is solved. What matters is where the diff goes and who holds the key. A hosted review bot means every diff from every private repository leaves your network, lands in a third party's logs, and lives under their retention policy. On a VPS (virtual private server) you own, the diff goes from GitHub to your box to the model API, and you can read the forty lines of code that decide what gets sent.

What you need before you start

• A VPS running Ubuntu 24.04 with a self-hosted GitHub Actions runner already registered to the repository. Give it the extra label pr-review when you register it, because the workflow below selects on that label.
• An Anthropic API key from the Claude Console.
• A repository where you control who can open a pull request. A private repository is the easy case. The fork section below covers the public case, and the answer there is less comfortable.

Install the reviewer on the VPS

The runner service runs as the unprivileged account you created when you ran ./svc.sh install. Install the reviewer under that same account so the job can execute it without sudo. Replace runner below with your account name.

sudo apt update && sudo apt install -y gh python3-venv
sudo install -d -m 755 -o runner -g runner /opt/pr-review
sudo -u runner python3 -m venv /opt/pr-review/venv
sudo -u runner /opt/pr-review/venv/bin/pip install anthropic
gh --version

gh --version prints gh version 2.45.0 on Ubuntu 24.04 as of August 2026. Any release from 2.20 onward has the --input flag used below. A Command 'gh' not found message means the universe component is not enabled, so run sudo add-apt-repository universe and try again.

Where the key and the token live

Two secrets, two different lifetimes. Neither one goes in the repository.

ANTHROPIC_API_KEY is a repository secret, set under Settings, then Secrets and variables, then Actions. GitHub encrypts it and injects it into the step's environment at run time. It is never a file on disk and never in the git history.

GITHUB_TOKEN works differently. Actions mints a fresh token for each job and destroys it when the job ends. What that token may do is set by the permissions: block in the workflow, so this is where least privilege actually happens:

permissions:
  contents: read
  pull-requests: write

That token can post a review. It cannot push a commit, merge a branch, edit a workflow file, or touch another repository. An agent that can comment is a reviewer. An agent that can push is a committer, and nobody agreed to that. Treat the model key with the same care, because it spends money on your account. There is more on this shape of problem in keeping secrets out of an AI agent's reach.

Actions replaces the exact secret string with *** in job logs. It matches the exact string only, so a key that you base64 encode, split across two lines, or print one character at a time appears in the clear. Do not add a debug step that dumps the environment.

Why a pull request from a fork never sees your API key

GitHub's rule is short: with the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository. So a pull_request run from a fork starts your script with no ANTHROPIC_API_KEY, and the first API call fails with invalid x-api-key.

The tempting fix is to switch the trigger to pull_request_target, which runs in the base repository's context and does get the secrets. Do not do that here. GitHub's own security guidance says those workflows "are privileged, which means they share the same cache of the main branch with other privileged workflow triggers, and may have repository write access and access to referenced secrets", and that the result "can be exploited to take over a repository".

The same guidance is blunt about the runner: "Self-hosted runners should almost never be used for public repositories on GitHub, because any user can open pull requests against the repository and compromise the environment."

That drives two design choices. The job carries a guard so it only runs on branches pushed to your own repository. And the workflow has no actions/checkout step at all. The agent never has the branch on disk, so a hostile pull request is only text that gets sent to a model. It cannot run a build script on your VPS, because nothing on your VPS ever runs it.

Fetch the diff, not the repository

One request gets the whole diff as plain text.

export GH_TOKEN=your_token   # in the workflow this comes from secrets.GITHUB_TOKEN
gh api /repos/OWNER/REPO/pulls/42 -H "Accept: application/vnd.github.diff"

The Accept: application/vnd.github.diff media type is what turns the response from a JSON object describing the pull request into the unified diff itself, and gh api prints that body unchanged. The first line you see should start with diff --git a/. A gh: Not Found (HTTP 404) means the token cannot see the repository, which on a fine-grained personal token almost always means the Pull requests permission was left off.

Filter before you spend a token

This section is the difference between a bot people read and a bot people mute. Each filter below runs before the model sees a single byte.

• Path filters. Lock files, vendored directories, minified bundles and generated code. A model comment on package-lock.json is pure noise, and those files are often most of the bytes in a diff.
• A size cap. Over the cap, skip the review and exit green. A 4,000 line refactor gets one honest line saying it was too large to review automatically, instead of sixty guesses.
• A severity threshold and a comment cap. Report high and medium findings, up to ten of them, highest severity first. Nobody reads comment eleven.

The script

Save this as /opt/pr-review/review.py. It reads its configuration from the environment, so the workflow can change models without a code change.

#!/usr/bin/env python3
"""Review only the changed lines of one pull request."""
import json
import os
import subprocess
import sys

import anthropic

REPO = os.environ["GITHUB_REPOSITORY"]
PR = os.environ["PR_NUMBER"]
MODEL = os.environ.get("REVIEW_MODEL", "claude-haiku-4-5-20251001")
MAX_DIFF_BYTES = int(os.environ.get("MAX_DIFF_BYTES", "120000"))
MIN_SEVERITY = os.environ.get("MIN_SEVERITY", "medium")
MAX_COMMENTS = 10
RANK = {"low": 0, "medium": 1, "high": 2}
SKIP = ("package-lock.json", "poetry.lock", "/vendor/", "/node_modules/", ".min.js")

raw_diff = subprocess.run(
    ["gh", "api", f"/repos/{REPO}/pulls/{PR}",
     "-H", "Accept: application/vnd.github.diff"],
    check=True, capture_output=True, text=True,
).stdout

Splitting the diff per file is what makes path filtering possible. Numbering each line is what makes the review comments land. GitHub accepts an inline comment only on a line that is part of the diff, so the model has to cite a real line number. Handing it the numbers means it can copy one instead of inventing one.

def per_file(diff_text):
    """Split a unified diff into one string per file."""
    sections, current = [], []
    for line in diff_text.splitlines():
        if line.startswith("diff --git ") and current:
            sections.append("\n".join(current))
            current = []
        current.append(line)
    if current:
        sections.append("\n".join(current))
    return sections


def annotate(section):
    """Prefix every line that exists in the new file with its line number."""
    out, n, in_hunk = [], 0, False
    for line in section.splitlines():
        if line.startswith("@@"):
            n = int(line.split("+")[1].split(",")[0].split(" ")[0])
            in_hunk = True
            out.append(line)
        elif not in_hunk or line.startswith(("-", "\\")):
            out.append(line)
        else:
            out.append(f"{n}\t{line}")
            n += 1
    return "\n".join(out)


kept = [s for s in per_file(raw_diff)
        if not any(p in s.split("\n", 1)[0] for p in SKIP)]
payload = "\n".join(annotate(s) for s in kept)

if not payload.strip():
    print("every changed file was filtered out")
    raise SystemExit(0)
if len(payload) > MAX_DIFF_BYTES:
    print(f"diff is {len(payload)} bytes, over the {MAX_DIFF_BYTES} cap")
    raise SystemExit(0)

The hunk header carries the numbering. @@ -12,7 +12,9 @@ says the new file's hunk starts at line 12, so the counter starts there and advances on added and unchanged lines only. Removed lines pass through unnumbered, because they do not exist in the new file. The guard on lines beginning with a backslash skips the no-newline marker git writes at the end of a file, which would otherwise push every following number out by one.

Both exits use status 0, not 1. A filtered or oversized pull request should show a green check. A red check that a human cannot act on gets ignored, and once one check is ignored, all of them are.

SYSTEM = (
    "You review one pull request diff. Every line that exists in the new file is "
    "prefixed with its line number and a tab character. "
    "Report only defects you can see in the lines shown: a crash, a resource leak, "
    "a security mistake, a wrong boundary condition, a broken contract with code "
    "that is visible in this diff. Do not comment on style, naming or formatting. "
    "Do not guess about code you cannot see. Leave out anything you are not "
    "certain about. An empty findings list is a normal and common answer. "
    'Reply with JSON only, in this shape: {"findings": [{"path": "src/app.py", '
    '"line": 42, "severity": "high", "comment": "what is wrong, then why"}]} '
    "Every line number must be one you can see in the left column of that file."
)

client = anthropic.Anthropic()
message = client.messages.create(
    model=MODEL,
    max_tokens=2000,
    system=SYSTEM,
    messages=[{"role": "user", "content": payload}],
)
print(f"stop={message.stop_reason} in={message.usage.input_tokens} "
      f"out={message.usage.output_tokens}", file=sys.stderr)

text = message.content[0].text
findings = json.loads(text[text.find("{"):text.rfind("}") + 1])["findings"]
findings = [f for f in findings if RANK.get(f["severity"], 0) >= RANK[MIN_SEVERITY]]
findings.sort(key=lambda f: -RANK.get(f["severity"], 0))
del findings[MAX_COMMENTS:]

if not findings:
    print("nothing above the severity threshold; posting no comment")
    raise SystemExit(0)

review = {
    "event": "COMMENT",
    "body": f"Automated review of the changed lines. {len(findings)} finding(s).",
    "comments": [
        {"path": f["path"].removeprefix("b/"), "line": f["line"], "side": "RIGHT",
         "body": f"**{f['severity']}** {f['comment']}"}
        for f in findings
    ],
}
subprocess.run(
    ["gh", "api", "-X", "POST", f"/repos/{REPO}/pulls/{PR}/reviews", "--input", "-"],
    input=json.dumps(review), text=True, check=True,
)

Three details in that block are load-bearing. The JSON is sliced between the first { and the last } because a model will sometimes wrap its answer in a code fence, and json.loads chokes on the fence. The path is stripped of a leading b/, since that prefix comes from the diff header and GitHub wants a repository-relative path. And --input - sends the whole review as one API call, so ten findings arrive as one notification rather than ten.

When there is nothing to report, the script posts nothing. A bot that writes "no issues found" on every pull request teaches people to skim past it, and then they skim past the one that mattered.

Wire it into the workflow

Save this as .github/workflows/pr-review.yml:

name: pr-review

on:
  pull_request:
    types: [opened, synchronize, reopened]
    paths-ignore:
      - '**.md'
      - 'docs/**'

permissions:
  contents: read
  pull-requests: write

concurrency:
  group: pr-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  review:
    if: github.event.pull_request.head.repo.full_name == github.repository && !contains(github.event.pull_request.labels.*.name, 'no-ai-review')
    runs-on: [self-hosted, linux, pr-review]
    steps:
      - name: Review the changed lines
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REVIEW_MODEL: claude-haiku-4-5-20251001
          MIN_SEVERITY: medium
        run: /opt/pr-review/venv/bin/python /opt/pr-review/review.py

GITHUB_REPOSITORY is not in that env: block because Actions sets it for every job already. The concurrency group matters for the bill: without it, pushing three quick fixes to a branch runs three full reviews and you pay for all three, and with it only the last one survives.

The if: line does two jobs. The first half skips pull requests from forks, which would fail anyway with no key. The second half gives your team an off switch: add the no-ai-review label to a pull request and the job does not run.

Open a pull request and watch what happens:

gh run list --workflow=pr-review.yml --limit 3
gh run view --log
gh pr view 42 --comments

A run that finishes in a few seconds with nothing above the severity threshold; posting no comment in the log is working correctly. On a small, clean pull request that is the expected result.

What does an automated pull request review cost?

The diff is nearly all of the input, so the size of the diff sets the price. Below is one measured 500 line diff plus the system prompt, counted with the token counting endpoint rather than estimated.

[
  {
    "label": "Haiku 4.5",
    "input_tokens": "8,000",
    "output_tokens": "1,200"
  },
  {
    "label": "Sonnet 5",
    "input_tokens": "10,400",
    "output_tokens": "1,560"
  },
  {
    "label": "Opus 5",
    "input_tokens": "10,400",
    "output_tokens": "1,560"
  }
]

That diff came to 8,000 input tokens on Haiku 4.5 and 10,400 on Sonnet 5. Same text, different count. Claude models from 4.7 onward use a newer tokenizer that produces roughly 30% more tokens for the same input, which Anthropic documents on its pricing page. Allow for that whenever you compare a newer model against an older one on price per million tokens alone.

List prices as of August 2026: Haiku 4.5 is $1 per million input tokens and $5 per million output. Sonnet 5 is $2 and $10 under introductory pricing that runs to 31 August 2026, then $3 and $15. Opus 5 is $5 and $25.

[
  {
    "label": "Haiku 4.5",
    "cost_per_pr_cents": 1.4,
    "cost_200_prs_usd": "2.80"
  },
  {
    "label": "Sonnet 5",
    "cost_per_pr_cents": 3.64,
    "cost_200_prs_usd": "7.28"
  },
  {
    "label": "Opus 5",
    "cost_per_pr_cents": 9.1,
    "cost_200_prs_usd": "18.20"
  }
]

That is 1.4 cents per pull request on Haiku 4.5 and 9.1 cents on Opus 5. A team merging 200 pull requests a month pays about $2.80 on Haiku 4.5, $7.28 on Sonnet 5, or $18.20 on Opus 5. From 1 September 2026, multiply the Sonnet 5 row by 1.5.

Two things push a real bill above that estimate. The synchronize trigger reviews every push, so an active branch with eight pushes costs eight reviews, and the concurrency rule only helps when pushes land close together. The figures also assume the path filters are working: one unfiltered lock file can double the input on its own.

Prompt caching does not help here. The cached prefix has to be byte-identical between calls, and the diff is different every time. The system prompt is the only stable part, and it sits far below the minimum cacheable length. For the general rule, see when prompt caching pays for itself, and for picking between the three models above, which Claude model to use for which job.

print(client.messages.count_tokens(
    model=MODEL, system=SYSTEM, messages=[{"role": "user", "content": payload}]
).input_tokens)

Why review bots get muted, and how to avoid it

Two behaviours destroy trust in these bots, and both have a fix in the code above.

Reviewing everything at once. A bot that leaves forty comments gets zero of them read. The severity threshold and the ten comment cap are not politeness, they are what keeps the real findings visible. Sorting by severity before truncating means the cap drops the least important findings rather than a random ten.

Commenting with confidence on something it cannot check. This is the one that makes engineers switch it off for good. A model shown 200 lines of a 40,000 line codebase will still write "this breaks the cache invalidation in redis_client.py" about a file it has never seen. The system prompt pushes back in plain language: report only defects visible in the lines shown, and leave out anything you are not certain about. Naming the failure directly works better than asking for accuracy in general, and telling the model that an empty result is normal is what stops it inventing something to say about a two line change.

Post the review as COMMENT, never as REQUEST_CHANGES. A model's opinion should not be able to block a merge, and the moment it can, someone with a deadline will remove the whole workflow rather than argue with it.

Failure modes, with the strings you will see

HTTP 422 when posting the review. gh prints gh: Unprocessable Entity (HTTP 422) and the response body names the field: Pull request review thread line must be part of the diff. GitHub cannot anchor that comment. The usual causes are a line number the model invented, a path that still carries the b/ prefix, or a comment on a removed line, which needs side set to LEFT instead of RIGHT. Print the review JSON before posting and check one comment by hand against the diff.

invalid x-api-key from the model API. The step fails on the first messages.create call. Either the ANTHROPIC_API_KEY secret is not set on the repository, or the pull request came from a fork so Actions passed no secrets at all. The fork guard in the if: line should have skipped it, so check that line first.

gh: Resource not accessible by integration (HTTP 403). The job token cannot write to pull requests. Add pull-requests: write to the permissions: block. If it is already there, look at Settings, then Actions, then General, where an organisation policy can cap what any workflow token is allowed to request.

json.decoder.JSONDecodeError. The model did not return parseable JSON. The common cause is a response that hit the token ceiling and stopped mid object. The log line prints stop_reason for exactly this: a value of max_tokens means raise max_tokens or lower MAX_COMMENTS.

The workflow never runs. gh run list shows nothing for the pull request. Check that paths-ignore did not filter every changed file, then the fork guard and the label guard, then whether the runner is alive with sudo systemctl status 'actions.runner.*' on the VPS. An offline runner leaves the job queued with no error message anywhere in the pull request.

Every review comes back empty. Set MIN_SEVERITY to low for one run. If findings appear, the threshold is doing its job. If nothing appears, print payload and confirm the filters have not removed the entire diff.

Running it alongside your other agents

The reviewer is small, so it is tempting to drop it on the box that already runs everything else. Keep it separate if the repository matters. This process holds a token that can comment on your code and a key that spends your money, and a self-hosted runner is by design a place where workflow code executes. A dedicated unprivileged account with no sudo rights, on a host running nothing else, is the baseline. If you also run interactive agents that do check out code, a disposable VM per agent is the pattern that holds up, and running a coding agent on a VPS covers the general setup. If the Anthropic API is new to you, a first Claude API app on a VPS is a smaller place to start than this.

FAQ

Does an AI PR review agent need write access to my repository?

No. It needs pull-requests: write to post a review and contents: read to fetch the diff. That is the whole list, and you set it in the permissions: block of the workflow, which caps what the per-job GITHUB_TOKEN can do. With those two lines the agent can comment on a pull request but cannot push a commit or merge a branch. Post reviews with event: COMMENT rather than REQUEST_CHANGES so it also cannot block a merge.

Why does my review comment fail with HTTP 422?

GitHub accepts an inline review comment only on a line that is part of the pull request diff, and returns Pull request review thread line must be part of the diff when it is not. Check that path is repository-relative with no b/ prefix from the diff header, and that the line number appears inside a hunk of that file. side must be RIGHT for an added or unchanged line and LEFT for a removed one. Prefixing every line of the diff with its new-file line number before sending it to the model is what stops the model inventing numbers in the first place.

Can I run this on a public repository with pull requests from forks?

Not with this design. GitHub does not pass secrets to a workflow triggered from a fork, so the model key is missing and the run fails. GitHub also states that self-hosted runners "should almost never be used for public repositories", because anyone can open a pull request that causes code to run on your machine. For a public project, either restrict the reviewer to branches pushed to the repository itself, which is what the if: guard does, or move the review step to a GitHub-hosted runner and accept that the diff leaves your own infrastructure.

Which model should I use for pull request review?

Start with Haiku 4.5. Reading a bounded diff against a fixed list of defect types is not a hard reasoning problem, and the cheapest model keeps the monthly bill at a number nobody argues about. Move up to Sonnet 5 if you find it missing real bugs in your language or framework, and measure that rather than assuming it. Opus 5 is the most expensive of the three by a wide margin per pull request, which is easier to justify on a release branch than on every push to every feature branch.