How to Run AI PR Review Agent for VPS

Wetin self-hosted PR review agent dey do

Self-hosted PR review agent na small program wey dey run for server wey you own. E dey read the diff of pull request (PR) and send only the lines wey change go model. Wetin come back, e post am as inline review comments. E no ever checkout your branch, and e no ever read file wey the pull request no touch. The only credentials wey e hold na one model API (application programming interface) key and one token wey fit comment and do nothing else.

Model fit read diff. That part don solve. Wetin matter na where the diff dey go and who hold the key. Hosted review bot mean say every diff from every private repository go comot from your network, enter third party logs, and stay under their retention policy. For VPS (virtual private server) wey you own, the diff go move from GitHub go your box, then go model API. You fit read the forty lines of code wey decide wetin go send.

Wetín you need before you start

• A VPS wey dey run Ubuntu 24.04, with self-hosted GitHub Actions runner wey don already register for the repository. Give am the extra label pr-review when you register am, because the workflow below dey select runner based on that label.
• An Anthropic API key from Claude Console.
• A repository where na you dey control who fit open pull request. Private repository na the easier case. The fork section below cover the public case, and the answer for there no too comfortable.

Install the reviewer for the VPS

Runner service dey run with the unprivileged account wey you create when you run ./svc.sh install. Install reviewer under that same account so the job fit run am without sudo. Replace runner below with your account name.

sudo apt update && sudo apt install -y gh python3-venv
sudo install -d -m 755 -o runner -g runner /opt/pr-review
sudo -u runner python3 -m venv /opt/pr-review/venv
sudo -u runner /opt/pr-review/venv/bin/pip install anthropic
gh --version

gh --version dey print gh version 2.45.0 for Ubuntu 24.04 as of August 2026. Any release from 2.20 go up get the --input flag wey dem use below. If you see Command 'gh' not found message, e mean say universe component no dey enabled. Run sudo add-apt-repository universe, then try again.

Where key and token dey live

Two secrets get different lifetimes. None of dem suppose enter the repository.

ANTHROPIC_API_KEY na repository secret. You set am under Settings, then Secrets and variables, then Actions. GitHub encrypts am and injects am into the step environment when e dey run. E never become file for disk, and e never enter git history.

GITHUB_TOKEN dey work differently. Actions mints fresh token for every job and destroys am when the job finish. The permissions: block for the workflow dey decide wetin that token fit do. Na here least privilege really happen:

permissions:
  contents: read
  pull-requests: write

That token fit post a review. E no fit push commit, merge branch, edit workflow file, or touch another repository. Agent wey fit comment na reviewer. Agent wey fit push na committer, and nobody agree to that. Treat model key with the same care, because e dey spend money from your account. You fit read more about this kind problem for keeping secrets away from AI agent.

Actions replaces the exact secret string with *** for job logs. E only matches the exact string. So if you base64 encode the key, split am across two lines, or print am one character at a time, e go show for clear text. No add debug step wey dumps the environment.

Why pull request wey come from fork no dey see your API key

GitHub rule short: apart from GITHUB_TOKEN, dem no dey pass secrets to the runner when workflow start from forked repository. So pull_request run from fork dey start your script without ANTHROPIC_API_KEY, and the first API call fail with invalid x-api-key.

The fix wey fit tempt person na to change the trigger to pull_request_target. E dey run for base repository context and e dey get the secrets. No do am here. GitHub own security guidance talk say those workflows "get privilege, meaning say dem share the same cache of the main branch with other privileged workflow triggers, and dem fit get repository write access plus access to referenced secrets", and the result "fit exploit to take over repository".

The same guidance talk plainly about the runner: "Self-hosted runners almost never suppose dey used for public repositories for GitHub, because any user fit open pull requests against the repository and compromise the environment."

This one lead to 2 design choices. The job get guard wey make am run only for branches wey dem push to your own repository. And the workflow no get actions/checkout step at all. The agent never get the branch for disk, so hostile pull request na only text wey dem send to model. E no fit run build script for your VPS, because nothing for your VPS ever run am. But text no automatically mean say e harmless: diff wey stranger write na untrusted input wey dey enter model, the same trust boundary wey you meet when you give agent web search, and the only thing wey contain am here na say this agent fit do nothing except post comment.

Fetch diff, no be repository

One request go collect the complete diff as plain text.

export GH_TOKEN=your_token   # in the workflow this comes from secrets.GITHUB_TOKEN
gh api /repos/OWNER/REPO/pulls/42 -H "Accept: application/vnd.github.diff"

The Accept: application/vnd.github.diff media type na wetin change the response from JSON object wey dey describe the pull request to the unified diff itself, and gh api go print that body without change. The first line wey you go see suppose start with diff --git a/. A gh: Not Found (HTTP 404) mean say the token no fit see the repository. For fine-grained personal token, this almost always mean say dem leave the Pull requests permission off.

Filter before you spend a token

This section na the difference between bot wey people go read and bot wey people go mute. Every filter here dey run before model see even one byte.

• Path filters. Lock files, vendored directories, minified bundles and generated code. Model comment for package-lock.json na pure noise, and those files often make up most of the bytes for one diff.
• A size cap. Once e pass the cap, skip the review and exit green. A 4,000 line refactor go get one honest line wey talk say e too large for automatic review, instead of sixty guesses.
• A severity threshold and a comment cap. Report high and medium findings, up to ten of dem, with highest severity first. Nobody dey read comment eleven.

Di script

Save am as /opt/pr-review/review.py. E dey read configuration from environment, so workflow fit change models without code change.

#!/usr/bin/env python3
"""Review only the changed lines of one pull request."""
import json
import os
import subprocess
import sys

import anthropic

REPO = os.environ["GITHUB_REPOSITORY"]
PR = os.environ["PR_NUMBER"]
MODEL = os.environ.get("REVIEW_MODEL", "claude-haiku-4-5-20251001")
MAX_DIFF_BYTES = int(os.environ.get("MAX_DIFF_BYTES", "120000"))
MIN_SEVERITY = os.environ.get("MIN_SEVERITY", "medium")
MAX_COMMENTS = 10
RANK = {"low": 0, "medium": 1, "high": 2}
SKIP = ("package-lock.json", "poetry.lock", "/vendor/", "/node_modules/", ".min.js")

raw_diff = subprocess.run(
    ["gh", "api", f"/repos/{REPO}/pulls/{PR}",
     "-H", "Accept: application/vnd.github.diff"],
    check=True, capture_output=True, text=True,
).stdout

Na splitting the diff per file dey make path filtering possible. Na numbering each line dey make review comments land for correct place. GitHub dey accept inline comment only for line wey dey part of the diff, so model must cite real line number. If you give am the numbers, e fit copy one instead of inventing one.

def per_file(diff_text):
    """Split a unified diff into one string per file."""
    sections, current = [], []
    for line in diff_text.splitlines():
        if line.startswith("diff --git ") and current:
            sections.append("\n".join(current))
            current = []
        current.append(line)
    if current:
        sections.append("\n".join(current))
    return sections


def annotate(section):
    """Prefix every line that exists in the new file with its line number."""
    out, n, in_hunk = [], 0, False
    for line in section.splitlines():
        if line.startswith("@@"):
            n = int(line.split("+")[1].split(",")[0].split(" ")[0])
            in_hunk = True
            out.append(line)
        elif not in_hunk or line.startswith(("-", "\\")):
            out.append(line)
        else:
            out.append(f"{n}\t{line}")
            n += 1
    return "\n".join(out)


kept = [s for s in per_file(raw_diff)
        if not any(p in s.split("\n", 1)[0] for p in SKIP)]
payload = "\n".join(annotate(s) for s in kept)

if not payload.strip():
    print("every changed file was filtered out")
    raise SystemExit(0)
if len(payload) > MAX_DIFF_BYTES:
    print(f"diff is {len(payload)} bytes, over the {MAX_DIFF_BYTES} cap")
    raise SystemExit(0)

Hunk header carry the numbering. @@ -12,7 +12,9 @@ dey show say new file hunk start for line 12, so counter start from there and advance only for added and unchanged lines. Removed lines pass through without number because dem no dey exist for new file. The guard for lines wey start with backslash dey skip the no-newline marker wey git writes for end of file. Without this guard, every number after am go shift by one.

Both exits use status 0, no be 1. Filtered or oversized pull request suppose show green check. Red check wey human no fit act on dey get ignored. Once person ignore one check, dem fit ignore all of dem.

SYSTEM = (
    "You review one pull request diff. Every line that exists in the new file is "
    "prefixed with its line number and a tab character. "
    "Report only defects you can see in the lines shown: a crash, a resource leak, "
    "a security mistake, a wrong boundary condition, a broken contract with code "
    "that is visible in this diff. Do not comment on style, naming or formatting. "
    "Do not guess about code you cannot see. Leave out anything you are not "
    "certain about. An empty findings list is a normal and common answer. "
    'Reply with JSON only, in this shape: {"findings": [{"path": "src/app.py", '
    '"line": 42, "severity": "high", "comment": "what is wrong, then why"}]} '
    "Every line number must be one you can see in the left column of that file."
)

client = anthropic.Anthropic()
message = client.messages.create(
    model=MODEL,
    max_tokens=2000,
    system=SYSTEM,
    messages=[{"role": "user", "content": payload}],
)
print(f"stop={message.stop_reason} in={message.usage.input_tokens} "
      f"out={message.usage.output_tokens}", file=sys.stderr)

text = message.content[0].text
findings = json.loads(text[text.find("{"):text.rfind("}") + 1])["findings"]
findings = [f for f in findings if RANK.get(f["severity"], 0) >= RANK[MIN_SEVERITY]]
findings.sort(key=lambda f: -RANK.get(f["severity"], 0))
del findings[MAX_COMMENTS:]

if not findings:
    print("nothing above the severity threshold; posting no comment")
    raise SystemExit(0)

review = {
    "event": "COMMENT",
    "body": f"Automated review of the changed lines. {len(findings)} finding(s).",
    "comments": [
        {"path": f["path"].removeprefix("b/"), "line": f["line"], "side": "RIGHT",
         "body": f"**{f['severity']}** {f['comment']}"}
        for f in findings
    ],
}
subprocess.run(
    ["gh", "api", "-X", "POST", f"/repos/{REPO}/pulls/{PR}/reviews", "--input", "-"],
    input=json.dumps(review), text=True, check=True,
)

Three details for that block dey carry important work. The JSON dey get sliced between the first { and the last } because model sometimes dey wrap its answer inside code fence, and json.loads no fit handle the fence. The path dey remove leading b/, because that prefix come from diff header and GitHub want repository-relative path. And --input - dey send the whole review as one API call, so ten findings go arrive as one notification instead of ten.

When there is nothing to report, the script no post anything. Bot wey write "no issues found" for every pull request dey teach people to skip am. Later, dem fit skip the one wey really matter too.

Liga am go workflow

Save dis one as .github/workflows/pr-review.yml:

name: pr-review

on:
  pull_request:
    types: [opened, synchronize, reopened]
    paths-ignore:
      - '**.md'
      - 'docs/**'

permissions:
  contents: read
  pull-requests: write

concurrency:
  group: pr-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  review:
    if: github.event.pull_request.head.repo.full_name == github.repository && !contains(github.event.pull_request.labels.*.name, 'no-ai-review')
    runs-on: [self-hosted, linux, pr-review]
    steps:
      - name: Review the changed lines
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REVIEW_MODEL: claude-haiku-4-5-20251001
          MIN_SEVERITY: medium
        run: /opt/pr-review/venv/bin/python /opt/pr-review/review.py

GITHUB_REPOSITORY no dey inside dat env: block because Actions dey set am for every job already. The concurrency group matter for the bill: if e no dey, pushing three quick fixes go one branch go run three full reviews and you go pay for all three. But with am, na only the last one go remain.

The if: line dey do two things. The first half skip pull requests from forks, wey for fail anyway because key no dey. The second half give your team off switch: add no-ai-review label to pull request and the job no go run.

Open pull request and monitor wetin happen:

gh run list --workflow=pr-review.yml --limit 3
gh run view --log
gh pr view 42 --comments

If run finish within few seconds and nothing above the severity threshold; posting no comment dey inside log, e mean say everything dey work correctly. For small, clean pull request, na so result suppose be.

Wetin automated pull request review dey cost?

The diff na almost all the input, so na the size of the diff dey set the price. Below na one measured 500 line diff plus the system prompt, wey dem count with the token counting endpoint instead of estimate.

[
  {
    "label": "Haiku 4.5",
    "input_tokens": "8,000",
    "output_tokens": "1,200"
  },
  {
    "label": "Sonnet 5",
    "input_tokens": "10,400",
    "output_tokens": "1,560"
  },
  {
    "label": "Opus 5",
    "input_tokens": "10,400",
    "output_tokens": "1,560"
  }
]

That diff use 8,000 input tokens for Haiku 4.5 and 10,400 for Sonnet 5. Na the same text, but the count different. Claude models from 4.7 upward dey use newer tokenizer wey dey produce roughly 30% more tokens for the same input, as Anthropic document for its pricing page. Remember this whenever you dey compare newer model with older one based only on price per million tokens.

List prices as of August 2026: Haiku 4.5 na $1 per million input tokens and $5 per million output. Sonnet 5 na $2 and $10 under introductory pricing wey go last reach 31 August 2026, then e go become $3 and $15. Opus 5 na $5 and $25.

[
  {
    "label": "Haiku 4.5",
    "cost_per_pr_cents": 1.4,
    "cost_200_prs_usd": "2.80"
  },
  {
    "label": "Sonnet 5",
    "cost_per_pr_cents": 3.64,
    "cost_200_prs_usd": "7.28"
  },
  {
    "label": "Opus 5",
    "cost_per_pr_cents": 9.1,
    "cost_200_prs_usd": "18.20"
  }
]

That na 1.4 cents per pull request for Haiku 4.5 and 9.1 cents for Opus 5. Team wey dey merge 200 pull requests every month go pay about $2.80 for Haiku 4.5, $7.28 for Sonnet 5, or $18.20 for Opus 5. From 1 September 2026, multiply the Sonnet 5 row by 1.5.

Two things fit make real bill pass this estimate. The synchronize trigger dey review every push, so active branch wey get eight pushes go cost eight reviews. The concurrency rule only help when the pushes land close together. The figures also assume say path filters dey work: one lock file wey no get filter fit double the input by itself.

Prompt caching no help for here. The cached prefix must be byte-identical between calls, and the diff dey change every time. The system prompt na the only stable part, but e dey below the minimum cacheable length. For the general rule, see when prompt caching dey pay for itself, and to choose between the three models above, which Claude model to use for which job.

print(client.messages.count_tokens(
    model=MODEL, system=SYSTEM, messages=[{"role": "user", "content": payload}]
).input_tokens)

Why review bot dey get muted, and how to avoid am

Two behaviours dey destroy trust for these bots, and both get fix for code above.

Reviewing everything at once. Bot wey leave forty comments, nobody go read any of dem. Severity threshold and ten-comment cap no be politeness; na dem dey keep real findings visible. If you sort by severity before you truncate, the cap go remove the findings wey matter least instead of any random ten.

Commenting with confidence on something e no fit check. Na this one make engineers switch am off for good. Model wey see 200 lines from 40,000-line codebase still fit write "this breaks the cache invalidation in redis_client.py" about file wey e never see. System prompt dey push back with plain language: report only defects wey dey visible for the lines wey dem show, and leave out anything wey you no sure about. If you name the failure directly, e work better than asking for general accuracy. Tell the model say empty result normal, and e no go invent something to say about a two-line change.

Post the review as COMMENT, never as REQUEST_CHANGES. Model opinion no suppose fit block a merge. Once e fit do that, person wey get deadline fit remove the whole workflow instead of arguing with am.

Failure modes, with the strings you go see

HTTP 422 when you dey post the review. gh dey print gh: Unprocessable Entity (HTTP 422) and the response body dey name the field: Pull request review thread line must be part of the diff. GitHub no fit anchor that comment. The usual causes na line number wey model invent, a path wey still get the b/ prefix, or comment for removed line, wey need side set to LEFT instead of RIGHT. Print the review JSON before you post am, then check one comment by hand against the diff.

invalid x-api-key from the model API. The step dey fail for the first messages.create call. Either the ANTHROPIC_API_KEY secret no dey set for the repository, or the pull request come from fork, so Actions no pass any secret at all. The fork guard for the if: line suppose skip am, so check that line first.

gh: Resource not accessible by integration (HTTP 403). The job token no fit write to pull requests. Add pull-requests: write to the permissions: block. If e already dey there, check Settings, then Actions, then General, where organisation policy fit limit wetin any workflow token fit request.

json.decoder.JSONDecodeError. The model no return JSON wey fit parse. The common cause na response wey reach the token ceiling and stop for middle of object. The log line dey print stop_reason exactly for this: value of max_tokens mean say make you raise max_tokens or lower MAX_COMMENTS.

The workflow no dey run. gh run list no show anything for the pull request. Check say paths-ignore no filter every changed file, then check the fork guard and label guard, then check whether runner dey alive with sudo systemctl status 'actions.runner.*' on the VPS. Offline runner go leave the job queued without error message anywhere for the pull request.

Every review dey come back empty. Set MIN_SEVERITY to low for one run. If findings show, the threshold dey work as e suppose. If nothing show, print payload and confirm say the filters no remove the whole diff.

E run amside your other agents

The reviewer small, so e fit tempt you make e run for the same box wey dey run everything else. Keep am separate if the repository important. This process hold token wey fit comment on your code and key wey fit spend your money, while self-hosted runner na deliberately place where workflow code dey execute. One dedicated unprivileged account wey no get sudo rights, for host wey no dey run anything else, na the baseline. If you still run interactive agents wey dey check out code, one disposable VM for each agent na the pattern wey dey work well, and how to run coding agent for VPS cover the general setup. If Anthropic API still new to you, your first Claude API app for VPS na smaller place to start than this.

FAQ

AI PR review agent need write access to my repository?

No. E need pull-requests: write to post review and contents: read to fetch the diff. Na the complete list be this, and na you dey set am inside the permissions: block of the workflow, wey dey limit wetin per-job GITHUB_TOKEN fit do. With those two lines, the agent fit comment on pull request but e no fit push commit or merge branch. Post reviews with event: COMMENT instead of REQUEST_CHANGES so e no fit block merge too.

Why my review comment dey fail with HTTP 422?

GitHub only accepts inline review comment for line wey dey inside the pull request diff, and e returns Pull request review thread line must be part of the diff when e no dey. Check say path dey relative to repository root, without b/ prefix from the diff header, and say the line number dey inside hunk for that file. side must be RIGHT for added or unchanged line, and LEFT for removed line. If you prefix every diff line with the new-file line number before you send am to the model, e go stop the model from inventing numbers from the beginning.

I fit run this for public repository with pull requests from forks?

No, this design no support am. GitHub no dey pass secrets to workflow wey fork trigger, so model key no dey and the run fails. GitHub still dey state say self-hosted runners "should almost never be used for public repositories", because anybody fit open pull request wey cause code to run for your machine. For public project, either restrict the reviewer to branches wey dem push to the repository itself, na wetin the if: guard dey do, or move the review step go GitHub-hosted runner and accept say the diff go comot from your own infrastructure.

Which model I suppose use for pull request review?

Start with Haiku 4.5. Reading bounded diff against fixed list of defect types no be hard reasoning problem, and the cheapest model go keep the monthly bill for amount wey nobody go argue about. Move up to Sonnet 5 if you notice say e dey miss real bugs for your language or framework, and measure am instead of assuming. Opus 5 cost pass the other two by far for each pull request, so e easier to justify for release branch than for every push to every feature branch.