Old Coder skill: review evidence, not code

What the Old Coder skill actually changes

The Old Coder skill replaces code review with document review. Your coding agent writes a SPEC before it writes any code, you approve that document, and only then does it implement. It then runs its own work through a fixed stack of automated checks called the gauntlet, and hands you an EVIDENCE report holding the exact commands and the actual numbers. You read two documents. You never read the diff.

That trade only works if the two documents carry the trust the diff used to carry. The SPEC carries it because you approved it before any code existed, so it could not be shaped around code the agent had already written. The EVIDENCE report carries it because every number in it comes from a command you can run yourself and watch produce the same number. If either half goes soft, you have swapped review for a summary of review, which is worse than reading the diff because it feels finished.

The skill is plain markdown, so it works with any agent that follows written instructions: Claude Code, Codex CLI, Cursor, or your own loop. It belongs to the same family as the Ponytail lazy senior developer persona skill. If the file format is new to you, what an agent skill is and how an agent loads one covers the mechanics.

The SPEC is the one decision you still make

The SPEC is a test plan written before the code exists. The skill file requires four things in it.

• Concrete scenarios: inputs, expected outputs, edge cases, error cases. divide(1, 0) raises ZeroDivisionError with message X, not "handles bad input".
• Negative constraints: what must not change, such as existing tests and public API signatures.
• A setup plan: every tool and every new dependency, each with one line saying why it is there.
• An absolute file path, so you can open the file without hunting for it.

Approve it, then commit it. A spec that can be edited after approval is not a contract, and the commit is what lets you check later that the evidence was measured against the thing you signed.

This is the only yes or no decision left to you, which is the point and also the risk. Treat it like a change to production, because that is what it is. If you already run an explicit approval gate in front of agent actions, spec approval slots into the same place in your workflow.

The EVIDENCE report: numbers with commands attached

The EVIDENCE report is what you read at the end. The skill requires each spec behaviour mapped to the test that verifies it, each gauntlet layer reported with the command that ran and its actual result, every number taken from one fresh run after the last code edit, and every skipped layer listed with its reason. Adjectives are not evidence. "All 41 tests pass, coverage 49/49 statements" is a result. "Well tested" is not.

The demo report in the repository, demo-rate-limiter/evidence.md, identifies its source state as a commit plus a sha256 tree hash. That line matters more than it looks, because it tells you which exact bytes produced those numbers. Without it a report can quietly describe a working tree that no longer exists.

Three anti-gaming rules keep the report honest, and the skill file states them as absolute. Never weaken a test to make it pass: no broadened assertions, no raised tolerances. Never edit a test and the implementation in the same step to reach green, because a simultaneous edit hides which one was wrong. Never report a layer that did not run: "skipped, no tool available, manual mutation instead" preserves trust, and an invented result destroys the whole scheme.

Install the skill, and pin the commit you installed

The repository is AmazingAng/old-coder, MIT licensed. Its README gives a one line install through the skills CLI:

npx skills add https://github.com/amazingang/old-coder

That installs whatever main holds at the moment you run it, and the CLI itself is a moving part. Check where the files landed before you assume the skill is active:

ls ~/.claude/skills/old-coder/

You should see SKILL.md and a references/ directory. Nothing there means the skill is not where Claude Code looks for it. Older releases of the skills CLI wrote into ~/.agents/skills/ without linking the result into ~/.claude/skills/, so the files existed on disk and the agent never loaded them. Running npx skills@latest add ... avoids the stale-CLI case.

Prefer the manual path, because it lets you record what you installed:

git clone https://github.com/AmazingAng/old-coder.git
cd old-coder
git checkout acc5a89
git rev-parse HEAD
mkdir -p ~/.claude/skills
cp -r skills/old-coder ~/.claude/skills/

acc5a89 was the tip of main on 17 August 2026. Pick your own and write it down. The repository is under active development, and the gauntlet reference, the templates and the verifier protocol have already moved between files. If your EVIDENCE reports do not say which version of the skill graded them, you cannot tell a change in your code from a change in the rules. Store that commit hash next to the SPEC, in the same repository as the code it governs.

For an agent that does not read ~/.claude/skills, add skills/old-coder/SKILL.md and skills/old-coder/references/gauntlet.md to its system prompt or rules file. That is the whole integration.

What runs inside the gauntlet

The gauntlet is a stack of layers, run once every spec behaviour is green. The skill names these:

• The full test suite, for regressions. Zero new failures, with any pre-existing failures recorded as a baseline first.
• Static types, plus lint and format, for whole classes of bug and for drift.
• Changed-line coverage, which must exit nonzero when the threshold is missed.
• Mutation testing, for tests that assert nothing.
• Property-based tests, for the edge cases nobody imagined.
• A complexity budget, one real execution of the actual program, a supply chain and secrets scan, and a suite-health run in randomized order.
• Domain layers chosen from the task's risk: concurrency stress, API compatibility, rollback rehearsal, latency benchmarks.

The demo wires these into one script, demo-rate-limiter/tools/gauntlet.sh, against a pinned toolchain in requirements-dev.txt: pytest, pytest-cov, coverage, hypothesis, mypy, ruff, pip-audit and pytest-randomly, each pinned to an exact version (pytest 9.1.1, ruff 0.16.0, as of August 2026). Run it:

cd demo-rate-limiter
python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt -e .
./tools/gauntlet.sh

The script prints one banner per layer, such as === tests + coverage === and === mutation ===, and ends with === gauntlet: all layers green ===. It runs under set -e, so the first broken layer stops it and that final banner never prints. Seeing the final banner is the check: it means every layer above it exited zero.

One detail in that script is worth copying into your own. The coverage layer is written like this:

pytest -q --cov=ratelimiter --cov-report=term-missing --cov-fail-under=100

Without --cov-fail-under, pytest --cov prints a percentage and exits 0 no matter how far coverage has fallen. That is a fail-open layer inside a script whose first line promises to stop at the first broken one. A gauntlet layer that cannot fail is decoration.

What mutation testing adds over coverage

Coverage answers one question: did the suite execute this line. It cannot answer the question you actually care about, which is whether any assertion would have failed had the line been wrong. A test that calls a function and checks nothing still reports full coverage on every line it touches. Coverage detects untested code. It does not detect tests that test nothing.

Mutation testing answers the second question directly. It changes the code on purpose, one small edit at a time, and reruns the suite. If the suite fails, the mutant is killed, which means some assertion was watching that behaviour. If the suite still passes, the mutant survived: the line ran, and nothing checked the result.

The demo does this with tools/mutants.py. It plants numbered single edits into src/ratelimiter/__init__.py, runs pytest after each one, then restores the file. The edits are the mistakes a tired human makes: >= becomes >, or a return value is dropped.

The kill rule in that runner is the part most home-grown mutation scripts get wrong. Only pytest exit code 1 counts as a kill, because 1 means the tests ran and at least one failed. Exit code 0 means the mutant survived. Anything else, a collection error or no tests collected at all, means nothing was verified and must not be counted. A script that treats "nonzero" as a kill scores its own crashes as successes, and that number only ever moves upward.

Two more honest details sit in the same file. One mutant, M11, is left out of the list because it is equivalent: pruning one expired entry instead of all expired entries produces the same observable behaviour under a monotone clock, so no test can ever kill it. And the runner sets PYTHONDONTWRITEBYTECODE=1 while the gauntlet deletes every __pycache__ first, because two mutants of identical size written in the same second can share a cached .pyc, and then the second one inherits the first one's verdict.

That hazard is why the gauntlet runs a negative control before the real mutation pass:

.venv/bin/python tools/mutants.py --negative-control
.venv/bin/python tools/mutants.py

The control runs two mutants under a pinned modification time: one that must be killed, and one that is strictly equivalent and must survive. If both come back killed, the bytecode cache leaked between runs and every kill count in the report is inflated. This is the skill's rule about checkers made concrete. pytest and mypy have earned their failure behaviour over many years. A script you wrote last week has not, so prove it can fail before you trust it when it passes, and record that proof in EVIDENCE.

[
  {
    "label": "Scenario tests",
    "mutants_killed": 22,
    "mutants_run": 22
  },
  {
    "label": "Property tests",
    "mutants_killed": 3,
    "mutants_run": 22
  }
]

The demo's own EVIDENCE report shows why a single aggregate number still hides things. The scenario suite killed 22 of 22 mutants. The property-based tests, rerun alone against the same mutants, killed 3. A kill is attributed to whichever test fails first, so a perfect total validates the suite as a whole and says nothing about any one layer inside it. Those properties still earn their place, because they catch input shapes nobody enumerated. They are not carrying the correctness load, and you only learn that by measuring each layer on its own.

Run the gauntlet on a server, not on your laptop

An EVIDENCE report is a claim that some commands produced some numbers. The claim is only checkable if someone else can produce the same numbers, and a laptop is the worst place to try. Your Python is a different patch release, and your PATH carries tools the next machine will not have. Mutation testing makes it worse, because it reruns the whole suite once per mutant, so the demo's list alone means 22 extra suite runs.

Put it in a container on a VPS. The container fixes the operating system and the interpreter, the pinned requirements-dev.txt fixes the tools, and the VPS gives you a machine that is not also running your browser.

FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
      python3 python3-venv git ca-certificates \
 && rm -rf /var/lib/apt/lists/*
WORKDIR /work

docker build -t gauntlet:24.04 .
docker run --rm -v "$PWD:/work" -w /work/demo-rate-limiter gauntlet:24.04 \
  sh -c 'python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt -e . && ./tools/gauntlet.sh'

A healthy run ends on === gauntlet: all layers green ===. Two steps need outbound network: pip installing the pinned tools, and the pip-audit layer, which checks your dependencies against a vulnerability service. An offline run does not skip that layer quietly. It fails, which is the behaviour you want from a gate.

For a run on every push, the same script becomes one CI job step. The repository runs its own gauntlet on GitHub Actions on ubuntu-latest with Python 3.12. Point runs-on at a self-hosted runner and the job executes on your VPS instead:

name: gauntlet
on:
  push:
    branches: [main]
jobs:
  gauntlet:
    runs-on: self-hosted
    defaults:
      run:
        working-directory: demo-rate-limiter
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: python -m venv .venv && .venv/bin/pip install -r requirements-dev.txt -e .
      - run: ./tools/gauntlet.sh

Keep the trigger on pushes to branches you control. A self-hosted runner that also builds pull requests from forks runs a stranger's code on your server with your runner's credentials. The same reasoning applies to the agent itself: give it a disposable VM you can throw away rather than your workstation. If you want the gauntlet result to arrive where the change is discussed, wire it into a self-hosted PR review agent.

Where this approach fails

The first failure is structural, and no amount of tooling removes it. The gauntlet turns the constraints in your SPEC into executable evidence. It cannot tell you the SPEC was right. Approve a spec that encodes the wrong requirement and you get a flawless EVIDENCE report for the wrong program: full coverage, every mutant killed, every layer green, and software that does something you did not want. Every hour you save by not reading the diff should go into reading the spec.

The second is the checkers. The repository is honest about this in its own evidence file. Its independent verification protocol ran six rounds, the sixth returned failed, and fixes made after that round were never re-verified, so the state that ships was not verified end to end. A shell lint layer is recorded as unavailable rather than as a pass. Earlier rounds found real behavioural defects and an unsound mutation runner sitting behind states that had already reported green. A green gauntlet does not authenticate itself.

The third is scope. A hand-written mutant list only covers the bugs somebody thought to plant, and every equivalent mutant pruned from that list is a judgement you are trusting. Off-the-shelf mutation tools (mutmut, cosmic-ray, Stryker, PIT) generate mutants systematically and are the better default wherever your language has one.

Calibrate the effort to the risk

The skill defines three tiers and expects the agent to declare which one it picked.

• Tier 1, trivial: a typo, a comment, a config value. Full suite plus lint, no new tests, with one sentence on why none were needed.
• Tier 2, normal: a bug fix or a small feature. The full loop, and a bug fix must start from a failing test that reproduces the bug, so yesterday's bug becomes tomorrow's regression test.
• Tier 3, high stakes: money, authentication, data loss, concurrency, a public API. Start from a failure model listing the ways this change can hurt, add a gauntlet layer for each mode, then run the full loop plus property tests, mutation testing, and one explicit pass attacking the implementation with hostile inputs.

Tier 3 also carries an experimental step. A second agent with a fresh context sees only the task contract, the approved SPEC and the source state, and tries to break the finished work before EVIDENCE is signed. It fixes nothing, it reports, and a human grades what it found. It reduces the correlation that comes from sharing a task context. It does not reduce the correlation that comes from sharing a model.

If you would rather build something in this shape than adopt this one, writing your own agent skill covers the file layout and the description field that decides when the agent loads it.

FAQ

What does mutation testing catch that code coverage misses?

Coverage records that a line ran. It cannot record whether any assertion would have failed if that line were wrong, so a test that calls a function and checks nothing still reports full coverage. Mutation testing changes the code on purpose, one edit at a time, and reruns the suite. A surviving mutant means the line executed and nothing verified the result. That is why the skill lists "never chase coverage numbers" as an absolute rule and names mutation as the layer that catches the gaming.

Do I still need to read the code my agent writes?

Under this workflow you read the SPEC before coding and the EVIDENCE report after, and you spot-check the report by rerunning the commands it cites. The diff becomes optional. The catch is that all of your judgement now lands on one document, because a spec with the wrong requirement in it produces a green gauntlet for a program you did not want. Spend the time you saved on the spec.

Can I run the Old Coder gauntlet in CI on my own server?

Yes, and that is the better place for it. Install the pinned toolchain from requirements-dev.txt inside a container and run the project's gauntlet script as one job step. On GitHub Actions, set runs-on: self-hosted and register a runner on your VPS. Keep the trigger on pushes to branches you control, because a self-hosted runner that builds pull requests from forks executes untrusted code with your runner's credentials.

Which version of the skill should I install?

Pin one. The repository is under active development and its reference files have already been split and moved, so a report produced last month may have been graded against different rules. Clone the repository, check out a specific commit, copy skills/old-coder into ~/.claude/skills/, and record that commit hash next to your SPEC. Then a change in your evidence means a change in your code.