# RunBoth

> RunBoth is a differential execution tool for Python. It runs both the old and the new version of
> a changed function on generated inputs, compares seven observation channels, and reports what
> actually behaves differently, including functions whose own source never changed. It works as a
> git commit-msg hook and as a GitHub Action, and it requires no test suite.

Site: https://runboth.dev
Source: https://github.com/runboth/runboth
Package: https://pypi.org/project/runboth/
Contact: hello@runboth.dev
Maker: Clouthier Simulation Labs (https://simgen.dev)
Licence: FSL-1.1-Apache-2.0. Free for every use except building a competing product, converting to
Apache 2.0 two years after each release.

## What it is, in one paragraph

An AI agent refactors your code and reports that nothing changed. RunBoth checks. It materialises
both commits, generates inputs for each changed function from its signature and from the literal
constants mined out of that function's own bytecode, runs both versions in separate sandboxed
subprocesses on the same inputs in the same order, and compares seven channels: return value,
exception raised, warnings, stdout, stderr, mutation of the arguments, and object state. When the
two disagree it reports the exact input that separates them.

## What it is not

- **Not a model checker.** Kani and CBMC translate code into logic, havoc the inputs, and ask a
  solver whether a bad state is reachable within a bound. They return a proof. RunBoth executes
  real code on concrete values. It can find a difference and reproduce it; it cannot prove absence,
  and it never claims to.
- **Not mutation testing.** Mutation testing damages your code to score your test suite. RunBoth
  damages nothing. Both versions come from git history, and no test suite is required.
- **Not an AI code reviewer.** There is no model in the product, no network call and no dependency.
  Every verdict is an execution you can re-run yourself.

## The three verdicts

- `changed`, with a witness: the exact arguments, the old result, the new result.
- `no_change at budget N`: N generated inputs found no difference across seven channels. Evidence,
  not proof.
- `abstained`, with a reason: it could not be checked, and why. Never counted as passing.

RunBoth never reports "safe". Distinguishing "cannot tell" from "no difference" is deliberate.

## Distinctive capabilities

- **Blast radius.** It walks the call graph outward from what changed and executes the callers,
  carrying the values already proven to differ. The caller's own source never changed, so no
  diff-based reviewer can see it.
- **Determinism gate.** The before version is run a second time in a third fresh process and
  compared against itself. If it disagrees with itself, RunBoth abstains rather than reporting a
  difference. This catches `uuid4`, `os.getpid`, `time.monotonic` and unseeded randomness.
- **Constant mining.** Inputs come from literals in the function's own bytecode, so a function
  containing `units >= 100` is tested at 99, 100 and 101, where the bugs are.
- **Honest silence.** Files it did not check, in other languages or in test and benchmark
  directories, are named explicitly rather than passed over quietly.

## Measured results (red team, 2026-09-12)

Eight public repositories the tool had never been tuned against, 2,548 functions adjudicated,
**zero false positives** under an automated oracle designed to catch the tool lying.

Independent measurement: Dristi and Dwyer, "A Differential Fuzzing-Based Evaluation of Functional
Equivalence in LLM-Generated Code Refactorings", arXiv:2602.15761 (February 2026), measured
LLM-generated refactorings as functionally non-equivalent 19 to 35% of the time across six models,
three datasets and two refactoring types. RunBoth's own audit of one frontier model over 36
functions landed at 44%, above that range and on a narrower sample. A control run of the same
instrument over 29 real human commits found zero witnessed behaviour changes.


| repository | layout | functions | abstained | median per commit |
|---|---|---|---|---|
| boltons | flat | 268 | 0.0% | 7.9s |
| sqlparse | flat | 230 | 0.9% | 14.3s |
| arrow | flat | 283 | 1.4% | 118s |
| cachetools | src/ | 325 | 1.8% | 60.9s |
| more-itertools | flat | 843 | 4.4% | 102.5s |
| packaging | src/ | 78 | 5.1% | 0.2s |
| pluggy | src/ | 157 | 6.4% | 33.5s |
| tenacity | async | 364 | 14.3% | 192.8s |

An adversarial corpus of 22 functions written specifically to induce false positives (object
addresses in default `repr`, `datetime.now`, unseeded `random`, `uuid4`, `os.getpid`, set iteration
order, mutable default arguments, module-level accumulators, generators, `__file__` paths, paths
inside exception messages) produced none.

## Install

    pip install runboth
    runboth install-hook

As a GitHub Action, running on your own runners:

    - uses: runboth/runboth@v0.1.0
      with:
        budget: 60

## Honest limits

- Function-level checking is Python only.
- Sampling finds differences; it cannot prove their absence.
- Functions that are nondeterministic, too slow, or need an object the tool cannot construct are
  abstained with a reason rather than guessed at.
- It is a process-level sandbox with resource limits, network blocked and filesystem writes
  blocked. It contains accidents. It is not a security boundary against hostile code, and no
  pure-Python sandbox is.