Articles

Proof, not claims: six verification rules from 17,468 AI agent commits

Six rules for CLAUDE.md and AGENTS.md from 17,468 commits by coding agents in 30 days, each with the measured failure that forced it, ready to paste into your own.

· 6 min read

Over 30 days (October 21 to November 20, 2026), our coding agents landed 17,468 commits across 2,483 tasks on a single main branch. That is a median of 512 commits a day, peaking at 1,061 in a single day, across a Rust, Dart, and Swift codebase.

At that pace nobody reads every diff, so whatever an agent reports has to be something a machine can check.

Every time an agent burned an hour or shipped a subtle false positive, we didn’t write a pep talk in the system prompt. We turned the measured failure into a hard engineering rule in CLAUDE.md (symlinked to AGENTS.md, so Claude Code, Codex and every other agent read the same rules).

Here are the six rules that saved us the most pain, each with the exact failure that forced it. You can adapt all six directly into your own CLAUDE.md or AGENTS.md.

A starved cargo check whose zero error lines were reported as green, against a cargo test that printed 'test result: ok. 128 passed; 0 failed', beside the article's six rules.

1. Ask for a positive token, never “zero errors”

The failure: A starved cargo check exited without finishing. It printed zero error: lines. The agent scanned for error lines, saw zero, and reported the build green. On another day, zsh expanded $C:tools/... as a history modifier, returned two empty file lists, compared 0 == 0, and declared two trees identical.

Why it happens: Most failure paths end at an empty value: 0, false, "", or an empty set. Checking “did anything fail?” cannot distinguish success from nothing ran.

The rule to paste into your CLAUDE.md:

Never treat the absence of errors as a pass. Require a positive completion token from the mechanism that owns the fact (Finished, test result: ok, or an exact blob from git show HEAD:<path>). A count below its expected floor is UNMEASURED, never equal or passing.

2. The missing state is always “I could not tell”

The failure: An agent wrote .status().map(|s| s.success()).unwrap_or(false) around a git ancestor check. When git failed to execute, unwrap_or(false) silently turned “git could not run” into “not a descendant”, freezing a pointer forever. In another script, a failed git status returned {}, which downstream read as “working tree clean” and folded another agent’s dirty file into the wrong commit.

Why it happens: bool and Option<T> are two-state types forced over a three-state reality: yes, no, and I could not ask. When “no answer” folds into the safe-looking branch, the bug is silent by construction.

The rule:

Never collapse an error or timeout into false, None, or {} unless a downstream record can still prove what happened. Make the third state (UNMEASURED or Unknown) explicit in the type. Prefer a self-describing record over a bare pointer, and an artifact on disk over a wall-clock timeout.

3. A test nobody has seen fail is a test nobody has tested

The failure: Agents love writing unit tests that pass on the first try, especially tests whose expected outcome equals the fail-safe fallback. You ship the guard, the test is green, and three weeks later a refactor orphans the guard completely while the test stays green forever.

What we do: Before trusting a new guard test, we run a mutation probe that saves a pristine copy of the file, collapses the guard with a targeted edit, runs the scoped test, and classifies the result into three states:

  • RED: the test actually caught the collapse.
  • VACUOUS: the test stayed green even with the guard deleted (worthless test).
  • UNMEASURED: no test result: line (didn’t compile or matched zero tests).

On exit, the probe merges back only its own mutation so concurrent edits by other agents aren’t clobbered.

The rule:

Pair every test so it fails if the code only takes the safe fallback. Assert the invariant a fix must preserve, never the symptom it removes. Never trust a guard test until you have watched it go RED with the guard collapsed.

4. A refactor is the operation that orphans a guard

The failure: An agent moved a call site during a refactor, leaving the old safety check function and all of its unit tests intact. Every test passed. grep still found callers (the tests themselves). Meanwhile, the live production path ran completely unguarded.

Why grep lies: grep cannot see transitive deadness. The only owner that knows whether production code still reaches a function is the compiler.

The rule:

Reachability is read from the compiler’s never used warning on a completed --all-targets check, never from grep. After any refactor that moves a call site, check for newly dead guards.

5. Claims are outputs, including error messages and landing reports

The failure: An agent wrote an error path that printed "Cause is in trigger.log" when nothing in that branch actually wrote to trigger.log. In another case, an agent saw its edit tool report +14 lines and reported that its change had landed on main, when those lines only lived in an unmerged candidate while HEAD still held the old file.

Why it happens: Proxies usually correlate with the truth right up until the moment they stop. A tool’s diff count describes the local candidate, not HEAD. A hardcoded error message describes what the author assumed would fail, not what the code actually observed.

The rule:

Claims enter the record only as the output of the mechanism that would falsify them. Read whether a change landed from git show HEAD:<path>, never from an edit tool’s line count. In error messages, say where to look; never assert what happened unless code actually observed it.

6. Separate high-velocity execution from irreversible authority

In a typical hour, 11 agent sessions land work on our main (31 at most). That works only if two things hold at once:

  1. Agents never block on each other’s ceremony. In our repo, every agent works directly on a shared main branch and hands its touched paths to one deterministic landing queue. Co-authored files split cleanly from edit journals; real overlaps fail loudly.
  2. Agents never cross an irreversible boundary alone. Sending an email, charging a card or deleting data pauses and asks a person (on the Mac or on the iPhone Lock Screen) while the rest of the crew keeps working.

The app behind these 17,468 commits

Those six rules are how we build Amia, a native Mac and iPhone workspace that runs Amia’s own agents alongside Claude Code, Codex, Cursor, Grok Build, Antigravity, and Muse Code as one crew.

Amia's Bookmarks on a Mac: the rules we give our agents, pinned and sent in one click, the first one open: "We're a startup… our number one rule is to keep things simple."
Real capture · Amia on a Mac
  • Every agent sits in one list (Needs you, Working, Done).
  • It works over every app on your Mac (⌥Space anywhere, a bar of actions on any text you select), and pays or signs in without ever seeing your card or password.
  • Agents can bring in other agents by name, so one can review another’s work (see how to use Claude Code and Codex together), and every step is on the record with what it spent.
  • For how Claude Code and Codex compared across 9,392 commits under these exact rules, see Claude Code vs Codex on one codebase.
  • Amia is free on your Mac with the AI accounts you already pay for, with up to 3 agents. The paid plans add more agents and a Cloud computer of its own, which runs Amia’s agents and Claude Code today (pricing).

Download Amia for Mac (Apple silicon, notarized) or join the iPhone beta on TestFlight.

Hand it off tonight.

Free with your own AI accounts · Plans from … a month