Collect evidence about changed behavior and affected callers

The message describes intent. Risk follows actual dependency edges. A named nearby test is not proof until it detects the proposed accident.

Follow-up 1 · Prove a named test

The tool says test_limit_zero catches an inverted condition. How do you verify that statement?

Follow-up 2 · The change crosses a service

The parser determines an outbound payload used by an independently deployed consumer. What evidence is missing?

Record the evidence and limitations

Build in three stops: reproduce the small case and baseline failure. Implement the protected boundary. Then replay both changed requirements with captured outputs. Record commands, fixtures, and observed results in your implementation README. A diagram is a prediction until those checks run.

Reproduce one real coverage gap and disprove one mistaken protection claim. Additional lead scope: Assign cross-service contract ownership and characterize collector blind spots. Completion demonstrates practice evidence. It does not establish interview readiness or multi-team delivery experience.

Detailed implementation and AI-assisted prompts

You end up with a repeatable interrogation for any diff, and one number from ten real ones: how often nothing would have caught an accidental change.

Build

A written procedure plus a small script that collects evidence and answers three questions of any diff: what behaviour changed, what could have changed accidentally, which existing test would catch the accident. Run it on ten real diffs from your history and tally how often the third answer is "none".

The thought process

A harness is not for replacing your reading. It is for making your tenth review of the day as good as your first. Attention degrades invisibly, and a procedure carries quality through fatigue — the reason experienced pilots still run the checklist. The commit message describes the intended change. The accidental one lives in the blast radius — every caller of a changed function, every other user of a touched helper or config key. That is greppable, so the script fetches evidence and holds no opinions.

The third question must end in a test's name or the word "none" — never "probably the auth tests". Both answers are checkable: break the behaviour and the named test must fail. Plant the accident and a true "none" leaves the suite green. If all ten diffs come back covered, ask what the answers would look like if coverage were bad — the same, and the harness is agreeing with you, not reviewing.

How to organise the prompts

1 — design the procedure against real diffs.

Here are three recent diffs from this repository. Draft the
procedure for interrogating any diff: for each of the three
questions, the mechanical evidence that answers it — commands, not
judgment. The third answer must be a test name or the word NONE.

The procedure must name commands you can run. Any step beginning "consider whether" is judgment smuggled in as evidence, so send it back.

2 — build the collector.

Write the evidence collector: given a diff, list the changed
functions, their callers, every other file using a helper or config
key this diff touches, and the tests exercising any of those.
Output file:line lists only. No prose, no conclusions.

Run it on a diff whose blast radius you already know — it must find the caller you know about, or it will never surface the ones you do not.

3 — the ten runs.

Here is diff 4 of 10 and the collector's output. Answer the three
questions; every claim must cite file:line from the evidence. If no
existing test would catch the accidental change, write NONE — not
the nearest test.

Spot-verify two NONEs by making the accidental change for real: a green suite means the NONE was true and the tally is data.

On AWS

It can run where the diff lives: GitHub Actions, on every pull request. Estimate runner usage for the current repository plan. AWS enters only if the harness calls a model per diff — then route it through Bedrock with invocation logging on, so each review has a visible cost in CloudWatch. A review bot nobody meters gets quietly expensive. And whatever runs it gets read-only credentials: it comments, it never merges.

What productionising it means

The tally is the real product: a none-rate over time, saying whether the suite grows with the code or falls behind it. The failure mode is ritual — people reading the harness instead of the diff — so it cites evidence and asks questions, never concludes "looks good". Alarm on the none-rate rising. That is the trend it exists to catch.

The learning

The dangerous part of a change is the part the message never mentions, and "none" is the most informative answer a review can produce — untooled reviews almost never do. Ten diffs teach you your real safety margin as no coverage percentage has.

How you would know it is wrong

• Plant an in-passing edit to a shared helper in a test diff. If question two does not list it, the blast-radius logic is decorative.
• Verify a named test the way you verify a NONE: break the behaviour. That test, specifically, must fail.
• Run the harness twice on one diff. Materially different answers mean a rumour generator. Pin every claim to collector output.
• Ten out of ten covered. In my own record, answers shaped that much like good news have usually been the instrument — check it before you believe it.

Back to the ordered project index