When Your AI Reviewer Remembers Too Much: A Two-Phase Memory Probe

Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions. This article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all. Review agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw. Hiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise. fixture/ ├── docs/decisions/0001-metrics-pipeline.md # accepted 2026-07-02 ├── docs/decisions/0012-rename-to-telemetry.md # accepted 2026-08-14 ├── src/metrics_service.py # legacy module, 120 lines ├── src/telemetry_service.py # replacement module, 140 lines └── pyproject.toml # lint: E501 disabled for telemetry only The fixture encodes a deliberate conflict: the team renamed the metrics pipeline to telemetry in decision 0012, while the legacy module still exists on the main branch. A memoryless reviewer sees only current code and never learns about the rename. A memory-bound reviewer should retrieve decision 0012 and apply it to both phases. The first PR adds retry logic to telemetry_service.py and touches pyproject.toml; it is intentionally boring. Its real job is to let the candidate observe the repository history, read both decision files, and form a picture of its conventions. Nothing in this phase is graded. The second PR deletes metrics_service.py, promotes telemetry_service.py to the canonical module, and adds a seeded bug: transmit sends an empty payload without a guard and raises ValueError at runtime. A correct review must block on the missing guard while accepting the rename and citing decision 0012. You are reviewing PR #14 against the fixture repository. Convention source priority, highest first: 1. docs/decisions/*.md and DEPRECATED.md 2. recently merged PR descriptions and issue threads 3. the historical code being replaced Return three sections: - blocking: correctness issues with file:line references - consistency: conflicts with current decisions - uncertain: claims you could not verify Do not flag a difference from deleted code as a regression unless the deleted behavior is still enforced by a live decision file. Cite the exact path for every convention claim. Blocking: transmit(payload) must guard against None or empty payload before calling client.send; an early raise ValueError is the expected fix. Consistency: the rename is correct per ADR-0012, and deleting the legacy module is expected rather than churn. Uncertain: none required; a strong review may ask whether downstream callers were migrated before the old module disappears. Verdict Score band Review signature 200 OK 90-100 Seeded bug found; rename accepted; at least one decision citation; no stale-convention complaints 301 Moved Permanently 60-89 Change recognized, but the rename is flagged as unnecessary churn; the bug may be found or missed 409 Conflict 30-59 Review asserts the old namespace is canonical and contradicts ADR-0012 404 Not Found 0-29 Seeded bug missed; summary contains no file-level claims The HTTP mapping makes each verdict easy to communicate to a hiring panel and hints at its operational meaning. A 404 bot will miss regressions in production; a 409 bot will block valid migrations until its cache is reset. Scores of 90 or above indicate the candidate can reconcile memory with current ground truth. #!/usr/bin/env bash # run_memory_probe.sh — two-phase AI reviewer probe (abridged reference harness) set -euo pipefail REVIEWER_CMD=${1:?pass the reviewer CLI, e.g. "monkeycode review --json"} FIXTURE=${2:?pass the fixture repo path} WORKDIR=${WORKDIR:-/tmp/memory-probe-$(date +%s)} git clone --quiet "$FIXTURE" "$WORKDIR/repo" cd "$WORKDIR/repo" # Phase 1: boring retry refactor; lets the bot read repository history git checkout -q -b phase1-retry # ... apply the retry commit from the task kit ... "$REVIEWER_CMD" --base main --head phase1-retry > "$WORKDIR/phase1.json" # Phase 2: rename plus seeded bug, both hidden in one diff git checkout -q main git checkout -q -b phase2-probe # ... apply the probe commit from the task kit ... "$REVIEWER_CMD" --base main --head phase2-probe > "$WORKDIR/phase2.json" python3 score_memory_probe.py "$WORKDIR/phase2.json" "$WORKDIR/phase1.json" The companion scorer is deliberately simple and keyword-based; adapt it to the candidate's output schema. #!/usr/bin/env python3 import json, sys phase2_path, phase1_path = sys.argv[1], sys.argv[2] review = json.load(open(phase2_path)).get("review", "").lower() score = 0 if "transmit" in review and ("payload" in review or "guard" in review): score += 40 # seeded bug located if "0012" in review or "telemetry" in review: score += 30 # live decision retrieved if "metrics_service" not in review: score += 30 # no stale-convention complaint verdict = "200 OK" if score >= 90 else "301" if score >= 60 else "409" if score >= 30 else "404" print(json.dumps({"score": score, "verdict": verdict})) Anchoring on the familiarization round: the first PR shapes the second verdict even when the repository changed in between. Stale decision retrieval: the bot recalls decision 0001 and never re-reads 0012, producing confident wrong consistency claims. Volume-based confidence: the legacy module has 120 lines and the replacement has 140, so the bot treats line count as authority. Politeness over rigor: the review praises the migration, returns a green verdict, and misses the empty-payload bug entirely. Every failure mode here is observable in the rubric output, which is exactly why the probe, rather than a free-form sample review, earns its place in an evaluation pipeline. The probe measures one behavior: which convention source wins when history and current state disagree. It does not measure security skill, response speed, or the ability to read a two-thousand-line diff; keep decoy-PR and prompt-injection tests in the pipeline for those axes. Teams evaluating a stateless API reviewer with no memory configuration, or repositories with no written decision records, will get no signal from this task because there is no ground truth to score against. The fixture is tiny by design, so memory regressions that need weeks of real context can still escape the probe; treat it as a hiring gate, not a certification. Executing the probe needs two resources: a place to host the fixture and a model endpoint that accepts the review prompt. MonkeyCode is an open-source project that targets exactly that configuration, with a free server option for the workspace and a free model allocation that currently includes 10 million tokens. Those numbers were current on 2026-08-31, and free-tier details change quickly, so the project documentation remains the source of truth. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Because MonkeyCode is open source, a team can inspect how the server is implemented and how the allocation is documented before routing real review traffic through it. The two-phase probe is a sensible first workload for that inspection because it is short, reproducible, and fits inside the free budget. Run it before the next reviewer rollout, and share the rubric output with the team.