Diff Every Tool Call: Replaying Agent Runs from a JSONL Trace

Production failed on Friday. My final transcript looked clean. The agent answered, cited sources, and summarized. The raw trace told a different story. It called the same endpoint three times with stale arguments. Re-running the agent wasted tokens and time. Replaying the trace took seconds. I built a diff-first replay harness. Logs became the source of truth for debugging. This post shows how to replay agent runs from JSONL traces. It also shows where a free server and a free model allowance fit in the loop. Re-running an agent is a roll of the dice. Temperature, tool latency, and cached state change every run. You pay tokens for each attempt. You also need live credentials and network access. Replaying from logs removes all three costs. The run becomes a static file. You inspect diffs instead of rerunning fate. Deterministic. Offline. Fast. Replay cannot fix missing logs. If you did not trace it, you cannot replay it. Start with logging. Everything else is downstream. Each line in my trace JSONL is one tool call. The schema is minimal. It stores run ID, step, tool name, arguments, an output hash, token count, and a timestamp. Enough to rebuild the call sequence and compare runs. {"run_id": "run_0042", "step": 1, "tool": "search_issues", "args": {"q": "broken:true", "page": 1}, "output_sha": "a1b2c3", "tokens": 812, "ts": "2026-08-31T09:12:04Z"} {"run_id": "run_0042", "step": 2, "tool": "search_issues", "args": {"q": "broken:true", "page": 1}, "output_sha": "a1b2c3", "tokens": 812, "ts": "2026-08-31T09:12:09Z"} {"run_id": "run_0043", "step": 1, "tool": "search_issues", "args": {"q": "broken:true", "page": 1}, "output_sha": "d4e5f6", "tokens": 817, "ts": "2026-08-31T09:20:11Z"} Run 0042 repeated the same call twice within five seconds. Run 0043 returned a different output hash for identical arguments. Both patterns are visible without re-executing anything. replay_diffs.py parses the trace and groups calls by argument signature. It flags identical repeats and cross-run drift. Then it replays each unique call against a mock executor. import hashlib, json from collections import defaultdict def sig(call): return hashlib.sha256( json.dumps(call["args"], sort_keys=True).encode() ).hexdigest() def load_trace(path): with open(path) as fh: return [json.loads(line) for line in fh if line.strip()] def diff_runs(run_a, run_b): a = load_trace(run_a) b = load_trace(run_b) by_sig = defaultdict(list) for call in a + b: by_sig[sig(call)].append(call) for key, calls in by_sig.items(): hashes = {c["output_sha"] for c in calls} if len(hashes) > 1: print(f"drift: {calls[0]['tool']} -> {len(hashes)} distinct outputs") Sample output: $ python replay_diffs.py trace_run_0042.jsonl trace_run_0043.jsonl repeat: run_0042 step 1 == step 2 (same args, same output) drift: search_issues -> 2 distinct outputs replay: 3 unique calls, 1 mock mismatch The script costs one SHA-256 per call. No model inference. No network. No credentials. It runs in milliseconds on a laptop. Replay is cheap. Shipping traces and classifying diffs is where costs accumulate. A trace collector must run 24/7. A summarizer needs model tokens to label each diff. MonkeyCode's free tier covers both stages in this setup. The project is open source. The current allowance includes 10M tokens, and the free server hosts the collector. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The collector is a 40-line FastAPI app plus SQLite. It stores every trace line. One endpoint returns the diff list between two runs. Check current terms before putting any free tier behind production traffic. Free allowances change. Do not read hundreds of diff lines by hand. Send them to a summarizer with a strict output format. Each diff returns one of three severities: expected, suspicious, or fatal. You receive a list of tool-call diffs between two agent runs. Return one line per diff: pattern, severity, suggested fix. Allowed severities: expected, suspicious, fatal. This is the only token-consuming stage. In the sample above, summarization cost a few hundred tokens. It is the cheapest part of the loop. Use this table when replay flags a diff. Pattern Evidence Fix Same args, same output, repeated Retry loop or duplicate call Cap tool calls per step Same args, different output External data changed Add freshness checks to context Different args, same tool LLM exploring too much Tighten tool budget or prompt Tool count spikes per step Agent browsing instead of deciding Reduce available tools Apply fixes in severity order. fatal first. Re-trace after each fix. The comparison window stays small, so feedback arrives in minutes. Run the agent with JSONL trace logging enabled. Ship each line to the collector on the free server. Run replay_diffs.py against the two latest runs. Ask the summarizer to classify all diffs. Fix the most severe diff. Re-trace. Compare again. Expect one or two iterations per defect class. The loop treats symptoms as data, not as failures. Replay only works if the trace is complete. Missing tool calls become invisible bugs. Mock executors drift from real APIs. Keep the mock minimal and update it with real responses. The summarizer can misclassify. Review fatal labels before changing code. Who should skip this approach? Teams with stateless single-shot scripts. If the agent never loops and never mutates state, replay adds ceremony. A plain log file is enough. Who benefits? Anyone running agents with tool budgets, retries, or external lookups. That is most agent workloads in production today. Start with one failed run. Add trace logging. Replay instead of re-running. The diff will show where the agent went wrong.