Devs Learning With AI Scored 17 Points Lower in Anthropic's Trial. Juniors Code Solo First

ve a developer an AI assistant while they learn a new library, and they finish about as fast as someone without one. Then quiz them on that library. In Anthropic's randomized trial, the AI group averaged 50%. The group that coded by hand averaged 67%.

That is a 17-percentage-point gap, and the biggest piece of it was on debugging questions. The speed difference was about two minutes and not statistically significant. My take: if your team hands new hires a coding agent on day one, you are skipping the weeks where they learn to debug. Juniors should write the first version of unfamiliar code themselves, use the agent to explain things, and earn delegation later.

The paper is How AI Impacts Skill Formation by Judy Hanwen Shen and Alex Tamkin, posted to arXiv on January 28, 2026. Anthropic's write-up describes the 52 participants as "mostly junior" software engineers. Everyone had used Python weekly for over a year, had tried AI coding assistants, and had never used Trio, a Python library for async concurrency. Each person got a short warm-up, then 35 minutes to build two Trio features: a timer that prints every second while other work runs, and a record fetcher that handles missing records.

Half could use an AI assistant during the Trio task. Half could not. Then everyone took a quiz: 14 questions, 27 points, covering seven Trio concepts through debugging, code reading and conceptual questions (paper, v2). The results, from the paper and Anthropic's post: AI group Hand-coding group Quiz average 50% 67% Task time about 23 min about 25 min Time difference not significant (p = 0.391) Quiz difference 4.15 of 27 quiz points (about 17% of the quiz), Cohen's d = 0.738, p = 0.010 The paper says the largest score gap was on debugging questions and the smallest on code reading.

It also counts errors. The median hand coder hit 3 errors during the task, the median AI user hit 1. Of the 12 people who finished both tasks with zero errors, only two coded by hand. That is the part I keep coming back to.

The AI group had a smoother hour and learned less from it. The authors suspect the errors were doing the teaching. The hand coders ran into Trio-specific mistakes the paper names: a RuntimeWarning for a coroutine that was never awaited, and a TypeError from passing a coroutine where Trio expected a function. Here is the first kind, in the shape of the study's timer task: import trio async def ticker(): for i in range(3): print("tick", i) trio.sleep(1) # bug: missing await async def main(): async with trio.open_nursery() as nursery: nursery.start_soon(ticker) trio.run(main) Without the await, trio.sleep(1) only creates a coroutine object and never runs it.

The ticks print back to back, and Python warns that the coroutine was never awaited. Figuring out why teaches you what a coroutine is. Pasting the warning into a chat window teaches you how to paste. The paper backs that up with its six interaction patterns.

Anthropic's post gives approximate averages for each: Iterative AI debugging (n=4): asked the AI to fix and verify code. Slowest group, about 31 minutes, quiz about 24%. Progressive AI reliance (n=4): a question or two, then handed over all the code. About 35%.

AI delegation (n=4): the AI wrote the code from the start. Fastest, about 19.5 minutes, quiz about 39%. Conceptual inquiry (n=7): asked only conceptual questions and fixed their own errors. About 22 minutes, quiz about 65%.

Hybrid code-explanation (n=3): asked for code plus an explanation. About 68%. Generation-then-comprehension (n=2): generated code, then asked follow-ups until they understood it. About 86%.

The lowest score in the table belongs to the people who used the AI to debug. The conceptual-inquiry group was the second fastest pattern overall, after AI delegation, and kept most of the learning. Asking "what does a nursery do" cost them almost nothing in time. Let the agent answer questions.

Let the human find the bug. Read this before you quote the 17 points in a planning meeting. Small sample. 52 people in total, 26 per arm.

The six patterns are subgroups of 2 to 7 people, and Anthropic's write-up says plainly that its pattern analysis "does not draw a causal link between interaction patterns and learning outcomes." The 86% group is two people. One hour. The quiz came right after the task. The paper notes real skill forms over "months to years", and nobody knows yet whether the gap closes, holds or grows.

A chat sidebar. The assistant was GPT-4o in a chat panel that could see the participant's code. The paper calls for research on agentic coding tools. Anthropic expects a larger effect from agents like Claude Code, though this study did not test one.

Not only juniors. The paper's balance table puts 29 of the 52 at seven or more years of coding experience. What every participant shared was being new to Trio. So the finding is about learning something unfamiliar, at any level.

Learning only. Anthropic's post points to earlier observational work with up to 80% time savings on tasks people already knew how to do. The caveats narrow the take: the risk is highest when someone is new to the thing they are building, and a junior is new to almost everything. Three stages, per area of the codebase, not per calendar month: First version solo.

For a new library, service or pattern, the junior writes the first working version without code generation. The agent is allowed in conceptual mode: "what does this error mean", "why would you use a nursery here". It does not write or fix the code. Then explanations on demand.

Once the first version works, the junior may generate code, but must ask follow-up questions until they can explain every line. This is the generation-then-comprehension pattern, the top scorer in the study (two people, so treat it as a hint). Delegation last. When the junior can debug that area unaided, the agent can write code there.

Debugging is the last thing to hand over. In the study, the four people who leaned on the AI to debug scored lowest of any pattern (about 24%), and debugging is where the quiz gap was widest. If your team uses Claude Code, its built-in Learning output style is a decent fit for stage 2. Claude handles routine code, explains its choices in Insight blocks, and leaves the design-decision pieces for you to write at a TODO(human) marker, then waits.

Switch with /output-style learning, or set it in .claude/settings.local.json: { "outputStyle": "Learning" } The docs say the value is case-sensitive, so learning in the file falls back to the default style. The docs also say Claude treats an output style as an instruction it may not always follow, so it won't enforce stage 1 for you. I build Hermes IDE, so weigh this section with that in mind. Hermes runs each agent in its own real terminal, and plain shells open the same way.

A junior can keep a shell for the solo version and an agent session for questions in the same window. Hermes 2.3.4: the bar above a Claude Code session, with the Commands list open. Since 2.3.4, the bar above each session puts Model, Plan and Review in the same place for every agent, and the Commands list includes your own commands and skills. A Claude Code command like /output-style still belongs to Claude Code.

Hermes just runs the agent's own interface. For stage 3, review matters more than the tool. Hermes saves every agent turn, and the Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows the changes by turn. A mentor can point at one turn and ask the junior to explain it, the same follow-up step the high scorers did on their own.

Hermes 2.3.4: the Review Desk by turn, with turn 1's diff and a commit message drafted from the turns. Hermes won't stop anyone from delegating on day one. Your team has to make that rule. It is source-available under BSL 1.1 and runs on macOS, Windows and Linux.

Download it here or read the code on GitHub. Hermes IDE is not affiliated with Nous Research or its Hermes Agent. Anthropic's write-up ends on the uncomfortable line: "Cognitive effort—and even getting painfully stuck—is likely important for fostering mastery." Teams are buying agents to remove exactly that. Do you let new hires use agents from day one, or do they write the first version alone?

And if you have watched someone learn to debug with an agent beside them, did it stick? Tell me in the comments. Interviews still grade the skill this study says AI can erode: reasoning through a problem yourself, out loud, without a tool to fall back on. My System Design Pocket Guide: Interviews walks through 15 real designs step by step.

Use it the way the study's best group used the AI: sketch your own design first, then read mine and argue with it.