The Most Dangerous Bias of Your AI Assistant Is That It Agrees with You – Part 2: Why We Also Need to Remove Rules Again

The first part of this series was about diagnosis: Part I What I deliberately left open there was this: What happens to those suggestions afterwards? How does “the AI agreed too quickly today” become a rule according to which it actually works differently tomorrow? And the part that is discussed even less: What happens when nobody needs a rule anymore for months? That is exactly what I thought through to the end in this session with my assistant. The result is a complete lifecycle for rules: emerge, classify, apply, fade, and, when necessary, return. The analysis layer from Part 1 produces a suggestion file at the end of the session with four sections: new rules, confirmed rules, drift warnings, recommendation. Each new rule also receives a suggested weight value (1–3; more on that shortly) and a frequency counter. Directly below each new rule suggestion there is a block that the analysis layer always leaves empty: **Classification:** - [ ] 🏛️ Global rule - [ ] 📁 Project rule → Project: _______________ - [ ] ⚖️ Weight confirmed (for weight 3: mandatory) This is not a minor detail. The analysis layer may make an observation (“agreement came too quickly here”), but it must not decide how far that observation reaches. Does it extend beyond the current project, meaning “this is how the assistant should generally behave”? Or is it bound to a specific context, meaning “this particular constraint applies in this project”? A pair of examples makes the difference tangible: “For architectural decisions, always formulate at least one opposing position before agreeing” → Global rule. This is a behavioral expectation, independent of context. “In this repository, property names are being migrated to camelCase; PascalCase in new mappings is an error, not a style choice” → Project rule. Outside this project, it would simply be wrong. I make that decision. Only afterwards are the rules transferred into the respective target file: global rules into an overarching rule set, project-specific rules into the project file. During the research, it became clear to me why this separation is not merely sensible “out of caution”, but actually mandatory — and this has to do with precisely the mechanism described in Part 1. Research on sycophancy (Perez et al. 2022, Sharma et al. 2023, both Anthropic) shows that the tendency toward agreement is not a glitch, but a training artifact — encouraged by training on human preference data, and generally more pronounced in larger models. A model that tends to agree would also tend to classify suggestions that restrict its own behavior more generously rather than narrowly. “Does this really apply globally, with all the consequences that entails?” is exactly the kind of question where an agreement-prone system tends toward the more convenient answer. That is why classification is the only step in the entire process that I keep fully manual. The analysis layer may suggest, categorize, and weight. But the decision “this now applies to every session, in every project, permanently” remains with me. And there is a second point where the same logic applies — one that only became obvious to me when I thought the archiving mechanism through to the end: the weight itself. Weight 3 permanently protects a rule from archiving (more on that shortly). But if the analysis layer were allowed to assign weights entirely on its own, the model would control the mechanism that makes rules immortal. An agreement-prone system that tends to consider its own observations important would assign weights too high rather than too low — and the rule set would gradually immunize itself against exactly the kind of curation it needs. That is why weight 3 is a suggestion, not a determination: the third checkbox in the classification block exists because archive immunity must be a human decision. This is not a declaration of mistrust toward the model. It is the direct consequence of the fact that precisely this kind of decision would be most affected by the drift that the system is supposed to combat. This limit was already present in Part 1, but it deserves a more precise justification because it solves two problems at the same time. First: overfitting to a single session. A long, intense session (like this one, with skill reviews, a reconstructed meeting protocol, and a discussion about rule archiving) inevitably produces many observations. Not every observation is a rule. Some are one-off incidents (an incorrectly named folder), some are too specific to generalize. Without a limit, the rule set would overrepresent the peculiarities of this one session simply because it was recent. Second — and this is the truly hard point: review capacity. I have to read, assess, and classify every new rule. That is not a formality; it is the core of the human-in-the-loop design from Part 1. A system that suggests 15 new rules per session will lead to me skimming rules or waving them through wholesale — and precisely at that moment, the control instance becomes what it was meant to prevent: a confirmation machine. The five-rule limit is not calibrated to model performance, but to my attention span for careful review. That is explicitly intentional. If a session produces more than five candidates, the five most impactful ones must be selected. The rest does not disappear because it is unimportant, but because it will either show up again next time (and then count), or it was indeed too situational to justify a general rule. This was the actual trigger for this session: a very simple question — “will the rule list now grow forever?” — and the answer was yes, unless countermeasures are introduced. Global rules apply always, in every session, independently of the project. That is their strength and, at the same time, their risk: every global rule is context that is given to the assistant in every single session, even if it has not been relevant in any session for months. A rule that emerged a year ago from a one-off incident and has never applied again since is, with high probability, either superseded by another rule, made obsolete by a workflow change, or was too specific to be “global” from the outset. In all three cases, it has become noise that dilutes real, active rules. That is why every global rule now gets a last_seen date: the point in time when it was last confirmed in a session. A rule is confirmed through the second section of the suggestion file — “confirmed rules”. If the analysis layer detects at the end of a session that an existing rule was relevant in that session (followed or violated; both count), it updates that rule’s last_seen and increments the counter. A rule that never appears in this section again ages. The logic: Weight 1 or 2 and last_seen older than 28 days → the rule moves to the archive (status: archive). It does not disappear; it is simply no longer actively carried along. The 28 days are not a law of nature, but an initial heuristic: four weeks cover a full work cycle for me, including topics that only come up every few sessions. If it turns out that rules are regularly archived and then have to be reactivated shortly afterwards, the threshold is too short — that is a parameter the system must calibrate through observation, not through decree. Weight 3 (critical) → never archived, regardless of date. A rule can be important without triggering frequently. Frequency and importance are two independent axes; that was already central in Part 1. A critical rule that only applies once per quarter, but would cause significant damage if violated, must not disappear because of inactivity. (And because weight 3 therefore means archive immunity, it is — as described above — assigned only by me, never by the analysis layer alone.) Reactivation: If an archived pattern appears again later, the analysis layer suggests reactivating the archived rule instead of placing a new, slightly differently worded rule next to it. This, too, is a suggestion in the output file, not a silent status change — otherwise the system would have a backdoor through which archived rules could return to the active rule set without review. Without this mechanism, however, archiving would lead to duplicates as soon as an old problem reappears. Project rules are deliberately excluded from this. Three reasons: Lower volume. A project accumulates a manageable number of specific constraints over its lifetime, not 30+ rules that would have to compete with and displace each other. Natural lifecycle. A project rule is tied to the project. When the project is completed or archived, the rule loses its relevance anyway. No separate expiration mechanism is needed for that; the project status handles it. Context specificity is not a risk here, but the point. A global rule that has not triggered for months is suspicious. A project rule that has not triggered for months is simply a rule for a part of the project that is currently dormant. That is normal and not a signal of obsolescence. The distinction can be summarized as follows: global rules are a shared resource pool in which every entry costs context for all future sessions. That pool must be actively curated. Project rules are locally bounded annotations whose scope is defined by the project itself. Looking back at both parts, the actual topic is not “how do I prevent the AI from agreeing with me”, but rather: Every mechanism intended to counteract drift needs an instance that reviews it, limits it, and, when necessary, rolls it back. The analysis layer reviews the session, but a human reviews the analysis layer. New rules improve behavior, but a limit prevents one session from dominating the rule set. Global rules apply everywhere, but an expiration date prevents them from accumulating uncontrollably. And the weight determines archive immunity, but assignment of the highest level remains with the human. None of these mechanisms works without the others. A rule set without a limit would collapse under a single session. A rule set without an expiration date would slowly suffocate under its own weight. And a rule set whose classification is left to the model would reproduce exactly the drift it is supposed to correct. That is, I think, the real lesson from these two sessions: a “self-improving system” is not a system that improves itself. It is a system in which human control is applied precisely where the system is least trustworthy. And that control has to remain practical enough that it actually happens.