I ran OpenAI's security audit 10 times: to review your own code, plain ChatGPT is enough

There's a pattern that keeps showing up on the timeline: someone vibecodes an app over a weekend, ships it, it does well, and a week later they're left with the nagging question — "is this thing secure?" And right on cue, in comes OpenAI's pitch: Daybreak Blue, the defensive lane of their cybersecurity stack, "for secure code review, vulnerability discovery, and patch validation." Which is, exactly, what that dev needs. I have access to Blue. So instead of taking the announcement at its word, I ran the experiment: I grabbed a real app, handed it the same request, word for word, to plain ChatGPT and to Blue, and compared. The question this answers for anyone who vibecodes is simple: to secure your app, does the special tier do anything that regular ChatGPT doesn't? I ran it once and got a clean, shareable answer. I almost published it. Good thing I ran it nine more times. Daybreak Blue isn't a different model: it runs on the same base model (Sol) with the cybersecurity safeguards loosened for authorized work. The one that does ship a separate model is the other tier, Red —GPT-5.6-Cyber—, which isn't available to individuals. In the Codex build I was granted, the program is picked with a flag, --cyber-access-program, that I set to standard (regular ChatGPT) or daybreak_blue; it doesn't appear in OpenAI's public documentation, I know because it's the one I run. So I audited the same base model, turning only that one knob. That's the only variable —at least the only one OpenAI says changes: from the outside I can't audit that Blue doesn't also touch the system prompt or the scaffold, so I take it for what it is, the vendor's word—. And heads up, this changes what you can reproduce: you don't get into Blue by ticking a box. You need a paid plan with Advanced Account Security, a registered physical FIDO2 key —yes, a YubiKey; that was the first post in this series—, identity verification with a selfie and ID at chatgpt.com/cyber, and a justification for what you'll use it for. OpenAI reviews case by case and can turn you down: completing the setup doesn't guarantee access (the official process, here). So in this experiment you can reproduce the standard side —that's the ChatGPT you already have— but not necessarily the daybreak_blue one. Target: anchor-cms, a PHP CMS, 3,300 stars, in real use but abandoned since 2018. Router, auth, and database layer all hand-rolled —the place where real bugs live, and also the easiest possible case for any model: old PHP, no framework covering anything up—. Same prompt for both, read-only sandbox (it reads the code, no network, no running the app), same machine, same commit. Here's the exact prompt, in case you want to run it on your own app: You are a security analyst doing defensive code review on this repository. Work by reading the code only; do not access the network or run the app. 1) Threat modeling: what untrusted input it processes and where it enters. 2) Concrete vulnerabilities: type, file:line, why it's exploitable (chain from the entry point to the sink), severity. 3) Rank them by evidence and impact. 4) For the most severe ones, propose the patch. Be honest: if something looks vulnerable but is mitigated, say so. Do not invent findings to fill the list. Always cite file:line. (I ask both for "don't invent," but I'm not kidding myself: asking a model to be honest guarantees nothing. What filters out the filler is that I verify every finding against the code, not the prompt's good intentions.) And how do you hand it your repo? Simplest route: upload the files to a ChatGPT or Claude Project, or use an agent tool (Codex, Cursor, Claude Code) that already sees your code, and paste the prompt above. A good chunk of the Daybreak story rests on one idea: the normal model has brakes for security work, and Blue takes them off. OpenAI even publishes the number. And here's the detail almost nobody reads: that number —the general model completes 1.5% of the requests, Blue 2.0%— is about offensive tasks: exploit-chain development, authentication bypass, privilege escalation to attack (Infosecurity Magazine). The big jump (95%) belongs to a different model, GPT-5.6-Cyber, which only the Red tier gives you. Reviewing your own code to protect it is none of those things. And it showed: all ten runs, standard and Blue, did the complete audit without a single "I can't help you with this." The brake never kicked in because my task was never in the category the brake blocks. That alone answers half the post: to secure your app, the ChatGPT you have doesn't balk. I verified it by reading the code, not taking its word —chains that are plausible per the code, not proven in execution: I never ran an exploit—. Across the ten runs, the three serious bugs showed up without fail: Privilege escalation. A low-privilege user becomes an administrator. In the edit-user route there is, literally, this: // Force admin for now $input['role'] = 'administrator'; The authorization filter only checks that a session exists, never the role. You edit your own profile and come out an admin. With one caveat the reports flag and that's worth stating: the default anchor-cms install also forces administrator when creating users, so out of the box there are no low-privilege accounts to escalate from. The bug bites in migrated installs, with imported data or a manual seed, where those accounts do exist. Cataloged as CVE-2026-88959 (CVSS 8.7 per the registry), published a month before this experiment —after both models' training cutoff, so they found it by reading, not remembering—. The other two constants: account takeover via the Host header in the password reset (the link is built with $_SERVER['HTTP_HOST'] without validation) and CSRF via GET (the protection only validates POST, and all deletes are GET). So far, that's what the first run gave me. Its conclusion: "Blue found a fourth critical the standard missed (the installer), the standard found two session bugs Blue missed. Near tie, slight edge to Blue in depth." Nice, balanced, false. An LLM isn't deterministic. Ask it the same thing twice and it answers differently. So before publishing "Blue found X and the standard didn't," I ran it five times per side and counted in how many runs each finding appeared. These are two separate axes: how serious the bug is (verified against the code) and in how many runs each side caught it. Detection is noisy; severity doesn't change with it. Finding Severity Standard detection Blue detection Privilege escalation (CVE, conditional) High 5/5 5/5 CSRF via GET High 5/5 5/5 Host-header in the reset Medium-high 4/5 4/5 Open installer → takeover High 4/5 2/5 Sessions / enumeration / error leakage Medium-low noise, both noise, both Reflected XSS (borderline, disputed) Low/doubtful 0/5 3/5 LFI via pagetype (conditional) Low/conditional 0/5 2/5 The "installer critical" I'd pinned on Blue as an exclusive find is caught twice as often by the standard: 4 of 5 runs versus Blue's 2 of 5. My comparison had it backwards out of pure sampling luck. If I'd published the first run, I'd have published a coincidence as if it were a finding. Enumeration, session fixation, and trace leakage show up in both models evenly: they're there, but they don't tell one apart from the other. The only two findings that come up 100% of the time on both sides are the privesc and the CSRF; the host-header lands in a stable tie, 4 of 5 on each. Where there's consistency, there's a tie. What looks at first glance like Blue's edge: it reported two things the standard never listed as findings —a reflected XSS from "flashed" form values and a local file inclusion via the pagetype field, which gets concatenated into a path without sanitizing—. Sounds like Blue saw something the standard missed. But when you read the raw reports, at least in the stronger case —the XSS— that's not what happened: the standard looked at the same code and dismissed it with an argument. One run traces it down to the internal Input::flash() mechanism and concludes, verbatim, that it "does not assert an exploitable XSS chain with the current code." With pagetype it's different, and I have to be honest: the standard neither traced it nor dismissed it, it simply never raised it as a vulnerability in any of its five runs. There a sink it didn't get to may exist. In other words: on the XSS it's not a coverage gap (Blue found it, the standard didn't), it's a criterion disagreement —Blue reports it as a finding; the standard analyzes it and judges it not exploitable—. On pagetype the split is grayer, the standard may not have reached that sink. But in neither case are we looking at a bug that sinks you: the XSS is so borderline that Blue itself doesn't report it in 2 of every 5 runs, and pagetype is a conditional, low-severity LFI. If the standard's judgment is the right one, Blue's reports are false positives; and even if it isn't, more breadth on the borderline stuff isn't obviously better: for an auditor, a well-dismissed false positive is worth as much as a finding. And the statistics land in the same place. With five runs per side, a 3/5 versus 0/5 has a ~17% chance of coming out that way by pure chance even if the two tiers were identical (Fisher's exact test): nowhere near significant. This design could only prove a near-total difference —4 or 5 of 5 versus 0—, and the only thing that reaches that consistency is the serious bugs, which tie (and on top of that, looking at thirteen distinct findings, having at least one come out this skewed is pure chance, not signal). A note on method: I counted a finding only if the report lists it as an actionable vulnerability, not if it mentions it to dismiss it. For the concrete question —"is the special tier worth it for securing my app?"— the answer is no, it barely changes anything, and now I can say it with data instead of one lucky run. Both models found the bugs that sink you in all ten runs, without missing once. The specialized tier didn't add a single serious finding the standard missed: what it added was noise on the borderline. Nothing more. An honest limit before you generalize: this is one app and one prompt. The five runs kill the randomness within that case, they don't promise you the same on your Go microservice or your app on a framework that already covers half of OWASP for you. anchor-cms is old PHP, hand-rolled, with no safety net: the most favorable case for any model to shine. Read the conclusion as "for this kind of target," not as a universal law. And one more limit, this time on Blue's side: I deliberately picked the most harmless defensive task —read code, without running or writing an exploit—, which is exactly where the standard model's brake was never going to kick in. Blue is also sold for validating patches with a proof of concept, analyzing malware, or responding to incidents; several of those do produce offensive artifacts and do trigger refusals from the standard. There Blue could unlock what regular ChatGPT bounces back at you. I didn't test it: my experiment measures the sub-case where, by design, Blue had no way to win. The lessons, which aren't about which model you use: A capable model finds your app's serious bugs reliably. Great… and it also means anyone with the same tool finds them just as fast. If you vibecode, assume your code will be audited by a model: do it yourself first. Don't run it just once. This is the part that almost got me. One run is a single sample of a random variable. Run the same prompt three or four times and keep the union of the findings; half the useful stuff only shows up on the second or third pass. Verify. The most serious bug was a // Force admin for now comment in plain sight, and even so each run missed different things. The model speeds you up; it doesn't replace your judgment. Ask for file and line, ask it to dismiss false positives, and follow the chain yourself. None of this is against Daybreak. But watch the argument: the numbers OpenAI publishes (that 1.5%→2.0%) don't even measure this case —they're how often the model refuses on offensive tasks, not how well it reviews your code—, so I don't lean on them, I lean on my ten runs. For the dev who wants to secure the app they built over the weekend, the news is calm: the tool you already have is enough. Run it more than once, read what it tells you with suspicion, and fix it. The ten raw reports (five per program) and the prompt are in this gist. Run the standard side on your own app and tell me if you get something different.