Asking an AI to review code for security is easy to do and hard to check, because nobody knows what it missed. So this series plants the answer first. An ASP.NET Core patient portal was written twice: once with care, once with 20 known vulnerabilities and six pieces of code that only look dangerous. Claude Code reviewed the vulnerable copy fifteen times in five set-ups, five free scanners scanned the same files, and every finding was counted against the list. A one-line prompt named 19 of the 20 in each of its three runs. A checklist skill named all 20 every time. The scanners together named 5. This part is the method and the count, including what the count does not show.

The app and what was planted

PatientPortal is a small clinic system on .NET 10: a minimal API with EF Core on SQLite, JWT bearer tokens for /api, a cookie for two server-rendered staff pages, 45 source files, 43 functional tests. Patients see their own records, appointments and documents; doctors see their patients; a receptionist and an admin do the rest. The secure version is the reference. The planted version differs from it by 20 commits, one weakness each, and the functional tests pass on both: nothing here is a bug a user would notice.

The 20 planted weaknesses and how often each was named: runs out of three per review set-up (A plain prompt, B checklist skill, C reviewer subagent, D endpoint walk, E built-in review), and the scanners whose message names it
IdWhat the code does wrongKindABCDEScanners
V01Search term concatenated into SQLone line33332.NET analyzers, Semgrep
V02Visit records returned without checking whose they arewho may do what33333none
V03Admin export open to anyone, signed in or notwho may do what33333none
V04Profile update applies role, email and password hash from the request bodyacross files33333none
V05Staff note rendered as raw HTMLone line33331none
V06Antiforgery switched off on the note formone line33330none
V07JWT signing key and mail password committed in appsettings.jsonconfiguration33330gitleaks (the mail password only)
V08Passwords stored as unsalted SHA-256one line33330none
V09Failed logins never counted, no lockouta control that is absent33330none
V10Token lifetime, issuer and audience not validatedone line33332.NET analyzers, Semgrep (lifetime only)
V11CORS answers any origin with credentialsone line33330none
V12Developer exception page in every environmentone line33331Semgrep
V13ImageSharp 3.1.12, six published advisories that day, with the NuGet audit switched offsupply chain03130NuGet audit
V14Document name from the query string reaches the file systemacross files33333none
V15Server fetches any URL it is given and follows redirectsacross files33331none
V16Password and issued token written to the logone line33333none
V17Sign-in cookie without HttpOnly, Secure and SameSiteone line33330none
V18Redirect to any returnUrl after sign-inacross files33330none
V19Patient list and export return the password hashacross files33332none
V20Password-reset code computed from the user id and the datewho may do what33333none

Six decoys measure false alarms. They are code a hurried reviewer flags and should not: FromSqlInterpolated with a parameter, MarkupString on a constant, a settings value not-a-real-secret that nothing reads, AllowAnyOrigin without credentials on the public health endpoint, SHA-256 used for a file checksum, and Path.Combine on the name of an enum value.

Behind the list sits a hidden suite of 53 attack tests, two to four per weakness. All 53 pass on the reference app; on the planted app 52 fail and one passes by design (a guard that login still works). No review session ever saw the suite, the list or the reference. Part 2 uses the same tests to judge fixes.

Five ways to ask

Every session was a headless claude -p run of Claude Code 2.1.287 on Opus 5.5, on a fresh copy of the planted app, with a read-only tool list (Read, Glob, Grep and a few dotnet, git and ls commands). Each set-up ran three times. All five end with the same report rule: a JSON array of findings with file, line, weakness, severity, how it is exploited and the fix, and the sentence "Report only what you would stand behind in front of the team: for each finding you must be able to say how it is exploited."

  • A, the plain prompt: "Review this repository for security vulnerabilities."
  • B, a checklist skill: the same sentence plus "Use the security-checklist skill.", a 56-line skill file in the repository that lists what to check in an ASP.NET Core app (access control, authentication, injection, outbound requests, forgery, redirects, data exposure, secrets, files, dependencies) and tells the reviewer to run the package audit.
  • C, a reviewer subagent: "Use the security-reviewer subagent to review this repository for security vulnerabilities, then give me its findings." The subagent is read-only and is told to think as someone who wants another person's data.
  • D, an endpoint walk: no extra files, only a longer prompt: list every route, write down for each who may call it, what it reads and where each input ends up, and only then review configuration, logging, secrets and packages.
  • E, the built-in command: /security-review, which ships with Claude Code. It reviews the diff between the branch and origin/HEAD, not a repository: on a plain copy it stopped with git's "ambiguous argument" error before the first turn. So for E the whole app was presented as one commit on top of an empty one.

A bias to know about: the checklist skill was written by the same author who chose the 20 weaknesses. It was written from the general classes, not from the list, but it covers every class on the list. Read B's 20 of 20 with that in mind; A and D, which have no such file, are the cleaner measurement.

How a finding was counted

A script matched findings to the list by file and line, then by keyword. That turned out to be too generous for scanners and too strict for people-like reports, so the count in this post is the author's reading: a weakness counts as found when a finding names it. The script's number is shown beside it. The difference matters in exactly three places. Two runs reported the path traversal at the line that joins the name into a path (DocumentStore.cs:15) instead of at the endpoint, so the script scored a miss and a decoy hit where the finding plainly says "Path traversal / arbitrary file read in document download". One run's CORS finding matched the harmless health-endpoint decoy by keyword although it describes the planted policy. And on the secure reference copy the script gave the .NET analyzers 9 of 20, because general warnings land inside the line ranges of the answer key.

What the reviews found

Fifteen review sessions on the planted app, Nov 14, 2026. Named = weaknesses of the 20 that a finding names. Meter = the notional API price the CLI reports; the sessions ran on a subscription.
Set-upNamed, per runScript's countDecoys flaggedOther findingsTurnsSecondsMeter per run
A plain prompt19, 19, 1918, 19, 1901, 1, 38 to 1077 to 87$0.53 to $0.75
B checklist skill20, 20, 2020, 20, 200012 to 4283 to 144$0.70 to $0.82
C reviewer subagent19, 19, 2019, 19, 2001, 2, 12 (plus the subagent's)186 to 221$0.73 to $0.85
D endpoint walk20, 20, 2019, 20, 201 (run 3)0, 1, 27 to 1091 to 131$0.53 to $0.62
E built-in /security-review8, 7, 128, 7, 120010 to 16225 to 314$1.30 to $1.93

The plain prompt's only miss, in all three runs, was V13, the package. The reason is in the transcripts: the three skill runs each ran dotnet list package --vulnerable, which the checklist tells them to, and reported ImageSharp with its advisories. No plain run ran that command or mentions it, although the tool list allowed it. Two of the endpoint-walk runs and one subagent run noticed NuGetAudit set to false without naming the package behind it. A review that only reads code does not know what was published about a version last month; the audit command does.

The weaknesses that need "who may do what" (V02, V03, V20) and the missing control (V09) were named in every run of A to D. That was the opposite of the guess written down before the runs, which expected a plain prompt to miss exactly those.

The built-in command is a different tool. It hands the review to sub-reviewers, and by its own choice it ran some of them on Sonnet 5.5, a smaller model, in all three runs; the other four set-ups ran on Opus 5.5 only. The sub-reviewers get a list of things not to report; the instructions one session passed down read "no DoS; no rate limiting; no hardening-only issues; no theoretical races; no outdated libs", and "Open redirects/tabnabbing only if extremely high confidence". On this app that list covers the missing lockout (V09), the package (V13) and the open redirect (V18), and arguably the cookie flags (V17). It never reported the committed secrets, the unsalted hashes, the CORS policy or the missing antiforgery check either, and it reported nothing else: no decoy and no extra finding in three runs. Six weaknesses it named every time (V02, V03, V04, V14, V16, V20). It reviews a diff and filters hard against false alarms, and that is how it behaved.

False alarms and everything else

One decoy was flagged in fifteen sessions: endpoint-walk run 3 listed the not-a-real-secret placeholder as a secret, severity low. Nobody flagged the parameterized FromSqlInterpolated, the constant MarkupString, the checksum or the enum path.

Twelve findings match neither list. None is invented. Four say AllowedHosts is * in the base settings file, three that there is no HTTPS redirection, three concern the reset endpoint (timing reveals whether an email exists, no rate limit, old tokens stay valid), one questions a rule that the route contract allows, and one is a second pointer to the path traversal. All but the last are rated low by the sessions themselves.

As a control, the plain prompt and the subagent each reviewed the secure reference app once. They reported 7 and 6 findings, none critical or high, none of the 20 and no decoy: gaps in the outbound address filter for unusual IPv6 forms, an image that is decoded before its dimensions are checked, a lockout counter that parallel requests can outrun, a staff cookie that outlives a password reset, and the same reset-endpoint remarks. None was tested here. The reference app was written to be free of the 20, not to be perfect, and a review of it says so.

What five free scanners found in the same files

Each tool run once on the planted copy and once on the secure copy, default or documented options, Nov 14, 2026
ToolReportedNames one of the 20Notes
.NET analyzers (SDK 11.0.100), default build1 compiler warningnonenothing security-related at the default level
.NET analyzers, AnalysisLevel=latest-all and AnalysisModeSecurity=All398 resultsV01 (CA2100), V10 (CA5404, three lines)four security diagnostics among 394 others
NuGet audit and dotnet list package --vulnerablesix advisoriesV13the switch in the API project did not silence them: they print for the test project
Semgrep 1.184.0, community packs3 findingsV01, V10 (lifetime line), V1272 rules on 55 files; C# 12 primary constructors parsed only in part
OWASP ZAP 2.17.0 baseline2 alerts from /, 23 from the login pagenoneunauthenticated; header and informational alerts, the same on both copies
gitleaks v8.30.11 secretV07, the mail passwordthe JWT signing key, a phrase, was not flagged

Together the tools name five of the twenty: V01, V07 (half of it), V10, V12 and V13. Six are reported by no tool under any way of counting: V03, V05, V06, V08, V11 and V17. What the tools do well is what a reading review did worst: the package audit knows the advisory list, and it costs nothing to run on every build. The 398 analyzer results also show why the security rules are off by default.

What this does not show

  • The app is small and the weaknesses are textbook cases, one per commit, each visible in a few lines once you look at the right file. A real codebase hides the same mistakes behind layers and scale.
  • Planting left hints. After the outbound check and the local-URL check were removed from their callers, the unused guard classes stayed in the code, as they would after a careless change. Several findings say so: "AccessRules never called", "OutboundAddressPolicy unused".
  • One model, one version, one day, three runs per set-up. The runs agree closely here; that is a fact about this app, not a promise.
  • A review names a weakness; it does not prove the fix. Part 2 measures that with the attack tests.

Do it yourself

The app in both versions, the attack suite, the answer key, the prompts and the skill are in one download.

  1. Download coder000-patientportal-security.zip (1.3 MB), unzip it into a short folder, open PowerShell in it and run the functional tests on the vulnerable app. They pass:
    dotnet test app-planted
  2. Run the attack suite against the same folder, then against the secure one (PowerShell; the suite takes the app's folder as a property):
    dotnet test attack/PatientPortal.Attack "-p:PortalRoot=$PWD\app-planted" --artifacts-path out\planted
    dotnet test attack/PatientPortal.Attack "-p:PortalRoot=$PWD\app-reference" --artifacts-path out\reference
  3. Copy app-planted somewhere outside that folder, so that the answer key is not next to it, and ask for a review there with the plain prompt and a read-only tool list:
    claude -p "Review this repository for security vulnerabilities." --allowedTools "Read,Glob,Grep,Bash(dotnet list:*),Bash(git log:*),Bash(ls:*)"
  4. Compare what you get with the answer key, then try the other four prompts, the checklist skill and the reviewer subagent.
  5. Run the package audit, the one check the plain review skipped:
    dotnet list app-planted package --vulnerable --include-transitive
  6. Read findings.csv, every finding of the fifteen sessions with what it was counted as, and the scanner commands.

What you should see: 43 functional tests passing on both apps; 52 attack tests failing and 1 passing on the planted app, 53 passing on the reference; and a review that lists most of the table above within two minutes.

Frequently asked

Can Claude Code replace a security scanner?
On this app the two did different jobs. The reviews named 19 or 20 of 20 planted weaknesses, including the access-control mistakes no scanner reported, and the plain prompt missed the one thing a scanner is best at: a package with published advisories. The package audit is free and runs on every build, so the measured answer is to use both.
Do I need a special prompt for a security review?
Not on this app. The one-line prompt named 19 of 20 in each of three runs. A checklist that tells the reviewer to run the package audit closed the gap; a longer endpoint-by-endpoint prompt did about as well. The built-in /security-review command reported 7 to 12, because it reviews a branch's changes and leaves several classes out by rule.
How many false alarms did the reviews produce?
One decoy out of six was flagged once in fifteen sessions, with low severity. Twelve other findings matched nothing planted; all were low-severity hardening remarks or duplicates, and none described a vulnerability that is not there.

Next: Part 2, where Claude Code fixes the 20 one at a time and the hidden attack tests decide whether each hole is closed.