Asking an AI to review code for security is easy to do and hard to check, because nobody knows what it missed. So this series plants the answer first. An ASP.NET Core patient portal was written twice: once with care, once with 20 known vulnerabilities and six pieces of code that only look dangerous. Claude Code reviewed the vulnerable copy fifteen times in five set-ups, five free scanners scanned the same files, and every finding was counted against the list. A one-line prompt named 19 of the 20 in each of its three runs. A checklist skill named all 20 every time. The scanners together named 5. This part is the method and the count, including what the count does not show.
The app and what was planted
PatientPortal is a small clinic system on .NET 10: a minimal API with EF Core on SQLite,
JWT bearer tokens for /api, a cookie for two server-rendered staff pages,
45 source files, 43 functional tests. Patients see their own records, appointments and
documents; doctors see their patients; a receptionist and an admin do the rest. The
secure version is the reference. The planted version differs from it by 20 commits, one
weakness each, and the functional tests pass on both: nothing here is a bug a user would
notice.
| Id | What the code does wrong | Kind | A | B | C | D | E | Scanners |
|---|---|---|---|---|---|---|---|---|
| V01 | Search term concatenated into SQL | one line | 3 | 3 | 3 | 3 | 2 | .NET analyzers, Semgrep |
| V02 | Visit records returned without checking whose they are | who may do what | 3 | 3 | 3 | 3 | 3 | none |
| V03 | Admin export open to anyone, signed in or not | who may do what | 3 | 3 | 3 | 3 | 3 | none |
| V04 | Profile update applies role, email and password hash from the request body | across files | 3 | 3 | 3 | 3 | 3 | none |
| V05 | Staff note rendered as raw HTML | one line | 3 | 3 | 3 | 3 | 1 | none |
| V06 | Antiforgery switched off on the note form | one line | 3 | 3 | 3 | 3 | 0 | none |
| V07 | JWT signing key and mail password committed in appsettings.json | configuration | 3 | 3 | 3 | 3 | 0 | gitleaks (the mail password only) |
| V08 | Passwords stored as unsalted SHA-256 | one line | 3 | 3 | 3 | 3 | 0 | none |
| V09 | Failed logins never counted, no lockout | a control that is absent | 3 | 3 | 3 | 3 | 0 | none |
| V10 | Token lifetime, issuer and audience not validated | one line | 3 | 3 | 3 | 3 | 2 | .NET analyzers, Semgrep (lifetime only) |
| V11 | CORS answers any origin with credentials | one line | 3 | 3 | 3 | 3 | 0 | none |
| V12 | Developer exception page in every environment | one line | 3 | 3 | 3 | 3 | 1 | Semgrep |
| V13 | ImageSharp 3.1.12, six published advisories that day, with the NuGet audit switched off | supply chain | 0 | 3 | 1 | 3 | 0 | NuGet audit |
| V14 | Document name from the query string reaches the file system | across files | 3 | 3 | 3 | 3 | 3 | none |
| V15 | Server fetches any URL it is given and follows redirects | across files | 3 | 3 | 3 | 3 | 1 | none |
| V16 | Password and issued token written to the log | one line | 3 | 3 | 3 | 3 | 3 | none |
| V17 | Sign-in cookie without HttpOnly, Secure and SameSite | one line | 3 | 3 | 3 | 3 | 0 | none |
| V18 | Redirect to any returnUrl after sign-in | across files | 3 | 3 | 3 | 3 | 0 | none |
| V19 | Patient list and export return the password hash | across files | 3 | 3 | 3 | 3 | 2 | none |
| V20 | Password-reset code computed from the user id and the date | who may do what | 3 | 3 | 3 | 3 | 3 | none |
Six decoys measure false alarms. They are code a hurried reviewer flags and should not:
FromSqlInterpolated with a parameter, MarkupString on a
constant, a settings value not-a-real-secret that nothing reads,
AllowAnyOrigin without credentials on the public health endpoint, SHA-256
used for a file checksum, and Path.Combine on the name of an enum value.
Behind the list sits a hidden suite of 53 attack tests, two to four per weakness. All 53 pass on the reference app; on the planted app 52 fail and one passes by design (a guard that login still works). No review session ever saw the suite, the list or the reference. Part 2 uses the same tests to judge fixes.
Five ways to ask
Every session was a headless claude -p run of Claude Code 2.1.287 on Opus
5.5, on a fresh copy of the planted app, with a read-only tool list (Read, Glob, Grep
and a few dotnet, git and ls commands). Each
set-up ran three times. All five end with the same report rule: a JSON array of
findings with file, line, weakness, severity, how it is exploited and the fix, and the
sentence "Report only what you would stand behind in front of the team: for each
finding you must be able to say how it is exploited."
- A, the plain prompt: "Review this repository for security vulnerabilities."
- B, a checklist skill: the same sentence plus "Use the security-checklist skill.", a 56-line skill file in the repository that lists what to check in an ASP.NET Core app (access control, authentication, injection, outbound requests, forgery, redirects, data exposure, secrets, files, dependencies) and tells the reviewer to run the package audit.
- C, a reviewer subagent: "Use the security-reviewer subagent to review this repository for security vulnerabilities, then give me its findings." The subagent is read-only and is told to think as someone who wants another person's data.
- D, an endpoint walk: no extra files, only a longer prompt: list every route, write down for each who may call it, what it reads and where each input ends up, and only then review configuration, logging, secrets and packages.
- E, the built-in command:
/security-review, which ships with Claude Code. It reviews the diff between the branch andorigin/HEAD, not a repository: on a plain copy it stopped with git's "ambiguous argument" error before the first turn. So for E the whole app was presented as one commit on top of an empty one.
A bias to know about: the checklist skill was written by the same author who chose the 20 weaknesses. It was written from the general classes, not from the list, but it covers every class on the list. Read B's 20 of 20 with that in mind; A and D, which have no such file, are the cleaner measurement.
How a finding was counted
A script matched findings to the list by file and line, then by keyword. That turned out
to be too generous for scanners and too strict for people-like reports, so the count in
this post is the author's reading: a weakness counts as found when a finding names it.
The script's number is shown beside it. The difference matters in exactly three places.
Two runs reported the path traversal at the line that joins the name into a path
(DocumentStore.cs:15) instead of at the endpoint, so the script scored a
miss and a decoy hit where the finding plainly says "Path traversal / arbitrary file
read in document download". One run's CORS finding matched the harmless health-endpoint
decoy by keyword although it describes the planted policy. And on the secure reference
copy the script gave the .NET analyzers 9 of 20, because general warnings land inside
the line ranges of the answer key.
What the reviews found
| Set-up | Named, per run | Script's count | Decoys flagged | Other findings | Turns | Seconds | Meter per run |
|---|---|---|---|---|---|---|---|
| A plain prompt | 19, 19, 19 | 18, 19, 19 | 0 | 1, 1, 3 | 8 to 10 | 77 to 87 | $0.53 to $0.75 |
| B checklist skill | 20, 20, 20 | 20, 20, 20 | 0 | 0 | 12 to 42 | 83 to 144 | $0.70 to $0.82 |
| C reviewer subagent | 19, 19, 20 | 19, 19, 20 | 0 | 1, 2, 1 | 2 (plus the subagent's) | 186 to 221 | $0.73 to $0.85 |
| D endpoint walk | 20, 20, 20 | 19, 20, 20 | 1 (run 3) | 0, 1, 2 | 7 to 10 | 91 to 131 | $0.53 to $0.62 |
| E built-in /security-review | 8, 7, 12 | 8, 7, 12 | 0 | 0 | 10 to 16 | 225 to 314 | $1.30 to $1.93 |
The plain prompt's only miss, in all three runs, was V13, the package. The reason is in
the transcripts: the three skill runs each ran dotnet list package --vulnerable,
which the checklist tells them to, and reported ImageSharp with its advisories. No plain
run ran that command or mentions it, although the tool list allowed it. Two of the
endpoint-walk runs and one subagent run noticed NuGetAudit set to false
without naming the package behind it. A review that only reads code does not know what
was published about a version last month; the audit command does.
The weaknesses that need "who may do what" (V02, V03, V20) and the missing control (V09) were named in every run of A to D. That was the opposite of the guess written down before the runs, which expected a plain prompt to miss exactly those.
The built-in command is a different tool. It hands the review to sub-reviewers, and by its own choice it ran some of them on Sonnet 5.5, a smaller model, in all three runs; the other four set-ups ran on Opus 5.5 only. The sub-reviewers get a list of things not to report; the instructions one session passed down read "no DoS; no rate limiting; no hardening-only issues; no theoretical races; no outdated libs", and "Open redirects/tabnabbing only if extremely high confidence". On this app that list covers the missing lockout (V09), the package (V13) and the open redirect (V18), and arguably the cookie flags (V17). It never reported the committed secrets, the unsalted hashes, the CORS policy or the missing antiforgery check either, and it reported nothing else: no decoy and no extra finding in three runs. Six weaknesses it named every time (V02, V03, V04, V14, V16, V20). It reviews a diff and filters hard against false alarms, and that is how it behaved.
False alarms and everything else
One decoy was flagged in fifteen sessions: endpoint-walk run 3 listed the
not-a-real-secret placeholder as a secret, severity low. Nobody flagged the
parameterized FromSqlInterpolated, the constant MarkupString,
the checksum or the enum path.
Twelve findings match neither list. None is invented. Four say AllowedHosts
is * in the base settings file, three that there is no HTTPS redirection,
three concern the reset endpoint (timing reveals whether an email exists, no rate
limit, old tokens stay valid), one questions a rule that the route contract allows, and
one is a second pointer to the path traversal. All but the last are rated low by the
sessions themselves.
As a control, the plain prompt and the subagent each reviewed the secure reference app once. They reported 7 and 6 findings, none critical or high, none of the 20 and no decoy: gaps in the outbound address filter for unusual IPv6 forms, an image that is decoded before its dimensions are checked, a lockout counter that parallel requests can outrun, a staff cookie that outlives a password reset, and the same reset-endpoint remarks. None was tested here. The reference app was written to be free of the 20, not to be perfect, and a review of it says so.
What five free scanners found in the same files
| Tool | Reported | Names one of the 20 | Notes |
|---|---|---|---|
| .NET analyzers (SDK 11.0.100), default build | 1 compiler warning | none | nothing security-related at the default level |
.NET analyzers, AnalysisLevel=latest-all and AnalysisModeSecurity=All | 398 results | V01 (CA2100), V10 (CA5404, three lines) | four security diagnostics among 394 others |
NuGet audit and dotnet list package --vulnerable | six advisories | V13 | the switch in the API project did not silence them: they print for the test project |
| Semgrep 1.184.0, community packs | 3 findings | V01, V10 (lifetime line), V12 | 72 rules on 55 files; C# 12 primary constructors parsed only in part |
| OWASP ZAP 2.17.0 baseline | 2 alerts from /, 23 from the login page | none | unauthenticated; header and informational alerts, the same on both copies |
| gitleaks v8.30.1 | 1 secret | V07, the mail password | the JWT signing key, a phrase, was not flagged |
Together the tools name five of the twenty: V01, V07 (half of it), V10, V12 and V13. Six are reported by no tool under any way of counting: V03, V05, V06, V08, V11 and V17. What the tools do well is what a reading review did worst: the package audit knows the advisory list, and it costs nothing to run on every build. The 398 analyzer results also show why the security rules are off by default.
What this does not show
- The app is small and the weaknesses are textbook cases, one per commit, each visible in a few lines once you look at the right file. A real codebase hides the same mistakes behind layers and scale.
- Planting left hints. After the outbound check and the local-URL check were removed from their callers, the unused guard classes stayed in the code, as they would after a careless change. Several findings say so: "AccessRules never called", "OutboundAddressPolicy unused".
- One model, one version, one day, three runs per set-up. The runs agree closely here; that is a fact about this app, not a promise.
- A review names a weakness; it does not prove the fix. Part 2 measures that with the attack tests.
Do it yourself
The app in both versions, the attack suite, the answer key, the prompts and the skill are in one download.
- Download coder000-patientportal-security.zip (1.3 MB), unzip it into a short folder, open PowerShell in it and run the functional tests on the vulnerable app. They pass:
dotnet test app-planted - Run the attack suite against the same folder, then against the secure one (PowerShell; the suite takes the app's folder as a property):
dotnet test attack/PatientPortal.Attack "-p:PortalRoot=$PWD\app-planted" --artifacts-path out\planted dotnet test attack/PatientPortal.Attack "-p:PortalRoot=$PWD\app-reference" --artifacts-path out\reference - Copy
app-plantedsomewhere outside that folder, so that the answer key is not next to it, and ask for a review there with the plain prompt and a read-only tool list:claude -p "Review this repository for security vulnerabilities." --allowedTools "Read,Glob,Grep,Bash(dotnet list:*),Bash(git log:*),Bash(ls:*)" - Compare what you get with the answer key, then try the other four prompts, the checklist skill and the reviewer subagent.
- Run the package audit, the one check the plain review skipped:
dotnet list app-planted package --vulnerable --include-transitive - Read findings.csv, every finding of the fifteen sessions with what it was counted as, and the scanner commands.
What you should see: 43 functional tests passing on both apps; 52 attack tests failing and 1 passing on the planted app, 53 passing on the reference; and a review that lists most of the table above within two minutes.
Frequently asked
- Can Claude Code replace a security scanner?
- On this app the two did different jobs. The reviews named 19 or 20 of 20 planted weaknesses, including the access-control mistakes no scanner reported, and the plain prompt missed the one thing a scanner is best at: a package with published advisories. The package audit is free and runs on every build, so the measured answer is to use both.
- Do I need a special prompt for a security review?
- Not on this app. The one-line prompt named 19 of 20 in each of three runs. A checklist that tells the reviewer to run the package audit closed the gap; a longer endpoint-by-endpoint prompt did about as well. The built-in /security-review command reported 7 to 12, because it reviews a branch's changes and leaves several classes out by rule.
- How many false alarms did the reviews produce?
- One decoy out of six was flagged once in fifteen sessions, with low severity. Twelve other findings matched nothing planted; all were low-severity hardening remarks or duplicates, and none described a vulnerability that is not there.
Next: Part 2, where Claude Code fixes the 20 one at a time and the hidden attack tests decide whether each hole is closed.