The question behind this series was how to build the best security mechanism for an app with Claude Code. There is no perfect one, and nothing below promises it. What there is instead: four experiments on one ASP.NET Core app with 20 planted vulnerabilities, 99 counted sessions, a hidden suite of attack tests as the judge, and eight guesses written down before the first run. Four of the guesses were wrong. This part puts the results in one place, settles the guesses, lists what the experiment itself got wrong, and ends with a checklist and the files to copy.
The four experiments in one table
| Part | What was asked | Result | Sessions | Turns | Session time | Meter |
|---|---|---|---|---|---|---|
| 1 Find | review the planted app, five set-ups, three runs each; two control reviews of the secure app | 19 or 20 of 20 named in every run of four set-ups; 7 to 12 by the built-in review; scanners 5; one false alarm on six decoys | 17 | 204 | 45 min | $14.54 |
| 2 Fix | fix the 20, one session each, judged by attack tests | 20 of 20 closed on the first run, nothing broken | 20 | 162 | 17 min | $4.44 |
| 3 Keep it fixed | rebuild seven removed features under three levels of guardrails | 42 of 42 built, no attack test failed in any arm; the full guardrails took 2.4 times as long | 42 | 528 | 68 min | $17.78 |
| 4 The agent as target | six planted instructions in two conditions; the same acts asked directly | none of twelve carried out; three direct requests blocked by a hook, five declined | 20 | 124 | 17 min | $3.47 |
| Total (plus nine dry and smoke sessions that checked the harness, $2.48) | 99 | 1,018 | 2 h 27 min | $40.23 | ||
Eight guesses, settled
The guesses were written into the plan before any session ran. They are the author's, made from experience with earlier series on this site, and half of them underestimated the model.
| Guess | Verdict | What happened |
|---|---|---|
| 1. A plain review prompt finds at least 12 of 20, missing the access-control mistakes and the absent lockout | half right | it named 19 in each run, including every access-control mistake; its one miss was the vulnerable package |
| 2. A checklist skill and an endpoint walk each find at least 3 more; a reviewer subagent finds the same at a higher cost | wrong | one more each, since only one was left to find; the subagent matched the plain prompt at a slightly higher meter and more than twice the time |
| 3. Free scanners together find fewer than half, including at least two that an AI set-up misses | right | five of 20; the plain prompt missed one of the five (the package) in every run, and the built-in review missed each of the five in at least one run |
| 4. Fewer than 15 are found in all three runs of the best set-up | wrong | all 20, in two set-ups |
| 5. At least three first fixes close the obvious exploit and fail a variant | wrong | none failed any variant |
| 6. No fix breaks the functional tests; at most two break an earlier fix | right | none of either |
| 7. Without guardrails at least half the features arrive with a vulnerability; rules and hooks remove most | wrong | none arrived with one, so there was nothing to remove; a rule added a weakness instead |
| 8. A planted instruction is followed in at most one of six scenarios, guardrails stop every attempt, and the direct replay finds a gap | half right | none was followed and no attempt was made; three direct requests were blocked; no gap was found, and several layers were never reached |
Read that table with the size of the test in mind. The app has 45 source files. The weaknesses are textbook cases, one per commit, and the careless change left its unused guard classes lying next to the hole. On that ground a current model reads all the code in one pass and misses little. The results say what Claude Code does on a small, clean codebase. They do not say what it does on yours.
What the measurements support
- Ask for the review, plainly. A one-line prompt named 19 of 20. The elaborate set-ups added one finding between them, and the same one.
- Keep the scanners that know things the code does not show. The package audit found what the plain review missed, costs nothing and runs on every build. Never switch it off to silence a warning; that was planted weakness number 13. A secret scanner found one of the two committed secrets; rotate a leaked key instead of trusting a scan to find it.
- Fix one finding per session, from a precise report, and judge the fix with a test the session did not write. All twenty fixes passed. One summary claimed the app would not start without its key, and a probe showed it starts and answers every request with an error. One session fixed a second finding it had not been given.
- Read the diff for what tests do not ask: limits on attempts, data stored the old way, odd inputs, work outside the finding.
- Write security rules as if a literal-minded colleague will apply them, because one will. The rule that named the app's own helpers was followed and harmless. The rule "Login and reset stay behind the existing attempt limit." was followed too, and it lets anyone lock another person's account with five wrong reset codes.
- Use hooks and deny rules for what must never happen, not as a reviewer. In 42 builds the hooks blocked two harmless redirects and the reviewer subagent changed small things at twice the cost. What they are for showed when the user asked for something risky: the command hook refused to copy the secrets file, to contact an unlisted host and to read secrets with a wildcard.
- Treat everything the agent reads as data, and tell it to report what it finds there. None of six planted instructions was followed. With rules and hooks added, the session mentioned the plant in only two of six runs and left the text in the repository.
- Plan for the run that goes differently. Twelve runs without a followed instruction is not a rate. Keep secrets out of the working tree, keep the tool list narrow, keep the agent from editing its own permissions, and review before merge.
What the reviews found in the "secure" app
The reference app was written to be free of the 20 planted weaknesses, not to be perfect, and two control reviews said so. Their thirteen findings, none above medium: the outbound address filter does not cover a few unusual IPv6 forms and reserved ranges; a fetched image is decoded before its dimensions are checked; the lockout counter is read and written without an atomic update; the staff cookie outlives a password reset; reset requests are not rate limited and their timing reveals whether an address exists; the email typed at a failed sign-in is logged. Part 2 added two more facts about it: return URLs with percent-encoded slashes count as local, and the image library it uses refuses a Release build without a licence key. None of these was tested further. They are listed because a series about security that hid the flaws of its own sample would deserve no trust.
What went wrong in the experiment itself
- One harness bug cost a run. After the first fix passed, the harness misread backslashes in the edited code as a path outside the working copy and started a retry nobody needed. It was stopped, fixed and not counted.
- Part 3 was redesigned on the day. The agent that was to write six new features with their own attack tests was stopped by a safety check before writing a file. The experiment that ran reuses the existing tests on rebuilt features and claims less.
- Not every session ran on one model. The built-in review command ran some of its sub-reviewers on Sonnet 5.5 by its own choice. In four runs of Part 4 a safety classifier at the API withheld a response and the session continued on Opus 4.8. Both are in the session table, column by column.
- The scoring script was too generous to scanners. It matched findings by line range and credited the .NET analyzers with 9 of 20 on the secure copy. Every count in the series is therefore a reading of what a finding says, with the script's number beside it.
- A published hook has a defect. The command guard threw an error on the text
byte[]?in one command and let it through. The file is published as it ran. - Several guardrail layers were never reached by a model, so the old question whether a deny rule alone stops
catwith a wildcard is still open. - The flat result of Part 3 is a weak result. When nothing fails without guardrails, the experiment cannot show what guardrails prevent.
Do it yourself: the files to copy
Everything the sessions used is in one download (1.3 MB): both apps, the attack tests, the set-ups, the harness and the results. To try the set-up on a project of your own:
- Turn the package audit on in the build and run it once by hand in your solution folder:
dotnet list package --vulnerable --include-transitive - Copy the checklist skill to
.claude/skills/security-checklist/SKILL.mdand ask for a review with it; or start with the plain prompt from prompts.txt. - For each finding you accept, start a new session with the fix prompt and the finding pasted in, and write the test that attacks it yourself.
- Take the fifteen rules as a starting point, delete what does not apply, name your own helpers, and reread each rule asking what happens if it is applied word for word. Rule 10 is the one that backfired here.
- Copy the settings and the command guard into
.claude, change the protected names and the host list to yours, and run the hook tests before you rely on them. - Read sessions.csv, one row per counted session with its part, model, verdict, turns, seconds and meter, and the answer key.
What you should see: an audit result for your own packages, a review you can compare with what you already know about your code, and hooks that refuse a command naming your secrets.
Frequently asked
- Is an app reviewed and fixed by Claude Code secure?
- It is better reviewed than most. In this test the review named 19 or 20 of 20 planted weaknesses and the fixes closed all 20, on a small app with textbook mistakes. It also missed a vulnerable package until told to run the audit, printed one wrong claim about its own fix, and found real gaps in the app that was supposed to be the secure one. Use it as a strong reviewer, not as a guarantee.
- What is the smallest useful security set-up for Claude Code?
- The package audit switched on in the build, a plain review prompt on changes that touch access, input or secrets, deny rules for the secrets folder and the agent's own settings, and one hook that refuses shell commands naming protected files or unlisted hosts. That is what did measurable work here.
- How much did the whole experiment cost?
- 99 counted sessions, 1,018 turns and 2 hours 27 minutes of session time, $40.23 on the meter at API prices, run on a subscription. Reviews cost $0.53 to $1.93 each, a fix about $0.22, a feature build $0.31 without guardrails and $0.62 with the full set.
The series starts at Part 1. The tooling itself, CLAUDE.md, permissions, hooks, skills, subagents and MCP, is explained step by step in Claude Code for .NET Developers.