The first time we published Ship Safe's false-positive benchmark, we put the uncomfortable sentence in the methodology ourselves: hand-adjudication was not done.
That sentence was not a footnote. It was the boundary of what the benchmark could claim. The clean corpus told us how much triage a user would inherit. The vulnerable corpus told us whether a noise reduction had broken detection. Neither told us whether a finding that Ship Safe called confirmed was actually confirmed.
We also wrote a warning that seemed obvious: a falling finding count can mean that the scanner got more precise, or that it stopped looking. Breaking detection can look exactly like noise reduction.
Both warnings became work today.
The question changed
The deterministic corpus asks whether a rule fires. The verdict benchmark asks whether the investigation layer reaches the right conclusion on paired fixtures. Both are useful, and both run in CI.
Neither is a person reading a real application.
So we added a third benchmark with a different job. It pins three real applications: an Express and Prisma REST API, an Express application with authentication and third-party integrations, and a FastAPI service with auth. These are the shapes that make absence rules, dataflow, and cross-file paths matter. A library has no request handler to follow. An application does.
The application corpus has no pass/fail gate. It prints the confirmations and the citations for a person to read. That is not a missing feature. It is the point.
The first audit put four confirmations in front of us.
One survived.
Three did not.
That is not a precision rate. It is one small audit, on three pinned repositories, and the sample is too small for a percentage to pretend to be a product metric. It is a much more useful result than another green synthetic run, because it showed us what the tool was saying about code written by people who were not trying to satisfy our fixtures.
The first defect: an argument is not an origin
One confirmation came from a line shaped like this:
const result = await getArticles(req.query, id)The rationale said that result came from the HTTP request because req.query appeared on the right-hand side.
It did not. The request was an argument to an unknown function. The return value came from whatever that function returned, in this case a database query. The tracer had mistaken a value passed into a call for the value produced by the call.
That is not an unusual edge case. It is one of the most common lines in a web application. Every const x = await something(req...) could be confirmed by the old logic, even when the request value was only an identifier used to fetch trusted data.
The correction is deliberately conservative. Known transforms that hand their input back, such as JSON.parse in the supported shapes, can still be followed. An unknown call is a barrier. If the investigator cannot see through the barrier, it cannot confirm a path on the other side of it.
The output may stay unresolved. That is not the scanner missing a vulnerability. It is the investigation layer refusing to manufacture a path from an argument it did not observe being returned.
The second defect: the tracer answered the wrong question
The other defect was a category error.
RAG_NO_RELEVANCE_THRESHOLD was confirmed because a query came from the request. That fact can be true. It does not answer whether a minimum similarity threshold exists in the retrieval configuration.
A dataflow trace answers a question about where a value goes. An absence rule answers a question about whether a control exists and applies in the relevant project. The source of a query does not establish the absence of a threshold any more than the source of a password establishes whether an audit log is configured.
Those questions now have separate investigators. The absence pass checks whether the project contains an application that could hold the control, then searches the project for the control and cites where it is applied. A control found in an import or a comment does not count. If the control is absent across the visible project, the finding can become likely; it does not become confirmed, because a gateway or proxy outside the repository may enforce it.
The distinction sounds narrow. It is the difference between evidence about a path and evidence about a boundary.
Why our green runs did not catch this
The existing automated corpora did what they were designed to do. The deterministic corpus checked that known rules still fired. The verdict corpus checked a collection of known paths and controls. Six runs passed.
They did not contain the exact shape that failed in the application audit: a returned database value with a request argument beside it, or a configuration-absence rule being answered by a request-to-sink trace.
The current state is 869 tests, a clean lint run, and three benchmark gates green. Those numbers are important. They are not a substitute for reading a confirmation.
A benchmark can protect a behavior it has named. It cannot adjudicate a new real-world shape it has not named. That is why the application corpus prints its evidence instead of hiding behind a gate.
A quieter report is not automatically a better report
The dangerous way to celebrate this work would be to say that confirmations went down and stop there.
If a rule stops confirming because the investigator became blind, the output is quieter and the product is worse. If a path is refuted because the cited control exists, the output is quieter for a reason. Those are different events, and a security tool should not collapse them into the same green number.
Ship Safe now keeps the sensor finding separate from the investigation verdict. audit can still show the deterministic match. investigate shows whether the evidence made it confirmed, likely, unresolved, or refuted. A fix is checked by whether the path closed, not only by whether the original pattern stopped matching.
That lets us say what happened without pretending that silence is proof.
What changed
The investigation layer now records the claim behind a conclusion: which pass made it, the rationale, and the files and lines it cited. The verdict is derived by fixed precedence. Reproduction outranks dataflow, chain, or presence; those outrank analysis; analysis outranks a heuristic match.
If a citation no longer resolves, that claim does not decide the verdict. If two passes at the same rank disagree, the result is unresolved. The system chooses uncertainty over whichever answer sounds more alarming.
The application audit also changed our release process. New confirmation behavior needs a real code reading before we call it trustworthy. The corpus is intentionally not a CI gate, because turning a human judgment into a checkbox would hide the thing we are trying to measure.
The claim we can make now
We cannot say that Ship Safe has solved false positives. We cannot say that one audit establishes a production precision rate. We cannot say that a clean score means every conclusion is true.
We can say that the scanner and the investigation layer now have different jobs, that the evidence for a conclusion is visible, and that our first application audit found defects the automated gates could not see.
That is a better starting point than a confident number.
The application corpus is public. The false-positive benchmark remains a proxy, not a verified false-positive rate. If you maintain a scanner, the useful question is not only how many findings it produces. It is whether someone can read the evidence and disagree with the conclusion for a specific reason.
Ship Safe is MIT licensed. Run npx ship-safe investigate . to see what the sensor found, what the evidence established, and what remains unresolved.