How We Dramatically Improved OdinScan Accuracy

TL;DR. We replaced our multi-agent audit stack with a single deep auditing agent that stays with your code from first read to final report. On the same model, our hardest benchmark contract went from F1 0.89 to 0.97: all 17 ground-truth vulnerabilities found, all 10 Highs included, one false positive instead of three, and recall from 94% to 100%. Our Solana pilot contract went from F1 0.67 to a perfect 1.00.

The dirty secret of AI scanners

Every AI security scanner makes the same promise: find bugs a human review would miss, and do it without wasting the reviewer's time. Most tools deliver at most one half of that promise. Either the model floods you with findings and you stop reading after the third "potential reentrancy" on a view function, or a conservative prompt quietly lets the real bug through. Precision and recall both improving at once is rare, and it usually takes a new model.

We just changed that, and we can show our work: same model, same benchmark corpus, same ground-truth labels. Only the architecture changed.

The old way: many agents, then a separate review of their work

Our original pipeline ran three frontier models - Claude, Gemini, and GLM - in parallel on the same repository, merged their findings, then ran a separate verification pass that re-read each candidate finding against the code and demoted or deleted anything it could not justify.

It worked. It was also the ceiling.

On sherlock-991, a DODO cross-chain DEX with 17 ground-truth vulnerabilities (10 High, 7 Medium), that stack topped out at 16 true positives, 3 false positives, 1 missed vulnerability, and F1 0.89. Read that result closely: a real bug got past the full multi-model stack, and three noise findings survived the separate verification pass.

The OdinBench v0.1 numbers we published earlier this year were from this same era: 156 raw false positives reduced to 86 after the verification pass. That 45% reduction was real progress, and it was generated entirely downstream of a pipeline that kept producing noise faster than a detached verifier could remove it.

We did not get stuck because the models were weak. We got stuck because the architecture was wrong.

The insight: context loss is the bug

Three independent agents each read the repository alone. They never shared reasoning: no shared invariants, no shared evidence, no shared memory of what had already been eliminated. Merging their findings produced the union of three isolated readings, not the output of one sustained analysis.

The verification pass was worse. It was a detached second step that re-read candidate findings with less context than the agent that produced them. It could verify that a cited line existed, but it could not revisit the agent's chain of reasoning, and it had no memory of the trace the agent had already worked through. If the agent missed a bug because it never read a file, the verifier could not recover that gap either, because it started from the finding, not from the code.

Fragmenting one audit into four processes does not multiply intelligence. It divides it. Every handoff loses reasoning and every merge is lossy.

The new way: one agent, one continuous audit

We removed the multi-agent orchestrator and the separate verification pass, and replaced both with a single audit agent that stays with your code from first read to final report. It holds the full context of the audit end to end - the files it read, the invariants it wrote down, the hypotheses it already killed - and it falsifies its own findings inside the same reasoning chain that produced them, rather than handing them to a detached reviewer with less context. There is no handoff because there is nobody to hand off to.

It is not a prompt tweak. The agent gets real tools - file reads, grep, and the ability to build the project and run its tests - and one methodology runs across EVM, Solana, CosmWasm, and Cosmos SDK with per-platform specialization on top.

The numbers

We re-ran everything on the same model, DeepSeek-V4-Flash, against the OdinScan benchmark corpus: our EVM and Solana pilot contracts. Same model both times - only the architecture changed.

EVM: the DODO cross-chain DEX

sherlock-991 is a DODO cross-chain DEX: 20 Solidity files and roughly 2,400 lines of Solidity, cross-chain messaging, a Solana adapter, and a custom ERC20 bridge. It is the hardest contract in our corpus and the one where the old stack looked worst.

Every High and every Medium is now surfaced with its exact file and line. The single remaining false positive is a Low about GatewaySend.onCall using a 2300-gas transfer(), which the corpus itself classifies as a non-issue. Even the "noise" it produces is a genuinely defensible observation, not a hallucination - and it is Low, not buried among fake Criticals.

The old pipeline missed one Medium on this contract and returned three extra findings the verifier could not kill. The old stack's recall ceiling was structural: no model in it read the whole codebase, so no finding could chain evidence across the three files where this bug lived.

Solana: PDA seeds

sol-anchor-pda-seeds is a small Anchor program with a single Critical: PDA seeds omit the unique discriminating field, so multiple users map to the same pool.

The old multi-agent run found the bug but also produced a false positive. The new agent found the bug, no noise, and its write-up called out the exact absence of the discriminator in seeds() with a precise exploit path.

Clean repo: no hallucinations

We run a clean, unmodified OpenZeppelin ERC20 as the hallucination test: a contract where every finding is, by construction, noise. The new agent produces 0 findings on it. No invented reentrancy, no speculative overflow, no filler - it reads the code, finds nothing wrong, and says so.

Why this is genuinely more accurate

Same model, so the gains are structural, not a model upgrade. Three things changed:

• One agent retains full context across the whole audit. No information loss between separate agents, no merge step, no summary of a summary.
• Falsification happens inside the same reasoning chain that produced the finding, so the skeptic has at least as much context as the discoverer.
• The agent chains primitives across files and functions instead of matching isolated patterns, so a stale value in one function and a profitable caller in another become one finding, not two.

Running 3-4 models in parallel also multiplied cost and latency, and we now run one. Static analyzers still run in the pipeline, and the agent treats their output as hypotheses to verify in code, never as ground truth.

What this means for users

• Fewer false alarms. 3 false positives became 1 on the hardest contract, and the clean-contract test is silent.
• No real bug buried. 100% recall on every EVM and Solana ground-truth bug in the corpus, including all 10 Highs on the hardest EVM contract.
• Partial results are still real results. The agent records findings as it works, so an audit that hits its budget ceiling returns what it actually proved instead of nothing.
• Tiers no longer mean more models. Every tier runs the same unified audit; the tier selects only the model and step budget. Before, a higher tier meant more models in parallel, which was exactly the wrong lever.

Honest limitations, and what is next

CosmWasm and Cosmos SDK run the same unified methodology and showed 100% recall on our planted-bug synthetics, but they still carry more noise (F1 0.40-0.67) - those platforms are good, not yet great, and custom detectors for their weak categories are in progress.

Two more items are on the roadmap: a larger corpus (we want the benchmark at 200 contracts, with more Solana coverage) and the side-by-side cross-tool comparison against Slither, Aderyn, and Mythril promised in OdinBench v0.2.

Try it

The unified agent is what runs when you scan today. Sign up for a free trial and run your first scan in under a minute. 7-day free trial, no credit card required. Questions about the methodology? Reach out at support@odinscan.ai.