Codex (and GPT-4) can’t beat humans on smart contract audits
Summary: Trail of Bits ran a four-month pilot called Toucan that integrated OpenAI's Codex into its Solidity auditing workflow, and concluded the technology is not yet ready for security audits. The team found Codex could not reason about higher-level concepts like re-entrancy, was easily misled by small syntax changes, and produced false positives and complete hallucinations at rates too high to be useful, with GPT-4 showing only incremental improvement.
Bottom line: In one larger test, five contracts produced 12 positives from Toucan, six of them false, including four complete hallucinations - a failure rate that would bury an auditor in noise rather than extend their reach.
Read the original at Trail of Bits Blog ↗
- Topics
- AI Security
- Related
- SECURITY AFFAIRS AI-CYBERSECURITY NEWSLETTER ROUND 8 Cyber Risks · Nov 15, 2026Pentagon Renames CDAO ‘Super Intelligence Office’ Cyber Risks · Nov 15, 2026186 Fake AI Domains Redirect Users to Click Fix Lures, Browser Malware, and Search Hijackers Cyber Risks · Nov 12, 2026