Cyber Risks · Trail of Bits Blog

Codex (and GPT-4) can’t beat humans on smart contract audits

Summary: Trail of Bits ran a four-month pilot called Toucan that integrated OpenAI's Codex into its Solidity auditing workflow, and concluded the technology is not yet ready for security audits. The team found Codex could not reason about higher-level concepts like re-entrancy, was easily misled by small syntax changes, and produced false positives and complete hallucinations at rates too high to be useful, with GPT-4 showing only incremental improvement.

Bottom line: In one larger test, five contracts produced 12 positives from Toucan, six of them false, including four complete hallucinations - a failure rate that would bury an auditor in noise rather than extend their reach.

Read the original at Trail of Bits Blog ↗

Topics
AI Security
Related