Project Naptime: Evaluating Offensive Security Capabilities of Large Language Models
Summary: Google Project Zero built a framework called Naptime that gives LLMs tools to browse code, run Python scripts, and use a debugger, raising CyberSecEval 2 benchmark scores by up to 20x over the original paper. The approach reached 1.00 on the Buffer Overflow tests (up from 0.05) and 0.76 on Advanced Memory Corruption (up from 0.24), though the team says substantial progress is still needed before these tools meaningfully affect daily security research.
Bottom line: Meta's CyberSecEval 2 concluded LLMs were unlikely to disrupt cyber exploitation in their present state; Project Zero's 20x improvement from methodology changes alone shows that conclusion was a measurement artifact, not a capability ceiling.
Read the original at Google Project Zero ↗
- Topics
- Vulnerabilities & Patching
- Related
- Attackers Exploit UnrealIRCd Flaw That Enables Server-Level Command Injection Cyber Risks · Nov 14, 2026U.S. CISA adds Check Point and UnrealIRCd flaws to its Known Exploited Vulnerabilities catalog Cyber Risks · Nov 14, 2026PostgreSQL Patches 52 Flaws, Including 22-Year-Old Bug Enabling Authenticated RCE Cyber Risks · Nov 13, 2026