Evaluating Solidity support in AI coding assistants

Summary: Trail of Bits evaluated popular code models on Solidity completion as of August 2024 and found Anthropic's Claude 3 Opus the best commercial model and the largest DeepSeek Coder model you can comfortably run the best local one, with local models outperforming commercial offerings on partial line completion. The firm added Solidity support to Continue.dev and Tabby and contributed upstream to tree-sitter-wasm, and it built a completion evaluation harness called CompChomper because Solidity appears in approximately zero code evaluation benchmarks.
Key Takeaway: Trail of Bits says it has encountered bugs in AI-generated Solidity during audits, with models emitting code that worked for known patterns but failed on the actual customized scenario, and it calls using AI assistance for anything beyond intelligent autocomplete an egregious risk.

Read the original at Trail of Bits Blog ↗