Can chatbots craft correct code?

Summary: A Trail of Bits engineer at the AI Engineer Code Summit in New York describes attendees arguing developers will soon never need to look at code again, and counters that the analogy to the shift from assembly to C misses determinism: compilers preserve semantic intent, while LLMs are nondeterministic by design. He cites Claude Opus 4.5, Gemini 3 Pro, and ChatGPT 5.2 all translating the same C snippet to Python with changed semantics around signed integer overflow, and a Claude-generated pull request that silenced a crash in his Vendetect tool by rewriting a bounded recursive function while breaking its core functionality.
Why it matters: The bug Claude confidently diagnosed and fixed was not a bug at all - the real cause was two files differing only in filename casing on a case-insensitive macOS filesystem - and the fix passed the eye test while emptying the tool's output and failing its unit tests. An assistant that reports no uncertainty while rewriting working code is the failure mode to plan for in any legacy codebase where the context that matters is not visible in the code.

Read the original at Trail of Bits Blog ↗