Detecting Bad OpenSSL Usage

Summary: A Trail of Bits intern prototyped Anselm, an LLVM pass that searches code for bad sequences of API calls, starting with OpenSSL usage such as calling EVP_EncryptUpdate and EVP_EncryptFinal_ex out of order or reusing initialization vectors. The tool performs depth-first search over a function's BasicBlocks, prunes irrelevant blocks to speed path-finding, caps path length to handle loops, and matches arguments and return values so a rule can require that every context from EVP_CIPHER_CTX_new is later initialized by EVP_EncryptInit_ex.
What's compelling: OpenSSL's low-level API inconsistently returns error codes and pointers, and without rigorous checking, unexpected program behavior and process termination can occur - and Anselm's detection is heuristic, so misidentifying both good and bad behavior is always possible.

Read the original at Trail of Bits Blog ↗