Benchmarking Open Search and Elasticsearch

Summary: Trail of Bits ran a four-month independent benchmark, requested by AWS, comparing OpenSearch and Elasticsearch on OpenSearch Benchmark workloads using AWS instances, and found OpenSearch v2.17.1 was 1.6x faster on the Big5 workload and 11% faster on the Vectorsearch workload than Elasticsearch v8.15.4 by geometric mean of query latencies. The study ran workloads once per day for 15 days (11 for Vectorsearch), discarded the first of five runs per instance to warm caches, and noted that both products released new versions during testing (Elasticsearch 8.17.0 and OpenSearch 2.18.0).
Why it matters: The benchmark found slow outlier runs in both engines that the researchers could not root-cause, and OpenSearch's outliers were more extreme than Elasticsearch's - so a latency advantage measured on medians does not tell you what your worst query will do.

Read the original at Trail of Bits Blog ↗