That can be a real pitfall for benchmarking. I recall looking at some promotional material touting the performance advantages of Rust against some other language (C?) It involved a loop so I kept increasing the loop count to try to get the Rust execution to take the same time as the other language. It took me a while but I finally figured out that the way the code was structured, the Rust compiler could see that the results of the computation were never used and the loop was simply optimized away.

Benchmarks can measure execution time of algorithms or if not careful, the ability of the ability of the compiler to optimize away unnecessary code.

(IMO the 'performance comparison' that was vastly in favor of Rust was a bit disingenuous.)