The argument is symmetric; once you've tuned the algorithm the only way to get >10x speedups is by working with those low-level things.

Both the broad-strokes algorithm and the fine details are responsible for huge performance differences.