Amdahl’s law says we’re only going to get marginal improvements in a lot of cases. But that’s ok because sometimes marginal improvements still pay off.
Increasing efficiency doesn’t always mean shorter runtime, sometimes it means more production from an equal runtime.
Lots of improvements to different places in the pipeline can pay off.
Sometimes an improvement in one part of the process can have synergistic knock-on effects in other, perhaps unexpected, parts of the larger system. It seems like these can be hard to predict without taking a deep look at the larger system and how all of its component parts interact.
At least twice during my career I’ve had to shrink some binary output so it can fit onto a fixed size part in some embedded system.
Once it was a FTTX CPE, and it needed to ship but the binary was a couple KB too big. The programmer on the project found a little bit of savings but not enough. I looked at the code, looked at the assembly output, and found a spot where a switch/case was being transformed into a huge jump table that was mostly sparse. (The embedded toolchain compiler wasn’t great at optimization.) A little refactor to this one function was enough savings to be just under the limit for whatever chip they were using. (I feel bad for the next person who has to fix a bug in that product.)
The relevant aspect to my story is that sometimes an improvement in one area can make another area worse! But that’s ok, because even though my function rewrite was almost certainly slower, we had cpu cycles to spare but not enough storage.
