With modern hardware it is very easy to serve millions of daily users from a single machine with simple software.
No one bothers because everyone loves their fast-as-a-snail framework in their favorite slow scripting language and they need massive infrastructure to scale it to serve less than 1k concurrent users.
Speak for yourself. Tools these days are much more ergonomic, not to mention faster than ever before. I host a website on DigitalOcean with 0.5 GB RAM and 1 vCPU that can support millions of requests and runs with just a git push. Try doing that in the 90s.
I always feel bad when I see novice programmers herded toward frameworks like Flask and Express for uncomplicated API cases, you can take your pick of mature C#, Rust, Java and C++ API frameworks that serve ~7,000,000 responses per second compared to Express' and Flask's ~123,000 and ~15,000 respectively.
This is a gross generalization. And I'm not sure where you are picking up those values from as well or their accuracy, for Flask is ~15,000 on uWsgi or Gunicorn? or the default werkzeug dev server? How you deploy matters very much I'm sure you'd understand.
However, Ultimately development teams make do with the skillset they have and saved development time is saved money. Who cares of the toolset, Most programmers forget about the business objectives ultimately. Instagram's backend was originally Django, they managed just fine for a long time. Using C++ is pretty overkill even for a simple API project and potentially a huge waste of developer time, people seem to forget developer time is expensive.
I don't think it's fair to call it a gross overgeneralization; good software engineering is being able to evaluate and select the right tool(s) for the job than try to bastardize their fave lang into everything.
The numbers were from skimming TechEmpower's benchmarks. I don't have a horse in the race as I don't do (much) web dev but I found the disparity amusing even if the reality isn't quite so bleak due to, as you mentioned, many other factors before you even hit the application itself.
Ultimately I agree with you, I'd rather see someone use a framework they're comfortable with if it makes their life easier - but I won't lie to them and pretend it's optimal.
You're completely ignoring the fact that most applications will never see 15,000 requests per second. Most applications are internal apps used by a few hundred employees during working hours, or small niche SaaS products used for specific tasks by a limited group of dedicated users.
If I'm working on one of those products, prioritizing requests per second over any other factor is bad engineering, period. That's not to say that I wouldn't end up choosing a mature compiled language in the end, but there are plenty of reasons why I might not, and you have no right to judge without knowing my constraints.
No one bothers because everyone loves their fast-as-a-snail framework in their favorite slow scripting language and they need massive infrastructure to scale it to serve less than 1k concurrent users.
Programmers have no one to blame but themselves.