You'll be extremely confused if you only think about very specific developmental disorders. That's not how highly polygenic traits like height and IQ work in general population. Some people are taller than others, and some people have higher IQ than others, and, to our best knowledge, this within population (or really, within cohort) variation overwhelmingly is not due to developmental disorders or environmental setbacks like e.g. malnutrition (though at the extreme levels, severe malnutrition or very debilitating disorders can and do have significant effects), but rather due to lucky draw of beneficial gene variants from the parents. As it turns out, some of the beneficial variants for height are also beneficial for IQ, this is called pleiotropy.

More directly to your question, yes, as I said in my previous comment, the effect is very real, and is not driven by some outliers. There have been dozens studies on this, with Ns in tens of thousands. Wikipedia has a sample of them: https://en.wikipedia.org/wiki/Height_and_intelligence