> a pattern I notice is that, a lot of these places are chuck full of reject physicists

Utter nonsense, these are some of the smartest people in the world who do incredibly valuable science.