And really who would choose to stay in 1938 Germany if you could leave. Even if you are some rich upper class Herr Doktor Professor, life for the next 20 years in Germany wasn't that great compared to England or the US. Why risk having your children killed paratrooping into Greenland. The world is still quite beautiful and quite full of kind people.
Interestingly, Werner Heisenberg was decidedly non-Nazi (and was regularly attacked as a “white jew”), and even though he had ample opportunity to leave, he chose to stay to work on the German nuclear fission program.
I don’t think it tarnished his scientific legacy, but it definitely created some friction in the post-war years.
It's funny how the politicization of science in Nazi Germany led to things like "Aryan physics" to counter the "Jewish physics" of Einstein's relativity: https://en.wikipedia.org/wiki/Deutsche_Physik
Along with the flight/expulsion/imprisonment/murder of top scientists who weren't ideologically or racially "pure", it's no wonder their nuclear program was such a failure.