If you have to go back to UK or Canada that is no big loss
If, on the other hand you have to go back to China or India
or AFrica then your entire family is #$@$#$%
Canada and the UK are nice places, but it's still a huge blow to have to pick up your family and move back to another country.
Source: I'm American, so I don't know this pain personally, but I was very close with a Canadian co-worker who was here on a work visa. He uprooted his wife and kids from Canada to come and work here. When our workplace soured, he had to do it all over again. Uproot his wife and children again, tell his children to say goodbye to their friends again, buy a house again, etc. Not easy!
find a job
get/consult a lawyer to make sure everything was good
do the whole change
and end up in the exact same situation
Situation: You are tied to ONE employer and if something goes wrong they can just end your job and kick you out
I've done the change so let's not pretend it is some super easy, doesn't matter thing
Also, if you have to leave US where do you have to go?
If you have to go back to UK or Canada that is no big loss
If, on the other hand you have to go back to China or India or AFrica then your entire family is #$@$#$%