I expect the accent is part of it, actually. Call it the Henry Higgins effect.

  Rose: If you ARE an alien, how come you sound like you're from the North?

  The Doctor: Lots of planets have a North!

(the joke lands better for American audiences if you mentally replace "North" with "South"; in many ways the economics/stereotypes are similar)

https://www.youtube.com/watch?v=svi-fBXZzqI