A large chunk of that average dose, 4 mSv, is from radon, where the dose varies a lot depending on where you live, and if you live in such a radon-prone location, whether you have sufficient ventilation in your house.
Yes, you're correct. The page I linked to has this to say:
"For example, the health risk caused by radon is estimated on the basis of epidemiologic examinations, not the effective dose. Every year, an average of 280 Finns die from lung cancer caused by radon. Of these cases, 240 deaths are induced by smoking in addition to radon."
So they're not actually measuring an average 4 mSv/y dose from radon, but rather going the other way, that is that 280 yearly deaths from radon would be consistent with an average dose of 4 mSv/y (assuming LNT, presumably).
A large chunk of that average dose, 4 mSv, is from radon, where the dose varies a lot depending on where you live, and if you live in such a radon-prone location, whether you have sufficient ventilation in your house.