It's an "all other things being equal" sort of thing, unless you're actually having a discussion that acknowledges intersectionality.

To use your example, if you're poor as fuck, is it worse if you're white or if you're not? For many, it's worse if you're not white - and that's white privilege.