This is a valid concern, I think. However I'd point out that extraneous detail and world building are not the same. If information serves the purpose of explaining the world then by definition it isn't extraneous. Though adding information certainly reaches a point of diminishing returns at some point.

I think the point of Chekhov's gun is more that unless you can justify the inclusion of the info, then it is extraneous and extraneous info should be removed.