> It just seems like it's always been used as a mildly derogatory term by people who really dislike sports and sports fans.

Yeah, actually saying 'sportsball' in conversation in a non-ironic way is an excellent way to signal that you have poor social skills. No need to yuck someone's yum.