Of course there should immigration regulations, but because that's the comment you chose to make at this moment in this context, people are probably assuming that you believe the talking points about there being some big huge problem with illegal immigration in America, that somehow necessitates things like building a giant border wall and enabling ICE, etc. That would be eminently downvote-worthy, but of course you didn't say any of that. It's sort of like someone making a "both sides" comment after Charlottesville, it's hard to make a comment like that without it sounding like a dog whistle for some really bad people.