It's not so much the fear and prejudice that's the problem; it's the idea that violence (or the threat of violence) is an acceptable way to force others to do what we want. You might think that most people don't think violence is acceptable, but every time someone says "there ought to be a law..." they are implicitly accepting the the use of force.