>"Because speech doesn’t cause physical harm or loss to others."

Nope, that is trivially wrong with the obvious counter examples being slander and libel.

Which is why they're excluded from the definition of free speech pretty much everywhere?

Couldn't "psychological harm/turmoil" fit somewhere in there aswell? Verbal abuse etc.

generally speaking that would fall under harassment.