I suspect you don't really guess that. There are differences between the two cases. For example, security threat models are a thing. Something can be secure against the threats it will face. I don't ssh -X into servers I don't already trust. There is no arbitrary code being run.