> It’s hard to come up with a nice visual for it though

Suggestion off the top of my head: down arrow / thumb down; circle / horizontal dash; up arrow / thumb up; star / heart.

The circle / horizontal dash could be ambiguous in isolation but should be clear in context.