I have a glass of whiskey a couple of times a year
I mostly just tell people I don't drink at all
If I tell people that I drink "rarely", they put it into their own framework based on how often they drink themselves. Heavy drinkers might assume "rarely" means one night a week. Moderate drinkers might think a couple times a month