I ended up moving away from Postgrey because the delays (especially coming from Gmail) were a huge problem. Since the mail would come from some random IP in Google's massive server farm, each time it tried to get delivered, postgrey would delay it again.