If the button is not in all caps, i.e. the first letter is substantially taller, then aligning by baseline & cap would make the text as a whole look too low (depending on x-height).

Icons can be less ambiguous, but still it depends on the shape and white space within one and the interplay with text label.

I think it all depends on your vision and its consistency.