That is how it works, I thought that was the whole point. It's a game about greed. You can either play it safe and go up 1 by 1, or go fast. The max you can get is 5 if you press immediately, seems to drop down to 1 after half a turn.

EDIT: Found the code

    (tapInterval < 500 ? 5 : tapInterval < 1000 ? 2 : 1)

So 5 points for under 0.5s, 2 points for under 1s, 1 point otherwise.