It's more than just the email. If you're in the breach, it might now publicly tie your email to things like your real name. You also have to worry if you reuse passwords (which you shouldn't do even if you haven't been in a breach), because now the password in the breach is known to be used with that email address, and attackers will pivot to other services to try those same credentials elsewhere.