This is about proving to Google that you're secure. Google doesn't know if the password you entered came from a password manager or not. But if you're using a hardware key, they know it's secure.
If you lose your hardware keys, you still have your other 2 factor options, so you are no worse off than your current situation.
Thing is, history has shown that nothing is reliably enough for Google, once it flags you suspicious. You've entered password and totp code? Nah, you're still suspicious. Gave one time backup-code? Hah, still suspicious. Have a hardware key? Nice, but you know you are really suspicious. How else can you prove that it's you?!
If you lose your hardware keys, you still have your other 2 factor options, so you are no worse off than your current situation.