Yes, the practical impossibility of recovering a private key from a security device is a feature. This implies you can't change security devices without changing public keys.
Yeah, I just didn't know how it works; didn't know it was stored there, but assumed it was just encrypted with the secret in the key, i.e. in the presence of the old and the new security key, it could be decrypted and re-encrypted with the new one.
