While this is good advice, best practice is to always get your yubikey in pairs and keep them synchronized. One should remain in your home, in a safe place (as you described) while the other should remain on your person or outside the home (e.g. in a safe deposit box)

It’s more of a pain in the ass for sure, but handles the theft scenario more effectively