Seems like you’re the only one mixing the two from what I can see (and furthermore adding to your confusion by conflating sex and gender as well). Sex is often the preferred term when referring to biology (for instance, “in many species, members of the male sex are larger than those of the female sex”), while gender is the preferred word when referring to behavioral, cultural, and psychological traits (for instance, “young people more likely to challenge the gender norms they grew up with.”). Sexuality is ones orientation in terms of sexual and romantic attraction. So you can have a woman (gender) who was born with male sex parts (sex) who happens to be pansexual (sexual orientation).

The reason people “care” so much is because if someone tells you how they wish to be identified and referred to and you not only refuse to do so but go out of your way to misidentify them, it shows blatant disregard and lack of respect for them as a human being. If you don’t care what people refer to you that’s fine. But that’s your personal choice and you are free to make it. Those who do care about what we refer to them as are likewise free to care. And while yes you have the freedom to not respect their request, it’s an inherently degrading, dehumanizing, and downright anti-social act. You’re going out of your way to make a big deal out of it and othering it rather than just using the terms people requested and moving the fuck on with your day.