It really doesn't. It only feels that way if you are incapable of differentiating between a sexual pervert and a trans person.
Society can accommodate gender neutral bathrooms (meaning you don't have this social construct of genital specific genital policed bathroom spaces) alongside gendered bathrooms where people presenting as either gender can go to the bathroom. Where there is no neutral option, the closer aligning gender specific bathroom is better than the genital specific bathroom.
Or you can have it your way with bathrooms where people's genital have to be declared, revealed, matched, or somehow verified to use them. That is the only way to achieve genital specific safe spaces that you're craving. Otherwise our society can go about accepting that people presenting as a gender to use their preferred bathroom regardless of their genital status.
So how do you want verify if someone is trans or not when they're using the bathroom? Keep in mind you've already demonstrated that you're incapable of differentiating between a sexual pervert and a trans person.
The sky is the limit here:
How do you want our society to verify genitalia before providing public bathroom access?
