That's because it's supposed to be. I was on Reddit for a decade until their management shit the bed, and these kinds of problems weren't a thing there despite the much larger userbase.
For the record, to me it's less about privacy and more about setting expectations. I'm not anonymous online, I'm pseudonymous, I've had this handle for a long time. I am my online identity, and when I post and vote I don't feel anonymous, even if I'm relatively protected from someone knocking on my door or messaging my boss about a statement.
If voting "ledgers" aren't presented in the discussion, that's because they aren't intended to be part of the discussion. This reduces the value of influential individuals votes (ooh Bill Gates liked X, Kamala Harris disliked Y etc.) and shifts focus to how the community values of the content. It’s the same reason that we follow communities rather than individuals. We get an internet "hive mind" of sorts without cult of personality.