the world is an interesting place, the very reasons you gave "for" it is why I was against it. I don't agree that it won't cause witch hunts, and from the POV of the commentor it might be nice, but from the POV of the person who is giving the vote, it's a severe downgrade.
Especially considering the fact that if the person downvoted but didn't leave a comment afterward they likely would not have downvoted in the first place if it wasn't anonymous because they don't want to have to deal with the social interaction of someone trying to push them to explain further. Not everything needs a detailed this is why I feel this way, that's why there is a upvo and down vote system in the first place, to prevent everyone from leaving a comment of I agree with this / I disagree with this / this is on topic / this is off topic
In addition to this, to say that no one's giving reasons of why voting should be private, I don't think that's a truthful statement there are plenty of reasons that people have provided via privacy, security and sometimes just mental state.
You mentioned that you want to have a system where you choose what people you see and the people you don't agree with don't appear., I think that type of environment is extremely unhealthy for a social media platform. It's why other platforms that have curated that content is starting to become a cesspool. I really don't want to see lemmy become one big Echo chamber, it's not healthy to have only one ideology that you see at all times and let's face it that's what that system you're proposing would introduce.
Additionally the system your proposing is going to run into the same issue as the other websites that have attempted to do, this sort of system leads to new people inadvertently getting filtered out as untrustworthy, which will mean that they're not getting activity on their posts/ comments as well which means that they're just going to move on to another platform.
Honestly, I think I would rather just have the score system be removed as a whole then see that type of system implemented