Is anyone else apprehensive to down voting even if you're being down voted?

I can't say I never downvote anyone but I get into these comment chains from time to time where really no matter what I post whoever I'm talking to downvotes every response but I generally shy away from downvoting even things I dislike. I don't want to fall into an echo chamber where all I see are only the things I agree with.