Then you're not a deontologist. It's definitional. Actions have inherent moral value, regardless of the situation. If something is ever not permissible, then it's always not permissible. That's what deontologists believe, isn't it? If not, then you're still taking a consequentialist stance on morality, but with extra steps that allow you to claim that your own inaction is actually the right thing to do, but only when you decided not to take action.
If you don't think violence is always wrong, then what's your excuse for not pulling the lever, or voting for the person who would cause less suffering? To my eyes, those are situations where the "violence" I'm commiting is permissible because it leads to less suffering than inaction would.