I kinda get where you're coming from... but just as an easy example...
well-informedly isn't a word.
I have now told you what is and isn't a word, dispote you explicitly not consenting to me doing so. What happens now?
In this case? It isn't a big deal. You can tell me to pound sand or ignore me or try to get into a flame war with me over it, but your lack of given consent doesn't inherently prevent me from doing anything to you so I wouldn't agree that its just the way things are naturally.
There are cases where it is much more important. If what you stated were true, there would be no concept of rape because nobody could do that to you without your consent, and if you have consent, it's not rape. Same with murder... the closest thing would be assisted suicide. I'd also argue that the Palestinians haven't consented to what is happening there. In each of these scenarios, one entity refused to consent, but it didn't impact the other perpetrator from continuing because the victim didn't have enough strength or weaponry or whatever (a bigger stick).
I honestly don't see how you can have a society without both a mechanism to determine what the shared rules are, someone with a big enough stick to enforce those shared rules. Obviously, the tricky part then becomes ensuring that the enforcers only use their stick when its appropriate. Otherwise, you just end up with authoritarianism.