Being such a person is considered such a bad thing that it can be used as an insult.
I mean, yes? Ultimately that's exactly what an insult is. Think of the other words we use as curses and insults. Asshole. Mother fucker. removed. Cunt. Dick. Shit(head). Dumbass. Pendejo. Cabron.
Do you want to be something stinky (asshole, cunt, dick, shit), or something disliked (removed, mother fucker)? Do you want to be like the people who the terms moron, idiot, imbecile, removed, handicapped, or disabled are describing? Hell, extend it to other things. If someone 6 feet tall was calling you, a 5'4" man, a midget, it's not like your height suddenly will change if the term is accepted for you, it's because you don't want to be perceived as someone short enough to be termed appropriately as such.
And as much as we can all consider everyone human to be just as valuable as any other human, people aren't suddenly going to want to be short, or have low intelligence / ability to learn/comprehend/adapt. This is why I ultimately have given up on policing the language in general. We are forever locked into the cycle of words becoming inappropriate, because the vast majority of folks genuinely abhor the idea of becoming something like those words are describing, whether its mental ability, height, likeability, worth, etc. You're not going to change that, ever.