I've always heard Buddhism referred to more as a spiritual practice than a religious one. Buddhism has no "deity," the Buddha would not have condoned the worship of himself as a God. Buddhism's whole thing deals with karma and the path the self enlightenment, which is the key difference between Buddhism (and other spiritual "religions"): spirituality is self contained, religion isn't.
Per most actual religions, you need to worship in certain places, listening to certain people, and many religions encourage the conversion of those outside of the religion (sometimes by force). The community aspect is a requirement of the religion, it's not considered "kosher" to just do it all by yourself in your house.
And thus, these religions tend to be organized: they have a hierarchy, a power structure, hell, some ever have their own governments/countries. They protect their own, shielding them from the law despite their pedophilic transgressions (cough cough Catholicism), they require tithes or their patrons but rarely offer financial assistance to their own parishes.
Buddhism can be practiced in groups (monks and monasteries), but those groups are not forcefully trying to convert outsiders (though they may welcome them), and they aren't necessary. You can practice Buddhism entirely on your own and still be faithfully practicing all of the tenets of the "religion," despite never interacting with any type of significant religious figures.
I've never met a Buddhist who sat down next to me at dinner and started asking when I'm going to accept Buddha and start going to meditation and find my inner chakra and give up all of my possessions for the pursuit of pure enlightenment, or tell me I'm going to reincarnate as a disfigured creature because I'm trans.
I have been told by various folks of various Abrahamic-Religion flavors that I'm going to hell for being trans, need to repent/kill myself, been asked when I'm going back to church, how I can turn my back on my creator, told religion will fix my depression/mental issues, and so on.
And the fact that the majority of religions rely on the indoctrination of children for their member base to remain consistent/strong enough, rather than allowing people to find their teachings and accept/deny them for themselves when they can consent as adults, should tell you most religions are more akin to cults than anything actually beneficial to society.