Personally I think the criticisms are valid if a bit overstated. Diamond perhaps didn't qualify his claims adequately and purported to explain more than he actually could. However, it's certainly worth considering the effect that environment has on societies. I think if you want to argue that European societies broadly had agency in their exploitation of the world, you have to explain how you would expect them to overcome millennia of violence and greed, develop guns and transcontinental sailing ships, then look at Asia and Africa and say "nah, let's leave them alone". So, individuals certainly had agency in whether or not they partook in the rape or bought into the post-facto rationalizations of scientific racism and the like, but societies amalgamate the competing interests of their component parts, and often the worst impulses win out (incidentally, this is the same reason societies "choose" to fail). And then, ask yourself if you think the Han Chinese would have handled it much differently if it had been them instead of the British. Consider the Han Chinese today. People are people, and people are racist.
None of that changes the fact that white people did, in fact, do those things, and in many (all?) cases owe reparations where none have yet been given. Chances are we'll be on to the next atrocious global hegemon long before those reparations ever happen. Maybe we'll figure it out someday.