Agreed. We've had lots of "team building" types of things that might have otherwise been fun (and mostly were), but ours always get turned into something super-competitive. It can't be just "get out and do something as a group", it has to have a set of winners. So our activities always get skewed to something like that. So the actual activity is the very opposite of "team building".
But I guess thing is that the very idea of a "team" at work is total bullshit anyway, at least as far as any company I've ever worked for. Ever notice people don't get compensated as a team? Hell, people don't even know how others are compensated and if the "team" has a success, it's not usually as if they all share in the rewards equally, or even merit-based. It's all gut-feel, combined with who has what title, no matter how much time/effort someone puts in. So someone who was only just added to a "team" weeks before a major initiative is finished gets showered with praise, money, benefits - if they are an executive or at least have the right title/relationship with people making such decisions.
I think workplace events (or offsite ones) just tend to annoy nearly everyone in the actual trenches, if you want to be honest. Add in something that is political in nature and I think it only gets worse.