Lemmy.world is down because of a DDOS attack

lemmy-world.statuspage.io Lemmy World Status

Welcome to Lemmy World's home for real-time and historical data on system performance.

106 留言

Someone is really out to get lemmy.world lately.
I feel like with every update there is at least one attack.
- We're having fun and trying to build a positive space. And we have real potential to succeed in growing large. Can you think of a single faster way to attract trolling on the internet?
  It's a lot more likely than someone like spez taking a break from plundering his company to piss off a modest number of internet randos in some internet corner somewhere, which would barely be a drop in the bucket of his problem anyway.
  The overall effect of this is so small, it almost has to be someone(s) with too much time on their hands. If they had any kind of real power, they wouldn't be wasting their time on these chump change attacks.
  
  Honestly, decentralized social media are probably bad news for the current state of the art of disinformation campaigns. The bullshit that has been thriving on Facebook and Twitter is not only a chorus of bigoted aunts and uncles, but (perhaps more importantly) a coordinated attack from state sponsored troll farms seeking, among other things, to destabilise Western democracies.
  The fediverse is, by design, less vulnerable to these attacks. Your trolls can generate activity around your disinformation content all they want: if nobody I follow boosts it, it's not going to show up in my Mastodon feed. And you can feel free to recreate r/conservative or whatever in the fediverse, but if it becomes a cesspool like on Reddit you'll be stuck with your trolls talking to each other on a defederated instance with no-one listening. Disinformation strategies currently employed successfully on centralized social media platforms are likely to fail here, causing a problem for bad actors.
  It is probably paranoid to think there's any geopolitical actor behind the current attack, but I fully expect the fediverse to become under attack from Russian troll farms as soon as they realize they're no longer reaching out to people on Twitter, Reddit or Facebook.
  
  Maybe it was their refusal to take a stance on Meta and Threads? The admins of .ml said it took them 2 minutes to decide to preemptively defederate. .World on the other hand came to an anti-corporate platform and publicly took a position that they would wait and see about federation with Meta.
  It's like saying "power to the people and viva revolution but we are also remaining open to licking boot depending on the circumstances."
  
  White hat attackers maybe?
- it's one of the biggest instances, not really surprising that bad actors are targeting it

This is a shame. Hosting a high visibility server is no joke, and I don't envy the admins and the very difficult work they do. It's simultaneously an argument for and against decentralization. For - a single instance can get knocked out without talking out the whole fediverse. Against - it seems as though high visibility communities are potentially fairly easy to target and take down.
I think that decentralization wins out here in the end, but it does feel like there may be a need for some sort of fallback mechanism to be in place at an instance/community level. I suspect this might evolve somehow over time. It would require some way to expand trust between instances and or portability of communities (which could be fraught with user trust/data integrity issues).
If things don't evolve it could grow into a whack-a-mole game for bad actors, or there might need to be more investment into server infrastructure (which could work against decentralization if only because of economies of scale).
Or maybe there's no issue after all? I'm just imagining potential implications of a scaling fediverse - it's fascinating and exciting stuff!
Thoughts?
- This is the primary reason why I'm ok for my instance to not grow massively. We got 10K people and we have pretty good traffic ,without overloading us or making too much of a target. We still get new users since we allow registrations, but the application requirements retain the quality
  
  I'm realizing that I signed up for a probably-at-risk instance (lemmy.ml). I'm quite left but not necessarily an anarchist so it would seem applying to lemmy.dbzer0.com wouldn't be a good move. (But I did enjoy reading your application requirements!) Recs on other small but reliable instances?
  
  Absolutely makes sense. If lemmy is going to have any truly large communities though, investment in infrastructure/ops as well as function/moderation will be absolutely needed. (It's an 'if', of course)
  Time will tell how the community will want to lead it.
- a single instance can get knocked out without talking out the whole fediverse
  Honestly, it may as well have in this case. LemmyWorld is the de facto "hub" for basically the entire Threadiverse right now. All the major communities are seeing the most activity through LemmyWorld. While I'm subscribed to a lot of communities from other instances, sometimes duplicates of ones found on LemmyWorld, losing LemmyWorld would still a huge chunk of the content that I'm trying to see.
  I really do wish that more specialized instances would sprout up and that some of these communities could cluster together across multiple pockets of the Threadiverse. I feel like this makes it less likely to lose huge chunks of content, and also makes fewer large targets for somebody to want to attack in the first place.
- You don't need to necessarily centralize to defend against DDos or similar attacks. You can add things like Cloudflare for DDos mitigations, CDN and maybe something like Kubernetes for horizontal scaling of servers (spin up more servers to handle extended load) transparently behind the scenes. This can also get you the benefits of low geographical latency, so a load-balancer fetches you data from the closest replica of a database geographically, etc.
  Of course, all this adds up in terms of cost, but I think this might be worth it for the largest instances. I suppose that can still be considered centralization.
  If we wanted to encourage small many small instances instead, perhaps there could be a transparent load-balancer layer for the fediverse that instances could sign up for, that is managed by a devops group. Alternatively, lemmy could have built-in load-balancing, caching, etc. as part of its codebase that instance operators can set up with their own accounts at Cloudflare, etc.
  
  Agreed. Ultimately, that's the point. There are solutions (with ongoing vigilance required) but it comes with an ongoing cost, be it server infrastructure or human resources).
  I think the federated load balancer might be interesting but I expect there are many pitfalls that need to be considered and addressed wrt security, trust and integrity of data.
  Anyway, it's amazing to see this all grow and evolve.
  
  Yeah everyone using Cloudflare is definitely centralisation, but maybe a kind of centralisation that allows for easier switching to something else if Cloudflare gets too crazy.
  DDoS is a war of attrition - and the best way to win a war of attrition is to make it cost much more than $1 to make you spend $1, and to be able to outspend the attackers (e.g. the whole community bands together to support the victims against the attacker). I think the best response depends on who is attacking.
  Network level DDoS is likely using stolen bandwidth - but the person directing the attack is probably paying someone for the use of it (i.e. they didn't compromise the equipment themselves, someone else builds botnets and rents them out). If you can identify what traffic is part of a DDoS, you can track down where it is coming from, and alert the owner of the network where it is coming from, which hurts the person providing the services to the attacker quite a lot. If I have a reputation of: if you attack me for someone else, I'll cost you a significant part of your business that will take you months to build back up, then you are not going to offer that service cheaply, or even at all.
  Application level DDoS usually relies on amplification of cost - I do something relatively inexpensive (like send a packet opening a connection), and it makes you do something really expensive involving databases, disk IO etc...; a good mitigation is to redesign the API to flip that on its head, so you do something expensive, and I do something relatively cheaper for you. There is an open issue about using Hashcash to do just that at: https://github.com/LemmyNet/lemmy/issues/3204 - the downside is that it forces users (even on mobile devices) to use more compute / power for every request to Lemmy, but I think there is a balance that can be struck there where it isn't too bad for users, but makes that type of attack infeasible.
- I think this might be interesting:
  permit separate, low-traffic, highly rate-limited, auth-only servers. They would be strictly rate-limited and only accept connections from whitelisted partner servers, because they only handle auth.
  any partner server can authenticate a user and handle content for the server/auth-server pair, but only does so under certain conditions (determined by the partner - all the time, when ping api call > n seconds, or manually, for example)
  user@lemmy.world can't log in, so the client tries the list of partnered servers. user succeeds at lemmy.partner.net.
  user@lemmy.world@partner.net says.. '..something' and all other servers accept it as being from user@lemmy.world
  lemmy.world recovers,, and claims all of the @lemmy.world@partner.net posts. Partners then forget the extra stuff they've been hosting.
  
  The problem with these types of redundancy schemes is that it simply takes a Internet backbone hiccough (or AWS fuck up) to cause there to be multiple primaries (i.e. lemmy.world is online still, but some portion of the internet can't see it, so a replica promotes itself to primary, people use both, how do you reconcile it).
  This is not even beginning to talk about the nightmare scenarios possible if someone hacks a replica.
  Edit: Still, this is a good thought and similar to how some actual software packages do things.
- Can't post to op... But... Somebody just s scared.

Drat, someone really doesn't like lemmy.world and how active it became.
- Big target. Either that or the butt wipe that was denied his Reddit username and started creating random long manned communities.
  I just sort of assumed we’d all get accounts in 2-3 instances so if one goes down we can still participate elsewhere.

Thanks for the update. Was wondering why I was having issues logging in.

Also the other instance I was using ... vlemmy.net ... has been down for like a week now. I wonder if it's completely done for?
- Definitely dead according to some of the other posts I've seen
  
  Do you know what happened to it? Just curious.

Someone is really out to get lemmy.world lately.
I feel like with every update there is at least one attack.
- maybe that dude with the mass random communities name
  
  Oh ya, I forgot about that guy. What even is the point of doing that?
  
  Wasn't that the person that hacked a few instances a week or two ago? @LMAO@lemmy.world

Logged on from Lemmy.world through voyager to see if this comment posts…
- I can see your comment from my instance page :)
  
  Lemmy.world seems nice and fast through the app at the moment but maybe the website is having trouble
- Ya. I'm still logged in through the Liftoff app and able to post. Idk why my first comment on this post was duplicated though.

Looks like lemmy.world is back up. vlemmy.net is still down.
- vlemmy.net has been down for a week or two at this point and the admins took down their donation links. I think it’s safe to say they’re not coming back.
  
  F

I have also had problems with my lemmy.ml account. Maybe the same attack??

I wonder who is behind this and what's the intention.
- Speznaz, probably. ( ͡° ͜ʖ ͡°)

106 留言