1y ago

Lemmy.world status update 2023-07-04

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

Loading is mostly OK, but sometimes things take forever
We (and you) see many 502 errors, resulting in empty pages etc.
System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

325 comments

I'm just excited to be back in the Wild West again -- all of the big players had bumps, at least this one is working to fix them.
- And giving updates!
  
  I’d rather have to deal with hiccups and bumps along the way, because the community only grows more each time.
- I still remember early reddit days of 502 it went through, 504 try once more.
  
  Early? Haha, reddit still shits the bed regularly
- Yee-haw, mother fuckers.

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Lemmy has a memory leak? Or, should I say, a "lemmory leak"?
- A pretty bad one at that...
  
  But... but... Rust...
- Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
  A lemmory meek, obviously!
- On some of the latest release candidates I think so.
- Wait isn't lemmy written in rust how do you create a memory leak in rust? Unsafe mode?
  
  undefined
  
  let mut memory_leak = Vec::<u8>::new(); loop { memory_leak.push(0); }

Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn't happen without you!

I want this to succeed so badly. I truly feel like it’s going to be sink or swim and will reflect how all enshitification efforts will play out.
Band together now and people see there’s a chance. Fail and we are doomed to corporate greed in every facet of our lives.
- Wow. Drama! Chill bruh. They'll get there. If it takes too long for some, they can switch to other instances.
  
  That’s just a lemming living up to their name!
  
  I'm here for the drama too!
  Fuck the cleanliness of the corporate web.

Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.
Seriously, you're doing a great job <3
- Great distraction from the collapse

i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.
but please, don't feel forced to overwork yourselves. i understand you want to do it soon so more people can move from Reddit, but i wouldn't like that Lemmy software and community developers overwork and feel miserable, as those things are some of the very motives you escaped from Reddit in first place.
in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.
this applies to all lemmy instances and communities, ofc. have a nice day you all! ^^
- Plus, slow steady growth means eventual success. Burnout is very real if you never take a break.
  
  so true, pal! slowly, with patience, no rushing, putting love into it, organizing ourselves, working smart is better than working hard and fast.
  because of the federated nature of fediverses like Lemmy, it is very possible that many people are doing the very same task without even knowing they are duping each other's efforts.
  and that's sad because if they knew, they could be teaming up, or splitting the task in two, in order to avoid wasting different efforts into dupe results.
  i have learnt a thing or two about burnout, it's better for me to make 40% planning, and 40 % self-care and so the 20 % of execution becomes piece of cake.
  but this is just my opinion. anyway, please take care, pals <3

As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I'm remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.
- I remember when Voat came out and the slight exedous that brought. I made an account and everything but it never properly took off. I checked on it two or three years later and it was just filled with alt-right/racist/transphobic garbage. Sad it never took off as a reddit alternative, reddit likely would have greatly benefited from a proper alternative, not sad it closed down after I saw what it ended up.
  So far the fediverse feels really different tho, very explicitly anti that type of shit. I'm sure it will pop up, they always do, but maybe now people know how to deal with it. Block it, defederate, deplatform.
  
  FYI, it has popped up explodingheads is a great example but many servers including lemmy.world became proactive in defederating from the instance
- Lemmy is a distributed system that can't quite be perma-killed, kind of like ye olde usenet. Anyone can create a new instance at will.

As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You're doing great with such fast exponential growth
💙 Thanks for your hard work!
- Amen. So happy to be done with Reddit.

This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.
- You said it: passion projects. While being paid is surely a motivator, seeing your pet project take off the way Lemmy is can be so intoxicating and rewarding! I plan to donate as soon as I get paid on Friday! I want to see this succeed, even if it is just to spite Reddit, and I am willing to pay for the pleasure.

What was that? We're going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?
Yeah, that sounds pretty reasonable, we can work with that.
- There is a sidebar?
  
  On web browser, yep.
  
  Every community has a sidebar. If you’re at the page for the community it should be at the top of the website. “sidebar+”
  If you’re using a 3rd party app it’s different for each one in terms of how to see it.
- Just signed up for the 8€/month tier, hopefully my blue checkmark will arrive soon!
  
  It’s here!
  ✅
  (Please do note that the blue checkmark is for our 1€ “Bean” tier subscribers.)
  We’re very pleased to provide you with our premium deluxe green checkmark for your generous 8€ subscription to our “Bacon” tier!
  Feel free to reach out to our Lemmy Cares team @ 1-800-FUC-SPEZ for any questions or concerns that you may have.

Could I get a discord invite? I’m an ex sysadmin with a. Lot of free time

@ruud That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻
- We have 128GB of RAM. It just skyrockets after a while!
  
  @ruud Oh damn. This spontaneously sounds crazy but I’m admittedly a novice at servers on this scale.
  
  I haven't checked into the code yet, but I imagine you can map out what all is in memory and force more aggressive garbage collection to find some middle ground.

Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.
- Not just that, but the code contributed to Lemmy by this debugging will make Lemmy run faster for everyone on every instance, which is makes the ecosystem that much better.

The work you're doing is greatly appreciated! It's like you invited half the internet into your house. I feel like I should've brought a cake or something
- You should consider setting up a small donation to the site, if you can afford it. If you throw $5 a month at them, you're not only covering your own hosting fees, but covering for some of the folks who can't afford to as well. And maybe Ruud gets to buy some beans.
  
  It appears I have no idea how to imbed an image, but good call. Subscribed.

Huge respect for what you've built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here
- Yes he's one of the other admins in our Discord, he's very helpful!

It’s a ton better this afternoon! Thank you!!
- I’ve noticed that too!

Of course these performance issues are a bit annoying, but I gotta say that I love these updates and explanations here. Great communication, keep it up, please!

.world is definitely running smoother than when I joined 3 days ago, back then it was impossible to comment and the lag was immense, right now I just have to occasionally reload the page, but that's nothing in comparison.
You guys are doing an amazing work! I'm broke, so here are some ~~coins 🪙🪙🪙🪙~~ beans 🫘🫘🫘🫘

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
who'd have thought memory leaks would be possible in Rust 🤯
(sorry not sorry Rust devs)
- Ackchually, leaking memory is totally possible in Rust! But so is hogging way too much of it because you're hanging on to too many things in memory at once🤷
- c/programmingcirclejerk when?

Keep up the good work!
I created an account in lemm.ee until the issues are fixed. Then I will happily go back to my lemmy.world account.
- Lemm.ee is also a good choice!
- If I have an account at lemmy.world, but have another instance selected, does that take some of the burden off the admins or do I need to do something more?

Cloud architect here— I’m sure someone’s probably already brought it up, but I’m curious if any cloud native services have been considered to take the place of what I’m sure are wildly expensive server machines. E.g. serve frontends from cloudfront, host the read-side API on Lambda@Edge so you can aggressively and regionally cache API responses, anything other than an SQL for the database — model it in DynamoDB for dirt cheap wicked speed, or Neptune for a graph database that’s more expensive but more featureful. Drop sync jobs for federated connections into SQS, have a lambda process that too, and it will scale as horizontally as you need to clear the queue in reasonable time.
It’s not quite as simple to develop and deploy as docker containers you can throw anywhere, but the massive scale you can achieve with that for fractions of the cost of servers or fargate with that much RAM is pretty great.
Or maybe you already tried/modeled this and discovered it’s terrible for you use case, in which case ignore me ;-)
- You were so close until you mentioned trying to ditch SQL. Lemmy is 100% tied hard to it, and trying to replicate what it does without ACID and Joins is going to require a massive rewrite. More importantly - Lemmy's docs suggest a docker-compose stack, not even k8s for now, it's trying really hard not to tie into a single cloud provider and avoid having three cloud deployment scripts. Which means SQS, lambdas and cloudfront out in the short term. Quick question, are there any STOMP compliant vendors for SQS and lambda equivalent yet?
  Also, the growth lemmy.world has seen has been far outside what any team could handle ime. Most products would have closed signups to handle current load and scale, well done to all involved!
  
  Rust FTW
  
  If Postgres becomes the bottleneck I wonder whether something like Citus could work to shard the data (relatively) transparently?
- cloudfront helps a lot with the client and is absolutely compatible with lemmy if you set it up correctly. possibly it could also help cache api responses, i haven't looked into that part yet.
  the database, on the other hand, would need a nearly full rewrite. lemmy uses postgres and dumping it for something else would be a huge pain for the entire federated community. it could probably tear it in half.
  there's also the issue of pictrs, which uses a stateful container and isn't yet able to use an external database which would allow you to scale it horizontally. resolving that one is on the roadmap though, and for the most part you can aggressively cache the pictrs get requests to alleviate the read-side load.
  but whatever the solution is, it kinda needs to be as simple as developing and deploying docker containers you can throw anywhere. the vendor-agnostic setup is a very important part of the open-source setup of lemmy. it's fine to build on top of that, but currently anyone with docker-compose installed can run the service and that really should be retained.
- Staying cloud agnostic is very important and CDN services like cloudflare/cloudfront have inherrent privacy issues. IMO the stack should remain hostable on anyones home server environment.

Please keep working on it, thank you for your effort.

good news, a fix might be in the works: https://github.com/LemmyNet/lemmy/pull/3482
- Yes phiresky is working with us improving performance, has helped a lot so far! (We're now running a custom build with some of his improvements)
  
  Just tried upvoting most of the comments here, and this time all of the upvotes went through and were extremely fast! No more 502 errors!
  
  I've noticed lemmy.world has been much more responsive today so something seems to be working!
  
  It’s working, it’s working!
- Thanks for sharing. It was very interesting to see the graphs showing the before and after.

Thank you for your effort!
- You're welcome! (testing comments now... )
  
  It looks like it's working.

I am very forgiving of the bugs I encounter on Lemmy instances because Lemmy is still growing and it's essentially still in beta. I am totally unforgiving of Reddit crashing virtually every day after almost two decades.
- Still a better product than Reddit Mobile lol

System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Shouldn't we be discussing closing registrations?
- There's a lot of momentum to move away from reddit right now, and closing registrations would be a wet blanket. Personally, I'll take the performance issues and transparency in the process over closing registrations.
  
  This. Don't stop the train. People need to be able to come over freely.
  
  Does Lemmy have the ability to replace default links?
  Basically, replace signup link with one that redirects to a page that gives a very simple as possible explanation what's going on, what fediverse is and gives s list of other instances to try.
  Reinforce "All are viable and can browse lemmy.world subs"... Or communities or whatever term we use here for lemmy equivalent of subreddits.
- Federation-wise it would be better if new users spread out. Between clueless redditors and impossible ideal, I prefer if they at least made an account and check out what Lemmy has to offer. The curious ones will eventually settle down and even redistribute into smaller instances.
  
  The curious ones will eventually settle down and even redistribute into smaller instances
  Absolutely. I migrated from lemmy.ml when that was having too many sign-ups, and I'm not opposed to migrating from lemmy.world to help with their load. I'm sure I'm one of many

The need to restart server every so often to avoid excessive ram usage bit is very interesting to me. This sounds like some issue with memory management. Not necessarily a leak, but maybe something like server keeping unnecessary references so the object cannot be dropped.
Anyway, from my experience Rust developers love debugging such kind of problems. Are Lemmy Devs aware of this issue? And do you publish server usage logs somewhere to look deeper into that?
- server keeping unnecessary references so the object cannot be dropped
  You indeed just described a memory leak :D
- Oh they're well aware, there's a few tickets open.
  For example there was some stuff about rate limiting using a lot of memory: https://github.com/LemmyNet/lemmy/pull/3111

How can I donate to lemmy.world?
- Just look at the sidebar:
  Donations
  If you would like to make a donation to support the cost of running this platform, please do so at the mastodon. world donation URLs:
  https://opencollective.com/mastodonworld
  https://patreon.com/mastodonworld
- Not an expert, but here is where I set up a recurring donation. Apparently Mastadon.world and Lemmy.world are run by the same admins. Took me a while to understand this, please correct me if I'm wrong.
- By moving to another instance

People of lemmy.world should encourage the new users joining up on lemmy.world to other create accounts on other instances instead

To all the folks that are worried, don't be. Let me tell you, Mastodon was a wreck when Musk took over Twitter and that all got sorted out within a week or so. The mods and sysadmin are obviously working hard to get things up and running, but growing pains are growing pains. To paraphrase an old adage: "Facebook wasn't built in a day." In the beginning, Zuck and Co literally limited signups to only people with college email accounts and only added universities a few domains at a time... scaling is very difficult, but it's not impossible. The way things are going, Lemmy is going to thrive!

Would donations help in the long run? How can we donate?
- If you would like to make a donation to support the cost of running this platform, please do so at the mastodon. world donation URLs:
  https://opencollective.com/mastodonworld https://patreon.com/mastodonworld
  
  Thank you, done!
- this is interesting indeed! could we help via donations?
- Donations definitely help. They are a real motivator and allow people to make more time for contributing. For instances it is also a way to cover hosting costs.
  You can donate directly to the Lemmy development or you can see if your instance accepts donations. Personally I just signed up to the Lemmy Patreon.
- Donations invested into REITs would fund Lemmy indefinitely, has anyone pondered this idea?

Thank you for everything! Can we donate to cover the costs? If more people throw in $5, we all will benefit. Can Lemmy server scale up without needing RAM exponentially growing with the user number? I hope the system will get better optimized for bigger user base as the time goes on..
- There are two donation options listed in the sidebar, idk if one is better than the other from their pov
  https://opencollective.com/mastodonworld
  https://patreon.com/mastodonworld

Thanks for giving us all these updates all the time!

Thanks team, love your work! Appreciate the updates and improvements but happy to have this place despite the bugs and quirks :)

I'm very much so okay putting up with hiccups right now. I know things will improve in the future.
Thank you and everyone for all your hard work!

It has become way smoother the last few hours. The work you guys from .world and Lemmy's devs have been doing is amazing!

Thanks again for all the hard work on Lemmy World. It feels fast today

I love the community here and am excited to see it grow. Thanks for giving a report, it awesome how invested y'all are and makes me want to stick with this instance even through the hardships. Keep up the good stuff!

I feel that lemmy runs smoothly the past few hours with very few hiccups and mostly on the upvoting and commenting side. I encountered no issues yet on the loading of the post.
Thank you so much for all your hard work.
- Agreed, it seems fine now, makes such a difference to whether I want to keep interacting with Lemmy.

The good thing about the fediverse is we can sign up on federated servers and still interact with everything on lemmy.world without the errors, then come back to our lemmy.world accounts when everything is smoother
- Yeah, when I understood this I joined another small local instance, it does improve performance by a lot!

I can't imagine how hard it is to rua server that has influx of large users. I Thank you for your hard work to run and maintain this instance. Hope it works out well and future will be a smooth sail.

Its amazing this site hasn't completely melted down yes it sucks trying to use it and having nothing load but the fact that the site has mostly worked when I want to use it and the fact you guys have less than 1% of 1% of reddits resources is mind blowing

I am sure the stress is unreal, but It is appreciated. You all stepped up when everyone was stepping down (or trying to sellout). I am hopeful this will turn into something great.

Can anyone give me a link to donate to help out? Ever since discovering lemmy I hardly use reddit and would like to contribute to this community more!
- Yep! Donating to both their Open Collective page for Mastodon.World (didn't grab that link by accident), as well as their Patreon page for Mastodon.World works!
  They have a note up on Patreon that donations there also contribute to World.Lemmy, as well as another Lemmy instance!
  Here's the link to their Patreon; https://patreon.com/mastodonworld

Thanks u/ruud!
How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.
- Same happened with mastodon.world in November. Family goes first, then work, and then all of my hobbies, of which this is one. (But the one taking up most time at the moment...)
  
  You’re a legend! Thanks so much Ruud!
  Glad to hear that your priorities are in the right order. Family comes before helping random strangers posts pictures of beans on the internet.
- Small remark instead of /u/.... you should use the at-sign @ to mention somebody. For example: Hey @Ghostalmedia@lemmy.world.
  
  How do you know which community the user is from?

Thank you for all of your effort on this. Adjusting to rapid growth is a huge challenge, and we all appreciate the effort you and everyone else is putting in to trying to make this instance run smoothly.

Thanks for the update, you're leading the way for Lemmy.
I was prepared for servers crashing and lagging, but you're keeping things moving thus far, despite receiving 85k new users in a month. Heroic stuff.

Thank you for your hard work! ♥️

Thanks for the transparent update!

The redirects to baseball games glitch is gone. I am unsure if this is because the glitch is fixed or if baseball season is over.
- 😆 sorry about that! Baseball season is still very much on.
  This issue was caused by the websockets replacing the current thread you were on with newly updated or created posts. Our game threads get updated frequently so they were constantly popping up in people's faces. We decreased the frequency of the updates pretty substantially in order to be less annoying to unrelated instances but it was still happening occasionally.
  Now that we're all through with websockets it won't be an issue anymore!

Thank you for the transparency @ruud@lemmy.world It is rare for platforms to scale this quickly so issues like this are inevitable. Good luck for the troubleshooting!