1y ago

lemm.ee just got faster! (infra upgrades)

Hey lemmings!

I wanted to share a quick update about our recent performance issues and how I have addressed them.

The last 24h have been a bit rough for lemm.ee.

Last night, I spent some time debugging federation issues with lemmy.world. We managed to significantly improve the situation - lemmy.world content is now reaching lemm.ee with a very high success rate - but this has had the effect of increasing incoming federation traffic on our servers significantly.

Additionally, we have been seeing steadily increasing normal user traffic over the past week, which is awesome from a community standpoint, but of course means that our servers have to do more work to keep up with all the new people.

To top things off, today there appeared a badly configured instance in the network, which was effectively launching a DoS attack against lemm.ee for several hours. Most likely it was unintentional, but unfortunately the end result was a sudden increase in our server load.

All these factors combined resulted in a really bad experience for most lemm.ee users today. Page load times have consistently been spiking into as much as 10 seconds or more for the whole day:

In fact, a lot of page loads just timed out with errors.

Fortunately, it seems I have managed to clear up the problems!

I have put a bunch of mitigations in place, and after monitoring the situation for the past hour, it seems that our performance issues have been resolved for now. So hopefully, you can enjoy browsing lemm.ee again without it feeling like torture!

Here are specific steps I took:

I have doubled the hardware resources for our backend servers and database.
I purchased a Cloudflare pro subscription for lemm.ee for 1 year. This took out a considerable chunk of my budget for lemm.ee, but in return it will allow me to analyze and optimize our cache usage to a far greater extent. I am already seeing vastly reduced load times for cacheable content (try opening https://lemm.ee a few times in a row as a logged out user - it should be blazing fast now!)
I have configured a rate limiter which will prevent future DoS from the specific method that was used against us today.

Of course, all of the above is costly. Luckily, lemm.ee users have been very generous with donations in the month of June, and in fact a significant amount of donors have opted for monthly recurring contributions. This all gives me the confidence to increase our spending for now, and I am currently expecting to NOT increase my personal planned contribution of 150€/month, as the increased costs so far are entirely being covered by donations!

Let me take this opportunity to thank the sponsors who made the upgrades possible! All lemm.ee users are now enjoying better performance thanks to you, I could not have done it without you awesome people.

On a final note, I just want to say that I hope a lot of these issues can be solved by optimizations in Lemmy software itself in the future. I have been personally contributing several optimizations to the Lemmy codebase, and I know many others are focused on optimizations as well. Just throwing extra resources at the problem will probably not be a sustainable solution for very long 😅. But I am optimistic that we are moving in the right direction with the software changes, and we'll be enjoying reduced resource needs before long.

That's all I wanted to share today, I wish you all a great weekend!

114 comments

What a phenomenal response. Every single day makes me happy that I chose lemme.ee as my home instance. Truly appreciate all of the hard work you are pouring into this. And we can see how difficult it is based off of your total transparency. Thank you!

I just fucking love the transparency of the admins of lemmy.world and lemm.ee. Cheers guys!
- You're welcome!

I'm brand new, this is my first comment. Thanks for your work! Where can we donate to this instance?
- I just signed up for a monthly donation to keep things running using their GitHub sponsorship page: http://github.com/sponsors/sunaurus
  Love this community!
  
  Thank you, signed up too!

Been browsing for a few days, decided to set up shop here, so to speak, upon seeing this.
Great work with this.
- Agreed! I was hopping around trying to figure out how to choose an instance (and only somewhat understanding how the site works so far lol), but these super transparent posts and the effort clearly being put in to keep things running smoothly + be welcoming to reddit "refugees" is what made it easy to pick this instance to sign up on first! Thank you for all the hard work!! :)

Yo, I'm a simple Reddit refugee, just trying to figure out how to make my way in the fediverse, and I signed up to lemm.ee not long after this post went up. I honestly chose this instance on a whim, and after a bit of exploring and learning about how Lemmy works, boy does this post make me glad I landed here!
Thanks for running this instance, and for housing us reddit noobs :)

I hope years from now you get to look back on these times as the beginning of something great not only for you, but also for the future of social media on the internet. Your dedication to this project has been admirable, and you are absolutely crushing it.

On a side note, really liking this 0.18.1 release candidate version, the 0.18.1 official release is going to be brilliant. The new compact view is beautiful and it scales with window width nicely. They just need to drop the post header size a bit and the compact view will be perfect. This release candidate seems to be pretty solid, only seeing fixes, no new bugs.

I have configured a rate limiter which will prevent future DoS from the specific method that was used against us today.
For those of us running other Lemmy instances, what should we be looking for?
- While a generally a good idea, this information is better shared privately

Cool! This is my first comment on Lemmy, and I'm excited to see what's in store for the future here!

Thanks for taking care, making upgrades, monitoring, fixing, contributing, and informing us so thoroughly.
Also thanks to all the donors! This is all great to hear 😊

I jumped from another instance, this one loads so much faster and has more accurate numbers on communities from other instances. Really cool stuff.
- I just joined up too after having lots of issues on lemmy.world, which is not surprising with how many people are flooding over. Someone commented about lemm.ee and I'm also really liking it. The performance difference is huge! I also love how the owner/dev talks about the instance. Seems like this will be a good one to stick with and use as primary.
  I am curious about membership counts on communities though. On other instances I'll see a community member count of around 1000 and on here, it shows the (same community) is nowhere near that high. Are you saying that the lower number is more accurate? Or maybe it's a syncing issue and with all the signups happening the numbers just haven't had a chance to get caught up?
  
  If you're viewing a community through an instance that isn't the one it's actually hosted on, you'll see a lower subscriber count because as far as I'm aware it's showing the number of subscribers from the instance you're viewing it on rather than the total.
  
  The instance its hosted on is probably the most accurate. The reason I said this one was more accurate is because the old one I used had many posts without any comments and some posts were missing completely while this was showing most if not all of them. It will probably never be 100% synced due how lemmy works I guess.
- Yeah @sunaurus@lemm.ee is really optimizing the hell out of this instance. A driver for me right now is he's running the release candidate for 0.18.1 and it's a huge improvement for me over 0.18.0. He's also a dev on the project so he's getting fixes in as well.
- Same here! So far this instance has ran really smooth for me.

I have to admit I don't really understand what you do, but I see some of your activities and I'm left feeling very impressed.
I'm not sure exactly what's necessary to make this site and these federations actually turn into a fully fledged alternative and perhaps even a successor to "the old country", and what mistakes to avoid becoming what made it turn worse. (I probably shouldn't even involve myself in figuring that out.)
I am however sure that I am seeing you do a lot to make sure lemm.ee even has a chance. There is a lot of stuff blooming on here.
- As a software engineer, this post convinced me to sign up for lemm.ee. So far it's the most stable instance I've tried. I've seen a lot of questionable technical choices and messaging from other larger instances, but it seems like we're in good hands here

You are awesome man. I wanted to wait until the instance matured before committing to a monthly donation but I am signing up now. You're the best instance admin anyone can hope for. Glad to see your patch make it to 0.18

I'm so grateful for your knowledge and persistence. My donation finally went through after fighting with my bank. Maybe I don't have to give up my Gen X card afterall.

Nice work!

I migrated a few hours ago when Apollo went down and have had a great experience so far! Nice work.

Wanted to share something from my experience running a pleroma instance: I was having an issue where postgresql was becoming more and more of my CPU utilization. It looked like I was going to have to buy a seriously upgraded server, my loads like 3-4 constantly.
I ran pg_repack during a lower traffic hour (site continued to run during the run but at reduced performance) and my loads were down by 90%, to much less than 1. Now I have it set to do a repack weekly (ymmv, it just seemed like a good frequency to me)
Haven't done it to my Lemmy server yet, but that's because of all my instances this one is the newest.

Alrighty, I had a browse around and I've settled on lemm.ee + Jerboa and it's looking good. Thanks for all your hard work handling the influx of reddfugees like myself. This gives me hope ❤️

I really appreciate how communicative you are about all the instance information. It’s what made me sign up just now!

Thank you @sunaurus@lemm.ee for all of your work. I am continually realizing that I chose the right instance with how transparent and responsive you have been.

You broke the https://lemm.ee homepage, it returns a json.
- It does load that JSON very quickly!
  
  Ah, yes, the optimist
  
  And really, who doesn't love json?
  I mean protocol buffers might be more efficient, but json is nice and readable. Much nicer than XML for an example. And significantly more readable than protobuf!
- It should be fixed now!
  
  It is, only that I had to clear the cache :)

Yay lemm.ee is back <3

Been lurking off and on since the start of the evil reddit shenanigans. Finally set up on here using Connect. This is legit! Great job
- Connect is great, nicest app I've used for Lemmy so far. I do wish it had a way to swap instances quicker without an account, as I get the need for posters, but I don't like posting.
- Same here. Lurker for weeks, subscriber for hours
- omg Dean Fogg! 🧙‍♂️ Welcome to Lemmy

Thank you. I did notice when it got significantly faster.

Can you tell me what you've cached? I'm not using cloudflare but I am using haproxy which has frontend caching builtin. It was next on my plan but if you share your caching setup I can try to replicate it on lemmy.dbzer0.com
- Mostly all images are served through a cache. I would like to also cache some static HTML (such as pages for unauthenticated users), but it breaks due to some users requesting these pages with an Accept header for an activitystream content-type, and I haven't had time to figure out a solution for accounting for the content type in my cache key unfortunately 😅. But if you can do that easily in your cache then for sure you could also cache any static pages for a minute or so.
  
  Ye I can cache differently depending on headers. Surprised that caching images helps a lot since your pictrs is hosted in an independent box anyway

Yeah the slowdown was a bit rough, been browsing off and on all day today. Thanks for fixing that. Seems to be working a lot better now. That's a bummer you had to increase expenses though.

Not sure if this is related to the infra upgrade, but my earlier issue with not seeing all the posts in the meta community is now fixed.
- Awesome news! I did make a small fix to a localization bug in Lemmy-ui, which was causing some people to not see posts, so it could have been that. But in any case, I'm glad it's sorted for you!

This is so awesome! Thank you for everything you've done. You continue to prove my belief that this is the best instance to be on.

It's great to see the good work you instance admins are doing on building the infrastructure. I may have settled on a different instance, but lemm.ee seems to have a great community and I love the openness here.

Thank you very much!!

It's feeling quite good at the moment so thank you for all your hard work.

Thank you for your work!

How can I donate? I want to help!
- https://github.com/sponsors/sunaurus

Thank you for operating this instance!
I hope you are tracking some dollar amount for your time as well. Even if you’re not actually taking any pay it would be good to have a sense of labor as a component of a thing’s upkeep cost.
- Thanks for the kind words! To be honest, I'm not tracking my time, as I'm not expecting to break even at all anyway, even before starting to account for the time I've spent 😄

Awesome instance. Really fast speed (compared to sh.itjust.works), there is no blocked instances/communities. Kudos❤️

Thanks!
I'm definitely seeing an increase in speed. I think at some point there will be a market for servers that users must pay to use. For the time being, I'm more than happy to contribute to a server that is open to all, but good speed needs to be there.

Thank you very much. I've been made to feel very welcome and have joined on of the many support groups, that his helping me remove typing r/ from my muscle memory. Have a great weekend.

Definitely appreciate the improved speed, but the persistent federation issues has left me in a permanent FOMO state.
- I know what you mean! The good news is that there are some huge improvements for federation in 0.18.1. These improvements depend on instances at both ends being on 0.18.1, so we'll start seeing it kick in shortly as more of the network upgrades.
  
  Is there a way to manually trigger a sync or so?
  Seeing other reply to you on another instance, and not being able to respond because those replies aren’t on lemm.ee is very frustrating.

I joined this instance after reading this post. My inclination is to operate my own instance; but it seems best to wait until some of the dust settles and some of the bugs get discovered and fixed, first. The admins here seem capable of doing just that, while providing a stable platform.

Once again, thank you for making this small pleasant corner of the Internet possible!

How much of the slowdown was caused by the bad instance VS the limitations of the previous hardware?
- The DoS was responsible for about 10-20% increased load on our system - it wasn't the root cause of the slowdowns, it was more like a nice cherry on top of the cake 😅 The bigger issue is the constantly increasing federation load.

I've just joined up to lemme.ee and subscribed to a community on another instance. I can see the posts in that community, but they all show up with zero comments. If I view the community on the web, there are several comments there. Is this a bug?
The community is !nrl@aussie.zone
- It is a federation thing. Newly federated communities don't load comments with their initial import. You'll start to see comments as you go.
- Just to add to the other reply, if there's an old comment chain you really want to reply to you can grab the URL of the last comment, search for it through your home instance and that'll force an index of the comment you searched for plus any parents.

You're doing a great job! Keep up the good work!

To this day I have still lots of "subscribe pending" in my communities options page, especially from lemmy.ml and lemmy.world
Should I try to cancel them and redo or just wait?
https://imgur.com/a/rJEH1Di
(I cannot upload images anymore, I get a JSON error now)
- You should be able to cancel and retry lemmy.world - there's a high chance those will go through now.
  With lemmy.ml, there's a much lower chance, better to wait until they upgrade to 0.18.1.

Love the instance and all your transparency! Keep up the good work. Heading over to the donation link now! I'm so stoked to be off of reddit for good.

I don’t mean to grossly oversimplify… But does this mean it’s time to upgrade to 0.18.1?
- Generally I would recommend against running RC builds. I was willing to take the risk in this case for lemm.ee, because:
  I feel comfortable with debugging and fixing issues in code as they crop up
  I have been working myself on stabilizing 0.18.1 and am pretty well aware of its current state
  The set of problems in 0.18.1-rc4 is a bit better than the different set of problems we had with 0.17.4 😅
  At the end of the day, you have to acknowledge the risks and see if they're worth it for you.

I would like to see a create a sub for dummies video or post. I followed the link to create a sub, and I’m a dummy. I don’t get it. Once subs can be created easily, this community will grow much faster now that Apollo is gone. RIP
EAT my chode spez
- I've not created any communities myself, but looking at the form it seems pretty straightforward. Is there a particular field that could be better explained, or are you getting some sort of error?

Beehaw upgraded too. Lemmy.ml seems to be delivering none of the messages for past few hours, it is erroring constantly for a local user.

thank you king

Amazing job guys!

You are a freaking hero!

114 comments