Remote View

sh.itjust.works Main Community @sh.itjust.works

2y ago

In preparation for the Reddit June 12-14 Black-out

As the reddit mods gets ready for the June 12-14 black-out, there some anticipation that an influx in user base will shift over to many of the lemmy instances as user seek out a home to post their internet memes and discuss their interests.

In anticipation of this increased volume I will be growing our current instance from

16 CPU
8 GB ram

24 CPU
64 GB ram

This server is currently equipped with SSDs that are configured in a raid 10 array (NVMEs will come in the next gen that get deployed)

Earlier today I also configured some monitoring that I'll be watching closely in order to have a better understanding on how the lemmy platform does under stress (for science!)

I'll be sharing graphs and some other insights in this thread for everyone that is interested. Feel free to ask anything you might be interested in knowing more of!

EDIT: I'll be posting and updating the graphs in this main post periodically! Last updated: 6:21AM ET June 12th

CPU - 48 hours

Memory - 48 hours

Network - 48 hours

Load Average - 48 hours

System Disk I/O - 48 hours

72 comments

For the icon: How about a Satisfied Seal? Because when sh.itjust.works, you know that feel!
- I love this!
  
  Let me know if you want me to tweak the design any and I can send an SVG as well. No copyright/usage restrictions - free to use or modify as you wish.
  
  Here’s a link to the SVG of the Satisfied Seal so you can tweak as needed. I’ll also post the link in the comment thread. Glad people seem to like it!
  https://svgshare.com/s/u8T

Impressive setup! Would you be interested to have your instance recommended on the Lemmy project website?
https://lemmy.ml/post/1171717
- That would be great! I checked your link and will work on completing the steps shortly :)
  
  Just joined you from that featured site, it just works. People are complaining about how hard it is to switch, I genuinely don’t know what the difficulty is
  
  let's goo! Great instance that deserves some growth! 🚀
- Joined from recommended a bit less than an hour ago. Tried lemmy.ml, got rejected, went for this one cuz it was top recommended and I figured with the name and the user count the likelihood of rejection was smaller.

I am from reddit migration. it's 100000000000 times better, e.g. more accessible with my screenreader, on reddit, I needed to use special app, called dystopia which handled voiceover more gracefully.
- damn that's good to hear

A little update for all of you interested. I allocated the additional resources to the VM and will post some updated graphs once they update with the new configurations.

For those who are like me and like looking at graphs here are some prior to the upgrade.
CPU - 48 hours

Memory - 48 hours

Network - 48 hours
- so if I'm reading this right, less than 10% CPU capacity and about 1/8th ram at peak times, before upgrades? gotta give you credit where it's due, that thing looks ready to take some abuse.
  
  The only part I don't have graphed yet is the disk IOs. I'm going to need to invest a little more time to get that metric going captured.
  Say you through 1000s of active users to this instance.. what would cause the bottle neck first? CPU, memory, network, disk? I'm thinking probably disk due to database optimization that need to be reworked on lemmy afterwards CPU and then memory.
  
  The server doesn't have any users atm though
- Hello Lemmy! I've just made an account to get my foot in the door incase the Reddit execs don't roll back after the blackout, let's sit back and watch the fireworks.

oooh pretty colors!

So the r/lemmymigration mod got his account unbanned from reddit, and it seems he might be willing to put together a currated and updated list of lemmy instances that are willing to host new users. Just in case that is something you would be interested in participating in, here is his comment on reddit: https://www.reddit.com/r/RedditAlternatives/comments/14316ai/the_guy_that_got_permanently_suspended_for/jn7k56l/
- Sure lets see how far we can get this server going!

Thanks for the transparency

shit just works

Is this an actual bare metal machine?
- No its on a vm.
  
  Which cloud?

Just discovered this place. Nice to see you guys are getting ready for the influx of users.

I’d love to see a few additional charts with the next update:
Disk space consumed
Subscribers
Communities federated
My guess is that’s not available vie SNMP for Librenms. :)
- I'm being a little private about the total amount of disk assigned to this instance for now. I do plan on sharing these details when I have implemented a viable solution. That being said I can tell you that with the amount of current users and activities this instance is growing by about 20GB per day in disk size.
  
  Holy shit... You tell us when you'll accept donations, right? This doesn't have to be your financial responsibility.
  Or do you and I just haven't found it yet?

Thanks @TheDude. I work in enterprise network/systems/cloud operations as a network/security engineer. Would love to contribute monetarily or with time.

Can this thing run on a Pi clone with 4GB RAM and 4 cores?
- It can definitely run on a Pi instance. The storage medium is going to be important as it will need to house a PostgreSQL database. Would be fine for a few users but not sure how many it would be able to handle concurrently.
  
  It's got 4 cores at 2.16GHz (I think 🤔), 4GB of RAM. Regarding the storage, that can be arranged. Currently there's a 32GB SD card in it, it can be swapped for a 512GB one, no prolem.
  So, how many users can an instance on a Pi like that serve?
- I was looking at that earlier too. I found LemmyNet on GitHub but haven't had a chance to test it. But "can it run" is also a fundamentally different question than "can it run smoothly for end users as it gets hit with a huge influx of users," and given the larger context of a potential Reddit exodus, that scalability concern is probably not negligible.
  
  My point was, those that can, make one, fill it till it can serve users, then just disable registrations, so it doesn't overload.

You may want to check this out: https://sh.itjust.works/post/2654
They are working on fixes, but they will certainly not be ready by the 12th

Any recommendations on VPS hosts. Thinking about creating my own.
- There are many to choose from. Where are you located?
  
  USA, IN
- Check out Vultr. Super customizable. Have not gone down since I got mine (currently has 900 days uptime).
- I've never hosted anything big but cloud fanatic is cheaper than digital ocean. I migrated from the latter to the former earlier this year to save. And then someone told me AWS is basically free up to a certain usage point. $200 free credits/mo? I can't remember what the "free" part implied. As always, do your own research, but you could join one and switch servers later on I guess.
  
  +1 Cloud Fanatic. They were Server Cheap until a few months ago, but I guess that name turned people off. I don't have any heavy usage so I cant really speak to that aspect, but I've never been over billed or unable to connect in 3 years.
- I've recently migrated a forum community (vbulletin -> discourse) using a vpn by Contabo (germany hosting). Their price are incredible and while you don't get an official SLA, it's around 99.5%.

I’m starting to get random 500’s, I guess that’s a sign I need to go to bed and let the server breathe lol. Can’t wait for the updated graphs!
- Looks like it was related to a setting that needed to be tuned. I just updated the graphs. Server is not on fire yet!
  
  Awesome, great to hear it!

Yeah I'm trying Lemmy anticipating that Relay Pro may stop working soon. Searched for my sub reddit subjects, seemed to work though some of them seem a bit empty so far.

wow, that's much more hardware than I thought.
- It probably doesn't need that much but I'm curious in seeing how much It can handle until it can't I have more resources available but it would make more sense after this size to start separating the different components of Lemmy into their own individual instances across multiple servers instead of trying to scale it vertically. Might require a little more time than I have but I'll see what I can manage!
  
  huh, interesting.
  How much room is there right now for lemmy to scale horizontally?
  
  I'm interested to see how this develops.
  I don't have a lot of experience administrating rust apps, but usually single instances hit bottlenecks when trying to utilize resources on a large enough server.
  Like others have said, I'm happy to help if there's anything you need to keep this going. I have many years of Linux and Windows admin experience, observability/instrumentation, and I can code to an extent.

The monitoring is with Prometheus / Grafana? Can you share your setup?
- Prometheus /grafana would be great but I'm using an existing librenms setup via snmp probing

72 comments