1y ago

Some system load graphs of last 24h

For those who find it interesting, enjoy!

199 comments

I really enjoy your transparency and style of communication!
- Comparing it to Spez and how Reddit became prior to the migration, this is such a refreshing change
  
  /u/Ruud is like /u/Spez but only if /u/Spez was actually cool.

It gets me every time seeing people using the product I build 🥹
- You worked on Grafana? Your product is awesome, I use it in my homelab for performance metrics
  
  Yes, I'm one of the designers 👍🏾
- Grafana is the most essential application in my job. I can use Notepad to code in a world without IDEs. I couldn't keep a damn thing running in the real world without Grafana. And I've been forced against my will to use alternatives in the past.
  
  How did you learn it?
- You work at Grafana?

It's been very snappy today, nice work! Is it all under Docker Compose with the node handling Nginx and Postgres as well?
- Yes.
  
  Why did you guys roll back the UI to .7 from .10? I enjoyed some of the UI improvements, but I guess there were some bugs?
  Edit: I see its back to .10 maybe I had a browser tab open from before that I never refreshed
  
  I‘m really grateful for your and your colleagues‘ work. Thank you for letting us lemmy around here!!!

Dang that's a lot of RAM
- mastodon.world has the same server but with twice the RAM :-)
  
  What chassis? I’ve got 256GB in an R720 but only 32 cores here!
- You should see some of our VM hosts at work...

I can’t believe how fast you’ve managed to crowdsource and fix things on this instance. I haven’t seen many problems at all sharing comments and things.

How can I throw some bucks in your direction?
- From the lemmy.world front page:
  undefined
  
  Donations If you would like to make a donation to support the cost of running this platform, please do so at the mastodon.world donation URLs: https://opencollective.com/mastodonworld https://patreon.com/mastodonworld
  
  Where in the frontpage can we see this?
  Edit: thank you all!

This is awesome! As a systems engineer for my day job, I love seeing stuff like this!

Some of my usage is in this data and I like that.

pretty gauges. the instance seems to be more stable/responsive today

Damn that’s a huge chunk of (what looks like) a 64 core CPU there. Impressive!
It’s cool it can aggressively cache that much. Although I am perplexed why one would have a swap file configured in this case? What does it give you here? Sorry not trying to be an elitist or anything just have no idea what advantage you get!
- To be honest I tend to use swap less and less. But this was in the build that Hetzner does and I didn't remove it.
- If your application goes wild with RAM usage, a properly configured swap will make sure the underlying OS remains responsive enough to deal with it.
  
  The OOM killer is usually triggered after it starts hitting the disk. Which means your system is unresponsive for a long time until it finally kills something.
  Using something like oomd can help trigger before it hits swap but then why are you using swap in the first place?
  The bigger issue is that the kernel sometimes ignores the swappiness and will evict code/data pages long before file cache even when set to 0 or 1. I'm still not sure if that was because of an Ubuntu patch or if it was an issue that's been resolved in the years since I last saw this

How much is that in beans?
- At least 1
  
  Possibly 2
- About tree fiddy

How far do you see lemmy.world capable of scaling to? One thing I've been noticing is the centralisation of Lemmy users on a few top servers, surely that cannot be healthy for federation? What are your thoughts on this?
- @ruud@lemmy.world They should post their clustered setup so others can replicate more easily. It sounded like they had several webservers in front of a database (hopefully a cache box too).
  
  Not entirely sure of what you are asking, but the only reason they need a clustered setup is simply because of their scale. Making the details of their setup public does not help with the issue I addressed, since in an ideal scenario, communities and users would be evenly distributed amongst the many Lemmy instances in the fediverse, making the need to do any sort of clustering for performance reasons unnecessary.

How much is this costing you? Also who is your host? Is it on a virtual machine?
- They have a dedicated server: https://lemmy.world/post/75556
  
  Whoa, cool. Thanks. Only a matter of time until it gets overloaded though. Can't Lemmy run in a container service like Cloud Run or AWS App Runner?
  
  It's actually pretty funny to see him mention the growth (almost 12k users!) considering they've added, what, 50k or so users recently?
  
  Dedicated means local?
  
  My homies love dedicated servers

I know that the RAM cache is just taking advantage of otherwise free RAM and will be dropped in favor of anything else, but it does stress me out a bit to see it "full" like that.
- It would stress me even more to see a lot of RAM doing nothing, that would be a shame! ;-)
  
  Difference between Windows and Linux. Windows would only use what it needs. Linux pre-empts more and fills the RAM for what coul dbe needed.
  It used to stress the shit out of me when I switched to Linux as I'd gotten used to opening task manager and seeing 90% free RAM. On Linux I'd be seeing 10% free and panicking thinking it was a resource hog.
  The Linux-way is the best way.
  I use Arch btw ;)
- That's how it supposed to work, free RAM does nothing :)
- It’s free real estate!
  If you had this much buffer memory what are the reasons to have swap space as well?
  With my servers I’m paranoid having swap enabled will inadvertently slow stuff down. Perhaps there’s a reason to have it that I’m unaware of?
  
  If you had this much buffer memory what are the reasons to have swap space as well?
  Many programs do stuff once during startup that they never do again, sometimes creating redundant data objects that will never get accessed in the configuration its being run in. Eventually the kernel memory manager figures out that some pages are never used but it can't just delete them. If swap is enabled it can swap them to disk instead. It frees up that RAM for something more important. It's usually minor but every few MB helps.
  
  I personally like having some swap as during low memory situations (which lemmy gets at least once a day on my small instance) everything slows down rather than getting culled by the oom killer. It's not a replacement for monitoring, but it does extend the timeframe to react to things.
- Memcache usually takes all the assigned memory regardless of usage so seeing high usage isn't always unusual. That's assuming the lemmy servers are using some kind of session caching solution.

I hate that radial graphs are so popular with *Grafana dashboards. Radial/pie charts are terrible representations for humans to interpret. I tend to try and convert them either to a stat with the line/time display or a bar chart. Humans are better judging linear relationships than radial.
- Who says I'm human?
  
  Or are you dancer?
- Radial graphs are a bit of a meme where I work as one of the C-suite managers despises them for precisely that reason.

Now that’s hot
- As a server admin, I really hope it's not hot
  
  2hot

Awesome. Gotta love Grafana!

Looks Awesome! Glad to see the patches seem to be working.

This is so cool to see. Thanks for posting! Lemmy.world has been super smooth today

Love me some grafana.

I was hoping to see some uptime, but thanks for the window into your server! Are you still having to kill the instance every half hour?
- It says uptime is 3.3 weeks in the top right.
  
  Hmmm... maybe the instance uptime is different from the server uptime.

Everytime I open a post and go back to previous page it scrolls back to top. Is this fixable? Im on windows 11, chrome.

Great stats. Thanks for posting!

Always fun to see system dashboards.

Quite a beefy setup 😄

Thanks for all the hard work. It has been running so well all day!

I notice your defederation list is completely depopulated today. Is that intentional?
- No it's just moved to the bottom of the page apparently. I preferred it on the side. Maybe a tab would be better.
- On 0.18.1-rc.10 the defederated instances are at the very bottom, not on the right hand side.
  
  OOoooooh! Thanks for the info.
  Also infinitely less clear and helpful...

Is the memory leak still there?
- No! Restarts are disabled and it's OK now!
  
  Great! 😁 I was just wondering because the memory graph showed sharp falls in memory usage every ~30 mins.

THE DROP???!! >:O
- Seriously! Talk about amazing optimization and debugging of the network service.

So I‘m currently on planning to host an instance myself. This graph helped me quite a lot to get an idea what system resources are required.
Do you use any reverse proxy in front of it?
- Nginx runs on the server , proxying to the lemmy docker containers
  
  That‘s what I had in mind. To run nginx on a seperate vps, so I can scale it easier. Run fediverse instances in the back, either all on one vps or on different vps. This way I could provide a hub while increase performance (due to compression and caching) and provide redundancy/load balancing if necessary.
  What‘s the typical traffic you experience? Peak (Gbit/s) and average/daily traffic (GB)
- Lemmy world has a lot of users. So your instance initially will require a lot less resources ✌️
  
  Yeah I saw that. I‘m a big fan of minimalistic, yet super performant architectures and I‘m just trying to get a feeling on how I could solve this problem. I try to avoid any downtime, whenever possible

I fucking love a sexy Grafana dashboard.

Can someone give me a hand. I see tons of posts of people talking about a picture in the OP but i see nothing. Am i doing something wrong? Is my connection bad? This seems to be happening quite a lot. For example the meme instance has almost zero pictures but i know just about every post should have one.
- hmm yeah it was gone.. need to investigate..

The entire team is doing an amazing job. Lemmy is getting smoother with each passing day. I hope it keeps growing (and none of you get too burnt out in the process)!

Ahh look at all those nice charts and diagrams, that's true server porn lol.
Again thank you very much for your awesome job. We all really appreciate that <3

This is indeed interesting, thanks again for the service!

🤤

Such pretty gobbledyremoved!

I think you can export the dashboard the way it looks to you - into Grafana cloud. Like a snapshot. Click "Share" then "upload" and share the link.
We won't be able to see historical data as it takes only dashboard snapshot with visible data.
Would be cool, isn't it?
- Used some provisioning templates to get started 😁

Sexy loads.
- I imagine sexy sax man playing in the background while watching these graphs.

Nice! That's a nice-looking dashboard, would you mind sharing its JSON config? Thanks!
- it's the popular one on grafana.com - https://grafana.com/grafana/dashboards/1860-node-exporter-full/
  
  Thanks!
- I could share the template, if ya like.
  
  Thanks for the offering, but no worries, some user posted it and I found it already

I have a love and hate relationship with Grafana but it probably feels the same

Does Lemmy have a memory leak?
- Lemmory meak?
  Yes at least until yesterday's version...
  
  Lemmory meak?
  heh
- From those graphs, memory usage is very low. Most of it is being used for disk caching, which is what linux does with memory it has no other use for (may as well use it for something).
  
  Yes, but we still restart the containers every 30 min. I'm gonna see if that's still needed after the recent changes.
  
  The consistent, sharp dips every 15 minutes made me assume that the container was being restarted.

You just can't beat the dopamine hit from "pointy chaos graph go smooth". Delicious. Great work!

Is that kibana or graphana?
- That's the grafana icon and it says grafana top right
- grafana, judging by the logo
- Graphana

Who you hosting with?
- They have a dedicated server: https://lemmy.world/post/75556
  
  I figured haha. I was wondering which company they used
  Whois shows Hetzner which answers my question :D

I am not seeing them. Are they gone?

Thats ~19 cores pegged at 100%, eating 128GiB of ram (OS disk cache included) and bleeding onto swap. 🤯
- I think you're misreading it. The olive green in the CPU chart is idle. RAM cache taking up most of system memory is also normal on most Linux systems, even on desktop. That cache is freed for applications to use as needed.
  
  Welp, my only calculation was "64 cpu threads * 30% load -> ~19 cores busy", I may be guilty of rounding up too much... The RAM usage is intresting however, since the kernel seems to be caching all it can, to point ejecting uneeded data into swap in order to retain the disk cache. If more ram is reserved by running processes, the (likely pict-rs, database services) disk access times will begin to degrade.

Pretty cool stuff.

I don't know what any of that means, but graag gedaan!

This is cool! Thanks for sharing

199 comments