Redis, Memcached, dragonfly for Lemmy and 2010 presentation on scaling....

There is some kind of social construct with Lemmy's 4.5 year development that foundational tools like Redis, Memcached, dragonfly were avoided when in May 2023 people were beating a path to the door of Lemmy. And lemmy servers crashed, one after the other, and I don't think there will be any statistics about all the times the pages didn't load and the 4 year old Lemmy app's SQL statements couldn't cope with the comments/post growth for even 20,000 users.

Sure, people will leave Reddit and Twitter/X and Threads and whatever again later in 2023 and in 2024. But it's still some pretty odd social situation that May 2023 came along and performance problems were holding the project back... and Redis, Memcached, dragonfly were not put on the table as routine tools of the trade.

MONDAY, MAY 17, 2010 Reddit spelled out all the performance and scaling problems they had, they gave an open presentation. The source code was open since June 18, 2008... long before this May 17 presentation.

If anything Reddit should have turned itself into a non-profit organization and kept selling reddit awards after ChatGPT came on the scene. The 2 month move to "charging for the API" was the wrong direction.

Twitter and Elon Musk with X, it just seems popular to turn things bad. Crashing servers, broken features, wild changes.

Clickbait news and anti-science popularity still seem to keep growing. Reddit or Lemmy users could replace headlines with sincere and earnest descriptions of news and information... but it seems Reddit and Elon Musk are in some kind of agreement that audiences don't really want that. In APRIL 2017, Wikipedia Founder Jimmy Wales tried to take on clickbait and fake news, but few have cared and clickbait headlines are still all over Reddit and Lemmy in 2023.. I don't get the appeal and attraction of junk clickbait all the time. But it's hard to ignore the upvotes it gets on Reddit and Lemmy both... you can watch it every day.

Leemy seems determined it doesn't want to optimize SQL statements and add scaling tools like Redis, Memcached, dragonfly to the platform. The crashes have been all over multiple sites since May, but the SQL problems and caching need is still ongoing in August.

The trend of Reddit, Twitter the ongoing favoring of clickbait seems intertwined.

They store more data now in Memcachedb than Postgres. It’s like memcache but stores to disk. Very fast.

And that was 7.5 million active users with 2010 hard drive technology. Hardware in 2023 is way faster. Lemmy is incredibly unique in it’s stance of not using Redis, Memcached, dragonfly… something. And all the CPU cores ... and still crashing.

For 4.5 year old code, I think the crashing speaks for itself. But this is the days of Elon Musk and Reddit changing their API with 1 month notice... the social aspect of social media seems to be making choices that just seem ... odd. And it isn't just Twitter, it isn't just Reddit...

I mean, May 2023 people flocking to the Lemmy door, your servers can't keep up with the poor-performing SQL... add scaling tools like like Redis, Memcached, dragonfly to the platform. There was a 40-day countdown to the Reddit API change.

I don't understand what has happened here on Lemmy since May any more than what Elon Musk is doing to Twitter to X. It's bewildering. Redis, Memcached, dragonfly are easily integrated...

"Redis has been a staple of the web ecosystem for years. It’s often used for caching, as a message broker, or simply as a database. In this guide, we’ll demonstrate how to use Redis inside a Rust web application.".... and the same exists for Rust and memcached, and Rust and dragonfly... Lemmy was in active development for 4 years, it's impossible to accept that nobody knows of tools like dragonfly and Redis.... its some kind of social choice at play, one that I find bewildering and frankly, has put me into tears as people seem to think the server crashes can't be addressed!...

the crashing servers speaks for itself, but I guess I spoke for it too. It's been a sad journey since May :( I don't understand what is gong on in social media audiences that Elon Musk and Reddit and Lemmy are making such choices.

I'll just say now that I haven't read everything, just skimmed through. Here's my 2¢

Considering Lemmy's contributors are volunteers, and Lemmy itself is written in rust, it'll probably be a really difficult one to try and attract knowledgeable volunteers willing to implement things like Redis and Memcached in a way that will hit the ground running.

Looking at how things are right now, I think the primary devs Dessalines and Nutomic are likely trying to power through the remaining outstanding features to get the next NLNet funding payout, so they can put food on the table.

After the funding comes to an end, I feel it's likely that we'll see a more ambitious roadmap calling on people to assist with implementing caching technologies, which will probably involve a big shakeup of the existing codebase