Remote View

1y ago

[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back

We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

351 comments

Thank you for the transparancy!
- I love post like this, another reason of why this instance is so good

Thanks for all your hard work @ruud@lemmy.world. It's not easy!

Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!
Dessalines:

Nutomic:
- Yeah they've been hard at work all month. But it's also okay if things aren't ready in time. Most of the people who matter are already here.
  Maybe we will blow up soon, maybe later, but the quality of content here is sufficient to drive growth regardless of whether or not we get the prophesized huge migrations from reddit
  
  Agreed. I'm here to stay with the content from the last couple weeks. 3 day poop embargoes alone is worth it. And recognizing users because the community is still small enough is SO different from they who shall not be named.
  
  It's adequate the way it is now. They should hold off on any major upgrades until after the next batch of new users come in. Lemmy breaking during that period would be far worse then missing some updates and new features.
- Oh god, so much coffee... it's also thrilling to grasp a sense of what's going on under the hood of such big social networks at a development level (not like I could understand it, but it was very interesting to see twitter's recommendation algorithm being open-sourced)

is that users can’t login anymore
Ouch, that's bad :(
Thank you for your work, as always :)
- I mean that’s a Tuesday on Reddit lmao.

As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.
We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.
- Although since we seem to be rolling more than a ship in a storm, I think a proportion of lemmings would end up hospitalised for alcohol poisoning.

Oh. Good luck with the next try!

Thanks for the hard work /u/ruud, we'll get there

Running one of the largest and most vibrant Lemmy instances certainly makes this deployment a great test case for the devs and the system overall. @ruud@lemmy.world this community really appreciates you!

- That doesn't make sense. Of course you can fail. Stop filling the boy's head with this nonsense.
  
  The motivational post is self-explanatory

pretty brutal bug ngl. was hoping I'd be using jerboa again soon
- Is the bug with Jerboa that no subscribed communities are showing up? From my point of view logged in now I only have access to lemmy.world communities
  
  Jerboa crashes for me
- Download v0.34 from F-droid or Obtainium, which both let you downgrade. It's compatible with Lemmy v0.17.

appreciate the transparency. how are things looking in the back in lemmy.world (server wise)? will we get to a point where it wont require complete rollbacks on the state when a botched update gets rolled out?
- Not having to do a database rollback is a really, really hard problem to solve, and it would almost certainly need to be on the Lemmy developers side, not the server owner's side. And if I'm them, that's a low priority issue, and probably not something I even think about until 1.0.
  Basically, they write code that says what to do in the event of a database version change. Usually this only handles upgrade cases, because that's what happening most of the time. One example of something you might do in a db upgrade is let's say you had a column where the data type was only numbers, but now you want to allow any alphanumeric character for some reason. You could have a line of code that converts the number to a string.
  Okay, but now you need to go back to the previous version. Okay, your db change code runs, but it's the old version of the db change code, not some new version that you wrote. You unfortunately didn't have a crystal ball when you wrote this code and couldn't predict that you were going to change the data to strings, so you didnt write code to change it from a string to a number.
  This is why most software doesn't support downgrades unless you wipe first. For example, if you updated your aging MacBook to the latest Mac OS version, then realized it slows down your laptop too much, you can only go back if you first wipe your laptop in the process. So it's just easier to just take a snapshot before an upgrade and revert to the snapshot if it fails. Some folks will even do "scheduled maintenance" time during the upgrade in which the whole system goes down for a short time so they don't have to risk losing data that happened after the snapshot.

I want to upvote, but I get this instead.
- Can you clear your browser cache? Should be ok after
- Oh man, for a sec there you made me thing I was getting errors 😅
- Same on thunder app
- I'm getting the same on my iPhone through lemmy.world

Suddenly my Jerboa won't change from local and I can't change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.
- What you're describing is an issue introduced with Jerboa 0.0.36-alpha and has been fixed in today released 0.0.37-alpha.
  
  Oh nice
- Mine randomly quit working a few days ago. All I get is a blank screen. I've switched to the website for now, though I need to look into the other app options. There are a couple more now.
  
  Wefwef.app is pretty interesting. Its browser based and so far seems more stable to me than a lot of the current Android apps. Not sure what happened, but Jerboa and the like all have become nearly completely unusable with bugs and crashes.
  Wefwef seems to unfortunately be lacking some very basic features tho.
- Mine's been doing it since yesterday, on all instances.
  There's an annoying workaround, but you can change your default sorting in settings and relaunch. It's getting stuck on whatever your default sorting is.
- Me too, I am using Liftoff for now.
  
  I am also trying out Liftoff
  The UI and design looks nice but the chunky 12 fps scrolling is a bit annoying. I hope it's gonna be better.

I am glad I didn't write my last post during the upgrade process.
Maybe next time, give a warning, or maintenance notice.
Thanks for trying though.
- That’s just a pain point of using modern asynchrony software and will never go away

Was worth a try! But yeah that was pretty bad 😬
- Yep that was terribly slow

I’ve found from hopping around some other instances that have upgraded to 0.18 that it is still pretty buggy. It does seem to be giving more information about the errors, instead of just failing like in 0.17, but spend any time browsing on those instances and you’re bound to be inundated with JSON and query errors. It also seems to get worse the longer you browse.
The UI changes are nice, and I do appreciate not having my feed auto-updating constantly, but I think you’d be making the right choice to hold off on upgrading until they can iron 0.18.1 out all the way. I’m not super knowledgeable about TS and Rust, but as a user it seems that switching from WebSocket created/shined a light on Lemmy’s issues with caching in general.
- but as a user it seems that switching from WebSocket created/shined a light on Lemmy’s issues with caching in general.
  Or just adding actual error messages instead of ignoring them and throwing up a spinning wheel.

Im already in lemmy 0.19.0
this comment is from the future
- How is it?
  
  Replies don’t work

Good luck! May the software gods bless you with a functioning website with minimal bugs!

Thanks for the transparency. Maybe it’s a good idea to have a test instance and some test cases/validation done there before updating the main instance. This is a regular process in any software/tech company/stack.
Testing should never be done directly on the prod instance.
- i think they tested it which worked fine, but it is totally different when there are users online
  
  Correct there was a test instance and that worked fine but once the live instance was upgraded everything became very slow. Can't really emulate that kind of load

Linux/Firefox After upgrade many things didn't work, but they continued not to work after rollback.
Turns out I needed to clear the cache, then everything worked fine again.

Dude I posted the funniest comment in those 10 minutes but you all must have missed it. Oh well 😅

I'm so new, I'm a baby, I have no idea what I'm doing. I did just create my account, I tried using the app but it wouldn't let me. Saying server version too low. I'm sorry for my ignorance, but is this what that means? And this means we can't use the app yet right?
Also, thank you for all of this. I will be here for it all, I'm staying forever! I already love this so much more than anything else 🥰
- You are probably on Jerboa. Either downgrade or use the web UI interim.
  
  Or use any of the other apps
  
  Sorry, I'm so dumb.. is there a place I can learn how to use Lemmy better? I don't know how to downgrade 🥲
  Thank you for your help.
- In Jerboa? Yeah. There's other apps for Android though in development.
  For example I'm enjoying Thunder: https://github.com/hjiangsu/thunder/releases/download/v0.2.1%2B7/thunder-v0.2.1+7.apk
  And there's also Liftoff and Connect for Lemmy on the Play store.
- Until we can get the instance updated, yeah the jerboa app doesn't really work. You could try another app, I'm using liftoff but that's harder to install, you could Google "thunder Lemmy" and try that one, I haven't tested it much but I've heard good things
  
  Thank you for your help 😊
- I'm a newbie too and that happened when I tried to sign into Jerboa. Using Thunder app now because I don't know how to fix the other one.

~~RC2~~ RC4 was just released, here's hoping it solves the issues you were experiencing with performance of RC1!
- I thought RC3 was just released?
  
  Dang! They're moving fast!

It seems that my Hot feed has been stuck for the last day or two.

Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.
Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though
any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?

the work you've done in the last month is incredible. I hope you can get some rest in between bursts of carrying the instance on your shoulders

Thats for all the work you're doing on this.
Just FYI, here is a different recurring issue that keeps coming back, I've strictly been using lemmy.world website, keep receiving MLB Game stats for the same day when opening random links, just happened when I opened this thread to view comments. So far if I just reload the page it will then show the correct content. But it keeps happening.
- Strange, probably one of those weird Websockets issues that should be solved in 0.18....
- I've had the same issue a few days ago, so far nothing too major since it can be refreshed like you mentioned, but still a nuisance.
- Same, it happens to me a lot!

Thank you for the update

@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?
With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.
- Yes, considering that. But we'll need people to use that when we will do testing...
  
  Well, you have at least one person who would be willing to use it ;D
  
  I'm not familiar with what the server architecture looks like, but is there a possibility of using a load balancer in front of the instance's server and swapping a "beta" server into the load balancer when you need to do testing? You could basically migrate your traffic with zero downtime, assuming Lemmy's architecture allows for it.
- Is there a way to have two instances writing to the same db? That'd help a lot in this situation so the test instance can still be accessing the same data as the stable instance. Otherwise we'll never have enough load to fully stress test and be in the same situation as the three existing Lemmy test instances.
  
  Seeing that with these updates often come with changes to the database as well, that could be very difficult if not impossible.

Jerboa is almost unusable for me at the moment. A lot of crashes, pages not loading, messages not sending etc. But I know that's inevitable at times. Restarting my phone seemed to make it worse. But I know you guys are on it! 👍
- I recommend you this web app
  https://wefwef.app
  It is almost as functional as Jerboa and very impressive for a web app.
  
  Already has a lot of what I liked about Apollo just in a web app, very cool!
  
  Wow, awesome design!!!
  
  I just switched to it, Had tried jerboa, connect and liftoff and this one is by far my favorite
  
  Currently rate-limited by Lemmy.world, per the dev. You might get a lot of red error boxes.
- Liftoff is a much better app. I used Jerboa as well but found an alternative cuz it was pretty much unusable. Just dowload Liftoff's latest apk and install
  
  Holy crap I just checked right now and Liftoff is on Playstore already!
  
  I am using it too, quite nice, it still miss some quality life features, but I am they will get to it.
- How did you install it? I was using the app through the Google Playstore and for some reason the app was really unstable and buggy. I ended up installing the app from F-Droid and it is drastically better and much more stable.

Bummer. Other instances apparently managed to upgrade, but none has the scale of lemmy.world, so that was bound to stretch things to their breaking point. Hopefully the devs are on top of whatever's causing it!

I'll admit this is probably something I can Google, but is there any scale-out options for a given Lemmy instance, or is it designed to be hosted in a single server?
- If I understand correctly this isn't a scaling problem as much as it is a compatibility problem that led to other downstream effects after it was remediated.
  
  That's fair, I'm kind of conflating scale out with HA. When you have a cluster, you can typically upgrade part without bringing down the whole instance, but it may just not be mature enough for that yet.

In the meantime I've migrated over to Connect for Lemmy on Android, and it is working significantly better than Jerboa did.
- maybe they could partner with other app developers, so they can spend more time in the lemmy code instead of the app
- Thunder has been making leaps in the last few updates as well.

thanks for your hard work ruud!

Appreciate the communication & the effort.

I had a weird issue after it upgraded. My browser (Firefox on Android) showed the Lemmy UI but showed no posts. I checked my profile to see if I was logged in, and it showed I was logged in as someone else! I cleared my cache and refreshed, it showed I wasn't logged in. Logging in worked fine, and showed my expected posts and my actual profile. In Liftoff app my account worked fine the whole time, didn't need to log in again.
- It was just a visual bug, happened to me too. As soon as I tried accessing the profile section I was logged out and couldn't login again.

Looks like we're still having issues, more's the pity.
Do we know what's causing the issue, or is it still a bit of a mystery for the time being?

After the update I can no longer log in or interact with Jerboa. Hopefully y'all resolve this soon, as I am really digging Jerboa, but thanks for letting us know what's going on, and keeping up the good work.
- There's a major breaking change in the 0.18 update (moving away from Websockets for posts and comments), that means that older Jerboa builds no longer work with instances on, or past, that version, and newer Jerboa builds won't work with servers older than 0.18.
  
  Version fracturing is probably going to be a fact of life on a decentralized platform like lemmy, a "good" app should probably handle cases for the latest few revisions instead of just the latest.
- I've been using Jerboa since I joined a few weeks ago, I haven't been liking any of the other clients so far, but today I tried thunder and I'm enjoying it more than Jerboa
- Strongly recommend you also try Liftoff, updates have been on fire the last few days and it's really quite nice: https://github.com/liftoff-app/liftoff

That's alright. Thanks for all your hard work!

Sucks that lemmy.world won't be working with Jerboa when the reddit apps quit working in a couple days.
- Jerboa still works fine with lemmy.world. It pops up a warning that the server version is unsupported, but it still seems to work fine.
  
  After I get the warning I can tap the hamburger menu and it shows my account name up top but when I try to upvote a post it states I need to log in.
- Better than being flooded with bots.
- alternatives like LiftOff is available
  https://play.google.com/store/apps/details?id=com.liftoffapp.liftoff&pli=1
  
  It's available from play store now? Awesome! Will definitely try out.
  
  Liftoff is amazing.
- It will work, I'm using it. Just ignore the warning message
  
  "Work" is subjective. I can ignore the warning message, but the app has become incredibly unstable to me to the point where it crashes every minute or so. It's almost completely unusable.
- On the plus side, I'm about to open the beta that handles both just fine (plus a lot of features no one else is doing yet). The deadline I for myself is Saturday when the Reddit clients go down.
  Unfortunately it was built by someone who lurks and comments, and I realized even though merging feeds from accounts on different servers is going to be awesome, other people like to post and I need to stop getting off track
  The stress of the timeline is starting to get to me and I'm terrified no one is going to give it a chance, but I've fixed most of the things that drove me nuts in jerboa... That should count for something, right?
- On the plus side, I'm about to open the beta that handles both just fine (plus a lot of features no one else is doing yet). The deadline I for myself is Saturday when the Reddit clients go down.
  Unfortunately it was built by someone who lurks and comments, and I realized even though merging feeds from accounts on different servers is going to be awesome, other people like to post and I need to stop getting off track
  The stress of the timeline is starting to get to me and I'm terrified no one is going to give it a chance, but I've fixed most of the things that drove me nuts in jerboa... That should count for something, right?
  
  Hol up, are you soft announcing the development of a new Jerboa-based app? What's it called? Good luck to you! I hope you achieve your goals with the project and that is well received. ☺️
- Liftoff is a much better app. It's working great
  
  Thanks, seems much better!

Thanks for the update, good luck!

Your time and efforts in keeping this instance going are most appreciated! Thanks again!

Thank you for your work!

If at first you don't succeed, try every other RC until proper :)
We have a good thing going, don't rush it, we'll wait :) and thank you for your service.

Thank you for the hard work Ruud!

Posted after Edit 2.
I would like to manage my expectations. Yesterday we were looking at 0.18.1 being sent out in about week from the devs. Was 0.18.1-rc.1 what we thought they'd have ready? Do we still think the 7-10 day window is likely?
- rc.1 is the first Release Candidate of that version - not the final one, so between now and a week a lot can still happen.
  
  Thanks. It's neat how active you and the other admins are. You guys dedicate a lot of time to individual comments, and not just on this thread.
- No they said in a week they would have it final. But maybe it will take a bit longer....
  
  Lemmy seems like a herculean undertaking. I can't imagine anyone doing the nuts and bolts for free or on any schedule, so it's quite amazing to me that we can talk about getting updates like this.

Thanks for all your hard work

Bummer. What keeps me from using Lemmy more atm is that it‘s unresponsive and gets stuck a lot on my end (iOS). I was hoping the update would improve on this. I hope future upgrades work better. Thanks for your work!

Jerboa crashes constantly for me now, doesn't even let me view anything. JSYK!
- You might want to try https://wefwef.app
  
  A variety of food scraps, seed, fruit for the parrots, and water is also very important.
  
  Finally got around to trying it and it works a bit better than the browser version and Liftoff, and 100% better than Jerboa. But it's very very slow. Thanks for the extra option, though!
- There's a fix the dev has just released. See reviews section on the app. He said it'll take a few hours to show in the playstore.
  
  This is good news! I haven't been able to try Jerboa fully yet since I use lemmy.world. I'm not seeing this review section you're talking about but I'll be on the lookout for an update. Thanks for sharing!

Thank you and good luck!

Thanks for trying and keeping us updated!

351 comments