1y ago

PSA: We're back, here's a post-mortem!

Hi Beeple!

Here's a vague version of events :

11PM EST: Lemmy.world got hacked
12:20AM EST: Blahaj.zone got hacked
12:25AM EST: I shut down the server
12:30AM EST: I make announcements to tell people about this
12:45AM EST: I have an idea of what the problem is but there is no fix
2:20AM EST: I go to sleep
8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied
11:40AM EST: We start testing things to make sure are working And well, now here we are.

If you have issues logging in or using an app:

Log out if you somehow are still logged in
Clear all cache, site data, etc.
Hard refresh Beehaw using CTRL+F5
Log back in.

If you still have issues, write to us at support@beehaw.org

To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.

Oh yeah, in case, you haven't, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw

86 comments

Huge props for being one of the few major instances to preemptively shut down!

shutting down the server early was best. the nature of open source software is what allows these incidents to be mitigated as quickly as they are. thanks a lot to you guys, and to all of the team at Lemmy who worked to resolve this.
heroes <3

Thank you for shutting down rather than "wait and see"! It was the right choice.

Agree with everyone else. Thanks for shutting it down.
I'll most likely do it anyway, but do you think password changes are necessary at this point?
- I don't think this is necessary.
  We had no messages on our database that had the vulnerability though some were federated from blahaj in the aftermath. The JWT, which is your session token, was changed as well so it seems very unlikely to me that this needs to be changed. There's no reason to believe the attack could've given access to passwords.
  
  Thanks! Was just making sure.
  Anyway, really glad you guys took action as soon as you realized what was happening. Super appreciated.
  
  I don’t think this is necessary.
  I'd add that it's basically useless. From what I've seen, resetting your password doesn't even invalidate previously issued JWT tokens, which would be the only reason to do it. But of course, you've already reset them all and so has lemmy.world.
  
  We had no messages on our database that had the vulnerability
  This is interesting. I actually commented about the use of emojis/emotes a couple days ago on a post on !foss@beehaw.org made by a federated user from lemmy.one, that has since been removed (😕), but I still have the bookmarked comment in which I copied the raw embed for the remote emote image in the federated comment I was responding to.
  Do I understand it correctly, that the latest fixes to stop the code injection, will still allow remote image embedding, so something like an "emote picker extension to embed animated GIFs from a remote server and/or remote instance's emoji list" would still be doable and wouldn't pose any risk?
  Or would such picker still have to include measures to prevent offering embeds with potential exploits?
- To add onto what @Lionir said, you'll never be wrong to change your password, even if much like in this case it isn't warranted. For future reference, my recommendation is "if you have to ask, rotate your password." Finding out later you didn't have to is so much better than finding out later you should have
  
  Oh, yeah, totally agreed.
  And I like the way you worded that. Really good rule of thumb and easy to remember for everyone.
  
  I disagree - rotating passwords comes at a cost especially for people who don't use a good password manager (and that is basically everyone). It's security theatre and generally creates distrust between people offering security advice and the people who (hopefully) are listening.
  There are times when it should be done, but don't do it without a reason.

Glad it's back up. I went outside. It was hot af and boring.

Good work.
Have a non custom beer 🍻

huge Ws, excellent work
also, thanks for the Mastodon link, i wasn't sure where to check on beehaw status during the outage

Awesome response, and a great succinct postmortem. Thanks for doing what you do!

The shutdown is a good call given the circumstances.
An idea of less-radical preventive action is placing the instance in read-only mode, either as a Lemmy feature, or through reverse proxy settings (eg reply 503 for any POST/PUT/DELETE request). But that'd require some development and/or preparation.
Doing that on the reserve proxy side would block any user-submitted content and more (logins, searches, ...). This would hopefully be efficient at blocking many attack vectors, while still keeping the instance partially online, even if that's a degraded mode.
- Note that if this were a Lemmy feature, if we had been infected, an admin could've gotten hacked and as a result, disabled that feature. I'm not really sure what can be done to make Beehaw foolproof. That said, the UI has since been hardened by CSP headers so this type of attack should no longer be possible.
- Would read-only mode help with XSS exploits though, like this particular one? Since the "damage was already done" by the time anybody noticed, wouldn't putting the site in read-only mode still have kept serving up the XSS payload? It'd stop "infected" people from making any state mutations on Lemmy, but eg. data exliftration would still happen

This is why I am on Beehaw. The Admins really care about the Instance and the content on it.
That's why I want to bring attention to the fact, that U can support them. https://opencollective.com/beehaw
I am not a Admin, Mod or anything else. I just really like Beehaw and support them. And you should too.

Thank you for all you do, from what I was hearing I was in no way expecting you to have the site back up within 12 hours. Many kudos.

Maybe post to https://hachyderm.io/@beehaw to spread the word outside of Beehaw.org

Good job on making the right call and preemptively shutting the server down. Thanks for being alert!
- Far more memorable than all the times the service was unavailable was all the times your data was breached. I'll always prefer the service being down to having it up, and vulnerable

morning thought: I've definitely joined the right instance. (also the start from the assumption of good faith guidelines linked to in Gaywallet's recent post)

Great job keeping the site safe guys!
Nice to see it back up again! It being offline was surprisingly palpable. Missed it!
I'm guessing it's probably not the last big thing that's going to hit Lemmy instances in the future, everything still being in early development and all. Only things we can do is keep an eye out, have vigilant admins and plenty of backups!
And patient users but we seem to have that. :)

12:30AM EST: I make announcements to tell people about this
I think it'd be beneficial to have more backup lines of communication for announcements than just Mastodon.
- We have Discord and Matrix channels as well. Do you have anything to suggest?
  
  Something like status-page is always nice. I haven't used it but it looks like https://cachethq.io/ could be a decent fit as well.
  
  Just something Google-friendly.

Welcome Back 🤗

Amazing job! This is not easy to go, given that you're working with an immature product and a changing landscape.

Anyone know where we can get updates on what is happening with lemmy.world? I have an account there as well but I'm afraid to even open the site now.
- Lemmy.world is fine now. They were able to get a handle of things.

Your work is greatly appreciated! Also happy to know that you got some sleep, very important for the process ☺

Content-Security-Policy will really help save your ~~bacon~~ beans and protect against XSS. Hopefully the Lemmy devs can apply a super strict policy to help. IMHO it's a must for any site with user generated content.
- This is what this PR has done as I understand it : https://github.com/LemmyNet/lemmy-ui/pull/1907

Great job! Being preemptive in a case like this is very good! Thanks for all your work!

yayyyyyy

Awesome work sidestepping the hack.

Was the hack related to the lasted patch update?
- No it was related to custom emojis.
- I don’t believe it was related to anything specifically added or removed in a recent patch, I think it was an existing exploit that just hadn’t been noticed.
- I'm not sure what you're asking.

86 comments