1y ago

PSA: We're back, here's a post-mortem!

Hi Beeple!

Here's a vague version of events :

11PM EST: Lemmy.world got hacked
12:20AM EST: Blahaj.zone got hacked
12:25AM EST: I shut down the server
12:30AM EST: I make announcements to tell people about this
12:45AM EST: I have an idea of what the problem is but there is no fix
2:20AM EST: I go to sleep
8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied
11:40AM EST: We start testing things to make sure are working And well, now here we are.

If you have issues logging in or using an app:

Log out if you somehow are still logged in
Clear all cache, site data, etc.
Hard refresh Beehaw using CTRL+F5
Log back in.

If you still have issues, write to us at support@beehaw.org

To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.

Oh yeah, in case, you haven't, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw

You're viewing a single thread.

86 comments

12:30AM EST: I make announcements to tell people about this
I think it'd be beneficial to have more backup lines of communication for announcements than just Mastodon.
- We have Discord and Matrix channels as well. Do you have anything to suggest?
  
  Something like status-page is always nice. I haven't used it but it looks like https://cachethq.io/ could be a decent fit as well.
  
  There you go, courtesy of @Penguincoder@beehaw.org
  
  Heck yeah! Thanks for getting this up
  
  Just something Google-friendly.
  
  Nah.
  
  I'll be blunt and say that unless you were already in-the-know, Beehaw pretty much ceased to exist when the server was shut down. Not the best result amidst a hacking scare.
  
  Much preferable to the announcement of Beehaw was hacked and lost your user credentials
  <or more>
  . Security trumps convenience.
  
  Having an entirely separate website, blog, or social media account for announcements that's accessible via a Google search wouldn't factor into how secure Beehaw is.
  
  Right in the sidebar.
  
  And how were users supposed to be able to see the sidebar while the server was offline?
  
  You could have checked it before and follow their Mastodon-style account and join their matrix and/or discord groups, like most of us did.
  Because everything they do server-wise is announced in those places, preemptive shutdowns included.
  Alternative ways to reach the admin team and to be kept aware of anything happening with the server exist. If you didn't take the time (seconds) to join at least one of them, that's not the server's owners fault.
  
  Like most of us did
  Considering the responses to the thread, I don't think that's true.
  Alternative ways to reach the admin team and to be kept aware of anything happening with the server exist.
  A lot of people, myself included, are still getting used to Lemmy. The status quo has been if stuff was happening to Reddit there was an easily accessible server status page you could search up. I tried to do the same this time around and Google came up with diddly-squat. I don't think googling Beehaw to figure out what's going on is that illogical of a response.
  
  Can you be more precise? What exactly do you recommend? I don't know what would be more "Google-friendly"
  
  Maybe the front page of the domain could be news and info with the actual forums down a level? Not sure if that works with the software.
  
  That's not supported by Lemmy unfortunately.. Most we could have is a status.beehaw.org, really.
  
  A status website would honestly be excellent.
  
  There you go, courtesy of @Penguincoder@beehaw.org
  
  Thank you both.

86 comments