Lemmy.world (and some others) were hacked

Very, seems like great work.

Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.

I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.

In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

uh, why did you have negative one dislike?

Deleting hospital fees/debt is very dangerous... In many HUGE regions in the US there's only one hospital and if that hospital suddenly can't pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.

In fact, something like this already happened:

https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/

Hospitals are special in that they're often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).

Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.

Before that can happen though people need a backup payment method that doesn't go through their systems and no: Cash won't work (there's not enough in circulation and it's dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).

As soon as people have a widely accepted payment option that doesn't go through Visa and MasterCard's middlemen (e.g. First Data) then hackers can take their gloves off! Until then though... Let's keep the payment infrastructure working, OK? Thanks!

There's no limit to the amount of good deeds hackers can do though. So let's encourage that! For example, there's plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces 👍

Considering some of the targeted instances and the stuff they left behind, it was likely some nazi.

“Marty we started this journey together!”

“It was a prank, Cos’!”

Ribbit

clearing credit card debt, deleting hospital fees, or something else that actually serves the public good,

Inflation does very clearly not serve the public good. That aside, causing havoc in banks and medical institutions would have other unpleasant effects.

Thank you.

In case anyone else is having trouble logging in, my password wasn't working so I had to reset it from the website.

Further 3rd party heads up -- for us nontech refugees:

If it looks like you are logged in, you may not be. I use Connect, and at your reminder, I clicked my acct and it says I was logged in. I tried to comment that Connect login was working, and my comment didn't show up.

I tried again, only to see an ”error: not logged in” message pop up.

Signed out, signed in again manually, and all is well.

So do a double check, Lemurs. Trust in your actions, not your eyes.

For capable people this is a minor annoyance but whenever there’s an “everyone needs to login again” issue, we will lose mere mortal users. In this case it wasn’t even clear that was needed - I appeared to be logged in but nothing worked. Ordinary users give up over things like this. I’ve seen it happen many times on sites where I had access to the analytics. I hope we regard this as a really bad thing to be avoided at all costs and not a “no big deal, just log in again.” Easy for you, easy for me, many others will just bail.

Thanks for the heads-up.

Thank you!

Thanks for the heads up. I was starting to get frustrated with not being able to upvote/downvote/subscribe/comment, but logging in again seems to have done the trick. Also, this is my first ever comment on Lemmy. Hooray.

EDIT: for anyone using Memmy on IOS, this might not be the only solution, but it is what worked for me - I didn’t see an option to sign out anywhere, so I went to the account settings and added a new account, putting in the details for my current and only account. Weirdly, this made my account show up as 2 separate accounts, which made the “delete account” button appear. I clicked delete on one of them and was immediately sent to the login screen. I logged back in from there and now things work fine.

Good point, I'll update the post.

Thanks for the info. We're looking into this.

If a valid browser token gets stolen like in this case, then MFA won't do much good because the stolen token will already have been authenticated. Linus Tech Tips experienced the same thing recently, you can check out their channel.

Out of curiosity, where would the regulators go for a case like this? There's no "company" running it per. se.

Can 2FA be enabled for all users? I don't see the link to activate it after saving.

edit

Yeah, this doesn’t work at all. The apps don’t open links anymore. I tried some github site that reads the link and generates a QR, but the codes don’t work. This is a complete waste of time.

Just curious if you turn yourself in to police everytime you speeding.

Good point. I did post about this on Mastodon @mwadmin@mastodon.world

Wow that's cool. How did malwarebytes know the website was compromised ?

Interestingly, here is a log when browsing Lemmy over the last week or so.

Believe the derp.foo and .today are both federated instances. Don't know what the other rows are.

Hey how do you check on that?

As of the time of me posting this comment, exploding heads is appearing in my feed with some anti lgbt posts. Idk what’s going on because I’m pretty sure they’re supposed to be defederated currently

Could you please link the issue? Thanks!

With Jerboa, try logging in and you'll get this bugged state. Now close the app from your multitasking view, open it up again. If the account shows up, sign out and clear cache. If the account doesn't show up, log in.

This should make it work right again, at least it worked for me.

Thanks for this comment, I had to wipe cache and cookies to login on desktop. Wasn't sure what was preventing me from logging in.

Yeah, I'm not sure what was going on.

1. JWT stands for JSON Web Token. It's basically a way for a server (lemmy's) to put a piece of information in your browser in a way that makes sure it come from the server. It (usually) uses some form of digital signature. You can think of it as a note someone gave you with their signature, assuming said signature is very hard/impossible to counterfeit. The next time you see that person, they don't have to remember you, they just have to check the signature. If it is valid, anything written on the note is taken at face value.

When you connect to a site, there are a few steps to validate that you are who you say you are (identification and authentication). Something like inputing you login/password. Since it would be tedious to do that on every requests, the first time you give your login/password to the server (this is the simplified version, this exchange is a bit more complex usually) the server gives you that JWT. For every subsequent requests, your browser automatically send that JWT that is simple to handle but hard to counterfeit, and the server safely knows that you're whoever is written in that JWT.

2. I assume there will be a post here when more details are known, or that this post itself will be updated. As with any online service, it's up to the service to decide if they want to communicate. (it may also be a legal requirements in some places to tell user when such an event occurs). Since we're talking about obtaining other user's authentication token including an admin, it is safe to assume that whatever an admin can see has leaked. This can range from basic user informations to more private stuff, although I am not familiar with the software behind lemmy. Note that this is a worst-case scenario; an admin impersonator could have access to anything an admin could see, it does not mean they immediately dumped everything. It depends on their motivation.
3. Protection against this kind of stuff Compromission of the JWT can happen in many ways and I don't know which way was used. But if there's a flaw in the software used (the lemmy's client-side code, for example) there is not much you can do. JWT can leak through many things :

• server compromission (out of your control)
• client-side compromission (only happens when using a browser; applications that uses API should be less susceptible to that)
• vulnerable extension: if you have browser extensions, they can easily peek into what's happening in any given page (that's their whole purpose). Malicious extensions, or extensions that allow outsider's some kind of control over them can leak data
• browser vulnerability: keep your browser up-to-date, and (this is controversial) stick to a family of well-enough known browser. That obscure browser that have 20 users worldwide and is based on a three years old version of chromium is not the best thing to use
• keep your data safe: only put the minimum required amount of data on any service. For lemmy, I assume an email address and your login/password is the bare minimum (well the email is already extra, but it's very convenient to have). Some services really likes to get everything they can out of you.

Basically, stay up to date and don't use shady stuff. Easy to say, I know.

1 - jwt is the authentication cookie u get when u sign in with ur password, ur browser stores the jwt and then any further interaction authenticates using it

2 - yes, userscripts, extensions, custom frontends, apps, all apps have access to jwt

No, passwords weren't compromised

I think it's good practice to change passwords after an attack no matter what

So, do we change passwords

If you don't use a randomly generated password, it's a good idea to change it anyway. Not because of this specific attack but in general. For the longest time the Lemmy software was just a hobby of a very small group of individuals. While the back-end is written in Rust and probably more robust than the PHP code over at Kbin, I don't think a proper security review was ever conducted, so there's a not so small chance there will be some additional growing pains in the somewhat near future.

According to the admin, no, but changing your password and keeping your data safer is always totally fine to do and you should probably do it every once in a while regardless.

Vulnerability strikes. Open source's lightning response strikes back. Again.

@nonagonOrc @ruud 👍🏻

Check the pinned post on liftoff community page.

https://lemmy.world/post/1292303

As a safety precaution logged-on sessions on many servers have been cancelled and you are required to logon again.

as someone who uses the app, extremely little effect from my experience, I didn't notice something was wrong at all until people pointed it out due to how liftoff does the whole sidebar thing for the instance.

It's still better to change your account password and clear your cache.

Was wondering this myself. Is there a way for users who where exposed to know about it?

(Edit) Eg if the exploit was through a post get notified if they saw the post?

Good communication is key to effective incident response. This is a good example of that. Kudos.

It's probably a good idea to have official announcements be signed, that way it's obvious when they're actually posting officially or if they are compromised.

What's the difference? JWT is already cryptographically signed, but tokens were stolen. That's the issue.

I think the lemmy.world admin posted on his official Mastodon.

https://mastodon.world/@mwadmin/110688515627268847

From the fix, I believe the custom emojis were not double checked after a user submits a post. The post data was used to display the emojis, and thus allowing injection.

The fix now is to search the emojis in the custom emojis list from the backend rather than the user post.

The JWT exploit bypasses 2FA requirements. It basically steals your active session and allows a third party to use it.

To answer one question, the admins are able to view email addresses I believe. My knowledge comes from "I read it in a comment awhile ago that sounded credible" so I could be wrong.