Recap of the Lemmy XSS incident & steps for mitigation

UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.

Original post follows:

This post is intended as a central place that admins can reference regarding the XSS incident from this morning.

What happened?

A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.

Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.

Am I vulnerable?

If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.

I had custom emojis on my instance, what should I do?

This should be enough to mitigate now:

Remove custom emoji

 undefined

    
DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;

Rotate your JWT secret (invalidates all current login sessions)

 undefined

    
-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();

Restart Lemmy server

If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)

Legal

If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402

More context:

https://github.com/LemmyNet/lemmy-ui/issues/1895

https://github.com/LemmyNet/lemmy-ui/pull/1897

You're viewing a single thread.

62 comments

Thanks for posting. There really should be a button which allows the admins to log everyone out for crisis situations like there I think
- Changing the JWT secret does this. So instead of a button, its a line of code, making it less likely to be done by mistake.
  
  True as that is, not every admin has DB access and those with DB access might be AFK.
  
  Surely this exploit proves that it's best to minimise the number of administrative actions available in the UI?
  
  Destructive actions, sure. This would not be one such actions and it would be self-defeating.
- In crisis situations like this, the last thing you want is to rely on some portion of the application UI working.
  
  If the UI doesn't work, there's always the DB directly. But it's good to have that option.
  
  JWT secret keys are not in the DB (speaking typically, maybe for Lemmy they are, but that would be very surprising), that's typically an environment variable or configuration file sort of thing.
  In any case, this isn't the part that's broken, it doesn't need fixed.
  
  undefined
  
  Rotate your JWT secret (invalidates all current login sessions)
  -- back up your secret first, just in case SELECT * FROM secret; -- generate a new secret UPDATE secret SET jwt_secret = gen_random_uuid();
  It's in the DB. Check the OP
  I am also not suggesting a fix. I am suggesting an improvement
  
  Oof, okay well that's not how I would've done it. The JWT secret in the database itself could be a vulnerability (e.g., someone that gains read only access to the database could then use that as a wedge to create any JWT they wanted). I'm not sure if that's actually worth bringing up or not (it's a bit of an odd case).
  
  afaik to generate those tokens, you configure a secret in an enviroment variable. You cannot generate tokens from looking at valid tokens within the database. Thus storing active tokens in the database is fine since you can always purge all active tokens as this post has also suggested.
  
  This is the secret, not the tokens.
  
  oh i see, they want to delete the secret instead of the active tokens. Yeah now i get what you mean. Seems kinda odd.
  
  I don't know enough about Lemmy's JWT design, but some JWT designs don't store the JWT in a database at all, so the only correct response is to regenerate the secret and kill all the sessions by them failing the validity checks.
  
  👍
  
  ~~...that's what i just said?~~ https://lemmy.dbzer0.com/comment/793036
  
  No, you said you can't generate valid tokens within the database. I just told you this is the secret, not the tokens (that is present in the database).

62 comments