Remote View

1y ago

WARNING: Lemmy Self-Hosters, There Have Been CSAM Attacks taking place against !lemmyshitpost@lemmy.world

cross-posted from: https://jamie.moe/post/113630

There have been users spamming CSAM content in !lemmyshitpost@lemmy.world causing it to federate to other instances. If your instance is subscribed to this community, you should take action to rectify it immediately. I recommend performing a hard delete via command line on the server.
I deleted every image from the past 24 hours personally, using the following command: sudo find /srv/lemmy/example.com/volumes/pictrs/files -type f -ctime -1 -exec shred {} \;

Note: Your local jurisdiction may impose a duty to report or other obligations. Check with these, but always prioritize ensuring that the content does not continue to be served.

Update

Apparently the Lemmy Shitpost community is shut down as of now.

156 comments

Someone is trying really hard to hurt Lemmy by continually attacking the most popular instance. Is this all coming from right-wingers upset that their nazi instances were defederated across basically the whole fediverse?
- The simplest explanation is 4chan types just doing it for the lulz.
  
  Could be, I'm surprised /g/ didn't create an instance
- My tin foil hat is telling me it’s one of the other social media companies funding a hacking group to do it. They stand to have the most to lose, and they’ve seemingly decided to enjoy changing the narrative regarding multiple topics. Lemmy stands directly against what the bigger social medias stand for.
  I have no evidence to back this though. As a business owner I just know that things become very consistent when people are being paid, and very inconsistent when they aren’t. These attacks are seemingly very consistent/organized.
  
  You think a company that is posed to go public is going to attack a competitor with a minuscule amount of traffic with extremely illegal material that could put them in prison for even having?
  
  You have a massively inflated view of Lemmy's importance in the social media market.
  
  There must be room under that tinfoil hat for the both of us, because this was my first thought too.
  
  I'd go with state actors first.
  When a particular social media platform is centralized, you can buy yourself a say percentage of stock and have sway over it (cough tencent), or have a useful idiot ruin the platform (cough musk), or another useful idiot to run propaganda you like anyway (cough truth social, cough fox news, cough newsmax...), or yet another that will sell out it's host country's citizens for cold hard cash (cough facebook).
  But when that social media platform is decentralized? Well, then you'd need to figure out how to poison the well early on to stave off adoption. The Saudi Arabias, UAEs, Chinas definitely don't like the idea of lemmy, and it'll be way harder for them to control if critical mass is hit.
- This makes the most sense to me. It's a pretty vitriolic attack, therefore I don't think it's simply a troll while at the same time I don't believe it's any corporate social media.
- Considering all the alt-right garbage that was popping up there the last couple of days this seems at least plausible. I sometimes envy their ability to utterly destroy anything they touch.
  
  I'm sure you'd love to link to some examples
  See people claim this constantly with no proof
- I wouldn't put it past the hexbear crazies throwing a tantrum. They claim to be left wing... Sure seem more like fascist trumper types though. Maybe it's just that they're all incels and incels all seem about the same.
  
  Throwing a tantrum about what exactly? They're one of the oldest-running Lemmy instances. Until now they were running a fork based on a pre-Federation version of the codebase.
  You believe they did a bunch of work migrating their database only to then negate that work by destroying the community they wanted to Federate with?
  
  they’re all incels and incels all seem about the same.
  Downvote from me there. I’ve seen plenty of examples of hexbear people being nice, interesting and good sports. They definitely seem to have more of shitposting culture than is normal on mainstream lemmy. But all in all it’s seemed fun to me from what I’ve seen.
  Beyond all that, this is just superficial and prejudicial. If you had some examples to link to or more substantial insights to share as to why it’d be “them”, that’d be worth reading.
  Otherwise, they’re an instance. Not one person, I’m sure some on hexbear are assholes and some awesome.

So, from memory there has been:
This recent attack
Regular DDOS attacks
Frequent attempts to spam community creation
That one time the instance got hacked and set to redirect to shock sites
Am I missing anything?
This seems like more than just a few trolls. Maybe someone really doesn't want to see user-owned social media take off.
- I see where you're going with this, but no, people really are just absolutely horrible. The fact is that with other social media they're just already very set up in managing this so we never see it. Lemmy wants to be open, this is the flipside of that openness.
  
  It's generally easy to crap on what's 'bad' about big players, while underestimating or undervaluing what they are doing right for product market fit.
  A company like Meta puts hundreds of people in foreign nations through PTSD causing hell in order to moderate and keep clean their own networks.
  While I hope that's not the solution that a community driven effort ends up with, it shows the breadth of the problems that can crop up with the product as it grows.
  I think the community will overcome these issues and grow beyond it, but jerks trying to ruin things for everyone will always exist, and will always need to be protected against.
  To say nothing for the far worse sorts behind the production and more typical distribution of such material, whom Lemmy will also likely eventually need to deal with more and more as the platform grows.
  It's going to take time, and I wouldn't be surprised if the only way a federated social network eventually can exist is within onion routing or something, as at a certain point the difference in resources to protect against content litigation between a Meta and someone hosting a Lemmy server is impossible to equalize, and the privacy of hosts may need to be front and center.
- It is very reminiscent of the trolls in the earlier web.

big F in chat for those of you dealing with this. my #1 fear about setting upand instance.
- It impacts everyone when this shit happens. It takes time for mods/admins to take down. And you can’t unsee it.
  I hope nobody else has the misfortune of stumbling on that shit
  
  There have been studies which found playing tetris for an hour or two after seeing something traumatic can prevent it taking root in our longterm memory.
  I tried it once after accidentally clicking a link on reddit that turned out to be gore, I can't remember exactly what it was now (about 9 months later) so it must have worked
  
  Yeah you really can’t. I’m pretty desensitized from earlier internet with death and other shock gore content but had managed to avoid CSAM until today. It was a lot worse than I expected, felt my heart drop. Worse, my app autoplays gifs in thumbnail so it kept going while I was reporting it.
  I’ve mostly forgotten and it wasn’t on my mind until I saw this thread (happened less than 24hr ago) but even the slightest reminder is oddly upsetting. Wish I’d thought of the Tetris thing.

Why do these deranged fucks do this
- You really think the trolls would pass up on this golden opportunity?
  
  This isn't trolling, this is just disgusting crime.
  
  Archive.org gets crapped up this way, too. Repulsive people posting repulsive stuff, "troll" is too kind a word.

I literally am going to give up social media in general if this doesn't stop
Seen it last night late around 3am shit made me sick I honestly almost cried but I just closed the app and tried not to think about it
Whatever the goal is it's a stark reminder that there is monsters creeping in the shadows every where you go

Likely scum moves from reddit patriots to destroy or weaken the fediverse.
I remember when Murdoch hired that Israeli tech company in Haifa to find weaknesses is TV smart cards and then leaked it to destroy their market by flooding counterfit smart cards.
They are getting desperate along with those DDOS attacks.
- Could be, but more likely it's just the result of having self hosted services, you have individuals exposing their own small servers to the wilderness of internet.
  These trols also try constantly to post their crap to mainstream social media but they have it more difficult there. My guess is that they noticed lemmy is getting a big traction and has very poor media content control. Easy target.
  Moderating media content is a difficult task and for sure centralized social media have better filters and actual humans in place to review content. Sadly, only big tech companies can pay for such infrastructure to moderate media content.
  I don't see an easy way for federated servers to cope with this.
  
  Yeah exactly. This is the main reason I decided not to attempt to self host a Lemmy instance. No way am I going to let anyone outside of my control have the ability to place a file of their choosing on my hardware. Big nope for me.

I got lucky. I am not subscribed to this community, and I am the only person on my instance. But what if I was subscribed and hadn't seen this post? This is too much responsibility for me.
I just shut down my instance until we can disable cached images. If that never happens, then I'm not bringing it back up.
Shout-out to https://github.com/wescode/lemmy_migrate. I moved my subscriptions over in a minute or two, and now, other than not having my post history, it's exactly the same.
- It is coming, don't worry.
  
  Do we have a PR or issue tracking that? That's a pretty major item I'm looking forward to

Yup. Nope.
Pictrs is just completely disabled now. Rather be safe, then sorry.
- Is this why I couldn't upload a meme to the Lemmy World servers earlier today?
  Fuck...
- Yeah... Just wow. I disabled pictrs and deleted all its images, which also means all my community images/uploaded images are gone, and it's more of a hassle to see other people's images, but in the end I think it's worth it.
  Through caching every image pictrs was also taking up a massive amount of space on my Pi, which I also use for Nextcloud. So that's another plus!
  
  Note, apparently, lemmy will get pretty pissy if pictrs isn't working..... and the "primary" lemmy GUI will straight-up stop working.
  Although, https://old.lemmyonline.com/ will still work.
  And- I am with you. My pictrs storage, has ended up taking up quite a bit of room.
  
  There has to be a more elegant way of dealing with this in the future, like de-coupling between Lemmy-account hosting (which effectively means acitivypub-fediverse account) and Lemmy-communities hosting.
- Is disabling Pictrs as simple as stopping the Docker container?
  
  Yup.
  I sent a step further, and commented out the pictrs related configuration from the lemmy.hjson too.
- Does that disable image saving and processing for one's instance?
  
  Yup.
  So far, mostly everything appears to work still. But, trying to upload an image, just throws an error.
  SyntaxError: Unexpected token 'R', "Request er"... is not valid JSON
  I don't see a way to actually "gracefully" disable it, but, this works.
  Edit- don't just stop pictrs.
  Lemmy gets very pissy... and b reaks.

How desperate to destroy Lemmy must you be to spam CSAM on communities and potentially get innocent people into trouble?
- Maybe you’re a dev on the Reddit team and own a lot of shares for what you know is about to go public?

What's CSAM?
- Child sexual abuse material - underage porn. For obvious reasons, you don't want this to be something you're hosting automatically out of your basement server.
  
  That's what I thought. Back in my days it was called CP.
- Child sexual abuse material
- Just google it.
  Edit: Not Safe For Life
  
  I kind of suspected it's better not to google it at work.

i’d love for a good tech journalist to look into how and why this is happening and do a full write-up on it. come on ars, verge, vice

Self hoster here, im nuking all of pictrs. People are sick. Luckily I did not see anything, however I was subscribed to the community.
Did a shred on my entire pictrs volume (all images ever):
sudo find /srv/lemmy/example.com/volumes/pictrs -type f -exec shred {} \;
Removed the pictrs config in lemmy.hjson
removed pictrs container from docker compose
Anything else I should to protect my instance, besides shutting down completely?

To be clear, if no one on a given instance sub to that particular /c, the content won't federate to said instance, correct?
- At this point, the community is clean. So unless more is posted, then you should be good. If someone searched for the community and caused a preview to load while the content was active though, then it could be an issue.
  
  Cool. Thanks. I cleaned up anything from the past 2 days, to be safe, and blocked that community.

I went ahead and just deleted my entire pictrs cache and will definitely disable caching other servers images when it becomes available.
- Anyone know if this work is tracked anywhere? I’m suddenly really suspicious of continuing to run my own instance.
  
  https://github.com/LemmyNet/lemmy/pull/3897
  It does say "thumbnails" but as far as I know, Lemmy (or pictrs) makes a copy of the full image too. I don't know if this PR includes full images.
  
  https://github.com/LemmyNet/lemmy/pull/3897

blocked lemmyshitpost some time age because it is trash anyway

I was looking into self hosting. What can I do to avoid dealing with this? Can I not cache images? Would I get in legal trouble for being federated with an instance being spammed?
- Someone else here mentioned not running "pictrs” at all
- Refer to this comment chain from this same thread https://poptalk.scrubbles.tech/comment/577589

I checked and there shouldn't be any images stored on the server when running lemmy 1.18.4. The post was made in high emotional distress and shouldn't be taken at a face value. If the posts are bothering you I advise purging the posts in question. (I have already done that)
- I'm on 1.18.4, once I deleted the most recent images, the former CSAM posts(among others) became broken images. So yes, it was pulling from local disk cache. Then I took care of the posts themselves after the content was invalidated.
- How did you check this? From my understanding, images from external servers are copied (and transcoded) over locally. At least in my server (running 0.18.4), they do.
  
  There is a possibility that my instance is buggy and it isn't caching images even though it should.
  
  It depends on how the image posted, the thumbnails might get federated. If the image is used in a post/comment body, usually the thumbnails are not federated.
  
  I shut down the pictrs or whatever docker container on my instance so all I host is containers and the database. All the images that I see on my instance are external links. I can check by just looking at the rendered HTML.
  https://files.catbox.moe/vm4yxl.png

What is CSAM?
- Child sexual abuse material, a.k.a. child pornography.
- https://en.wikipedia.org/wiki/Child_pornography?wprov=sfti1

If the source deletes the post. Won’t that remove it from all the instances ?

I'm not subscribed to that community, but I guess I'm glad Pictrs doesn't work for me, since I am using the Yunohost version of Lemmy. The creators of the Yunohost package couldn't get it to work. I haven't really missed it honestly.
- Can you run lemmy without pictrs? What behavior is different?
  
  It just means that you can't upload pictures, including banners or avatars. However, when I want to create an image post, I just make the post on Pixelfed and then mention the Lemmy community I want to post to at the bottom of the post body. Supposedly there's a way to reference a remote image for a banner or an avatar, but I haven't figured that out yet.

Likely Spez’s personal jailbait collection

Could someone please ELI5 that script. I'm all for keeping things clean, but old enough to remember the days of console based trolling.
- Looks fairly sane, finds every file in the given directory that was created in the last 24 hours and deletes them. Personally if you are dealing with CSAM I'd be using shred instead of just rm
  
  shred -u will shred and then remove.
- sudo
  As root
  find /srv/lemmy/example.com/volumes/pictrs/files
  Find files in /srv/lemmy... that:
  -type f
  Are plain files (not directories, symlinks, etc; includes images)
  -ctime -1
  And were created within an amount of time (probably last day, haven't used this flag in a while)
  -exec rm {} \\;
  For each matching file found execute rm on it (delete it).
  
  I don't think rm is gonna cut it if you have that shit on disk
- sudo find /srv/lemmy/example.com/volumes/pictrs/files -type f -ctime -1 -exec rm {} \;
  sudo: run as root
  find /srv/lemmy/example.com/volumes/pictrs/files -type f: find files (f) in directory
  -ctime -1: which have been created in the last day
  -exec rm {} execute the command rm (remove) on each of them

I am using the Lemmy easy deploy would this command works?
- You'll need to find where the actual container files are being stored. I'm unfortunately not familiar with Lemmy Easy Deploy, but you should have a folder that has some files/folders like docker-compose.yml, volumes, lemmy.hjson.
  The important one is the volumes/pictrs/files folder, take the full path of that folder and replace it with the /srv/lemmy/example.com... path from the original post, and then that command should work.

There was a weird JSON error I was getting in the last few minutes. I'm not sure if this is at all related.

Couldn't this be stopped with automatic filtering of bad content? There are open source tools and libraries that do this already
- That's what we're pushing the lemmy devs to do. Honestly even if they want to use proprietary tools for this instance I'm okay, I'll happily go register an Azure account and plop an API key into the UI so it can start scanning. Lemmy should have the guardrails to prevent this from ever hitting our servers.
  In the meantime, services like cloudflare will handle the recognizing and blocking access to images like that, but the problem still comes down to the federation of images. Most small hosters do not want the risk of hosting images from the whole of the internet, and it sounds like there is code in the works to disable that. Larger hosters who allow open registrations can do what they please and host what they please, but for us individual hosters we really need tools to block this.
  
  Proprietary software isnt necessary there are plenty of project that detect scam

Locking the thread. Information relevant to self-hosters has already been shared. Too many reports of off-topic comments to leave this open.

Is it possible to prune pictrs by community name?
- Not really. You could technically locate the images and determine precisely which ones they are from their filenames, but that means you actually have to view the images long enough to pull the URL. I had no desire to view them for even a moment, and just universally removed them.
  As mentioned in my edit above though, ensure you are in compliance with local regulations when dealing with the material in case you have to do any preservation for law enforcement or something.
  
  There is a purge community option available for admins but that did nothing.

That's it, I'm defederating from lemmy.world. the admins let their users make death threats against users of other instances on top of this.

156 comments