Selfhosted @lemmy.world bmygsbvur @lemmy.ca 2y ago

PSA: You can upload images to a Lemmy instance without anyone knowing

EDIT

TO EVERYONE ASKING TO OPEN AN ISSUE ON GITHUB, IT HAS BEEN OPEN SINCE JULY 6: https://github.com/LemmyNet/lemmy/issues/3504

June 24 - https://github.com/LemmyNet/lemmy/issues/3236

TO EVERYONE SAYING THAT THIS IS NOT A CONCERN: Everybody has different laws in their countries (in other words, not everyone is American), and whether or not an admin is liable for such content residing in their servers without their knowledge, don't you think it's still an issue anyway? Are you not bothered by the fact that somebody could be sharing illegal images from your server without you ever knowing? Is that okay with you? OR are you only saying this because you're NOT an admin? Different admins have already responded in the comments and have suggested ways to solve the problem because they are genuinely concerned about this problem as much as I am. Thank you to all the hard working admins. I appreciate and love you all.

ORIGINAL POST

cross-posted from: https://lemmy.ca/post/4273025

You can upload images to a Lemmy instance without anyone knowing that the image is there if the admins are not regularly checking their pictrs database.

To do this, you create a post on any Lemmy instance, upload an image, and never click the "Create" button. The post is never created but the image is uploaded. Because the post isn't created, nobody knows that the image is uploaded.

You can also go to any post, upload a picture in the comment, copy the URL and never post the comment. You can also upload an image as your avatar or banner and just close the tab. The image will still reside in the server.

You can (possibly) do the same with community icons and banners.

Why does this matter?

Because anyone can upload illegal images without the admin knowing and the admin will be liable for it. With everything that has been going on lately, I wanted to remind all of you about this. Don't think that disabling cache is enough. Bad actors can secretly stash illegal images on your Lemmy instance if you aren't checking!

These bad actors can then share these links around and you would never know! They can report it to the FBI and if you haven't taken it down (because you did not know) for a certain period, say goodbye to your instance and see you in court.

Only your backend admins who have access to the database (or object storage or whatever) can check this, meaning non-backend admins and moderators WILL NOT BE ABLE TO MONITOR THESE, and regular users WILL NOT BE ABLE TO REPORT THESE.

Aren't these images deleted if they aren't used for the post/comment/banner/avatar/icon?

NOPE! The image actually stays uploaded! Lemmy doesn't check if the images are used! Try it out yourself. Just make sure to copy the link by copying the link text or copying it by clicking the image then "copy image link".

How come this hasn't been addressed before?

I don't know. I am fairly certain that this has been brought up before. Nobody paid attention but I'm bringing it up again after all the shit that happened in the past week. I can't even find it on the GitHub issue tracker.

I'm an instance administrator, what the fuck do I do?

Check your pictrs images (good luck) or nuke it. Disable pictrs, restrict sign ups, or watch your database like a hawk. You can also delete your instance.

Good luck.

118

118 comments

seems like the solution to this should be to automatically remove images that haven't been posted, after like 3 minutes
- Or make it like 1hr and don't let the user know the url of the uploaded image until they post it, that way it wouldn't be able to be shared or reported.
  
  It's difficult to display an image without the client knowing the URL, but it would be possible to use a temporary URL that only works for that signed-in user.
- This is one way to solve it.
- 3 minutes is way too short. You could upload and it be gone by the time you post.
- Or you set a flag that says something like “incomplete image” and then only once user completes whatever operation by hitting “submit” do you then set it to complete.
  
  And maybe while an image is not yet complete, only the uploading user can view the image.
This is not unique to Lemmy. You can do the same on Slack, Discord, Teams, GitHub, ... Finding unused resources isn't trivial, and you're usually better off ignoring the noise.

If you upload illegal content somewhere, and then tell the FBI about it, being the only person knowing the URL, let me know how that turns out.
- Imagine if the image link is shared to other people and you aren't aware of it. You think that's acceptable?
  
  I do not. As far as I'm aware, this is usually countered through a proper way to follow through on reports. If you host user-generated content, have an abuse contact who will instantly act on reports, delete reported content, and report whatever metadata came along with the upload to the authorities if necessary.
  
  The bookkeeping code for keeping track of unused uploads has a cost attributed to it. I claim that most providers are not willing to pay that cost proactively, and prefer to act on reports.
  
  I can only extrapolate from my own experience though. No idea how the industry at large really handles or reasons about this.
I'm usually pretty relaxed when it comes to disclosure of vulnerabilities but this is the kind of issues where I think it would have been better to privately report the issue to the Lemmy dev and wait ( a long time probably) for it to be fixed before disclosing.

Especially since currently there is multiple people abusing the image hosting feature.

Not a big deal, but sometimes it is actually a better practice to give an opportunity to the dev to fix something before forcing them to do so in a hurry.
- I've mentioned this before to a similar reply. But I'll say it again: this was already publicly known months ago. People just forgot about it because they didn't think it was a big deal. Now that they realize CSAM is a real issue, I made this post to remind everyone about it again. Bad actors already know about this and really, it isn't hard to figure out how this work.
  
  Then why didn't you contact the devs or opened a bug report on GitHub?
- Eh... Better make it public so you don't have people taking a chance with hosting CSAM!
- Nah. Where’s the drama and FUD in behaving like adults? Much better to make a brand new account and spam moral panic all over the fediverse. /smh
Or just disable image uploads completely. We got by on Reddit without any built-in image hosting functionality for over a decade, so Lemmy should be fine without it as well - especially considering that we don't really have many image-heavy communities, besides the NSFW instances. I mean, storage costs money you know, and with Lemmy being run by volunteers, it makes even more sense to get rid of image hosting to save costs.
- I don't have the pictrs container running on my instance.
- 💯 I really hope this is in the next update
Note, my tools is the only solution that exists (currently) for this in regards to csam

https://github.com/db0/lemmy-safety
- Not hosting images is a far better solution, and also exists.
  
  What's the best away for be to disable the pictrs directory? Is there a setting to flip in my instance?
- Appreciate your work.
- Your contributions to the Lemmy ecosystem are much appreciated 🙏🏼
- I hate how everything is a double edged sword, because this is now also the perfect tool for making sure your CSAM doesn't trip the filter. Also, it uses CLIP so a simple obfuscation overlay would render it useless.
  
  can you elaborate on what an obfuscation overlay is?
This is how it works. Since pictrs and Lemmy are two completely different applications (they even run in two different containers with two different databases) they do not communicate and tracking what images belong to what post or comment simply isn't possible in the current state I guess.

How come this hasn’t been addressed before?

This is how the Fediverse works. There is so much bad practices, so much haphazardly implemented functionality and so much bad API documentation all over the place that I wonder why nothing has extremely exploded so far. We don't even have proper data protection and everything is replicated to everywhere causing a shitload of legal issues all over the workd but no-one seems to care so far.
- Sounds like the Internet Protocol I grew up with 😍
- This isn’t unique to Lemmy or haphazard coding. It’s a common technique to get pictures into Github READMEs this way. You’d create a PR, upload an image, copy the link, delete the PR, and then paste the link elsewhere on Github for use.
- The difference between the Fediverse and a closed system like reddit is that it's open and we're privy to haphazardly implemented functionality and bad API documentation.
  
  I work on big closed source web apps for a living; they're just as haphazard and badly documented, it's just all closed.
Why does Lemmy even ship its own image host? There are plenty of places to upload images you want to post that are already good at hosting images, arguably better than pictrs is for some applications. Running your own opens up whole categories of new problems like this that are inessential to running a federated link aggregator. People selfhost Lemmy and turn around and dump the images for "their" image host in S3 anyway.

We should all get out of the image hosting business unless we really want to be there.
- Convenience for end-users and avoiding link rot is probably one of the reasons.
  
  and avoiding link rot
  
  Lemmy seems built to destroy information, rot links. Unlike Reddit has been for 15 years, when a person deletes their account Lemmy removes all posts and comments, creating a black hole.
  
  Not only are the comments disappeared from the person who deleted their account, all the comments made by other users disappear on those posts and comments.
  
  Right now, a single user just deleting one comment results in the entire branch of comment replies to just disappear.
  
  Installing an instance was done pretty quickly... over 1000 new instances went online in June because of the Reddit API change. But once that instance goes offline, all the communities hosted there are orphaned and no cleanup code really exists to salvage any of it - because the whole system was built around deleting comments and posts - and deleting an instance is pretty much a purging of everything they ever created in the minds of the designers.
  
  Seems to not be paying off though; having whole communities and instances close is pretty inconvenient.
- S3 is expensive, while if you use a third party like img.bb or imgur, you never know when they will close, accidentally lose your data, or decide to delete it.
I can’t be the only one getting bored with the 8-hr-old accounts spreading FUD.

If you have a legitimate concern, post it from your proper account. Otherwise it looks like you’re just trolling for Spez. It’s pathetic, really.
- Additionally this isn't the community where this needs to be addressed. Either contact the admins or open an issue on GitHub.
  
  Issue has been opened two months ago: https://github.com/LemmyNet/lemmy/issues/3236 https://github.com/LemmyNet/lemmy/issues/3504
- You're not concerned because you're not an admin. Of course you only bothered to check my account profile and not the actual post. If the issue I stated above doesn't bother you, then it's only a matter of time until people start seriously abusing it. Or who knows, somebody already is and we just aren't aware of it yet.
  
  If you’re not just trolling, what is your main account?
  
  I’m not concerned because people smarter than us have said it isn’t a concern. So long as they preserve their safe harbor shield, instance admins are not generally liable for content posted by users.
the admin will be liable for it.

...

These bad actors can then share these links around and you would never know! They can report it to the FBI and if you haven’t taken it down (because you did not know) for a certain period, say goodbye to your instance and see you in court.

In most jurisdictions this is not now it would work. Even a less tech savvy investigator would figure out that it was an online community not obviously affiliated with CSAM, and focus on alerting you and getting the content removed.

There's this misunderstanding that CSAM is some sort of instant go-to-prison situation, but it really does depend on context. It's generally not so easy to just plant illegal files and tip off the FBI, because the FBI is strategic enough not to be weaponized like that. Keep an eye on your abuse and admin email inboxes, and take action as soon as you see something, and nobody is going to shut you down or drag you to court.
- Doesn't change the fact that this is an issue that needs to be resolved.
  
  It's not. Image hosting sites have existed for decades. Websites are not liable unless they have actual knowledge of illegal content and ignore takedown requests. Stop fearmongering.
  
  Never said otherwise, I just want to make sure we're not scaring people away from Lemmy administration and moderation, as if they were risking going to prison as a child sex offender or something.
In theory also possible to just be a nuisance by filling out the instances available space? That sounds like it's gonna get fixed one way or another.
- Yes - that's possible.
This is just like how someone could put printed CSAM behind a bush in my yard or something and some authorities could decide to hold me responsible.
- To take your analogy, it could be someone hosts a collection of material in your yard and invites all the pedos to use your yard to see and share other material.
  
  These are interesting thought experiments.
  
  If I live in say Oregon, and own 20 acres in Montana. Am I responsible for an hourly or daily sweep of my distant property to ensure no CSAM exists lest I be held responsible? Would I need to hire guards to ensure nobody uses a hole in a tree on my property to stash CSAM otherwise be responsible?
  
  IMHO, it is or should be more what's reasonable. Obviously hourly sweeps of acres or property is ridiculous but if you run a magazine stand then it should be have some processes to ensure what is sold is legal. Similarly, so long as a lemmy server operator is running by current best practices and isn't grossly negligent it seems like a bad idea to hold them responsible. As time goes on, best practices will evolve better methods of keeping shit out.
  
  TBH, my worry is that the owners of Reddit could pay some bad actors to post CSAM to lemmy servers to take out their competition.
- So you're telling me you're NOT bothered if CSAM was sitting on your server and shared with others without your knowledge? Do you think all countries have the same laws? You don't think any of this is an issue?
  
  You're a regarded one. I won't bother to answer such dumb questions.
Yeah, this is a big issue. I know Lemmy blew up a bit before it was truly ready for prime time but I hope this cleans up.
In the USA, admins being liable is not really true
- Most admins aren't in the USA. But that's not really the issue here is it?
- Are individuals granted the same 230 protections as organizations when it comes to self-hosting an instance? I doubt people are forming non-profits for their self hosting endeavors
Did you submit an issue?
- I'm not on GitHub. Someone else can submit it this and I'm very sure the Lemmy devs are aware. They just have different priorities.
Funny, I couldn't even get pict-rs working on my instance. I don't need it, either. I just upload to an FTP server when I need to share something.
- Good. It's better that way until the issues are resolved.
There really needs to be an option for instances to upload images to imgur using their API.

imgur has been hosting images for years, and has the resources and experience to deal with stuff like CSAM.

It shouldn’t be the default/only option that hosting an instance means having to open the floodgates for anyone to upload images to their servers.

From a liability standpoint alone, it’s an absurd thing to just expect every instance to accept.
- Yes. This is a great alternative solution.
I'm not using lemmy. But I was thinking of making a process to periodically scan the object storage and check for a reference to a post, comment etc and if none are found delete it. In most cases the images are deleted but sometimes they don't seem to be.

Probably lemmy could have a similar process created.
It would not be difficult to use SQL to delete any images that are not associated with a post or active as an avatar etc. So, set that to be run periodically and it would solve this problem.
- Checking every single image ID against all stored text blobs is not trivial. Most platforms don't do this. It's cheaper to just ignore the unused images.
  
  Yeah, this is only if what OP was saying was a real legal threat, which I don’t think it is.
- I'm not knowledgeable with SQL. If you know or if anyone knows how to fix it with a script or built into Lemmy, please share.
  
  I haven't worked with Lemmy, but I certainly could craft a script to do that if I was familiar with the database structure. Perhaps I'll try installing it and running an instance. In the meantime, surely there's someone with an instance and SQL skills who could figure that out.
- Isn't it more likely that paths are used to reference resources like images rather than a db fk?
  
  Not familiar with Lemmy specifically, but usually in an app like this, while of course the files are stored on a filesystem, IDs and metadata are stored in the DB and associated with each other through relations. It seems in this case one way to express it would be 'don't delete every image that is associated with a valid post or in-use avatar, but delete everything else'.
  
  Take this random image for instance: https://lemmy.world/pictrs/image/ede63269-7b8a-42a4-a1fa-145beea682cb.jpeg
  associated with this post: https://lemmy.world/post/4130981
  
  Highly likely the way it works is there is an entry for post 4130981 that says it uses ede63269-7b8a-42a4-a1fa-145beea682cb, or an image table with a relation to the post table where an image entry (with whatever ID) that is ede63269-7b8a-42a4-a1fa-145beea682cb says it is related to post 4130981. Whatever the specifics, it would be possible.
So many weird issues... Thanks for posting this.
Yes, an app designed to receive user generated content will allow users to upload arbitrary content. This is not really an issue unique to Lemmy, and can be fixed by banning abusive users
- Except admins need to go through the database to catch it. Mods can't catch it. Admins without access to the database or object storage can't catch it.
- As far as I know, people don’t homebrew Reddit.
  
  Sites like reddit could have about the same problem. You can post to some obscure profile and use reddit as image hosting and it might take quite some time before anyone notices. This is a little worse though because it wouldn't even be listed on a profile page.
Just my two cents, but I feel it's quite irresponsible to post a "how to exploit this platform" guide ON the platform.
- This has been known forever. Any bad actor already knows about this. There's no reason to hide this. I am reminding people so solutions can be solved sooner. I will keep reminding until the problem is solved.
  
  Yeah, it's been acknowledged and solutions have been discussed as future Todos
  
  https://github.com/LemmyNet/lemmy/pull/1809#discussion_r889164824
  
  https://github.com/LemmyNet/lemmy/issues/3504
  
  I imagine it should be prioritized though, along with some of the other malicious upload issue mitigation bugs.
  
  Meh. I main 4chan. All sorts of shit get uploaded on 4chan, yet it still exists. I'm not saying nothing should be done, but no need to panic. Quietly delete the images periodically. In terms of what users can do, I suggest a report system where after a certain number of similar reports, the media gets auto pulled for moderation.

118 comments