Skip Navigation
Technology @beehaw.org
trashhalo @beehaw.org

Stanford researchers find Mastodon has a massive child abuse material problem

Mastodon, an alternative social network to Twitter, has a serious problem with child sexual abuse material according to researchers from Stanford University. In just two days, researchers found over 100 instances of known CSAM across over 325,000 posts on Mastodon. The researchers found hundreds of posts containing CSAM related hashtags and links pointing to CSAM trading and grooming of minors. One Mastodon server was even taken down for a period of time due to CSAM being posted. The researchers suggest that decentralized networks like Mastodon need to implement more robust moderation tools and reporting mechanisms to address the prevalence of CSAM.

124 comments

  • While the study itself is a good read and I agree with the conclusions—Mastodon, and decentralized social media need better moderation tools—it’s hard to not read the Verge headline as misleading. One of the study authors gives more context here https://hachyderm.io/@det/110769470058276368. Basically most of the hits came from a large Japanese instance that no one federates with; the author even calls out that the blunt instrument most Mastodon admins use is to blanket defederate with instances hosted in Japan due to their more lax (than the US) laws around CSAM. But the headline seems to imply that there’s a giant seedy underbelly to places like mastodon.social[1] that are rife with abuse material. I suppose that’s a marketing problem of federated software in general.

    1. There is a seedy underbelly of mainstream Mastodon instances, but it’s mostly people telling you how you’re supposed to use Mastodon if you previously used Twitter.

    • In my opinion the biggest issue the author points out is that cached materials are sometimes retained even after moderator action. Which honestly just sounds like a straight up bug more than anything. Though if I were running an instance, the feds showing up at my door with a warrant because I've been accidentally distributing CSAM would be my nightmare scenario. And of course jurisdiction plays a part, too: an American user on a Canadian server might see drawn depictions of sexualized minors, think "weird but not illegal," and now the Canadian admin has content that's illegal in Canada on their Canadian server and has no idea.

      IMO I think the best solution to this is something similar to what Renaud Chaput (Mastodon's resident infra boffin) described in his recent blog post. Effectively, give admins a way to hand this off to pluggable third-party services. Admins that are worried about this sort of thing can then have some degree of safety via e.g. PhotoDNA, whereas others can take on additional risk and preserve additional privacy.

      All that said: yeah the headline makes it sound like .social is some 8chan-esque hellhole, whereas in reality my feed is 99% German programmers sharing milquetoast political takes.

    • The person outright rejects defederation as a solution when it IS the solution, if an instance is in favor of this kind of thing you don't want to federate with them, period.

      I also find worrying the amount of calls for a "Fediverse police" in that thread, scanning every image that gets uploaded to your instance with a 3rd party tool is an issue too, on one side you definitely don't want this kinda shit to even touch your servers and on the other you don't want anybody dictating that, say, anti-union or similar memes are marked, denounced and the person who made them marked, targeted and receiving a nice Pinkerton visit.

      This is a complicated problem.

      Edit: I see somebody suggested checking the observations against the common and well used Mastodon blocklists, to see if the shit is contained on defederated instances, and the author said this was something they wanted to check, so i hope there's a followup

  • I'm not fully sure about the logic and perhaps hinted conclusions here. The internet itself is a network with major CSAM problems (so maybe we shouldn't use it?).

    • It doesn't help to bring whataboutism into this discussion. This is a known problem with the open nature of federation. So is bigotry and hate speech. To address these problems, it's important to first acknowledge that they exist.

      Also, since fed is still in the early stages, now is the time to experiment with mechanisms to control them. Saying that the problem is innate to networks is only sweeping it under the rug. At some point there will be a watershed event that'll force these conversations anyway.

      The challenge is in moderating such content without being ham-fisted. I must admit I have absolutely no idea how, this is just my read of the situation.

      • @mudeth @pglpm you really don't beyond our current tools and reporting to authorities.

        This is not a single monolithic platform, it's like attributing the bad behavior of some websites to HTTP.

        Our existing moderation tools are already remarkably robust and defederating is absolutely how this is approached. If a server shares content that's illegal in your country (or otherwise just objectionable) and they have no interest in self-moderating, you stop federating with them.

        Moderation is not about stamping out the existence of these things, it's about protecting your users from them.

        If they're not willing to take action against this material on their servers, then the only thing further that can be done is reporting it to the authorities or the court of public opinion.

      • Maybe my comment wasn't clear or you misread it. It wasn't meant to be sarcastic. Obviously there's a problem and we want (not just need) to do something about it. But it's also important to be careful about how the problem is presented - and manipulated - and about how fingers are pointed. One can't point a finger at "Mastodon" the same way one could point it at "Twitter". Doing so has some similarities to pointing a finger at the http protocol.

        Edit: see for instance the comment by @while1malloc0@beehaw.org to this post.

    • This is exactly what I thought. The story here is that the human race has a massive child abuse material problem.

      • The problem is even bigger: some places (ejem Reddit) you will get deplatformed for explaining and documenting why there is a problem.

        (even here, I'll censor myself, and hopefully restrict to content not too hard to moderate)

    • The internet itself is a network with major CSAM problems

      Is it, though?

      Over the last year, I've seen several reports on TV of IRL group abuse of children, by other children... which left everyone scratching their heads as to what to do since none of the perpetrators are legally imputable.

      During that same time, I've seen exactly 0 (zero) instances of CSAM on the Internet.

      Sounds to me like IRL has a major CSAM, and general sex abuse, problem.

  • So instances that are actually supporting CSAM material can and should be dealt with by law enforcement. That much is simple (and I'm surprised it hasn't been done with certain ... instances, to be honest). But I think the apparently less clearly solved issues have known and working solutions that apply to other parts of the web as well. No content moderation is perfect, but in general, if admins are acting in good faith, I don't think there should be too much of a problem:

    • For when federation inadvertently spreads some of the material through to other instances' databases: Isn't this the same situation as when ISP's used to cache web traffic to save on bandwidth costs? In that situation, too, browsed web pages would end up in the ISP's cache which could then harbour whatever material the user was looking at. As I recall, the ISP would just ban CSAM and other illegal material in their terms of service, and remove anyone reported as violating the rule, and that sufficed.
    • As for "bad" instances/users: It's impossible to block all instances and all users that might disseminate this material as you'd have to go to a "block everything, then allow known entities" rule which would break the Fediverse model. Again, users or site admins found to be acting in bad faith should be blocked and reported (either automatically or manually). Some may slip through the net, but as long as admins are seen to be doing the best they can, that should be enough.

    There seem to be concerns about "surveillance" of material on Mastodon, which strikes me as a bit odd. Mastodon isn't a private platform. People who want private messaging should use an E2EE messaging app like Signal, not a social networking platform like Mastodon (or Twitter, Threads etc.). Mastodon data is already public and is likely already being surveilled, and will be so regardless of what anyone involved with the network wants, because there's no access control on it anyway. Having Mastodon itself contain code to keep the network clean, even if it only applies to part of the network, just allows those Mastodon admins who are running that part of the code to take some of the responsibility on themselves for doing so, reducing the temptation for third parties to do it for them.

  • Mastodon.art doesn't.

    And the beauty of Mastodon is you can block an entire instance, as can your admin, when something awful is posted. Mastodon even has a hashtag they use as an alert for this kind of thing. (#Fediblock)

  • I don't trust stanford to not work on behalf of the CIA or other 3 alphabet orgs. They kind of turn a blind eye to CSA in churches but a federated media? This sounds like a smear job.

    • I really don't think the CIA cares to be honest...

    • Total tangent, but we kid ourselves if we think the fediverse is somehow censorship-immune in comparison to Reddit or Twitter.

      There are more moderators and administrators across all instances which can federate/defederate at will and can delete posts and propagate this deletion through the network. At the same time governments don’t need to negotiate with a large company, but only need to hint they could destroy one person’s livelihood to remove undesirable content from the network. And to avoid the Streisand effect instead of requesting to delete one specific piece of subversive content (which could backfire), just insinuate some illegal material (CSAM being the most obvious, but anything goes, really) has been found to force shut down or takeover of the whole instance.

      The same goes for big companies instead of governments: if a large corporation has launched their own Mastodon clone, the first thing they’d reasonably fund are smearpieces by “journalists” and/or “scientists” hinting at harm to befall server owners by continuing to host Mastodon instances.

      I personally hate, what crypto has become (if I wanted to destroy crypto, I’d have invented crypto bros as a psy op), but the fediverse isn’t really federated enough to be resistant to influence by corporations and governments and something blockchain adjacent could have been the solution. For example: if the server admin and their hoster is totally unable to decrypt whatever is stored on their own server and the network as a whole is distributing all the content probabilistically across every federated server, the network would only get stronger and more censorship resistant with each new instance. If the government is forcing you for any reason to take down your server your content is not gone but stored with all the other nodes. If you are able to retrieve your key, you could even move to a new instance and authenticate as your old instance (don’t forget: you are not “sending” BTC from one wallet to another, you are only telling as much nodes as sensible that BTC on the chain belongs to a new key now; the same would go for content. Take down one node with a “wallet” doesn’t change which wallet the BTC on the chain belongs to. I propose the same, just with content). If federation between instances would work in a comparable way as it is now, this would additionally increase the probability to root out bad faith actors trying to flood the whole network with illegal content, since their content would be stored on much less nodes in a pseudo-predictable way: as soon as each major instance would defederate, their content would not be stored on their nodes and unfederated third-party-nodes.

  • not surprised at all. this is a growing pain here too because this was previously a thing handled invisibly by platforms and federation makes it fall to individual sysadmins and whoever they have on staff. the tools for this stuff are, in general, not here yet--and as people have noted there are potential conflicts with some of the principles of federation introduced by those tools that can't be totally handwaved.

124 comments