3w ago

My Lemmy instance where the top 80% most generic posts are deleted

I made a Lemmy instance with a custom algorithm that keeps only the top 20% most unique (=interesting?) posts. It does this by calculating a similarity score between every post on my instance and all posts that came before it. The top 80% of posts with the highest self-similarity get removed instantly.

The idea would be that this allows me to cut through the noise that's running through the communities, similar to how xkcd-signal attempted to do 20 years ago.

The instance is mostly meant for reading, not posting. So it has a very open federation policy (for now).

If anything, this is experimental. So please let me know what you think! You can see the type of stuff that gets removed in the modlog (https://lemmy.coffee/modlog).

32 comments

If there are a bunch of posts on a particular topic, shouldn’t it keep at least one of them? Otherwise it would tend to completely filter out the most significant or interesting topics.
- Yeah, that's one of the potential issues that I'm currently looking out for. So far the main thing I can tell is that memes get removed like crazy (https://lemmy.coffee/c/memes@lemmy.world?dataType=Post&sort=New) and the posts on the homepage are generally much less meme-intensive when compared to instances like lemmy.world or lemmy.ml.

Interesting. One of my instance's guiding philosophies is "Quality over Quantity". I've taken different steps toward achieving that (defederate from the Reddit repost instances, disallow pretty much all content bots, manually/locally mod duplicate posts, etc).
Do you plan to publish your algorithm/filter? Would be interested in seeing if it could be tuned and possibly reduce some of the workload for me.
- Do you plan to publish your algorithm/filter?
  In an ideal world sure. But I'd have to think about that some more, because in principle I don't want people to game it :)
  
  Lemmy's license is AGPL, so you would need to at least publish changes to Lemmy itself 😉
  (I don't know if e.g. the code for the algorithm is separate, in order to have a closed source algorithm with an open source Lemmy fork)

Oh, kinda like r9k!
- That's the idea, yes.

I was curious what would happen to the ratio of political posts, specifically Trump/Elon, to other communities, but it feels >= the amount as All on lemmy.world.
None of my superb owls look to have made it through, but I didn't see them removed in the mod log. We're a pretty large community, so I'd have thought some would have gotten through. I don't recall if I saw stuff from any of the animal comms.
- I'll take a look at Superbowl@lemmy.world!
  The political posts thing is interesting. So far, mainly news posts with text descriptions remain on the front page (which I personally prefer over the memes, but that's just my preference). I am having a hard time deciding if I actually find them interesting though.. ^^ I think I actually prefer it over my reddit homepage, there tends to be much of the same, usually.
  For comparison, see: https://imgur.com/a/xbzMXmQ
  
  I do look to have made it to your feed now! It deleted my 3 posts that were photos with short blurbs and kept my news article that is mostly text.
  We rarely get the meme type posts, but if your setup is looking at anything like text:photo ratio, that could be doing it.
  I have most meme communities block myself, so I understand how overtaking they can be to a feed. 😁
  I'm interested in what you're doing because I am interested in news and politics, but it would be nice to get rid of most of the garbage US news and the dupe posts, and let a better variety of news come in. I've gone to keyword blocking, but while I have "Trump" blocked, I'd still like to see things from other countries about their responses to his actions., for example.

I've been through a few pages, could only find this post about Lemmy apps from !buyfromeu@feddit.org ; https://lemmy.coffee/post/6860?scrollToComments=true , with a single comment (mine).
No posts from !buyeuropean@feddit.uk, while it is much more active. Do you know why?
On the other hand, !world@lemmy.world , !news@lemmy.world and !europe@feddit.org seem to be doing fine
- I added the larger communities before starting to remove posts, so there may be historical posts still hanging around. Maybe everything from BuyFromEU was deleted?
  You can see the kind of stuff that stays best via homepage ALL > Top last N hours
  
  Like this, for example: https://lemmy.coffee/?dataType=Post&listingType=All&sort=TopSixHour
  
  Seems like !buyeuropean@feddit.uk votes and comments aren't federated to your instance: https://lemmy.coffee/c/buyeuropean@feddit.uk

Cool Idea!, Created an account to check it out.
- Cool. Let me know what you think!

This is very interesting. I've been thinking of how a similar but different system could be implemented. The front page of any instance always seems to have two to four posts by the same person, and I've been following a rule that if I notice it I block them to remove their clutter of posts from my feed. Unfortunately, most of these accounts are brand new ones posting memes, so it feels like for every two I swat down, four more take their place.
I was looking for an instance that allowed me to mute all new accounts, or less likely an instance that filters out posts by the same person, but it doesn't seem such a thing has been created.
This isn't the same that I wanted, and if I'm not mistaken it only applies to posts inside this instance, since the algorithm is removing them? Or does it filter out all similar posts across the fediverse feed? Still, it's close in concept.
- PieFed places an icon next to the username to help highlight such aspects of a person's account. The one for "new account, less than 2 weeks old" is very useful. Others need tweaking such as "potential unregistered bot that posts far more often than comments", and "contentious user with far more downvotes than upvotes". I find it useful not to block people but to simply scroll past or to tailor my response to knowing that info.
  In addition to PieFed, there are some Lemmy apps that will do this too, although I am not sure which ones (perhaps check out Sync and Connect) - the trick here is ofc to auto apply it to all accounts as you read through your feed.
  
  I would still argue that thunder is the best app for lemmy
  https://github.com/thunder-app/thunder
- From what I can tell, the larger instances have frontpages without much user duplication. They have a LOT more memes than I do tho :)
  https://imgur.com/a/xbzMXmQ

32 comments