Technology @lemmy.ml moe90 @feddit.nl 7mo ago

Reddit is now blocking major search engines and AI bots — except the ones that pay

www.theverge.com Reddit is now blocking major search engines and AI bots — except the ones that pay

Sorry, Bing users.

71 comments

The cycle continues:

Hey you guys can have everything for free

WTF this is expensive to provide, I think I’m gonna start taking advantage of you guys which someone will pay me to do

WTF where’s everyone going

WTF I’m still losing money and always have been

Screw you guys, screw everybody, I didn’t want y’all anyway

(fades into irrelevance, gets bought by someone and stripped for parts)

Idk it’s not as pithy as Cory Doctorow’s version I guess

Anyway we’re at step 5 at this point
- Yeah, Reddit is Digging its own grave.
  
  It's getting Fark'd
- honestly I'm not convinced step 6 is inevitable. I think enough people are okay with whatever reddit does.
  
  Enough consumers are okay with it, but the core geeks and nerds that created, curated, and moderated the content have jumped ship.
  
  The cruise line is still sailing and there are still drinks and snacks so nobody has noticed the staff have jumped ship. There's management, low level volunteers, and thousands of kids, moms, and dads.
  
  But sooner or later people are gonna get tired of snacks and flip their shit when management tells them the people who know how to make the steaks have just all fuckin ✨ inexplicably disappeared✨
- The key capitalistic trick is to time your step 2 just when you have a critical mass on your platform. Upper management has understood that our shitty paywall will remove x% of our users from our platform. But if (100-x)% of our users can pay $y annually, we can sustain our business model and make $z of profit each year. PR will take care of all the backlash but it's all calculated.
- I freaking wish..
Tangentially related- I fucking hate discord
- Discord is fine for chatting, voice, and iterating quickly on projects. I have no idea why people want to think it's a forum. That's ridiculous.
  
  Its pretty awful for all those things if you care about privacy or can't signup for an account
  
  We use it for our friend group, as we have pub nights, group meals, vacations etc. we also all do each other's cat care when we're out of town, so we have a channel devoted to pet photos etc. works well enough for us.
  
  Unpopular opinion: I never liked discord for chatting either. I found it strangely confusing trying to keep track of logins for each group
  
  Edit: I am indeed thinking of slack
- I fucking hate discord
  
  It's Cancer, have an upvote.
thanks to them for making my deredditification that much easier!
The only way they get my clicks now are when I Google something and they come up.

They really keep making sure that I don't end up there.
- libredirect helps with that on desktop
  
  (browser extension that turns links to sites like reddit, youtube, etc into links to redlib, invidious)
Too bad. Hey, crazy idea: let's create an open alternative for reddit with good content! Maybe something in the fediverse or so.
- I think you're onto something
- that would never work
There are numerous occasions where someone has a lingering question on Reddit that I see and know the answer to. It’s too bad it’s on Reddit because I no longer contribute to that website, and refuse to.
- All the decent answers I find are from 5+ years ago. I check the user’s activity and they normally quit the place. Warms the heart.
just begin with site:reddit.com test for ddg and it still works
- Are they new posts or old ones? They are blocking new ones, not old ones.
  
  new posts do not work
  
  this post in /r/selfhosted is from 8hr ago: SWEKIT v0.1 - an open source library to build software engineering agents (DEVIN) in a agentic framework agnostic manner!
  
  reddit/redlib: https://redlib.kylrth.com/r/selfhosted/comments/1eb86lf/swekit_v01_an_open_source_library_to_build/
  
  doesn’t appear in DDG results: https://duckduckgo.com/?q=site%3Areddit.com+SWEKIT+v0.1+-+an+open+source+library+to+build+software+engineering+agents+(DEVIN)+in+a+agentic+framework+agnostic+manner!&t=ffit
  
  I tested and it got lots of reddit queries from even 2 years ago afaik.
- Based on my testing if you filter results by the last week or last day you get nothing. Past month works.
- For old posts. I can't find new posts on DDG. I find them on Google but not on DDG.
  
  I tried brave search begin with site:reddit.com test and it still works
LMAO searching "____ reddit" is the only time I visit their site.

They just really have no clue.
Would lemmy instances do this?

I know they can't afford to now, but hypothetically? A lot of people here don't seem to like data scraping for AI.
- Your Lemmy posts are already being scraped for AI
  
  The level of effort it would take to prevent would be infeasible to ask of even a non volunteer admin let alone a volunteer let alone literally all of them
  
  Your Lemmy posts are already being scraped for AI
  
  Good, hopefully it’ll make AI that is slightly less toxic than the rest of the internet.
  
  It always baffles me that people don’t want their content represented in an AI - every word you write that gets indexed is a vote for how future AI will behave.
  
  That's what I figured, but I am envisioning a future where lemmy is huge and the network of admins is quite sizable.
  
  I guess that doesn't change much?
- You don't need to scrape. If you want to get all the content on Lemmy, just set up an instance and subscribe to all the top communities, and the instances will just send you all the content.
  
  So there isn't really a way to monetise or block it. I guess you could only federate to a whitelist, but the biggest instances will federate by default with any new instances until they are given a reason to defederate.
- Some Lemmy instances disallow indexing in robots.txt, however indexers can choose to ignore that and actually blocking them takes a lot more effort.
  
  Some places on a "budget" like Ao3 just rate limit hard.
  
  I don't like that solution at all though.
The users who wrote the content are going to get a share of the money, right Reddit? Riiight? /s
So glad I found this alternative. reddit, mods are psychos and the average user not much better
Brave search got an option for that.
- begin with site:reddit.com test is much more accurate to get reddit search on brave search tbh
- Someone should make this feature but for ALL public web content you browse. Just download an extension to share the content of pages you browse to everyone (with cross-checking for accuracy), and you can view a fair share of what others have shared based on how much you contributed to the platform yourself. Basically crowd-sourced, unblockable web scraping.
- We need that for DDG. Opt-in, of course, but with a banner that makes it clear why is that really needed
Good, their answers are generally crap, and I wish they wouldn't show in searches anyway.
- I mostly feel the opposite.
  
  Reddit is one of the only search results that actually has content made by humans.
  
  I mean, you're right if by humans you mean kids.
  
  Depending on the subject, I encounter more and more threads with <deleted by user> content. And billions and billions and billions of results that are either spam or written by unprofessionals.
  
  The smart crowd is not there anymore. The smart crowd that once was there, has removed the content that Reddit was worth visiting for. Let the Googzz have them and sell ads to each other.
lol - fine by me. My private searx-ng instance already filters out Reddit from the results, and my Pi-holes block all known Reddit domains.
Your fault for using a major search engine honestly

71 comments