Programming.dev Meta @programming.dev lucoso @programming.dev 2y ago

what about data scrapers?

reddit and twitter (suposedly) jacked up their API prices because of data scrapers, what could lemmy to do try to stop them?

i dont think we can do anything

7 comments

Why would we? I posted on the internet with the knowledge that it would be used by others
- Yeah, I actually think lemmy could benefit from improved scraping and indexing. For example, it'd really help if more search engines could natively understand Lemmy's federated nature, e.g:
  
  Deduplicate links by prioritising results for instances hosting the community that a post was originally submitted to.
  
  Include and denote cross posts by recognizing order of submission timestamp and prioritizing popularity via vote ratios, comment counts, and lurker click-through traffic.
  
  Do the same deduplication and prioritization across instances, but for comments as well.
  
  Another use case besides search engines would be for internet archive projects, helping to preserve historic internet content even in the face of lemmy instances falling offline and disappearing. For example, much knowledge was lost to us due to the Twitter APIoplicips and Reddit Blackout: E.g:
  
  https://wiki.archiveteam.org/index.php/Warrior_projects
  
  https://tracker.archiveteam.org/reddit/
  
  Most of the above will only ever be possible due to improved scraping or even federation APIs.
Well, they did it because they're for profit companies looking to wring money out of people's interactions. Lemmy is open source and, by nature, publicly and readily available to whatever observers want to federate. It would be dead simple to create an activitypub server that does nothing but listen and save a copy of everything everyone ever said or posted (at least, those you get it to federate with).
- Yeah absolutely, I'm actually tempted to implement some kind of encompassing "all" instance that listens to all communities of all known lemmy instances (to emulate something like reddits r/all equivalent for lemmy), which I think would be pretty useful, to get an idea of the "size" of the lemmy community. Probably without any users itself (also to avoid being blacklisted from other instances).
  
  I wonder how stuff like this would affect the scalability of the fediverse, I think having 1 instance like you describe would be good, but having many of them could become an issue maybe
I think this is the beauty of federation. Everything is open and free to all rather than a company being able to lock in your personally created content.

For example, I wanted to learn about NLP and am working on building a bot to monitor sentiment and check for hate speech in lemmy content. I am still at the brainstorming/research phase, but the accessibility of lemmy makes it really nice.

Pythorhead was made for this exact purpose.

7 comments