How do I use Open Source scrapers? (Selenium, Scrapy, etc.)

I have quite an extensive history of scraping web sites for various data over the years, I'd be happy to help you out but I can't really know how to help without knowing what website your trying to scrape, different sites have their own challenges (maybe behind a login, or using JavaScript to load content - in which case a http response won't give you what you're after, or any number of things really).

If you give me a link to a book you want to download as an example I can take a look and help guide you through it

It needs a driver and the web-browser to be executed in headless mode. For Chrome that's chrome-driver. You can get it here.

To make a script for it, I recommend talking to a LLM. I have asked it to build scrapers before, so it does the job.

If you want a practical use of Selenium being demonstrated, you can see it in LucidWebSearch plugin for Oobabooga.

There is no simplification that you're looking for. It seems you don't have a programing background. If you really need to scrape something, you need to learn a programing language, HTTP, HTML, and maybe javascript. AFAIK, there is no easy way or point and click scrapper building tool. You will need to invest time and learn. Don't worry, you should be able to get it done in 2-3 months if you do invest your time in.

Selenium is really more of a testing framework for frontend developers, and could theoretically be used for scraping, but that would be somewhat like buying a car based on the paint and not looking in detail under the hood.

I can't say I've ever worked with scrappy, but the tool I would use for web scraping with Python is BeautifulSoup. This tutorial seems decent enough, but you will need to understand basic web concepts like IDs, classes, tags, and tag attributes to get the most out of the tutorial: https://geekpython.medium.com/web-scraping-in-python-using-beautifulsoup-3207c038723b

W3Schools will also be your friend if you have questions about HTML/CSS selectors in general: https://www.w3schools.com/html/default.asp

Understanding regular expressions and/or xpath would also be very helpful, but are probably best considered to be extra credit in most cases.

I'll try to respond if you have any issues or questions, but hopefully that gives you enough to get started.

Depending on what you want to scape, that's a lot of overkill and overcomplication. Full website testing frameworks may not be necessary to scrape. Python with it's tooling and package management may not be necessary.

I've recently extracted and downloaded stuff via Nushell.

1. Requirement: Knowledge of CSS Selectors
2. Inspect Website DOM in Webbrowser web developer tools
   1. Identify structure
   2. Identify adequate selectors; testable via browser dev tools console document.querySelectorAll()
3. Get and query data

For me, my command line terminal and scripting language of choice is Nushell:

let $html = http get 'https://example.org/'
let $meta = $html | query web --query '#infobox .title, #infobox .tags' |  | { title: $in.0.0 tags: $in.1.0 }
let $content = $html | query web --query 'main img' --attribute data-src
$meta | save meta.json

or

1..30 | each {|x| http get $'https://example.org/img/($x).jpg' | save $'($x).jpg'; sleep 100ms }

Depending on the tools you use, it'll be quite similar or very different.

Selenium is an entire web-browser driver meaning it does a lot more and has a more extensive interface because of it; and you can talk to it through different interfaces and languages.

Selenium is a “driver” that controls browsers, you would need some type of software to actually drive it. If you have programming experience it’s pretty easy to get going.

Personally, I use it in Ruby on Rails development for unit testing but I also use it to log in to websites and perform some actions on behalf of a user (where the websites don’t offer an API).

I don’t have experience with the others, but thought my comment may or may not be useful.

I am no expert, but I have used Python in a professional environment, and helped on board a Python newbie to build out his first project.

It would be helpful to know what your environment looks like (what OS you are running, Python version, terminal interface -- are you running cmd, powershell, terminal) and which steps prompts the reported error messages.

Starting from the first time running Python using a Windows computer, the first steps should be

Launch Powershell as admin and type in the following commands:

set-executionpolicy remotesigned

winget install python

mkdir python

cd python

python -m venv scraper

.\scraper\Scripts\activate

Following that you should be able to use pip to install more modules or packages. I have Visual Studio Code as my IDE, and that means from there I can also run code to open the text editor to write whatever code I intend to run. Be sure to save it to C:\Users\youruseraccount\python If your scripts are saved to that folder, you can run them from powershell by just typing in their filename. Any time you run scripts, open powershell and type cd python and then .\scraper\Scripts\activate Hit enter, then type in the name of the script you want to run.

This information dump is not the most detailed, but it should get you to the point that you can run your scripts.

We use node.js with puppeteer for some of our web crawling at work. It's pretty straightforward once you have a basic script to launch it. If you havent already I'd highly suggest installing vs code. You install node.js, then using npm (node package manager) install puppeteer and whatever other dependencies you might have. Someone out there probably has a basic js file out there that will open chrome, or just ask an LLM (I just use ChatGPT, they're all the same shit). From there you just need to navigate to your pages, then use a queryselector and .click() to click on your elements. It's all javascript from there.

Pro tip: write your queryselectors in your browser using the inspect element/console tab, then put it in your JS file. Nothing is worse than being 10 minutes into a crawl and you've got a queerselector.

I'm an automation engineer. For scripting with with python and selenium what's the issue

Ask dr gpt