Stealthy Scraping Tools

Do not use puppeteer and playwright for scraping. Explanation.

We only use the CDP to obtain the page source and to get the absolute coordinates for an arbitrary CSS selector. That's all what is needed for efficient scraping.

To obtain the page source of the browser's current page. Implemented in page_source.js
To get the absolute coordinates for an arbitrary CSS selector. Implemented in coords.js

Mouse movements and typing is handled by pyautogui or other means, but not with JavaScript or with the CDP! Reason: Browser based mouse and keyboard emulation is very easy detectable!

Theory

Analyzing key strokes: TypeNet: Deep Learning Keystroke Biometrics
Research how to mimic human mouse movements: BeCAPTCHA-Mouse: Synthetic Mouse Trajectories and Improved Bot Detection

Full Example

The bot challenge that can be found here bot.incolumitas.com/#botChallenge will be solved in the following quick tutorial.

The example code can be found in example.py.

I am using an Ubuntu 18.04 system with Python3 (with pipenv) and a recent Node version.

The browser google-chrome must be installed.

Clone the repo:

git clone https://github.com/NikolaiT/stealthy-scraping-tools
cd stealthy-scraping-tools

Activate a pipenv environment with:

pipenv shell

Then install pyautogui:

pip install pyautogui

Install node modules:

npm install chrome-remote-interface

And then run the bot with:

python example.py

TODO

Look at Kernel/OS level mouse/keyboard control commands (Ditch pyautogui)
Use the math from ghost-cursor
Create a set of typign recordings and use it to derive rules for bot writing

Scraping Thailand COVID-19 data from the DDC's tableau dashboard

Scraping COVID-19 data from DDC Dashboard Scraping Thailand COVID-19 data from the DDC's tableau dashboard. Data is updated at 07:30 and 08:00 daily.

5 Jan 4, 2022

A Web Scraping Program.

Web Scraping AUTHOR: Saurabh G. MTech Information Security, IIT Jammu. If you find this repository useful. I would appreciate if you Star it and Fork

2 Dec 14, 2022

PyQuery-based scraping micro-framework.

demiurge PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x. Documentation: http://demiurge.readthedocs.org Installing demiurge $ pip

109 Jul 20, 2022

Universal Reddit Scraper - A comprehensive Reddit scraping command-line tool written in Python.

543 Jan 3, 2023

A web scraping pipeline project that retrieves TV and movie data from two sources, then transforms and stores data in a MySQL database.

New to Streaming Scraper An in-progress web scraping project built with Python, R, and SQL. The scraped data are movie and TV show information. The go

1 Mar 28, 2022

Comments

`.getBoundingClientRect()`

https://github.com/NikolaiT/stealthy-scraping-tools/blob/9d0fa1c56dada4fc1a7d9838e469da9aa41964cf/behavior/sst_utils.py#L53

I have had "the same" issue, I guess.

If you use JSON.stringify({offsetY: window.screen.availHeight - window.innerHeight, offsetX: window.screen.width - window.availWidth}) (I'm using availHeight) the coordinates seem to be accurate.

I don't know if you mean this issue with your comment.

opened by 0x7357 0
I guess pyautogui.scroll requires an integer?

https://github.com/NikolaiT/stealthy-scraping-tools/blob/9d0fa1c56dada4fc1a7d9838e469da9aa41964cf/behavior/behavior.py#L96

At least it throws an error if I pass (like you did) a float value. Passing an integer works.

opened by 0x7357 0
Alot of issues
Conflict between doubleHit function and doubleHits variable.

subprocess.CalledProcessError: Command '/usr/bin/node coords.js "li:nth-of-type(3) a"'
opened by Mobeen-Ghaffar 1