👁️ Tool for Data Extraction and Web Requests.

Last update: Dec 5, 2021

Overview

httpmapper 👁️

Project • Technologies • Installation • How it works • License

Project 🚧

For educational purposes.

This is a project that I developed, which is basically a Web crawler that navigate in the web, extracting source codes, links, cookies and more. I also did to learn more about requests and data-extraction.

Technologies 🛠️

This project was developed with the following technologies:

Python

Installation 🚀

# cloning repository
git clone https://github.com/vLeeH/httpmapper

# enter on folder 
cd httpmapper 

# update 
sudo apt update 

# running 
python install.py

How it works 🔧

# using
python install.py 

examples 

# website for this example: https://github.com 

   python install.py 

   Choice: 5
   Website: https://github.com 

   [+] Cookie Name = _octo - Cookie Value = GH1.1.413278149.1633841686
   [+] Cookie Name = logged_in - Cookie Value = no
   [+] Cookie Name = _gh_sess - Cookie Value = ejqBvu%2BSIjM68y7f8niePF8U%2FyrwbGVoKc8iW6FWLil8%2BtsOtGcYSaxw52b%2FhCg%2F275eqHG18jSe4wZ7TFzvlD5Xx6tqvddoSy%2BEdOUlooL7gEpchhK1W8i0Y%2Fg1ARBhrK3saX43%2FjlBEMJX45km%2BPHf39gxk1fO8fc6ytX%2Fp7uX2F1z3hMIep76ooxirYuFzSwBefa3EZU5fZq2OQoV2is8xjiInY72lDSxErMjPKKS6%2B1cjUp9NW7bS5G63%2B9AJCPMwjpdg15qa8aulJ%2FLZg%3D%3D--qTvdBCfTpQiV75Hr--liTEY8bhr%2B0QHWrLVyJZ8w%3D%3D
   [-] Cookie extracter finished!

Note: You need to identify which browser is being used for the header variable.

Contributing 🔨

How can I contribute to the project?

1. Create a fork from httpmapper repository.
2. git clone https://github.com/vLeeH/httpmapper.git
3. cd httpmapper/
4. Make your changes.
5. Commit and make a git push.
6. Open a pull request.

License 📝

This project is under the MIT License.

Scraping web pages to get data

Scraping Data Get public data and save in database This is project use Python How to run a project 1 - Clone the repository 2 - Install beautifulsoup4

2 Nov 1, 2021

Web Crawlers for Data Labelling of Malicious Domain Detection & IP Reputation Evaluation

Web Crawlers for Data Labelling of Malicious Domain Detection & IP Reputation Evaluation This repository provides two web crawlers to label domain nam

1 Nov 5, 2021

This is a web scraper, using Python framework Scrapy, built to extract data from the Deals of the Day section on Mercado Livre website.

Deals of the Day This is a web scraper, using the Python framework Scrapy, built to extract data such as price and product name from the Deals of the

1 Jan 12, 2022

A tool for scraping and organizing data from NewsBank API searches

nbscraper Overview This simple tool automates the process of copying, pasting, and organizing data from NewsBank API searches. Curerntly, nbscrape onl

0 Jun 17, 2021

A tool to easily scrape youtube data using the Google API

YouTube data scraper To easily scrape any data from the youtube homepage, a youtube channel/user, search results, playlists, and a single video itself

7 Dec 3, 2022

An application that on a given url, crowls a web page and gets all words, sorts and counts them.

Web-Scrapping-1 An application that on a given url, crowls a web page and gets all words, sorts and counts them. Installation Using the package manage

1 Jan 16, 2022

This repo has the source code for the crawler and data crawled from auto-data.net

This repo contains the source code for crawler and crawled data of cars specifications from autodata. The data has roughly 45k cars

5 Nov 22, 2022

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Pattern Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google, Twitter, Wikipedia), web crawler, HTML DOM par

Computational Linguistics Research Group

8.4k Jan 8, 2023

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

AutoScraper: A Smart, Automatic, Fast and Lightweight Web Scraper for Python This project is made for automatic web scraping to make scraping easy. It

4.8k Jan 4, 2023

👁️ Tool for Data Extraction and Web Requests.

Related tags

Overview

httpmapper 👁️

Project 🚧

Technologies 🛠️

Installation 🚀

How it works 🔧

Contributing 🔨

License 📝

You might also like...

Scraping web pages to get data

Web Crawlers for Data Labelling of Malicious Domain Detection & IP Reputation Evaluation

This is a web scraper, using Python framework Scrapy, built to extract data from the Deals of the Day section on Mercado Livre website.

A tool for scraping and organizing data from NewsBank API searches

A tool to easily scrape youtube data using the Google API

An application that on a given url, crowls a web page and gets all words, sorts and counts them.

This repo has the source code for the crawler and data crawled from auto-data.net

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Owner

Simple Web scrapper Bot to scrap webpages using Requests, html5lib and Beautifulsoup.

Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data.

A simple proxy scraper that utilizes the requests module in python.

A web scraping pipeline project that retrieves TV and movie data from two sources, then transforms and stores data in a MySQL database.

News, full-text, and article metadata extraction in Python 3. Advanced docs:

robobrowser - A simple, Pythonic library for browsing the web without a standalone web browser.

Web scrapping tool written in python3, using regex, to get CVEs, Source and URLs.

A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

Video Games Web Scraper is a project that crawls websites and APIs and extracts video game related data from their pages.

This is a web crawler that works on employ email data by gmane.org and visualizes it in different ways.