A Tool to scrape URLs for a given domain from wayback machine, Commoncrawl and OTX Alienvault

Overview

Mr_URL

Mr.URL fetches known URLs for a given domain from Wayback Machine, Commoncrawl and OTX Alienvault. It also finds old versions of any given URL using WaybackMachine. Inspired by GAU and Waybackurls.

Installation:

$ git clone https://github.com/Sachin-v3rma/Mr_URL
$ cd Mr_URL && pip install -r requirements.txt
$ python3 Mr_URL.py -h

Usage:

Examples:

$ echo example.com | python3 Mr_URL.py
$ echo example.com | python3 Mr_URL.py -subs
$ cat live_subs.txt | python3 Mr_URL.py -q -t5
$ cat js_links.txt | python3 Mr_URL.py -v
Flag Description
-h, --help show this help message and exit
-subs Include Subdomains
-q Quick Mode (Uses Only WaybackMachine)
-v Get Old versions of a URL
-t Number of threads (Default : 10)

Buy Me A Coffee

You might also like...
A Discord bot to scrape textfiles from messages and put them to Hastebin

A Discord bot to scrape textfiles from messages and put them to Hastebin. Intended to use on support servers to help users read textfiles on mobile.

Scrape the Twitter Frontend API without authentication.

Twitter Scraper 🇰🇷 Read Korean Version Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has

Scrape Twitter for Tweets
Scrape Twitter for Tweets

Backers Thank you to all our backers! 🙏 [Become a backer] Sponsors Support this project by becoming a sponsor. Your logo will show up here with a lin

A basic API to scrape Craigslist.
A basic API to scrape Craigslist.

CLAPI A basic API to scrape Craigslist. Most useful for viewing posts across a broad geographic area or for viewing posts within a specific timeframe.

Automatically scrape all of your artifacts in Genshin Impact.

Genshin Artifact Scraper Automatically scrape all of your artifacts in Genshin Impact. Features: Simple recalibration (2 steps). GUI to select OCR reg

Instagram bot that upload images for you which scrape posts from 9gag meme website or other Instagram users , which is 24/7 Automated Runnable.
Instagram bot that upload images for you which scrape posts from 9gag meme website or other Instagram users , which is 24/7 Automated Runnable.

Autonicgram Automates your Instagram posts by taking images from sites like 9gag or other Instagram accounts and posting it onto your page. Features A

Python SCript to scrape members from a selected Telegram group.

A python script to scrape all the members in a telegram group anad save in a CSV file. REGESTRING Go to this link https://core.telegram.org/api/obtain

Telegram bot to scrape images from the reddit universe
Telegram bot to scrape images from the reddit universe

Telegram bot to scrape images from the reddit universe

Python script to scrape users/id/badges/creation-date from a Discord Server Memberlist
Python script to scrape users/id/badges/creation-date from a Discord Server Memberlist

Discord Server Badges Scraper - Credits to bytixo he made this Custom Discord Vanity Creator How To Use Install discum 1.2 Open a terminal and execute

Owner
Stinger
403 Forbidden
Stinger
GET-ACQ is a python tool used to gather all companies acquired by a given company domain name.

get-acq ?? GET-ACQ is a python tool used to gather all companies acquired by a given company domain name. It is done by calling SecurityTrails API. Us

Milan 7 Dec 19, 2022
This bot can stream audio or video files and urls in telegram voice chats :)

Voice Chat Streamer This bot can stream audio or video files and urls in telegram voice chats :) ?? Follow me and star this repo for more telegram bot

Anjana Madu 63 Dec 25, 2022
:electric_plug: Generating short urls with python has never been easier

pyshorteners A simple URL shortening API wrapper Python library. Installing pip install pyshorteners Documentation https://pyshorteners.readthedocs.i

Ellison 351 Jan 3, 2023
Create custom Vanity URLs for Discord without 30 boosts

CustomVanity - Made by udp#6666 aka Apolo - OpenSource Custom Discord Vanity Creator How To Use Open CustomVanity.py Write your server invite code Wri

apolo 17 Aug 23, 2022
domhttpx is a google search engine dorker with HTTP toolkit built with python, can make it easier for you to find many URLs/IPs at once with fast time.

domhttpx is a google search engine dorker with HTTP toolkit built with python, can make it easier for you to find many URLs/IPs at once with fast time

Naufal Ardhani 59 Dec 4, 2022
A simple telegram bot that resolves video urls using yt-dlp

URL to Video Telegram Bot A simple telegram bot that resolves video urls using yt-dlp Copyright (C) 2021 Vítor Vasconcellos This program is free softw

Vítor 1 Nov 18, 2021
A github actions + python code to extract URLs to code repositories to put into standard form, starting with github

A github actions + python code to extract URLs to code repositories to put into standard form, starting with github ---- NOTE: JUS

Justin Gosses 2 Nov 15, 2021
A quick-and-dirty script to scrape the daily menu of Leipzig University Mensa and send it to a telegram channel.

Feed me Mensa UL A quick-and-dirty script to scrape the daily menu of Leipzig University Mensa and send it to a telegram channel. For food and cat lov

null 3 Apr 8, 2022
This is a story bot, that will scrape stories from r/stories subreddit and convert it into an Audio File.

Introduction This is a story bot, that will scrape stories from r/stories subreddit and convert it into an Audio File. Installation pip install -r req

Yasho 11 Jun 30, 2022
API which uses discord+mojang api to scrape NameMC searches/droptime/dropping status of minecraft names, and texture links

API which uses discord+mojang api to scrape NameMC searches/droptime/dropping status of minecraft names, and texture links

null 2 Dec 22, 2021