find all the URL of a site with a specific Regex

Overview

href

this program will find all the link with a spesfic Regex pattern from a site.

what it will do

in any site there are a lots of url that may you need the file behind them, this program will find all the tag, then list the href of the tags. you can use Regex to find the special link(s)

all the finded url's have some special charater's, so the Regex pattern will try to match with all finded url, if match, the url will return. if not match, try for next url in the list.

if you do not write any pattern the program will print all link of site, defualt pattern is: .*

and the last thing is that the program is case-insensitive

how to use

usage:

    --url 'the url of site'

    --pattern 'Regex pattern'

    --load-headers` path header file

    href.py --url 'URL' --pattern 'RegegPattern' --load-headers ./headers

example:

    href.py --url 'https://guitarmusic.ir/hayedeh-songs/' --pattern '.*mp3.*'

example:

href

note:

  1. all the switch have a small way to use

--help: -h

--url: -u

--pattern: -p

--load-headers: -l

  1. use pipe

to use the program some time you need to pipe or redirect the result

some site repeated their link to preview a video or music before download them, so you can pipe the result to uniq command for prevent link duplicate.

and for having the link in a text file, you should redirect the result to a file. href.py -u "URL" -p "patternt" > links.txt

  1. run easy

to run the program witch oud cd to the source dir or wite the full path every time, you can link it to your ~//.local/bin/href do it by this command: ln -s href.py ~/.local/bin/href

and do not forget to make it executable

  1. headers

if you got any http status code try to use a the header of site, or use the default header in the directory by this switch --load-header [file]

Owner
Arya Shabane
The Only Currency Left In This World Is POWER | Knowledge is POWER
Arya Shabane
python3 flask based python-url-shortener microservice.

python-url-shortener This repository is for managing all public/private entity specific api endpoints for an organisation. In this case we have entity

Asutosh Parida 1 Oct 18, 2021
A simple, immutable URL class with a clean API for interrogation and manipulation.

purl - A simple Python URL class A simple, immutable URL class with a clean API for interrogation and manipulation. Supports Pythons 2.7, 3.3, 3.4, 3.

David Winterbottom 257 Oct 9, 2021
Astra is a tool to find URLs and secrets.

Astra finds urls, endpoints, aws buckets, api keys, tokens, etc from a given url/s. It combines the paths and endpoints with the given domain and give

Stinger 128 Sep 30, 2021
A URL builder for genius :D

genius-url A URL builder for genius :D Usage from gurl import genius_url

ꌗᖘ꒒ꀤ꓄꒒ꀤꈤꍟ 12 Aug 14, 2021
🌐 URL parsing and manipulation made easy.

furl is a small Python library that makes parsing and manipulating URLs easy. Python's standard urllib and urlparse modules provide a number of URL re

Ansgar Grunseid 2.1k Oct 18, 2021
A python code for url redirect check

A python code for url redirect check

Fayas Noushad 1 Oct 24, 2021
🔗 Generate Phishing URLs 🔗

URLer ?? Generate Phishing URLs ?? URLer Table Of Contents General Information Preview Installation Disclaimer Credits Social Media Bug Report General

mrblackx 4 Sep 27, 2021
URL Shortener in Flask - Web service using Flask framework for Shortener URLs

URL Shortener in Flask Web service using Flask framework for Shortener URLs Install Create Virtual env $ python3 -m venv env Install requirements.txt

Rafnix Guzman 1 Sep 21, 2021
A url redirect status check module for python

A url redirect status check module for python

Fayas Noushad 2 Oct 24, 2021
a url shortener project from semicolonworld

Url Shortener With Django Written by Semicolon World

null 3 Aug 24, 2021
This is a no-bullshit file hosting and URL shortening service that also runs 0x0.st. Use with uWSGI.

This is a no-bullshit file hosting and URL shortening service that also runs 0x0.st. Use with uWSGI.

mia 1.5k Oct 25, 2021
Have you ever wondered: Where does this link go? The REDLI Tool follows the path of the URL.

Have you ever wondered: Where does this link go? The REDLI Tool follows the path of the URL. It allows you to see the complete path a redirected URL goes through. It will show you the full redirection path of URLs, shortened links, or tiny URLs.

JAYAKUMAR 28 Oct 20, 2021
declutters url lists for crawling/pentesting

uro Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; uro aims to solve that.

Somdev Sangwan 253 Oct 19, 2021
encurtador de links feito com python

curt-link encurtador de links feito com python! instalação Linux: $ git clone https://github.com/bydeathlxncer/curt-link $ cd curt-link $ python3 url.

bydeathlxncer 4 Sep 16, 2021
:electric_plug: Generating short urls with python has never been easier

pyshorteners A simple URL shortening API wrapper Python library. Installing pip install pyshorteners Documentation https://pyshorteners.readthedocs.i

Ellison 316 Oct 25, 2021