ProxyBroker
ProxyBroker is an open source tool that asynchronously finds public proxies from multiple sources and concurrently checks them.
Features
- Finds more than 7000 working proxies from ~50 sources.
- Support protocols: HTTP(S), SOCKS4/5. Also CONNECT method to ports 80 and 23 (SMTP).
- Proxies may be filtered by type, anonymity level, response time, country and status in DNSBL.
- Work as a proxy server that distributes incoming requests to external proxies. With automatic proxy rotation.
- All proxies are checked to support Cookies and Referer (and POST requests if required).
- Automatically removes duplicate proxies.
- Is asynchronous.
Requirements
Installation
To install last stable release from pypi:
$ pip install proxybroker
The latest development version can be installed directly from GitHub:
$ pip install -U git+https://github.com/constverum/ProxyBroker.git
Usage
CLI Examples
Find
Find and show 10 HTTP(S) proxies from United States with the high level of anonymity:
$ proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 10
Grab
Find and save to a file 10 US proxies (without a check):
$ proxybroker grab --countries US --limit 10 --outfile ./proxies.txt
Serve
Run a local proxy server that distributes incoming requests to a pool of found HTTP(S) proxies with the high level of anonymity:
$ proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High
Run
proxybroker --help for more information on the options available.
Run
proxybroker --help for more information on a command.
Basic code example
Find and show 10 working HTTP(S) proxies:
import asyncio
from proxybroker import Broker
async def show(proxies):
while True:
proxy = await proxies.get()
if proxy is None: break
print('Found proxy: %s' % proxy)
proxies = asyncio.Queue()
broker = Broker(proxies)
tasks = asyncio.gather(
broker.find(types=['HTTP', 'HTTPS'], limit=10),
show(proxies))
loop = asyncio.get_event_loop()
loop.run_until_complete(tasks)
Documentation
https://proxybroker.readthedocs.io/
TODO
- Check the ping, response time and speed of data transfer
- Check site access (Google, Twitter, etc) and even your own custom URL's
- Information about uptime
- Checksum of data returned
- Support for proxy authentication
- Finding outgoing IP for cascading proxy
- The ability to specify the address of the proxy without port (try to connect on defaulted ports)
Contributing
- Fork it: https://github.com/constverum/ProxyBroker/fork
- Create your feature branch:
git checkout -b my-new-feature - Commit your changes:
git commit -am 'Add some feature' - Push to the branch:
git push origin my-new-feature - Submit a pull request!
License
Licensed under the Apache License, Version 2.0
This product includes GeoLite2 data created by MaxMind, available from http://www.maxmind.com.
1 Jun 1, 2022
43 May 11, 2022
3 Feb 13, 2022
2 Feb 14, 2022
6 Sep 14, 2022
32 Dec 26, 2022
1 Nov 9, 2021
12 Dec 13, 2022
3 Jan 12, 2022
7 Apr 28, 2022
7 Aug 17, 2022
2 Nov 26, 2021
108 Dec 29, 2022
1 Dec 17, 2021
441 Dec 23, 2022
70 Oct 3, 2022
1 Jan 2, 2022
3 May 22, 2022