Yet another googlesearch - A Python library for executing intelligent, realistic-looking, and tunable Google searches.

Overview

yagooglesearch - Yet another googlesearch

Overview

yagooglesearch is a Python library for executing intelligent, realistic-looking, and tunable Google searches. It simulates real human Google search behavior to prevent rate limiting by Google (the dreaded HTTP 429 response), and if HTTP 429 blocked by Google, logic to back off and continue trying. The library does not use the Google API and is heavily based off the googlesearch library. The features include:

  • Tunable search client attributes mid searching
  • Returning a list of URLs instead of a generator
  • HTTP 429 / rate-limit detection (Google is blocking your IP for making too many search requests) and recovery
  • Randomizing delay times between retrieving paged search results (i.e., clicking on page 2 for more results)
  • HTTP(S) and SOCKS5 proxy support
  • Leveraging requests library for HTTP requests and cookie management
  • Adds "&filter=0" by default to search URLs to prevent any omission or filtering of search results by Google
  • Console and file logging
  • Python 3.6+

Terms and Conditions

This code is supplied as-is and you are fully responsible for how it is used. Scraping Google Search results may violate their Terms of Service. Another Python Google search library had some interesting information/discussion on it:

Google's preferred method is to use their API.

Installation

pip

pip install yagooglesearch

setup.py

git clone https://github.com/opsdisk/yagooglesearch
cd yagooglesearch
virtualenv -p python3.7 .venv  # If using a virtual environment.
source .venv/bin/activate  # If using a virtual environment.
python setup.py install

Usage

import yagooglesearch

query = "site:github.com"

client = yagooglesearch.SearchClient(
    query,
    tbs="li:1",
    max_search_result_urls_to_return=100,
    http_429_cool_off_time_in_minutes=45,
    http_429_cool_off_factor=1.5,
    proxy="socks5h://127.0.0.1:9050",
    verbosity=5,
)
client.assign_random_user_agent()

urls = client.search()

len(urls)

for url in urls:
    print(url)

Google is blocking me!

Low and slow is the strategy when executing Google searches using yagooglesearch. If you start getting HTTP 429 responses, Google has rightfully detected you as a bot and will block your IP for a set period of time. yagooglesearch is not able to bypass CAPTCHA, but you can do this manually by performing a Google search from a browser and proving you are a human.

The criteria and thresholds to getting blocked is unknown, but in general, randomizing the user agent, waiting enough time between paged search results (7-17 seconds), and waiting enough time between different Google searches (30-60 seconds) should suffice. Your mileage will definitely vary though. Using this library with Tor will likely get you blocked quickly.

HTTP 429 detection and recovery (optional)

If yagooglesearch detects an HTTP 429 response from Google, it will sleep for http_429_cool_off_time_in_minutes minutes and then try again. Each time an HTTP 429 is detected, it increases the wait time by a factor of http_429_cool_off_factor.

The goal is to have yagooglesearch worry about HTTP 429 detection and recovery and not put the burden on the script using it.

If you do not want yagooglesearch to handle HTTP 429s and would rather handle it yourself, pass yagooglesearch_manages_http_429s=False when instantiating the yagooglesearch object. If an HTTP 429 is detected, the string "HTTP_429_DETECTED" is added to a list object that will be returned, and it's up to you on what the next step should be. The list object will contain any URLs found before the HTTP 429 was detected.

import yagooglesearch

query = "site:twitter.com"

client = yagooglesearch.SearchClient(
    query,
    tbs="li:1",
    verbosity=4,
    num=10,
    max_search_result_urls_to_return=1000,
    minimum_delay_between_paged_results_in_seconds=1,
    yagooglesearch_manages_http_429s=False,  # Add to manage HTTP 429s.
)
client.assign_random_user_agent()

urls = client.search()

if "HTTP_429_DETECTED" in urls:
    print("HTTP 429 detected...it's up to you to modify your search.")

    # Remove HTTP_429_DETECTED from list.
    urls.remove("HTTP_429_DETECTED")

    print("URLs found before HTTP 429 detected...")

    for url in urls:
        print(url)

http429_detection_string_in_returned_list.png

HTTP and SOCKS5 proxy support

yagooglesearch supports the use of a proxy. The provided proxy is used for the entire life cycle of the search to make it look more human, instead of rotating through various proxies for different portions of the search. The general search life cycle is:

  1. Simulated "browsing" to google.com
  2. Executing the search and retrieving the first page of results
  3. Simulated clicking through the remaining paged (page 2, page 3, etc.) search results

To use a proxy, provide a proxy string when initializing a yagooglesearch.SearchClient object:

client = yagooglesearch.SearchClient(
    "site:github.com",
    proxy="socks5h://127.0.0.1:9050",
)

Supported proxy schemes are based off those supported in the Python requests library (https://docs.python-requests.org/en/master/user/advanced/#proxies):

  • http
  • https
  • socks5 - "causes the DNS resolution to happen on the client, rather than on the proxy server." You likely do not want this since all DNS lookups would source from where yagooglesearch is being run instead of the proxy.
  • socks5h - "If you want to resolve the domains on the proxy server, use socks5h as the scheme." This is the best option if you are using SOCKS because the DNS lookup and Google search is sourced from the proxy IP address.

HTTPS proxies and SSL/TLS certificates

If you are using a self-signed certificate for an HTTPS proxy, you will likely need to disable SSL/TLS verification when either:

  1. Instantiating the yagooglesearch.SearchClient object:
import yagooglesearch

query = "site:github.com"

client = yagooglesearch.SearchClient(
    query,
    proxy="http://127.0.0.1:8080",
    verify_ssl=False,
    verbosity=5,
)
  1. or after instantiation:
query = "site:github.com"

client = yagooglesearch.SearchClient(
    query,
    proxy="http://127.0.0.1:8080",
    verbosity=5,
)

client.verify_ssl = False

Multiple proxies

If you want to use multiple proxies, that burden is on the script utilizing the yagooglesearch library to instantiate a new yagooglesearch.SearchClient object with the different proxy. Below is an example of looping through a list of proxies:

import yagooglesearch

proxies = [
    "socks5h://127.0.0.1:9050",
    "socks5h://127.0.0.1:9051",
    "http://127.0.0.1:9052",  # HTTPS proxy with a self-signed SSL/TLS certificate.
]

search_queries = [
    "python",
    "site:github.com pagodo",
    "peanut butter toast",
    "are dragons real?",
    "ssh tunneling",
]

proxy_rotation_index = 0

for search_query in search_queries:

    # Rotate through the list of proxies using modulus to ensure the index is in the proxies list.
    proxy_index = proxy_rotation_index % len(proxies)

    client = yagooglesearch.SearchClient(
        search_query,
        proxy=proxies[proxy_index],
    )

    # Only disable SSL/TLS verification for the HTTPS proxy using a self-signed certificate.
    if proxies[proxy_index].startswith("http://"):
        client.verify_ssl = False

    urls_list = client.search()

    print(urls_list)

    proxy_rotation_index += 1

&tbs= URL filter clarification

The &tbs= parameter is used to specify either verbatim or time-based filters.

Verbatim search

&tbs=li:1

verbatim.png

time-based filters

Time filter &tbs= URL parameter Notes
Past hour qdr:h
Past day qdr:d Past 24 hours
Past week qdr:w
Past month qdr:m
Past year qdr:y
Custom cdr:1,cd_min:1/1/2021,cd_max:6/1/2021 See yagooglesearch.get_tbs() function

time_filters.png

Limitations

Currently, the .filter_search_result_urls() function will remove any url with the word "google" in it. This is to prevent the returned search URLs from being polluted with Google URLs. Note this if you are trying to explicitly search for results that may have "google" in the URL, such as site:google.com computer

License

Distributed under the BSD 3-Clause License. See LICENSE for more information.

Contact

@opsdisk

Project Link: https://github.com/opsdisk/yagooglesearch

Acknowledgements

Comments
  • Got no results

    Got no results

    I wrote such a simple function and ran it. It returned an empty array. This was my 1st time using the module so shouldn't be a rate-limiting thing. I also waited for a long time and retried, still no results.

        gg_query = "topic cluster"
        gg_search = yagooglesearch.SearchClient(
            gg_query,
            verbosity=4, # Logging level: DEBUG (CRITICAL:50, ERROR: 40, WARNING: 30, INFO: 20 -> 4, DEBUG: 10 -> 5, NOTSET: 0 -> 6)
        )
        gg_search.assign_random_user_agent()
    
        urls = gg_search.search()
    

    Result:

    2021-11-06 09:07:23,558 [MainThread  ] [INFO] Requesting URL: https://www.google.com/
    2021-11-06 09:07:23,727 [MainThread  ] [INFO] Stats: start=0, num=100, total_valid_links_found=0 / max_search_result_urls_to_return=100
    2021-11-06 09:07:23,727 [MainThread  ] [INFO] Requesting URL: https://www.google.com/search?hl=en&q=topic+cluster&num=100&btnG=Google+Search&tbs=0&safe=off&cr=en&filter=0
    2021-11-06 09:07:23,906 [MainThread  ] [INFO] The number of valid search results (0) was not the requested max results to pull back at once num=(100) for this page.  That implies there won't be any search results on the next page either.  Moving on...
    
    

    whereas with the original googlesearch library, I get a result with this code:

        from googlesearch import search
        for url in search(gg_query, stop=20):
            print(url)
    
    opened by LeMoussel 22
  • Always no results on next page

    Always no results on next page

    First of all, thanks for the great tool, anyway it seems it fails to perform the search for any 2nd page: "No valid search results found on this page. Moving on..."

    opened by Hoculus 9
  • added captcha handle with unicaps

    added captcha handle with unicaps

    hello is there any chance to added captcha handle with unicaps library? it might nice to have alternative to escaped blocked by google

    what you think about this?

    opened by tjengbudi 8
  • Add output param

    Add output param

    I needed the list of titles, description and urls of the searches

    I added the output parameter that in case of taking the value "complete" returns a list [{"title": title, "desc": description, "url": url}, ...], for any other value returns the original output

    I tested it with the following code

    import yagooglesearch
    
    query = "site:github.com"
    
    client = yagooglesearch.SearchClient(
        query,
        tbs="li:1",
        max_search_result_urls_to_return=100,
        http_429_cool_off_time_in_minutes=45,
        http_429_cool_off_factor=1.5,
        verbosity=5,
        output="complete"  # "normal" : Only url list // "complete" : List of {title, desc, url}
    )
    client.assign_random_user_agent()
    
    urls = client.search()
    
    len(urls)
    
    for url in urls:
        print(url)
    

    The default is "normal" which does not modify the original behavior of the program I also added the output parameter to the documentation

    I'll use the modification from my repo, but if you are interested you can merge it

    Thank you very much for this software and Merry Christmas

    KennBro

    opened by KennBro 8
  • Is there a way to turn off cool_off_time and make search request fail instead of retry?

    Is there a way to turn off cool_off_time and make search request fail instead of retry?

    Thanks for the great library, but as far as I understand it does not provide a way to handle 429 response by yourself(or at least I didn't find one) and it's just trying to make another request after certain cool_off time. It would be great if there was some parameter like "retry" that could be passed to client something like "retry=False" to make it raise an Error if 429 response was received.

    opened by Cyber-Cowboy 8
  • use paid proxy and get error while tunneling

    use paid proxy and get error while tunneling

    hello, i want to ask about some question about proxy, i have a paid proxy that used authentication i trying with pure "requests" in blank project to fetch google and it works fine. but after i added it at yagooglesearch it have error like this Traceback (most recent call last): File "C:\Users\budi\.virtualenvs\article-30Om8FEk\lib\site-packages\urllib3\connectionpool.py", line 700, in urlopen self._prepare_proxy(conn) File "C:\Users\budi\.virtualenvs\article-30Om8FEk\lib\site-packages\urllib3\connectionpool.py", line 996, in _prepare_proxy conn.connect() File "C:\Users\budi\.virtualenvs\article-30Om8FEk\lib\site-packages\urllib3\connection.py", line 369, in connect self._tunnel() File "C:\Users\budi\AppData\Local\Programs\Python\Python310\lib\http\client.py", line 924, in _tunnel raise OSError(f"Tunnel connection failed: {code} {message.strip()}") OSError: Tunnel connection failed: 407 Proxy Authentication Required

    i check several time and it can used in pure requests code. is there any clue for this?

    opened by tjengbudi 2
Releases(v1.6.1)
  • v1.6.1(Jul 30, 2022)

  • v1.6.0(Jan 21, 2022)

    • Added verbose_output (bool) switch to return just a list of URLs (verbose_output=False) or a list of dictionaries containing the URL, rank, title, and description (verbose_output=True). Thanks to @KennBro for submitting https://github.com/opsdisk/yagooglesearch/pull/9
    Source code(tar.gz)
    Source code(zip)
  • v1.5.0(Jan 1, 2022)

    • Improved logic to return more search results https://github.com/opsdisk/yagooglesearch/pull/10
    • Added info log message if duplicate URL is found
    Source code(tar.gz)
    Source code(zip)
  • v1.4.0(Dec 30, 2021)

    • Added yagooglesearch_manages_http_429s option. Determines if yagooglesearch will handle HTTP 429 cool off and retries. Disable if you want to manage HTTP 429 responses. Idea submitted by @Cyber-Cowboy in https://github.com/opsdisk/yagooglesearch/issues/7
    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(Dec 6, 2021)

    • If the IP address is sourcing from a European Union country, the cookie has to be modified. Thank you to @LeMoussel and @kusinhavre for reporting and providing feedback in https://github.com/opsdisk/yagooglesearch/issues/5
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Sep 5, 2021)

  • v1.1.0(Aug 31, 2021)

  • v1.0.0(Aug 30, 2021)

Owner
null
Simple algorithm search engine like google in python using function

Mini-Search-Engine-Like-Google I have created the simple algorithm search engine like google in python using function. I am matching every word with w

Sachin Vinayak Dabhade 5 Sep 24, 2021
This is a Telegram Bot written in Python for searching data on Google Drive.

This is a Telegram Bot written in Python for searching data on Google Drive. Supports multiple Shared Drives (TDs). Manual Guide for deploying the bot

Levi 158 Dec 27, 2022
Google Project: Search and auto-complete sentences within given input text files, manipulating data with complex data-structures.

Auto-Complete Google Project In this project there is an implementation for one feature of Google's search engines - AutoComplete. Autocomplete, or wo

Hadassah Engel 10 Jun 20, 2022
Google Search Engine Results Pages (SERP) in locally, no API key, no signup required

Local SERP Google Search Engine Results Pages (SERP) in locally, no API key, no signup required Make sure the chromedriver and required package are in

theblackcat102 4 Jun 29, 2021
GitScanner is a script to make it easy to search for Exposed Git through an advanced Google search.

GitScanner Legal disclaimer Usage of GitScanner for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to

Kaio Gomes 3 Oct 28, 2022
Google Drive file searcher

Google Drive file searcher

Hafitz Setya 25 Dec 9, 2022
A Python web searcher library with different search engines

Robert A simple Python web searcher library with different search engines. Install pip install roberthelper Usage from robert import GoogleSearcher

null 1 Dec 23, 2021
A library for fast import of Windows NT Registry(REGF) into Elasticsearch.

A library for fast import of Windows NT Registry(REGF) into Elasticsearch.

S.Nakano 3 Apr 1, 2022
A library for fast parse & import of Windows Prefetch into Elasticsearch.

prefetch2es Fast import of Windows Prefetch(.pf) into Elasticsearch. prefetch2es uses C library libscca. Usage When using from the commandline interfa

S.Nakano 5 Nov 24, 2022
A real-time tech course finder, created using Elasticsearch, Python, React+Redux, Docker, and Kubernetes.

A real-time tech course finder, created using Elasticsearch, Python, React+Redux, Docker, and Kubernetes.

Dinesh Sonachalam 130 Dec 20, 2022
Eland is a Python Elasticsearch client for exploring and analyzing data in Elasticsearch with a familiar Pandas-compatible API.

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

elastic 463 Dec 30, 2022
A fast, efficiency python package for searching and getting search results with many different search engines

search A fast, efficiency python package for searching and getting search results with many different search engines. Installation To install the pack

Neurs 0 Oct 6, 2022
document organizer with tags and full-text-search, in a simple and clean sqlite3 schema

document organizer with tags and full-text-search, in a simple and clean sqlite3 schema

Manos Pitsidianakis 152 Oct 29, 2022
High level Python client for Elasticsearch

Elasticsearch DSL Elasticsearch DSL is a high-level library whose aim is to help with writing and running queries against Elasticsearch. It is built o

elastic 3.6k Dec 30, 2022
Pysolr — Python Solr client

pysolr pysolr is a lightweight Python client for Apache Solr. It provides an interface that queries the server and returns results based on the query.

Haystack Search 626 Dec 1, 2022
Whoosh indexing capabilities for Flask-SQLAlchemy, Python 3 compatibility fork.

Flask-WhooshAlchemy3 Whoosh indexing capabilities for Flask-SQLAlchemy, Python 3 compatibility fork. Performance improvements and suggestions are read

Blake VandeMerwe 27 Mar 10, 2022
Senginta is All in one Search Engine Scrapper for used by API or Python Module. It's Free!

Senginta is All in one Search Engine Scrapper. With traditional scrapping, Senginta can be powerful to get result from any Search Engine, and convert to Json. Now support only for Google Product Search Engine (GShop, GVideo and many too) and Baidu Search Engine.

null 33 Nov 21, 2022
esguard provides a Python decorator that waits for processing while monitoring the load of Elasticsearch.

esguard esguard provides a Python decorator that waits for processing while monitoring the load of Elasticsearch. Quick Start You need to launch elast

po3rin 5 Dec 8, 2021
a Telegram bot writen in Python for searching files in Drive. Based on SearchX-bot

Drive Search Bot This is a Telegram bot writen in Python for searching files in Drive. Based on SearchX-bot How to deploy? Clone this repo: git clone

Hafitz Setya 25 Dec 9, 2022