googlenewsscraper
Getting Started
Installation
pip install GoogleNewsScraper
Reference
Importing
from GoogleNewsScraper import GoogleNewsScraper
Instantiating Scraper
GoogleNewsScraper(driver)
Constructor Parameters
| Name | Type | Required |
|---|---|---|
| driver | web driver | no |
Possible values:
'chrome': The driver will default to use this package's chrome driver- A path to some driver (FireFox, for instance) stored on the user's system
Methods
This method is both public and private, though it really should only be used by the class
locate_html_element(self, driver, element, selector, wait_seconds)
| Name | Type | Required | Description |
|---|---|---|---|
| driver | web driver | yes | A web driver (Chrome, FireFox, etc) |
| element | string | yes | Id or class selector of an HTML element |
| selector | Module import | yes | see below |
| wait_seconds | int | no | Waits a certain number of seconds in order to locate an HTML element |
To configure the 'selector' param:
First install selenium
pip install selenium
Then import By
from selenium.webdriver.common.by import By
Possible values:
By.IDBy.CLASS_NAMEBy.CSS_SELECTORBy.LINK_TEXTBy.NAMEBy.PARTIAL_LINK_TEXTBy.TAG_NAMEBy.XPATH
GoogleNewsScraper(...args).search(search_text, date_range, pages, pagination_pause_per_page, cb) -> list or None
| Name | Type | Required | Description |
|---|---|---|---|
| search_text | str | yes | A series of word(s) that will be inputted into the Google search engine |
| date_range | str | no | Filters article by date. Possible values: Past hours, Past 24 hours, Past week, Past month, Past year, Archives |
| pages | str or int | no | Number of pages that should be scraped (defaults to 'max') |
| pagination_pause_per_page | int | no | Waits a certain amount of seconds before a new page is scraped (defaults to 2). Time may have to be increased if Google prevents you from scraping all pages. |
| cb | function | no | Will return all article data on a single page for every page scraped (defaults to False) |
- Example using 'cb' paramater:
def handle_page_data(page_data: list):
# Do something with page_data
GoogleNewsScraper(...args).search(...args, cb=handle_page_data)
NOTE:
- If no argument is provided for 'cb,' the scrape method will return a two-dimensional list
- Each list will contain an object of news article data for every news article on that page
Example of the data that every article-object will contain:
'id': A unique id for every article data object'description': The preview description of the news article'title': The title of the news article'source': The source of news article (New York Times, for instance)'image_url': The url of the preview news article image'url': A link to the news article'date_time': A datetime string that represents the date of when the article was published
12.3k Jan 7, 2023
3 Feb 13, 2022
1 Nov 13, 2021
2 Jul 22, 2022
1 Nov 7, 2021
0 Nov 7, 2022
4 Jun 2, 2022
3.8k Jan 2, 2023
46.4k Jan 3, 2023
16 Oct 8, 2022
2 Jan 24, 2022
13 Dec 21, 2022
0 Nov 22, 2021
0 Nov 17, 2021
2 Jun 6, 2022
1 Dec 5, 2021
18 Dec 14, 2022
2 Jan 7, 2022
2 Dec 22, 2021