Automated data scraper for Thailand COVID-19 data

Related tags

Web Crawling the-researcher-covid-data
Overview
Comments
  • Github Actions are failing

    Github Actions are failing

    Looks like most of the Github Actions are all failing. get-vaccine-delivery: IndexError: list index out of range https://github.com/porames/the-researcher-covid-data/runs/4284380038?check_suite_focus=true

    bot-task-scheduler - get-moh-prompt: selenium TimeoutException https://github.com/porames/the-researcher-covid-data/runs/4282943528?check_suite_focus=true

    opened by nathakits 11
  • Add update date/time to vaccination data

    Add update date/time to vaccination data

    Read หมอพร้อม dashboard header for update date/time and put it in the dataset.

    {
  "update_date": date,
  "data": []
}

    And check if dashboard has newer data before attempt to scrape.

    opened by porames 1
  • Power Bi blocks Github Actions

    Power Bi blocks Github Actions

    Thanks to Microsoft, selenium can't render power bi dashboard for หมอพร้อม. For now, Circle CI works fine but making it run using GitHub actions would be better. @plumsirawit try vpn or something?

    opened by porames 1
  • fix: update link for cases dataset

    fix: update link for cases dataset

    • Provincial cases dataset has been splited again so we need to update links to the new data set and merge these dataset together.
    • Old dataset have been put into their separate zip file.
    opened by noppakorn 0
  • Question - Why is the scraped vaccine data different from the dashboard?

    Question - Why is the scraped vaccine data different from the dashboard?

    Hi. For the vaccine data by doses national-vaccination-timeseries, the scraped data is slightly different from the dashboard. Already minus the 4th dose.

    For example on the 4th October, the scraped data:

    {
    "date": "2021-10-04",
    "daily_vaccinations": 667141,
    "data_anomaly": null,
    "first_dose": 33232868,
    "second_dose": 21187548,
    "third_dose": 1467466,
    "total_doses": 55887882
  }

    Moh Promp Dashboard. Checked data at 9pm 4th October

    1st Dose: 33,232,871
2nd Dose: 21,187,550
3rd Dose: 1,467,466
Total: 55,887,887

    Also if possible, it would be nice to add 4th dose to the scraper as well.

    opened by nathakits 0
  • refactor: vaccination timeseries

    refactor: vaccination timeseries

    • Separate national and manufacturer into 2 files
    • PEP 8
    • Descriptive variable name
    • Do multiple operation in one line
    • Replace some pandas function with numpy for efficiency
    • Export national timeseries as csv
    • Convert data from float to int to save storage spaces
    opened by noppakorn 0
Owner
Porames Vatanaprasan
Medical Student. Faculty of Medicine, Ramathibodi Hospital, Mahidol University.
Porames Vatanaprasan
GitHub
Shopee Scraper - A web scraper in python that extract sales, price, avaliable stock, location and more of a given seller in Brazil

Shopee Scraper A web scraper in python that extract sales, price, avaliable stock, location and more of a given seller in Brazil. The project was crea

Paulo DaRosa 5 Nov 29, 2022
An automated, headless YouTube Watcher and Scraper

Searches YouTube, queries recommended videos and watches them. All fully automated and anonymised through the Tor network. The project consists of two independently usable components, the YouTube automation written in Python and the dockerized Tor Browser.

null 44 Oct 18, 2022
An Automated udemy coupons scraper which scrapes coupons and autopost the result in blogspot post

Autoscraper-n-blogger An Automated udemy coupons scraper which scrapes coupons and autopost the result in blogspot post and notifies via Telegram bot

GOKUL A.P 13 Dec 21, 2022
Github scraper app is used to scrape data for a specific user profile created using streamlit and BeautifulSoup python packages

Github Scraper Github scraper app is used to scrape data for a specific user profile. Github scraper app gets a github profile name and check whether

Siva Prakash 6 Apr 5, 2022
This is a web scraper, using Python framework Scrapy, built to extract data from the Deals of the Day section on Mercado Livre website.

Deals of the Day This is a web scraper, using the Python framework Scrapy, built to extract data such as price and product name from the Deals of the

David Souza 1 Jan 12, 2022
Video Games Web Scraper is a project that crawls websites and APIs and extracts video game related data from their pages.

Video Games Web Scraper Video Games Web Scraper is a project that crawls websites and APIs and extracts video game related data from their pages. This

Albert Marrero 1 Jan 12, 2022
A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

Alex Papadopoulos 1 Nov 13, 2021
A universal package of scraper scripts for humans

Scrapera is a completely Chromedriver free package that provides access to a variety of scraper scripts for most commonly used machine learning and data science domains.

null 299 Dec 15, 2022
A Smart, Automatic, Fast and Lightweight Web Scraper for Python

AutoScraper: A Smart, Automatic, Fast and Lightweight Web Scraper for Python This project is made for automatic web scraping to make scraping easy. It

Mika 4.8k Jan 4, 2023
A web scraper that exports your entire WhatsApp chat history.

WhatSoup ?? A web scraper that exports your entire WhatsApp chat history. Table of Contents Overview Demo Prerequisites Instructions Frequen

Eddy Harrington 87 Jan 6, 2023
Python scraper to check for earlier appointments in Clalit Health Services

clalit-appt-checker Python scraper to check for earlier appointments in Clalit Health Services Some background If you ever needed to schedule a doctor

Dekel 16 Sep 17, 2022
A Web Scraper built with beautiful soup, that fetches udemy course information. Get udemy course information and convert it to json, csv or xml file

Udemy Scraper A Web Scraper built with beautiful soup, that fetches udemy course information. Installation Virtual Environment Firstly, it is recommen

Aditya Gupta 15 May 17, 2022
🤖 Threaded Scraper to get discord servers from disboard.org written in python3

Disboard-Scraper Threaded Scraper to get discord servers from disboard.org written in python3. Setup. One thread / tag If you whant to look for multip

Ѵιcнч 11 Nov 1, 2022
A simple proxy scraper that utilizes the requests module in python.

Proxy Scraper A simple proxy scraper that utilizes the requests module in python. Usage Depending on your python installation your commands may vary.

null 3 Sep 8, 2021
A simple python web scraper.

Dissec A simple python web scraper. It gets a website and its contents and parses them with the help of bs4. Installation To install the requirements,

null 11 May 6, 2022
Twitter Scraper

Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse–engineered. No API rate limits. No restrictions. Extremely fast.

Tayyab Kharl 45 Dec 30, 2022
Kusonime scraper using python3

Features Scrap from url Scrap from recommendation Search by query Todo [+] Search by genre Example # Get download url >>> from kusonime import Scrap >

MhankBarBar 2 Jan 28, 2022
simple http & https proxy scraper and checker

simple http & https proxy scraper and checker

Neospace 11 Nov 15, 2021
Nekopoi scraper using python3

Features Scrap from url Todo [+] Search by genre [+] Search by query [+] Scrap from homepage Example # Hentai Scraper >>> from nekopoi import Hent >>>

MhankBarBar 9 Apr 6, 2022