Scraping Thailand COVID-19 data from the DDC's tableau dashboard

Related tags

Web Crawling python scraping covid-19
Overview

Scraping COVID-19 data from DDC Dashboard

Scraping Thailand COVID-19 data from the DDC's tableau dashboard.

fetch-today-data

Data is updated at 07:30 and 08:00 daily.

About

Thailand COVID-19 data is not avalible in raw form from the government. The dashboard make the situation a bit better, but still it's not good enough.

Getting the Data

  • National Timeseries : CSV, JSON
  • Data dates back to 2021-01-01
  • Raw data can be downloaded by cloning this repo wiki.

Data Visualisation

Data Source

You might also like...
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)

trafilatura: Web scraping tool for text discovery and retrieval Description Trafilatura is a Python package and command-line tool which seamlessly dow

Adrien Barbaresi 704 Jan 6, 2023
🥫 The simple, fast, and modern web scraping library
🥫 The simple, fast, and modern web scraping library

About gazpacho is a simple, fast, and modern web scraping library. The library is stable, actively maintained, and installed with zero dependencies. I

Max Humber 692 Dec 22, 2022
Transistor, a Python web scraping framework for intelligent use cases.
Transistor, a Python web scraping framework for intelligent use cases.

Web data collection and storage for intelligent use cases. transistor About The web is full of data. Transistor is a web scraping framework for collec

BOM Quote Manufacturing 212 Nov 5, 2022
A pure-python HTML screen-scraping library

Scrapely Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely con

Scrapy project 1.8k Dec 31, 2022
A repository with scraping code and soccer dataset from understat.com.
A repository with scraping code and soccer dataset from understat.com.

UNDERSTAT - SHOTS DATASET As many people interested in soccer analytics know, Understat is an amazing source of information. They provide Expected Goa

douglasbc 48 Jan 3, 2023
Minimal set of tools to conduct stealthy scraping.

Stealthy Scraping Tools Do not use puppeteer and playwright for scraping. Explanation. We only use the CDP to obtain the page source and to get the ab

Nikolai Tschacher 88 Jan 4, 2023
crypto currency scraping
crypto currency scraping

SCRYPTO What ? Crypto currencies scraping (At the moment, only bitcoin and ethereum crypto currencies are supported) How ? A python script is running

null 15 Sep 1, 2022
A Web Scraping Program.

Web Scraping AUTHOR: Saurabh G. MTech Information Security, IIT Jammu. If you find this repository useful. I would appreciate if you Star it and Fork

Saurabh G. 2 Dec 14, 2022
PyQuery-based scraping micro-framework.

demiurge PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x. Documentation: http://demiurge.readthedocs.org Installing demiurge $ pip

Matias Bordese 109 Jul 20, 2022
Comments
  • Try to use weekly testing data to reverse the real test value

    Try to use weekly testing data to reverse the real test value

    The issue

    • Testing data provided in the dashboard is displayed as a 14-day average value image

    Goal

    • It would be nice if we can know the real value

    Possible Solution

    • There is a dataset which contains testing data but it is updated on a weekly basis which is not ideal for tracking
    • If we can use the historical data to prove that the average value is calculated from the same dataset we can pin point the real value from the average value
    opened by noppakorn 0
  • Testing pipeline will not work between 00:00 - 07:30

    Testing pipeline will not work between 00:00 - 07:30

    • This is because the scraping pipeline check if dashboard date match today's date and throw an error if the date does not match
    • Maybe for create a special CLI argument for testing pipeline that just fetch the data and not check for date
    opened by noppakorn 0
  • New dump format

    New dump format

    • Dump tableau data in one file per day to save space
    • Makes the workflow a bit easier
    • Put more data fields in timeseries
    • Create timeseries in both csv and json
    opened by noppakorn 0
Owner
Noppakorn Jiravaranun
Noppakorn Jiravaranun
GitHub
A web scraping pipeline project that retrieves TV and movie data from two sources, then transforms and stores data in a MySQL database.

New to Streaming Scraper An in-progress web scraping project built with Python, R, and SQL. The scraped data are movie and TV show information. The go

Charles Dungy 1 Mar 28, 2022
A tool for scraping and organizing data from NewsBank API searches

nbscraper Overview This simple tool automates the process of copying, pasting, and organizing data from NewsBank API searches. Curerntly, nbscrape onl

null 0 Jun 17, 2021
Example of scraping a paginated API endpoint and dumping the data into a DB

Provider API Scraper Example Example of scraping a paginated API endpoint and dumping the data into a DB. Pre-requisits Python >= 3.9 Pipenv Setup # i

Alex Skobelev 1 Oct 20, 2021
Scraping web pages to get data

Scraping Data Get public data and save in database This is project use Python How to run a project 1 - Clone the repository 2 - Install beautifulsoup4

Soccer Project 2 Nov 1, 2021
A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

Alex Papadopoulos 1 Nov 13, 2021
Web Scraping Framework

Grab Framework Documentation Installation $ pip install -U grab See details about installing Grab on different platforms here http://docs.grablib.

null 2.3k Jan 4, 2023
Visual scraping for Scrapy

Portia Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web pag

Scrapinghub 8.7k Jan 5, 2023
Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapy Overview Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pag

Scrapy project 45.5k Jan 7, 2023
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Pattern Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google, Twitter, Wikipedia), web crawler, HTML DOM par

Computational Linguistics Research Group 8.4k Jan 8, 2023
Async Python 3.6+ web scraping micro-framework based on asyncio

Ruia ??️ Async Python 3.6+ web scraping micro-framework based on asyncio. ⚡ Write less, run faster. Overview Ruia is an async web scraping micro-frame

howie.hu 1.6k Jan 1, 2023