pip3 install scrapy
pip3 install bs4
pip3 install lxml
create new topic in kafka:
bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --topic x_news_1 --bootstrap-server localhost:9092
Run consumer kafka in package: xnews.kafkaconsumer.consumer.
cd to crawler folder and run command:
scrapy crawl news
Transform application Lifecycle Data and Design and ETL pipeline architecture for ingesting data from multiple sources to redshift This project is composed of two parts: Part1 and Part2
This project contains the process from building a web crawler to extract the raw data of house price to create ETL pipelines using Google Could Platform services.
SPARK JOINS This project is to perform inner, all outer joins and semi joins. create_df.py: load_data.py : helps to put data into Spark data frames. d
Evaluate the performance of BigDL (Distributed Deep Learning on Apache Spark) in big data analysis problems.
Movies-ETL ETL pipeline on movie data using Python and postgreSQL Overview This project consisted on a automated Extraction, Transformation and Load p
DataComPy DataComPy is a package to compare two Pandas DataFrames. Originally started to be something of a replacement for SAS's PROC COMPARE for Pand
Spark app can't be debugged using normal procedure. This mini project showcase how to build and debug Apache Spark application using Python programming language. There are also options to run Spark application on Spark container
ETL Pipeline with Airflow, Spark, s3, MongoDB and Amazon Redshift
How Long Does It Take Ordinary People To "Get Good" At Chess? TL;DR: According to 5.5 years of data from 2.3 million players and 450 million games, mo
Streaming Data Pipeline - Kafka + ELK Stack Streaming weather data using Apache Kafka and Elastic Stack. Data source: https://openweathermap.org/api O
Realtime Financial Market Data Visualization and Analysis Introduction This repo shows my project about real-time stock data pipeline. All the code is
Sentiment analysis on streaming twitter data using Spark Structured Streaming & Python This project is a good starting point for those who have little
PySpark-Structured-Streaming-ROS-Kafka-ApacheSpark-Cassandra The purpose of this project is to demonstrate a structured streaming pipeline with Apache
An ETL Pipeline of a large data set from a fictitious music streaming service named Sparkify. The ETL process flows from AWS's S3 into staging tables in AWS Redshift.
Using Data Science with Machine Learning techniques (ETL pipeline and ML pipeline) to classify received messages after disasters.
SNV Pipeline SNV calling pipeline developed explicitly to process individual or trio vcf files obtained from Illumina based pipeline (grch37/grch38).
Two phase pipeline + Streamlit This is an example project that demonstrates how to create a pipeline that consists of two phases of execution. In betw
udacity-api-reporting-pipeline In this exercise, you'll use portions of each of
Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.
1 Jan 19, 2022
1 Nov 22, 2021
1 Dec 14, 2021
1 Jan 6, 2022
0 Jul 7, 2021
259 Dec 24, 2022
1 Dec 29, 2021
214 Jan 2, 2023
182 Nov 11, 2022
2 Jan 20, 2022
6 Sep 7, 2022
2 Dec 4, 2021
5 Nov 13, 2022
1 Feb 11, 2022
1 Feb 11, 2022
1 Nov 2, 2021
1 Nov 17, 2021
1 Feb 15, 2022
2 Nov 20, 2021