ETL python utilizando API do Spotify

Overview

author

Processo de ETL com Python e Airflow usando API do Spotify

Sobre

Projeto de ETL(Extract, Transform e Load) utilizando Python com API do Spotify e Airflow para automatização. Com um token gerado pela própria plataforma do Spotify é possível fazer um request com os dados das músicas que você ouviu, pegando apenas as que eu ouvi nas ultimas 24 horas, montei um processo onde manualmente ou automaticamente com o Airflow esses dados são salvos em um banco de dados SQL. O projeto foi proposto pela youtuber Inglesa Karolina Sowinska e tem como objetivo passar mesmo que de forma básica pelos processos de ETL.

Tecnologias

As seguintes ferramentas foram usadas na construção do projeto:

  • Python
  • Pandas
  • SQLAlchemy
  • Airflow

Extract

Com uma conta gratuita do Spotify, é possível gerar um token que dará acesso as musicas que você ouviu, vale notar que o token expira em algumas horas o ideal seria encontrar uma maneira de atualizar o token automaticamente.

Transform

Depois de extrair os dados, eles são transformados em um arquivo JSON, onde eu posso escolher as informações que eu desejo e coloca-las em um dicionario python. Depois eu transformo esse dicionario em um DataFrame do pandas para facilitar a manipulação.

Load

Utilizando o SQLAlchemy eu crio um banco de dados, esse banco de dados é então alimentado com os dados do DataFrame.

Validação

Há algumas validações básicas, uma que indica se o DataFrame está vazio, outra indica se alguma chave primaria está repetida e uma que checa se existe valores nulos.

Airflow

Construi apenas uma DAG simples para entender como funciona o Airflow, ela rodaria a cada dia de forma automática, executando o código de ETL.

Sobre Mim

Recém-formado em Ciências da Computação, e apaixonado por Inteligência Artificial e dados, busco oportunidade para adquirir experiência com Data Science, Machine Learning, Data Engineer e Big Data.

You might also like...
Python script to backup/convert your Spotify playlists into the XSPF format.

Python script to backup/convert your Spotify playlists into the XSPF format.

Python based Spotify account generator.
Python based Spotify account generator.

Spotify Account Generator Python based Spotify account generator. Installation Download the latest release, open command prompt in the folder, run pip

Telegram Group Calls Streaming bot with some useful features, written in Python with Pyrogram and Py-Tgcalls. Supporting platforms like Youtube, Spotify, Resso, AppleMusic, Soundcloud and M3u8 Links.
Telegram Group Calls Streaming bot with some useful features, written in Python with Pyrogram and Py-Tgcalls. Supporting platforms like Youtube, Spotify, Resso, AppleMusic, Soundcloud and M3u8 Links.

Yukki Music Bot Yukki Music Bot is a Powerful Telegram Music+Video Bot written in Python using Pyrogram and Py-Tgcalls by which you can stream songs,

Script for polybar to display and control media(not only Spotify) using DBus.
Script for polybar to display and control media(not only Spotify) using DBus.

polybar-now-playing Script for polybar to display and control media(not only Spotify) using DBus Python script to display and control current playing

RevSpotify is a fast, useful telegram bot to have Spotify music on Telegram.
RevSpotify is a fast, useful telegram bot to have Spotify music on Telegram.

RevSpotify A Telegram Bot that can download music from Spotify RevSpotify is a fast, useful telegram bot to have Spotify music on Telegram. ✨ Features

Web app for spotify playlist management with last.fm integration
Web app for spotify playlist management with last.fm integration

Music Tools Set of utility tools for Spotify and Last.fm. Built on my other libraries for Spotify (spotframework), Last.fm (fmframework) and interfaci

Converts between Spotify's new lyrics (and their proprietary format) to an LRC file for local playback.

spotify-lyrics-to-lrc Converts between Spotify's new lyrics (and their proprietary format) to an LRC file for local playback. How to use: Open Spotify

Converts a text file of songs to a playlist on your Spotify account.
Converts a text file of songs to a playlist on your Spotify account.

Playlist Converter Convert a text file of songs to a playlist on your Spotify account. Create your playlists faster instead of manually searching for

A simple program to display current playing from Spotify app on your desktop
A simple program to display current playing from Spotify app on your desktop

WallSpot A simple program to display current playing from Spotify app on your desktop How to Use: Linux: Currently Supports GNOME and KDE. If you want

Owner
Leonardo
Recém-formado em Ciências da Computação, e apaixonado por Inteligência Artificial.
Leonardo
An interactive App to play with Spotify data, both from the Spotify Web API and from CSV datasets.

An interactive App to play with Spotify data, both from the Spotify Web API and from CSV datasets.

Caio Lang 3 Jan 24, 2022
Spotify Top Lists - get the current top lists of a user from the Spotify API and display them in a Flask app

Spotify Top Lists This is a simple script that will get the current top lists of a user from the Spotify API and display them in a Flask app. Requirem

Yasin 0 Oct 16, 2022
A really easy way to display your spotify listening status on spotify.

Spotify playing README A really easy way to display your spotify listening status on READMEs and Websites too! Demo Here's the embed from the site. Cu

Sunrit Jana 21 Nov 6, 2022
Construindo API's robustas utilizando Python

?? Construindo API's robustas utilizando Python Neste tutorial vamos aprender a construir API's utilizando Python e FastAPI, integrá-las a serviços ex

luizalabs 296 Dec 13, 2022
ETL for tononkira.serasera.org

python-tononkiramalagasy-api Api Endpoints: ### get artists - /artists/<int:page> [page_offset = 20] ### get artist's songs, index was given by

Titosy Manankasina 1 Dec 24, 2021
A light weight Python library for the Spotify Web API

Spotipy A light weight Python library for the Spotify Web API Documentation Spotipy's full documentation is online at Spotipy Documentation. Installat

Paul Lamere 4.2k Jan 6, 2023
Spotify Web API client for Python 3

Welcome to the GitHub repository of Tekore! We provide a client for the Spotify Web API for Python, complete with all available endpoints and authenti

Felix Hildén 186 Dec 22, 2022
Module to use some statistics from Spotify API

statify Module to use some statistics from Spotify API To use it you have to import the functions into your own project. You have also to authenticate

Miguel Cózar 2 Jun 2, 2022
JAKYM, Just Another Konsole YouTube-Music. A command line based Youtube music player written in Python with spotify and youtube playlist support

Just Another Konsole YouTube-Music Overview I wanted to create this application so that I could use the command line to play music easily. I often pla

Mayank Jha 73 Jan 1, 2023
A Python script to update Spotify Playlist data every 5 minutes.

Spotify Playlist Updater A Python script to update Spotify Playlist data every 5 minutes. Description An automatic playlist updater using Spotify API

null 6 Nov 24, 2022