Warning: the project is very much work in progress.
Valves is a collection of functions for your data .pipe()-lines.
This project aimes to host a few performant implementations of functions that are common in industry. This gives us an opportunity to share sensible implementations but it also allows us to compare the performance across libraries. For now the project mainly targets polars, pandas and dask.
Currently, the subtitle of the project lists "data pipelines built with polars".
Considering that we don't just support polars here, might it be better to call it "general functions for your data .pipe()-lines."?
If you're going to make a random sample on your data you don't always want to uniformly sample the rows. Instead, you may want to uniformly sample users and all their rows. That way, you'll have all interactions/sessions of the user in your subset.
Someone asked if we could support this: https://pandas.pydata.org/pandas-docs/stable/user_guide/window.html#window-exponentially-weighted
I haven't looked at it much, but it seems like this should be possible with some cumulative expression kung fu.
Given a log of weighted user-item interactions, can we generate a item-item recommendation table and a user-item recommendation table?
Kind of! We can calculate p(item_a | item_b) and p(item_a) which is can be reweighed into a table with recommendations. We can also do something similar for users. After all, a user that interactive with items a, b and c will have a score for item x defined via;
p(item_x | user) = p(item_x | item_a, item_b, item_c)
\propto p(item_x | item_a) p(item_x| item_b) p(item_x|item_c)
As we compare different tools here. It would be cool to run benchmarks from this repo.
Maybe in CI, and later maybe even a dedicated runner.
These can could then be shown on the website. I am already assuming here that polars does great. :smile:
@ritchie46 Do we have any preference for documentation? I could go for something like mkdocs but I figured checking in first because it wouldn't fit the current documentation style.
Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has similar Python APIs to Apache Spark or Dask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set.
Streamz helps you build pipelines to manage continuous streams of data. It is simple to use in simple cases, but also supports complex pipelines that involve branching, joining, flow control, feedback, back pressure, and so on.
logdata-anomaly-miner This tool parses log data and allows to define analysis pipelines for anomaly detection. It was designed to run the analysis wit
This project contains the process from building a web crawler to extract the raw data of house price to create ETL pipelines using Google Could Platform services.
PyFunctional Features PyFunctional makes creating data pipelines easy by using chained functional operators. Here are a few examples of what it can do
unipipeline simple way to build the declarative and distributed data pipelines. Why you should use it Declarative strict config Scaffolding Fully type
PipeChain Motivation PipeChain is a utility library for creating functional pipelines. Let's start with a motivating example. We have a list of Austra
Tablexplore is an application for data analysis and plotting built in Python using the PySide2/Qt toolkit.
Elicited Helper tools to construct probability distributions built from expert elicited data for use in monte carlo simulations. Credit to Brett Hoove
Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.
Data lineage made simple, reliable, and automated. Effortlessly track the flow of data, understand dependencies and analyze impact. Features Visualiza
TheMachineScraper ???? is a tool made purely for analysing machine data for any reason.
???? ??. The purpose of the panel-chemistry project is to make it really easy for you to do DATA ANALYSIS and build powerful DATA AND VIZ APPLICATIONS within the domain of Chemistry using using Python and HoloViz Panel.
Fast Data Science, AKA fds, is a CLI for Data Scientists to version control data and code at once, by conveniently wrapping git and dvc
A data parser for the internal syncing data format used by Fog of World. The parser is not designed to be a well-coded library with good performance, it is more like a demo for showing the data structure.
Functional Data Analysis Python package
WhiteBox Utilities Toolkit: Tools to make your life easier Fancy data functions that will make your life as a data scientist easier. Installing To ins
Processing NYC Taxi Data using PySpark ETL pipeline Description This is an project to extract, transform, and load large amount of data from NYC Taxi
Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.
791 Jan 4, 2023
1.1k Dec 28, 2022
32 Nov 27, 2022
1 Nov 22, 2021
2.1k Jan 5, 2023
0 Jan 26, 2022
2 Aug 7, 2022
81 Dec 26, 2022
3 Nov 4, 2022
3.7k Jan 3, 2023
898 Jan 9, 2023
5 Dec 1, 2022
97 Dec 8, 2022
359 Dec 22, 2022
40 Dec 12, 2022
184 Dec 27, 2022
3 Oct 3, 2022
2 Dec 12, 2021
2 Nov 20, 2021