Prodmodel is a build system for data science pipelines. Users, testers, contributors are welcome!
Motivation · Concepts · Installation · Usage · Contributing · Contact · Licence
Motivation
- Performance. No need to rerun things, everything is cached, switching between multiple versions is super easy. Prodmodel can figure out if a particular partial code path has already been executed using a particular piece of data and just use the cached output.
- Easy debugging. Every single dependency - code or data - is version controlled and tracked.
- Deploy to production. Models are more than just a file. Prodmodel makes sure that the correct version of label encoders, feature transformation code and data and model files are all packaged together.
Concepts
A build system is a DAG of rules (transformations), inputs and targets. In Prodmodel inputs can be
- data,
- Python code,
- and configuration.
A rule is transforming any of the above to an output (which can in turn be depended on by other rules). Therefore rules need to be re-executed (and their outputs re-created) if any of their dependencies change. Prodmodel keeps track all of these dependencies.
The outputs of the rules are targets. Every target corresponds to an output (e.g. a model or a dataset). These outputs are cached and version controlled.
Prodmodel therefore ensures
- correctness, by executing every code (e.g. feature transformation, model building, tests) which can potentially be affected by a change, and
- performance, by executing only the necessary code, saving time compared to rerunning the whole pipeline.
Rules
Every rule is a statically typed function, where the inputs are targets, data, or configs. The execution of a rule outputs some data (e.g. a different feature set or a model), which can be used in other rules.
In order to use Prodmodel your code has to be structured as functions which the rules can call into.
Targets
Targets are created by rule functions. Targets can be executed to generate output files. IterableDataTarget is a special target which can be used as an iterable of dicts to make iterating over datasets easier. Regular DataTargets can represent any Python object.
Installation
Prodmodel requires at least Python3.6. Use pip to install prodmodel.
pip install prodmodel --user
Usage
Create a build.py file in your data science folder. The build file contains references to your inputs and the build rules you can execute.
from prodmodel.rules import rules
csv_data = rules.data_source(file='data.csv', type='csv', dtypes={...})
my_model = rules.transform(objects={'data': csv_data}, file='kmeans.py', fn='compute_kmeans')
Now you can build your model by running prodmodel my_model from the directory of build.py, or prodmodel <path_to_my_directory>:my_model from any directory.
Prodmodel creates a .prodmodel directory under the home directory of the user to store log and config files.
Documentation
Check out a complete example project for more examples.
The complete list of build rules can be found here.
Prodmodel searches for a config file under <user home dir>/.prodmodel/config. The config file can be created manually based on this template.
Arguments
--force_external: Some data sources are remote (e.g. an SQL server), therefore tracking changes is not always feasible. This argument gives the user manual control over when to reload these data sources.--cache_data: Cache local data files if changed. This can be useful for debugging / reproducibility by making sure every data source used for a specific build is saved.--output_format: One ofnone,str,bytesandlog. The output format of the data produced by the build target written to stdout.
List targets in build file
- Run
prodmodel ls <path_to_build>to list targets in a build file where<path_to_build>to the build file or its directory.
Cleaning old cache files
- Run
prodmodel clean <target> --cutoff_date=<cutoff datetime>to delete output cache files of a target created before the cutoff datetime, which has to be in%Y-%m-%dT%H:%M%S(YYYY-mm-ddTHH:MM:SS) format.
Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Contact
Feel free to email me at [email protected] if you have any question, need help or would like to contribute to the code.
2 Jan 5, 2022
791 Jan 4, 2023
66 Nov 23, 2022
18 Dec 9, 2022
30 Oct 13, 2022
2.9k Jan 6, 2023
0 Jan 26, 2022
1.7k Dec 30, 2022
1.1k Dec 28, 2022
121 Dec 28, 2022
10 Dec 1, 2021
16.2k Jan 1, 2023
102 Dec 17, 2022
2.2k Jan 9, 2023
70 Nov 18, 2022
53 Jan 2, 2023
10 Oct 22, 2022
32 Nov 27, 2022
997 Jan 6, 2023
185 Dec 20, 2022
1 Dec 12, 2021
188 Nov 24, 2022
1.1k Jan 1, 2023
401 Dec 19, 2022
11 Jul 5, 2022
7 Apr 8, 2022
1 Nov 30, 2021
5 Sep 27, 2021
6 Oct 30, 2022