ZenML 🙏: MLOps framework to create reproducible ML pipelines for production machine learning.

Overview
Join our Slack Slack Community and become part of the ZenML family
Give us a Slack GitHub star to show your love
NEW: Vote Vote on the next ZenML features

What is ZenML?

ZenML is an extensible, open-source MLOps framework to create production-ready machine learning pipelines. It has a simple, flexible syntax, is cloud and tool agnostic, and has interfaces/abstractions that are catered towards ML workflows.

At its core, ZenML pipelines execute ML-specific workflows from sourcing data to splitting, preprocessing, training, all the way to the evaluation of results and even serving. There are many built-in batteries as things progress in ML development. ZenML is not here to replace the great tools that solve these individual problems. Rather, it integrates natively with many popular ML tooling, and gives standard abstraction to write your workflows.

Why do I need it?

Ichi Wa Zen, Zen Wa Ichi.

We built ZenML because we could not find an easy framework that translates the patterns observed in the research phase with Jupyter notebooks into a production-ready ML environment. ZenML follows the paradigm of Pipelines As Experiments (PaE), meaning ZenML pipelines are designed to be written early on the development lifecycle, where the users can explore their pipelines as they develop towards production.

By using ZenML at the early stages of development, you get the following features:

  • Reproducibility of training and inference workflows.
  • Managing ML metadata, including versioning data, code, and models.
  • Getting an overview of your ML development, with a reliable link between training and deployment.
  • Maintaining comparability between ML models.
  • Scaling ML training/inference to large datasets.
  • Retaining code quality alongside development velocity.
  • Reusing code/data and reducing waste.
  • Keeping up with the ML tooling landscape with standard abstractions and interfaces.

Who is it for?

ZenML is built for ML practitioners who are ramping up their ML workflows towards production. It is created for data science / machine learning teams that are engaged in not only training models, but also putting them out in production. Production can mean many things, but examples would be:

  • If you are using a model to generate analysis periodically for any business process.

  • If you are using models as a software service to serve predictions and are consistently improving the model over time.

  • If you are trying to understand patterns using machine learning for any business process.

  • In all of the above, there will be team that is engaged with creating, deploying, managing and improving the entire process. You always want the best results, the best models, and the most robust and reliable results. This is where ZenML can help. In terms of user persona, ZenML is created for producers of the models. This role is classically known as 'data scientist' in the industry and can range from research-minded individuals to more engineering-driven people. The goal of ZenML is to enable these practitioners to own their models until deployment and beyond.

Release 0.5.0 and what lies ahead

The current release is bare bones (as it is a complete rewrite). We are missing some basic features which used to be part of ZenML 0.3.8 (the previous release):

  • Standard interfaces for TrainingPipeline.
  • Individual step interfaces like PreprocessorStep, TrainerStep, DeployerStep etc. need to be rewritten from within the new paradigm. They should be included in the non-RC version of this release.
  • A proper production setup with an orchestrator like Airflow.
  • A post-execution workflow to analyze and inspect pipeline runs.
  • The concept of Backends will evolve into a simple mechanism of transitioning individual steps into different runners.
  • Support for KubernetesOrchestrator, KubeflowOrchestrator, GCPOrchestrator and AWSOrchestrator are also planned.
  • Dependency management including Docker support is planned.

However, bare with us: Adding those features back in should be relatively faster as we now have a solid foundation to build on. Look out for the next email!

Roadmap and Community

ZenML is being built in public. The roadmap is a regularly updated source of truth for the ZenML community to understand where the product is going in the short, medium, and long term.

ZenML is managed by a core team of developers that are responsible for making key decisions and incorporating feedback from the community. The team oversee's feedback via various channels, but you can directly influence the roadmap as follows:

Contributing

We would love to receive your contributions! Check our Contributing Guide for more details on how best to contribute.

Copyright

ZenML is distributed under the terms of the Apache License Version 2.0. A complete version of the license is available in the LICENSE.md in this repository.

Any contribution made to this project will be licensed under the Apache License Version 2.0.

Credit

ZenML is built on the shoulders of giants: we leverage, and would like to give credit to, existing open-source libraries like TFX. The goal of our framework is neither to replace these libraries, nor to diminish their usage. ZenML is simply an opinionated, higher-level interface with the focus being purely on easy-of-use and coherent intuitive design. You can read more about why we actually started building ZenML at our blog.

Legacy [Updated Soon Q4 2021]

From this point onwards, the README is intended to give a glimpse as to what lies ahead. We have redesigned our public roadmap to showcase better the timeline in which these features will be complete.

Quickstart

The quickest way to get started is to create a simple pipeline.

Step 0: Installation

ZenML is available for easy installation into your environment via PyPI:

pip install zenml

Alternatively, if you’re feeling brave, feel free to install the bleeding edge: NOTE: Do so on your own risk, no guarantees given!

pip install git+https://github.com/zenml-io/zenml.git@main --upgrade

Step 1: Initialize a ZenML repo from within a git repo

git init
zenml init

Step 2: Assemble, run, and evaluate your pipeline locally

tf.keras.Model: """A simple Keras Model to train on the data.""" model = tf.keras.Sequential() model.add(tf.keras.layers.Flatten(input_shape=(28, 28))) model.add(tf.keras.layers.Dense(10)) model.compile( optimizer=tf.keras.optimizers.Adam(0.001), loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"], ) model.fit(X_train, y_train) # write model return model @step def evaluator( X_test: np.ndarray, y_test: np.ndarray, model: tf.keras.Model, ) -> float: """Calculate the accuracy on the test set""" test_acc = model.evaluate(X_test, y_test, verbose=2) return test_acc @pipeline def mnist_pipeline( importer, trainer, evaluator, ): """Links all the steps together in a pipeline""" X_train, y_train, X_test, y_test = importer() model = trainer(X_train=X_train, y_train=y_train) evaluator(X_test=X_test, y_test=y_test, model=model) pipeline = mnist_pipeline( importer=importer(), trainer=trainer(), evaluator=evaluator(), ) pipeline.run() ">
import numpy as np
import tensorflow as tf

from zenml.pipelines import pipeline
from zenml.steps import step
from zenml.steps.step_output import Output


@step
def importer() -> Output(
    X_train=np.ndarray, y_train=np.ndarray, X_test=np.ndarray, y_test=np.ndarray
):
    """Download the MNIST data store it as numpy arrays."""
    (X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()
    return X_train, y_train, X_test, y_test


@step
def trainer(
    X_train: np.ndarray,
    y_train: np.ndarray,
) -> tf.keras.Model:
    """A simple Keras Model to train on the data."""
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Flatten(input_shape=(28, 28)))
    model.add(tf.keras.layers.Dense(10))

    model.compile(
        optimizer=tf.keras.optimizers.Adam(0.001),
        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=["accuracy"],
    )

    model.fit(X_train, y_train)

    # write model
    return model


@step
def evaluator(
    X_test: np.ndarray,
    y_test: np.ndarray,
    model: tf.keras.Model,
) -> float:
    """Calculate the accuracy on the test set"""
    test_acc = model.evaluate(X_test, y_test, verbose=2)
    return test_acc


@pipeline
def mnist_pipeline(
    importer,
    trainer,
    evaluator,
):
    """Links all the steps together in a pipeline"""
    X_train, y_train, X_test, y_test = importer()
    model = trainer(X_train=X_train, y_train=y_train)
    evaluator(X_test=X_test, y_test=y_test, model=model)

pipeline = mnist_pipeline(
    importer=importer(),
    trainer=trainer(),
    evaluator=evaluator(),
)
pipeline.run()

Leverage powerful integrations

Once code is organized into a ZenML pipeline, you can supercharge your ML development with powerful integrations and on multiple MLOps stacks.

Work locally but switch seamlessly to the cloud

Switching from local experiments to cloud-based pipelines doesn't need to be complex.

pipeline.run('airflow_gcp_stack')

Versioning galore: Use caching across experiments

ZenML makes sure for every pipeline you can trust that:

Code is versioned
Data is versioned
Models are versioned
Configurations are versioned

Use caching to help iterate quickly through ML experiments.

Automatically detect schema

# See the schema of your data [COMING SOON]
pipeline.view_schema()

Automatic schema dection

View statistics

# See statistics of train and eval [COMING SOON]
pipeline.view_statistics()

ZenML statistics visualization

Evaluate the model using built-in evaluators

# Creates a notebook for evaluation [COMING SOON]
training_pipeline.evaluate()

Tensorboard built-in

Compare training pipelines

# COMING SOON

ZenML built-in pipeline comparison

Distribute preprocessing to the cloud

Leverage distributed compute powered by Apache Beam:

# COMING SOON

ZenML distributed processing

Deploy models automatically

Automatically deploy each model with powerful Deployment integrations like Ray.

# COMING SOON

The best part is that ZenML is extensible easily, and can be molded to your use-case. You can create your own custom logic or create a PR and contribute to the ZenML community, so that everyone can benefit.

Comments
  • Upgraded pyyaml to version 6

    Upgraded pyyaml to version 6

    Describe changes

    I just upgraded pyyaml to version 6.

    Pre-requisites

    Please ensure you have done the following:

    • [X] I have read the CONTRIBUTING.md document.
    • [X] If my change requires a change to docs, I have updated the documentation accordingly.
    • [X] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [X] I have added tests to cover my changes. (No tests needed)

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [X] Other (add details above) package upgrade
    enhancement hacktoberfest-accepted 
    opened by HadiSDev 60
  • Add support for Label Studio OCR config generation

    Add support for Label Studio OCR config generation

    Describe changes

    Hi, as part of the MLOps competition, I wanted to use Label Studio for OCR labelling task. Since ZenML currently supports only generation of template for image classification and object detection labelling tasks, I implemented support for OCR to support my use case. I thought of contributing this back since it might be useful for others too.
    As a user of zenml, I felt that all the features supported by label studio should be accessible via the zenml integration as well. So I'd be happy to submit a similar PR to support other kinds of labelling tasks supported by label studio if you guys would be interested in it.

    Do let me know if the current PR requires any changes.

    However, in case, this PR is not something you guys are looking to add at the moment then do let me know. I will revoke it.

    Pre-requisites

    Please ensure you have done the following:

    • [x] I have read the CONTRIBUTING.md document.
    • [x] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [x] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    enhancement 
    opened by shivalikasingh95 39
  • Ignore columns for drift detection

    Ignore columns for drift detection

    Describe changes

    Added ignored_columns to Evidently StandardStep to filter dataframe columns, to enable drift detection on selected columns.

    • This fixes the (Issue : #600 )

    Pre-requisites

    Please ensure you have done the following:

    • [x] I have read the CONTRIBUTING.md document.
    • [x] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table.
    • [x] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [x] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)

    @strickvl Hi, Submitting another PR, on the same issue.

    enhancement 
    opened by SangamSwadiK 38
  • BentoML Deployer Integration

    BentoML Deployer Integration

    …ZenML

    Describe changes

    A raw implementation of the BentoML Deployer Integration

    Pre-requisites

    Please ensure you have done the following:

    • [ ] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • Mimiced the KServe Integration, but remodelled that to fit the BentoML Service Type
    • unit tests still to come
    hacktoberfest-accepted 
    opened by Timothy102 35
  • ZenServer

    ZenServer

    Describe changes

    • ZenML can now run as a server that can be accessed via REST API and comes with a visual user interface. This server can be deployed in arbitrary environments (local, on-prem, via Docker, on AWS / GCP / Azure / ...) and supports user management, project scoping, and more.
    • All metadata is now stored, tracked, and managed by ZenML itself.
    • Stack components can now be registered without having the required integrations installed. As part of this change, we split all existing stack component definitions into three classes: An implementation class that defines the logic of the stack component, a config class that defines the attributes and performs input validations, and a flavor class that links implementation and config classes together. See #895 for more details.
    • To further improve reproducibility, pipelines themselves are now tracked in ZenML and exposed as first-level citizens. Each pipeline clearly defines what steps are used in the pipeline and in what order the steps are executed. By default, pipeline runs are now scoped by pipeline.

    As part of these changes, the following concepts have been removed:

    • Metadata Stores,
    • Profiles,
    • Local YAML ZenStores

    New CLI commands:

    • zenml server connect / disconnect / down / up / explain / list / logs / status
    • zenml config describe / explain / set
    • zenml pipeline list / runs / delete
    • zenml profile list / migrate

    List of other PRs that have become part of this one:

    • Decouple metadata store from stack #826
    • API design for the new and improved Zen Server #831
    • Update ZenStore SQL Models #856
    • Implement ZenServer API Endpoints #859
    • Remove profiles and local zenml store #865
    • Local daemon and docker ZenML server deployments #884
    • component flavor models #895

    Pre-requisites

    Please ensure you have done the following:

    • [ ] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    breaking-change 
    opened by fa9r 34
  • Integration test examples on kubeflow pipelines

    Integration test examples on kubeflow pipelines

    Describe changes

    Running with: pytest tests/integration/test_examples.py -s --on-kubeflow --use-virtualenv

    Will now run the chosen examples on kubeflow. This should manly happen from within the github actions and is less intended for local use.

    All examples that rely on a configurable stack component get their secrets from the gh action secret manager. Some examples are still missing from this but should follow in the future (wandb_tracking, whylogs, step operator, cloud_secrets_manager).

    These tests will be run when '''LTKF!' has been commented on a PR - on success a comment is left on the PR:

    TestingQuo

    Pre-requisites

    Please ensure you have done the following:

    • [ ] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    enhancement internal 
    opened by AlexejPenner 31
  • Add nlp example

    Add nlp example

    Describe changes

    I implemented/fixed _ to achieve _.

    Pre-requisites

    Please ensure you have done the following:

    • [X] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    opened by Ankur3107 21
  • Convert `examples` to conform to best practices

    Convert `examples` to conform to best practices

    Describe changes

    I have updated some of our examples to conform to our best practices - :heavy_check_mark: is done - :hamburger: is not done and should be done in the next few days - airflow currently does not work with our best practices structure.

    :heavy_check_mark: add_your_own :heavy_check_mark: airflow_orchestration
    :heavy_check_mark: cloud_secrets_manager
    :heavy_check_mark: evidently_drift_detection
    :heavy_check_mark: facets_visualize_statistics
    :heavy_check_mark: feast_feature_store :heavy_check_mark: github_actions_orchestration :heavy_check_mark: : huggingface :heavy_check_mark: kubeflow_pipelines_orchestration
    :heavy_check_mark: lightgbm
    :heavy_check_mark: mlflow_deployment
    :heavy_check_mark: mlflow_tracking
    :heavy_check_mark: neural_prophet
    :heavy_check_mark: pytorch
    :heavy_check_mark: quickstart
    :heavy_check_mark: scipy
    :cactus: seldon_deployment - done - Need to be tested with AWS cluster where seldon is installed :heavy_check_mark: slack_alert
    :heavy_check_mark: step_operator_remote_training
    :heavy_check_mark: vertex_ai_orchestration
    :heavy_check_mark: wandb_tracking
    :heavy_check_mark: whylogs_data_profiling
    :heavy_check_mark: xgboost

    https://docs.zenml.io/links-and-resources/resources/best-practices

    Besides the obvious many changes to the example structure there have been a few changes in the core repo. The pipeline-run code has moved out of the cli into the pipelines module to be callable from code without invoking subprocess calls or cli-runners.

    On top of that the source utils have been adjusted to make sure that zenml pipeline run ... -c config.yaml works for our recommended repo structure. Special attention should be payed to the changes in src/zenml/utils/source_utils.py during code review (:eyes: @schustmi :wink: )

    Pre-requisites

    Please ensure you have done the following:

    • [ ] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    enhancement internal 
    opened by AlexejPenner 20
  • Invalidating artifact/metadata store if there is a change in one of them

    Invalidating artifact/metadata store if there is a change in one of them

    Describe changes

    I implemented a mechanism that puts a stop to any operation if the corresponding artifact store/metadata store pair is not properly associated.

    It works as follows:

    • The ZenStore now features an association table.
    • Once you register a stack, the uuids of the artifact- and metadata stores are saved in pairs in this table.
    • Stacks get validated through a validate method when you run a pipeline, register/update stacks, etc. This method now also checks whether the stack in question has the correct entry in this table for its artifact- metadata store pair.
    • It is possible to bypass this by using a flag -r when creating/updating stack over the CLI, which will reset the previous associations of these components and establish a new one.

    TODO:

    • There are some failing tests. I am working on it at the moment.

    Pre-requisites

    Please ensure you have done the following:

    • [x] I have read the CONTRIBUTING.md document.
    • [x] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [x] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    bug internal 
    opened by bcdurak 18
  • [BUG]: StepInterfaceError: Unable to find materializer for output 'output' of type `<class 'keras.engine.training.Model'>` in step 'trainer'.

    [BUG]: StepInterfaceError: Unable to find materializer for output 'output' of type `` in step 'trainer'.

    Contact Details [Optional]

    [email protected]

    System Information

    ZenML version: 0.13.1 Install path: /home//.pyenv/versions/3.7.6/lib/python3.7/site-packages/zenml Python version: 3.7.6 Platform information: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '18.04'} Environment: native Integrations: ['kubeflow', 'kubernetes', 'tensorboard', 'tensorflow']

    What happened?

    When I try to implement this code "https://github.com/zenml-io/zenml/tree/main/examples/kubeflow_pipelines_orchestration"; at the step "Run pipeline": _

    Initialize the pipeline

    first_pipeline = mnist_pipeline( importer=importer(), normalizer=normalizer(), trainer=trainer(), evaluator=evaluator(), )

    first_pipeline.run()

    _

    When I try to run the above snippet, I face StepInterfaceError

    StepInterfaceError: Unable to find materializer for output 'output' of type <class 'keras.engine.training.Model'> in step 'trainer'. Please make sure to either explicitly set a materializer for step outputs using step.with_return_materializers(...) or registering a default materializer for specific types by subclassing BaseMaterializer and setting its ASSOCIATED_TYPES class variable. For more information, visit https://docs.zenml.io/developer-guide/advanced-usage/materializer.

    Reproduction steps

    You can get the entire source code from here: https://github.com/zenml-io/zenml/tree/main/examples/kubeflow_pipelines_orchestration

    Initialize the pipeline

    first_pipeline = mnist_pipeline( importer=importer(), normalizer=normalizer(), trainer=trainer(), evaluator=evaluator(), )

    first_pipeline.run()

    Relevant log output

    Creating run for pipeline: mnist_pipeline
    Cache enabled for pipeline mnist_pipeline
    Using stack default to run pipeline mnist_pipeline...
    
    ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
    │ <ipython-input-19-3e8dc075b972>:9 in <module>                                                    │
    │                                                                                                  │
    │ /usr/local/lib/python3.7/dist-packages/zenml/pipelines/base_pipeline.py:449 in run               │
    │                                                                                                  │
    │   446 │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True                                 │
    │   447 │   │   try:                                                                               │
    │   448 │   │   │   return_value = stack.deploy_pipeline(                                          │
    │ ❱ 449 │   │   │   │   self, runtime_configuration=runtime_configuration                          │
    │   450 │   │   │   )                                                                              │
    │   451 │   │   finally:                                                                           │
    │   452 │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = False                            │
    │                                                                                                  │
    │ /usr/local/lib/python3.7/dist-packages/zenml/stack/stack.py:832 in deploy_pipeline               │
    │                                                                                                  │
    │   829 │   │   )                                                                                  │
    │   830 │   │                                                                                      │
    │   831 │   │   return_value = self.orchestrator.run(                                              │
    │ ❱ 832 │   │   │   pipeline, stack=self, runtime_configuration=runtime_configuration              │
    │   833 │   │   )                                                                                  │
    │   834 │   │                                                                                      │
    │   835 │   │   # Put pipeline level cache policy back to make sure the next runs                  │
    │                                                                                                  │
    │ /usr/local/lib/python3.7/dist-packages/zenml/orchestrators/base_orchestrator.py:250 in run       │
    │                                                                                                  │
    │   247 │   │   # Create the protobuf pipeline which will be needed for various reasons            │
    │   248 │   │   # in the following steps                                                           │
    │   249 │   │   pb2_pipeline: Pb2Pipeline = Compiler().compile(                                    │
    │ ❱ 250 │   │   │   create_tfx_pipeline(pipeline, stack=stack)                                     │
    │   251 │   │   )                                                                                  │
    │   252 │   │                                                                                      │
    │   253 │   │   self._configure_node_context(                                                      │
    │                                                                                                  │
    │ /usr/local/lib/python3.7/dist-packages/zenml/orchestrators/utils.py:50 in create_tfx_pipeline    │
    │                                                                                                  │
    │    47 │   │   │   the pipeline.                                                                  │
    │    48 │   """                                                                                    │
    │    49 │   # Connect the inputs/outputs of all steps in the pipeline                              │
    │ ❱  50 │   zenml_pipeline.connect(**zenml_pipeline.steps)                                         │
    │    51 │                                                                                          │
    │    52 │   tfx_components = {                                                                     │
    │    53 │   │   step.name: step.component for step in zenml_pipeline.steps.values()                │
    │ <ipython-input-18-3f1c2fa0b83f>:11 in mnist_pipeline                                             │
    │                                                                                                  │
    │ /usr/local/lib/python3.7/dist-packages/zenml/steps/base_step.py:707 in __call__                  │
    │                                                                                                  │
    │   704 │   │   }                                                                                  │
    │   705 │   │                                                                                      │
    │   706 │   │   # make sure we have registered materializers for each output                       │
    │ ❱ 707 │   │   materializers = self.get_materializers(ensure_complete=True)                       │
    │   708 │   │                                                                                      │
    │   709 │   │   # Prepare the output artifacts and spec                                            │
    │   710 │   │   for key, value in self.OUTPUT_SIGNATURE.items():                                   │
    │                                                                                                  │
    │ /usr/local/lib/python3.7/dist-packages/zenml/steps/base_step.py:366 in get_materializers         │
    │                                                                                                  │
    │   363 │   │   │   │   │   │   f"registering a default materializer for specific "                │
    │   364 │   │   │   │   │   │   f"types by subclassing `BaseMaterializer` and setting "            │
    │   365 │   │   │   │   │   │   f"its `ASSOCIATED_TYPES` class variable.",                         │
    │ ❱ 366 │   │   │   │   │   │   url="https://docs.zenml.io/developer-guide/advanced-usage/materi   │
    │   367 │   │   │   │   │   )                                                                      │
    │   368 │   │                                                                                      │
    │   369 │   │   return materializers                                                               │
    ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
    StepInterfaceError: Unable to find materializer for output 'output' of type `<class 'keras.engine.training.Model'>`
    in step 'trainer'. Please make sure to either explicitly set a materializer for step outputs using 
    `step.with_return_materializers(...)` or registering a default materializer for specific types by subclassing 
    `BaseMaterializer` and setting its `ASSOCIATED_TYPES` class variable. For more information, visit 
    https://docs.zenml.io/developer-guide/advanced-usage/materializer.
    

    Code of Conduct

    • [X] I agree to follow this project's Code of Conduct
    bug 
    opened by Coder-Vishali 15
  • Add exception for missing system requirements

    Add exception for missing system requirements

    Pre-requisites

    Please ensure you have done the following:

    • [x] I have read the CONTRIBUTING.md document.
    • [x] If my change requires a change to the documentation, I have updated the documentation accordingly.
    • [x] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [x] Breaking change (fix or feature that would cause existing functionality to change)

    Describe changes

    Briefly describe the changes you have introduced.

    I have modified the check_installation function and the Integration class to look for missing system requirements particularly pertaining to the graphviz integration. The new code raises an Exception during zenml init were it mentions the library whose package is missing from the system. I tried to make it raise the error during zenml example run but for that to be done the changes should be done in example cli and it was mentioned in the following issue #266 that an override for check_installation is preferable.

    opened by kamalesh0406 14
  • Add example for Hyperparameter Tuning with ZenML

    Add example for Hyperparameter Tuning with ZenML

    Describe changes

    I implemented a hyperparameter tuning pipeline example.

    Pre-requisites

    Please ensure you have done the following:

    • [x] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [x] Other (add details above)
    opened by nitay93 1
  • [BUG]: Cannot load service with unregistered service type

    [BUG]: Cannot load service with unregistered service type

    Contact Details [Optional]

    No response

    System Information

    ZenML version: 0.23.0 Python version: 3.8.13 Platform information: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'} Environment: native Integrations: ['evidently', 'mlflow', 'plotly', 'scipy', 'sklearn', 'tensorboard', 'tensorflow']

    What happened?

    I have created custom model deployer flavor and wrote all functionalities by following mlflow model deployer steps. When I run the pipeline it says. TypeError: Cannot load service with unregistered service type: type='model-serving' flavor='demo' name='demo-deployment' description='Demo detection service' I have debug to findout the reason of this error, noticed that ServiceRegistry class get_service_type() method returns none value for demo service and the error triggered. I searched about registering the service on the doc but did not find any information about it.

    Reproduction steps

    1. Prepared a demo model deployer by flollowing mlflow model deployer
    2. Prepare an api service with flask same as mlflow
    3. registered the flavor
    4. now add it to the pipline and run ...

    Relevant log output

    Updating an existing Demo deployment service: DemoDeploymentService[547692ed-8f3c-4938-b1a0-42cf73821c86] (type: model-serving, flavor: demo)
    Timed out waiting for service DemoDeploymentService[547692ed-8f3c-4938-b1a0-42cf73821c86] (type: model-serving, flavor: demo) to become active:
      Administrative state: active
      Operational state: inactive
      Last status message: 'service daemon is not running'
    For more information on the service status, please see the following log file: /home/mushfiq/.config/zenml/local_stores/634ae6dd-fa52-4871-a827-afd1a4494676/547692ed-8f3c-4938-b1a0-42cf73821c86/service.log
    
    ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
    │ /home/mushfiq/Projects/zenml-test/demo/demo_detection/run.py:21 in <module>   │
    │                                                                              │
    │   18                                                                         │
    │   19                                                                         │
    │   20 if __name__ == "__main__":                                              │
    │ ❱ 21 │   main()                                                              │
    │   22                                                                         │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/demo/demo_detection/run.py:9 in main        │
    │                                                                              │
    │    6                                                                         │
    │    7 def main():                                                             │
    │    8 │   # initialize and run the training pipeline                          │
    │ ❱  9 │   lstm_training_pipeline(                                             │
    │   10 │   │   training_data_loader=training_data_loader(),                    │
    │   11 │   │   feature_generator=feature_generator(),                          │
    │   12 │   │   pre_processor=pre_processor(),                                  │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/pip │
    │ elines/base_pipeline.py:526 in run                                           │
    │                                                                              │
    │   523 │   │   # behavior                                                     │
    │   524 │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True             │
    │   525 │   │   try:                                                           │
    │ ❱ 526 │   │   │   return_value = stack.deploy_pipeline(pipeline_deployment)  │
    │   527 │   │   finally:                                                       │
    │   528 │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = False        │
    │   529                                                                        │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/sta │
    │ ck/stack.py:677 in deploy_pipeline                                           │
    │                                                                              │
    │   674 │   │   Returns:                                                       │
    │   675 │   │   │   The return value of the call to `orchestrator.run_pipeline │
    │   676 │   │   """                                                            │
    │ ❱ 677 │   │   return self.orchestrator.run(deployment=deployment, stack=self │
    │   678 │                                                                      │
    │   679 │   def _get_active_components_for_step(                               │
    │   680 │   │   self, step_config: "StepConfiguration"                         │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
    │ hestrators/base_orchestrator.py:286 in run                                   │
    │                                                                              │
    │   283 │   │   """                                                            │
    │   284 │   │   self._prepare_run(deployment=deployment)                       │
    │   285 │   │                                                                  │
    │ ❱ 286 │   │   result = self.prepare_or_run_pipeline(                         │
    │   287 │   │   │   deployment=deployment, stack=stack                         │
    │   288 │   │   )                                                              │
    │   289                                                                        │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
    │ hestrators/local/local_orchestrator.py:76 in prepare_or_run_pipeline         │
    │                                                                              │
    │    73 │   │   │   │   │   step.config.name,                                  │
    │    74 │   │   │   │   )                                                      │
    │    75 │   │   │                                                              │
    │ ❱  76 │   │   │   self.run_step(                                             │
    │    77 │   │   │   │   step=step,                                             │
    │    78 │   │   │   )                                                          │
    │    79                                                                        │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
    │ hestrators/base_orchestrator.py:377 in run_step                              │
    │                                                                              │
    │   374 │   │   │   stack.prepare_step_run(info=step_run_info)                 │
    │   375 │   │   │   step_failed = False                                        │
    │   376 │   │   │   try:                                                       │
    │ ❱ 377 │   │   │   │   execution_info = self._execute_step(component_launcher │
    │   378 │   │   │   except:  # noqa: E722                                      │
    │   379 │   │   │   │   self._publish_failed_run(run_name_or_id=run_model.id)  │
    │   380 │   │   │   │   step_failed = True                                     │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/orc │
    │ hestrators/base_orchestrator.py:556 in _execute_step                         │
    │                                                                              │
    │   553 │   │   │   "_publish_failed_execution",                               │
    │   554 │   │   │   types.MethodType(_new_publish_failed_execution, tfx_launch │
    │   555 │   │   )                                                              │
    │ ❱ 556 │   │   execution_info = tfx_launcher.launch()                         │
    │   557 │   │   if execution_failed:                                           │
    │   558 │   │   │   raise RuntimeError(                                        │
    │   559 │   │   │   │   "Failed to execute step. This is probably because some │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
    │ stration/portable/launcher.py:549 in launch                                  │
    │                                                                              │
    │   546 │   │     self._executor_operator.with_execution_watcher(              │
    │   547 │   │   │     executor_watcher.address)                                │
    │   548 │   │     executor_watcher.start()                                     │
    │ ❱ 549 │   │   executor_output = self._run_executor(execution_info)           │
    │   550 │     except Exception as e:  # pylint: disable=broad-except           │
    │   551 │   │   execution_output = (                                           │
    │   552 │   │   │   e.executor_output if isinstance(e, _ExecutionFailedError)  │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
    │ stration/portable/launcher.py:424 in _run_executor                           │
    │                                                                              │
    │   421 │                                                                      │
    │   422 │   outputs_utils.make_output_dirs(execution_info.output_dict)         │
    │   423 │   try:                                                               │
    │ ❱ 424 │     executor_output = self._executor_operator.run_executor(execution │
    │   425 │     code = executor_output.execution_result.code                     │
    │   426 │     if code != 0:                                                    │
    │   427 │   │   result_message = executor_output.execution_result.result_messa │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
    │ stration/portable/python_executor_operator.py:135 in run_executor            │
    │                                                                              │
    │   132 │   │   pipeline_info=execution_info.pipeline_info,                    │
    │   133 │   │   pipeline_run_id=execution_info.pipeline_run_id)                │
    │   134 │   executor = self._executor_cls(context=context)                     │
    │ ❱ 135 │   return run_with_executor(execution_info, executor)                 │
    │   136                                                                        │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/tfx/orche │
    │ stration/portable/python_executor_operator.py:58 in run_with_executor        │
    │                                                                              │
    │    55 │   │   artifact.read()                                                │
    │    56                                                                        │
    │    57   output_dict = copy.deepcopy(execution_info.output_dict)              │
    │ ❱  58   result = executor.Do(execution_info.input_dict, output_dict,         │
    │    59 │   │   │   │   │      execution_info.exec_properties)                 │
    │    60   if not result:                                                       │
    │    61 │   # If result is not returned from the Do function, then try to      │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/ste │
    │ ps/utils.py:461 in Do                                                        │
    │                                                                              │
    │   458 │   │   │   step_run_info=step_run_info,                               │
    │   459 │   │   │   cache_enabled=self.configuration.enable_cache,             │
    │   460 │   │   ):                                                             │
    │ ❱ 461 │   │   │   return_values = step_function(**function_params)           │
    │   462 │   │                                                                  │
    │   463 │   │   output_annotations = parse_return_type_annotations(spec.annota │
    │   464 │   │   if len(output_annotations) > 0:                                │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/blackwidow/steps/demo_deployer.py:114 in    │
    │ demo_model_deployer_step                                                      │
    │                                                                              │
    │   111 │                                                                      │
    │   112 │   new_service = cast(                                                │
    │   113 │   │   DemoDeploymentService,                                          │
    │ ❱ 114 │   │   active_model_deployer.deploy_model(                            │
    │   115 │   │   │   replace=True,                                              │
    │   116 │   │   │   config=predictor_cfg,                                      │
    │   117 │   │   │   timeout=params.timeout,                                    │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/blackwidow/model_deployers/demo_model_deplo │
    │ yer.py:147 in deploy_model                                                   │
    │                                                                              │
    │   144 │   │   │   config.root_runtime_path = self.local_path                 │
    │   145 │   │   │   service.stop(timeout=timeout, force=True)                  │
    │   146 │   │   │   service.update(config)                                     │
    │ ❱ 147 │   │   │   service.start(timeout=timeout)                             │
    │   148 │   │   else:                                                          │
    │   149 │   │   │   # create a new DemoDeploymentService instance               │
    │   150 │   │   │   service = self._create_new_service(timeout, config)        │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/ser │
    │ vices/local/local_service.py:432 in start                                    │
    │                                                                              │
    │   429 │   │   │   │   the service status.                                    │
    │   430 │   │   """                                                            │
    │   431 │   │   if not self.config.blocking:                                   │
    │ ❱ 432 │   │   │   super().start(timeout)                                     │
    │   433 │   │   else:                                                          │
    │   434 │   │   │   self.run()                                                 │
    │   435                                                                        │
    │                                                                              │
    │ /home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/ser │
    │ vices/service.py:392 in start                                                │
    │                                                                              │
    │   389 │   │   │   self.provision()                                           │
    │   390 │   │   │   if timeout > 0:                                            │
    │   391 │   │   │   │   if not self.poll_service_status(timeout):              │
    │ ❱ 392 │   │   │   │   │   raise RuntimeError(                                │
    │   393 │   │   │   │   │   │   f"Failed to start service {self}\n"            │
    │   394 │   │   │   │   │   │   + self.get_service_status_message()            │
    │   395 │   │   │   │   │   )                                                  │
    ╰──────────────────────────────────────────────────────────────────────────────╯
    RuntimeError: Failed to start service 
    DemoDeploymentService[547692ed-8f3c-4938-b1a0-42cf73821c86] (type: model-serving,
    flavor: demo)
      Administrative state: `active`
      Operational state: `inactive`
      Last status message: 'service daemon is not running'
    For more information on the service status, please see the following log file: 
    /home/mushfiq/.config/zenml/local_stores/634ae6dd-fa52-4871-a827-afd1a4494676/54
    7692ed-8f3c-4938-b1a0-42cf73821c86/service.log
    
    
    Process finished with exit code 1
    And Traceback from log file:
    Traceback (most recent call last):
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/local/local_daemon_entrypoint.py", line 92, in run
        launch_service(config_file)
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/utils/daemon.py", line 99, in daemon
        run_as_daemon(
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/utils/daemon.py", line 270, in run_as_daemon
        daemon_function(*args, **kwargs)
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/local/local_daemon_entrypoint.py", line 84, in launch_service
        service = ServiceRegistry().load_service_from_json(config)
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/service_registry.py", line 212, in load_service_from_json
        return self.load_service_from_dict(service_dict)
      File "/home/mushfiq/Projects/zenml-test/venv/lib/python3.8/site-packages/zenml/services/service_registry.py", line 191, in load_service_from_dict
        raise TypeError(
    TypeError: Cannot load service with unregistered service type: type='model-serving' flavor='demo' name='demo-deployment' description='Demo detection service'
    Cleanup: terminating children processes...
    Cleanup: removing PID file /home/mushfiq/.config/zenml/local_stores/634ae6dd-fa52-4871-a827-afd1a4494676/547692ed-8f3c-4938-b1a0-42cf73821c86/service.pid
    

    Code of Conduct

    • [x] I agree to follow this project's Code of Conduct
    bug 
    opened by mushfiqulIslam 1
  • [BUG]: ZenML GPU scheduling

    [BUG]: ZenML GPU scheduling

    Contact Details

    [email protected]

    System Information

    ZenML version: 0.30.0
    Install path: /Users/user19/Library/Caches/pypoetry/virtualenvs/ktp-object-detection-DhOqw35J-py3.8/lib/python3.8/site-packages/zenml
    Python version: 3.8.11
    Platform information: {'os': 'mac', 'mac_version': '12.6'}
    Environment: native
    Integrations: ['kubernetes', 'mlflow', 'pillow', 'scipy', 'sklearn', 'tensorboard', 'tensorflow']
    

    What happened?

    How to schedule a specific resource such as GPU when run a ZenML pipeline on Kubernetes orchestrator?

    How to parse this k8s manifest manifest for a Pod that requests a GPU in the Kubernetes settings?

    resources:
        limits:
         gpu-vendor.example/example-gpu: 1 # requesting 1 GPU
    

    I tried to add those manifest but apparently it does not schedule the process using GPU.

    bug 
    opened by swicaksono 1
  • Test for `dashboard_utils`

    Test for `dashboard_utils`

    2 tests for one of the dashboard_utils functions.

    Pre-requisites

    Please ensure you have done the following:

    • [ ] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    internal tests 
    opened by strickvl 0
  • Tests for `network_utils`

    Tests for `network_utils`

    Added tests for the network_utils code.

    Wasn't fully sure of the best way to test this kind of network calls / scans, so happy to rework following any feedback that might come my way.

    Pre-requisites

    Please ensure you have done the following:

    • [ ] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    internal tests 
    opened by strickvl 0
  • Tests for `io_utils` and removal of duplicate code

    Tests for `io_utils` and removal of duplicate code

    Tests for the methods inside io_utils and a couple of places where I thought the original source needed small changes.

    Found some duplicate code which existed inside fileio as well as in io_utils so I removed it, leaving fileio.py as the canonical place for most of those kinds of functions.

    Pre-requisites

    Please ensure you have done the following:

    • [ ] I have read the CONTRIBUTING.md document.
    • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
    • [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
    • [ ] I have added tests to cover my changes.

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Other (add details above)
    internal tests 
    opened by strickvl 0
Releases(0.31.0)
  • 0.31.0(Dec 23, 2022)

    The highlights of this release are:

    • our Materializers have been redesigned to be more flexible and easier to use
    • we have added a new integration test framework
    • the SageMaker orchestrator has been added to our list of supported orchestrators
    • pipeline runs and artifacts can now be deleted from the ZenML database via the CLI or the Client API
    • some integrations have been updated to a more recent version: Kubeflow, Seldon Core and Tekton

    This release also includes a few bug fixes and other minor improvements to existing features.

    What's Changed

    • Fix installation instructions in readme and docs by @schustmi in https://github.com/zenml-io/zenml/pull/1167
    • Fix broken TOC for scheduling docs by @strickvl in https://github.com/zenml-io/zenml/pull/1169
    • Ensure model string fields have a max length by @strickvl in https://github.com/zenml-io/zenml/pull/1136
    • Integration test framework by @stefannica in https://github.com/zenml-io/zenml/pull/1099
    • Check if all ZenML server dependencies are installed for local zenml deployment using zenml up by @dnth in https://github.com/zenml-io/zenml/pull/1144
    • Persist the server ID in the database by @stefannica in https://github.com/zenml-io/zenml/pull/1173
    • Tiny docs improvements by @strickvl in https://github.com/zenml-io/zenml/pull/1179
    • Changing some interactions with analytics fields by @bcdurak in https://github.com/zenml-io/zenml/pull/1174
    • Fix PyTorchDataLoaderMaterializer for older torch versions by @fa9r in https://github.com/zenml-io/zenml/pull/1178
    • Redesign Materializers by @fa9r in https://github.com/zenml-io/zenml/pull/1154
    • Fixing the error messages when fetching entities by @bcdurak in https://github.com/zenml-io/zenml/pull/1171
    • Moved the active_user property onto the client, implemented get_myself as zenstore method by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1161
    • Bugfix/bump evidently version by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1183
    • Alembic migration to update size of flavor config schema by @fa9r in https://github.com/zenml-io/zenml/pull/1181
    • Deleting pipeline runs and artifacts by @fa9r in https://github.com/zenml-io/zenml/pull/1164
    • Signer email checked before setting in google cloud scheduler by @htahir1 in https://github.com/zenml-io/zenml/pull/1184
    • Fix zenml helm chart to not leak analytics events by @stefannica in https://github.com/zenml-io/zenml/pull/1190
    • Tests for dict_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1196
    • Adding exception tracking to zeml init by @bcdurak in https://github.com/zenml-io/zenml/pull/1192
    • Prevent crashes during Airflow server forking on MacOS by @schustmi in https://github.com/zenml-io/zenml/pull/1186
    • add alpha as server deployment type by @wjayesh in https://github.com/zenml-io/zenml/pull/1197
    • Bugfix for custom flavor registration by @bcdurak in https://github.com/zenml-io/zenml/pull/1195
    • Tests for uuid_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1200
    • Sagemaker orchestrator integration by @strickvl in https://github.com/zenml-io/zenml/pull/1177
    • Fix Pandas Materializer Index by @safoinme in https://github.com/zenml-io/zenml/pull/1193
    • Add support for deploying custom stack recipes using the ZenML CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1188
    • Add cloud CI environments by @stefannica in https://github.com/zenml-io/zenml/pull/1176
    • Fix project scoping for artifact list through ZenServer by @fa9r in https://github.com/zenml-io/zenml/pull/1203
    Source code(tar.gz)
    Source code(zip)
  • 0.30.0(Dec 9, 2022)

    In this release, ZenML finally adds Mac M1 support, Python 3.10 support and much greater flexibility and configurability under the hood by deprecating some large dependencies like ml-pipelines-sdk.

    Scheduling

    Based on some community feedback around scheduling, this release comes with improved docs concerning scheduling in general. Additionally, the Vertex AI orchestrator now also supports scheduling.

    Slimmer Dependencies

    By removing dependencies on some of the packages that ZenML was built on, this version of ZenML is slimmer, faster and more configurable than ever. This also finally makes ZenML run natively on Macs with M1 processors without the need for Rosetta. This also finally enables ZenML to run on Python 3.10.

    Breaking Changes

    • The removal of ml-pipelines-sdk and tfx leads to some larger changes in the database that is tracking your pipeline runs and artifacts. Note: There is an automatic migration to upgrade this automatically, However, please note that downgrading back down to 0.23.0 is not supported.
    • The CLI commands to export and import pipeline runs have been deprecated. Namely: zenml pipeline runs export and zenml pipeline runs import These commands were meant for migrating from zenml<0.20.0 to 0.20.0<=zenml<0.30.0.
    • The azure-ml integration dependency on azureml-core has been upgraded from 1.42 to 1.48

    What's Changed

    • Remove stack extra from installation, enable re-running the quickstart by @schustmi in https://github.com/zenml-io/zenml/pull/1133
    • Secrets manager support to experiment trackers docs by @safoinme in https://github.com/zenml-io/zenml/pull/1137
    • Updating the README files of our examples by @bcdurak in https://github.com/zenml-io/zenml/pull/1128
    • Prevent running with local ZenStore and remote code execution by @schustmi in https://github.com/zenml-io/zenml/pull/1134
    • Remove ml-pipelines-sdk dependency by @schustmi in https://github.com/zenml-io/zenml/pull/1103
    • Fix Huggingface dataset materializer by @safoinme in https://github.com/zenml-io/zenml/pull/1142
    • Disallow alembic downgrades for 0.30.0 release by @fa9r in https://github.com/zenml-io/zenml/pull/1140
    • Fix Client flavor-related methods by @schustmi in https://github.com/zenml-io/zenml/pull/1153
    • Replace User Password with Token in docker images by @safoinme in https://github.com/zenml-io/zenml/pull/1147
    • Remove zenml pipeline runs export / import CLI commands by @fa9r in https://github.com/zenml-io/zenml/pull/1150
    • Context manager to track events by @bcdurak in https://github.com/zenml-io/zenml/pull/1149
    • Made explicit is not None calls to allow for empty pwd again by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1159
    • Add Neptune exp tracker into flavors table by @dnth in https://github.com/zenml-io/zenml/pull/1156
    • Fix step operators by @schustmi in https://github.com/zenml-io/zenml/pull/1155
    • Display correct name when updating a stack component by @schustmi in https://github.com/zenml-io/zenml/pull/1160
    • Update mysql database creation by @schustmi in https://github.com/zenml-io/zenml/pull/1152
    • Adding component conditions to experiment tracker examples and adding to the environmental variable docs by @bcdurak in https://github.com/zenml-io/zenml/pull/1162
    • Increase dependency range for protobuf by @schustmi in https://github.com/zenml-io/zenml/pull/1163
    • Scheduling documentation by @strickvl in https://github.com/zenml-io/zenml/pull/1158
    • Adding scheduling for Vertex Pipelines by @htahir1 in https://github.com/zenml-io/zenml/pull/1148
    • Fix alembic migration for sqlite<3.25 by @fa9r in https://github.com/zenml-io/zenml/pull/1165
    • Fix pandas Series materializer by @jordandelbar in https://github.com/zenml-io/zenml/pull/1146

    New Contributors

    • @jordandelbar made their first contribution in https://github.com/zenml-io/zenml/pull/1146
    Source code(tar.gz)
    Source code(zip)
  • 0.30.0rc3(Dec 8, 2022)

    This pre-release builds on top of 0.30.0rc2.

    Major changes:

    • Improved database migration from previous versions
    • Fixes for step operators

    Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

    Source code(tar.gz)
    Source code(zip)
  • 0.30.0rc2(Dec 6, 2022)

    This pre-release builds on top of 0.30.0rc1.

    Major changes:

    • Docker images are now built for python 3.10
    • The database tables for pipeline and step runs are cleaned up.

    Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

    Source code(tar.gz)
    Source code(zip)
  • 0.30.0rc1(Dec 5, 2022)

    This pre-release builds on top of 0.30.0rc0, specifically with a few bug fixes including fixing the quickstart

    Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

    Source code(tar.gz)
    Source code(zip)
  • 0.30.0rc0(Dec 2, 2022)

    This pre-release removes the ml-pipeline-sdk dependency and adds support for Python 3.10 as well as M1 macs.

    Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

    Source code(tar.gz)
    Source code(zip)
  • 0.23.0(Dec 2, 2022)

    This release comes with a brand-new Neptune integration to track your ML experiments

    as well as lots of performance improvements!

    Neptune integration

    The new Neptune integration includes a Neptune experiment tracker component that allows you to track your machine learning experiments using Neptune.

    Performance Optimization

    The 0.20.0 release introduced our new server but brought with it a few performance and scalability issues. Since then we've made many improvements to it, and this release is the final and biggest boost in performance. We reduced the amount of server calls needed for almost all CLI commands and greatly improved the speed of the dashboard as well.

    PyArrow dependency removal

    We've removed PyArrow as a dependency of the zenml python package. As a consequence of that, our NumPy and Pandas materializer no longer read and write their artifacts using PyArrow but instead use native formats instead. If you still want to use PyArrow to serialize your NumPy arrays and Pandas dataframes, you'll need to install it manually like this: pip install pyarrow

    In future releases we'll get rid of other unnecessary dependencies to further slim down the zenml package.

    Breaking Changes

    The following changes introduces with this release mey require some manual intervention to update your current installations:

    • If your code calls some methods of our Client class, it might need to be updated to the new model classes introduced by the performance optimization changes explained above
    • The CLI command to remove an attribute from a stack component now takes no more dashes in front of the attribute names: zenml stack-component remove-attribute <COMPONENT_NAME> <ATTRIBUTE_NAME>
    • If you're using a custom stack component and have overridden the cleanup_step_run method, you'll need to update the method signature to include a step_failed parameter.

    What's Changed

    • Docs regarding roles and permissions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1081
    • Add global config dir to zenml status by @schustmi in https://github.com/zenml-io/zenml/pull/1084
    • Remove source pins and ignore source pins during step spec comparisons by @schustmi in https://github.com/zenml-io/zenml/pull/1083
    • Docs/links for roles permissions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1091
    • Bugfix/eng 1485 fix api docs build by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1089
    • fix bento builder step parameters to match bentoml by @safoinme in https://github.com/zenml-io/zenml/pull/1096
    • Add bentoctl to BentoML docs and example by @safoinme in https://github.com/zenml-io/zenml/pull/1094
    • Fix BaseParameters sample code in docs by @jcarlosgarcia in https://github.com/zenml-io/zenml/pull/1098
    • zenml logs defaults to active stack without name_or_id by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1101
    • Fixed evidently docs by @htahir1 in https://github.com/zenml-io/zenml/pull/1111
    • Update sagemaker default instance type by @schustmi in https://github.com/zenml-io/zenml/pull/1112
    • The ultimate optimization for performance by @bcdurak in https://github.com/zenml-io/zenml/pull/1077
    • Update stack exporting and importing by @schustmi in https://github.com/zenml-io/zenml/pull/1114
    • Fix readme by @schustmi in https://github.com/zenml-io/zenml/pull/1116
    • Remove Pyarrow dependency by @safoinme in https://github.com/zenml-io/zenml/pull/1109
    • Bugfix for listing the runs filtered by a name by @bcdurak in https://github.com/zenml-io/zenml/pull/1118
    • Neptune.ai integration by @AleksanderWWW in https://github.com/zenml-io/zenml/pull/1082
    • Add YouTube video explaining Stack Components Settings vs Config by @dnth in https://github.com/zenml-io/zenml/pull/1120
    • Add failed Status to component when step fails by @safoinme in https://github.com/zenml-io/zenml/pull/1115
    • Add architecture diagrams to docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1119
    • Remove local orchestrator restriction from step operator docs by @schustmi in https://github.com/zenml-io/zenml/pull/1122
    • Validate Stack Before Provision by @safoinme in https://github.com/zenml-io/zenml/pull/1110
    • Bugfix/fix endpoints for dashboard development by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1125
    • Skip kubeflow UI daemon provisioning if a hostname is configured by @schustmi in https://github.com/zenml-io/zenml/pull/1126
    • Update Neptune Example by @safoinme in https://github.com/zenml-io/zenml/pull/1124
    • Add debugging guide to docs by @dnth in https://github.com/zenml-io/zenml/pull/1097
    • Fix stack component attribute removal CLI command by @schustmi in https://github.com/zenml-io/zenml/pull/1127
    • Improving error messages when fetching entitites by @bcdurak in https://github.com/zenml-io/zenml/pull/1117
    • Introduce username and password to kubeflow for more native multi-tenant support by @htahir1 in https://github.com/zenml-io/zenml/pull/1123
    • Add support for Label Studio OCR config generation by @shivalikasingh95 in https://github.com/zenml-io/zenml/pull/1062
    • Misc doc updates by @schustmi in https://github.com/zenml-io/zenml/pull/1131
    • Fix Neptune run cleanup by @safoinme in https://github.com/zenml-io/zenml/pull/1130

    New Contributors

    • @jcarlosgarcia made their first contribution in https://github.com/zenml-io/zenml/pull/1098
    • @AleksanderWWW made their first contribution in https://github.com/zenml-io/zenml/pull/1082
    • @shivalikasingh95 made their first contribution in https://github.com/zenml-io/zenml/pull/1062

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.22.0...0.23.0

    Source code(tar.gz)
    Source code(zip)
  • 0.22.0(Nov 18, 2022)

    This release comes with a new BentoML integration as well as a reworked Airflow orchestrator. Additionally, it greatly improves the server performance as well as other small fixes and updates to our docs!

    BentoML integration

    The new BentoML integration includes a BentoML model deployer component that allows you to deploy your models from any of the major machine learning frameworks on your local machine.

    See example here.

    Airflow orchestrator v2

    The previous Airflow orchestrator was limited to running locally and had many additional unpleasant constraints that made it hard to work with. This release includes a completely rewritten, new version of the Airflow orchestrator that now relies on Docker images to run your pipelines and works both locally and with remote Airflow deployments.

    See what changed in this video and check out the brand-new example here.

    Notable bugfixes

    • Further improvements to the synchronization that transfers pipeline run information from the MLMD database to the ZenML Server.
    • The ZenML Label Studio integration can now be used with non-local (i.e. deployed) instances. For more information see the Label Studiodocs.
    • The Spark example is fixed and now works again end-to-end.

    Breaking Changes

    The following changes introduces with this release mey require some manual intervention to update your current installations:

    • the Airflow orchestrator now requires a newer version of Airflow (run zenml integration install airflow to upgrade) and Docker installed to work.

    What's Changed

    • Fix bug when running non-local annotator instance. by @sheikhomar in https://github.com/zenml-io/zenml/pull/1045
    • Introduce Permissions, Link Permissions to Roles, Restrict Access to endpoints based on Permission by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1007
    • Fix copy-pasted log message for annotator by @strickvl in https://github.com/zenml-io/zenml/pull/1049
    • Add warning message for client server version mismatch by @schustmi in https://github.com/zenml-io/zenml/pull/1047
    • Fix path to ingress values in ZenServer recipes by @wjayesh in https://github.com/zenml-io/zenml/pull/1053
    • Prevent deletion/update of default entities by @stefannica in https://github.com/zenml-io/zenml/pull/1046
    • Fix Publish API docs workflow by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1054
    • Fix multiple alembic heads warning by @fa9r in https://github.com/zenml-io/zenml/pull/1051
    • Fix Null Step Configuration/Parameters Error by @fa9r in https://github.com/zenml-io/zenml/pull/1050
    • Fix role permission migration by @schustmi in https://github.com/zenml-io/zenml/pull/1056
    • Made role assignment/revokation possible through zen_server by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1059
    • Bugfix/make role assignment work with enum by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1063
    • Manually set scoped for each endpoint by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1064
    • Add run args to local docker orchestrator settings by @schustmi in https://github.com/zenml-io/zenml/pull/1060
    • Docker ZenML deployment improvements and docs by @stefannica in https://github.com/zenml-io/zenml/pull/1061
    • Bugfix Mlflow service cleanup configuration by @safoinme in https://github.com/zenml-io/zenml/pull/1067
    • Rename DB Tables and Fix Foreign Keys by @fa9r in https://github.com/zenml-io/zenml/pull/1058
    • Paginate secrets in AWSSecretsManager by @chiragjn in https://github.com/zenml-io/zenml/pull/1057
    • Add explicit dashboard docs by @strickvl in https://github.com/zenml-io/zenml/pull/1052
    • Added GA and Gitlab to envs by @htahir1 in https://github.com/zenml-io/zenml/pull/1068
    • Add Inference Server Predictor to KServe and Seldon Docs by @safoinme in https://github.com/zenml-io/zenml/pull/1048
    • Rename project table to workspace by @fa9r in https://github.com/zenml-io/zenml/pull/1073
    • Airflow orchestrator v2 by @schustmi in https://github.com/zenml-io/zenml/pull/1042
    • Add get_or_create_run() ZenStore method by @fa9r in https://github.com/zenml-io/zenml/pull/1070
    • Fix the flaky fileio tests by @schustmi in https://github.com/zenml-io/zenml/pull/1072
    • BentoML Deployer Integration by @safoinme in https://github.com/zenml-io/zenml/pull/1044
    • Sync Speedup by @fa9r in https://github.com/zenml-io/zenml/pull/1055
    • Fixed broken links in docs and examples. by @dnth in https://github.com/zenml-io/zenml/pull/1076
    • Make additional stack component config options available as a setting by @schustmi in https://github.com/zenml-io/zenml/pull/1069
    • Rename step_run_artifact table to step_run_input_artifact by @fa9r in https://github.com/zenml-io/zenml/pull/1075
    • Update Spark Example to ZenML post 0.20.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1071
    • Always set caching to false for all Kubeflow based orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/1079
    • Feature/eng 1402 consolidate stack sharing by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1036

    New Contributors

    • @sheikhomar made their first contribution in https://github.com/zenml-io/zenml/pull/1045
    • @chiragjn made their first contribution in https://github.com/zenml-io/zenml/pull/1057

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.21.1...0.22.0

    Source code(tar.gz)
    Source code(zip)
  • 0.21.1(Nov 4, 2022)

    This is an ad-hoc release to fix some bugs introduced the 0.21.0 release that made the local ZenML dashboard unusable.

    What's Changed

    • Include latest (not oldest) three runs in HydratedPipelineModel by @schustmi in https://github.com/zenml-io/zenml/pull/1039
    • Update docs to use pip install [server] by @strickvl in https://github.com/zenml-io/zenml/pull/1037
    • Docs fix for Deepchecks by @strickvl in https://github.com/zenml-io/zenml/pull/1040
    • Fix the pipeline run sync on sqlite and the --blocking zenml server deployment by @stefannica in https://github.com/zenml-io/zenml/pull/1041

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.21.0...0.21.1

    Source code(tar.gz)
    Source code(zip)
  • 0.21.0(Nov 3, 2022)

    This release primarily fixes a number of bugs that were introduced as part of the 0.20.0 ZenServer release. These significantly improve the stability when using ZenML with the ZenML Server.

    Notable fixes include:

    • Improved the synchronization that transfers pipeline run information from the MLMD database to the ZenML Server. This helps fix a number of issues with missing steps in the post-execution workflow, model deployment steps and other issues.
    • The Label Studio example is fixed and now works again end-to-end.
    • The ZenML Label Studio integration can now be used with non-local (i.e. deployed) instances. For more information see the Label Studiodocs.

    New features and other improvements:

    • ZenML now uses alembic for automated database migrations. The migrations happen automatically after every ZenML update.
    • New zenml pipeline runs export / import / migrate CLI commands are now available to export, import and migrate pipeline runs from older, pre-0.20.0 versions of ZenML. The ZenML server now also automatically picks up older pipeline runs that have been logged in the metadata store by ZenML prior to 0.20.0.
    • An MLMD gRPC service can now be deployed with the ZenML Helm chart to act as a proxy between clients, orchestrators and the MySQL database. This significantly reduces the time it takes to run pipelines locally.
    • You can now specify affinity and tolerations and node selectors to all Kubernetes based orchestrators with the new Kubernetes Pod settings feature.

    Breaking Changes

    The following changes introduces with this release mey require some manual intervention to update your current installations:

    • the zenml server helm chart values.yaml file has been restructured to make it easier to configure and to clearly distinguish between the zenml server component and the newly introduced gRPC service component. Please update your values.yaml copies accordingly.
    • the Azure integration dependency versions have been updated. Please run zenml integration install azure to update your current installation, if you're using Azure.

    What's Changed

    • Implement automatic alembic migration by @AlexejPenner in https://github.com/zenml-io/zenml/pull/990
    • Fix GCP Artifact Store listdir empty path by @safoinme in https://github.com/zenml-io/zenml/pull/998
    • Add flavors mini-video to docs by @strickvl in https://github.com/zenml-io/zenml/pull/999
    • Remove the Client() warning when used inside a step by @stefannica in https://github.com/zenml-io/zenml/pull/1000
    • Fix broken links caused by updated by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1002
    • Fix FileNotFoundError with remote path in HuggingFace Dataset materializer by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/995
    • Add zenml pipeline runs export / import / migrate CLI commands by @fa9r in https://github.com/zenml-io/zenml/pull/977
    • Log message when activating a stack as part of registration by @schustmi in https://github.com/zenml-io/zenml/pull/1005
    • Minor fixes in Migration to 0.20.0 documentation by @alvarobartt in https://github.com/zenml-io/zenml/pull/1009
    • Doc updates by @htahir1 in https://github.com/zenml-io/zenml/pull/1006
    • Fixing broken links in docs by @dnth in https://github.com/zenml-io/zenml/pull/1018
    • Label Studio example fix by @strickvl in https://github.com/zenml-io/zenml/pull/1021
    • Docs for using CUDA-enabled docker images by @strickvl in https://github.com/zenml-io/zenml/pull/1010
    • Add social media heading on docs page by @dnth in https://github.com/zenml-io/zenml/pull/1020
    • Add executing custom command for getting requirements by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/1012
    • Delay user instruction in dockerfile generation by @schustmi in https://github.com/zenml-io/zenml/pull/1004
    • Update link checker configs for faster, more accurate checks by @dnth in https://github.com/zenml-io/zenml/pull/1022
    • Add pip install zenml[server] to relevant examples by @dnth in https://github.com/zenml-io/zenml/pull/1027
    • Add Tolerations and NodeAffinity to Kubernetes executor by @wefner in https://github.com/zenml-io/zenml/pull/994
    • Support pydantic subclasses in BaseParameter attributes by @schustmi in https://github.com/zenml-io/zenml/pull/1023
    • Unify run names across orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/1025
    • Add gRPC metadata service to the ZenML helm chart by @stefannica in https://github.com/zenml-io/zenml/pull/1026
    • Make the MLMD pipeline run information transfer synchronous by @stefannica in https://github.com/zenml-io/zenml/pull/1032
    • Add console spinner back by @strickvl in https://github.com/zenml-io/zenml/pull/1034
    • Fix Azure CLI auth problem by @wjayesh in https://github.com/zenml-io/zenml/pull/1035
    • Allow non-local Label Studio instances for annotation by @strickvl in https://github.com/zenml-io/zenml/pull/1033
    • Before deleting the global zen_server files, spin it down by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1029
    • Adding zenserver integration to stack recipe CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1017
    • Add support for Azure ZenServer by @wjayesh in https://github.com/zenml-io/zenml/pull/1024
    • Kubernetes Pod settings by @schustmi in https://github.com/zenml-io/zenml/pull/1008

    New Contributors

    • @alvarobartt made their first contribution in https://github.com/zenml-io/zenml/pull/1009
    • @wefner made their first contribution in https://github.com/zenml-io/zenml/pull/994

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.5...0.21.0

    Source code(tar.gz)
    Source code(zip)
  • 0.20.5(Oct 21, 2022)

    ZenML 0.20.5 fixes another series of minor bugs, significantly improves the performance of the CLI, and adds an option to specify APT packages in Docker images.

    What's Changed

    • Fix accessing local zen store and artifact store in containers by @stefannica in https://github.com/zenml-io/zenml/pull/976
    • K3d local registry pod spec updated by @wjayesh in https://github.com/zenml-io/zenml/pull/972
    • Update readme page by @dnth in https://github.com/zenml-io/zenml/pull/985
    • Remove beam dependency by @schustmi in https://github.com/zenml-io/zenml/pull/986
    • Fix error message when registering secret without secrets manager by @schustmi in https://github.com/zenml-io/zenml/pull/981
    • Update cheat sheet up to zenml==0.20.4 by @dnth in https://github.com/zenml-io/zenml/pull/987
    • Example fixes (part 2) by @strickvl in https://github.com/zenml-io/zenml/pull/971
    • Allow duplicate step classes inside a pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/989
    • Include deployment in azureml docker build by @schustmi in https://github.com/zenml-io/zenml/pull/984
    • Automatically open browser upon zenml up command by @dnth in https://github.com/zenml-io/zenml/pull/978
    • Add a just_mine flag for zenml stack list by @strickvl in https://github.com/zenml-io/zenml/pull/979
    • Add option to specify apt packages by @schustmi in https://github.com/zenml-io/zenml/pull/982
    • Replace old flavor references, fix the windows local ZenML server and other fixes by @stefannica in https://github.com/zenml-io/zenml/pull/988
    • Improve docker and k8s detection by @schustmi in https://github.com/zenml-io/zenml/pull/991
    • Update GH actions example by @schustmi in https://github.com/zenml-io/zenml/pull/993
    • Update MissingStepParameterError exception message by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/996
    • Seprated code docs into core and integration docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/983
    • Add docs/mkdocstrings_helper.py to format script sources by @fa9r in https://github.com/zenml-io/zenml/pull/997
    • Further CLI optimization by @bcdurak in https://github.com/zenml-io/zenml/pull/992

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.4...0.20.5

    Source code(tar.gz)
    Source code(zip)
  • 0.20.4(Oct 15, 2022)

    This release fixes another series of minor bugs that were introduced in 0.20.0.

    What's Changed

    • Detect failed executions by @schustmi in https://github.com/zenml-io/zenml/pull/964
    • Only build docker images for custom deployments by @schustmi in https://github.com/zenml-io/zenml/pull/960
    • M1 Mac Installation Tutorial by @fa9r in https://github.com/zenml-io/zenml/pull/966
    • Update ZenBytes links in docs by @fa9r in https://github.com/zenml-io/zenml/pull/968
    • Fix the API docs builder by @stefannica in https://github.com/zenml-io/zenml/pull/967
    • Fix gpu_limit condition in VertexOrchestrator by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/963
    • Add simple node affinitiy configurations by @schustmi in https://github.com/zenml-io/zenml/pull/973
    • First iteration of the CLI optimization by @bcdurak in https://github.com/zenml-io/zenml/pull/962

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.3...0.20.4

    Source code(tar.gz)
    Source code(zip)
  • 0.20.3(Oct 12, 2022)

    This release fixes another series of minor bugs that were introduced in 0.20.0 and add full support docs for Kubeflow multi-tenancy deployments.

    What's Changed

    • Fixed GitHub/Colab JSON formatting error on quickstart. by @fa9r in https://github.com/zenml-io/zenml/pull/947
    • Update YAML config template by @htahir1 in https://github.com/zenml-io/zenml/pull/952
    • correct code from merge and fix import by @wjayesh in https://github.com/zenml-io/zenml/pull/950
    • Check for active component using id instead of name by @schustmi in https://github.com/zenml-io/zenml/pull/956
    • Tekton fix by @htahir1 in https://github.com/zenml-io/zenml/pull/955
    • Improve zenml up/down UX and other fixes by @stefannica in https://github.com/zenml-io/zenml/pull/957
    • Update kubeflow docs for multi-tenant deployments by @htahir1 in https://github.com/zenml-io/zenml/pull/958
    • Update kubeflow.md by @abohmeed in https://github.com/zenml-io/zenml/pull/959
    • Add additional stack validation for step operators by @schustmi in https://github.com/zenml-io/zenml/pull/954
    • Fix pipeline run dashboard URL for unlisted runs by @fa9r in https://github.com/zenml-io/zenml/pull/951
    • Support subclasses of registered types in recursive materialization by @fa9r in https://github.com/zenml-io/zenml/pull/953

    New Contributors

    • @abohmeed made their first contribution in https://github.com/zenml-io/zenml/pull/959

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.2...0.20.3

    Source code(tar.gz)
    Source code(zip)
  • 0.20.2(Oct 7, 2022)

    After a successful release of the new ZenML server and dashboard paradigm, we set to ironing out some bugs that slipped through.

    What's Changed

    • Capitalize all docs page titles. by @fa9r in https://github.com/zenml-io/zenml/pull/937
    • Increase field sizes for docstrings and step parameters. by @fa9r in https://github.com/zenml-io/zenml/pull/940
    • Fixing the bug in the registration of custom flavors by @bcdurak in https://github.com/zenml-io/zenml/pull/938
    • Implemented docstring Attribute of StepModel by @fa9r in https://github.com/zenml-io/zenml/pull/936
    • Fix shared stack emoji by @strickvl in https://github.com/zenml-io/zenml/pull/941
    • Fix shared stacks not being allowed to be set as active. by @fa9r in https://github.com/zenml-io/zenml/pull/943
    • Typo fix by @strickvl in https://github.com/zenml-io/zenml/pull/944
    • Update Kubernetes Orchestrator Example by @fa9r in https://github.com/zenml-io/zenml/pull/942
    • Add code and instructions to run quickstart on Colab. by @fa9r in https://github.com/zenml-io/zenml/pull/939
    • Fixing the interaction in getting stacks/components by @bcdurak in https://github.com/zenml-io/zenml/pull/945
    • Fix Kubeflow run name by @safoinme in https://github.com/zenml-io/zenml/pull/946
    • VertexOrchestrator apply node selector constraint if gpu_limit > 0 by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/935

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.1...0.20.2

    Source code(tar.gz)
    Source code(zip)
  • 0.20.0(Oct 5, 2022)

    The ZenML 0.20.0 release brings a number of big changes to its architecture and a lot of cool new features, some of which are not backwards compatible with previous versions.

    These changes are only covered briefly in the release notes. For a detailed view on what happened and how you can get the most out of the 0.20.0 release, please head over to our "ZenML 0.20.0: Our Biggest Release Yet" blog post.

    Warning: Breaking Changes

    Updating to ZenML 0.20.0 needs to be followed by a migration of your existing ZenML Stacks and you may also need to make changes to your current ZenML pipeline code. Please read the migration guide carefully and follow the instructions to ensure a smooth transition. The guide walks you through these changes and offers instructions on how to migrate your existing ZenML stacks and pipelines to the new version with minimal effort and disruption to your existing workloads.

    If you have updated to ZenML 0.20.0 by mistake or are experiencing issues with the new version, you can always go back to the previous version by using pip install zenml==0.13.2 instead of pip install zenml when installing ZenML manually or in your scripts.

    Overview of Changes

    What's Changed

    • Fix error in checking Great Expectations results when exit_on_error=True by @TimovNiedek in https://github.com/zenml-io/zenml/pull/889
    • feat(user-dockerfile): Add user argument to DockerConfiguration by @cjidboon94 in https://github.com/zenml-io/zenml/pull/892
    • Minor doc updates for backporting by @htahir1 in https://github.com/zenml-io/zenml/pull/894
    • Removed feature request and replaced with hellonext board by @htahir1 in https://github.com/zenml-io/zenml/pull/897
    • Unit tests for (some) integrations by @strickvl in https://github.com/zenml-io/zenml/pull/880
    • Fixed integration installation command by @edshee in https://github.com/zenml-io/zenml/pull/900
    • Pipeline configuration and intermediate representation by @schustmi in https://github.com/zenml-io/zenml/pull/898
    • [Bugfix] Fix bug in auto-import of stack after recipe deploy by @wjayesh in https://github.com/zenml-io/zenml/pull/901
    • Update TOC on CONTRIBUTING.md by @strickvl in https://github.com/zenml-io/zenml/pull/907
    • ZenServer by @fa9r in https://github.com/zenml-io/zenml/pull/879
    • Update kserve README by @strickvl in https://github.com/zenml-io/zenml/pull/912
    • Confirmation prompts were not working by @htahir1 in https://github.com/zenml-io/zenml/pull/917
    • Stacks can be registered in Click<8.0.0 now by @AlexejPenner in https://github.com/zenml-io/zenml/pull/920
    • Made Pipeline and Stack optional on the HydratedPipelineRunModel by @AlexejPenner in https://github.com/zenml-io/zenml/pull/919
    • Renamed all references from ZenServer to ZenML Server in logs and comments by @htahir1 in https://github.com/zenml-io/zenml/pull/915
    • Prettify pipeline runs list CLI output. by @fa9r in https://github.com/zenml-io/zenml/pull/921
    • Warn when registering non-local component with local ZenServer by @strickvl in https://github.com/zenml-io/zenml/pull/904
    • Fix duplicate results in pipeline run lists and unlisted flag. by @fa9r in https://github.com/zenml-io/zenml/pull/922
    • Fix error log by @htahir1 in https://github.com/zenml-io/zenml/pull/916
    • Update cli docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/913
    • Fix Pipeline Run Status by @fa9r in https://github.com/zenml-io/zenml/pull/923
    • Change the CLI emoji for whether a stack is shared or not. by @fa9r in https://github.com/zenml-io/zenml/pull/926
    • Fix running pipelines from different locations. by @fa9r in https://github.com/zenml-io/zenml/pull/925
    • Fix zenml stack-component describe CLI command. by @fa9r in https://github.com/zenml-io/zenml/pull/929
    • Update custom deployment to use ArtifactModel by @safoinme in https://github.com/zenml-io/zenml/pull/928
    • Fix the CI unit test and integration test failures by @stefannica in https://github.com/zenml-io/zenml/pull/924
    • Add gcp zenserver recipe by @wjayesh in https://github.com/zenml-io/zenml/pull/930
    • Extend Post Execution Class Properties by @fa9r in https://github.com/zenml-io/zenml/pull/931
    • Fixes for examples by @strickvl in https://github.com/zenml-io/zenml/pull/918
    • Update cheat sheet by @dnth in https://github.com/zenml-io/zenml/pull/932
    • Fix the docstring attribute of pipeline models. by @fa9r in https://github.com/zenml-io/zenml/pull/933
    • New docs post ZenML Server by @htahir1 in https://github.com/zenml-io/zenml/pull/927

    New Contributors

    • @TimovNiedek made their first contribution in https://github.com/zenml-io/zenml/pull/889
    • @cjidboon94 made their first contribution in https://github.com/zenml-io/zenml/pull/892
    • @edshee made their first contribution in https://github.com/zenml-io/zenml/pull/900

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.2...0.20.0

    Source code(tar.gz)
    Source code(zip)
  • 0.13.2(Sep 9, 2022)

    ZenML 0.13.2 comes with a new local Docker orchestrator and many other improvements and fixes:

    • You can now run your pipelines locally in isolated Docker containers per step
    • @gabrielmbmb updated our MLFlow experiment tracker to work with Databricks deployments 🎉
    • Documentation updates for cloud deployments and multi-tenancy Kubeflow support

    What's Changed

    • Update GitHub Actions by @fa9r in https://github.com/zenml-io/zenml/pull/864
    • Raise zenml exception when cyclic graph is detected by @schustmi in https://github.com/zenml-io/zenml/pull/866
    • Add source to segment identify call by @htahir1 in https://github.com/zenml-io/zenml/pull/868
    • Use default local paths/URIs for the local artifact and metadata stores by @stefannica in https://github.com/zenml-io/zenml/pull/873
    • Implement local docker orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/862
    • Update cheat sheet with latest CLI commands from 0.13.0 by @dnth in https://github.com/zenml-io/zenml/pull/867
    • Add a note about importing proper DockerConfiguration module by @jsuchome in https://github.com/zenml-io/zenml/pull/877
    • Bugfix/misc by @schustmi in https://github.com/zenml-io/zenml/pull/878
    • Fixed bug in tfx by @htahir1 in https://github.com/zenml-io/zenml/pull/883
    • Mlflow Databricks connection by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/882
    • Refactor cloud guide to stack deployment guide by @wjayesh in https://github.com/zenml-io/zenml/pull/861
    • Add cookie consent by @strickvl in https://github.com/zenml-io/zenml/pull/871
    • Stack recipe CLI improvements by @wjayesh in https://github.com/zenml-io/zenml/pull/872
    • Kubeflow workaround added by @htahir1 in https://github.com/zenml-io/zenml/pull/886

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.1...0.13.2

    Source code(tar.gz)
    Source code(zip)
  • 0.13.1(Aug 26, 2022)

    ZenML 0.13.1 is here and it comes with several quality of life improvements:

    • You can now specify the exact order in which your pipelines steps should be executed, e.g., via step_b.after(step_a)
    • TensorBoard was moved to a separate integration so you can use it with Pytorch and other modeling frameworks
    • You can now configure the Evidently integration to ignore specific columns in your datasets.

    This release also contains a lot of documentation on how to deploy custom code (like preprocessing and postprocessing code) with our KServe and Seldon integrations.

    What's Changed

    • Fix flag info on recipes in docs by @wjayesh in https://github.com/zenml-io/zenml/pull/854
    • Fix some materializer issues by @schustmi in https://github.com/zenml-io/zenml/pull/852
    • Add ignore columns for evidently drift detection by @SangamSwadiK in https://github.com/zenml-io/zenml/pull/851
    • TensorBoard Integration by @fa9r in https://github.com/zenml-io/zenml/pull/850
    • Add option to specify task dependencies by @schustmi in https://github.com/zenml-io/zenml/pull/858
    • Custom code readme and docs by @safoinme in https://github.com/zenml-io/zenml/pull/853

    New Contributors

    • @SangamSwadiK made their first contribution in https://github.com/zenml-io/zenml/pull/851

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.13.0...0.13.1

    Source code(tar.gz)
    Source code(zip)
    Release.0.13.1.gif(1.22 MB)
  • 0.13.0(Aug 17, 2022)

    ZenML version 0.13.0 is chock-full with exciting features.

    Custom Code Deployment is the continuation of the Model Deployment story that we have been working on over the last few releases. Now it is possible to deploy custom code along with your models using Kserve or Seldon.

    With Spark this release also brings distributed processing into the ZenML toolkit.

    Spinning up and configuring infrastructure is a difficult part of the MLOps journey and can easily become a barrier to entry. Using our mlops-stacks repository, it is now possible to spin up perfectly configured infrastructure with the corresponding ZenML stack using the ZenML CLI.

    As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.

    Breaking Changes

    This release introduces a breaking change to the CLI by adjusting the access to the stack component specific resources for secret-managers and model-deployers to be more explicitly linked to the component. Here is how:

    # `zenml secret register ...` becomes 
    zenml secrets-manager secret register ...
    
    # `zenml served_models list` becomes 
    zenml model-deployer models list
    

    What's Changed

    • Link checker by @dnth in https://github.com/zenml-io/zenml/pull/818
    • Update Readme with latest info from docs page by @dnth in https://github.com/zenml-io/zenml/pull/810
    • Typo on Readme by @dnth in https://github.com/zenml-io/zenml/pull/821
    • Update kserve installation to 0.9 on kserve deployment example by @safoinme in https://github.com/zenml-io/zenml/pull/823
    • Allow setting caching via the config.yaml by @strickvl in https://github.com/zenml-io/zenml/pull/827
    • Handle file-io with context manager by @aliabbasjaffri in https://github.com/zenml-io/zenml/pull/825
    • Add automated link check github actions by @dnth in https://github.com/zenml-io/zenml/pull/828
    • Fix the SQL zenstore to work with MySQL by @stefannica in https://github.com/zenml-io/zenml/pull/829
    • Improve label studio error messages if secrets are missing or of wrong schema by @schustmi in https://github.com/zenml-io/zenml/pull/832
    • Add secret scoping to the Azure Key Vault by @stefannica in https://github.com/zenml-io/zenml/pull/830
    • Unify CLI concepts (removing secret, feature and served-models) by @strickvl in https://github.com/zenml-io/zenml/pull/833
    • Put link checker as part of CI by @dnth in https://github.com/zenml-io/zenml/pull/838
    • Add missing requirement for step operators by @schustmi in https://github.com/zenml-io/zenml/pull/834
    • Fix broken links from link checker results by @dnth in https://github.com/zenml-io/zenml/pull/835
    • Fix served models logs formatting error by @safoinme in https://github.com/zenml-io/zenml/pull/836
    • New Docker build configuration by @schustmi in https://github.com/zenml-io/zenml/pull/811
    • Secrets references on stack component attributes by @schustmi in https://github.com/zenml-io/zenml/pull/817
    • Misc bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/842
    • Pillow Image materializer by @strickvl in https://github.com/zenml-io/zenml/pull/820
    • Add Tekton orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/844
    • Put Slack call to action at the top of README page. by @dnth in https://github.com/zenml-io/zenml/pull/846
    • Change Quickstart to Use Tabular Data by @fa9r in https://github.com/zenml-io/zenml/pull/843
    • Add sleep before docker builds in release GH action by @schustmi in https://github.com/zenml-io/zenml/pull/849
    • Implement Recursive Built-In Container Materializer by @fa9r in https://github.com/zenml-io/zenml/pull/812
    • Custom deployment with KServe and Seldon Core by @safoinme in https://github.com/zenml-io/zenml/pull/841
    • Spark Integration by @bcdurak in https://github.com/zenml-io/zenml/pull/837
    • Add zenml stack recipe CLI commands by @wjayesh in https://github.com/zenml-io/zenml/pull/807

    New Contributors

    • @aliabbasjaffri made their first contribution in https://github.com/zenml-io/zenml/pull/825

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.12.0...0.13.0

    Source code(tar.gz)
    Source code(zip)
  • 0.12.0(Aug 2, 2022)

    The 0.12.0 release comes with the third implementation of the ZenML Model Deployer abstraction: The KServe integration allows you to deploy any PyTorch, TensorFlow or SKLearn from within your ZenML pipelines!

    We also added functionality to specify hardware resources on a step level to control the amount of memory, CPUs and GPUs that each ZenML step has access to. This is curretly limited to the Kubeflow and Vertex orchestrator but will be expanded in upcoming releases.

    Additionally, we've added support for scoped secrets in our AWS, GCP and Vault Secrets Managers. These updated Secrets Managers allow you to configure a scope which determines if secrets are shared with other ZenML Secrets Managers using the same backend.

    As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.

    What's Changed

    • Fix Links on the examples by @safoinme in https://github.com/zenml-io/zenml/pull/782
    • Fix broken links in source code by @schustmi in https://github.com/zenml-io/zenml/pull/784
    • Invalidating artifact/metadata store if there is a change in one of them by @bcdurak in https://github.com/zenml-io/zenml/pull/719
    • Fixed broken link in README by @htahir1 in https://github.com/zenml-io/zenml/pull/785
    • Embed Cheat Sheet in a separate docs page by @fa9r in https://github.com/zenml-io/zenml/pull/790
    • Add data validation documentation by @stefannica in https://github.com/zenml-io/zenml/pull/789
    • Add local path for mlflow experiment tracker by @schustmi in https://github.com/zenml-io/zenml/pull/786
    • Improve Docker build logs. by @fa9r in https://github.com/zenml-io/zenml/pull/793
    • Allow standard library types in steps by @stefannica in https://github.com/zenml-io/zenml/pull/799
    • Added small description by @AlexejPenner in https://github.com/zenml-io/zenml/pull/801
    • Replace the restriction to use Repository inside step with a warning by @stefannica in https://github.com/zenml-io/zenml/pull/792
    • Adjust quickstart to data validators by @fa9r in https://github.com/zenml-io/zenml/pull/797
    • Add utility function to deprecate pydantic attributes by @schustmi in https://github.com/zenml-io/zenml/pull/778
    • Fix the mismatch KFP version between Kubeflow and GCP integration by @safoinme in https://github.com/zenml-io/zenml/pull/796
    • Made mlflow more verbose by @htahir1 in https://github.com/zenml-io/zenml/pull/802
    • Fix links by @dnth in https://github.com/zenml-io/zenml/pull/798
    • KServe model deployer integration by @stefannica in https://github.com/zenml-io/zenml/pull/655
    • retrieve pipeline requirement within running step by @safoinme in https://github.com/zenml-io/zenml/pull/805
    • Fix --decouple_stores error message by @strickvl in https://github.com/zenml-io/zenml/pull/814
    • Support subscripted generic step output types by @fa9r in https://github.com/zenml-io/zenml/pull/806
    • Allow empty kubeconfig when using local kubeflow orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/809
    • fix the secret register command in kserve docs page by @safoinme in https://github.com/zenml-io/zenml/pull/815
    • Annotation example (+ stack component update) by @strickvl in https://github.com/zenml-io/zenml/pull/813
    • Per-step resource configuration by @schustmi in https://github.com/zenml-io/zenml/pull/794
    • Scoped secrets by @stefannica in https://github.com/zenml-io/zenml/pull/803
    • Adjust examples and docs to new pipeline and step fetching syntax by @fa9r in https://github.com/zenml-io/zenml/pull/795

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.11.0...0.12.0

    Source code(tar.gz)
    Source code(zip)
  • 0.11.0(Jul 20, 2022)

    Our 0.11.0 release contains our new annotation workflow and stack component. We've been blogging about this for a few weeks, and even started maintaining our own repository of open-source annotation tools. With ZenML 0.11.0 you can bring data labeling into your MLOps pipelines and workflows as a first-class citizen. We've started our first iteration of this functionality by integrating with Label Studio, a leader in the open-source annotation tool space.

    This release also includes a ton of updates to our documentation. (Seriously, go check them out! We added tens of thousands of words since the last release.) We continued the work on our data validation story from the previous release: Deepchecks is the newest data validator we support, and we updated our Evidently and whylogs integrations to include all the latest and greatest from those tools.

    Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user. For a detailed look at what's changed, give our full release notes a glance.

    🏷 Data Annotation with Label Studio

    We've been hard at work on our new stack component and integration with Label Studio, the first of our data annotation tools. 🥳

    Object Detection with ZenML and Label Studio

    Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks you have ready for use.

    Data annotation/labeling is a core part of MLOps that is frequently left out of the conversation. With this release, ZenML now supports annotation as an integrated and first-class citizen as part of the MLOps lifecycle.

    Got feedback or just want to let us know how you feel? Connect with us or join us for a Community Meetup 👋 which happens every Wednesday!

    🔎 More Data Validation with Deepchecks, Evidently & whylogs

    We continued the work started in the previous release, adding a new integration with Deepchecks. Deepchecks is a feature-rich data validation open-source library to painlessly do data validation. Deepchecks can do a variety of data validation tasks, from data integrity checks that work with a single dataset to data+model evaluation to data drift analyses. All this can be done with minimal configuration input from the user, or customized with specialized conditions that the validation checks should perform. Check out our example if you want to see it in action!

    We also updated our integrations with Great Expectations, Evidently and whylogs to support their latest releases. These tools have added lots of great new features in recent weeks and we now support most of them, all under the new data validator stack component and abstraction.

    📖 Documentation & User Guides

    We made a significant overhaul of our documentation since the last release:

    • The developer guide section is reworked to be more complete and beginner-friendly
    • We wrote a whole new 'MLOps stack' section, which contains detailed explanations for all MLOps stack components and their various implementations in ZenML
    • A new 'Cloud Guide' section contains complete setup guides for multiple cloud stacks. This will help you get started quickly.
    • We added a new ZenML cheatsheet that you can use to remind you of important CLI commands.

    ⌨️ CLI Improvements

    We fixed a bug that was preventing users who upgraded to 0.10.0 from pulling new examples. This now works without any problem.

    ➕ Other Updates, Additions and Fixes

    The latest release includes several smaller features and updates to existing functionality:

    • We fixed a bug in our Feast integration that prevented registration as a stack component.
    • We updated the structure of all our examples so that they now conform to all of the 'best practices' guidance we've made available in our docs.
    • We fixed some module and path resolution errors that were happening for Windows users.
    • We have combined all the MetadataStore contexts to speed up calls to the metadata store. This speeds up pipeline execution.
    • We now prevent providing extra attributes when initializing stack components. This could have led to unexpected behaviors so we now just prevent this behavior.
    • We've built several new Docker images. You can view them all over at dockerhub.
    • The facets magic display now works on Google Colab.
    • Our Azure Secret Schema now works with the secrets manager. An issue with how Azure handles secret names was preventing this, but we encoded the secret names to get around this shortcoming on the Azure platform.
    • @Val3nt-ML added a nested MLflow parameter (on the @enable_mlflow decorator) which will allow the creation of nested runs for each step of a ZenML pipeline in MLflow.
    • We enabled the fetching of secrets from within a step.
    • We now allow the fetching of pipelines and steps by name, class or instance.
    • You can now also add optional machine specs to VertexAI orchestrators, thanks to a PR from @felixthebeard.
    • We fixed a bug that was preventing users from importing pipeline requirements via a requirements.txt file if the file ended with a newline.

    Breaking Changes

    The 0.11.0 release remodels the Evidently and whylogs integrations as Data Validator stack components, in an effort to converge all data profiling and validation libraries around the same abstraction. As a consequence, you now need to configure and add a Data Validator stack component to your stack if you wish to use Evidently or whylogs in your pipelines:

    • for Evidently:

      zenml data-validator register evidently -f evidently
      zenml stack update -dv evidently
      
    • for whylogs:

      zenml data-validator register whylogs -f whylogs
      zenml stack update -dv whylogs
      

    In this release, we have also upgraded the Evidently and whylogs libraries to their latest and greatest versions (whylogs 1.0.6 and evidently 0.1.52). These versions introduce non-backwards compatible changes that are also reflected in the ZenML integrations:

    • Evidently profiles are now materialized using their original evidently.model_profile.Profile data type and the builtin EvidentlyProfileStep step now also returns a Profile instance instead of the previous dictionary representation. This may impact your existing pipelines as you may have to update your steps to take in Profile artifact instances instead of dictionaries.

    • the whylogs whylogs.DatasetProfile data type was replaced by whylogs.core.DatasetProfileView in the builtin whylogs materializer and steps. This may impact your existing pipelines as you may have to update your steps to return and take in whylogs.core.DatasetProfileView artifact instances instead of whylogs.DatasetProfile objects.

    • the whylogs library has gone through a major transformation that completely removed the session concept. As a result, the enable_whylogs step decorator was replaced by an enable_whylabs step decorator. You only need to use the step decorator if you wish to log your profiles to the Whylabs platform.

    Please refer to the examples provided for Evidently and whylogs to learn more about how to use the new integration versions:

    🙌 Community Contributions

    We received several new community contributions during this release cycle. Here's everybody who contributed towards this release:

    • @jsuchome made their first contribution in https://github.com/zenml-io/zenml/pull/740
    • @Val3nt-ML made their first contribution in https://github.com/zenml-io/zenml/pull/742
    • @felixthebeard contributed a PR to allow for optional machine specs to be passed in for the VertexAI orchestrator in https://github.com/zenml-io/zenml/pull/762

    👩‍💻 Contribute to ZenML!

    Join our Slack to let us know if you have an idea for a feature or something you'd like to contribute to the framework.

    We have a new home for our roadmap where you can vote on your favorite upcoming feature or propose new ideas for what the core team should work on. You can vote without needing to log in, so please do let us know what you want us to build!

    Source code(tar.gz)
    Source code(zip)
  • 0.10.0(Jun 28, 2022)

    The 0.10.0 release continues our streak of extending ZenML with support for new orchestrators, this time by adding the Kubernetes Native Orchestrator. This orchestrator is a lightweight alternative to other distributed orchestrators like Airflow or Kubeflow that gives our users the ability to run pipelines in any Kubernetes cluster without having to install and manage additional tools or components.

    This release features another integration that we are really excited about: the popular data profiling and validation library Great Expectations is our first Data Validator, a new category of stack components that we are in the process of standardizing, that will make data quality a central feature of ZenML. The ZenML Great Expectations integration eliminates the complexity associated with configuring the store backends for Great Expectations by reusing our Artifact Store concept for that purpose and gives ZenML users immediate access to Great Expectations in both local and cloud settings.

    Last but not least, the release also includes a new secrets manager implementation, courtesy of our contributor @karimhabush, that integrates ZenML with the Hashicorp Vault Server as well as a few other bug fixes and improvements.

    What's Changed

    • Fix broken link by @strickvl in https://github.com/zenml-io/zenml/pull/707
    • Add stack component copy command by @schustmi in https://github.com/zenml-io/zenml/pull/705
    • Remove force flag from secrets managers' implementation by @strickvl in https://github.com/zenml-io/zenml/pull/708
    • Fixed wrong example README by @AlexejPenner in https://github.com/zenml-io/zenml/pull/712
    • Fix dead links in integrations docs. by @fa9r in https://github.com/zenml-io/zenml/pull/710
    • Fixing link to guide by @chethanuk-plutoflume in https://github.com/zenml-io/zenml/pull/716
    • Adding azure-keyvault-secrets to azure integration dependencies by @safoinme in https://github.com/zenml-io/zenml/pull/717
    • Fix MLflow repeated deployment error by @fa9r in https://github.com/zenml-io/zenml/pull/715
    • Replace alerter standard steps by Slack-specific steps to fix config issue. by @fa9r in https://github.com/zenml-io/zenml/pull/714
    • Fix broken links on README by @dnth in https://github.com/zenml-io/zenml/pull/722
    • Invalidate cache by @strickvl in https://github.com/zenml-io/zenml/pull/724
    • Skip Cleaning Trace on tests by @safoinme in https://github.com/zenml-io/zenml/pull/725
    • Kubernetes orchestrator by @fa9r in https://github.com/zenml-io/zenml/pull/688
    • Vault Secrets Manager integration - KV Secrets Engine by @karimhabush in https://github.com/zenml-io/zenml/pull/689
    • Add missing help text for CLI commands by @safoinme in https://github.com/zenml-io/zenml/pull/723
    • Misc bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/713
    • Great Expectations integration for data validation by @strickvl in https://github.com/zenml-io/zenml/pull/555
    • Fix GCP artifact store by @schustmi in https://github.com/zenml-io/zenml/pull/730

    New Contributors

    • @chethanuk-plutoflume made their first contribution in https://github.com/zenml-io/zenml/pull/716
    • @dnth made their first contribution in https://github.com/zenml-io/zenml/pull/722
    • @karimhabush made their first contribution in https://github.com/zenml-io/zenml/pull/689

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.9.0...0.10.0

    Source code(tar.gz)
    Source code(zip)
  • 0.9.0(Jun 14, 2022)

    0.9.0

    It's been a couple of weeks, so it's time for a new release! 0.9.0 brings two whole new orchestrators, one of which was contributed by a community member just one day after we unveiled new documentation for orchestrator extensibility! The release also includes a new secrets manager, a Slack integration and a bunch of other smaller changes across the codebase. (Our new orchestrators are exciting enough that they'll get their own blog posts to showcase their strengths in due course.)

    Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user.

    What's Changed

    • Pass secret to release linting workflow by @schustmi in https://github.com/zenml-io/zenml/pull/642
    • Fix typo in example by @anencore94 in https://github.com/zenml-io/zenml/pull/644
    • Added SecretExistsError in register_secret() method by @hectorLop in https://github.com/zenml-io/zenml/pull/648
    • Fix broken GCP Secrets example CLI command by @strickvl in https://github.com/zenml-io/zenml/pull/649
    • Upgrade to ml-pipelines-sdk v1.8.0 by @strickvl in https://github.com/zenml-io/zenml/pull/651
    • Fix example list CLI command name by @schustmi in https://github.com/zenml-io/zenml/pull/647
    • Fix README by @strickvl in https://github.com/zenml-io/zenml/pull/657
    • Fix broken links in docs by @safoinme in https://github.com/zenml-io/zenml/pull/652
    • Add VertexOrchestrator implementation by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/640
    • Fix index page links and Heading links. by @safoinme in https://github.com/zenml-io/zenml/pull/661
    • Add docstring checks to pre-commit script by @strickvl in https://github.com/zenml-io/zenml/pull/481
    • Pin MLflow to <1.26.0 to prevent issues when matplotlib is not installed by @fa9r in https://github.com/zenml-io/zenml/pull/666
    • Making utils more consistent by @strickvl in https://github.com/zenml-io/zenml/pull/658
    • Fix linting failures on develop by @strickvl in https://github.com/zenml-io/zenml/pull/669
    • Add docstrings for config module by @strickvl in https://github.com/zenml-io/zenml/pull/668
    • Miscellaneous bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/660
    • Make ZenServer dependencies optional by @schustmi in https://github.com/zenml-io/zenml/pull/665
    • Implement Azure Secrets Manager integration by @strickvl in https://github.com/zenml-io/zenml/pull/654
    • Replace codespell with pyspelling by @strickvl in https://github.com/zenml-io/zenml/pull/663
    • Add Community Event to README by @htahir1 in https://github.com/zenml-io/zenml/pull/674
    • Fix failing integration tests by @strickvl in https://github.com/zenml-io/zenml/pull/677
    • Add io and model_deployers docstring checks by @strickvl in https://github.com/zenml-io/zenml/pull/675
    • Update zenml stack down to use --force flag by @schustmi in https://github.com/zenml-io/zenml/pull/673
    • Fix class resolving on windows by @schustmi in https://github.com/zenml-io/zenml/pull/678
    • Added pipelines docstring checks by @strickvl in https://github.com/zenml-io/zenml/pull/676
    • Docstring checks for cli module by @strickvl in https://github.com/zenml-io/zenml/pull/680
    • Docstring fixes for entrypoints and experiment_trackers modules by @strickvl in https://github.com/zenml-io/zenml/pull/672
    • Clearer Contributing.md by @htahir1 in https://github.com/zenml-io/zenml/pull/681
    • How to access secrets within step added to docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/653
    • FIX: Log a warning instead of raising an AssertionError by @ketangangal in https://github.com/zenml-io/zenml/pull/628
    • Reviewer Reminder by @htahir1 in https://github.com/zenml-io/zenml/pull/683
    • Fix some docs phrasings and headers by @strickvl in https://github.com/zenml-io/zenml/pull/670
    • Implement SlackAlerter.ask() by @fa9r in https://github.com/zenml-io/zenml/pull/662
    • Extending Alerters Docs by @fa9r in https://github.com/zenml-io/zenml/pull/690
    • Sane defaults for MySQL by @htahir1 in https://github.com/zenml-io/zenml/pull/691
    • pd.Series materializer by @Reed-Schimmel in https://github.com/zenml-io/zenml/pull/684
    • Add docstrings for materializers and metadata_stores by @strickvl in https://github.com/zenml-io/zenml/pull/694
    • Docstrings for the integrations module(s) by @strickvl in https://github.com/zenml-io/zenml/pull/692
    • Add remaining docstrings by @strickvl in https://github.com/zenml-io/zenml/pull/696
    • Allow enabling mlflow/wandb/whylogs with the class-based api by @schustmi in https://github.com/zenml-io/zenml/pull/697
    • GitHub Actions orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/685
    • Created MySQL docs, Vertex AI docs, and step.entrypoint() by @AlexejPenner in https://github.com/zenml-io/zenml/pull/698
    • Update ignored words by @strickvl in https://github.com/zenml-io/zenml/pull/701
    • Stack Component registering made easier by @AlexejPenner in https://github.com/zenml-io/zenml/pull/695
    • Cleaning up the docs after the revamp by @bcdurak in https://github.com/zenml-io/zenml/pull/699
    • Add model deployer to CLI docs by @safoinme in https://github.com/zenml-io/zenml/pull/702
    • Merge Cloud Integrations and create a Vertex AI Example by @AlexejPenner in https://github.com/zenml-io/zenml/pull/693
    • GitHub actions orchestrator example by @schustmi in https://github.com/zenml-io/zenml/pull/703

    New Contributors

    • @anencore94 made their first contribution in https://github.com/zenml-io/zenml/pull/644
    • @hectorLop made their first contribution in https://github.com/zenml-io/zenml/pull/648
    • @gabrielmbmb made their first contribution in https://github.com/zenml-io/zenml/pull/640
    • @ketangangal made their first contribution in https://github.com/zenml-io/zenml/pull/628
    • @Reed-Schimmel made their first contribution in https://github.com/zenml-io/zenml/pull/684

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.8.1...0.9.0

    Source code(tar.gz)
    Source code(zip)
  • 0.8.1(May 25, 2022)

    0.8.1

    ZenML 0.8.1 is here and it comes with support for Python 3.9 🎉. It also includes major updates to our documentation, fixes some broken links in our examples and improves the zenml go command which helps you get started with ZenML.

    We did this fast turn-around to fix some bugs in 0.8.0 and get a quick release out for Python 3.9 which was a highly requested feature! #502

    What's Changed

    • Hotfix/fix failing release by @AlexejPenner in https://github.com/zenml-io/zenml/pull/611
    • Remove autocomplete + alerter from documentation by @strickvl in https://github.com/zenml-io/zenml/pull/612
    • Support Python 3.9 by @htahir1 in https://github.com/zenml-io/zenml/pull/605
    • Revert README by @htahir1 in https://github.com/zenml-io/zenml/pull/624
    • Don't build cuda image on release by @schustmi in https://github.com/zenml-io/zenml/pull/623
    • Update quickstart for zenml go by @fa9r in https://github.com/zenml-io/zenml/pull/625
    • Improve kubeflow manual setup logs by @schustmi in https://github.com/zenml-io/zenml/pull/622
    • Added missing space to error message by @AlexejPenner in https://github.com/zenml-io/zenml/pull/614
    • Added --set flag to register stack command by @AlexejPenner in https://github.com/zenml-io/zenml/pull/613
    • Fixes for multiple examples by @schustmi in https://github.com/zenml-io/zenml/pull/626
    • Bring back the served_model format to the keras materializer by @stefannica in https://github.com/zenml-io/zenml/pull/629
    • Fix broken example links by @schustmi in https://github.com/zenml-io/zenml/pull/630
    • FAQ edits by @strickvl in https://github.com/zenml-io/zenml/pull/634
    • Fix version parsing by @schustmi in https://github.com/zenml-io/zenml/pull/633
    • Completed Best Practices Page by @AlexejPenner in https://github.com/zenml-io/zenml/pull/635
    • Comments on Issues should no longer trigger gh actions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/636
    • Revise CONTRIBUTING.md by @strickvl in https://github.com/zenml-io/zenml/pull/615
    • Alerter Component for Slack Integration by @fa9r in https://github.com/zenml-io/zenml/pull/586
    • Update zenml go to open quickstart/notebooks. by @fa9r in https://github.com/zenml-io/zenml/pull/631
    • Update examples by @schustmi in https://github.com/zenml-io/zenml/pull/638
    • More detailed instructions on creating an integration by @AlexejPenner in https://github.com/zenml-io/zenml/pull/639
    • Added publish api docs to release workflow by @AlexejPenner in https://github.com/zenml-io/zenml/pull/641
    • Added *.md to ignore paths by @AlexejPenner in https://github.com/zenml-io/zenml/pull/637
    • Update README and Docs with new messaging and fix broken links by @htahir1 in https://github.com/zenml-io/zenml/pull/632

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.8.0...0.8.1

    Source code(tar.gz)
    Source code(zip)
  • 0.8.0(May 19, 2022)

    🧘‍♀️ Extensibility is our middle name

    • The ability to register custom stack component flavors (and renaming types to flavor (Registering custom stack component flavors by @bcdurak in https://github.com/zenml-io/zenml/pull/541)
    • The ability to easily extend orchestrators
    • Documentation for stacks, stack components and flavors by @bcdurak inhttps://github.com/zenml-io/zenml/pull/607
    • Allow configuration of s3fs by @schustmi in https://github.com/zenml-io/zenml/pull/532
    • Ability to use SSL to connect to MySQL clients (That allows for connecting to Cloud based MYSQL deployments)
    • New MySQL metadata stores by @bcdurak in https://github.com/zenml-io/zenml/pull/580!
    • Docs and messaging change
    • Make Orchestrators more extensible and simplify the interface by @AlexejPenner in https://github.com/zenml-io/zenml/pull/581
    • S3 Compatible Artifact Store and materializers file handling by @safoinme in https://github.com/zenml-io/zenml/pull/598

    Manage your stacks

    • Update stack and stack components via the CLI by @strickvl in https://github.com/zenml-io/zenml/pull/497
    • Add stack delete confirmation prompt by @strickvl in https://github.com/zenml-io/zenml/pull/548
    • Add zenml stack export and zenml stack import commands by @fa9r in https://github.com/zenml-io/zenml/pull/560

    Collaboration

    • User management by @schustmi in https://github.com/zenml-io/zenml/pull/500

    CLI improvements

    • CLI speed improvement by @bcdurak in https://github.com/zenml-io/zenml/pull/567
    • Ensure rich CLI displays full text and wraps table text by @strickvl in https://github.com/zenml-io/zenml/pull/577
    • Add CLI command to remove stack component attribute by @strickvl in https://github.com/zenml-io/zenml/pull/590
    • Beautify CLI by grouping commands list into tags by @safoinme in https://github.com/zenml-io/zenml/pull/546

    New integrations:

    • Add PyTorch example by @htahir1 in https://github.com/zenml-io/zenml/pull/559
    • Added GCP as secret manager by @AlexejPenner in https://github.com/zenml-io/zenml/pull/556

    Documentation / ZenBytes etc

    • ZenBytes update (and ZenFiles)
    • Beautification of Examples by @AlexejPenner in https://github.com/zenml-io/zenml/pull/491
    • Document global configuration and repository by @stefannica in https://github.com/zenml-io/zenml/pull/579
    • ZenML Collaboration docs by @stefannica in https://github.com/zenml-io/zenml/pull/597

    ➕ Other Updates, Additions and Fixes

    • Experiment tracker stack components by @htahir1 in https://github.com/zenml-io/zenml/pull/530
    • Secret Manager improvements and Seldon Core secret passing by @stefannica in https://github.com/zenml-io/zenml/pull/529
    • Pipeline run tracking by @schustmi in https://github.com/zenml-io/zenml/pull/601
    • Stream model deployer logs through CLI by @stefannica in https://github.com/zenml-io/zenml/pull/557
    • Fix various usability bugs by @stefannica in https://github.com/zenml-io/zenml/pull/561
    • Replace -f and --force with -y and --yes by @strickvl inhttps://github.com/zenml-io/zenml/pull/566
    • Make it easier to submit issues by @htahir1 in https://github.com/zenml-io/zenml/pull/571
    • Sync the repository and local store with the disk configuration files and other fixes by @stefannica in https://github.com/zenml-io/zenml/pull/588
    • Add ability to give in-line pip requirements for pipeline by @strickvl in https://github.com/zenml-io/zenml/pull/583
    • Fix evidently visualizer on Colab by @fa9r in https://github.com/zenml-io/zenml/pull/592

    🙌 Community Contributions

    • @Ankur3107 made their first contribution in https://github.com/zenml-io/zenml/pull/467
    • @MateusGheorghe made their first contribution in https://github.com/zenml-io/zenml/pull/523
    • Added support for scipy sparse matrices by @avramdj in https://github.com/zenml-io/zenml/pull/534
    Source code(tar.gz)
    Source code(zip)
  • 0.7.3(Apr 28, 2022)

    📊 Experiment Tracking Components

    PR #530 adds a new stack component to ZenMLs ever-growing list: experiment_trackers allows users to configure your experiment tracking tools with ZenML. Examples of experiment tracking tools are Weights&Biases, mlflow, Neptune, amongst others.

    Existing users might be confused, as ZenML has had MLflow and wandb support for a while now without such a component. However, this component allows uses more control over the configuration of MLflow and wandb with the new MLFlowExperimentTracker and WandbExperimentTracker components. This allows these tools to work in more scenarios than the currently limiting local use-cases.

    🔎 XGBoost and LightGBM support

    XGBoost and LightGBM are one of the most widely used boosting algorithm libraries out there. This release adds materializers for native objects for each library.

    Check out both examples here and PR's #544 and #538 for more details.

    📂 Parameterized S3FS support to enable non-AWS S3 storage (minio, ceph)

    A big complaint of the S3 Artifact Store integration was that it was hard to parameterize it in a way that it supports non-AWS S3 storage like minio and ceph. The latest release made this super simple! When you want to register an S3ArtifactStore from the CLI, you can now pass in client_kwargs, config_kwargs or s3_additional_kwargs as a JSON string. For example:

    zenml artifact-store register my_s3_store --type=s3 --path=s3://my_bucket \
        --client_kwargs='{"endpoint_url": "http://my-s3-endpoint"}'
    

    See PR #532 for more details.

    🧱 New CLI commands to update stack components

    We added functionality to allow users to update stacks that already exist. This shows the basic workflow:

    zenml orchestrator register local_orchestrator2 -t local
    zenml stack update default -o local_orchestrator2
    zenml stack describe default
    zenml container-registry register local_registry --type=default --uri=localhost:5000
    zenml container-registry update local --uri='somethingelse.com'
    zenml container-registry rename local local2
    zenml container-registry describe local2
    zenml stack rename default new_default
    zenml stack update new_default -c local2
    zenml stack describe new_default
    zenml stack remove-component -c
    

    More details are in the CLI docs. Users can add new stack components to a pre-existing stack, or they can modify already-present stack components. They can also rename their stack and individual stack components.

    🐛 Seldon Core authentication through ZenML secrets

    The Seldon Core Model Deployer stack component was updated in this release to allow the configuration of ZenML secrets with credentials that authenticate Seldon to access the Artifact Store. The Seldon Core integration provides 3 different secret schemas for the 3 flavors of Artifact Store: AWS, GCP, and Azure, but custom secrets can be used as well. For more information on how to use this feature please refer to our Seldon Core deployment example.

    Lastly, we had numerous other changes such as ensuring the PyTorch materializer works across all artifact stores and the Kubeflow Metadata Store can be easily queried locally.

    Detailed Changelog

    • Fix caching & mypy errors by @strickvl in https://github.com/zenml-io/zenml/pull/524
    • Switch unit test from local_daemon to multiprocessing by @jwwwb in https://github.com/zenml-io/zenml/pull/508
    • Change Pytorch materializer to support remote storage by @safoinme in https://github.com/zenml-io/zenml/pull/525
    • Remove TODO from Feature Store init docstring by @strickvl in https://github.com/zenml-io/zenml/pull/527
    • Fixed typo predicter -> predictor by @MateusGheorghe in https://github.com/zenml-io/zenml/pull/523
    • Fix mypy errors by @strickvl in https://github.com/zenml-io/zenml/pull/528
    • Replaced old local_* logic by @htahir1 in https://github.com/zenml-io/zenml/pull/531
    • capitalize aws username in ECR docs by @wjayesh in https://github.com/zenml-io/zenml/pull/533
    • Build docker base images quicker after release by @schustmi in https://github.com/zenml-io/zenml/pull/537
    • Allow configuration of s3fs by @schustmi in https://github.com/zenml-io/zenml/pull/532
    • Update contributing and fix ci badge to main by @htahir1 in https://github.com/zenml-io/zenml/pull/536
    • Added XGboost integration by @htahir1 in https://github.com/zenml-io/zenml/pull/538
    • Added fa9r to .github/teams.yml. by @fa9r in https://github.com/zenml-io/zenml/pull/539
    • Secret Manager improvements and Seldon Core secret passing by @stefannica in https://github.com/zenml-io/zenml/pull/529
    • User management by @schustmi in https://github.com/zenml-io/zenml/pull/500
    • Update stack and stack components via the CLI by @strickvl in https://github.com/zenml-io/zenml/pull/497
    • Added lightgbm integration by @htahir1 in https://github.com/zenml-io/zenml/pull/544
    • Fix the Kubeflow metadata store and other stack management improvements by @stefannica in https://github.com/zenml-io/zenml/pull/542
    • Experiment tracker stack components by @htahir1 in https://github.com/zenml-io/zenml/pull/530

    New Contributors

    • @MateusGheorghe made their first contribution in https://github.com/zenml-io/zenml/pull/523
    • @fa9r made their first contribution in https://github.com/zenml-io/zenml/pull/539

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.7.2...0.7.3 Blog Post: https://blog.zenml.io/zero-seven-two-three-release/

    Source code(tar.gz)
    Source code(zip)
  • 0.7.2(Apr 14, 2022)

    0.7.2 is a minor release which quickly patches some bugs found in the last release to do with Seldon and Mlflow deployment.

    This release also features initial versions of two amazing new community-led integrations: HuggingFace and Weights&Biases!

    richard_socher_shoutout

    • HuggingFace models are now supported to be passed through ZenML pipelines [see full example] -> Huge shoutout to @Ankur3107 for PR #467.
    • You can now track your pipeline runs with Weights&Biases with the new enable_wandb decorator [see full example] -> Huge shoutout to @soumik12345 for PR #486.

    Continuous model deployment with MLflow has been improved with ZenML 0.7.2. A new MLflow Model Deployer Stack component is now available and needs to be part of your stack to be able to deploy models:

    zenml integration install mlflow
    zenml model-deployer register mlflow --type=mlflow
    zenml stack register local_with_mlflow -m default -a default -o default -d mlflow
    zenml stack set local_with_mlflow
    

    The MLflow Model Deployer is yet another addition to the list of Model Deployers available in ZenML. You can read more on deploying models to production with MLflow in our [Continuous Training and Deployment documentation section](https://docs.zenml.io/features/continuous training-and-deployment) and our MLflow deployment example.

    What's Changed

    • Fix the seldon deployment example by @htahir1 in https://github.com/zenml-io/zenml/pull/511
    • Create base deployer and refactor MLflow deployer implementation by @wjayesh in https://github.com/zenml-io/zenml/pull/489
    • Add nlp example by @Ankur3107 in https://github.com/zenml-io/zenml/pull/467
    • Fix typos by @strickvl in https://github.com/zenml-io/zenml/pull/515
    • Bugfix/hypothesis given doesnt work with fixture by @jwwwb in https://github.com/zenml-io/zenml/pull/513
    • Bug: fix long Kubernetes labels in Seldon deployments by @stefannica in https://github.com/zenml-io/zenml/pull/514
    • Change prediction_uri to prediction_url in MLflow deployer by @stefannica in https://github.com/zenml-io/zenml/pull/516
    • Simplify HuggingFace Integration by @AlexejPenner in https://github.com/zenml-io/zenml/pull/517
    • Weights & Biases Basic Integration by @htahir1 in https://github.com/zenml-io/zenml/pull/518

    New Contributors

    • @Ankur3107 made their first contribution to the HuggingFace integration in https://github.com/zenml-io/zenml/pull/467
    • @soumik12345 from Weights&Biases also made their first contribution to the wandb integration in https://github.com/zenml-io/zenml/pull/486!

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.7.1...0.7.2

    Source code(tar.gz)
    Source code(zip)
  • 0.7.1(Apr 11, 2022)

    0.7.1

    Feast-Seldon

    The release introduces the Seldon Core ZenML integration, featuring the Seldon Core Model Deployer and a Seldon Core standard model deployer step. The Model Deployer is a new type of stack component that enables you to develop continuous model deployment pipelines that train models and continuously deploy them to an external model serving tool, service or platform. You can read more on deploying models to production with Seldon Core in our Continuous Training and Deployment documentation section and our Seldon Core deployment example.

    We also see two new integrations with Feast as ZenML's first feature store integration. Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use within a team or wider organization. ZenML now supports connecting to a Redis-backed Feast feature store as a stack component integration. Check out the full example to see it in action!

    0.7.1 also brings an addition to ZenML training library integrations with NeuralProphet. Check out the new example for more details, and the docs for more further detail on all new features!

    What's Changed

    • Add linting of examples to pre-commit by @strickvl in https://github.com/zenml-io/zenml/pull/490
    • Remove dev-specific entries in .gitignore by @strickvl in https://github.com/zenml-io/zenml/pull/488
    • Produce periodic mocked data for Segment/Mixpanel by @AlexejPenner in https://github.com/zenml-io/zenml/pull/487
    • Abstractions for artifact stores by @bcdurak in https://github.com/zenml-io/zenml/pull/474
    • enable and disable cache from runtime config by @AlexejPenner in https://github.com/zenml-io/zenml/pull/492
    • Basic Seldon Core Deployment Service by @stefannica in https://github.com/zenml-io/zenml/pull/495
    • Parallelise our test suite and make errors more readable by @alex-zenml in https://github.com/zenml-io/zenml/pull/378
    • Provision local zenml service by @jwwwb in https://github.com/zenml-io/zenml/pull/496
    • bugfix/optional-secrets-manager by @safoinme in https://github.com/zenml-io/zenml/pull/493
    • Quick fix for copying folders by @bcdurak in https://github.com/zenml-io/zenml/pull/501
    • Pin exact ml-pipelines-sdk version by @schustmi in https://github.com/zenml-io/zenml/pull/506
    • Seldon Core model deployer stack component and standard step by @stefannica in https://github.com/zenml-io/zenml/pull/499
    • Fix datetime test / bug by @strickvl in https://github.com/zenml-io/zenml/pull/507
    • Added NeuralProphet integration by @htahir1 in https://github.com/zenml-io/zenml/pull/504
    • Feature Store (Feast with Redis) by @strickvl in https://github.com/zenml-io/zenml/pull/498
    Source code(tar.gz)
    Source code(zip)
  • 0.7.0(Mar 28, 2022)

    With ZenML 0.7.0, a lot has been revamped under the hood about how things are stored. Importantly what this means is that ZenML now has system-wide profiles that let you register stacks to share across several of your projects! If you still want to manage your stacks for each project folder individually, profiles still let you do that as well.

    Most projects of any complexity will require passwords or tokens to access data and infrastructure, and for this purpose ZenML 0.7.0 introduces the Secrets Manager stack component to seamlessly pass around these values to your steps. Our AWS integration also allows you to use AWS Secrets Manager as a backend to handle all your secret persistence needs.

    Finally, in addition to the new AzureML and Sagemaker Step Operators that version 0.6.3 brought, this release also adds the ability to run individual steps on GCP's Vertex AI.

    Beyond this, some smaller bugfixes and documentation changes combine to make ZenML 0.7.0 a more pleasant user experience.

    What's Changed

    • Added quick mention of how to use dockerignore by @AlexejPenner in https://github.com/zenml-io/zenml/pull/468
    • Made rich traceback optional with ENV variable by @htahir1 in https://github.com/zenml-io/zenml/pull/472
    • Separate stack persistence from repo implementation by @jwwwb in https://github.com/zenml-io/zenml/pull/462
    • Adding safoine username to github team by @safoinme in https://github.com/zenml-io/zenml/pull/475
    • Fix zenml stack describe bug by @strickvl in https://github.com/zenml-io/zenml/pull/476
    • ZenProfiles and centralized ZenML repositories by @stefannica in https://github.com/zenml-io/zenml/pull/471
    • Add examples folder to linting script by @strickvl in https://github.com/zenml-io/zenml/pull/482
    • Vertex AI integration and numerous other changes by @htahir1 in https://github.com/zenml-io/zenml/pull/477
    • Fix profile handing in the Azure ML step operator by @stefannica in https://github.com/zenml-io/zenml/pull/483
    • Copy the entire stack configuration into containers by @stefannica in https://github.com/zenml-io/zenml/pull/480
    • Improve some things with the Profiles CLI output by @stefannica in https://github.com/zenml-io/zenml/pull/484
    • Secrets manager stack component and interface by @AlexejPenner in https://github.com/zenml-io/zenml/pull/470
    • Update schedule.py (#485) by @avramdj in https://github.com/zenml-io/zenml/pull/485

    New Contributors

    • @avramdj in https://github.com/zenml-io/zenml/pull/485

    Full Changelog: https://github.com/zenml-io/zenml/compare/0.6.3...0.7.0rc

    Source code(tar.gz)
    Source code(zip)
  • 0.6.3(Mar 14, 2022)

    0.6.3

    ZenInTheClouds_3

    With ZenML 0.6.3, you can now run your ZenML steps on Sagemaker and AzureML! It's normal to have certain steps that require specific hardware on which to run model training, for example, and this latest release gives you the power to switch out hardware for individual steps to support this.

    We added a new Tensorboard visualisation that you can make use of when using our Kubeflow Pipelines integration. We handle the background processes needed to spin up this interactive web interface that you can use to visualise your model's performance over time.

    Behind the scenes we gave our integration testing suite a massive upgrade, fixed a number of smaller bugs and made documentation updates. For a detailed look at what's changed, see below:

    What's Changed

    • Fix typo by @wjayesh in https://github.com/zenml-io/zenml/pull/432
    • Remove tabulate dependency (replaced by rich) by @jwwwb in https://github.com/zenml-io/zenml/pull/436
    • Fix potential issue with local integration tests by @schustmi in https://github.com/zenml-io/zenml/pull/428
    • Remove support for python 3.6 by @schustmi in https://github.com/zenml-io/zenml/pull/437
    • Create clean test repos in separate folders by @michael-zenml in https://github.com/zenml-io/zenml/pull/430
    • Copy explicit materializers before modifying, log correct class by @schustmi in https://github.com/zenml-io/zenml/pull/434
    • Fix typo in mysql password parameter by @pafpixel in https://github.com/zenml-io/zenml/pull/438
    • Pytest-fixture for separate virtual environments for each integration test by @AlexejPenner in https://github.com/zenml-io/zenml/pull/405
    • Bugfix/fix failing tests due to comments step by @AlexejPenner in https://github.com/zenml-io/zenml/pull/444
    • Added --use-virtualenvs option to allow choosing envs to run by @AlexejPenner in https://github.com/zenml-io/zenml/pull/445
    • Log whether a step was cached by @strickvl in https://github.com/zenml-io/zenml/pull/435
    • Added basic integration tests for remaining examples by @strickvl in https://github.com/zenml-io/zenml/pull/439
    • Improve error message when provisioning local kubeflow resources with a non-local container registry. by @schustmi in https://github.com/zenml-io/zenml/pull/442
    • Enable generic step inputs and outputs by @schustmi in https://github.com/zenml-io/zenml/pull/440
    • Removed old reference to a step that no longer exists by @AlexejPenner in https://github.com/zenml-io/zenml/pull/452
    • Correctly use custom kubernetes context if specified by @schustmi in https://github.com/zenml-io/zenml/pull/451
    • Fix CLI stack component describe/list commands by @schustmi in https://github.com/zenml-io/zenml/pull/450
    • Ignore type of any tfx proto file by @schustmi in https://github.com/zenml-io/zenml/pull/453
    • Another boyscout pr on the gh actions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/455
    • Upgrade TFX to 1.6.1 by @jwwwb in https://github.com/zenml-io/zenml/pull/441
    • Added ZenFiles to README by @htahir1 in https://github.com/zenml-io/zenml/pull/457
    • Upgrade rich from 11.0 to 12.0 by @strickvl in https://github.com/zenml-io/zenml/pull/458
    • Add Kubeflow tensorboard viz and fix tensorflow file IO for cloud back-ends by @stefannica in https://github.com/zenml-io/zenml/pull/447
    • Implementing the explain subcommand by @bcdurak in https://github.com/zenml-io/zenml/pull/460
    • Implement AzureML and Sagemaker step operators by @schustmi in https://github.com/zenml-io/zenml/pull/456

    New Contributors

    • @pafpixel made their first contribution in https://github.com/zenml-io/zenml/pull/438
    Source code(tar.gz)
    Source code(zip)
  • 0.6.2(Feb 23, 2022)

    ZenML0-6-2

    ZenML 0.6.2 brings you the ability to serve models using MLflow deployments as well as an updated CLI interface! For a real continuous deployment cycle, we know that ZenML pipelines should be able to handle everything — from pre-processing to training to serving to monitoring and then potentially re-training and re-serving. The interfaces we created in this release are the foundation on which all of this will build.

    We also improved how you interact with ZenML through the CLI. Everything looks so much smarter and readable now with the popular rich library integrated into our dependencies.

    Smaller changes that you'll notice include updates to our cloud integrations and bug fixes for Windows users. For a detailed look at what's changed, see below.

    What's Changed

    • Updated notebook for quickstart by @htahir1 in https://github.com/zenml-io/zenml/pull/398
    • Update tensorflow base image by @schustmi in https://github.com/zenml-io/zenml/pull/396
    • Add cloud specific deployment guide + refactoring by @wjayesh in https://github.com/zenml-io/zenml/pull/400
    • add cloud sub page to toc.md by @wjayesh in https://github.com/zenml-io/zenml/pull/401
    • fix tab indent by @wjayesh in https://github.com/zenml-io/zenml/pull/402
    • Bugfix for workflows failing due to modules not being found by @bcdurak in https://github.com/zenml-io/zenml/pull/390
    • Improve github workflows by @schustmi in https://github.com/zenml-io/zenml/pull/406
    • Add plausible script to docs.zenml.io pages by @alex-zenml in https://github.com/zenml-io/zenml/pull/414
    • Add orchestrator and ECR docs by @wjayesh in https://github.com/zenml-io/zenml/pull/413
    • Richify the CLI by @alex-zenml in https://github.com/zenml-io/zenml/pull/392
    • Allow specification of required integrations for a pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/408
    • Update quickstart in docs to conform to examples by @htahir1 in https://github.com/zenml-io/zenml/pull/410
    • Updated PR template with some more details by @htahir1 in https://github.com/zenml-io/zenml/pull/411
    • Bugfix on the CLI to work without a git installation by @bcdurak in https://github.com/zenml-io/zenml/pull/412
    • Added Ayush's Handle by @ayush714 in https://github.com/zenml-io/zenml/pull/417
    • Adding an info message on Windows if there is no application associated to .sh files by @bcdurak in https://github.com/zenml-io/zenml/pull/419
    • Catch matplotlib crash when running IPython in terminal by @strickvl in https://github.com/zenml-io/zenml/pull/416
    • Automatically activate integrations when unable to find stack component by @schustmi in https://github.com/zenml-io/zenml/pull/420
    • Fix some code inspections by @halvgaard in https://github.com/zenml-io/zenml/pull/422
    • Prepare integration tests on kubeflow by @schustmi in https://github.com/zenml-io/zenml/pull/423
    • Add concepts back into glossary by @strickvl in https://github.com/zenml-io/zenml/pull/425
    • Make guide easier to follow by @wjayesh in https://github.com/zenml-io/zenml/pull/427
    • Fix httplib to 0.19 and pyparsing to 2.4 by @jwwwb in https://github.com/zenml-io/zenml/pull/426
    • Wrap context serialization in try blocks by @jwwwb in https://github.com/zenml-io/zenml/pull/397
    • Track stack configuration when registering and running a pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/429
    • MLflow deployment integration by @stefannica in https://github.com/zenml-io/zenml/pull/415
    Source code(tar.gz)
    Source code(zip)
Owner
ZenML
Building production MLOps tooling.
ZenML
Model Validation Toolkit is a collection of tools to assist with validating machine learning models prior to deploying them to production and monitoring them after deployment to production.

Model Validation Toolkit is a collection of tools to assist with validating machine learning models prior to deploying them to production and monitoring them after deployment to production.

FINRA 25 Dec 28, 2022
A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.

Master status: Development status: Package information: TPOT stands for Tree-based Pipeline Optimization Tool. Consider TPOT your Data Science Assista

Epistasis Lab at UPenn 8.9k Jan 9, 2023
Upgini : data search library for your machine learning pipelines

Automated data search library for your machine learning pipelines → find & deliver relevant external data & features to boost ML accuracy :chart_with_upwards_trend:

Upgini 175 Jan 8, 2023
Evidently helps analyze machine learning models during validation or production monitoring

Evidently helps analyze machine learning models during validation or production monitoring. The tool generates interactive visual reports and JSON profiles from pandas DataFrame or csv files. Currently 6 reports are available.

Evidently AI 3.1k Jan 7, 2023
Production Grade Machine Learning Service

This project is made to help you scale from a basic Machine Learning project for research purposes to a production grade Machine Learning web service

Abdullah Zaiter 10 Apr 4, 2022
Machine learning that just works, for effortless production applications

Machine learning that just works, for effortless production applications

Elisha Yadgaran 16 Sep 2, 2022
Iris-Heroku - Putting a Machine Learning Model into Production with Flask and Heroku

Puesta en Producción de un modelo de aprendizaje automático con Flask y Heroku L

Jesùs Guillen 1 Jun 3, 2022
⏳ Tempo: The MLOps Software Development Kit

Tempo provides a unified interface to multiple MLOps projects that enable data scientists to deploy and productionise machine learning systems.

Seldon 36 Jun 20, 2021
Pragmatic AI Labs 421 Dec 31, 2022
End to End toy example of MLOps

churn_model MLOps Toy Example End to End You might find below links useful Connect VSCode to Git MLFlow Port Heroku App Project Organization ├── LICEN

Ashish Tele 6 Feb 6, 2022
ClearML - Auto-Magical Suite of tools to streamline your ML workflow. Experiment Manager, MLOps and Data-Management

ClearML - Auto-Magical Suite of tools to streamline your ML workflow Experiment Manager, MLOps and Data-Management ClearML Formerly known as Allegro T

ClearML 4k Jan 9, 2023
Azure MLOps (v2) solution accelerators.

Azure MLOps (v2) solution accelerator Welcome to the MLOps (v2) solution accelerator repository! This project is intended to serve as the starting poi

Microsoft Azure 233 Jan 1, 2023
A demo project to elaborate how Machine Learn Models are deployed on production using Flask API

This is a salary prediction website developed with the help of machine learning, this makes prediction of salary on basis of few parameters like interview score, experience test score.

null 1 Feb 10, 2022
A project based example of Data pipelines, ML workflow management, API endpoints and Monitoring.

MLOps template with examples for Data pipelines, ML workflow management, API development and Monitoring.

Utsav 33 Dec 3, 2022
Scikit-Learn useful pre-defined Pipelines Hub

Scikit-Pipes Scikit-Learn useful pre-defined Pipelines Hub Usage: Install scikit-pipes It's advised to install sklearn-genetic using a virtual env, in

Rodrigo Arenas 1 Apr 26, 2022
easyNeuron is a simple way to create powerful machine learning models, analyze data and research cutting-edge AI.

easyNeuron is a simple way to create powerful machine learning models, analyze data and research cutting-edge AI.

Neuron AI 5 Jun 18, 2022
Python Extreme Learning Machine (ELM) is a machine learning technique used for classification/regression tasks.

Python Extreme Learning Machine (ELM) Python Extreme Learning Machine (ELM) is a machine learning technique used for classification/regression tasks.

Augusto Almeida 84 Nov 25, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

Vowpal Wabbit 8.1k Dec 30, 2022
CD) in machine learning projectsImplementing continuous integration & delivery (CI/CD) in machine learning projects

CML with cloud compute This repository contains a sample project using CML with Terraform (via the cml-runner function) to launch an AWS EC2 instance

Iterative 19 Oct 3, 2022