Open MLOps - A Production-focused Open-Source Machine Learning Framework
Open MLOps is a set of open-source tools carefully chosen to ease user experience of conducting machine learning experiments and deploying machine learning models. Read our announcement blog post with more background here.
We also provide a step by step set-up guide and some other getting started tutorials.
In this repository, we provide these applications as Terraform modules with which the user will be able to install them into a Kubernetes cluster. The tools we provide are the following:
- Prefect for data flow automation
- Jupyter Hub for experimenting lab
- Dask for distributed computing
- Feast for Feature Store and Serving
- MLFlow for model registry and experiment tracking
- Seldon for model deployment
Other repositories
Modules
Jupyter Hub
With the Jupyter Hub, we enabled a multi-user environment in which each of them can spawn a Jupyter server to do their experiments. Users can work on different environments being able to install any library necessary to meet their needs.
We provide a default Jupyter server image that comes with most of the data science packages installed. Users can use their own Jupyter server images as well.
Configuration
Below we provide a lists of the configurable parameters available and their default values.
| Parameter (* required parameter) | Description | Default |
|---|---|---|
jupyterhub_namespace |
Namespace to install jupyterhb | jhub |
Proxy configuration
The proxy receives the requests from the client’s browser and forwards all requests to the Hub. In the JupyterHub docs you can find a more in-depth explanation.
* Required parameters
| Parameter | Description | Default |
|---|---|---|
jhub_proxy_https_enabled |
Indicator to set whether HTTPS should be enabled or not on the proxy | false |
jhub_proxy_https_hosts |
You domains in list form. Required for automatic HTTPS | [] |
jhub_proxy_secret_token * |
A 32-byte cryptographically secure randomly generated string used to secure communications between the hub and the configurable-http-proxy (for example, generated by openssl rand -hex 32) |
nil |
jhub_proxy_https_letsencrypt_contact_email |
The contact email to be used for automatically provisioned HTTPS certificates by Let’s Encrypt | "" |
Authentication configuration
JupyterHub’s OAuthenticator has support for enabling your users to authenticate via a third-party OAuth2 identity provider such as GitHub.
You can configure authentication using GitHub accounts and restrict what users are authorized based on membership in a GitHub organization.
See details on how to set up a GitHub Oauth here.
If you choose not to use GitHub to authenticate users, the DummyAuthenticator will be used as default. The Dummy Authenticator lets any user log in with the given password.
The dummy password is: a-shared-secret-password.
* Required parameters ** Required when oauth_github_enable is enabled
| Parameter | Description | Default |
|---|---|---|
oauth_github_enable |
Defines whether the authentication will be handled by github oauth. Required when oauth_github_enable is enabled |
false |
oauth_github_client_id ** |
Github client id used on GitHubOAuthenticator. | "" |
oauth_github_client_secret ** |
Github secret used to authenticate with github. | "" |
oauth_github_admin_users |
List of github user names to allow as administrator | [] |
oauth_github_callback_url |
The URL that people are redirected to after they authorize your GitHub App to act on their behalf | "" |
oauth_github_allowed_organizations |
List of Github organization to restrict access to the members | [""] |
User configuration
Single user configuration refers to the default settings for each user logged in the JupyterHub.
A user can choose a Docker image to spawn a new Jupyter server. Each Docker image can have different libraries and environments installed. We use the singleuser_profile_list parameter to set up a list of default images available to the user. This parameter receives a list of maps that describes the image details such as the image location and description.
See an example:
[{
display_name = "Prefect"
description = "Notebook with prefect installed"
default = true
kubespawner_override = {
image = "drtools/prefect:notebook-prefect"
}
}]
You must pass the image pull secret if you provide an image located in a private container registry. The image pull secret parameter is defined as below:
default = [{
name = ""
}]
| Parameter | Description | Default |
|---|---|---|
singleuser_profile_list |
List of images which the user can select to spawn a server | |
singleuser_image_pull_secrets |
List of image secrets | nil |
singleuser_image_pull_policy |
Image pull policy | Always |
singleuser_memory_guarantee |
How much memory will be guarateed to the user | 1G |
singleuser_storage_capacity |
How much storage capacity a user will have | 1G |
singleuser_storage_mount_path |
Storage mount path | /home/jovyan/persistent |
Prefect
...
| Parameter | Description | Default |
|---|---|---|
namespace |
Namespace name to deploy the application | `prefect |
prefect_version_tag |
Configures the default tag for prefect images | latest |
Agent
According to Prefect docs, Agents are lightweight processes for orchestrating flow runs. Agents run inside a user's architecture, and are responsible for starting and monitoring flow runs. During operation the agent process queries the Prefect API for any scheduled flow runs, and allocates resources for them on their respective deployment platforms.
| Parameter | Description | Default |
|---|---|---|
agent_enabled |
determines if the Prefect Kubernetes agent is deployed | True |
agent_prefect_labels |
Defines what scheduling labels (not K8s labels) should be associated with the agent | [""] |
agent_image_name |
Defines the prefect agent image name | prefecthq/prefect |
agent_image_tag |
Defines agent image tag | " |
agent_image_pull_policy |
Defines the image pull policy | Always |
Postgresql
| Parameter | Description | Default |
|---|---|---|
postgresql_database |
Defines the postgresql database name | prefect |
postgresql_username |
Defines the username to authenticate with | prefect |
postgresql_existing_secret |
Configures which secret should be referenced for access to the database. | "" |
postgresql_service_port |
Configures the port that the database should be accessed at | 5432 |
postgresql_external_hostname |
Defines the address to contact an externally managed postgres database instance at | "" |
postgresql_use_subchart |
Determines if a this chart should deploy a user-manager postgres database or use an externally managed postgres instance | true |
postgresql_persistence_enabled |
Enables a PVC that stores the database between deployments. If making changes to the database deployment, this PVC will need to be deleted for database changes to take effect. This is especially notable when the authentication password changes on redeploys. | false |
postgresql_persistence_size |
Defines the persistence storage size for postgres | 8G |
postgresql_init_user |
Defines the initial db username | postgres |
Dask
...
| Parameter | Description | Default |
|---|---|---|
namespace |
Namespace name to deploy the application | dask |
worker_name |
Dask worker name | worker |
worker_replicas |
Default number of workers | 3 |
worker_image_repository |
Containe image repository | daskdev/dask |
worker_image_tag |
Container image tag | 2.30.0 |
worker_image_pull_policy |
Container image pull policy. | IfNotPresent |
worker_image_dask_worker_command |
ask worker command. E.g dask-cuda-worker for GPU worker. |
dask-worker |
worker_image_pull_secret |
Container image pull secrets | [{name: ""}] |
worker_environment_variables |
Environment variables. See values.yaml for example values. | [{}] |
Feast
...
| Parameter | Description | Default |
|---|---|---|
namespace |
Namespace name to deploy the application | feast |
feast_core_enabled |
Defines whether to install feast core | True |
feast_online_serving_enabled |
Defines whether to install feast server | True |
feast_jupyter_enabled |
Defines whether to install feast jupyther hub | False |
feast_jobservice_enabled |
Defines whether to install feast job service | True |
feast_posgresql_enabled |
Defines whether to enable postgresql | True |
feast_postgresql_password * |
Postgress password | "" |
feast_kafka_enabled |
Defines whether to enable kafka | False |
feast_redis_enabled |
Defines whether to enable redis | True |
feast_redis_use_password |
Defines whether to enable redis password | False |
feast_prometheus_enabled |
Defines whether to install prometheys | False |
feast_prometheus_statsd_exporter_enabled |
Defines whether to enable statsd exporter | False |
feast_grafana_enabled |
Defines whether to enable grafana | True |
MLFlow
...
| Parameter | Description | Default |
|---|---|---|
namespace |
Namespace name to deploy the application | mlflow |
db_host |
Database host address | `` |
db_username |
Database username | mlflow |
db_password * |
Database password | `` |
database_name |
Database name | mlflow |
db_port |
Database port | 5432 |
default_artifact_root |
local or remote filepath to store model artifacts. It is mandatory when specifying a database backend store | /tmp |
image_pull_policy |
Docker image pull policy | IfNotPresent |
image_repository |
Docker image repository | drtools/mlflow |
image_tag |
Docker image tag | 1.13.1 |
service_type |
Kubernetes service type | NodePort |
docker_registry_server |
Docker Registry Server | `` |
docker_auth_key |
Base64 Enconded combination of {registry_username}:{registry_password}. Can be found in ~/.docker/config.json | `` |
docker_private_repo |
Whether the MLFlow's image comes from a private repository or not. If true, docker_registry_server and docker_auth_key will be required |
false |
Note: The variables docker_registry_server and docker_auth_key are optional and should only be used when pulling MLFlow's image from a private repository.
Seldon
| Parameter | Description | Default |
|---|---|---|
namespace |
Namespace name to deploy the application | mlflow |
istio_enabled |
Whether to install istio as ingress controller | true |
usage_metrics_enabled |
Whether to enable usage metrics | true |
Exposing Services
In order to access the services from outside the cluster, we need to expose them. Usually, this is done through Kubernetes Ingress resources. In this project, since we rely on Seldon to expose our prediction endpoints, we use Ambassador API Gateway as our ingress controller. Seldon Core works well with Ambassador, allowing a single ingress to be used to expose ambassador and running machine learning deployments can then be dynamically exposed through seldon-created ambassador configurations.
Ambassador
Ambassador is a Kubernetes-native API Gateway built on the Envoy Proxy. In addition to the classical routing capabilities of an ingress, it can perform sophisticated traffic management functions, such as load balancing, circuit breakers, rate limits, and automatic retries. Also, it has support for independent authentication systems, such as the ORY ecosystem.
Exposing a service in Ambassador
Ambassador is designed around a declarative, self-service management model. The core resource used to support application development teams who need to manage the edge with Ambassador is the Mapping resource. This resource allows us to define custom routing rules to our services. This routing configuration can achieved by applying a custom Kubernetes Resource like the following
# mapping.yaml
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
name: httpbin-mapping
spec:
prefix: /httpbin/
service: httpbin.httpbin_namespace
By applying this configuration with kubectl apply -f httpbin-mapping.yaml.
Terraform
Since this project uses Terraform to manage resources and, with the current version, it's still not possible to apply custom Kubernetes resource definitions, we need to add this YAML file inside the services annotation. One way to do this is by using Service's Metadata field
resource "kubernetes_service" "httpbin" {
metadata {
...
annotations = {
"getambassador.io/config" = <<YAML
---
apiVersion: getambassador.io/v2
kind: Mapping
name: httpbin-mapping
service: httpbin.httpbin_namespace
prefix: /httpbin/
YAML
}
}
}
This will produce the same behaviour as applying the custom yaml file described above.
Authentication
Since we're exposing our services in the Internet, we need an Authentication and Authorization system to prevent unwanted users to accessing our services. Ambassador API Gateway can control the access by using an External Authentication Service resource (AuthService). An AuthService is an API that has a verification endpoint, which determines if the user can access this resource (returning 200 or not, 401). In this project, we rely on ORY ecosystem to enable authentication. ORY is an open-source ecosystem of services with clear boundaries that solve authentication and authorization.
Session Lifespan
The session lifespan of authenticated users can be managed through the /ory/kratos/values.yaml file. By default, the session lifespan is 24h, but it is currently set to 30 days.
kratos:
config:
...
session:
cookie:
domain: ${cookie_domain}
lifespan: 720h
ORY Oathkeeper
ORY Oathkeeper is an Identity and Access Proxy. It functions as a centralized way to manage different Authentication and Authorization methods, and inform the gateway, whether the HTTP request is allowed or not. The Oathkeeper serves perfectly as an Ambassador's External AuthService.
Zero-Trust and Unauthorized Resources
Oathkeeper is rooted in the principle of "never trust, always verify,". This means that if no additional configuration is provided, the Oathkeeper will always block the incoming request. In practice, all endpoints exposed in Ambassador will be blocked by external requests, until further configuration is made.
Access Rules
To configure an access rule to ORY Oathkeeper, the file access-rule-oathkeeper.yaml is used. Example:
Allow all incoming requests
- id: oathkeeper-access-rule
match:
url: <{http,https}>://${hostname}/allowed-service/<**>
methods:
- GET
authenticators:
- handler: anonymous
authorizer:
handler: allow
mutators:
- handler: noop
credentials_issuer:
handler: noop
This configuration will register all the incoming requests as a guest user, thus, not performing any credentials validation.
Authorize on KRATOS
- id: httpbin-access-rule
match:
url: <{http,https}>://${hostname}/blocked-service/<**>
methods:
- GET
authenticators:
- handler: cookie_session
authorizer:
handler: allow
mutators:
- handler: id_token
credentials_issuer:
handler: noop
errors:
- handler: redirect
config:
to: http://${hostname}/auth/login
This configuration will force authenticating all incoming requests by checking a cookie_session, which configuration is specified in config-oathkeeper.yaml
3.1k Jan 7, 2023
10 Apr 4, 2022
16 Sep 2, 2022
1 Jun 3, 2022
36 Jun 20, 2021
421 Dec 31, 2022
6 Feb 6, 2022
4k Jan 9, 2023
3 Sep 16, 2022
233 Jan 1, 2023
1 Feb 10, 2022
26 Nov 6, 2022
2 Jul 29, 2021
1.8k Jan 1, 2023
1.4k Dec 27, 2022
1.8k Jan 3, 2023
23.3k Dec 31, 2022
8.9k Jan 9, 2023
84 Nov 25, 2022