This PR is the initial work to re-write k3ai into a more flexible tool. This PR implements:

• [x] #3
• [x] #4
• [x] #5
• [ ] #6
• [x] #7
• [x] #8
• [ ] #9

This PR also include the Issues in the Plugin repo:

• [x] https://github.com/k3ai/plugins/issues/1
• [x] https://github.com/k3ai/plugins/issues/2
• [x] https://github.com/k3ai/plugins/issues/3

Describe the bug After following the installation instructions, the following error is reported:

k3ai: /lib/x86_64-linux-gnu/libc.so.6: versionGLIBC_2.28' not found (required by k3ai)`

To Reproduce Steps to reproduce the behavior:

1. curl -LO https://get.k3ai.in | sh -
2. k3ai up

Expected behavior It should have spun up the cluster!

OS: Ubuntu 18.04

This PR address:

• #10
• #14 Also introduce -x as Extra and -e ad Entrypoint

Examples:

k3ai run -s https://github.com/alefesta/sample/mlflow -b mlflow -t <clustername>
k3ai run -s https://github.com/alefesta/sample/kfp -b kfp -e condition.py -t <clustername>

For MLFlow remains to manage the need of boto3 in the conda.yaml file that is a requirement to run on K8s. This need to be addressed before merge this PR. We may:

• try to inject boto3 in the conda.yaml file at runtime
• Force the user to fork the example first , change conda and run k3ai The first seems more compliant with k3ai goals of making life of user easier

Describe the bug I was trying to follow the kubeflow pipelines tutorial as described in the k3ai website. It seems the final step of running the pipeline fails because the quickstart repository for kubeflow pipelines does not exist.

To Reproduce Steps to reproduce the behavior:

1. k3ai up
2. k3ai cluster deploy -t k3s -n myk3scluster
3. k3ai plugin deploy -n kf-pa -t myk3scluster
4. k3ai run -s https://github.com/k3ai/quickstart/kfp -b kfp -e condition.py -t mycluster

Expected behavior Pipeline to run successfully.

Actual behavior Pipeline run fails.

🚀 Is your feature request related to a problem? Please describe. Currently, k3ai doesn't have support for k3d.

k3s is known to have issues with WSL2 deployment (systemd requirement, etc.), so it would be better to have k3d support.

💡 Describe the solution you'd like We can add k3d support to k3ai in a subsequent release. (would require some work on pkg/io/execution).

Fixed 100+ issues related to ineffectual assignments and added error checks. Added golangci-lint workflow to check linting issues while pushing code.

The log level used while error checking is Fatal (log.Fatal(err)).

Describe the bug

While running the MLFlow plugin, the endpoint URI displayed by k3ai is not accessible.

The following endpoints are not accessible:

• http://172.29.170.187:30500/ (displayed by k3ai)
• http://172.29.170.187:5000/
• http://10.96.150.194:30500/
• http://10.244.0.7:30500/

The IP address for the WSL2 machine is (through wsl hostname -I): 172.29.170.187

WSL2 uses dynamic IP allocation.

To Reproduce Steps to reproduce the behavior:

k3ai run -s https://github.com/k3ai/quickstart -b mlflow

Expected behavior The MLFlow endpoint exposed through k3ai should have worked.

K3ai should implement an automatice system domain (i.e: sslip.io or nip.io) so that any plugin installed could be exposed with the standard: <plugin-name>.<clusterIP>.nip.io This way we may use the same IP in cases like:

• WSL
• Laptops

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Follow quickstart steps:

k3ai up
k3ai cluster deploy -t k3s -n mycluster
k3ai plugin deploy -n mlflow -t mycluster

2. Try running quickstart: $ k3ai run -s https://github.com/k3ai/quickstart -b mlflow
3. Receive error:

🧪	Initializing code...
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/k3ai/pkg/runner.Loader({0x7fff7fe7bb4e, 0x22}, {0x0, 0x0}, {0x7fff7fe7bb74, 0x6}, {0x0, 0x0}, {0x0, 0x0})
	/home/joshec/git/k3ai/pkg/runner/run.go:78 +0x10f6
github.com/k3ai/cmd.runCommand.func1(0xc000403680, {0xc0003d57c0, 0x0, 0x4})
	/home/joshec/git/k3ai/cmd/run.go:71 +0x58b
github.com/spf13/cobra.(*Command).execute(0xc000403680, {0xc0003d5780, 0x4, 0x4})
	/home/joshec/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0x2254960)
	/home/joshec/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	/home/joshec/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/k3ai/cmd.Execute(...)
	/home/joshec/git/k3ai/cmd/root.go:34
main.main()
	/home/joshec/git/k3ai/main.go:10 +0x25

Expected behavior A clear and concise description of what you expected to happen.

A successful run with proper artifact storage and tracking URI settings

Screenshots If applicable, add screenshots to help explain your problem.

On running golangci-lint on my local machine, I was able to find 40+ linting issues.

10 of them were deadcode issues, so it can be ignored as they're a part of adding code for future releases.

The rest are ineffectual assignments and skipped error checks. We can log the error message for the skipped error checks; it would help us more in debugging.

I know this sounds like a minor issue, but with more code coming in the subsequent releases, addressing this earlier can help us save a lot of time maintaining good quality code.

Suggested Fix: Add golangci-lint action as workflow to check linting issues. We can add a rule for excluding deadcode issues for now.

Hello - I am trying out the Mlflow deployment as in the tutorials and I get a stream of logs that say "invalid argument" and after a while I get "We tried to publish MLFLow at:http://172.17.0.2:30500" .. but when I go to this page there is no Mlflow server.

Would appreciate the help. Thanks.

Great work btw! this library is amazing!

Describe the bug I am trying to run the kubeflow plugin on a single node 8vcpu / 16gb ram.

To Reproduce curl -sfL https://get.k3ai.in | sh - k3ai up k3ai cluster deploy --type k3s -n mycluster k3ai plugin deploy -n kf-pa -t mycluster

Issue Installation never ends, seems the pods are not being started correctly

ubuntu:~$ k3s kubectl get pods -n kubeflow
NAME                                              READY   STATUS                   RESTARTS        AGE
workflow-controller-b7f95d6c6-q2wkf               1/1     Running                  0               4m22s
ml-pipeline-scheduledworkflow-5c549bc5f5-drkmn    1/1     Running                  0               4m23s
ml-pipeline-viewer-crd-7555c4d55f-fpd2m           1/1     Running                  0               4m23s
metadata-envoy-deployment-7654b98955-rkt2g        1/1     Running                  0               4m24s
ml-pipeline-ui-656466fdc9-qg9xv                   1/1     Running                  0               4m23s
mysql-55778745b6-g4vbd                            1/1     Running                  0               4m22s
minio-6d6d45469f-xgmz2                            1/1     Running                  0               4m24s
cache-deployer-deployment-6f8ff5b986-tvwn4        1/1     Running                  0               4m24s
metadata-grpc-deployment-5c8599b99c-b45jf         1/1     Running                  1 (3m17s ago)   4m24s
ml-pipeline-8995b746f-dhznz                       1/1     Running                  1 (2m31s ago)   4m23s
cache-server-74494cbf5-k956w                      0/1     Pending                  0               2m20s
cache-server-74494cbf5-6v5lj                      0/1     ContainerStatusUnknown   0               4m24s
ml-pipeline-persistenceagent-59689585f6-s8dhd     1/1     Running                  1 (2m5s ago)    4m23s
ml-pipeline-visualizationserver-6b8fb8c44-mmrk8   0/1     ContainerStatusUnknown   0               4m22s
ml-pipeline-visualizationserver-6b8fb8c44-svm25   0/1     Pending                  0               113s
metadata-writer-fd965db48-9lw22                   0/1     Error                    0               4m24s
metadata-writer-fd965db48-rqt7d                   0/1     Pending                  0               82s

Pod metadata-writer-fd965db48-9lw22 error : message: 'The node was low on resource: ephemeral-storage. Container main was using 392Ki, which exceeds its request of 0. '

Any ideas? Thanks!

Describe the bug in postgres pod: Bus error (core dumped)

running on: (base) pax@ithaki:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.4 LTS Release: 20.04 Codename: focal (base) pax@ithaki:~$

To Reproduce

k3ai cluster deploy --type k3s --name arrakis rk3ai plugin deploy -n mlflow -t arrakis k3s kubectl logs postgres-0

Expected behavior successful mlflow startup

Screenshots (base) pax@ithaki:~$ kubectl get all -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/local-path-provisioner-6c79684f77-996dg 1/1 Running 0 7m28s kube-system pod/coredns-d76bd69b-8zqzz 1/1 Running 0 7m28s kube-system pod/metrics-server-7cd5fcb6b7-6zwwl 1/1 Running 0 7m28s default pod/minio-0 1/1 Running 0 6m40s default pod/mlflow-7c6768c4c-m6j6d 1/1 Running 0 6m23s default pod/postgres-0 0/1 CrashLoopBackOff 6 (20s ago) 6m31s

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.43.0.1 443/TCP 7m43s kube-system service/kube-dns ClusterIP 10.43.0.10 53/UDP,53/TCP,9153/TCP 7m40s kube-system service/metrics-server ClusterIP 10.43.39.68 443/TCP 7m39s default service/minio-service ClusterIP 10.43.144.140 9000/TCP 6m40s default service/postgres-service ClusterIP 10.43.236.158 5432/TCP 6m23s default service/mlflow-service NodePort 10.43.192.251 5000:30500/TCP 6m8s

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system deployment.apps/local-path-provisioner 1/1 1 1 7m40s kube-system deployment.apps/coredns 1/1 1 1 7m40s kube-system deployment.apps/metrics-server 1/1 1 1 7m39s default deployment.apps/mlflow 1/1 1 1 6m23s

NAMESPACE NAME DESIRED CURRENT READY AGE kube-system replicaset.apps/local-path-provisioner-6c79684f77 1 1 1 7m29s kube-system replicaset.apps/coredns-d76bd69b 1 1 1 7m29s kube-system replicaset.apps/metrics-server-7cd5fcb6b7 1 1 1 7m29s default replicaset.apps/mlflow-7c6768c4c 1 1 1 6m23s

NAMESPACE NAME READY AGE default statefulset.apps/minio 1/1 6m40s default statefulset.apps/postgres 0/1 6m31s

(base) pax@ithaki:~$ kubectl logs postgres-0 The files belonging to this database system will be owned by user "postgres". This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/mlflow/data ... ok creating subdirectories ... ok selecting default max_connections ... 20 selecting default shared_buffers ... 400kB selecting default timezone ... Etc/UTC selecting dynamic shared memory implementation ... posix creating configuration files ... ok

Describe the bug There's bug on kubeflow part, they currently don't support k8s 1.22, so at the moment kubeflow pipelines seems to work, with k3s, but e.g. kf-dashboard is failing, which might be related to unsupported k8s API version.

...
 ⏳     Working...
 🚀 unable to recognize "/home/user/.k3ai/git/common/istio-1-9/istio-crds/base": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
...
 🚀 unable to recognize "/home/user/.k3ai/git/common/istio-1-9/istio-install/base": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
...
 🚀 unable to recognize "/home/user/.k3ai/git/common/istio-1-9/istio-install/base": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
 🚀 unable to recognize "/home/user/.k3ai/git/common/istio-1-9/istio-install/base": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
...
 🚀 unable to recognize "/home/user/.k3ai/git/common/istio-1-9/istio-install/base": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
 🚀 unable to recognize "/home/user/.k3ai/git/common/istio-1-9/istio-install/base": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
...

To Reproduce

k3ai plugin deploy -n kf-dashboard -t myk3scluster

Expected behavior

Successful deployment of all kubeflow components on k3s.

🚀 Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

See Issue https://github.com/k3ai/k3ai/issues/53 regarding k3ai run not returning a useful error when a required argument was missing

💡 Describe the solution you'd like A clear and concise description of what you want to happen.

a better CLI handler with more descriptive errors, or at least a fix for this bug on this command's handling

🤩 Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

None yet

Hey, great work with K3ai. Works For most of the operations pretty smooth.

After experimenting a bit with Kubeflow I wanted to remove a plugin, but it seems that the command is not implemented:

➜ k3ai  plugin remove --name kf-pa
Remove a given plugin based on NAME

Usage:
  k3ai[options] plugin remove [-n NAME] [other flags]

Flags:
  -n, --name string     NAME of plugin to be created/deleted
  -t, --target string   Target from where to remove plugin.
  -q, --quiet           Suppress output messages. Useful when k3ai is used within scripts.
  -c, --config string   Configure K3ai using a custom config file.[-c /path/tofile] [-c https://urlToFile]

See here: https://github.com/k3ai/k3ai/blob/main/cmd/plugin.go#L138

I am not sure, whether I am missing something but couldn't find anything related in the issues or Roadmap.

• Your operating system name and version: Ubuntu 18.04
• Detailed steps to reproduce the bug: Follow exact steps from documentation or README to deploy a plugin

🚀 Is your feature request related to a problem? Please describe. Related:

• #37

Using the metrics report exported through the executor, we can use Github Actions workflows to create automated issues containing the reports.

💡 Describe the solution you'd like Create issues with the exported report as the content using Github Actions.