Implements case 1 and case 2 in the new kfctl semantics Design doc reference

• [x] make platform args optional
• [x] implement new semantics for build and apply

Current workflow for kfctl -

New sematics single command install:

# move into empty directory
kfctl apply -f <path to config file / URL>

New semantics build, edit, apply:

# move into empty directory
kfctl build -f <path to config file / URL>
kfctl apply

Old semantics:

kfctl init kf-app --config <path to config file / URL> 
cd kf-app
kfctl generate <platform>
kfctl apply <platform>

/cc @yanniszark @jlewi @kkasravi

Related to: #3518

This change is 

Trying to install Kubeflow v.70 on on-prem cluster (kubernetes v15.7/Ubuntu 18.04) but getting below error:

root@0939-jdeml-m03:/opt/kubeflow/kf-test# kfctl apply -V -f ${CONFIG_FILE} INFO[0000]

Notice anonymous usage reporting enabled using spartakus To disable it If you have already deployed it run the following commands: cd $(pwd) kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/

filename="coordinator/coordinator.go:120" INFO[0000] Deleting cachedir /opt/kubeflow/kf-test/.cache/manifests because Status.ReposCache is out of date filename="kfconfig/types.go:464" INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /opt/kubeflow/kf-test/.cache/manifests filename="kfconfig/types.go:485" Error: failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF Usage: kfctl apply -f ${CONFIG} [flags]

Flags: -f, --file string Static config file to use. Can be either a local path: export CONFIG=./kfctl_gcp_iap.yaml or a URL: export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml kfctl apply -V --file=${CONFIG} -h, --help help for apply -V, --verbose verbose output default is false

failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF

The Kubeflow installation was working fin when tried it a few days ago. Trying to install it in a new cluster and getting this error. Can you please help?

Thanks, Job Varkey

/kind process

We need to identify who will be driving the 1.1 release. These folks should then

• Identify timelines for the release
  • e.g. cutoff dates for branch cuts
  • Target dates for RCs

This would probably be a good opportunity to update some of the processes and policies around releases. https://github.com/kubeflow/kubeflow/blob/master/docs_dev/releasing.md

Area | release czar | Tracking Issue --- | --- | --- aws | @Jeffwan | ~~#5057~~ | centraldashboard | | ~~#5068~~ docs | | kubeflow/website#1984 fairing | @jinchihe | ~~kubeflow/fairing#503~~ | feast | @woop | | gcp | @jlewi | kubeflow/gcp-blueprints#46 | katib | @andreyvelich | kubeflow/katib#1211 | kfctl | @krishnadurai, @crobby | kubeflow/kfctl#352 | kfserving | @yuzisun @animeshsingh | kubeflow/kfserving#648 | manifests | @krishnadurai | kubeflow/manifests#1252 metadata | | minikf | @vkoukis | | multiuser | @yanniszark @bmorphism | #5067, #5068 notebooks | @kimwnasptd , @jtfogarty | #5060, #5068 | pipelines | @Bobgy | kubeflow/pipelines#3961 | training | @johnugeorge @andreyvelich @Jeffwan | kubeflow/common#97

Target Dates

• Branch cut June 19

We'd like to make it super easy to go from writing code in a notebook to training that model distributed.

Experience might be something like

• User writes code in notebook and executes in Jupyter lab
• User clicks a button which allows user to fill in various settings e.g. number of GPUs
• User clicks train

Under the hood this would cause

• A docker image to be built
• A TFJob/PyTorch/K8s Job to be created and fired off.

I think the biggest challenge is that we probably don't want to execute all code in the notebook. Typically, there's some amount of refactoring that needs to be done to convert a notebook into a python module suitable for execution in a bash job.

As a concrete example

Here's the notebook for our GitHub Issue summarization example

Here's the corresponding python module used when training in a K8s job.

The python module only executes a subset of cells in particular those to

1. Define model architecture
2. train the model

Rather than try to auto-convert a notebook like the github issue example, I think we should require users structure their code to facilitate the conversion.

My suggestion would be to allow any functions defined in the notebook to be used as entry points. So for the GitHub issues summarization the user would have a cell like the following

from keras.callbacks import CSVLogger, ModelCheckpoint

def train_model(output)
  script_name_base = 'tutorial_seq2seq'
  csv_logger = CSVLogger('{:}.log'.format(script_name_base))
  model_checkpoint = ModelCheckpoint('{:}.epoch{{epoch:02d}}- 
  val{{val_loss:.5f}}.hdf5'.format(script_name_base),
                                                  save_best_only=True)

  batch_size = 1200
  epochs = 7
  history = seq2seq_Model.fit([encoder_input_data, decoder_input_data], 
  np.expand_dims(decoder_target_data, -1),
           batch_size=batch_size,
           epochs=epochs,
           validation_split=0.12, callbacks=[csv_logger, model_checkpoint])

   seq2seq_Model.save(output)

train('seq2seq_model_tutorial.h5')

If user structures their code this way, we should be able to manually create and invoke a suitable container entry point. Something like the following

• Use nbconvert to convert from ipynb to python code
• Post process the python code
  • Strip out any statements not inside a function (except imports)
  • Create a CLI for the functions using a library like PyFire
• Build a Docker image that is Notebook image + code

A variant of this idea would be to use metaml (by @wbuchwalter ). metaml uses metaparticle to allow people to annotate their python code with information needed to then run it on K8s (e.g. distributed using TFJob). If we went this approach I think the flow would be

• Run nbconvert to go from ipynb -> py
• Use metaparticle/metaml tool chain to build the docker image and submit the job.

@willingc @yuvipanda Is there existing tooling in the Jupyter community other than nbconvert to convert notebooks to code suitable for asynchronous batch execution?

/cc @wbuchwalter @gaocegege @yuvipanda @willingc

I want to add tolerations to the pods for my Jupyter server. I am able to inject other things like Labels and Annotations, this would allow me to also use PodDefault to inject tolerations.

This change is 

/kind bug

When trying to change namespace on Kubeflow main page, the drop down of namespaces is showing behind the left blue pane, which completely obstructs the list

• Kubeflow version: 0.6
• kfctl version: kfctl v0.6.0-0-g71aea0a9
• Kubernetes platform: OpenShift
• Kubernetes version: kubernetes v1.13.4+c62ce01

/kind bug

What steps did you take and what happened: Installed kubeflow 1.0. While trying to create notebook server I am getting errors like:

User None is not authorized to list kubeflow.org.v1beta1.notebooks for namespace: anonymous

and

User None is not authorized to list .v1.persistentvolumeclaims for namespace: anonymous

What did you expect to happen: Successfully create notebook servers

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.0
• kfctl version: (use kfctl version): 1.0-rc3
• Kubernetes platform: (e.g. minikube) AWS kops
• Kubernetes version: (use kubectl version): 1.15.1
• OS (e.g. from /etc/os-release): Ubuntu 18.04

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

Executed the workaround in https://github.com/kubeflow/kubeflow/issues/3810#issuecomment-527687504

get errors like the below

no matches for kind "deployment" in version "extensions/v1beta1" error: unable to recognize "STDIN": no matches for kind "Deployment" in version "apps/v1beta2"

What did you expect to happen:

all the pods to show up in kubectl get pods -n kubeflow

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment: Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): kfctl version: (use kfctl version): kfctl v0.6.2-0-g47a0e4c7 Kubernetes platform: (e.g. minikube) Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.3 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

Related to #1057 - pytorch notebook image Related to #258 - Kaggle image PTAL /cc @jlewi /cc @ankushagarwal /cc @johnugeorge

Launched for me but obviously not exhaustively tested. Created a contrib dir to give it that YMMV vibe.

Notes (IMHO):

• py3 only
• we should just manually build/push to gcr.io once per release, no other support
• definitely should not be in any argo or release workflows
• waaaay too big an image, there may be some slimming that could be done with my mods but almost certainly it would remain north of 20 Gb

This change is 

Adds a Central UI dashboard which provides links to all other components (jobs dashboard, jupyterhub) as a separate component.
Addresses Issue #146.

Depends on #11. A secure proxy is needed for the links to point to their respective components.

This change is 

/kind process

Time line:

• RC Release: Nov 7
• Official Release: Nov 16

WGs and projects

Area | release czar | Tracking Issue --- | --- | --- centraldashboard | | https://github.com/kubeflow/kubeflow/pull/5412 docs |@RFMVasconcelos | kubeflow/website#2322 fairing | @jinchihe | | feast | @woop | | katib | @andreyvelich | https://github.com/kubeflow/manifests/pull/1593 | kfctl | @Jeffwan | https://github.com/kubeflow/kfctl/issues/421 | kfserving | @yuzisun @animeshsingh @cliveseldon | https://github.com/kubeflow/manifests/pull/1600 https://github.com/kubeflow/manifests/pull/1575 | manifests | @Jeffwan | kubeflow/manifests#1597 metadata | | notebooks | @kimwnasptd , @jtfogarty | | pipelines | @Bobgy | https://github.com/kubeflow/manifests/pull/1605 | training | @johnugeorge @gaocegege @terrytangyuan | |

Platforms Platforms | release czar | Tracking Issue --- | --- | --- aws | @PatrickXYS | | azure | @aronchick | | ibm | @animeshsingh @adrian555 @shawnzhu | | gcp | @jlewi @Bobgy | | minikf | @vkoukis | |

Volumes Viewer

About two years ago, @DavidSpek proposed a pvcviewer. We've been using and enjoying this feature quite a lot since then and I'm convinced this feature needs to find its way to Kubeflow's core. Unfortunately, the PR staled and never got merged even though it sparked serious interest among the community.

This is my attempt to move forward with this feature, making it available for all users.

Pre-Existing Work

@DavidSpek provided a functional prototype of the volumes viewer. His work comprises a pvcviewer resource definition, a resource controller and changes to the volumes UI. His changes already got partially merged into the volumes UI but are only available within the rok distribution.

The initial controller was based off the tensorboard controller and was WIP in many regards.

My Work

I've re-implemented the volumes viewer, and made it compatible with the current volumes UI's master.

Also, the controller has been re-written from scratch and is well-tested. It now comprises new features, such as restarting pods on RWO-Nodename changes.

API

I've included an example of a viewer object. The API is similar to what @kimwnasptd descibed in another thread. Creating a VolumesViewer currently looks like this (comments included explaining its functionality).

apiVersion: kubeflow.org/v1alpha1
kind: VolumesViewer
metadata:
  name: volumesviewer-sample
  namespace: kubeflow-user-example-com
spec:
  # The podTemplate is applied to the deployment.Spec.Template.Spec
  # and thus, represents the core viewer's application
  podTemplate:
    #...
  service:
    # Specifies the application's target port used by the Service
    targetPort: 8080
    # If defined, an istio VirtualService is created, pointing to the Service
    virtualService:
      # The base prefix is suffixed by '/namespace/name' to create the
      # VirtualService's prefix and a unique URL for each started viewer
      basePrefix: "/volumesviewer"
      # You may specify the VirtualService's rewrite.
      # If not set, the prefix's value is used
      rewrite: "/"
      # By default, no timeout is set
      # timeout: 30s
  rwoScheduling:
    # If set to true, the controller detects RWO-Volumes referred to by the
    # podTemplate and uses affinities to schedule the viewer to nodes
    # where the volume is currently mounted. This enables the viewer to
    # access RWO-Volumes, even though they might already be mounted.
    enabled: true
    # Using the rwoScheduling feature, the viewer might block other application
    # from (re-starting). Setting restart to true instructs the controller to
    # re-compute the affinity in case Pods start using the viewer's RWO-Volumes.
    # Thus, the viewer might restart on another node without blocking new Pods.
    restart: true

How to install

Using the current kubeflow master:

1. Install the kustomize applications
   • kustomize build components/crud-web-apps/volumes/manifests/overlays/istio | kubectl apply -f -
   • kustomize build components/volumes-viewer/config/overlays/kubeflow | kubectl apply -f -
2. Build the container images and set them accordingly (alternatively, use my prebuilt images):
   • kubectl -n kubeflow set image deploy/volumes-web-app-deployment volumes-web-app=tobiasgoerke/kubeflow-volumes-web-app:test-v2
   • kubectl -n kubeflow set image deploy/volumes-viewer-controller-manager manager=tobiasgoerke/kubeflow-volumes-viewer:test-v2

Outlook and Future Work

• There is a generic viewer controller managed by the pipelines WG. The volumes viewer would integrate into this controller nicely. However, attempts to integrate the viewer have failed and it seems the viewer itself is not moving forward, too. Thus, I've decided to create a new controller (like @DavidSpek did in his PR) and call the integration into the pipelines WG optional and out-of-scope. In case this PR gets merged, I'll rename the VolumesViewer controller and push for it to be accepted as a generic viewer. This would enable the volumes viewer, notebooks, tensorboard etc. to use a common implementation and controller. For this PR, the CRD and controller could be dropped then, leaving us with only changes to the volumes UI. I've designed this PR with this possible step in mind so that a generic viewer would only require one file to change.
• I've created a PR in the filebrowser project which enables Filebrowser to support the tus.io protocol for resumable and chunked uploads. This comes in very handy with big uploads that may get disrupted or proxies that block large requests. This PR is currently in review.

Fix an issue in JWA's form and change the volume's mount path when the volume's name changes. This feature is already working for workspace volumes. Also, add one more e2e test for the volume's mount path.

This PR adds functionality for adding pod default related labels for tensorboard pods in tensorboard controller. This can be used for various things like configuring cloud storage parameters using environment variables(#6493), service account, tolerations etc.

User experience:

1. User creates a tensorboard CR with required labels (metadata.labels)
2. Controller will create the deployment with labels on the pod copied from the tensorboard CR
3. Admission webhook will take care of injecting necessary configurations according to poddefault

Updated the image version because v2.1.0 did not work with IAM roles. Using the latest version would require building and maintaining an image because some functionality is moved out to a different package(tensorflow-io). This package is not installed by default in tensorflow image and I would like to focus on completing the core functionality first.

@kimwnasptd @kandrio @elikatsis

Is there anyone who can help with implementing the webapp frontend related changes or I would appreciate if anyone can give me a quick intro on the frontend Angular part of the code so I can make the changes myself?

This PR addresses https://github.com/kubeflow/kubeflow/issues/6872

Currently there is a single build-all rule in the top-level Makefile under components/ dir that calls all sub-makefiles for building all KF images in a serial manner. This makes the whole process very time-consuming as we can't use the -j option of make to run jobs in parallel.

Changes to the top-level Makefile for building all KF images:

• Create a single rule for each directory containing a sub-makefile
• Have each directory rule as a dependency to the build-all rule in the central Makefile

This way every directory rule can run in parallel with other rules.

Similarly, the central makefile for building all the example-notebook-servers images calls each sub-Makefile for each of the notebook servers in a serial manner.

Changes to the central Makefile for building all notebook-server-images:

• Split the single target rule into multiple rules which perform recursive make calls (sub-makefiles) to build all the notebook-server-images.
• Use the variable MAKE for recursive make commands instead of explicit make command:
  • https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html#MAKE-Variable

Signed-off-by: Apostolos Gerakaris [email protected]

/kind feature

https://github.com/kubeflow/kubeflow/pull/6555 introduced a mechanism for building all Kubeflow images via a top-level central-Makefile. The central-Makefile has a build-all rule that calls all the sub-Makefiles of the different components in a serial manner. This makes the whole process very time-consuming since we can't use the -j option of make to run jobs in parallel.

We could speed-up this process by:

• creating a single rule for each directory containing a sub-Makefile.
• having each directory rule as a dependency to the build-all rule in the central Makefile

This way every directory rule can run in parallel with other rules.

The same applies for the central Makefile for building all notebook-server-images. In this case though we should follow a slightly different approach as we need to build the images in a specific order since most of the Dockerfiles expect a base image to build on top. The following graph shows the dependencies for building the images:

So, we could split the single docker-build-all rule into multiple rules which perform recursive make calls to sub-Makefiles to build all the notebook-server-images. We can introduce target rules for the following sub-Makefiles:

• jupyter-pytorch-full:
  • jupyter-pytorch-full-cpu
  • jupyter-pytorch-full-cuda
• jupyter-scipy
• jupyter-tensorflow-full:
  • jupyter-tensorflow-full-cpu
  • jupyter-tensorflow-full-cuda
• codeserver-python
• rstudio-tidyverse

This way we ensure that all notebook-server-images will be built as each sub-Makefile is responsible for building all the required base images.

What steps did you take and what happened: When trying to set limitation on cpu like this: op.set_cpu_limit(16) on ContainerOp and compile it using: kfp.compiler.Compiler().compile(my_pipeline, 'my-pipeline.zip') I get: TypeError: expected string or bytes-like object which is different when using gpu limit: op.set_gpu_limit(16) that compile well and working

What did you expect to happen: I expect those two function to act similarly.

Anything else you would like to add: I solved it by using op.set_cpu_limit('16')

Environment: kfp==1.6.2 python 3.8