Bodywork deploys machine learning projects developed in Python, to Kubernetes.

This PR contains a single, but very large unit of refactoring. It removes/relocates the logic contained in:

• bodywork.workflow.BodyworkProject
• bodywork.stages.Stage
• bodywork.stages.BatchStage
• bodywork.stages.ServiceStage

And relocates it into complete refactoring of bodywork.config.BodyworkConfig. This scope of this class has changed enormously - because, it now opens and parses a YAML configuration file containing all of the configuration, for the entire project.

The change to a single config file enabled the opportunity to validate the config for the entire project in one pass, before any workflow has been started - this is now what is happening in bodywork.config.BodyworkConfig and why it is so large.

The Builder pattern has been implemented in bodywork.config.BodyworkConfig to create an easily extensible Bodywork config validator. Collectively, this refactoring has brought the following major advantages and new features:

• We no longer need config.ini and requirements.txt files in every stage directory. This completely separates code from config as can be seen in the new bodywork-test-project, which is now structured as a conventional Python project with a bodywork.yaml in the root directory - in precisely the same manner as many CICD services. that hook into GitHub, require.
• We now have the option of breaking the relationship between the name given to a stage, and the name of the directory that the executable module resides in. For any given stage, you now just provide the path to the executable to be used for that stage. This will allow users to re-use an executable module for multiple stages, without having to copy-paste anything. Also, this way Bodywork imposes absolutely nothing on the structure of the codebase 🙂 To build on this flexibility, I built-in the capability to pass parameters to each executable module, via the config file, so that a single executable module can be re-used in different ways, depending in which stage it's running in. This is demonstrated in bodywork-test-project, for stage_1.
• Because all the config can be validated in one pass, it has been possible to accumulate a running log of all config errors. This has been used to implement a new CLI command - $ bodywork validate that prints a validation report to stdout, that can be used to debug the config, before deployment is tested. This is great for user-experience!
• It is now much, much easier to extend and test config related code, without having to spawn new test resources (e.g. directories containing config.ini and requirements.txt files), each time a new parameter is added. This is basically how I slipped-in new features like passing args in config, so quickly.
• All the config and validation is now in one module, under a single interface, which makes the codebase easier to navigate and comprehend.

This PR improves the exception handling in git.py, so that users receive the git error when trying to download a git repo. This closes #37

Test fixture was also fixed to be compatible with Windows OS.

This PR closes #116 .

Changes mostly follow those described in #116.

When replicating the secrets to the target namespace the workflow will run in, all the secrets that belong to the secrets group specified will be copied. This occurs once per workflow before any stages are executed.

Created a separate Integration test file for Secrets as another small step towards splitting them out and cleaning up our tests.

This PR closes #111. Tweaked the existing code for setting up a namespace so the job-deployment accounts have permission to create namespaces. This in turn is called by the new command to configure the cluster.

This PR close #71. The commit hash is retrieved and added to both batch and service stages.

Some refactoring was also done in workflow_execution.py which consisted of breaking down the run_workflow method into smaller units.

This PR closes #120.

In addition to closing the original PR this also includes the removal of the service commands so these are all available under deployment. bodywork deployment display --namewill show all services belonging to that deployment andbodywork deployment display --service_name myserviceis equivalent toservice display --name myservice`.

During this I found out that the configure-cluster command was not giving the right permissions to the workflow service account because this was the first time the bodywork namespace was deleted since the V3 namespace related changes (this includes the cluster role). I effectively had to add all the permissions we had before, to the clusterrole, so that we can have the same permissions for multiple namespaces.

Because these permissions are being added to the entire cluster, I removed the permissions giving '*' access and went for a more restrictive set.

Integration tests have also been improved so that they use single stage projects unless they require the multi-stage project. This should reduce the strain on the cluster and also stop tests impacting each other as each project uses a different namespace.

This PR closes #112 .

Implementation differs from the original issue requirements in the following way:

• Job name is post-fixed with a small random number instead of a timestamp because there is a 52 character limit on the job name.
• Cronjobs will have a static name as defined by the user as this best fits how people like to use cronjobs.
• Namespace left as a parameter in lower level package functions as I see no gain in removing them and might come useful in later iterations.

When trying to deploy a private repo via SSH using,

$ bodywork deployment create \
    --namespace=arc-cpre \
    --name=d1 \
    [email protected]/everlution/arc-cpre.git \
    --git-repo-branch=rest-api-definition \
    -L

I got the following unhandled exception:

testing with local workflow-controller - retries are inactive
namespace=arc-cpre is setup for use by Bodywork
2021-06-22 19:48:29,882 - INFO - workflow_execution.run_workflow - attempting to run workflow for [email protected]/everlution/arc-cpre.git on branch=rest-api-definition in kubernetes namespace=arc-cpre
2021-06-22 19:48:29,939 - ERROR - workflow_execution.run_workflow - failed to execute workflow for rest-api-definition branch of project repository at [email protected]/everlution/arc-cpre.git: Unable to setup SSH for Github and you are trying to connect via SSH: 'failed to setup SSH for github.com - cannot find BODYWORK_GIT_SSH_PRIVATE_KEY environment variable'
Traceback (most recent call last):
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/git.py", line 63, in download_project_code_from_repo
    setup_ssh_for_git_host(hostname)
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/git.py", line 137, in setup_ssh_for_git_host
    raise KeyError(msg)
KeyError: 'failed to setup SSH for github.com - cannot find BODYWORK_GIT_SSH_PRIVATE_KEY environment variable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/workflow_execution.py", line 81, in run_workflow
    download_project_code_from_repo(repo_url, repo_branch, cloned_repo_dir)
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/git.py", line 74, in download_project_code_from_repo
    raise BodyworkGitError(msg)
bodywork.exceptions.BodyworkGitError: Unable to setup SSH for Github and you are trying to connect via SSH: 'failed to setup SSH for github.com - cannot find BODYWORK_GIT_SSH_PRIVATE_KEY environment variable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/workflow_execution.py", line 156, in run_workflow
    if config.project.run_on_failure and type(e) not in [
UnboundLocalError: local variable 'config' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/workflow_execution.py", line 167, in run_workflow
    f"Error executing failure stage: {config.project.run_on_failure}"
UnboundLocalError: local variable 'config' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/bin/bodywork", line 8, in <module>
    sys.exit(cli())
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/cli/cli.py", line 302, in cli
    args.func(args)
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/cli/cli.py", line 324, in wrapper
    func(*args, **kwargs)
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/cli/cli.py", line 393, in deployment
    workflow(pass_through_args)
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/cli/cli.py", line 324, in wrapper
    func(*args, **kwargs)
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/cli/cli.py", line 562, in workflow
    run_workflow(
  File "/Users/alexioannides/Dropbox/bodywork_client_repos/arc-CPRE/.venv/lib/python3.8/site-packages/bodywork/workflow_execution.py", line 176, in run_workflow
    if config is not None and config.project.usage_stats:
UnboundLocalError: local variable 'config' referenced before assignment

This PR fulfills the changes required to Bodywork-core for #73. Every time the workflow is executed it will ping the usage trackage server (unless explicitly configured not to) and fail silently if it cannot be reached.

The Constant containing the URL for the tracking server needs to be updated with the tracking server URL once this is available.

Fixed so that the git-branch is shown and removed the stage name field as this is the same as the name field, therefore displaying duplicate data.

This commit also contains a fix for the CI/CD build:

• Upgrade deprecated Ubuntu image for docker builds.
• Fix Docker Dev Build Step after upgrade.

This hotfix targets two separate issues:

1. While putting together an example pipeline project for the next blog post, I run into several issues when using a pipeline/ folder for the pipeline code and then trying to import modules as if pipeline was a Python package module. These can be solved by adding the project's root directory to PYTHONPATH. While this isn't strictly necessary, it leads to a much better developer experience (and probably reduces the risk of confusion).

• By accident, I tried to run a workflow without a bodywork.yaml file and I got a cascade of exceptions in the face. This is because BodyworkConfig throws a FileExistsError exception, which we don't handle in run_workflow. I thought about creating a custom exception that inherits from BodyworkConfigError, but that seemed overkill (need your opinion here).

"As a ML Engineer, I would like to be able to use Bodywork images hosted on a private image registry, so that I can use images that have passed security scanning as mandated by my place of work."

Tasks

• Investigate how image pull secrets could be configured for private image repositories. Could we use the existing secrets mechanism?
• Investigate how images hosted on private image repositories can be pinged to check that they exist.
• Plan the work required to develop the required functionality.

"As a ML Engineer, I would like Bodywork to support the newest version of Kuberentes possible, so that I can use Bodywork with cluster running later versions of Kubernetes."

Tasks

• Upgrade the Kubernetes Python client and check to see what versions of Kubernetes it is officially compatible with.
• Run integration tests on clusters with the lowest and highest supported versions of Kubernetes (via Minikube).
• Update the documentation with the latest supported version of Kubernetes.

"As a ML Engineer, I would like to know why my Jupyter notebook stage has failed, so that I can fix the issue."

Tasks

• Investigate what exceptions nbconvert.preprocessors.ExecutePreprocessor().preprocess throws when a notebook raises an exception.
• Within bodywork.stage_execution.run_stage ensure that these exceptions are handled individually and that the resulting error message to the user is informative.
• Consider executing the notebook via the CLI, if this makes managing the notebook execution easier.

"As a ML Engineer, I would like error messages that contain suggestions for how to fix the problem, so that I am enabled to make progress with my deployments."

Tasks

• Review all exceptions in bodywork.workflow_execution.run_workflow and ensure that, where possible, they contain suggestions for how to fix the problem.
• Review all exceptions in bodywork.stage_execution.run_stage and ensure that, where possible, they contain suggestions for how to fix the problem.
• When creating a secret or cronjob, without having configured the cluster first, ensure that the message to the user informs them that they ought to run bw configure-cluster.

"As a newly recruited Bodywork Developer, I would like to know how to configure all developer tools, so that I can start making contributions quickly."

Tasks

• Add a Contributing section to the Bodywork docs.
• Add a section that links to it in the project's README.md.
• Add a CONTRIBUTING.md file to the project.

"As a Bodywork Developer, I would like code formatting to be executed automatically prior to making a commit, so that I don't have to remember to run all the formatting tools and each commit is consistently formatted."

Tasks

• Read about the pre-commit package.
• Setup the pre-commit package.
• Add Black, isort and pydocstyle as pre-commit hooks.
• Document the installation steps required by new developers.

Requires #197, #196 to be completed first.