Automate issue discovery for your projects against Lightning nightly and releases.

Last update: Dec 24, 2022

Overview

Automated Testing for Lightning EcoSystem Projects

Automate issue discovery for your projects against Lightning nightly and releases.
You get CPUs, Multi-GPUs testing for free, and Slack notification alerts if issues arise!

How do I add my own Project?

Pre-requisites

Here are pre-requisites for your project before adding to the Lightning EcoSystem CI:

Your project already includes some Python tests with PyTorch Lightning as a dependency
You'll be a contact/responsible person to resolve any issues that the CI finds in the future for your project

Adding your own project config

First, fork this project (with CLI or in browser) to be able to create a new Pull Request, and work within a specific branch.
```
gh repo fork PyTorchLightning/ecosystem-ci
cd ecosystem-ci/
```
Copy the template file in configs folder and call it <my_project_name>.yaml.
```
cp configs/template.yaml configs/<my_project_name>.yaml
```
At the minimum, modify the HTTPS variable to point to your repository. See Configuring my project for more options.
```
target_repository:
  HTTPS: https://github.com/MyUsername/MyProject.git
...
```
If your project tests multiple configurations or you'd like to test against multiple Lightning versions such as master and release branches, create a config file for each one of them. As an example, have a look at metrics master and metrics release CI files.

Add your config filename to either/both the GitHub CPU CI file or the Azure GPU CI file.

For example, for the GitHub CPU CI file we append our config into the pytest parametrization:

...
jobs:
  pytest:
    ...
        config:
          - "PyTorchLightning/metrics_pl-release.yaml"
          - "PyTorchLightning/transformers_pl-release.yaml"
          - "MyUsername/myproject-release.yaml"
        include:
          - {os: "ubuntu-20.04", python-version: "3.8", config: "PyTorchLightning/metrics_pl-master.yaml"}
          - {os: "ubuntu-20.04", python-version: "3.9", config: "PyTorchLightning/transformers_pl-master.yaml"}
          - {os: "ubuntu-20.04", python-version: "3.9", config: "MyUsername/my_project-master.yaml"}
        exclude:
          - {os: "windows-2019", config: "PyTorchLightning/transformers_pl-release.yaml"}
...

For example, in the Azure GPU CI file file:

...
jobs:
- template: testing-template.yml
  parameters:
    configs:
    - "PyTorchLightning/metrics_pl-master.yaml"
    - "PyTorchLightning/metrics_pl-release.yaml"
    - "MyUsername/my_project-master.yaml"

Add the responsible person(s) to CODEOWNERS for your organization folder or just the project.
```
# MyProject
/configs/Myusername/MyProject*    @Myusername
```
Finally, create a draft PR to the repo!

(Optional). [wip] join our Slack channel to be notified if your project is breaking

Configuring my project

The config include a few different sections:

target_repository include your project
env (optional) define any environment variables required when running tests
dependencies listing all dependencies which are taken outside pip
testing defines specific pytest arguments and what folders shall be tested

All dependencies as well as the target repository is sharing the same template with the only required field HTTPS and all others are optional:

target_repository:
  HTTPS: https://github.com/PyTorchLightning/metrics.git
  username: my-nick  # Optional, used when checking out private/protected repo
  password: dont-tell-anyone # Optional, used when checking out private/protected repo
  token: authentication-token # Optional, overrides the user/pass when checking out private/protected repo
  checkout: master # Optional, checkout a particular branch or a tag
  install_extras: all # Refers to standard pip option to install some additional dependencies defined with setuptools, typically used as `<my-package>[<install_extras>]`.

# Optional, if any installation/tests require some env variables
env:
   MY_ENV_VARIABLE: "VAR"

copy_tests:
    - integrations # copied folder from the original repo into the running test directory
    # this is copied as we use the helpers inside integrations as regular python package
    - tests/__init__.py
    - tests/helpers

# Optional, additional pytest arguments and control which directory to test on
testing:
  dirs:
    - integrations
  pytest_args: --strict

Note: If you define some files as done above, and they are using internal-cross imports, you need to copy the __init__.py files from each particular package level.

The testing section provides access to the pytest run args and command.

testing:
  # by default pytest is called on all copied items/tests
  dirs:
    - integrations
  # OPTIONAL, additional pytest arguments
  pytest_args: --strict

Comments

Fix flash CI
Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[ ] Did you set runtimes in config for GitHub action integration?

[ ] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Fixes # (issue) [add a link to the created issue in your repository if any] OR link your project for clarity.

Did you have fun?

Make sure you had fun coding 🙃
bug / fix
opened by ethanwharris 4
Add mridc
Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[X] Did you create/update your configuration file?

[X] Did you add your config to CI - GitHub action and Azure pipeline?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Fixes # (issue).

Did you have fun?

Make sure you had fun coding 🙃
New project
opened by wdika 4
Dockers for GPU testing
Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[ ] Did you create/update your configuration file?

[ ] Did you add your config to CI - GitHub action and Azure pipeline?

[x] Are all integration tests passing?

What does this PR do? [optional]

creating own docker image with PyTorch and sudo

Did you have fun?

Make sure you had fun coding 🙃
enhancement CI/CD
opened by Borda 4
Add NeMo
Before submitting

Starting with the release/CPU test

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[ ] Did you add your config to CI - GitHub action and Azure pipeline?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Resolves https://github.com/NVIDIA/NeMo/issues/3502

Did you have fun?

Make sure you had fun coding 🙃
New project
opened by SeanNaren 4
e2e gallery tests, invideo search app
Before submitting

[ ] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[ ] Did you create/update your configuration file?

[ ] Did you set runtimes in config for GitHub action integration?

[ ] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Fixes # (issue) [add a link to the created issue in your repository if any] OR link your project for clarity.

Did you have fun?

Make sure you had fun coding 🙃
opened by manskx 3
update org & repo name
Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[x] Did you set runtimes in config for GitHub action integration?

[x] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[x] Are all integration tests passing?

What does this PR do? [optional]

Update the org and repo name for pulling PL heads

Did you have fun?

Make sure you had fun coding 🙃
bug / fix Priority
opened by Borda 3
torchdyn
Before submitting

[ ] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[x] Did you set runtimes in config for GitHub action integration?

[x] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[x] Are all integration tests passing?

What does this PR do? [optional]

Integrates a torchdyn test that is coupled with PyTorchLightning Fixes # (issue) [add a link to the created issue in your repository if any] OR link your project for clarity.

Did you have fun?

Make sure you had fun coding 🙃
New project won't fix
opened by joglekara 3
Add DeepSpeed!
Closes https://github.com/microsoft/DeepSpeed/issues/1674.

Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[x] Did you add your config to CI - GitHub action and Azure pipeline?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Fixes # (issue).

Did you have fun?

Make sure you had fun coding 🙃
New project
opened by SeanNaren 3
add Solo-learn
Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[x] Did you add your config to CI - GitHub action and Azure pipeline?

[x] Are all integration tests passing?

What does this PR do? [optional]

Resolves https://github.com/vturrisi/solo-learn/issues/219

Did you have fun?

Make sure you had fun coding 🙃
New project
opened by Borda 3
Add differential build for PR

🚀 Feature

When a change is just for particular configs we want to build a configuration with includes these configs as in principal al configs shall be completely independent and so isolated... We may use GH CLI, ask if the change is only the one config and then skip all others...

Our CLI: input PR number and the config path and return bool if the config shall be build Logic: build particular configs in all changes are only in configs and this is one of the changed GH action: for execution steps add if: ... with skip if no need for building

Motivation

we want to be as efficient as possible so do not build anything which is not really needed also taking too many resources with growing ecosystem size may suck pool for other our projects as the pool org-wide shared

Pitch

the targeted build will speed up PR checks and lower the costs

Alternatives

Additional context
enhancement Priority CI/CD

opened by Borda 3
First pass at README changes
Before submitting

TODO:

Let's get a logo for the repo like lightning: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/docs/source/_static/images/logo.png

Add the eco system badges somewhere

Validate commands all work

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[ ] Did you create/update your configuration file?

[ ] Did you add your config to CI - GitHub action and Azure pipeline?

[x] Are all integration tests passing?

What does this PR do? [optional]

Fixes # (issue).

Did you have fun?

Make sure you had fun coding 🙃
documentation
opened by SeanNaren 3
adding metrics with `torch`
Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[x] Did you set runtimes in config for GitHub action integration?

[x] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Adding compatibility with future pyTorch

Did you have fun?

Make sure you had fun coding 🙃
enhancement
opened by Borda 1
adding PL & lite with `torch`
Before submitting

[x] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[x] Did you set runtimes in config for GitHub action integration?

[x] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Adding compatibility check for future PyTorch release

Did you have fun?

Make sure you had fun coding 🙃
enhancement
opened by Borda 1
Produce collection of passing integrations

🚀 Feature

see: https://github.com/Lightning-AI/lightning/issues/14137#issuecomment-1210635121

Motivation

Showcase all passing integrations on a special PL page

Pitch

Get more visibility of active collaborations

Alternatives

Additional context
enhancement

opened by Borda 0
Add PyKEEN to ecosystem-ci
Before submitting

[ ] Was this discussed/approved via a GitHub issue? (no need for typos and docs improvements)

[x] Did you create/update your configuration file?

[x] Did you set runtimes in config for GitHub action integration?

[x] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Project: https://github.com/pykeen/pykeen

PyL integration via

https://github.com/pykeen/pykeen/pull/905

https://github.com/pykeen/pykeen/pull/917

https://github.com/pykeen/pykeen/pull/930

Did you have fun?

Make sure you had fun coding 🙃
opened by mberr 6
Add NN Template
Before submitting

[x] Was this discussed/approved via a GitHub issue? Yes

[x] Did you create/update your configuration file? (WIP)

[ ] Did you set runtimes in config for GitHub action integration?

[ ] Did you add your config to CI in Azure pipeline (only projects with 100+ GitHub stars)?

[ ] Are all integration tests passing?

What does this PR do? [optional]

Add support for nn-template.

Did you have fun?

Always 🙃

Missing

Cookiecutter integration

The nn-template uses cookiecutter to generate a parametrized project from a template. Thus it is not enough to clone/checkout a repository to obtain a working project.

In our CI we are using the following to: (1) generate the project with cookiecutter and (2) by-pass the interactive setup (with the hacky echo command)

- name: Install Dependencies shell: bash -l {0} run: | pip install cookiecutter - name: Generate Repo shell: bash -l {0} run: | echo -e 'n\nn\nn\n' | cookiecutter . --no-input project_name=${{ env.COOKIECUTTER_PROJECT_NAME }}

Dependencies

This project uses a combination of conda and pip, most of the dependencies are specified in the setup.cfg file.

The idea is that the conda dependencies specified in the env.yaml are part of the "environment" (e.g. enabling the use of the torch docker image), and thus are not dependencies when installing the package with pip.

To configure an environment, thus, there are two options (after changing the working directory to the cookiecutter-generated project):

Using conda: conda env create -f env.yaml

Install the package into an existing environment with pip install ".[dev]"

I am not sure which is the best option here, if we want to test the development setup I think the 1. would be more adequate.

Complete config

Part of the file is still borrowed from the template config, e.g. the dependencies, runtimes and before_test (p.s. the template file indicated in the README as configs/template.yaml does not exists anymore).

These are the main challenges I see at the moment. Once the environment is active to run the tests (and maybe the pre-commits) it is enough to:

pre-commit install pre-commit run -v --all-files --show-diff-on-failure pytest -v

Please feel free to contribute to this PR if you have any spare time! Otherwise I will try to understand better how to solve these problems in the Lightning's ecosystem CI (not in the immediate future though!).

@Borda @rasbt @Flegyas
New project
opened by lucmos 1
Conda environment for specific dependencies

🚀 Feature

While the current container tests are run against bare python, there are many important libraries that require conda to be available to properly install dependencies. Therefore conda support would be the request.

Motivation

A use case for this in NeMo is Numba - we utilize it for custom CUDA kernels for RNNT loss, custom spec augment kernels and in the future for jit compiled CPU code as well. We have plenty of unit tests that are optional (and hence skipped) when these are not installed.

Another is Pynini (for WFST based text normalization and inverse text normalization) - however since the graph building takes significant time if not cached, we do not recommend performing this CPU test during your ecosystem tests. It is just an example.

The reason we prefer Numba to be installed from conda (whereas it does have a pip install path) is that often an increment in numba version is accompanied by an increment in the llvmlite library too - and this causes significant issues. llvmlite is a library that will often throw errors during upgrade, and it can be forced via --force-reinstall (for pip) but that brings about an unstable environment. Conda bypasses all of this.

Pitch

Install miniconda on the fly (or use a base container that supports conda).

Alternatives

Leave conda tests out, and request ecosystem members to make their tests skippable if dependencies are not installed. NeMo can do this (and currently does skip numba tests if its not installed or doesn't support a recent cuda version), it may be cumbersome on a large scale.
enhancement help wanted CI/CD let's do it

opened by titu1994 1

Owner

Pytorch Lightning

GitHub

This solves the autonomous driving issue which is supported by deep learning technology. Given a video, it splits into images and predicts the angle of turning for each frame.

Self Driving Car An autonomous car (also known as a driverless car, self-driving car, and robotic car) is a vehicle that is capable of sensing its env

4 Sep 4, 2021

Implementation for "Seamless Manga Inpainting with Semantics Awareness" (SIGGRAPH 2021 issue)

Seamless Manga Inpainting with Semantics Awareness [SIGGRAPH 2021](To appear) | Project Website | BibTex Introduction: Manga inpainting fills up the d

101 Jan 1, 2023

Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

Manga Filling with ScreenVAE SIGGRAPH ASIA 2020 | Project Website | BibTex This repository is for ScreenVAE introduced in the following paper "Manga F

30 Dec 24, 2022

Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

520 Dec 30, 2022

The spiritual successor to knockknock for PyTorch Lightning, get notified when your training ends

Who's there? The spiritual successor to knockknock for PyTorch Lightning, to get a notification when your training is complete or when it crashes duri

70 Oct 6, 2022

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

SoC-Winter Checkout some cool self-projects you can try your hands on to curb your boredom this December! These are short projects that you can do you