RAD Lab enables users to deploy infrastructure on Google Cloud Platform (GCP) to support specific use cases. Infrastructure is created and managed through Terraform in conjunction with support scripts written in Python. The templates, code, and documentation for each use case are bundled into modules.

Google Cloud Platform

Last update: Jan 4, 2023

Related tags

Deep Learning rad-lab

Overview

RAD Lab

RAD Lab enables users to deploy infrastructure on Google Cloud Platform (GCP) to support specific use cases. Infrastructure is created and managed through Terraform in conjunction with support scripts written in Python. The templates, code, and documentation for each use case are bundled into modules.

Each module comes with an initial sample configuration that can be deployed via the RAD Lab Launcher. Teams with experience in Terraform can use the Terraform modules in the /modules-directory directly, to integrate the codebase with their existing CI/CD infrastructure.

For any issues, please create an issue in the Issue tracker of the repository, following the provided templates.

Disclaimer

This is not an officially supported Google product

Quick Start

To create a module in an existing GCP environment, please click on the “Open In Cloud Shell” button. This will clone the entire repository to Cloud Shell and automatically select the main-branch.

Prerequisites

terraform version 1.0 or higher.
gcloud SDK version 360.0.0 or higher. (ONLY for RAD Lab Launcher)
python version 3.7.3 or higher. (ONLY for RAD Lab Launcher)

Users can run installer_prereq.py included in the radlab-launcher directory to validate these prerequisites and install any missing dependencies.

RAD Lab Launcher

An installation script is included to enable users without prior cloud or Terraform experience to explore the advantages of GCP. The deployment wizard will create the necessary infrastructure for each module. More instructions on the launcher can be found here.

Modules

The modules directory contains Terraform modules that can be integrated in existing CI/CD pipelines. We recommend creating a fork from this repository and use that as part of your overall workflow to create infrastructure. While we will make an effort to provide backwards compatibility, we cannot guarantee it with every release.

Permissions

Each module will list the minimum IAM permissions users require to successfully create the infrastructure.

Additional permissions are required when deploying the RAD Lab modules via RAD Lab Launcher

Repository Structure

The repository has the following structure:

/docs: General documentation about the repository and how to use it.
/modules: Customisable, reusable Terraform modules to create GCP infrastructure & supporting scripts.
/radlab-launcher: RAD Lab Launcher and associated scripts to support the modules.
/tools: Automation tools to generate the necessary documentation and license checks.

Contributing

We welcome all contributions! Please read CONTRIBUTING.md for more information on how to publish your contributions.

Comments

modules/silicon_design: use deeplearning images
replace container base image with deeplearning image

provision eda tooling from conda-eda

add daisy-based workflow to build compute engine image

add compute binding for cloudbuild service account

reduce image build time to ~15min

module.simple.null_resource.build_and_push_image: Creation complete after 16m18s [id=8907064954708688601]

/cc @mithro @QuantamHD @hzeller
opened by proppy 89
modules: add silicon design solution
Add module for silicon/EDA solution:

forked from datascience module

use container based notebook

add cloud build recipe for container image (opennlane + pdk + jupyterlab)

Note: left the original data_science commit to ease review (diff) but happy to rebase and squash if needed.
opened by proppy 10
RE #29 | Enhancement on RAD Lab Launcher
Below enhancements are included in this PR:

Spinning up radlab modules via RAD Lab Launcher without an Organization

Moving away from using 'gsutil' to using code/SDK for cloud storage certain operations
opened by guptamukul-google 5
Releasing new RAD Lab Module: Web Hosting

This RAD Lab module builds a 3-tier application architecture which is a modular client-server architecture that consists of a presentation tier, an application tier and a data tier. We spin up a sample app aka registration form which takes data from UI (Presentation Layer), process the data on GCE servers (application layer) and selects or insert data into Cloud SQL Postgres DB (database layer)

opened by guptamukul-google 4
WORKAROUND: Associating PubSub to Billing budget

Introducing Workaround for Billing Budget Issue [1]: Billing Budget creation failing when adding pubsub_topic rule without monitoring_notification_channels

[1] https://github.com/hashicorp/terraform-provider-google/issues/12613

opened by guptamukul-google 4
RAD Lab Module Feature Enhancements
Added capability to deploy modules via Service Account Identity.

Added capability to enable Billing Budgets to all the modules.

Added Monitoring channels for Billing Budgets.

(For future purposes) Added UIMeta to variables.tf of every module.
opened by guptamukul-google 4
Updating modules & RAD Lab Launcher for #29
Usage of rad-lab with a personal account i.e. without access to an organization.

Made Org ID as one of the optional variable.

Either Org ID or Folder ID will be required to spin up the project.
opened by guptamukul-google 4
Feature/project ownership

I've added the ability to create additional Owners for the different modules. This gives the ability to grant more people access to the different projects.

opened by debakkerb 3

cromwell-genomics: "caller does not have permission, forbidden" when deploying outside an organization

What are you trying to accomplish? I am deploying into an existing project.

tfvars:

billing_account_id = "<BILLING-ACCOUNT>"
project_name       = "<PROJECT>"
create_project     = false
organization_id    = null

For organization ID, I also tried "" with the same result.

What challenge are you running into?

I see this error, despite the fact that I am not running inside an organization.

google_project_organization_policy.shielded_vm_policy[0]: Creating...
google_project_organization_policy.restrict_vpc_peering_policy[0]: Creating...
google_project_organization_policy.trustedimage_project_policy[0]: Creating...
╷
│ Error: googleapi: Error 403: The caller does not have permission, forbidden
│ 
│   with google_project_organization_policy.restrict_vpc_peering_policy[0],
│   on orgpolicy.tf line 33, in resource "google_project_organization_policy" "restrict_vpc_peering_policy":
│   33: resource "google_project_organization_policy" "restrict_vpc_peering_policy" {
│ 
╵
╷
│ Error: googleapi: Error 403: The caller does not have permission, forbidden
│ 
│   with google_project_organization_policy.shielded_vm_policy[0],
│   on orgpolicy.tf line 50, in resource "google_project_organization_policy" "shielded_vm_policy":
│   50: resource "google_project_organization_policy" "shielded_vm_policy" {
│ 
╵
╷
│ Error: googleapi: Error 403: The caller does not have permission, forbidden
│ 
│   with google_project_organization_policy.trustedimage_project_policy[0],
│   on orgpolicy.tf line 63, in resource "google_project_organization_policy" "trustedimage_project_policy":
│   63: resource "google_project_organization_policy" "trustedimage_project_policy" {
│ 
╵

What have your tried so far?

See above.

question

opened by seandavi 3

Map type fields are not readable in RAD Lab UI deployment
Describe the bug Map type fields are not readable in RAD Lab UI deployment

To Reproduce Steps to reproduce the behavior:

Go to RAD Lab UI

Try deploying NEON module which has map field

See the default value of Ah Listing Dataset Map

Expected behavior RAD Lab UI should show clear and readable values in the map type fields

Screenshots
bug
opened by guptamukul-google 1
RAD Lab UI spinning for Outputs of the Deleted module
Describe the bug RAD Lab UI spinning for Outputs of the Deleted module.

To Reproduce Steps to reproduce the behavior:

Delete a RAD Lab module via UI

Go to the Deployment specific Overview page

Select Output tab

See the spinning wheel with below message:

Terraform outputs will be generated when a module is deployed successfully.

Expected behavior After the modules are deployed no outputs are suppose to get generated. Thus once the deployment is successfully deleted remove all the outputs and display some other message.

Screenshots
bug
opened by guptamukul-google 1

Releases(v10.0.0)

v10.0.0(Nov 15, 2022)
Released RAD Lab Documentation site & RAD Lab UI.

Released New RAD Lab Module : NEON Module

Added/Changed/Fixed RAD Lab modules functionalities:

Fixed deployment issue for Silicon Design module.

Updated nodepool image in AppModElastic module.

Removed hardcoded references for zones in WebHosting module.

Source code(tar.gz)
Source code(zip)
v9.1.1(Oct 21, 2022)
Added/Changed/Fixed RAD Lab modules functionalities:

Added Project Owners to the RAD Lab module deployment.

Fixed permission issues for Managed notebooks in Data Science module.

Source code(tar.gz)
Source code(zip)
v9.0.0(Oct 20, 2022)
Released New RAD Lab Module : Web Hosting (3-tier architecture) Module

Added/Changed/Fixed RAD Lab modules functionalities:

Added PubSub Topic & Subscription for Billing Budgets.

Using deeplearning images in Silicon Design module.

Added permission to trusted users/groups for Genomics Cromwell Module.

Fixed the deployment issue for App Mod Elastic RAD Lab module.

Source code(tar.gz)
Source code(zip)
v8.0.0(Oct 20, 2022)
Added/Changed/Fixed RAD Lab modules functionalities:

Added capability to deploy modules via Service Account Identity.

Added capability to enable Billing Budgets to all the modules.

Added Monitoring channels for Billing Budgets.

Added/Changed/Fixed RAD Lab Launcher functionalities:

Updated RAD Lab Launcher to support changing folder_id.

Source code(tar.gz)
Source code(zip)
v7.0.0(May 25, 2022)
Released New RAD Lab Module : GCP Project Billing Budget Module

Added/Changed/Fixed RAD Lab modules functionalities:

Fixed Cloud NAT issue in modules/genomics_cromwell/

Added/Changed/Fixed RAD Lab Launcher functionalities:

Fixed the execution of gcloud commands within provisioner "local-exec"

Added/Changed/Fixed RAD Lab repo functionalities:

Added a tool to create Terraform based GCP Service Catalog solutions for RAD Lab modules

Source code(tar.gz)
Source code(zip)
v6.1.1(Apr 26, 2022)
Added/Changed/Fixed RAD Lab Launcher functionalities:

Listing Existing Deployments when Updating/Deleting any deployment.

Fix for empty line in the file of --varfile used to cause error of "index out of range" and fail the execution of python3 radlab.py --module.... --varfile ./xxxxx.

Source code(tar.gz)
Source code(zip)
v6.0.0(Apr 19, 2022)

Released New RAD Lab Module : AlphaFold Module
Source code(tar.gz)
Source code(zip)
v5.1.1(Mar 18, 2022)
Added/Changed/Fixed RAD Lab modules functionalities:

Added option to create container image Vertex AI notebooks in Data Science Module

Added capability for users to upload their own Jupyter notebooks to Vertex AI notebooks on Data Science Module deployment.

Added/Changed/Fixed RAD Lab Launcher functionalities:

Enhancement for Launcher to accept Command Line arguments.

Added a CLI argument / flag to disable RADLab Permission pre-checks.

Fixed links in the Error Messages.

Fixing a bug to also check for Owner role (i.e. roles/owner) which already contains all required permission for pre-checks for RAD Lab management project

Source code(tar.gz)
Source code(zip)
v5.0.0(Feb 24, 2022)

Released New RAD Lab Module : Silicon Design Module
Source code(tar.gz)
Source code(zip)
v4.2.1(Feb 18, 2022)
Added/Changed/Fixed RAD Lab modules functionalities:

Resolved conflicting iam binding of notebooks.admin for data_science module.

Added/Changed/Fixed RAD Lab Launcher functionalities:

Skipped installation of Terraform binaries on Cloud Shell as TF is automatically authenticated and is integrated with Cloud Shell.

Source code(tar.gz)
Source code(zip)
v4.1.0(Feb 15, 2022)
Added/Changed/Fixed RAD Lab modules functionalities:

Standardized Project ID prefix for all RAD Lab Module

Added/Changed/Fixed RAD Lab Launcher functionalities:

Pre-Check IAM Permissions before deploying RAD Lab Modules via RAD Lab Launcher.

RAD Lab Launcher to prompt for either proceed with the existing GOOGLE_APPLICATION_CREDENTIALS or go through the login again.

Source code(tar.gz)
Source code(zip)
v4.0.0(Jan 14, 2022)
Released New RAD Lab Module : Genomics-Cromwell Module

Added/Changed/Fixed RAD Lab modules functionalities:

Updated AI Notebook Image to include sample notebook for Data Science RAD Lab Module.

Added/Changed/Fixed RAD Lab Launcher functionalities:

Spinning up radlab modules via RAD Lab Launcher without an Organization.

Launcher to prompt for Project ID for RAD Lab management.

Added/Changed/Fixed Repository functionalities:

Set up permissions in the GitHub Action workflows.

Fixed GitHub Action Workflow - Build Module README.

Source code(tar.gz)
Source code(zip)
v3.0.0(Dec 6, 2021)
Released New RAD Lab Module : Genomics-databiosphere dsub Module

Source code(tar.gz)
Source code(zip)
v2.1.0(Nov 23, 2021)
Added/Changed/Fixed RAD Lab Launcher functionalities:

Added optional Input file parameter to set variables & create terraform.tfvars file

Changed how module selection works & is now compatible with any other module in future without any edits on the radlab.py

Source code(tar.gz)
Source code(zip)
v2.0.0(Nov 16, 2021)
Released New RAD Lab Module : Application Modernization Module (w/ Elasticsearch)

Added/Changed/Fixed RAD Lab modules functionalities:

Deploy RAD Lab modules in existing GCP projects / networks / subnets

Added timer to wait for Org Policy to get rolled out.

Added/Changed/Fixed RAD Lab Launcher functionalities:

List existing RAD Lab deployments for specific modules

Prompt for manually input Org ID & Billing A/C ID

Source code(tar.gz)
Source code(zip)
v1.1.0(Nov 2, 2021)
Added/Changed/Fixed Repository functionalities:

Added supporting tools , documentation & scripts to support RAD Lab modules.

Added Issue Templates, Workflows, Codeowners & Changelog to support RAD Lab repo.

Source code(tar.gz)
Source code(zip)
v1.0.0(Oct 7, 2021)
Released New RAD Lab Module : Data Science Module

Source code(tar.gz)
Source code(zip)

Related tags

Overview

RAD Lab

Disclaimer

Quick Start

Prerequisites

RAD Lab Launcher

Modules

Permissions

Repository Structure

Contributing

Comments

Releases(v10.0.0)

v10.0.0(Nov 15, 2022)

v9.1.1(Oct 21, 2022)

v9.0.0(Oct 20, 2022)

v8.0.0(Oct 20, 2022)

v7.0.0(May 25, 2022)

v6.1.1(Apr 26, 2022)

v6.0.0(Apr 19, 2022)

v5.1.1(Mar 18, 2022)

v5.0.0(Feb 24, 2022)

v4.2.1(Feb 18, 2022)

v4.1.0(Feb 15, 2022)

v4.0.0(Jan 14, 2022)

v3.0.0(Dec 6, 2021)

v2.1.0(Nov 23, 2021)

v2.0.0(Nov 16, 2021)

v1.1.0(Nov 2, 2021)

v1.0.0(Oct 7, 2021)

Owner

Google Cloud Platform

The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

GRaNDPapA: Generator of Rad Names from Decent Paper Acronyms

A python-image-classification web application project, written in Python and served through the Flask Microframework. This Project implements the VGG16 covolutional neural network, through Keras and Tensorflow wrappers, to make predictions on uploaded images.

Image Captioning on google cloud platform based on iot

piSTAR Lab is a modular platform built to make AI experimentation accessible and fun. (pistar.ai)

A trusty face recognition research platform developed by Tencent Youtu Lab

ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS.

Torch-mutable-modules - Use in-place and assignment operations on PyTorch module parameters with support for autograd

Source code for our paper "Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash"

Interactive Terraform visualization. State and configuration explorer.

PyStan, a Python interface to Stan, a platform for statistical modeling. Documentation: https://pystan.readthedocs.io

This is a Machine Learning Based Hand Detector Project, It Uses Machine Learning Models and Modules Like Mediapipe, Developed By Google!

This solves the autonomous driving issue which is supported by deep learning technology. Given a video, it splits into images and predicts the angle of turning for each frame.

In the case of your data having only 1 channel while want to use timm models

Turning SymPy expressions into PyTorch modules.

Covid19-Forecasting - An interactive website that tracks, models and predicts COVID-19 Cases

I have created this Virtual Paint Program, in this you can paint(draw) on your screen using hand gestures, created in Python-3 using OpenCV and Mediapipe library. Gestures :- Index Finger for drawing and Index+Middle Finger for changing position and objects.

Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)