RAD Lab enables users to deploy infrastructure on Google Cloud Platform (GCP) to support specific use cases. Infrastructure is created and managed through Terraform in conjunction with support scripts written in Python. The templates, code, and documentation for each use case are bundled into modules.

Overview

RAD Lab

RAD Lab enables users to deploy infrastructure on Google Cloud Platform (GCP) to support specific use cases. Infrastructure is created and managed through Terraform in conjunction with support scripts written in Python. The templates, code, and documentation for each use case are bundled into modules.

Each module comes with an initial sample configuration that can be deployed via the RAD Lab Launcher. Teams with experience in Terraform can use the Terraform modules in the /modules-directory directly, to integrate the codebase with their existing CI/CD infrastructure.

For any issues, please create an issue in the Issue tracker of the repository, following the provided templates.

Disclaimer

This is not an officially supported Google product

Quick Start

To create a module in an existing GCP environment, please click on the “Open In Cloud Shell” button. This will clone the entire repository to Cloud Shell and automatically select the main-branch.

Open in Cloud Shell

Prerequisites

Users can run installer_prereq.py included in the radlab-launcher directory to validate these prerequisites and install any missing dependencies.

RAD Lab Launcher

An installation script is included to enable users without prior cloud or Terraform experience to explore the advantages of GCP. The deployment wizard will create the necessary infrastructure for each module. More instructions on the launcher can be found here.

Modules

The modules directory contains Terraform modules that can be integrated in existing CI/CD pipelines. We recommend creating a fork from this repository and use that as part of your overall workflow to create infrastructure. While we will make an effort to provide backwards compatibility, we cannot guarantee it with every release.

Permissions

Each module will list the minimum IAM permissions users require to successfully create the infrastructure.

Additional permissions are required when deploying the RAD Lab modules via RAD Lab Launcher

Repository Structure

The repository has the following structure:

  • /docs: General documentation about the repository and how to use it.
  • /modules: Customisable, reusable Terraform modules to create GCP infrastructure & supporting scripts.
  • /radlab-launcher: RAD Lab Launcher and associated scripts to support the modules.
  • /tools: Automation tools to generate the necessary documentation and license checks.

Contributing

We welcome all contributions! Please read CONTRIBUTING.md for more information on how to publish your contributions.

Comments
  • modules/silicon_design: use deeplearning images

    modules/silicon_design: use deeplearning images

    • replace container base image with deeplearning image
    • provision eda tooling from conda-eda
    • add daisy-based workflow to build compute engine image
    • add compute binding for cloudbuild service account
    • reduce image build time to ~15min
    module.simple.null_resource.build_and_push_image: Creation complete after 16m18s [id=8907064954708688601]
    

    /cc @mithro @QuantamHD @hzeller

    opened by proppy 89
  • modules: add silicon design solution

    modules: add silicon design solution

    Add module for silicon/EDA solution:

    • forked from datascience module
    • use container based notebook
    • add cloud build recipe for container image (opennlane + pdk + jupyterlab)

    Note: left the original data_science commit to ease review (diff) but happy to rebase and squash if needed.

    opened by proppy 10
  • RE #29 | Enhancement on RAD Lab Launcher

    RE #29 | Enhancement on RAD Lab Launcher

    Below enhancements are included in this PR:

    • Spinning up radlab modules via RAD Lab Launcher without an Organization
    • Moving away from using 'gsutil' to using code/SDK for cloud storage certain operations
    opened by guptamukul-google 5
  • Releasing new RAD Lab Module: Web Hosting

    Releasing new RAD Lab Module: Web Hosting

    This RAD Lab module builds a 3-tier application architecture which is a modular client-server architecture that consists of a presentation tier, an application tier and a data tier. We spin up a sample app aka registration form which takes data from UI (Presentation Layer), process the data on GCE servers (application layer) and selects or insert data into Cloud SQL Postgres DB (database layer)

    opened by guptamukul-google 4
  • WORKAROUND: Associating PubSub to Billing budget

    WORKAROUND: Associating PubSub to Billing budget

    Introducing Workaround for Billing Budget Issue [1]: Billing Budget creation failing when adding pubsub_topic rule without monitoring_notification_channels

    [1] https://github.com/hashicorp/terraform-provider-google/issues/12613

    opened by guptamukul-google 4
  • RAD Lab Module Feature Enhancements

    RAD Lab Module Feature Enhancements

    • Added capability to deploy modules via Service Account Identity.
    • Added capability to enable Billing Budgets to all the modules.
    • Added Monitoring channels for Billing Budgets.
    • (For future purposes) Added UIMeta to variables.tf of every module.
    opened by guptamukul-google 4
  • Updating modules & RAD Lab Launcher for #29

    Updating modules & RAD Lab Launcher for #29

    Usage of rad-lab with a personal account i.e. without access to an organization.

    • Made Org ID as one of the optional variable.
    • Either Org ID or Folder ID will be required to spin up the project.
    opened by guptamukul-google 4
  • Feature/project ownership

    Feature/project ownership

    I've added the ability to create additional Owners for the different modules. This gives the ability to grant more people access to the different projects.

    opened by debakkerb 3
  • cromwell-genomics:

    cromwell-genomics: "caller does not have permission, forbidden" when deploying outside an organization

    What are you trying to accomplish? I am deploying into an existing project.

    tfvars:

    billing_account_id = "<BILLING-ACCOUNT>"
    project_name       = "<PROJECT>"
    create_project     = false
    organization_id    = null 
    

    For organization ID, I also tried "" with the same result.

    What challenge are you running into?

    I see this error, despite the fact that I am not running inside an organization.

    google_project_organization_policy.shielded_vm_policy[0]: Creating...
    google_project_organization_policy.restrict_vpc_peering_policy[0]: Creating...
    google_project_organization_policy.trustedimage_project_policy[0]: Creating...
    ╷
    │ Error: googleapi: Error 403: The caller does not have permission, forbidden
    │ 
    │   with google_project_organization_policy.restrict_vpc_peering_policy[0],
    │   on orgpolicy.tf line 33, in resource "google_project_organization_policy" "restrict_vpc_peering_policy":
    │   33: resource "google_project_organization_policy" "restrict_vpc_peering_policy" {
    │ 
    ╵
    ╷
    │ Error: googleapi: Error 403: The caller does not have permission, forbidden
    │ 
    │   with google_project_organization_policy.shielded_vm_policy[0],
    │   on orgpolicy.tf line 50, in resource "google_project_organization_policy" "shielded_vm_policy":
    │   50: resource "google_project_organization_policy" "shielded_vm_policy" {
    │ 
    ╵
    ╷
    │ Error: googleapi: Error 403: The caller does not have permission, forbidden
    │ 
    │   with google_project_organization_policy.trustedimage_project_policy[0],
    │   on orgpolicy.tf line 63, in resource "google_project_organization_policy" "trustedimage_project_policy":
    │   63: resource "google_project_organization_policy" "trustedimage_project_policy" {
    │ 
    ╵
    

    What have your tried so far?

    See above.

    question 
    opened by seandavi 3
  • Map type fields are not readable in RAD Lab UI deployment

    Map type fields are not readable in RAD Lab UI deployment

    Describe the bug Map type fields are not readable in RAD Lab UI deployment

    To Reproduce Steps to reproduce the behavior:

    1. Go to RAD Lab UI
    2. Try deploying NEON module which has map field
    3. See the default value of Ah Listing Dataset Map

    Expected behavior RAD Lab UI should show clear and readable values in the map type fields

    Screenshots Screen Shot 2022-12-16 at 3 17 55 PM

    bug 
    opened by guptamukul-google 1
  • RAD Lab UI spinning for Outputs of the Deleted module

    RAD Lab UI spinning for Outputs of the Deleted module

    Describe the bug RAD Lab UI spinning for Outputs of the Deleted module.

    To Reproduce Steps to reproduce the behavior:

    1. Delete a RAD Lab module via UI
    2. Go to the Deployment specific Overview page
    3. Select Output tab
    4. See the spinning wheel with below message:

    Terraform outputs will be generated when a module is deployed successfully.

    Expected behavior After the modules are deployed no outputs are suppose to get generated. Thus once the deployment is successfully deleted remove all the outputs and display some other message.

    Screenshots Screen Shot 2022-12-15 at 10 29 21 AM

    bug 
    opened by guptamukul-google 1
Releases(v10.0.0)
  • v10.0.0(Nov 15, 2022)

    • Released RAD Lab Documentation site & RAD Lab UI.
    • Released New RAD Lab Module : NEON Module

    Added/Changed/Fixed RAD Lab modules functionalities:

    • Fixed deployment issue for Silicon Design module.
    • Updated nodepool image in AppModElastic module.
    • Removed hardcoded references for zones in WebHosting module.
    Source code(tar.gz)
    Source code(zip)
  • v9.1.1(Oct 21, 2022)

    Added/Changed/Fixed RAD Lab modules functionalities:

    • Added Project Owners to the RAD Lab module deployment.
    • Fixed permission issues for Managed notebooks in Data Science module.
    Source code(tar.gz)
    Source code(zip)
  • v9.0.0(Oct 20, 2022)

    • Released New RAD Lab Module : Web Hosting (3-tier architecture) Module

    Added/Changed/Fixed RAD Lab modules functionalities:

    • Added PubSub Topic & Subscription for Billing Budgets.
    • Using deeplearning images in Silicon Design module.
    • Added permission to trusted users/groups for Genomics Cromwell Module.
    • Fixed the deployment issue for App Mod Elastic RAD Lab module.
    Source code(tar.gz)
    Source code(zip)
  • v8.0.0(Oct 20, 2022)

    Added/Changed/Fixed RAD Lab modules functionalities:

    • Added capability to deploy modules via Service Account Identity.
    • Added capability to enable Billing Budgets to all the modules.
    • Added Monitoring channels for Billing Budgets.

    Added/Changed/Fixed RAD Lab Launcher functionalities:

    • Updated RAD Lab Launcher to support changing folder_id.
    Source code(tar.gz)
    Source code(zip)
  • v7.0.0(May 25, 2022)

    • Released New RAD Lab Module : GCP Project Billing Budget Module
    • Added/Changed/Fixed RAD Lab modules functionalities:
    1. Fixed Cloud NAT issue in modules/genomics_cromwell/
    • Added/Changed/Fixed RAD Lab Launcher functionalities:
    1. Fixed the execution of gcloud commands within provisioner "local-exec"
    • Added/Changed/Fixed RAD Lab repo functionalities:
    1. Added a tool to create Terraform based GCP Service Catalog solutions for RAD Lab modules
    Source code(tar.gz)
    Source code(zip)
  • v6.1.1(Apr 26, 2022)

    Added/Changed/Fixed RAD Lab Launcher functionalities:

    • Listing Existing Deployments when Updating/Deleting any deployment.
    • Fix for empty line in the file of --varfile used to cause error of "index out of range" and fail the execution of python3 radlab.py --module.... --varfile ./xxxxx.
    Source code(tar.gz)
    Source code(zip)
  • v6.0.0(Apr 19, 2022)

  • v5.1.1(Mar 18, 2022)

    • Added/Changed/Fixed RAD Lab modules functionalities:
    1. Added option to create container image Vertex AI notebooks in Data Science Module
    2. Added capability for users to upload their own Jupyter notebooks to Vertex AI notebooks on Data Science Module deployment.
    • Added/Changed/Fixed RAD Lab Launcher functionalities:
    1. Enhancement for Launcher to accept Command Line arguments.
    2. Added a CLI argument / flag to disable RADLab Permission pre-checks.
    3. Fixed links in the Error Messages.
    4. Fixing a bug to also check for Owner role (i.e. roles/owner) which already contains all required permission for pre-checks for RAD Lab management project
    Source code(tar.gz)
    Source code(zip)
  • v5.0.0(Feb 24, 2022)

  • v4.2.1(Feb 18, 2022)

    • Added/Changed/Fixed RAD Lab modules functionalities:
    1. Resolved conflicting iam binding of notebooks.admin for data_science module.
    • Added/Changed/Fixed RAD Lab Launcher functionalities:
    1. Skipped installation of Terraform binaries on Cloud Shell as TF is automatically authenticated and is integrated with Cloud Shell.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.0(Feb 15, 2022)

    • Added/Changed/Fixed RAD Lab modules functionalities:
    1. Standardized Project ID prefix for all RAD Lab Module
    • Added/Changed/Fixed RAD Lab Launcher functionalities:
    1. Pre-Check IAM Permissions before deploying RAD Lab Modules via RAD Lab Launcher.
    2. RAD Lab Launcher to prompt for either proceed with the existing GOOGLE_APPLICATION_CREDENTIALS or go through the login again.
    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(Jan 14, 2022)

    • Released New RAD Lab Module : Genomics-Cromwell Module
    • Added/Changed/Fixed RAD Lab modules functionalities:
    1. Updated AI Notebook Image to include sample notebook for Data Science RAD Lab Module.
    • Added/Changed/Fixed RAD Lab Launcher functionalities:
    1. Spinning up radlab modules via RAD Lab Launcher without an Organization.
    2. Launcher to prompt for Project ID for RAD Lab management.
    • Added/Changed/Fixed Repository functionalities:
    1. Set up permissions in the GitHub Action workflows.
    2. Fixed GitHub Action Workflow - Build Module README.
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Dec 6, 2021)

  • v2.1.0(Nov 23, 2021)

    • Added/Changed/Fixed RAD Lab Launcher functionalities:
    1. Added optional Input file parameter to set variables & create terraform.tfvars file
    2. Changed how module selection works & is now compatible with any other module in future without any edits on the radlab.py
    Source code(tar.gz)
    Source code(zip)
  • v2.0.0(Nov 16, 2021)

    • Released New RAD Lab Module : Application Modernization Module (w/ Elasticsearch)
    • Added/Changed/Fixed RAD Lab modules functionalities:
    1. Deploy RAD Lab modules in existing GCP projects / networks / subnets
    2. Added timer to wait for Org Policy to get rolled out.
    • Added/Changed/Fixed RAD Lab Launcher functionalities:
    1. List existing RAD Lab deployments for specific modules
    2. Prompt for manually input Org ID & Billing A/C ID
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Nov 2, 2021)

    • Added/Changed/Fixed Repository functionalities:
    1. Added supporting tools , documentation & scripts to support RAD Lab modules.
    2. Added Issue Templates, Workflows, Codeowners & Changelog to support RAD Lab repo.
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Oct 7, 2021)

Owner
Google Cloud Platform
Google Cloud Platform
The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

Andrés Romero 37 Nov 27, 2022
GRaNDPapA: Generator of Rad Names from Decent Paper Acronyms

GRaNDPapA: Generator of Rad Names from Decent Paper Acronyms Trying to publish a new machine learning model and can't write a decent title for your pa

null 264 Nov 8, 2022
Image Captioning on google cloud platform based on iot

Image-Captioning-on-google-cloud-platform-based-on-iot - Image Captioning on google cloud platform based on iot

Shweta_kumawat 1 Jan 20, 2022
piSTAR Lab is a modular platform built to make AI experimentation accessible and fun. (pistar.ai)

piSTAR Lab WARNING: This is an early release. Overview piSTAR Lab is a modular deep reinforcement learning platform built to make AI experimentation a

piSTAR Lab 0 Aug 1, 2022
A trusty face recognition research platform developed by Tencent Youtu Lab

Introduction TFace: A trusty face recognition research platform developed by Tencent Youtu Lab. It provides a high-performance distributed training fr

Tencent 956 Jan 1, 2023
ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS.

ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS. It currently supports four examples for you to quickly experience the power of ONNX Runtime Web.

Microsoft 58 Dec 18, 2022
Torch-mutable-modules - Use in-place and assignment operations on PyTorch module parameters with support for autograd

Torch Mutable Modules Use in-place and assignment operations on PyTorch module p

Kento Nishi 7 Jun 6, 2022
Source code for our paper "Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash"

Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash Abstract: Apple recently revealed its deep perceptual hashing system NeuralHash to

ml-research@TUDarmstadt 11 Dec 3, 2022
Interactive Terraform visualization. State and configuration explorer.

Rover - Terraform Visualizer Rover is a Terraform visualizer. In order to do this, Rover: generates a plan file and parses the configuration in the ro

Tu Nguyen 2.3k Jan 7, 2023
PyStan, a Python interface to Stan, a platform for statistical modeling. Documentation: https://pystan.readthedocs.io

PyStan NOTE: This documentation describes a BETA release of PyStan 3. PyStan is a Python interface to Stan, a package for Bayesian inference. Stan® is

Stan 229 Dec 29, 2022
This is a Machine Learning Based Hand Detector Project, It Uses Machine Learning Models and Modules Like Mediapipe, Developed By Google!

Machine Learning Hand Detector This is a Machine Learning Based Hand Detector Project, It Uses Machine Learning Models and Modules Like Mediapipe, Dev

Popstar Idhant 3 Feb 25, 2022
This solves the autonomous driving issue which is supported by deep learning technology. Given a video, it splits into images and predicts the angle of turning for each frame.

Self Driving Car An autonomous car (also known as a driverless car, self-driving car, and robotic car) is a vehicle that is capable of sensing its env

Sagor Saha 4 Sep 4, 2021
In the case of your data having only 1 channel while want to use timm models

timm_custom Description In the case of your data having only 1 channel while want to use timm models (with or without pretrained weights), run the fol

null 2 Nov 26, 2021
Turning SymPy expressions into PyTorch modules.

sympytorch A micro-library as a convenience for turning SymPy expressions into PyTorch Modules. All SymPy floats become trainable parameters. All SymP

Patrick Kidger 89 Dec 13, 2022
Covid19-Forecasting - An interactive website that tracks, models and predicts COVID-19 Cases

Covid-Tracker This is an interactive website that tracks, models and predicts CO

Adam Lahmadi 1 Feb 1, 2022
Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)

Representation Robustness Evaluations Our implementation is based on code from MadryLab's robustness package and Devon Hjelm's Deep InfoMax. For all t

Sicheng 19 Dec 7, 2022