Implementation of OpenAI paper with Simple Noise Scale on Fastai V2

Related tags

Deep Learning fastai-batch-size-finder
Overview

README

Implementation of OpenAI paper "An Empirical Model of Large-Batch Training" for Fastai V2.

The code is based on the batch size finder implementation for Fastai V1 by DanyWind (repo V1 / blog / discussion).

This implementation differs on:

  1. It implements exactly the original article and not an aproximation (by default).
  2. Fixes a couple of bugs in noise and scale values. However, they didn't affect on Simple Noise Scale value.

However, you could use the DanyWind aproximation by settting simulate_multi_gpus to False. DanyWind aproximation is faster but numerically more inestable and finds a Simple Noise Scale smaller than the original Simple Noise Scale.

It's tested with fastai 2.1. It should work with fastai>=2.0

TODO:

You might also like...
An Agnostic Computer Vision Framework - Pluggable to any Training Library: Fastai, Pytorch-Lightning with more to come
An Agnostic Computer Vision Framework - Pluggable to any Training Library: Fastai, Pytorch-Lightning with more to come

IceVision is the first agnostic computer vision framework to offer a curated collection with hundreds of high-quality pre-trained models from torchvision, MMLabs, and soon Pytorch Image Models. It orchestrates the end-to-end deep learning workflow allowing to train networks with easy-to-use robust high-performance libraries such as Pytorch-Lightning and Fastai

airctic 789 Dec 29, 2022
Code of our paper "Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning"

CCOP Code of our paper Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning Requirement Install OpenSelfSup Install Detectron2

Chenhongyi Yang 21 Dec 13, 2022
The trained model and denoising example for paper : Cardiopulmonary Auscultation Enhancement with a Two-Stage Noise Cancellation Approach

The trained model and denoising example for paper : Cardiopulmonary Auscultation Enhancement with a Two-Stage Noise Cancellation Approach

ycj_project 1 Jan 18, 2022
PyTorch implementation for Partially View-aligned Representation Learning with Noise-robust Contrastive Loss (CVPR 2021)
PyTorch implementation for Partially View-aligned Representation Learning with Noise-robust Contrastive Loss (CVPR 2021)

2021-CVPR-MvCLN This repo contains the code and data of the following paper accepted by CVPR 2021 Partially View-aligned Representation Learning with

XLearning Group 33 Nov 1, 2022
PyTorch implementation of "A Two-Stage End-to-End System for Speech-in-Noise Hearing Aid Processing"

Implementation of the Sheffield entry for the first Clarity enhancement challenge (CEC1) This repository contains the PyTorch implementation of "A Two

null 10 Aug 19, 2022
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

Phil Wang 4.4k Jan 3, 2023
A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.
A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.

Ryan Murdock has done it again, combining OpenAI's CLIP and the generator from a BigGAN! This repository wraps up his work so it is easily accessible to anyone who owns a GPU.

Phil Wang 2.3k Jan 9, 2023
Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

DALL-E in Pytorch Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch. It will also contain CLIP for ranking the ge

Phil Wang 5k Jan 4, 2023
Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI
Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Hourglass Transformer - Pytorch (wip) Implementation of Hourglass Transformer, in Pytorch. It will also contain some of my own ideas about how to make

Phil Wang 61 Dec 25, 2022
Comments
  • Issue with fast.ai==2.2.7

    Issue with fast.ai==2.2.7

    Hi.

    Starting from fasta.i==2.2.7 bs_finder stopped working. Could you advise how to fix this issue, please?

    Message: KeyError: "None of ['n_iter'] are in the columns"

Detailed traceback:
  File "D:\a\fastai\fastai\check\fastai.Rcheck\fastai\python\fastaibuilt\bs_finder.py", line 161, in bs_find
    with self.no_logging(): self.fit(n_epoch, lr, cbs=cb)
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastai\learner.py", line 211, in fit
    self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastai\learner.py", line 162, in _with_events
    self(f'after_{event_type}');  final()
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastai\learner.py", line 141, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastcore\foundation.py", line 154, in map
    def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastcore\basics.py", line 666, in map_ex
    return list(res)
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastcore\basics.py", line 651, in __call__
    return self.func(*fargs, **kwargs)
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastai\learner.py", line 145, in _call_one
    for cb in self.cbs.sorted('order'): cb(event_name)
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\fastai\callback\core.py", line 44, in __call__
    if self.run and _run: res = getattr(self, event_name, noop)()
  File "D:\a\fastai\fastai\check\fastai.Rcheck\fastai\python\fastaibuilt\bs_finder.py", line 126, in after_fit
    df.set_index('n_iter', inplace=True)
  File "C:\Users\RUNNER~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\pandas\core\frame.py", line 4727, in set_index
    raise KeyError(f"None of {missing} are in the columns")
    opened by turgut090 4
  • AttributeError: 'str' object has no attribute '__stored_args__'

    AttributeError: 'str' object has no attribute '__stored_args__'

    Hi. I get error with fastai 2.1 when I run the following.

    model.bs_find(lr = 1e-3)

     AttributeError: 'str' object has no attribute '__stored_args__'

    Is there a way to fix this?

    opened by turgut090 3
  • fastai 2.1 breaking changes

    fastai 2.1 breaking changes

    Hi. I ran the notebook again. It worked after your suggestion. Thanks! Could you link this PR to that issue, so it will be closed automatically as well, please?

    hal-314 edit: fixes #1

    opened by turgut090 0
Owner
null
GitHub
Official implementation of "Open-set Label Noise Can Improve Robustness Against Inherent Label Noise" (NeurIPS 2021)

Open-set Label Noise Can Improve Robustness Against Inherent Label Noise NeurIPS 2021: This repository is the official implementation of ODNL. Require

Hongxin Wei 12 Dec 7, 2022
🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI

PyTorch implementation of OpenAI's Finetuned Transformer Language Model This is a PyTorch implementation of the TensorFlow code provided with OpenAI's

Hugging Face 1.4k Jan 5, 2023
A general-purpose, flexible, and easy-to-use simulator alongside an OpenAI Gym trading environment for MetaTrader 5 trading platform (Approved by OpenAI Gym)

gym-mtsim: OpenAI Gym - MetaTrader 5 Simulator MtSim is a simulator for the MetaTrader 5 trading platform alongside an OpenAI Gym environment for rein

Mohammad Amin Haghpanah 184 Dec 31, 2022
Official implementation of our paper "Learning to Bootstrap for Combating Label Noise"

Learning to Bootstrap for Combating Label Noise This repo is the official implementation of our paper "Learning to Bootstrap for Combating Label Noise

null 21 Apr 9, 2022
Simple implementation of OpenAI CLIP model in PyTorch.

It was in January of 2021 that OpenAI announced two new models: DALL-E and CLIP, both multi-modality models connecting texts and images in some way. In this article we are going to implement CLIP model from scratch in PyTorch. OpenAI has open-sourced some of the code relating to CLIP model but I found it intimidating and it was far from something short and simple. I also came across a good tutorial inspired by CLIP model on Keras code examples and I translated some parts of it into PyTorch to build this tutorial totally with our beloved PyTorch!

Moein Shariatnia 226 Jan 5, 2023
Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

GPT2-Pytorch with Text-Generator Better Language Models and Their Implications Our model, called GPT-2 (a successor to GPT), was trained simply to pre

Tae-Hwan Jung 775 Jan 8, 2023
The fastai deep learning library

Welcome to fastai fastai simplifies training fast and accurate neural nets using modern best practices Important: This documentation covers fastai v2,

fast.ai 23.2k Jan 7, 2023
The fastai deep learning library

Welcome to fastai fastai simplifies training fast and accurate neural nets using modern best practices Important: This documentation covers fastai v2,

fast.ai 20.4k Feb 12, 2021
tsai is an open-source deep learning package built on top of Pytorch & fastai focused on state-of-the-art techniques for time series classification, regression and forecasting.

Time series Timeseries Deep Learning Pytorch fastai - State-of-the-art Deep Learning with Time Series and Sequences in Pytorch / fastai

timeseriesAI 2.8k Jan 8, 2023
fastgradio is a python library to quickly build and share gradio interfaces of your trained fastai models.

fastgradio is a python library to quickly build and share gradio interfaces of your trained fastai models.

Ali Abdalla 34 Jan 5, 2023