Materials for upcoming beginner-friendly PyTorch course (work in progress).

Daniel Bourke

Last update: Dec 29, 2022

Overview

Learn PyTorch for Deep Learning (work in progress)

I'd like to learn PyTorch. So I'm going to use this repo to:

Add what I've learned.
Teach others in a beginner-friendly way.

Stay tuned to here for updates, course materials are being actively worked on.

Launch early-mid 2022.

Course materials/outline

Note: This is rough and subject to change.
Course focus: code, code, code, experiment, experiment, experiment
Teaching style: https://sive.rs/kimo

Section	What does it cover?	Exercises & Extra-curriculum	Slides
00 - PyTorch Fundamentals	Many fundamental PyTorch operations used for deep learning and neural networks.	Go to exercises & extra-curriculum	Go to slides
01 - PyTorch Workflow	Provides an outline for approaching deep learning problems and building neural networks with PyTorch.	Go to exercises & extra-curriculum	Go to slides
02 - PyTorch Neural Network Classification	Uses the PyTorch workflow from 01 to go through a neural network classification problem.	Go to exercises & extra-curriculum	Go to slides
03 - PyTorch Computer Vision	Let's see how PyTorch can be used for computer vision problems using the same workflow from 01 & 02.	Go to exercises & extra-curriculum	Go to slides
04 - PyTorch Custom Datasets	How do you load a custom dataset into PyTorch? Also we'll be laying the foundations in this notebook for our modular code (covered in 05).	Go to exercises & extra-curriculum	Go to slides
05 - PyTorch Going Modular	PyTorch is designed to be modular, let's turn what we've created into a series of Python scripts (this is how you'll often find PyTorch code in the wild).	Go to exercises & extra-curriculum	Go to slides
Coming soon: 06 - PyTorch Transfer Learning	Let's take a well performing pre-trained model and adjust it to one of our own problems.	Go to exercises & extra-curriculum	Go to slides
Coming soon: 07 - Milestone Project 1: PyTorch Experiment Tracking	We've built a bunch of models... wouldn't it be good to track how they're all going?	Go to exercises & extra-curriculum	Go to slides
Coming soon: 08 - Milestone Project 2: PyTorch Paper Replicating	PyTorch is the most popular deep learning framework for machine learning research, let's see why by replicating a machine learning paper.	Go to exercises & extra-curriculum	Go to slides
Coming soon: 09 - Milestone Project 3: Model deployment	So you've built a working PyTorch model... how do you get it in the hands of others? Hint: deploy it to the internet.	Go to exercises & extra-curriculum	Go to slides

Old outline version (will update this if necessary)

PyTorch fundamentals - ML is all about representing data as numbers (tensors) and manipulating those tensors so this module will cover PyTorch tensors.
PyTorch workflow - You'll use different techniques for different problem types but the workflow remains much the same:

data -> build model -> fit model to data (training) -> evaluate model and make predictions (inference) -> save & load model

Module 1 will showcase an end-to-end PyTorch workflow that can be leveraged for other problems.

PyTorch classification - Let's take the workflow we learned in module 1 and apply it to a common machine learning problem type: classification (deciding whether something is one thing or another).
PyTorch computer vision - We'll get even more specific now and see how PyTorch can be used for computer vision problems though still using the same workflow from 1 & 2. We'll also start functionizing the code we've been writing, for example: def train(model, data, optimizer, loss_fn): ...
PyTorch custom datasets - How do you load a custom dataset into PyTorch? Also we'll be laying the foundations in this notebook for our modular code (covered in 05).
Going modular - PyTorch is designed to be modular, let's turn what we've created into a series of Python scripts (this is how you'll often find PyTorch code in the wild). For example:

code/
    data_setup.py <- sets up data
    model_builder.py <- builds the model ready to be used
    engine.py <- training/eval functions for the model
    train.py <- trains and saves the model

PyTorch transfer learning - Let's improve upon the models we've built ourselves using transfer learning.
PyTorch experiment tracking - We've built a bunch of models... wouldn't it be good to track how they're all going?
PyTorch paper replicating - Let's see why PyTorch is the most popular deep learning framework for machine learning research by replicating a machine learning research paper with it.
PyTorch model deployment - How do you get your PyTorch models in the hands of others?

Each notebook will teach a maximum of 3 big ideas.

Status

Working on: shooting videos for 05
Total video count: 162
Done skeleton code for: 00, 01, 02, 03, 04, 05, 06, 07
Done annotations (text) for: 00, 01, 02, 03, 04, 05
Done images for: 00, 01, 02, 03, 04, 05
Done keynotes for: 00, 01, 02, 03, 04, 05
Done exercises and solutions for: 00, 01, 02, 03, 04, 05
Done vidoes for: 00, 01, 02, 03, 04

TODO

See the project page for specifics - https://github.com/users/mrdbourke/projects/1

High-level overview of things to do:

How to use this repo (e.g. env setup, GPU/no GPU) - all notebooks should run fine in Colab and locally if needed.
Finish skeleton code for notebooks 00 - 07 ✅
Write annotations for 00 - 07
Make images for 00 - 07
Make slides for 00 - 07
Record videos for 00 - 07

Log

Almost daily updates of what's happening.

12 May 2022 - added exercises and solutions for 05
11 May 2022 - clean up part 1 and part 2 notebooks for 05, make slides for 05, start on exercises and solutions for 05
10 May 2022 - huuuuge updates to the 05 section, see the website, it looks pretty: https://www.learnpytorch.io/05_pytorch_going_modular/
09 May 2022 - add a bunch of materials for 05, cleanup docs
08 May 2022 - add a bunch of materials for 05
06 May 2022 - continue making materials for 05
05 May 2022 - update section 05 with headings/outline
28 Apr 2022 - recorded 13 videos for 04, finished videos for 04, now to make materials for 05
27 Apr 2022 - recorded 3 videos for 04
26 Apr 2022 - recorded 10 videos for 04
25 Apr 2022 - recorded 11 videos for 04
24 Apr 2022 - prepared slides for 04
23 Apr 2022 - recorded 6 videos for 03, finished videos for 03, now to 04
22 Apr 2022 - recorded 5 videos for 03
21 Apr 2022 - recorded 9 videos for 03
20 Apr 2022 - recorded 3 videos for 03
19 Apr 2022 - recorded 11 videos for 03
18 Apr 2022 - finish exercises/solutions for 04, added live-coding walkthrough of 04 exercises/solutions on YouTube: https://youtu.be/vsFMF9wqWx0
16 Apr 2022 - finish exercises/solutions for 03, added live-coding walkthrough of 03 exercises/solutions on YouTube: https://youtu.be/_PibmqpEyhA
14 Apr 2022 - add final images/annotations for 04, begin on exercises/solutions for 03 & 04
13 Apr 2022 - add more images/annotations for 04
3 Apr 2022 - add more annotations for 04
2 Apr 2022 - add more annotations for 04
1 Apr 2022 - add more annotations for 04
31 Mar 2022 - add more annotations for 04
29 Mar 2022 - add more annotations for 04
27 Mar 2022 - starting to add annotations for 04
26 Mar 2022 - making dataset for 04
25 Mar 2022 - make slides for 03
24 Mar 2022 - fix error for 03 not working in docs (finally)
23 Mar 2022 - add more images for 03
22 Mar 2022 - add images for 03
20 Mar 2022 - add more annotations for 03
18 Mar 2022 - add more annotations for 03
17 Mar 2022 - add more annotations for 03
16 Mar 2022 - add more annotations for 03
15 Mar 2022 - add more annotations for 03
14 Mar 2022 - start adding annotations for notebook 03, see the work in progress here: https://www.learnpytorch.io/03_pytorch_computer_vision/
12 Mar 2022 - recorded 12 videos for 02, finished section 02, now onto making materials for 03, 04, 05
11 Mar 2022 - recorded 9 videos for 02
10 Mar 2022 - recorded 10 videos for 02
9 Mar 2022 - cleaning up slides/code for 02, getting ready for recording
8 Mar 2022 - recorded 9 videos for section 01, finished section 01, now onto 02
7 Mar 2022 - recorded 4 videos for section 01
6 Mar 2022 - recorded 4 videos for section 01
4 Mar 2022 - recorded 10 videos for section 01
20 Feb 2022 - recorded 8 videos for section 00, finished section, now onto 01
18 Feb 2022 - recorded 13 videos for section 00
17 Feb 2022 - recorded 11 videos for section 00
16 Feb 2022 - added setup guide
12 Feb 2022 - tidy up README with table of course materials, finish images and slides for 01
10 Feb 2022 - finished slides and images for 00, notebook is ready for publishing: https://www.learnpytorch.io/00_pytorch_fundamentals/
01-07 Feb 2022 - add annotations for 02, finished, still need images, going to work on exercises/solutions today
31 Jan 2022 - start adding annotations for 02
28 Jan 2022 - add exercies and solutions for 01
26 Jan 2022 - lots more annotations to 01, should be finished tomorrow, will do exercises + solutions then too
24 Jan 2022 - add a bunch of annotations to 01
21 Jan 2022 - start adding annotations for 01
20 Jan 2022 - finish annotations for 00 (still need to add images), add exercises and solutions for 00
19 Jan 2022 - add more annotations for 00
18 Jan 2022 - add more annotations for 00
17 Jan 2022 - back from holidays, adding more annotations to 00
10 Dec 2021 - start adding annoations for 00
9 Dec 2021 - Created a website for the course (learnpytorch.io) you'll see updates posted there as development continues
8 Dec 2021 - Clean up notebook 07, starting to go back through code and add annotations
26 Nov 2021 - Finish skeleton code for 07, added four different experiments, need to clean up and make more straightforward
25 Nov 2021 - clean code for 06, add skeleton code for 07 (experiment tracking)
24 Nov 2021 - Update 04, 05, 06 notebooks for easier digestion and learning, each section should cover a max of 3 big ideas, 05 is now dedicated to turning notebook code into modular code
22 Nov 2021 - Update 04 train and test functions to make more straightforward
19 Nov 2021 - Added 05 (transfer learning) notebook, update custom data loading code in 04
18 Nov 2021 - Updated vision code for 03 and added custom dataset loading code in 04
12 Nov 2021 - Added a bunch of skeleton code to notebook 04 for custom dataset loading, next is modelling with custom data
10 Nov 2021 - researching best practice for custom datasets for 04
9 Nov 2021 - Update 03 skeleton code to finish off building CNN model, onto 04 for loading custom datasets
4 Nov 2021 - Add GPU code to 03 + train/test loops + helper_functions.py
3 Nov 2021 - Add basic start for 03, going to finish by end of week
29 Oct 2021 - Tidied up skeleton code for 02, still a few more things to clean/tidy, created 03
28 Oct 2021 - Finished skeleton code for 02, going to clean/tidy tomorrow, 03 next week
27 Oct 2021 - add a bunch of code for 02, going to finish tomorrow/by end of week
26 Oct 2021 - update 00, 01, 02 with outline/code, skeleton code for 00 & 01 done, 02 next
23, 24 Oct 2021 - update 00 and 01 notebooks with more outline/code
20 Oct 2021 - add v0 outlines for 01 and 02, add rough outline of course to README, this course will focus on less but better
19 Oct 2021 - Start repo 🔥 , add fundamentals notebook draft v0

Comments

Convolutional - Section 7 model issue

Hey, weird stuff's happening. In the section 7.2 when we're setting up CNN and have to figure out what shapes of the input features should be. Below you can take a look for a print screen from collab. I first encountered this mismatch while writing my own code. But next I tried Yours code and the same problem - it looks like somehow the inputs aren't flatten so instead of 1x490 we have 10x49.

Anyone here with similar problems?

opened by SquareGraph 5
One of the notebooks is invalid

Path: pytorch-deep-learning/extras/solutions/03_pytorch_computer_vision_exercise_solutions.ipynb

Invalid Notebook The Notebook Does Not Appear to Be Valid JSON

This is the only invalid notebook in this repository.

opened by Cheerstoast 4
Notebook 03. Mistake with confusion matrix (10 section)?

Hey, In the screenshot below we apply softmax function with dim=0, but y_logit has the shape [32, 10]. If we apply accuracy metric to evaluate score between y_pred_tensor and test_data.targets we get the lower results than it were before.

Should we use dim=1 in softmax instead? In this case we get the same results (in terms of accuracy) Maybe it depends on PyTorch version.

Thank you!

opened by AnatolyAdamovich 3

PyTorch syntax error (torch undefined)

After writing the following code:-

## Create a random tensor of size (3, 4)
random_tensor = torch.rand(1, 10, 10)
random_tensor

The response I get is this:-

_**NameError Traceback (most recent call last)

in
1 ## Create a random tensor of size (3, 4)
----> 2 random_tensor = torch.rand(1, 10, 10)
3 random_tensor_

NameError: name 'torch' is not defined**_

Please help me on how to solve this issue

opened by potaytoo 2

Part4_ make predictions and get model 0 results

I am getting very less accuracy score when I wrote the function for model evaluation. All other parts of the code are fine and even the accuracy metrics while I run the epochs in part 3 are also close to your ones.

I went through the code several times but I could not figure out the issue.

You can access my notebook from this link.

Can you please guide me through what is wrong ?

opened by ulfat191 2
Computer Vision Notebook 3

I tried the CNN model with Adam optimizer with the same code , however the validation accuracy didn't increase and stayed a 10 % in all the epochs , I am not sure why this happens

opened by Hrithik2212 2
How to save time to compare between model_results in dictionary?

In comparing part of three models in 03. Pytorch Computer Vision, How to show in table about time difference of model_1_results, model_2_results and model_3_results for CPU or GPU?

opened by JasOlean 2
Create template "How to ask a question for this course" in GitHub discussions
Create a template to include things like:

Video timestamp

Resource watching the video (e.g. YouTube, ZTM etc)

"What I've tried so far..."

"Where the code is running..." (e.g. Colab)
opened by mrdbourke 2
Review Notebook 03
The "Input and output shapes" image shows NHWC. Might be worthwhile to change to NCHW.

In addition to being more computationally efficient, mini-batching also gives the model more opportunities to improve. Batch Gradient Descent would calculate the gradient once per epoch and that's it. Mini-batch SGD gives you dataset_size // batch_size weight updates.

After reading this, I'm not 100% sure that torch.manual_seed(42) will give the same results for everybody.

I understood that model 0 will be ran on both CPU and GPU, but it is only run on CPU.

Image above 7.2 also uses NHWC instead of NCHW

While it is possible to type hint the accuracy_fn parameter, I decided against it, as this is a "noisy" type hint

I'm running on PyTorch 1.11 and apparently, running conv_layer(test_image) actually works without a batch dimension and returns [10, 62, 62] shape result

Added type hints where missing

Changed the model vs accuracy bar chart to be horizontal, that way the model names are more readable 🙂
opened by beneyal 2
Notebook 03 is breaking the documentation building process

Not sure what's happening with Notebook 03... seems to be breaking the documentation building process

Might have to manually turn it into a new notebook.

As in copy all of the contents of different cells to new cells.

Not ideal... but I'll try this tomorrow.

See the action breakage here: https://github.com/mrdbourke/pytorch-deep-learning/runs/5621310937?check_suite_focus=true

opened by mrdbourke 2
01_pytorch_workflow_exercises_solutions file Updated

Solving the issue #208 . This also decreased the loss of the model and also aligned the assignment with the video as it was done in the video plz review it and merge it @mrdbourke

opened by dhruvkej9 1

PyTorch and numpy - when converting from tensor to numpy they are mapped to same memory location

for pytorch version:'1.11.0'

# converting from tensor to numpy
t = torch.ones(7)
np_array = t.numpy()
t , t.dtype , np_array , np_array.dtype

output:

(tensor([1., 1., 1., 1., 1., 1., 1.]),
 torch.float32,
 array([1., 1., 1., 1., 1., 1., 1.], dtype=float32),
 dtype('float32'))

After changing the tensor

# change the tensor see what happens to the numpy array
t += 10
t , t.dtype , np_array , np_array.dtype

output:

(tensor([11., 11., 11., 11., 11., 11., 11.]),
 torch.float32,
 array([11., 11., 11., 11., 11., 11., 11.], dtype=float32),
 dtype('float32'))

opened by ashwinshetgaonkar 1

Notebook 03: Different shape outputs between PyTorch 1.10, 1.11, 1.12

Hi all,

With the pytorch version 1.11 (and 1.12), the trick that Daniel uses (hidden_units*7*7) doesn;'t work. It worked I believe because the output in 1.10 of Conv_layer_2 =[1,10,7,7]. Multiplying each unit 10*7*7 = 490 and delivers [1,490] and thus solving this by using hidden_units*7*7 works in 1.10.

In 1.11 and 1.12, the output of conv_layer_2 is however is [10, 7, 7], leading to 7*7 and a size of [10*49]. Hence, you cannot solve the input by doing hidden*7*7 (results in 490) but rather, simply 7*7.

thus the linear layer becomes:

nn.Linear(in_features=7*7, out_features=output_shape)

Using this the shapes match and it will work on a single image,

Yet when training you will need the hidden*7*7 setup as it wont work otherwise.

Originally posted by @aronvandepol in https://github.com/mrdbourke/pytorch-deep-learning/discussions/68

opened by mrdbourke 2

Materials for upcoming beginner-friendly PyTorch course (work in progress).

Related tags

Overview

Learn PyTorch for Deep Learning (work in progress)

Course materials/outline

Old outline version (will update this if necessary)

Status

TODO

Log

Comments

Owner

Daniel Bourke

Code for the upcoming CVPR 2021 paper

Official repository of my book: "Deep Learning with PyTorch Step-by-Step: A Beginner's Guide"

This repository builds a basic vision transformer from scratch so that one beginner can understand the theory of vision transformer.

Space-invaders - Simple Game created using Python & PyGame, as my Beginner Python Project

Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

Collection of in-progress libraries for entity neural networks.

ShuttleNet: Position-aware Fusion of Rally Progress and Player Styles for Stroke Forecasting in Badminton (AAAI 2022)

A tiny, friendly, strong baseline code for Person-reID (based on pytorch).

Unofficial PyTorch implementation of MobileViT based on paper "MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer".

This repository contains all the code and materials distributed in the 2021 Q-Programming Summer of Qode.

Pre-trained model, code, and materials from the paper "Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation" (MICCAI 2019).

This repo contains research materials released by members of the Google Brain team in Tokyo.

Useful materials and tutorials for 110-1 NTU DBME5028 (Application of Deep Learning in Medical Imaging)

The materials used in the SaxonJS tutorial presented at Declarative Amsterdam, 2021

An SE(3)-invariant autoencoder for generating the periodic structure of materials

Workshop Materials Delivered on 28/02/2022

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".

PyTorch implementation of CloudWalk's recent work DenseBody

Scalable, event-driven, deep-learning-friendly backtesting library