Official repository of my book: "Deep Learning with PyTorch Step-by-Step: A Beginner's Guide"

Overview

Deep Learning with PyTorch Step-by-Step

This is the official repository of my book "Deep Learning with PyTorch Step-by-Step". Here you will find one Jupyter notebook for every chapter in the book.

Each notebook contains all the code shown in its corresponding chapter, and you should be able to run its cells in sequence to get the same outputs as shown in the book. I strongly believe that being able to reproduce the results brings confidence to the reader.

There are three options for you to run the Jupyter notebooks:

Google Colab

You can easily load the notebooks directly from GitHub using Colab and run them using a GPU provided by Google. You need to be logged in a Google Account of your own.

You can go through the chapters already using the links below:

Part I - Fundamentals

Part II - Computer Vision

Part III - Sequences

Part IV - Natural Language Processing

Binder

You can also load the notebooks directly from GitHub using Binder, but the process is slightly different. It will create an environment on the cloud and allow you to access Jupyter's Home Page in your browser, listing all available notebooks, just like in your own computer.

If you make changes to the notebooks, make sure to download them, since Binder does not keep the changes once you close it.

You can start your environment on the cloud right now using the button below:

Binder

Local Installation

This option will give you more flexibility, but it will require more effort to set up. I encourage you to try setting up your own environment. It may seem daunting at first, but you can surely accomplish it following seven easy steps:

1 - Anaconda

If you don’t have Anaconda’s Individual Edition installed yet, that would be a good time to do it - it is a very handy way to start - since it contains most of the Python libraries a data scientist will ever need to develop and train models.

Please follow the installation instructions for your OS:

Make sure you choose Python 3.X version since Python 2 was discontinued in January 2020.

2 - Conda (Virtual) Environments

Virtual environments are a convenient way to isolate Python installations associated with different projects.

First, you need to choose a name for your environment :-) Let’s call ours pytorchbook (or anything else you find easier to remember). Then, you need to open a terminal (in Ubuntu) or Anaconda Prompt (in Windows or macOS) and type the following command:

conda create -n pytorchbook anaconda

The command above creates a conda environment named pytorchbook and includes all anaconda packages in it (time to get a coffee, it will take a while...). If you want to learn more about creating and using conda environments, please check Anaconda’s Managing Environments user guide.

Did it finish creating the environment? Good! It is time to activate it, meaning, making that Python installation the one to be used now. In the same terminal (or Anaconda Prompt), just type:

conda activate pytorchbook

Your prompt should look like this (if you’re using Linux)...

(pytorchbook)$

or like this (if you’re using Windows):

(pytorchbook)C:\>

Done! You are using a brand new conda environment now. You’ll need to activate it every time you open a new terminal or, if you’re a Windows or macOS user, you can open the corresponding Anaconda Prompt (it will show up as Anaconda Prompt (pytorchbook), in our case), which will have it activated from start.

IMPORTANT: From now on, I am assuming you’ll activate the pytorchbook environment every time you open a terminal / Anaconda Prompt. Further installation steps must be executed inside the environment.

3 - PyTorch

It is time to install the star of the show :-) We can go straight to the Start Locally section of its website and it will automatically select the options that best suit your local environment and it will show you the command to run.

Your choices should look like:

  • PyTorch Build: "Stable"
  • Your OS: your operating system
  • Package: "Conda"
  • Language: "Python"
  • CUDA: "None" if you don't have a GPU, or the latest version (e.g. "10.1"), if you have a GPU.

The installation command will be shown right below your choices, so you can copy it. If you have a Windows computer and no GPU, you'd have to run the following command in your Anaconda Prompt (pytorchbook):

(pytorchbook) C:\> conda install pytorch torchvision cpuonly -c pytorch

4 - TensorBoard

TensorBoard is a powerful tool and we can use it even if we are developing models in PyTorch. Luckily, you don’t need to install the whole TensorFlow to get it, you can easily install TensorBoard alone using conda. You just need to run this command in your terminal or Anaconda Prompt (again, after activating the environment):

(pytorchbook)C:\> conda install -c conda-forge tensorboard

5 - GraphViz and TorchViz (optional)

This step is optional, mostly because the installation of GraphViz can be challenging sometimes (especially on Windows). If, for any reason, you do not succeed in installing it correctly, or if you decide to skip this installation step, you will still be able to execute the code in this book (except for a couple of cells that generate images of a model’s structure in the Dynamic Computation Graph section of Chapter 1).

We need to install GraphViz to be able to use TorchViz, a neat package that allows us to visualize a model’s structure. Please check the installation instructions for your OS.

If you are using Windows, please use the installer at GraphViz's Windows Package. You also need to add GraphViz to the PATH (environment variable) in Windows. Most likely, you can find GraphViz executable file at C:\ProgramFiles(x86)\Graphviz2.38\bin. Once you found it, you need to set or change the PATH accordingly, adding GraphViz's location to it. For more details on how to do that, please refer to How to Add to Windows PATH Environment Variable.

For additional information, you can also check the How to Install Graphviz Software guide.

If you installed GraphViz successfully, you can install the torchviz package. This package is not part of Anaconda Distribution Repository and is only available at PyPI , the Python Package Index, so we need to pip install it.

Once again, open a terminal or Anaconda Prompt and run this command (just once more: after activating the environment):

(pytorchbook)C:\> pip install torchviz

6 - Git

It is way beyond the scope of this guide to introduce you to version control and its most popular tool: git. If you are familiar with it already, great, you can skip this section altogether!

Otherwise, I’d recommend you to learn more about it, it will definitely be useful for you later down the line. In the meantime, I will show you the bare minimum, so you can use git to clone this repository containing all code used in this book - so you have your own, local copy of it and can modify and experiment with it as you please.

First, you need to install it. So, head to its downloads page and follow instructions for your OS. Once installation is complete, please open a new terminal or Anaconda Prompt (it's OK to close the previous one). In the new terminal or Anaconda Prompt, you should be able to run git commands. To clone this repository, you only need to run:

(pytorchbook)C:\> git clone https://github.com/dvgodoy/PyTorchStepByStep.git

The command above will create a PyTorchStepByStep folder which contains a local copy of everything available on this GitHub’s repository.

7 - Jupyter

After cloning the repository, navigate to the PyTorchStepByStep and, once inside it, you only need to start Jupyter on your terminal or Anaconda Prompt:

(pytorchbook)C:\> jupyter notebook

This will open your browser up and you will see Jupyter's Home Page containing this repository's notebooks and code.

Congratulations! You are ready to go through the chapters' notebooks!

Issues
  • Went from

    Went from "train_step" in helper fn 1, to "step" in helper fn 2?

    I am missing the place where "step" is used as the returned function to "train_step"

    First: train_step = make_train_step(model, loss_fn, optimizer) loss = **train_step**(x_train_tensor, y_train_tensor)

    In model_training/v2.py we see mini_batch_loss = **train_step**(x_batch, y_batch)

    in helper function #2 we see mini_batch_loss = **step**(x_batch, y_batch)

    So far, I have been able to follow the thread of higher level functions. But I missed the above.

    Thank You Tom

    opened by minertom 4
  • No Issue. I purchased the book. Can you hint towards future chapters?

    No Issue. I purchased the book. Can you hint towards future chapters?

    HI, I just purchased the book as I am in the process of learning Pytorch. I was wondering if you could give a hint towards future chapters.

    Also, How would I be notified when future chapters are available?

    Thank You Tom

    opened by minertom 4
  • Link 35 is broken

    Link 35 is broken

    Another thing I found in the book, link 35 doesn't seem to work.

    Thanks! :)

    opened by Leon0402 3
  • Why do we have detach() in self.alphas = alphas.detach() in Attention class in Chapter 9.

    Why do we have detach() in self.alphas = alphas.detach() in Attention class in Chapter 9.

    I wonder why alphas is "detach()"ed and before saved to self.alphas in Attention class. I tried self.alphas = alphas, that is, without detach and trained the model. There is no difference in performance. So I believe the reason is in something else.

    Thank you for your great teaching in your great book!

    opened by eisthf 2
  • On the calculation of

    On the calculation of "w_grad".

    In chapter0 and chapter1 there are a couple of calculations for the gradient of b and the gradient of w.

    Given that yhat = b +w(x_train), where b is random and error is (yhat -y_train), that makes error = (b +w(x_train) - y_train).

    b_grad is given as 2(error.mean() ) which is 2(b + w(x_train) - y_train).mean() , so it seems to me, and I could be wrong, that the so called gradient of b also includes a healthy helping of w.

    w_grad is given as 2(x_train(error).mean() ) which expands out to 2((x_train)b + (x_train**2)w - (x_train)(y_train)).mean() )

    It is the (x_train)**2 term that triggered something in my mind. As well as the fact that the w_grad term also has a healthy helping of b.

    My intuition, and I could be wrong here, is that there is a partial derivative missing such that the gradient of b would be based upon differentiating b while holding w constant and similarly, the gradient of w would be done with holding b as a constant.

    Also, the (x_train)**2 term is confusing here.

    I would be deeply grateful for a clarification.

    Thank You Tom

    opened by minertom 2
  • A simple question about how you are using matplotlib?

    A simple question about how you are using matplotlib?

    Hi, and i realize that this may be trivial.

    In your "figures" you are using python functions that are defined in the "plots" directory.

    In the jupyter notebook you use figure1(x_train, y_train, x_val, y_val

    This calls the function `def figure1(x_train, y_train, x_val, y_val): fig, ax = plt.subplots(1, 2, figsize=(12, 6))

    ax[0].scatter(x_train, y_train)
    ax[0].set_xlabel('x')
    ax[0].set_ylabel('y')
    ax[0].set_ylim([0, 3.1])
    ax[0].set_title('Generated Data - Train')
    
    ax[1].scatter(x_val, y_val, c='r')
    ax[1].set_xlabel('x')
    ax[1].set_ylabel('y')
    ax[1].set_ylim([0, 3.1])
    ax[1].set_title('Generated Data - Validation')
    fig.tight_layout()
    
    return fig, ax
    

    ` I notice that there is no plt.show() statement that is needed, either in the function or in the jupyter notebook.

    However, if I am at the command line (not using the jupyter notebook) and I enter the python code line by line and make the "figure" call, I don't get a plot unless I use "plt.show()" as the next line after the figure call.

    Can you tell me why plt.show() is not necessary for the jupyter notebook but it is necessary when calling the function from the command line?

    Thank You Tom

    opened by minertom 2
  • PyTorchStepByStep is under advertised.

    PyTorchStepByStep is under advertised.

    Danial, I wanted to let you know that I am really finding your Tutorial Series very helpful but you could be selling it a bit better.

    1. I am finding the tutorial a fantastic help in terms of learning numpy. Yes, the CS231 numpy tutorial give a good starting point but there is very little of practical use compared to what is in PyTorchStepByStep.

    2. Your tutorial provides a good learning point for matplotlib. I was forced to dig through the functions which generated the figures. I got everything working inside and outside of a python notebook.

    I just wanted to let you know how much I appreciate this tutorial.

    Tom

    opened by minertom 2
  • Typo Figure 0.6

    Typo Figure 0.6

    Hi there,

    not sure if it's the right place, but it's the easiest one for me :)

    In Figure 0.6 in your book the diagrams has "w" on is x-axis instead of "b".

    Great book, hopefully there will be more content soon :)

    opened by Leon0402 1
  • Everything works locally, except figure graphics

    Everything works locally, except figure graphics

    Hi, My usual practice when learning from python notebooks is to get everything that would work in the notebook to work locally.

    What program is used to generate the figures? Clearly, something not local in the notebook. Perhaps something called from google colab?

    I have installed everything properly and the data output is correct. However, I can't get the figures to display, locally. I did install matplotlib but I don't know how to get the same graphics to generate as are generated via the python notebook .

    figure1(x_train, y_train, x_val, y_val) gives me

    (<Figure size 864x432 with 2 Axes>, (<matplotlib.axes._subplots.Axes3DSubplot at 0x7fc942d92fd0>, <matplotlib.axes._subplots.AxesSubplot at 0x7fc9430497d0>))

    Just like it does in the jupyter notebook but with no picture. I know that I have to get some kind of graphical engine going as is going in the notebook but I don't know which one to start.

    I realize that this is a pretty minor issue.

    Thank You Tom

    opened by minertom 0
  • Ch4

    Ch4

    opened by dvgodoy 0
  • Chapter 01 - negative sign for gradients

    Chapter 01 - negative sign for gradients

    Prior to "Linear Regression in Numpy" section you do not add a negative sign in front of calculated gradients, while you do so later. I believe later is correct as gradients need to point towards the minima. Is that right?

    opened by nisargvp 7
  • How does nn.conv2D implement the feature map table of LeNet-5?

    How does nn.conv2D implement the feature map table of LeNet-5?

    I have been going through the pytorch documentation of conv2D. Reading the docs from pytorch I see

    torch.nn.Conv2d(in_channels: int, out_channels: int, kernel_size: Union[T, Tuple[T, T]], stride: Union[T, Tuple[T, T]] = 1, padding: Union[T, Tuple[T, T]] = 0, dilation: Union[T, Tuple[T, T]] = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros')

    From the LeNet paper of November 1998 I see that the third convolution layer is implemented with 6 input layers and 16 output layers. The 16 output layers are made from a combination of the 6 input layers according to a table, also in the paper:

    LenetFeatureMap

    From the chapter 5 tutorial, C3 is implemented with

    lenet.add_module('C3', nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5))

    What I do not see is how pytorch implements this feature map table. It seems a little like a "loaves and fishes" issue :-).

    P.S. If I might kill two birds with one stone, can i ask which plots are being implemented with IPython? I have observed that usually the plots that you generate are implemented in the plots directories from each chapter, using matplotlib. I was looking for HTML plots that were generated with IPython, because you call the IPython.core.display module and import display, HTML. But, I could not seem to spot how it was used.

    Thank You Tom

    opened by minertom 10
  • Missing input in helpers.py for function

    Missing input in helpers.py for function "make_balanced_sampler" and WeightedRandomSampler?

    I just downloaded the latest full zip file.

    While running the jupyter notebook for Chapter05, I noticed a couple of errors, relating to

    ` in 22 23 # Builds a weighted random sampler to handle imbalanced classes ---> 24 sampler = make_balanced_sampler(y_train_tensor) 25 26 # Uses sampler in the training set to get a balanced data loader

    ~/projects/pytorchstepbystep_2/PyTorchStepByStep-master/helpers.py in make_balanced_sampler(y) 79 num_samples=len(sample_weights), 80 generator=generator, ---> 81 replacement=True 82 ) 83 return sampler

    TypeError: init() got an unexpected keyword argument 'generator' `

    I think that the issue stems from the "WeightedRandomSampler" function, which might be missing an input. From the pytorch documentation CLASStorch.utils.data.WeightedRandomSampler(weights: Sequence[float], num_samples: int, replacement: bool = True, generator=None)

    That would be 4 inputs. "replacement" and "generator" are satisfied but either "weights" or "num_samples" seems to be missing from the call to WeightedRandomSampler from make_balanced_sampler. My guess would be that "num_samples" is the missing input.

    Is that the case?

    Thank You Tom

    opened by minertom 5
  • How to extract/save weights after training?

    How to extract/save weights after training?

    OK, here I am displaying my utter ignorance again. I did find a post on towards data science entitled "everything you need to know about saving weights in pytorch".

    https://towardsdatascience.com/everything-you-need-to-know-about-saving-weights-in-pytorch-572651f3f8de

    Now I am stuck. Having saved the weights in the example project, I am aware that the file is not in a human readable format.

    So my question now becomes is there a way to take this file of weights which is in pth format and convert it to numpy, which would be human readable? I would like to be able to do further manipulation of the weights in numpy.

    Thank You for your patients Tom

    opened by minertom 2
  • The use of

    The use of "log" vs "ln", chapter 3 - Loss

    I hope that this is not trivial. I was confused by it for a while so I thought that I should bring it up.

    Normally, in engineering, when I see Log, without any base, it is assumed to be logarithm to the base 10. In the following from chapter 3, Loss first_summation = torch.log(positive_pred).sum()

    Printing this first summation I noticed the value = tensor(-0.1054) I was going though the math and realized that this is not equal to log 10 of .9, which is -.045.

    Going to the pytorch documentation I saw that "log Returns a new tensor with the natural logarithm of the elements of input."

    Of course, in the "From Logits to Probablilties" there is shown the relationship which "kind of" hints towards natural logarithms or log to the base e, but the whole confusion can be avoided by using the symbol "ln" as opposed to "log".

    Do you agree?

    Thank You Tom

    opened by minertom 2
Owner
Daniel Voigt Godoy
Data scientist, developer, teacher and writer. Author of "Deep Learning with PyTorch Step-by-Step: A Beginner's Guide".
Daniel Voigt Godoy
A research toolkit for particle swarm optimization in Python

PySwarms is an extensible research toolkit for particle swarm optimization (PSO) in Python. It is intended for swarm intelligence researchers, practit

Lj Miranda 847 Oct 22, 2021
null 114 Oct 20, 2021
Deal or No Deal? End-to-End Learning for Negotiation Dialogues

Introduction This is a PyTorch implementation of the following research papers: (1) Hierarchical Text Generation and Planning for Strategic Dialogue (

Facebook Research 1.3k Oct 15, 2021
🛠 All-in-one web-based IDE specialized for machine learning and data science.

All-in-one web-based development environment for machine learning Getting Started • Features & Screenshots • Support • Report a Bug • FAQ • Known Issu

Machine Learning Tooling 2.2k Oct 23, 2021
All course materials for the Zero to Mastery Deep Learning with TensorFlow course.

All course materials for the Zero to Mastery Deep Learning with TensorFlow course.

Daniel Bourke 1.7k Oct 24, 2021
Advanced Deep Learning with TensorFlow 2 and Keras (Updated for 2nd Edition)

Advanced Deep Learning with TensorFlow 2 and Keras (Updated for 2nd Edition)

Packt 1.1k Oct 21, 2021
A repo with study material, exercises, examples, etc for Devnet SPAUTO

MPLS in the SDN Era --> DevNet SPAUTO Get right to the study material: Checkout the Wiki! A lab topology based on MPLS in the SDN era book used for 30

Hugo Tinoco 29 Aug 1, 2021
Free Book about Deep-Learning approaches for Chess (like AlphaZero, Leela Chess Zero and Stockfish NNUE)

Free Book about Deep-Learning approaches for Chess (like AlphaZero, Leela Chess Zero and Stockfish NNUE)

Dominik Klein 75 Oct 22, 2021
TorchIO is a Medical image preprocessing and augmentation toolkit for deep learning. Part of the PyTorch Ecosystem.

Medical image preprocessing and augmentation toolkit for deep learning. Part of the PyTorch Ecosystem.

Fernando Pérez-García 1.2k Oct 20, 2021
Official repository of my book: "Deep Learning with PyTorch Step-by-Step: A Beginner's Guide"

This is the official repository of my book "Deep Learning with PyTorch Step-by-Step". Here you will find one Jupyter notebook for every chapter in the book.

Daniel Voigt Godoy 133 Oct 21, 2021
Solutions of Reinforcement Learning 2nd Edition

Solutions of Reinforcement Learning, An Introduction

YIFAN WANG 1000 Oct 14, 2021
Jupyter notebooks for the code samples of the book "Deep Learning with Python"

Jupyter notebooks for the code samples of the book "Deep Learning with Python"

François Chollet 13.8k Oct 23, 2021
Sample code from the Neural Networks from Scratch book.

Neural Networks from Scratch (NNFS) book code Code from the NNFS book (https://nnfs.io) separated by chapter.

Harrison 110 Oct 4, 2021
Learning Intents behind Interactions with Knowledge Graph for Recommendation, WWW2021

Learning Intents behind Interactions with Knowledge Graph for Recommendation This is our PyTorch implementation for the paper: Xiang Wang, Tinglin Hua

null 87 Oct 19, 2021
BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization

BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization Authors: Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong,

Salesforce 67 Oct 13, 2021
Providing the solutions for high-frequency trading (HFT) strategies using data science approaches (Machine Learning) on Full Orderbook Tick Data.

Modeling High-Frequency Limit Order Book Dynamics Using Machine Learning Framework to capture the dynamics of high-frequency limit order books. Overvi

Chang-Shu Chung 861 Oct 15, 2021
This project uses reinforcement learning on stock market and agent tries to learn trading. The goal is to check if the agent can learn to read tape. The project is dedicated to hero in life great Jesse Livermore.

Reinforcement-trading This project uses Reinforcement learning on stock market and agent tries to learn trading. The goal is to check if the agent can

Deepender Singla 1.4k Oct 20, 2021
Springer Link Download Module for Python

♞ pupalink A simple Python module to search and download books from SpringerLink. ?? This project is still in an early stage of development. Expect br

Pupa Corp. 19 Sep 14, 2021
A fast and easy to use, moddable, Python based Minecraft server!

PyMine PyMine - The fastest, easiest to use, Python-based Minecraft Server! Features Note: This list is not always up to date, and doesn't contain all

PyMine 72 Oct 8, 2021