What size cards are these networks tested and trained on? I just tried running "09 - Image Captioning" and I immediately get errors. I am testing this on a Titan X with 12 GB of memory:

Namespace(batch_size=128, caption_path='./data/annotations/captions_train2014.json', crop_size=224, embed_size=256, hidden_size=512, image_dir='./data/resized2014', learning_rate=0.001, log_step=10, model_path='./models/', num_epochs=5, num_layers=1, num_workers=2, save_step=1000, vocab_path='./data/vocab.pkl')
loading annotations into memory...
Done (t=0.89s)
creating index...
index created!
THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
Traceback (most recent call last):
  File "train.py", line 119, in <module>
    main(args)
  File "train.py", line 66, in main
    features = encoder(images)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/pytorch/pytorch-tutorial/tutorials/09 - Image Captioning/model.py", line 26, in forward
    features = self.resnet(images)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/.venv/local/lib/python2.7/site-packages/torchvision/models/resnet.py", line 146, in forward
    x = self.layer3(x)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 64, in forward
    input = module(input)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/.venv/local/lib/python2.7/site-packages/torchvision/models/resnet.py", line 85, in forward
    out = self.bn3(out)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/batchnorm.py", line 43, in forward
    self.training, self.momentum, self.eps)
  File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/functional.py", line 463, in batch_norm
    return f(input, weight, bias)
RuntimeError: cuda runtime error (2) : out of memory at /b/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu:66
Command exited with non-zero status 1

Hi , am new to PyTorch in model.py you do ,

embeddings = self.embed(captions)
embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)

that means you are only passing the context(from the EncoderCNN ) as input to the first time step, hence your outputs will always be 1+captions size as you are running the lstm for captions+1 number of time steps in the decoder , but looks like you have targets and outputs of same size. What am I missing here ? Thanks in advance .

@yunjey How we can train your image captioning code on another dataset? If we prepare a JSON file like (http://msvocds.blob.core.windows.net/annotations-1-0-3/captions_train-val2014.zip) the code can be trained on a new dataset?

Hi, I am running the 09 image captioning model. When I used the pytorch 0.10, I can set the batchsize to be 512 (takes around 8G for the GPU memory). But when I upgrade to pytorch 0.11， the maximum batchsize I can set is only around 30 (takes around 11G GPU memory). I am confused since the codebase is mainly the basic CNN and lstm models. Do you know where the problem comes from? Have you tried to upgrade to pytorch 0.11 to run it?

By the way, I am using Tesla K40c GPU. pytorch 0.10: 0.1.10+ac9245a pytorch 0.11: same problem with (0.1.11_5 and 0.1.11+b13b701)

Hello @yunjey . Nice to meet you and thanks for your work. I'm studying pytorch with your source.

I think the resolution of input image shoud be changed "256x256" to "224x224".

• Environment

  • hardware: MacBook Pro (Retina, 15-inch, Early 2013)
  • os: macOS Sierra 10.12.6

• Version

  • torch==0.3.0.post4
  • torchvision==0.2.0

• Code in error: pytorch-tutorial/tutorials/01-basics/pytorch_basics/main.py line 152-154

  # For test.
  images = Variable(torch.randn(10, 3, 256, 256)) # => randn(10, 3, 224, 224)
  outputs = resnet(images)
  

• Error message

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-7-55315f838129> in <module>()
      1 images = Variable(torch.randn(10, 3, 256, 256))
----> 2 outputs = resnet(images)
      3 print (outputs.size())   # (10, 100)

~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

~/.virtualenvs/fira/lib/python3.6/site-packages/torchvision/models/resnet.py in forward(self, x)
    149         x = self.avgpool(x)
    150         x = x.view(x.size(0), -1)
--> 151         x = self.fc(x)
    152 
    153         return x

~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     53 
     54     def forward(self, input):
---> 55         return F.linear(input, self.weight, self.bias)
     56 
     57     def __repr__(self):

~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
    833     if input.dim() == 2 and bias is not None:
    834         # fused op is marginally faster
--> 835         return torch.addmm(bias, input, weight.t())
    836 
    837     output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [10 x 2048], m2: [512 x 100] at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:1416

I try to train model by using my own datasets, but i have a problem of character encoding, i try to change the caption into Chinese, but the vab.pkl has many ' S'\xe7\xba\xa6' ' and when i use my trained model,it always gives /</start/>/</unk/>/</end/>/ ,so, could you please tell me how to solve this problem?Thank you very much.

When I follow the tutorials to train this network, I can not get effective image results. What should I do？？

I made the following modifications

Save the sampled images

fake_images = fake_images.view(fake_images.size(0), 3, 32, 32) torchvision.utils.save_image(fake_images.data, './sample/fake_samples_%d.png'

I downloaded the pretrained model to use. But I don't have the models/encoder-2 1000.ckpt' file. What I have from the model.zip are just "decoder-5 3000.pkl" and "encoder-5 3000.pkl".

In the Image Captioning tutorial in the DecoderRNN:

def forward(self, features, captions, lengths):
  embeddings = self.embed(captions)
  embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)
  packed = pack_padded_sequence(embeddings, lengths, batch_first=True)
  hiddens, _ = self.lstm(packed)
  outputs = self.linear(hiddens[0])
  return outputs

shouldn't the lenghts inside pack_padded be lenghts+1 to take into account the increased length due to the features added with cat?

e.g. (assuming numbers are captions, e is the embedding, f are the features and batch_size = 4) if

embeds:
e(126)  e(1214)  e(14)    e(4033)
e(126)  e(6)     e(84)    e(4033)
e(126)  e(3002)  e(4033)  e(0)
e(126)  e(3002)  e(4033)  e(0)

has lengths = [4,4,3,3] then

embeds_cat:
f_0  e(126)  e(1214)  e(14)    e(4033)
f_1  e(126)  e(6)     e(84)    e(4033)
f_2  e(126)  e(3002)  e(4033)  e(0)
f_3  e(126)  e(3002)  e(4033)  e(0) 

should have lengths = [5,5,4,4], right?

Hi, yunnjey. thanks for your work. I run your demo, but it doesn't seem to work. what's strange is that there is tf events under the log files. so is there any thing wrong?

When I Run the evaluation command python sample.py --image='png/example.png'

I got this error

Traceback (most recent call last): File "sample.py", line 97, in main(args) File "sample.py", line 61, in main sampled_ids = decoder.sample(feature) File "/users/PAS1273/osu8235/pytorch/pytorch-tutorial/tutorials/03-advanced/image_captioning/model.py", line 62, in sample hiddens, states = self.lstm(inputs, states) # (batch_size, 1, hidden_size), File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call result = self.forward(*input, **kwargs) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/modules/rnn.py", line 162, in forward output, hidden = func(input, self.all_weights, hx) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 351, in forward return func(input, *fargs, **fkwargs) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 284, in _do_forward flat_output = super(NestedIOFunction, self)._do_forward(*flat_input) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 306, in forward result = self.forward_extended(*nested_tensors) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 293, in forward_extended cudnn.rnn.forward(self, input, hx, weight, output, hy) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/backends/cudnn/rnn.py", line 208, in forward 'input must have 3 dimensions, got {}'.format(input.dim())) RuntimeError: input must have 3 dimensions, got 2

您好，仓库内容很全面，非常受益， 可否引荐下本人的笔记，把我对自动驾驶的理解分享给大家，希望大家和我一起不断完善相关内容，谢谢您！

Hello, the content of the repository is very comprehensive and very beneficial. Could you introduce my notes and share my understanding of autopilot with others? I hope you can continue to improve the relevant content with me, thank you!

Autopilot-Updating-Notes

I think the correct way the code the training is that

    optimizer.zero_grad()
    # Forward pass
    outputs = model(images)
    loss = criterion(outputs, labels)
    
    # Backward and optimize
    loss.backward()
    optimizer.step()

not that

    # Forward pass
    outputs = model(images)
    loss = criterion(outputs, labels)
    
    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/01-basics/pytorch_basics/main.py:

root@sriov-guest:~/dev-learn/gpu/pytorch/tutorial# python3 pytorch-main.py
tensor(2.)
tensor(1.)
tensor(1.)
w:  Parameter containing:
tensor([[-0.4404, -0.1190,  0.4829],
        [ 0.4956, -0.3760,  0.1435]], requires_grad=True)
b:  Parameter containing:
tensor([0.4760, 0.5684], requires_grad=True)
loss:  1.0722367763519287
dL/dw:  tensor([[-0.4775,  0.0949,  0.0065],
        [ 0.0698, -0.4295, -0.0664]])
dL/db:  tensor([0.3632, 0.3266])
loss after 1 step optimization:  1.0655767917633057
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ../../data/cifar-10-python.tar.gz
170499072it [00:32, 5230791.96it/s]
Extracting ../../data/cifar-10-python.tar.gz to ../../data/
torch.Size([3, 32, 32])
6
Traceback (most recent call last):
  File "pytorch-main.py", line 157, in <module>
    shuffle=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 270, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py", line 103, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0
root@sriov-guest:~/dev-learn/gpu/pytorch/tutorial#

Traceback (most recent call last): File "D:/PycharmWorkspace/pytorch-tutorial/tutorials/01-basics/pytorch_basics/main.py", line 154, in train_loader = torch.utils.data.DataLoader(dataset=custom_dataset, File "D:\anaconda3\envs\torch\lib\site-packages\torch\utils\data\dataloader.py", line 262, in init sampler = RandomSampler(dataset, generator=generator) # type: ignore File "D:\anaconda3\envs\torch\lib\site-packages\torch\utils\data\sampler.py", line 103, in init raise ValueError("num_samples should be a positive integer " ValueError: num_samples should be a positive integer value, but got num_samples=0