Hi, thanks for this project. Just opening for future investigation. I am finding that training with more than one GPU using the basic pytorch DPP demo on CIFAR-10 results in NaN outputs after a few epochs. Training using a single gpu works great within the DPP framework.

The implementation from [meliketoy](https://github.com/meliketoy/wide-resnet.pytorch works fine, but uses more gpu memory.

Hey there,

I'm trying WideResNet on the following dataset, which consists of 256x256x3 images which should be classified into 17 classes.

However I have a size issue on the network. With a batch size of 1, I expected a 1x17 output but I get a 64x17 output. With a batch size of 8 I get 512 x 17 output.

I'm trying to understand the shapes of the tensors at each stage with with the following prints:

def forward(self, x):
        print(x)
        out = self.conv1(x)
        print(out)
        out = self.block1(out)
        print(out)
        out = self.block2(out)
        print(out)
        out = self.block3(out)
        print(out)
        out = self.relu(self.bn1(out))
        print(out)
        out = F.avg_pool2d(out, 8)
        print(out)
        out = out.view(-1, self.nChannels)
        print(out)
        out = self.fc(out)
        print(out)
        return out

$  python main.py
Variable containing:
...
[torch.FloatTensor of size 1x3x256x256]

Variable containing:
...
[torch.FloatTensor of size 1x16x256x256]

Variable containing:
...
[torch.FloatTensor of size 1x64x256x256]

Variable containing:
...
[torch.FloatTensor of size 1x128x128x128]

Variable containing:
...
[torch.FloatTensor of size 1x256x64x64]

Variable containing:
...
[torch.FloatTensor of size 1x256x64x64]

Variable containing:
...
[torch.FloatTensor of size 1x256x8x8]

Variable containing:
...
[torch.FloatTensor of size 64x256]

Variable containing:
...
[torch.FloatTensor of size 64x17]

Will allow importing the model like so:

model = torch.hub.load(
    'xternalz/WideResNet-pytorch', 
    'WideResNet', 
    depth=28, 
    num_classes=10, 
    widen_factor=2
)

Example: https://colab.research.google.com/drive/1CnYutDFEk9SSjdsEeE5nY5SPPZ843MNm?usp=sharing

@xternalz I think your avgpool https://github.com/xternalz/WideResNet-pytorch/blob/master/wideresnet.py#L81 and the original avgpool https://github.com/szagoruyko/wide-residual-networks/blob/master/pytorch/resnet.py#L56 use different strides (yours=kernelsize=8, original=1).

In your code, you add dropout after relu2, your process is like dropout(relu2(bn2(conv1(relu1(bn1(x)))))).

But it this code, he adds dropout after conv1, his process is like relu2(bn2(dropout(conv1(relu1(bn1(x)))))).

Does it matter? What is the difference on performance between the two methods? I'm troubled, I can't reproduce the performance on CIFAR10(I only get accuracy at 93.2%) using the second method.