I'm trying to make DenseNet for ImageNet dataset. But, it doesn't converge well. Have you ever try DenseNet to ImageNet dataset? Please share it if you have any successful densenet network for imagenet.

Hi there and great work! I've actually also figured out the very same concept myself prior to finding out you guys have already tested and published it. ✍(◔◡◔) Some of the design decisions I've made were different, so I'd like to compare.

Where you're reporting results on Cifars, if you could also add the number of parameters you are using and, possibly, an estimated amount of computation, that would be highly beneficial. It's really necessary for serious comparisons and ability to perfect even this very architecture. Also, if you could add your training logs that would also be of great insight.

As for how to measure the amount of computation, that's quite a tough thing to do, so I'd recommend to at least measure training time, which is a very inexact measure, but, well, provides at least some insights.

I've had 19.5% on Cifar-100+ with mean and std not adjusted (whole dataset just scaled to [0..1] values) with 24m params and forward+backwards running for 220 sec / epoch on GTX Titan X with the best dense-type architecture that I designed prior (I could only experiment on a single GTX Titan X, don't really have a lot of computational resources). It didn't have preactivation. It would most likely at least match the results those you've published for DenseNet (L=100, k=24) CIFAR-100+ if I used the right dataset (with std and mean adjusted). My code https://github.com/ibmua/Breaking-Cifar/blob/master/models/hoard-2-x.lua (uses 4-spaced tabs. To achieve that result I used depth=2 sequences=2 , here's a log of the end of training https://github.com/ibmua/Breaking-Cifar/blob/master/logs/load_59251794/log.txt ). Mind that I used groups, which are only accessible via Soumith's "cudnn", so if you'll want to try this you probably want to clone the whole thing. Also, not that I didn't use any Droupout (haven't even tried)

I tested Wide-DenseNet-BC (L=40, k=48) on CIFAR-10 augmentation, see https://github.com/seasonyc/densenet/blob/bf99d7f459ca7754c37ff58c6610eb76e93f7990/cifar10-test.py#L217 in https://github.com/seasonyc/densenet but could only get 4.5% error rate.

I tried to tune some hyper parameters, e.g. dropout, weight decay, learning rate... but always couldn't get better result. Now I am testing to follow the lr decay of wide resnet training, i.e. initial 0.1, by 0.2 per 60 epochs, but I very suspect if it will take effect...

Would you like to give me any suggestion for it?

Thanks YC

Hi author, as you claimed in both repo and paper, the numer of parameters of densenet-100-12 is 7.0M and densenet-100-24 is 27.72M. However when I examine the parameters in following way

-- main.lua, line 32
-- Create model
local model, criterion = models.setup(opt, checkpoint)

params = model:getParameters()
print(#params)

I got 4.06M for densenet-100-12 and 16.11M for densenet-100-24. Did I count it in a wrong way?

Hi, I saw this caffe implementation which is memory efficient. https://github.com/Tongcheng/DN_CaffeScript

And I also notice this in wiki

Memory efficient implementation (newly added feature on June 6, 2017)

There is an option -optMemory which is very useful for reducing GPU memory footprint when training a DenseNet. By default, the value is set to 2, which activates the shareGradInput function 
....

Does that caffe use the above memory efficient way to implementation?

Thanks.

I may be misunderstanding the architecture, but why does DenseNet decide to concatenate feature maps from the current layer to pass backward instead of using "true" residual connections?

In your network architecture for CIFAR and imagenet dataset, what does purpose use the first convolution (before pooling-denseblock1)? In the imagenet, you use two convolutions block before entering the dense-block, while CIFAR just one, any reason? Thanks

Congrats for best paper on CVPR 2017! I'm troubled with the memory problem with densenet. Would you share your wide-densenet implementation and pre-train models publicly?

Best!

I was wondering if you ever tried the extreme case growth_rate = 1 with a very deep network? Just as an exercise I implemented a fully-connected dense block with growth_rate = 1 and depth = 50 on a 2D dataset so I could visualize what each neuron was learning, the results where very nice.

I've seen that you use:

BN -> ReLU -> Conv3x3 -> Dropout

on the normal case, or

BN -> ReLU -> Conv1x1 -> Dropout -> BN -> ReLU -> Conv3x3 -> Dropout

when using bottleneck. The question is why? Most networks use e.g.

Conv3x3 -> BN -> ReLU -> Dropout

Why did you invert the order? Did you get better results this way?

Thanks in advance!

I've followed one of Tensorflow implementations of DenseNet (https://github.com/ikhlestov/vision_networks) to reproduce DenseNet-BC-100-12. It seemed to me that the tensorflow implementation is nearly equivalent with one from this repo, but I couldn't reach to ~4.5 % error (the best one was about ~4.8 %, by the way) Could you give me any reasons why it is? I already compared two codes very carefully, but couldn't find.

I take the liberty to bother you. I want to ask you a question about image classification, but images are not images in the usual sense. The experiment is to detect different objects with the collected WiFi signals. The above figure shows the rssi and phase change curves of the WiFi signal when a bottle is placed indoors. Now I want to classify objects according to these curves. Currently, Resnet50 DenseNet201 is used but the accuracy is not high and only 80。So I would like to ask what kind of network structure is better to use deep learning to classify this kind of image?

Hi,

What's the receptive field of densenet121 and densenet161? The Distill blog post doesn't mention it. Is it more than that of resnet101 (RF: 1027)?

Hi! Thank you for publishing this. I'm following pytorch implementation of DenseNet, specifically I'm using densenet161 for extracting features from images. I'm wondering, in your implementation here, why after the last Denseblock are you adding additional transition layer, consisting of batch-normalization and ReLU? I don't see any notion of those operation in the paper. Am I missing something? I'm asking because I'm wondering how those transitions are influencing quality of learnt features when we use DenseNet not for classifying images but rather as feature extractor. Thanks!

Hi, Thanks for your great work In section 3 in paper "However, the identity function and the output of H_l are combined by summation, which may impede the information flow" I did not understand that, there is also similiar question on stackoverflow https://stackoverflow.com/questions/52696110/resnet-how-could-summation-impede-the-information-flow Looking forward to your reply

Hello, I want to reproduce the results on densenet-cifar10/cifar100, but got lower accuracy on tensorflow implementation. There is one question on model architecture, In the paper Implementation Details part: "Before entering the first dense block, a convolutional with 16(or twice the growth eate for DenseNet-BC) output channels is performed on the input images." However the code seemingly all use twice the growth rate on first block. https://github.com/liuzhuang13/DenseNet/blob/master/models/densenet.lua#L15 https://github.com/liuzhuang13/DenseNet/blob/master/models/densenet.lua#L70 Since I did not get the same accuracy level with pytorch implementaion (25.53% on cifar100 d_40_k_12_no_bottleneck, 24.42% in paper. following the same data augmentation as your official code), I am wondering whether this caused the difference? (since I am a beginner on pytorch, maybe it is in other part of the code, can you point it out) Thanks in advance