When I trained the classification network(using both pretrained vgg and resnet weights), the loss didn't decrease succesfully using given hyperparamters. For example, the loss of vgg network vibrated around 0.24 after 1k iters, I also tried the learning rate of 0.01, it also failed. Could you give me some suggestions? Thanks~

Hi Jiwoon,

thanks for sharing this nice work! I'm trying to generate the CAMs with a model I trained myself, but the mIoU I get is quite low, 42.28 (on PASCAL VOC2012 / train). I first ran

python3 train_cls.py --lr 0.1 --batch_size 16 --max_epoches 15 --crop_size 448 --network network.resnet38_cls --voc12_root ./data --weights weights/ilsvrc-cls_rna-a1_cls1000_ep-0001.pth --wt_dec 5e-4

and then generated the CAMs with

python3 infer_cls.py --infer_list voc12/train.txt --voc12_root ./data --network network.resnet38_cls --weights res38_cls.pth --out_cam_pred ./cams

Is there anything amiss? To get mIoU of 48% in the paper, should I use the dCRF? Here I don't.

More details, in case it helps:

| Class | # | IoU | Pr | Re | |:---:|---:|---:|---:|---:| | background | 10581 | 71.6 | 83.4 | 83.9 | aeroplane | 586 | 37.8 | 40.8 | 93.8 | bicycle | 485 | 43.6 | 50.3 | 86.1 | bird | 698 | 34.1 | 39.6 | 86.1 | boat | 460 | 28.1 | 35.1 | 79.0 | bottle | 651 | 27.0 | 31.9 | 81.9 | bus | 385 | 61.3 | 77.2 | 79.3 | car | 1079 | 39.9 | 50.1 | 81.1 | cat | 1000 | 43.8 | 73.3 | 57.9 | chair | 1063 | 34.0 | 44.1 | 68.3 | cow | 262 | 42.2 | 54.8 | 78.0 | diningtable | 520 | 39.2 | 54.6 | 65.4 | dog | 1176 | 41.9 | 66.4 | 63.4 | horse | 444 | 40.4 | 58.4 | 65.6 | motorbike | 481 | 51.9 | 62.8 | 83.0 | person | 3876 | 37.2 | 50.5 | 66.7 | potted-plant | 485 | 35.0 | 45.8 | 76.8 | sheep | 299 | 43.8 | 50.6 | 86.4 | sofa | 474 | 43.5 | 65.9 | 60.0 | train | 499 | 52.6 | 65.9 | 79.6 | tv/monitor | 548 | 38.9 | 45.3 | 87.4 | ambiguous | 330 | 0.0 | 0.0 | 0.0

mIou: 42.28 (background included)

Best, Nikita

hi @jiwoon-ahn @hardBird123 , I wanted to know if you trained the vgg16 model from scratch or used a imagenet pre-trained model and used only specific layers of it for training on voc12?

@jiwoon-ahn hi, Thanks for your nice work! I load your shared weights and achieve the same mIoU as you showed in README.

While I meet some trouble to train the model by myself and only achieve 59.077% mIoU (instead of 60.2%) on val set by ResNet38_aff. As you mentioned in other issues that the default params setting is just for vgg16, could you share the params setting for resnet38_aff?

Here is my res38_aff training setttings: lr=0.01 gpu=4 (may cause different batch number on each gpu) batchsize=8 max_epoch=8 loss_weight=1/4 * bg + 1/4 * fg + 1/2 * neg pretrained_model=res38_cls.pth (provided by you) alpha=4/16/32 other default params...

in train_cls.py,when the learning rate is set to 0.1,the loss is increasing,so I tried to set it to 0.01,it works,,so can you verify the learning rate in train_cls.py and train_aff.py?Thanks in advance!

suppose the augmented VOC dataset can be splited into train_aug(10582)+val(1449)+test(1456), when you report results on test split in your paper, do you use train_aug + val for training or just use train_aug? thx!

Thanks for the code!

For how many epochs did you train your model to achieve the results in the paper? I am wondering what a good practice would be for reporting the results since there isn't really a validation set for early stopping. Thanks!

Thanks for the code! How did you download the pretrained weights for ResNet-38? When I click on the link here (Optional) Mxnet and ResNet-38 pretrained weights [ilsvrc-cls_rna-a1_cls1000_ep-0001.params], I am just taken to another github repo. without instructions on how to get the pretrained weights. Thanks

Here index 0 is mapped to 2 and 2 mapped to 0 - https://github.com/jiwoon-ahn/psa/blob/master/network/vgg16d.py#L15

Image is initially loaded in RGB Mode - https://github.com/jiwoon-ahn/psa/blob/9493e59bef16687ecb4821387e38e6460857f508/voc12/data.py#L69

Finally, the transform is simply applied to RGB image, no where there is need for channel swapping. https://github.com/jiwoon-ahn/psa/blob/master/train_cls.py#L82

P.S. It could be a bug due to some legacy code using OpenCV to open image, which defaults to BGR mode.

Hi, thanks for your brilliant work! When I tried to reproduce the ResNet38 segmentation network, I found the repo you mentioned (https://github.com/jiwoon-ahn/psa/issues/7#issuecomment-430170981_) didn't give the training command. So I'm wondering if you can share your training command for ResNet-38 segmentation network?

Thank author for the code. I have two small questions:

1. regarding this small block of code below to obtain cam: def forward_cam(self, x): x = super().forward(x) x = self.fc8(x) x = F.relu(x) x = torch.sqrt(x) return x what's x=torch.sqrt(x) for?
2. regarding the pretrained caffe weghts, i.e. vgg16_20M.prototxt, on what dataset was it trained on? were you meant to use the deeplab_LargeFOV model of vgg16 version as the network to compute CAM, but why setting fc6_dilation=1 not 12 as in deeplab v1 paper?

At line https://github.com/jiwoon-ahn/psa/blob/master/network/vgg16_cls.py#L36. CAM is weighted combination of feature maps, so not sure why sqrt is used.

Please, could you provide some clarification.

Hello, I note that the order of paramters (params lr wd) in PolyOptimizer is different from official SGD(params lr momentum). So I think the value of wd will actually be assigned to momentum. Is it so?

class PolyOptimizer(torch.optim.SGD):

    def __init__(self, params, lr, weight_decay, max_step, momentum=0.9):
        super().__init__(params, lr, weight_decay)

Hi, I notice that the last fc layer in resnet38_cls.py is defined as self.fc8 = nn.Conv2d(4096, 20, 1, bias=False), while I found another implementation define the last layer as a linear self.fc = nn.Linear(2048, num_classes), I wonder why do you use convolution instead, and is there any difference?

I run the project and got the result out_cam/out_la_crf/out_ha_crf/out_rw. Where should I download GT? How should I calculate the mIOU? Can you provide the code to calculate the mIOU?

Hi Jiwoon,

when training the segmentation net on the final masks, what schedule did you use (e.g. learning rate/decay, num of iterations/epochs, frozen/learned BN)? My guess is that you might need early stopping to avoid overfitting to the errors in the masks used for pseudo-supervision.

Thanks, Nikita