From my understanding, we know that tensorflow and pytorch can calculate gradients automatically. In original coral implementation, they used caffe which cannot calculate gradients automatically and thats why they used back-propagation. So can we use simply the following correlation loss in pytorch as well? (https://github.com/tensorflow/models/blob/master/research/domain_adaptation/domain_separation/losses.py)

def correlation_loss(source_samples, target_samples):
    source_samples -= tf.reduce_mean(source_samples, 0)
    target_samples -= tf.reduce_mean(target_samples, 0)
    source_samples = tf.nn.l2_normalize(source_samples, 1)
    target_samples = tf.nn.l2_normalize(target_samples, 1)
    source_cov = tf.matmul(tf.transpose(source_samples), source_samples)
    target_cov = tf.matmul(tf.transpose(target_samples), target_samples)
    corr_loss = tf.reduce_mean(tf.square(source_cov - target_cov))
    return corr_loss

In your code you are using pretrained model. For an example, I will not use pretrained model, then what can I do? I am writing a data loader code. Hopefully, I will get good accuracy as paper and then I will send pull request.

Please help me so that I can use your code without pretrained model.

Thanks in advanced.

Hello, I have some questions about your testing codes in main.py: 1.

# sum up batch loss
test_loss += torch.nn.functional.cross_entropy(out, target, size_average=False).data[0]
# get the index of the max log-probability
pred = out.data.max(1, keepdim=True)[1]
correct += pred.eq(target.data.view_as(pred)).cpu().sum()

why don't you directly calculate the loss as you do for training like this: classification_loss = torch.nn.functional.cross_entropy(out1, source_label) Instead, you sum up the loss then divide by the times you sum up. Is there any reason for that?

out1, out2 = model(data, data)
out = out1 if mode == 'source' else out2

This part looks a little bit weird to me. Please correct me if I am wrong. out1 is the output from forwarding data to the source branch, and out2 is the output from forwarding data to the target branch. I thought we still use source branch for testing since the parameters learned in the source branch will be affected by the target branch through the CORAL loss. My concern is that the target branch is not trained using the source labels, so I am not sure whether it will produce the correct labels for the testing data. It would be great if you can clarify my concern, or you can show the result forwarding the testing data through the source branch. Thank you!!!

    out1, out2 = model(source_data, target_data)

    classification_loss = torch.nn.functional.cross_entropy(out1, source_label)
    coral = models.CORAL()
    coral_loss = coral(out1, out2)

    sum_loss = _lambda*coral_loss + classification_loss

From your codes, the input to the CORAL loss function is the prediction. Although it seems the case in the original DeepCORAL paper, it's not really reasonable to me. To my understanding, the main goal of domain adaptation is to match the feature distribution between source and target domain, so I think it makes more sense to calculate the loss function using the feature vectors from previous layers (e.g. fc6, fc7). In addition, in some famous DA works: https://arxiv.org/pdf/1412.3474.pdf http://proceedings.mlr.press/v37/ganin15.pdf https://arxiv.org/pdf/1502.02791.pdf They all have loss functions for those layers. Maybe you can try different positions of the CORAL loss function and see whether it improve the performance or not. Thanks.

I think that there's an error in the calculation of covariance matrices. In the line xm = torch.mean(source, 1, keepdim=True) - source, imo 1 must be replaced by 0, as I would like to calculate the mean of each feature along the entire batch. Please correct me if I am wrong. Thanks, wonderful work by the way.

Hi,

I was using the code from models.py to try and improve transfer learning results between two datasets. It made me wonder what the point of self.target_fc is in the DeepCORAL class since it is never actually used in the forward pass. Instead, `self.source_fc' is applied to both the source and target data.

Also, in main.py the test function has a mode argument which selects the output to return. Given the fact that it calls model(data, data) and that there is no difference between how these two inputs will be processed, is such indexing redundant?

I noticed in the Deep CORAL paper that the CORAL loss is calculated by loss = loss / (4 * d * d) But line 26 of models.py defines the CORAL loss as loss = loss/(4*d*4) I do not understand. Is there a mistake?

Hello, in models DeepCoral you declare a target_fc but it seems to me it's never used:

class DeepCORAL(nn.Module):
    def __init__(self, num_classes=1000):
        super(DeepCORAL, self).__init__()
        self.sharedNet = AlexNet()
        self.source_fc = nn.Linear(4096, num_classes)
        self.target_fc = nn.Linear(4096, num_classes)

        # initialize according to CORAL paper experiment
        self.source_fc.weight.data.normal_(0, 0.005)
        self.target_fc.weight.data.normal_(0, 0.005)

    def forward(self, source, target):
        source = self.sharedNet(source)
        source = self.source_fc(source)

        target = self.sharedNet(target)
        target = self.source_fc(target)
        return source, target

Am I missing something?

I run successfully python3 main.py but when I test, then got error. Can you fix it please?

~$ python -m DeepCORAL.tests.test -v

(pytorch_python3) user@PA00074589:~$ python -m DeepCORAL.tests.test -v test_CORAL_backward (main.TestCORAL) ... ERROR test_CORAL_forward (main.TestCORAL) ... ERROR test_feature_covariance_mat (main.TestCORAL) ... ERROR test_forbenius_norm (main.TestCORAL) ... ok

====================================================================== ERROR: test_CORAL_backward (main.TestCORAL)

Traceback (most recent call last): File "/home/user/DeepCORAL/tests/test.py", line 37, in test_CORAL_backward valid = torch.autograd.gradcheck(coral, test[x], eps=1e-3) File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/autograd/gradcheck.py", line 154, in gradcheck output = func(*inputs) File "/home/user/DeepCORAL/models.py", line 31, in forward cs = feature_covariance_mat(ns, source) File "/home/user/DeepCORAL/models.py", line 13, in feature_covariance_mat tmp = ones_t.matmul(d) File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/tensor.py", line 180, in matmul return torch.matmul(self, other) File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/functional.py", line 173, in matmul return torch.mm(tensor1, tensor2) TypeError: torch.mm received an invalid combination of arguments - got (torch.cuda.FloatTensor, torch.FloatTensor), but expected one of:

• (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat2) didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)
• (torch.cuda.sparse.FloatTensor source, torch.cuda.FloatTensor mat2) didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)

====================================================================== ERROR: test_CORAL_forward (main.TestCORAL)

Traceback (most recent call last): File "/home/user/DeepCORAL/tests/test.py", line 29, in test_CORAL_forward res = coral.forward(*test[x]) File "/home/user/DeepCORAL/models.py", line 31, in forward cs = feature_covariance_mat(ns, source) File "/home/user/DeepCORAL/models.py", line 13, in feature_covariance_mat tmp = ones_t.matmul(d) File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/tensor.py", line 180, in matmul return torch.matmul(self, other) File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/functional.py", line 173, in matmul return torch.mm(tensor1, tensor2) TypeError: torch.mm received an invalid combination of arguments - got (torch.cuda.FloatTensor, torch.FloatTensor), but expected one of:

• (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat2) didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)
• (torch.cuda.sparse.FloatTensor source, torch.cuda.FloatTensor mat2) didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)

====================================================================== ERROR: test_feature_covariance_mat (main.TestCORAL)

Traceback (most recent call last): File "/home/user/DeepCORAL/tests/test.py", line 20, in test_feature_covariance_mat res = feature_covariance_mat(*test[x]) File "/home/user/DeepCORAL/models.py", line 13, in feature_covariance_mat tmp = ones_t.matmul(d) File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/tensor.py", line 180, in matmul return torch.matmul(self, other) File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/functional.py", line 173, in matmul return torch.mm(tensor1, tensor2) TypeError: torch.mm received an invalid combination of arguments - got (torch.cuda.FloatTensor, torch.FloatTensor), but expected one of:

• (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat2) didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)
• (torch.cuda.sparse.FloatTensor source, torch.cuda.FloatTensor mat2) didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)

Ran 4 tests in 1.179s

FAILED (errors=3)

There is a difference between the CORAL Loss used in the paper and the one described in this code. Is it because they both are giving the same result ? The Forbenius Norm is also different.

The results were also different from the ones in the paper. Is there any idea why the results have changed? Any response would be helpful.

DEEP_CORAL_LOSS: def CORAL(source, target): d = source.data.shape[1] ns = source.data.shape[0] nt = target.data.shape[0]

# source covariance
xm = torch.mean(source, 0, keepdim=True) - source
xc = (xm.t() @ xm) / (ns-1)

# target covariance
xmt = torch.mean(target, 0, keepdim=True) - target
xct = xmt.t() @ xmt / (nt-1)

print(xc, xct)

# frobenius norm between source and target
loss = torch.sum(torch.mul((xc - xct), (xc - xct)))
loss = loss/(4*d*d)
return loss

I'm sorry i met the following error using the office31 dataset

[INFO] Loading datasets: amazon [INFO] Loading datasets: webcam Traceback (most recent call last): File "main.py", line 151, in res = train(model, optimizer, e+1, _lambda) File "main.py", line 28, in train source, target = list(enumerate(source_loader)), list(enumerate(target_loader)) File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 201, in next return self._process_next_batch(batch) File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch raise batch.exc_type(batch.exc_msg) TypeError: Traceback (most recent call last): File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 40, in _worker_loop samples = collate_fn([dataset[i] for i in batch_indices]) File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 40, in samples = collate_fn([dataset[i] for i in batch_indices]) File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 67, in getitem img = self.transform(img) File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torchvision/transforms.py", line 29, in call img = t(img) File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torchvision/transforms.py", line 139, in call ow = int(self.size * w / h) TypeError: unsupported operand type(s) for /: 'tuple' and 'int'

I noticed that both the covariance and the Frobenius norm are computed differently in your implementation.

You compute the Frobenius norm as below: # frobenius norm between source and target loss = torch.mean(torch.mul((xc - xct), (xc - xct))) However as stated here http://mathworld.wolfram.com/FrobeniusNorm.html , after squaring each element and summing them, should be computed the square root of the sum not the mean of the squared elements.

In the original paper the covariances are computed as below : https://arxiv.org/abs/1607.01719

While in your implementation:

# source covariance 
xm = torch.mean(source, 0, keepdim=True) - source
xc = xm.t() @ xm

# target covariance
xmt = torch.mean(target, 0, keepdim=True) - target
xct = xmt.t() @ xmt  

I got this accuracy for SHVN -----> MNIST

###Test Source: Epoch: 2460, avg_loss: 0.0003, Accuracy: 73255/73257 (100.00%) ###Test Target: Epoch: 2460, avg_loss: 21.5493, Accuracy: 35242/60000 (58.74%)

That means data loading does not matter to get good accuracy. Because for SHVN---->MNIST, I just use default data processing.