This repository contains the database and code used in the paper Embedding Arithmetic for Text-driven Image Transformation

Hi! I'm trying to reproduce SIMAT score of zero shot CLIP in paper. (Table1 , Fig5) I used eval.py for reproducing this score, and change it slightly to remove adaptation layer.

def simat_eval(args):
    #img_head, txt_head, emb_key='clip', lbds=[1], test=True:, tau
    # get heads !
    emb_key = 'clip'
    heads = torch.load(f'data/head_{emb_key}_t={args.tau}.pt')
    output = {}
    transfos = pd.read_csv('simat_db/transfos.csv', index_col=0)
    transfos = transfos[transfos.is_test == (args.domain == 'test')]
    img_embs = torch.load('data/clip_simat_2.pt').float()


    img_embs = img_embs.normalize()
    #img_embs = heads['img_head'](clip_simat).normalize()
    value_embs = torch.stack([img_embs[did] for did in transfos.dataset_id])

    word_embs = dict(torch.load(f'data/simat_words_{emb_key}_2.ptd'))
    #w2v = {k:heads['txt_head'](v.float()).normalize() for k, v in word_embs.items()}
    w2v = {k:(v.float()).normalize() for k, v in word_embs.items()}
    delta_vectors = torch.stack([w2v[x.target] - w2v[x.value] for i, x in transfos.iterrows()])

    oscar_scores = torch.load('simat_db/oscar_similarity_matrix.pt')
    weights = 1/np.array(transfos.norm2)**.5
    weights = weights/sum(weights)

    for lbd in args.lbds:
        target_embs = value_embs + lbd*delta_vectors

        nnb = (target_embs @ img_embs.T).topk(5).indices

        nnb_notself = [r[0] if r[0].item() != t else r[1] for r, t in zip(nnb, transfos.dataset_id)]

        scores = np.array([oscar_scores[ri, tc] for ri, tc in zip(nnb_notself, transfos.target_ids)]) > .5


        output[lbd] = 100*np.average(scores, weights=weights)
    return output

when i use your embedding file (clip_simat.pt , simat_words_clip.ptd) it results well , but I cannot get full performance with the embedding that I made with encode.py.
(I used the original CLIP model in openAI repository, so maybe it is not problem of CLIP model.)

I would be appreciated if you check it. Thank you

This is pull request was created automatically because we noticed your project was missing a Code of Conduct file.

Code of Conduct files facilitate respectful and constructive communities by establishing expected behaviors for project contributors.

This PR was crafted with love by Facebook's Open Source Team.

This is pull request was created automatically because we noticed your project was missing a Contributing file.

CONTRIBUTING files explain how a developer can contribute to the project - which you should actively encourage.

This PR was crafted with love by Facebook's Open Source Team.

Hi,

When I ran the command python adaptation.py --backbone ViT-B/32 --lr 0.001 --tau 0.1 --batch_size 512 --gpus 2, I encountered the error. The error is as follows :

  ...
  File "/usr/.conda/envs/simat/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 93, in forward
    return self.module.validation_step(*inputs, **kwargs)
  File "adaptation.py", line 99, in validation_step
    img_ = self.core.encode_image(img).detach()
  File "/usr/.conda/envs/simat/lib/python3.8/site-packages/clip/model.py", line 341, in encode_image
    return self.visual(image.type(self.dtype))
  File "/usr/.conda/envs/simat/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/.conda/envs/simat/lib/python3.8/site-packages/clip/model.py", line 224, in forward
    x = self.conv1(x)  # shape = [*, width, grid, grid]
  File "/usr/.conda/envs/simat/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/.conda/envs/simat/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/.conda/envs/simat/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [768, 3, 32, 32], expected input[1, 1536, 224, 224] to have 3 channels, but got 1536 channels instead

I thought the error occurred since the size of the batch and the channel are not decoupled. To resolve the error, I inserted a line of code img = img.reshape(-1, 3, img.shape[-1], img.shape[-1]) which separates the batch and the channel at Line77 and Line90 in adaptation.py. Then, I could fully run the code without the error.

Is my modification the right solution? I had another reproduction problem in this code, but I'm worried if it's the cause here.

Thanks, Janet

Hi @PhazCode,

Thanks for this great work.

I tried to reproduce training new adaptation layers of CLIP on the COCO dataset. After training, I compute SIMAT scores with my trained weights and the results are as follows :

lbd=1.0: 13.09
lbd=2.0: 25.78
lbd=3.0: 29.94
lbd=4.0: 28.73
lbd=5.0: 26.21

When I compute SIMAT scores with provided weights, the results are as follows :

lbd=1.0: 47.59
lbd=2.0: 35.78
lbd=3.0: 29.04
lbd=4.0: 26.36
lbd=5.0: 24.52

For training, I followed the hyperparameters mentioned in paper. The other parameters were initialized with the default values at adaptation.py. The hyperparameters are as follows :

• max_epochs : 30
• batch_size : 4096
• lr : 1e-3
• tau : 0.1
• sched_step_size : 25
• sched_gamma : 0.1

The command that I used was python adaptation.py --backbone ViT-B/32 --lr 1e-3 --tau 0.1 --batch_size 4096 --wandb --max_epochs 30 And I also ran the command written in README.md python adaptation.py --backbone ViT-B/32 --lr 0.001 --tau 0.1 --batch_size 512

Unfortunately I couldn't retreive the same or similar results of the provided weights in both commands. Is there any mistakes that I did or any solution to resolve this problem?

My settings are as follows:

• Pytorch : 1.12.1
• CUDA : 11.3
• Python : 3.8.13
• gpu : 8 Tesla V100 GPUs

Thanks, Janet

Hi,

Congrats on the amazing work!

I am trying to reproduce the results of your fine-tuning CLIP on MS-COCO experiment (figure 5, section 5.2 in the paper) -- however I am running into issues while doing the exact fine-tuning and am getting lower SIMAT scores than reported in your paper.

These are the exact training loss and validation loss plots while doing the fine-tuning:

Adaption at tau=0.01:

Adaption at tau=0.1:

I used the exact same hyperparameter settings as you have done in your adaptation script:

• learning rate=1e-3
• lr decay schedule with step_size=25 and gamma=0.1
• num_epochs=50
• gradient clipping at norm=1

Do you have any insights into where the training script might be going wrong and why the loss seems to be stagnating as we step through training? After adaptation (with these training plots), I get a SIMAT score of 37.10 compared to your 47.5 (at tau=0.1, lambda=1). Similarly, I get a SIMAT score of around 16.61 compared to your 17.10 (at tau=0.01, lambda=1).

Note: I had to reimplement chunks of your code in pytorch as I believe parts of your adaptation script were incomplete in Pytorch lightning -- I would be happy to share my adaptation script with you if that would help!

Hoping for a prompt response.