So I implemented a text-to-image search where I query a text - text encoder then image through the image encoder and retrieve top images for the query but it doesn't work well with CLOOB?

What is the main reasoning behind this?

I changed the checkpoint to RN50x4 and I got an error in run(model, classifier, ...) function about mismatch sizes (below). Any idea what could be the issue?

Calculating the text embeddings for all classes of the dataset
100%|██████████| 500/500 [00:02<00:00, 171.59it/s]Calculating the image embeddings for all images of the dataset

  0%|          | 0/8 [00:50<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-41-9989560228e7> in <module>()
      4 classifier = zero_shot_classifier(model, classnames, prompt_templates, device)
      5 print("Calculating the image embeddings for all images of the dataset", flush=True)
----> 6 accuracy = run(model, classifier, dataloader, device, accuracy_score)
      7 print('Zeroshot accuracy: ', accuracy.round(2))

5 frames
<ipython-input-32-7f9a8b257980> in run(model, classifier, dataloader, device, accuracy_metric)
     22 
     23             # predict
---> 24             image_features = model.encode_image(images)
     25             image_features /= image_features.norm(dim=-1, keepdim=True)
     26             logits = image_features @ classifier

/content/drive/My Drive/MLExperiments/cloob/src/clip/model.py in encode_image(self, image)
    388 
    389     def encode_image(self, image):
--> 390         return self.visual(image.type(self.dtype))
    391 
    392     def encode_text(self, text):

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/content/drive/My Drive/MLExperiments/cloob/src/clip/model.py in forward(self, x)
    162         x = self.layer3(x)
    163         x = self.layer4(x)
--> 164         x = self.attnpool(x)
    165 
    166         return x

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/content/drive/My Drive/MLExperiments/cloob/src/clip/model.py in forward(self, x)
     69         x = x.reshape(x.shape[0], x.shape[1], x.shape[2] * x.shape[3]).permute(2, 0, 1)  # NCHW -> (HW)NC
     70         x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0)  # (HW+1)NC
---> 71         x = x + self.positional_embedding[:, None, :].to(x.dtype)  # (HW+1)NC
     72         x, _ = F.multi_head_attention_forward(
     73             query=x, key=x, value=x,

RuntimeError: The size of tensor a (50) must match the size of tensor b (82) at non-singleton dimension 0

Hi,

Are there any more checkpoints? In the model_configs folder, I see several jsons but we can only download the RN50 and RN50x4 checkpoints.

Also, is it possible to add zero shot comparison with original clip models?

Hello,

I am trying to train a similar architecture in a self-supervised way. However, for the pre-text task the loss plateaus relatively quickly. What are the various graphs that were generated when this architecture was trained, both for accuracy and loss? Thank you.

Great work! Is it possible to have both pretrained model and configuration files to test the notebook? I found models and datasets in: https://ml.jku.at/research/CLOOB/downloads but when I run the notebook I need the config file RN50.json

Thank you very much

Why is the existing code for the Modern Hopfield Net (which has its own Github repository) not used here? And if I wanted to use it instead, what arguments would I have to call it with to get the same result as here?

Hi, after cloning the github repository I ran the command "conda env create --file environment.yml" in my conda base environment. The environment is not generated and the following error appears:

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - lz4-c==1.9.3=h9c3ff4c_1
  - _openmp_mutex==4.5=1_llvm
  ...

Looking at the conda version (conda --version), results in version 4.11.0. How can I do to correctly install the environment? Thanks in advice

The CLOOB paper mentioned that it used CUML-based logistic regression with L-BFGS algorithm to utilize GPUs for efficiency. My implementation works fine on small datasets (e.g., CIFAR), but CUDA out of memory occurred when dealing with large-scale ImageNet.

I have been stuck here for a pretty long time, and I cannot find useful support from the document or the Internet. Is it possible to provide a few code examples highlighting how to fix this problem?