PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

First I want to thank you. Based on my playing over at huggingface this seems to be the best piece of software I have hit on for image captioning. I am trying to get it to run locally on Windows 10.

I keep winding up with the following error:

OSError: [Errno 22] Invalid argument: 'C:\Users\Matthew/.cache\torch\hub\checkpoints\model*_base_caption.pth'

It seems to mangle the path to the .pth file. I tried putting some print statements in to try and figure out what was going on and it seems to be something around os.path.dirname, but I got lost. I tried a second tact and just downloaded all the .pth files and put them in the proper directory but that does the same thing. I tried a third tact and changed the model URL to a file:// that points at the file, and that results in a RuntimeError: checkpoint url or path is invalid error.

Any help is appreciated!

Is there any example scripts about how to use the pretrain checkpoint to Bootstrap captions from noisy image-text pairs, then get the much more cleaning train data?

I want to process a folder of images that I will use for comparing to an input text (which will be given at a different time). How do I use your colab to extract features from images and then at a later time, compare them to an input text? All the examples involve passing in an image and a text at the same time.

Hello,

When I try the demo on Huggingface Spaces (https://huggingface.co/spaces/Salesforce/BLIP) with a sample image, the output captions are different every time I enter the submit button with same decoding strategy.

But if I try the code on Colab with the same image and same decoding strategy, the output captions are always the same.

May I know why this is happening?

Thanks.

Thanks a lot for your awesome work! I have some question about the pre-training pipeline of BLIP. Considering the application of Captioner and Filter to generate pretraining data, I wonder if the process for the complete training is:

1. pre-training on noisy data
2. fine-tuning on COCO to obtain captioner and filter
3. using the captioner and filter from 2 to generate the final pretraining dataset
4. pre-training on the dataset obtained in 3

I used the pre-trained model for the 20th epoch,

and fine-tuning the caption task for 20 epochs on custom dataset,

the best model‘s results are as follows:

"caption": "post post post post post post post post post post post post post post post post"

The loss value dropped from 6.761 to 6.419.

Does the model converge? Do you have any good suggestions?

Hi, it is mentioned in the paper that you pretrain the model on two 16-GPU nodes with batch-size=2880, I wonder how long does it take to pretrain on the filtered CC + COCO + SBU + VG dataset, and could you share your log with me?

Thanks a lot.

Hi, Thank your nice work! I have a question, when I test captioning I find that vocab_size should be 30524, but do pre-training should be set 30522, am I right? Thanks.

I know how to ask the same question for multiple images at the same time and it will return different results to different images; how do I swap? I mean: can I ask multiple questions about the same image and return multiple different answers simultaneously(run the model only once)? If so, how can I do it? I tried feeding multiple questions as a list into the blip_vqa model but it raise an error seems like dimension of tensors mismatch. Thank you for your excellent work and look forward to your reply.

In the pre-training mode, each epoch is saved.

It seems that the code does not filter the optimal model by the accuracy or loss value.

What is your strategy?

Hello,

We are running the pretraining script and got this error:

Traceback (most recent call last):
  File "pretrain.py", line 184, in <module>
    main(args, config)
  File "pretrain.py", line 141, in main
    train_stats = train(model, data_loader, optimizer, epoch, device, config) 
  File "pretrain.py", line 69, in train
    loss_ita, loss_itm, loss_lm = model(image, caption, alpha = alpha)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jupyter/codebase/BLIP/models/blip_pretrain.py", line 163, in forward
    neg_idx = torch.multinomial(weights_t2i[b], 1).item()
RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)

Any idea how we can solve it please? Thanks

Hi,

At first I would like to say thank you for your great work which inspires me a lot.

I would like to know, is a BLIP w/ ViT-L + CapFilt-L model (use vit large as encoder and CapFilt for data augment) exist? I believe it should be stronger compared with BLIP w/ ViT-B + CapFilt-L and BLIP w/ ViT-L.

Thanks

I keep getting this error, I migrated from original BLIP to the new REPO here but the error keeps following. I did remove all existing models so it would download fresh copies to prevent me having wrong models, did not help, I try your example code kinda close to what its in your example but still it failes. Funny thing is it did work weeks ago and now it keeps failing so it might still be an issue with my box but I fail to find it, could you tell which direction I get shoot at?

I try this

raw_img = Image.open("00000-0-1.png").convert("RGB")

# setup device to use
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

vis_processor = load_processor("blip_image_eval").build(image_size=384)

model_type= "BLIP_large"

if model_type.startswith("BLIP"):
    blip_type = model_type.split("_")[1].lower()
    model = load_model_cache(
        "blip_caption",
        model_type=f"{blip_type}_coco",
        is_eval=True,
        device=device,
    )

use_beam = False #did try True either but same result
img = vis_processor(raw_img).unsqueeze(0).to(device)
captions = generate_caption(
    model=model, image=img, use_nucleus_sampling=not use_beam
)

Dear authors,

We have a working implementation of BLIP and 3 of its variants in huggingface transformers (image captioning, visual question answering, image text retrieval): https://github.com/huggingface/transformers/pull/20716 that is not merged yet

The license of the repository and model states that:

3. Neither the name of [Salesforce.com](http://salesforce.com/) nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

We would like to promote the addition of this architecture to transformers library. Therefore I would like to ask you the permission for promoting this contribution

Thank you very much in advance

Hi, thank you for the work!

I've played with the code and noticed that examples do not preserve the original aspect ration during the resize. E.g. https://github.com/salesforce/BLIP/blob/d10be550b2974e17ea72e74edc7948c9e5eab884/predict.py#L93 or the colab example.

I wonder if it is done on purpose?

Thanks for your great work and well-written code. We are evaluating the performance of Zero-shot Retrieval based on the checkpoint and evaluation code you provided. Our testing results are as follows and there is about 10 points' gap between those in your paper. We guess there maybe some bug, Could you please supply your evaluation results based on this repo.

vit-large zero-shot retrieval {"val_txt_r1": 91.42011834319527, "val_txt_r5": 97.534516765286, "val_txt_r10": 98.91518737672584, "val_txt_r_mean": 95.95660749506904, "val_img_r1": 79.30966469428007, "val_img_r5": 94.04339250493096, "val_img_r10": 96.44970414201184, "val_img_r_mean": 89.93425378040763, "val_r_mean": 92.94543063773833, "test_txt_r1": 89.9, "test_txt_r5": 98.8, "test_txt_r10": 99.7, "test_txt_r_mean": 96.13333333333333, "test_img_r1": 80.38, "test_img_r5": 94.88, "test_img_r10": 97.12, "test_img_r_mean": 90.79333333333334, "test_r_mean": 93.46333333333334}

vit-large finetune retrieval {"val_txt_r1": 85.99605522682445, "val_txt_r5": 97.33727810650888, "val_txt_r10": 98.22485207100591, "val_txt_r_mean": 93.85272846811307, "val_img_r1": 77.85009861932939, "val_img_r5": 93.68836291913215, "val_img_r10": 96.60749506903353, "val_img_r_mean": 89.38198553583169, "val_r_mean": 91.61735700197238, "test_txt_r1": 85.4, "test_txt_r5": 97.9, "test_txt_r10": 99.0, "test_txt_r_mean": 94.10000000000001, "test_img_r1": 77.72, "test_img_r5": 94.2, "test_img_r10": 96.88, "test_img_r_mean": 89.60000000000001, "test_r_mean": 91.85000000000001}