@HYPJUDY I am unable to setup the requirements for the project using the "requirements.txt" on my Windows machine as it produces the following error:

I did find a way around this using PyTorch previous versions:

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

There is additionally an issue with conflicting tensorboard versions. I commented out the tensorboard in the "requirements.txt" for now:

Also I believe the yacs and apex libraries are required, as they are also not in the requirements and produce errors when running the sample_images.py file.

Hello, I'm interested in your work!

I want to know what should i do if i want to infer the model by using my own datasets.

Should i build the file like dataset_coco.json, and re-extract cluster and grid features?

Thank you!

In train.py(103-110)

103: # for image
104: _visual_mask = torch.zeros((batch_size, visual_token_num), dtype=torch.float32, device=device)
105: # need to mask token content in selected_idx for prediction/generation
106: num_masks = random.randint(max(1, int(0.1 * visual_token_num)), visual_token_num)
107: selected_idx = random.sample(range(visual_token_num), num_masks)
108: _visual_mask[:, selected_idx] = 1
109: mask_position = (_visual_mask == 1).to(torch.long).view(-1)
110: mask_position = mask_position.nonzero().squeeze()

I think '_visual_mask = 1' means the model can see it, '_visual_mask = 0' is the opposite. The above codes randomly sample mask position, which selects which grid(8*8) the model can see(_visual_mask=1). The position that really needs to be masked is the position where the _visual_mask is equal to 0. So the code on line 109 should be changed to mask_position = (_visual_mask == 0).to(torch.long).view(-1) is this right?