Hi, would you be interested in sharing a web demo on Huggingface Spaces for lang-seg?

It would make this model more accessible as it would allow people to try out the model directly from the browser. Some other recent machine learning model repos have set up Spaces for easy access:

github: https://github.com/salesforce/BLIP Spaces: https://huggingface.co/spaces/akhaliq/BLIP

github: https://github.com/facebookresearch/omnivore Spaces: https://huggingface.co/spaces/akhaliq/omnivore

Spaces is completely free, and I can help setup a Gradio Space. Here are some getting started instructions if you'd prefer to do it yourself: https://huggingface.co/blog/gradio-spaces

Hi,

Thanks for open-sourcing such great work. I have some questions when using this code:

1. Does the test_lseg.py script support multi-GPU inference? When using a single GPU, it takes about 2~3 hours for inference on ade20k.
2. I tried to evaluate the provided demo_e200.ckpt on ade20k and got (pixAcc: 0.8078, mIoU: 0.3207), is that correct? It seems lower than the values in the paper.
3. I trained a model on ade20k (the same config as train.sh, backbone is vit_l16_384) with 8*V100 but found it needs ~90 hours for training 240 epochs. Is it reasonable (it seems much longer than you said in #7)?
4. When I use this code for other datasets like cityscapes, what changes should I make? The only difference I found is get_labels()in lseg_module.py. Have you evaluated the mIoU on cityscapes?

Thanks in advance.

Hi, I have a question on the difference of settings between the demo in your README and the experiment in your paper.

In the README, you published the pre-trained weight for demo. It says while training the backbones for both image and text are ViT-L/16. The section 5.1 in your paper says

We used LSeg with DPT and a smaller ViT-B/32 backbone together with the CLIP ViT-B/32 text encoder ...

When reproducing your results in 5.1, does that require a full-scratch training with ViT-B/32 backbone for the images? Also, are there any other differences, such as batch size? More specifically, How do I change the arguments in train.sh ?

Finally, is it possible to share with us (or me) the weight used for your results?

Thank you in advance.

Congrats on your paper accepted to ICLR 2022!

Do you have your pretrained models on 4 folds of Pascal-5i and COCO-20i? Can you share them?

I really appreciate your response.

Hi! Thanks for the great work

I can't seem to import encoding after following the installation steps. The error I got is "cannot import name 'gpu' from partially initialized module 'encoding' (most likely due to a circular import)". Can you please let me know whether you know the cause of this issue? Thanks!

Hi, I would like to tried your code, but the error shows up when I tried to install the pytorch-encoding. Could you give us your environment info of CUDA GPU python g++ and so on？Do you have any advice about installing it，which I see a lot of people get the error.

My env: OS: Ubuntu 18.04 gcc: 7.5.0 GPU:3090 driver：515 CUDA: tried 11.7 and 10.2 pytorch: tried 1.12 and 1.7.1

This is a question on an interesting report in the paper. The paper reported

We also evaluated on a model initialized with the CLIP image encoder with the same setup and hyperparameters, but observed worse performance than using the ViT initialization.

It seems surprising that CLIP image encoder, which is already well-aligned to the text encoder, is not helpful for the task. Do authors have any guesses about the reason? And, was the performance much worse or a little worse?

Hello,

This is great work @Boyiliee ! I'm excited to try this out.

I have a quick question: what kind of system requirements are necessary to train and run inference on this model? Specifically I am wondering about the type of GPU(s) needed to train LSeg.

Hi Thanks for your great work! When I tried to add LSegNet into my own framework, there was a RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. My train function is(run on ADE20K):

def train(self, cur_epoch, optim, train_loader, scheduler=None, print_int=10, logger=None):
        device = self.device
        model = self.model
        criterion = nn.CrossEntropyLoss(ignore_index=-1)
        model.train()
        for cur_step, (images, labels) in enumerate(train_loader):
            images = images.to(device, dtype=torch.float32)
            labels = labels.to(device, dtype=torch.long)
            optim.zero_grad()
            outputs = model(images, labelset='')
            loss = criterion(outputs, labels)
            self.scaler.scale(loss)
            loss.backward()
            optim.step()
            if scheduler is not None:
                scheduler.step()

The model is LSegNet and I didn't modify lseg_net.py. I think maybe some optimizations have been made by Pytorch-lighting. Could you give me some suggestions? Thank you!

Hi! Thanks for your interesting work! I am trying to reproduce the zero-shot experiments in the paper recently, but like https://github.com/isl-org/lang-seg/issues/19#issue-1213501618 , it gets mIoU much lower than yours.

Here is my scripts:

train_lseg_zs.py:

from modules.lseg_module_zs import LSegModuleZS
from utils import do_training, get_default_argument_parser

if __name__ == "__main__":
    parser = LSegModuleZS.add_model_specific_args(get_default_argument_parser())
    args = parser.parse_args()
    do_training(args, LSegModuleZS)

command:

python -u train_lseg_zs.py --backbone clip_resnet101 --exp_name lsegzs_pascal_f0 --dataset pascal \
--widehead --no-scaleinv --arch_option 0 --ignore_index 255 --fold 0 --nshot 0 --batch_size 8 \

Default aruguments: base_lr=0.004, weight_decay=1e-4, momentum=0.9

I wonder where the problem is. And could you please share your training scripts for the zero-shot experiment?

Hi, I have a reproduction issue for Table 5. In the paper, the LSeg with ViT-B/32 backbone achieves 79.7 pixAcc and 37.8 mIoU. However, I only get 78.9 pixAcc and 33.7 mIoU by using the released code. The reproduced pixAcc/mIoU are not as expected.

Our reproduction command is as follows on 8 GPU cards.

python -u train_lseg.py --dataset ade20k --data_path datasets --batch_size 4 --exp_name lseg_ade20k_b32_240e --base_lr 0.004 --weight_decay 1e-4 --no-scaleinv --max_epochs 240 --widehead --accumulate_grad_batches 2 --backbone clip_vitb32_384

So what is the reason of the performance gap? I may miss some detail settings.

By the way, I encounter a warning when running the released code. [W reducer.cpp:283] Warning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance. grad.sizes() = [512, 768, 1, 1], strides() = [768, 1, 768, 768]

Have ever met this warning? May the performance gap caused by this warning?

Look forward to your reply.