Thank you for releasing the great code! I enjoyed the notebook a lot.

By the way, can we train a model from scratch or fine-tuning? Although I read experiment_setup.py, I didn't find how to run the training directly.

So when I run the code I get a mask that seems to be made of 32x32 large square on a whole 512x512 pixels image.

Is there a way to increase the resolution of the prediction and thus the mask generated?

Thanks a lot in advance.

Hi,

Thanks for this awesome work. As I really liked the approach of adapting CLIP for zero and one-shot image segmentation, I implemented your model as a branch of 🤗 Transformers.

The model is soon going to be added to the main library (see https://github.com/huggingface/transformers/pull/20066). Here's a Colab notebook to showcase usage: https://colab.research.google.com/drive/1ijnW67ac6bMnda4D_XkdUbZfilyVZDOh?usp=sharing.

Would you like to create an organization on the hub, under which the checkpoints can be hosted?

Currently I host them under my own username: https://huggingface.co/models?other=clipseg.

Thanks!

Niels, ML Engineer @ HF

When I run: "! git clone https://github.com/timojl/clipseg" in google colab, I get this error: Error downloading object: weights/rd16-uni.pth (61545cd): Smudge error: Error downloading weights/rd16-uni.pth (61545cdb3a28f99d33d457c64a9721ade835a9dfbda604c459de6831c504167a): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Here is the FULL ERROR: Cloning into 'clipseg'... remote: Enumerating objects: 168, done. remote: Counting objects: 100% (77/77), done. remote: Compressing objects: 100% (61/61), done. remote: Total 168 (delta 36), reused 39 (delta 16), pack-reused 91 Receiving objects: 100% (168/168), 1.21 MiB | 4.54 MiB/s, done. Resolving deltas: 100% (77/77), done. Downloading weights/rd16-uni.pth (1.1 MB) Error downloading object: weights/rd16-uni.pth (61545cd): Smudge error: Error downloading weights/rd16-uni.pth (61545cdb3a28f99d33d457c64a9721ade835a9dfbda604c459de6831c504167a): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Errors logged to /content/clipseg/clipseg/clipseg/.git/lfs/objects/logs/20220922T222731.926515854.log Use git lfs logs last to view the log. error: external filter 'git-lfs filter-process' failed fatal: weights/rd16-uni.pth: smudge filter lfs failed warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry the checkout with 'git checkout -f HEAD'

I was running the Quickstart notebook in my local, but I was getting the following error.

---> model.load_state_dict(torch.load('weights/rd16-uni.pth', map_location=torch.device('cpu')), strict=False)

File /scratch/miniconda3/envs/clipseg-environment/lib/python3.10/site-packages/torch/serialization.py:713, in load(f, map_location, pickle_module, **pickle_load_args)
    711             return torch.jit.load(opened_file)
    712         return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--> 713 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File /scratch/miniconda3/envs/clipseg-environment/lib/python3.10/site-packages/torch/serialization.py:920, in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    914 if not hasattr(f, 'readinto') and (3, 8, 0) <= sys.version_info < (3, 8, 2):
    915     raise RuntimeError(
    916         "torch.load does not work with file-like objects that do not implement readinto on Python 3.8.0 and 3.8.1. "
    917         f"Received object of type \"{type(f)}\". Please update to Python 3.8.2 or newer to restore this "
    918         "functionality.")
--> 920 magic_number = pickle_module.load(f, **pickle_load_args)
    921 if magic_number != MAGIC_NUMBER:
    922     raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: invalid load key, 'v'.

In pfe_dataset.py line 5 : from datasets.lvis_oneshot3 import blend_image_segmentation I can not find the lvis_oneshot3.py file, Thanks! Beside, how can I run the zero-shot experiment?

Hi, great job! I realy love this CLIP based segment. I am not so familiar with the code right now, I am just wondering how can I use a image as a prompt ?

On line 21 (models/clipseg.py) , I got an import error. from models.clip_prompts import imagenet_templates In fact , there has no file or function called clip_prompts in the models directory. Thank you very much!

When I was running the code, something wrong happened in models/clipseg.py (line 10 and line 17). Please correct the file import error. Thank you for your excellent work!

Trying to get the same output consitently for video frame-by-frame use. Can't find any sort of paramater/argument/function to call to set a custom seed...

Thanks for the awesome project! I needed to create a setup.py so that downstream projects can automatically install this package. Sharing a PR in case helpful. Also added .gitignore because pip puts the egg in the directory.