OpenAI's CLIP — REST API

This is a container wrapping OpenAI's CLIP model in a RESTful interface.

Running the container locally

First, build the container:

docker build -t clip-container:latest .

Then, you can run it:

docker run -it -p 8080:8080 --name "clip-container" --rm clip-container:latest /opt/ml/code/serve

Sending requests:

The container exposes two different endpoints:

GET /ping: Returns 200 status if the container is working properly.
POST /invocations: Processes a list of images and returns the list of labels with their corresponding probabilities.

Here is an example request assuming the container is listening in port 8080:

curl --location --request POST 'http://localhost:8080/invocations' \
--header 'Content-Type: application/json' \
--data-raw '{
    "images": [
        "https://images.unsplash.com/photo-1597308680537-1ba44407ffc0?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1834&q=80",
        "https://images.unsplash.com/photo-1589270216117-7972b3082c7d?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1834&q=80"],
    "classes": ["person", "bag", "person with a bag", "woman riding a horse", "woman with a bag", "woman with black shirt and a bag"]
}'

The response looks like this:

[
    {
        "url": "https://images.unsplash.com/photo-1597308680537-1ba44407ffc0?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1834&q=80", 
        "labels": [
            "woman with black shirt and a bag", 
            "woman with a bag", 
            "person with a bag", 
            "bag", "person"
        ], 
        "probs": [1.0, 1.7488513970320696e-09, 1.1663764917350243e-19, 4.179975909038141e-30, 3.77612043676229e-30]
    }, 
    {
        "url": "https://images.unsplash.com/photo-1589270216117-7972b3082c7d?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1834&q=80", 
        "labels": [
            "person with a bag", 
            "woman with black shirt and a bag", 
            "bag", 
            "woman with a bag", 
            "person"
        ], 
        "probs": [1.0, 2.4879632576357835e-08, 2.065714813830402e-13, 7.658033346455602e-15, 1.1307645811408335e-23]
    }
]

SageMaker Integration

This container is compatible with SageMaker so you should be able to host it as a SageMaker endpoint with no modifications. The code supports GPU and CPU instances.

Hi svpino,

this happens when running the container:

root@megacool:/tmp/clip-container# docker run -it -p 8080:10000 --name "clip-container" --rm clip-container:latest /opt/ml/code/serve Starting the inference server with 16 workers. [2022-07-08 07:31:12 +0000] [10] [INFO] Starting gunicorn 20.1.0 [2022-07-08 07:31:12 +0000] [10] [INFO] Listening at: unix:/tmp/gunicorn.sock (10) [2022-07-08 07:31:12 +0000] [10] [INFO] Using worker: gevent [2022-07-08 07:31:12 +0000] [13] [INFO] Booting worker with pid: 13 [2022-07-08 07:31:12 +0000] [14] [INFO] Booting worker with pid: 14 [2022-07-08 07:31:12 +0000] [15] [INFO] Booting worker with pid: 15 [2022-07-08 07:31:12 +0000] [16] [INFO] Booting worker with pid: 16 [2022-07-08 07:31:12 +0000] [17] [INFO] Booting worker with pid: 17 [2022-07-08 07:31:12 +0000] [18] [INFO] Booting worker with pid: 18 [2022-07-08 07:31:12 +0000] [19] [INFO] Booting worker with pid: 19 [2022-07-08 07:31:12 +0000] [20] [INFO] Booting worker with pid: 20 [2022-07-08 07:31:12 +0000] [51] [INFO] Booting worker with pid: 51 [2022-07-08 07:31:12 +0000] [52] [INFO] Booting worker with pid: 52 [2022-07-08 07:31:12 +0000] [68] [INFO] Booting worker with pid: 68 [2022-07-08 07:31:12 +0000] [69] [INFO] Booting worker with pid: 69 [2022-07-08 07:31:12 +0000] [70] [INFO] Booting worker with pid: 70 [2022-07-08 07:31:12 +0000] [71] [INFO] Booting worker with pid: 71 [2022-07-08 07:31:12 +0000] [72] [INFO] Booting worker with pid: 72 [2022-07-08 07:31:12 +0000] [88] [INFO] Booting worker with pid: 88 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 0%| | 0.00/338M [00:00<?, ?iB/s]/root/.local/lib/python3.8/site-packages/clip/clip.py:57: UserWarning: /root/.cache/clip/ViT-B-32.pt exists, but the SHA256 checksum does not match; re-downloading the file warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file") 0%| | 0.00/338M [00:00<?, ?iB/s]/root/.local/lib/python3.8/site-packages/clip/clip.py:57: UserWarning: /root/.cache/clip/ViT-B-32.pt exists, but the SHA256 checksum does not match; re-downloading the file warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file") 15%|█████▋ | 50.2M/338M [00:18<01:42, 2.95MiB/s]

This file gets redownloaded all the time resulting in a loop.

I hope you can help. :)

Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

458 Jan 2, 2023

Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.

2.3k Jan 4, 2023

Loop when running the container

Hi svpino,

this happens when running the container:

root@megacool:/tmp/clip-container# docker run -it -p 8080:10000 --name "clip-container" --rm clip-container:latest /opt/ml/code/serve Starting the inference server with 16 workers. [2022-07-08 07:31:12 +0000] [10] [INFO] Starting gunicorn 20.1.0 [2022-07-08 07:31:12 +0000] [10] [INFO] Listening at: unix:/tmp/gunicorn.sock (10) [2022-07-08 07:31:12 +0000] [10] [INFO] Using worker: gevent [2022-07-08 07:31:12 +0000] [13] [INFO] Booting worker with pid: 13 [2022-07-08 07:31:12 +0000] [14] [INFO] Booting worker with pid: 14 [2022-07-08 07:31:12 +0000] [15] [INFO] Booting worker with pid: 15 [2022-07-08 07:31:12 +0000] [16] [INFO] Booting worker with pid: 16 [2022-07-08 07:31:12 +0000] [17] [INFO] Booting worker with pid: 17 [2022-07-08 07:31:12 +0000] [18] [INFO] Booting worker with pid: 18 [2022-07-08 07:31:12 +0000] [19] [INFO] Booting worker with pid: 19 [2022-07-08 07:31:12 +0000] [20] [INFO] Booting worker with pid: 20 [2022-07-08 07:31:12 +0000] [51] [INFO] Booting worker with pid: 51 [2022-07-08 07:31:12 +0000] [52] [INFO] Booting worker with pid: 52 [2022-07-08 07:31:12 +0000] [68] [INFO] Booting worker with pid: 68 [2022-07-08 07:31:12 +0000] [69] [INFO] Booting worker with pid: 69 [2022-07-08 07:31:12 +0000] [70] [INFO] Booting worker with pid: 70 [2022-07-08 07:31:12 +0000] [71] [INFO] Booting worker with pid: 71 [2022-07-08 07:31:12 +0000] [72] [INFO] Booting worker with pid: 72 [2022-07-08 07:31:12 +0000] [88] [INFO] Booting worker with pid: 88 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 /root/.local/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.) return torch._C._cuda_getDeviceCount() > 0 0%| | 0.00/338M [00:00<?, ?iB/s]/root/.local/lib/python3.8/site-packages/clip/clip.py:57: UserWarning: /root/.cache/clip/ViT-B-32.pt exists, but the SHA256 checksum does not match; re-downloading the file warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file") 0%| | 0.00/338M [00:00<?, ?iB/s]/root/.local/lib/python3.8/site-packages/clip/clip.py:57: UserWarning: /root/.cache/clip/ViT-B-32.pt exists, but the SHA256 checksum does not match; re-downloading the file warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file") 15%|█████▋ | 50.2M/338M [00:18<01:42, 2.95MiB/s]

This file gets redownloaded all the time resulting in a loop.

I hope you can help. :)

opened by KarstenVogt 0

A containerized REST API around OpenAI's CLIP model.

Related tags

Overview

OpenAI's CLIP — REST API

Running the container locally

Sending requests:

SageMaker Integration

You might also like...

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

CLIP+FFT text-to-image

Navigating StyleGAN2 w latent space using CLIP

RANZCR-CLiP 7th Place Solution

A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.

A PyTorch Lightning solution to training OpenAI's CLIP from scratch.

CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)

Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.

Comments

Loop when running the container

Owner

Santiago Valdarrama

CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

BisQue is a web-based platform designed to provide researchers with organizational and quantitative analysis tools for 5D image data. Users can extend BisQue by implementing containerized ML workflows.

Run containerized, rootless applications with podman

Simple implementation of OpenAI CLIP model in PyTorch.

Feed forward VQGAN-CLIP model, where the goal is to eliminate the need for optimizing the latent space of VQGAN for each input prompt

Simple image captioning model - CLIP prefix captioning.

CLIPImageClassifier wraps clip image model from transformers

Working demo of the Multi-class and Anomaly classification model using the CLIP feature space

A simple rest api serving a deep learning model that classifies human gender based on their faces. (vgg16 transfare learning)

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)