Hi,

I am a little confused that should the resolution of images in 2 folders be the same or different (folder1: 256x256, folder2: 1024x1024)? If not, can we use PIL to resize it or use torch transform resize?

Many thanks.

We follow the StyleGAN-ADA to extract FFHQ256 from tfrecord file. However, when I computed the statistic, it results in different statistics from your pre-computed trainval70K. I want to know whether the calculation steps have changed?

However, it is required when using fid.make_custom_stats(). I used pip.

Also, the README instructs one to pip install -r requirements.txt while such a file is not present.

My dataset has nonsquare samples, so on training, I'm doing random crop and for validation, I'm calculating fid with center cropped copy of dataset. Would be cool to have some option for passing my own transform, for example.

Hello, I tried using the package, but its throwing this runtime error. I checked image size mismatch, whether they are corrupted or not but no leads as to what is causing this.

Recently, I've come across a post on LinkedIn that describes how we should carefully choose the right resize function while stressing the fact that using different libraries/frameworks leads to different results. So, I decided to test it myself. Click here to find the post that I took the inspiration from.

The following is the code snippet that I've edited(using this colab notebook) to give the correct way of using resize methods in different frameworks.

import numpy as np
import torch
import torchvision.transforms.functional as F
from torchvision import transforms
from torchvision.transforms.functional import InterpolationMode
from PIL import Image
import tensorflow as tf
import cv2
import matplotlib.pyplot as plt
from skimage import draw

image = np.ones((128, 128), dtype=np.float64)
rr, cc = draw.circle_perimeter(64, 64, radius=45, shape=image.shape)
image[rr, cc] = 0
plt.imshow(image, cmap='gray')
print(f"Unique values of image: {np.unique(arr)}")
print(image.dtype)
output_size = 17
def inspect_img(*, img):
    plt.imshow(img, cmap='gray')
    print(f"Value of pixel with coordinates (14,9): {img[14, 9]}")

def resize_PIL(*, img, output_size):
    img = Image.fromarray(image)
    img = img.resize((output_size, output_size), resample=Image.BICUBIC)
    img = np.asarray(img,dtype=np.float64)
    inspect_img(img=img)
    return img
def resize_pytorch(*, img, output_size):
    img = F.resize(Image.fromarray(np.float64(img)), # Provide a PIL image rather than a Tensor.
                   size=output_size, 
                   interpolation=InterpolationMode.BICUBIC)
    img = np.asarray(img, dtype=np.float64) 
    inspect_img(img=img)
    return img
def resize_tensorflow(*, img, output_size):
    img = img[tf.newaxis, ..., tf.newaxis]
    img = tf.image.resize(img, size = [output_size] * 2, method="bicubic", antialias=True)
    img = img[0, ..., 0].numpy()
    inspect_img(img=img)
    return img
image_PIL = resize_PIL(img=image, output_size=output_size)
image_pytorch = resize_pytorch(img=image, output_size=output_size)
image_tensorflow = resize_tensorflow(img=image, output_size=output_size)
assert np.array_equal(image_PIL, image_pytorch) == True, 'Not Identical!'
# assert np.array_equal(image_PIL, image_tensorflow) == True, 'Not Identical!'  --> fails
assert np.allclose(image_PIL, image_tensorflow) == True, 'Not Close!'
# assert np.array_equal(image_tensorflow, image_pytorch) == True, 'Not Identical!'  --> fails 
assert np.allclose(image_tensorflow, image_pytorch) == True, 'Not Close!'
# tensorflow gives a slightly different values than pytorch and PIL.

which gives us the following results:

Therefore, TensorFlow, PyTorch, and PIL give similar results if the resize method is used properly like in the above snippet code.

You can read my comments on linkedin to find out how I came to this solution.

The only remaining library is OpenCV which I'll test in the future.

Have a great day/night!

I am working on vid2vid GAN model from Nvidia-Imaginaire library. It uses clean-fid library. While running the model on google colab. I encounter this error related to the clean-fid.

from imaginaire.evaluation import compute_fid File "/content/imaginaire/imaginaire/evaluation/init.py", line 5, in from .fid import compute_fid, compute_fid_data File "/content/imaginaire/imaginaire/evaluation/fid.py", line 10, in from imaginaire.evaluation.common import load_or_compute_activations File "/content/imaginaire/imaginaire/evaluation/common.py", line 14, in from cleanfid.resize import build_resizer File "/usr/local/lib/python3.7/dist-packages/cleanfid/resize.py", line 10, in from cleanfid.utils import * File "/usr/local/lib/python3.7/dist-packages/cleanfid/utils.py", line 5, in from cleanfid.resize import build_resizer ImportError: cannot import name 'build_resizer' from 'cleanfid.resize' (/usr/local/lib/python3.7/dist-packages/cleanfid/resize.py)

The Issue is it is not able import the build_resizer from cleanfid.resize. I don't have this issue like two days back now I am having this issue. Is it due to the new version.

Thanks in advance

Running the code on my windows machine, gave me the two following errors: the /tmp is different on windows and DataLoader crashes while using num_workers > 0. I made the /tmp folder general and added a condition for the num_workers argument.

Hi Gaurav, thanks for sharing this amazing tool! I spot a block of suspicious lines that might worth your attention. Specifically, this resizing function of the default "clean" resizer: https://github.com/GaParmar/clean-fid/blob/d2a10b1f4f44e79ea08717a10702fb1c674b1830/cleanfid/resize.py#L43-L52

It seems to me that the output_size in L47 (s1, s2) supposes to be a (w, h) tuple whilie L48 expects it as (h, w). What do you think? This might mean that the default resizer only works for square output resolution.

Shouldn't be a big problem since it does not affect default behavior.

Passing device="cpu" and model_name="clip_vit_b_32" crashes with the following error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument weight in method wrapper___slow_conv2d_forward)

Looks like the actual number of generations is rounded up to the nearest multiple of batch size:

https://github.com/GaParmar/clean-fid/blob/fca67180659eae81d2ea207b11e12acd84735171/cleanfid/fid.py#L199

for the Line 391 and 396 in compute_kid(), I think the 'None' should actually be 'feat_model' https://github.com/GaParmar/clean-fid/blob/55ec1683ce3b2615bdbee12cb611f6ea0dc6457f/cleanfid/fid.py#L391

Hi! Uppercase JPEG extension ignored by get_folder_features I've noticed this issue while making custom statistics from a folder with .JPEG files. It seems like it has been thought of here for processing .zip https://github.com/GaParmar/clean-fid/blob/b1d8934d7ebb7e0c471f7bdb4c12872fe62f6cc4/cleanfid/fid.py#L138 but not here for processing folders https://github.com/GaParmar/clean-fid/blob/b1d8934d7ebb7e0c471f7bdb4c12872fe62f6cc4/cleanfid/fid.py#L140-L141

Probably the easiest fix is to expand the EXTENSIONS with the upper-case versions

I am trying to compute KID, but it is generating negative values. Can KID be a negative number?

Here is the code that I used: ` from cleanfid import fid fdir1 = my_folder1_path fdir2 = my_folder2_path

kid_score = fid.compute_kid(fdir1, fdir2) `

Each folder has only 6 images. My kid_score is -0.0406.

Could someone please help me understand why the KID is less that zero?

Thank you, Chandrakanth

This code is from fid.py line 459-470.

elif gen is not None:
    if not verbose:
        print(f"compute FID of a model with {dataset_name}-{dataset_res} statistics")
    score = fid_model(gen, dataset_name, dataset_res, dataset_split,
            model=feat_model, z_dim=z_dim, num_gen=num_gen,
            mode=mode, num_workers=num_workers, batch_size=batch_size,
            device=device, verbose=verbose)
    return score

# compute fid for a generator, using images in fdir2
elif gen is not None and fdir2 is not None:

There is no way we enter the last elif, so I can't compare my generator with images in fdir2. Is this intentional?

Hi, thanks for sharing your work. In my case, I have a reference folder (fdir1), and about 50 target folders (fdir2), each containing hundreds of images. It takes a long time to calculate the score by default fid.compute_fid(fdir1, fdir2). It seems it is chocked by the CPU. Is there any way to speed up it?