Dear authors,'

Thanks for this awesome work!

1. Could you please let me know how you compute the T60 mean, and std in Table 3? Does it correspond to the 'test.py' script and take 1957 errors in %=(estimate-real)/real), and directly compute the mean and std?
2. I guess the pretrained model corresponds to the checkpoint under `resources', however, when I compute the T60 metric in point 1, it replies mean=60， std=253, which is different from the paper?

Thanks for your help!

Dear authors,

For the dataset description, you said "We curated a dataset of 265 different spaces totalling 1169 images and 738 IRs. From these, we produced a total of 11234 paired examples with a train- validation-test split of 9743-154-1957."

So, for the 11234 paired examples, did you duplicate the IRs i.e., some IRs in the provided dataset are exactly the same? And how do you choose which IR to duplicate?

Dear authors,

Could you please let me know how you select the best model? From the validation set? or report the best test results in the paper?

BTW, have you tried to remove the adversarial scheme to see the results with only a generator?

Dear authors,

Could you please let me know in the paper, how could you deal with the sound location? You assume it co-located with the microphone? Because RIR is related with the microphone-sound path, i didn;t find the information in the paper. Do you randomly locate a sound?

Newest PyTorch seems to have FFT routines that support complex tensors. Also added the finite-difference instantaneous angular frequency representation from GANSynth, etc.

Hi, I was trying to run your very interesting model on the input images from the project page, but the generated IRs are always kind of the same and do not resemble your output examples.

This is the code I used:

import numpy as np
import torch
from PIL import Image
import torchvision.transforms as transforms

from image2reverb.model import Image2Reverb
from image2reverb.stft import STFT

checkpoint_path = "./checkpoints/model.ckpt"
encoder_path = "./checkpoints/resnet50_places365.pth.tar"
depthmodel_path = "./checkpoints/mono_640x192"
constant_depth = None
latent_dimension = 512

model = Image2Reverb(encoder_path, depthmodel_path, constant_depth=constant_depth, spec="stft")
m = torch.load(checkpoint_path, map_location=model.device)
model.load_state_dict(m["state_dict"])

image_transforms = transforms.Compose([
    transforms.Resize([224, 224], transforms.functional.InterpolationMode.BICUBIC),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])

image_path = "examples/input.5416407f.png"

image = Image.open(image_path).convert("RGB")
label = image_transforms(image).unsqueeze(0)

with torch.no_grad():
    f, img = model.enc.forward(label)

    shape = (
        f.shape[0],
        (latent_dimension - f.shape[1]) if f.shape[1] < latent_dimension else f.shape[1],
        f.shape[2],
        f.shape[3]
    )
    z = torch.cat((f, torch.randn(shape, device=model.device)), 1)

    spec = model.g(z)

stft = STFT()
ir = stft.inverse(spec.squeeze())

import soundfile as sf
sf.write("ir.wav", ir, 22050)

It's possible I'm doing something wrong or am missing a step. Any ideas? The generated IR always seems to be the same kind of exponentially decaying noise burst, regardless of what the input image looks like.

Dear authors,

Thanks for the open sourced code.

Do you select the anechonic signals and AIRs from the OpenAIR dataset? Or some samples are recorded by yourself?

I am investigating other possible acoustic dataset with AIR from different environments, do you have something in mind?

Thanks in advance for your help! Best, Xinyuan

Hi Nikhil,

Thanks for open-sourcing the code!

I have a question regarding the differentiability of the proxy T_60 error. Firstly, in the in compare_t60 in image2reverb.util, detach is being called on both predictions and ground truths, which would make any further computation non-differentiable. Secondly, inside estimate_t60, which is being called from compare_t60, there exists a bunch of torch.where operations, which are again non-differentiable. So it looks like the proxy T_60 error won't be differentiable and upon adding it to the other losses, shouldn't really change the model training as the model won't have any control over it.

However, in Sec. 4.2 (Ablation Study) of the paper, it looks like not having the proxy T_60 error term in the objective is creating a difference. Am I missing something here?

Best, Sagnik

Hi Nikhil,

Thank you for sharing the code! However, I can't find any information about downloading the data. Could you share more details of that?

Thanks, Changan