Hi, I am trying to execute the code for image fitting problem. I am setting batch size =1 (default value) as I have 4gb GPU. Still the training stops due to GPU out of memory. Can anyone let me know how could I solve this problem? Thanks.

 python experiment_scripts/train_img.py --model_type=sine --experiment_name=output

SingleBVPNet(
  (image_downsampling): ImageDownsampling()
  (net): FCBlock(
    (net): MetaSequential(
      (0): MetaSequential(
        (0): BatchLinear(in_features=2, out_features=256, bias=True)
        (1): Sine()
      )
      (1): MetaSequential(
        (0): BatchLinear(in_features=256, out_features=256, bias=True)
        (1): Sine()
      )
      (2): MetaSequential(
        (0): BatchLinear(in_features=256, out_features=256, bias=True)
        (1): Sine()
      )
      (3): MetaSequential(
        (0): BatchLinear(in_features=256, out_features=256, bias=True)
        (1): Sine()
      )
      (4): MetaSequential(
        (0): BatchLinear(in_features=256, out_features=1, bias=True)
      )
    )
  )
)

  0%|                                                                          | 0/10000 [00:00<?, ?it/s]

Traceback (most recent call last):
  File "experiment_scripts/train_img.py", line 62, in <module>
    model_dir=root_path, loss_fn=loss_fn, summary_fn=summary_fn)
  File "/home/mz/code/siren/training.py", line 92, in train
    summary_fn(model, model_input, gt, model_output, writer, total_steps)
  File "/home/mz/code/siren/utils.py", line 334, in write_image_summary
    img_gradient = diff_operators.gradient(model_output['model_out'], model_output['model_in'])
  File "/home/mz/code/siren/diff_operators.py", line 42, in gradient
    grad = torch.autograd.grad(y, [x], grad_outputs=grad_outputs, create_graph=True)[0]
  File "/home/mz/anaconda3/envs/siren/lib/python3.6/site-packages/torch/autograd/__init__.py", line 158, in grad
    inputs, allow_unused)

RuntimeError: CUDA out opython experiment_scripts/train_img.py --model_type=sine --experiment_name=output1
SingleBVPNet(
  (image_downsampling): ImageDownsampling()
  (net): FCBlock(
    (net): MetaSequential(
      (0): MetaSequential(
        (0): BatchLinear(in_features=2, out_features=256, bias=True)
        (1): Sine()
      )
      (1): MetaSequential(
        (0): BatchLinear(in_features=256, out_features=256, bias=True)
        (1): Sine()
      )
      (2): MetaSequential(
        (0): BatchLinear(in_fef memory. Tried to allocate 256.00 MiB (GPU 0; 3.94 GiB total capacity; 2.76 GiB already allocated; 215.06 MiB free; 2.78 GiB reserved in total by PyTorch) (malloc at /opt/conda/conda-bld/pytorch_1587428091666/work/c10/cuda/CUDACachingAllocator.cpp:289)

I reimplemented Siren in TensorFlow 2.5. The network easily learns images, but I can not reproduce result with audio. On the sample file from the paper loss gets stuck at relatively high value (~0.0242), and network's output turns very quiet (max(abs(x)) ~= 0.012). Just curious if anyone has faced the same issue when reimplementing Siren on their own.

What I've tried so far:

1. doublechecked omega - it is set to 3000.0 (input), 30.0, 30.0, 30.0 (inner) layers
2. Changing batch size to full length of the sample (I used to do randomized batches of 8*1024)
3. Using float64 to avoid potential issues with numerical overflows/underflows
4. Checked network weights: all are finite numbers
5. Using SGD as a more stable optimizer
6. Increasing network width/adding more layers

Essentially, all the above actions still led to the same result with loss ~0.0242

When i try to train sdf, i found it will take a long time to call model.cuda() in file train_sdf.py. (just after printing the model) Also when testing sdf, self.model.load_state_dict(torch.load(opt.checkpoint_path)) need a lot of time.

My pytorch and cudatookit are installed according to file environment.yml. ( pytorch=1.5.0=py3.6_cuda10.1.243_cudnn7.6.3_0 )

How can i solve the issue? Is it normal to have such a problem?

I noticed in all the examples and looking at the code that images and video are equal in column and row counts (e.g., 512x512). I am considering arbitrary sized images (without resizing), how do I do that? thanks

Does it seem feasible to train the CNN+SIREN scheme on larger images? The given example (train_img_neural_process.py) uses 32x32 pixel images, and I haven't managed to generate visually pleasing results on a more useful size (eg 256x256px).

Either the image quality is very poor, or the memory blows up when I try increasing the number of parameters.

Another example (train_img.py) uses the same SIREN network to generate larger images (512x512) so the SIREN shouldn't be the problem, but its weight generation process is. It seems that the output of the CNN is reduced to only 256 features, first in ConvImgEncoder, which does something like

torch.nn.Linear(1024,1)(torch.rand([batch_size, 256, 32, 32]).view(4,256,-1)).squeeze(-1).shape                                                                                      
torch.Size([batch_size, 256])

Then the HyperParameter network (FCBlock) does the same according to the hidden layer's (hyper) hidden features.

This seems to be very little when we are generating weights for a network which has 256*256 weights per layer. I guess it would work on 32x32px but it makes sense that this would not scale up.

Do you have any insight as how to generate the SIREN weights from the output of a CNN without blowing up memory?

Is it not possible to scale the input linear space instead ?

Also, not at all an issue, but I'd like to share the following Mathematica code attempting to replicate this, or at least the part for fitting an image :

image = ImageResize[ExampleData[{"TestImage", "Lena"}], 128]
{width, height } = ImageDimensions[image];
output = Flatten[ImageData[image] // #*2 - 1 &, 1];
linspace[n_] := Array[# &, n, {-#, #}] &[(n - 1)/2]
input = Pi/2*Tuples[{linspace[height], linspace[width]}] // N;
layer[n_, in_] := LinearLayer[
    n,
    "Weights" -> RandomReal[{-#, #}, {n, in}],
    "Biases" -> RandomReal[{-#, #}, {n}],
    "Input" -> in
] &[Sqrt[6/in]]
net = NetChain[
  {
   128, Sin,
   layer[128, 128], Sin,
   layer[128, 128], Sin,
   layer[128, 128], Sin,
   layer[3, 128], Sin
  },
  "Input" -> 2
]
net = NetTrain[net, (input -> output)]
Image[Partition[(# + 1)/2 &[net /@ input], width], ColorSpace -> "RGB"]
Table[NetExtract[net, {i, "Weights"}] // Flatten // Histogram, {i, {1, 3, 5, 7, 9}}]
Table[NetExtract[net, {i, "Biases"}] // Flatten // Histogram, {i, {1, 3, 5, 7, 9}}]

It works surprisingly well, but I haven't used omega0, I scaled the input instead. Also performance is better when the weights in the first layer are not initialized as advised in the paper. Not sure if it's related with the input scaling or what.

Hello @vsitzmann ,

Thanks for your nice paper and implementations, Currently I try to use sine activation function on some implicit image restoration functions,

In here I have little question about w0 implementation

1. Multiply of w0

In paper w0=30 in initial layer is represented with y = sin(w0*wx+b) in last sentence of paragraph 3.2 But in Implementation,

MetaSequential(BatchLinear(in_features, hidden_features), nl)) BatchLinear(in_features, hidden_features) for (wx+b) and nl make y = sin(w0*(wx+b))

I check that performance goes well in your distributed experiments, but is there any problems to use w0 multiply on bias? It just make smaller bias only?

2. Initialize of w

In paper paragraph 3.2 and supplement 1.5

Initialize of w should goes with -sqrt(6/n), sqrt(6/n) in paragraph 3.2 or -sqrt(6/n)/w0, sqrt(6/n)/w0when use with w0 value,

but In distributed source,
first layer use w0=30 but it's initialization of w is -1/n , 1/n where can I find the reason for this first layer initialization?

In section 4.4 of your paper you go into an interesting hypernetwork idea that can generate the siren parameters for a space of "functions" (images in that case).

In section 9 of the appendix, you go more into details, and I specifically care about the part where you predict the siren parameters from the input:

As far as I understand

the weights of a 5-layer SIREN with hidden features of size 256

Are:

W1 ∈ R(2, 256) # maps x,y
b1 ∈ R(256)

W2,W3,W4 ∈ R(256,256)
b2,b3,b4 ∈ R(256)

W5 ∈ R(256,3) # maps to rgb
b5 ∈ R(3)

Total params: 2*256 + 256 + 3*(256*256 + 256) + 256*3 + 3 =198,915

So, do I understand correctly that your hypernetwork takes the input from the convnet, input, and does the following:

h1 = relu(U1*input + c1)
h2 = U2*h1 + c2

Where

U1 ∈ R(|input|, 256)
c1 ∈  R(256)

U2 ∈ R(256, 198915) 
c2 ∈ R(198915) 

This doesn't feel right to me.

When I run the code python experiment_scripts/train_img.py --model_type=sine given in README.md, I get the error

usage: train_img.py [-h] [-c CONFIG_FILEPATH] [--logging_root LOGGING_ROOT]
                    --experiment_name EXPERIMENT_NAME
                    [--batch_size BATCH_SIZE] [--lr LR]
                    [--num_epochs NUM_EPOCHS]
                    [--epochs_til_ckpt EPOCHS_TIL_CKPT]
                    [--steps_til_summary STEPS_TIL_SUMMARY]
                    [--model_type MODEL_TYPE]
                    [--checkpoint_path CHECKPOINT_PATH]
train_img.py: error: the following arguments are required: --experiment_name

Hi, thanks for this great work. I recently adopted SIREN in my problem, but during training the loss becomes divergent. I've set the omega to 30 by default. Could you please give suggestion about this. Thanks!

from modules.py at line62, the sine_init is

m.weight.uniform_(-np.sqrt(6 / num_input) / 30, np.sqrt(6 / num_input) / 30)

but I think is:

m.weight.uniform_(-np.sqrt(6 / num_input) * 30, np.sqrt(6 / num_input) * 30)

follow is my test code with 101 sin layers, the std of outputs is stability equals 0.7 when * 30， and when I replace * factor with / factor, the std of output reduced to 0.

import torch
import numpy as np

dim = 1000
factor = 30
alpha = 1
x = torch.Tensor(np.zeros(dim, dtype=np.float32)).reshape([dim, 1])
w = torch.Tensor(np.zeros([dim,dim], dtype=np.float32))

inputs = torch.nn.init.normal_(x, 0, 1)
weights = torch.nn.init.uniform_(w, -alpha/dim, alpha/dim)

outputs = torch.sin(alpha*factor * weights@inputs)
print(torch.mean(outputs), torch.std(outputs))

weights = torch.nn.init.uniform_(w, -np.sqrt(6 / dim) * factor / alpha, np.sqrt(6 / dim) * factor / alpha)

for _ in range(100):
    outputs = torch.sin(alpha*weights@outputs)
    print(torch.mean(outputs), torch.std(outputs))

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

In paper, it is written sin(\omega_0 W x + b).

But in implementation, the explore_siren notebook as well as modules.py, the output of linear layer is multiplied to \omega_0. In other words, sin(\omega_0 (Wx+b)).

I find this difference drastically changes the network convergence behavior. The actual implemented network performs much better.

Could you clarify this issue?

In Colab notebook implementation of Class Siren foward def forward(self, coords): coords = coords.clone().detach().requires_grad_(True) # allows to take derivative w.r.t. input output = self.net(coords) return output, coords

why are coords set require_grad_(True)? Shouldn't the weights of the network only updated, not coordinates?

Hi,

how do you usually initialize the frequency of the network (leaving aside the magic number 30)? As it is very much problem-dependent, I'm wondering if it is currently just a trial-and-error task or if there is a more reasonable approach, even if just for finding general scale (I saw you mentioned an value of 3000 for audio data in another issue).

Thanks already in advance, and also for this nice code-base.

In Section 4 of supplementary material, it is mentioned that points with their normals are collected. Would you please explain which norm is calculated (Euclidean, etc)? Also, in which coordination system ( real-world or voxel coordination) the points are collected?

To compare SIREN layer with RELU +MLP, we implement two models.

1. audio signal (B, T, 1) --> Linear(B, T, 128) +RELU --> Linear(B, T, 128) +RELU --> Linear(B, T, 128) +RELU --> Linear(B, T, 1) --> reproduced signal (B, T, 1)

2. audio signal (B, T, 1) -->SIREN layer --> SIREN layer -->SIREN layer --> Linear(B, T, 1) --> reproduced signal (B, T, 1)

In your paper, RELU+MLP is not able to reproduce the audio signal, However, First model can reproduce audio signal even better than SIREN...

SIREN is also very instability so i used lower learning rate. But the loss fluctuated.

Could you explain why the siren is better than others in audio reproduction domain?