Hi, this is really an excellent paper and code! I can run it and I found that it takes about 1 hour (time=4258.89s) to train 500 iterations on a single GPU with num_res_blocks=5, n_batch_train=256.

In the paper, you said you train CIFAR-10 240k iterations(final version) or 50k (under review version). So I wonder the total training time you need to finish the training for CIFAR10. Thanks so much!

In the following code for computing the (unnormalized) log probability, the network output is divided by b0.

https://github.com/ruiqigao/recovery_likelihood/blob/c77cc0511dedcb8d9ab928438d80acb62aeca96f/model.py#L154

I wonder if there is a legitimate explanation for this division.

b0 is supposed to be step_size_square, which usually has a very small value. https://github.com/ruiqigao/recovery_likelihood/blob/c77cc0511dedcb8d9ab928438d80acb62aeca96f/model.py#L184

I wonder if dividing by this b0 makes the gradient too large and harms the training in some settings.

Hello! I tried to train the model with multi GPUs. And I found that you have released train_distributed.py So I tried to use tf.distribute.MirroredStrategy() as strategy to achieve distributed training. But I got an error as follows:

 RuntimeError: `merge_call` called while defining a new graph or a tf.function. This can often happen if the function `fn` passed to `strategy.experimental_run()` is decorated with `@tf.function` (or contains a nested `@tf.function`), and `fn` contains a synchronization point, such as aggregating gradients. This behavior is not yet supported. Instead, please wrap the entire call `strategy.experimental_run(fn)` in a `@tf.function`, and avoid nested `tf.function`s that may potentially cross a synchronization boundary.

Looking forward to your help!!

I met an error when I tried to eval the model. I don't know what I should do

ValueError: in user code: /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:1947 split ** axis=axis, num_split=num_or_size_splits, value=value, name=name) /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py:9723 split "Split", split_dim=axis, value=value, num_split=num_split, name=name) /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:744 _apply_op_helper attrs=attr_protos, op_def=op_def) /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py:595 _create_op_internal compute_device) /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:3327 _create_op_internal op_def=op_def) /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1817 init control_input_ops, op_def) /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1657 _create_c_op raise ValueError(str(e))

ValueError: Dimension size must be evenly divisible by 3 but is 64
    Number of ways to split should evenly divide the split dimension for '{{node split}} = Split[T=DT_UINT8, num_split=3](split/split_dim, input_tensor)' with input shapes: [], [64,32,32,3] and with computed input tensors: input[0] = <0>.

Hi,

Thanks a lot for your amazing work. I'm recently reproducing your work, but I find that the code implementation for calculating the NLL metric (i.e., Table 4 in the paper) is missing. Will you release the code for evaluating the NLL metric? Or could you provide a pseudo code for reference?

Thanks.

Hello, I have cloned your code and run it with the default settings from config.py. However, it still converges with high negative contrastive losses and nan grads. Here present the log.

output.log

Logging ........ 2021-07-16 20:11:02,055 : gpus=0 2021-07-16 20:11:02,056 : {'logtostderr': False, 'alsologtostderr': False, 'log_dir': '', 'v': 0, 'verbosity': 0, 'logger_levels': {}, 'stderrthreshold': 'fatal', 'showprefixforinfo': True, 'run_with_pdb': False, 'pdb_post_mortem': False, 'pdb': None, 'run_with_profiling': False, 'profile_file': None, 'use_cprofile_for_profiling': True, 'only_check_args': False, 'runtime_oom_exit': True, 'op_conversion_fallback_to_while_loop': False, 'test_srcdir': '', 'test_tmpdir': '/tmp/absl_testing', 'test_random_seed': 301, 'test_randomize_ordering_seed': '', 'xml_output_file': '', 'jobid': 0, 'logdir': '', 'eager': False, 'ckpt_load': None, 'device': '0', 'tpu': False, 'tpu_name': None, 'tpu_zone': None, 'rnd_seed': 1, 'problem': 'cifar10', 'n_batch_train': 64, 'lr': 0.0001, 'beta_1': 0.9, 'n_iters': 1000000, 'grad_clip': False, 'warmup': 1000, 'n_batch_per_iter': 1, 'cosine_decay': False, 'opt': 'adam', 'eval': False, 'include_xpred_freq': 1, 'eval_fid': False, 'fid_n_samples': 64, 'fid_n_iters': 40000, 'fid_n_batch': 64, 'num_res_blocks': 8, 'num_diffusion_timesteps': 6, 'randflip': True, 'dropout': 0.0, 'normalize': None, 'act': 'lrelu', 'final_act': 'relu', 'use_attention': False, 'resamp_with_conv': False, 'spec_norm': True, 'res_conv_shortcut': True, 'res_use_scale': True, 'ma_decay': 0.999, 'noise_scale': 1.0, 'mcmc_num_steps': 30, 'mcmc_step_size_b_square': 0.0002, 'tfhub_cache_dir': None, 'tfhub_model_load_format': 'AUTO', '?': False, 'help': False, 'helpshort': False, 'helpfull': False, 'helpxml': False, 'output': './output/main/2021-07-16-20-10-55--num_res_blocks=8--n_batch_train=64'} 2021-07-16 20:11:02,238 : output dir ./output/main/2021-07-16-20-10-55--num_res_blocks=8--n_batch_train=64 2021-07-16 20:40:31,668 : ========== begin training ========= 2021-07-16 20:42:38,251 : dir=2021-07-16-20-10-55--num_res_blocks=8--n_batch_train=64 i= 0 loss=-2228.8938 learning grads mean= nan grads max=448971.1250 disp=2.339, 15.666, 28.026, 40.882, 53.527, 42.221, 42.221 loss_ts=19446.877, 16269.05, 13148.938, 11828.364, 11097.337, 5871.38, 5871.38 f_ts=17548.744, 19199.219, 23628.014, 21098.88, 26118.945, 14722.812, 14722.812 is_accepted_ts= 0.0000 lr=0.00000010 time=126.57s 2021-07-16 20:43:13,702 : early exit due to nan 2021-07-16 20:43:13,703 : done

Can you tell me how to reinterpret the result in the paper? Thanks.