Thanks for your astonishing work.

I notice that some voxels have negative density or zero density. I was wondering how the density is defined if it's negative. According to Formulas 1 and 2 in the plenoxel paper, if density is negative, the result of RGB may be unreasonable. However, I did some experiments to clip the density in the trained model，and the PSNR decreased.

I didn't find a value domain constraint on density in the code. Maybe I missed something. So I hope to get your opinion.

Thanks!

I test the code on Titan V 12G GPU card and 128G memory, it runs out of memory. The paper presents the optimization per scene on a single GPU, I would like to know how much memory need for optimization.

Traceback (most recent call last): File "plenoptimize.py", line 479, in main() File "plenoptimize.py", line 434, in main mse, data_grad = jax.value_and_grad(lambda grid: get_loss_rays((indices, grid), batch_rays, target_s, FLAGS.resolution, radius, FLAGS.harmonic_degree, FLAGS.jitter, FLAGS.uniform, subkeys, sh_dim, occupancy_penalty, FLAGS.interpolation, FLAGS.nv))(data) File "plenoptimize.py", line 434, in mse, data_grad = jax.value_and_grad(lambda grid: get_loss_rays((indices, grid), batch_rays, target_s, FLAGS.resolution, radius, FLAGS.harmonic_degree, FLAGS.jitter, FLAGS.uniform, subkeys, sh_dim, occupancy_penalty, FLAGS.interpolation, FLAGS.nv))(data) File "plenoptimize.py", line 293, in get_loss_rays rgb, disp, acc, weights, voxel_ids = plenoxel.render_rays(data_dict, rays, resolution, key, radius, harmonic_degree, jitter, uniform, interpolation, nv) jax._src.traceback_util.FilteredStackTrace: RuntimeError: Resource exhausted: Out of memory while trying to allocate 196608000 bytes.

The stack trace above excludes JAX-internal frames. The following is the original exception that occurred, unmodified.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "plenoptimize.py", line 479, in main() File "plenoptimize.py", line 434, in main mse, data_grad = jax.value_and_grad(lambda grid: get_loss_rays((indices, grid), batch_rays, target_s, FLAGS.resolution, radius, FLAGS.harmonic_degree, FLAGS.jitter, FLAGS.uniform, subkeys, sh_dim, occupancy_penalty, FLAGS.interpolation, FLAGS.nv))(data) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 139, in reraise_with_filtered_traceback return fun(*args, **kwargs) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/api.py", line 809, in value_and_grad_f ans, vjp_py = _vjp(f_partial, *dyn_args) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/api.py", line 1878, in _vjp out_primal, out_vjp = ad.vjp(flat_fun, primals_flat) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/ad.py", line 114, in vjp out_primals, pvals, jaxpr, consts = linearize(traceable, *primals) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/ad.py", line 101, in linearize jaxpr, out_pvals, consts = pe.trace_to_jaxpr(jvpfun_flat, in_pvals) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/partial_eval.py", line 498, in trace_to_jaxpr jaxpr, (out_pvals, consts, env) = fun.call_wrapped(pvals) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/linear_util.py", line 166, in call_wrapped ans = self.f(*args, **dict(self.params, **kwargs)) File "plenoptimize.py", line 434, in mse, data_grad = jax.value_and_grad(lambda grid: get_loss_rays((indices, grid), batch_rays, target_s, FLAGS.resolution, radius, FLAGS.harmonic_degree, FLAGS.jitter, FLAGS.uniform, subkeys, sh_dim, occupancy_penalty, FLAGS.interpolation, FLAGS.nv))(data) File "plenoptimize.py", line 293, in get_loss_rays rgb, disp, acc, weights, voxel_ids = plenoxel.render_rays(data_dict, rays, resolution, key, radius, harmonic_degree, jitter, uniform, interpolation, nv) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 139, in reraise_with_filtered_traceback return fun(*args, **kwargs) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/api.py", line 332, in cache_miss out_flat = xla.xla_call( File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1402, in bind return call_bind(self, fun, *args, **params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1393, in call_bind outs = primitive.process(top_trace, fun, tracers, params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1405, in process return trace.process_call(self, fun, tracers, params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/ad.py", line 308, in process_call result = call_primitive.bind(f_jvp, *primals, *nonzero_tangents, **new_params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1402, in bind return call_bind(self, fun, *args, **params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1393, in call_bind outs = primitive.process(top_trace, fun, tracers, params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1405, in process return trace.process_call(self, fun, tracers, params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/partial_eval.py", line 191, in process_call jaxpr, out_pvals, consts, env_tracers = self.partial_eval( File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/partial_eval.py", line 299, in partial_eval out_flat, (out_avals, jaxpr, env) = app(f, *in_consts), aux() File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1402, in bind return call_bind(self, fun, *args, **params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1393, in call_bind outs = primitive.process(top_trace, fun, tracers, params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 1405, in process return trace.process_call(self, fun, tracers, params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/core.py", line 600, in process_call return primitive.impl(f, *tracers, **params) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/xla.py", line 579, in _xla_call_impl return compiled_fun(*args) File "/home/hdr/anaconda3/lib/python3.8/site-packages/jax/interpreters/xla.py", line 830, in _execute_compiled out_bufs = compiled.execute(input_bufs) RuntimeError: Resource exhausted: Out of memory while trying to allocate 196608000 bytes.

Hi,

thanks for your awesome work! in the Plenoctree paper and in this implementation, you apply the sigmoid function to the SH sum: https://github.com/sarafridov/plenoxels/blob/975d2619a75b4e6fadd8b72e9edbf30fdf7e559c/plenoxel.py#L38

Whereas in the CUDA implementation it seems to me like you apply clamp(x+0.5, 0, inf): https://github.com/sxyu/svox2/blob/59984d6c4fd3d713353bafdcb011646e64647cc7/svox2/csrc/render_lerp_kernel_cuvol.cu#L102 outv += weight * fmaxf(lane_color_total + 0.5f, 0.f); // Clamp to [+0, infty)

What's the reason for the different implementations and which is more accurate? If it is the second one: why do you add constant 0.5f, whats the intuition for this factor?

Thanks a lot! :)

Hi I'm wondering about the trilinear interpolation of sigmas. From my understanding going from sigma to alpha looks like this: α = 1 - exp(-max(sum(σᵢ * wᵢ), 0))) where σᵢ, wᵢ are the neighbor sigmas and weights for trilinear interpolation.

I think it's possible for the sigmas to take on small negative values -- which could possibly cancel out other small positive sigmas. For example consider the 8 neighboring sigmas with values ([-0.001, -0.001, -0.001, -0.001, -0.001, -0.001, -0.001, 0.007]) and weights = 1/8 -- the result is a interpolated sigma of zero even though there are non-zero sigmas in the neighborhood.

Applying the relu function to sigmas before interpolating would fix this problem -- if it is actually a problem. Maybe the the effect is negligible?

Exception has occurred: TypeError mul got incompatible shapes for broadcasting: (800, 800, 1, 1536), (800, 1, 800, 3). File "/plenoxels/plenoptimize.py", line 479, in main() File "/plenoxels/plenoptimize.py", line 419, in main mse, data_grad = jax.value_and_grad(lambda grid: get_loss((indices, grid), c2w, gt, H, W, focal, FLAGS.resolution, radius, FLAGS.harmonic_degree, FLAGS.jitter, FLAGS.uniform, subkeys, sh_dim, occupancy_penalty, FLAGS.interpolation, FLAGS.nv))(data) File "/plenoxels/plenoptimize.py", line 419, in mse, data_grad = jax.value_and_grad(lambda grid: get_loss((indices, grid), c2w, gt, H, W, focal, FLAGS.resolution, radius, FLAGS.harmonic_degree, FLAGS.jitter, FLAGS.uniform, subkeys, sh_dim, occupancy_penalty, FLAGS.interpolation, FLAGS.nv))(data) File "/plenoxels/plenoptimize.py", line 285, in get_loss rgb, disp, acc, weights, voxel_ids = plenoxel.render_rays(data_dict, rays, resolution, key, radius, harmonic_degree, jitter, uniform, interpolation, nv) File "/plenoxels/plenoxel.py", line 490, in render_rays pts = rays_o[:, jnp.newaxis, :] + intersections[:, :, jnp.newaxis] * rays_d[:, jnp.newaxis, :] # [n_rays, n_intersections, 3]

Hi, thanks for your work. There is a sentence I cannot understand in "opt/util/dataset_base.py". def init(self): self.ndc_coeffs = (-1, -1) self.use_sphere_bound = False self.should_use_background = True # a hint self.use_sphere_bound = True As above, self.use_sphere_bound appears twice. Is it a mistake?

Thanks for your astonishing work.

“but is much slower (roughly 1 hour per epoch) than the CUDA implementation https://github.com/sxyu/svox2 (roughly 1 minute per epoch)”

As mentioned above, Jax is much slower than CUDA implementation. As far as I know, Jax could be also accelerated by GPU(cuda). Why is the speed gap so large? And is the fast training of plenoxels due to the acceleration of cuda implementation instead of not using the neural network?

rays = np.stack([get_rays_np(H, W, focal, p) for p in train_c2w[:,:3,:4]], 0) # [N, ro+rd, H, W, 3]

if change it by the following codes:

_# Precompute all the training rays and shuffle them
tp = train_c2w[:,:3,:4]
for p in tp:
    a = get_rays_np(H, W, focal, p)
    rays = np.stack([a], 0)
    print(p)_

the shape of :

rays is (1, 2, 800, 800, 3)
multi_lowpass(train_gt[:,None], FLAGS.resolution).astype(np.float32) is (100, 1, 800, 800, 3)

when np.concatenate the two value, it produces: ValueError: all the input array dimensions for the concatenation axis must match exactly,

many thanks for your great work! I have question about the code, can you give me some more detailed explanation？ ` #Compute when the rays enter and leave the grid

offsets_pos = jax.lax.stop_gradient((radius - rays_o) / rays_d) offsets_neg = jax.lax.stop_gradient((-radius - rays_o) / rays_d) offsets_in = jax.lax.stop_gradient(jnp.minimum(offsets_pos, offsets_neg)) offsets_out = jax.lax.stop_gradient(jnp.maximum(offsets_pos, offsets_neg)) start = jax.lax.stop_gradient(jnp.max(offsets_in, axis=-1, keepdims=True)) stop = jax.lax.stop_gradient(jnp.min(offsets_out, axis=-1, keepdims=True)) first_intersection = jax.lax.stop_gradient(rays_o + start * rays_d) `