I've managed to render several of the fitted scenes on Ubuntu 22.04 LTS, with an NVIDIA GeForce 940MX GPU (2 GB RAM). However, I faced several challenges while doing so. Therefore, I am going to document these challenges with their solutions here in case someone finds them helpful.
Note: this guide was written based on commit 5bc6f8d18e61978f167f7dbb21787771fbd59bf6.
Step 0
Before starting, make sure you have gcc and OpenGL tools installed. Installing these will save you from dealing with a lot of problems later. This can be done by running:
$ sudo apt update
$ sudo apt install build-essential # gcc
$ sudo apt-get install libglu1-mesa-dev freeglut3-dev mesa-common-dev libglfw3-dev # OpenGL
More details on how to install OpenGL can be found here.
Next, make sure you have an appropriate NVIDIA driver installed, and that you have CUDA installed. These can be installed via a quick google search.
Finally, make sure you have Anaconda installed.
Step 1
Clone the repository
git clone https://github.com/alievk/npbg.git
then run install_deps.sh
cd npbg
source scripts/install_deps.sh
Step 2
Download the fitted scenes and the rendering network weights as described here. Place the downloads folder in the root directory (where the README.md file is located).
Step 3
As described in the README.md file, try running the Person 1 fitted scene:
$ python viewer.py --config downloads/person_1.yaml --viewport 2000,1328 --origin-view
This will likely not work, and you will observe an AttributeError:
Traceback (most recent call last):
File "viewer.py", line 9, in <module>
from npbg.gl.render import OffscreenRender, create_shared_texture, cpy_tensor_to_buffer, cpy_tensor_to_texture
File "/home/mhdadk/Documents/npbg2/npbg/gl/render.py", line 14, in <module>
import torch
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/__init__.py", line 280, in <module>
from .functional import *
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/functional.py", line 2, in <module>
import torch.nn.functional as F
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/__init__.py", line 1, in <module>
from .modules import * # noqa: F401
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/__init__.py", line 2, in <module>
from .linear import Identity, Linear, Bilinear
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 5, in <module>
from .. import functional as F
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/functional.py", line 14, in <module>
from .._jit_internal import boolean_dispatch, List
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/_jit_internal.py", line 595, in <module>
import typing_extensions
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/typing_extensions-4.2.0-py3.6.egg/typing_extensions.py", line 159, in <module>
class _FinalForm(typing._SpecialForm, _root=True):
AttributeError: module 'typing' has no attribute '_SpecialForm'
To fix this AttributeError, run
$ pip uninstall typing-extensions
$ pip install typing-extensions
Try to run the Person 1 fitted scene again:
$ python viewer.py --config downloads/person_1.yaml --viewport 2000,1328 --origin-view
However, this will not work again, and you will observe the following error:
Traceback (most recent call last):
File "viewer.py", line 434, in <module>
my_app = MyApp(args)
File "viewer.py", line 103, in __init__
_config = yaml.load(f)
TypeError: load() missing 1 required positional argument: 'Loader'
The solution to this error can be found here. To fix this error, run
$ pip install pyyaml==5.4.1
Again, try to run the Person 1 fitted scene:
$ python viewer.py --config downloads/person_1.yaml --viewport 2000,1328 --origin-view
Unfortunately, this will still throw an error:
loading pointcloud...
=== 3D model ===
VERTICES: 3072078
EXTENT: [-16.5258255 -14.02860832 -30.76384926] [ 10.81384468 2.03398347 -20.07631493]
================
new viewport size (2000, 1328)
libGL error: MESA-LOADER: failed to open iris: /usr/lib/dri/iris_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
libGL error: failed to load driver: iris
libGL error: MESA-LOADER: failed to open iris: /usr/lib/dri/iris_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
libGL error: failed to load driver: iris
libGL error: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
libGL error: failed to load driver: swrast
[w] b'GLX: Failed to create context: GLXBadFBConfig'
[x] Window creation failed
The solution to this error can be found in this answer. To fix this error, run
$ export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6
One more time: try to run the Person 1 fitted scene:
$ python viewer.py --config downloads/person_1.yaml --viewport 2000,1328 --origin-view
Still not quite there yet. You will observe the following error:
loading pointcloud...
=== 3D model ===
VERTICES: 3072078
EXTENT: [-16.5258255 -14.02860832 -30.76384926] [ 10.81384468 2.03398347 -20.07631493]
================
new viewport size (2000, 1328)
[i] HiDPI detected, fixing window size
[w] Cannot read STENCIL size from the framebuffer
Unable to load numpy_formathandler accelerator from OpenGL_accelerate
[i] Using GLFW (GL 4.6)
Traceback (most recent call last):
File "/home/mhdadk/Documents/npbg2/npbg/gl/render.py", line 29, in _init_buffers
import pycuda.gl.autoinit # this may fails in headless mode
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/gl/autoinit.py", line 10, in <module>
context = make_default_context(lambda dev: cudagl.make_context(dev))
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/tools.py", line 228, in make_default_context
"on any of the %d detected devices" % ndevices
RuntimeError: make_default_context() wasn't able to create a context on any of the 1 detected devices
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "viewer.py", line 434, in <module>
my_app = MyApp(args)
File "viewer.py", line 177, in __init__
clear_color=args.clear_color)
File "/home/mhdadk/Documents/npbg2/npbg/gl/render.py", line 19, in __init__
self._init_buffers(viewport_size, out_buffer_location)
File "/home/mhdadk/Documents/npbg2/npbg/gl/render.py", line 31, in _init_buffers
raise RuntimeError('PyCUDA init failed, cannot use torch buffer')
RuntimeError: PyCUDA init failed, cannot use torch buffer
The solution to this RuntimeError can be found in issue #12. To fix this error, replace the code in viewer.py with the code given in the gist in this reply. In other words, here is the code to put into viewer.py:
import argparse
import threading
import yaml
import re
from glumpy import app, gloo, glm, gl, transforms
from glumpy.ext import glfw
from npbg.gl.render import OffscreenRender, create_shared_texture, cpy_tensor_to_buffer, cpy_tensor_to_texture
from npbg.gl.programs import NNScene
from npbg.gl.utils import load_scene_data, get_proj_matrix, crop_intrinsic_matrix, crop_proj_matrix, \
setup_scene, rescale_K, FastRand, nearest_train, pca_color, extrinsics_from_view_matrix, extrinsics_from_xml
from npbg.gl.nn import OGL
from npbg.gl.camera import Trackball
import os, sys
import time
import numpy as np
import torch
import cv2
def get_args():
parser = argparse.ArgumentParser(description='')
parser.add_argument('-c', '--config', type=str, default=None, required=True, help='config path')
parser.add_argument('--viewport', type=str, default='', help='width,height')
parser.add_argument('--keep-fov', action='store_true', help='keep field of view when resizing viewport')
parser.add_argument('--init-view', type=str, help='camera label for initial view or path to 4x4 matrix')
parser.add_argument('--use-mesh', action='store_true')
parser.add_argument('--use-texture', action='store_true')
parser.add_argument('--rmode', choices=['trackball', 'fly'], default='trackball')
parser.add_argument('--fps', action='store_true', help='show fps')
parser.add_argument('--light-position', type=str, default='', help='x,y,z')
parser.add_argument('--replay-camera', type=str, default='', help='path to view_matrix to replay at given fps')
parser.add_argument('--replay-fps', type=float, default=25., help='view_matrix replay fps')
parser.add_argument('--supersampling', type=int, default=1, choices=[1, 2])
parser.add_argument('--clear-color', type=str)
parser.add_argument('--nearest-train', action='store_true')
parser.add_argument('--gt', help='like /path/to/images/*.JPG. * will be replaced with nearest camera label.')
parser.add_argument('--pca', action='store_true')
parser.add_argument('--origin-view', action='store_true')
parser.add_argument('--temp-avg', action='store_true')
parser.add_argument('--checkpoint')
args = parser.parse_args()
args.viewport = tuple([int(x) for x in args.viewport.split(',')]) if args.viewport else None
args.light_position = [float(x) for x in args.light_position.split(',')] if args.light_position else None
args.clear_color = [float(x) for x in args.clear_color.split(',')] if args.clear_color else None
return args
def get_screen_program(texture):
vertex = '''
attribute vec2 position;
attribute vec2 texcoord;
varying vec2 v_texcoord;
void main()
{
gl_Position = <transform>;
v_texcoord = texcoord;
} '''
fragment = '''
uniform sampler2D texture;
varying vec2 v_texcoord;
void main()
{
gl_FragColor = texture2D(texture, v_texcoord);
} '''
quad = gloo.Program(vertex, fragment, count=4)
quad["transform"] = transforms.OrthographicProjection(transforms.Position("position"))
quad['texcoord'] = [( 0, 0), ( 0, 1), ( 1, 0), ( 1, 1)]
quad['texture'] = texture
return quad
def start_fps_job():
def job():
print(f'FPS {app.clock.get_fps():.1f}')
threading.Timer(1.0, job).start()
def load_camera_trajectory(path):
if path[-3:] == 'xml':
view_matrix, camera_labels = extrinsics_from_xml(path)
else:
view_matrix, camera_labels = extrinsics_from_view_matrix(path)
return view_matrix
def fix_viewport_size(viewport_size, factor=16):
viewport_w = factor * (viewport_size[0] // factor)
viewport_h = factor * (viewport_size[1] // factor)
return viewport_w, viewport_h
class MyApp():
def __init__(self, args):
with open(args.config) as f:
_config = yaml.load(f)
# support two types of configs
# 1 type - config with scene data
# 2 type - config with model checkpoints and path to scene data config
if 'scene' in _config: # 1 type
self.scene_data = load_scene_data(_config['scene'])
net_ckpt = _config.get('net_ckpt')
texture_ckpt = _config.get('texture_ckpt')
else:
self.scene_data = load_scene_data(args.config)
net_ckpt = self.scene_data['config'].get('net_ckpt')
texture_ckpt = self.scene_data['config'].get('texture_ckpt')
self.viewport_size = args.viewport if args.viewport else self.scene_data['config']['viewport_size']
self.viewport_size = fix_viewport_size(self.viewport_size)
print('new viewport size ', self.viewport_size)
# crop/resize viewport
if self.scene_data['intrinsic_matrix'] is not None:
K_src = self.scene_data['intrinsic_matrix']
old_size = self.scene_data['config']['viewport_size']
sx = self.viewport_size[0] / old_size[0]
sy = self.viewport_size[1] / old_size[1]
K_crop = rescale_K(K_src, sx, sy, keep_fov=args.keep_fov)
self.scene_data['proj_matrix'] = get_proj_matrix(K_crop, self.viewport_size)
elif self.scene_data['proj_matrix'] is not None:
new_proj_matrix = crop_proj_matrix(self.scene_data['proj_matrix'], *self.scene_data['config']['viewport_size'], *self.viewport_size)
self.scene_data['proj_matrix'] = new_proj_matrix
else:
raise Exception('no intrinsics are provided')
if args.init_view:
if args.init_view in self.scene_data['view_matrix']:
idx = self.scene_data['camera_labels'].index(args.init_view)
init_view = self.scene_data['view_matrix'][idx]
elif os.path.exists(args.init_view):
init_view = np.loadtxt(args.init_view)
else:
init_view = self.scene_data['view_matrix'][0]
if args.origin_view:
top_view = np.eye(4)
top_view[2, 3] = 20.
init_view = top_view
if np.allclose(self.scene_data['model3d_origin'], np.eye(4)):
print('Setting origin as mass center')
origin = np.eye(4)
origin[:3, 3] = -np.percentile(self.scene_data['pointcloud']['xyz'], 90, 0)
self.scene_data['model3d_origin'] = origin
else:
# force identity origin
self.scene_data['model3d_origin'] = np.eye(4)
self.trackball = Trackball(init_view, self.viewport_size, 1, rotation_mode=args.rmode)
args.use_mesh = args.use_mesh or _config.get('use_mesh') or args.use_texture
# this also creates GL context necessary for setting up shaders
self.window = app.Window(width=self.viewport_size[0], height=self.viewport_size[1], visible=True, fullscreen=False)
self.window.set_size(*self.viewport_size)
if args.checkpoint:
assert 'Texture' in args.checkpoint, 'Set path to descriptors checkpoint'
ep = re.search('epoch_[0-9]+', args.checkpoint).group().split('_')[-1]
net_name = f'UNet_stage_0_epoch_{ep}_net.pth'
net_ckpt = os.path.join(*args.checkpoint.split('/')[:-1], net_name)
texture_ckpt = args.checkpoint
need_neural_render = net_ckpt is not None
# self.out_buffer_location = 'torch' if need_neural_render else 'opengl'
self.out_buffer_location = 'numpy' if need_neural_render else 'opengl'
# setup screen image plane
self.off_render = OffscreenRender(viewport_size=self.viewport_size, out_buffer_location=self.out_buffer_location,
clear_color=args.clear_color)
if self.out_buffer_location == 'torch':
screen_tex, self.screen_tex_cuda = create_shared_texture(
np.zeros((self.viewport_size[1], self.viewport_size[0], 4), np.float32)
)
else:
screen_tex, self.screen_tex_cuda = self.off_render.color_buf, None
gl_view = screen_tex.view(gloo.TextureFloat2D)
gl_view.activate() # force gloo to create on GPU
gl_view.deactivate()
screen_tex = gl_view
self.screen_program = get_screen_program(screen_tex)
self.scene = NNScene()
if need_neural_render:
print(f'Net checkpoint: {net_ckpt}')
print(f'Texture checkpoint: {texture_ckpt}')
self.model = OGL(self.scene, self.scene_data, self.viewport_size, net_ckpt,
texture_ckpt, out_buffer_location=self.out_buffer_location, supersampling=args.supersampling, temporal_average=args.temp_avg)
else:
self.model = None
if args.pca:
assert texture_ckpt
tex = torch.load(texture_ckpt, map_location='cpu')['state_dict']['texture_']
print('PCA...')
pca = pca_color(tex)
pca = (pca - np.percentile(pca, 10)) / (np.percentile(pca, 90) - np.percentile(pca, 10))
pca = np.clip(pca, 0, 1)
self.scene_data['pointcloud']['rgb'] = np.clip(pca, 0, 1)
setup_scene(self.scene, self.scene_data, args.use_mesh, args.use_texture)
if args.light_position is not None:
self.scene.set_light_position(args.light_position)
if args.replay_camera:
self.camera_trajectory = load_camera_trajectory(args.replay_camera)
else:
self.camera_trajectory = None
self.window.attach(self.screen_program['transform'])
self.window.push_handlers(on_init=self.on_init)
self.window.push_handlers(on_close=self.on_close)
self.window.push_handlers(on_draw=self.on_draw)
self.window.push_handlers(on_resize=self.on_resize)
self.window.push_handlers(on_key_press=self.on_key_press)
self.window.push_handlers(on_mouse_press=self.on_mouse_press)
self.window.push_handlers(on_mouse_drag=self.on_mouse_drag)
self.window.push_handlers(on_mouse_release=self.on_mouse_release)
self.window.push_handlers(on_mouse_scroll=self.on_mouse_scroll)
self.mode0 = NNScene.MODE_COLOR
self.mode1 = 0
self.point_size = 1
self.point_mode = False
self.draw_points = not args.use_mesh
self.flat_color = True
self.neural_render = need_neural_render
self.show_pca = False
self.n_frame = 0
self.t_elapsed = 0
self.last_frame = None
self.last_view_matrix = None
self.last_gt_image = None
self.mouse_pressed = False
self.args = args
def run(self):
if self.args.fps:
start_fps_job()
app.run()
def render_frame(self, view_matrix):
self.scene.set_camera_view(view_matrix)
if self.neural_render:
frame = self.model.infer()['output'].flip([0])
else:
self.scene.set_mode(self.mode0, self.mode1)
if self.point_mode == 0:
self.scene.set_splat_mode(False)
self.scene.program['splat_mode'] = int(0)
elif self.point_mode == 1:
self.scene.set_splat_mode(True)
self.scene.program['splat_mode'] = int(0)
elif self.point_mode == 2:
self.scene.set_splat_mode(False)
self.scene.program['splat_mode'] = int(1)
if not self.scene.use_point_sizes:
self.scene.set_point_size(self.point_size)
self.scene.set_draw_points(self.draw_points)
self.scene.set_flat_color(self.flat_color)
frame = self.off_render.render(self.scene)
return frame
def print_info(self):
print('-- start info')
mode = [m[0] for m in NNScene.__dict__.items() if m[0].startswith('MODE_') and self.mode0 == m[1]][0]
print(mode)
n_mode = [m[0] for m in NNScene.__dict__.items() if m[0].startswith('NORMALS_MODE_') and self.mode1 == m[1]][0]
print(n_mode)
print(f'point size {self.point_size}')
print(f'splat mode: {self.point_mode}')
print('-- end info')
def save_screen(self, out_dir='./data/screenshots'):
os.makedirs(out_dir, exist_ok=True)
get_name = lambda s: time.strftime(f"%m-%d_%H-%M-%S___{s}")
img = self.last_frame.cpu().numpy()[..., :3][::-1, :, ::-1] * 255
cv2.imwrite(os.path.join(out_dir, get_name('screenshot') + '.png'), img)
np.savetxt(os.path.join(out_dir, get_name('pose') + '.txt'), self.last_view_matrix)
def get_next_view_matrix(self, frame_num, elapsed_time):
if self.camera_trajectory is None:
return self.trackball.pose
n = int(elapsed_time * args.replay_fps) % len(self.camera_trajectory)
return self.camera_trajectory[n]
# ===== Window events =====
def on_init(self):
pass
def on_key_press(self, symbol, modifiers):
KEY_PLUS = 61
if symbol == glfw.GLFW_KEY_X:
self.mode0 = NNScene.MODE_XYZ
self.neural_render = False
elif symbol == glfw.GLFW_KEY_N:
self.mode0 = NNScene.MODE_NORMALS
self.neural_render = False
elif symbol == glfw.GLFW_KEY_C:
self.mode0 = NNScene.MODE_COLOR
self.neural_render = False
elif symbol == glfw.GLFW_KEY_U:
self.mode0 = NNScene.MODE_UV
self.neural_render = False
elif symbol == glfw.GLFW_KEY_D:
self.mode0 = NNScene.MODE_DEPTH
self.neural_render = False
elif symbol == glfw.GLFW_KEY_L:
self.mode0 = NNScene.MODE_LABEL
self.neural_render = False
elif symbol == glfw.GLFW_KEY_Y:
self.neural_render = True
self.show_pca = False
elif symbol == glfw.GLFW_KEY_T:
self.neural_render = True
self.show_pca = True
elif symbol == glfw.GLFW_KEY_Z:
self.mode1 = (self.mode1 + 1) % 5
elif symbol == KEY_PLUS:
self.point_size = self.point_size + 1
elif symbol == glfw.GLFW_KEY_MINUS:
self.point_size = max(0, self.point_size - 1)
elif symbol == glfw.GLFW_KEY_P:
self.point_mode = (self.point_mode + 1) % 3
elif symbol == glfw.GLFW_KEY_Q:
self.draw_points = not self.draw_points
elif symbol == glfw.GLFW_KEY_F:
self.flat_color = not self.flat_color
elif symbol == glfw.GLFW_KEY_I:
self.print_info()
elif symbol == glfw.GLFW_KEY_S:
self.save_screen()
else:
print(symbol, modifiers)
def on_draw(self, dt):
self.last_view_matrix = self.get_next_view_matrix(self.n_frame, self.t_elapsed)
self.last_frame = self.render_frame(self.last_view_matrix)
if self.out_buffer_location == 'torch':
cpy_tensor_to_texture(self.last_frame, self.screen_tex_cuda)
elif self.out_buffer_location == 'numpy':
if isinstance(self.last_frame, torch.Tensor):
rendered = self.last_frame.cpu().numpy()
elif isinstance(self.last_frame, np.ndarray):
rendered = self.last_frame
if rendered.shape[2] == 3:
rendered = np.concatenate([rendered, np.ones_like(rendered[..., [0]])], axis=2)
self.screen_program['texture'] = rendered
self.window.clear()
gl.glDisable(gl.GL_CULL_FACE)
# ensure viewport size is correct (offline renderer could change it)
gl.glViewport(0, 0, self.viewport_size[0], self.viewport_size[1])
self.screen_program.draw(gl.GL_TRIANGLE_STRIP)
self.n_frame += 1
self.t_elapsed += dt
if self.args.nearest_train:
ni = nearest_train(self.scene_data['view_matrix'], np.linalg.inv(self.scene_data['model3d_origin']) @ self.last_view_matrix)
label = self.scene_data['camera_labels'][ni]
assert self.args.gt, 'you must define path to gt images'
path = self.args.gt.replace('*', str(label))
if not os.path.exists(path):
print(f'{path} NOT FOUND!')
elif self.last_gt_image != path:
self.last_gt_image = path
img = cv2.imread(path)
max_side = max(img.shape[:2])
s = 1024 / max_side
img = cv2.resize(img, None, None, s, s)
cv2.imshow('nearest train', img)
cv2.waitKey(1)
def on_resize(self, w, h):
print(f'on_resize {w}x{h}')
self.trackball.resize((w, h))
self.screen_program['position'] = [(0, 0), (0, h), (w, 0), (w, h)]
def on_close(self):
pass
def on_mouse_press(self, x, y, buttons, modifiers):
# print(buttons, modifiers)
self.trackball.set_state(Trackball.STATE_ROTATE)
if (buttons == app.window.mouse.LEFT):
ctrl = (modifiers & app.window.key.MOD_CTRL)
shift = (modifiers & app.window.key.MOD_SHIFT)
if (ctrl and shift):
self.trackball.set_state(Trackball.STATE_ZOOM)
elif ctrl:
self.trackball.set_state(Trackball.STATE_ROLL)
elif shift:
self.trackball.set_state(Trackball.STATE_PAN)
elif (buttons == app.window.mouse.MIDDLE):
self.trackball.set_state(Trackball.STATE_PAN)
elif (buttons == app.window.mouse.RIGHT):
self.trackball.set_state(Trackball.STATE_ZOOM)
self.trackball.down(np.array([x, y]))
# Stop animating while using the mouse
self.mouse_pressed = True
def on_mouse_drag(self, x, y, dx, dy, buttons):
self.trackball.drag(np.array([x, y]))
def on_mouse_release(self, x, y, button, modifiers):
self.mouse_pressed = False
def on_mouse_scroll(self, x, y, dx, dy):
self.trackball.scroll(dy)
if __name__ == '__main__':
args = get_args()
my_app = MyApp(args)
my_app.run()
FINALLY, try to run the Person 1 fitted scene again:
$ python viewer.py --config downloads/person_1.yaml --viewport 2000,1328 --origin-view
Depending on how much RAM your GPU has, you may see a CUDA out of memory error:
loading pointcloud...
=== 3D model ===
VERTICES: 3072078
EXTENT: [-16.5258255 -14.02860832 -30.76384926] [ 10.81384468 2.03398347 -20.07631493]
================
new viewport size (2000, 1328)
[i] HiDPI detected, fixing window size
[w] Cannot read STENCIL size from the framebuffer
Unable to load numpy_formathandler accelerator from OpenGL_accelerate
[i] Using GLFW (GL 4.6)
Net checkpoint: downloads/scenes/person_1/02-20_08-42-55/UNet_stage_0_epoch_39_net.pth
Texture checkpoint: downloads/scenes/person_1/02-20_08-42-55/PointTexture_stage_0_epoch_39.pth
SUPERSAMPLING: 2
[i] Running at 60 frames/second
on_resize 1920x1009
Traceback (most recent call last):
File "viewer.py", line 452, in <module>
my_app.run()
File "viewer.py", line 256, in run
app.run()
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/glumpy/app/__init__.py", line 317, in run
clock = __init__(clock=clock, framerate=framerate, backend=__backend__)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/glumpy/app/__init__.py", line 277, in __init__
window.dispatch_event('on_resize', window._width, window._height)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/glumpy/app/window/event.py", line 396, in dispatch_event
if getattr(self, event_type)(*args):
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/glumpy/app/window/window.py", line 221, in on_resize
self.dispatch_event('on_draw', 0.0)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/glumpy/app/window/event.py", line 386, in dispatch_event
if handler(*args):
File "viewer.py", line 366, in on_draw
self.last_frame = self.render_frame(self.last_view_matrix)
File "viewer.py", line 262, in render_frame
frame = self.model.infer()['output'].flip([0])
File "/home/mhdadk/Documents/npbg2/npbg/gl/nn.py", line 121, in infer
out, net_input = self.model(input_dict, return_input=True)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/mhdadk/Documents/npbg2/npbg/models/compose.py", line 194, in forward
out1 = self.net(*input_multiscale, **kwargs)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/mhdadk/Documents/npbg2/npbg/models/unet.py", line 271, in forward
up2 = self.up2(up3, down1)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/mhdadk/Documents/npbg2/npbg/models/unet.py", line 130, in forward
output= self.conv(torch.cat([in1_up, inputs2_], 1))
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/mhdadk/Documents/npbg2/npbg/models/unet.py", line 77, in forward
features = self.block.act_f(self.block.conv_f(x))
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/modules/activation.py", line 329, in forward
return F.elu(input, self.alpha, self.inplace)
File "/home/mhdadk/miniconda3/envs/npbg/lib/python3.6/site-packages/torch/nn/functional.py", line 992, in elu
result = torch._C._nn.elu(input, alpha)
RuntimeError: CUDA out of memory. Tried to allocate 82.00 MiB (GPU 0; 1.96 GiB total capacity; 1.36 GiB already allocated; 71.62 MiB free; 98.74 MiB cached)
In this case, change the viewport parameter from 2000,1328 to something smaller, like 500,500:
$ python viewer.py --config downloads/person_1.yaml --viewport 500,500 --origin-view
This should now work. In case you see the message:
"viewer.py" is not responding.
This just means that it is taking some time to load the render. Click on "Wait" and the render should load soon.
Project 
Video 
Paper
662 Jan 1, 2023
119 Jan 2, 2023
259 Dec 25, 2022
616 Jan 6, 2023
63 Dec 9, 2022
43 Dec 5, 2022
75 Nov 24, 2022
78 Dec 27, 2022
65 Dec 26, 2022
1k Jan 6, 2023
892 Dec 28, 2022
59 Dec 17, 2022
44 Feb 24, 2022
27 Oct 6, 2022
34 Nov 30, 2022
149 Dec 15, 2022
114 Dec 29, 2022
86 Oct 5, 2022