Thanks for sharing your code. I have a problem with running preprocess_mask_rcnn.py. I get the following error: RuntimeError: No such operator detectron2::nms_rotated. I followed all your instructions for creating the required environment, and it seems that detectron2 is installed successfully based on this line: Successfully installed antlr4-python3-runtime-4.8 detectron2-0.4+cu101 future-0.18.2 fvcore-0.1.3.post20210317 google-auth-1.35.0 iopath-0.1.9 omegaconf-2.1.2 portalocker-2.4.0 pycocotools-2.0.4 pydot-1.4.2 tabulate-0.8.9 termcolor-1.1.0 yacs-0.1.8 I can import detectron2 without any problem, but after running from detectron2 import model_zoo, I get the error below:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/neural_atlases/lib/python3.7/site-packages/detectron2/model_zoo/__init__.py", line 8, in <module>
    from .model_zoo import get, get_config_file, get_checkpoint_url, get_config
  File "/anaconda3/envs/neural_atlases/lib/python3.7/site-packages/detectron2/model_zoo/model_zoo.py", line 9, in <module>
    from detectron2.modeling import build_model
  File "/anaconda3/envs/neural_atlases/lib/python3.7/site-packages/detectron2/modeling/__init__.py", line 2, in <module>
    from detectron2.layers import ShapeSpec
  File "/anaconda3/envs/neural_atlases/lib/python3.7/site-packages/detectron2/layers/__init__.py", line 5, in <module>
    from .nms import batched_nms, batched_nms_rotated, nms, nms_rotated
  File "/anaconda3/envs/neural_atlases/lib/python3.7/site-packages/detectron2/layers/nms.py", line 16, in <module>
    nms_rotated_func = torch.ops.detectron2.nms_rotated
  File "/.local/lib/python3.7/site-packages/torch/_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator detectron2::nms_rotated

I am wondering to find out why this error happens and how I can solve it. Thank you.

Hi there! This project is phenomenol, seriously excited to try this, see what others make of it and how people use it.

Update: Managed to fix this! I realised that I just hadn't cloned the Raft project properly.

I'm trying to run this in colab, it appears to work up until the point Im trying to run preprocess_optical_flow.py where it gives me. "" /content/drive/MyDrive/VideoEdit/layered-neural-atlases Traceback (most recent call last): File "/content/drive/MyDrive/VideoEdit/layered-neural-atlases/preprocess_optical_flow.py", line 4, in from raft_wrapper import RAFTWrapper File "/content/drive/MyDrive/VideoEdit/layered-neural-atlases/raft_wrapper.py", line 12, in from utils.utils import InputPadder ModuleNotFoundError: No module named 'utils' "" I have imported the models under RAFT/models but don't see a utils file or folder anywhere, other than unwrap_utils.py.

Hoping you might have time to answer my query. Again amazing work!

Hello thanks for your amazing work.

I have run the train and edit based on your pretrained model, everything goes fine.

However, after I train 40w model on blackswan and use your edited altas, I cannot get correct results. Should I make new atlas images based on the new trained model? Then where I could get the raw atlas(unedit from the model) images?

Thanks in advance.

(neural_atlases) D:\layered-neural-atlases-main>python only_edit.py --trained_mo del_folder=pretrained_models/checkpoints/libby --video_name=libby --data_folder= data/libby --output_folder=editing_outputs --use_edit_frame --edit_frame_index= 7 --edit_frame_path=pretrained_models/edit_inputs/libby/edit_frame_.png

Model has 264706 params Model has 133122 params Model has 416379 params Model has 402945 params Traceback (most recent call last): File "only_edit.py", line 472, in main(training_folder, frame_edit, frames_folder, mask_rcnn_folder, frame_edi t_file, edit_tex1_file, edit_tex2_file, File "only_edit.py", line 378, in main edit_im1, edit_im2 = texture_edit_from_frame_edit(edit_frame, frame_number, model_F_mapping1, model_F_mapping2, File "only_edit.py", line 204, in texture_edit_from_frame_edit maxx2, minx2, maxy2, miny2, edge_size2 = get_mapping_area(model_F_mapping2, model_alpha, mask_frames > -1, larger_dim, File "D:\layered-neural-atlases-main\evaluate.py", line 149, in get_mapping_ar ea relisa = np.array_split(relis_i.numpy(), np.ceil(relis_i.shape[0] / 100000))

File "<array_function internals>", line 5, in array_split File "D:\Anaconda3\envs\neural_atlases\lib\site-packages\numpy\lib\shape_base. py", line 778, in array_split raise ValueError('number sections must be larger than 0.') from None ValueError: number sections must be larger than 0.

Thanks for sharing this amazing code! I'm trying to implement the Multi Foreground Atlases feature referenced by the Section 4.3 on arxiv paper. But, i can't understand this sentence:

Unlike the one foreground object case, to support occlusions between different foreground objects, the sparsity loss is applied directly on the atlas, by applying 𝑙1 regularization on randomly sampled UV coordinates from foreground regions in the atlas.

What this means in practice? I need to apply the l1 regularization equation error(y, ŷ) + λ * Σ |w|?

if yes:

• What is the lambda value used in the lucia results on paper?
• The error(y, ŷ) term is the single layer case sparsity loss equation (Eq. 14)?
• The |w| term is the values from uv coordinates given by the multiple foreground mapping models?

if not:

• How to calculate, in practice, the sparsity loss on multi foreground object case?
• Please, explain with a equation for easier understanding.

Another questions:

• The equations that calculates the losses for each mapping model (like the rigidity_loss and the optical_flow_loss) need to be applied for each foreground mapping model and summed at the end?

• What is the coefficient values used for the user scribble losses on the equations:

  • l_red = -log(alpha_red) Eq. 20
  • l_green = -log(alpha_green) Eq. 21

• What is this βtv = 100 variable at Section 3.5?

I would be grateful if you can answer these questions. Thanks!

python only_edit.py --trained_model_folder=pretrained_models/checkpoints/libby --video_name=libby --data_folder=data --output_folder=editing_outputs --use_edit_frame --edit_frame_index=7 --edit_frame_path=pretrained_models/edit_inputs/libby/edit_frame_.png

how set edit_frame_index = ? 1 2 3 ... ?

Thanks for your excellent work and source code.

I saw that in your provided config.json, height is 432 and width is 768. But in my case, the height is 768 and width is 432, and I only changed the corresponding config file as: "resx": 768, "resy": 432, However, the reconstructed video seem strange and cannot work well.

Wonder if there are other codes I should change when I deal with this portrait video. Thanks a lot ~