Is there any way to add interpolation for animation? The current version of course knows how to make animation smoother, but with a bunch of artifacts. Yes, version 4 has become better compared to the past, but it's still better to make a separate model for animation, as they did for example in real-esrgan

Starting from the v3 model, the .py files will be packaged together in a compressed package for developers to change the code.

请不要在本帖下回复 Please do not reply under this post

inference_img.py - line 99

with bugs:

res.append(model.inference(img0, img1, (i+1) * 1. / (n+1), args.scale))

fixed:

img_list.append(model.inference(img0, img1, (i+1) / n))

1. There's no res variable, so I assumed it should be img_list
2. Formula was incorrect, resulting in a 0.333 ratio instead of 0.5 for x2 interpolation
3. There's no args.scale argument, so I removed it, but you can add it in the list of arguments to keep it

File "/Users/x/Practical-RIFE/inference_video.py", line 96, in model = Model() NameError: name 'Model' is not defined

I downloaded the model and added it to a train_log folder. How do I define the model?

Interpolating 720p or 1440p video with RIFE 4.0 throws an error after some frames:

File "D:\Code\GitHub\flowframes\Code\bin\x64\Release\FlowframesData\pkgs\rife-cuda\arch\RIFE_HDv3.py", line 59, in inference
flow, mask, merged = self.flownet(imgs, timestep, scale_list)
File "D:\Software\Python38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Code\GitHub\flowframes\Code\bin\x64\Release\FlowframesData\pkgs\rife-cuda\arch\IFNet_HDv3.py", line 99, in forward
f0, m0 = block[i](torch.cat((warped_img0[:, :3], warped_img1[:, :3], timestep, mask), 1), flow, scale=scale_list[i])
RuntimeError: Sizes of tensors must match except in dimension 2. Got 768 and 736 (The offending index is 2)

1080p works fine, it seems to be some scaling/cropping issue?

I can reliably reproduce this problem on 4.0, but not on 3.9, so it seems to be caused by IFNet_HDv3.py changes.

When testing version 4, a bug was found, because of which the video stupidly accelerates, instead of an upscale. That is, it is improving, but in some moments it stupidly accelerate

I've always seen plenty artefacts when interpolating anime vids using rife. Could you please make a model that's specially trained on anime to create the best results for this content? Besides rife is commonly used to interpolate anime too so I think a special model for anime would satisfy lots of people!

Upon running it for the first time, I got

Please download our model from model list
Traceback (most recent call last):
  File "[snip]/Practical-RIFE/inference_video.py", line 96, in <module>
    model = Model()
NameError: name 'Model' is not defined

I had installed a model. When I ran the line in IDLE:

>>> from train_log.RIFE_HDv3 import Model
    Traceback (most recent call last):
      File "/usr/lib64/python3.10/idlelib/run.py", line 578, in runcode
        exec(code, self.locals)
      File "<pyshell#2>", line 1, in <module>
      File "[snip]/repos/Practical-RIFE/train_log/RIFE_HDv3.py", line 11, in <module>
        from model.loss import *
      File "[snip]/repos/Practical-RIFE/model/loss.py", line 5, in <module>
        import torchvision.models as models
    ModuleNotFoundError: No module named 'torchvision'
>>>

I ran pip install torchvision and tried again, and it worked. I'm guessing this means that torchvision should be added to requirements.txt.

Says some crap about siize mismatch, it does not interpolate , in colab it should not have any errors , you just run it and it works, but it doesnt.Maybe it wasnt checked in a while and is broken for a long time.this is local macgine error : size mismatch for block2.conv0.0.0.weight: copying a param with shape torch.size

blah blah. Why is this happening ?

Hi Hzwer, thank you for great work! I'm currently testing model v3.8 and frame interpolation looks rather peculiar in comparison to v2.4, here is the example:

https://vimeo.com/577224538

Hi,

I fun side effect is that it also interpolates between cuts/scenes. Is there a way to avoid this? Set a threshold so it doesn't do any interpolating when the image changes too much?

10-bit video content is becoming more and more common with a lot of modern TVs, displays, and cameras (mobile phones or dedicated) being capable of recording or playing 10 bit video files. Popular standards like HDR10 have 10 bit video as a requirement.

Currently, RIFE doesn't support source frames with more than 8 bit pixel depth per component. It would be very useful if RIFE could read and write 10-bit frames without losing color information during the interpolation. Having 12/14/16-bit support for the future may also be useful, but supporting at least 10-bit content would be a big improvement, since such content is very common already.

rgb48le is one of the suitable pixel formats for the internal processing to support up to 16 bits (x3 components). But when Practical-RIFE receives such rgb48le frames instead of rgb24 frames from the ffmpeg-based reader, the pixel data gets truncated and corrupted back to the 8-bit format while going through the inference logic.

Would it be possible to make the inference logic compatible with 10+ bit content?

Thank you for your work on RIFE!

你好：

我写了个python脚本来转换模型为torchscript格式（cvtTorchScript.zip)，当我用torch.script.trace()函数转换时发生错误，报告系统内存不够（错误信息，torch.jit.trace.zip),然后我改用torch.jit.script()来转换，同样发生错误（错误信息torch.jit.script.zip），所以请问下我该如何转换模型为TorchScript格式，谢谢！

Hi, thank you for making your code available. The latest model release (v4.6) works really well for my project. I would like to fine tune the model on my dataset. Do you plan to make the script for the fine tuning available any time soon? Thank you.

I'm using 4.1 model. When I try to interpolarate a video, I get such a ERROR:

2022-07-24 19:29:27,296 - one_line_shot_args - 772 - INFO - Initial New Interpolation Project: project_dir: C:/Users/Globefish/Downloads/Compressed/SVFI.3.2.Community\\u3010MAD\u3011 Genuine article \u3010\u4ffa\u30ac\u30a4\u30eb\u3011_1, INPUT_FILEPATH: C:/Users/Globefish/Desktop/\u3010MAD\u3011 Genuine article \u3010\u4ffa\u30ac\u30a4\u30eb\u3011_1.mp4
2022-07-24 19:29:27,296 - one_line_shot_args - 774 - INFO - Changing working dir to C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package
2022-07-24 19:29:27,296 - one_line_shot_args - 781 - WARNING - Not find selected ffmpeg, use default
2022-07-24 19:29:27,376 - one_line_shot_args - 255 - INFO - 
Input Video Info
{'index': 0, 'width': 960, 'height': 540, 'color_range': 'tv', 'color_space': 'smpte170m', 'color_transfer': 'smpte170m', 'color_primaries': 'smpte170m', 'r_frame_rate': '30/1', 'duration': '230.500000', 'nb_frames': '6915', 'disposition': {'default': 1, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}, 'tags': {'creation_time': '2022-07-24T04:25:41.000000Z', 'language': 'eng', 'handler_name': '\x1fMainconcept Video Media Handler', 'encoder': 'AVC Coding'}}
2022-07-24 19:29:27,376 - one_line_shot_args - 273 - INFO - Auto Find FPS in r_frame_rate: 30.0
2022-07-24 19:29:27,376 - one_line_shot_args - 280 - INFO - Auto Find frames cnt in nb_frames: 6915
2022-07-24 19:29:27,390 - one_line_shot_args - 832 - INFO - Check Interpolation Source, FPS: 30.0, TARGET FPS: 60.0, FRAMES_CNT: 13830, EXP: 1
2022-07-24 19:29:27,407 - one_line_shot_args - 859 - INFO - Buffer Size to 677
2022-07-24 19:29:29,077 - one_line_shot_args - 1412 - INFO - Start VRAM Test: 960x540 with scale 1.0
2022-07-24 19:29:30,384 - one_line_shot_args - 1420 - ERROR - VRAM Check Failed, PLS Lower your presets
Traceback (most recent call last):
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\one_line_shot_args.py", line 1416, in nvidia_vram_test
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\Utils\inference.py", line 174, in generate_interp
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\Utils\inference.py", line 87, in __make_inference
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\model\RIFE_HDv3.py", line 60, in inference
flow, mask, merged = self.flownet(imgs, timestep, scale_list)
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\model\IFNet_HDv3.py", line 99, in forward
f0, m0 = block[i](torch.cat((warped_img0[:, :3], warped_img1[:, :3], timestep, mask), 1), flow, scale=scale_list[i])
RuntimeError: Sizes of tensors must match except in dimension 3. Got 576 and 544 (The offending index is 2)

Traceback (most recent call last):
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\one_line_shot_args.py", line 2151, in <module>
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\one_line_shot_args.py", line 1871, in run
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\one_line_shot_args.py", line 1416, in nvidia_vram_test
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\Utils\inference.py", line 174, in generate_interp
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\Utils\inference.py", line 87, in __make_inference
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\model\RIFE_HDv3.py", line 60, in inference
flow, mask, merged = self.flownet(imgs, timestep, scale_list)
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\GLOBEF~1\DOWNLO~1\COMPRE~1\SVFI32~1.COM\Package\model\IFNet_HDv3.py", line 99, in forward
f0, m0 = block[i](torch.cat((warped_img0[:, :3], warped_img1[:, :3], timestep, mask), 1), flow, scale=scale_list[i])
RuntimeError: Sizes of tensors must match except in dimension 3. Got 576 and 544 (The offending index is 2)
INFO - ONE LINE SHOT ARGS 6.6.1 2021/6/26
Warning: Find Empty Args at 'ffmpeg_customized'
Warning: Find Empty Args at 'slow_motion_fps'
Warning: Find Empty Args at 'resize'
Warning: Find Empty Args at 'crop'
Warning: Find Empty Args at 'use_sr_model'
INFO - FP16 mode switch success
INFO - Loaded v3.x HD model.

If i use a "scale_factor" of 0.5 (though the original video is only 960x540), it can work well. I'm using a RTX3060 Laptop with 6GB VRAM, I guess VRAM is enough. The same video can be use as input for 3.9 model with 1.0 scale factor.

I wonder how to solve this issue, thanks.

I'm using model 4.1, on this video, downloaded in what youtube-dl/yt-dlp identifies as format number 248 (the default if you exclude resolutions above 1080p).

There is a "cut" between scenes/shots at 15 frames past the 11-second mark in the input video. In the output video (with default settings), this turns into a downright disorienting transition. Looking frame by frame, the first two frames after the cut are heavily distorted.

Here are those two frames, plus the next two for context:

The culprit is the if block starting at line 227:

if ssim > 0.996:
    frame = read_buffer.get() # read a new frame
    if frame is None:
        break_flag = True
        frame = lastframe
    else:
        temp = frame
    I1 = torch.from_numpy(np.transpose(frame, (2,0,1))).to(device, non_blocking=True).unsqueeze(0).float() / 255.
    I1 = pad_image(I1)
    I1 = model.inference(I0, I1, args.scale)
    I1_small = F.interpolate(I1, (32, 32), mode='bilinear', align_corners=False)
    ssim = ssim_matlab(I0_small[:, :3], I1_small[:, :3])
    frame = (I1[0] * 255).byte().cpu().numpy().transpose(1, 2, 0)[:h, :w]

If I change the initial threshold from 0.996 to 1 (which I think disables the block because ssim can never be >1), the issue disappears: