Hi~ Thank you for the wonderful work. I find your work on the alphamatting.com, and I am very glad that you open source your code so quickly. However, I do not find the training code in the repository. Would you like also open source the training code?

Hello!

I am trying to apply the FBA Matting method to removing green screen from images (portraits of people shot on a green screen). Overall, it works well but it leaves some green pixels mixed with the hair (I attach here an example).

I can see 2 possible reasons for this issue:

1. As the result heavily depends on the quality of trimaps, better trimaps are needed. I am currently generating trimaps using DeepLabV3. I tried to adjust some parameters (conf_threshold, dilation), it got better but not perfect yet. Do you know if there are any other parameters that should be tuned or models I can try for trimaps creation?
2. This model was trained on Adobe Image Matting Dataset that does not contain portraits with green screen. Do you think it would make a big difference if I retrained this model on my custom dataset of green screen portraits?

Thank you in advance for any help! I would truly appreciate it!

Thanks for the great work! Your results are really impressive. I noticed that you suggested using laplacian loss for training. May I ask for the implementation details of it?

Hi, thank you for sharing your code and paper!

Recently, I’m reproducing your work. I have some questions about the training and test details.

1. In the paper, it is said that RAdam optimizer is used with momentum 0.9 and weight decay 0.0001. But I didn’t found “momentum” parameter in the official RAdam optimzier code. Did you modify the official code or just set beta1 to 0.9?

2. About the weight decay, there are two descriptions in the paper. a) weight decay 0.0001 in RAdam optimizer, b) weight decay of 0.005, 1e-5 to convolutional weights and GB parameters. How did you set them in your code? I tried to set the weight decay in the optimizer to 0.0001, and add L2 loss to conv weights and GN weights & bias with weight 0.005 and 1e-5. Because I think L2 loss here is equivalent to weight decay. Is it same as yours?

3. The input resolution for training patches. The training patches of size 640, 480, 320, are randomly cropped during training. After that, did your resize them to a certain size? If not, how to train the model with batch size 6?

4. The input channel for test. In section 3.6, “During inference, the full-resolution input images and trimaps are concatenated as 4-channel input and fed into the network.” But in the code, 11-channel input is used. Is it a typo?

5. About the re-estimated foreground. I tried to re-estimate the training foreground images, but only succeeded in 411 of 431 fg. 20 of them failed to be optimized. Did you have the same problem? Besides, when calculating the alpha * foreground error during test, are the ground-truth foreground images re-estimated by closed form?

Thank you !

In paper 3.1 :

First, we increase the number of input channels from 3 to 9 to allow for the extra trimap. We encode the trimap using Gaussian blurs of the definite foreground and background masks at three different scales (in a similar way to the method of [19] in interactive segmentation). This encoding differs from existing approaches in deep image matting, as they usually encode the trimap as a single channel with value 1 if foreground, 0.5 for unknown and 0 for background.

I know the output channels is 7 (a=1, F=3, B=3) But why the input channels is 9? In you code,I saw the input are image and trimap,then input channels will be 4. So,why the input channels is 9 in paper 3.1?

https://patents.google.com/patent/US11004208B2/en

patent seems to cover the idea of user interaction to create binary matte -> trimap -> final matte and every variation of that.

in fba_decoder, there is a fba_fusion() function. Is the loss function calculated by the output of the fba_fusion()? or the output of the conv_up4()->clamp and sigmoid

Hi, I have read your paper on arxiv. In your paper, the color-spill problem was proposed and an example was given in Fig.2. I tried to reproduce your result in Fig.2 using the same foreground in Composition-1k dataset. But the composite image is totally different from yours. Besides, I tried to use Closed-form method to estimate a new foreground, but got a different one. My new fg is black in the area where alpha equals to zero.

So how did you composite the original foreground onto the background? And how did you re-estimate the new foreground? Can you please share your code?

Thanks!

https://github.com/MarcoForte/FBA_Matting/blob/master/networks/models.py#L208, here conv_out[-6] is conv_out[0], which is https://github.com/MarcoForte/FBA_Matting/blob/master/networks/models.py#L99 the actual image, so we're concatenating the image twice basically

am i correct or am i missing something?

I use Code for foreground extension for foreground prediction on adobe dataset. But when processing some pictures, there will be insufficient memory.

I noticed that you mentioned that a custom multi-level solver written in C. Could you please provide this code？

The effect of your proposed model and training method is really amazing. “This, combined with longer training at 45 epochs and TTA, has a bigger impact than the choices of loss functions”. It seems TTA also plays an important role. So I want to konw the detail of TTA, such as the example code or related info. Looking forward to your reply!

Using FBA Matting.ipynb in Gooble Colab, at the source code below,

class Args:
  encoder = 'resnet50_GN_WS'
  decoder = 'fba_decoder'
  weights = 'FBA.pth'
args=Args()
try:
    model = build_model(args)
except:
    !gdown  https://drive.google.com/uc?id=1T_oiKDE_biWf2kqexMEN7ObWqtXAzbB1
    model = build_model(args)

the following error occurred.

modifying input layer to accept 11 channels
Downloading...
From: https://drive.google.com/uc?id=1T_oiKDE_biWf2kqexMEN7ObWqtXAzbB1
To: /content/drive/MyDrive/FBA Matting/FBA.pth
100% 139M/139M [00:01<00:00, 81.2MB/s]
modifying input layer to accept 11 channels
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/torch/serialization.py](https://localhost:8080/#) in _check_seekable(f)
    307     try:
--> 308         f.seek(f.tell())
    309         return True

AttributeError: 'Args' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
14 frames
AttributeError: 'Args' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
AttributeError: 'Args' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/torch/serialization.py](https://localhost:8080/#) in raise_err_msg(patterns, e)
    302                                 + " Please pre-load the data into a buffer like io.BytesIO and"
    303                                 + " try to load from it instead.")
--> 304                 raise type(e)(msg)
    305         raise e
    306 

AttributeError: 'Args' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

Please tell me the cause of the error and how to fix it.

Hello, thnak you for your great work I am trying to make the training code. So i started by making alpha loss, composition loss and regression loss, and i used radam optimizer RAdam(group_weight(net, 1e-5, 1e-5, 0.0001)) as you mentionned. To test the convegenve and evrithing is right in my code i took only 2 images with hair from adobe dataset and start training with these 2 images. But it's seems the loss is not converging, however when using the classical adam optimizer i got convergence in few iterations. the group_weight function seems dealing with parameters of the resnet encoder, i think i should use a pretrained resnet. could you share the pretrained resnet you used and if possible the code to load it in matting module. thanks in advance

In paper you wrote:

To further increase the dataset diversity, we randomly composite a new foreground object with 50% probability, as in [34].

Could you please provide more details on how exactly you did it. Did you compose foregrounds before or after solving colors in areas where alpha is 0. And did you do something (maybe mean?) to save correct foreground colors during composition where both fg's alpha is 0?

I noticed that if i compose solved foregrounds it results to dominating "background" fg colors where alpha is 0.

Here is an example of compositioning 2 foregrounds in different order. Look at "orange"/green background.

And here is what i've got if solve combined fg one more time (but as i think, double solving could lead to incorrect colors in semitransparent regions)

As far as i understand, here https://github.com/MarcoForte/FBA_Matting/blob/master/networks/models.py#L230 resnet backbone will return such feature maps: [original_image, conv_bn_relu out, layer1 out, layer2 out, layer3 out, layer4 out]

In the decoder https://github.com/MarcoForte/FBA_Matting/blob/master/networks/models.py#L350 you concatenate: (x, conv_out[-6][:, :3], img, two_chan_trimap).

But conv_out[-6][:, :3] is the same as img. Are you sure that image should be concatenated twice?

Hi in the repo https://github.com/huochaitiantang/pytorch-deep-image-matting/

the alpha prediction loss is calculated only for pixels that are unknown in trimap

""" wi = torch.zeros(trimap.shape) wi[trimap == 128] = 1. t_wi = wi.cuda() unknown_region_size = t_wi.sum() """

while training are you doing something similar or is it for all pixels?

Sorry couldn't find this in paper or code