Official PyTorch code for WACV 2022 paper "CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows"

great job i am new about abnomal detection. what i want to know is that ,in actually scene ，when i have trained one model with bottle class in mvtec, i give another bottle image, should the model give the class (nomal or abnomal),and location the abnomal area like GT? could you give some demo code for that (load a model and predict single image)?

First of all, thank you for presenting your work as open source. When I tested the bottle class with the

python3 main.py --gpu 0 --pro -enc mobilenet_v3_large --dataset mvtec --action-type norm-test -inp INPUT --class-name CLASS --checkpoint PATH/FILE.PT

command, I got results. DET_AUROC: last: 100.00 max: 100.00 epoch_max: 0 SEG_AUROC: last: 98.93 max: 98.93 epoch_max: 0 SEG_AUPRO: last: 96.48 max: 96.48 epoch_max: 0

The question is how can i display heatmap on images.

First, it's a great job, thanks for your sharing. And, when I test bottle, train loss is very small, 10 epochs later, 0.06. But the test loss growth quickly, larger than 27000. At the 25th epoch, test loss is 6636421 while train loss 0.05. So, is this normal? I use default settings.

Then, a problem confused me a lot time. Without GTs, how can I chose thresholds of SEG? Both the auroc socres and F1 scores require gt_mask, but the fact is no gt information for new data with only normal samples. Could you give some ideas? thks!

Hi! I faced the RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient when tracing encoder (-enc mobilenet_v3_large) from the cflow-ad network with torch.jit.trace() function. I fed in trace function simplified version of test_meta_epoch function and single image (as tensor torch.Size([1, 3, 512, 512])). Script fails on line with _ = c.encoder(image).

Please explain to me, what should I change in code to resolve this task?

Traceback (most recent call last):
  File "convert_cflow-ad.py", line 187, in <module>
    traced_model = torch.jit.trace(test_model, image)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/jit/_trace.py", line 780, in trace
    traced = torch._C._create_function_from_trace(
  File "convert_cflow-ad.py", line 72, in test_model
    _ = c.encoder(image)  # BxCxHxW
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 443, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/Users/user/Projects/project/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 439, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient

Hi,

When I increase hyperparameter N, for test_meta_epoch only, to 8192 I get an fps increase of 300% and SUM scores which are identical to N=256. I do not fully understand the code yet, so could you answer if this is wise to do?

I'm a student who studies anomaly detection, Thank you for providing an interesting paper and well-designed code. I would like to ask a question about image resizing and evaluation methods in the localization task.

The input image is resized when training in your code. And the information about the size of the resized image is given when the main.py is executed like this. python3 main.py --gpu 0 --pro -inp 128 --dataset mvtec --class-name transistor The original size of "transistor" class in MVTec dataset is 1024, so I guess the size of this image is resized from 1024 to 128 here.

My question: Are the predicted mask image and GT mask image also resized when they are tested? (like 1024 -> 128 in "transistor" class) If so, would reducing the resolution of the image used for testing not improve the evaluation value?

I'd appreciate it if you could answer my questions.

Hello, running the command below.

python3 main.py --gpu 0 --pro -inp 512 --dataset mvtec --class-name bottle

Can you tell me why it takes more time to train than the existing Anomaly algorithm and why it takes more time to bring dataloaders? Thank you for sharing code

Is it possible to get the detection performance from the anomaly map (segmentation)? I mean like the way that get the top k highest anomaly score in the anomaly map and calculate the mean of them as the threshold for decide as anomaly or not.

Is there any misunderstanding or something that I miss about this kind of way to calculate detection performance instead of calculate with label (1,0) ?

Because I observe that you have a better performance in segmentation than the detection in this model maybe your model is flow-based (generative-based) so it have a better behavior in pixel level?

Hello @gudovskiy .

I was measuring the proc time of this work in my GPU. I noted that the proc time per iteration increases. I wonder if there are some accumulation formula that could decrease the speed. You can see next:

I am sharing the code I'm using below. I already saw that you have a similar method but I want to measure all the pipeline untill the scores are achieved

input_data = torch.rand_like(next(iter(loader))[0], requires_grad=False, device=torch.device('cuda'))

#Warmup phase 
....
# end of warmup

torch.cuda.synchronize()
    print("Start timing...")
    timings = []
    with torch.no_grad():
        for i in tqdm(range(500), 'Measuring inference speed...'):
            start_time = time.time()
            _ = encoder(input_data)  # BxCxHxW
            # test decoder
            for l, layer in enumerate(pool_layers):
                e = activation[layer]  # BxCxHxW
                B, C, H, W = e.size()
                S = H * W
                E = B * S
                if i == 0:  # get stats
                    height.append(H)
                    width.append(W)
                p = positionalencoding2d(P, H, W).to(c.device).unsqueeze(0).repeat(B, 1, 1, 1)
                c_r = p.reshape(B, P, S).transpose(1, 2).reshape(E, P)  # BHWxP
                e_r = e.reshape(B, C, S).transpose(1, 2).reshape(E, C)  # BHWxC
                decoder = decoders[l]
                FIB = E // N + int(E % N > 0)  # number of fiber batches
                for f in range(FIB):
                    if f < (FIB - 1):
                        idx = torch.arange(f * N, (f + 1) * N)
                    else:
                        idx = torch.arange(f * N, E)
                    c_p = c_r[idx]  # NxP
                    e_p = e_r[idx]  # NxC
                    z, log_jac_det = decoder(e_p, [c_p, ])
                    decoder_log_prob = get_logp(C, z, log_jac_det)
                    log_prob = decoder_log_prob / C  # likelihood per dim
                    test_dist[l] = test_dist[l] + log_prob.detach().cpu().tolist()

            test_map = [list() for p in pool_layers]
            for l, p in enumerate(pool_layers):
                test_norm = torch.tensor(test_dist[l], dtype=torch.double)  # EHWx1
                test_norm -= torch.max(test_norm)  # normalize likelihoods to (-Inf:0] by subtracting a constant
                test_prob = torch.exp(test_norm)  # convert to probs in range [0:1]
                test_mask = test_prob.reshape(-1, height[l], width[l])
                test_map[l] = F.interpolate(test_mask.unsqueeze(1), size=c.crp_size, mode='bilinear', align_corners=True).squeeze().numpy()

            score_map = np.zeros_like(test_map[0])
            for l, p in enumerate(pool_layers):
                score_map += test_map[l]
            score_mask = score_map
            super_mask = score_mask.max() - score_mask
            score_label = np.max(super_mask)

            torch.cuda.synchronize()
            end_time = time.time()
            timings.append(end_time - start_time)
            if i % 100 == 0:
                print("It: {} average time: {}".format(i, np.mean(timings) * 1000))

    print("Input shape {}".format(input_data.shape))
    print("Average time {}".format(np.mean(timings) * 1000))

Thanks

In the line 334 of train file, the detection AUROC is calculated using truth label and score label. Why the truth label is bolean and score label is a float? Im trying to replicate using the leather class and the score label values are between [0.97,2.7].

According to your paper, what we should use is log(pz) in equation 8. Why the log_prob is divided by the number of channels here?

https://github.com/gudovskiy/cflow-ad/blob/0d39848c988ad1c6b3c9b7f4bad40d52c1a75ceb/train.py#L77

Hello Denis, Thanks a lot for your proposed CFlow method! Do you have any plans to export the model to 'onnx' format? In my attempts to export, I had to use 'torch.jit.script' as an intermediate method to implement 'onnx' due to the presence of the loop in the forward code. But this seems to be difficult and the Freia library always fails to export. Sincerely looking forward to your reply!

Hello Denis, I have been studying and adjusting this great repository to fit my needs in defect detection. I have trained several models successfully(using the mobilenetv3_large backbone) and achieved good results. However, I have eliminated some parts of your code, including the snippets that calculate the "seg_threshold" parameter from the ground truth. Therefore, I am choosing it by hand (through trial&error), and although the results are OK, I think they can be further improved.

My questions are: 1- How can I choose a value for "seg_threshold" reliably in my given case? 2- What parameters do you recommend fine-tuning when training a new model (knowing that I have a good and balanced dataset of the same product but with different colours)? 3- My last question is related to exporting the model to "onnx" format, do you have any comments on how to achieve that? Do you plan on adding that capability? 4- When should I stop training ?

Thank you in advance, your work is truly inspiring.

Hello!

I wonder if the detection AUROC snd segmentation AUROC have the same behavior. In other words, when the model achieves the best seg_auroc,should it be when it has also the best detection_auroc? Or can it have the best det_auroc and the best seg_auroc in diferent epochs?

PDT: congrats for this work

In train.py, you normalize the scores according to:

test_map = [list() for p in pool_layers]
for l, p in enumerate(pool_layers):
    test_norm = torch.tensor(test_dist[l], dtype=torch.double)  # EHWx1
    test_norm-= torch.max(test_norm) # normalize likelihoods to (-Inf:0] by subtracting a constant
    test_prob = torch.exp(test_norm) # convert to probs in range [0:1]
    test_mask = test_prob.reshape(-1, height[l], width[l])
    test_mask = test_prob.reshape(-1, height[l], width[l])
    # upsample
    test_map[l] = F.interpolate(test_mask.unsqueeze(1),
        size=c.crp_size, mode='bilinear', align_corners=True).squeeze().numpy()
# score aggregation
score_map = np.zeros_like(test_map[0])
for l, p in enumerate(pool_layers):
    score_map += test_map[l]

This normalization is fine as long as it is done for only one map since this normalization function is monotonically increasing. By adding up the maps from the different layers, this makes no sense to me since the relative weighting of the score maps for aggregation (last line) depends on the test set or to be more precise on the maxima of the individual maps over the test set. Am I missing something here or is this normalization improper?