In anomaly_detection.py line 105, you calculate the standard deviation of the mean value (which is a scalar) and it makes no sense. I think it should be window_std = np.std(window_elts)

hi, thanks for your code, when I read the code, I have a little question, when you calculate gp_loss. The original paper is written 1-lipschitz penalty should (gp_loss-1)^2, but your code is only gp_loss. I understand that you limit the loss to small and meet the requirements. I want to know, is the result obtained by the experiment? Looking forward to your reply.

• Python version:3.6
• Operating System:Ubuntu 18.04.5 LTS

Description

The dataset for ur code should have a column name as signal containing the signals and a column with name anomaly containing the true labels. But in the paper of TADGAN, e.g.,NAB(Art) dataset is only has two COLUMN ( timestamp and value). how can i use the dataset like NAB?

Hi,

Thanks for the translation of TadGan into pytorch. I'm trying to run the code present in this github just running:

python main.py

and I'm having the following error

Traceback (most recent call last):
  File "main.py", line 251, in <module>
    anomaly_detection.test(test_loader, encoder, decoder, critic_x)
  File "/content/drive/MyDrive/TadGAN/anomaly_detection.py", line 27, in test
    find_scores(y_true, y_predict)
  File "/content/drive/MyDrive/TadGAN/anomaly_detection.py", line 131, in find_scores
    precision = tp / (tp + fp)
ZeroDivisionError: division by zero

what could be problem causing this error?

In addition I'd try other dataset mentioned in the TadGan paper, like MSL for example. Is that possible with the code already present?

Thanks in advance.

Alessandro

Hi,

Thanks for making this available. I see that you just have one commit. Is this ready to use?

I will look more carefully at your paper, but does the technique work reasonably well for signals that very little to no periodic structure?

Thanks

Hi guys, awesome work!

I was exploring it and couldn't figure out how to use multivariate time series data as also told in your paper. Looking forward to the instructions.

Hi, Thanks for the translation of TadGan into pytorch. when I apply prune_false_positive program, I found some confusion about this, i think we should find every abnormal sequences that the max anomaly score ,and the max normal score, so , I modify the code of prune_false_positive, can you help me to determine that the code is correct, Looking forward to your reply

def prune_false_positive(is_anomaly, anomaly_score, change_threshold):
    #The model might detect a high number of false positives.
    #In such a scenario, pruning of the false positive is suggested.
    #Method used is as described in the Section 5, part D Identifying Anomalous
    #Sequence, sub-part - Mitigating False positives
    #TODO code optimization
    seq_details = []
    delete_sequence = 0
    start_position = 0
    end_position = 0
    anomaly_score = np.abs(anomaly_score)  # calculate standard deviations from the mean of the window
    max_seq_element = anomaly_score[0]
    for i in range(1, len(is_anomaly)):
        if is_anomaly[i] == 1 and is_anomaly[i-1] == 0:  # anomaly start
            start_position = i  # anomaly start position
            max_seq_element = anomaly_score[i]  # first anomaly score
        if is_anomaly[i] == 1 and is_anomaly[i-1] == 1 and anomaly_score[i] > max_seq_element:  # continuous anomaly, compare anomaly score
            max_seq_element = anomaly_score[i]
        if i+1 == len(is_anomaly) and is_anomaly[i] == 1:  # last is anomaly
            seq_details.append([start_position, i, max_seq_element, delete_sequence])
        elif is_anomaly[i] == 1 and is_anomaly[i+1] == 0:  # anomaly end
            end_position = i  # anomaly end postion
            seq_details.append([start_position, end_position, max_seq_element, delete_sequence])

    max_elements = list()
    max_elements.append(max(anomaly_score[is_anomaly==0]))  # normal data max score
    for i in range(0, len(seq_details)):
        max_elements.append(seq_details[i][2])

    max_elements.sort(reverse=True)
    max_elements = np.array(max_elements)
    change_percent = abs(max_elements[1:] - max_elements[:-1]) / max_elements[1:]

    # Appending 0 for the 1 st element which is not change percent
    delete_seq = np.append(np.array([0]), change_percent < change_threshold)

    # Mapping max element and seq details
    for i, max_elt in enumerate(max_elements):
        for j in range(0, len(seq_details)):
            if seq_details[j][2] == max_elt:
                seq_details[j][3] = delete_seq[i]

    for seq in seq_details:
        if seq[3] == 1: # Delete sequence
            is_anomaly[seq[0]:seq[1]+1] = [0] * (seq[1] - seq[0] + 1)
 
    return is_anomaly

After reading the code carefully, I found that the loss function of the encoder consists of three parts.like this loss_enc = mse + critic_score_valid_x - critic_score_fake_x i can understand 'mse',but i can not understand "critic_score_valid_x , critic_score_fake_x". 'critic_score_valid_x' and 'critic_score_fake_x' I think it is not related to the encoder. Is it replaced with 'critic_score_valid_z' and 'critic_score_fake_z'?

i have use the model in my own dataset ,But after a long time of training, the loss is still very high.Can you tell me how to reduce my loss?

DEBUG:root:critic x loss -30.026 critic z loss 0.416 encoder loss 1265.846 decoder loss 1235.256

Hi,

I have tried to run the code with the current setup (number of epochs is 30) but I get

File TadGAN/anomaly_detection.py", line 129, in find_scores precision = tp / (tp + fp) ZeroDivisionError: division by zero

Any ideas about what is going on ?

With Kind Regards, Roberto