Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" (ICLR 2022 Spotlight)

Overview

Anomaly-Transformer

Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" (ICLR 2022 Spotlight)

Unsupervised detection of anomaly points in time series is a challenging problem, which requires the model to learn informative representation and derive a distinguishable criterion. In this paper, we propose the Anomaly Transformer in these three folds:

  • An inherent distinguishable criterion as Association Discrepancy for detection.
  • A new Anomaly-Attention mechanism to compute the association discrepancy.
  • A minimax strategy to amplify the normal-abnormal distinguishability of the association discrepancy.

Get Started

  1. Install Python 3.6, PyTorch 1.4.0. (Note: Higher PyTorch version may cause error.)
  2. Download data. You can obtain four benchmarks from Tsinghua Cloud. All the datasets are well pre-processed. For the SWaT dataset, you can apply for it by following its official tutorial.
  3. Train and evaluate. We provide the experiment scripts of all benchmarks under the folder ./scripts. You can reproduce the experiment results as follows:
bash ./scripts/SMD.sh
bash ./scripts/MSL.sh
bash ./scripts/SMAP.sh
bash ./scripts/PSM.sh

Main Result

We compare our model with 15 baselines, including THOC, InterFusion, etc. Generally, Anomaly-Transformer achieves SOTA.

Citation

If you find this repo useful, please cite our paper.

@inproceedings{
xu2022anomaly,
title={Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy},
author={Jiehui Xu and Haixu Wu and Jianmin Wang and Mingsheng Long},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=LzQQ89U1qm_}
}

Contact

If you have any question, please contact [email protected], [email protected].

Comments
  • Wrongly valid on test_loader. Unfair evaluation.

    Wrongly valid on test_loader. Unfair evaluation.

    It seems like the validation is on test_loader while not vali_loader, which is unfair to some extent and would make the results a little bit different.

    https://github.com/thuml/Anomaly-Transformer/blob/72a71e5f0847bd14ba0253de899f7b0d5ba6ee97/solver.py#L196

    Moreover, directly using thre_loader to find thresholds would cause test datasets leakage, since thre_loader is built on test_data while not valid_data.

    https://github.com/thuml/Anomaly-Transformer/blob/72a71e5f0847bd14ba0253de899f7b0d5ba6ee97/solver.py#L254

    https://github.com/thuml/Anomaly-Transformer/blob/72a71e5f0847bd14ba0253de899f7b0d5ba6ee97/data_factory/data_loader.py#L66-L69

    opened by dqgdqg 4
  • Some questions about Prior-Association

    Some questions about Prior-Association

    Thanks for submitting such a great job, but I have some questions with prior-association of the code

    https://github.com/thuml/Anomaly-Transformer/blob/72a71e5f0847bd14ba0253de899f7b0d5ba6ee97/model/attn.py#L48-L49

    In this part, some mathematical processing is done on the sigma Could you please explain some reasons for doing this? I can not find any reference in your paper.

    opened by A-New-Page 4
  • About 3090

    About 3090

    Hello, looking at other questions, I found that you mentioned that your environment is 3090, but I found that the cuda version supported by 3090 is above 11. How did you solve it?

    opened by xuchunyu123 4
  • Some results inconsistent with the original paper of other methods

    Some results inconsistent with the original paper of other methods

    Hi ! Thx for your attention. In your paper, I found some results inconsistent with the original paper of other methods, like "OmniAnomaly" and "InterFusion". Is there some thing different in experiment detail?

    opened by Conearth 4
  • The model scored poorly after annotating the

    The model scored poorly after annotating the "detection adjustment" code

    Hi, this is an amazing job. Here I come across a small problem. On MSL dataset, the model performed good, looks like: ======================TEST MODE====================== Threshold : 0.0017330783803481142 pred: (73700,) gt: (73700,) pred: (73700,) gt: (73700,) Accuracy : 0.9853, Precision : 0.9161, Recall : 0.9473, F-score : 0.9314

    But after I annotated the "detection adjustment" code, the score was poorly, looks like: ======================TEST MODE====================== Threshold : 0.0017330783803481142 pred: (73700,) gt: (73700,) pred: (73700,) gt: (73700,) Accuracy : 0.8866, Precision : 0.1120, Recall : 0.0109, F-score : 0.0199

    And I'm sure only the "detection adjustment" code was annotated.

    Can you help me out of this problem? thx.

    opened by Conearth 4
  • Why the optimizer.step() write twice?

    Why the optimizer.step() write twice?

    https://github.com/thuml/Anomaly-Transformer/blob/bfe075e4f3a0be789f168b2aeee7a4ce30482ce5/solver.py#L189-L192 When the first optimizer.step() execute all the gradient relate to loss1 will update,but some variable in loss2 are common in loss1.So this maybe cause some problem.

    opened by San-ctuary 3
  • 运行时卡在模型实例化

    运行时卡在模型实例化

    运行时会在实例化solver卡住,具体是在 self.build_model() -> self.model = AnomalyTransformer(win_size=self.win_size, enc_in=self.input_c, c_out=self.output_c, e_layers=3) -> self.encoder = Encoder(...) 并没有报错,只是运行到这里时就卡住了,不知道是什么原因导致的? python 3.6, cuda 10.1, pytorch 1.4.0 GPU型号A100

    opened by hhhercules 2
  • RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

    RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

    Hello, When I run bash ./scripts/MSL.sh, I get the following error: Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512, 55]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

    opened by Naji-Najari 2
  • loss2.backward():runtime error

    loss2.backward():runtime error

    Thanks for the open source code! And there is a problem confuing me. When i running the code with [python main.py],there is a Pytorch error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512, 38]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

    Code error is located in solver.py,line 188 //Minimax strategy loss1.backward(retain_graph=True) self.optimizer.step() loss2.backward() self.optimizer.step()

    when i change the code to loss1.backward(retain_graph=True) //self.optimizer.step() loss2.backward() self.optimizer.step()

    It worked! I wonder if the first [self.optimizer.step()] should be annotated,if not, how to resolve the error. Thank you.

    opened by Apollo-asleep 2
  • solver.py use gt before auc test

    solver.py use gt before auc test

    good work,i have a simple question. solver.py line 338 # detection adjustment in the detection adjustment stage, the code use gt to adjust pred result.But gt can't be used before auc test. Why?Is that ok to do this?

    opened by Apollo-asleep 1
  • About NeurIPS-TS dataset

    About NeurIPS-TS dataset

    I want to know the details about what you experimented using NeurIPS-TS dataset. According to the paper, It includes univariate time series which have total length of 50,000. I couldn't find any information about how to generate or sample that train, test samples. Could you explain more detailed design of your experiment using this dataset? Is there any future plans to update the related codes?

    opened by DRMBSGN 1
  • questions on validation and early stop

    questions on validation and early stop

    hello, when i trained the model with the PSM data, i found that it is difficult to escape from early stop. is it not easy to get the best vali loss1 score and vali loss2 score at the same time so that training process stop so early? vali loss1 score= -rec_loss + AssDiss vali loss2 score= -rec_loss - AssDiss

    opened by k1e3v1i4n 0
  • How should I reproduce the performance of Anomaly-Transformer on SWaT

    How should I reproduce the performance of Anomaly-Transformer on SWaT

    I want to reproduce the performance of Anomaly-Transformer on the SWaT dataset, but I don't know how the SWaT dataset is preprocessed. Can you help me? Thank you!

    opened by LyBjer 0
  • F1 and F1-PA

    F1 and F1-PA

    Hello, I've encountered the exactly same issue as previous issue, and want to ask your opinion.

    To provide more, Along with the code without point adjustment (commenting out the PA part as in previous issue), I personally got the following result:

    | Dataset | F1-PA | F1-PA (paper) | F1 | | ----------- | ----------- | ----------- | ----------- | | MSL | 0.9500 | 0.9359 | 0.0209 | | PSM | 0.9750 | 0.9789 | 0.0217 | | SMAP | 0.9636 | 0.9669 | 0.0189| | SMD | 0.8944 | 0.9233 | 0.0201|

    Although I agree that the F1-PA algorithm has practical justification (abnormal time point will cause an alert and further make the whole segment noticed in real-world applications.), (1) F1 seems to aggravate too much, and (2) AAAI paper raises concern about F1-PA metrics: even random guessing can achieive high F1-PA depending on data distribution.

    I want to ask your opinion on these results. Thanks in advance.

    opened by carrtesy 0
  • Some issues about baseline and SOTA methods

    Some issues about baseline and SOTA methods

    Many thanks for your excellent work!

    I would like to know about the details of the baselines and SOTA method codes. I can not find any official code for the LSTM, LSTM-VAE, BeatGAN, and THOC methods. Can you provide some links and code repo for us to reproduce?

    Thanks again! Look forward to your kind reply!!

    opened by yyysjz1997 0
  • questions on validation set and threshold selection algorithm

    questions on validation set and threshold selection algorithm

    Hello,

    I have a question regarding your work.

    1. How validation set are selected here? from dataloader code, validation set seems to be test set. Did I get the codes right?
    2. Anomaly Transformer's thresholding mechanism. Is the model using test dataset rather than validation set to set the threshold? From Appendix H, validation set (which is consisted of normal data only) is used to pick an appropriate threshold. However, the code seems to utilize test dataset for thresholding.
      Did I miss some points?

    Thanks a lot for your answer in advance.

    opened by carrtesy 2
Owner
THUML @ Tsinghua University
Machine Learning Group, School of Software, Tsinghua University
THUML @ Tsinghua University
USAD - UnSupervised Anomaly Detection on multivariate time series

USAD - UnSupervised Anomaly Detection on multivariate time series Scripts and utility programs for implementing the USAD architecture. Implementation

null 116 Jan 4, 2023
Code for the paper "TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks"

TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks This is a Python3 / Pytorch implementation of TadGAN paper. The associated

Arun 92 Dec 3, 2022
Official PyTorch code for WACV 2022 paper "CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows"

CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows WACV 2022 preprint:https://arxiv.org/abs/2107.1

Denis 156 Dec 28, 2022
Time-series-deep-learning - Developing Deep learning LSTM, BiLSTM models, and NeuralProphet for multi-step time-series forecasting of stock price.

Stock Price Prediction Using Deep Learning Univariate Time Series Predicting stock price using historical data of a company using Neural networks for

Abdultawwab Safarji 7 Nov 27, 2022
[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks, ICLR 2021 (Spotlight) Demo | Paper [NEW!] Time to play with our interac

Shengyu Zhao 373 Jan 2, 2023
[ICLR 2021 Spotlight Oral] "Undistillable: Making A Nasty Teacher That CANNOT teach students", Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, Zhangyang Wang

Undistillable: Making A Nasty Teacher That CANNOT teach students "Undistillable: Making A Nasty Teacher That CANNOT teach students" Haoyu Ma, Tianlong

VITA 71 Dec 28, 2022
Code for "The Intrinsic Dimension of Images and Its Impact on Learning" - ICLR 2021 Spotlight

dimensions Estimating the instrinsic dimensionality of image datasets Code for: The Intrinsic Dimensionaity of Images and Its Impact On Learning - Phi

Phil Pope 41 Dec 10, 2022
Official Implementation of 'UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers' ICLR 2021(spotlight)

UPDeT Official Implementation of UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers (ICLR 2021 spotlight) The

hhhusiyi 96 Dec 22, 2022
Deep Anomaly Detection with Outlier Exposure (ICLR 2019)

Outlier Exposure This repository contains the essential code for the paper Deep Anomaly Detection with Outlier Exposure (ICLR 2019). Requires Python 3

Dan Hendrycks 464 Dec 27, 2022
PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short-Term Transformer for Online Action Detection".

Long Short-Term Transformer for Online Action Detection Introduction This is a PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short

null 77 Dec 16, 2022
Imposter-detector-2022 - HackED 2022 Team 3IQ - 2022 Imposter Detector

HackED 2022 Team 3IQ - 2022 Imposter Detector By Aneeljyot Alagh, Curtis Kan, Jo

Joshua Ji 3 Aug 20, 2022
A Pytorch implement of paper "Anomaly detection in dynamic graphs via transformer" (TADDY).

TADDY: Anomaly detection in dynamic graphs via transformer This repo covers an reference implementation for the paper "Anomaly detection in dynamic gr

Yue Tan 21 Nov 24, 2022
This codebase is the official implementation of Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS2021, Spotlight)

Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization This codebase is the official implementation of Test-Time Classifier A

null 47 Dec 28, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 274 Jan 5, 2023
[NeurIPS 2021 Spotlight] Aligning Pretraining for Detection via Object-Level Contrastive Learning

SoCo [NeurIPS 2021 Spotlight] Aligning Pretraining for Detection via Object-Level Contrastive Learning By Fangyun Wei*, Yue Gao*, Zhirong Wu, Han Hu,

Yue Gao 139 Dec 14, 2022
Demo project for real time anomaly detection using kafka and python

kafkaml-anomaly-detection Project for real time anomaly detection using kafka and python It's assumed that zookeeper and kafka are running in the loca

Rodrigo Arenas 36 Dec 12, 2022
CLEAR algorithm for multi-view data association

CLEAR: Consistent Lifting, Embedding, and Alignment Rectification Algorithm The Matlab, Python, and C++ implementation of the CLEAR algorithm, as desc

MIT Aerospace Controls Laboratory 30 Jan 2, 2023
null 190 Jan 3, 2023
graph-theoretic framework for robust pairwise data association

CLIPPER: A Graph-Theoretic Framework for Robust Data Association Data association is a fundamental problem in robotics and autonomy. CLIPPER provides

MIT Aerospace Controls Laboratory 118 Dec 28, 2022