Neural Tangent Generalization Attacks (NTGA)
ICML 2021 Video | Paper | Quickstart | Results | Unlearnable Datasets | Competitions | Citation
Overview
This is the repo for Neural Tangent Generalization Attacks, Chia-Hung Yuan and Shan-Hung Wu, In Proceedings of ICML 2021.
We propose the generalization attack, a new direction for poisoning attacks, where an attacker aims to modify training data in order to spoil the training process such that a trained network lacks generalizability. We devise Neural Tangent Generalization Attack (NTGA), a first efficient work enabling clean-label, black-box generalization attacks against Deep Neural Networks.
NTGA declines the generalization ability sharply, i.e. 99% -> 15%, 92% -> 33%, 99% -> 72% on MNIST, CIFAR10 and 2- class ImageNet, respectively. Please see Results or the main paper for more complete results. We also release the unlearnable MNIST, CIFAR-10, and 2-class ImageNet generated by NTGA, which can be found and downloaded in Unlearnable Datasets, and also launch learning on unlearnable data competitions. The following figures show one clean and the corresponding poisoned examples.
| Clean | NTGA |
|---|---|
 |
 |
Installation
Our code uses the Neural Tangents library, which is built on top of JAX, and TensorFlow 2.0. To use JAX with GPU, please follow JAX's GPU installation instructions. Otherwise, install JAX on CPU by running
pip install jax jaxlib --upgrade
Once JAX is installed, clone and install remaining requirements by running
git clone https://github.com/lionelmessi6410/ntga.git
cd ntga
pip install -r requirements.txt
If you only want to examine the effectiveness of NTGAs, you can download datasets here and evaluate with evaluate.py or any code/model you prefer. To use evaluate.py, you do not need to install JAX externally, instead, all dependencies are specified in requirements.txt.
Usage
NTGA Attack
To generate poisoned data by NTGA, run
python generate_attack.py --model_type fnn --dataset cifar10 --save_path ./data/
There are few important arguments:
--model_type: A string. Surrogate model used to craft poisoned data. One offnnorcnn.fnnandcnnstands for the fully-connected and convolutional networks, respectively.--dataset: A string. One ofmnist,cifar10, orimagenet.--t: An integer. Time step used to craft poisoned data. Please refer to main paper for more details.--eps: A float. Strength of NTGA. The default settings for MNIST, CIFAR-10, and ImageNet are0.3,8/255, and0.1, respectively.--nb_iter: An integer. Number of iteration used to generate poisoned data.--block_size: An integer. Block size of B-NTGA algorithm.--batch_size: An integer.--save_path: A string.
In general, the attacks based on the FNN surrogate have greater influence against the fully-connected target networks, while the attacks based on the CNN surrogate work better against the convolutional target networks. The hyperparameter t plays an important role in NTGA, which controls when an attack will take effect during the training process of a target model. With a smaller t, the attack has a better chance to affect training before the early stop.
Both eps and block_size influence the effectiveness of NTGA. Larger eps leads to stronger but more distinguishable perturbations, while larger block_size results in better collaborative effect (stronger attack) in NTGA but also induces both higher time and space complexities. If you encounter out-of-memory (OOM) errors, especially when using --model_type cnn, please try to reduce block_size and batch_size to save memory usage.
For ImageNet or another custom dataset, please specify the path to the dataset in the code directly. The original clean data and the poisoned ones crafted by NTGA can be found and downloaded in Unlearnable Datasets.
Evaluation
Next, you can examine the effectiveness of the poisoned data crafted by NTGA by calling
python evaluate.py --model_type densenet121 --dataset cifar10 --dtype NTGA \
--x_train_path ./data/x_train_cifar10_ntga_cnn_best.npy \
--y_train_path ./data/y_train_cifar10.npy \
--batch_size 128 --save_path ./figure/
If you are interested in the performance on the clean data, run
python evaluate.py --model_type densenet121 --dataset cifar10 --dtype Clean \
--batch_size 128 --save_path ./figures/
This code will also plot the learning curve and save it in --save_path ./figures/. The following figures show the results of DenseNet121 trained on the CIFAR-10 dataset. The left figure demonstrates the normal learning curve, where the network is trained on the clean data, and the test accuracy achieves ~93%. On the contrary, the figure on the right-hand side shows the remarkable result of NTGA, where the training accuracy is ~100%, but test accuracy drops sharply to ~37%, in other word, the model fails to generalize.
 |
 |
There are few important arguments:
--model_type: A string. Target model used to evaluate poisoned data. One offnn,fnn_relu,cnn,resnet18,resnet34, ordensenet121.--dataset: A string. One ofmnist,cifar10, orimagenet.--dtype: A string. One ofCleanorNTGA, used for figure's title.--x_train_path: A string. Path for poisoned training data. Leave it empty for clean data (mnist or cifar10).--y_train_path: A string. Path for training labels. Leave it empty for clean data (mnist or cifar10).--x_val_path: A string. Path for validation data.--y_val_path: A string. Path for validation labels.--x_test_path: A string. Path for testing data. The ground truth (y_test) is hidden. You can submit the prediction to Competitions.--epoch: An integer.--batch_size: An integer.--save_path: A string.
Visualization
How does the poisoned data look like? Is it truly imperceptible to a human? You can visualize the poisoned data and their normalized perturbations by calling
python plot_visualization.py --dataset cifar10 \
--x_train_path ./data/x_train_cifar10.npy \
--x_train_ntga_path ./data/x_train_cifar10_ntga_fnn_t1.npy \
--save_path ./figure/
The following figure shows some poisoned CIFAR-10 images. As we can see, they look almost the same as the original clean data. However, training on the clean data can achieve ~92% test accuracy, while training on the poisoned data the performance decreases sharply to ~35%.
Here we also visualize the high-resolution ImageNet dataset and find even more interesting results:
The perturbations are nearly invisible. The only difference between the clean and poisoned images is the hue!
There are few important arguments:
--dataset: A string. One ofmnist,cifar10, orimagenet.--x_train_path: A string. Path for clean training data.--x_train_ntga_path: A string. Path for poisoned training data.--num: An integer. Number of data to be visualized. The valid value is 1-5.--save_path: A string.
Results
Here we briefly report the performance of NTGA and two baselines (RFA and DeepConfuse) equipped with the FNN and CNN surrogates. NTGA(·) denotes an attack generated by NTGA with a hyperparameter t mentioned in NTGA Attack, and NTGA(best) represents the results of the best hyperparameter of the specific dataset and surrogate combination. NTGA(1) is the most imperceptible poisoned data which has the lowest-frequency perturbations.
As we can see, NTGA attack has remarkable transferability across a wide range of models, including Fully-connected Networks (FNNs) and Convolutional Neural Networks (CNNs), trained under various conditions regarding the optimization method, loss function, etc.
FNN Surrogate
| Target\Attack | Clean | RFA | DeepConfuse | NTGA(1) | NTGA(best) |
|---|---|---|---|---|---|
| Dataset: MNIST | |||||
| FNN | 96.26 | 74.23 | - | 3.95 | 2.57 |
| FNN-ReLU | 97.87 | 84.62 | - | 2.08 | 2.18 |
| CNN | 99.49 | 86.99 | - | 33.80 | 26.03 |
| Dataset: CIFAR-10 | |||||
| FNN | 49.57 | 37.79 | - | 36.05 | 20.63 |
| FNN-ReLU | 54.55 | 43.19 | - | 40.08 | 25.95 |
| CNN | 78.12 | 74.71 | - | 48.46 | 36.05 |
| ResNet18 | 91.92 | 88.76 | - | 39.72 | 39.68 |
| DenseNet121 | 92.71 | 88.81 | - | 46.50 | 47.36 |
| Dataset: ImageNet | |||||
| FNN | 91.60 | 90.20 | - | 76.60 | 76.60 |
| FNN-ReLU | 92.20 | 89.60 | - | 80.00 | 80.00 |
| CNN | 96.00 | 95.80 | - | 77.80 | 77.80 |
| ResNet18 | 99.80 | 98.20 | - | 76.40 | 76.40 |
| DenseNet121 | 98.40 | 96.20 | - | 72.80 | 72.80 |
CNN Surrogate
| Target\Attack | Clean | RFA | DeepConfuse | NTGA(1) | NTGA(best) |
|---|---|---|---|---|---|
| Dataset: MNIST | |||||
| FNN | 96.26 | 69.95 | 15.48 | 8.46 | 4.63 |
| FNN-ReLU | 97.87 | 84.15 | 17.50 | 3.48 | 2.86 |
| CNN | 99.49 | 94.92 | 46.21 | 23.89 | 15.64 |
| Dataset: CIFAR-10 | |||||
| FNN | 49.57 | 41.31 | 32.59 | 28.84 | 28.81 |
| FNN-ReLU | 54.55 | 46.87 | 35.06 | 32.77 | 32.11 |
| CNN | 78.12 | 73.80 | 44.84 | 41.17 | 40.52 |
| ResNet18 | 91.92 | 89.54 | 41.10 | 34.74 | 33.29 |
| DenseNet121 | 92.71 | 90.50 | 54.99 | 43.54 | 37.79 |
| Dataset: ImageNet | |||||
| FNN | 91.60 | 87.80 | 90.80 | 75.80 | 75.80 |
| FNN-ReLU | 92.20 | 87.60 | 91.00 | 80.00 | 80.00 |
| CNN | 96.00 | 94.40 | 93.00 | 79.00 | 79.00 |
| ResNet18 | 99.80 | 96.00 | 92.80 | 76.40 | 76.40 |
| DenseNet121 | 98.40 | 90.40 | 92.80 | 80.60 | 80.60 |
Unlearnable Datasets
Here we publicly release the poisoned datasets generated by NTGA. We provide 5 versions for each dataset. FNN(·) denotes an attack generated by NTGA from the FNN surrogate with a hyperparameter t. The best hyperparameter t is selected according to the empirical results. For the 2-class ImageNet, we choose n01560419 and n01910747 (bulbul v.s. jellyfish) from the original ImageNet dataset. Please refer to the main paper and supplementary materials for more details.
- MNIST
- FNN(best) = FNN(64)
- CNN(best) = CNN(64)
- CIFAR-10
- FNN(best) = FNN(4096)
- CNN(best) = CNN(8)
- ImageNet
- FNN(best) = FNN(1)
- CNN(best) = CNN(1)
Please support the project by hitting a star if you find this code or dataset is helpful for your research.
| Dataset\Attack | Clean | FNN(1) | FNN(best) | CNN(1) | CNN(best) |
|---|---|---|---|---|---|
| MNIST | Download | Download | Download | Download | Download |
| CIFAR-10 | Download | Download | Download | Download | Download |
| ImageNet | Download | Download | Download | Download | Download |
We do not provide the test label (y_test.npy) for each dataset since we launched Competitions. Nevertheless, if you are a researcher and need to use these data for academic purpose, we are willing to provide the complete dataset to you. Please send an email to [email protected]. Last but not least, using these data to participate in the competition defeats the entire purpose. So seriously, don't do that.
Competitions
We launch 3 competitions on Kaggle, where we are interested in learning from unlearnable MNIST, CIFAR-10, and 2-class ImageNet created by Neural Tangent Generalization Attack. Feel free to give it a shot if you are interested. We welcome people who can successfully train the model on the unlearnable data and overturn our conclusions.
| Kaggle Competitions | Unlearnable MNIST | Unlearnable CIFAR-10 | Unlearnable ImageNet |
|---|
For instance, you can create a submission file by calling:
python evaluate.py --model_type resnet18 --dataset cifar10 --dtype NTGA \
--x_train_path ./data/x_train_cifar10_unlearn.npy \
--y_train_path ./data/y_train_cifar10.npy \
--x_val_path ./data/x_val_cifar10.npy \
--y_val_path ./data/y_val_cifar10.npy \
--x_test_path ./data/x_test_cifar10.npy \
--save_path ./figure/
The results will be stored as y_pred_cifar10.csv. Please specify --x_test_path for the test data.
Citation
If you find this code or dataset is helpful for your research, please cite our ICML 2021 paper.
@inproceedings{yuan2021neural,
title={Neural Tangent Generalization Attacks},
author={Yuan, Chia-Hung and Wu, Shan-Hung},
booktitle={International Conference on Machine Learning},
pages={12230--12240},
year={2021},
organization={PMLR}
}
25 Sep 14, 2022
12 Dec 5, 2021
69 Dec 21, 2022
10 Sep 26, 2022
22 Dec 2, 2022
1 Nov 20, 2021
4 Nov 14, 2022
3 Mar 31, 2022
7 Aug 23, 2022
57 Nov 21, 2022
44 Dec 7, 2022
8 Dec 12, 2022
145 Sep 16, 2022
892 Dec 28, 2022
163 Nov 21, 2022
51 Dec 1, 2022
25 Dec 8, 2022
50 Dec 11, 2022
21 Oct 6, 2022