Toolkit for Building Robust ML models that generalize to unseen domains (RobustDG)
Divyat Mahajan, Shruti Tople, Amit Sharma
Privacy & Causal Learning (ICML 2020) | MatchDG: Causal View of DG (ICML 2021) | Privacy & DG Connection paper
For machine learning models to be reliable, they need to generalize to data beyond the train distribution. In addition, ML models should be robust to privacy attacks like membership inference and domain knowledge-based attacks like adversarial attacks.
To advance research in building robust and generalizable models, we are releasing a toolkit for building and evaluating ML models, RobustDG. RobustDG contains implementations of domain generalization algorithms and includes evaluation benchmarks based on out-of-distribution accuracy and robustness to membership privacy attacks. We will be adding evaluation for adversarial attacks and more privacy attacks soon.
It is easily extendable. Add your own DG algorithms and evaluate them on different benchmarks.
Installation
To use the command-line interface of RobustDG, clone this repo and add the folder to your system's PATH (or alternatively, run the commands from the RobustDG root directory).
Load dataset
Let's first load the rotatedMNIST dataset in a suitable format for the resnet18 architecture.
python data/data_gen_mnist.py --dataset rot_mnist --model resnet18 --img_h 224 --img_w 224 --subset_size 2000
Train and evaluate ML model
The following commands would train and evalute the MatchDG method on the Rotated MNIST dataset.
python train.py --dataset rot_mnist --method_name matchdg_ctr --match_case 0.0 --match_flag 1 --epochs 50 --batch_size 64 --pos_metric cos --match_func_aug_case 1
python train.py --dataset rot_mnist --method_name matchdg_erm --penalty_ws 0.1 --match_case -1 --ctr_match_case 0.0 --ctr_match_flag 1 --ctr_match_interrupt 5 --ctr_model_name resnet18 --epochs 25
python test.py --dataset rot_mnist --method_name matchdg_erm --penalty_ws 0.1 --match_case -1 --ctr_match_case 0.0 --ctr_match_flag 1 --ctr_match_interrupt 5 --ctr_model_name resnet18 --epochs 25 --test_metric acc
python test.py --dataset rot_mnist --method_name matchdg_ctr --match_case 0.0 --match_flag 1 --pos_metric cos --test_metric match_score
Demo
A quick introduction on how to use our repository can be accessed here in the Getting Started notebook.
If you are interested in reproducing results from the MatchDG paper, check out the Reproducing results notebook.
Roadmap
- Support for more domain generalization algorithms like CSD and IRM. If you are an author of a DG algorithm and would like to contribute, please raise a pull request here or get in touch.
- More evaluation metrics based on adversarial attacks, privacy attacks like model inversion. If you'd like to see an evaluation metric implemented, please raise an issue here.
Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.
25 Dec 28, 2022
6.9k Jan 5, 2023
14.5k Jan 7, 2023
1.4k Dec 27, 2022
1.4k Jan 6, 2023
11.6k Jan 2, 2023
65 Dec 20, 2022
3.1k Jan 6, 2023
35 Jan 2, 2023
2.3k Jan 5, 2023
47 Dec 30, 2022
8.9k Jan 9, 2023
84 Nov 25, 2022
8.1k Dec 30, 2022
19 Oct 3, 2022
2 Aug 23, 2022
23.3k Dec 31, 2022
1.6k Dec 31, 2022