Behavioral Testing of Clinical NLP Models
This repository contains code for testing the behavior of clinical prediction models based on patient letters. For a detailed description of the testing framework see our paper What Do You See in this Patient? Behavioral Testing of Clinical NLP Models.
Usage
Install requirements: pip install -r requirements.txt
Run main.py, e.g. for diagnosis prediction test on gender, age and ethnicity:
python main.py
--test_set_path ./path_to_test_set
--model_path bvanaken/CORe-clinical-diagnosis-prediction
--task diagnosis
--shift_keys gender,age,ethnicity
--save_dir ./results
--gpu False
| Parameter | Description |
|---|---|
| test_set_path | Path to original test set file |
| model_path | Path to model or Huggingface model hub checkpoint |
| task | Current options: diagnosis, mortality |
| shift_keys | Which patient characteristics to test. Current options: age, gender, ethnicity, weight, intersectional (gender + ethnicity) |
| save_dir | Directory to save results, default: "./results" |
| gpu | Whether to use a gpu during inference or not, default: False |
Using Non-Transformer models
The framework currently focuses on testing Transformer-based models. However, it is easy to extend it to any other prediction model. To do so, simply create a new class implementing the Predictor interface and add it to the TASK_MAP in main.py.
Cite
@inproceedings{vanAken2021,
author = {Betty van Aken and
Sebastian Herrmann and
Alexander Löser},
title = {What Do You See in this Patient? Behavioral Testing of Clinical NLP Models},
booktitle = {Bridging the Gap: From Machine Learning Research to Clinical Practice,
Research2Clinics Workshop @ NeurIPS 2021},
year = {2021}
}
165 Dec 29, 2022
1 Jan 5, 2022
195 Dec 22, 2022
8.5k Jan 2, 2023
74 Dec 15, 2022
1 Sep 1, 2022
3 Apr 26, 2022
184 Dec 11, 2022
68 Jul 18, 2022
80 Dec 8, 2022
67 Dec 20, 2022
101 Dec 19, 2022
605 Jan 2, 2023
76 Dec 22, 2022
180 Dec 19, 2022
1 Jan 10, 2022
26 Jul 13, 2022
109 Dec 14, 2022
71 Oct 25, 2022