Easy and comprehensive assessment of predictive power, with support for neuroimaging features

Related tags

Deep Learning machine-learning scikit-learn cross-validation report easy-to-use neuroimaging pattern-recognition nilearn tractography structural-imaging anatomical-mri functional-connectivity tract-based-statistics resting-state
Overview

docs/logo_neuropredict.png

https://landscape.io/github/raamana/neuropredict/master/landscape.svg?style=flat https://api.codacy.com/project/badge/Grade/501e560b8a424562a1b8f7cd2f3cadfe https://travis-ci.org/raamana/neuropredict.svg?branch=master

Documentation: https://raamana.github.io/neuropredict/

News

  • As of v0.6, neuropredict now supports regression applications i.e. predicting continuous targets (in addition to categorical classes), as well as allow you to regress out covariates / confounds within the nested-CV (following all the best practices). Utilizing this feature requires the input datasets be specified in the pyradigm data structures: code @ https://github.com/raamana/pyradigm, docs @ https://raamana.github.io/pyradigm/. Check the changelog below for more details.

Older news

  • neuropredict can handle missing data now (that are encoded with numpy.NaN). This is done respecting the cross-validation splits without any data leakage.

Overview

On a high level,

roleofneuropredict

On a more detailed level,

roleofneuropredict

Long term goals

neuropredict, the tool, is part of a broader initiative described below to develop easy, comprehensive and standardized predictive analysis:

longtermgoals

Citation

If neuropredict helped you in your research in one way or another, please consider citing one or more of the following, which were essential building blocks of neuropredict: - Pradeep Reddy Raamana. (2017, November 18). neuropredict: easy machine learning and standardized predictive analysis of biomarkers (Version 0.4.5). Zenodo. http://doi.org/10.5281/zenodo.1058993 - Raamana et al, (2017), Python class defining a machine learning dataset ensuring key-based correspondence and maintaining integrity, Journal of Open Source Software, 2(17), 382, doi:10.21105/joss.00382

Change Log - version 0.6

  • Major feature: Ability to predict continuous variables (regression)
  • Major feature: Ability to handle confounds (regress them out, augmenting etc)
  • Redesigned the internal structure for easier extensibility
  • New CVResults class for easier management of a wealth of outputs generated in the Classification and Regression workflows
  • API access is refreshed and easier

Change Log - version 0.5.2

  • Imputation of missing values
  • Additional classifiers such as XGBoost, Decision Trees
  • Better internal code structure
  • Lot more tests
  • More precise tests, as we vary number of classes wildly in test suites
  • several bug fixes and enhancements
  • More cmd line options such as --print_options from a previous run
Comments
  • ImportError: cannot import name 'MultiDatasetClassify'

    ImportError: cannot import name 'MultiDatasetClassify'

    Hi,

    When I try to run neuropredict on the command line using neuropredict -m ~/NeuroPredict/meta_data.csv -o ~/NeuroPredict/results/ --user_feature_paths ~/NeuroPredict/face/ ~/NeuroPredict/cnlg/ ~/NeuroPredict/glmn/ ~/NeuroPredict/cnsr/ ~/NeuroPredict/bimndir/ ~/NeuroPredict/bimnemo/ ~/NeuroPredict/msep/ -t 0.8 -n 20 -g 'light' I get the error that ImportError: cannot import name 'MultiDatasetClassify' from 'pyradigm.multiple' It appears that the multiple.py file that I've downloaded does not match the current version that is in your pyradigm repository. I installed it just now though, using pip3 install neuropredict. Do you know why I am getting this problem?

    Thanks

    opened by cpappas18 13
  • Add new classifier: need of probability output?

    Add new classifier: need of probability output?

    Hi @raamana ,

    I would like to add LinearSVC classifier based on liblinear implementation. Does the current implementation of neuropredict need that predictions are based on probability values? Because, LinearSVC doesn't allow prediction of probabilities.

    opened by mattvan83 12
  • Implement option to let user select number of features to select

    Implement option to let user select number of features to select

    To do:

    • [x] add an argument: --num_features
    • [x] describe the argument thoroughly in help text
    • [x] checks on the input types
    • [x] checks on the input ranges
    • [ ] checks on whether all the feature sets have the same number of features
    • [ ] make the new argument interface with all the other methods, if need be.
    • [ ] add option to pass on subset of feature names to visualize.feature_importance_map

    We can add/remove as we realize them.

    enhancement 
    opened by raamana 12
  • How to add multiple features?

    How to add multiple features?

    Hi Pradeep,

    I can see the file structure for adding a single feature is:

    subject1/features.txt subject2/features.txt subject3/features.txt subject4/features.txt

    Where the feature is a vector in each case. To clarify, if I wanted to add a second feature, would this be a second column in the features.txt file of each participant?

    Thanks,

    John

    question 
    opened by johnaeanderson 10
  • Classification blocked with multiple 1-dimensional features

    Classification blocked with multiple 1-dimensional features

    Hello,

    I have problems when using multiple 1-D features with the following command:

          neuropredict -m meta_data.csv \
      -d mask_WMpet_av45_early.ero1.25mm.csv mask_WMpet_av45_early.ero1.5mm.csv \
      -o outdir -t 0.8 -n 250 -k 'all' \
     --gs_level 'exhaustive' --classifier "LinearSVC"

    The CV trial were well launched for the first subgroup CN,MCI for both 1-D features but it seemed that at the end of CV trials the process got stuck at this stage:

    >

>  Python 3.7.3
> > SGE recognized, job set up with 40 slots.
> > Positive class inferred for AUC calculation: CN
> > Running neuropredict 0.5+34.g220af55.dirty
> > 
> > Requested features for analysis:
> > get_data_matrix from /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/mask_WMpet_av45_early.ero1.25mm.csv
> > get_data_matrix from /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/mask_WMpet_av45_early.ero1.5mm.csv
> > Ignoring imputation strategy chosen, as no missing data were found!
> > 
> > Data import is done.
> > 
> > 
> > Requested processing for the following subgroups:
> > DIS,CN
> > 
> > --------------------------------------------------------------------------------
> > Processing subgroup : DIS,CN (1/1)
> > --------------------------------------------------------------------------------
> > SGE recognized, job set up with 40 slots.
> > Training percentage      : 0.8
> > Number of CV repetitions : 250
> > Classifier chosen        : linearsvc
> > Feature selection chosen : variancethreshold
> > Level of grid search     : exhaustive
> > Number of processors     : 40
> > Saving the results to 
> >   /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/linearsvc/binary_WMmasks/CN_DIS
> > 
> > -------------------------
> > All datasets contain:
> >  
> > 101 samples, 2 classes, 1 features
> > Class  CN : 71 samples
> > Class DIS : 30 samples
> > -------------------------
> > 
> > Estimated chance accuracy : 0.500
> > 
> > Different classes in the training set are stratified to match the smallest class!
> > Parallelizing the repetitions of CV with 40 processes ...
> > CV trial 52     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525 	 weighted AUC: 0.7482
> > CV trial 78     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 	 weighted AUC: 0.8227
> > CV trial 50     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5993 	 weighted AUC: 0.6596
> > CV trial 72     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6082 	 weighted AUC: 0.6596
> > CV trial 46     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7021
> > CV trial 58     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8475
> > CV trial 38     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 	 weighted AUC: 0.7518
> > CV trial 16     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 	 weighted AUC: 0.7801
> > CV trial 12     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8794
> > CV trial 0      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738 	 weighted AUC: 0.6950
> > CV trial 68     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7057 	 weighted AUC: 0.8369
> > CV trial 14     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8794
> > CV trial 22     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8333
> > CV trial 6      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 	 weighted AUC: 0.7234
> > CV trial 48     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8298
> > CV trial 10     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7624
> > CV trial 64     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8191
> > CV trial 20     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8688
> > CV trial 4      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8688
> > CV trial 56     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 	 weighted AUC: 0.8262
> > CV trial 70     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8440
> > CV trial 74     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7553
> > CV trial 36     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8050
> > CV trial 2      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.9043
> > CV trial 54     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8830
> > CV trial 24     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 	 weighted AUC: 0.9043
> > CV trial 76     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 	 weighted AUC: 0.9220
> > CV trial 42     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 	 weighted AUC: 0.8759
> > CV trial 26     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.9078
> > CV trial 62     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8103
> > CV trial 66     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418 	 weighted AUC: 0.7624
> > CV trial 60     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8511 	 weighted AUC: 0.8972
> > CV trial 72     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6294 	 weighted AUC: 0.6418
> > CV trial 40     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.9149
> > CV trial 52     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7340
> > CV trial 16     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7411
> > CV trial 46     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 	 weighted AUC: 0.6489
> > CV trial 38     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6507 	 weighted AUC: 0.7376
> > CV trial 78     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 	 weighted AUC: 0.8014
> > CV trial 50     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6099 	 weighted AUC: 0.6454
> > CV trial 58     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8298
> > CV trial 6      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.6879
> > CV trial 14     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8688
> > CV trial 36     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844 	 weighted AUC: 0.8156
> > CV trial 18     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8014
> > CV trial 0      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904 	 weighted AUC: 0.6454
> > CV trial 20     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8582
> > CV trial 68     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 	 weighted AUC: 0.7979
> > CV trial 4      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8511 	 weighted AUC: 0.8440
> > CV trial 74     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.7358
> > CV trial 56     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8050
> > CV trial 34     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5816 	 weighted AUC: 0.7518
> > CV trial 12     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8723
> > CV trial 66     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7376
> > CV trial 70     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844 	 weighted AUC: 0.8227
> > CV trial 64     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 	 weighted AUC: 0.8121
> > CV trial 24     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8794
> > CV trial 76     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 	 weighted AUC: 0.9007
> > CV trial 79     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5904 	 weighted AUC: 0.7021
> > CV trial 53     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7766
> > CV trial 2      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8936
> > CV trial 73     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.9255
> > CV trial 54     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8759
> > CV trial 62     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8014
> > CV trial 17     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 	 weighted AUC: 0.7624
> > CV trial 22     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8191
> > CV trial 39     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 	 weighted AUC: 0.7695
> > CV trial 60     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 	 weighted AUC: 0.9007
> > CV trial 51     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8404 	 weighted AUC: 0.8227
> > CV trial 26     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.9007
> > CV trial 42     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8546
> > CV trial 48     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8121
> > CV trial 59     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8014
> > CV trial 69     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7695
> > CV trial 47     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7695
> > CV trial 21     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 	 weighted AUC: 0.9220
> > CV trial 28     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8316 	 weighted AUC: 0.9007
> > CV trial 37     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8475
> > CV trial 57     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8582
> > CV trial 1      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8440
> > CV trial 40     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.9149
> > CV trial 30     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8582
> > CV trial 18     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 	 weighted AUC: 0.7589
> > CV trial 65     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7837
> > CV trial 75     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8617
> > CV trial 25     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7943
> > CV trial 13     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8121
> > CV trial 79     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 	 weighted AUC: 0.6915
> > CV trial 61     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 	 weighted AUC: 0.7766
> > CV trial 71     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8156
> > CV trial 32     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8316 	 weighted AUC: 0.9291
> > CV trial 3      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8440
> > CV trial 53     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7411
> > CV trial 55     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8298
> > CV trial 15     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8582
> > CV trial 23     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7872
> > CV trial 34     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5284 	 weighted AUC: 0.7376
> > CV trial 59     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.7713
> > CV trial 43     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.7766
> > CV trial 77     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7305
> > CV trial 49     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 	 weighted AUC: 0.8528
> > CV trial 27     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8191
> > CV trial 73     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 	 weighted AUC: 0.8936
> > CV trial 39     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5798 	 weighted AUC: 0.7305
> > CV trial 47     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7234
> > CV trial 69     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7199
> > CV trial 21     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 	 weighted AUC: 0.9149
> > CV trial 44     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8369
> > CV trial 65     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.7943
> > CV trial 41     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 	 weighted AUC: 0.7092
> > CV trial 80     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6011 	 weighted AUC: 0.7305
> > CV trial 51     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738 	 weighted AUC: 0.8121
> > CV trial 57     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7128 	 weighted AUC: 0.8369
> > CV trial 75     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8156
> > CV trial 1      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8191
> > CV trial 63     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 	 weighted AUC: 0.8582
> > CV trial 82     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8475
> > CV trial 3      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8050
> > CV trial 23     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7270
> > CV trial 61     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.7553
> > CV trial 55     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 	 weighted AUC: 0.8014
> > CV trial 43     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 	 weighted AUC: 0.7270
> > CV trial 77     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7234
> > CV trial 25     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613 	 weighted AUC: 0.7872
> > CV trial 84     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.9362
> > CV trial 30     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 	 weighted AUC: 0.8333
> > CV trial 19     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8298
> > CV trial 37     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8298
> > CV trial 88     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8121
> > CV trial 96     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6507 	 weighted AUC: 0.7270
> > CV trial 86     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8759
> > CV trial 13     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.7837
> > CV trial 41     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 	 weighted AUC: 0.6809
> > CV trial 5      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617 	 weighted AUC: 0.9043
> > CV trial 102    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8333
> > CV trial 80     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 	 weighted AUC: 0.7163
> > CV trial 92     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.9220
> > CV trial 15     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8227
> > CV trial 104    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6436 	 weighted AUC: 0.7660
> > CV trial 35     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8333
> > CV trial 106    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5585 	 weighted AUC: 0.7163
> > CV trial 108    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418 	 weighted AUC: 0.7163
> > CV trial 110    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.6844
> > CV trial 94     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617 	 weighted AUC: 0.9255
> > CV trial 98     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8652
> > CV trial 82     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8546
> > CV trial 100    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7890 	 weighted AUC: 0.8652
> > CV trial 49     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8440
> > CV trial 63     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 	 weighted AUC: 0.8475
> > CV trial 112    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8475
> > CV trial 90     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8511 	 weighted AUC: 0.8511
> > CV trial 118    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525 	 weighted AUC: 0.7411
> > CV trial 27     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.7943
> > CV trial 44     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8014
> > CV trial 124    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8085
> > CV trial 84     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 	 weighted AUC: 0.9255
> > CV trial 110    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645 	 weighted AUC: 0.6454
> > CV trial 31     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.9149
> > CV trial 96     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 	 weighted AUC: 0.7021
> > CV trial 108    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.6738
> > CV trial 126    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8582
> > CV trial 81     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 	 weighted AUC: 0.7589
> > CV trial 116    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.9184
> > CV trial 104    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7163
> > CV trial 106    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5585 	 weighted AUC: 0.7163
> > CV trial 67     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7996 	 weighted AUC: 0.9007
> > CV trial 92     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.9043
> > CV trial 88     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7695
> > CV trial 120    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 	 weighted AUC: 0.7695
> > CV trial 122    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 	 weighted AUC: 0.8511
> > CV trial 130    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 	 weighted AUC: 0.7376
> > CV trial 19     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8085
> > CV trial 102    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8262
> > CV trial 98     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8688
> > CV trial 112    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 	 weighted AUC: 0.8298
> > CV trial 114    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8652
> > CV trial 86     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8475
> > CV trial 35     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8121
> > CV trial 100    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7057 	 weighted AUC: 0.8475
> > CV trial 71     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7996
> > CV trial 118    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 	 weighted AUC: 0.6986
> > CV trial 94     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8901
> > CV trial 83     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738 	 weighted AUC: 0.8475
> > CV trial 128    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8369
> > CV trial 85     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 	 weighted AUC: 0.7695
> > CV trial 17     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6631 	 weighted AUC: 0.7447
> > CV trial 90     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404 	 weighted AUC: 0.8511
> > CV trial 45     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617 	 weighted AUC: 0.8191
> > CV trial 111    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6188 	 weighted AUC: 0.8333
> > CV trial 124    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.7713
> > CV trial 97     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8582
> > CV trial 109    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8652
> > CV trial 105    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 	 weighted AUC: 0.8369
> > CV trial 120    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.7340
> > CV trial 113    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 	 weighted AUC: 0.6702
> > CV trial 87     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.9362
> > CV trial 81     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 	 weighted AUC: 0.7305
> > CV trial 136    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5904 	 weighted AUC: 0.8050
> > CV trial 132    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738 	 weighted AUC: 0.7589
> > CV trial 116    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8901
> > CV trial 99     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8972
> > CV trial 119    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021 	 weighted AUC: 0.8759
> > CV trial 103    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.9113
> > CV trial 95     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.6879
> > CV trial 134    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 	 weighted AUC: 0.8014
> > CV trial 130    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.6986
> > CV trial 93     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8121
> > CV trial 125    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 	 weighted AUC: 0.7482
> > CV trial 101    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8652
> > CV trial 91     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021 	 weighted AUC: 0.9255
> > CV trial 85     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7482
> > CV trial 128    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8262
> > CV trial 107    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 	 weighted AUC: 0.7801
> > CV trial 111    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6188 	 weighted AUC: 0.7979
> > CV trial 5      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8865
> > CV trial 31     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.9220
> > CV trial 138    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8901
> > CV trial 114    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8333
> > CV trial 121    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8546
> > CV trial 113    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.6312
> > CV trial 97     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8546
> > CV trial 28     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8528 	 weighted AUC: 0.8617
> > CV trial 89     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418 	 weighted AUC: 0.6702
> > CV trial 109    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8688
> > CV trial 122    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8422
> > CV trial 87     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.9149
> > CV trial 105    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8191
> > CV trial 83     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8475
> > CV trial 95     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826 	 weighted AUC: 0.6879
> > CV trial 140    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8546
> > CV trial 7      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8830
> > CV trial 125    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.6844
> > CV trial 142    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8723
> > CV trial 136    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 	 weighted AUC: 0.7908
> > CV trial 103    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8759
> > CV trial 119    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 	 weighted AUC: 0.8262
> > CV trial 99     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8759
> > CV trial 132    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6436 	 weighted AUC: 0.7234
> > CV trial 129    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7021
> > CV trial 91     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915 	 weighted AUC: 0.9184
> > CV trial 126    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8493
> > CV trial 150    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.7376
> > CV trial 93     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 	 weighted AUC: 0.8014
> > CV trial 144    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.7872
> > CV trial 148    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.7323
> > CV trial 146    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.7624
> > CV trial 107    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6294 	 weighted AUC: 0.7411
> > CV trial 152    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8723
> > CV trial 101    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950 	 weighted AUC: 0.8617
> > CV trial 115    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7234 	 weighted AUC: 0.9007
> > CV trial 154    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8759
> > CV trial 134    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 	 weighted AUC: 0.8014
> > CV trial 29     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8582
> > CV trial 158    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7128 	 weighted AUC: 0.8121
> > CV trial 166    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 	 weighted AUC: 0.7872
> > CV trial 137    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7234
> > CV trial 168    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7057
> > CV trial 160    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8440
> > CV trial 123    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8121
> > CV trial 117    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8652
> > CV trial 164    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8369
> > CV trial 45     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950 	 weighted AUC: 0.7979
> > CV trial 170    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6011 	 weighted AUC: 0.7766
> > CV trial 150    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.7057
> > CV trial 7      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8723
> > CV trial 121    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8050
> > CV trial 174    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8298
> > CV trial 176    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8369
> > CV trial 162    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 	 weighted AUC: 0.7660
> > CV trial 156    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.7908
> > CV trial 127    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8014
> > CV trial 8      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8723 	 weighted AUC: 0.9326
> > CV trial 172    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6117 	 weighted AUC: 0.7589
> > CV trial 144    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 	 weighted AUC: 0.7589
> > CV trial 148    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5691 	 weighted AUC: 0.7021
> > CV trial 129    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4965 	 weighted AUC: 0.6809
> > CV trial 180    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.8121
> > CV trial 140    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8191
> > CV trial 146    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.7624
> > CV trial 142    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8652
> > CV trial 154    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8475
> > CV trial 115    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915 	 weighted AUC: 0.8865
> > CV trial 29     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 	 weighted AUC: 0.8227
> > CV trial 166    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 	 weighted AUC: 0.7553
> > CV trial 89     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 	 weighted AUC: 0.6348
> > CV trial 133    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8298
> > CV trial 168    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.6702
> > CV trial 184    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5780 	 weighted AUC: 0.7589
> > CV trial 137    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 	 weighted AUC: 0.6950
> > CV trial 158    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.7837
> > CV trial 152    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234 	 weighted AUC: 0.8262
> > CV trial 151    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5479 	 weighted AUC: 0.7305
> > CV trial 131    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.9255
> > CV trial 160    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8227
> > CV trial 123    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 	 weighted AUC: 0.7766
> > CV trial 32     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.0709
> > CV trial 135    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.7713
> > CV trial 170    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 	 weighted AUC: 0.7270
> > CV trial 188    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.6454
> > CV trial 145    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7234 	 weighted AUC: 0.7553
> > CV trial 117    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 	 weighted AUC: 0.8369
> > CV trial 156    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.7837
> > CV trial 127    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7890
> > CV trial 186    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 	 weighted AUC: 0.7979
> > CV trial 178    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.7624
> > CV trial 141    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 	 weighted AUC: 0.8511
> > CV trial 147    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 	 weighted AUC: 0.8759
> > CV trial 180    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 	 weighted AUC: 0.7979
> > CV trial 174    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8085
> > CV trial 149    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8582
> > CV trial 162    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6507 	 weighted AUC: 0.7163
> > CV trial 172    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904 	 weighted AUC: 0.7270
> > CV trial 184    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5674 	 weighted AUC: 0.7340
> > CV trial 151    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826 	 weighted AUC: 0.7163
> > CV trial 176    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677 	 weighted AUC: 0.7872
> > CV trial 194    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525 	 weighted AUC: 0.7092
> > CV trial 188    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645 	 weighted AUC: 0.5993
> > CV trial 169    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553 	 weighted AUC: 0.8688
> > CV trial 155    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 	 weighted AUC: 0.7872
> > CV trial 171    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 	 weighted AUC: 0.7766
> > CV trial 159    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6223 	 weighted AUC: 0.7482
> > CV trial 167    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8511
> > CV trial 153    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8652
> > CV trial 133    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465 	 weighted AUC: 0.7766
> > CV trial 143    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8617
> > CV trial 198    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8333
> > CV trial 161    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8475
> > CV trial 202    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.6950
> > CV trial 182    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.8333
> > CV trial 164    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844 	 weighted AUC: 0.8191
> > CV trial 186    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 	 weighted AUC: 0.7411
> > CV trial 8      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8830 	 weighted AUC: 0.0674
> > CV trial 157    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 	 weighted AUC: 0.9574
> > CV trial 141    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021 	 weighted AUC: 0.8617
> > CV trial 135    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6631 	 weighted AUC: 0.7695
> > CV trial 145    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 	 weighted AUC: 0.7199
> > CV trial 190    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7553
> > CV trial 181    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.7979
> > CV trial 196    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7766
> > CV trial 33     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8103 	 weighted AUC: 0.8936
> > CV trial 192    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8422 	 weighted AUC: 0.8546
> > CV trial 138    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784 	 weighted AUC: 0.8901
> > CV trial 200    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.9468
> > CV trial 149    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8546
> > CV trial 173    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8475
> > CV trial 177    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8014
> > CV trial 178    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465 	 weighted AUC: 0.7340
> > CV trial 204    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8404
> > CV trial 206    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.7872
> > CV trial 163    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8085
> > CV trial 185    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8298
> > CV trial 171    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7801
> > CV trial 202    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 	 weighted AUC: 0.6950
> > CV trial 169    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8440
> > CV trial 167    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8369
> > CV trial 153    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8369
> > CV trial 131    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 	 weighted AUC: 0.9326
> > CV trial 159    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950 	 weighted AUC: 0.7340
> > CV trial 194    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418 	 weighted AUC: 0.6879
> > CV trial 187    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 	 weighted AUC: 0.7553
> > CV trial 155    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738 	 weighted AUC: 0.7660
> > CV trial 212    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.8546
> > CV trial 198    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826 	 weighted AUC: 0.8191
> > CV trial 189    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8794
> > CV trial 165    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8652
> > CV trial 157    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340 	 weighted AUC: 0.9397
> > CV trial 208    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8546
> > CV trial 139    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7855
> > CV trial 190    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7305
> > CV trial 214    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5780 	 weighted AUC: 0.6879
> > CV trial 181    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7128 	 weighted AUC: 0.7660
> > CV trial 210    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8121
> > CV trial 143    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 	 weighted AUC: 0.8404
> > CV trial 9      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039 	 weighted AUC: 0.8511
> > CV trial 196    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738 	 weighted AUC: 0.7518
> > CV trial 182    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8227
> > CV trial 147    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.8582
> > CV trial 175    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.9113
> > CV trial 216    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7943
> > CV trial 173    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8440
> > CV trial 177    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.7979
> > CV trial 206    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 	 weighted AUC: 0.7518
> > CV trial 179    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8121
> > CV trial 204    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8440
> > CV trial 218    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8759
> > CV trial 185    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8014
> > CV trial 226    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6312 	 weighted AUC: 0.8227
> > CV trial 224    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8475
> > CV trial 220    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.7801
> > CV trial 195    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7872
> > CV trial 200    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8511 	 weighted AUC: 0.9291
> > CV trial 163    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465 	 weighted AUC: 0.7908
> > CV trial 161    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8156
> > CV trial 230    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6401 	 weighted AUC: 0.8227
> > CV trial 199    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8050
> > CV trial 212    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613 	 weighted AUC: 0.8262
> > CV trial 187    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7447
> > CV trial 232    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6844 	 weighted AUC: 0.7837
> > CV trial 207    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.4752 	 weighted AUC: 0.6064
> > CV trial 33     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.0975
> > CV trial 228    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.8440
> > CV trial 203    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 	 weighted AUC: 0.7766
> > CV trial 139    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 	 weighted AUC: 0.7624
> > CV trial 183    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 	 weighted AUC: 0.8262
> > CV trial 208    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8404
> > CV trial 165    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8191
> > CV trial 234    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.9468
> > CV trial 214    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 	 weighted AUC: 0.6631
> > CV trial 216    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7589
> > CV trial 191    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8511
> > CV trial 240    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8440
> > CV trial 236    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.8262
> > CV trial 210    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.7943
> > CV trial 242    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8582
> > CV trial 224    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553 	 weighted AUC: 0.7943
> > CV trial 189    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979 	 weighted AUC: 0.8387
> > CV trial 220    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7163
> > CV trial 213    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6950 	 weighted AUC: 0.8156
> > CV trial 238    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6844 	 weighted AUC: 0.8085
> > CV trial 218    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8582
> > CV trial 230    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401 	 weighted AUC: 0.7730
> > CV trial 226    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6312 	 weighted AUC: 0.8014
> > CV trial 179    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8014
> > CV trial 201    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6507 	 weighted AUC: 0.8298
> > CV trial 207    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645 	 weighted AUC: 0.5816
> > CV trial 246    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 	 weighted AUC: 0.8121
> > CV trial 248    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6117 	 weighted AUC: 0.7163
> > CV trial 195    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.7518
> > CV trial 199    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.7837
> > CV trial 183    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525 	 weighted AUC: 0.8298
> > CV trial 244    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 	 weighted AUC: 0.8298
> > CV trial 203    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.7270
> > CV trial 209    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340 	 weighted AUC: 0.8511
> > CV trial 205    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5691 	 weighted AUC: 0.7801
> > CV trial 217    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.9149
> > CV trial 240    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.8191
> > CV trial 221    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660 	 weighted AUC: 0.9291
> > CV trial 225    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358 	 weighted AUC: 0.8262
> > CV trial 232    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6436 	 weighted AUC: 0.7660
> > CV trial 211    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8262
> > CV trial 234    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 	 weighted AUC: 0.9326
> > CV trial 236    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7943
> > CV trial 246    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7730
> > CV trial 213    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404 	 weighted AUC: 0.8050
> > CV trial 209    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447 	 weighted AUC: 0.8316
> > CV trial 219    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631 	 weighted AUC: 0.7305
> > CV trial 248    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5071 	 weighted AUC: 0.7092
> > CV trial 201    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613 	 weighted AUC: 0.7979
> > CV trial 67     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7163 	 weighted AUC: 0.8617
> > CV trial 228    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8121
> > CV trial 9      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8262
> > CV trial 217    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8830
> > CV trial 215    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766 	 weighted AUC: 0.9255
> > CV trial 244    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8121
> > CV trial 175    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8688
> > CV trial 231    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7979
> > CV trial 227    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8369
> > CV trial 241    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465 	 weighted AUC: 0.8652
> > CV trial 211    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571 	 weighted AUC: 0.8014
> > CV trial 205    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039 	 weighted AUC: 0.7660
> > CV trial 235    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933 	 weighted AUC: 0.8156
> > CV trial 191    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8475
> > CV trial 249    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826 	 weighted AUC: 0.6773
> > CV trial 233    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6206 	 weighted AUC: 0.7234
> > CV trial 225    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358 	 weighted AUC: 0.7908
> > CV trial 247    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021 	 weighted AUC: 0.8865
> > CV trial 237    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8652
> > CV trial 242    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8333
> > CV trial 229    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7872
> > CV trial 219    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904 	 weighted AUC: 0.7199
> > CV trial 221    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8901
> > CV trial 238    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011 	 weighted AUC: 0.7943
> > CV trial 245    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298 	 weighted AUC: 0.8528
> > CV trial 222    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8794
> > CV trial 215    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.9078
> > CV trial 249    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.6312
> > CV trial 231    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145 	 weighted AUC: 0.7447
> > CV trial 235    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720 	 weighted AUC: 0.7943
> > CV trial 241    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 	 weighted AUC: 0.8582
> > CV trial 247    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915 	 weighted AUC: 0.8511
> > CV trial 229    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933 	 weighted AUC: 0.7624
> > CV trial 233    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206 	 weighted AUC: 0.6915
> > CV trial 237    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872 	 weighted AUC: 0.8546
> > CV trial 227    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660 	 weighted AUC: 0.8262
> > CV trial 239    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8901
> > CV trial 245    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404 	 weighted AUC: 0.8121
> > CV trial 243    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7996 	 weighted AUC: 0.8688
> > CV trial 222    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085 	 weighted AUC: 0.8723
> > CV trial 239    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252 	 weighted AUC: 0.8901
> > CV trial 192    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8316 	 weighted AUC: 0.8298
> > CV trial 197    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.9539
> > CV trial 197    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766 	 weighted AUC: 0.9433
> > CV trial 223    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5709 	 weighted AUC: 0.7908
> > CV trial 243    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7890 	 weighted AUC: 0.8617
> > CV trial 193    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191 	 weighted AUC: 0.9255
> > CV trial 223    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4982 	 weighted AUC: 0.7943
> > CV trial 193    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191 	 weighted AUC: 0.9397
>

    If I launch the previous command line with only one 1-D feature as follows, everything is going well:

          neuropredict -m meta_data.csv \
      -d mask_WMpet_av45_early.ero1.25mm.csv \
      -o outdir -t 0.8 -n 250 -k 'all' \
      --gs_level 'exhaustive' --classifier "LinearSVC"

    Do you have any idea?

    Best,

    Matthieu

    Sources.zip

    opened by mattvan83 6
  • IO error, unable to load features

    IO error, unable to load features

    Hi Pradeep,

    I finally got neuropredict installed with the dependencies using python 2.7.13! I'm having some issues with running the command, see the error message below,

    Cheers,

    John

    (py2713) Johns-MacBook-Pro-2:KNN John$ neuropredict -m /Users/John/Desktop/KNN/METADATAFILE.csv -o /Users/John/Desktop/KNN/features/results -u /Users/John/Desktop/KNN/features

    Requested features for analysis: get_dir_of_dirs from /Users/John/Desktop/KNN/features Traceback (most recent call last): File "/Users/John/anaconda3/envs/py2713/lib/python2.7/site-packages/neuropredict/neuropredict.py", line 333, in getfeatures data, feat_names = getmethod(featdir, subjid) File "/Users/John/anaconda3/envs/py2713/lib/python2.7/site-packages/neuropredict/neuropredict.py", line 274, in get_dir_of_dirs raise IOError('Unable to load features from \n{}'.format(featfile))

    opened by johnaeanderson 5
  • Unable to launch multi-class classifier

    Unable to launch multi-class classifier

    Hi Pradeep,

    When trying to launch multi-class classifier (whatever the classifier chosen) on a 3-class (CN, MCI, AD) problem, I got the following problem:

    Traceback (most recent call last):
  File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/bin/neuropredict", line 8, in <module>
    sys.exit(main())
  File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/__main__.py", line 11, in main
    run_workflow.cli()
  File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 969, in cli
    grid_search_level, classifier, feat_select_method = parse_args()
  File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 525, in parse_args
    class_set, subgroups, positive_class = validate_class_set(classes, user_args.sub_groups, user_args.positive_class)
  File "/homes_unix/mvanhoutte/Soft/anaconda3/envs/neuropredict/lib/python3.7/site-packages/neuropredict/run_workflow.py", line 659, in validate_class_set
    ''.format(comb))
ValueError: Subgroup all does not contain 2 unique classes! Each subgroup must contain atleast two classes for classification experiments.

    I don't understand since I have well defined 3-class in the metadata.csv file and used the following command line:

      neuropredict -m meta_data.csv \
  -d metaROI.csv metaROI_split.csv HCP_parcellation.csv \
  -t 0.8 -n 250 -k 'all' --sub_groups 'all' --gs_level 'exhaustive' \
  --feat_select_method 'variancethreshold' --classifier ${classifier} --make_vis ./visu \
  --num_procs ${NCPU} --print_options ./options

    Could you help me? Best, Matthieu

    opened by mattvan83 4
  • Regression output visualizations: residual plots

    Regression output visualizations: residual plots

    A good output in the regression version would be residual plots

    Good example is from yellowbrick: http://www.scikit-yb.org/en/latest/api/regressor/residuals.html

    opened by raamana 2
  • CLI flag to print chosen options from an existing run

    CLI flag to print chosen options from an existing run

    normal run would print only the most important options such as which classifier, cross-validation details etc, whereas verbose option would print which subjects and more info into feature sets etc.

    opened by raamana 1
  • Add documentation to help interpret the results and visualizations

    Add documentation to help interpret the results and visualizations

    Have a separate markdown file

    • [ ] describe each output figure comprehensively
    • [ ] describe each output CSV file
    • [ ] And what else they can do with results
    enhancement help wanted 
    opened by raamana 1
  • feature request: covariates and ability to regress them

    feature request: covariates and ability to regress them

    few ideas:

    CSV input

    • [ ] have column names, specify which columns are class, id, covariates and feature anmes!
    • [ ] get the class ID from a specified colum (e.g. first, or specified)
    • [ ] get the subject ID from a specified column (or the second or specified)
    sbuject_id,class,feature1,feature2,age,sex
acbde_01,healthy,1.2,2.3,3.4,74,female
fghijk_01,healthy,1.2,2.3,3.4,76,male
...
lmnop_01,alzheimer,1.2,2.3,3.4,70,female
...
qrstu_01,depression,1.2,2.3,3.4,71,male

    Directory of directories input

    • [ ] covariates.txt in addition to features.txt within each subject folder
    enhancement Hackathon project 
    opened by raamana 1
  • Enabling plug-in user-chosen models or hyper-parameters

    Enabling plug-in user-chosen models or hyper-parameters

    Sometimes some advanced users would like to choose their model or hyper-parameters for many good reasons including model not existing in sklearn etc or needing deep control over them etc, then having the API or CLI allowing users to do it would be great!

    opened by raamana 0
  • Differences betw. frequency misclassified vs frequently correctly classified

    Differences betw. frequency misclassified vs frequently correctly classified

    besides the misclassification frequency plots (which can be a helpful diagnostic), it may be useful to offer an option to enable the user to compare and study the differences characteristics between frequency misclassified vs. frequently correctly classified..

    Characteristics could be the input features themselves, or the other associated meta data and attributes (e.g. provided as part of pyradigm)

    examples include: those frequently misclassified were mostly the older patients , and those frequently correctly classified has "property X" etc.

    opened by raamana 0
  • Add docs for all API, esp. CVResults

    Add docs for all API, esp. CVResults

    CVResults might be individually even more useful than others in the API. So document it thoroughly align with the rest with a tutorial or two for use by dev and reuse by the replicator or meta-analytic researcher.

    Hackathon project 
    opened by raamana 0
  • update all docs and graphics to indicate users can try different models also

    update all docs and graphics to indicate users can try different models also

    not just compare different feature sets with a fixed builtin model, but users can also input a model of their own choice. it does not limit exploration of new models or pipelines - they can use implementation of best practices while evaluating the such new models on features of their choice

    opened by raamana 1
  • Attribute-contrained performance estimates

    Attribute-contrained performance estimates

    Often, performance is estimated in aggregate on the entire test-set (in a split), regardless of their covariate- or attribute characteristics.. hence looking into performance estimates segregated by certain characteristics of covariates would reveal additional insights - such as difference in AUC in men vs women, or across different age bins, or between sites etc.

    Hackathon project 
    opened by raamana 2
Releases(0.6)

  • 0.6(Jun 8, 2020)

    • Major feature: Ability to predict continuous variables (regression)
    • Major feature: Ability to handle confounds (regress them out, augmenting etc)
    • Redesigned the internal structure for easier extensibility
    • New CVResults class for easier management of a wealth of outputs generated in the Classification and Regression workflows
    • API access is refreshed and easier
    Source code(tar.gz)
    Source code(zip)

  • 0.5(Jan 31, 2019)

    • Good news: neuropredict can handle missing data now (that are encoded with numpy.NaN). This is done respecting the cross-validation splits without any data leakage.
    • Few other useful utilities
    Source code(tar.gz)
    Source code(zip)

  • 0.4.5(Nov 18, 2017)

    • new classifier : SVM
    • new flag to choose a feature selection method
    • user chosen options now saved to disk, to better handle complex interactions between options
    • code clean up and faster tests
    Source code(tar.gz)
    Source code(zip)

  • 0.4.1(Nov 6, 2017)

    • Parallelizing the main the CV loop, leading to great reduction in total time for report generation!
    • More options, including choice of different classifiers (Random Forest and Extra Trees classifiers)
    • Support for dataset in Weka's ARFF format
    • Better visualizations (handling small/nan values in feature importance, layouts and design)
    • auto versioning!
    • Ability to read meta data from pyradigms or ARFF files, without having to specify that separately.
    • Dropping support for Python 2.7 :(
    Source code(tar.gz)
    Source code(zip)

  • v0.3-DOI(Sep 26, 2017)

Owner
Pradeep Reddy Raamana
Neuroscientist trying to bridge the gap between clinic & computer science. Interests: Machine learning, Neuroimaging, Brain disorders, Informatics, Open science
Pradeep Reddy Raamana
GitHub https://raamana.github.io/neuropredict/
Machine learning for NeuroImaging in Python

nilearn Nilearn enables approachable and versatile analyses of brain volumes. It provides statistical and machine-learning tools, with instructive doc

null 919 Dec 25, 2022
improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

null 310 Dec 28, 2022
Multiple types of NN model optimization environments. It is possible to directly access the host PC GUI and the camera to verify the operation. Intel iHD GPU (iGPU) support. NVIDIA GPU (dGPU) support.

mtomo Multiple types of NN model optimization environments. It is possible to directly access the host PC GUI and the camera to verify the operation.

Katsuya Hyodo 24 Mar 2, 2022
Static Features Classifier - A static features classifier for Point-Could clusters using an Attention-RNN model

Static Features Classifier This is a static features classifier for Point-Could

ABDALKARIM MOHTASIB 1 Jan 25, 2022
A user-friendly research and development tool built to standardize RL competency assessment for custom agents and environments.

Built with ❤️ by Sam Showalter Contents Overview Installation Dependencies Usage Scripts Standard Execution Environment Development Environment Benchm

SRI-AIC 1 Nov 18, 2021
An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates neural fields, predictive coding, top-down-bottom-up, and attention (consensus between columns)

GLOM - Pytorch (wip) An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates neural fields, predictive coding,

Phil Wang 173 Dec 14, 2022
A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

Website, Tutorials, and Docs    Uncertainty Toolbox A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualizatio

Uncertainty Toolbox 1.4k Dec 28, 2022
MagFace: A Universal Representation for Face Recognition and Quality Assessment

MagFace MagFace: A Universal Representation for Face Recognition and Quality Assessment in IEEE Conference on Computer Vision and Pattern Recognition

Qiang Meng 523 Jan 5, 2023
Repo for 2021 SDD assessment task 2, by Felix, Anna, and James.

SoftwareTask2 Repo for 2021 SDD assessment task 2, by Felix, Anna, and James. File/folder structure: helloworld.py - demonstrates various map backgrou

null 3 Dec 13, 2022
MRQy is a quality assurance and checking tool for quantitative assessment of magnetic resonance imaging (MRI) data.

Front-end View Backend View Table of Contents Description Prerequisites Running Basic Information Measurements User Interface Feedback and usage Descr

Center for Computational Imaging and Personalized Diagnostics 58 Dec 2, 2022
To propose and implement a multi-class classification approach to disaster assessment from the given data set of post-earthquake satellite imagery.

To propose and implement a multi-class classification approach to disaster assessment from the given data set of post-earthquake satellite imagery.

Kunal Wadhwa 2 Jan 5, 2022
Predictive AI layer for existing databases.

MindsDB is an open-source AI layer for existing databases that allows you to effortlessly develop, train and deploy state-of-the-art machine learning

MindsDB Inc 12.2k Jan 3, 2023
Predictive AI layer for existing databases.

MindsDB is an open-source AI layer for existing databases that allows you to effortlessly develop, train and deploy state-of-the-art machine learning

MindsDB Inc 3.2k Feb 12, 2021
Official implementation for NIPS'17 paper: PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs.

PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning The predictive learning of spatiotemporal sequences aims to generate future

THUML: Machine Learning Group @ THSS 243 Dec 26, 2022
PyTorch Code of "Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics"

Memory In Memory Networks It is based on the paper Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spati

Yang Li 12 May 30, 2022
Real-Time Multi-Contact Model Predictive Control via ADMM

Here, you can find the code for the paper 'Real-Time Multi-Contact Model Predictive Control via ADMM'. Code is currently being cleared up and optimize

null 17 Dec 28, 2022
Predictive Maintenance LSTM

Predictive-Maintenance-LSTM - Predictive maintenance study for Complex case study, we've obtained failure causes by operational error and more deeply by design mistakes.

Amir M. Sadafi 1 Dec 31, 2021
EfficientMPC - Efficient Model Predictive Control Implementation

efficientMPC Efficient Model Predictive Control Implementation The original algo

Vin 8 Dec 4, 2022
Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions

Natural Posterior Network This repository provides the official implementation o

Oliver Borchert 54 Dec 6, 2022