Need to carry out following tests.

• [x] binary classification
• [x] multi-class classification
• [x] regression

First, we need to create dataset(train & test) for above problems.

Next, define several models.

Finally, do stacking.

Hi I find something maybe you don't notice 1. First of all, run 'pip install stacking' , it's not work for macOS because the xgboost package can't install here automatic，so we need install xgboost first manually. 2. I notice that in "examples/multi_class/scripts/multiclass.py" , something wrong in line 25 ,it should be "from keras.regularizers import l1, l2" ,little mistake but make 'bash run.sh' can't work 3. 'No module named paratext' is another problem ,the solution is git from https://github.com/wiseio/paratext, and don't forget to install swig by using 'brew install swig' if brew have been installed

That's it ,now i can run the EXAMPLE, i suggest the environment should write in README.md just in case. BTW, project is wonderful！

In stacking(i.e., cross-validation), validation data can be used for checking training process. XGBoost and Keras are now done because this models show the evaluation during training. ( #8 )

before(XGB):

Fold 0
[0] train-mlogloss:1.963271
[1] train-mlogloss:1.716260
[2] train-mlogloss:1.519960
[3] train-mlogloss:1.358485
[4] train-mlogloss:1.224851
[5] train-mlogloss:1.111999
[6] train-mlogloss:1.012638
[7] train-mlogloss:0.925325
[8] train-mlogloss:0.846453
[9] train-mlogloss:0.778393
logloss:  0.947207616242

after(XGB):

Fold 0
[0] train-mlogloss:1.963271 validation-mlogloss:2.006139
[1] train-mlogloss:1.716260 validation-mlogloss:1.782591
[2] train-mlogloss:1.519960 validation-mlogloss:1.601844
[3] train-mlogloss:1.358485 validation-mlogloss:1.462138
[4] train-mlogloss:1.224851 validation-mlogloss:1.343135
[5] train-mlogloss:1.111999 validation-mlogloss:1.242986
[6] train-mlogloss:1.012638 validation-mlogloss:1.152776
[7] train-mlogloss:0.925325 validation-mlogloss:1.078933
[8] train-mlogloss:0.846453 validation-mlogloss:1.009647
[9] train-mlogloss:0.778393 validation-mlogloss:0.947208
logloss:  0.947207616242

before(Keras):

Fold 0
Train on 1072 samples, validate on 275 samples
Epoch 1/15
1072/1072 [==============================] - 0s - loss: 0.9323 - acc: 0.7015
Epoch 2/15
1072/1072 [==============================] - 0s - loss: 0.3831 - acc: 0.8843
Epoch 3/15
1072/1072 [==============================] - 0s - loss: 0.3377 - acc: 0.8909
Epoch 4/15
1072/1072 [==============================] - 0s - loss: 0.2932 - acc: 0.9002
Epoch 5/15
1072/1072 [==============================] - 0s - loss: 0.2936 - acc: 0.9198
Epoch 6/15
1072/1072 [==============================] - 0s - loss: 0.3889 - acc: 0.8983
Epoch 7/15
1072/1072 [==============================] - 0s - loss: 0.3491 - acc: 0.9179
Epoch 8/15
1072/1072 [==============================] - 0s - loss: 0.3642 - acc: 0.9114
Epoch 9/15
1072/1072 [==============================] - 0s - loss: 0.3045 - acc: 0.9216
Epoch 10/15
1072/1072 [==============================] - 0s - loss: 0.4251 - acc: 0.9114
mlogloss = 0.4170

after(Keras):

Fold 0
Train on 1072 samples, validate on 275 samples
Epoch 1/15
1072/1072 [==============================] - 0s - loss: 0.9323 - acc: 0.7015 - val_loss: 0.2980 - val_acc: 0.9018
Epoch 2/15
1072/1072 [==============================] - 0s - loss: 0.3831 - acc: 0.8843 - val_loss: 0.1917 - val_acc: 0.9309
Epoch 3/15
1072/1072 [==============================] - 0s - loss: 0.3377 - acc: 0.8909 - val_loss: 0.2097 - val_acc: 0.9309
Epoch 4/15
1072/1072 [==============================] - 0s - loss: 0.2932 - acc: 0.9002 - val_loss: 0.1793 - val_acc: 0.9382
Epoch 5/15
1072/1072 [==============================] - 0s - loss: 0.2936 - acc: 0.9198 - val_loss: 0.2713 - val_acc: 0.9309
Epoch 6/15
1072/1072 [==============================] - 0s - loss: 0.3889 - acc: 0.8983 - val_loss: 0.2549 - val_acc: 0.9382
Epoch 7/15
1072/1072 [==============================] - 0s - loss: 0.3491 - acc: 0.9179 - val_loss: 0.3072 - val_acc: 0.9418
Epoch 8/15
1072/1072 [==============================] - 0s - loss: 0.3642 - acc: 0.9114 - val_loss: 0.2830 - val_acc: 0.9418
Epoch 9/15
1072/1072 [==============================] - 0s - loss: 0.3045 - acc: 0.9216 - val_loss: 0.3789 - val_acc: 0.9236
Epoch 10/15
1072/1072 [==============================] - 0s - loss: 0.4251 - acc: 0.9114 - val_loss: 0.4170 - val_acc: 0.9273
mlogloss = 0.4170

Added multi-class classification methods in base_fixed_fold.py and modified the way of holding predictions. ( #3 )

And tested each function for multi-class classification. ( #9 ) Also tested binary classification and regression problem.

Scikit-learn methods are not tested. So it is needed.

Added regression test under test/regression/. ( #9 ) Run following code sh run.sh

Fixed normalization in KerasRegressor.

Modified the way of setting problem type.

Need to set like following code above your model-defined script.

# ----- Set problem type!! -----
problem_type = 'regression'
classification_type = ''
eval_type = 'rmse'

BaseModel.set_prob_type(problem_type, classification_type, eval_type)

I used class variables like BaseModel.problem_type in base_fixed_fold.py instead of global variables. Discussion is need.

Tested binary prediction under test/binary/. ( #9 done binary classification) Run following code sh run.sh
XGBoost and Keras are tested.

And modified base_fixed_fold.py

• Modified PATH variables
• Fixed how to create CV-fold index file ( #4, #6 )
  • User need to define own create_cv_id()
• Delete arguments of keras(i.e., show_accuracy)
• Delete save_pred_as_submit_format() ( #5, #6 )
  • User need to create submission format file(i.e., adding prediction ID, name...)

In stacking, test data(out-of-fold data) is not passed to models as validation data (XGBoost and NN). That is, validation scores are calculated after model training is done. It will be convenient to check the validation score every epoch. So need to pass out-of-fold data in model training as well.

Current load_data() use the same train and test features. (related #2 ) And target is included in train dataset.

FEATURE_LIST_stage1 = {
                'train':(INPUT_PATH + 'train.csv',
                         FEATURES_PATH + 'train_log.csv',

                        ),#target is in 'train'
                'test':(INPUT_PATH + 'test.csv',
                        FEATURES_PATH + 'test_log.csv',
                        ),
                }

However, in level 2 stacking, the following codes cause an error if target data is passed only in train dataset. This error occurs due to the difference of length of lists between train and test.

    FEATURE_LIST_stage2 = {
                'train':(INPUT_PATH + 'target.csv',

                         TEMP_PATH + 'v1_stage1_all_fold.csv',
                         TEMP_PATH + 'v2_stage1_all_fold.csv',
                         TEMP_PATH + 'v3_stage1_all_fold.csv',
                         TEMP_PATH + 'v4_stage1_all_fold.csv',
                         TEMP_PATH + 'v5_stage1_all_fold.csv',
                         TEMP_PATH + 'v6_stage1_all_fold.csv',

                        ),#target is in 'train'
                'test':(
                         TEMP_PATH + 'v1_stage1_test.csv',
                         TEMP_PATH + 'v2_stage1_test.csv',
                         TEMP_PATH + 'v3_stage1_test.csv',
                         TEMP_PATH + 'v4_stage1_test.csv',
                         TEMP_PATH + 'v5_stage1_test.csv',
                         TEMP_PATH + 'v6_stage1_test.csv',                       
                        ),
                }

This bug is fixed by using each length of list in reading. But for more general library, new key of feature dictionary, i.e., target, should be made like:

    FEATURE_LIST_stage2 = {
                'train':(
                         TEMP_PATH + 'v1_stage1_all_fold.csv',
                         TEMP_PATH + 'v2_stage1_all_fold.csv',
                         TEMP_PATH + 'v3_stage1_all_fold.csv',
                        ),#target is not in 'train'

                'target':(
                         INPUT_PATH + 'target.csv',
                        ),#target is in 'target'

                'test':(
                         TEMP_PATH + 'v1_stage1_test.csv',
                         TEMP_PATH + 'v2_stage1_test.csv',
                         TEMP_PATH + 'v3_stage1_test.csv',
                        ),
                }

This is very reasonable.

Now data directories(i.e., data/input(output)/...) are made in importing stacking if the directories does not exist. But if this is simply implemented, user can make directories without intent. It is useful that one can select if making them in the first import.

like:

Can new directories for input and output data be created? [Y/n]
Y(N) <---- input

Should change task-dependent functions to virtual functions?

User need to define such functions themselves. (e.g., CV-fold index, save_pred_as_submit_format, )

• previous version for CV-fold file. Using index.

train:

[2,3,4,5,6,7,8,9]

[0,1,4,5,6,7,8,9]

[0,1,2,3,6,7,8,9]

[0,1,2,3,4,5,8,9]

[0,1,2,3,4,5,6,7]

test:

[0,1]

[2,3]

[4,5]

[6,7]

[8,9]

• current version for CV-fold file(better than previous one). Using fold ID.

[0,0,1,1,2,2,3,3,4,4]

But current BaseModel uses previous version architectures. If current version is used, it is changed to previous format. So need to change that to using original format. And need to change .ix to .iloc for stable behavior.

Need to change global CV-fold file name with new CV-fold file name, if new CV-fold be created.

Now path is defined using slash / like

PATH = "foo/"
PATH + 'hoge'

, but it seems good to use os.path.join like

import os

PATH = "foo"
os.path.join(PATH, 'hoge')

Training and test data are now loaded at the beginning of model building.

However, it is not time-efficient if the data is very large.

So, if the data used is not changed through staking script, the data should be stored somewhere of base.py.

STORED_X = pd.DataFrame()

def load_data(stored=True):
    if STORED_X:
        return STORED_X

    else:
        ...

    if stored:
        STORED_X = X.copy()
        return STORED_X

    else:
        return X

Refactor hard coding for stability in preparation for the version up of libraries, such as sklearn, keras, and XGBoost. Specifically, initialization of BaseModel class.

In Keras-implemented neural net, to avoid recompile, initial weights after compilation is saved and used at the next beginning of training in cross validation. However, the initial weights are same in all fold-training. So initial weights should be changed at each training.

A possible solution is passing the argument of compilation (e.g., optimizer, loss, and metrics). In binary_class.py,

class ModelV2(BaseModel):
        def build_model(self):
            model = Sequential()
            model.add(Dense(64, input_shape=nn_input_dim_NN, init='he_normal'))
            model.add(LeakyReLU(alpha=.00001))
            model.add(Dropout(0.5))

            model.add(Dense(output_dim, init='he_normal'))
            model.add(Activation('softmax'))
            sgd = SGD(lr=0.1, decay=1e-5, momentum=0.9, nesterov=True)

            compile_options = {
                                           'optimizer': sgd, 
                                           'loss': 'categorical_crossentropy', 
                                           'metrics': ["accuracy"]
                                           }

            return KerasClassifier(nn=model, compile_options=compile_options, **self.params)

In base.py,

import copy

class KerasClassifier(BaseEstimator, ClassifierMixin):
    def __init__(self,nn):
        self.nn = nn

    def fit(self, X, y, X_test=None, y_test=None):
        self.compiled_nn = copy.copy(self.nn)
        self.compiled_nn.compile(**self.compile_options)

        return self.compiled_nn(X, y)

But this approach leads memory consumption...