I want to know about metrics used for measuring in evaluate_oversamplers about smote_variants HOT 10 CLOSED

Hi @arjunpuri7, sure,

• gacc refers to the geometric mean of accuracies computed for the individual classes. For example, having TP, FP, TN, FN true positives, false positives, true negatives and false negatives, the accuracy of the positive class only (also called sensitivity - SENS) is SENS= TP/(TP + FN); while the accuracy of the negative class only (also called the specificity - SPEC) is SPEC= TN/(TN + FP), then, GACC is the geometric mean of these class specific accuracies, that is, GACC = SQRT(SENS*SPEC). This score is expected to take into account class imbalance as the "accuracy" of predicing positive samples (SENS) is taken into account with the same weight as the "accuracy" of predicting negative samples (SPEC). For more, see https://stats.stackexchange.com/questions/235710/auc-geometric-mean-for-classifying-imbalanced-classes
• Brier-score is basically the mean squared error of predicting probabilities, that is, the difference of the predicted positive class probability and the observed probability (0/1) is taken, squared and averaged. For more, see https://en.wikipedia.org/wiki/Brier_score
• You can also take a look on this one, giving an overview of performance measures commonly accepted in imbalanced learning: https://www.researchgate.net/publication/267671515_Learning_from_Imbalanced_Data_Evaluation_Matters

from smote_variants.

from smote_variants.

I want to know a little more about the evaluator_Oversamplers
as they are performing oversampling using stratified cross sampling:

is oversampling is applied to only training part of datasets in cross validation or it may use to apply oversampling on whole datasets as preprocessing??

from smote_variants.

Hi @arjunpuri7, this is a crucial question: in each round of cross-validation, oversampling is applied ONLY to the training set. The test set (which is, say, 1/8th of the entire dataset in a given split) is NOT affected by oversampling. With this approach, we can avoid any data leakage, no information from the test set is used to influence the oversampling of the training set in the cross-validation rounds.

from smote_variants.

from smote_variants.

hello sir,
i am facing another problem with this code when i running evaluate_oversamplers with different type of oversampling methods using different datasets. it gives me same result of my initial dataset and not work on other datasets. please help me. results of two different datasets are attach as below.
New folder.zip

from smote_variants.

Hi @arjunpuri7, could you please send over the code too?

from smote_variants.

hello sir,
sorry for delay in reply. I gone to some where.I solve this problem.
but another problem is how to set evaluate_oversamplers cache in colab. if you have any idea then please share it with me.

from smote_variants.

Hi @arjunpuri7, that's fine. I have limited experience with colab. The caching mechanism should work if some path is available for the caching system. As far as I know, google drive can be attached to colab as a folder and then you can use that folder for caching. For more details, please take a look at https://gist.github.com/Joshua1989/dc7e60aa487430ea704a8cb3f2c5d6a6

from smote_variants.

thank sir,
finally issue is resolved.

from smote_variants.