Git Product home page Git Product logo

hyperparametertunability's Introduction

To tune or not to tune? A meta-leaning approach for recommending important hyperparameters

The following repository contains all metrails for repoducing the paper "To tune or not to tune? A meta-leaning approach for recommending important hyperparameters":

  • the scripts for collecting performance data of 6 machine learning algorithms on 200 classification tasks from OpenML environment.

  • the collected performance data of SVM, Decision Tree, Random Forest, AdaBoost, Gradient Boosting and Extra Trees Classifiers.

  • Several notebooks that each performs one experiment and conducts the results.

  • Based on PerformanceData, created new datasets that all are in output_csv folders.

  • tools for:

    • Importing and modifying the collected data
    • Searching correlation between the dataset metafeatures and classifier performances.
    • Conducting statistical tests to compare performance of the classifiers over the tasks.
    • Computing the best value for each important hyperparameter.
    • Computing Wilcoxon test for verifing the result.
  • script for extracting metafeatures of the datasets

  • script for performing fANOVA on the performance data

To start collecting data for a given classifier over all datasets

from DataCollection.functions import *

path_to_datasets = 'Datasets/'
classification_per_algorithm(path=path_to_datasets, algorithm='DecisionTree')

Conduct fANOVA on the data

from fANOVA.fanova_functions import *
do_fanova(dataset_name='PerformanceData/AB_results_total.csv', algorithm='AdaBoost')

Extract Metafeatures

from tools.metafeatures import *
extract_for_all(path_to_datasets)

Create the Database object to import the collected data in desired formats

from Tools.database import Database
db = Database()
per_dataset_acc = db.get_per_dataset_accuracies()
per_dataset_acc.head()
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}
</style>
dataset AB ET RF DT GB SVM
0 AP_Breast_Omentum.csv 0.981060 0.976235 0.976462 0.973912 0.983555 0.914538
1 AP_Breast_Prostate.csv 0.995238 0.995238 0.995238 0.995238 0.995238 0.961498
2 AP_Endometrium_Lung.csv 0.968363 0.958392 0.957018 0.929240 0.968363 0.894591
3 AP_Endometrium_Prostate.csv 0.992857 0.992857 0.992857 1.000000 1.000000 0.984615
4 AP_Endometrium_Uterus.csv 0.854854 0.837953 0.859561 0.827924 0.860409 0.758801
metafeatures = db.get_metafeatures()

hyperparametertunability's People

Contributors

mbahmani avatar nshanpotikyan avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.