My personal implementation of xgboost-style gradient boosting trees.

• Python 3.4.3
• Cython 0.23.4
• numpy 1.10.4

pip install git+https://github.com/khyh/papermill.git --user

from papermill import Papermill


ppm = Papermill(seed = 0, loss_type = "log_loss", eta = 0.01,
    subsample = 0.8, colsample_bytree = 0.8, num_round = 300)


# simple fit
ppm.fit(data_train, label_train)

pred_test = ppm.predict(data_test)


# early stopping
ppm.fit(
    data_train, label_train, # train data
    data_valid, label_valid, # additional data for validation
    "roc_auc_score",         # metrics to monitor. "rmse", "log_loss", "roc_auc_score" is supported
    10)                      # validation error needs to decrease at every 10 rounds

pred_test = ppm.predict(data_test) # best round is used by default

• num_threads [default=-1]
  • number of threads to be used.
  • maximum number of threads are used by default.
• seed [default=0]
  • random seed.
• silent [default=False]
  • be silent or not.
• loss_type ["mse" or "log_loss", default="mse"]
  • loss to minimize. "mse" or "log_loss" is supported for now.
  • aka objective.
• eta [default=0.3]
  • shrinkage parameter.
• max_depth [default=6]
  • maximum depth of trees.
• min_child_weight [default=1.0]
  • minimum sum of instance hessian needed in a child.
• lambda_ [default=1.0]
  • L2 regularization term on weights.
• gamma [default=0.0]
  • minimum gain to make a node split while pruning a tree.
  • pruning is not performed when gamma is smaller than gamma_zero.
• subsample [default=1.0]
  • sampling ratio of training samples.
• colsample_bytree [default=1.0]
  • sampling ratio of columns used by a single tree.
• normalize_target [True or False, default=False]
  • set bias to (sum of gradient) / (sum of hessian).
  • when False, bias is set to 0.0.
• gamma_zero [default=1.0e-5]
  • minimum gain to make a node split while growing a tree.
• num_round [default=100]
  • number of maximum trees to train.

• dmlc/xgboost by Distributed (Deep) Machine Learning Community
• Parallel Gradient Boosting Decision Trees by Zhanpeng Fang