Comments (5)
Hi, this is the line for split enumeration on CPU
xgboost/src/tree/hist/evaluate_splits.h
Line 304 in 1450aeb
It might take some time to get around it and enable the feature you want. We implement sampling by setting the hessian to zero, which is quite different from your feature request.
from xgboost.
I don't think we have a way to evaluate based on out-of-bag samples at the moment.
from xgboost.
I think the way to do this is to simultaneously build a tree with the out-of-sample data and apply each split to it. For the new split candidates in the tree built with the sample data, evaluate gain reduction in the out-of-sample tree and choose the candidate that maximally improves in-sample, among those for which gamma is also greater than X out-of-sample.
I think this double-checking is very useful in high noise / low signal problems.
I don't know how hold predictions are done, if doing something similar (building hold tree in parallel) or predicting hold with each new tree.
from xgboost.
For statistical inference algorithms like gradient boosting, the standard way is simply to run cross-validation, which is reasonably effective. GBM is built upon the concept that we can stack many weak models to obtain a strong model, instead of preventing the tree from splitting, you can simply train small/shallow trees and try to prevent XGBoost from adding more trees, at least in theory this should have the similar effect as your feature request.
from xgboost.
Yes, that's the theory behind gradient boosting and it works very well in almost all contexts. The scenario where there is an extremely low signal-to-noise ratio is different. Let me justify why I am interested in this functionality.
I am referring to datasets for stock market prediction, with level of rank correlation is on the order of 0.02 - 0.03.
At the same time, these predictions exhibit extreme variability. For example, changing the starting seed without changing any other parameter produces changes on the order of 15% in Spearman correlation. This can give us the noise level.
In this scenario, every time a model fits signal with a weak model, that signal becomes unrecoverable later by another weak model because its level is so low that once the information is fragmented, it dissipates into noise.
That's why I believe that a double-check is very important before using ('destroying') the signal in a tree (weak model). I may be wrong, but my intuition tells me that by preserving the signal and using it only in splits that are useful in both a sample and out-of-sample double-check, better results will be achieved than by adding new trees that won't be able to extract the remaining little signal diluted in the noise.
from xgboost.
Related Issues (20)
- An error occurred: jemalloc unsupported system page size HOT 2
- Make external CUDA stream wait.
- Docs on params and Python tutorial for regression HOT 1
- Docs for custom objectives could cover more details HOT 9
- Which one is `y_true` when I use a custom loss for regression mode. HOT 3
- clarification needed for model/saving loading HOT 2
- Missing XGBoostRanker in xgboost4j-spark jvm package HOT 3
- multi label support in Scala xgboost. HOT 8
- SparkXGBClassifier does not validate params HOT 6
- feature_weights only compatible with CPU ? HOT 1
- Improve XGBoost quantile predictions HOT 3
- Latest version training crashes HOT 12
- Error when trying to build HOT 4
- Model provides different results for different Python versions HOT 2
- error in the docs for ranking HOT 1
- Defining a callback to write hessians of train observations to a csv file HOT 2
- Slow inference on sphoradic stremaing data HOT 2
- XGBoost GPU Warning When Working with BayesSearchCV (XGBoost is running on: cuda:0, while the input data is on: cpu.)
- help installing xgboost with gpu HOT 2
- Potential Documentation Inaccuracy Regarding Feature Interaction Constraints
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xgboost.