Comments (5)
crarlus
commented on July 27, 2024
1
Hi,
we experienced a problem when using ranger version 0.5. Apparantly something has changed since version 0.4 in the handling of the label data. In a simple two-class problem, it seems to rename the labels internally in numerical or lexicographical order, thus providing misleading predictions.
It is easily demonstrated with a toy example:
# create trainingsdata
n = 1000
a1 = c(rnorm(n, 3, sd = 2), rnorm(n, 8, sd = 2))
a2 = c(rnorm(n, 8, sd = 2), rnorm(n, 3, sd = 2))
# create labels for data
labels <- as.factor(c(rep("0", n), rep("1", n)))
labels.rev <- as.factor(c(rep("1", n), rep("0", n))) # switch label -> expect switch of predictions
# training
test_forest <- ranger::ranger(dependent.variable.name = "label", data = data.frame(label = labels, a1, a2), write.forest = T, probability = T)
test_forest.rev_labels <- ranger::ranger(dependent.variable.name = "label", data = data.frame(label = labels.rev, a1, a2), write.forest = T, probability = T)
Whereas in ranger 0.4 the predictions are as expected, the first few predictions either have high probabilities for label "0" or for label "1" with reversed labels:
> head(test_forest$predictions)
0 1
[1,] 1.0000000 0.0000000000
[2,] 1.0000000 0.0000000000
[3,] 0.7267204 0.2732796452
[4,] 0.9916530 0.0083470170
[5,] 0.9613803 0.0386196857
[6,] 0.9994949 0.0005050505
> head(test_forest.rev_labels$predictions)
0 1
[1,] 0.00000000 1.0000000
[2,] 0.00000000 1.0000000
[3,] 0.29646739 0.7035326
[4,] 0.01103671 0.9889633
[5,] 0.04867796 0.9513220
[6,] 0.00000000 1.0000000
However, in the case of ranger 0.5 the predictions are always in favor of class "0" although in this test examples the labels have been reversed:
> head(test_forest_version5$predictions)
0 1
[1,] 1.0000000 0.00000000
[2,] 1.0000000 0.00000000
[3,] 0.6686391 0.33136095
[4,] 1.0000000 0.00000000
[5,] 0.9776536 0.02234637
[6,] 1.0000000 0.00000000
> head(test_forest.rev_labels_version5$predictions)
0 1
[1,] 1.0000000 0.00000000
[2,] 1.0000000 0.00000000
[3,] 0.7530120 0.24698795
[4,] 1.0000000 0.00000000
[5,] 0.9757576 0.02424242
[6,] 1.0000000 0.00000000The predictions for test_forest.rev_labels_version5 should show high predictions for class "1".
Some more comments:
- The same issue persists when choosing different labels, e.g. strings.
- Also, the same is true for the prediction of independent test data, it seems to be an issue of the training.
- Predicting a forest trained in version 0.4 yields the correct prediction when loaded in version 0.5.
- Vice versa, a forest trained in version 0.5 does not predict correct in version 0.4.
from ranger.
mnwright
commented on July 27, 2024
Thanks for this detailed analysis. I can confirm that this is a bug. The prediction is made on the levels in the order they appear in the data but the labels saved with R's levels() function, which sorts them alphabetically.
from ranger.
mnwright
commented on July 27, 2024
Fixed.
from ranger.
Eric2333
commented on July 27, 2024
When will this be available on CRAN?
from ranger.
mnwright
commented on July 27, 2024
There will probably be a CRAN release in October.
from ranger.
Related Issues (20)
- Performance Evaluation of Models HOT 3
- Perfect Association Between Covariates and Outcome Does Not Have Perfectly Ranked Survival Function Sums HOT 2
- Issue with respect.unordered.factors HOT 1
- Could ranger record samples used for each tree? HOT 1
- `treeInfo()`might be wrong if `class.values` are not ordered in ascending order HOT 2
- Error: Missing data in columns: pop when running random forest regression HOT 1
- split candidates HOT 1
- Can not specify the classes of a prediction outcome HOT 4
- Is it possible to add the cross-entropy method to the splitrule options? HOT 2
- Unexpected behaviour with combined `always.split` and `split.select.weights` HOT 1
- Define a custom loss function HOT 2
- R squared in stand alone version
- Error: Missing data in columns: predictor1, predictor2.... HOT 2
- Incorrect bounds check in beta log probability HOT 2
- Different predictions based on variable importance setting HOT 4
- `ranger()` finished, but `rsession` is still running HOT 8
- Trying to spatially predict(type="se")$se and the output has many null values and what look like spatial offsets? HOT 6
- compilation failed for package ‘ranger’ HOT 34
- ranger-cli fails to build HOT 1
- treeInfo() fails for probability trees with non-factor response HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ranger.