mayer79 / flashlight Goto Github PK
View Code? Open in Web Editor NEWMachine learning explanations
Home Page: https://mayer79.github.io/flashlight/
License: GNU General Public License v2.0
Machine learning explanations
Home Page: https://mayer79.github.io/flashlight/
License: GNU General Public License v2.0
Add a flag "center" to light_ice
to center the curves at first (or better: middle) evaluation point. Can use something like `ave(..., FUN = scale, scale = FALSE) to achieve the task.
see title
Dear @mayer79,
I am trying to create a "human readable" models in social sciences. Researchers here understand
y = ax + bz + c
Where y and x and z were historically averages of multiple observations. It has been a domain idiosyncrasy. From what I see in DALEX, the SHAP scores are for individual profiles. Is it possible to generate additive SHAP scores for aggregate profiles in flashlight to make ML palettable in social sciences? Or is that a mathematical absurdity?
The current default leads to very unstable results.
Proposal:
Furthermore, pairwise = TRUE
could be considered as the better default as FALSE
.
The current implementation of pairwise interaction results in nC2 combinations where N is the number of variables v. Is there a way to create interaction between two groups? For example Group A = (a,b,c,d) and Group B (j,k) . In my case I am not interested in scenarios ab, bc, or jk, but only want to focus on aj, ak etc....
I see an option "by" Not sure if that can accomplish what I seek.
Hello,
Firstly, thank you very much for your very comprehensive package an your very clear vignette that I see as a concise but good reference for model's explainability.
I am a little bit doubtful about how to use flashlight
functionalities with h2o
models. I have not seen it covered in your vignette.
With some kind of guideline I can test it and share results.
Thanks again!
Carlos.
See title
Add option "mean" to light_ice
option center_at
. Use it for light_interaction
as default.
See title
Add Murphy diagrams as in https://cran.r-project.org/web/packages/murphydiagram/index.html, see also https://doi.org/10.1111/rssb.12154 and mayer79/MetricsWeighted#7.
Created a working model with xgboost tidymodels workflow. Checked if the model predicts accordingly. All seems ok. We get probability of conversion for a bunch of customer visitors to our page. All target factors are present and mutated to TRUE/FALSE
We get an error when plotting performance
mydata <- customer_data %>% dplyr::mutate(convert= if_else(convert== "converted", TRUE,FALSE))
fl <- flashlight(
model = xgb_fit,
label = "conversion",
y = "convert",
data = mydata,
metrics = list(AUC = AUC, f1_score = f1_score, recall = recall, logLoss = logLoss),
predict_function = function(model, data) predict(model, data, type = "prob")$.pred_converted
)
fl
Flashlight conversion
Model: Yes
y: convert
w: No
by: No
data dim: 53958 28
predict_fct default: FALSE
linkinv default: TRUE
metrics: AUC f1_score recall logLoss
SHAP: No
light_performance(fl) %>%
plot(fill = "orange") +
xlab(element_blank())
Error: Problem with `summarise()` input `..1`.
i `..1 = core_fun(cur_data())`.
x all(predicted == 0 | predicted == 1) is not TRUE
i The error occurred in group 1: label = "conversion".
We reproduced the error with vignette example just setting iris (ir) virginica column to virginica/notvirginica
Thanks
expand_grid
was introduced not before tidyr
1.0.0 and is required in flashlight
. Should add as minimum requirement.
flashlight is able to calculate approximate SHAP values. Next step is to provide tools to analyse these SHAP values (shap_importance, shap_profile, ...).
I am having trouble running the functions for classifications using Ranger, i.e.,
library(ranger)
library(flashlight)
library(MetricsWeighted)
test1<-data.frame(
resp=as.logical(round(runif(100))),
x1=rnorm(100),
x3=rnorm(100,10,5)
)
m1<-ranger(resp~.,
data=test1,
probability =F
)
fl <- flashlight(model = m1, data = test1, label = "rf",
metrics = list(logLoss = logLoss, AUC = AUC),
predict_function = function(mod, X) predict(mod, X)$predictions)
plot(light_importance(fl))
#Error: Problem with `summarise()` input `..1`.
#x all(predicted > 0 & predicted < 1) is not TRUE
#i Input `..1` is `core_fun(cur_data())`.
#i The error occurred in group 1: label = "rf".
#Run `rlang::last_error()` to see where the error occurred.
See topic. The idea is to be able to permute multiple times in order to get more stable results (and have a measure of uncertainty).
This issue collects a couple of ideas for the next CRAN release 0.8.0
kmeans
or quantile_inner
?) -> postpone until interface is clear.options(flashlight.metric_name = "metric", ...)
. This amounts to a major code update.Relevant PRs:
Ticked boxes mean the feature is already available in the master branch.
Hi @mayer79
I am curious how I can accomplish this article using your package. https://arxiv.org/abs/2006.04628
Here is my scenario
Hope you can advice. It does not need to be the above article. I want to hear your approach.
Profile plots could benefit from a trim_sd
argument, which would remove outliers before binning numeric variables. An outlier would e.g. be defined as a value > trim_sd
standard deviations away from the mean .
pr <- light_profile(flash, v = "some_variable", trim_sd = 3)
As alternative to partial dependence profiles, it would be nice to have ALE-profiles in light_profile
(and consequently also in light_effects).
Computation would be as follows:
Create break-vector (b_i) of x values as in light_profile
For each i >1, call light_ice
for the data subset with x in [b_{i-1}, b_{j}] (including limits) with just two evaluation points b_{i-1}, b_{j}.
For each i >1, aggregate the results per "by" level by averaging the differences of light_ice
. (If none, return NA)
cumsum within "by" levels.
Calibrate result (how?)
Hi,
First of all, I'd like to share my amazement with that package!
However, during my research work, I've encountered a weird behavior of function that generates variable profiles. Let me show it
data(titanic_imputed, package = "DALEX")
ranger_model <- ranger::ranger(survived~., data = titanic_imputed, classification = TRUE, probability = TRUE)
custom_predict <- function(X.model, new_data) {
predict(X.model, new_data)$predictions[,1]
}
fl <- flashlight(model = ranger_model, data = titanic_imputed, y = "survived", label = "Titanic Ranger",
metrics = list(auc = AUC), predict_function = custom_predict)
ale <- light_profile(fl, v = "fare", type = "ale")
plot(ale)
Here we see rather correct ALE plot for provided data. The general direction in those data should be the more particular passenger had paid, it's more possible he survived. However the plot has been creating using the probability of 0 class, this will be important. Now let's create pdp plot
custom_predict <- function(X.model, new_data) {
predict(X.model, new_data)$predictions[,1]
}
fl <- flashlight(model = ranger_model, data = titanic_imputed, y = "survived", label = "Titanic Ranger",
metrics = list(auc = AUC), predict_function = custom_predict)
pdp <- light_profile(fl, v = "fare", type = "partial dependence")
plot(pdp)
Using the same column, which contains the probability of belonging to 0 class, we get an inversed pdp plot, it shows that probability decreases along with fare value increase. To get a plot that seems proper I had to swap columns in custom_predict
so it indicates the probability of belonging to 1 class.
custom_predict <- function(X.model, new_data) {
predict(X.model, new_data)$predictions[,2]
}
fl <- flashlight(model = ranger_model, data = titanic_imputed, y = "survived", label = "Titanic Ranger",
metrics = list(auc = AUC), predict_function = custom_predict)
pdp <- light_profile(fl, v = "fare", type = "partial dependence")
plot(pdp)
Overall it looks like one of the function inverses the probabilities. Is it intended?
Best regards
Szymon Maksymiuk
Error in weighted_mean(preds, if (!is.null(x$w)) data[[x$w]], na.rm = TRUE) :
length(x) == length(w) is not TRUE
-> light_profile fix
The current calibration of ALE curves is biased towards the prediction of small v values. (Binary classification example in vignette even has negative values). This should be improved.
Not sure how to implement as plot function needs to be involved as well.
I use tidymodels framework for my models, and I guess if flashlight could be suitable for model explanation.
I have been searching for code examples but no success so far
any hint? (xgboost classification model)
thanks in advance
Will help to prevent problems with integer variables with a lot of different values if the predict function uses an "as.factor".
During the next months, "flashlight" will be revised with the following changes in mind:
::
instead of importFrom
#54ggplot2::aes_string()
by rlang::.Data
#54geom_line()
instead of size #54coord_flip()
in plot.light_breakdown()
and plot.light_importance()
#57add_shap()
in favour of {kernelshap} and {shapviz}. This will greatly simplify the code. The current implementation is too slow to be useful. #56plot_counts()
. Might be replaced by {ggside} later, see below. Plus heavy dependency with {cowplot} #57stats = "quartiles"
argument in profile functions. Too much code for too little gain. #57light_recode()
by plot.light_effects(..., recode_labels = ....)
#60light_*.multiflashlight()
functions in order to see type hints of RStudiolibrary(ggplot2)
library(ggside)
ggplot(iris, aes(Species, Sepal.Width)) +
geom_jitter(aes(color = Species)) +
geom_xsidebar(aes(fill=Species)) +
geom_ysidehistogram(bins = 9) +
theme_ggside_void()
see topic
light_importance function for permutation variable importance is not working with models built using mlr pipeline.
Decision Tree Model from Rpart
fl_rpart <- flashlight(model = dt_model, data = train, y = "x1", label = "rpart", metrics = list(auc = AUC), predict_function = function(mod, X) predict(mod, X)[,2])
imp_rpart <- light_importance(fl_rpart, m_repetitions = 5)
works as expected, but when i build same model on same dataset using mlr package -
fl_rpart_mlr <- flashlight(model = dt_model_mlr, data = train, y = "x1", label = "rpart_mlr", metrics = list(auc = AUC), predict_function = function(model, newdata) predict(model, newdata = train)$data$prob.1)
imp_rpart_mlr <- light_importance(fl_rpart_mlr, m_repetitions = 5)
I get importance of all the variables = 0.
I checked that output of predict_function is both the cases is same.
Can you highlight, if i am doing something wrong or the package can't be used with mlr models?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.