mayer79 / flashlight Goto Github PK

Add Murphy diagrams as in https://cran.r-project.org/web/packages/murphydiagram/index.html, see also https://doi.org/10.1111/rssb.12154 and mayer79/MetricsWeighted#7.

Created a working model with xgboost tidymodels workflow. Checked if the model predicts accordingly. All seems ok. We get probability of conversion for a bunch of customer visitors to our page. All target factors are present and mutated to TRUE/FALSE

We get an error when plotting performance

mydata <- customer_data %>% dplyr::mutate(convert= if_else(convert== "converted", TRUE,FALSE))
fl <- flashlight(
   model = xgb_fit,
   label = "conversion",
   y = "convert",
   data = mydata,
   metrics = list(AUC = AUC, f1_score = f1_score, recall = recall, logLoss = logLoss),
   predict_function = function(model, data) predict(model, data, type = "prob")$.pred_converted
 )

fl

Flashlight conversion 

Model:			 Yes
y:			 convert
w:			 No
by:			 No
data dim:		 53958 28
predict_fct default:	 FALSE
linkinv default:	 TRUE
metrics:		 AUC f1_score recall logLoss
SHAP:			 No

 light_performance(fl) %>% 
   plot(fill = "orange") +
   xlab(element_blank())

Error: Problem with `summarise()` input `..1`.
i `..1 = core_fun(cur_data())`.
x all(predicted == 0 | predicted == 1) is not TRUE
i The error occurred in group 1: label = "conversion".

We reproduced the error with vignette example just setting iris (ir) virginica column to virginica/notvirginica

Thanks

See title

As alternative to partial dependence profiles, it would be nice to have ALE-profiles in light_profile (and consequently also in light_effects).

Computation would be as follows:

1. Create break-vector (b_i) of x values as in light_profile

2. For each i >1, call light_ice for the data subset with x in [b_{i-1}, b_{j}] (including limits) with just two evaluation points b_{i-1}, b_{j}.

3. For each i >1, aggregate the results per "by" level by averaging the differences of light_ice. (If none, return NA)

4. cumsum within "by" levels.

5. Calibrate result (how?)

See title

This issue collects a couple of ideas for the next CRAN release 0.8.0

• 2D partial dependence plots, further 2D profile plots (residual, response, predicted, shap)
• Extend options to create grid points for profile plots (e.g. kmeans or quantile_inner?) -> postpone until interface is clear.
• More graceful error messages
• Add a cool logo
• Move the "name" arguments to options(flashlight.metric_name = "metric", ...). This amounts to a major code update.

Relevant PRs:

• #41
• #43
• #44

Ticked boxes mean the feature is already available in the master branch.

Add a flag "center" to light_ice to center the curves at first (or better: middle) evaluation point. Can use something like `ave(..., FUN = scale, scale = FALSE) to achieve the task.

when using the light_profile function the by option only works when one variable is entred, when more than one variable is used the error listed bellow is displayed. Previously one could enter multiple rows and get a data frame that has predictions based on the desired variables.
 
fit <- lm(Sepal.Length ~ ., data = iris)
fl <- flashlight(model = fit, label = "iris", data = iris, y = "Sepal.Length")

light_profile(fl, v = "Petal.Length",
by = c("Species","Sepal.Width"),
type = 'predicted')

Error in nm %in% nms && !is.null(x[[nm]]) && !(x[[nm]] %in% colnames(x$data)) :
'length = 2' in coercion to 'logical(1)'

light_importance function for permutation variable importance is not working with models built using mlr pipeline.

Decision Tree Model from Rpart
fl_rpart <- flashlight(model = dt_model, data = train, y = "x1", label = "rpart", metrics = list(auc = AUC), predict_function = function(mod, X) predict(mod, X)[,2])

imp_rpart <- light_importance(fl_rpart, m_repetitions = 5)

works as expected, but when i build same model on same dataset using mlr package -

fl_rpart_mlr <- flashlight(model = dt_model_mlr, data = train, y = "x1", label = "rpart_mlr", metrics = list(auc = AUC), predict_function = function(model, newdata) predict(model, newdata = train)$data$prob.1)

imp_rpart_mlr <- light_importance(fl_rpart_mlr, m_repetitions = 5)

I get importance of all the variables = 0.

I checked that output of predict_function is both the cases is same.

Can you highlight, if i am doing something wrong or the package can't be used with mlr models?

The current calibration of ALE curves is biased towards the prediction of small v values. (Binary classification example in vignette even has negative values). This should be improved.

The current implementation of pairwise interaction results in nC2 combinations where N is the number of variables v. Is there a way to create interaction between two groups? For example Group A = (a,b,c,d) and Group B (j,k) . In my case I am not interested in scenarios ab, bc, or jk, but only want to focus on aj, ak etc....

I see an option "by" Not sure if that can accomplish what I seek.

flashlight is able to calculate approximate SHAP values. Next step is to provide tools to analyse these SHAP values (shap_importance, shap_profile, ...).

Hi,

First of all, I'd like to share my amazement with that package!

However, during my research work, I've encountered a weird behavior of function that generates variable profiles. Let me show it

data(titanic_imputed, package = "DALEX")
ranger_model <- ranger::ranger(survived~., data = titanic_imputed, classification = TRUE, probability = TRUE)

custom_predict <- function(X.model, new_data) {
  predict(X.model, new_data)$predictions[,1]
}
fl <- flashlight(model = ranger_model, data = titanic_imputed, y = "survived", label = "Titanic Ranger",
                 metrics = list(auc = AUC), predict_function = custom_predict)

ale <- light_profile(fl, v = "fare", type = "ale")
plot(ale)

Here we see rather correct ALE plot for provided data. The general direction in those data should be the more particular passenger had paid, it's more possible he survived. However the plot has been creating using the probability of 0 class, this will be important. Now let's create pdp plot

custom_predict <- function(X.model, new_data) {
  predict(X.model, new_data)$predictions[,1]
}
fl <- flashlight(model = ranger_model, data = titanic_imputed, y = "survived", label = "Titanic Ranger",
                 metrics = list(auc = AUC), predict_function = custom_predict)

pdp <- light_profile(fl, v = "fare", type = "partial dependence")
plot(pdp)

Using the same column, which contains the probability of belonging to 0 class, we get an inversed pdp plot, it shows that probability decreases along with fare value increase. To get a plot that seems proper I had to swap columns in custom_predict so it indicates the probability of belonging to 1 class.

custom_predict <- function(X.model, new_data) {
  predict(X.model, new_data)$predictions[,2]
}
fl <- flashlight(model = ranger_model, data = titanic_imputed, y = "survived", label = "Titanic Ranger",
                 metrics = list(auc = AUC), predict_function = custom_predict)

pdp <- light_profile(fl, v = "fare", type = "partial dependence")
plot(pdp)

Overall it looks like one of the function inverses the probabilities. Is it intended?

Best regards
Szymon Maksymiuk

I am having trouble running the functions for classifications using Ranger, i.e.,

library(ranger)
library(flashlight)
library(MetricsWeighted)

test1<-data.frame(
  resp=as.logical(round(runif(100))),
  x1=rnorm(100),
  x3=rnorm(100,10,5)
)

m1<-ranger(resp~.,
            data=test1,
            probability =F
)

fl <- flashlight(model = m1, data = test1, label = "rf",
                 metrics = list(logLoss = logLoss, AUC = AUC),
                 predict_function = function(mod, X) predict(mod, X)$predictions)

plot(light_importance(fl))

#Error: Problem with `summarise()` input `..1`.
#x all(predicted > 0 & predicted < 1) is not TRUE
#i Input `..1` is `core_fun(cur_data())`.
#i The error occurred in group 1: label = "rf".
#Run `rlang::last_error()` to see where the error occurred.

see title

Not sure how to implement as plot function needs to be involved as well.

Will help to prevent problems with integer variables with a lot of different values if the predict function uses an "as.factor".

Hello,

Firstly, thank you very much for your very comprehensive package an your very clear vignette that I see as a concise but good reference for model's explainability.

I am a little bit doubtful about how to use flashlight functionalities with h2o models. I have not seen it covered in your vignette.

With some kind of guideline I can test it and share results.

Thanks again!
Carlos.

I use tidymodels framework for my models, and I guess if flashlight could be suitable for model explanation.

I have been searching for code examples but no success so far

any hint? (xgboost classification model)

thanks in advance

Error in weighted_mean(preds, if (!is.null(x$w)) data[[x$w]], na.rm = TRUE) :
length(x) == length(w) is not TRUE

-> light_profile fix

Profile plots could benefit from a trim_sd argument, which would remove outliers before binning numeric variables. An outlier would e.g. be defined as a value > trim_sd standard deviations away from the mean .

pr <- light_profile(flash, v = "some_variable", trim_sd = 3)

Hi @mayer79

I am curious how I can accomplish this article using your package. https://arxiv.org/abs/2006.04628

Here is my scenario

• 1000 predictors, a lot of whom are correlated with each other, However, I do not know which ones. There is no way for me to manually group them.
• I need to generate top-5 predictors for a field study.
• everybody tells me simple light_importance is wrong approach for my problem, but I do not have a guidance on what is correct way to generate top-5 from 1000 predictors

Hope you can advice. It does not need to be the above article. I want to hear your approach.

See topic. The idea is to be able to permute multiple times in order to get more stable results (and have a measure of uncertainty).

expand_grid was introduced not before tidyr 1.0.0 and is required in flashlight. Should add as minimum requirement.

The current default leads to very unstable results.

Proposal:

• Higher n_max of 1000 instead of 300
• Higher grid_size of 200 instead of 30.

Furthermore, pairwise = TRUE could be considered as the better default as FALSE.

Dear @mayer79,

I am trying to create a "human readable" models in social sciences. Researchers here understand
y = ax + bz + c
Where y and x and z were historically averages of multiple observations. It has been a domain idiosyncrasy. From what I see in DALEX, the SHAP scores are for individual profiles. Is it possible to generate additive SHAP scores for aggregate profiles in flashlight to make ML palettable in social sciences? Or is that a mathematical absurdity?

During the next months, "flashlight" will be revised with the following changes in mind:

• Modern code formatting #54
• :: instead of importFrom #54
• Replace ggplot2::aes_string() by rlang::.Data #54
• Use linewidth aesthetic in geom_line() instead of size #54
• Git actions
• Shorter (and less) vignettes #54
• Reduce package dependencies, e.g. {caret}, {xgboost}, {ranger}, {moderndive}, {mlr3}, {mlr3learners} #54
• Flip x and y coordinates, and remove coord_flip() in plot.light_breakdown() and plot.light_importance() #57
• Deprecate add_shap() in favour of {kernelshap} and {shapviz}. This will greatly simplify the code. The current implementation is too slow to be useful. #56
• Deprecate plot_counts(). Might be replaced by {ggside} later, see below. Plus heavy dependency with {cowplot} #57
• Deprecate stats = "quartiles" argument in profile functions. Too much code for too little gain. #57
• Remove the unnecessarly complex configuration of special columns like "label", "metric" etc. #57
• Hide internal functions #56 #57 #58
• Better examples #58
• Replace light_recode() by plot.light_effects(..., recode_labels = ....) #60
• Consider using nice default colors in the plots (e.g., via options()).
• Better interface to initialize and update flashlights and multiflashlights
• Consider using explicit argument lists in light_*.multiflashlight() functions in order to see type hints of RStudio
• Better unit tests

Backlog

• Consider using the "collapse" package for grouped (weighted) averages, counts, medians.

library(ggplot2)
library(ggside)

ggplot(iris, aes(Species, Sepal.Width)) + 
  geom_jitter(aes(color = Species)) + 
  geom_xsidebar(aes(fill=Species)) +
  geom_ysidehistogram(bins = 9) +
  theme_ggside_void()

Add option "mean" to light_ice option center_at. Use it for light_interaction as default.

see topic