model_diagnostics fails due to no rows to aggregate about modelstudio HOT 10 CLOSED

Hi, I presume your modelstudio is working just fine and you are only worried about the warning?

As the message suggests, something went wrong in the ingredients::describe(<ceteris paribus object>) function call, which produces textual description of the ceteris paribus explanation. This is an experimental feature to display text when you hover on the D button next to your plot.

For me to debug it further, it would be best to obtain the updated model/data.

from modelstudio.

Hi @MJimitater, without code and/or any reproducible example, e.g. on a different data/model, I might not be able to help you.

from modelstudio.

Hi @hbaniecki, this is the code that I used

explain <- DALEX::explain(trained_model, data = train[1:train_size,], y = as.numeric(train[1:train_size,]$PRUEFERGEBNIS)-1, type = "classification", label = "xgboost", predict_function = 
                              function(trained_model, obs){
                                previous_na_action <- options('na.action')
                                options(na.action='na.pass')
                                sparse_matrix_test <- sparse.model.matrix(PRUEFERGEBNIS ~., data = obs)
                                options(na.action=previous_na_action$na.action)
                                
                                results_test <- predict(trained_model, sparse_matrix_test, type = "response")
                                round(results_test,3)
                              }, predict_function_target_column = 0)
  
  new_observation <- test[ind_obs, , drop = FALSE]
  
  xai_dashboard <- modelStudio(explain, new_observation = new_observation)

I hope this helps somewhat.
If this doesn't help much, then we can see how to provide the model and data. Thanks

from modelstudio.

Hi @MJimitater, of what class is the object passed to data? Does the following example help you by any chance https://modelstudio.drwhy.ai/articles/ms-r-python-examples.html#xgboost-dashboard?

Googling the error message, I'm unsure if this issue is related to DALEX and modelStudio.

Might try to debug it if you provide data/code.

from modelstudio.

Hi @hbaniecki , the data object passed to explain() is a "data.table" "data.frame" and train[1:train_size,] the first train_size -rows.

Yes, Im also unsure if this issue is related to modelStudio.. is there way to provide data/code confidentially?

from modelstudio.

Can you serialize both in R, so I can load them into my environment and share with me at [email protected]? Data can be small (a few rows). You can also create a simpler xgboost model where the error still occurs--perhaps a small subset of features suffices.

Anyhow, I don't have your email to reach out.

from modelstudio.

@hbaniecki Thanks so much for your help! Just now, I sent you an email with the model and data in .rds format. I hope you can reproduce, keep me posted ;) Cheers

from modelstudio.

Hi @hbaniecki sorry for my late reply, Thanks again for your excellent help in debugging!
I have thoroughly examined the data for NA-values and have removed or imputed them in the numeric features, and changed them to "(Missing)" in the factorial features (by fct_explicit_na()) hoping that the problem with the variable_splits is solved. I do hope that this is a valid approach to generating error-free model- and predict profile explanations.
However , running modelStudio(explain, new_observation, B=3) with the updates runs in the following error:

 Calculating ingredients::ceteris_paribus (1)         
  Calculating ...                                                                                             
    Elapsed time: 00:10:37 ETA:10s Warning messages:
1: In value[[3L]](cond) : 
Error occurred in ingredients::describe.ceteris_paribus function: missing value where TRUE/FALSE needed
2: In value[[3L]](cond) : 
Error occurred in ingredients::describe.ceteris_paribus function: missing value where TRUE/FALSE needed

Now Im trying to find where the bug occurs, but seemingly everything works so far:

#doesn't work:
ms <- modelStudio(explain, new_observation, B=3)

#works:
mp <- model_performance(explain)

#works:
mr <- model_diagnostics(explain)

#works:
fi <- model_parts(explain, B=2)
plot(fi)

#works:
pdp <- model_profile(explain)

#works:
pd <- ingredients::partial_dependence(explain) 

#works:
pp <- predict_profile(explain, new_observation)

#both work:
cp <- ingredients::ceteris_paribus(explain, new_observation)
pdp <- ingredients::aggregate_profiles(cp, type = "partial")

Now Im wondering that cp <- ingredients::ceteris_paribus(explain, new_observation) worked, but somehow still throws this error in modelStudio. What else can I try? What other functions can I try to unit test in order to debug? Do you think this is still a problem with the "(Missing)"-values in the categories? Thank you so much

from modelstudio.

Thanks for your fast reply, I'll get back to you next week! ;)

from modelstudio.

Actually you are right, modelStudio works just fine! Finally the explanation of single test cases works without errors!

BTW, would be nice to have a faster way of explaining and visualizing individual test cases, than waiting a lengthy time for modelStudio to finish calculating, but I guess this the way it is ;) Great software!

Thanks again for the excellent support, closing for now

from modelstudio.