SAGE with NLP/Huggingface about sage HOT 5 CLOSED

Yes I think that's right - ideally the global explanation would have confidence intervals, and these could be derived from the individual SHAP explanations. The SHAP values in each local explanation would need to have their own uncertainty (unfortunately not supported in all SHAP-related repositories), and the global value (the mean of the individual SHAP values) can be understood as a stratified sampling estimator (see here for more details). It could certainly be done!

Happy to help, and good luck with your project!

from sage.

Hi there, thanks for checking out the package. I haven't personally tested SAGE with anything NLP-related so it would be cool to try to make that work. But first, let's make sure this is a good problem for SAGE.

Typically, SAGE is used with tabular datasets because we can see what impact removing a feature has on the model's loss. With tabular datasets, each feature is always in the same place, whereas with text data the features are words but each can be located in different places in different sentences. In your case, is the goal to see how important a word is when it's included in a sentence? Or is it something else?

from sage.

Thanks so much for responding!! I think that's right. To clarify more, my primary goal is global explainability. Ideally, this would find some average importance of the word across a diverse set of contexts within the training data.

As I'm sure you're aware, the original SHAP package has support for NLP transformers. I've been using average SHAP values for global token importance, but since this package uses SHAP instead of loss SHAP, I think it misses out on the great theoretical properties discussed in your paper. I'd even be interested in seeing NLP support for Shapley Regression - you have too many cool papers to try out haha.

from sage.

Okay great, finding the global word importance is a good idea!

I think that SAGE could be modified to work in this setting, although it would require a bit more than just supporting Huggingface models. It would also require a different kind of estimation routine (most likely an extension of the permutation sampling approach, but I'll spare you the details for now). In the end, what you would get is a score for each word/token that basically represents how important it is, in the sense of how critical it is to make a good prediction.

That would be useful, but I think your original idea is just as useful, if not more: while this variation on SAGE would say how important a word is, the mean SHAP value would also give some directionality to the importance, indicating whether the word has a strong influence on pushing the predictions up or down on average. (I suppose whether this is interesting to you may depend on what problem you're looking at, e.g., sentiment analysis could be cool.)

So let me know what you think about that, if you're still interested in the SAGE route we can talk about the best way to get it implemented.

Also, thanks for checking out the other packages! Some quick thoughts on those:

1. Shapley regression probably should provide easier support for NLP models (it could be run with them right now, but it would require writing a little extra code), but it may not be that interesting to you as it's basically an improved version of KernelSHAP (it automatically uses the right number of sampling iterations and provides confidence intervals).
2. FastSHAP is a very recent paper about quickly estimating Shapley values using neural networks (vs. KernelSHAP, which runs a relatively slow estimation separately for each input to be explained). The paper focuses on tabular and image data, but at some point we (or someone else) will probably look into how it works with NLP models as well. Again though, it's basically a way to calculate local Shapley values (or local LossSHAP values) that is faster and/or more accurate than what you would use otherwise, it's not really about global explanations.

from sage.

I'll trust your expertise. You're actually not too far off the mark haha - my application is indeed sentiment related, so I agree average SHAP lends itself well to interpretability. I guess I'm content with what I have then. I suppose it would be nice to have confidence intervals, though maybe this should be up for future work. Perhaps you can pool together the local SHAP means/SEs from FastSHAP or Shapley regression to make a global explanation?

Thanks again for your very helpful comments and your incredible work in xAI!!

from sage.