ENH: increase transparency of background dataset sub-sampling about shap HOT 4 OPEN

Thanks for the report and your effort to investigate this. Your description is absolutely accurate and the reason for this is the default in the tabular masker.

Here is an issue where this problem was already discussed including workaround: #3174.

We probably should throw at least a warning if max_samples < len(X). What do you thing @connortann ? This issue seems to come up and is confusing users.

from shap.

I agree with your analysis, this seems to be a consequence of sampling. I'll remove the bug label as I think this is indented behaviour.

We probably should throw at least a warning if max_samples < len(X)

I'm not sure if I agree. To me, warnings are generally used to indicate undesirable situations in which the user should probably update their code to fix the warning. In this case I think for the majority of users the subsampling is expected and desirable behaviour. Many parts of shap are sampling-based and only offer approximate results.

Would log.info() be more appropriate?

from shap.

logging.info is fine for me as well. I would be fine with a print as well, just to make sure that users do not have to investigate a couple hours to find the reason for the inconsistency between values and theory

from shap.

I would much prefer logging over print statements, as prints are much harder to configure and disable. I think adding a print would risk annoying a large majority of shap users.

I've renamed the title accordingly to reflect the plan.

from shap.