Incorrect AUC ROC score and log loss. about evidently HOT 6 CLOSED

Hi @vikrant-sahu!
You got it right from the comments! We are aware of it and will fix it really soon :)
Could you share your use case when you got a wrong result? We will add it to the testing now.

from evidently.

If you want the data. Then sorry I can't share it as it a confidential dataset and can't be shared outside our network.

from evidently.

Hi @vikrant-sahu
Could share something of the below:

• the number of classes in the dataset
• the rough number of observations (are we talking thousands, or tens of thousands, or more)
• whether the metric was incorrect or did not compute at all.

That would help!

We understand it can be sensitive - appreciate the response in any case.

from evidently.

Cool..

• the number of classes in the dataset - 2 classes
• the rough number of observations (are we talking thousands, or tens of thousands, or more) - thousands, within 10K
• whether the metric was incorrect or did not compute at all. It was incorrect. I used sklearn and it gave different metric. Even AUC curve was incorrect. Also please check other curves too.

from evidently.

Hi @vikrant-sahu,

Apologies for the late reply! (🦠)

Thanks for your answers! We are still struggling to reproduce the bug. Here are a couple ideas we have on why the calculation is wrong.

1. Could you please check if your target (positive) class is the first in the column-mapping ['prediction'] list?

In other tools including sklearn, in order to calculate the ROC AUC metric, you have to explicitly pass the probabilities for the target class. Evidently expects the class columns to follow the order. If it is reversed, the metrics will be calculated incorrectly.

2. Do you pass the target function as is (categorical label) or there is some post-processing (e.g. you pass the class label like "0" or "1")?

If you pass a categorical label to Evidently, it should automatically binarize it. However, the problem might have happened at this stage, and we'd look further!

Overall, we believe that if you:

• pass a binarized target label to the Evidently (1 for target class and 0 for other); and
• make sure that positive class is the first in the column-mapping ['prediction'] list
  the calculation should be correct.

If it is not, please let us know!

from evidently.

Hi @vikrant-sahu!

We found a bug and fixed it in #122

from evidently.