Getting different f1-score result about cutpointr HOT 3 CLOSED

Hi, thanks for the report. We have a unit test that checks if the F1-scores from cutpointr are identical to the ones returned by ROCR, so I hope the results from cutpointr are correct.

I'll look into it and will try to check why the difference occurs.

If you have a look at cutpointr::F1_score you'll see that the code is basically identical to yours:

function(tp, fp, tn, fn, ...) {
    f <- (2 * tp) / (2 * tp + fp + fn)
    f <- matrix(f, ncol = 1)
    colnames(f) <- "F1_score"
    return(f)
}

from cutpointr.

Upon reviewing the issue, I'm noticing that you're hardcoding the positive class to be 1, the negative class to be 0 and direction = ">=", so that values larger than the threshold indicate the positive class. As the output from cutpointr() states (also via the messages #> Assuming the positive class is 0 and #> Assuming the positive class has higher x values), it assumes that the positive class is 0 because it is the class with the higher median value for df_test$resp.

If you simply specify classes and direction as follows, you'll get an F1-score that is identical to the one from your function:

> cp <- cutpointr(data = df_test, x = pred, class = resp, 
+                 method = maximize_metric, metric = F1_score,
+                 pos_class = 1, direction = ">=")
> cp %>% select(optimal_cutpoint, F1_score)
# A tibble: 1 x 2
  optimal_cutpoint F1_score
             <dbl>    <dbl>
1      0.000465349 0.663995
> dice(pred = df_test$pred, label = df_test$resp, thr = cp$optimal_cutpoint)
[1] 0.6639947

from cutpointr.

I guess I was somehow confused ;-)
Thanks a lot for taking the time to review this and also for pointing out the reasonable messages.
Indeed super clear.

from cutpointr.