Creating Custom Metrics about keras HOT 6 CLOSED

Actually you might be able to do it by converting to R vectors and back using the K$eval() and K$constant() functions. For example (haven't tried this!):

metric_custom <- function( y_true, y_pred ) {
    # convert tensors to R objects
    K <- backend()
    y_true <- K$eval(y_true)
    y_pred <- K$eval(y_pred)
     
    # calculate the metric
    metric <- mean( y_true[ y_pred >= quantile(y_pred, .985) ] ) 

    # convert to tensor
    K$constant(metric)
}

If this works as expected for you I'll add this as an example to the documentation. Let me know!

from keras.

Hey, thanks for looking at it again.

Compiling with your function above produced an error:
InvalidArgumentError: Shape [-1,-1] has negative dimensions

Edit - is your last line valid? Coercing a scalar to a tensor?

from keras.

Custom metrics operate on tensors so you need to build these expressions using backend tensor functions. For example:

K <- backend()
mean_pred <- function(y_true, y_pred) {
  K$mean(y_pred) 
}

Docs on available back end functions are here: https://rstudio.github.io/keras/articles/backend.html

Note that even once you get this working the name of the metric won't show up properly in training progress (it will just say "python_function". Working on a change now to remedy this.

from keras.

I've added the ability to provide names for custom metrics and enhanced the docs a bit: file:///Users/jjallaire/packages/keras/docs/reference/metric_binary_accuracy.html#custom-metrics

from keras.

Yeah, I don't see that I'll be able to do it within those confines, BUT, to anyone else who might stumble on this thread - you can always Run an epoch, manually predict on validation, check your metric of interest, loop. Since prediction is slow with keras/TF it'll add a bit of time, but it's worth it in my case.

from keras.

Yes, you can indeed convert a scalar to a tensor, but perhaps Keras is expecting a different type? (there are other arguments to K$constant which can provide type/shape/etc

from keras.