Git Product home page Git Product logo

keras-metrics's Introduction

Keras Metrics

Deprecation Warning

Since Keras version 2.3.0, it provides all metrics available in this package. It's preferrable to use metrics from the original Keras package.

This package will be maintained for older version of Keras (<2.3.0).

Build Status

This package provides metrics for evaluation of Keras classification models. The metrics are safe to use for batch-based model evaluation.

Installation

To install the package from the PyPi repository you can execute the following command:

pip install keras-metrics

Usage

The usage of the package is simple:

import keras
import keras_metrics as km

model = models.Sequential()
model.add(keras.layers.Dense(1, activation="sigmoid", input_dim=2))
model.add(keras.layers.Dense(1, activation="softmax"))

model.compile(optimizer="sgd",
              loss="binary_crossentropy",
              metrics=[km.binary_precision(), km.binary_recall()])

Similar configuration for multi-label binary crossentropy:

import keras
import keras_metrics as km

model = models.Sequential()
model.add(keras.layers.Dense(1, activation="sigmoid", input_dim=2))
model.add(keras.layers.Dense(2, activation="softmax"))

# Calculate precision for the second label.
precision = km.binary_precision(label=1)

# Calculate recall for the first label.
recall = km.binary_recall(label=0)

model.compile(optimizer="sgd",
              loss="binary_crossentropy",
              metrics=[precision, recall])

Keras metrics package also supports metrics for categorical crossentropy and sparse categorical crossentropy:

import keras_metrics as km

c_precision = km.categorical_precision()
sc_precision = km.sparse_categorical_precision()

# ...

Tensorflow Keras

Tensorflow library provides the keras package as parts of its API, in order to use keras_metrics with Tensorflow Keras, you are advised to perform model training with initialized global variables:

import numpy as np
import keras_metrics as km
import tensorflow as tf
import tensorflow.keras as keras

model = keras.Sequential()
model.add(keras.layers.Dense(1, activation="softmax"))
model.compile(optimizer="sgd",
              loss="binary_crossentropy",
              metrics=[km.binary_true_positive()])

x = np.array([[0], [1], [0], [1]])
y = np.array([1, 0, 1, 0])

# Wrap model.fit into the session with global
# variables initialization.
with tf.Session() as s:
    s.run(tf.global_variables_initializer())
    model.fit(x=x, y=y)

keras-metrics's People

Contributors

aronhoff avatar ashutosh1919 avatar chrissype avatar dgrahn avatar janstrohbeck avatar jeffgreenca avatar michetonu avatar ybubnov avatar yiqinzhao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

keras-metrics's Issues

Example for how to use metric with multi-label (ValueError for more than 2 classes)

Since this issue (#23) it should be possible to have a y array of longer than 2, however I cannot get it to work.

I have 5 classes, so I tried to use precision for 1 label with:

precision = keras_metrics.precision(label=0)

However, this results in the error:

~/anaconda3/envs/dl/lib/python3.6/site-packages/keras_metrics/metrics.py in _categorical(self, y_true, y_pred, dtype)
     46             return self._binary(y_true, y_pred, dtype, label=1)
     47         elif labels > 2:
---> 48             raise ValueError("With 2 and more output classes a "
     49                              "metric label must be specified")
     50 

ValueError: With 2 and more output classes a metric label must be specified

It seems like it only looks at the y shape, and not whether a label is specified?

Would this be better?:

def _categorical(self, y_true, y_pred, dtype, label=None):
    labels = y_pred.shape[-1]
    if labels == 2:
        label=1

    if labels > 2 and label:  # label != None
        raise ValueError("With 2 and more output classes a metric label must be specified")
    else:
        return self._binary(y_true, y_pred, dtype, label=label)

sparse_categorical metrics do not work for tensorflow 2.7

This is the great work!
Though tf and keras have official recall() and precision() metrics, the metrics only worked for binary classfication.
We had a problem to classify three categories. And we would like to figure out the recall, precision metrics for the each class. In our model, the last layer is Dense layer with active function 'softmax'. The loss function is 'sparse_categorical_crossentropy', as we used class label for y.

output = Dense(3, activation='softmax')(attention_mul)
model.compile(loss=tf.keras.losses.sparse_categorical_crossentropy, optimizer='Adam', metrics=['accuracy'])

The output of the prediction result is a vector of the probability of the each class, e.g. [0.3, 0.5, 0.2].To get the class label, we need to apply np.argmax() for the prediction results. And thus, the official recall and precision metrics do not work!
I found there are sparse_categorical metrics in this project. I tried the following code to get the metric for the class label 1, and passed the metrics to model compile.

recall_1 = km.sparse_categorical_recall(label=1)
precision_1 = km.sparse_categorical_precision(label=1)
metrics = ['accuracy', recall_1, precision_1]

Here are error messages we got:

Epoch 1/2000
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:00.580176 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:00.776434 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
/usr/local/python3.8/lib/python3.8/site-packages/keras_metrics/metrics.py:26: UserWarning: `layer.updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
  return object.__getattribute__(self, name)
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:01.025048 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:01.229791 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:03.521585 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:03.548538 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:03.596331 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:03.622677 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:11.411682 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:11.435597 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:11.480696 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:`add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
W0901 17:11:11.506137 139653739816768 base_layer.py:1764] `add_update` `inputs` kwarg has been deprecated. You no longer need to pass a value to `inputs` as it is being automatically inferred.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras_metrics.metrics.recall object at 0x7f02f39d6d30>, because it is not built.
W0901 17:11:14.928323 139653739816768 save_impl.py:71] Skipping full serialization of Keras layer <keras_metrics.metrics.recall object at 0x7f02f39d6d30>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras_metrics.metrics.precision object at 0x7f02f2d1dac0>, because it is not built.
W0901 17:11:14.929263 139653739816768 save_impl.py:71] Skipping full serialization of Keras layer <keras_metrics.metrics.precision object at 0x7f02f2d1dac0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras_metrics.metrics.true_positive object at 0x7f02f3701700>, because it is not built.
W0901 17:11:14.929968 139653739816768 save_impl.py:71] Skipping full serialization of Keras layer <keras_metrics.metrics.true_positive object at 0x7f02f3701700>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras_metrics.metrics.false_negative object at 0x7f02f36ca8e0>, because it is not built.
W0901 17:11:14.930354 139653739816768 save_impl.py:71] Skipping full serialization of Keras layer <keras_metrics.metrics.false_negative object at 0x7f02f36ca8e0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras_metrics.metrics.true_positive object at 0x7f02f2dc4970>, because it is not built.
W0901 17:11:14.930832 139653739816768 save_impl.py:71] Skipping full serialization of Keras layer <keras_metrics.metrics.true_positive object at 0x7f02f2dc4970>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras_metrics.metrics.false_positive object at 0x7f02f2d24580>, because it is not built.

Traceback (most recent call last):
  File "main.py", line 537, in <module>
    tf.compat.v1.app.run()
  File "/usr/local/python3.8/lib/python3.8/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/python3.8/lib/python3.8/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/usr/local/python3.8/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "main.py", line 476, in main
    model.train(stock_data_list, test_ratio)
  File "/root/work/branch/infinity_stock3/models/model_attention_three_category.py", line 101, in train
    self.model.fit(train_g, steps_per_epoch=math.ceil(train_len / self.batch_size),
  File "/usr/local/python3.8/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/python3.8/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py", line 53, in _python_properties_internal
    metadata.update(get_serialized(self.obj))
ValueError: dictionary update sequence element #0 has length 1; 2 is required

Failure when model.fit parameter verbose set to 2

Tensorflow backend
Model parameters: 2 GRU followed by 1 dense, monitor binary crossentropy

Model tested with metrics precision and recall, fails with parameter verbose set to 2, passes when parameter is set to 0 or 1.

model = Sequential()
model.add(GRU(45, input_shape=[maximum_smiles_size, char_vec_len], return_sequences=True, go_backwards=True,
activation='tanh', recurrent_regularizer=rgr.l2(l2)))
model.add(GRU(30, activation='tanh', recurrent_dropout=.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer=RMSprop(lr=0.0007), metrics=[keras_metrics.precision(), keras_metrics.recall()])
history = model.fit(train_x[0:7000,:,:], train_y[0:7000], epochs=200, batch_size=100,
validation_data=(test_x, test_y), verbose=2)
Traceback (most recent call last):

File "mypathdatanotshowingitonreddit", line 220, in <module>

validation_data=(test_x, test_y), verbose=2)

File "mypathdatanotshowingitonreddit/lib/python3.5/site-packages/keras/models.py", line 963, in fit

validation_steps=validation_steps)

File "mypathdatanotshowingitonreddit/lib/python3.5/site-packages/keras/engine/training.py", line 1705, in fit

validation_steps=validation_steps)

File "mypathdatanotshowingitonreddit/lib/python3.5/site-packages/keras/engine/training.py", line 1255, in _fit_loop

callbacks.on_epoch_end(epoch, epoch_logs)

File "mypathdatanotshowingitonreddit/lib/python3.5/site-packages/keras/callbacks.py", line 77, in on_epoch_end

callback.on_epoch_end(epoch, logs)

File "mypathdatanotshowingitonreddit/lib/python3.5/site-packages/keras/callbacks.py", line 339, in on_epoch_end

self.progbar.update(self.seen, self.log_values)

File "mypathdatanotshowingitonreddit/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 426, in update

self._values[k][0] / max(1, self._values[k][1]))

IndexError: invalid index to scalar variable.

Average recall returns zero value for a single class

Given the following model:

y_pred = [[0], [1], [1]]

model = keras.models.Sequential()
model.add(keras.layers.Lambda(lambda x: x+y_pred))
model.compile(optimizer="sgd", loss="binary_crossentropy",
              metrics=[km.binary_average_recall(classes=1)])

x = numpy.array([[0],[0],[0]])
y = numpy.array([0, 1, 1])

model.fit(x, y, epochs=1)
AR = model.evaluate(x, y)[1:]
print(AR)

The expected average recall is 1.0, while the output is 0.0

What about multi-class classifiers?

Was just going through the code, not sure if I understand this correctly.

Are these metrics only good for binary classifiers? How does one interpret the output for precision/recall if a multi-class classifier is being trained?

Binary classification with 2 units gives wrong precision and recall

Hi, I've seen the similar issue that was about evaluation after training on multi-class classification, mine is about the training. If one has only one unit at the end, everything is correct, however if dense layer with two units is used for the last layer with a softmax activation, true positive is always equals to true negative, and false positive is always equals to false negative in every batch during the training. And when you sum all of them you obtain a number that is exactly equals to twice of batch size. Please kindly check my code below.
Note: I'm aware of the fact that these metrics shouldn't be checked per batch basis, but seeing something wrong pushed me to create this issue. By the way, thanks for the library and effort.

https://colab.research.google.com/drive/1lmQ-hWcN4tsGMicd4dKnSjeTD-BdgJuE

Input support for Tensorflow.data.Dataset

Hello! :)
Firstly I would like to thank you for this greatly useful repository!

Secondly I would like to ask:
Do the metrics functions you provide support an input of Tensorflow.data.Dataset, like the normal metrics of keras does?

"ValueError: too many values to unpack (expected 2)" on using precision and recall metrics for binary segmentation

I'm trying to solve a binary segmentation problem with a highly imbalanced dataset, so tracking the recall and precision is useful. I'm using a keras implementation of a CNN.

imgs_train, imgs_mask_train, imgs_test, imgs_validation, mask_validation = self.load_data()
model = self.get_unet()
weighted_loss=weighted_pixelwise_crossentropy(np.array([1,self.loss_weight]))

adam = optimizers.Adam()
precision = keras_metrics.precision() 
recall = keras_metrics.recall() model.compile(loss=weighted_loss,optimizer=adam, metrics=['accuracy', precision, recall])
model_checkpoint = ModelCheckpoint(self.experiment_id+'.hdf5', monitor='loss',verbose=1, save_best_only=True, save_weights_only= True)
csv_logger = CSVLogger(self.augmented_datapath+"_"+self.experiment_id+'.csv', append=True, separator=';')
model.fit(imgs_train, imgs_mask_train, validation_data=[imgs_validation, mask_validation], epochs=20, verbose=1, callbacks=[model_checkpoint, csv_logger])

Here's a snippet of the code I'm using. load_data() returns numpy arrays for the training, testing, and validation data, and the model returns a (512, 512, 2) array, with 1s in one of the two columns for each pixel.

Traceback (most recent call last):
  File "job_array_nnet.py", line 74, in <module>
    job_array(args.array_id)
  File "job_array_nnet.py", line 66, in job_array
    nnet.train_neuralnet()
  File "/mainfs/home/jw22g14/DeepSVM/neural_net_model.py", line 172, in train_neuralnet
    model.compile(loss=weighted_loss,optimizer=adam, metrics=['accuracy', precision, recall])
  File "/home/jw22g14/.conda/envs/python3/lib/python3.6/site-packages/keras/engine/training.py", line 440, in compile
    handle_metrics(output_metrics)
  File "/home/jw22g14/.conda/envs/python3/lib/python3.6/site-packages/keras/engine/training.py", line 409, in handle_metrics
    mask=masks[i])
  File "/home/jw22g14/.conda/envs/python3/lib/python3.6/site-packages/keras/engine/training_utils.py", line 403, in weighted
    score_array = fn(y_true, y_pred)
  File "/home/jw22g14/.conda/envs/python3/lib/python3.6/site-packages/keras_metrics/metrics.py", line 218, in __call__
    tp = self.tp(y_true, y_pred)
  File "/home/jw22g14/.conda/envs/python3/lib/python3.6/site-packages/keras_metrics/metrics.py", line 72, in __call__
    y_true, y_pred = self.cast(y_true, y_pred)
  File "/home/jw22g14/.conda/envs/python3/lib/python3.6/site-packages/keras_metrics/metrics.py", line 26, in cast
    return self.cast_strategy(y_true, y_pred, dtype=dtype)
  File "/home/jw22g14/.conda/envs/python3/lib/python3.6/site-packages/keras_metrics/metrics.py", line 44, in _categorical
    _, labels = y_pred.shape

And here's the stack trace.

load_model problem

keras-metrics is useful in the process of training

model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.Adadelta(), metrics=['accuracy', keras_metrics.precision(), keras_metrics.recall()])

during the training, it reported this:

1766016/1839646 [=====>..] - ETA: 2s - loss: 0.3919 - acc: 0.8308 - precision: 0.6595 - recall: 0.3231
1767424/1839646 [=====>..] - ETA: 2s - loss: 0.3919 - acc: 0.8308 - precision: 0.6595 - recall: 0.3231
1768832/1839646 [=====>..] - ETA: 2s - loss: 0.3920 - acc: 0.8307 - precision: 0.6594 - recall: 0.3231
1770368/1839646 [=====>..] - ETA: 2s - loss: 0.3920 - acc: 0.8307 - precision: 0.6594 - recall: 0.3232
1771776/1839646 [=====>..] - ETA: 2s - loss: 0.3919 - acc: 0.8308 - precision: 0.6595 - recall: 0.3232

and I save the model with the code:
model.save(model_save_path + '/mlp_pssm_%s.h5' % short_mid_long)

things going well, so far.

however, when I was trying to load the model with the code:
clf = load_model(models_path + '/%s_%s.h5' % (model_type,contact_type))

I got this error:
ValueError: Unknown metric function: {'class_name': 'precision', 'config': {'name': 'precision', 'trainable': True}}

How to fix this error?
Thanks!

Metrics are not pickleable

Instantiated metrics are not pickleable, as __init__ instantiates tf variables, which cannot be pickled.

This means that keras models trained with one of the metrics, and then saved, cannot be opened without knowing the dict structure for the 'custom_objects' argument when loading the model. Without using dill, one cannot pickle said dict, which would make delivering a complete model possible. Tools like mlflow are able to track and return complete keras models, however only if the model and custom_objects are both pickleable.

I've come up with a moderately ugly solution that waits until a metric is called to instantiate any tf variables, let me know if you want to see it.

What does the `reset_states` do?

It seems that the method reset_state in metrics resets the stored values. However, I am not sure when it should be used. Is it for resetting states at the end of each epoch?

According to my understanding, the keras-metrics is designed to avoid the incorrect approximation of recall on each batch. Thus, a practical solution is computing the metrics on the end of each epoch independently.

But in the README.md, the given example is

import keras
import keras_metrics

model = models.Sequential()
model.add(keras.layers.Dense(1, activation="sigmoid", input_dim=2))
model.add(keras.layers.Dense(1, activation="softmax"))

model.compile(optimizer="sgd",
              loss="binary_crossentropy",
              metrics=[keras_metrics.precision(), keras_metrics.recall()])

which directly pass the keras_metrics.recall() as metrics for batch-based usage. The problem in the demo is that the states may* not be resetted. Therefore, the recall value of each epoch will be dependent on previous epochs.

* I am not sure if the reset_states method is called at the end of each epoch.

TypeError: 'NoneType' object is not callable

I tried to customize keras-metrics, because i want to use a balanced_accuracy as metric. So i added keras-metrics in a folder of my project, but i can't modify ìt as well.
In metrics.py:
def __init__(self, name=None, label=0, cast_strategy=None, **kwargs): super(layer, self).__init__(name=name, **kwargs) self.stateful = True self.label = label self.cast_strategy = cast_strategy self.epsilon = K.constant(K.epsilon(), dtype="float64")

but cast_strategy is called, there is
return self.cast_strategy( y_true, y_pred, dtype=dtype, label=self.label)
cast strategy is not callable! Can someone help me? I'm working for my master thesis.

f1_score: nan

At some low threshold of precision and recall, f1_score becomes nan. I'm getting outputs like this:

2306/10000 [=====>........................] - ETA: 21:25 - loss: 0.7265 - precision: 0.5095 - recall: 0.4593 - f1_score: nan

Precision and Recall and F1-Score > 1 ?

Hi everybody,
I was using the library on my training and everything looks good.
This is an example:

22262/39088 [================>.............] - ETA: 1:02:06 - loss: 5.7241 - acc: 0.9064 - precision: 0.7208 - recall: 0.9263 - f1_score: 0.8108

Until it arrives to the end of the epoch where it has some weird behaviors:

Epoch 1/100
39088/39088 [==============================] - 8798s 225ms/step - loss: 9.6921 - acc: 0.8581 - precision: 2.9728 - recall: 1.2057 - f1_score: 1.7156 - val_loss: 7.8221 - val_acc: 0.8764 - val_precision: 0.6613 - val_recall: 0.9001 - val_f1_score: 0.7624

The precision / recall / f1-score for the validation look good, but for the training they have a value bigger than 1.
They should remain always less then 1, is it?
Thanks

Non-Sensical Print Values For Metrics

Hi,
While training a binary classifier with binary_crossentropy output layer, and binary labels in Keras (TF - GPU backend) using your keras_metrics package, I've been getting lots of f1-score values printed as: 0.0000e+00

It wasn't making any sense as I was seeing a drop in loss, and an increase in accuracy (either/both for train/val). I decided to dig it up, and started printing everything available in keras_metrics.

Realized I am getting all the metrics from keras_metrics printed as 0.0000e+00 !

I get true positive, true negatives, false positives, false negatives, precision, recall, f1-score (each is binary like binary_recall etc), all of them printed as 0.0000e+00.

There must either be a bug, or a catastrophic division or something propagating.

I have the following module versions:

sklearn '0.21.3'
tensorflow gpu '0.21.3'
keras '2.2.4'
numpy '1.16.4'
scipy '1.3.1'
pandas '0.25.0'
keras_metrics '1.1.0'

Do you know what may be the issue? It stopped my progress big time, before I even realize.

Best,
Onur

Metrics does not work with tf.keras.estimator.model_to_estimator

I am trying to use tf.keras.estimator.model_to_estimator to convert tf.keras model to be distributed, however, I found that keras-metrics does not work as desired, Is there any idea or work around for me ? thanks

Traceback:

Traceback (most recent call last):
  File "1.py", line 204, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "1.py", line 190, in main
    config=Rcfg
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/estimator/__init__.py", line 73, in model_to_estimator
    config=config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 486, in model_to_estimator
    config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 354, in _save_first_checkpoint
    custom_objects)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 201, in _clone_and_build_model
    optimizer_iterations=global_step)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 511, in clone_and_build_model
    target_tensors=target_tensors)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/checkpointable/base.py", line 442, in _method_wrapper
    method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training.py", line 499, in compile
    sample_weights=self.sample_weights)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1844, in _handle_metrics
    return_stateful_result=return_stateful_result))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1800, in _handle_per_output_metrics
    stateful_metric_result = _call_stateful_fn(stateful_fn)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1773, in _call_stateful_fn
    fn, y_true, y_pred, weights=weights, mask=mask)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_utils.py", line 852, in call_metric_function
    return metric_fn(y_true, y_pred, sample_weight=weights)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/metrics.py", line 438, in __call__
    update_op = self.update_state(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/metrics.py", line 160, in inner
    return func.__get__(instance_ref(), cls)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/metrics.py", line 98, in decorated
    update_op = update_state_fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/metrics.py", line 649, in update_state
    matches = self._fn(y_true, y_pred, **self._fn_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras_metrics/metrics.py", line 192, in __call__
    tp = self.tp(y_true, y_pred)
  File "/usr/local/lib/python2.7/dist-packages/keras_metrics/metrics.py", line 50, in __call__
    tp_update = K.update_add(self.tp, tp)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 986, in update_add
    return tf.assign_add(x, increment)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 190, in assign_add
    ref, value, use_locking=use_locking, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 107, in assign_add
    "AssignAdd", ref=ref, value=value, use_locking=use_locking, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 350, in _apply_op_helper
    g = ops._get_graph_from_inputs(_Flatten(keywords.values()))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 5713, in _get_graph_from_inputs
    _assert_same_graph(original_graph_element, graph_element)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 5649, in _assert_same_graph
    original_item))
ValueError: Tensor("metrics/precision/Sum:0", shape=(), dtype=int32) must be from the same graph as Tensor("Variable:0", shape=(), dtype=int32_ref).

Buggy code (a little bit messy...)

model.py.zip

If I dont add keras_metrics.sparse_categorical_precision() into Accuracy part, it DOES work but fail when I add sparse_categorical_precision...

Tested in Py2.7/3.7 TF 1.13.1

Discrepancy between keras-metrics and scikit-learn

Hi all,

Wondering if you might be able to shed some light on what's going on here. Is this a bug? Thanks.

I'm using:
tensorflow gpu 1.13.1
keras 2.2.4 (very latest pip installed form github repo)
keras-metrics 1.1.0
numpy 1.16.4
scikit-learn 0.21.2

Here's the situation...

I'm training a ResNet on a multiclass problem (seven classes total). I'm trying to track the precision, recall and F1 for each class at each epoch. If I compare the validation set output from the last epoch with the values in that scikit learn calculates in its classification report after calling predict, they are vastly different.

For example, after 3 epochs the precision, recall and F1 of each class in the validation set is:

val_precision: 0.5000
val_precision_1: 0.3333
val_precision_2: 0.6000
val_precision_3: 0.3333
val_precision_4: 0.5641
val_precision_5: 0.8972
val_precision_6: 0.3500

val_recall: 0.0312
val_recall_1: 0.0196
val_recall_2: 0.0275
val_recall_3: 0.0909
val_recall_4: 0.1982
val_recall_5: 0.8075
val_recall_6: 0.5000

val_f1_score: 0.0588
val_f1_score_1: 0.0370
val_f1_score_2: 0.0526
val_f1_score_3: 0.1429
val_f1_score_4: 0.2933
val_f1_score_5: 0.8500
val_f1_score_6: 0.4118

But the scikit-learns confusion matrix and classification report shows:

Confusion matrix
[[ 0 0 28 0 4 0 0]
[ 0 0 44 0 7 0 0]
[ 0 0 102 0 7 0 0]
[ 0 0 11 0 0 0 0]
[ 0 0 99 0 12 0 0]
[ 0 0 657 0 13 0 0]
[ 0 0 14 0 0 0 0]]

Classification Report
precision recall f1-score support

       0       0.00      0.00      0.00        32
       1       0.00      0.00      0.00        51
       2       0.11      0.94      0.19       109
       3       0.00      0.00      0.00        11
       4       0.28      0.11      0.16       111
       5       0.00      0.00      0.00       670
       6       0.00      0.00      0.00        14

    accuracy                                0.11       998
 macro avg       0.06      0.15      0.05       998

weighted avg 0.04 0.11 0.04 998

Here's my code:

import numpy as np
np.random.seed(1)

import tensorflow as tf
tf.set_random_seed(1)

import random as rn
rn.seed(1)

import keras
from keras import layers, models, optimizers
from keras.preprocessing.image import ImageDataGenerator
from sklearn.metrics import confusion_matrix, classification_report
from keras_applications.resnet import ResNet50
from math import ceil
import keras_metrics as km



train_images = np.load('path to tensor')
train_labels = np.load('path to tensor')

validation_images = np.load('path to tensor')
validation_labels = np.load('path to tensor')

input_height = 150
input_width = 150
input_depth = 3

num_train_images = len(train_images)
num_validation_images = len(validation_images)


steps_per_epoch = ceil(num_train_images / 32)
validation_steps = ceil(num_validation_images / 32)


train_labels = keras.utils.to_categorical(train_labels, 7)
validation_labels = keras.utils.to_categorical(validation_labels, 7)


train_datagen = ImageDataGenerator(rescale=1./255,
                                   dtype='float32')


val_datagen = ImageDataGenerator(rescale=1./255,
                                 dtype='float32')

train_datagen.fit(train_images)
val_datagen.fit(validation_images)


train_generator = train_datagen.flow(train_images,
                                     train_labels,
                                     batch_size=32)

validation_generator = val_datagen.flow(validation_images,
                                               validation_labels,
                                               batch_size=32)

pretrained = ResNet50(weights='imagenet',
                     backend=keras.backend,
                     layers=keras.layers,
                     models=keras.models,
                     utils=keras.utils,
                     include_top=False,
                     input_shape=(input_height, input_width, input_depth))


model = models.Sequential()
model.add(pretrained)
model.add(layers.Flatten())
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(7, activation='softmax'))


model.compile(optimizer=optimizers.RMSprop(lr=0.00001),
                         loss='categorical_crossentropy',
                        metrics=['categorical_accuracy',
                       km.categorical_precision(label=0),
                       km.categorical_precision(label=1),
                       km.categorical_precision(label=2),
                       km.categorical_precision(label=3),
                       km.categorical_precision(label=4),
                       km.categorical_precision(label=5),
                       km.categorical_precision(label=6),
                       km.categorical_recall(label=0),
                       km.categorical_recall(label=1),
                       km.categorical_recall(label=2),
                       km.categorical_recall(label=3),
                       km.categorical_recall(label=4),
                       km.categorical_recall(label=5),
                       km.categorical_recall(label=6),
                       km.categorical_f1_score(label=0),
                       km.categorical_f1_score(label=1),
                       km.categorical_f1_score(label=2),
                       km.categorical_f1_score(label=3),
                       km.categorical_f1_score(label=4),
                       km.categorical_f1_score(label=5),
                       km.categorical_f1_score(label=6)
                       ])

with tf.Session() as s:
    s.run(tf.global_variables_initializer())
    history = model.fit_generator(train_generator,
                              steps_per_epoch=steps_per_epoch,
                              epochs=3,
                              validation_data=validation_generator,
                              validation_steps=validation_steps,
                              shuffle=True,
                              verbose=1)

    predictions = model.predict(validation_images)

    predicted_classes = np.argmax(predictions, axis=1)

    validation_labels = np.argmax(validation_labels, axis=1)

    c_matrix = confusion_matrix(validation_labels, predicted_classes)
    print(c_matrix)

    report = classification_report(validation_labels, predicted_classes)
    print(report)

"FailedPreconditionError" upon using precision and recall metrics

I wanted to run a simple model with precision and recall reported using keras-metrics, since keras itself had removed them, but I get a pretty drastic error because of them. This is the code I am using. I hope I am not using precision and recall in a wrong way somehow?

%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd

import tensorflow as tf
from tensorflow import keras
import sklearn as sk

import keras_metrics as km

import seaborn as sns
sns.set_style("darkgrid")

## Spiral data generation
def twospirals(n_points, noise=.5):
    """
     Returns the two spirals dataset.
    """
    n = np.sqrt(np.random.rand(n_points,1)) * 780 * (2*np.pi)/360
    d1x = -np.cos(n)*n + np.random.rand(n_points,1) * noise
    d1y = np.sin(n)*n + np.random.rand(n_points,1) * noise
    return (np.vstack((np.hstack((d1x,d1y)),np.hstack((-d1x,-d1y)))), 
            np.hstack((np.zeros(n_points),np.ones(n_points))))

x, y = twospirals(1000)

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(2,)),
    keras.layers.Dense(3, activation='sigmoid'),
    keras.layers.Dense(3, activation='sigmoid'),
    keras.layers.Dense(1, activation=keras.activations.sigmoid)
])

## Choosing my optimizer algorithm and loss function
grad_opt = tf.train.GradientDescentOptimizer(learning_rate=.003)
mse = keras.losses.mean_squared_error

model.compile(optimizer=grad_opt, 
              loss=mse,
              metrics=[km.recall(), km.precision()])

model.fit(x=x, y=y, epochs=300)

Feature request: Multi-class average recall metric

Hi! I have been using multi-class average recall in most of my research projects in the past. I used to calculate them by callbacks, now I can finally add a metric.

I have posted my implementation at this gist. Would be convenient to have a look at my work? If you are interested in it, I will be glad to submit a PR.

The multi-class average recall is a widely used metric in emotion classification and many other research problems. I think it could certainly benefit lots of people.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.