CORN layer inherits fromDense, so it all usual **kwargs can be passed.

Coral layer inherits from Layer, so kernel_regualizer/bias_regularizer (and other kwargs that would make sense for Coral) dont work as expected (passing them fails init since Layer does not recognized those kwarggs)

Hello,

i hope this is the right place to ask this.

I have a dataset of ordinal data where the classes significantly decrease in frequency, similar to older ages being less common in a large population. How would you adress this in a coral-ordinal model optimally?

Is using compute_class_weight from sklearn.utils.class_weight on the one hot encoded labels a valid approach? How would you pass the results to the class_weight parameter of the keras model.fit() function?

Thank you!

When compiling the model with:

model.compile(loss = coral.OrdinalCrossEntropy(num_classes = num_labels),
                metrics = [coral.MeanAbsoluteErrorLabels()],
                optimizer=opt)

it causes the following exception:

TypeError: MeanAbsoluteErrorLabels() missing 2 required positional arguments: 'y_true' and 'y_pred'

Is there a workaround / easy fix for this?

There is a bug in the MNIST example in the Colab notebook:
shuffle() was called on test_dataset, but later you compared your prediction on the test dataset against the pre-shuffled mnist_labels_test. This is why the accuracy was only 10% (i.e. random), and the MAE computed doesn't match the one reported by evaluate().

A simple fix would be to remove shuffle() from test_dataset. After fix, the accuracy becomes 57% for labels and 67% for labels2. MAE(version 2) was close to what evaluate(test_dataset) was reporting (0.541 vs. 0.538). So there might still be some subtle differences between the two ways of computing them.

Thank you for making the code available!

Epoch 1/5
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-d352eb85a539> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', '\n# This takes about 5 minutes on CPU.\nhistory = model.fit(dataset, epochs = 5, validation_data = val_dataset,\n                    callbacks = [tf.keras.callbacks.EarlyStopping(patience = 3, restore_best_weights = True)])')

13 frames
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2115             magic_arg_s = self.var_expand(line, stack_depth)
   2116             with self.builtin_trap:
-> 2117                 result = fn(magic_arg_s, cell)
   2118             return result
   2119 

<decorator-gen-60> in time(self, line, cell, local_ns)

/usr/local/lib/python3.6/dist-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/usr/local/lib/python3.6/dist-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
   1191         else:
   1192             st = clock2()
-> 1193             exec(code, glob, local_ns)
   1194             end = clock2()
   1195             out = None

<timed exec> in <module>()

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
    106   def _method_wrapper(self, *args, **kwargs):
    107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
--> 108       return method(self, *args, **kwargs)
    109 
    110     # Running inside `run_distribute_coordinator` already.

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1096                 batch_size=batch_size):
   1097               callbacks.on_train_batch_begin(step)
-> 1098               tmp_logs = train_function(iterator)
   1099               if data_handler.should_sync:
   1100                 context.async_wait()

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781 
    782       new_tracing_count = self._get_tracing_count()

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    821       # This is the first call of __call__, so we have to initialize.
    822       initializers = []
--> 823       self._initialize(args, kwds, add_initializers_to=initializers)
    824     finally:
    825       # At this point we know that the initialization is complete (or less

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    695     self._concrete_stateful_fn = (
    696         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 697             *args, **kwds))
    698 
    699     def invalid_creator_scope(*unused_args, **unused_kwds):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2853       args, kwargs = None, None
   2854     with self._lock:
-> 2855       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2856     return graph_function
   2857 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    598         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    599         # the function a weak reference to itself to avoid a reference cycle.
--> 600         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    601     weak_wrapped_fn = weakref.ref(wrapped_fn)
    602 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

AttributeError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/coral_ordinal/loss.py:58 call  *
        y_pred = ops.convert_to_tensor_v2(y_pred)

    AttributeError: module 'tensorflow.python.ops' has no attribute 'convert_to_tensor_v2'

I tried running your colab notebook with the mnist data and the amazon data but came up with this error

Hi,
I'm a beginner on deep learning and have been trying to turn a multi-class classification task into the ordinal regression. Here's the code regarding the CNN architecture:

`class TrainingModel:
def init(self, hyperparameters, model_type, target_height, target_width, class_names):
""" Creates and prepares a model for training.

    Args:
        hyperparameters (dict): The configuration's hyperparameters.
        model_type (str): Type of model to create.
        target_height (int): Height of input.
        target_width (int): Width of input.
        class_names (list of str): A list of classes.
        
    Returns:
        model (keras Model): The prepared keras model.
        model_type (str): The name of the model type.
    """

    # Catch if the model is not in the model list
    if model_type not in model_list:
        print(colored(f"Warning: Model '{model_type}' not found in the list of possible models: {list(model_list.keys())}"))
        
        # TODO: Create a custom model

    self.model_type = model_type
        
    # Get the model base
    base_model = model_list[model_type](
        include_top=False,
        weights=None,
        input_tensor=None,
        input_shape=(target_height, target_width, hyperparameters['channels']),
        pooling=None
    )

    # Return the prepared model
    avg = keras.layers.GlobalAveragePooling2D()(base_model.output)
    #out = coral.CoralOrdinal(len(class_names))(avg)
    out = keras.layers.Dense(len(class_names), activation="softmax")(avg)
    out = coral.CoralOrdinal(len(class_names))(out)
    self.model = keras.models.Model(inputs=base_model.input, outputs=out)
    
    # Create optimizer and add to model
    optimizer = keras.optimizers.SGD(
        lr=hyperparameters['learning_rate'], 
        momentum=hyperparameters['momentum'], 
        nesterov=True, 
        decay=hyperparameters['decay']
    )
    self.model.compile(
        #loss="sparse_categorical_crossentropy", 
        loss = coral.OrdinalCrossEntropy(),
        optimizer=optimizer,
        #metrics=["accuracy"]
        metrics = ["accuracy",coral.MeanAbsoluteErrorLabels()] 
    )

`

Am I doing this correctly by adding an ordinal layer, changing the loss function and the metrics? Also, does the accuracy actually make sense since I'm getting extremely low value on accuracy (around 0.05, it's around 0.95 when it's a classification). Any help would be greatly appreciated, thanks!!!

Edit: my classes are in the form of ["class1": 0, "class2": 1, "class3": 2, "class4": 3, "class5": 4]

Hi @ck37 ,

Thanks for sharing this repo with Corn loss implementation. I see here that you have implemented both the corn layer as well as the CORN loss.

In my experiments, I already have a neural network trying to do ordinal regression (or classification) with N-1 output neurons (meaning i have N classes) and sigmoid activation on all the output neurons and the loss function is the sum of the individual Binary cross entropy loss from all the output neurons. Hence, I already have the ground truth outputs encoded in ordinal format.
I am using tensorflow 2.14.
Question:

1. Can I use the "loss = coral.OrdinalCrossEntropy()" in the place where i calculate the loss?
2. Does the "loss = coral.OrdinalCrossEntropy()" expect the ground truth outputs to be in integer labelled format (0 to N-1)?

Looking forward to your opinion.

Hi there,
for a pet project of nn-based classifier of an ordinal variable, I tried to use coral-ordinal, and it kept predicted only 0 instead of other values in the range (I didn't experience this problem when using standard multiclass-related parameters).
In an attempt to isolate the problem, I wrote the following simple code just to test whether I'm able to get coral-ordinal to work. It creates a data set of 1000 items, each with three randomized independent variables 0<x1,x2,x3<1, and a randomized dependent integer variable 0 <= y <= 4. For whatever reason, the predictions are always 0. Any idea why?
Thanks,
Roy.

import tensorflow as tf
from tensorflow import keras
import numpy as np
from sklearn.model_selection import train_test_split
import coral_ordinal as coral
import random
from sklearn import metrics
import pandas
from keras import models,layers,optimizers

df = pandas.DataFrame()
df.insert(len(df.columns),"x1", np.random.rand(1000))
df.insert(len(df.columns),"x2", np.random.rand(1000))
df.insert(len(df.columns),"x3", np.random.rand(1000))
y = np.random.randint(5,size=1000)

X_train, X_test, y_train, y_test = train_test_split(df,y,test_size=0.2,random_state=41)

nn = keras.models.Sequential()
nn.add(layers.Dense(7, input_shape=(3,), activation="relu"))
nn.add(coral.CoralOrdinal(num_classes=5))
print(nn.summary())
loss_fn = coral.OrdinalCrossEntropy(num_classes=5)
optim_fn = optimizers.legacy.Adam()
nn.compile(optimizer=optim_fn,loss=loss_fn,metrics=[coral.MeanAbsoluteErrorLabels()])
history = nn.fit(X_train,y_train,epochs=100,validation_split=0.2)
test_pred = nn.predict(X_test)
_ ,test_acc, *dummy = nn.evaluate(X_test,y_test)
print(test_acc)
y_pred = np.array([np.argmax(x) for x in test_pred])

print("y_test y_pred")
for i in range(len(y_test)):
print(y_test[i],y_pred[i])

How to use coral_ordinal (CoralOrdinal, OrdinalCrossEntropy, MeanAbsoluteErrorLabels) with the functional API rather then sequential model?

updated loss function to adapt with the tf2.2 version in label_to_levels function

label_vec = tf.repeat(1, tf.cast(tf.argmax(label), tf.int32)) 
    
    # This line requires that label values begin at 0. If they start at a higher
    # value it will yield an error.
num_zeros = self.num_classes - 1 - tf.cast(tf.argmax(label), tf.int32)
    
zero_vec = tf.zeros(shape = (num_zeros), dtype = tf.int32)
    
levels = tf.concat([label_vec, zero_vec], axis = 0)

return tf.cast(levels, tf.float32)

Hello.

I found out that the ordinal softmax function yields an output of list which contains NUM_CLASSES - 1. In other words, it's a list. I have been wondering how to have the layer to give an output of the predicted class in an integer. I found some activation functions in the activations module: 1) coral_cumprobs and 2) cumprobs_to_label.

Correct me if I'm wrong. If my goal is to have a single integer of predicted class in the output. I could use both functions mentioned above in order. Hence, I tried to created a custom activation function, something like this

def coral_labels(x: tf.Tensor):
  threshold = 0.5

  result = tf.math.sigmoid([x])
  predict_levels = tf.cast(result > threshold, dtype=tf.int32)
  predicted_labels = tf.reduce_sum(predict_levels, axis=1)
  return predicted_labels

However, the model summary still tells that the output layer is a list of (1, NUM_CLASSES - 1)

Could you help me on this?

The coral-pytorch authors recently published an update on CORAL ordinal regression, called CORN ordinal regression. tl;dr: it's a stick breaking process implementation of ordinal regression, which means no weight constraints are necessary to keep rank consistency.

See here for template
https://github.com/Raschka-research-group/coral-pytorch/blob/main/coral_pytorch/losses.py#L86

TF keras implementation can be completely analogous using tf.math.log_sigmoid().

Reading the readme in full helps!

Note that the minimum value of the ordinal variable needs to be 0. If your labeled data ranges from 1 to 5, you will need to subtract 1 so that it is scaled to be 0 to 4.

For future people if you see this error: you need to subtract 1 from your labels (or increase your num_classes

InvalidArgumentError:  Dimension -1 must be >= 0
	 [[{{node ordinal_crossent/map/while/body/_1/zeros}}]] [Op:__inference_train_function_657979]

The serialization of CornLayer is based on the the config() values returned by get_config(). the issue is that 'num_classes' from CornLayer init is passed to Dense(units=num_classes-1, **kwargs), which means if **kwargs contains "units" the init will fail. A (human) caller would not do pass units in the first place, however, because get_config() uses the super().get_config() call first, and then just updates it with "num_classes" -- the returned dict by get_config() has a "units" argument --> this leads to failure when loading a serialized model with the CornLayer using programmatic way in https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model

proposed solution: drop "units" from return dict of get_config() (alternatively, remove "units" from **kwargs at init time if present -- and leave get_config() the same.

Hi,

Is it possible use this for segmentation?

Thanks

Is it possible to use the class_weight parameter of the fit() function while using coral.OrdinalCrossEntropy()?

OrdinalCrossEntropy inherits from baseline Loss class with reduction argument. However, the reduction argument is then ignored in the actual loss computation of the helper function: ordinal_loss() (it uses reduce_mean() even if reduction != tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE or AUTO)

I run the following (very basic) code:

import coral_ordinal as coral
model = keras.Sequential()
model.add(keras.layers.Dense(32, activation = "relu"))
model.add(coral.CoralOrdinal(num_classes = 5)) # Ordinal variable has 5 labels, 0 through 4.
model.compile(loss = coral.OrdinalCrossEntropy(), metrics = [coral.MeanAbsoluteErrorLabels])

And get the following error:
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

This error originates in coral.OrdinalCrossEntropy(), see traceback:
` in
3 model.add(keras.layers.Dense(32, activation = "relu"))
4 model.add(coral.CoralOrdinal(num_classes = 5)) # Ordinal variable has 5 labels, 0 through 4.
----> 5 model.compile(loss = coral.OrdinalCrossEntropy(), metrics = [coral.MeanAbsoluteErrorLabels])

~\Anaconda3\lib\site-packages\coral_ordinal\loss.py in init(self, num_classes, importance, from_type, name, **kwargs)
28 #self.importance_weights = importance
29 if importance is None:
---> 30 self.importance_weights = tf.ones(self.num_classes - 1, dtype = tf.float32)
31 else:
32 self.importance_weights = tf.cast(importance, dtype = tf.float32)`

I am using Tensorflow 2.4.1 and numpy 1.19.5 and python 3.8.5 so based on the requirements it should work. What could cause this problem?

Thanks for a great package.

One slight inconvenience is that OrdinalCrossEntropy() has a required (!) argument to specify num_classes, which makes the loss function non-standard compared to other standard keras losses (e.g., crosscategory_entropy does not require to specify num_classes either). This however is not required to pass at __init__(), but can be inferred from y_pred.shape[1] . Any particular reason to not implement it this way?

I can add a PR to add this if needed (also can be backwards compatible by leaving num_classes: Optional[int] = None).