I think it could be useful to add L2 regularization.

Trying to run python setup.py build_ext fails on OS X 10.8.3 (Python 2.7.2) with Cython 0.17.

» python setup.py build_ext      fonnescj@Cepeda
running build_ext
Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    requires = ['numpy', 'scipy'],
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Library/Python/2.7/site-packages/Cython/Distutils/build_ext.py", line 159, in run
    if self.cython_gdb or [1 for ext in self.extensions
TypeError: 'NoneType' object is not iterable

This is an interesting project! However, the code seems to be doing different things from what you describe in your blog post.

1. Shouldn't the sentence "We will then assign the class j if the prediction wTX lies in the interval [θj−1,θj[" translate into:

def ordinal_logistic_predict(w, theta, X):
    """
    Parameters
    ----------
    w : coefficients obtained by ordinal_logistic
    theta : thresholds
    """
    unique_theta = np.sort(np.unique(theta))
    out = X.dot(w)
    unique_theta[-1] = np.inf # p(y <= max_level) = 1
    tmp = out[:, None].repeat(unique_theta.size, axis=1)
    return np.argmax(tmp < unique_theta, axis=1)

If I make this change, I obtain much better performance, but it seems almost too perfect:

MEAN ABSOLUTE ERROR (ORDINAL LOGISTIC):    2.88859180036
MEAN ABSOLUTE ERROR (LOGISTIC REGRESSION): 3.83957219251
MEAN ABSOLUTE ERROR (RIDGE REGRESSION):    3.5623885918

1. Why is the threshold vector of the same size as the number of levels K? I believe it should have size K-1, but I am not sure of how to modify the gradient to have this...

Hi Fabian,

I'm working on a dense problem (n_samples=1000, n_features=32000)
for which classical formulation of ordinal logistic regression is somewhat
too constrained.

As an alternative I was considering standard multinomial logit
with the constraint that the sigmoid functions should be parallel.

This would be less restricted than ordinal logistic regression since
the "slopes" of the sigmoids would be allowed to be different
while avoiding any crossing of the curves.

The questions are:

1. Do you thing such an optimization target loss would be useful?
   Would it be in scope with minirank? Are you aware of any
   paper related to this approach?

2. Since manual derivation of gradient and hessian is quite heavy and
   error prone, is there any automatic differentiation framework that you
   have tested or you would suggest? Have you done any experiment
   with automatic differentiation?

3. While being definitely out of scope for minirank. I would be nice to have
   a l2 penalized logistic package for dense/sparse problems based on
   pytron solving multinomial logit (with/without parallel constraint) and
   ordinal regression based on pytron ...

Thanks!
Paolo