Initial stepsize not exposed in LBFGS constructor [question/bug?] about jaxopt HOT 6 CLOSED

Hi @erdmann!

Indeed, LBFGS currently doesn't support a stepsize option at the moment.

ProximalGradient and GradientDescent do support this option but the behavior is slightly different from what you want:

• If stepsize is a float and <= 0, use linesearch
• If stepsize is a a float and >=, use this value as constant step size
• If stepsize is a Callable of the form stepsize(iter_num), then use the output of this function

I think for consistency we should implement the same behavior in LBFGS.

Regarding the ability to initialize the linesearch with a value of the user's choosing, currently this is not possible. We would need to think it it's really useful (normally, without numerical instability issues, the line search is supposed to auto-tune the stepsize) and how to integrate it in the API. Currently, we only support backtracking line search but we plan to support more line search strategies and to let the user pass a line search object. Maybe the option you propose could be added in this context.

In the meantime, I would try 64-bit precision to see if the numerical issues disappear.

from jaxopt.

I'm finding myself in a similar position, even with x64 enabled. The LBFGS implementation initial stepsize is 1.0 * increase_factor which results in a large number of iterations compared to the SciPy wrapper implementation (see here for example). Without x64 enabled it fails to proceed beyond its initialization due to nan/ model misspecification.

It would be fantastic to have a more flexible initialization at some point in the future. Thank you and the team for developing such a useful library.

from jaxopt.

What do you think of replacing init_stepsize = state.stepsize * self.increase_factor with init_stepsize = min(state.stepsize * self.increase_factor, 1.0) as a safegard? (see this line)

from jaxopt.

Thanks @mblondel . It might need to wait until configurable LineSearch objects can be initialized and passed in to create a more flexible starting point.

I found a simpler example where LBFGS chokes compared to other methods. See here

from jaxopt.

@mblondel seems like latest release fixed the issues reported in my above example. Thanks so much!

from jaxopt.

Glad to hear that and sorry it took some time!

Thanks #236 and #239, it's now possible to set the initial step size using the max_stepsize option.

I am closing this issue but feel free to re-open if necessary.

from jaxopt.