Derivatives of max and min about diffrules.jl HOT 2 OPEN

I think the more useful result is to pick a something like a sub-gradient. The use case to imagine is some optimisation problem, where equality might be the best or worst point. We'd like gradient descent to be able to end up there, or not to get stuck there, both of which want a finite gradient.

You could argue that these should pick a symmetric convention, though:

julia> ForwardDiff.gradient(x -> max(x...), [1,1])
2-element Vector{Int64}:
 0
 1

julia> ForwardDiff.derivative(x -> clamp(x, 1, 2), 1)
1

julia> ForwardDiff.derivative(x -> clamp(x, 1, 2), 2)
1

from diffrules.jl.

Yes, the problem I had was encountered when optimizing a function with max(x, y) inside it. I was checking the gradients using ForwardDiff and realized that problem of no symmetry. Worse, though, is FiniteDiff:

julia> FiniteDiff.finite_difference_gradient(x -> max(x...), [1.0, 1.0])
2-element Vector{Float64}:
 0.4999999999934427
 0.4999999999934427

from diffrules.jl.