Divergences.jl is a Julia package that makes it easy to evaluate divergence measures between two vectors. The package allows calculating the gradient and the diagonal of the Hessian of several divergences. These divergences are used to good effect by the MomentBasedEstimators package.

The package defines an abstract Divergence type with the following suptypes:

• Kullback-Leibler divergence KullbackLeibler
• Chi-square distance ChiSquared
• Reverse Kullback-Leibler divergence ReverseKullbackLeibler
• Cressie-Read divergences CressieRead

These divergences differ from the equivalent ones defined in the Distances package because they are normalized. Also, the package provides methods for calculating their gradient and the (diagonal elements of the) Hessian matrix.

The constructors for the types above are straightforward

KullbackLeibler()
ChiSqaured()
ReverseKullbackLeibler()

The CressieRead type define a family of divergences indexed by a parameter alpha. The constructor for CressieRead is

CR(::Real)

The Hellinger divergence is obtained by CR(-1/2). For a certain value of alpha, CressieRead correspond to a divergence that has a specific type defined. For instance CR(1) is equivalent to ChiSquared although the underlying code for evaluation and calculation of the gradient and Hessian are different.

Three versions of each divergence in the above list are implemented currently. A vanilla version, a modified version, and a fully modified version. These modifications extend the domain of the divergence.

The modified version takes an additional argument that specifies the point at which the divergence is modified by a convex extension.

ModifiedKullbackLeibler(theta::Real)
ModifiedReverseKullbackLeibler(theta::Real)
ModifiedCressieRead(alpha::Real, theta::Real)

Similarly, the fully modified version takes two additional arguments that specify the points at which the divergence is modified by a convex extensions.

FullyModifiedKullbackLeibler(phi::Real, theta::Real)
FullyModifiedReverseKullbackLeibler(phi::Real, theta::Real)
FullyModifiedCressieRead(alpha::Real, phi::Real, theta::Real)

Each divergence corresponds to a divergence type. You can always compute a certain divergence between two vectors using the following syntax

d = evaluate(div, x, y)

Here, div is an instance of a divergence type. For example, the type for Kullback Leibler divergence is KullbackLeibler (more divergence types are described in some details in what follows), then the Kullback Leibler divergence between x and y can be computed

d = evaluate(KullbackLeibler(), x, y)

We can also calculate the diverge between the vector x and the unit vector

r = evaluate(KullbackLeibler(), x)

The Divergence type is a subtype of PreMetric defined in the Distances package. As such, the divergences can be evaluated row-wise and column-wise for X::Matrix and Y::Matrix.

rowise(div, X, Y)

colwise(div, X, Y)

To calculate the gradient of div::Divergence with respect to x::AbstractArray{Float64, 1} the gradient method can be used

g = gradient(div, x, y)

or through its in-place version

gradient!(Array(Float64, size(x)), div, x, y)

The hessian method calculate the Hessian of the divergence with respect to x

h = hessian(div, x, y)

Its in-place variant is also defined

hessian!(Array(Float64, size(x)), div, x, y)

Notice that the Hessian of a divergence is sparse, where the diagonal entries are the only ones different from zero. For this reason, hessian(div, x, y) return an Array{Float64,1} with the diagonal entries of the hessian.

[To be added]