[MacOS/Linux]	Coverage	Documentation
		TBA

This is a convenience package gathering functionalities to solve a number of generalised linear regression/classification problems which, inherently, correspond to an optimisation problem of the form

L(y, Xθ) + P(θ)

where L is a loss function and P is a penalty function (both of those can be scaled or composed). Additional regression/classification methods which do not directly correspond to this formulation may be added in the future.

The core aims of this package are:

• make these regressions models "easy to call" and callable in a unified way,
• interface with MLJ.jl,
• focus on performance including in "big data" settings exploiting packages such as Optim.jl, IterativeSolvers.jl,
• use a "machine learning" perspective, i.e.: focus essentially on prediction, hyper-parameters should be obtained via a data-driven procedure such as cross-validation.

1. "0" stands for no penalty
2. Analytical means the solution is computed in "one shot" using the \ solver,
3. CG = conjugate gradient
4. (Accelerated) Proximal Gradient Descent
5. Huber, Andrews, Bisquare, Logistic, Fair and Talwar weighing functions available
6. Iteratively re-Weighted Least Squares where each system is solved iteratively via CG
7. In other packages such as Scikit-Learn, a scale factor is estimated along with the parameters, this is a bit ad-hoc and corresponds more to a statistical perspective, further it does not work well with penalties; we recommend using cross-validation to set the parameter of the Huber Loss. (TODO: document)

Unless otherwise specified:

• Newton-like solvers use Hager-Zhang line search (default in Optim.jl)
• ISTA, FISTA solvers use backtracking line search and a shrinkage factor of β=0.8

• The models are built and tested assuming n > p; if this doesn't hold, tricks should be employed to speed up computations; these have not been implemented yet.
• Stochastic solvers that would be appropriate for huge models have not yet been implemented.
• "Meta" functionalities such as One-vs-All or Cross-Validation are left to other packages such as MLJ.

• Quantile reg with ADMM, IWLS, (? IP, Frisch Newton)
• LAD with ADMM

• (⭒) should be added soon

There are a number of other regression models that may be included in this package in the longer term but may not directly correspond to the paradigm Loss+Penalty introduced earlier.

In some cases it will make more sense to just use GLM.jl.

Sklearn's list: https://scikit-learn.org/stable/supervised_learning.html#supervised-learning

While the functionalities in this package overlap with a number of existing packages, the hope is that this package will offer a general entry point for all of them in a way that won't require too much thinking from an end user (similar to how someone would use the tools from sklearn.linear_model). If you're looking for specific functionalities/algorithms, it's probably a good idea to look at one of the packages below:

• SparseRegression.jl
• Lasso.jl
• QuantileRegression.jl
• (unmaintained) Regression.jl
• (unmaintained) LARS.jl
• (unmaintained) FISTA.jl

There's also GLM.jl which is more geared towards statistical analysis for reasonably-sized datasets and does (as far as I'm aware) lack a few key functionalities for ML such as penalised regressions or multinomial regression.

• Minka, Algorithms for Maximum Likelihood Regression, 2003. For a review of numerical methods for the binary Logistic Regression.
• Beck and Teboulle, A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems, 2009. For the ISTA and FISTA algorithms.
• Raman et al, DS-MLR: Exploiting Double Separability for Scaling up DistributedMultinomial Logistic Regression, 2018. For a discussion of multinomial regression.
• Robust regression
  • Mastronardi, Fast Robust Regression Algorithms for Problems with Toeplitz Structure, 2007. For a discussion on algorithms for robust regression.
  • Fox and Weisberg, Robust Regression, 2013. For a discussion on robust regression and the IWLS algorithm.
  • Statsmodels, M Estimators for Robust Linear Modeling. For a list of weight functions beyond Huber's.
  • O'Leary, Robust Regression Computation using Iteratively Reweighted Least Squares, 1990. Discussion of a few common robust regressions and implementation with IWLS.

• Probit Loss --> via StatsFuns // Φ(x) (normcdf); ϕ(x) (normpdf); -xϕ(x)
• Newton, LBFGS take linesearches, seems NewtonCG doesn't
• several ways of doing backtracking (e.g. https://archive.siam.org/books/mo25/mo25_ch10.pdf); for FISTA many though see http://www.seas.ucla.edu/~vandenbe/236C/lectures/fista.pdf; probably best to have "decent safe defaults"; also this for FISTA http://150.162.46.34:8080/icassp2017/pdfs/0004521.pdf ; https://github.com/tiepvupsu/FISTA#in-case-lf-is-hard-to-find ; https://hal.archives-ouvertes.fr/hal-01596103/document; not so great https://github.com/klkeys/FISTA.jl/blob/master/src/lasso.jl ;
• https://www.ljll.math.upmc.fr/~plc/prox.pdf
• proximal QN http://www.stat.cmu.edu/~ryantibs/convexopt-S15/lectures/24-prox-newton.pdf; https://www.cs.utexas.edu/~inderjit/public_papers/Prox-QN_nips2014.pdf; https://github.com/yuekai/PNOPT; https://arxiv.org/pdf/1206.1623.pdf
• group lasso http://myweb.uiowa.edu/pbreheny/7600/s16/notes/4-27.pdf

• LBFGSB https://www.researchgate.net/profile/Jose_Morales23/publication/220493046_Remark_on_Algorithm_778_L-BFGS-B_Fortran_Subroutines_for_Large-Scale_Bound_Constrained_Optimization/links/546ec9920cf2b5fc17607f33/Remark-on-Algorithm-778-L-BFGS-B-Fortran-Subroutines-for-Large-Scale-Bound-Constrained-Optimization.pdf