Git Product home page Git Product logo

gdnm's Introduction

Generalized Damped Newton Method for Solving Lasso - GDNM

In [kmpt21], we propose a new algorithm for solving Lasso regession which is the optimization problem of the form

image image

The formulation for general proximal mapping can be found in [kmpt21].

image

The Matlab code for this algorithm can be found in the file GDNM.m or below.

Input

function x=lasso_GDNM(A,b,mu)

For simplicity, we choose \beta=1/2 and \sigma=0.1. The choice of y^0 can be found in the iterating step below.

Find gamma.

    ATA=A'*A;
    gamma=0.5/eigs(ATA,1);

Calculate Proximal mapping

    function [value,zeros_index]=prox_of_function(gamma,y,mu)
        value=[];
        zeros_index=[];
        for i=1:length(y)
            if y(i)>mu*gamma
                value=[value;y(i)-mu*gamma];
            elseif y(i)<-mu*gamma
                value=[value;y(i)+mu*gamma];
            else 
                value=[value;0];
                zeros_index=[zeros_index;i];
            end
        end
    end   

Calculate matrices Q, P, and vector c

    Q=inv(eye(size(ATA))-gamma*ATA);
    P=Q-eye(size(Q));
    c=-gamma*Q*transpose(A)*b;

Calculate value of Lasso function, value of function phi (1) and its gradient (2)

    function value=value_of_function(A,b,mu,x)
        value=0.5*norm(A*x-b)^2+mu*norm(x,1);
    end
    function value_of_phi=phi(A,b,gamma,mu,y)
        [prox_y,zeros_index]=prox_of_function(gamma,y,mu);
        value_of_phi=0.5*transpose(P*y)*y+transpose(c)*y+gamma*mu*norm(prox_y,1)+0.5*norm(y-prox_y)^2;
    end
    function grad=finding_gradient(A,b,gamma,mu,y)
        [prox_y,zeros_index]=prox_of_function(gamma,y,mu);
        grad=Q*y-prox_y+c;
    end

Find direction dk, step size tk

    function dk=finding_dk(A,b,gamma,mu,y)
        grad=finding_gradient(A,b,gamma,mu,y);
        [prox_y,zeros_index]=prox_of_function(gamma,y,mu);
        X=P;
        for i=zeros_index
            X(i,:)=Q(i,:);
        end
        dk=-linsolve(X,grad);
    end
    function tau=finding_tk(A,b,gamma,mu,y)
        tau=1;
        dk=finding_dk(A,b,gamma,mu,y);
        grad=finding_gradient(A,b,gamma,mu,y);
        while phi(A,b,gamma,mu,y+tau*dk)-phi(A,b,gamma,mu,y)-0.1*tau*grad'*dk>0
            tau=tau/2;
        end
    end

Start iterating

   y=zeros(length(c),1);
   iter=0;
   while iter<100
       y=y+finding_tk(A,b,gamma,mu,y)*finding_dk(A,b,gamma,mu,y);
       iter=iter+1
       value=0.5*norm(A*(Q*y+c)-b)^2+mu*norm(Q*y+c,1)
   end

Transform y to x

   x=Q*y+c;
end

The entire code can be found in file GDNM.m

Let us do a simple example from [Example 7.5,kmp20] for illustration. In this example, we choose

image

The implementations for this are as follows

A=[4/7,3,6;12/7,2,-3;6/7,-6,2;0,24,0];
b=[104/49;347/49;-649/49;0];
mu=1/3;
x=lasso_GDNM(A,b,mu)

Reader can find implementations for this example via Octave Online in the link GDNM Example.

References

[kmp20] Khanh, P. D., Mordukhovich, B. S., Phat, V. T.: A Generalized Newton Method for Subgradient Systems, submitted, https://arxiv.org/abs/2009.10551

[kmpt21] Khanh, P. D., Mordukhovich, B. S., Phat, V. T., Tran, D. B.: Generalized Damped Newton Algorithms in Nonsmooth Optimization with Applications to Lasso Problems, submitted, https://arxiv.org/abs/2101.10555

gdnm's People

Contributors

he9180 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.