This repository contains a Keras implementation of the paper "LiSHT: Non-Parametric Linearly Scaled Hyperbolic Tangent Activation Function for Neural Networks" - (link to the arXiv page)

Roy, Kumar S., Manna, S., Dubey, Ram S., and Chaudhuri, Baran B., 2019. LiSHT: Non-Parametric Linearly Scaled 
Hyperbolic Tangent Activation Function for Neural Networks. arXiv preprint arXiv:1901.05894.

The activation function in neural network is one of the important aspects which facilitates the deep training by introducing the non-linearity into the learning process. However, because of zero-hard rectification, some the of existing activations function such as ReLU and Swish miss to utilize the negative input values and may suffer from the dying gradient problem. Thus, it is important to look for a better activation function which is free from such problems. As a remedy, this paper proposes a new non-parametric function, called Linearly Scaled Hyperbolic Tangent (LiSHT) for Neural Networks (NNs). The proposed LiSHT activation function is an attempt to scale the non-linear Hyperbolic Tangent (Tanh) function by a linear function and tackle the dying gradient problem.

Most neural networks work by interleaving linear projections and simple (fixed) activation functions, like the ReLU function:

A LiSHT is instead a parametric activation function defined as non-parametric approximator:

The effectiveness of the proposed LiSHT activation function is evaluated on six benchmark datasets:

• Car Evaluation Dataset
• Iris Flower Dataset
• MNIST Dataset
• CIFAR-10 Dataset
• CIFAR-100 Dataset
• Twitter140 Dataset

The classification performance of MLP for different activations over Cars Evaluation, Iris and MNIST datasets.

The classification performance of ResNet for different activations over MNIST and CIFAR-10/100 datasets.

The classification performance of LSTM for different activations over twitter140 dataset.

If you use this code or a derivative thereof in your research, we would appreciate a citation to the original paper:

@article{roy2019lisht,
        title={LiSHT: Non-Parametric Linearly Scaled Hyperbolic Tangent Activation Function for Neural Networks},
        author={Roy, Swalpa Kumar and Manna, Suvojit and Dubey, Shiv Ram and Chaudhuri, Bidyut B},
        journal={arXiv preprint arXiv:1901.05894},
        year={2019}
    }

The code is released under the MIT License. See the attached LICENSE file.