Hi!
This is not an issue per se and can be closed.

First of all, thank you for advancing progress in deep learning.

I'm just a random guy that want to implement an AGI (lol) and like many Nlp engeeners, I need HIGHLY accurate neural networks for fundamental NLP tasks (e.g POS tag, NER, dep parsing, Coref resolution, WSD, etc)
They are all not very accurate (often sub 95% F1 score) and their errors add up.

Such limitations make Nlp not yet suitable for many things.
This is why improving the state of the art (which can be observed on paperswithcode.com) is a crucial priority from academicians.

Effectively, many researchers have smart ideas to improve the state of the art and often slightly improve it by:
Having a "standard neural network" for the task and mix with it their new fancy idea.

I talk from knowledge, I've read most papers from state of the art leaderboards from most fundamental NLP tasks.
Almost always they have this common baseline + one idea, theirs.
The common baseline sometimes slowly evolve (e.g now it's often a pre trained model (say BERT) + fine tuning + their idea.

Sorry to say, but "this" is to me retarded
Where "this" mean the fact that by far, most researchers work in isolation, not integrating others ideas (or with such a slow inertia).
I would have wished that state of the art in one Nlp task would be a combination of e.g 50 innovative and complementary ideas from researchers.
You are researchers, do you have an idea why that is the case? If someone actually tried to merge all good complementary and compatible ideas, would they have the best, unmatchable state of the art?
Why facebookresearch, Microsoft, Google don't try the low hanging fruit in addition to producing X new shiny ideas per month, actually try to merge them in a coherent, synergetic manner??
I would like you to tell me what you think of this major issue that slow AI progress.

As an example of such inertia let's talk about Swish, Mish or RAdam :
Those things are incredibly easy to try and see "hey does it give to my neural network free accuracy gains?"
Yet not any paper on state of the art leaderboards has tried Swish, Mish or RAdam despite being soo simple to try (you don't need to change the neural network)
Not even pre trained models where so many papers depend on them (I opened issues for each of them).

Thank you for reading.

Dear authors:
Thanks for your contribution in activation functions.
Now I have a question about a statistical index "mean loss" in Table.1. Could you please tell me what the meaning is?
Thanks again. And looking forward to your reply.

ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network
This paper addresses representational bottleneck in a network and proposes new design principles.
The authors apply this method on MobileNetV1 and V2 and gain performance that scales much better than EfficientNets

One of the tools they use is Swish. In this instance, they actually provide theoretical and empirical justification for this. (This involves the rank of inputs and outputs, and is explained within the paper)

The authors do not discuss or use Mish however, which is shown to be better in extensive use cases.

Since the ReXNet is quite simple, it would seem to be relatively straight forward to conduct studies investigating the replacement of swish with mish in the relevant settings.

I hope this is of interest to you.

Hi, do you have any tips on how to implement Mish as an inplace operation in PyTorch? Could this work? Is there a way of making the F.softplus() operation also inplace?

class Mish_(nn.Module):
    def __init__(self):
        super().__init__()
        
    def forward(self, x):
        softplus_res = F.softplus(x)
        torch.tanh_(softplus_res)
        return x * softplus_res

Hi. I trained a model with the TF-Keras version of MIsh and saved it. However, I cannot figure out how to load the Mish custom layer object when attempting to load the model.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-99cc25741e5d> in <module>
    208     learning_rate=1e-4, base_lr=1e-4, max_lr=2e-4,
    209     img_size=300, train_img_folder="data/train_supercropboy/", valid_img_folder="data/train_supercropboy/",
--> 210     df_valid=None, pretrain=False, freeze_weights=False, freeze_weights_epochs=5)

<ipython-input-36-99cc25741e5d> in regression_cv(df, nfolds, epochs, batch_size, learning_rate, base_lr, max_lr, train_img_folder, valid_img_folder, img_size, df_valid, start_fold, pretrain, freeze_weights, freeze_weights_epochs)
    137         # Create model
    138 
--> 139         model = create_model(img_size=img_size, pretrain=pretrain, freeze_weights=freeze_weights)
    140 
    141         if freeze_weights:

<ipython-input-36-99cc25741e5d> in create_model(img_size, pretrain, freeze_weights)
     22         model = load_model("effnet_pretrain.h5",
     23                           custom_objects = {"root_mse": root_mse,
---> 24                                             "Mish": Mish})
     25 
     26     return(model)

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\saving\save.py in load_model(filepath, custom_objects, compile)
    144       h5py is not None and (
    145           isinstance(filepath, h5py.File) or h5py.is_hdf5(filepath))):
--> 146     return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
    147 
    148   if isinstance(filepath, six.string_types):

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py in load_model_from_hdf5(filepath, custom_objects, compile)
    210     model_config = json.loads(model_config.decode('utf-8'))
    211     model = model_config_lib.model_from_config(model_config,
--> 212                                                custom_objects=custom_objects)
    213 
    214     # set weights

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\saving\model_config.py in model_from_config(config, custom_objects)
     53                     '`Sequential.from_config(config)`?')
     54   from tensorflow.python.keras.layers import deserialize  # pylint: disable=g-import-not-at-top
---> 55   return deserialize(config, custom_objects=custom_objects)
     56 
     57 

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\layers\serialization.py in deserialize(config, custom_objects)
     87       module_objects=globs,
     88       custom_objects=custom_objects,
---> 89       printable_module_name='layer')

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
    190             custom_objects=dict(
    191                 list(_GLOBAL_CUSTOM_OBJECTS.items()) +
--> 192                 list(custom_objects.items())))
    193       with CustomObjectScope(custom_objects):
    194         return cls.from_config(cls_config)

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\engine\network.py in from_config(cls, config, custom_objects)
   1119     # First, we create all layers and enqueue nodes to be processed
   1120     for layer_data in config['layers']:
-> 1121       process_layer(layer_data)
   1122     # Then we process nodes in order of layer depth.
   1123     # Nodes that cannot yet be processed (if the inbound node

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\engine\network.py in process_layer(layer_data)
   1103       from tensorflow.python.keras.layers import deserialize as deserialize_layer  # pylint: disable=g-import-not-at-top
   1104 
-> 1105       layer = deserialize_layer(layer_data, custom_objects=custom_objects)
   1106       created_layers[layer_name] = layer
   1107 

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\layers\serialization.py in deserialize(config, custom_objects)
     87       module_objects=globs,
     88       custom_objects=custom_objects,
---> 89       printable_module_name='layer')

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
    192                 list(custom_objects.items())))
    193       with CustomObjectScope(custom_objects):
--> 194         return cls.from_config(cls_config)
    195     else:
    196       # Then `cls` may be a function returning a class.

~\Anaconda3\envs\r-tensorflow\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in from_config(cls, config)
    444         A layer instance.
    445     """
--> 446     return cls(**config)
    447 
    448   def compute_output_shape(self, input_shape):

TypeError: __init__() got an unexpected keyword argument 'name'

Any advice on how to successfully load a saved model?

Flatten-T Swish is relu(x) * sigmoid(x) - 0.2
That activation showed better results than vanilla Swish in my experiments.

Page link for testing with Jupyter notebook leads to "404 page not found" error.

(As a side note)
There are papers that noted information regarding the larger variety of CNNs for ImageNet testing.

• https://github.com/BertMoons/Comparing-CNN-Architectures
  • https://www.researchgate.net/publication/328490160_Embedded_Deep_Neural_Networks_Algorithms_Architectures_and_Circuits_for_Always-on_Neural_Network_Processing
• https://github.com/CeLuigi/models-comparison.pytorch
  • https://arxiv.org/pdf/1810.00736.pdf
• https://github.com/yeephycho/nasnet-tensorflow
  • https://arxiv.org/pdf/1707.07012.pdf

I read your research paper on Mish, and from the graphs I saw, it is clearly the best activation function.

You had two different implementations of Mish with widely different computational overheads. There were standard Mish and Mish-CUDA, where Mish-CUDA has almost zero computational cost. I'm optimizing DL4S for Metal, and looking to optimize Mish so that it can be the top-recommended function. However, I need to know how you implemented Mish-CUDA.

Could you please explain how Mish-CUDA was faster than the standard Mish, and how I might implement it in a generic GPGPU context? I have a lot of experience with assembly-level optimization and I know the differences between Apple GPUs/Metal and Nvidia GPUs/CUDA well, if that helps with your explanation.

Hello! Great work on this activation function! I've been using it in some of my projects with great success.

I want to let you know I found what the gain should be set at during kaiming weight initialization for Mish.

I found this experimentally using this code:

import torch
import torch.nn.functional as F
import pandas as pd
import numpy as np

device = 'cpu'

def mish(x):
    return x * (torch.tanh(F.softplus(x)))

aa = []
bb = []
for n in range(100):
    with torch.no_grad():
        a = torch.randn(5000, 5000, device=device)
        b = a
        x = 0.0 + 0.00001 * n
        for i in range(10):
            l = torch.nn.Linear(5000, 5000, bias=False).to(device)
            torch.nn.init.kaiming_uniform_(l.weight, a=x)
            b = mish(l(b))
        aa.append(b.std().item())
        bb.append(x)
        print(x)
        print (f"in: {a.std().item():.8f}, out: {b.std().item():.8f}")
pd.DataFrame(data=aa, index=bb).plot(figsize=(20,8))

which was talked about here. The "a" hyperparameter for init.kaiming_uniform_ is not actually the gain but the negative slope of a leaky relu, so really I experimentally found the equivalent negative slope of mish for kaiming_uniform_ init. The actual gain is found internally by math.sqrt(2.0 / (1 + a ** 2)).

This is an example of the code output. I found through repeated experiments that 0.0003 results in the most consistently efficient throughput through the network, so it is almost the zero slop of relu but not quite. a=0 did produce okay results, as did say 0.001, but the best averaged over many runs is 0.0003. This is important because for deep networks the pytorch default value of sqrt(5) for initializing conv layers is not a good default value if using mish.

I now use something like

for m in self.modules():
    if isinstance(m, (nn.Conv1d, nn.Linear)):
        torch.nn.init.kaiming_uniform_(m.weight, a=0.0003)

Just wondering if all Activation Functions have been addressed in the ReadME.

• https://www.semanticscholar.org/paper/A-comparative-performance-analysis-of-different-in-Farzad-Mashayekhi/bcfdfe54796c501a90c3b353661a19e9c161d2c8/figure/0
• https://www.semanticscholar.org/paper/Activation-Functions-for-Generalized-Learning-A-Villmann-Ravichandran/04d54996bcbe44b3547da889d7eab8aab3660990/figure/0
• https://www.semanticscholar.org/paper/Comparison-of-new-activation-functions-in-neural-Gomes-Ludermir/9b37079041bdaca4248ab4f62f1a63013a50f067/figure/1
• https://www.semanticscholar.org/paper/Searching-for-Activation-Functions-Ramachandran-Zoph/c8c4ab59ac29973a00df4e5c8df3773a3c59995a/figure/2

Gaussian Error Linear Units (GELUs)

First, congratulation on your Mish paper accepted.

I've been thinking about activation functions, and how they can be improved, what properties are most important for a long time, and also inspired by your paper (I noticed the days old update), and e.g. recent SharkFin (my own unpublished idea has some similarities).

I was just thinking if I could ask you some questions, I'm not sure if this is the right place.

It seems like almost any function can do (all except polynominal proven to work for shallow wide networks, and that restriction eliminated by deep-narrow networks):

Universal Approximation with Deep Narrow Networks
https://arxiv.org/pdf/1905.08539.pdf

we show that the class of neural networks of arbitrary depth, width n+m+2, and activation function ρ, is dense [..] This covers every activation function possible to use in practice, and also includes polynomial activation functions [..]

We refer to these as enhanced neurons. [..]

4.2. Square Model
Lemma 4.3 One layer of two enhanced neurons, with square activation function, may exactly represent the multiplication function (x,y)→xy on R^2 [..]

Remark 4.7 Lemma 4.5 is key to the proof of Proposition 4.6. It was fortunate that the reciprocal function may be approximated by a network of width two - note that even if Proposition 4.6 were already known, it would have required a network of width three. It remains unclear whether an arbitrary-depth network of width two, with square activation function, is dense in C(K). [..]

Remark 4.8 Note that allowing a single extra neuron in each layer would remove the need for the trick with the reciprocal, as it would allow [..] Doing so would dramatically reduce the depth of the network. We are thus paying a heavy price in depth in order to reduce the width by a single neuron.

[I've yet to read much further, but this seems very important.]

So my reading is all but identity function can work as activation function (when more than one hidden layer), and a network no wider than 4 (or 5 better, optimal?) can approximate all four elementary arithmetic operation. Could also approximate e.g. sine and exponential with such narrow network (through Fourier-theorem), I think.

Have you looked at Capsule networks, and deep variant?
https://arxiv.org/pdf/1904.09546.pdf

May main worry is that by thinking about better activation functions (of if there can be one best), I'm wasting my time, with them and/or (traditional) backpropagation going away, with the thousand-brain theory and more. Capsule networks seem similar, with a voting mechanism. It at least has ReLU in the first layer (I didn't look at more in detail).

Have you looked at BERT and variants? I assume they could use your functions, or do you know if there are exceptions, making GELU better for them? I'm thinking it's maybe just ignorance (or authors extending, want to change one thing at a time):

https://arxiv.org/pdf/1909.11942.pdf

The backbone of the ALBERT architecture is similar to BERT in that it uses a transformer encoder (Vaswani et al., 2017) with GELU nonlinearities

The Reversible Residual Network: Backpropagation Without Storing Activations
https://arxiv.org/pdf/1707.04585.pdf

I know that it is minor, but it would ease things out to rename the folder TF-Keras.
Then we can just git clone repo & import.

Otherwise python doesn't like dashes in module/package names.

For starters I would like to mention that https://github.com/kozistr/Awesome-GANs and https://github.com/nightrome/really-awesome-gan exists for people to check references.

slim.conv2d it is relu ,how to change it to mish?

Hi, thanks for this interesting new activation function. I've tested it with YOLOv3-SPP on a V100 from https://github.com/ultralytics/yolov3 and have mixed feedback. The performance improves slightly, but the training time is much slower and the GPU memory requirements are much higher vs LeakyReLU(0.1). Any suggestions on how to improve speed/memory in PyTorch? Thanks!

From AlexeyAB/darknet#3114 (comment):

class Swish(nn.Module):
    def __init__(self):
        super(Swish, self).__init__()

    def forward(self, x):
        return x.mul_(torch.sigmoid(x))


class Mish(nn.Module):  # https://github.com/digantamisra98/Mish
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x.mul_(F.softplus(x).tanh())

My implementation is
inputs = inputs * tf.math.tanh(tf.math.softplus(inputs))
Tensorflow 1.14
When I use this activation function for my own model, the training time is doubled compared to swish

I've tested mish vs elu on a simple feedforward network with L2 and dropout and got this result:

You can check it with Colab Notebook, press ctrl+f9.

"Repsoitory" is written instead of "Repository," didn't know if anyone had caught that or if it was intentional for whatever reason that I'm not aware of lol

When using in an RNN, I get {NameError}name 'uses_learning_phase' is not defined when applying a Dense layer on the output of an LSTM.

Tensorflow 1.14

Hello, thanks for the great work. Using the exponential identity for tanh, you can remove two of the transcendental operations (exp, log) and get what, hopefully, should be a faster implementation.

Since

you can express Mish as:

or equivalently (to avoid overflow when x is large)

NB: With a little tweak, there is an interesting connection to the GELU approximated with a logistic distribution ("Logistic Error Linear Unit"?) (i.e. Swish)

c.f. the approximation x \sigma(1.702 x) from the GELU paper.

What are the computational cost (CPU/GPU cycles per node) of Mish vs GELU vs Swish?
If it is possible to reduce the CPU/GPU cycles of the computation through simplification,
it can save both time and energy, leading to lower green house gas emissions.
Of course convergence is important, RELU isn't that computational heavy, but it is weak,
and right now the second derivative of both GELU and Swish are symmetric.

In function call: return inputs * tf.math.tanh(tf.math.softplus(x))
Should be inputs instead of x inside softplus

Keras and Tensorflow links lead to "404" error.

Through the mish landing page I found this paper
TanhExp: A Smooth Activation Function with High Convergence Speed for Lightweight Neural Networks Paper

Long story short it is an adjustment on mish.
In many of the evaluations that they have done, they show consistent improvement across many benchmarks over mish.

Whether this improvement holds upon other datasets (ImageNet) or networks remain to be seen.
The authors definitely have not done as extensive testing as the author of mish has.

I do have problem with the following section
"4.6 Comparison of the Computation Speed"

Here they argue that their TanhExp function is faster then mish in its original form(forward pass), 1st derivative(backward pass) and 2nd derivative form.

However, I would like to point out that for at least the original form, they used the original formulation of mish and not the (potentially) faster mish discussed here before.

I have conducted my own tests and found that for CPU calculations, fastmish is faster than tanhexp.
Unfortunately, I have not been able to test backward passes. But to the best of my knowledge, the 1st and 2nd derivatives do not differ between the original formulation and the faster formulation. (This is because they are based on an identity and the speed gain comes from CPU friendly formulas)

On GPU, TanhExp was faster than both versions of mish.

Since they do not have a separate github page I decided to at least let you know so that you could raise an objection if need be.
I tried thinking of a faster formulation of TanhExp to be fair, but it is beyond my knowledge.

To be frank I am interested in the future of TanhExp since it shows consistent accuracy and training speed and stability gains in the limited scope they tested compared to mish.

However, activation functions come and go real fast(remember swish?) and for simplicity's sake, ReLu isnt going anywhere. Also, previous investigations into this issue showed that minor differences in activation function speed might be eclipsed by other bottlenecks.

Anyway here is my notebook.
TanhExp.zip

When trying to change cifar-10 to cifar-100(In cifar-10-senet-18-mish), I try to change the code in the second cell like this:
def get_training_dataloader(train_transform, batch_size=128, num_workers=0, shuffle=True):
""" return training dataloader
Args:
train_transform: transfroms for train dataset
path: path to cifar100 training python dataset
batch_size: dataloader batchsize
num_workers: dataloader num_works
shuffle: whether to shuffle
Returns: train_data_loader:torch dataloader object
"""

transform_train = train_transform
cifar100_training = torchvision.datasets.CIFAR100(root='.', train=True, download=True, transform=transform_train)
cifar100_training_loader = DataLoader(
    cifar100_training, shuffle=shuffle, num_workers=num_workers, batch_size=batch_size)

return cifar100_training_loader

define test dataloader

def get_testing_dataloader(test_transform, batch_size=128, num_workers=0, shuffle=True):
""" return training dataloader
Args:
test_transform: transforms for test dataset
path: path to cifar100 test python dataset
batch_size: dataloader batchsize
num_workers: dataloader num_works
shuffle: whether to shuffle
Returns: cifar100_test_loader:torch dataloader object
"""

transform_test = test_transform
cifar100_test = torchvision.datasets.CIFAR100(root='.', train=False, download=True, transform=transform_test)
cifar100_test_loader = DataLoader(
    cifar100_test, shuffle=shuffle, num_workers=num_workers, batch_size=batch_size)

return cifar100_test_loader

However, it raises RuntimeError: CUDA error: device-side assert triggered.
Could you help me of how to do cifar-100?

Besides, I'd like to know how to use stats.ipynb...Thanks a lot!!!

@digantamisra98

https://github.com/digantamisra98/Mish/blob/master/output_landscape.py is not found
Please fix it

Hi, I tried to use mish instead of relu on mask r-cnn, including res-net and three head, but the AP dropped 0.3 in the end. What is the possible problem?
Thank you!

Hi, I am trying to save weights of trained model utilizing Keras model_checkpoint.

The error I am getting (last couple of rows):

File "C:\Users\SParkhonyuk\AppData\Local\Continuum\anaconda3\envs\DL\lib\site-packages\keras\engine\network.py", line 860, in get_config
layer_config = layer.get_config()

File "C:\Users\SParkhonyuk\repos...\mish.py", line 36, in get_config
return dict(list(base_config.items()) + list(config.items()))

NameError: name 'config' is not defined

Some global configuration is missing in mish.py ( in fact, it is a straightforward function).

Can you please direct me to what exactly needs to be added to make the model save-able?

Although Swish by Google researchers is more popular, it is the result of NAS and not very well-justified. Actually Swish is a modification of the earlier activation function SiLU (Sigmoid-weighted linear units), which shows better results than Swish on several benchmarks and simpler conceptually. Moreover, SiLU is more theoretical justified, for example the idea of self-regularization was proposed. Please see the following paper.

Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. arXiv preprint arXiv:1702.03118, 2017.