Hello, thanks for the great work. Using the exponential identity for tanh, you can remove two of the transcendental operations (exp, log) and get what, hopefully, should be a faster implementation.

Since

\tanh(x) = (e^2x - 1) / (e^2x + 1)

you can express Mish as:

y = e^x
mish(x) = x y (y + 2) / (y^2 + 2 y + 2)

or equivalently (to avoid overflow when x is large)

y = e^-x
mish(x) = x (1 + 2 y) / (1 + 2 y + 2 y^2) 

NB: With a little tweak, there is an interesting connection to the GELU approximated with a logistic distribution ("Logistic Error Linear Unit"?) (i.e. Swish)

x \tanh(0.5 \log( 1 + e^x) ) = x \sigma(x - \log 2)

c.f. the approximation x \sigma(1.702 x) from the GELU paper.

First, congratulation on your Mish paper accepted.

I've been thinking about activation functions, and how they can be improved, what properties are most important for a long time, and also inspired by your paper (I noticed the days old update), and e.g. recent SharkFin (my own unpublished idea has some similarities).

I was just thinking if I could ask you some questions, I'm not sure if this is the right place.

It seems like almost any function can do (all except polynominal proven to work for shallow wide networks, and that restriction eliminated by deep-narrow networks):

Universal Approximation with Deep Narrow Networks https://arxiv.org/pdf/1905.08539.pdf

we show that the class of neural networks of arbitrary depth, width n+m+2, and activation function ρ, is dense [..] This covers every activation function possible to use in practice, and also includes polynomial activation functions [..]

We refer to these as enhanced neurons. [..]

4.2. Square Model Lemma 4.3 One layer of two enhanced neurons, with square activation function, may exactly represent the multiplication function (x,y)→xy on R^2 [..]

Remark 4.7 Lemma 4.5 is key to the proof of Proposition 4.6. It was fortunate that the reciprocal function may be approximated by a network of width two - note that even if Proposition 4.6 were already known, it would have required a network of width three. It remains unclear whether an arbitrary-depth network of width two, with square activation function, is dense in C(K). [..]

Remark 4.8 Note that allowing a single extra neuron in each layer would remove the need for the trick with the reciprocal, as it would allow [..] Doing so would dramatically reduce the depth of the network. We are thus paying a heavy price in depth in order to reduce the width by a single neuron.

[I've yet to read much further, but this seems very important.]

So my reading is all but identity function can work as activation function (when more than one hidden layer), and a network no wider than 4 (or 5 better, optimal?) can approximate all four elementary arithmetic operation. Could also approximate e.g. sine and exponential with such narrow network (through Fourier-theorem), I think.

Have you looked at Capsule networks, and deep variant? https://arxiv.org/pdf/1904.09546.pdf

May main worry is that by thinking about better activation functions (of if there can be one best), I'm wasting my time, with them and/or (traditional) backpropagation going away, with the thousand-brain theory and more. Capsule networks seem similar, with a voting mechanism. It at least has ReLU in the first layer (I didn't look at more in detail).

Have you looked at BERT and variants? I assume they could use your functions, or do you know if there are exceptions, making GELU better for them? I'm thinking it's maybe just ignorance (or authors extending, want to change one thing at a time):

https://arxiv.org/pdf/1909.11942.pdf

The backbone of the ALBERT architecture is similar to BERT in that it uses a transformer encoder (Vaswani et al., 2017) with GELU nonlinearities

The Reversible Residual Network: Backpropagation Without Storing Activations https://arxiv.org/pdf/1707.04585.pdf

Just wondering if all Activation Functions have been addressed in the ReadME.

• https://www.semanticscholar.org/paper/A-comparative-performance-analysis-of-different-in-Farzad-Mashayekhi/bcfdfe54796c501a90c3b353661a19e9c161d2c8/figure/0
• https://www.semanticscholar.org/paper/Activation-Functions-for-Generalized-Learning-A-Villmann-Ravichandran/04d54996bcbe44b3547da889d7eab8aab3660990/figure/0
• https://www.semanticscholar.org/paper/Comparison-of-new-activation-functions-in-neural-Gomes-Ludermir/9b37079041bdaca4248ab4f62f1a63013a50f067/figure/1
• https://www.semanticscholar.org/paper/Searching-for-Activation-Functions-Ramachandran-Zoph/c8c4ab59ac29973a00df4e5c8df3773a3c59995a/figure/2

When using in an RNN, I get {NameError}name 'uses_learning_phase' is not defined when applying a Dense layer on the output of an LSTM.

Tensorflow 1.14

I've tested mish vs elu on a simple feedforward network with L2 and dropout and got this result:

You can check it with Colab Notebook, press ctrl+f9.

I read your research paper on Mish, and from the graphs I saw, it is clearly the best activation function.

You had two different implementations of Mish with widely different computational overheads. There were standard Mish and Mish-CUDA, where Mish-CUDA has almost zero computational cost. I'm optimizing DL4S for Metal, and looking to optimize Mish so that it can be the top-recommended function. However, I need to know how you implemented Mish-CUDA.

Could you please explain how Mish-CUDA was faster than the standard Mish, and how I might implement it in a generic GPGPU context? I have a lot of experience with assembly-level optimization and I know the differences between Apple GPUs/Metal and Nvidia GPUs/CUDA well, if that helps with your explanation.

Hi! This is not an issue per se and can be closed.

First of all, thank you for advancing progress in deep learning.

I'm just a random guy that want to implement an AGI (lol) and like many Nlp engeeners, I need HIGHLY accurate neural networks for fundamental NLP tasks (e.g POS tag, NER, dep parsing, Coref resolution, WSD, etc) They are all not very accurate (often sub 95% F1 score) and their errors add up.

Such limitations make Nlp not yet suitable for many things. This is why improving the state of the art (which can be observed on paperswithcode.com) is a crucial priority from academicians.

Effectively, many researchers have smart ideas to improve the state of the art and often slightly improve it by: Having a "standard neural network" for the task and mix with it their new fancy idea.

I talk from knowledge, I've read most papers from state of the art leaderboards from most fundamental NLP tasks. Almost always they have this common baseline + one idea, theirs. The common baseline sometimes slowly evolve (e.g now it's often a pre trained model (say BERT) + fine tuning + their idea.

Sorry to say, but "this" is to me retarded Where "this" mean the fact that by far, most researchers work in isolation, not integrating others ideas (or with such a slow inertia). I would have wished that state of the art in one Nlp task would be a combination of e.g 50 innovative and complementary ideas from researchers. You are researchers, do you have an idea why that is the case? If someone actually tried to merge all good complementary and compatible ideas, would they have the best, unmatchable state of the art? Why facebookresearch, Microsoft, Google don't try the low hanging fruit in addition to producing X new shiny ideas per month, actually try to merge them in a coherent, synergetic manner?? I would like you to tell me what you think of this major issue that slow AI progress.

As an example of such inertia let's talk about Swish, Mish or RAdam : Those things are incredibly easy to try and see "hey does it give to my neural network free accuracy gains?" Yet not any paper on state of the art leaderboards has tried Swish, Mish or RAdam despite being soo simple to try (you don't need to change the neural network) Not even pre trained models where so many papers depend on them (I opened issues for each of them).

Thank you for reading.

What are the computational cost (CPU/GPU cycles per node) of Mish vs GELU vs Swish? If it is possible to reduce the CPU/GPU cycles of the computation through simplification, it can save both time and energy, leading to lower green house gas emissions. Of course convergence is important, RELU isn't that computational heavy, but it is weak, and right now the second derivative of both GELU and Swish are symmetric.

Hi, thanks for this interesting new activation function. I've tested it with YOLOv3-SPP on a V100 from https://github.com/ultralytics/yolov3 and have mixed feedback. The performance improves slightly, but the training time is much slower and the GPU memory requirements are much higher vs LeakyReLU(0.1). Any suggestions on how to improve speed/memory in PyTorch? Thanks!

From https://github.com/AlexeyAB/darknet/issues/3114#issuecomment-554784601:

| [email protected] | mAP0.5:0.95 | GPU memory | Epoch time --- | --- | --- | --- | --- LeakyReLU(0.1) | 48.9 | 29.6 | 4.0G | 31min Mish() | 50.9 | 31.2 | 11.1G | 46min

class Swish(nn.Module):
    def __init__(self):
        super(Swish, self).__init__()

    def forward(self, x):
        return x.mul_(torch.sigmoid(x))


class Mish(nn.Module):  # https://github.com/digantamisra98/Mish
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x.mul_(F.softplus(x).tanh())

When trying to change cifar-10 to cifar-100(In cifar-10-senet-18-mish), I try to change the code in the second cell like this: def get_training_dataloader(train_transform, batch_size=128, num_workers=0, shuffle=True): """ return training dataloader Args: train_transform: transfroms for train dataset path: path to cifar100 training python dataset batch_size: dataloader batchsize num_workers: dataloader num_works shuffle: whether to shuffle Returns: train_data_loader:torch dataloader object """

transform_train = train_transform
cifar100_training = torchvision.datasets.CIFAR100(root='.', train=True, download=True, transform=transform_train)
cifar100_training_loader = DataLoader(
    cifar100_training, shuffle=shuffle, num_workers=num_workers, batch_size=batch_size)

return cifar100_training_loader

define test dataloader

def get_testing_dataloader(test_transform, batch_size=128, num_workers=0, shuffle=True): """ return training dataloader Args: test_transform: transforms for test dataset path: path to cifar100 test python dataset batch_size: dataloader batchsize num_workers: dataloader num_works shuffle: whether to shuffle Returns: cifar100_test_loader:torch dataloader object """

transform_test = test_transform
cifar100_test = torchvision.datasets.CIFAR100(root='.', train=False, download=True, transform=transform_test)
cifar100_test_loader = DataLoader(
    cifar100_test, shuffle=shuffle, num_workers=num_workers, batch_size=batch_size)

return cifar100_test_loader

However, it raises RuntimeError: CUDA error: device-side assert triggered. Could you help me of how to do cifar-100?

Besides, I'd like to know how to use stats.ipynb...Thanks a lot!!!

@digantamisra98

Hi, do you have any tips on how to implement Mish as an inplace operation in PyTorch? Could this work? Is there a way of making the F.softplus() operation also inplace?

class Mish_(nn.Module):
    def __init__(self):
        super().__init__()
        
    def forward(self, x):
        softplus_res = F.softplus(x)
        torch.tanh_(softplus_res)
        return x * softplus_res

Hello! Great work on this activation function! I've been using it in some of my projects with great success.

I want to let you know I found what the gain should be set at during kaiming weight initialization for Mish.

I found this experimentally using this code:

import torch
import torch.nn.functional as F
import pandas as pd
import numpy as np

device = 'cpu'

def mish(x):
    return x * (torch.tanh(F.softplus(x)))

aa = []
bb = []
for n in range(100):
    with torch.no_grad():
        a = torch.randn(5000, 5000, device=device)
        b = a
        x = 0.0 + 0.00001 * n
        for i in range(10):
            l = torch.nn.Linear(5000, 5000, bias=False).to(device)
            torch.nn.init.kaiming_uniform_(l.weight, a=x)
            b = mish(l(b))
        aa.append(b.std().item())
        bb.append(x)
        print(x)
        print (f"in: {a.std().item():.8f}, out: {b.std().item():.8f}")
pd.DataFrame(data=aa, index=bb).plot(figsize=(20,8))

which was talked about here. The "a" hyperparameter for init.kaiming_uniform_ is not actually the gain but the negative slope of a leaky relu, so really I experimentally found the equivalent negative slope of mish for kaiming_uniform_ init. The actual gain is found internally by math.sqrt(2.0 / (1 + a ** 2)).

This is an example of the code output. I found through repeated experiments that 0.0003 results in the most consistently efficient throughput through the network, so it is almost the zero slop of relu but not quite. a=0 did produce okay results, as did say 0.001, but the best averaged over many runs is 0.0003. This is important because for deep networks the pytorch default value of sqrt(5) for initializing conv layers is not a good default value if using mish.

I now use something like

for m in self.modules():
    if isinstance(m, (nn.Conv1d, nn.Linear)):
        torch.nn.init.kaiming_uniform_(m.weight, a=0.0003)