Hi,

In MNIST, Cifar-10 tutorials there is Selu as well as alpha dropout used and result after experiment is that SNN outperforms ReLU, ELU based models. Mnist based models (Lenet) can work without dropout, batchNorm with quite good accuracy, so my question is if Selu alone (no dropout and Batch Norm) is according to your observations, increasing accuracy? What I mean is that I have model that is working on MNIST and is a basic CNN eg. convolutions, ReLU, Fully Connected and softmax, and assuming that initialization of weights and normalization of input was done correctly can I expect increased accuracy?

So I used skchem pipeline to feature extract meaningful features from the sdf train file mention in the official Tox21 challenge website. But I am not able to get the 801 features that are there in the zip csv file . Can someone help me with that py code. My aim is to experiment with the architucture with SeLU and technique and not get into domain specific feature extraction details.

https://github.com/bioinf-jku/SNNs/commit/1a366963a18b328c4e5d13f95fb9993fe990e4fb#diff-c7c21fc90a9f9340db7b45c70ef0e393R13

Could you please give me some hints?

Thank you

With the UCI data, how did you preprocess the categorical and continuous variables?

Did you enforce a min/max or did you just standardize the continuous variables? And for the categorical, did you use one-hot/dummy coding or standardized them?

Edit: Also, what batch size did you use? Did it depend on the sample size?

Thanks!

https://github.com/bioinf-jku/SNNs/blob/f992b229795712a54c67266995d8ea522cd10770/selu.py#L31 and many other examples (e.g. Keras) do an additional trick where samples are resampled if they're not within two standard deviations of the mean. I'm curious how much of an effect this truncation has on the fix points derivation? Are they analytically identical for a normal distribution and a truncated normal distribution?

I read in the paper that "Uniform and truncated Gaussian distributions with these moments led to networks with similar behavior." but this feels unsatisfactory to me. Maybe a small discrepancy becomes really problematic for deeper networks? This aligns with my experience that it's still beneficial to have batchnorm/layernorm with SELU.

Thank you for sharing the code. I have successfully applied them (SELU and alpha_dropout) building a pure fully connected network (7 layers) for a regression problem (the R2 between the predicted and the observed variable is greater than 0.99!).

Right now I'm trying to replace the RELU in cnn_graph with SELU. Unlike the standard cnn, cnn_graph performs the convolution on the graph Fourier transformed inputs (a recursive process involves multiple matrix multiplications between the layer-specific graph Laplacian and the layer inputs). The original normalized input is shifted to some unknown distribution by the graph Fourier transform. Therefore, I don't know how to apply the SELU even on the first cnn_graph layer. Could you give me some suggestions on this?

Besides, can I use some normalization on the output of cnn_graph before feeding them into fully connected network using SELU as activation function?

Thanks!

@bioinf-jku,thank you for your nice work! I am a new to deep learning,and I have some simply questions,since the net in your test code is not very deep,It makes no big different to add batch normalization layers after each convolution layers,but if the net is very deep,is it necessary to add batch normalization layer after each convolution layers?Or there is no need to do so since the activation function selu has the ability to batch normalize the input layer? thank you in advance!

I think the proposed SELU is an powerful non-linearity for MLPs. The self-normalizing property comes from the derivation of the forward propagation. This property could be confirmed by the following codes.

import torch
f = torch.nn.functional.selu
x = torch.randn(1024, 1024) * 456 + 123
lin = torch.nn.Linear(1024, 1024, bias=False)
_ = torch.nn.init.kaiming_normal_(lin.weight, nonlinearity="linear")
with torch.no_grad():
    for i in range(100): x = f(lin(x))
print(f"mean = {x.mean()}") # 0.00253
print(f"var = {(x ** 2).mean()}") # 1.05135

However, the self-normalizing property only holds for the forward pass, it does not hold for the backward pass (is it right?). Noisy gradients will definitely be harmful for the learning processes. The proposed SELU is based on ELU, which is based on a selective preference. I wonder if there could exists a more general non-linearity that has self-normalizing properties for both the forward and backward propagations. If it is possible, how could we find it?

I've been experimenting with SELUs, and found they provide an improvement in terms of computation time during training with respect to batchnorm, thank you for your work.

I just have a question regarding the effect of bias in linear layers. As I understand it, every neuron should have mean zero in order to stay in the self regularizing zone, but bias precisely shifts that mean. In my experiments however I didn't see much of an effect either removing or adding biases. I see that in the tutorial notebook bias is used, and I wonder wether you've considered the issue.

Hi, thank you for the paper and the code. Could you please tell us how you scale the inputs to zero mean and unit variance in step (1) in selu.py

Thank you

Hi,

Just a question: when applying the alpha dropouts, on the prediction phase, not the training, do you scale by p where p is the probability to be kept? https://stats.stackexchange.com/questions/205932/dropout-scaling-the-activation-versus-inverting-the-dropout