nninit
Weight initialization schemes for PyTorch nn.Modules. This is a port of the popular nninit for Torch7 by @kaixhin.
##Update
This repo has been merged into PyTorch's nn module, I recommend you use that version going forward.
###PyTorch Example
import nninit
from torch import nn
import torch.nn.init as init
import numpy as np
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(5, 10, (3, 3))
init.xavier_uniform(self.conv1.weight, gain=np.sqrt(2))
init.constant(self.conv1.bias, 0.1)
network = Net()
##Installation Clone the repo and run python setup install
##Usage
import nninit
from torch import nn
import numpy as np
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(5, 10, (3, 3))
nninit.xavier_uniform(self.conv1.weight, gain=np.sqrt(2))
nninit.constant(self.conv1.bias, 0.1)
network = Net()
##Supported Schemes
nninit.uniform(tensor, a=0, b=1)- Fillstensorwith values from a uniform, U(a,b)nninit.normal(tensor, mean=0, std=1)- Fillstensorwith values drawn from a normal distribution with the given mean and stdnninit.constant(tensor, val)- Fillstensorwith the constantvalnninit.xavier_uniform(tensor, gain=1)- Fillstensorwith values according to the method described in "Understanding the difficulty of training deep feedforward neural networks" - Glorot, X. and Bengio, Y., using a uniform distribution.nninit.xavier_normal(tensor, gain=1)- Fillstensorwith values according to the method described in "Understanding the difficulty of training deep feedforward neural networks" - Glorot, X. and Bengio, Y., using a normal distribution.nninit.kaiming_uniform(tensor, gain=1)- Fillstensorwith values according to the method described in "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification" - He, K. et al. using a uniform distribution.nninit.kaiming_normal(tensor, gain=1)- Fillstensorwith values according to the method described in "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification" - He, K. et al. using a normal distribution.nninit.orthogonal(tensor, gain=1)- Fills thetensorwith a (semi) orthogonal matrix. Reference: "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks" - Saxe, A. et al.nninit.sparse(tensor, sparsity, std=0.01)- Fills the 2Dtensoras a sparse matrix, where the non-zero elements will be drawn from a normal distribution with mean=0 and std=std.
145 Dec 30, 2022
31 Dec 7, 2022
18 Oct 12, 2022
357 Jan 4, 2023
759 Jan 8, 2023
45 Dec 8, 2022
36 Nov 15, 2022
117 Nov 21, 2022
440 Dec 18, 2022
45 Dec 29, 2022
138 Dec 28, 2022
3 Nov 18, 2021
52 Nov 30, 2022
123 Jan 4, 2023
91 Dec 2, 2022
89 Dec 13, 2022
1 Dec 17, 2021
7 Jun 6, 2022
101 Dec 16, 2022