Distiller is an open-source Python package for neural network compression research.

Hello there, I am wondering about the state of the ADC implementation, and what remains to bring it to a functional state. In the ADC merge commit message, you mentioned that it is still WiP and that it is using an unreleased version of Coach. Is that still the case? Also, is there any documentation for how to use ADC in Distiller?

Thanks

Hi Neta,

I tried to run the sensitivity analysis for filter with the following command 'python3 compress_classifier.py -a resnet20_cifar --data ../../../data.cifar10/ -j 12 --resume=../ssl/checkpoints/checkpoint_trained_dense.pth.tar --sense=filter', but got an error, detailed log:

Logging to TensorBoard - remember to execute the server:

tensorboard --logdir='./logs'

=> loading checkpoint ../ssl/checkpoints/checkpoint_trained_dense.pth.tar Checkpoint keys: arch optimizer compression_sched state_dict best_top1 epoch best top@1: 92.540 Loaded compression schedule from checkpoint (epoch 179) => loaded checkpoint '../ssl/checkpoints/checkpoint_trained_dense.pth.tar' (epoch 179) Optimizer Type: <class 'torch.optim.sgd.SGD'> Optimizer Args: {'lr': 0.1, 'momentum': 0.9, 'dampening': 0, 'weight_decay': 0.0001, 'nesterov': False} Files already downloaded and verified Files already downloaded and verified Dataset sizes: training=45000 validation=5000 test=10000 Running sensitivity tests Testing sensitivity of module.conv1.weight [0.0% sparsity] Traceback (most recent call last): File "compress_classifier.py", line 782, in main() File "compress_classifier.py", line 339, in main return sensitivity_analysis(model, criterion, test_loader, pylogger, args) File "compress_classifier.py", line 750, in sensitivity_analysis group=args.sensitivity) File "/home/chongyu/application/distiller/distiller/sensitivity.py", line 108, in perform_sensitivity_analysis scheduler.on_epoch_begin(0) File "/home/chongyu/application/distiller/distiller/scheduler.py", line 112, in on_epoch_begin policy.on_epoch_begin(self.model, self.zeros_mask_dict, meta) File "/home/chongyu/application/distiller/distiller/policy.py", line 123, in on_epoch_begin self.is_last_epoch = meta['current_epoch'] == (meta['ending_epoch'] - 1) TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

It looks like there is no valid value for meta['ending_epoch']. Can you kindly suggest how to solve it? Thanks.

I tried to use distiller for my cascade rcnn model but it did not work for me and i need some help.

I used AGP pruning method and followed the schedule introduced in the guide doc to change my training code, and the backbone resnet50(used in senet model) parameters were selected to prune. However in the process the total sparsity just was 0. It failed. What is the problem and how should i adjust??

in epoch loop: in train function： when i want to see the mask info, i get this error: i don't know why the mask is not working and i think the masks are set when epoch begin. ps: i just can't see any model compression project support object detection, including distiller and pocketflow? Why? What is the difference between object detection problems and classification model when we try to compress them ??

looking forward to you answer~3q

Hi~, I implemented my model quantization like this: `import distiller from distiller.quantization import PostTrainLinearQuantizer

quantizer = PostTrainLinearQuantizer(model) quantizer.prepare_model(torch.rand(*your_input_shape)) apputils.save_checkpoint(0, 'mymodel', model, optimizer=None, name='model', dir='quantization')`

But output model did not resize to 1/4,just not changed, would you please to help me? Thanks!

Hello,

Im trying to quantise my CNN classification model using a simple post training quantisation. The paper by Ramakrishnan on Quantisizing CNN for efficient inference suggests that we can try to two ways to go about this.

1. Weight Only quantisation- Either quantise the tensors layer wise or channel wise, they recommend channel wise quantisation.
2. Activations and weights both.

I tried to perform weight only quantisation but am getting horrible results. My 99% MNIST model was degraded to an accuracy of 11% after post training quantisation. The code for quantising the pytorch tensors is given here:

QTensor = namedtuple('QTensor', ['tensor', 'scale', 'zero_point'])

def quantize_tensor(x, num_bits=8):
    qmin = 0.
    qmax = 2.**num_bits - 1.
    min_val, max_val = x.min(), x.max()

    scale = (max_val - min_val) / (qmax - qmin)

    initial_zero_point = qmin - min_val / scale

    zero_point = 0
    if initial_zero_point < qmin:
        zero_point = qmin
    elif initial_zero_point > qmax:
        zero_point = qmax
    else:
        zero_point = initial_zero_point

    zero_point = int(zero_point)
    q_x = zero_point + x / scale
    q_x.clamp_(qmin, qmax).round_()
    q_x = q_x.round().byte()
    return QTensor(tensor=q_x, scale=scale, zero_point=zero_point)

def dequantize_tensor(q_x):
    return q_x.scale * (q_x.tensor.float() - q_x.zero_point)

And the function I run the 99% trained model to perform post training quantisation is as follows:

def quantizeModel(model):
  """
  Post Training Quantisation
  """
  q_tensors = {}
  
  for k, m in enumerate(model.modules()):

      if  isinstance(m, nn.Conv2d):
          weight= m.weight.data
          q_weight = weight.clone()
        
          num_inp_channels = weight.shape[0]
          num_output_channels = weight.shape[1]
          
          # per channel quantisation
          for i in range(num_inp_channels):
            for j in range(num_output_channels):
              quantized_weight = quantize_tensor(weight[i,j])
              q_weight[i,j] = quantized_weight.tensor.float()
          m.weight.data = q_weight
        else:
          # for fully connected layers, layer wise quantisation
          weight = m.weight.data
          quantized_weight = quantize_tensor(weight)
          m.weight.data = quantized_weight.tensor.float()
  return model

What am I doing wrong here? Any help would deeply appreciated ! I know this isn't distiller code but I was trying to build a very minimal example of post training quantisation and you folks are the experts. I am using a very simple model for MNIST classificatoin. My model code is given here:

class Net(nn.Module):
    def __init__(self, mnist=True):
      
        super(Net, self).__init__()
        if mnist:
          num_channels = 1
        else:
          num_channels = 3
          
        self.conv1 = nn.Conv2d(num_channels, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)

    
    def quantize(self):
      """
      Post Training Quantisation
      """
      q_tensors = {}

      for k, m in enumerate(self.modules()):

        if isinstance(m, nn.Conv2d):   
          weight= m.weight.data
          q_weight = weight.clone()

          num_inp_channels = weight.shape[0]
          num_output_channels = weight.shape[1]

          # per channel quantisation
          for i in range(num_inp_channels):
            for j in range(num_output_channels):
              quantized_weight = quantize_tensor(weight[i,j])
              q_weight[i,j] = quantized_weight.tensor.float()

          m.weight.data = q_weight
        
      
      
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4*4*50)        
        x = F.relu(self.fc1(x))
        x = self.fc2(x)

        return F.log_softmax(x, dim=1)
    

Best, Karanbir Chahal

Hello: I want to use MultiStepMultiGammaLR scheduler in my pruning lr_scheduler. When I using the compress_classifier.py to pruning the res_net20_cifar from the begining and define the lr_scheduler in the yaml file, it works well. But when I using the checkpoint to train and prune, the lr_scheduler defined in the yaml file doesn't work. The lr doesn't decay when the epoch achieve defined milestone.

I use the script below: python3 compress_classifier.py --arch resnet20_cifar dataset/ -p=50 --lr=0.3 --epochs=150 -b 128 --compress=resnet20_cifar_ele_pruning.yaml -j=1 --vs 0 --deterministic --resume=logs/resnet20_cifar_baseline/checkpoint.pth.tar

Below is my yaml setting

version: 1
pruners:
  low_pruner:
    class: AutomatedGradualPruner
    initial_sparsity : 0.05
    final_sparsity: 0.60
    weights: [module.layer1.2.conv1.weight,  module.layer1.2.conv1.weight,
              module.layer1.0.conv1.weight,  module.layer1.0.conv2.weight,
              module.layer1.1.conv1.weight,  module.layer1.1.conv2.weight]

  mid_pruner:
    class:  AutomatedGradualPruner
    initial_sparsity : 0.05
    final_sparsity: 0.67
    weights: [module.layer2.2.conv1.weight,  module.layer2.2.conv2.weight,
              module.layer2.0.conv2.weight,  module.layer2.0.downsample.1.weight,
              module.layer2.0.conv1.weight,  module.layer2.0.downsample.0.weight,
              module.layer2.1.conv1.weight,  module.layer2.1.conv2.weight]

  high_pruner:
    class:  AutomatedGradualPruner
    initial_sparsity : 0.05
    final_sparsity: 0.76
    weights: [module.layer3.0.conv1.weight,  module.layer3.1.conv1.weight,
              module.layer3.1.conv2.weight,  module.layer3.0.conv2.weight,
              module.layer3.0.downsample.0.weight, module.layer3.0.downsample.1.weight,
              module.fc.weight]
lr_schedulers:
  training_lr:
    class: MultiStepMultiGammaLR
    milestones: [300, 302, 400]
    gammas: [0.1, 0.1, 0.5]

policies:
    - pruner:
        instance_name: low_pruner
      starting_epoch: 300
      ending_epoch: 400
      frequency: 2
    - pruner:
        instance_name: mid_pruner
      starting_epoch: 300
      ending_epoch: 400
      frequency: 2
    - pruner:
        instance_name: high_pruner
      starting_epoch: 300
      ending_epoch: 400
      frequency: 2
    - lr_scheduler:
        instance_name: training_lr
      starting_epoch: 0
      ending_epoch: 400
      frequency: 1

Is there any problem in my script and yaml setting?

This PR is a fix for issue #422.

The file data_loader had fixed classification image size for ImageNet as [1, 3, 224, 224]. However, all Inception models requires an input image size of [1, 3, 299, 299].

To fix this issue, I modified the apputils/image_classifier.py file to add a new parameter to the load_data function. This function calls apputils.load_data, so I changed the corresponding function in apputils/data_loader.py.

Also, image_classifier.py is modified to consider both losses from the normal classifier and aux_logits classifier of inception network, and the classification accuracy is calculated only from normal classifier.

@nzmora : Please review the changes.

PS: My fork is based on PyTorch 1.3, so all additional changes for PyTorch1.3 support are present in this PR.

If I want to train a dorefa-Net quantization on alexnet , is the command line like this ? python compress_classifier.py -a alexnet /ImageNet_Share/ --compress=/distiller/examples/quantization/quant_aware_train/alexnet_bn_dorefa.yaml We do not need to modify the code in compress_classifier.py to do this training?

I installed all the environments and packages according to the tutorial and got this error when running the code: ModuleNotFoundError: No module named 'gitdb.utils.compat'. Ask for help, how can I resolve this error? thank you very much!

Hi, for the last two days, I have been trying to prune two models, (mobilenetv1 and resnet50) using distillers CompressionScheduler and AutomatedGradualPruner. first of all let me say this, that I didnt use the yaml config file for this. I directly incorporated distiller in my own code as it seemed a better choice without much hassle. So based on the tiny example which is provided in the compress_classifier.py which by the way reads as :

Integrating compression is very simple: simply add invocations of the appropriate
compression_scheduler callbacks, for each stage in the training.  The training skeleton
looks like the pseudo code below.  The boiler-plate Pytorch classification training
is speckled with invocations of CompressionScheduler.
For each epoch:
    compression_scheduler.on_epoch_begin(epoch)
    train()
    validate()
    compression_scheduler.on_epoch_end(epoch)
    save_checkpoint()
train():
    For each training step:
        compression_scheduler.on_minibatch_begin(epoch)
        output = model(input)
        loss = criterion(output, target)
        compression_scheduler.before_backward_pass(epoch)
        loss.backward()
        compression_scheduler.before_parameter_optimization(epoch)
        optimizer.step()
        compression_scheduler.on_minibatch_end(epoch)

So I did this and nothing worked! but before I show the snippet I used (the way I implemented this) and ask my question, may I ask one right now ? Does compression_scheduler by itself, does any compression ? I mean by default simply invoking it like above! would it run a default policy? or should we always add a policy for it to work? tlrd, I used the bare compression scheduler, and obviously nothing happened, except, I got a bit lower accuracy. I then added the agp pruner to it like this, and again nothing happened! :

agp = distiller.pruning.AutomatedGradualPruner('retinaface', 0.15, 0.5, weights_agp_resnet_no_bn)
scheduler = distiller.scheduler.CompressionScheduler(net)
scheduler.add_policy(agp, epochs=[max_epoch], starting_epoch=50, ending_epoch=max_epoch, frequency=1)

and this is what I have come up so far :

you said do : 
"""
train():
    For each training step:
        compression_scheduler.on_minibatch_begin(epoch)
        output = model(input)
        loss = criterion(output, target)
        compression_scheduler.before_backward_pass(epoch)
        loss.backward()
        compression_scheduler.before_parameter_optimization(epoch)
        optimizer.step()
        compression_scheduler.on_minibatch_end(epoch)
"""

and I did :

def train_one_epoch(net, data_loader, optimizer, criterion, cfg, gamma, current_epoch, step_index,scheduler, device=torch.device('cpu')
                     ):
    
    net.train()
    epoch_size = len(data_loader)
    # load train data
    for iteration, (images, targets) in enumerate(data_loader):
        load_t0 = time.time()
        images = images.to(device)
        targets = [anno.to(device) for anno in targets]

        scheduler.on_minibatch_begin(current_epoch, iteration, epoch_size)

        # forward
        out = net(images)

        loss_l, loss_c, loss_landm = criterion(out, priors, targets)
        loss = cfg['loc_weight'] * loss_l + loss_c + loss_landm

        scheduler.before_backward_pass(current_epoch, iteration, epoch_size, loss, optimizer)

       # backprop
        optimizer.zero_grad()
        loss.backward()

        scheduler.before_parameter_optimization(current_epoch, iteration, epoch_size, optimizer)
        
        optimizer.step()

        scheduler.on_minibatch_end(current_epoch, iteration, epoch_size, optimizer)

        lr = adjust_learning_rate(optimizer, gamma, current_epoch, step_index, iteration, epoch_size)

        load_t1 = time.time()
        batch_time = load_t1 - load_t0
        eta = int(batch_time * (len(data_loader) - iteration))
        
        print('Epoch:{}/{} || Iter: {}/{} || Loc: {:.4f} Cla: {:.4f} Landm: {:.4f} || LR: {:.8f} || Batchtime: {:.4f} s || ETA: {}'
             .format(current_epoch, max_epoch, iteration + 1, epoch_size,
              loss_l.item(), loss_c.item(), loss_landm.item(),
             lr, batch_time, str(datetime.timedelta(seconds=eta))))
        
    return loss.item() 

def adjust_learning_rate(optimizer, gamma, epoch, step_index, iteration, epoch_size):
    """Sets the learning rate
    # Adapted from PyTorch Imagenet example:
    # https://github.com/pytorch/examples/blob/master/imagenet/main.py
    """
    warmup_epoch = -1
    if epoch <= warmup_epoch:
        lr = 1e-6 + (initial_lr-1e-6) * iteration / (epoch_size * warmup_epoch)
    else:
        lr = initial_lr * (gamma ** (step_index))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
    return lr

and this is my main training loop : you said, do :

"""
For each epoch:
    compression_scheduler.on_epoch_begin(epoch)
    train()
    validate()
    compression_scheduler.on_epoch_end(epoch)
    save_checkpoint()
"""

and I did :

if __name__ == '__main__':
    # train()
    ...
    if torch.cuda.is_available():
    ...
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    optimizer = optim.SGD(net.parameters(), lr=initial_lr, momentum=momentum, weight_decay=weight_decay)
    
    criterion = MultiBoxLoss(num_classes, 0.35, True, 0, True, 7, 0.35, False)
    priorbox = PriorBox(cfg, image_size=(img_dim, img_dim))
    print('Training will happen on device {}'.format(device))
    with torch.no_grad():
        priors = priorbox.forward()
        priors = priors.to(device)
    ...
    #  I got this list by running this snippet : 
    #     model =  retinaface.RetinaFace(cfg=cfg_resnet50, phase='test')
    #     w=torch.load(weight_path, map_location=torch.device('cpu'))
    #     model.load_state_dict(w, strict=False)
    #     w_lst = [name for name,_ in w.items() if 'weight' in name]
    #     print (w_list) 
    weights_agp_resnet_no_bn = ['module.body.conv1.weight', 'module.body.layer1.0.conv1.weight', 'module.body.layer1.0.conv2.weight', 
                                'module.body.layer1.0.conv3.weight', 'module.body.layer1.0.downsample.0.weight',
                                'module.body.layer1.0.downsample.1.weight', 'module.body.layer1.1.conv1.weight',
                                'module.body.layer1.1.conv2.weight', 'module.body.layer1.1.conv3.weight', 
                                'module.body.layer1.2.conv1.weight', 'module.body.layer1.2.conv2.weight', 
                                'module.body.layer1.2.conv3.weight', 'module.body.layer2.0.conv1.weight', 
                                'module.body.layer2.0.conv2.weight', 'module.body.layer2.0.conv3.weight', 
                                'module.body.layer2.0.downsample.0.weight', 'module.body.layer2.0.downsample.1.weight', 
                                'module.body.layer2.1.conv1.weight', 'module.body.layer2.1.conv2.weight', 
                                'module.body.layer2.1.conv3.weight', 'module.body.layer2.2.conv1.weight', 'module.body.layer2.2.conv2.weight', 
                                'module.body.layer2.2.conv3.weight', 'module.body.layer2.3.conv1.weight', 'module.body.layer2.3.conv2.weight', 
                                'module.body.layer2.3.conv3.weight', 'module.body.layer3.0.conv1.weight', 'module.body.layer3.0.conv2.weight', 
                                'module.body.layer3.0.conv3.weight', 'module.body.layer3.0.downsample.0.weight', 'module.body.layer3.0.downsample.1.weight', 
                                'module.body.layer3.1.conv1.weight', 'module.body.layer3.1.conv2.weight', 'module.body.layer3.1.conv3.weight', 
                                'module.body.layer3.2.conv1.weight', 'module.body.layer3.2.conv2.weight', 'module.body.layer3.2.conv3.weight', 
                                'module.body.layer3.3.conv1.weight', 'module.body.layer3.3.conv2.weight', 'module.body.layer3.3.conv3.weight', 
                                'module.body.layer3.4.conv1.weight', 'module.body.layer3.4.conv2.weight', 'module.body.layer3.4.conv3.weight',
                                'module.body.layer3.5.conv1.weight', 'module.body.layer3.5.conv2.weight', 'module.body.layer3.5.conv3.weight', 
                                'module.body.layer4.0.conv1.weight', 'module.body.layer4.0.conv2.weight', 'module.body.layer4.0.conv3.weight',
                                'module.body.layer4.0.downsample.0.weight', 'module.body.layer4.0.downsample.1.weight', 
                                'module.body.layer4.1.conv1.weight', 'module.body.layer4.1.conv2.weight', 'module.body.layer4.1.conv3.weight', 
                                'module.body.layer4.2.conv1.weight', 'module.body.layer4.2.conv2.weight', 'module.body.layer4.2.conv3.weight', 
                                'module.fpn.output1.0.weight', 'module.fpn.output1.1.weight', 'module.fpn.output2.0.weight', 
                                'module.fpn.output2.1.weight', 'module.fpn.output3.0.weight', 'module.fpn.output3.1.weight', 
                                'module.fpn.merge1.0.weight', 'module.fpn.merge1.1.weight', 'module.fpn.merge2.0.weight', 
                                'module.fpn.merge2.1.weight', 'module.ssh1.conv3X3.0.weight', 'module.ssh1.conv3X3.1.weight', 
                                'module.ssh1.conv5X5_1.0.weight', 'module.ssh1.conv5X5_1.1.weight', 'module.ssh1.conv5X5_2.0.weight',
                                'module.ssh1.conv5X5_2.1.weight', 'module.ssh1.conv7X7_2.0.weight', 'module.ssh1.conv7X7_2.1.weight', 
                                'module.ssh1.conv7x7_3.0.weight', 'module.ssh1.conv7x7_3.1.weight', 'module.ssh2.conv3X3.0.weight', 
                                'module.ssh2.conv3X3.1.weight', 'module.ssh2.conv5X5_1.0.weight', 'module.ssh2.conv5X5_1.1.weight', 
                                'module.ssh2.conv5X5_2.0.weight', 'module.ssh2.conv5X5_2.1.weight', 'module.ssh2.conv7X7_2.0.weight',
                                'module.ssh2.conv7X7_2.1.weight', 'module.ssh2.conv7x7_3.0.weight', 'module.ssh2.conv7x7_3.1.weight',
                                'module.ssh3.conv3X3.0.weight', 'module.ssh3.conv3X3.1.weight', 'module.ssh3.conv5X5_1.0.weight', 
                                'module.ssh3.conv5X5_1.1.weight', 'module.ssh3.conv5X5_2.0.weight', 'module.ssh3.conv5X5_2.1.weight', 
                                'module.ssh3.conv7X7_2.0.weight', 'module.ssh3.conv7X7_2.1.weight', 'module.ssh3.conv7x7_3.0.weight', 
                                'module.ssh3.conv7x7_3.1.weight', 'module.ClassHead.0.conv1x1.weight', 'module.ClassHead.1.conv1x1.weight', 
                                'module.ClassHead.2.conv1x1.weight', 'module.BboxHead.0.conv1x1.weight', 'module.BboxHead.1.conv1x1.weight',
                                'module.BboxHead.2.conv1x1.weight', 'module.LandmarkHead.0.conv1x1.weight', 'module.LandmarkHead.1.conv1x1.weight', 
                                'module.LandmarkHead.2.conv1x1.weight']

    weights_agp= ['body.stage1.0.0.weight', 'body.stage1.0.1.weight', 'body.stage1.1.0.weight', 'body.stage1.1.1.weight',
          'body.stage1.1.3.weight', 'body.stage1.1.4.weight', 'body.stage1.2.0.weight', 'body.stage1.2.1.weight',
          'body.stage1.2.3.weight', 'body.stage1.2.4.weight', 'body.stage1.3.0.weight', 'body.stage1.3.1.weight',
          'body.stage1.3.3.weight', 'body.stage1.3.4.weight', 'body.stage1.4.0.weight', 'body.stage1.4.1.weight',
          'body.stage1.4.3.weight', 'body.stage1.4.4.weight', 'body.stage1.5.0.weight', 'body.stage1.5.1.weight',
          'body.stage1.5.3.weight', 'body.stage1.5.4.weight', 'body.stage2.0.0.weight', 'body.stage2.0.1.weight',
          'body.stage2.0.3.weight', 'body.stage2.0.4.weight', 'body.stage2.1.0.weight', 'body.stage2.1.1.weight',
          'body.stage2.1.3.weight', 'body.stage2.1.4.weight', 'body.stage2.2.0.weight', 'body.stage2.2.1.weight',
          'body.stage2.2.3.weight', 'body.stage2.2.4.weight', 'body.stage2.3.0.weight', 'body.stage2.3.1.weight',
          'body.stage2.3.3.weight', 'body.stage2.3.4.weight', 'body.stage2.4.0.weight', 'body.stage2.4.1.weight',
          'body.stage2.4.3.weight', 'body.stage2.4.4.weight', 'body.stage2.5.0.weight', 'body.stage2.5.1.weight',
          'body.stage2.5.3.weight', 'body.stage2.5.4.weight', 'body.stage3.0.0.weight', 'body.stage3.0.1.weight',
          'body.stage3.0.3.weight', 'body.stage3.0.4.weight', 'body.stage3.1.0.weight', 'body.stage3.1.1.weight',
          'body.stage3.1.3.weight', 'body.stage3.1.4.weight'] 
        #   ,'fpn.output1.0.weight', 'fpn.output1.1.weight', 'fpn.output2.0.weight', 'fpn.output2.1.weight',
        #   'fpn.output3.0.weight', 'fpn.output3.1.weight', 'fpn.merge1.0.weight', 'fpn.merge1.1.weight',
        #   'fpn.merge2.0.weight', 'fpn.merge2.1.weight', 
        #   'ssh1.conv3X3.0.weight', 'ssh1.conv3X3.1.weight', 'ssh1.conv5X5_1.0.weight', 'ssh1.conv5X5_1.1.weight',
        #   'ssh1.conv5X5_2.0.weight', 'ssh1.conv5X5_2.1.weight', 'ssh1.conv7X7_2.0.weight', 'ssh1.conv7X7_2.1.weight',
        #   'ssh1.conv7x7_3.0.weight', 'ssh1.conv7x7_3.1.weight', 'ssh2.conv3X3.0.weight', 'ssh2.conv3X3.1.weight',
        #   'ssh2.conv5X5_1.0.weight', 'ssh2.conv5X5_1.1.weight', 'ssh2.conv5X5_2.0.weight', 'ssh2.conv5X5_2.1.weight',
        #   'ssh2.conv7X7_2.0.weight', 'ssh2.conv7X7_2.1.weight', 'ssh2.conv7x7_3.0.weight', 'ssh2.conv7x7_3.1.weight',
        #   'ssh3.conv3X3.0.weight', 'ssh3.conv3X3.1.weight', 'ssh3.conv5X5_1.0.weight', 'ssh3.conv5X5_1.1.weight',
        #   'ssh3.conv5X5_2.0.weight', 'ssh3.conv5X5_2.1.weight', 'ssh3.conv7X7_2.0.weight', 'ssh3.conv7X7_2.1.weight',
        #   'ssh3.conv7x7_3.0.weight', 'ssh3.conv7x7_3.1.weight'#]

        #  ,'ClassHead.0.conv1x1.weight', 'ClassHead.1.conv1x1.weight', 'ClassHead.2.conv1x1.weight',
        #  'BboxHead.0.conv1x1.weight', 'BboxHead.1.conv1x1.weight', 'BboxHead.2.conv1x1.weight',
        #  'LandmarkHead.0.conv1x1.weight', 'LandmarkHead.1.conv1x1.weight','LandmarkHead.2.conv1x1.weight'] 

    agp = distiller.pruning.AutomatedGradualPruner('retinaface', 0.15, 0.5, weights_agp_resnet_no_bn)
    scheduler = distiller.scheduler.CompressionScheduler(net)
    scheduler.add_policy(agp, epochs=[max_epoch], starting_epoch=50, ending_epoch=max_epoch, frequency=1)
    compression_algorithm = agp.__class__.__name__

    interval = 20
    for i in range(max_epoch):
        scheduler.on_epoch_begin(i)
        net = net.to(device)
        train_one_epoch(net, data_loader, optimizer, criterion, cfg, gamma, i, step_index, scheduler, device)
        scheduler.on_epoch_end(i, optimizer)
        if i%interval==0:
            torch.save(net.state_dict(), save_folder + cfg['name'] + '_epoch_{}.pth'.format(i))
        
    fname = '_Final_{0}.pth'.format(compression_algorithm)
    torch.save(net.state_dict(), save_folder + cfg['name'] + fname)
    print('Pruning by {0} has been successfully done.'.format(compression_algorithm))
    

So I guess based on what is explained in the mentioned source code, I have done everything, yet I dont get any pruning! at first I thought maybe, my model is so small and compact that all the weights are equally important and when for example agp messes with one weight, it finds out that its important and thus, leave it be. therefore, the model doesnt shrink! However, when I tried resnet50! the very same thing happens, the model snapshot on disk takes 109MB of space! and I have set its pruning level to be 50%! yet, at epoch 80/100, its still 109MB! signaling no pruning has occured.

Am I doing something wrong here or is this simply normal, and mobilenet and resnet cant be pruned like this? Thanks a lot in advance

Hi, I'm learning to use the distiller. When I learn to use the quantization, it has the problem. In examples/classifier_compression/compress_classifier.py, I change the model to my small net and use the cifar dataset. my small net defined as below:

class Net(nn.Module): def init(self): super(Net, self).init() self.conv1 = nn.Conv2d(3, 6, 5) self.pool = nn.MaxPool2d(2, 2) self.conv2 = nn.Conv2d(6, 16, 5) self.fc1 = nn.Linear(16 * 5 * 5, 120) self.fc2 = nn.Linear(120, 84) self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

and change the yaml file as below: version: 1

quantizers: dorefa_quantizer: class: DorefaQuantizer bits_activations: 4 bits_weights: 4 bits_bias: 4 overrides: conv1: bits_weights: null bits_activations: null fc3: bits_weights: null bits_activations: null

lr_schedulers: pruning_lr: class: ExponentialLR gamma: 0.9

policies:

• quantizer: instance_name: 'dorefa_quantizer' starting_epoch: 1 ending_epoch: 30 frequency: 1

• lr_scheduler: instance_name: pruning_lr starting_epoch: 24 ending_epoch: 30 frequency: 1

other args is use the default or None. When I run the compress_classifier.py I get the error:

File "compress_classifier.py", line 786, in main() File "compress_classifier.py", line 292, in main msglogger.info(distiller.masks_sparsity_tbl_summary(model, compression_scheduler)) File "/usr/lib/python3.5/contextlib.py", line 77, in exit self.gen.throw(type, value, traceback) File "/home/zhang/distiller/distiller/data_loggers/collector.py", line 469, in collectors_context yield collectors_dict File "compress_classifier.py", line 287, in main loggers=[tflogger, pylogger], args=args) File "compress_classifier.py", line 399, in train optimizer.step() File "/usr/local/lib/python3.5/dist-packages/torch/optim/sgd.py", line 93, in step d_p.add_(weight_decay, p.data) RuntimeError: expected type torch.FloatTensor but got torch.cuda.FloatTensor

And I also check all the tensor in model. It's the torch.cuda.FloatTenor. How can I fix it?

Hi everyone

I saw in your documentation that you support post training quantization of LSTMs by replacing the PyTorch modules by your own implementation. Do you also support quantization aware training of LSTMs with this technique? Do you already have a tutorial for this?

Thanks a lot for your help!

I found model without DataParallel wrapping it will fail to prune. i.e. --load-serialized will disable pruning.

When I run

python compress_classifier.py -a=resnet20_cifar -p=50 ../../../data/cifar10/ -j=22 --epochs=1 --lr=0.001 --masks-sparsity --compress=../agp-pruning/resnet18.schedule_agp.yaml --load-serialized

The total sparsity will always be 0.00

Total sparsity: 0.00

But if I run the same command line without --load-serialized

python compress_classifier.py -a=resnet20_cifar -p=50 ../../../data/cifar10/ -j=22 --epochs=1 --lr=0.001 --masks-sparsity --compress=../agp-pruning/resnet18.schedule_agp.yaml

The total sparsity will be 1.53 after 1 epoch

Total sparsity: 1.53

Does anyone know how to apply my original dataset to distiller? It seems that Distiller only supports very basic dataset like MNIST, cifar10 and Imagenet.

I try to use Post Training Quantization to convert my float32 model to int8 follow the tutorial of quantizing GNMT. I change the model code to a distiller style and get a quantized model. This is some information about the quantized model:

(context_attn): MultiHeadedAttention(
          (linear_keys): RangeLinearQuantParamLayerWrapper(
            weights_quant_settings=(num_bits=8 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
            output_quant_settings=(num_bits=8 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
            accum_quant_settings=(num_bits=32 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
              inputs_quant_auto_fallback=True, forced_quant_settings_for_inputs=None
            scale_approx_mult_bits=None
            preset_activation_stats=True
              output_scale=16.876938, output_zero_point=0.000000
            weights_scale=194.250000, weights_zero_point=0.000000
            (wrapped_module): Linear(in_features=512, out_features=512, bias=True)
          )
          (linear_values): RangeLinearQuantParamLayerWrapper(
            weights_quant_settings=(num_bits=8 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
            output_quant_settings=(num_bits=8 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
            accum_quant_settings=(num_bits=32 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
              inputs_quant_auto_fallback=True, forced_quant_settings_for_inputs=None
            scale_approx_mult_bits=None
            preset_activation_stats=True
              output_scale=13.410025, output_zero_point=0.000000
            weights_scale=204.500000, weights_zero_point=0.000000
            (wrapped_module): Linear(in_features=512, out_features=512, bias=True)
          )
          (linear_query): RangeLinearQuantParamLayerWrapper(
            weights_quant_settings=(num_bits=8 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
            output_quant_settings=(num_bits=8 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
            accum_quant_settings=(num_bits=32 ; quant_mode=SYMMETRIC ; clip_mode=NONE ; clip_n_stds=None ; clip_half_range=False ; per_channel=False)
              inputs_quant_auto_fallback=True, forced_quant_settings_for_inputs=None
            scale_approx_mult_bits=None
            preset_activation_stats=True
              output_scale=15.167286, output_zero_point=0.000000
            weights_scale=200.000000, weights_zero_point=0.000000
            (wrapped_module): Linear(in_features=512, out_features=512, bias=True)
          )

My model is a transformer model from OpenNMT. It seems I do get a correct compressed model but the model file is larger than the original model, like 200MB -> 280 MB. Is there any method can reduce the model size as I think it's the most important feature from quantization. The code script is like this:

model_path= sys.argv[1]

parser = ArgumentParser(description='translate.py')
opts.config_opts(parser)
opts.translate_opts(parser)
src = "test.en"
output = src+".distiller_out"
opt = parser.parse_args(f"--model {model_path} --src {src} --output {output} --gpu 0")

#translate(opt)
ArgumentParser.validate_translate_opts(opt)
logger = init_logger(opt.log_file)

translator = build_translator(opt, report_score=True)
stats_file = "./acts_quantization_stats.yaml"

def evaluate(model, output, num_batches=None):
    src_shards = split_corpus(opt.src, opt.shard_size)
    tgt_shards = split_corpus(opt.tgt, opt.shard_size)
    shard_pairs = zip(src_shards, tgt_shards)

    for i, (src_shard, tgt_shard) in enumerate(shard_pairs):
        logger.info("Translating shard %d." % i)
        translator.translate(
            src=src_shard,
            tgt=tgt_shard,
            src_dir=opt.src_dir,
            batch_size=opt.batch_size,
            batch_type=opt.batch_type,
            attn_debug=opt.attn_debug,
            align_debug=opt.align_debug
            )

    print("translate end")

output = "output_file_distiller"

if not os.path.isfile(stats_file): # Collect stats.
    #model_copy = deepcopy(model)
    model_copy = translator.model
    distiller.utils.assign_layer_fq_names(model_copy)
    
    def eval_for_stats(model):
        evaluate(model, output + '.temp', num_batches=None)
    collect_quant_stats(model_copy, eval_for_stats, save_dir='.')
    #del model_copy
    torch.cuda.empty_cache()

quantizer = PostTrainLinearQuantizer(deepcopy(translator.model),
                                    mode="SYMMETRIC",  # As was suggested in GNMT's paper
                                    model_activation_stats=stats_file)
for t, rf in quantizer.replacement_factory.items():
    if rf is not None:
        print("Replacing '{}' modules using '{}' function".format(t.__name__, rf.__name__))

fake_input = torch.tensor([4,115,1480,73,12,4,18125,1424,234,26,12,3658,16278,36], dtype=torch.long)
fake_input = fake_input.unsqueeze(-1).unsqueeze(-2)
length  = torch.tensor([14])
fake_inputs = (fake_input, fake_input, length)
dummy_input = (torch.ones(1, 1, 2).to(dtype=torch.long),
                torch.ones(1, 1, 2).to(dtype=torch.long),
                torch.tensor([1]).to(dtype=torch.long),)
quantizer.prepare_model(fake_inputs)
print(quantizer.model)

I want to reduce the size of yolov3 model( format is keras(.h5) or darknet(.weight) ) using pruning method.

According to issue #62, I found the discription 「Sure, you can use Distiller with any network, not just the models that come from TorchVision. 」.

Therefore, can I reduce model size of keras(.h5) or darknet(.weight) without converting these models into pytorch model?? In addition, I would appreciate it if you could let me know how to prune yolov3 model ( keras(.h5) or darknet(.weight) )