The changes recommended in https://github.com/apache/singa-doc/pull/14 to simply the APIs of the examples I have reduced the number of arguments in the train_mnist_cnn

Seems that adding nccl and mpich is okay in conda build of singa, but need to check further and add other thing such as python "deprecated"

======================================================================
ERROR: test_tensor (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_tensor
Traceback (most recent call last):
  File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_tensor.py", line 24, in <module>
    from singa import tensor
  File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
    from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'


======================================================================
FAIL: test_conv2d (test_mkldnn.TestPythonOperation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_mkldnn.py", line 78, in test_conv2d
    self.assertAlmostEqual(4.0, _dW[0], places=5)
AssertionError: 4.0 != 0.040000003 within 5 places

----------------------------------------------------------------------
Ran 13 tests in 0.003s

FAILED (failures=1, errors=12)
TEST CONV2D FORWARD
TEST CONV2D DATA BACKWARD
TEST CONV2D WEIGHT BACKWARD
+ exit 0

Resource usage statistics from testing singa:
   Process count: 1
   CPU time: unavailable
   Memory: 3.0M
   Disk usage: 656B
   Time elapsed: 0:00:02.1

TEST END: /root/miniconda/conda-bld/linux-64/singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36.tar.bz2
Renaming work directory,  /root/miniconda/conda-bld/singa_1583767754024/work  to  /root/miniconda/conda-bld/singa_1583767754024/work_moved_singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36_linux-64_main_build_loop
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /root/miniconda/conda-bld/linux-64/singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []
####################################################################################
Resource usage summary:

Total time: 0:07:23.3
CPU usage: sys=0:00:06.5, user=0:02:17.0
Maximum memory usage observed: 777.8M
Total disk usage observed (not including envs): 252.4K


####################################################################################
Source and build intermediates have been left in /root/miniconda/conda-bld.
There are currently 4 accumulated.
To remove them, you can run the ```conda build purge``` command
root@3c17fd6cb72e:~/dcsysh/singa/tool/conda/singa#

After merging the PR #590, today I am trying to solve the problem that the mnist cnn hangs as reported in PR #589. When I run mnist_cnn after changing to cpu, it hangs. So the training has some problem.

The Layer class in Autograd is to maintain the model parameters. It passes the parameters into the operation and thus operations are stateless.

Typically the parameter size depends on the input and layer configuration. Currently, we require the users to provide the input size in the layer constructor. Then we can create the parameter tensor and initialize it in the constructor, e.g., Linear layer. One potential problem is that the initialization operation may not be buffered. @XJDKC Is it an issue? For some layers like RNN implemented using cudnn, although we can get the input size, the parameter size is unknown until the cudnn handle is created, which is done until the data is forwarded through the layer.

Another way is to delay the parameter tensor creation until the layer is called for forward propagation. At that time, we have the input tensor (and its device). Then in the layer constructor, we do not need the user to provide the input size. The drawback is that after the layer is created, the get_params() function would still fail to get the parameter tensors as they are not created yet. @dcslin To switch to this approach, we need to change the constructors of existing layer classes and examples. We also need to provide an initializer function/class into the constructor for initializing the parameter tensors after they are created.

Please add your comments.

Updated on May 15

class Layer:
   def get_params(self):
       """the params of this layer and sublayers as a dict;  param name is: layername.param
           e.g., self.W = Tensor(), self.b=Tensor()
                  name of W and b is  like conv1.W and conv1.b  
       """

    def get_states(self):
       """states of this layer as sublayers that are necessary for model evaluation/inference.
           the states include the params and others, e.g., the running mean and var of batchnorm.
      """

class Module(Layer):   
  def compile(self ...):
     """set the name of each layer and sublayers, which will be used to create the dict 
          for get_params and get_states. Then no need to manually config the layer name 
         the __init__ method of a layer.
 
        For instance,
        class Blk(Layer):
             def __init__(self):
                  self.conv1= Conv2d()
                  self.conv2 = Conv2d()

        class MyModel(Module):
              def __init__(self):         
                 self.blk1 = Blk() --> blk1.conv1, blk1.conv2
                 self.blk2 = Blk()  --> blk2.conv1, blk2.conv2
   """

  # high priority
  def save(self, fpath, ckp_states={}):
      """Save the model and optionally some states.
      
      Args:
         fpath: output file path (without the extension)
         ckp_states(dict): states for checkpoint that are not attributes of Module, e.g., epoch ID.
      """
      cust_states = {}
      if ckp_states is not None:
         cust_states = ckp_states + model (include sublayers) attributes - get_states()
      save model states via onnx with customized field for the cust_states

  def load(self, fpath, dev, use_graph, graph_alg):
      """Load the model onto dev
   
       Args:
         path: input file path (without the extension)

       Returns:
          dict for the ckp_states.
      ```
      load model states + cust_states
      model attributes = model states + attributes from cust_states
      self.compile()
      restore the model attributes
      return the rest states as a dict

# lower priority
def save(fpath, model, ckp_states):
    attributes <-- model
    replace all tensors in attributes + ckp_states into dict name -->(shape, dtype)
    dump the tensors via numpy.savez_compressed
    dump model via pickle

def load(fpath, dev, use_graph, graph_alg):
     load model via pickle
     load tensors via numpy.load
     restore the tensors 
     return the ckp_states

Clarification:

• Params: layer parameters (Tensor) that are updated via SGD. Layer.get_params()
• States: Params + other variables that are necessary for model evaluation/inference. Superset of params. Layer.get_states()
• Attributes: members of a class instance class.__dict__. Superset of states.

todo-list:

1. sonnx to suit new API(almost done),
2. sonnx to use new autograd API,
3. test cases,
4. debug test operations,
5. debug sonnx backend test cases,
6. examples
7. add re-training examples
8. (if time is enough) add more example(will move to another PR)

#688 is refactoring the autograd module. Here are some comments about the current APIs in autograd.

1. Relationship between the classes and functions in autograd. Operator implements the forward and backward method for autograd. For each Operator class, there is a function that creates an Operator instance and calls the forward method. Layer stores the states (handles and parameters) and calls Operator function for the real computation. Note that a layer class can have sub-layers (as states) for creating complex and deep models. Issue: When we create a network using the Module API, there are both stateless (e.g., flatten) and statefull (conv2d) operations. Currently, we create layers in __init__ of Module and calls the layers and operator function in forward method. Therefore, Layer and Operator are mixed, which may confuse the users. A better way is to use Layer instances only. For every operator, we create a corresponding layer class to replace the layer function.

2. Layer API. issue: when and how to initialize the parameters and (handle) of a layer?

• do initialization in __init__ method OR when the data is forwarded for the first time #674
• pass an initializer function to the __init__ method of a layer and use it to initialize the parameters OR pass an initializer function to the __init__ method of the Module class and use it to initialize the parameter (through get_params) of the layers after forwarding the layers for once. The second approach requires the Module class's __init__ to do a forward pass of all layers and then get_params of each layer for initialization. To do that, it needs at least the shapes of the input tensors and the device. The drawback of the first approach is that we need to include the initializer in every Layer constructor.

comments are welcomed.

Overview

This PR adds the computational graph with memory optimization. It is based on the code developed by @chrishkchris and @XJDKC and some discussions with @nudles.

Features

There are four main features in this PR, namely the construction of the computational graph, lazy allocation, automatic recycling, and synchronization pipeline. Details as follows:

• Computational graph construction: Construct a computational graph based on the user-defined neural network or expressions and then run the graph to accomplish the training task. The computational graph also includes operations like synch and fused synch in the communicator.
• Lazy allocation: When blocks need to be allocated, devices won't allocate memory for them immediately. Only when an operation uses this block for the first time, memory allocation will be performed.
• Automatic recycling: Automatically deallocate the intermediate tensors which won't be used again in the following operations when we are running the graph in an iteration.
• Synchronization pipeline: In previous synchronization operations, buffers were used to synchronize multiple tensors at once. But the communicator needs to collect all the tensors before copying them into the buffer. Synchronization pipeline can copy tensors to the buffer separately, which reduces the time for synchronous operations.

Design

1. Computational graph construction
   • Use the technique of delayed execution to falsely perform operations in the forward propagation and backward propagation once. Buffer all the operations and the tensors read or written by each operation.
   • Calculate dependencies between all the operations to decide the order of execution. (Support directed cyclic graph)
   • Execute all the operations in the order we just calculated to update all the parameters.
   • The system will only analyze the same graph once. If new operations are added to the graph, the calculation graph will be re-analyzed.
   • Provided a module class for users to use this feature more conveniently.
2. Lazy allocation
   • When a device needs to create a new block, just pass the device to that block instead of allocating a piece of memory from the mempool and passing the pointer to that block.
   • When the block is accessed for the first time, let the device corresponding to the block allocate memory and then access it.
3. Automatic recycling
   • When calculating dependencies between the operations during the graph construction, the reference count of tensors can also be calculated.
   • When an operation is completed, we can decrease the reference count of tensors the operation used.
   • If a tensor's reference count reaches zero, it means the tensor won't be accessed by latter operations and we can recycle its memory.
   • The program will track the usage of the block. If a block is used on the python side, it will not be recycled, which is convenient for debugging on the python side.
4. Synchronization pipeline
   • If a tensor needs fusion synchronization, it will be copied to the buffer immediately and don't need to gather all the tensors. Because the copy operation is advanced, it takes less time to do real synchronization. This optimizes the use of the GPU.

Changes

• Tensor&Operation&Communicator
  • Change the capture type of tensors in lambda expressions to achieve delayed execution.
  • Change the type of input and output parameters to ensure that the input and output of the operation are tensors.
  • Submit synchronous operations to the device for execution.
• Device: Add code for
  • Buffering operations
  • RunGraph
• Scheduler
  • Add operations in the graph to construct the graph.
  • Analyse the graph, calculating dependencies and deciding the reallocation of the blocks.
  • Execute the operations in the graph.
• Block: Add a member variable of type device to help to do the lazy allocation. Add a function to help to do automatic recycling.
• Communicator and opt.py: Support the synchronization pipeline.
• Swig: add some interfaces
• Module: Provide a Module class on the python side for users to use the graph more conveniently.
• Examples: Add some examples with operations buffering by using Module class.

Evaluation

Single node

• Experiment settings
  • Modle
    • dev branch: ResNet50 in resnet.py
    • this pr: ResNet50 in resnet_module.py
  • GPU: Nvidia RTX 2080Ti
• Explanation
  • s ：second
  • it ： iteration
  • Memory：peak memory usage of single GPU
  • Throughout：number of pictures processed per second
  • Time：total time
  • Speed：iterations per second
  • Reduction：the memory usage reduction rate compared with dev branch
  • Seepdup: speedup ratio compared with dev branch
• Result:

Multi processes

• Experiment settings
  • Model
    • dev branch: ResNet50 in resnet_dist.py
    • this pr: ResNet50 in resnet_module.py
  • GPU: Nvidia RTX 2080Ti * 2
  • MPI: two MPI processes on one node
• Explanation: the same as above
• Result:

From the table above, we can know:

• This PR does not affect training time and memory usage if the graph is disabled (has backward compatibility).
• This PR can significantly reduce memory usage and training time by using the graph.

How to use

• An example of CNN:

class CNN(module.Module):

    def __init__(self, optimizer):
        super(CNN, self).__init__()

        self.conv1 = autograd.Conv2d(1, 20, 5, padding=0)
        self.conv2 = autograd.Conv2d(20, 50, 5, padding=0)
        self.linear1 = autograd.Linear(4 * 4 * 50, 500)
        self.linear2 = autograd.Linear(500, 10)
        self.pooling1 = autograd.MaxPool2d(2, 2, padding=0)
        self.pooling2 = autograd.MaxPool2d(2, 2, padding=0)

        self.optimizer = optimizer

    def forward(self, x):
        y = self.conv1(x)
        y = autograd.relu(y)
        y = self.pooling1(y)
        y = self.conv2(y)
        y = autograd.relu(y)
        y = self.pooling2(y)
        y = autograd.flatten(y)
        y = self.linear1(y)
        y = autograd.relu(y)
        y = self.linear2(y)
        return y

    def loss(self, x, ty):
        return autograd.softmax_cross_entropy(x, ty)

    def optim(self, loss):
        self.optimizer.backward_and_update(loss)

# initialization other objects
# ......
model = CNN(sgd)

# Train
for b in range(num_train_batch):
    # Generate the patch data in this iteration
    # ......

    # Copy the patch data into input tensors
    tx.copy_from_numpy(x)
    ty.copy_from_numpy(y)

    # Train the model
    out = model(tx)
    loss = model.loss(out, ty)
    model.optim(loss)

• More examples:
  • MLP
  • CNN
  • ResNet

Plan

• [ ] Computation graph optimization: replace a subgraph of the input computation graph with another subgraph which is functionally equivalent to the original one.
• [ ] Support recalculation and swapping out variables from the GPU.
• [ ] Perform operations in the graph in the order of DFS.
• [ ] Performing operations in parallel.

AssertionError with the onnx testcase: https://github.com/apache/singa/blob/master/examples/onnx/training/train.py

$ cd examples/onnx
$ python3 training/train.py --model vgg16

Then I get the following error msg:

File "training/train.py", line 437, in <module>
    args.onnx_model_path, args.data, sgd, args.graph, args.verbosity)
  File "training/train.py", line 295, in run
    model.compile([tx], is_train=True, use_graph=graph, sequential=sequential)
  File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/model.py", line 177, in compile
    self.forward(*inputs)
  File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py", line 63, in wrapper
    return func(self, *args, **kwargs)
  File "training/train.py", line 191, in forward
    y = self.linear(y)
  File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py", line 110, in __call__
    return self.forward(*args, **kwargs)
  File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py", line 61, in wrapper
    self.initialize(*args, **kwargs)
  File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py", line 45, in wrapper
    'initialize function expects PlaceHolders or Tensors')
AssertionError: initialize function expects PlaceHolders or Tensors

Something maybe wrong with the layer initialization?

singa version: 3100(the latest build from the source code of master branch) Python version: 3.5.2 ONNX version: 1.5.0

I used the clang-formatter with VS-code after I alter the tensor.h file, it results in different format with the dev branch.

The tensor.cc should have re-formatted before in PR #581. So, did I use incorrect setting in clang-formatter?

In the log of travis CI CPU version build, it displays the error in test_onnx that cannot import libprotobuf.so.20 https://travis-ci.org/github/apache/singa/jobs/664251025#L3998

======================================================================
3966
ERROR: test_onnx (unittest.loader._FailedTest)
3967
----------------------------------------------------------------------
3968
ImportError: Failed to import test module: test_onnx
3969
Traceback (most recent call last):
3970
  File "/home/travis/conda-bld-1971.5/singa_1584596418932/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
3971
    module = self._get_module_from_name(name)
3972
  File "/home/travis/conda-bld-1971.5/singa_1584596418932/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
3973
    __import__(name)
3974
  File "/home/travis/conda-bld-1971.5/singa_1584596418932/test_tmp/test/python/test_onnx.py", line 24, in <module>
3975
    from singa import sonnx
3976
  File "/home/travis/conda-bld-1971.5/singa_1584596418932/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages/singa/sonnx.py", line 23, in <module>
3977
    import onnx.utils
3978
  File "/home/travis/conda-bld-1971.5/singa_1584596418932/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages/onnx/__init__.py", line 8, in <module>
3979
    from .onnx_cpp2py_export import ONNX_ML
3980
ImportError: libprotobuf.so.20: cannot open shared object file: No such file or directory

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Need to print intermediate information for cnn and cifar_distributed_cnn examples since it may take quite long for large models to finish one epoch of training

Hi apache/singa!

This is not an automatic, 🤖-generated PR, as you can check in my GitHub profile, I work for GitHub and I am part of the GitHub Security Lab which is helping out with the migration of LGTM configurations to Code Scanning. You might have heard that we've integrated LGTM's underlying CodeQL analysis engine natively into GitHub. The result is GitHub code scanning!

With LGTM fully integrated into code scanning, we are focused on improving CodeQL within the native GitHub code scanning experience. In order to take advantage of current and future improvements to our analysis capabilities, we suggest you enable code scanning on your repository. Please take a look at our blog post for more information.

This pull request enables code scanning by adding an auto-generated codeql.yml workflow file for GitHub Actions to your repository — take a look! We tested it before opening this pull request, so all should be working :heavy_check_mark:. In fact, you might already have seen some alerts appear on this pull request!

Where needed and if possible, we’ve adjusted the configuration to the needs of your particular repository. But of course, you should feel free to tweak it further! Check this page for detailed documentation.

Questions? Check out the FAQ below!

FAQ

Bumps protobuf-java from 2.6.1 to 3.16.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This issue is open to discuss different options for adding GPU build and test to Github workflows.

To enable this feature, SINGA must provide a real or virtual machine with GPU as host machine for running the workflow. Then use the self-hosted runner feature of Github Actions. See also this MLOps video tutorial.

The team need to take some decisions:

1. Which machine(s) should we use? (e.g. virtual machine(s) on AWS, dedicated server(s) at NUS, ..)
2. Which operating systems we test on? (Only Linux? or also Mac).
3. When we run the GPU build and test workflow? (with every pull request? once per day at night? once per week? only on master branch?, ...)
4. Should we keep the machines always on? or use them only when the scheduled test is running and shut down them when there is no workflow runs? Assuming we run the GPU build and test only at scheduled time.
5. Should we add workflows to run examples and test the Jupyter notebooks? note that some examples may take hours or days to complete the training. But automating the test of examples can be very useful to speed up the development.

What do you think?