The training process is interrupted by a segmentation fault during the very first epoch as part of the pretraining process. The error encountered is as follows:

[dynet] random seed: 1678755796 [dynet] allocating memory: 512MB [dynet] memory allocation done. [dynet] random seed: 2822143777 [dynet] using autobatching [dynet] allocating memory: 10000MB [dynet] memory allocation done. train_single-source.sh: line 65: 1310309 Segmentation fault (core dumped) python3 postcorrection/multisource_wrapper.py --dynet-mem $dynet_mem --dynet-autobatch 1 --pretrain_src1 $pretrain_src --pretrain_tgt $pretrain_tgt $params --single --vocab_folder $expt_folder/vocab --output_folder $expt_folder --model_name $pretrained_model_name --pretrain_only

I was able to narrow down the problem to the following piece of code: batch_loss.backward() in line 80: lm_trainer.py

The following issue may hint towards the potential problem: [(https://github.com/clab/dynet/issues/308)]

Has anyone encountered this problem before?

Hi, I am facing some issues while installing Dynet-GPU for CUDA 11.1. Can you please inform, whether you have used the CPU version of dynet or GPU ? If GPU, then can you inform the version of dynet and eigen that you have used?

System Specifications: Cuda Version - 11.1

I have tried the following versions.

Build Command: To avoid Unsupported GPU architecture for compute_30 during build time, the below command is used.

cmake .. -DEIGEN3_INCLUDE_DIR=../eigen -DPYTHON='which python' -DBACKEND=cuda -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.1

| Dynet | Eigen | Error | | ----------- | ----------- | ----------- | | Latest(master branch) | Eigen 3.2 | CX11 folder is not present| | Latest(master branch) | Eigen 3.3 | identifier std::round is undefined in device code| | Latest(master branch) | Eigen 3.3.7 | identifier std::round is undefined in device code| | Latest(master branch) | Eigen 3.4 | Error while running make| | Dynet 2.0.3 | Eigen-2355b22 | Unsupported GPU architecture for compute_30 (while running make) | | Dynet 2.1 | Eigen-b2e267dc99d4.zip | Unsupported GPU architecture for compute_30 (while running make) |

I have installed dynet with gpu compatibility as mentioned in the docs. Also the --dynet-mem is set in the train_single-source.sh file. Even then I got this error. Following is the Traceback of the entire error.

[dynet] Device Number: 2 [dynet] Device name: GeForce GTX 1080 Ti [dynet] Memory Clock Rate (KHz): 5505000 [dynet] Memory Bus Width (bits): 352 [dynet] Peak Memory Bandwidth (GB/s): 484.44 [dynet] Memory Free (GB): 11.5464/11.7215 [dynet] Device(s) selected: 2 [dynet] random seed: 2652333402 [dynet] using autobatching [dynet] allocating memory: 6000MB [dynet] memory allocation done. Param, load_model: None Traceback (most recent call last): File "/mnt/data/souvik/sanskrit/ocr-post-correction/postcorrection/multisource_wrapper.py", line 65, in pretrainer = PretrainHandler( File "/mnt/data/souvik/sanskrit/ocr-post-correction/postcorrection/pretrain_handler.py", line 81, in init self.pretrain_model(pretrain_src1, pretrain_src2, pretrain_tgt, epochs) File "/mnt/data/souvik/sanskrit/ocr-post-correction/postcorrection/pretrain_handler.py", line 88, in pretrain_model self.seq2seq_trainer.train( File "/mnt/data/souvik/sanskrit/ocr-post-correction/postcorrection/seq2seq_trainer.py", line 55, in train batch_loss.backward() File "_dynet.pyx", line 823, in _dynet.Expression.backward File "_dynet.pyx", line 842, in _dynet.Expression.backward ValueError: Dynet does not support both dynamic increasing of memory pool size, and automatic batching or memory checkpointing. If you want to use automatic batching or checkpointing, please pre-allocate enough memory using the --dynet-mem command line option (details http://dynet.readthedocs.io/en/latest/commandline.html).

Hi @shrutirij thanks for the great work and very well-documented and clean codebase, I greatly appreciate it!

I adapted and tried running this codebase for some conceptually similar experiments and I've observed a few quirks that I wanted to run by you to get your thoughts since I haven't really used Dynet before.

1. I ran the codebase with --dynet-gpus 8 (after also modifying opts.py to support this arg) and found that although 8 processes are spawned and attached to 8 GPUs, only the first process has > 0% GPU utilization. It appears that this codebase doesn't support distributed training in its current form. Is that accurate? Is there an equivalent to PyTorch's DistributedDataParallel and DistributedSampler that I can use to perform data-parallel training and inference with Dynet? It would greatly speed up my experiments.
2. It appears that the training time with a CPU is the same as the training time with GPU when using 1 GPU via provision of the --dynet-gpu flag. Is this what you noticed too during your runs? If not, could you suggest how I can get this to run faster with a GPU?
3. It appears that the Dynet auto-batching feature isn't working, because I tried running the code with and without the --dynet-autobatch 1 flag and the run-time doesn't seem to change. I see the main training loop looks like the following (where minibatch_size is always set to 1 here):

            for i in range(0, len(train_data), minibatch_size):
                cur_size = min(minibatch_size, len(train_data) - i)
                losses = []
                dy.renew_cg()
                for (src1, src2, tgt) in train_data[i : i + cur_size]:
                    losses.append(self.model.get_loss(src1, src2, tgt))
                batch_loss = dy.esum(losses)
                batch_loss.backward()
                trainer.update()
                epoch_loss += batch_loss.scalar_value()
            logging.info("Epoch loss: %0.4f" % (epoch_loss / len(train_data)))

Doesn't this mean that cur_size is always 1, causing the inner for loop to just iterate over a list of size 1 by default? If I were to override minibatch_size to, say, 32, how does Dynet ensure that 1 forward operation occurs per batch of 32 examples instead of 32 separate forward passes?

Thanks a lot for your time and thanks again for the great work toward protecting endangered languages!