I run your code MitoEM-R-A.yaml in the MitoEM challenge, but many H5 files appear in the inference time . How can I merge them? The H5 file list is as follow:

Hi!

While running the tutorial on Synaptic Cleft Segmentation (https://zudi-lin.github.io/pytorch_connectomics/build/html/tutorials/cremi.html), I encountered the following error:

`Traceback (most recent call last):
  File "pytorch_connectomics/scripts/main.py", line 72, in <module>
    main()
  File "pytorch_connectomics/scripts/main.py", line 65, in main
    trainer = Trainer(cfg, mode, args.checkpoint, device)
  File "/n/home11/kguliani/pytorch_connectomics/connectomics/engine/trainer.py", line 29, in _init_
    self.model = build_model(self.cfg, self.device)
  File "/n/home11/kguliani/pytorch_connectomics/connectomics/model/_init_.py", line 27, in build_model
    model = model.to(device)
  File "/n/home11/kguliani/.conda/envs/py3_torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 431, in to
    device, dtype, non_blocking, convert_to_format = torch._C._nn._parse_to(*args, **kwargs)
RuntimeError: Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu device type at start of device string: train`

Steps to reproduce the error in the shell-

`$ srun --pty -p cox -t 2-00:00 --mem 16000 -n 1 --gres=gpu:4 /bin/bash 
 $ module load cuda/9.2.88-fasrc01 cudnn/7.1.4-fasrc01
 $ module load cuda/9.2.88-fasrc01
 $ module load Anaconda/2019.10

 $ source activate py3_torch 

 $ PATH=/usr/local/cuda/bin:$PATH
 $ echo $PATH
 $ CPATH=/usr/local/cuda/include:$CPATH
 $ echo $CPATH

 $ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -u pytorch_connectomics/scripts/main.py --config-file pytorch_connectomics/configs/CREMI-Synaptic-Cleft.yaml`

Additionally,

`$ python -c 'import torch; print(torch.version.cuda)'

9.2

$ nvcc --version

9.2

@zudi-lin Have a look pls

Thanks!

Describe the bug Following the demo 'segmentation.ipynb', there will be keyerror 'seg' in paragraph 4.

Screenshots

System Specifications Desktop (please complete the following information):

• Operating system: Ubuntu 18.04LTS
• CUDA version: 10.0
• python version: 3.6
• pytorch version: 1.1.0
• Anything else that seems relevant: zwatershed version:1e8528c commit

In configs/NucMM/NucMM-Mouse-Base.yaml, MODEL.INPUT_SIZE is [33, 97, 97] and INFERENCE.STRIDE is [26, 128, 128] . Doesn't this gap between the model input size and inference stride lead to some pixels uncovered during the test stage?

Hi,

when I load the MitoEM dataset, I meet an error "IndexError: index 1 is out of bounds for axis 2 with size 1".

data_io.py

def vast2Seg(seg):
    # convert to 24 bits
    if seg.ndim==2:
        return seg
    else: #vast: rgb
        return seg[:,:,0].astype(np.uint32)*65536+seg[:,:,1].astype(np.uint32)*256+seg[:,:,2].astype(np.uint32)  # error!!

I find the shape of "seg" is (2859, 2859, 1). Because the labels of the MitoEM dataset are grey image. It seems that you haven't considered the case where the label is a grayscale image?

The following link for the documentation of data augmentation does not work: https://zudi-lin.github.io/pytorch_connectomics/build/html/modules/augmentation.html

Hi,

Thank you for your open-source awesome work!

I meet a problem of GPU underutilization. I can run the code successfully. But when I use 4 Titan XP, only 2 are used. When I use 8 GPUs, only 3 are in use.

| ID | Name | Serial | UUID || GPU temp. | GPU util. | Memory util. || Memory total | Memory used | Memory free || Display mode | Display active |

| 0 | TITAN Xp | 0321118041854 | GPU-e47e3aa6-63e6-cccc-9575-740c0932425a || 60C | 0% | 52% || 12196MB | 6306MB | 5890MB || Disabled | Disabled | | 1 | TITAN Xp | 0321118043078 | GPU-99bacaab-e6a7-68a4-8f91-631df2578104 || 74C | 0% | 49% || 12196MB | 6024MB | 6172MB || Disabled | Disabled | | 2 | TITAN Xp | 0321118040179 | GPU-73abffcb-a391-bf5d-3095-0271e9919f8c || 70C | 0% | 49% || 12196MB | 6024MB | 6172MB || Disabled | Disabled | | 3 | TITAN Xp | 0321118042097 | GPU-ce8fed4b-4882-01e6-73f8-03019d2d1b5e || 24C | 0% | 0% || 12196MB | 11MB | 12185MB || Disabled | Disabled | | 4 | TITAN Xp | 0321118040143 | GPU-f9d9e962-3254-456d-47a8-c5da5f13551c || 29C | 0% | 0% || 12196MB | 11MB | 12185MB || Disabled | Disabled | | 5 | TITAN Xp | 0321118042010 | GPU-4130bd87-b82b-f901-6953-d925dc5fc039 || 30C | 0% | 0% || 12196MB | 11MB | 12185MB || Disabled | Disabled | | 6 | TITAN Xp | 0321118040854 | GPU-314d4746-9237-14e0-071b-cc2ee9c3dac6 || 27C | 0% | 0% || 12196MB | 11MB | 12185MB || Disabled | Disabled | | 7 | TITAN Xp | 0321118042171 | GPU-d9d858b8-721f-99a0-3da5-90b52bb2d78b || 27C | 0% | 0% || 12196MB | 11MB | 12185MB || Disabled | Disabled |

System

• ubuntu
• pytorch 1.1
• cuda 9.0

(Due to hardware limitations, I can only use this version.) Could you help me?

hi，

thank you for the impressive work！I am concerned about the format that should be followed in the json file

Could you give an official sample of it？

After 49 iterations, the model always stops training and runs into this error. I am training without CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

Traceback (most recent call last):
  File "pytorch_connectomics/scripts/main.py", line 67, in <module>
    main()
  File "pytorch_connectomics/scripts/main.py", line 62, in main
    trainer.train()
  File "/n/home00/nwendt/zebrafish/pytorch_connectomics/connectomics/engine/trainer.py", line 92, in train
    GPUtil.showUtilization(all=True)
  File "/n/home00/nwendt/anaconda3/envs/py3_torch/lib/python3.7/site-packages/GPUtil/GPUtil.py", line 210, in showUtilization
    GPUs = getGPUs()
  File "/n/home00/nwendt/anaconda3/envs/py3_torch/lib/python3.7/site-packages/GPUtil/GPUtil.py", line 102, in getGPUs
    deviceIds = int(vals[i])
ValueError: invalid literal for int() with base 10: 'No devices were found'

For inference, current logic load all volumes first and then run inference for each volume.

To save memory, it's better to load the volume when it's needed

Need to extend test_augmentor to also support 2D models.

• data/augmentation/test_augmentor.py: add an attribute for the TestAugmentor object for 2D or 3D model
• engine/trainer.py: specify whether it's a 2D or 3D model

Hi, Is it possible to add the evaluation code for Neuron Segmentation tutorial, and also to provide the pre-trained weights for the U-Net for Neuron segmentation please?

In file connectomics/data/utils/data_segmentation.py:

• Added missing bracket

• Added type casting to int62

Indexing requires int or boolean. But the mask used for indexing where of format float without decimal ( like 124432.) since originally being of type float32.

Steps to reproduce

Use VolumeDataset without specifying a list of labels.

In line 48, the label list is initialized with None.

In line 88 with set self.label_vol_ratio = self.sample_label_size / self.sample_volume_size if self.label is not None.

However, in line 232 self.label_vol_ratio is referenced even if the label list was not initialized and consequently self.label_vol_ratio was never defined.

Current behavior (bug)

Raises AttributeError

Expected behavior (correct)

Should be able to process a data volume without providing a list of labels since label has a default value of None.

/label ~Bug

Thank you very much for your contributions! :)

I'm implementing MALA's network in this pipeline. It saves memory by using convolution without padding, therefore can afford a larger input size during training (for example [64, 268, 268] with batch size 4 on a single GPU).

However, the data loading time became unaffordable under this input size, where 90% of the time is spent on data-loading. I found that this is caused by SMOOTH, the post-process of the label after augmentation.

I wonder if you are aware of this? Will discarding smooth influence training much?

Merry Christmas :)

Hello, I'm interested in the pytorch_connectomics. So I try to learn it, however, I encounter a problem. In the notebook, the data you gave is some pictures in png format, but the code seems to require the .h5 format, which makes me fail when I try to run the code. Could you do me a favor? Thank you!