Thanks for your great project, and I need to use jit.trace convert model to libtorch(C++) model ,code as below

model.to(device) model.cuda() model.eval() example = torch.rand(1, 3, INPUT_HEIGHT,INPUT_WIDTH).cuda() traced_script_module = torch.jit.trace(model, example) traced_script_module.save("model_for_libtorch.pt")

RuntimeError: Could not export Python function call 'GradientMultiplyLayer'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to constants:

First, thanks for your public nice code!

I just wonder whether the AP metric in paper is about classification rather than semantic segmentation. And in your "end2end.py " L246, it seems only evaluate for "predictions" which is result of classification.

By the way, I think "weakly supervision" is not precise in your mix supervision setting, because if your final evaluation is only about classification, class-level, pixel-level labels will all be full-supervised label for classification.

hello, the number of the positive data is 246, all same in five different splits, is there anything wrong? I think that the number in the name of the split represents the number of the positive data, do i make something wrong?

Hi , can you please tell me as to what parameters to use to train the model to get the AP 99.99% (as mentioned in paper)? I tried to train using below parameters on STEEL dataset but achieved AP 75% only. Also could you please tell me how is the class information given to the model for this dataset?

python -u train_net.py  \
        --GPU=0 \
        --DATASET=STEEL \
        --RUN_NAME=steel_server_1 \
        --DATASET_PATH=datasets/STEEL \
        --RESULTS_PATH=datasets/steel_result_1 \
        --SAVE_IMAGES=True \
        --DILATE=7 \
        --EPOCHS=50 \
        --VALIDATION_N_EPOCHS=5 \
        --LEARNING_RATE=0.1 \
        --DELTA_CLS_LOSS=0.1 \
        --BATCH_SIZE=500 \
        --WEIGHTED_SEG_LOSS=True \
        --WEIGHTED_SEG_LOSS_P=2 \
        --WEIGHTED_SEG_LOSS_MAX=1 \
        --DYN_BALANCED_LOSS=True \
        --GRADIENT_ADJUSTMENT=True \
        --FREQUENCY_SAMPLING=True \
        --TRAIN_NUM=3000 \
        --NUM_SEGMENTED=3000 \
        --FOLD=0

Thanks

Thank you so much for sharing the scripts.

In line 129 of "End2End.py", "training_iteration()" returns "total_loss_seg + total_loss_dec" instead of "total_loss". Is it supposed to mean something?

end2end.py, line113: total_correct += (decision > 0.5).item() == is_pos_.item() decision is the output of model(fc layer)，it‘s not a probability, Is function sigmoid() missed?

Result

Code

import os
import cv2
import numpy as np
from scipy.ndimage.morphology import distance_transform_edt
import matplotlib.pyplot as plt

def distance_transform(mask: np.ndarray, max_val: float, p: float) -> np.ndarray:
	dst_trf = distance_transform_edt(mask)
	if dst_trf.max() > 0:
		dst_trf = (dst_trf / dst_trf.max())
		dst_trf = (dst_trf ** p) * max_val
	dst_trf[mask == 0] = 1
	return np.array(dst_trf, dtype=np.float32)

def distance_transform_new(mask, max_val, p): 
    h, w = mask.shape[:2]
    dst_trf = np.zeros((h, w))
    num_labels, labels = cv2.connectedComponents(mask, connectivity=8)
    for idx in range(1, num_labels):
        mask_roi= np.zeros((h, w))
        k = labels == idx
        mask_roi[k] = 255
        dst_trf_roi = distance_transform_edt(mask_roi)
        if dst_trf_roi.max() > 0:
            dst_trf_roi = (dst_trf_roi / dst_trf_roi.max())
            dst_trf_roi = (dst_trf_roi ** p) * max_val
        dst_trf += dst_trf_roi

    dst_trf[mask == 0] = 1
    return np.array(dst_trf, dtype=np.float32)

image_name = './KSDD2/train/20922.png'
mask_name = './KSDD2/train/20922_GT.png'

img = cv2.imread(image_name)
img = cv2.resize(img, dsize=(224, 600))

lbl = cv2.imread(mask_name, cv2.IMREAD_GRAYSCALE)
lbl = cv2.resize(lbl, dsize=(224, 600))

dilate_lbl = cv2.dilate(lbl, np.ones((7, 7)))

distance_transform_lbl = distance_transform(dilate_lbl, max_val=3.0, p=2)
distance_transform_lbl_new = distance_transform_new(dilate_lbl, max_val=3.0, p=2)

plt.figure(figsize=(10, 6))
plt.suptitle("KSDD2 train 20922.png and 20922_GT.png")
plt.subplot(151)
plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
plt.title("img")
plt.subplot(152)
plt.imshow(lbl, cmap='gray')
plt.title("lbl")
plt.subplot(153)
plt.imshow(dilate_lbl, cmap='gray')
plt.title("dilate7_lbl")
plt.subplot(154)
plt.imshow(distance_transform_lbl, cmap='gray')
plt.title("DT_lbl")
plt.subplot(155)
plt.imshow(distance_transform_lbl_new, cmap='gray')
plt.title("DT_lbl_new")
plt.tight_layout()
plt.show()

# if args.ON_DEMAND_READ is not None: self.TRAIN_NUM = args.ON_DEMAND_READ if args.ON_DEMAND_READ is not None: self.ON_DEMAND_READ = args.ON_DEMAND_READ # if args.REPRODUCIBLE_RUN is not None: self.TRAIN_NUM= args.REPRODUCIBLE_RUN if args.REPRODUCIBLE_RUN is not None: self.REPRODUCIBLE_RUN = args.REPRODUCIBLE_RUN # if args.MEMORY_FIT is not None: self.TRAIN_NUM = args.MEMORY_FIT if args.MEMORY_FIT is not None: self.MEMORY_FIT = args.MEMORY_FIT

KSDD datasets are binary classification. If every picture in my datasets is an object with two or more defects, how can I modify the algorithm to be multi-classified?I'd appreciate it if you could help me！Thanks！

Thansk for your great work!

I was just wondering if it is possible to get image resulst from running demo_eval_single_image.py It seems that only score is availale as a result. An input image with a localization of defects is needed.

Problems when using my own dataset. How can I annotation multi-tagged images?Is it possible to use pseudo-color annotation like paddle?https://github.com/PaddlePaddle/PaddleSeg/blob/release/2.5/docs/data/marker/marker.md

Directory structure

COCO

--IMAGE

--LABEL

D:\mixed-segdec-net-comind2021-master\mixed-segdec-net-comind2021-master>python -u train_net.py  --GPU=1 --DATASET=COCO --RUN_NAME=COCO --DATASET_PATH=datasets/COCO --RESULTS_PATH=save/results --SAVE_IMAGES=True --DILATE=7 --EPOCHS=50 --LEARNING_RATE=1.0 --DELTA_CLS_LOSS=0.01 --BATCH_SIZE=1 --WEIGHTED_SEG_LOSS=True --WEIGHTED_SEG_LOSS_P=2 --WEIGHTED_SEG_LOSS_MAX=1 --DYN_BALANCED_LOSS=True --GRADIENT_ADJUSTMENT=True --FREQUENCY_SAMPLING=True  --NUM_SEGMENTED=0 --FOLD=2
COCO Executing run with path save/results\COCO\COCO
COCO BATCH_SIZE                : 1
COCO DATASET                   : COCO
COCO DATASET_PATH              : datasets/COCO
COCO DELTA_CLS_LOSS            : 0.01
COCO DILATE                    : 7
COCO DYN_BALANCED_LOSS         : True
COCO EPOCHS                    : 50
COCO FOLD                      : 2
COCO FREQUENCY_SAMPLING        : True
COCO GPU                       : 1
COCO GRADIENT_ADJUSTMENT       : True
COCO INPUT_CHANNELS            : 1
COCO INPUT_HEIGHT              : 400
COCO INPUT_WIDTH               : 2448
COCO LEARNING_RATE             : 1.0
COCO MEMORY_FIT                : 1
COCO NUM_SEGMENTED             : 0
COCO ON_DEMAND_READ            : False
COCO REPRODUCIBLE_RUN          : False
COCO RESULTS_PATH              : save/results
COCO SAVE_IMAGES               : True
COCO TRAIN_NUM                 : None
COCO USE_BEST_MODEL            : False
COCO VALIDATE                  : True
COCO VALIDATE_ON_TEST          : True
COCO VALIDATION_N_EPOCHS       : 5
COCO WEIGHTED_SEG_LOSS         : True
COCO WEIGHTED_SEG_LOSS_MAX     : 1.0
COCO WEIGHTED_SEG_LOSS_P       : 2.0
datasets/COCO
268
264
536
datasets/COCO
268
264
532
536
536
COCO Saving current model state to save/results\COCO\COCO\models\ep_00.pth
COCO Returning seg_loss_weight 1.0 and dec_loss_weight 0.0
COCO Returning dec_gradient_multiplier 0
Traceback (most recent call last):
  File "train_net.py", line 63, in <module>
    end2end.train()
  File "D:\mixed-segdec-net-comind2021-master\mixed-segdec-net-comind2021-master\end2end.py", line 60, in train
    train_results = self._train_model(device, model, train_loader, loss_seg, loss_dec, optimizer, validation_loader, tensorboard_writer)
  File "D:\mixed-segdec-net-comind2021-master\mixed-segdec-net-comind2021-master\end2end.py", line 160, in _train_model
    for iter_index, (data) in enumerate(train_loader):
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 517, in __next__
    data = self._next_data()
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "D:\mixed-segdec-net-comind2021-master\mixed-segdec-net-comind2021-master\data\dataset.py", line 49, in __getitem__
    ix = self.neg_imgs_permutation[ix]
IndexError: index 266 is out of bounds for axis 0 with size 264

input_coco.py

class COCODataset(Dataset):
    def __init__(self, kind: str, cfg):
        super(COCODataset, self).__init__(cfg.DATASET_PATH, cfg, kind)
        self.read_contents()

    def read_contents(self):
        pos_samples, neg_samples = [], []
        print(self.path)
        img_num = 0
        for image_name in glob.glob(self.path + '/IMAGE/*.jpg'):
            img_num += 1
            image_path = image_name
            image = self.read_img_resize(image_path, self.grayscale, self.image_size)
            img_name_short = image_name[:-4]
            jpg_name = img_name_short.split('\\')[-1]
            seg_mask_path = os.path.join(self.path, "LABEL", f"{jpg_name}.jpg")
            if os.path.exists(seg_mask_path):
                seg_mask, _ = self.read_label_resize(seg_mask_path, self.image_size, dilate=self.cfg.DILATE)
                image = self.to_tensor(image)
                seg_loss_mask = self.distance_transform(seg_mask, self.cfg.WEIGHTED_SEG_LOSS_MAX,
                                                        self.cfg.WEIGHTED_SEG_LOSS_P)
                seg_mask = self.to_tensor(self.downsize(seg_mask))
                seg_loss_mask = self.to_tensor(self.downsize(seg_loss_mask))
                pos_samples.append((image, seg_mask, seg_loss_mask, True, image_path, None, img_name_short))
            else:
                seg_mask = np.zeros_like(image)
                image = self.to_tensor(image)
                seg_loss_mask = self.to_tensor(self.downsize(np.ones_like(seg_mask)))
                seg_mask = self.to_tensor(self.downsize(seg_mask))
                neg_samples.append((image, seg_mask, seg_loss_mask, True, image_path, seg_mask_path, img_name_short))
        self.pos_samples = pos_samples
        self.neg_samples = neg_samples
        self.num_pos = len(pos_samples)
        print(len(pos_samples))
        self.num_neg = len(neg_samples)
        print(len(neg_samples))
        self.len = 2 * len(pos_samples) if self.kind in ['TRAIN'] else len(pos_samples) + len(neg_samples)
        print(self.len)
        self.init_extra()