您好～ 我在尝试跑CPM-2-Pretrain代码时发现以下问题：

1. 在“src-->data-->dataset_utils.py”中compile_helper()与deepspeed相比 发现缺失以下代码 ret = subprocess.run(['make', '-C', path]) 如若不加此行代码会出现 import helper导入不成功，如果加上上述代码会出现2的问题。 请问该行代码是否需要加上？

2. 当我加入ret = subprocess.run(['make', '-C', path]) 进行预训练会出现数组越界问题 IndexError：index 53004 is out of bounds for axis 0 with size 62 问题出现在“src-->data-->enc_dec_dataset.py” line228. 我检查了一下似乎出现在了offset=tmp_target_offset[2 * x]这个上面。 不知道问题出在哪里。

很抱歉不能把原版错误粘贴出来。。问题困扰了我两天，还望作者能帮忙解答一下此处问题！万分感谢❤️❤️❤️

我单机测试两台机器都能正常，但是多机器并行后会出现环境问题

ip:   File "/path/to//src/pretrain_enc_dec.py", line 823, in <module>
ip:     main()
ip:   File "/path/to//src/pretrain_enc_dec.py", line 684, in main
ip:     model, optimizer, lr_scheduler = setup_model_and_optimizer(args, tokenizer.vocab_size)
ip:   File "/path/to//src/pretrain_enc_dec.py", line 157, in setup_model_and_optimizer
ip:     model, optimizer, _, lr_scheduler = deepspeed.initialize(
ip:   File "/path/to//lib/python3.8/site-packages/deepspeed/__init__.py", line 110, in initialize
ip:     engine = DeepSpeedEngine(args=args,
ip:   File "/path/to//lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 198, in __init__
ip:     util_ops = UtilsBuilder().load()
ip:   File "/path/to//lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 176, in load
ip:     return self.jit_load(verbose)
ip:   File "/path/to//lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 204, in jit_load
ip:     op_module = load(
ip:   File "/path/to//lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1079, in load
ip:     return _jit_compile(
ip:   File "/path/to//lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1292, in _jit_compile
ip:     _write_ninja_file_and_build_library(
ip:   File "/path/to//lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1373, in _write_ninja_file_and_build_library
ip:     verify_ninja_availability()
ip:   File "/path/to//lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1429, in verify_ninja_availability
ip:     raise RuntimeError("Ninja is required to load C++ extensions")
ip: RuntimeError: Ninja is required to load C++ extensions

您好，我在加载moe的模型时候遇到了问题： RuntimeError: Error(s) in loading state_dict for EncDecModel: size mismatch for word_embeds.weight: copying a param with shape torch.Size([6496, 4096]) from checkpoint, the shape in current model is torch.Size([6592, 4096]). size mismatch for lm_head.weight: copying a param with shape torch.Size([6496, 4096]) from checkpoint, the shape in current model is torch.Size([6592, 4096]). size mismatch for encoder.word_embeds.weight: copying a param with shape torch.Size([6496, 4096]) from checkpoint, the shape in current model is torch.Size([6592, 4096]). size mismatch for decoder.word_embeds.weight: copying a param with shape torch.Size([6496, 4096]) from checkpoint, the shape in current model is torch.Size([6592, 4096]). 这里的词表用的是中英文词表： https://github.com/TsinghuaAI/CPM-2-Pretrain/blob/moe/bpe_cn_en/vocab.txt

我的疑问是： 这个词表的大小是52736=6592乘以8，为什么加载过程中报错，显示模型文件中的词表大小是6496乘以8？

这个问题非常困扰我，还请作者帮忙解答，万分感谢~~

还有一个小疑问，中英文词表的中文字符最后一行26050行是不是有点问题？（与中文词表的26050行不太一样） https://github.com/TsinghuaAI/CPM-2-Pretrain/blob/moe/bpe_cn_en/vocab.txt#L26050

May i know how is the model partitioned in CPM-2 MoE? It seems each rank only takes 1 expert, and each expert is further partitioned (i.e., 256 model partitions in total)?

Thank you for your info.

How to complete the transformation： Deepspeed_ckpt to mp_model(tensor parallel) to single_model(single ckpt) with zero-1/2 and mp(tensor parallel) training. Furthermore,Deepspeed version should be? thanks very much.

Not found the definition of TransposedSampler in sampler.py. I have found it in https://github.com/NVIDIA/sentiment-discovery/blob/master/data_utils/samplers.py. Not sure if it is the right on.

class TransposedSampler(data.sampler.Sampler):
    """
    Instead of performing sequential sampling, samples array in a transposed fashion given the
    batch size to sampled. Instead of generating the following indices for a batch size of 2
        1 3 5
        2 4 6
    It will generate
        1 2 3
        4 5 6
    """
    def __init__(self, data_source, batch_size, data_sampler=None):
        self.data_source = data_source
        self.batch_size = batch_size
        self.len_ds = len(data_source)
        self.strat_width = self.len_ds//batch_size
        #self.strat_width = math.ceil(self.len_ds/batch_size)
        self.data_sampler = data_sampler
        self.wrap_around = 0

    def transpose_helper(self, x):
        """computes index corrseponding to transpose of index x"""
        return ((x%self.batch_size)*self.strat_width+(x//self.batch_size))%self.len_ds
        x += self.wrap_around
        return ((x%self.batch_size)*self.strat_width+(x//self.batch_size))%self.len_ds

    def __iter__(self):
        if self.data_sampler is None:
            return iter(map(self.transpose_helper, range(len(self))))
        return iter(map(self.transpose_helper, iter(self.data_sampler)))

    def __len__(self):
        #return self.len_ds
        return self.strat_width*self.batch_size

I install bminf from docker just like:

docker run -it --gpus 1 -v ${100000_MODEL_FILE_PATH}:/root/.cache/bigmodels --rm openbmb/bminf python3 examples/fill_blank.py

Where 100000_MODEL_FILE_PATH containers 4 .pt files in my localhost. However, I got the following error

Loading model
Failed to connect to the source server
Traceback (most recent call last):
  File "examples/fill_blank.py", line 29, in <module>
    main()
  File "examples/fill_blank.py", line 24, in main
    cpm2 = bminf.models.CPM2()
  File "/usr/local/lib/python3.6/dist-packages/bminf/models/cpm2.py", line 60, in __init__
    super().__init__(config)
  File "/usr/local/lib/python3.6/dist-packages/bminf/arch/t5/model.py", line 72, in __init__
    model_path = data.ensure_file(config.MODEL_NAME, "checkpoint.pt")
  File "/usr/local/lib/python3.6/dist-packages/bminf/data/__init__.py", line 49, in ensure_file
    raise ConnectionError("Failed to connect to the source server")
ConnectionError: Failed to connect to the source server

I also tried to convert the 4 .pt files into one .pt file, however, the same error occurred. How to solve the server connect problem?

您好～ 我在尝试跑CPM-2-Pretrain代码时发现以下问题：

在“src-->data-->dataset_utils.py”中compile_helper()与deepspeed相比 发现缺失以下代码 ret = subprocess.run(['make', '-C', path]) 如若不加此行代码会出现 import helper导入不成功，如果加上上述代码会出现2的问题。 请问该行代码是否需要加上？

当我加入ret = subprocess.run(['make', '-C', path]) 进行预训练会出现数组越界问题 IndexError：index 53004 is out of bounds for axis 0 with size 62 问题出现在“src-->data-->enc_dec_dataset.py” line228. 我检查了一下似乎出现在了offset=tmp_target_offset[2 * x]这个上面。 不知道问题出在哪里。

很抱歉不能把原版错误粘贴出来。。问题困扰了我两天，还望作者能帮忙解答一下此处问题！万分感谢❤️❤️❤️

@zzy14 @t1101675 What confuses me is this default parameter setting, shouldn't it be d_ffn (10240) * 32 for MOE？？？https://github.com/TsinghuaAI/CPM-2-Pretrain/blob/a00b3dd70d71a796a1ed2a925ddf7902e0209ab3/src/configs/model/enc_dec_xlarge_config.json#L3

您好，当我使用您的代码做数据预处理时(具体文件为/src/tokenization_enc_dec.py)， 发现位于第182行的jieba.cut(text, cut_all=False)无法处理诸如'<s>'这样的特殊token。 jieba会将其分为'<', 's','>'再进行编码，请问这里是否有问题？ 还望解答，谢谢！

目前bminf 和智源的模型都是一个模型，我尝试了使用cpm2-pretrain进行微调，发现加载模型时，会有问题：

由于下载的模型已经合并成为一个，能不能把模型并行的多个子模型合并成一个模型的代码，以及如何拆分成一个模型成为模型并行的模型加载的代码开源？

顺便提一下在另一个issues中的一个小问题：https://github.com/TsinghuaAI/CPM-2-Pretrain/issues/16#issuecomment-983374591