我在尝试用CPM2.1进行文本生成时，为了生成更长的结果，我修改了下面这行代码，使程序不会在生成标点符号时就停止。 https://github.com/OpenBMB/BMInf/blob/59e8903366ed53615d8af0a61e15b9f932042dcc/bminf/models/cpm2.py#L216

我的调用方法如下：

在修改代码后，我发现生成的结果result中会有换行符（由词表中id为3的token转换而来），并且在换行后，上下文就不再连贯了，像是另起了一个段，有时候甚至话题都变了，如下图。

这个例子在开始生成时直接换行了。

这个例子在换行后话题出现了较大变化。

1. 这种现象的出现是因为训练时就是以这种方式分隔段落的吗？
2. 可否允许从generate函数传入自定义的”停止字符“来控制生成行为？
3. 可否给出一个CPM2.1用来生成长篇文本的示例？

Describe the bug

用的CPM-2提供的docker镜像，出现该bug，其他环境没有尝试。

Minimal steps to reproduce

Expected behavior

运行examples下面的generate以及quick start中的代码都报错

Screenshots

Environment:

EVA报错

In [11]: eva2 = bminf.models.EVA2()

KeyError Traceback (most recent call last) in () ----> 1 eva2 = bminf.models.EVA2()

~/anaconda3/envs/yhs/lib/python3.6/site-packages/bminf/models/eva2.py in init(self, device, memory_limit, config) 56 raise ValueError("Memory is not enough") 57 ---> 58 super().init(config) 59 60 def dialogue(self,

~/anaconda3/envs/yhs/lib/python3.6/site-packages/bminf/arch/t5/model.py in init(self, config) 73 vocab_path = data.ensure_file(config.MODEL_NAME, "vocab.txt") 74 ---> 75 self.tokenizer = T5Tokenizer(vocab_path) 76 77 self.device = config.DEVICE

~/anaconda3/envs/yhs/lib/python3.6/site-packages/bminf/arch/t5/tokenizer.py in init(self, vocab_path, max_len, max_sentinels) 81 self.translator_dec = str.maketrans("\u2582\u2583", " \n") 82 ---> 83 self.sentinel_list = [self.encoder['<s_{}>'.format(i)] for i in range(max_sentinels)] 84 85 @property

~/anaconda3/envs/yhs/lib/python3.6/site-packages/bminf/arch/t5/tokenizer.py in (.0) 81 self.translator_dec = str.maketrans("\u2582\u2583", " \n") 82 ---> 83 self.sentinel_list = [self.encoder['<s_{}>'.format(i)] for i in range(max_sentinels)] 84 85 @property

KeyError: '<s_0>'

running the example file fill_blank.py, it raise error as follows:

Loading model
Start
Input:  北京环球度假区相关负责人介绍，北京环球影城指定单日门票将采用____制度，即推出淡季日、平季日、旺季日和特定日门票。____价格为418元，____价格为528元，____价格为638元，____价格为____元。北京环球度假区将提供90天滚动价格日历，以方便游客提前规划行程。
Traceback (most recent call last):
  File "abc.py", line 28, in <module>
    main()
  File "abc.py", line 25, in main
    fill_blank(cpm2, input_text)
  File "abc.py", line 9, in fill_blank
    for result in cpm2.fill_blank(text,
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/models/cpm2.py", line 245, in fill_blank
    for token in res:
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/models/cpm2.py", line 129, in _gen_iter
    self._model.embedding(
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/arch/t5/model.py", line 165, in embedding
    self.input_embedding.embedding_forward(ctx, tensor_ids, x_out)
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/layers/embedding.py", line 27, in embedding_forward
    ck.embedding_forward(
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/embedding.py", line 25, in embedding_forward
    embedding_kernel.cu_embedding_forward(
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/base.py", line 48, in __call__
    func = self._prepare_func()
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/base.py", line 40, in _prepare_func
    self._module.get_module(), self._func_name
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/base.py", line 23, in get_module
    Device(curr_device).use()   # force initialize context
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/device/__init__.py", line 152, in use
    self._device.use()
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/device/__init__.py", line 120, in use
    self.cublasLtHandle = cublaslt.cublasLtCreate()
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/library/base.py", line 94, in wrapper
    return f(*args, **kwargs)
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/library/cublaslt.py", line 105, in cublasLtCreate
    checkCublasStatus(cublasLt.cublasLtCreate(ctypes.byref(handle)))
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/library/cublaslt.py", line 98, in checkCublasStatus
    raise RuntimeError("CUBLAS error: {}".format(
RuntimeError: CUBLAS error: CUBLAS_STATUS_NOT_INITIALIZED

Environment: Python 3.8.10 cudatoolkit 11.3.1

请问该工具是否直接支持 cpm2-finetune配套的cpm2模型（需要到智源页面申请）； 我下载了中英文模型，100亿参数，vocab大小51967；本来有4个单独的文件，我按照官方脚本将其合并成1个单文件模型，测试显示没问题； 修改一些参数后，用bminf 下的 example/generate_cpm2.py加载我合并的单文件模型进行测试，无法加载，错误如下：

更新1

1. 我发现应该是要加载压缩量化等技术处理之后的模型，tool下有个migrate_cpm2.py，我用它做了量化工作，得到11g的模型；建议可以把文档写的详细一点。

2. 用 migrate_cpm2.py量化后，重新微调是如何做的，类似 训练过程量化（quantized-aware）吗？

3. 我用generate_cpm2.py加载上面 量化后的11g模型，推理时设置最多生成100个字，查看显存占用要用13g+（A100），不知道要怎么做到你们 doc 说的 显存调度，可以在2080ti下跑推理(2080ti只有11g显存)？

4. 请问怎么样把模型的模块拆分到不同gpu？这样可以解决第3步11g显存不够用的问题。比如把encoder、decoder分配到不同的gpu。我看模型构建并不是用torch等框架，数据迁移到显存主要靠with device 和 allocator好像，所以没太懂怎么把不同模块分配到不同gpu；

@a710128 期望回复，谢谢

运行examples/generate.py时报错，上层调用栈是functions/gemm.py的第249行。 https://github.com/OpenBMB/BMInf/blob/d40c6f5d5678e1cba771048ecd0923bceae176e2/bminf/functions/gemm.py#L249

使用的环境为： cuda 10.1（cublas版本为10.1.0.63） BMInf 0.0.4 通过clone + python setup.py install方式安装 torch 1.7.1

将输入改成 input_text = "近日，北京智源人工智能研究院和清华大学研究团队发布了以中文为核心的大规模预训练语言模型 CPM-LM，参数规模达 26 亿，预训练中文数据规模 100 GB。" 会报错 "Unexpected model output: 26239" 请问fill_blank输入的文本有什么要求？或者对要填空的词有什么要求？ 用的是 cpm2 = bminf.models.CPM2()

用pip 安装的，bminf-1.0.0

运行generate_cpm2.py 报value error

(EVAAA) [root@localhost examples]# python generate_cpm2.py Loading model Input: 天空是蔚蓝色，窗外有 Output: 天空是蔚蓝色，窗外有Traceback (most recent call last): File "generate_cpm2.py", line 32, in main() File "generate_cpm2.py", line 29, in main generate(cpm2_1, input_text) File "generate_cpm2.py", line 11, in generate value, stoped = model.generate( ValueError: too many values to unpack (expected 2)

when running examples/fill_blank.py, get error: AttributeError: type object 'cublasLt' has no attribute 'cublasLtHandle_t'

cuda version is 10.0 have successfully installed bminf 0.4.0 any idea how to solve this problem?

I am not familiar with int8. But i suppose it can not be trained like other fp32 models? Any suggestion about how to finetune it?

And does cpm2.1 has any report or paper? I did not find it anywhere.

Thank you!

Is your feature request related to a problem? Please describe.

有时候跑模型的服务器是物理断网的，需要手动下载模型上传后再加载。 从前（0.0.4版本）可以通过设置config的MODEL_NAME实现本地加载，但代码更新到1.0.0以后不能这样做了（除非修改BMInf的源码）。

Describe the solution you'd like

在初始化模型时提供一个接口，指定本地路径进行加载（可能通过修改现有的version字段实现）。

Describe alternatives you've considered

无。

其他：请问0.0.4到1.0.0之间，CPM2.1模型是否更新过？使用1.0.0的代码加载0.0.4时期下载的模型时报错了。

模型代码：

self.model = MyBert.from_pretrained(pretrained_model_name_or_path=model_path,)
self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
self.model.to(self.device)
self.model = bminf.wrapper(self.model)

错误信息：

input_embed = self.model.bert(**input_tokenized)["last_hidden_state"]
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_bert.py", line 1022, in forward
    encoder_outputs = self.encoder(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_bert.py", line 611, in forward
    layer_outputs = layer_module(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_bert.py", line 497, in forward
    self_attention_outputs = self.attention(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_bert.py", line 427, in forward
    self_outputs = self.self(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_bert.py", line 293, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/bminf/quantization/__init__.py", line 81, in forward
    out = OpLinear.apply(x, self.weight_quant, self.weight_scale)
  File "/usr/local/lib/python3.8/dist-packages/bminf/quantization/__init__.py", line 31, in forward
    gemm_int8(
  File "/usr/local/lib/python3.8/dist-packages/cpm_kernels/kernels/gemm.py", line 139, in gemm_int8
    assert m % 4 == 0 and n % 4 == 0 and k % 4 == 0
AssertionError

Describe the bug 使用docker环境，运行三个demo时后台都报错误 File "/usr/local/lib/python3.6/dist-packages/bminf/arch/t5/model.py", line 238, in encode True File "/usr/local/lib/python3.6/dist-packages/bminf/layers/transformer_block.py", line 42, in forward x = self.self_attention.forward(allocator, x, attention_mask, self_attn_position_bias) File "/usr/local/lib/python3.6/dist-packages/bminf/layers/attention.py", line 63, in forward qkv_i32 File "/usr/local/lib/python3.6/dist-packages/bminf/functions/gemm.py", line 86, in igemm _igemm(allocator, a, aT, b, bT, c, device, stream) File "/usr/local/lib/python3.6/dist-packages/bminf/functions/gemm.py", line 265, in _igemm stream.ptr File "/usr/local/lib/python3.6/dist-packages/bminf/backend/cublaslt.py", line 101, in checkCublasStatus raise RuntimeError("cublas error: %s" % cublas_errors[cublas_status]) RuntimeError: cublas error: CUBLAS_STATUS_NOT_SUPPORTED

请问是什么原因，是哪个版本有问题吗？

Environment: cuda：10.1 模型：EVA-int8 显存：12G

1. 您好，如图所述，我想查看 cupy操作cuda的函数的具体定义和用法，但是可能是因为cupy封装了c/c++代码，所以看不到，请问可以去哪里看呢？ 能帮忙解释一下图中第3个参数 routine 里面 4个函数执行顺序吗（我了解大概是 创建结构体、计算对称量化的scale）

跳到定义处，就只有这样的doc

2. 请问下图红框内为什么那样写？

3. 想问一下为什么选择使用cupy直接操作cuda呢，比如allocator、igemm、fgemm的应用？这样相比使用框架（如pytorch等）实现量化有更大的好处吗？感觉cupy+cuda实现方式 要求挺高的

非常感谢

@a710128

Describe the bug

运行路径https://github.com/OpenBMB/BMInf 的demo时候，出现RuntimeError: Library cublasLt is not initialized错误

Minimal steps to reproduce

import bminf #成功导入 cpm2 = bminf.models.CPM2() #成功定义 cpm2.fill_blank('好') #报错 RuntimeError: Library cublasLt is not initialized Expected behavior

Screenshots

Environment:

NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 NVIDIA A40
CUDA Version: 11.3 Memory:45634MiB

Describe the bug

输入: import bminf cpm2 = bminf.models.CPM2() result = cpm2.fill_blank("有一个服装品牌叫做<span>专门设计彩绘T恤", top_p=0.5, top_n=5, temperature=0.5, frequency_penalty=0, presence_penalty=0 )

报错信息: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3331, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 6, in presence_penalty=0 File "/usr/local/lib/python3.6/dist-packages/bminf/models/cpm2.py", line 252, in fill_blank raise RuntimeError("Unexpected model output: %d" % token) RuntimeError: Unexpected model output: 26239

请帮忙看看是什么原因

Environment: python3.6 torch1.8.1