Google and Stanford University released a new pre-trained model called ELECTRA

比较 pretraining 不同 steps 的 checkpoint。同 step 对应的 checkpoint，electra 100% label 学习的优势，在 finetuning 效果上，论文里是显著快于 bert 的。

不知道复现是否有这个结论呢？我们在做一个类似的策略，收敛速度上并没有论文显著。

我根据readme的使用方法进行操作，提示“writing predictions is not supported for this task”，于是我在代码“ if task.name in ["cola", "mrpc", "mnli", "sst", "rte", "qnli", "qqp", "sts"]:”中增加了lcqmc。但是屏幕只输出loss没有Accuracy。请问是哪里出错了？如何复现出readme中所展示的效果？

transformers 4.0.1加载chinese-electra-small-ex-generator输出如下log，请问这个是正常的吗？

Some weights of the model checkpoint at hfl/chinese-electra-small-ex-generator were not used when initializing BertForMultiTaskClassification: ['electra.embeddings.word_embeddings.weight', 'electra.embeddings.position_embeddings.weight', 'electra.embeddings.token_type_embeddings.weight', 'electra.embeddings.LayerNorm.weight', 'electra.embeddings.LayerNorm.bias', 'electra.embeddings_project.weight', 'electra.embeddings_project.bias', 'electra.encoder.layer.0.attention.self.query.weight', 'electra.encoder.layer.0.attention.self.query.bias', 'electra.encoder.layer.0.attention.self.key.weight', 'electra.encoder.layer.0.attention.self.key.bias', 'electra.encoder.layer.0.attention.self.value.weight', 'electra.encoder.layer.0.attention.self.value.bias', 'electra.encoder.layer.0.attention.output.dense.weight', 'electra.encoder.layer.0.attention.output.dense.bias', 'electra.encoder.layer.0.attention.output.LayerNorm.weight', 'electra.encoder.layer.0.attention.output.LayerNorm.bias', 'electra.encoder.layer.0.intermediate.dense.weight', 'electra.encoder.layer.0.intermediate.dense.bias', 'electra.encoder.layer.0.output.dense.weight', 'electra.encoder.layer.0.output.dense.bias', 'electra.encoder.layer.0.output.LayerNorm.weight', 'electra.encoder.layer.0.output.LayerNorm.bias', 'electra.encoder.layer.1.attention.self.query.weight', 'electra.encoder.layer.1.attention.self.query.bias', 'electra.encoder.layer.1.attention.self.key.weight', 'electra.encoder.layer.1.attention.self.key.bias', 'electra.encoder.layer.1.attention.self.value.weight', 'electra.encoder.layer.1.attention.self.value.bias', 'electra.encoder.layer.1.attention.output.dense.weight', 'electra.encoder.layer.1.attention.output.dense.bias', 'electra.encoder.layer.1.attention.output.LayerNorm.weight', 'electra.encoder.layer.1.attention.output.LayerNorm.bias', 'electra.encoder.layer.1.intermediate.dense.weight', 'electra.encoder.layer.1.intermediate.dense.bias', 'electra.encoder.layer.1.output.dense.weight', 'electra.encoder.layer.1.output.dense.bias', 'electra.encoder.layer.1.output.LayerNorm.weight', 'electra.encoder.layer.1.output.LayerNorm.bias', 'electra.encoder.layer.2.attention.self.query.weight', 'electra.encoder.layer.2.attention.self.query.bias', 'electra.encoder.layer.2.attention.self.key.weight', 'electra.encoder.layer.2.attention.self.key.bias', 'electra.encoder.layer.2.attention.self.value.weight', 'electra.encoder.layer.2.attention.self.value.bias', 'electra.encoder.layer.2.attention.output.dense.weight', 'electra.encoder.layer.2.attention.output.dense.bias', 'electra.encoder.layer.2.attention.output.LayerNorm.weight', 'electra.encoder.layer.2.attention.output.LayerNorm.bias', 'electra.encoder.layer.2.intermediate.dense.weight', 'electra.encoder.layer.2.intermediate.dense.bias', 'electra.encoder.layer.2.output.dense.weight', 'electra.encoder.layer.2.output.dense.bias', 'electra.encoder.layer.2.output.LayerNorm.weight', 'electra.encoder.layer.2.output.LayerNorm.bias', 'electra.encoder.layer.3.attention.self.query.weight', 'electra.encoder.layer.3.attention.self.query.bias', 'electra.encoder.layer.3.attention.self.key.weight', 'electra.encoder.layer.3.attention.self.key.bias', 'electra.encoder.layer.3.attention.self.value.weight', 'electra.encoder.layer.3.attention.self.value.bias', 'electra.encoder.layer.3.attention.output.dense.weight', 'electra.encoder.layer.3.attention.output.dense.bias', 'electra.encoder.layer.3.attention.output.LayerNorm.weight', 'electra.encoder.layer.3.attention.output.LayerNorm.bias', 'electra.encoder.layer.3.intermediate.dense.weight', 'electra.encoder.layer.3.intermediate.dense.bias', 'electra.encoder.layer.3.output.dense.weight', 'electra.encoder.layer.3.output.dense.bias', 'electra.encoder.layer.3.output.LayerNorm.weight', 'electra.encoder.layer.3.output.LayerNorm.bias', 'electra.encoder.layer.4.attention.self.query.weight', 'electra.encoder.layer.4.attention.self.query.bias', 'electra.encoder.layer.4.attention.self.key.weight', 'electra.encoder.layer.4.attention.self.key.bias', 'electra.encoder.layer.4.attention.self.value.weight', 'electra.encoder.layer.4.attention.self.value.bias', 'electra.encoder.layer.4.attention.output.dense.weight', 'electra.encoder.layer.4.attention.output.dense.bias', 'electra.encoder.layer.4.attention.output.LayerNorm.weight', 'electra.encoder.layer.4.attention.output.LayerNorm.bias', 'electra.encoder.layer.4.intermediate.dense.weight', 'electra.encoder.layer.4.intermediate.dense.bias', 'electra.encoder.layer.4.output.dense.weight', 'electra.encoder.layer.4.output.dense.bias', 'electra.encoder.layer.4.output.LayerNorm.weight', 'electra.encoder.layer.4.output.LayerNorm.bias', 'electra.encoder.layer.5.attention.self.query.weight', 'electra.encoder.layer.5.attention.self.query.bias', 'electra.encoder.layer.5.attention.self.key.weight', 'electra.encoder.layer.5.attention.self.key.bias', 'electra.encoder.layer.5.attention.self.value.weight', 'electra.encoder.layer.5.attention.self.value.bias', 'electra.encoder.layer.5.attention.output.dense.weight', 'electra.encoder.layer.5.attention.output.dense.bias', 'electra.encoder.layer.5.attention.output.LayerNorm.weight', 'electra.encoder.layer.5.attention.output.LayerNorm.bias', 'electra.encoder.layer.5.intermediate.dense.weight', 'electra.encoder.layer.5.intermediate.dense.bias', 'electra.encoder.layer.5.output.dense.weight', 'electra.encoder.layer.5.output.dense.bias', 'electra.encoder.layer.5.output.LayerNorm.weight', 'electra.encoder.layer.5.output.LayerNorm.bias', 'electra.encoder.layer.6.attention.self.query.weight', 'electra.encoder.layer.6.attention.self.query.bias', 'electra.encoder.layer.6.attention.self.key.weight', 'electra.encoder.layer.6.attention.self.key.bias', 'electra.encoder.layer.6.attention.self.value.weight', 'electra.encoder.layer.6.attention.self.value.bias', 'electra.encoder.layer.6.attention.output.dense.weight', 'electra.encoder.layer.6.attention.output.dense.bias', 'electra.encoder.layer.6.attention.output.LayerNorm.weight', 'electra.encoder.layer.6.attention.output.LayerNorm.bias', 'electra.encoder.layer.6.intermediate.dense.weight', 'electra.encoder.layer.6.intermediate.dense.bias', 'electra.encoder.layer.6.output.dense.weight', 'electra.encoder.layer.6.output.dense.bias', 'electra.encoder.layer.6.output.LayerNorm.weight', 'electra.encoder.layer.6.output.LayerNorm.bias', 'electra.encoder.layer.7.attention.self.query.weight', 'electra.encoder.layer.7.attention.self.query.bias', 'electra.encoder.layer.7.attention.self.key.weight', 'electra.encoder.layer.7.attention.self.key.bias', 'electra.encoder.layer.7.attention.self.value.weight', 'electra.encoder.layer.7.attention.self.value.bias', 'electra.encoder.layer.7.attention.output.dense.weight', 'electra.encoder.layer.7.attention.output.dense.bias', 'electra.encoder.layer.7.attention.output.LayerNorm.weight', 'electra.encoder.layer.7.attention.output.LayerNorm.bias', 'electra.encoder.layer.7.intermediate.dense.weight', 'electra.encoder.layer.7.intermediate.dense.bias', 'electra.encoder.layer.7.output.dense.weight', 'electra.encoder.layer.7.output.dense.bias', 'electra.encoder.layer.7.output.LayerNorm.weight', 'electra.encoder.layer.7.output.LayerNorm.bias', 'electra.encoder.layer.8.attention.self.query.weight', 'electra.encoder.layer.8.attention.self.query.bias', 'electra.encoder.layer.8.attention.self.key.weight', 'electra.encoder.layer.8.attention.self.key.bias', 'electra.encoder.layer.8.attention.self.value.weight', 'electra.encoder.layer.8.attention.self.value.bias', 'electra.encoder.layer.8.attention.output.dense.weight', 'electra.encoder.layer.8.attention.output.dense.bias', 'electra.encoder.layer.8.attention.output.LayerNorm.weight', 'electra.encoder.layer.8.attention.output.LayerNorm.bias', 'electra.encoder.layer.8.intermediate.dense.weight', 'electra.encoder.layer.8.intermediate.dense.bias', 'electra.encoder.layer.8.output.dense.weight', 'electra.encoder.layer.8.output.dense.bias', 'electra.encoder.layer.8.output.LayerNorm.weight', 'electra.encoder.layer.8.output.LayerNorm.bias', 'electra.encoder.layer.9.attention.self.query.weight', 'electra.encoder.layer.9.attention.self.query.bias', 'electra.encoder.layer.9.attention.self.key.weight', 'electra.encoder.layer.9.attention.self.key.bias', 'electra.encoder.layer.9.attention.self.value.weight', 'electra.encoder.layer.9.attention.self.value.bias', 'electra.encoder.layer.9.attention.output.dense.weight', 'electra.encoder.layer.9.attention.output.dense.bias', 'electra.encoder.layer.9.attention.output.LayerNorm.weight', 'electra.encoder.layer.9.attention.output.LayerNorm.bias', 'electra.encoder.layer.9.intermediate.dense.weight', 'electra.encoder.layer.9.intermediate.dense.bias', 'electra.encoder.layer.9.output.dense.weight', 'electra.encoder.layer.9.output.dense.bias', 'electra.encoder.layer.9.output.LayerNorm.weight', 'electra.encoder.layer.9.output.LayerNorm.bias', 'electra.encoder.layer.10.attention.self.query.weight', 'electra.encoder.layer.10.attention.self.query.bias', 'electra.encoder.layer.10.attention.self.key.weight', 'electra.encoder.layer.10.attention.self.key.bias', 'electra.encoder.layer.10.attention.self.value.weight', 'electra.encoder.layer.10.attention.self.value.bias', 'electra.encoder.layer.10.attention.output.dense.weight', 'electra.encoder.layer.10.attention.output.dense.bias', 'electra.encoder.layer.10.attention.output.LayerNorm.weight', 'electra.encoder.layer.10.attention.output.LayerNorm.bias', 'electra.encoder.layer.10.intermediate.dense.weight', 'electra.encoder.layer.10.intermediate.dense.bias', 'electra.encoder.layer.10.output.dense.weight', 'electra.encoder.layer.10.output.dense.bias', 'electra.encoder.layer.10.output.LayerNorm.weight', 'electra.encoder.layer.10.output.LayerNorm.bias', 'electra.encoder.layer.11.attention.self.query.weight', 'electra.encoder.layer.11.attention.self.query.bias', 'electra.encoder.layer.11.attention.self.key.weight', 'electra.encoder.layer.11.attention.self.key.bias', 'electra.encoder.layer.11.attention.self.value.weight', 'electra.encoder.layer.11.attention.self.value.bias', 'electra.encoder.layer.11.attention.output.dense.weight', 'electra.encoder.layer.11.attention.output.dense.bias', 'electra.encoder.layer.11.attention.output.LayerNorm.weight', 'electra.encoder.layer.11.attention.output.LayerNorm.bias', 'electra.encoder.layer.11.intermediate.dense.weight', 'electra.encoder.layer.11.intermediate.dense.bias', 'electra.encoder.layer.11.output.dense.weight', 'electra.encoder.layer.11.output.dense.bias', 'electra.encoder.layer.11.output.LayerNorm.weight', 'electra.encoder.layer.11.output.LayerNorm.bias', 'electra.encoder.layer.12.attention.self.query.weight', 'electra.encoder.layer.12.attention.self.query.bias', 'electra.encoder.layer.12.attention.self.key.weight', 'electra.encoder.layer.12.attention.self.key.bias', 'electra.encoder.layer.12.attention.self.value.weight', 'electra.encoder.layer.12.attention.self.value.bias', 'electra.encoder.layer.12.attention.output.dense.weight', 'electra.encoder.layer.12.attention.output.dense.bias', 'electra.encoder.layer.12.attention.output.LayerNorm.weight', 'electra.encoder.layer.12.attention.output.LayerNorm.bias', 'electra.encoder.layer.12.intermediate.dense.weight', 'electra.encoder.layer.12.intermediate.dense.bias', 'electra.encoder.layer.12.output.dense.weight', 'electra.encoder.layer.12.output.dense.bias', 'electra.encoder.layer.12.output.LayerNorm.weight', 'electra.encoder.layer.12.output.LayerNorm.bias', 'electra.encoder.layer.13.attention.self.query.weight', 'electra.encoder.layer.13.attention.self.query.bias', 'electra.encoder.layer.13.attention.self.key.weight', 'electra.encoder.layer.13.attention.self.key.bias', 'electra.encoder.layer.13.attention.self.value.weight', 'electra.encoder.layer.13.attention.self.value.bias', 'electra.encoder.layer.13.attention.output.dense.weight', 'electra.encoder.layer.13.attention.output.dense.bias', 'electra.encoder.layer.13.attention.output.LayerNorm.weight', 'electra.encoder.layer.13.attention.output.LayerNorm.bias', 'electra.encoder.layer.13.intermediate.dense.weight', 'electra.encoder.layer.13.intermediate.dense.bias', 'electra.encoder.layer.13.output.dense.weight', 'electra.encoder.layer.13.output.dense.bias', 'electra.encoder.layer.13.output.LayerNorm.weight', 'electra.encoder.layer.13.output.LayerNorm.bias', 'electra.encoder.layer.14.attention.self.query.weight', 'electra.encoder.layer.14.attention.self.query.bias', 'electra.encoder.layer.14.attention.self.key.weight', 'electra.encoder.layer.14.attention.self.key.bias', 'electra.encoder.layer.14.attention.self.value.weight', 'electra.encoder.layer.14.attention.self.value.bias', 'electra.encoder.layer.14.attention.output.dense.weight', 'electra.encoder.layer.14.attention.output.dense.bias', 'electra.encoder.layer.14.attention.output.LayerNorm.weight', 'electra.encoder.layer.14.attention.output.LayerNorm.bias', 'electra.encoder.layer.14.intermediate.dense.weight', 'electra.encoder.layer.14.intermediate.dense.bias', 'electra.encoder.layer.14.output.dense.weight', 'electra.encoder.layer.14.output.dense.bias', 'electra.encoder.layer.14.output.LayerNorm.weight', 'electra.encoder.layer.14.output.LayerNorm.bias', 'electra.encoder.layer.15.attention.self.query.weight', 'electra.encoder.layer.15.attention.self.query.bias', 'electra.encoder.layer.15.attention.self.key.weight', 'electra.encoder.layer.15.attention.self.key.bias', 'electra.encoder.layer.15.attention.self.value.weight', 'electra.encoder.layer.15.attention.self.value.bias', 'electra.encoder.layer.15.attention.output.dense.weight', 'electra.encoder.layer.15.attention.output.dense.bias', 'electra.encoder.layer.15.attention.output.LayerNorm.weight', 'electra.encoder.layer.15.attention.output.LayerNorm.bias', 'electra.encoder.layer.15.intermediate.dense.weight', 'electra.encoder.layer.15.intermediate.dense.bias', 'electra.encoder.layer.15.output.dense.weight', 'electra.encoder.layer.15.output.dense.bias', 'electra.encoder.layer.15.output.LayerNorm.weight', 'electra.encoder.layer.15.output.LayerNorm.bias', 'electra.encoder.layer.16.attention.self.query.weight', 'electra.encoder.layer.16.attention.self.query.bias', 'electra.encoder.layer.16.attention.self.key.weight', 'electra.encoder.layer.16.attention.self.key.bias', 'electra.encoder.layer.16.attention.self.value.weight', 'electra.encoder.layer.16.attention.self.value.bias', 'electra.encoder.layer.16.attention.output.dense.weight', 'electra.encoder.layer.16.attention.output.dense.bias', 'electra.encoder.layer.16.attention.output.LayerNorm.weight', 'electra.encoder.layer.16.attention.output.LayerNorm.bias', 'electra.encoder.layer.16.intermediate.dense.weight', 'electra.encoder.layer.16.intermediate.dense.bias', 'electra.encoder.layer.16.output.dense.weight', 'electra.encoder.layer.16.output.dense.bias', 'electra.encoder.layer.16.output.LayerNorm.weight', 'electra.encoder.layer.16.output.LayerNorm.bias', 'electra.encoder.layer.17.attention.self.query.weight', 'electra.encoder.layer.17.attention.self.query.bias', 'electra.encoder.layer.17.attention.self.key.weight', 'electra.encoder.layer.17.attention.self.key.bias', 'electra.encoder.layer.17.attention.self.value.weight', 'electra.encoder.layer.17.attention.self.value.bias', 'electra.encoder.layer.17.attention.output.dense.weight', 'electra.encoder.layer.17.attention.output.dense.bias', 'electra.encoder.layer.17.attention.output.LayerNorm.weight', 'electra.encoder.layer.17.attention.output.LayerNorm.bias', 'electra.encoder.layer.17.intermediate.dense.weight', 'electra.encoder.layer.17.intermediate.dense.bias', 'electra.encoder.layer.17.output.dense.weight', 'electra.encoder.layer.17.output.dense.bias', 'electra.encoder.layer.17.output.LayerNorm.weight', 'electra.encoder.layer.17.output.LayerNorm.bias', 'electra.encoder.layer.18.attention.self.query.weight', 'electra.encoder.layer.18.attention.self.query.bias', 'electra.encoder.layer.18.attention.self.key.weight', 'electra.encoder.layer.18.attention.self.key.bias', 'electra.encoder.layer.18.attention.self.value.weight', 'electra.encoder.layer.18.attention.self.value.bias', 'electra.encoder.layer.18.attention.output.dense.weight', 'electra.encoder.layer.18.attention.output.dense.bias', 'electra.encoder.layer.18.attention.output.LayerNorm.weight', 'electra.encoder.layer.18.attention.output.LayerNorm.bias', 'electra.encoder.layer.18.intermediate.dense.weight', 'electra.encoder.layer.18.intermediate.dense.bias', 'electra.encoder.layer.18.output.dense.weight', 'electra.encoder.layer.18.output.dense.bias', 'electra.encoder.layer.18.output.LayerNorm.weight', 'electra.encoder.layer.18.output.LayerNorm.bias', 'electra.encoder.layer.19.attention.self.query.weight', 'electra.encoder.layer.19.attention.self.query.bias', 'electra.encoder.layer.19.attention.self.key.weight', 'electra.encoder.layer.19.attention.self.key.bias', 'electra.encoder.layer.19.attention.self.value.weight', 'electra.encoder.layer.19.attention.self.value.bias', 'electra.encoder.layer.19.attention.output.dense.weight', 'electra.encoder.layer.19.attention.output.dense.bias', 'electra.encoder.layer.19.attention.output.LayerNorm.weight', 'electra.encoder.layer.19.attention.output.LayerNorm.bias', 'electra.encoder.layer.19.intermediate.dense.weight', 'electra.encoder.layer.19.intermediate.dense.bias', 'electra.encoder.layer.19.output.dense.weight', 'electra.encoder.layer.19.output.dense.bias', 'electra.encoder.layer.19.output.LayerNorm.weight', 'electra.encoder.layer.19.output.LayerNorm.bias', 'electra.encoder.layer.20.attention.self.query.weight', 'electra.encoder.layer.20.attention.self.query.bias', 'electra.encoder.layer.20.attention.self.key.weight', 'electra.encoder.layer.20.attention.self.key.bias', 'electra.encoder.layer.20.attention.self.value.weight', 'electra.encoder.layer.20.attention.self.value.bias', 'electra.encoder.layer.20.attention.output.dense.weight', 'electra.encoder.layer.20.attention.output.dense.bias', 'electra.encoder.layer.20.attention.output.LayerNorm.weight', 'electra.encoder.layer.20.attention.output.LayerNorm.bias', 'electra.encoder.layer.20.intermediate.dense.weight', 'electra.encoder.layer.20.intermediate.dense.bias', 'electra.encoder.layer.20.output.dense.weight', 'electra.encoder.layer.20.output.dense.bias', 'electra.encoder.layer.20.output.LayerNorm.weight', 'electra.encoder.layer.20.output.LayerNorm.bias', 'electra.encoder.layer.21.attention.self.query.weight', 'electra.encoder.layer.21.attention.self.query.bias', 'electra.encoder.layer.21.attention.self.key.weight', 'electra.encoder.layer.21.attention.self.key.bias', 'electra.encoder.layer.21.attention.self.value.weight', 'electra.encoder.layer.21.attention.self.value.bias', 'electra.encoder.layer.21.attention.output.dense.weight', 'electra.encoder.layer.21.attention.output.dense.bias', 'electra.encoder.layer.21.attention.output.LayerNorm.weight', 'electra.encoder.layer.21.attention.output.LayerNorm.bias', 'electra.encoder.layer.21.intermediate.dense.weight', 'electra.encoder.layer.21.intermediate.dense.bias', 'electra.encoder.layer.21.output.dense.weight', 'electra.encoder.layer.21.output.dense.bias', 'electra.encoder.layer.21.output.LayerNorm.weight', 'electra.encoder.layer.21.output.LayerNorm.bias', 'electra.encoder.layer.22.attention.self.query.weight', 'electra.encoder.layer.22.attention.self.query.bias', 'electra.encoder.layer.22.attention.self.key.weight', 'electra.encoder.layer.22.attention.self.key.bias', 'electra.encoder.layer.22.attention.self.value.weight', 'electra.encoder.layer.22.attention.self.value.bias', 'electra.encoder.layer.22.attention.output.dense.weight', 'electra.encoder.layer.22.attention.output.dense.bias', 'electra.encoder.layer.22.attention.output.LayerNorm.weight', 'electra.encoder.layer.22.attention.output.LayerNorm.bias', 'electra.encoder.layer.22.intermediate.dense.weight', 'electra.encoder.layer.22.intermediate.dense.bias', 'electra.encoder.layer.22.output.dense.weight', 'electra.encoder.layer.22.output.dense.bias', 'electra.encoder.layer.22.output.LayerNorm.weight', 'electra.encoder.layer.22.output.LayerNorm.bias', 'electra.encoder.layer.23.attention.self.query.weight', 'electra.encoder.layer.23.attention.self.query.bias', 'electra.encoder.layer.23.attention.self.key.weight', 'electra.encoder.layer.23.attention.self.key.bias', 'electra.encoder.layer.23.attention.self.value.weight', 'electra.encoder.layer.23.attention.self.value.bias', 'electra.encoder.layer.23.attention.output.dense.weight', 'electra.encoder.layer.23.attention.output.dense.bias', 'electra.encoder.layer.23.attention.output.LayerNorm.weight', 'electra.encoder.layer.23.attention.output.LayerNorm.bias', 'electra.encoder.layer.23.intermediate.dense.weight', 'electra.encoder.layer.23.intermediate.dense.bias', 'electra.encoder.layer.23.output.dense.weight', 'electra.encoder.layer.23.output.dense.bias', 'electra.encoder.layer.23.output.LayerNorm.weight', 'electra.encoder.layer.23.output.LayerNorm.bias', 'generator_predictions.LayerNorm.weight', 'generator_predictions.LayerNorm.bias', 'generator_predictions.dense.weight', 'generator_predictions.dense.bias', 'generator_lm_head.weight', 'generator_lm_head.bias']
- This IS expected if you are initializing BertForMultiTaskClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMultiTaskClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMultiTaskClassification were not initialized from the model checkpoint at hfl/chinese-electra-small-ex-generator and are newly initialized: ['embeddings.word_embeddings.weight', 'embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.LayerNorm.weight', 'embeddings.LayerNorm.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.9.output.dense.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.10.attention.self.value.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.11.output.dense.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.11.output.LayerNorm.bias', 'encoder.layer.12.attention.self.query.weight', 'encoder.layer.12.attention.self.query.bias', 'encoder.layer.12.attention.self.key.weight', 'encoder.layer.12.attention.self.key.bias', 'encoder.layer.12.attention.self.value.weight', 'encoder.layer.12.attention.self.value.bias', 'encoder.layer.12.attention.output.dense.weight', 'encoder.layer.12.attention.output.dense.bias', 'encoder.layer.12.attention.output.LayerNorm.weight', 'encoder.layer.12.attention.output.LayerNorm.bias', 'encoder.layer.12.intermediate.dense.weight', 'encoder.layer.12.intermediate.dense.bias', 'encoder.layer.12.output.dense.weight', 'encoder.layer.12.output.dense.bias', 'encoder.layer.12.output.LayerNorm.weight', 'encoder.layer.12.output.LayerNorm.bias', 'encoder.layer.13.attention.self.query.weight', 'encoder.layer.13.attention.self.query.bias', 'encoder.layer.13.attention.self.key.weight', 'encoder.layer.13.attention.self.key.bias', 'encoder.layer.13.attention.self.value.weight', 'encoder.layer.13.attention.self.value.bias', 'encoder.layer.13.attention.output.dense.weight', 'encoder.layer.13.attention.output.dense.bias', 'encoder.layer.13.attention.output.LayerNorm.weight', 'encoder.layer.13.attention.output.LayerNorm.bias', 'encoder.layer.13.intermediate.dense.weight', 'encoder.layer.13.intermediate.dense.bias', 'encoder.layer.13.output.dense.weight', 'encoder.layer.13.output.dense.bias', 'encoder.layer.13.output.LayerNorm.weight', 'encoder.layer.13.output.LayerNorm.bias', 'encoder.layer.14.attention.self.query.weight', 'encoder.layer.14.attention.self.query.bias', 'encoder.layer.14.attention.self.key.weight', 'encoder.layer.14.attention.self.key.bias', 'encoder.layer.14.attention.self.value.weight', 'encoder.layer.14.attention.self.value.bias', 'encoder.layer.14.attention.output.dense.weight', 'encoder.layer.14.attention.output.dense.bias', 'encoder.layer.14.attention.output.LayerNorm.weight', 'encoder.layer.14.attention.output.LayerNorm.bias', 'encoder.layer.14.intermediate.dense.weight', 'encoder.layer.14.intermediate.dense.bias', 'encoder.layer.14.output.dense.weight', 'encoder.layer.14.output.dense.bias', 'encoder.layer.14.output.LayerNorm.weight', 'encoder.layer.14.output.LayerNorm.bias', 'encoder.layer.15.attention.self.query.weight', 'encoder.layer.15.attention.self.query.bias', 'encoder.layer.15.attention.self.key.weight', 'encoder.layer.15.attention.self.key.bias', 'encoder.layer.15.attention.self.value.weight', 'encoder.layer.15.attention.self.value.bias', 'encoder.layer.15.attention.output.dense.weight', 'encoder.layer.15.attention.output.dense.bias', 'encoder.layer.15.attention.output.LayerNorm.weight', 'encoder.layer.15.attention.output.LayerNorm.bias', 'encoder.layer.15.intermediate.dense.weight', 'encoder.layer.15.intermediate.dense.bias', 'encoder.layer.15.output.dense.weight', 'encoder.layer.15.output.dense.bias', 'encoder.layer.15.output.LayerNorm.weight', 'encoder.layer.15.output.LayerNorm.bias', 'encoder.layer.16.attention.self.query.weight', 'encoder.layer.16.attention.self.query.bias', 'encoder.layer.16.attention.self.key.weight', 'encoder.layer.16.attention.self.key.bias', 'encoder.layer.16.attention.self.value.weight', 'encoder.layer.16.attention.self.value.bias', 'encoder.layer.16.attention.output.dense.weight', 'encoder.layer.16.attention.output.dense.bias', 'encoder.layer.16.attention.output.LayerNorm.weight', 'encoder.layer.16.attention.output.LayerNorm.bias', 'encoder.layer.16.intermediate.dense.weight', 'encoder.layer.16.intermediate.dense.bias', 'encoder.layer.16.output.dense.weight', 'encoder.layer.16.output.dense.bias', 'encoder.layer.16.output.LayerNorm.weight', 'encoder.layer.16.output.LayerNorm.bias', 'encoder.layer.17.attention.self.query.weight', 'encoder.layer.17.attention.self.query.bias', 'encoder.layer.17.attention.self.key.weight', 'encoder.layer.17.attention.self.key.bias', 'encoder.layer.17.attention.self.value.weight', 'encoder.layer.17.attention.self.value.bias', 'encoder.layer.17.attention.output.dense.weight', 'encoder.layer.17.attention.output.dense.bias', 'encoder.layer.17.attention.output.LayerNorm.weight', 'encoder.layer.17.attention.output.LayerNorm.bias', 'encoder.layer.17.intermediate.dense.weight', 'encoder.layer.17.intermediate.dense.bias', 'encoder.layer.17.output.dense.weight', 'encoder.layer.17.output.dense.bias', 'encoder.layer.17.output.LayerNorm.weight', 'encoder.layer.17.output.LayerNorm.bias', 'encoder.layer.18.attention.self.query.weight', 'encoder.layer.18.attention.self.query.bias', 'encoder.layer.18.attention.self.key.weight', 'encoder.layer.18.attention.self.key.bias', 'encoder.layer.18.attention.self.value.weight', 'encoder.layer.18.attention.self.value.bias', 'encoder.layer.18.attention.output.dense.weight', 'encoder.layer.18.attention.output.dense.bias', 'encoder.layer.18.attention.output.LayerNorm.weight', 'encoder.layer.18.attention.output.LayerNorm.bias', 'encoder.layer.18.intermediate.dense.weight', 'encoder.layer.18.intermediate.dense.bias', 'encoder.layer.18.output.dense.weight', 'encoder.layer.18.output.dense.bias', 'encoder.layer.18.output.LayerNorm.weight', 'encoder.layer.18.output.LayerNorm.bias', 'encoder.layer.19.attention.self.query.weight', 'encoder.layer.19.attention.self.query.bias', 'encoder.layer.19.attention.self.key.weight', 'encoder.layer.19.attention.self.key.bias', 'encoder.layer.19.attention.self.value.weight', 'encoder.layer.19.attention.self.value.bias', 'encoder.layer.19.attention.output.dense.weight', 'encoder.layer.19.attention.output.dense.bias', 'encoder.layer.19.attention.output.LayerNorm.weight', 'encoder.layer.19.attention.output.LayerNorm.bias', 'encoder.layer.19.intermediate.dense.weight', 'encoder.layer.19.intermediate.dense.bias', 'encoder.layer.19.output.dense.weight', 'encoder.layer.19.output.dense.bias', 'encoder.layer.19.output.LayerNorm.weight', 'encoder.layer.19.output.LayerNorm.bias', 'encoder.layer.20.attention.self.query.weight', 'encoder.layer.20.attention.self.query.bias', 'encoder.layer.20.attention.self.key.weight', 'encoder.layer.20.attention.self.key.bias', 'encoder.layer.20.attention.self.value.weight', 'encoder.layer.20.attention.self.value.bias', 'encoder.layer.20.attention.output.dense.weight', 'encoder.layer.20.attention.output.dense.bias', 'encoder.layer.20.attention.output.LayerNorm.weight', 'encoder.layer.20.attention.output.LayerNorm.bias', 'encoder.layer.20.intermediate.dense.weight', 'encoder.layer.20.intermediate.dense.bias', 'encoder.layer.20.output.dense.weight', 'encoder.layer.20.output.dense.bias', 'encoder.layer.20.output.LayerNorm.weight', 'encoder.layer.20.output.LayerNorm.bias', 'encoder.layer.21.attention.self.query.weight', 'encoder.layer.21.attention.self.query.bias', 'encoder.layer.21.attention.self.key.weight', 'encoder.layer.21.attention.self.key.bias', 'encoder.layer.21.attention.self.value.weight', 'encoder.layer.21.attention.self.value.bias', 'encoder.layer.21.attention.output.dense.weight', 'encoder.layer.21.attention.output.dense.bias', 'encoder.layer.21.attention.output.LayerNorm.weight', 'encoder.layer.21.attention.output.LayerNorm.bias', 'encoder.layer.21.intermediate.dense.weight', 'encoder.layer.21.intermediate.dense.bias', 'encoder.layer.21.output.dense.weight', 'encoder.layer.21.output.dense.bias', 'encoder.layer.21.output.LayerNorm.weight', 'encoder.layer.21.output.LayerNorm.bias', 'encoder.layer.22.attention.self.query.weight', 'encoder.layer.22.attention.self.query.bias', 'encoder.layer.22.attention.self.key.weight', 'encoder.layer.22.attention.self.key.bias', 'encoder.layer.22.attention.self.value.weight', 'encoder.layer.22.attention.self.value.bias', 'encoder.layer.22.attention.output.dense.weight', 'encoder.layer.22.attention.output.dense.bias', 'encoder.layer.22.attention.output.LayerNorm.weight', 'encoder.layer.22.attention.output.LayerNorm.bias', 'encoder.layer.22.intermediate.dense.weight', 'encoder.layer.22.intermediate.dense.bias', 'encoder.layer.22.output.dense.weight', 'encoder.layer.22.output.dense.bias', 'encoder.layer.22.output.LayerNorm.weight', 'encoder.layer.22.output.LayerNorm.bias', 'encoder.layer.23.attention.self.query.weight', 'encoder.layer.23.attention.self.query.bias', 'encoder.layer.23.attention.self.key.weight', 'encoder.layer.23.attention.self.key.bias', 'encoder.layer.23.attention.self.value.weight', 'encoder.layer.23.attention.self.value.bias', 'encoder.layer.23.attention.output.dense.weight', 'encoder.layer.23.attention.output.dense.bias', 'encoder.layer.23.attention.output.LayerNorm.weight', 'encoder.layer.23.attention.output.LayerNorm.bias', 'encoder.layer.23.intermediate.dense.weight', 'encoder.layer.23.intermediate.dense.bias', 'encoder.layer.23.output.dense.weight', 'encoder.layer.23.output.dense.bias', 'encoder.layer.23.output.LayerNorm.weight', 'encoder.layer.23.output.LayerNorm.bias', 'pooler.dense.weight', 'pooler.dense.bias', 'new_classifiers.ocnli.weight', 'new_classifiers.ocnli.bias', 'new_classifiers.tnews.bias', 'new_classifiers.tnews.weight', 'new_classifiers.ocemotion.weight', 'new_classifiers.ocemotion.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

您好 ， 关于咱们这个项目和 Google electra 的区别。 我个人理解就是两个不同 一个是我们提供了已有的中文预训练模型， 另一个是词表 我们用的21128 也即是bert中文的那个词表 。假如我们要继续预训练 我们要用 build_dataset.py 用21128词表产生tfrecord 然后进行。 其余的 比如 run_pretraining.py run_finetuning.py 和 configure_finetune configure_pretrain 几个文件都和google那个英文版完全一样的 请问是这样吗。 谢谢

您好，下载模型及相关文件时（https://huggingface.co/hfl/chinese-electra-180g-small-discriminator/tree/main）未发现tokenizer.json，能否生成然后上传一下。 看google/electra-base-discriminator(https://huggingface.co/google/electra-base-discriminator)是有tokenizer.json的，这个tokenizer.json是不能直接用于chinese-electra-180g-small-discriminator模型吧

ELECTRA由生成器+判决器构成，生成器负责把[MASK]替换成实际tokens，判决器负责区分替换结果是否和实际data中相同

但是，论文提及：

1. Typically k = [0.15n], i.e., 15% of the tokens are masked out

2. if the generator happens to generate the correct token, that token is considered “real” instead of “fake

也就是说，只有 (1 - generator_inference_acc) * 0.15的token会被预测成fake，训练到后期就是一个极度不均衡的二分类问题（对于判决器而言），为什么判决器不会受到影响呢？

Hello Maintainer of Github repo ymcui/Chinese-ELECTRA!

Thank you for your work on ymcui/Chinese-ELECTRA. This GitHub project is interesting, and we think that it would be a great addition to make this work instantly discoverable & available as an API for all your users, to quickly try and use it in their applications.

The list of model card(s) covered by this PR are:

• https://console.tiyaro.ai/explore/hfl-chinese-bert-wwm
• https://console.tiyaro.ai/explore/hfl-chinese-bert-wwm-ext
• https://console.tiyaro.ai/explore/hfl-chinese-macbert-base
• https://console.tiyaro.ai/explore/hfl-chinese-roberta-wwm-ext
• https://console.tiyaro.ai/explore/hfl-chinese-roberta-wwm-ext-large
• https://console.tiyaro.ai/explore/hfl-rbt3
• https://console.tiyaro.ai/explore/hfl-rbt6

On Tiyaro, every model in ymcui/Chinese-ELECTRA will get its own:

• Dedicated model card (e.g. https://console.tiyaro.ai/explore/hfl-chinese-bert-wwm
• Model demo (e.g. https://console.tiyaro.ai/explore/hfl-chinese-bert-wwm/demo)
• Unique Inference API (e.g. https://api.tiyaro.ai/explore/huggingface/1//hfl/chinese-bert-wwm)
• Sample code snippets and swagger spec for the API

Users will also be able to compare your model with other models of similar types on various parameters using Tiyaro Experiments (https://tiyaro.ai/blog/ocr/)

—- I am from Tiyaro.ai (https://tiyaro.ai/). We are working on enabling developers to instantly evaluate, use and customize the world’s best AI. We are constantly working on adding new features to Tiyaro EasyTrain, EasyServe & Experiments, to make the best use of your ML model, and making AI more accessible for anyone.

Sincerely, I-Jong Lin

环境：

PaddleHub 1.8.1 PaddlePaddle 1.8.4 paddle-gpu-serving >=0.8.2 ujson >=1.35

# coding: utf8
from paddlehub.serving.bert_serving import bs_client

if __name__ == "__main__":
    # 初始化bert_service客户端BSClient
    bc = bs_client.BSClient(module_name="ernie", server="127.0.0.1:8866")

    # 输入要做embedding的文本
    # 文本格式为[["文本1"]]
    input_text = [
        ["西风吹老洞庭波"]
    ]

    # BSClient.get_result()获取结果
    result = bc.get_result(input_text=input_text)

    # 打印输入文本的embedding结果
    for item in result:
        print(item)

在使用 bert_service_client 时，引入erine 模型的 get_result是一个数组，但是 chinese-electra-base 就只有一个值, 请问是为什么呢？然后可以让 chinese-electra-base 模型的输出结果也是一个和 erine 结果维度一样的数组吗？

在Bert中若处理输入为两个句子的相关任务（例如语义相似性打分等），常使用token_type_embedding对两个句子分别加上不同的embedding；这一做法只需要在transformers的API中设置token_type_id（一个句子全为0，另一个全为1）即可实现。 然而，electra的预训练好像取消了NSP任务，相应的也就没有训练这个句子embedding（抱歉我不是很确定，只是看了一下论文好像没写这一点😂），所以我想请教一下使用token_type_id这一做法是否在electra中也可以通用呢？如果不行，对于两个句子的输入，推荐的处理方法是什么呢？谢谢！

提供开源模型的老师，您好，我通过讯飞云下载的模型， 解析ELECTRA-small模型发现有五个权重找不到。 error electra/encoder/layer_9/intermediate/dense/kernel error electra/encoder/layer_9/output/LayerNorm/beta error electra/encoder/layer_9/output/LayerNorm/gamma error electra/encoder/layer_9/output/dense/bias error electra/encoder/layer_9/output/dense/kernel

请检查是否存在该问题，十分感谢。

换了个源下载是ok的，抱歉

解压后发现没有json文件，而不少框架都是根据json文件来读取模型基本结构的，建议还是配上。

比如small版

{
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 256,
  "initializer_range": 0.02,
  "intermediate_size": 1024,
  "max_position_embeddings": 512,
  "num_attention_heads": 4,
  "num_hidden_layers": 12,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 21128,
  "embedding_size": 128
}

还有，不知道为啥ckpt的命名不加上ckpt...

最后，最新版bert4keras（0.6.4）已经能加载electra了，只需要在build_transformer_model里边传入model='electra'，欢迎用bert4keras调用哈哈～