2021搜狐校园文本匹配算法大赛baseline
分享了一个搜狐文本匹配的baseline，主要是通过条件LayerNorm来增加模型的多样性，以实现同一模型处理不同类型的数据、形成不同输出的目的。
线下验证集F1约0.74，线上测试集F1约0.73。预训练模型是RoFormer，也欢迎对比其他预训练模型的效果。
测试环境：tensorflow 1.14 + keras 2.3.1 + bert4keras 0.10.5，如果在其他环境组合下报错，请根据错误信息自行调整代码。
详情请看：https://kexue.fm/archives/8337
QQ交流群：808623966，微信群请加机器人微信号spaces_ac_cn
my env configuration: keras==2.3.1, tensorflow-GPU==2.2.0
I try to support multi-GPUs in one machcine, so I add simple code as below to include all model related codes:
strategy = tf.distribute.MirroredStrategy() print('Number of devices: {}'.format(strategy.num_replicas_in_sync)) with strategy.scope(): xxx xxx xxx xxx
I also try to set os.environ['TF_KERAS'] to "0" or "1".
I can see the process in two GPUs, but the last GPU-Util is always 0% as below:
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla P40 Off | 00000000:00:0E.0 Off | 0 | | N/A 40C P0 155W / 250W | 21699MiB / 22919MiB | 100% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla P40 Off | 00000000:00:0F.0 Off | 0 | | N/A 31C P0 49W / 250W | 21659MiB / 22919MiB | 0% Default | +-------------------------------+----------------------+----------------------+
so what else need I do if I want to train in multi-GPUs???
During the training, It seems evaluate() always returns 0 result.
so if it has problems with these accuracy caculation: total_a += ((y_pred + y_true) * (flag == 0)).sum() right_a += ((y_pred * y_true) * (flag == 0)).sum() total_b += ((y_pred + y_true) * (flag == 1)).sum() right_b += ((y_pred * y_true) * (flag == 1)).sum() should be? total_a += (flag == 0).sum() right_a += ((y_pred == y_true) * (flag == 0)).sum() total_b += (flag == 1).sum() right_b += ((y_pred == y_true) * (flag == 1)).sum()
and f1 should remove "2*" also : f1_a = right_a / total_a f1_b = right_b / total_b
