Dataset IMDB has multi labels. But I am wondering how to save the file for evaluation, cause I don't see it in run_multi.py. And else, I want to know what API in sklearn to evaluate multi-label.

the result just can't reach the result in the paper. When I ran IMDB, the result was like macro-f1=0.44, micro-f1=0.56 or so. I use the parameter described in the paper. "We set 𝑑 = 64, 𝑛ℎ = 8 for all datasets. For IMDB, we set 𝑠 = 0.1 and 𝐿 = 5. We use feat = 0 for IMDB."

I wonder why this happened. Thanks for the attention

Thanks for your team's good work. But I have a doubt about the last L2 norm trick used in Simple HGN. From the paper's ablation study, we can see that the performance of Simple HGN drops greatly without last L2 norm trick(even decrease 5-8% , it seems that the high performance of Simple HGN comes from this trick), which is quite strange.

Also, the website https://www.biendata.xyz/competition/hgb-1 has a problem recently. When I make a submission, the website will stop at "evaluating" for quite a long time and won't return the results. Can you check it? Thank you

After I made a submission to node classification leaderboard, the interface is stuck at 100% upload and a greyed-out "Evaluating" button forever without returning any results. Is there any issue in the evaluation server? Submitting to link prediction worked fine. Thanks!

您好，我在上传结果到节点分类排行榜时，界面卡在了100%上传，没有返回评测结果。上传到链路预测的时候没有问题。请问是否是评测服务器有问题？谢谢！

(Possible duplicate of #14 )

您好，在论文中，ACM的元路径设置如图，但在代码中 HAN 的meta-path不是这样https://github.com/THUDM/HGB/blob/master/NC/benchmark/methods/HAN/utils.py#L170 这需要自己改动吗

以及我跑了benchmark中HAN的代码，按照代码的路径设置，验证集最好 macro f1 为 0.87，micro f1 为 0.96，5 次后提交到 biendata，测试集的 macro f1 和 micro f1 都是 0.86 左右，与论文的 90.8% 左右还是相差较多 如果把路径按论文设置（即没有 ptp，添加额外的引用/参考关系），验证集最好 macro f1 为 0.78，micro f1 为 0.90

不知道其他人跑 ACM 结果如何，请问 ACM 的实验有什么要注意的吗

Dear authors,

I find this repo only provides datasets which are available to download but doe not provide the preprocessing code for the datasets.

I have a question about the dataset ACM and Freebase, I wonder what is the input features for target type nodes and how you assign features for other non-target-type nodes? Thanks!

Hi, I was checking the convolution, and apparently there are expensive layers there that can be completely eliminated:

The code is:

e_feat = self.edge_emb(e_feat)
e_feat = self.fc_e(e_feat).view(-1, self._num_heads, self._edge_feats)
ee = (e_feat * self.attn_e).sum(dim=-1).unsqueeze(-1)
el = (feat_src * self.attn_l).sum(dim=-1).unsqueeze(-1)
er = (feat_dst * self.attn_r).sum(dim=-1).unsqueeze(-1)
graph.srcdata.update({'ft': feat_src, 'el': el})
graph.dstdata.update({'er': er})
graph.edata.update({'ee': ee})
graph.apply_edges(fn.u_add_v('el', 'er', 'e'))
e = self.leaky_relu(graph.edata.pop('e')+graph.edata.pop('ee'))

Problem 1:

self.edge_emb = nn.Embedding(num_etypes, edge_feats)
e_feat = self.edge_emb(e_feat)
e_feat = self.fc_e(e_feat).view(-1, self._num_heads, self._edge_feats)

Is it necessary to run a fully connected layer over embeddings? As far as I understand, the embeddings can naturally learn the same projection emb = self.fc_c(emb). This becomes even more expensive when we think that the conv might have only 20 types of edges, but it is running this fully connected layer hundreds of thousands of times for the same repeated 20 types.

Problem 2:

self.attn_l = nn.Parameter(th.FloatTensor(size=(1, num_heads, out_feats)))
self.attn_r = nn.Parameter(th.FloatTensor(size=(1, num_heads, out_feats)))
self.attn_e = nn.Parameter(th.FloatTensor(size=(1, num_heads, edge_feats)))

ee = (e_feat * self.attn_e).sum(dim=-1).unsqueeze(-1)
el = (feat_src * self.attn_l).sum(dim=-1).unsqueeze(-1)
er = (feat_dst * self.attn_r).sum(dim=-1).unsqueeze(-1)

e_feat, feat_src and feat_dst are the products of an MLP. Is it necessary to multiply it by a constant (called attention here)? I guess the MLP can naturally achieve the same value. We can just say that:

y = (a * x+ b)  
y = d(a*x + b)
y = d*a*x + b*d
y = new_a*x + new_b

If you remove these two parts, then attention can be calculated just as right + left + edge_emb (graph.apply_edges(fn.u_add_v('feat_src', 'feat_dst', 'e_feat'))), without doing all these transformations beforehand.