Hi,

Great work!

I want to confirm whether my understanding of equations 11~12 is correct.

I understand equation 12 in this way: (Q h_i * K h_j / sqrt(d_k)) is a scalar, and (E e_ij) is a d_k-dim vector. Then a scalar multiplying a vector gives a d_k-dim vector. In equation 11, this d_k-dim vector is transformed to a scalar by computing w_1+w_2+...+w_dk. Is it correct?

Hi Vijay,

Thanks for your repo!

Question: I see your doing sign flipping of eigen pos_enc during training, but it seems that you are not doing so during eval time. I understand that we want to make deterministic predictions so we don't have random flipping when evaluating it. Do you have further comments or justification for this?

Best Kezhi

Hi ! Congratulations for your paper and thank you for making the implementation publicly available as well.

Quick question on this function :

    def func(edges):
        return {out_field: (edges.src[src_field] * edges.dst[dst_field])}
    return func

Why do you do a multiplication of K and Q and not a dot product? The dimensions of the scores are [num_edges, num_heads, hidden_dim/num_heads]. But I expect a [num_edges,num_edges] matrix .

You can also reach me here : [email protected] Hope to hear from you soon , Pietro Bonazzi

Hello there, First of all, thank you for providing such an amazing work. I'd like to know how can I leverage graphtransformer on Graph Classification task with textual data, for instance, I first extract nodes and edge info from the text data, given node features and edge information (only one type of edge in my case), the model generate binary targets based on those given features.

Kind Regards Michael

For graphs with large difference in the number of vertices, how to determine the value of pos_enc_dim? For example, the number of vertices of a graph is 7, and the number of vertices of another graph is 3? Do you take the pos_enc_dim is 8 because the number of vertices in your experimental data set is greater than 8?

Thanks for the innovative work! Could you please tell me how can we get a full graph？Did full graph mean the full attention map？Did sparse graph mean that we only retain the immediate neighbor nodes‘ value of full graph？

Thanks for sharing codes for your interesting paper. I'm interested in applying this method to MNIST superpixel dataset (your benchmarking work) where the task is graph classification, graphs have both node features and edge features, and the number of nodes/edges are different between graphs. What modifications should I made to the current code?

Hello, about the problem of calculating attention, the attention of node i and its adjacent nodes is calculated in the formula, but I find that the final calculation is the attention of all nodes, and it does not distinguish whether the nodes are connected. Is there a problem with my understanding?

Great work!

I have a question concerning the implementation of softmax in the graph_transformer_edge_layer.py

When you define the softmax, you use the following function:

def exp(field):
    def func(edges):
        # clamp for softmax numerical stability
        return {field: torch.exp((edges.data[field].sum(-1, keepdim=True)).clamp(-5, 5))}
    return func

Shouldn't the attention weights/scores be scalars? From what I see, each head has an 8-dimensional score vector which you then compute .sum() on. The graph_transformer_layer.py layer does not have this .sum() function.

def scaled_exp(field, scale_constant):
    def func(edges):
        # clamp for softmax numerical stability
        return {field: torch.exp((edges.data[field] / scale_constant).clamp(-5, 5))}

    return func

Would appreciate any clarification on this :)

Best, Devin

Hi, thanks for the great paper :)

I was just curious as to what the 'z' variable is in line 59 of the graph_transformer_layer.py code? I cannot seem to find the equivalent in the paper. It seems you are normalizing the output heads by the sum of the attention weights?

Would appreciate a little point :)

Thanks, Devin

I'm trying to setup the environment on mac, followed your instructions carefully however I got the below error when building the env by conda: ResolvePackageNotFound:

• h5py=2.9.0
• tensorboard=1.14.0
• requests==2.22.0
• ipython=7.7.0
• ipykernel=5.1.2
• notebook=6.0.0
• pip=19.2.3
• scikit-image=0.15.0
• scipy=1.3.0
• torchvision==0.7.0
• pytorch=1.6.0
• matplotlib=3.1.0
• pillow==6.1
• mkl=2019.4
• dgl=0.6.1
• scikit-learn=0.21.2
• python=3.7.4

g.send_and_recv(eids, fn.src_mul_edge('V_h', 'score', 'V_h'), fn.sum('V_h', 'wV')) this only update target node; head_out = g.ndata['wV']/g.ndata['z'] so this only update target node;the source node not update?

Hi there,

I was reading your code on graphtransformer, I'm kind of curious on the operation shown below. Why did you divide the wV score by the w(or so called 'score' term), I didn't see any terms shown in your equation 4 or equation 9 in the paper. Could you illustrated that? https://github.com/graphdeeplearning/graphtransformer/blob/c9cd49368eed4507f9ae92a137d90a7a9d7efc3a/layers/graph_transformer_edge_layer.py#L112

Thanks

Could you provide some additional information about the memory consumption using your Graph Transformer?

You state, that sparse-attention favors both computation time and memory consumption, but do not provide actual measurements of the second property in your evaluation or do not state clearly, if and how your implementation is able to take advantage of it. Some peak memory measurements of your experiments as an addendum to your evaluation of the computation times (e.g. Table 1) could be beneficial to others, too. As in my case the quadratic growth of the memory consumption w.r.t. the sequence length prevents an efficient use of Transformers for some task, where connectivity information is given and can be simply modeled by masking out (-Inf) the attention scores in the attention matrix.

Also some exemplary or artificial data could be interesting, e.g. (Mean) number of nodes n = {128, 1024, 2048, 4096}, (mean) number of edges per node e = {4, 8, 16, 64, 128}, to get an impression of the resource consumption of your Graph Transformer with Sparse Graph vs. NLP-Transformer (Full Graph with masking).

(I hope, that I could run the experiments myself, but I suppose your evaluation pipeline is already running and data provided by the original author should be more precise and trustworthy to other researchers, too.)