Hi,

Thank you for the great work! This work is really wonderful. When I try to use GCN model for node classification by running: python main_SBMs_node_classification.py --dataset SBM_PATTERN --gpu_id 0 --seed 41 --config 'configs/SBMs_node_clustering_GCN_PATTERN_100k.json' I found that when I set dgl_builtin to false, the test acc is 63.77, which is consistent with the results reported in the paper; however, when I set dgl_builtin to true, the test acc became 85.56.

I do not think this behavior is normal. But I did not figure out why the performances are so different after struggling for some time. I would appreciate it if you could help me. Thank you! Have a nice day!

Best, Yongcheng

Hi, I encountered an error when I loaded the MNIST.pkl and CIFAR10.pkl

<stdout>:cuda available with GPU: Tesla P100-PCIE-16GB
<stdout>:[I] Loading dataset CIFAR10...
<stderr>:Traceback (most recent call last):
<stderr>:  File "tools/train_superpixels_graph_classification.py", line 444, in <module>
<stderr>:    main() 
<stderr>:  File "tools/train_superpixels_graph_classification.py", line 325, in main
<stderr>:    dataset = LoadData(DATASET_NAME, data_dir=args.dataDir)
<stderr>:  File "/var/storage/shared/nextmsra/sys/jobs/application_1583577754071_28152/gnn/data/data.py", line 18, in LoadData
<stderr>:    return SuperPixDataset(DATASET_NAME, data_dir)
<stderr>:  File "/var/storage/shared/nextmsra/sys/jobs/application_1583577754071_28152/gnn/data/superpixels.py", line 271, in __init__
<stderr>:    f = pickle.load(f)
<stderr>:  File "/tmp/cache/python/lib/python3.7/site-packages/dgl/graph_index.py", line 53, in __setstate__
<stderr>:    num_nodes, readonly, src, dst = state
<stderr>:ValueError: too many values to unpack (expected 4)

Here is my environment. cuda version :10.0 pytorch version : pytorch 1.4 with cudatoolkit 10.0 dgl version: v0.4.3

I wonder if the new dgl version causes the inconsistency. Because I can run well when dgl is v0.4.2., but now I don't find how to downgrade my dgl to v0.4.2 explicitly. Thanks for solving this :) Or can you publish your code of how to generate the dgl graph from CIFAR and MNIST dataset? Thank you

Hi,

First of all, congratulations on your great work.

Maybe I've caught a minor mistake. It seems that you're leaving out the nearest neighbor when computing the edges list in the file data/superpixels.py (method: compute_edges_list):

knns = np.argpartition(A, new_kth - 1, axis=-1)[:, new_kth:-1] 
knn_values = np.partition(A, new_kth - 1, axis=-1)[:, new_kth:-1]  

I think it should be

knns = np.argpartition(A, new_kth, axis=-1)[:, new_kth+1:] 
knn_values = np.partition(A, new_kth, axis=-1)[:, new_kth+1:]  

Could you please verify that?

Thanks.

I have a question about these two datasets. there are three subset trainset, valset, and testset. How to combine these datasets into one whole set? Thank you.

Hi, thanks for creating this awesome project. However, I've encountered some problems on reproduction of results present in the paper and the leaderboard. Generally speaking, results are very unstable in my experiments.

1. Results fluctuate heavily.

On ZINC dataset, here are my results of GAT model :

• Trial 1: Mean test MAE = 0.38158. Under each seed (95, 41, 12, 35 and the same below): 0.3972, 0.3788, 0.3800, 0.3703
• Trial 2: Mean test MAE = 0.37115. Under each seed: 0.3636, 0.3785, 0.3769, 0.3656.

My results of GCN model:

• Trial 1: Mean test MAE = 0.37575. Under each seed: 0.3770, 0.3659, 0.3742, 0.3859
• Trial 2: Mean test MAE = 0.38070. Under each seed: 0.3946, 0.3668, 0.3764, 0.3850

I think we can not neglect these fluctuating errors since sometimes errors are around 0.01. And some recent SOTA models only outperform 0.01 over the best model in the benchmark paper.

It seems setting random seeds does not really work in this case. Is there any other requirements for reproduction? Do you think such fluctuation is tolerable?

2. I cannot reproduce results using checkpoints.

After I trained a GCN model on ZINC dataset, I reload the latest checkpoint and then try to reproduce test MAE through evaluate_network_sparse function. Unfortunately, test MAE from the checkpoint (0.3670) is different than test MAE from the result file (0.3757). Can we reproduce results via checkpoints? Could you please provide me some code of testing pipeline in case that I missed some important steps during reproduction?

Note: I changed readout to 'sum'. Besides, I didn't modify other hyper-parameters. My experiments are conducted on Quadro P6000.

Hello, which GNN model can be applied on weighted graph? like take into consideration the edge features/weights? and is there any GNN for unsupervised clustering? and what's the largest my graph can be?(not semi supervised). Your answer will be much appreciated :))

Thanks for providing the benchmark!

As a Ph.D student working on GNNs in China, I'd like to ask you considering providing additional mirror for the datasets, e.g., on github or other websites, since dropbox may not be accessble for us.

(Just to mention, similar problems have happend in other packages as well, e.g., see https://github.com/rusty1s/pytorch_geometric/issues/1116#issuecomment-612580946)

Thank you for your work in providing a standard repository for graph benchmarking

Some applications might require using the SMILES to build different types of graphs than the one provided by the benchmarking platform. I know that the ZINC dataset come from the JT-VAE paper where the SMILES are provided. However, this paper is only a subset of the original dataset, and the train-val-test split is different.

I tried going from the DGLGraph back to SMILES, but it is not possible since I don't know which node label corresponds to which atom.

This PR changes the reducers of GraphSage/GCN/GatedGCN from user-defined functions to builtin functions. Such changes brought a significant speedup for both 0.5.2 and 0.4.2 on these models:

EDIT: The dataset is SBM_PATTERN-100k and I ran the benchmark on a g3.16x AWS instance.

| Model | 0.4.2 w/ UDF | 0.4.2 w/ builtin reducers | DGL-master w/ UDF | DGL-master w/ builtin reducers | |:-:|:-:|:-:|:-:|:-:| | GraphSage | 326.21 | 43.0335 | 316.99 | 17.8404 | | GatedGCN | 573.42 | 207.3135 | 598.05 | 177.4308 |

DGL-master represents the current master branch of DGL that incorporated some performance optimization on graph batching and readouts (today). The code should be available in the upcoming nightly build 201125.

This PR is for DGL 0.5.2. I'll submit a separate PR for 0.4.2 once this PR has been reviewed since the code changes are exactly the same.

Is it possible to add a graph feature (a vector for each graph) to be used togheter with the graph itself to predict the label, in the graph classification/ graph regression settings? If it is, how can I do so? I'm not very experienced with torch and tensorflow. Thank you in advance.

Hi, I see your code in https://github.com/graphdeeplearning/benchmarking-gnns/blob/92c762a5a8713f5804c69c1808128a6b3e8f4535/data/superpixels.py#L114-L144

, find the superpixel graph is directed graph not an undirected graph? Isn't it true? I see Figure 1 of your paper, you draw MNIST and CIFAR10 as undirected graphs, but it should be directed graphs based on your code?

Thank you for your work in such a meaningful job.

I tried going from zinc SMILES to .pkl file, but it is not possible since I don't know how to get the right format.Could you please provide the code about how to prepare the data?

Thanks for your time.

Just as the title said, when I run the 'main_superpixels_graph_classification.py', I got the error"'Column' object has no attribute 'deferred_dtype'" in the file 'superpixels.py' when getting the feature of the graph data at lin290. line290for idx, graph in enumerate(graphs): line291 graphs[idx].ndata['feat'] = graph.ndata['feat'].float() line292 graphs[idx].edata['feat'] = graph.edata['feat'].float() line293batched_graph = dgl.batch(graphs) Unfortunately, I can't tackle this by myself. Have you encountered the same error? I indeed need your help. Thank you greatly!