I install Graphormer follow the guide:

1. To create and activate a conda environment with Python3.9

conda create -n graphormer python=3.9
conda activate graphormer

2. Run the following commands

git clone --recursive https://github.com/microsoft/Graphormer.git
cd Graphormer
bash install.sh

3. To train a Graphormer-slim on ZINC-500K on a single GPU card

cd examples/property_prediction/
bash zinc.sh

I ran this command for half an hour and got no response: Occasional reactions occur, but the following error is reported:

2022-03-17 17:00:38 | WARNING | root | The OGB package is out of date. Your version is 1.3.2, while the latest ve
rsion is 1.3.3.
Traceback (most recent call last):
  File "/home/linjiayi/anaconda3/envs/graphormer/bin/fairseq-train", line 8, in <module>
    sys.exit(cli_main())
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq_cli/train.py", line 512, in
cli_main
    parser = options.get_training_parser()
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq/options.py", line 38, in get
_training_parser
    parser = get_parser("Trainer", default_task)
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq/options.py", line 234, in ge
t_parser
    utils.import_user_module(usr_args)
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq/utils.py", line 497, in impo
rt_user_module
    import_tasks(tasks_path, f"{module_name}.tasks")
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq/tasks/__init__.py", line 117
, in import_tasks
    importlib.import_module(namespace + "." + task_name)
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/linjiayi/Graphormer/graphormer/tasks/graph_prediction.py", line 25, in <module>
    from ..data.dataset import (
  File "/home/linjiayi/Graphormer/graphormer/data/dataset.py", line 9, in <module>
    from .wrapper import MyPygGraphPropPredDataset
  File "/home/linjiayi/Graphormer/graphormer/data/wrapper.py", line 6, in <module>
    from ogb.graphproppred import PygGraphPropPredDataset
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/ogb/graphproppred/__init__.py", line
 5, in <module>
    from .dataset_pyg import PygGraphPropPredDataset
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/ogb/graphproppred/dataset_pyg.py", l
ine 1, in <module>
    from torch_geometric.data import InMemoryDataset
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch_geometric/__init__.py", line 5
, in <module>
    import torch_geometric.data
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch_geometric/data/__init__.py", l
ine 1, in <module>
    from .data import Data
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch_geometric/data/data.py", line
8, in <module>
    from torch_sparse import coalesce, SparseTensor
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch_sparse/__init__.py", line 41,
in <module>
    from .tensor import SparseTensor  # noqa
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch_sparse/tensor.py", line 13, in
 <module>
    class SparseTensor(object):
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch/jit/_script.py", line 1128, in
 script
    _compile_and_register_class(obj, _rcb, qualified_name)
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch/jit/_script.py", line 138, in
_compile_and_register_class
    script_class = torch._C._jit_script_class_compile(qualified_name, ast, defaults, rcb)
RuntimeError:
object has no attribute sparse_csr_tensor:
  File "/home/linjiayi/anaconda3/envs/graphormer/lib/python3.9/site-packages/torch_sparse/tensor.py", line 511
            value = torch.ones(self.nnz(), dtype=dtype, device=self.device())

        return torch.sparse_csr_tensor(rowptr, col, value, self.sizes())
               ~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

I try to train a Graphormer-base on PCQM4M dataset on multiple GPU cards using bash pcqv1.sh, no response either. Is there a problem with the data set download? How to solve the problem?

Hi authors, thanks for your great work. As I have been trying to reproduce the No.1 result on ogb-pcba board, I didn't find the checkpoints mentioned in your paper that pretrained for the PCBA task. Therefore, I turned to use the PCQM checkpoint you provided for the PCQM task. But during loading the checkpoints, an error occured even I have set the hidden dimension and ffn dimension from 1024 to 768: `RuntimeError: Error(s) in loading state_dict for Graphormer:

    size mismatch for atom_encoder.weight: copying a param with shape torch.Size([4737, 768]) from checkpoint, the shape in current model is torch.Size([4609, 768]).

    size mismatch for edge_encoder.weight: copying a param with shape torch.Size([769, 32]) from checkpoint, the shape in current model is torch.Size([1537, 32]).`

Thus, may I ask two questions about the reproducing process:

1. Can you provide the checkpoints that can be used to reproduce the PCBA result?
2. Is there a reason why the code cannot load the previous PCQM checkpoints even though having changed the ffn and hidden dimension?

Looking forward to your reply. Thank you!

I have completed the installation with Conda (using the install.sh script), but could not successfully run the example script. When I run the pcqv2.sh script, it just hangs without any output or any error message. I'm not sure anybody else has faced the same problem. Can you give me some advice to resolve the issue?

For more information, I'm using the GCP with NVIDIA V100 GPUs and CUDA11.1. Within the same environment, I have checked running the fairseq NMT example codes and the Graphormer v1. codes. Both of them did not produce any errors.

Hi! This is Stella from Seoul National University, I'm getting a lot of help from your code. I have a question about entry.py line 87. Originally it has metric = 'valid_' + get_dataset(dm.dataset_name)['metric'] but when I run model, I faced error like this: 'pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='valid_mae') not found in the returned metrics: ['train_loss']. HINT: Did you call self.log('valid_mae', value) in the LightningModule?'

So I changed the line 87 as metric = 'train_loss' and it runs well.

I'm quite afraid that I'm doing something wrong, is it right way to modify the code? Here are some useful information for my project:

1. task: regression
2. input type : integer (originally continuous value, but discretized)
3. target type : real value
4. eval metric : rmse
5. features from data.py

•         'num_class': 1,
  

•         'loss_fn': F.l1_loss,
  

•         'metric': 'mae',
  

•         'metric_mode': 'min',
  

Hi! this is Stella from Seoul National University. I'd like to ask how can I implement regression task on Graphormer. I adjusted ogb module for our data, and setted num_class as -1 like other regression datasets. And I faced problem with editing model dimensions at model.py, line 62 ~ 75. I think that 512*9+1 is something like vocabulary size, which is calculated by 512 * (number of categories of node features) + 1. Is my guess right? And you said that it should be greater than the number of the class of all categories inissue #32, and how can I set this number in regression task? maybe number of graphs?

Thank you!

Hi, I trained some models on PCQM4M-LSC, ogbg-molhiv, and ZINC following the setting in the paper, and the results of PCQM4M-LSC and ogbg-molhiv are same as the paper. I also run experiment on ZINC several times, but the MAE is always more than 0.14 (with or without adding --intput_dropout_rate 0), which should be about 0.12 according to the paper. Here is my command:

python3 entry.py --dataset_name ZINC --hidden_dim 80 --ffn_dim 80 --num_heads 8 --tot_updates 400000 --batch_size 256 --warmup_updates 40000 --precision 16 --intput_dropout_rate 0 --gradient_clip_val 5 --num_workers 8 --gpus 1 --accelerator ddp --max_epochs 10000

https://github.com/microsoft/Graphormer/blob/740e6ff09a5de29d61def5ea6af7dfd04cee719e/graphormer/model.py#L20

When you re-initialize the weight of embedding, the weight of 0th index is also initialized by normal distribution, whose padding vector in the feature input will be non-zero. It should be wrong.

Hi,

Thanks for your interesting work. I have a problem regarding the evaluation. I downloaded your checkpoints from here, then I run the following command as mentioned in the Readme (for all_fold_seed0 checkpoint):

conda activate graphormer-lsc
export arch="--ffn_dim 768 --hidden_dim 768 --attention_dropout_rate 0.1 --dropout_rate 0.1 --n_layers 12 --peak_lr 2e-4 --edge_type multi_hop --multi_hop_max_dist 20 --weight_decay 0.0 --intput_dropout_rate 0.0"
export ckpt_path="checkpoints"
export ckpt_name="all_fold_seed0.ckpt"
bash inference.sh

The output log is:

Global seed set to 1
 > PCQM4M-LSC loaded!
{'num_class': 1, 'loss_fn': <function l1_loss at 0x7fc2381b3950>, 'metric': 'mae', 'metric_mode': 'min', 'evaluator': <ogb.lsc.pcqm4m.PCQM4MEvaluator object at 0x7fc1995d2110>, 'dataset': MyPygPCQM4MDataset2(3803453), 'max_node': 128}
 > dataset info ends
total params: 47167841
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Global seed set to 1
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
len(val_dataloader) 1487
Validating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1487/1487 [03:04<00:00,  7.42it/s]
0.027769196778535843
--------------------------------------------------------------------------------                                                                                                                           
DATALOADER:0 VALIDATE RESULTS
{'valid_mae': 0.027769196778535843}
--------------------------------------------------------------------------------
[{'valid_mae': 0.027769196778535843}]

I assumed that I should get results near Table 1 "validate MAE column", but it's different from that. Do I miss something?

Thanks for your help.

Hi authors, Thank you for your great work! I noticed that the result of 'the mean of the 4 cards ap' is different from the result of 'gather all pred and labels of different cards and do evaluation once'. And the latter method's result tends to be lower than the former one. It seems that when doing evaluation, Graphormer is using the former method. May I know that if you have the valid and test result of Graphormer model evulating the whole dataset once? Thank you!

Hi,

I have a few questions about the node padding.

Firstly, is my assumption correct, that the adding of -inf values in "pad_attn_bias_unsqueeze" has the same purpose as the attention_mask in BERT, so that there will be no attention to padded nodes?

If this is correct, why do you add +1 to x in the padding functions? As the attention is restricted not to attend there anyway, there could be any values in the padded nodes, so 0 could still be just as a regular feature value.

I talk about the padding like in

def pad_2d_unsqueeze(x, padlen):
    x = x + 1  # pad id = 0 -> THIS LINE
    xlen, xdim = x.size()
    if xlen < padlen:
        new_x = x.new_zeros([padlen, xdim], dtype=x.dtype)
        new_x[:xlen, :] = x
        x = new_x
    return x.unsqueeze(0)

which is used to pad x.

I'm trying to reproduce the reported results on OGB and ZINC datasets, but I failed to achieve the performance.

I first directly run the provided scripts hiv.sh to train a graphormer on MolHiv dataset without pretraining. The final AUC is 73.10%. Then I followed the instructions and hyper-parameter settings in the paper to do pre-training. I pre-trained on the PCQM4M for 20 epochs (until the loss converge) and fine-tuned the model on MolHiv for 8 epochs (as specified in the script) The best result turn out to be 76.25%.

Despite some improvement, the final AUC is not as high as it was reported in the paper. I also tried to reproduce the result on ZINC via the example script. But the best MAE is 0.1576, which is lower than 0.122 reported in the paper.

I'm wondering what I'm likely to miss that results in my poor performance. Can I know more reproduction details? My python environment is elaborated as below:

pytorch==1.9.0
pytorch-geometric==1.7,2
pytorch-scatter==2.0.8
pytorch-sparse==0.6.11
pytorch-lightning==1.3.0
ogb==1.3.1
cudatoolkit==11.1

I'd really appreciate it if someone could share their reproduced results and give me some suggestions.

I am training graphormer-slim on a custom dataset with 5K graphs and the loss is coming down nicely. However, the GPU utilization is close to zero while the CPU processes (16) are very busy, which means that the CPU is the bottleneck and I could be training much faster...

Is there anyway to make the CPU side faster?

details: graphs in DGL format 50-100 nodes per graph 1-3 edges per node

我最近在做Graphormer相关工作，我查找了GitHub未能找到您graphormer v1.0（ https://github.com/microsoft/Graphormer/tree/v1.0）提供的预训练模型，重新训练时间太长，所以向您求助graphormer v1.0在pcqm4m v1上的训练权重。

I've read the instructions of preparing the customized dataset, and I saw the example customized dataset is the "QM9" from dgl. And this dataset object extends the dgl.data.DGLDataset class, so does it mean that my customized dataset must extend the DGLDataset class and override the parental methods? I think it might be very tricky. So, could you please tell me more details if I want to prepare the customized dataset based on my self-curated data?

I want to use algos.pyx in my own file, but it can not be imported in that it can not be found. My own file can not recognize this pyx file. Is it not able to use in other files? Or could you tell me how to make it ? Thanks a lot!

Hi,

When running install.sh, getting the following error in the fairseq part -

Have you seen this before? Could this be due to different pip versions supporting different --use-feature options? I am currently running pip 22.3.1. Thank you.

Hallo everone, I wanted to train a customized dataset. Following the instructions in https://graphormer.readthedocs.io/en/latest/Datasets.html#id5 I added my code before the create_customized_dataset function to build a dgl dataset class. Then I wrote a shell file to run the training process. But I got ModuleNotFoundError error when I started the training. Here was the error information

Traceback (most recent call last):
  File "/root/anaconda3/envs/graphormer/bin/fairseq-train", line 8, in <module>
    sys.exit(cli_main())
  File "/root/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq_cli/train.py", line 528, in cli_main
    distributed_utils.call_main(cfg, main)
  File "/root/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 369, in call_main
    main(cfg, **kwargs)
  File "/root/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq_cli/train.py", line 85, in main
    task = tasks.setup_task(cfg.task)
  File "/root/anaconda3/envs/graphormer/lib/python3.9/site-packages/fairseq/tasks/__init__.py", line 46, in setup_task
    return task.setup_task(cfg, **kwargs)
  File "/workspace/Graphormer/graphormer/tasks/graph_prediction.py", line 179, in setup_task
    return cls(cfg)
  File "/workspace/Graphormer/graphormer/tasks/graph_prediction.py", line 142, in __init__
    self.__import_user_defined_datasets(cfg.user_data_dir)
  File "/workspace/Graphormer/graphormer/tasks/graph_prediction.py", line 165, in __import_user_defined_datasets
    importlib.import_module(module_name)
  File "/root/anaconda3/envs/graphormer/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'customized_dataset'

Here was the shell file

#!/usr/bin/env bash

CUDA_VISIBLE_DEVICES=0 fairseq-train \
--user-dir ../../graphormer \
--user-data-dir /workspace/Graphormer/examples/customized_dataset \
--num-workers 16 \
--ddp-backend=legacy_ddp \
--dataset-name MonomerTg_dataset \
--task graph_prediction \
--criterion l1_loss \
--arch graphormer_slim \
--num-classes 1 \
--attention-dropout 0.1 --act-dropout 0.1 --dropout 0.0 \
--optimizer adam --adam-betas '(0.9, 0.999)' --adam-eps 1e-8 --clip-norm 5.0 --weight-decay 0.01 \
--lr-scheduler polynomial_decay --power 1 --warmup-updates 60000 --total-num-update 400000 \
--lr 2e-4 --end-learning-rate 1e-9 \
--batch-size 64 \
--fp16 \
--data-buffer-size 20 \
--encoder-layers 12 \
--encoder-embed-dim 80 \
--encoder-ffn-embed-dim 80 \
--encoder-attention-heads 8 \
--max-epoch 10000 \
--save-dir ./ckpts

Anyone knows why it happens and maybe a solution for it? Thank you very much!