Implementation of "GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings" in PyTorch

Overview

PyGAS: Auto-Scaling GNNs in PyG


PyGAS is the practical realization of our GNNAutoScale (GAS) framework, which scales arbitrary message-passing GNNs to large graphs, as described in our paper:

Matthias Fey, Jan E. Lenssen, Frank Weichert, Jure Leskovec: GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings (ICML 2021)

GAS prunes entire sub-trees of the computation graph by utilizing historical embeddings from prior training iterations, leading to constant GPU memory consumption in respect to input mini-batch size, and maximally expressivity.

PyGAS is implemented in PyTorch and utilizes the PyTorch Geometric (PyG) library. It provides an easy-to-use interface to convert a common or custom GNN from PyG into its scalable variant:

from torch_geometric.nn import SAGEConv
from torch_geometric_autoscale import ScalableGNN
from torch_geometric_autoscale import metis, permute, SubgraphLoader

class GNN(ScalableGNN):
    def __init__(self, num_nodes, in_channels, hidden_channels,
                 out_channels, num_layers):
        # * pool_size determines the number of pinned CPU buffers
        # * buffer_size determines the size of pinned CPU buffers,
        #   i.e. the maximum number of out-of-mini-batch nodes

        super().__init__(num_nodes, hidden_channels, num_layers,
                         pool_size=2, buffer_size=5000)

        self.convs = ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        for _ in range(num_layers - 2):
            self.convs.append(SAGEConv(hidden_channels, hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

    def forward(self, x, adj_t, *args):
        for conv, history in zip(self.convs[:-1], self.histories):
            x = conv(x, adj_t).relu_()
            x = self.push_and_pull(history, x, *args)
        return self.convs[-1](x, adj_t)

perm, ptr = metis(data.adj_t, num_parts=40, log=True)
data = permute(data, perm, log=True)
loader = SubgraphLoader(data, ptr, batch_size=10, shuffle=True)

model = GNN(...)
for batch, *args in loader:
    out = model(batch.x, batch.adj_t, *args)

A detailed description of ScalableGNN can be found in its implementation.

Requirements

pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-geometric

where ${TORCH} should be replaced by either 1.7.0 or 1.8.0, and ${CUDA} should be replaced by either cpu, cu92, cu101, cu102, cu110 or cu111, depending on your PyTorch installation.

Installation

pip install git+https://github.com/rusty1s/pyg_autoscale.git

or

python setup.py install

Project Structure

  • torch_geometric_autoscale/ contains the source code of PyGAS
  • examples/ contains examples to demonstrate how to apply GAS in practice
  • small_benchmark/ includes experiments to evaluate GAS performance on small-scale graphs
  • large_benchmark/ includes experiments to evaluate GAS performance on large-scale graphs

We use Hydra to manage hyperparameter configurations.

Cite

Please cite our paper if you use this code in your own work:

@inproceedings{Fey/etal/2021,
  title={{GNNAutoScale}: Scalable and Expressive Graph Neural Networks via Historical Embeddings},
  author={Fey, M. and Lenssen, J. E. and Weichert, F. and Leskovec, J.},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2021},
}
Comments
  • Error raised when reproducing large_benchmark

    Error raised when reproducing large_benchmark

    We run the main.py on the document large_benchmark ,but we meet the following issues,please help us.The following are our issues and enviroment.

    Error message

    Loading data... Processing...
    Error executing job with overrides: ['model=pna', 'dataset=flickr', 'root=/tmp/datasets', 'device=0', 'log_every=1']
    Traceback (most recent call last):
      File "main.py", line 78, in main
        data, in_channels, out_channels = get_data(conf.root, conf.dataset.name)
      File "/root/anaconda3/lib/python3.8/site-packages/torch_geometric_autoscale/data.py", line 126, in get_data
        return get_flickr(root)
      File "/root/anaconda3/lib/python3.8/site-packages/torch_geometric_autoscale/data.py", line 81, in get_flickr
        dataset = Flickr(f'{root}/Flickr', pre_transform=T.ToSparseTensor())
      File "/root/anaconda3/lib/python3.8/site-packages/torch_geometric/datasets/flickr.py", line 37, in __init__
        super().__init__(root, transform, pre_transform)
      File "/root/anaconda3/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 60, in __init__
        super().__init__(root, transform, pre_transform, pre_filter)
      File "/root/anaconda3/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 86, in __init__
        self._process()
      File "/root/anaconda3/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 165, in _process
        self.process()
      File "/root/anaconda3/lib/python3.8/site-packages/torch_geometric/datasets/flickr.py", line 69, in process
        x = np.load(osp.join(self.raw_dir, 'feats.npy'))
      File "/root/anaconda3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 444, in load
        raise ValueError("Cannot load file containing pickled data "
    ValueError: Cannot load file containing pickled data when allow_pickle=False
    
    Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
    

    Environment

    Package                            Version
    ---------------------------------- -------------------
    alabaster                          0.7.12
    anaconda-client                    1.7.2
    anaconda-navigator                 1.10.0
    anaconda-project                   0.8.3
    antlr4-python3-runtime             4.8
    argh                               0.26.2
    argon2-cffi                        20.1.0
    asn1crypto                         1.4.0
    astroid                            2.4.2
    astropy                            4.0.2
    async-generator                    1.10
    atomicwrites                       1.4.0
    attrs                              20.3.0
    autopep8                           1.5.4
    Babel                              2.8.1
    backcall                           0.2.0
    backports.functools-lru-cache      1.6.1
    backports.shutil-get-terminal-size 1.0.0
    backports.tempfile                 1.0
    backports.weakref                  1.0.post1
    beautifulsoup4                     4.9.3
    bitarray                           1.6.1
    bkcharts                           0.2
    bleach                             3.2.1
    bokeh                              2.2.3
    boto                               2.49.0
    Bottleneck                         1.3.2
    brotlipy                           0.7.0
    certifi                            2020.6.20
    cffi                               1.14.3
    chardet                            3.0.4
    click                              7.1.2
    cloudpickle                        1.6.0
    clyent                             1.2.2
    colorama                           0.4.4
    conda                              4.11.0
    conda-build                        3.20.5
    conda-package-handling             1.7.2
    conda-verify                       3.4.2
    contextlib2                        0.6.0.post1
    cryptography                       3.1.1
    cycler                             0.10.0
    Cython                             0.29.21
    cytoolz                            0.11.0
    dask                               2.30.0
    decorator                          4.4.2
    defusedxml                         0.6.0
    dgl-cu110                          0.6.1
    diff-match-patch                   20200713
    distributed                        2.30.1
    docutils                           0.16
    entrypoints                        0.3
    et-xmlfile                         1.0.1
    fastcache                          1.1.0
    filelock                           3.0.12
    flake8                             3.8.4
    Flask                              1.1.2
    fsspec                             0.8.3
    future                             0.18.2
    gevent                             20.9.0
    glob2                              0.7
    gmpy2                              2.0.8
    googledrivedownloader              0.4
    greenlet                           0.4.17
    h5py                               2.10.0
    HeapDict                           1.0.1
    html5lib                           1.1
    hydra-core                         1.1.1
    idna                               2.10
    imageio                            2.9.0
    imagesize                          1.2.0
    importlib-metadata                 2.0.0
    importlib-resources                5.4.0
    iniconfig                          1.1.1
    intervaltree                       3.1.0
    ipykernel                          5.3.4
    ipython                            7.19.0
    ipython-genutils                   0.2.0
    ipywidgets                         7.5.1
    isodate                            0.6.0
    isort                              5.6.4
    itsdangerous                       1.1.0
    jdcal                              1.4.1
    jedi                               0.17.1
    jeepney                            0.5.0
    Jinja2                             2.11.2
    joblib                             0.17.0
    json5                              0.9.5
    jsonschema                         3.2.0
    jupyter                            1.0.0
    jupyter-client                     6.1.7
    jupyter-console                    6.2.0
    jupyter-core                       4.6.3
    jupyterlab                         2.2.6
    jupyterlab-pygments                0.1.2
    jupyterlab-server                  1.2.0
    keyring                            21.4.0
    kiwisolver                         1.3.0
    lazy-object-proxy                  1.4.3
    libarchive-c                       2.9
    littleutils                        0.2.2
    llvmlite                           0.34.0
    locket                             0.2.0
    lxml                               4.6.1
    MarkupSafe                         1.1.1
    matplotlib                         3.3.2
    mccabe                             0.6.1
    mistune                            0.8.4
    mkl-fft                            1.2.0
    mkl-random                         1.1.1
    mkl-service                        2.3.0
    mock                               4.0.2
    more-itertools                     8.6.0
    mpmath                             1.1.0
    msgpack                            1.0.0
    multipledispatch                   0.6.0
    navigator-updater                  0.2.1
    nbclient                           0.5.1
    nbconvert                          6.0.7
    nbformat                           5.0.8
    nest-asyncio                       1.4.2
    networkx                           2.5
    nltk                               3.5
    nose                               1.3.7
    notebook                           6.1.4
    numba                              0.51.2
    numexpr                            2.7.1
    numpy                              1.19.2
    numpydoc                           1.1.0
    ogb                                1.3.1
    olefile                            0.46
    omegaconf                          2.1.1
    openpyxl                           3.0.5
    outdated                           0.2.1
    packaging                          20.4
    pandas                             1.1.3
    pandocfilters                      1.4.3
    parso                              0.7.0
    partd                              1.1.0
    path                               15.0.0
    pathlib2                           2.3.5
    pathtools                          0.1.2
    patsy                              0.5.1
    pep8                               1.7.1
    pexpect                            4.8.0
    pickleshare                        0.7.5
    Pillow                             8.0.1
    pip                                20.2.4
    pkginfo                            1.6.1
    pluggy                             0.13.1
    ply                                3.11
    prometheus-client                  0.8.0
    prompt-toolkit                     3.0.8
    psutil                             5.7.2
    ptyprocess                         0.6.0
    py                                 1.9.0
    pycodestyle                        2.6.0
    pycosat                            0.6.3
    pycparser                          2.20
    pycurl                             7.43.0.6
    pydocstyle                         5.1.1
    pyflakes                           2.2.0
    Pygments                           2.7.2
    pylint                             2.6.0
    pyodbc                             4.0.0-unsupported
    pyOpenSSL                          19.1.0
    pyparsing                          2.4.7
    pyrsistent                         0.17.3
    PySocks                            1.7.1
    pytest                             0.0.0
    python-dateutil                    2.8.1
    python-jsonrpc-server              0.4.0
    python-language-server             0.35.1
    python-louvain                     0.15
    pytz                               2020.1
    PyWavelets                         1.1.1
    pyxdg                              0.27
    PyYAML                             5.3.1
    pyzmq                              19.0.2
    QDarkStyle                         2.8.1
    QtAwesome                          1.0.1
    qtconsole                          4.7.7
    QtPy                               1.9.0
    rdflib                             5.0.0
    regex                              2020.10.15
    requests                           2.24.0
    rope                               0.18.0
    Rtree                              0.9.4
    ruamel-yaml                        0.15.87
    scikit-image                       0.17.2
    scikit-learn                       0.23.2
    scipy                              1.5.2
    seaborn                            0.11.0
    SecretStorage                      3.1.2
    Send2Trash                         1.5.0
    setuptools                         50.3.1.post20201107
    simplegeneric                      0.8.1
    singledispatch                     3.4.0.3
    sip                                4.19.13
    six                                1.15.0
    snowballstemmer                    2.0.0
    sortedcollections                  1.2.1
    sortedcontainers                   2.2.2
    soupsieve                          2.0.1
    Sphinx                             3.2.1
    sphinxcontrib-applehelp            1.0.2
    sphinxcontrib-devhelp              1.0.2
    sphinxcontrib-htmlhelp             1.0.3
    sphinxcontrib-jsmath               1.0.1
    sphinxcontrib-qthelp               1.0.3
    sphinxcontrib-serializinghtml      1.1.4
    sphinxcontrib-websupport           1.2.4
    spyder                             4.1.5
    spyder-kernels                     1.9.4
    SQLAlchemy                         1.3.20
    statsmodels                        0.12.0
    sympy                              1.6.2
    tables                             3.6.1
    tblib                              1.7.0
    terminado                          0.9.1
    testpath                           0.4.4
    threadpoolctl                      2.1.0
    tifffile                           2020.10.1
    toml                               0.10.1
    toolz                              0.11.1
    torch                              1.8.0+cu111
    torch-cluster                      1.5.9
    torch-geometric                    1.7.2
    torch-geometric-autoscale          0.0.0
    torch-scatter                      2.0.7
    torch-sparse                       0.6.10
    torch-spline-conv                  1.2.1
    torchaudio                         0.8.0
    torchvision                        0.9.0+cu111
    tornado                            6.0.4
    tqdm                               4.50.2
    traitlets                          5.0.5
    typing-extensions                  3.7.4.3
    ujson                              4.0.1
    unicodecsv                         0.14.1
    urllib3                            1.25.11
    watchdog                           0.10.3
    wcwidth                            0.2.5
    webencodings                       0.5.1
    Werkzeug                           1.0.1
    wheel                              0.35.1
    widgetsnbextension                 3.5.1
    wrapt                              1.11.2
    wurlitzer                          2.0.1
    xlrd                               1.2.0
    XlsxWriter                         1.3.7
    xlwt                               1.3.0
    xmltodict                          0.12.0
    yapf                               0.30.0
    zict                               2.0.0
    zipp                               3.4.0
    zope.event                         4.5.0
    zope.interface                     5.1.2
    
    opened by yanliang3612 10
  • RuntimeError: Not compiled with CUDA support

    RuntimeError: Not compiled with CUDA support

    Hello, thank you very much for your contribution in the large-scale graph training. I also try to run it on the machine. There is a problem now, I hope you can help solve it ,thank you! 360截图187903157388109

    opened by fishysq 10
  • A Question about the Code

    A Question about the Code

    Hello. I appreciate that your work is really good and influenced me a lot. But excuse me, could you tell me in pyg_autoscale/torch_geometric_autoscale/history.py, within the member of the class History, push(), the branch else, why is it self.emb[dst_o:dst_o + c] = x[src_o:src_o + c]? Where is the indices vector n_id? I thought self.emb, also "history" as mentioned in your paper, recorded node v's most recent embedding in its v-th block or self.emb[v]. So in History's push(), within the last elif, self.emb is assigned values from x as the batch feature vectors only for x in n_id. But why in else, it has nothing to do with n_id? Or I want to know, if it should be self.emb[n_id[dst_o:dst_o + c]] = x[src_o:src_o + c], or the offset and count are not exactly like what you've described in pyg_autoscale/torch_geometric_autoscale/models/base.py, within ScalableGNN: __call__() ? Since I can't find the source code of torch.ops.torch_sparse.partition (trust me, I have tried a lot question_gas_author )

    opened by shuoyinn 6
  • Installation Failure

    Installation Failure

    My environment: centos7, GCC 5.4, nvidia driver 470.82.01, cuda: 11.4, anaconda, python 3.9.13 pytorch 1.10.1, pyg 2.0.4

    Install from source, with pip install -e .

    20F9D08A-3F08-47DD-AABC-422863F8B617

    Could you help me install this package? Thanks!

    opened by joneswong 5
  • Impossible to install on AWS SageMaker (gcc < 5.0)

    Impossible to install on AWS SageMaker (gcc < 5.0)

    When I am trying to install pyg_autoscale at AWS SageMaker I am getting a following error:

    pip install git+https://github.com/rusty1s/pyg_autoscale.git -q

    ERROR: Command errored out with exit status 1: command: /home/ec2-user/anaconda3/envs/python3/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-wjznfuqn/setup.py'"'"'; file='"'"'/tmp/pip-req-build-wjznfuqn/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ajrg7df1 cwd: /tmp/pip-req-build-wjznfuqn/ Complete output (46 lines): running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/init.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/metis.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/pool.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/loader.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/history.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/utils.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/data.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale creating build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/gcn2.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/init.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/gcn.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/gat.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/appnp.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/pna_jk.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/base.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/pna.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models running build_ext /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/utils/cpp_extension.py:312: UserWarning:

                                 !! WARNING !!
    

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Your compiler (g++ 4.8.5) may be ABI-incompatible with PyTorch! Please use a compiler that is ABI-compatible with GCC 5.0 and above. See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

    See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6 for instructions on how to install GCC 5 or higher. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                                !! WARNING !!
    
    warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
    

    building 'torch_geometric_autoscale._relabel' extension creating build/temp.linux-x86_64-3.6 creating build/temp.linux-x86_64-3.6/csrc creating build/temp.linux-x86_64-3.6/csrc/cpu gcc -pthread -B /home/ec2-user/anaconda3/envs/python3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -Icsrc -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include/TH -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda-10.0/include -I/home/ec2-user/anaconda3/envs/python3/include/python3.6m -c csrc/relabel.cpp -o build/temp.linux-x86_64-3.6/csrc/relabel.o -DAT_PARALLEL_OPENMP -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=_relabel -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14 gcc: error: unrecognized command line option ‘-std=c++14’ error: command 'gcc' failed with exit status 1

    ERROR: Failed building wheel for torch-geometric-autoscale ERROR: Command errored out with exit status 1: command: /home/ec2-user/anaconda3/envs/python3/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-wjznfuqn/setup.py'"'"'; file='"'"'/tmp/pip-req-build-wjznfuqn/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-o2qjs31r/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/anaconda3/envs/python3/include/python3.6m/torch-geometric-autoscale cwd: /tmp/pip-req-build-wjznfuqn/ Complete output (46 lines): running install running build running build_py creating build creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/init.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/metis.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/pool.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/loader.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/history.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/utils.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale copying torch_geometric_autoscale/data.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale creating build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/gcn2.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/init.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/gcn.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/gat.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/appnp.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/pna_jk.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/base.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models copying torch_geometric_autoscale/models/pna.py -> build/lib.linux-x86_64-3.6/torch_geometric_autoscale/models running build_ext /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/utils/cpp_extension.py:312: UserWarning:

                                   !! WARNING !!
    
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    Your compiler (g++ 4.8.5) may be ABI-incompatible with PyTorch!
    Please use a compiler that is ABI-compatible with GCC 5.0 and above.
    See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
    
    See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
    for instructions on how to install GCC 5 or higher.
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    
                                  !! WARNING !!
    
      warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
    building 'torch_geometric_autoscale._relabel' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/csrc
    creating build/temp.linux-x86_64-3.6/csrc/cpu
    gcc -pthread -B /home/ec2-user/anaconda3/envs/python3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -Icsrc -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include/TH -I/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda-10.0/include -I/home/ec2-user/anaconda3/envs/python3/include/python3.6m -c csrc/relabel.cpp -o build/temp.linux-x86_64-3.6/csrc/relabel.o -DAT_PARALLEL_OPENMP -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=_relabel -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    gcc: error: unrecognized command line option ‘-std=c++14’
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
    

    ERROR: Command errored out with exit status 1: /home/ec2-user/anaconda3/envs/python3/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-wjznfuqn/setup.py'"'"'; file='"'"'/tmp/pip-req-build-wjznfuqn/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-o2qjs31r/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/anaconda3/envs/python3/include/python3.6m/torch-geometric-autoscale Check the logs for full command output.

    gcc --version gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28) Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    Is 5.0 gcc version is really needed?

    opened by CaypoH 4
  • Why `cudaMemcpyDeviceToHost` in `read_async_cuda`?

    Why `cudaMemcpyDeviceToHost` in `read_async_cuda`?

    Hi @rusty1s , awesome work. I'm reading through the codebase and found you were using cudaMemcpyDeviceToHost when copying data from src (which resides in CPU) and dst (which resides in CUDA):

    https://github.com/rusty1s/pyg_autoscale/blob/a17350c04910c95b0486110b70b742b5e3f56de5/csrc/cuda/async_cuda.cu#L74-L76

    I wonder is it a typo?

    opened by yzh119 3
  • pickling error

    pickling error

    Hi

    thank you a lot for this brilliant work, it works like a charm, and is very impressive (a perfect addition to already fantastic torch_geometric pack)

    I have an issue that seems far beyond my python competences. When i am trying to use several workers with SubgraphLoader, I get a very interesting error:

    _pickle.PicklingError: Can't pickle <class 'torch_geometric.data.batch.Batch'>: it's not the same object as torch_geometric.data.batch.Batch
    

    (made a few google searches, but it did not help)

    my setup is a bit more involved than your examples, I will try to describe it, and hopefully give useful information without too much noise.

    I have an outer torch_geometric.loader.DataLoader that read samples from disk this samples are of the form MyGraph(torch_geometric.data.Data) and redefine __cat_dim__ and the constructor

    (I did not redefine the collation functions, so I get a Batch out of this) this outer dataloader has a fixed batch size of 1, as I am scaling up to larger graphs

    this samples are passed (one by one as my bs is 1) to a code that split it like in your examples:

    perm, ptr = metis(d.adj_t, num_parts=num_parts, log=self.debug)                                 
    data = permute(d, perm, log=self.debug)                        
    

    and then it works like a charm if I do:

    loader = SubgraphLoader(data, ptr, batch_size=inner_batch_size, shuffle=True) 
    

    but I get the above error if i do:

    loader = SubgraphLoader(data, ptr, batch_size=inner_batch_size, shuffle=True, num_workers=2, persistent_workers=True) 
    

    So I am wondering I you have any clue on this, or if you know where I should investigate.

    Thanks a lot, and congratulations again for this piece of work and for the whole torch_geometric pack.

    opened by fantes 3
  • Plan to implement the VR-GCN model

    Plan to implement the VR-GCN model

    Hi, thanks for your solid work! I am wondering if you plan to implement the VR-GCN[1] by pytorch_geometric since it is similar to GNNAutoScale.

    [1] Stochastic training of graph convolutional networks with variance reduction. In ICML2018.

    opened by huangtinglin 2
  • init ScalableGNN

    init ScalableGNN

    Currently, the example in README.md could not be run because one could only import ScalableGNN from torch_geometric_autoscale.models. I have imported ScalableGNN to torch_geometric_autoscale so that the example would not fail. Alternatively, one could fix the example code in README.md, but as ScalableGNN is useful in many places, perhaps making it easier to be imported would be better.

    opened by SherylHYX 1
  • Whether autoscale supports heterogeneous graph

    Whether autoscale supports heterogeneous graph

    Thanks for this great works. What i should do if I want to use autoscale for heterogeneous graph? Can these three ways in the docs be applied with autoscale to create models on heterogeneous graph data? image

    opened by leethu2012 1
  • Failed to import package torch_geometric_autoscale

    Failed to import package torch_geometric_autoscale

    my env is torch 1.12.0 torch-cluster 1.6.0 torch-geometric 2.0.4 torch-geometric-autoscale 0.0.0 torch-scatter 2.0.9 torch-sparse 0.6.15 torch-spline-conv 1.2.1

    The import command is blocked until ctrl+c, then throw exception

    from torch_geometric_autoscale import History, AsyncIOPool ^CTraceback (most recent call last): File "", line 1, in File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/torch_geometric_autoscale/init.py", line 12, in from .data import get_data # noqa File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/torch_geometric_autoscale/data.py", line 9, in from ogb.nodeproppred import PygNodePropPredDataset File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/ogb/nodeproppred/init.py", line 1, in from .evaluate import Evaluator File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/ogb/nodeproppred/evaluate.py", line 1, in from sklearn.metrics import roc_auc_score File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/sklearn/init.py", line 82, in from .base import clone File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/sklearn/base.py", line 17, in from .utils import _IS_32BIT File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/sklearn/utils/init.py", line 26, in from . import _joblib File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/sklearn/utils/_joblib.py", line 7, in import joblib File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/joblib/init.py", line 113, in from .memory import Memory, MemorizedResult, register_store_backend File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/joblib/memory.py", line 32, in from ._store_backends import StoreBackendBase, FileSystemStoreBackend File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/joblib/_store_backends.py", line 15, in from .backports import concurrency_safe_rename File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/joblib/backports.py", line 7, in from distutils.version import LooseVersion File "", line 1007, in _find_and_load File "", line 982, in _find_and_load_unlocked File "", line 925, in _find_spec File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/_distutils_hack/init.py", line 97, in find_spec return method() File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/_distutils_hack/init.py", line 108, in spec_for_distutils mod = importlib.import_module('setuptools._distutils') File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/setuptools/init.py", line 16, in import setuptools.version File "/home/ma-user/anaconda3/envs/py3.9/lib/python3.9/site-packages/setuptools/version.py", line 1, in import pkg_resources File "", line 211, in _lock_unlock_module File "", line 107, in acquire KeyboardInterrupt

    opened by EnjianGong 1
  • Have trouble on installing the package

    Have trouble on installing the package

        warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
      building 'torch_geometric_autoscale._async' extension
      creating build/temp.linux-x86_64-cpython-38
      creating build/temp.linux-x86_64-cpython-38/csrc
      gcc -pthread -B /gpfs/gibbs/pi/zhao/tl688/conda_envs/graphnn/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc -I/home/tl688/.local/lib/python3.8/site-packages/torch/include -I/home/tl688/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/tl688/.local/lib/python3.8/site-packages/torch/include/TH -I/home/tl688/.local/lib/python3.8/site-packages/torch/include/THC -I/gpfs/gibbs/pi/zhao/tl688/conda_envs/graphnn/include/python3.8 -c csrc/async.cpp -o build/temp.linux-x86_64-cpython-38/csrc/async.o -DAT_PARALLEL_OPENMP -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_async -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
      gcc: error: unrecognized command line option ‘-std=c++14’
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
    

    note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

    Here is the error I met, but I did not fail when I intend to install pyg. Could you please help me solve this problem? Thanks a lot!

    opened by HelloWorldLTY 3
  • SubgraphLoader for heterogeneous graph

    SubgraphLoader for heterogeneous graph

    Hi, I am trying to apply pyg_autoscale to heterogeneous graph and have to modify the compute_subgraph method in SubgraphLoader class. I was wondering would you like to elaborate on what offset, count are and what is relabel_fn doing? My current understanding is that compute_subgraph is basically taking the sub-graph spanned by n_id. Is this understanding accurate? Many thanks!

        def compute_subgraph(self, batches: List[Tuple[int, Tensor]]) -> SubData:
            batch_ids, n_ids = zip(*batches)
            n_id = torch.cat(n_ids, dim=0)
            batch_id = torch.tensor(batch_ids)
    
            # We collect the in-mini-batch size (`batch_size`), the offset of each
            # partition in the mini-batch (`offset`), and the number of nodes in
            # each partition (`count`)
            batch_size = n_id.numel()
            offset = self.ptr[batch_id]
            count = self.ptr[batch_id.add_(1)].sub_(offset)
    
            rowptr, col, value = self.data.adj_t.csr()
            rowptr, col, value, n_id = relabel_fn(rowptr, col, value, n_id,
                                                  self.bipartite)
    
            adj_t = SparseTensor(rowptr=rowptr, col=col, value=value,
                                 sparse_sizes=(rowptr.numel() - 1, n_id.numel()),
                                 is_sorted=True)
    
            data = self.data.__class__(adj_t=adj_t)
            for k, v in self.data:
                if isinstance(v, Tensor) and v.size(0) == self.data.num_nodes:
                    data[k] = v.index_select(0, n_id)
    
            return SubData(data, batch_size, n_id, offset, count)
    
    opened by Chen-Cai-OSU 1
  • question about emb_device and device for History class

    question about emb_device and device for History class

    Hello Matthias, Thank you very much for the code. Nice work as always. I was wondering what is the difference between emb_device and device for the History class?

    When I initialized a GCN like this

    model = GCN(10, 10, 10, 10, 5, device='cpu')
    print(model)
    

    I get

    GCN(
      (histories): ModuleList(
        (0): History(10, 10, emb_device=cpu, device=cpu)
        (1): History(10, 10, emb_device=cpu, device=cpu)
        (2): History(10, 10, emb_device=cpu, device=cpu)
        (3): History(10, 10, emb_device=cpu, device=cpu)
      )
      (lins): ModuleList()
      (convs): ModuleList(
        (0): GCNConv(10, 10)
        (1): GCNConv(10, 10)
        (2): GCNConv(10, 10)
        (3): GCNConv(10, 10)
        (4): GCNConv(10, 10)
      )
      (bns): ModuleList(
        (0): BatchNorm1d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): BatchNorm1d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): BatchNorm1d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (3): BatchNorm1d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): BatchNorm1d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    
    

    but I noticed that there is a process in cuda:0 (I have multiple gpus), which I don't understand why. Is this desirable behavior? Also, in general, should I always set the device in GCN class as none? I noticed this is what you did in the large_benchmark/main.py.

    opened by Chen-Cai-OSU 4
  • The code cannot exit properly

    The code cannot exit properly

    Hi, thank you very much for your extraordinary work, it helps me a lot. Now I have a problem when I run the the following command in the examples directory: python train_gcn.py --root=/my_data --device=0. There is no error raised, but the code just cannot exit properly after the training is finished, and I have to exit manually using Ctrl+C as following. I have tried to add "sys.exit(0)" in the train_gcn.py, but it doesn't work.

    Traceback (most recent call last): File "/home/qchen/anaconda3/envs/py38/lib/python3.8/threading.py", line 1388, in _shutdown lock.acquire() KeyboardInterrupt:

    My dependencies are python38 + pytorch 1.10.1 + pyg 2.0.3. Do you have any idea by what it is caused? Thank you again for your contribution!

    opened by 7qchen 3
Owner
Matthias Fey
PhD student @ TU Dortmund University - Interested in Representation Learning on Graphs and Manifolds; PyTorch, CUDA, Vim and macOS Enthusiast
Matthias Fey
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
RealFormer-Pytorch Implementation of RealFormer using pytorch

RealFormer-Pytorch Implementation of RealFormer using pytorch. Includes comparison with classical Transformer on image classification task (ViT) wrt C

Simo Ryu 90 Dec 8, 2022
A PyTorch implementation of the paper Mixup: Beyond Empirical Risk Minimization in PyTorch

Mixup: Beyond Empirical Risk Minimization in PyTorch This is an unofficial PyTorch implementation of mixup: Beyond Empirical Risk Minimization. The co

Harry Yang 121 Dec 17, 2022
A pytorch implementation of Pytorch-Sketch-RNN

Pytorch-Sketch-RNN A pytorch implementation of https://arxiv.org/abs/1704.03477 In order to draw other things than cats, you will find more drawing da

Alexis David Jacq 172 Dec 12, 2022
PyTorch implementation of Advantage async actor-critic Algorithms (A3C) in PyTorch

Advantage async actor-critic Algorithms (A3C) in PyTorch @inproceedings{mnih2016asynchronous, title={Asynchronous methods for deep reinforcement lea

LEI TAI 111 Dec 8, 2022
Pytorch-diffusion - A basic PyTorch implementation of 'Denoising Diffusion Probabilistic Models'

PyTorch implementation of 'Denoising Diffusion Probabilistic Models' This reposi

Arthur Juliani 76 Jan 7, 2023
Fang Zhonghao 13 Nov 19, 2022
RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

RETRO - Pytorch (wip) Implementation of RETRO, Deepmind's Retrieval based Attent

Phil Wang 556 Jan 4, 2023
HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

HashNeRF-pytorch Instant-NGP recently introduced a Multi-resolution Hash Encodin

Yash Sanjay Bhalgat 616 Jan 6, 2023
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code here will be included in upstream Pytorch eventually. The intention of Apex is to make up-to-date utilities available to users as quickly as possible.

NVIDIA Corporation 6.9k Jan 3, 2023
Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

Mayur 119 Nov 24, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 360 Dec 10, 2022
The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

Ritchie Ng 9.2k Jan 2, 2023
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 359 Jan 5, 2023
A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

Vince 0 Jul 13, 2021
PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

Subin An 8 Nov 21, 2022
PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Amin Rezaei 157 Dec 11, 2022