Codebase for Attentive Neural Hawkes Process (A-NHP) and Attentive Neural Datalog Through Time (A-NDTT)

Alan Yang

Last update: Dec 12, 2022

Related tags

Deep Learning anhp-andtt

Overview

Introduction

Codebase for the paper Transformer Embeddings of Irregularly Spaced Events and Their Participants.

This codebase contains two packages:

anhp: Attentive-Neural Hawkes Process (A-NHP)
andtt: Attentive-Neural Datalog Through Time (A-NDTT).

Author: Chenghao Yang ([email protected])

Reference

If you use this code as part of any published research, please acknowledge the following paper (it encourages researchers who publish their code!):

@article{yang-2021-transformer,
  author =      {Chenghao Yang and Hongyuan Mei and Jason Eisner},
  title =       {Transformer Embeddings of Irregularly Spaced Events and Their Participants},
  journal =     {arXiv preprint arxiv:2201.00044},
  year =        {2021}
}

Instructions

Here are the instructions to use the code base.

Dependencies and Installation

This code is written in Python 3, and I recommend you to install:

Anaconda that provides almost all the Python-related dependencies;

This project relies on Datalog Utilities in NDTT project, please first install it. (please remove the torch version (1.1.0) in setup.py of NDTT project, because that is not the requirement of this project and we only use non-pytorch part of NDTT. We recommend using torch>=1.7 for this project.).

Then run the command line below to install the package (add -e option if you need an editable installation):

pip install .

Dataset Preparation

Download datasets and programs from here.

Organize your domain datasets as follows:

domains/YOUR_DOMAIN/YOUR_PROGRAMS_AND_DATA

(A-NDTT-only) Build Dynamic Databases

Go to the andtt/run directory.

To build the dynamic databases for your data, try the command line below for detailed guide:

python build.py --help

The generated dynamic model architectures (represented by database facts) are stored in this directory:

domains/YOUR_DOMAIN/YOUR_PROGRAMS_AND_DATA/tdbcache

Train Models

To train the model specified by your Datalog probram, try the command line below for detailed guide:

python train.py --help

The training log and model parameters are stored in this directory:

# A-NHP
domains/YOUR_DOMAIN/YOUR_PROGRAMS_AND_DATA/ContKVLogs
# A-NDTT
domains/YOUR_DOMAIN/YOUR_PROGRAMS_AND_DATA/Logs

Example command line for training:

# A-NHP
python train.py -d YOUR_DOMAIN -ps ../../ -bs BATCH_SIZE -me 50 -lr 1e-4 -d_model 32 -teDim 10 -sd 1111 -layer 1
# A-NDTT
python train.py -d YOUR_DOMAIN -db YOUR_PROGRAM -ps ../../ -bs BATCH_SIZE -me 50 -lr 1e-4 -d_model 32 -teDim 10 -sd 1111 -layer 1

Test Models

To test the trained model, use the command line below for detailed guide:

python test.py --help

Example command line for testing:

python test.py -d YOUR_DOMAIN -fn FOLDER_NAME -s test -sd 12345 -pred

To evaluate the model predictions, use the command line below for detailed guide:

python eval.py --help

Example command line for testing:

python eval.py -d YOUR_DOMAIN -fn FOLDER_NAME -s test

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgements

The transformer component implementation used in this repo is based on widely-recognized Annotated Transformer.
The code structure is inspired by Prof. Hongyuan Mei's Neural Datalog Through Time

You might also like...

Ranger - a synergistic optimizer using RAdam (Rectified Adam), Gradient Centralization and LookAhead in one codebase

Ranger-Deep-Learning-Optimizer Ranger - a synergistic optimizer combining RAdam (Rectified Adam) and LookAhead, and now GC (gradient centralization) i

1.1k Dec 21, 2022

Codebase for the solution that won first place and was awarded the most human-like agent in the 2021 NeurIPS Competition MineRL BASALT Challenge.

KAIROS MineRL BASALT Codebase for the solution that won first place and was awarded the most human-like agent in the 2021 NeurIPS Competition MineRL B

37 Oct 30, 2022

Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

2.9k Jan 4, 2023

Comments

Reproducing Results for Figure2
Hi authors, Thank you for sharing the great work along with fixed versions for the previous works: THP and SAHP. While I was reproducing the results for THP, I ran into a problem.

NLL: I was able to reproduce the results on stackoverflow but the results on MIMIC easily go down to ~5, which is far below than the reported number ~8. I have only changed lr from 1e-4 to 1e-3. Is it expected or am I missing something here?

RMSE: it seems like the code for RMSE is not fixed the current version of the code. I believe all the dummy values for event_time and prediction need to be ruled out here: https://github.com/yangalan123/anhp-andtt/blob/b907f3808ed2ce1616edb1bc2229993a6742cee9/thp/thp_training/Utils.py#L103 Could you confirm that it has not been fixed yet? If so, do you have a plan to fix it anytime soon?

Lastly, according to the datasets in google drive, each dataset is divided into several fold. Have you used all the folds together to report the numbers in Figure2? Or used a particular fold? Thank you in advance.
opened by won-bae 6
Reproducing the prediction results using the thinning algoritm.

Hi Alan,

Thank you for your response.

I quite enjoyed the paper you recommended. I believe that I have some level of understaning about the thinning algorithm, but still having a hard time understanding the mathmetical details behind it.

Also, It got me wondering about the difference between the method used in Neural Hawkes Process where you directly approximate the conditional expectation of t_i and the thinnin algorithm approach. What would be the benefit of using the thinning algorithm?

Again, thank you for sharing a great work!

opened by JardinDelSol 2
请教个关于 bos 参数的问题

你好，请教个关于代码细节的问题。

在 manager.py 中加载数据的时候 add_bos=False, 但是在模型中却强制写成 add_bos=True，我改成 add_bos=False会报错。

dataset = NHPDataset(_data[_split], event_types, concurrent=False, add_bos=False, add_eos=False) line 35，https://github.com/yangalan123/anhp-andtt/blob/master/anhp/esm/manager.py

self.add_bos = True line 64，https://github.com/yangalan123/anhp-andtt/blob/master/anhp/model/xfmr_nhp_fast.py

从我个人理解来看，model.add_bos 和 dataset.add_bos 含义不一样对吗？ model.add_bos 是指每次预测下一个点，所以label序列的长度比输入序列的长度要少一个，后面代码会因此有特殊的处理。

如果我的理解正确的话，那是不是所有的模型 model.add_bos 都应该是 True，这样代码里面一些 if else 的判断就不需要了，比如 line 158-165, https://github.com/yangalan123/anhp-andtt/blob/master/anhp/model/xfmr_nhp_fast.py，这部分实际上都不会执行。

非常感谢。

opened by iLampard 2

Codebase for Attentive Neural Hawkes Process (A-NHP) and Attentive Neural Datalog Through Time (A-NDTT)

Related tags

Overview

Introduction

Reference

Instructions

Dependencies and Installation

Dataset Preparation

(A-NDTT-only) Build Dynamic Databases

Train Models

Test Models

License

Acknowledgements

You might also like...

Ranger - a synergistic optimizer using RAdam (Rectified Adam), Gradient Centralization and LookAhead in one codebase

Codebase for the solution that won first place and was awarded the most human-like agent in the 2021 NeurIPS Competition MineRL BASALT Challenge.

Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

Official codebase used to develop Vision Transformer, MLP-Mixer, LiT and more.

Code for the paper "How Attentive are Graph Attention Networks?"

Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding (AAAI 2020) - PyTorch Implementation

Dynamic Attentive Graph Learning for Image Restoration, ICCV2021 [PyTorch Code]

A Structured Self-attentive Sentence Embedding

The first public PyTorch implementation of Attentive Recurrent Comparators

Comments

Reproducing Results for Figure2

Reproducing the prediction results using the thinning algoritm.

请教个关于 bos 参数的问题

Owner

Alan Yang

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

Codebase for Amodal Segmentation through Out-of-Task andOut-of-Distribution Generalization with a Bayesian Model

A python-image-classification web application project, written in Python and served through the Flask Microframework. This Project implements the VGG16 covolutional neural network, through Keras and Tensorflow wrappers, to make predictions on uploaded images.

A PaddlePaddle implementation of Time Interval Aware Self-Attentive Sequential Recommendation.

This codebase is the official implementation of Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS2021, Spotlight)

code for "AttentiveNAS Improving Neural Architecture Search via Attentive Sampling"

Simple codebase for flexible neural net training

A general and strong 3D object detection codebase that supports more methods, datasets and tools (debugging, recording and analysis).

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Code in both PyTorch and TensorFlow

X-modaler is a versatile and high-performance codebase for cross-modal analytics.