Curvlearn, a Tensorflow based non-Euclidean deep learning framework.

Overview

English | 简体中文

Why Non-Euclidean Geometry

Considering these simple graph structures shown below. Nodes with same color has 2-hop distance whereas 1-hop distance between nodes with different color. Now how could we embed these structures in Euclidean space while keeping these distance unchanged?

Actually perfect embedding without distortion, appearing naturally in hyperbolic (negative curvature) or spherical (positive curvature) space, is infeasible in Euclidean space [1].

As shown above, due to the high capacity of modeling complex structured data, e.g. scale-free, hierarchical or cyclic, there has been an growing interest in building deep learning models under non-Euclidean geometry, e.g. link prediction [2], recommendation [3].

What's CurvLearn

In this repository, we provide a framework, named CurvLearn, for training deep learning models in non-Euclidean spaces.

The framework implements the non-Euclidean operations in Tensorflow and remains the similar interface style for developing deep learning models.

Currently, CurvLearn serves for training several recommendation models in Alibaba. We implement CurvLearn on top of our distributed (graph/deep learning) training engines including Euler and x-deeplearning. The figure below shows how the category tree is embedded in hyperbolic space by using CurvLearn.

Why CurvLearn

CurvLearn has the following major features.

  1. Easy-to-Use. Converting a Tensorflow model from Euclidean space to non-Euclidean spaces with CurvLearn is graceful and undemanding, due to the manifold operations are decoupled from model architecture and similar to vanilla Tensorflow operations. For researchers, CurvLearn also reserves lucid interfaces for developing novel manifolds and optimizers.
  2. Comprehensive methods. CurvLearn is the first Tensorflow based non-Euclidean deep learning framework and supports several typical non-Euclidean spaces, e.g. constant curvature and mixed-curvature manifolds, together with necessary manifold operations and optimizers.
  3. Verified by tremendous industrial traffic. CurvLearn is serving on Alibaba's sponsored search platform with billions of online traffic in several key scenarios e.g. matching and cate prediction. Compared to Euclidean models, CurvLearn can bring more revenue and the RPM (revenue per mille) increases more than 1%.

Now we are working on exploring more non-Euclidean methods and integrating operations with Tensorflow. PR is welcomed!

CurvLearn Architecture

Manifolds

We implemented several types of constant curvature manifolds and the mixed-curvature manifold.

  • curvlearn.manifolds.Euclidean - Euclidean space with zero curvature.
  • curvlearn.manifolds.Stereographic - Constant curvature stereographic projection model. The curvature can be positive, negative or zero.
  • curvlearn.manifolds.PoincareBall - The stereographic projection of the Lorentz model with negative curvature.
  • curvlearn.manifolds.ProjectedSphere - The stereographic projection of the sphere model with positive curvature.
  • curvlearn.manifolds.Product - Mixed-curvature space consists of multiple manifolds with different curvatures.

Operations

To build a non-Euclidean deep neural network, we implemented several basic neural network operations. Complex operations can be decomposed into basic operations explicitly or realized in tangent space implicitly.

  • variable(t, c) - Defines a riemannian variable from manifold or tangent space at origin according to its name.
  • to_manifold(t, c, base) - Converts a tensor t in the tangent space of base point to the manifold.
  • to_tangent(t, c, base) - Converts a tensor t in the manifold to the tangent space of base point.
  • weight_sum(tensor_list, a, c) - Computes the sum of tensor list tensor_list with weight list a.
  • mean(t, c, axis) - Computes the average of elements along axis dimension of a tensor t.
  • sum(t, c, axis) - Computes the sum of elements along axis dimension of a tensor t.
  • concat(tensor_list, c, axis) - Concatenates tensor list tensor_list along axis dimension.
  • matmul(t, m, c) - Multiplies tensor t by euclidean matrix m.
  • add(x, y, c) - Adds tensor x and tensor y.
  • add_bias(t, b, c) - Adds a euclidean bias vector b to tensor t.
  • activation(t, c_in, c_out, act) - Computes the value of activation function act for the input tensor t.
  • linear(t, in_dim, out_dim, c_in, c_out, act, scope) - Computes the linear transformation for the input tensor t.
  • distance(src, tar, c) - Computes the squared geodesic/distance between src and tar.

Optimizers

We also implemented several typical riemannian optimizers. Please refer to [4] for more details.

  • curvlearn.optimizers.rsgd - Riemannian stochastic gradient optimizer.
  • curvlearn.optimizers.radagrad - Riemannian Adagrad optimizer.
  • curvlearn.optimizers.radam - Riemannian Adam optimizer.

How to use CurvLearn

To get started with CurvLearn quickly, we provide a simple binary classification model as a quick start and three representative examples for the application demo. Note that the non-Euclidean model is sensitive to the hyper-parameters such as learning rate, loss functions, optimizers, and initializers. It is necessary to tune those hyper-parameters when transferring to other datasets.

Installation

CurvLearn requires tensorflow~=1.15, compatible with both python 2/3.

The preferred way for installing is via pip.

pip install curvlearn

Quick Start

Here we show how to build binary classification model using CurvLearn. Model includes Stereographic manifold, linear operations , radam optimizer, etc.

Instructions and implement details are shown in Quick Start.

HGCN on Link Prediction [2]

HGCN (Hyperbolic Graph Convolutional Neural Network) is the first inductive hyperbolic GCN that leverages both the expressiveness of GCNs and hyperbolic geometry to learn inductive node representations for hierarchical and scale-free graphs. Run the command to check the accuracy on the OpenFlight airport dataset. Running environment and performance are listed in hgcn.

python examples/hgcn/train.py

HyperML on Recommendation Ranking [3]

HyperML (Hyperbolic Metric Learning) applies hyperbolic geometry to recommender systems through metric learning approach and achieves state-of-the-art performance on multiple benchmark datasets. Run the command to check the accuracy on the Amazon Kindle-Store dataset. Running environment and performance are listed in hyperml.

python examples/hyperml/train.py

Hyper Tree Pre-train Model

In the real-world, data is often organized in tree-like structure or can be represented hierarchically. It has been proven that hyperbolic deep neural networks have significant advantages over tree-data representation than Euclidean models. In this case, we present a hyperbolic graph pre-train model for category tree in Taobao. The further details including dataset description, model architecture and visualization of results can be found in CateTreePretrain.

python examples/tree_pretrain/run_model.py

References

[1] Bachmann, Gregor, Gary Bécigneul, and Octavian Ganea. "Constant curvature graph convolutional networks." International Conference on Machine Learning. PMLR, 2020.

[2] Chami, Ines, et al. "Hyperbolic graph convolutional neural networks." Advances in neural information processing systems 32 (2019): 4868-4879.

[3] Vinh Tran, Lucas, et al. "Hyperml: A boosting metric learning approach in hyperbolic space for recommender systems." Proceedings of the 13th International Conference on Web Search and Data Mining. 2020.

[4] Bécigneul, Gary, and Octavian-Eugen Ganea. "Riemannian adaptive optimization methods." arXiv preprint arXiv:1810.00760 (2018).

License

This project is licensed under the Apache License, Version 2.0, unless otherwise explicitly stated.

You might also like...
Deep Learning Package based on TensorFlow
Deep Learning Package based on TensorFlow

White-Box-Layer is a Python module for deep learning built on top of TensorFlow and is distributed under the MIT license. The project was started in M

Realtime Face Anti Spoofing with Face Detector based on Deep Learning using Tensorflow/Keras and OpenCV
Realtime Face Anti Spoofing with Face Detector based on Deep Learning using Tensorflow/Keras and OpenCV

Realtime Face Anti-Spoofing Detection 🤖 Realtime Face Anti Spoofing Detection with Face Detector to detect real and fake faces Please star this repo

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform
TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform

StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion

StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion Yinghao Aaron Li, Ali Zare, Nima Mesgarani We pres

Adversarial Framework for (non-) Parametric Image Stylisation Mosaics
Adversarial Framework for (non-) Parametric Image Stylisation Mosaics

Fully Adversarial Mosaics (FAMOS) Pytorch implementation of the paper "Copy the Old or Paint Anew? An Adversarial Framework for (non-) Parametric Imag

Custom Implementation of Non-Deep Networks
Custom Implementation of Non-Deep Networks

ParNet Custom Implementation of Non-deep Networks arXiv:2110.07641 Ankit Goyal, Alexey Bochkovskiy, Jia Deng, Vladlen Koltun Official Repository https

PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020).
PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020).

NHDRRNet-PyTorch This is the PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020). 0. Differences between Original Paper and

Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity
Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

This repository is the official PyTorch implementation of Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

Deep learning (neural network) based remote photoplethysmography: how to extract pulse signal from video using deep learning tools

Deep-rPPG: Camera-based pulse estimation using deep learning tools Deep learning (neural network) based remote photoplethysmography: how to extract pu

Comments
  • Help needed

    Help needed

    Hello, I'm trying to implement HyperML on my data using the manifold PoincareBall. I'm not familiar with TensorFlow so it has been a little hard for me to understand if I need to change some parameters in order for the model to work well. Until now, I've made these changes: image I tried to implement the model but it got stuck on step 1000 and the HR metric doesn't print. You can see that below: image

    I am doing something wrong? Another issue relates to the results. When I get to the end of the training, how can I check if my model worked and get some recommendations for a specific user?

    Thanks in advance!

    opened by tita1999 4
  • How can i run with the product of manifolds

    How can i run with the product of manifolds

    Hi teams, thank you for the quick support. I am trying to run these models with the product of manifolds. For example, with HGCN, in the config.py, I used manifold with Product(Euclidean(), Stereographic()). But it seems doesn't work and has a bug. Did I do any wrong? Thank you so much.

    opened by nguyentuc 0
Owner
Alibaba
Alibaba Open Source
Alibaba
A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

Confluence: A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection 1. 介绍 用以替代 NMS,在所有 bbox 中挑选出最优的集合。 NMS 仅考虑了 bbox 的得分,然后根据 IOU 来

null 44 Sep 15, 2022
KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

null 80 Dec 27, 2022
This is a template for the Non-autoregressive Deep Learning-Based TTS model (in PyTorch).

Non-autoregressive Deep Learning-Based TTS Template This is a template for the Non-autoregressive TTS model. It contains Data Preprocessing Pipeline D

Keon Lee 13 Dec 5, 2022
A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation

Aboleth A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation [1] with stochastic gradient variational Bayes

Gradient Institute 127 Dec 12, 2022
This is the repository for paper NEEDLE: Towards Non-invertible Backdoor Attack to Deep Learning Models.

This is the repository for paper NEEDLE: Towards Non-invertible Backdoor Attack to Deep Learning Models.

null 1 Oct 25, 2021
ESGD-M - A stochastic non-convex second order optimizer, suitable for training deep learning models, for PyTorch

ESGD-M - A stochastic non-convex second order optimizer, suitable for training deep learning models, for PyTorch

Katherine Crowson 53 Dec 29, 2022
Ivy is a templated deep learning framework which maximizes the portability of deep learning codebases.

Ivy is a templated deep learning framework which maximizes the portability of deep learning codebases. Ivy wraps the functional APIs of existing frameworks. Framework-agnostic functions, libraries and layers can then be written using Ivy, with simultaneous support for all frameworks. Ivy currently supports Jax, TensorFlow, PyTorch, MXNet and Numpy. Check out the docs for more info!

Ivy 8.2k Jan 2, 2023
Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.

Deep Image Search - AI-Based Image Search Engine Deep Image Search is an AI-based image search engine that includes deep transfer learning features Ex

null 139 Jan 1, 2023