Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Karn Watcharasupat

Last update: Sep 8, 2022

Related tags

Overview

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Latte

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

Functional API

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Modular API

import latte
from latte.metrics.core.disentanglement import MutualInformationGap
import numpy as np

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.compute()

TorchMetrics API

import latte
from latte.metrics.torch.disentanglement import MutualInformationGap
import torch

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update(z, attributes)

mig_val = mig.compute()

Keras Metric API

import latte
from latte.metrics.keras.disentanglement import MutualInformationGap
from tensorflow import keras as tfk

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.result()

Documentation

https://latte.readthedocs.io/en/latest

Supported metrics

Metric	Latte Functional	Latte Modular	TorchMetrics	Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG)	🧪	🧪	🧪	??
📝 Dependency-blind Mutual Information Gap (DMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Mutual Information Gap (XMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Latent Information Gap (DLIG)	🧪	🧪	🧪	🧪
📝 Separate Attribute Predictability (SAP)	🧪	🧪	🧪	🧪
📝 Modularity	🧪	🧪	🧪	🧪
📝 β-VAE Score	👀	👀	👀	👀
📝 FactorVAE Score	👀	👀	👀	👀
📝 DCI Score	👀	👀	👀	👀
📝 Interventional Robustness Score (IRS)	👀	👀	👀	👀
📝 Consistency	👀	👀	👀	👀
📝 Restrictiveness	👀	👀	👀	👀
Interpolatability Metrics
📝 Smoothness	🧪	🧪	🧪	🧪
📝 Monotonicity	🧪	🧪	🧪	🧪
📝 Latent Density Ratio	🕣	🕣	🕣	🕣
📝 Linearity	👀	👀	👀	👀

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle	Latte Functional	Latte Modular	TorchMetrics	Keras Metric	Included
Dependency-aware Disentanglement	🧪	🧪	🧪	🧪	MIG, DMIG, XMIG, DLIG
LIAD-based Interpolatability	🧪	🧪	🧪	🧪	Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.

If you find our package useful please cite our repository and arXiv preprint as

@article{
  watcharasupat2021latte,
  author = {Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},
  title = {{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},
  eprint={2112.10638},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url = {https://github.com/karnwatcharasupat/latte}
  doi = {10.5281/zenodo.5786402}
}

Comments

Documentation: Metric Descriptions

Might be nice to provide a short description for each metric in addition to the paper links. The readme might get too long with it, but either some doc in the repo or maybe on a github.io page?
type: documentation priority: high

opened by alexanderlerch 2
Add Smoothness and Monotonicity support
Smoothness

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

Monotonicity

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add Modularity support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add SAP support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add DMIG, DLIG, XMIG support
DMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

XMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

DLIG

[ x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add MIG support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Support issue for on-the-fly computation in TF2 graph mode
The current delegate-to-NumPy technique used in TF is only compatible with TF2 eager mode since Tensor.numpy() would not work in graph mode. As a result, graph-mode users will only be able to use Latte in the evaluation stage when the model weights are no longer changing but not on-the-fly during the training stage.

However, certain computation steps required for some metrics (especially MI-based ones) necessarily require scikit-learn ops and there is no (maintainable) way to create consistent TF mirrors of those functions.

One potential solution is to wrap the core functions in tf.numpy_function or tf.py_function but we will have to figure out a way to make the wrapper less painful to implement/maintain since the variable args/kwargs option currently used by the dtype converter is not allowed in these functions. A naive workaround would be to make a tf.numpy_function wrapper for every highest-possible level function with fixed args but this would be considered a last-resort solution.

Links:

https://www.tensorflow.org/api_docs/python/tf/numpy_function

https://www.tensorflow.org/api_docs/python/tf/py_function

type: enhancement priority: medium !! needs more brains !!
opened by karnwatcharasupat 3

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)
What's Changed

Add contributing guide by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/16

[ADD] add example notebooks by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/18

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha3...v0.0.1-alpha5
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha3(Dec 16, 2021)
Added metric bundle support

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha2...v0.0.1-alpha3
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha2(Dec 9, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha1...v0.0.1-alpha2
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha1(Dec 1, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-dev5...v0.0.1-alpha1
Source code(tar.gz)
Source code(zip)

Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Related tags

Overview

Latte

Installation

Running tests locally

Example

Functional API

Modular API

TorchMetrics API

Keras Metric API

Documentation

Supported metrics

Bundled metric modules

Cite

Comments

Documentation: Metric Descriptions

Add Smoothness and Monotonicity support

Smoothness

Monotonicity

Add Modularity support

Add SAP support

Add DMIG, DLIG, XMIG support

DMIG

XMIG

DLIG

Add MIG support

Support issue for on-the-fly computation in TF2 graph mode

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)

What's Changed

v0.0.1-alpha3(Dec 16, 2021)

v0.0.1-alpha2(Dec 9, 2021)

v0.0.1-alpha1(Dec 1, 2021)

Owner

Karn Watcharasupat

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

Framework for joint representation learning, evaluation through multimodal registration and comparison with image translation based approaches

High-Resolution Image Synthesis with Latent Diffusion Models

PyTorch implementation and pretrained models for XCiT models. See XCiT: Cross-Covariance Image Transformer

PyTorch implementation of "Learning to Discover Cross-Domain Relations with Generative Adversarial Networks"

Official implementation of "Learning to Discover Cross-Domain Relations with Generative Adversarial Networks"

PyTorch implementation of the cross-modality generative model that synthesizes dance from music.

CVPR 2021 Official Pytorch Code for UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Cross Quality LFW: A database for Analyzing Cross-Resolution Image Face Recognition in Unconstrained Environments

Official Implementation of LARGE: Latent-Based Regression through GAN Semantics

Attention-based Transformation from Latent Features to Point Clouds (AAAI 2022)

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

An evaluation toolkit for voice conversion models.

A large-scale video dataset for the training and evaluation of 3D human pose estimation models

A large-scale video dataset for the training and evaluation of 3D human pose estimation models

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment