hyppo is an open-source software package for multivariate hypothesis testing.

Last update: Dec 18, 2022

Overview

hyppo

hyppo (HYPothesis Testing in PythOn, pronounced "Hippo") is an open-source software package for multivariate hypothesis testing. We decided to develop hyppo for the following reasons:

With the increase in the amount of data in many fields, hypothesis testing for high dimensional and nonlinear data is important
Libraries in R exist, but their interfaces are inconsistent and most are not available in Python

hyppo intends to be a comprehensive multivariate hypothesis testing package and runs on all major versions of operating systems. It also includes novel tests not found in other packages. It is quick to install and free of charge. If you need to use multivariate hypothesis testing, be sure to give hyppo a try!

Website: https://hyppo.neurodata.io/

Installation

Dependencies

hyppo requires the following:

python (>= 3.6)
numba (>= 0.46)
numpy (>= 1.17)
scipy (>= 1.4.0)
scikit-learn (>= 0.22)
joblib (>= 0.17.0)

User installation

The easiest way to install hyppo is using pip:

pip install hyppo

To upgrade to a newer release use the --upgrade flag:

pip install --upgrade hyppo

The documentation includes more detailed installation instructions. hyppo is free software; you can redistribute it and/or modify it under the terms of the license.

Changelog

See the changelog for a history of notable changes to hyppo.

Development

We welcome new contributors of all experience levels. The hyppo community goals are to be helpful, welcoming, and effective. The contributor guide has detailed information about contributing code, documentation and tests.

Official source code: https://github.com/neurodata/hyppo/tree/master/hyppo
Download releases: https://pypi.org/project/hyppo/
Issue tracker: https://github.com/neurodata/hyppo/issues

Note: We have recently moved our benchmarks (with relevant figure replication code for our papers) folder to a new repo! We aim to add test development code and paper figure replication code there, and will add relevant tests (with tutorials) to hyppo.

Project History

hyppo is a rebranding of mgcpy, which was founded in Novemember 2018. mgcpy was designed and written by Satish Palaniappan, Sambit Panda, Junhao Xiong, Sandhya Ramachandran, and Ronak Mehtra. hyppo was designed and written by Sambit Panda and first released February 2020.

Comments

MGC redundancy warning (#125)
Reference issue

Add warning when X or Y have redundant rows; close #125

Type of change

Documentation

What does this implement/fix?

Added a function check_redundancy in mgc.py to check if redundancies are present in x or y and report a warning to the user.

Additional information

I used the following code to find if redundant rows exist

def check_redundancy(self, x, y): """Check if there are redundant rows in input arrays x and y""" combined = np.column_stack([x, y]) unique_rows = np.unique(combined, axis=0) redundancy_flag = combined.shape != unique_rows.shape if redundancy_flag: warnings.warn("Redundant rows exist")

Added a test in test_mgc.py for checking if UserWarning is printed successfully.
opened by Verathagnus 24
Add discriminability (one sample and two sample)

Closes https://github.com/sampan501/mgc/issues/13.

Added functionality for one sample test (@sampan501 ). This performs a one-sample test for whether the discriminability differs from random chance, as described in: https://www.biorxiv.org/content/10.1101/802629v1.full
enhancement

opened by jdey4 13
Redundancy check for MGC
Reference issue

Add warning when X or Y have redundant rows; close #125

Type of change

Documentation

What does this implement/fix?

Added a function _check_redundancy in mgc.py to check if redundancies are present in x or y and report a warning to the user.

Additional information

I used the following code to find if redundant rows exist

def _check_redundancy(self, x, y, mgc_dict): """Check if there are redundant rows in input arrays x and y""" if x.shape[0] == 1: return if [len(x), len(y)] > mgc_dict["opt_scale"]: warnings.warn("Redundant rows exist")

Added a test in test_mgc.py for checking if UserWarning is printed successfully.
opened by Verathagnus 10
Allow for passing a random state to test objects and use rng instead of np.random

Is your feature request related to a problem? Please describe. I don't see any way to control the random state for most tests (e.g. dcorr)

Is there some way that I am missing?

Describe the solution you'd like Be able to pass a random state/seed to the independence (or other) test object to enable reproducibility

Describe alternatives you've considered setting the global numpy random state - but I'm not sure how this plays with parallelization, different platforms, etc. and have found this to be unreliable in the past.

Example https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html as an example of passing a random state
enhancement question

opened by bdpedigo 10
Discriminability `y` should allow arbitrary type or make docs more clear

Is your feature request related to a problem? Please describe. Currently, the y argument to Discriminability.test(x, y) will raise an error if a vector of strings is passed. The error is just that a string array cannot be cast to float.

The docs currently just say nd.array and don't clarify what type of values are accepted. (See here)

Describe the solution you'd like I don't see why the y argument needs to be cast to float (here), but maybe I'm missing something? From a quick look at the code I don't think anything would break if you just didn't cast this array. Especially since most of the time, these will be ints, i'd imagine.

Describe alternatives you've considered If you disagree or I'm wrong, the docs should at least be clarified as to what types are accepted, and a check for that data type in the array seems reasonable.
documentation good first issue hyppo.discrim

opened by bdpedigo 10
Added kernel goodness of fit

Reference issue
Close #103
Type of change
Addition of functionality
What does this implement/fix?
This will implement the kernel goodness of fit (Jitkrittum, Xu, Szabo, Fukumizu & Gretton 2017): https://proceedings.neurips.cc/paper/2017/file/979d472a84804b9f647bc185a877a8b5-Paper.pdf. A new module "kgof" is added in hyppo/tools, I am attaching a .txt file with the code
kgof in hyppo demonstration.txt to demonstrate it. All credit goes to Jitkrittum et al. for the kernel goodness of fit. Note: This comes with an added setup requirement for hyppo: autograd>=1.1.7 since it computes derivatives for optimization and replaces NumPy.

Additional information
This is only the implementation of the first link given in #103 -- the second, (Jitkrittum, Szabo & Gretton 2017), is not implemented.
Why I think CircleCI "run code linting" is failing for tools: Since now there is a new requirement of autograd, I linted goftest.py (where the error originated) to find 2 errors: E: 15, 0: Unable to import 'autograd' (import-error) E: 16, 0: Unable to import 'autograd.numpy' (import-error)

But I've added autograd in the setup.py and requirements.txt. So I'm guessing CircleCI isn't still considering autograd a requirement.. or I'm missing another spot to specify autograd as the new requirement (apart from setup.py and requirements.txt). Runs fine locally.
enhancement

opened by GM-git-dotcom 9
Edited the types in the documentation section.

Reference issue

closes #140 closes #245

Type of change

Documentation

What does this implement/fix?

As of issue #140 the types of both x and y are changed to ndarray of floats in the documentation.

Additional information

opened by ghost 8
potential k-sample tests to add
[x] https://projecteuclid.org/euclid.aos/1176344722

[x] https://arxiv.org/abs/1506.04725

[x] http://papers.nips.cc/paper/6148-interpretable-distribution-features-with-maximum-testing-power

enhancement ndd hyppo.ksample
opened by sampan501 8
Fix #228

Adds rng.permutation reproducibility to perm_test @kareef928 from issue #228 . Should all work, but someone can PR to this or extend the modifications.

opened by rflperry 7

[BUG] scipy 1.8 breaks hyppo import

Reproducing code example:

import hyppo.discrim

Error message

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/hyppo/discrim/__init__.py:1: in <module>
    from .discrim_one_samp import DiscrimOneSample
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/hyppo/discrim/discrim_one_samp.py:5: in <module>
    from ._utils import _CheckInputs
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/hyppo/discrim/_utils.py:4: in <module>
    from ..tools import check_ndarray_xy, check_reps, contains_nan, convert_xy_float64
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/hyppo/tools/__init__.py:1: in <module>
    from .common import *
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/hyppo/tools/common.py:6: in <module>
    from scipy.stats.stats import _contains_nan
E   ImportError: cannot import name '_contains_nan' from 'scipy.stats.stats' (/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/scipy/stats/stats.py)

The function _contains_nan() was moved to a different file in scipy 1.8. In general, this is why it's a bad idea to rely on importing private functions.

Recommend just copy-pasting the private function into hyppo explicitly.

bug

opened by bdpedigo 6

Fast HHG Test

Reference issue

Aims to close #106: Add Fast HHG.

Type of change

Added fast version of independence test into hhg.py and edited independence tutorial.

What does this implement/fix?

Adds fast HHG independence test.

Additional information

Compared to fast Dcorr, I made fast HHG opt-in rather than auto performing on 1D data due to low relative power of fast HHG in most scenarios. Ran through Black formatting check and successfully built through Sphinx.

opened by TacticalFallacy 6
Discriminability for a multivariate measure (64k vertices)

Hi there,

I've got a measure of shape (subjects, sessions, vertices). That is, the measure consists of multiple values per session per subject. The input shape required by hyppo.discrim.DiscrimOneSample is (subjects, sessions) which implies a univariate measure per session, as far as I understood it. What would you recommend to circumvent this?

Also, it would be useful if code examples in the documentation involved data resembling real data (e.g. fMRI, connectivity) and an example of a research question concerning reproducibility assessment.

Thanks in advance
question

opened by victoris93 0
Energy statistic is wrong
I was trying to compare the energy statistic implemented in hyppo with the one implemented in my own package, dcor. However, I found that the statistic in hyppo implements the energy statistic based in Theorem 2 from https://arxiv.org/pdf/1910.08883.pdf, but does it in a wrong way.

In https://github.com/neurodata/hyppo/blob/d0450180d0717a2f73ae55e1f8c1c8413e048a5e/hyppo/ksample/energy.py#L118 the coefficient should be inverted (as one implements the energy from the distance correlation, and not otherwise). Moreover the x passed to distance covariance should be the concatenation of x and y, and the y passed should be a label containing zero for samples in x and 1 for samples in y.

Reproducing code example:

from hyppo.ksample import Energy import dcor import numpy as np x = np.arange(100.0) y = x**2 print("dcor", dcor.energy_distance(x, y)) print("paper formula", dcor.distance_covariance_sqr(np.concatenate([x, y]), np.concatenate([np.zeros(len(x)), np.ones(len(y))])) * 200**4 / (2 * 100**2 * 100**2)) print("hyppo", hyppo.ksample.Energy(bias=True).statistic(x[:, None], y[:, None]))

Results

dcor 3146.5236000000004 paper formula 3146.5235999999986 hyppo 5501.374807500008

Version information

OS: Any

Python Version: Any

Package Version: 0.3.2

bug
opened by vnmabus 1
Conditional distance correlation for conditional independence?

Is your feature request related to a problem? Please describe. Wondering what jovo / your thoughts are on including a submodule for CI based testing?

Describe the solution you'd like E.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4749041/

I skimmed it and it introduces the conditional dcorr function which could be used for CI tests.

Describe alternatives you've considered

Additional context (e.g. screenshots)
enhancement

opened by adam2392 0
Add Range of Friedman Rafsky Test Statistic

My issue is about ... Updating the documentation to include the range of the Friedman Rafsky test statistic

Snapshot of documentation error:

Additional context

This is a sub-issue of issue 121.

I have been investigating the ranges of some test statistics, and I have made some progress. I would like to contribute the ranges I have determined, starting with Friedman Rafsky.
documentation

opened by oakla 0

Releases(v0.3.2)

v0.3.2(Feb 10, 2022)

This release just fixes the bug of using emojis in the long description so the package can build on Windows.

Authors

Issues Closed

N/A

PRs Merged

N/A
Source code(tar.gz)
Source code(zip)
v0.3.1(Feb 10, 2022)

This release just fixes the bug of not adding autograd as a dependency.

Authors

Issues Closed

N/A

PRs Merged

N/A
Source code(tar.gz)
Source code(zip)
v0.3.0(Feb 10, 2022)
hyppo v0.3.0 is the culmination of 12 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations.

This release requires Python 3.6+.

Highlights of this release

New features

Added goodness of fit module with FSSD

Added d-variate independence testing module with dHsic

Added fast HHG

Added Friedman Rafsky test

Added Smooth CF Test

Added Mean Embedding Test

Bug fixes

Copy _contains_nan function from SciPy since it is gone

Documentation

Made documentation more accessible with a new theme

Added example for block permutation

Mentioned types of inputs of ndarrays

Maintenance

Drafted PR to include Dcorr in SciPy

Bumped iPython for security reasons

Authors

Issues Closed

#103: Create a goodness of fit module in hyppo

#104: Add dHsic to hyppo

#106: Add fast HHG

#140: Discriminability y should allow arbitrary type or make docs more clear

#187: potential k-sample tests to add

#211: Fast Two-Sample Testing with Analytic Representations of Probability Measures

#235: Add existing permutation tree simulation notebook to docs

#245: Add more description about types of ndarray in all methods in documentation

#249: Add contributors to README

#303: [BUG] scipy 1.8 breaks hyppo import

PRs Merged

#232: Creating a goodness-of-fit module in hyppo

#233: Adding dHSIC

#234: Fast tstest

#238: Fast HHG Test

#239: Friedman Rafsky PR

#242: add permutation test example to docs

#244: Edited the types in the documentation section.

#299: Bump ipython from 7.19.0 to 7.31.1 in /docs

#304: Copy SciPy private _contains_nan function

Source code(tar.gz)
Source code(zip)
v0.2.2(Dec 7, 2021)
Note: People with a "+" by their names contributed a patch for the first time.

Release date: 7 December 2021 Supports: Python 3.6+.

New features

Functions now can be reproducible using a random_state parameter

Add warning for redundant rows when running :class:hyppo.independence.MGC

Tests now also retun a namedtuple

Bug Fixes

Change from L1 to L2 distance for the median heuristic

Make median heuristic default for RBF and Gaussian kernels

Documentation

Update license in setup.py

Change reference style and store reference in a refs.bib file

Updated papers in documentation from preprint to published versions

Rename badges in README

Maintenance

Added support for Python 3.9

Update pytest orbs version

Make repository citeable with the new GitHub feature

Authors

@sampan501

@kareef928 +

@rflperry

@Verathagnus +

@PSSF23 +

@hadasarik +

Source code(tar.gz)
Source code(zip)
v0.2.1(Feb 25, 2021)
Note: People with a "+" by their names contributed a patch for the first time.

Release date: 25 February 2021 Supports: Python 3.6+.

Bug Fixes

hyppo.independence.Dcorr and hyppo.independence.Hsic when auto=True had low finite power, fixed.

Fast exact statistic for hyppo.independence.Dcorr was incorrect, fixed

Documentation

Fix PyPi description rendering issues

Add more descriptive contributing guidelines

Add ROLES.md for specification about maintainers

Added power computation for independence increasing sample size and dimension

Add benchmark section and move relevant examples there

Add base classes

Maintenance

Fix gaussian kernel to prevent division by 0

Add checks for Type I error

Authors

Sambit Panda

Source code(tar.gz)
Source code(zip)
v0.2.0(Feb 8, 2021)
Note: People with a "+" by their names contributed a patch for the first time.

Release date: 08 February 2021 Supports: Python 3.6+.

New features

Added restricted permutation functionality for Dcorr

Kernel functions now wrap scikit-learn and support keyword arguments

hyppo.ksample.Energy

hyppo.ksample.DISCO

hyppo.ksample.MMD

Fast 1D exact Dcorr O(n log n)

hyppo.ksample.Hotelling

hyppo.ksample.MANOVA

hyppo.independence.MaxMargin

Bug Fixes

Fixed error check for k-sample tests to be between samples instead of within

Time series doesn't clip negative values

Fix docs not building on netlify

Fix p-value calculations for permutation tests to be more in line with the literature

Fix hyppo.independence.Dcorr and hyppo.independence.Hsic incorrect stats

Documentation

Update badges and README to FIRM guidelines

Incorrect equation in :meth:hyppo.tools.circle docstring

Update README to be in line with scikit-learn

Remove literature reference section in docstrings, add links to papers

Add docstrings for :mod:hyppo.tools functions

Add overview.py file for an overview of the package

Add tutorials folder, rewrite so it is more user-friendly (port independence, k-sample, time series)

Add examples folder with more information about unique features

Move to sphinx-gallery instead of nbconvert

Use automodule instead of autoclass

Make clear about the package requirements and docs requirements

Make changelog into a single file

Add external links to neurodata and code of conduct

Add citing page to cite the package papers

Make index page a clone of README

Update MakeFile for more options

Add intersphinx mapping with links externally (numpy, scipy, etc.) and internally

Add docs for statistic function

Add discriminability tutorial

Maintenance

Fix typos in warning commits

Updated tests to precalculate distance matrix

Moved from Travis CI to Circle CI

Raise base requirements.txt to fix failing tests on CircleCI

Add code coverage config files

Add documentation folders and files to .gitignore

Remove reps warning test

Cache numba after first call to speed up runs

Fix netlify config to new doc build structure

Authors

Sambit Panda

Vivek Gopalakrishnan +

Ronak Mehta

Ronan Perry +

Source code(tar.gz)
Source code(zip)
v0.1.3(Jul 24, 2020)
hyppo 0.1.3 does not include any new features, just fixes. This release requires Python 3.6+.

Fixes

k-sample testing

Prevent division by zero when calculating using default gaussian median kernel.

Other

Used chi2.sf instead of 1 - chi2.cdf

Authors

This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order by last name):

Benjamin Pedigo +

Anton Alayakin +

People with a "+" by their names contributed a patch for the first time.
Source code(tar.gz)
Source code(zip)
v0.1.2(May 6, 2020)
hyppo 0.1.2 does not include any new features, just fixes. This release requires Python 3.6+.

Fixes

k-sample testing

Fixed MMD/k-samp;e Hsic not running bug.

Authors

This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order by last name):

Sambit Panda

A total of 1 person contributed to this release. People with a "+" by their names contributed a patch for the first time.
Source code(tar.gz)
Source code(zip)
v0.1.1(Apr 29, 2020)
hyppo 0.1.1 does not include any new features, just removal of extraneous features.

Release highlights:

Kernel matrices calculated once before calculating p-values

Pearson, Kendall, and Spearman are no longer tests

This release requires Python 3.6+.

Improvements

Independence testing

Null distribution added as a class atribute. Calculate kernel once before calculating p-value.

k-sample testing

Null distribution added as a class atribute. Calculate kernel once before calculating p-value. Upper and lower-case inputs are available for indep_test.

Time series

Reference docs and tutorials added to Time Series module.

Other changes

OS/Software requirements and license changes updated in ReadME. pairwise_distances from sklearn used instead of cdist from scipy.

Removed features

Pearson, Spearman, and Kendall are no longer tests within the package. arXiv badge added to docs. Python 3.5 no longer supported.

Authors

This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order by last name):

Ronak Mehta +

Sambit Panda

Bijan Varjavand +

A total of 3 people contributed to this release. People with a "+" by their names contributed a patch for the first time.
Source code(tar.gz)
Source code(zip)
v0.1.0(Feb 26, 2020)
hyppo 0.1.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. All users are encouraged to use this release instead of the mgcpy, which this package is replacing, are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on this branch, and on adding new features on the development master branch.

Release highlights (as compared to mgcpy):

New time series and discriminability modules

More user-friendly implementation of k-Sample Testing

This release requires Python 3.5+.

New features

Independence testing

Tests have been given a unique class as compared to mgcpy. Parallelization in this module makes tests faster and Hsic has been added as a standalone test within this package (uses exact equivalence implementation).

k-sample testing

k-Sample testing is now organized within a class that is modeled similarly to how independence tests are run. More information about the specifics of how this works can be found in the docs.

Time series

Time series based independence tests have been included as a separate module from independence tests.

Simulations

Simulations have been included as a separate module and hyppo.sims now includes k-sample simulations and time series simulations.

Discriminability

Discriminability has been included from the r-mgc package and has been changed so it conforms to the hyppo API.

Benchmarks

A benchmarks folder as been added that contains notebooks comparing statistical power, algorithm wall times, and test statistics comparisons between the algorithms and sometimes between the respective R implementations. Many of those notebooks have been condensed into tutorials in the documentation.

Other changes

API has been changed as compared to mgcpy and is modeled after scikit-learn with the inclusion of the base.py containing an abstract class within each module and energy with a .test method calculating test statistic in p-value.

Authors

This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order by last name):

Jayanta Dey +

Sambit Panda +

A total of 2 people contributed to this release. People with a "+" by their names contributed a patch for the first time.
Source code(tar.gz)
Source code(zip)

Owner

neurodata

GitHub https://hyppo.neurodata.io/

PacketPy is an open-source solution for stress testing network devices using different testing methods

PacketPy About PacketPy is an open-source solution for stress testing network devices using different testing methods. Currently, there are only two c

4 Sep 22, 2022

PENBUD is penetration testing buddy which helps you in penetration testing by making various important tools interactive.

penbud - Penetration Tester Buddy PENBUD is penetration testing buddy which helps you in penetration testing by making various important tools interac

15 Feb 1, 2022

A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax

1.0 Release See here for details about breaking changes with the upcoming 1.0 release: https://github.com/taverntesting/tavern/issues/495 Easier API t

909 Dec 15, 2022

hyppo is an open-source software package for multivariate hypothesis testing.

Related tags

Overview

hyppo

Installation

Dependencies

User installation

Changelog

Development

Project History

Comments

Reference issue

Type of change

What does this implement/fix?

Additional information

Reference issue

Type of change

What does this implement/fix?

Additional information

Reference issue

Type of change

What does this implement/fix?

Additional information

Reference issue

Type of change

What does this implement/fix?

Additional information

Reproducing code example:

Error message

Reference issue

Type of change

What does this implement/fix?

Additional information

Reproducing code example:

Results

Version information

Snapshot of documentation error:

Additional context

Releases(v0.3.2)

v0.3.2(Feb 10, 2022)

Authors

Issues Closed

PRs Merged

v0.3.1(Feb 10, 2022)

Authors

Issues Closed

PRs Merged

v0.3.0(Feb 10, 2022)

Highlights of this release

New features

Bug fixes

Documentation

Maintenance

Authors

Issues Closed

PRs Merged

v0.2.2(Dec 7, 2021)

New features

Bug Fixes

Documentation

Maintenance

Authors

v0.2.1(Feb 25, 2021)

Bug Fixes

Documentation

Maintenance

Authors

v0.2.0(Feb 8, 2021)

New features

Bug Fixes

Documentation

Maintenance

Authors

v0.1.3(Jul 24, 2020)

Fixes

k-sample testing

Other

Authors

v0.1.2(May 6, 2020)

Fixes