50% faster, 50% less RAM Machine Learning. Numba rewritten Sklearn. SVD, NNMF, PCA, LinearReg, RidgeReg, Randomized, Truncated SVD/PCA, CSR Matrices all 50+% faster

Overview

[Due to the time taken @ uni, work + hell breaking loose in my life, since things have calmed down a bit, will continue commiting!!!] [By the way, I'm still looking for new contributors! Please help make HyperLearn no1!!]

drawing

HyperLearn is what drives Umbra's AI engines. It is open source to everyone, everywhere, and we hope humanity can rise to the stars.

[Notice - I will be updating the package monthly or bi-weekly due to other commitments]


drawing https://hyperlearn.readthedocs.io/en/latest/index.html

Faster, Leaner GPU Sklearn, Statsmodels written in PyTorch

GitHub issues Github All Releases

50%+ Faster, 50%+ less RAM usage, GPU support re-written Sklearn, Statsmodels combo with new novel algorithms.

HyperLearn is written completely in PyTorch, NoGil Numba, Numpy, Pandas, Scipy & LAPACK, and mirrors (mostly) Scikit Learn. HyperLearn also has statistical inference measures embedded, and can be called just like Scikit Learn's syntax (model.confidence_interval_) Ongoing documentation: https://hyperlearn.readthedocs.io/en/latest/index.html

I'm also writing a mini book! A sneak peak: drawing

drawing

Comparison of Speed / Memory

Algorithm n p Time(s) RAM(mb) Notes
Sklearn Hyperlearn Sklearn Hyperlearn
QDA (Quad Dis A) 1000000 100 54.2 22.25 2,700 1,200 Now parallelized
LinearRegression 1000000 100 5.81 0.381 700 10 Guaranteed stable & fast

Time(s) is Fit + Predict. RAM(mb) = max( RAM(Fit), RAM(Predict) )

I've also added some preliminary results for N = 5000, P = 6000 drawing

Since timings are not good, I have submitted 2 bug reports to Scipy + PyTorch:

  1. EIGH very very slow --> suggesting an easy fix #9212 https://github.com/scipy/scipy/issues/9212
  2. SVD very very slow and GELS gives nans, -inf #11174 https://github.com/pytorch/pytorch/issues/11174

Help is really needed! Message me!


Key Methodologies and Aims

1. Embarrassingly Parallel For Loops

2. 50%+ Faster, 50%+ Leaner

3. Why is Statsmodels sometimes unbearably slow?

4. Deep Learning Drop In Modules with PyTorch

5. 20%+ Less Code, Cleaner Clearer Code

6. Accessing Old and Exciting New Algorithms


1. Embarrassingly Parallel For Loops

  • Including Memory Sharing, Memory Management
  • CUDA Parallelism through PyTorch & Numba

2. 50%+ Faster, 50%+ Leaner

3. Why is Statsmodels sometimes unbearably slow?

  • Confidence, Prediction Intervals, Hypothesis Tests & Goodness of Fit tests for linear models are optimized.
  • Using Einstein Notation & Hadamard Products where possible.
  • Computing only what is necessary to compute (Diagonal of matrix and not entire matrix).
  • Fixing the flaws of Statsmodels on notation, speed, memory issues and storage of variables.

4. Deep Learning Drop In Modules with PyTorch

  • Using PyTorch to create Scikit-Learn like drop in replacements.

5. 20%+ Less Code, Cleaner Clearer Code

  • Using Decorators & Functions where possible.
  • Intuitive Middle Level Function names like (isTensor, isIterable).
  • Handles Parallelism easily through hyperlearn.multiprocessing

6. Accessing Old and Exciting New Algorithms

  • Matrix Completion algorithms - Non Negative Least Squares, NNMF
  • Batch Similarity Latent Dirichelt Allocation (BS-LDA)
  • Correlation Regression
  • Feasible Generalized Least Squares FGLS
  • Outlier Tolerant Regression
  • Multidimensional Spline Regression
  • Generalized MICE (any model drop in replacement)
  • Using Uber's Pyro for Bayesian Deep Learning

Goals & Development Schedule

Will Focus on & why:

1. Singular Value Decomposition & QR Decomposition

* SVD/QR is the backbone for many algorithms including:
    * Linear & Ridge Regression (Regression)
    * Statistical Inference for Regression methods (Inference)
    * Principal Component Analysis (Dimensionality Reduction)
    * Linear & Quadratic Discriminant Analysis (Classification & Dimensionality Reduction)
    * Pseudoinverse, Truncated SVD (Linear Algebra)
    * Latent Semantic Indexing LSI (NLP)
    * (new methods) Correlation Regression, FGLS, Outlier Tolerant Regression, Generalized MICE, Splines (Regression)

On Licensing: HyperLearn is under a GNU v3 License. This means:

  1. Commercial use is restricted. Only software with 0 cost can be released. Ie: no closed source versions are allowed.
  2. Using HyperLearn must entail all of the code being avaliable to everyone who uses your public software.
  3. HyperLearn is intended for academic, research and personal purposes. Any explicit commercialisation of the algorithms and anything inside HyperLearn is strictly prohibited.

HyperLearn promotes a free and just world. Hence, it is free to everyone, except for those who wish to commercialise on top of HyperLearn. Ongoing documentation: https://hyperlearn.readthedocs.io/en/latest/index.html [As of 2020, HyperLearn's license has been changed to BSD 3]

Issues
  • license

    license

    do you mind to pass to a MIT/BSD license instead of GPL?

    opened by mprosperi 5
  • New Documentation - ReadTheDocs !

    New Documentation - ReadTheDocs !

    Hey! I started documentation @ https://hyperlearn.readthedocs.io/en/latest/index.html

    opened by danielhanchen 5
  • Fix readme typo

    Fix readme typo

    null

    opened by erjanmx 1
  • Create CODE_OF_CONDUCT.md

    Create CODE_OF_CONDUCT.md

    opened by danielhanchen 0
  • ReadTheDocs Cython ImportError: Cannot import module ...

    ReadTheDocs Cython ImportError: Cannot import module ...

    This is just a post to highlight how the problem has been solved. Building locally works, however within RTD, the error showcases WARNING: autodoc: failed to import module 'hyperlearn.base'; the following exception was raised: No module named 'hyperlearn.cython.base'

    I've checked: https://stackoverflow.com/questions/13238736/how-to-document-cython-function-on-readthedocs/13364247, https://stackoverflow.com/questions/13372068/importerror-for-my-code-on-readthedocs, https://stackoverflow.com/questions/10324393/sphinx-build-fail-autodoc-cant-import-find-module, https://github.com/rtfd/readthedocs.org/issues/3634, https://github.com/rtfd/readthedocs.org/issues/2511, https://docs.readthedocs.io/en/latest/faq.html, https://github.com/astropy/astropy-healpix/issues/4, and https://github.com/mozilla/treeherder/commit/f17bcf82051300ce1ff012dc7f1d42919137800a.

    All didn't solve the problem. To solve it, in conf.py change sys.path.insert(0, os.path.abspath('.')) sys.path.insert(0, os.path.abspath('..')) sys.path.insert(0, os.path.abspath('../../'))

    to sys.path.insert(0, os.path.abspath('.hyperlearn'))

    Only the one line is enough to fix the problem!

    opened by danielhanchen 0
  • Install instructions?

    Install instructions?

    Hi there, Thanks for the package! It looks very cool and seems like you put alot of work into it. Kudos!!! Sadly I can't find installation instructions anywhere. Can you point me to them? Thanks,

    opened by hoangthienan95 3
  • [ANNOUNCEMENT] New plans for HyperLearn

    [ANNOUNCEMENT] New plans for HyperLearn

    Hello all. I was a bit busy during December - May due to uni (I'm a 4th year student :) ) + I'm part of NVIDIA's cuML team (check out cuML's algos! Theyre super fast on GPUs!)

    In terms of plans, I have a 3 week "holiday", so will maintain HyperLearn more. I've had many things I wanted to do, and plan to make things runnable without compiling via C, C++ etc.

    My main aims are:

    1. Delete all C/C++ dependencies, and rely solely on Numba + Scipy LAPACK.

    2. Focus on specific algos that are extremely relevant.

    a. Randomized + Dense PCA on Sparse Matrices (without converting sparse to dense). decomposition.PCA

    b. Porting Eigendecomposition via MRRR for the top K eigenvectors. [Scipy still hasn't..] decomposition.EIGH

    c. Fixing all memory copies for SVD, and support Randomized SVD on sparse matrices. decomposition.SVD

    d. Fix up LinearSolve and place it into 1 module with LSMR (super fast sparse solve), combine with Cholesky, SVD, Eig, etc solving + Ridge. linear_model.solve

    e. And finally, introduce a Python only modified version of Spotify's ANNOY library (a limited nearest neighbor KD-Tree based on other heuristics I found to be useful).

    If anyone has other suggestions, please mention them! If anyone wants to help, please msg / email me @ [email protected]!

    Thanks for all the support!

    opened by danielhanchen 0
  • Compiling numba code failes on Ubuntu 16.04

    Compiling numba code failes on Ubuntu 16.04

    numba.errors.LoweringError: Failed at nopython (nopython mode backend) Buffer dtype cannot be buffer

    File "hyperlearn/numba.py", line 126: def squaresum(v): s = v[0]**2 for i in prange(1,len(v)): ^ [1] During: lowering "id=6[LoopNest(index_variable = parfor_index.395, range = ($const28.2, $28.5, 1))]{44: <ir.Block at /home/lee/hyperlearn/hyperlearn/numba.py (126)>}Var(parfor_index.395, /home/lee/hyperlearn/hyperlearn/numba.py (126))" at /home/lee/hyperlearn/hyperlearn/numba.py (126)

    opened by MCDM2018 1
  • Undefined function in `randomized.linalg.svd`

    Undefined function in `randomized.linalg.svd`

    https://github.com/danielhanchen/hyperlearn/blob/a2af0ba0420be6a2f036c761e7ecae9f0e66c21a/hyperlearn/randomized/linalg.py#L299

    The name _min is not defined. Seems like that existed in hyperlearn_old/numba but not the current one.

    opened by themightyoarfish 1
  • Compiling Numba code fails on 16.04

    Compiling Numba code fails on 16.04

    At the end of python setup.py install, it says to run

    #### If you want to compile Numba code, please run:
        >>>>   python numba_compile.py
    

    to compile Numba. Doing this gives me

    Progress: |||||||||||||||Traceback (most recent call last):
      File "numba_compile.py", line 6, in <module>
        from hyperlearn.numba.funcs import *
      File "/home/share/software/sources/hyperlearn/hyperlearn/numba/funcs.py", line 4, in <module>
        from ..cfuncs import uinteger
      File "/home/share/software/sources/hyperlearn/hyperlearn/cfuncs.py", line 3, in <module>
        from .cython.base import isComplex, available_memory, isList
    ModuleNotFoundError: No module named 'hyperlearn.cython.base'
    
    opened by themightyoarfish 1
  • Cannot import hyperlearn subpackages

    Cannot import hyperlearn subpackages

    After python setup.py install I get

    ModuleNotFoundError: No module named 'hyperlearn.cython'
    

    when trying to import hyperlearn.(random|linalg|…). What could the problem be?

    opened by themightyoarfish 4
  • `setup.py` install fails on Ubuntu 16.04

    `setup.py` install fails on Ubuntu 16.04

    This is what I see when running python setup.py install (gcc8, python 3.6):

    running install
    running build
    running build_py
    file hyperlearn.py (for module hyperlearn) not found
    file hyperlearn.py (for module hyperlearn) not found
    running install_lib
    warning: install_lib: 'build/lib' does not exist -- no Python modules to install
    
    running install_egg_info
    running egg_info
    writing hyperlearn.egg-info/PKG-INFO
    writing dependency_links to hyperlearn.egg-info/dependency_links.txt
    writing requirements to hyperlearn.egg-info/requires.txt
    writing top-level names to hyperlearn.egg-info/top_level.txt
    file hyperlearn.py (for module hyperlearn) not found
    reading manifest file 'hyperlearn.egg-info/SOURCES.txt'
    writing manifest file 'hyperlearn.egg-info/SOURCES.txt'
    Copying hyperlearn.egg-info to /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/hyperlearn-0.0.1-py3.6.egg-info
    running install_scripts
    ###########
    >>> Building HyperLearn C extensions. <<<
    running build_ext
    building 'DEFINE' extension
    creating build/temp.linux-x86_64-3.6
    gcc -pthread -B /home/share/miniconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -I/home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include -I/home/share/miniconda3/envs/torch/include/python3.6m -c DEFINE.c -o build/temp.linux-x86_64-3.6/DEFINE.o
    In file included from /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:0,
                     from /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                     from /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from DEFINE.c:580:
    /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it by " \
      ^
    gcc -pthread -shared -B /home/share/miniconda3/envs/torch/compiler_compat -L/home/share/miniconda3/envs/torch/lib -Wl,-rpath=/home/share/miniconda3/envs/torch/lib -Wl,--no-as-needed -Wl,--sysroot=/ -O3 -march=native build/temp.linux-x86_64-3.6/DEFINE.o -o /home/share/software/sources/hyperlearn/hyperlearn/cython/DEFINE.cpython-36m-x86_64-linux-gnu.so
    building '__temp__' extension
    gcc -pthread -B /home/share/miniconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -I/home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include -I/home/share/miniconda3/envs/torch/include/python3.6m -c __temp__.c -o build/temp.linux-x86_64-3.6/__temp__.o
    In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/immintrin.h:79:0,
                     from __temp__.c:578:
    __temp__.c: In function ‘__pyx_f_8__temp____mult_add’:
    /usr/lib/gcc/x86_64-linux-gnu/5/include/fmaintrin.h:55:1: error: inlining failed in call to always_inline ‘_mm_fmadd_ps’: target specific option mismatch
     _mm_fmadd_ps (__m128 __A, __m128 __B, __m128 __C)
     ^
    __temp__.c:1084:3: error: called from here
       _mm_store_ps1(__pyx_v_x, _mm_fmadd_ps(__pyx_v_mult, _mm_load_ps1(__pyx_v_x), __pyx_v_shift));
       ^
    In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/immintrin.h:79:0,
                     from __temp__.c:578:
    /usr/lib/gcc/x86_64-linux-gnu/5/include/fmaintrin.h:55:1: error: inlining failed in call to always_inline ‘_mm_fmadd_ps’: target specific option mismatch
     _mm_fmadd_ps (__m128 __A, __m128 __B, __m128 __C)
     ^
    __temp__.c:1084:3: error: called from here
       _mm_store_ps1(__pyx_v_x, _mm_fmadd_ps(__pyx_v_mult, _mm_load_ps1(__pyx_v_x), __pyx_v_shift));
       ^
    error: command 'gcc' failed with exit status 1
    ###########
    >>> Successfully built C extensions. <<<
    

    Not sure what to make of this.

    opened by themightyoarfish 7
  • GPU RAM consumption?

    GPU RAM consumption?

    It seems like the RAM consumption refers to the CPU RAM, right? However, GPU computation is highly limited to its memory capacity. Computation over super large dataset such as K-Means/t-SNE tends to cause out-of-memory w/o careful breakdown. Does HyperLean specifically deal with super large input (Hosted on CPU memory but cannot reside on GPU memory all at once) regardless GPU memory size limit?

    opened by farleylai 2
  • Installation

    Installation

    It would be nice to have PyPI package, so it can be installed using pip.

    opened by SoundBot 4
  • Shocking Confusing Speed / Timing results of Algorithms (Sklearn, Numpy, Scipy, Pytorch, Numba) | Prelim results

    Shocking Confusing Speed / Timing results of Algorithms (Sklearn, Numpy, Scipy, Pytorch, Numba) | Prelim results

    Anyways, I didn't update the code a lot, but that's because I was busily testing and finding out which algos were the most stable and best.

    Key findings for N = 5,000 P = 6,000 [more features than N near square matrix]

    1. For pseudoinverse, (used in Linear Reg, Ridge Reg, lots of other algos), JIT, Scipy MKL, PinvH, Pinv2 and HyperLearn's Pinv are very similar. PyTorch's is clearly problematic, having close to over x4 slower than Scipy MKL.

    2. For Eigh (used in PCA, LDA, QDA, other algos), Sklearn's PCA utilises SVD. Clearly, not a good idea, since it is much better to compute the eigenvec / eigenval on XTX. JIT Eigh is the clear winner at 14.5 seconds on XTX, whilst Numpy is 2x slower. Torch likewise is slower once again...

    3. So, for PCA, a speedup of 3 times is seem if using JIT compiled Eigh when compared to Sklearn's PCA

    4. To solve X*theta = y, Torch GELS is super unstable. Like really. If you use Torch GELS, don't forget to call theta_hat[np.isnan(theta_hat) | np.isinf(theta_hat)] = 0, or else results are problematic. All other algos have very similar MSEs, and HyperLearn's Regularized Cholesky Solve takes a mere 0.635 seconds when compared to say using Sklearn's next fastest Ridge Solve (via cholesky) by over 100% (after considering matrix multiplication time) --> HyperLearn 2.89s vs 4.53s Sklearn.

    So to conclude:

    1. HyperLearn's Pseudoinverse has no speed improvement

    2. HyperLearn's PCA will have over 2 times speed boost. (200% improvement)

    3. HyperLearn's Linear Solvers will be over 1 times faster. (100% improvement)

    Help make HyperLearn better! All contributors are welcome, as this is truly an overwhelming project...

    drawing research 
    opened by danielhanchen 1
Owner
Daniel Han-Chen
Fast energy efficient machine learning algorithms
Daniel Han-Chen
STUMPY is a powerful and scalable Python library for computing a Matrix Profile, which can be used for a variety of time series data mining tasks

STUMPY STUMPY is a powerful and scalable library that efficiently computes something called the matrix profile, which can be used for a variety of tim

TD Ameritrade 1.8k Jun 13, 2021
A complete guide to start and improve in machine learning (ML)

A complete guide to start and improve in machine learning (ML), artificial intelligence (AI) in 2021 without ANY background in the field and stay up-to-date with the latest news and state-of-the-art techniques!

Louis-François Bouchard 1.4k Jun 16, 2021
Python package for stacking (machine learning technique)

vecstack Python package for stacking (stacked generalization) featuring lightweight functional API and fully compatible scikit-learn API Convenient wa

Igor Ivanov 624 Jun 10, 2021
Microsoft contributing libraries, tools, recipes, sample codes and workshop contents for machine learning & deep learning.

Microsoft contributing libraries, tools, recipes, sample codes and workshop contents for machine learning & deep learning.

Microsoft 87 Jun 7, 2021
PySpark + Scikit-learn = Sparkit-learn

Sparkit-learn PySpark + Scikit-learn = Sparkit-learn GitHub: https://github.com/lensacom/sparkit-learn About Sparkit-learn aims to provide scikit-lear

Lensa 1.1k Jun 3, 2021
A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.

Master status: Development status: Package information: TPOT stands for Tree-based Pipeline Optimization Tool. Consider TPOT your Data Science Assista

Epistasis Lab at UPenn 8.1k Jun 13, 2021
Distributed scikit-learn meta-estimators in PySpark

sk-dist: Distributed scikit-learn meta-estimators in PySpark What is it? sk-dist is a Python package for machine learning built on top of scikit-learn

Ibotta 268 Jun 10, 2021
Automated Machine Learning Pipeline with Feature Engineering and Hyper-Parameters Tuning

The mljar-supervised is an Automated Machine Learning Python package that works with tabular data. I

MLJAR 1.3k Jun 13, 2021
Implementation of different ML Algorithms from scratch, written in Python 3.x

Implementation of different ML Algorithms from scratch, written in Python 3.x

Gautam J 168 May 23, 2021
Factorization machines in python

Factorization Machines in Python This is a python implementation of Factorization Machines [1]. This uses stochastic gradient descent with adaptive re

Corey Lynch 845 Jun 13, 2021
A Collection of Conference & School Notes in Machine Learning 🦄📝🎉

Machine Learning Conference & Summer School Notes. ??????

null 528 Jun 4, 2021
Quantum Machine Learning

The Machine Learning package simply contains sample datasets at present. It has some classification algorithms such as QSVM and VQC (Variational Quantum Classifier), where this data can be used for experiments, and there is also QGAN (Quantum Generative Adversarial Network) algorithm.

Qiskit 70 Jun 12, 2021
cuML - RAPIDS Machine Learning Library

cuML - GPU Machine Learning Algorithms cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions t

RAPIDS 2.2k Jun 16, 2021
🔬 A curated list of awesome machine learning strategies & tools in financial market.

?? A curated list of awesome machine learning strategies & tools in financial market.

GeorgeZou 1k Jun 11, 2021