A Python 3 library making time series data mining tasks, utilizing matrix profile algorithms

Describe the bug We need protobuf > 3.11.2 for other dependencies.

To Reproduce Steps to reproduce the behavior:

1. pip install matrixprofile
2. pip install --upgrade protobuf Should install protobuf v3.17.3 which it does. It also errors saying matrixprofile 1.1.10 requires protobuf==3.11.2, but you have protobuf 3.17.3 which is incompatible. despite the fact that matrixprofile/requirements.txt in v1.1.10 says protobuf>=3.11.2,<4.0.0
3. pip install matrixprofile This uninstalls protobuf and installs protobuf==3.11.2, which is the wrong behavior

Expected behavior I expect protobuf to be left alone, since the stated requirements for matrixprofile is just protobuf>=3.11.2,<4.0.0

Additional context I believe the problem is that matrixprofile/setup.py required protobuf==3.11.2 until March 8, 2021, and the latest release was released on January 16, 2021. I think the bug would be fixed by creating another release.

pip installing matrixprofile from the commit where setup.py relaxed the protobuf version requirement resolves the issue

Is your feature request related to a problem? Please describe. In some corporate environments, it's common to import external libraries into an internal repository for internal dependency resolution. In order to import matrixprofile for use, a tar.gz file is needed similar to the python libraries requests or pytest.

Describe the solution you'd like A tar.gz file uploaded to pypi as part of the release. See requests or pytest as examples. Links below.

Describe alternatives you've considered None.

Additional context

• https://pypi.org/project/requests/#files
• https://pypi.org/project/pytest/#files

Is your feature request related to a problem? Please describe. Currently, conda users have no easy way of downloading the matrixprofile library. This is a problem both from an accessibility standpoint, and for various initiatives on the MPF roadmap.

Describe the solution you'd like matrixprofile can be downloaded through conda-install matrixprofile

Describe alternatives you've considered N/A

Additional context N/A

I'll be taking this task in tandem with @frankiecancino , unless other members of the community would like to help :)

I installed matrixprofile a week ago and I am trying to reproduce the example in the documentation https://matrixprofile.docs.matrixprofile.org/examples/ECG_Annotation_Vectors.html but as soon as I arrive to

import matrixprofile as mp
import numpy as np

from matplotlib import pyplot as plt
%matplotlib inline

ecg = mp.datasets.load('ecg-heartbeat-av')
ts = ecg['data']
window_size = 150

I get the error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-11-93a2bc01725a> in <module>
      6 
      7 
----> 8 ecg = mp.datasets.load('ecg-heartbeat-av')
      9 ts = ecg['data']
     10 window_size = 150

AttributeError: module 'matrixprofile' has no attribute 'datasets'

I even tried mp.compute and I got the error

AttributeError: module 'matrixprofile' has no attribute 'compute'

Am I doing something wrong or are there missing packages or....?

The re-design currently leverages Cython, which in turn requires openmp for parallelization. The version of gcc installed on Mac is actually clang without openmp, so the code cannot compile properly. I believe updating to a different version of gcc (w/openmp) should do the trick, but unfortunately I can't figure out how to update setup.py to use a different version of gcc (maybe just change environment variables via os.environ?).

Also, we should think through what this means for our Mac customer base. I worry that requiring the installation of a new compiler (no matter how easy) will be an additional step for our user base that will reduce adoption.

Error in question:

(ds_env) Andrews-MacBook-Pro:matrixprofile-ts-redesign andrewvanbenschoten$ python setup.py install /usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type' warnings.warn(msg) running install running bdist_egg running egg_info writing matrixprofile_ts.egg-info/PKG-INFO writing dependency_links to matrixprofile_ts.egg-info/dependency_links.txt writing requirements to matrixprofile_ts.egg-info/requires.txt writing top-level names to matrixprofile_ts.egg-info/top_level.txt reading manifest file 'matrixprofile_ts.egg-info/SOURCES.txt' writing manifest file 'matrixprofile_ts.egg-info/SOURCES.txt' installing library code to build/bdist.macosx-10.13-x86_64/egg running install_lib running build_py running build_ext building 'matrixprofile.algorithms.cympx' extension clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/Users/andrewvanbenschoten/code/python3/ds_env/include -I/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c matrixprofile/algorithms/cympx.c -o build/temp.macosx-10.13-x86_64-3.6/matrixprofile/algorithms/cympx.o -O3 -march=native -fopenmp clang: error: unsupported option '-fopenmp' error: command 'clang' failed with exit status 1

Overview Right now all of the algorithms implemented support one-dimensional matrix profiles. There have been a handful of requests on our discord channel asking for multi-dimensional support. This issue is used to keep track of the request and give an opportunity to the community for contributions.

Todos

• Data structure supporting multiple dimensions
• Motif discovery
• Discord discovery
• Figure out what algorithms should be updated. I'm guessing MPX and SCRIMP++ only?
• Update analyze and compute higher-level functions to pass off multi-dimensional time series to the right algorithm.
• Update I/O module

Relevant papers Matrix Profile VI: Meaningful Multidimensional Motif Discovery. Chin-Chia Michael Yeh, Nickolas Kavantzas, Eamonn Keogh. [pdf] ICDM 2017.

Domain Agnostic Online Semantic Segmentation for Multi-Dimensional Time Series. Shaghayegh Gharghabi, Chin-Chia Michael Yeh, Yifei Ding, Wei Ding, Paul Hibbing, Samuel LaMunion, Andrew Kaplan, Scott E. Crouter, Eamonn Keogh. Data Mining and Knowledge Discovery.

Added unit testing, documentation, citations, and functionality for annotation vectors in the matrix profile library as per specifications on the main website.

There is an off-by-one error when calculating the exclusion zones. I noticed the error while testing my own implementation against this repository. I believe the issue can be fixed by changing https://github.com/matrix-profile-foundation/matrixprofile/blob/f0d354fa8363911f860fbf378c78f7ce05ef41d4/matrixprofile/algorithms/cympx.pyx#L81 to for diag in prange(minlag+1, profile_len, num_threads=n_jobs, nogil=True):

Describe the bug When calling mp.compute on one timeseries of the robot example dataset (e.g. for id = 1), matrixprofile first throws a warning

/home/nils/anaconda3/envs/tsfresh/lib/python3.7/site-packages/matrixprofile/algorithms/skimp.py:339: RuntimeWarning: No windows found with given threshold, try to set a lower threshold

and then raises an exception

TypeError                                 Traceback (most recent call last)
<ipython-input-14-d031a7c608e1> in <module>
----> 1 mp.compute(ts)

~/anaconda3/envs/tsfresh/lib/python3.7/site-packages/matrixprofile/compute.py in compute(ts, windows, query, sample_pct, threshold, n_jobs, preprocessing_kwargs)
    120         # from 8 in steps of 2 until upper w
    121         start = 8
--> 122         windows = range(start, profile['upper_window'] + 1)
    123 
    124         # compute the pmp

TypeError: 'float' object is not subscriptable

Looking at the code, I think that maximum_subsequence returns a NaN when it has not found a window (here), which is not the format expected by compute (which expects a dictionary of some sort).

The root cause for this, is probably that the timeseries under study is quite short (length 15), so I would be happy to understand if this is a systematic problem for short time series (the fact that no windows are found, not the fact that the exception is raised :-))

To Reproduce

> conda install tsfresh
> pip install matrixprofile

from tsfresh.examples.robot_execution_failures import download_robot_execution_failures, load_robot_execution_failures
download_robot_execution_failures()
df, _ = load_robot_execution_failures()
ts = df[df.id == 1]['F_x'].values

import matrixprofile as mp
mp.compute(ts, threshold=0.0)

Expected behavior No exception :-)

Desktop (please complete the following information):

• OS: ubuntu 20.04
• Version: 1.1.6

Additional context Related to https://github.com/blue-yonder/tsfresh/issues/785

Please note that if you are using tsfresh's example robot data, one timeseries are all values for a single id (so the full data sample has actually 88 time series in it)

Hello,

Thanks for creating this amazing package. It helps others to use it for different applications. I have three things to discuss regarding the snippets:

(1) It would be great if the locations of all snippets could be returned as well. (By that, I mean all the subsequences that are close to a snippet which is considered as the representative of those subsequences).

(2) "Probable" Bug: I have an hourly data recorded in a year (array: 8760-hr time series). I tried to find two snippets (with length 24, and window size=12) and I got a result. Then, I just deleted the first 12 samples (i.e. new_array = array[12:]). Rationally speaking, there should be no change in the snippets, but that is not the case as the results have been changed drastically. I also tried array[1:] and the result was the same as the first one but with one hour shift. I noticed that, in all three cases, the index (given in the result), from which the snippet starts, is multiple of 24 (my snippet size) which might be a little bit strange.

(3) I tried to read the snippet paper published by Dr. Eamon Keogh's team. I found the subsequent length m, the snippet size. However, I couldn't find the purpose of window size, the one used in the snippet function in this package. It would be great if you could give some elaboration on this matter.

Best, Nima

Our wheel build system is reliant on the package cibuildwheel. Currently, our TravisCI script has some environment issues with Python and pulling down the required version of cibuildwheel for Python 3.9 support. The version required for cibuildwheel to build wheels on Python 3.9 is v1.6.2.

See these logs for Linux and MacOS issues:

https://travis-ci.com/github/matrix-profile-foundation/matrixprofile/jobs/464939617 https://travis-ci.com/github/matrix-profile-foundation/matrixprofile/jobs/464939618

Windows is successfully building wheel for Python 3.9.

CVE-2022-40898 - Medium Severity Vulnerability

Vulnerable Library - wheel-0.36.2-py2.py3-none-any.whl

A built-package format for Python

Library home page: https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt,/python2-requirements.txt

Dependency Hierarchy:

• :x: wheel-0.36.2-py2.py3-none-any.whl (Vulnerable Library)

Found in base branch: master

Vulnerability Details

An issue discovered in Python Packaging Authority (PyPA) Wheel 0.37.1 and earlier allows remote attackers to cause a denial of service via attacker controlled input to wheel cli.

Publish Date: 2022-12-23

URL: CVE-2022-40898

CVSS 3 Score Details (5.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Local
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Release Date: 2022-12-23

Fix Resolution: wheel 0.38.0

Step up your Open Source Security Game with Mend here

Hi all, I'm a student using Matrix Profile to do time series comparisons, and I'm having a hard time understanding what "neighbors" are in MPF. I understand the concept of a "nearest neighbor" being a subsequence with the smallest Euclidean distance from a given subsequence, but there is an input to at least one MPF function called "maxneighbors." I couldn't find any explanation of this input on the API or any other resources I could find. What are neighbors, why do they appear on graphs of motifs that MPF generates, and what distinguishes them from motifs?

Thank you for any help!

Hi matrix-profile-foundation/matrixprofile!

This is a one-off automatically generated pull request from LGTM.com :robot:. You might have heard that we’ve integrated LGTM’s underlying CodeQL analysis engine natively into GitHub. The result is GitHub code scanning!

With LGTM fully integrated into code scanning, we are focused on improving CodeQL within the native GitHub code scanning experience. In order to take advantage of current and future improvements to our analysis capabilities, we suggest you enable code scanning on your repository. Please take a look at our blog post for more information.

This pull request enables code scanning by adding an auto-generated codeql.yml workflow file for GitHub Actions to your repository — take a look! We tested it before opening this pull request, so all should be working :heavy_check_mark:. In fact, you might already have seen some alerts appear on this pull request!

Where needed and if possible, we’ve adjusted the configuration to the needs of your particular repository. But of course, you should feel free to tweak it further! Check this page for detailed documentation.

Questions? Check out the FAQ below!

FAQ

The install fails when building from source and using the --use-pep517 option of pip. This means that installation from source is not possible with poetry.

To Reproduce Steps to reproduce the behavior:

1. docker run -it --platform linux/arm64/v8 python:3.8 /bin/bash (arm64 wheel not available, builds from source)
2. pip install --use-pep517 matrixprofile

Expected behavior Build should succeed

Desktop (please complete the following information):

• OS: Linux WSL2
• Version matrixprofile 1.1.10

Additional context

Collecting matrixprofile
  Downloading matrixprofile-1.1.10.tar.gz (331 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 331.6/331.6 KB 6.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [41 lines of output]
      /usr/local/bin/python: No module named pip
      /tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      Traceback (most recent call last):
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
          subprocess.check_call(cmd)
        File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpj8u2em6x', '--quiet', 'numpy>=1.16.2']' returned non-zero exit status 1.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 482, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 4, in <module>
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
          resolved_dists = pkg_resources.working_set.resolve(
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py", line 789, in resolve
          dist = best[req.key] = env.best_match(
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1075, in best_match
          return self.obtain(req, installer)
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1087, in obtain
          return installer(requirement)
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
          return fetch_build_egg(self, req)
        File "/tmp/pip-build-env-w_o96bhe/overlay/lib/python3.8/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
          raise DistutilsError(str(e)) from e
      distutils.errors.DistutilsError: Command '['/usr/local/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpj8u2em6x', '--quiet', 'numpy>=1.16.2']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.