DeepMReye: magnetic resonance-based eye tracking using deep neural networks

Overview

License: GPL v3 py38 status Build Status NatNeuro Paper Logo

DeepMReye: magnetic resonance-based eye tracking using deep neural networks

This Jupyter Notebook provides a step-by-step walkthrough of the code. It includes eyeball coregistration, voxel extraction, model training and test as well as basic performance measures. Alternatively, here is a Colab Notebook.

This Data Repository includes exemplary data for model training and test, source data of all paper figures as well as pre-trained model weights.

Moreover, here are additional User Recommendations as well as a Frequently-Asked-Questions (FAQ) page. If you have other questions, please reach out to us.

deepMReye video

Installation - Option 1: CPU version

Pip installation

Install DeepMReye with a CPU version of TensorFlow using the following command.

pip install git+https://github.com/DeepMReye/DeepMReye.git

Anaconda / Miniconda installation

Install Anaconda or miniconda and clone this repository:

git clone https://github.com/DeepMReye/DeepMReye.git
cd DeepMReye

Create a virtual environment for DeepMReye with the following commands:

conda create --name deepmreye python=3.7
conda install --file requirements.txt
conda activate deepmreye

If installation of ANTsPy fails try to manually install it via:

git clone https://github.com/ANTsX/ANTsPy
cd ANTsPy
python3 setup.py install

This CPU version runs on Windows, Mac and Linux, but it takes substantially more time to compute than the GPU version (see below).

Installation - Option 2: GPU version (recommended)

Install DeepMReye with a GPU version of TensorFlow using following command. This version is substantially faster than the CPU version, but it requires CUDA and a NVIDIA GPU (not supported by Mac). The GPU version runs on Windows and Linux.

conda install tensorflow-gpu

Note that you might need to install cudnn first (conda install -c conda-forge cudnn). After installing the GPU version of tensorflow in the conda environment install DeepMReye via pip:

pip install git+https://github.com/DeepMReye/DeepMReye.git

Installation - Option 3: Colab

We provide a Colab Notebook showcasing model training and evaluation on a GPU provided by Google Colab. To use your own data, preprocess your data locally and upload only the extracted eyeball voxels. This saves space and avoids data privacy issues. See the Jupyter Notebook for the preprocessing and eyeball-extraction code.

Model Training & Evaluation

Colab Walkthrough

Data formats

The fMRI data should be organized in 4D NIFTI files (.nii), containing the realigned 3D images acquired over time. The pipeline then extracts the eyeball voxels automatically and saves them as Python Pickle files, which serve as model input. For model training, you additionally need training labels, a numpy array containing 10 gaze coordinates per functional volume. These gaze coordinates can either be camera-based eye-tracking labels or the coordinates of a fixation target, and many file formats can be easily read (e.g. .npy, .npz, .mat, .csv etc.).

Please see our FAQ page for more details on data formats and preprocessing.

Hardware requirements

The GPU version of DeepMReye requires a NVIDIA GPU.

Software requirements

The following python dependencies are being automatically installed when installing DeepMReye (specified in requirements.txt):

tensorflow-gpu (2.2.0)
numpy (1.19.1)
pandas (1.0.5)
matplotlib (3.2.2)
scipy (1.5.0)
ipython (7.13.0)
plotly (4.14.3)

Version in parentheses indicate the ones used for testing the framework. Its extensively tested on Linux 16.04 but should run on all OS (Windows, Mac, Linux) supporting a Python version >3.6 and pip. It is recommended to install the framework and dependencies in a virtual environment (e.g. conda).

Correspondence

If you have questions, comments or inquiries, please check out the online User documention and reach out to us: markus.frey[at]ntnu.no and matthias.nau[at]ntnu.no

Comments
  • Explicitly add files in masks folder for install

    Explicitly add files in masks folder for install

    Address #6 (and possibly closes it).

    This is a quick fix attempt to close #6. In theory what you expressed should have been enough, in practice let's see if adding more specificity to the setup helps. Worth a try - works for me on Linux.

    Another cause of this issue might be some local configuration that doesn't allow for nifti files of files bigger than x MB.

    opened by smoia 11
  • in the Notebook: No module named 'tensorflow.python.tools'

    in the Notebook: No module named 'tensorflow.python.tools'

    Trying to run the jupyter notebook.

    I am getting this error:

    ModuleNotFoundError: No module named 'tensorflow.python.tools'

    See the traceback and the output of pip list below

    Also: I had to place the notebook in the root of the repo to find the DeepMReye module.

    ---------------------------------------------------------------------------
    ModuleNotFoundError                       Traceback (most recent call last)
    /tmp/ipykernel_5653/3206132536.py in <module>
          8 
          9 # DeepMReye imports
    ---> 10 from deepmreye import architecture, train, analyse, preprocess
         11 from deepmreye.util import util, data_generator, model_opts
         12 
    
    ~/github/DeepMReye/deepmreye/__init__.py in <module>
    ----> 1 from . import util
          2 from . import preprocess
          3 from . import architecture
          4 from . import train
          5 from . import analyse
    
    ~/github/DeepMReye/deepmreye/util/__init__.py in <module>
    ----> 1 from . import util
          2 from . import model_opts
          3 from . import data_generator
          4 from . import data_io
    
    ~/github/DeepMReye/deepmreye/util/util.py in <module>
          6 from sklearn.metrics import r2_score
          7 import warnings
    ----> 8 import tensorflow.keras.backend as K
          9 from tensorflow.keras.callbacks import LearningRateScheduler
         10 import tensorflow as tf
    
    ~/github/DeepMReye/env/lib/python3.8/site-packages/tensorflow/__init__.py in <module>
         39 import sys as _sys
         40 
    ---> 41 from tensorflow.python.tools import module_util as _module_util
         42 from tensorflow.python.util.lazy_loader import LazyLoader as _LazyLoader
         43 
    
    ModuleNotFoundError: No module named 'tensorflow.python.tools'
    
    Package                      Version
    ---------------------------- ---------
    absl-py                      1.0.0
    antspyx                      0.2.9
    anyio                        3.3.4
    argon2-cffi                  21.1.0
    astunparse                   1.6.3
    attrs                        21.2.0
    Babel                        2.9.1
    backcall                     0.2.0
    bleach                       4.1.0
    cachetools                   4.2.4
    certifi                      2021.10.8
    cffi                         1.15.0
    charset-normalizer           2.0.7
    chart-studio                 1.1.0
    cycler                       0.11.0
    debugpy                      1.5.1
    decorator                    5.1.0
    defusedxml                   0.7.1
    entrypoints                  0.3
    flatbuffers                  2.0
    fonttools                    4.28.1
    gast                         0.4.0
    google-auth                  2.3.3
    google-auth-oauthlib         0.4.6
    google-pasta                 0.2.0
    grpcio                       1.42.0
    h5py                         3.6.0
    idna                         3.3
    imageio                      2.10.5
    importlib-metadata           4.8.2
    importlib-resources          5.4.0
    ipykernel                    6.5.0
    ipython                      7.29.0
    ipython-genutils             0.2.0
    jedi                         0.18.1
    Jinja2                       3.0.3
    joblib                       1.1.0
    json5                        0.9.6
    jsonschema                   4.2.1
    jupyter-client               7.0.6
    jupyter-core                 4.9.1
    jupyter-server               1.11.2
    jupyterlab                   3.2.4
    jupyterlab-pygments          0.1.2
    jupyterlab-server            2.8.2
    keras                        2.7.0
    Keras-Preprocessing          1.1.2
    kiwisolver                   1.3.2
    libclang                     12.0.0
    Markdown                     3.3.6
    MarkupSafe                   2.0.1
    matplotlib                   3.5.0
    matplotlib-inline            0.1.3
    mistune                      0.8.4
    nbclassic                    0.3.4
    nbclient                     0.5.8
    nbconvert                    6.3.0
    nbformat                     5.1.3
    nest-asyncio                 1.5.1
    networkx                     2.6.3
    nibabel                      3.2.1
    notebook                     6.4.6
    numpy                        1.21.4
    oauthlib                     3.1.1
    opt-einsum                   3.3.0
    packaging                    21.2
    pandas                       1.3.4
    pandocfilters                1.5.0
    parso                        0.8.2
    patsy                        0.5.2
    pexpect                      4.8.0
    pickleshare                  0.7.5
    Pillow                       8.4.0
    pip                          21.3.1
    plotly                       5.4.0
    prometheus-client            0.12.0
    prompt-toolkit               3.0.22
    protobuf                     3.19.1
    ptyprocess                   0.7.0
    pyasn1                       0.4.8
    pyasn1-modules               0.2.8
    pycparser                    2.21
    Pygments                     2.10.0
    pyparsing                    2.4.7
    pyrsistent                   0.18.0
    python-dateutil              2.8.2
    pytz                         2021.3
    PyWavelets                   1.2.0
    PyYAML                       6.0
    pyzmq                        22.3.0
    requests                     2.26.0
    requests-oauthlib            1.3.0
    retrying                     1.3.3
    rsa                          4.7.2
    scikit-image                 0.18.3
    scikit-learn                 1.0.1
    scipy                        1.7.2
    Send2Trash                   1.8.0
    setuptools                   58.2.0
    setuptools-scm               6.3.2
    six                          1.16.0
    sniffio                      1.2.0
    statsmodels                  0.13.1
    tenacity                     8.0.1
    tensorboard                  2.7.0
    tensorboard-data-server      0.6.1
    tensorboard-plugin-wit       1.8.0
    tensorflow                   2.7.0
    tensorflow-estimator         2.7.0
    tensorflow-io-gcs-filesystem 0.22.0
    termcolor                    1.1.0
    terminado                    0.12.1
    testpath                     0.5.0
    threadpoolctl                3.0.0
    tifffile                     2021.11.2
    tomli                        1.2.2
    tornado                      6.1
    traitlets                    5.1.1
    typing_extensions            4.0.0
    urllib3                      1.26.7
    wcwidth                      0.2.5
    webcolors                    1.11.1
    webencodings                 0.5.1
    websocket-client             1.2.1
    Werkzeug                     2.0.2
    wheel                        0.37.0
    wrapt                        1.13.3
    zipp                         3.6.0
    
    opened by Remi-Gau 10
  • [INFRA] drop testing on python 3.7 and install via setup.cfg

    [INFRA] drop testing on python 3.7 and install via setup.cfg

    Would require relaxing the requirements on tenserflow. https://discuss.tensorflow.org/t/support-python-3-10/124

    Could possibly consider removing python 3.7 or wait for its end of life: https://endoflife.date/python

    opened by Remi-Gau 7
  • [INFRA & DOC] add tests for notebooks

    [INFRA & DOC] add tests for notebooks

    Changes summary

    • Runs the pretrained-weight notebook in CI to use as a "system test".
    • Simplifies running the demo by facilitating data download from OSF

    Details

    • [x] add makefile to facilitate downloading some of the data from OSF in the demo folder
    • [x] adapt notebooks to run with demos in the notebook folders so users do not have to manually change the path
    • [x] update notebooks to explain how the data has to be structure
    • [x] add github action workflow to test the notebook with the nbmake pytest plugin
    • [x] lint notebooks content with nb-balck
    • [x] replace path creation using string concatenation with using os.path.join
    • [x] run on macOS
    • [x] install from upstream after merging of #10

    Note

    This does not run the "full" training notebook in CI

    opened by Remi-Gau 6
  • masks can't be found

    masks can't be found

    After a

    pip install git+https://github.com/DeepMReye/DeepMReye.git
    

    I get this when trying to load the masks.

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    /tmp/ipykernel_31249/7865041.py in <module>
          1 # Preload masks to save time within participant loop
    ----> 2 (eyemask_small, eyemask_big, dme_template, mask, x_edges, y_edges, z_edges) = preprocess.get_masks()
    
    ~/github/CPP_deepMReye/code/env/lib/python3.8/site-packages/deepmreye/preprocess.py in get_masks(data_path)
        118         Edges of mask in z-dimension
        119     """     
    --> 120     eyemask_small = ants.image_read(data_path + 'eyemask_small.nii')
        121     eyemask_big = ants.image_read(data_path + 'eyemask_big.nii')
        122     dme_template = ants.image_read(data_path + 'dme_template.nii')
    
    ~/github/CPP_deepMReye/code/env/lib/python3.8/site-packages/ants/core/ants_image_io.py in image_read(filename, dimension, pixeltype, reorient)
        513         filename = os.path.expanduser(filename)
        514         if not os.path.exists(filename):
    --> 515             raise ValueError("File %s does not exist!" % filename)
        516 
        517         hinfo = image_header_info(filename)
    
    ValueError: File ../deepmreye/masks/eyemask_small.nii does not exist!
    

    Given the default of get_masks and I am not sure that they get "installed" (or at all) by the pip install

    get_masks(data_path='../deepmreye/masks/'):
    

    Will pass a value to the path to the masks in the repo on my local machine but there may be a smarter way of doing this.

    opened by Remi-Gau 6
  • Setup a config setup file and a manifest file

    Setup a config setup file and a manifest file

    Address #6. This is a first and quick attempt at setting up a configuration file (to have a package-like setup) and solve the missing masks in some setups. ~It's still a WIP (I need to fix a couple of errors).~

    The real part that you need is in lines 42-50, that expressively adds files in the masks folder.

    opened by smoia 5
  • [MNT] packaging

    [MNT] packaging

    • [x] sets up makefile to help build wheel locally and push to pypi
    • [x] update config and remove extra dependencies
    • [x] add github action to push to pypi when making a github release
    opened by Remi-Gau 4
  • could not install tensorflow dependency on apple M1

    could not install tensorflow dependency on apple M1

    tried to install and got the following error

    this seems to be due to the fact that I am using an M1 apple computer: see here

    pip install git+https://github.com/DeepMReye/DeepMReye.git
    Collecting git+https://github.com/DeepMReye/DeepMReye.git
      Cloning https://github.com/DeepMReye/DeepMReye.git to /private/var/folders/4s/b4y96bxd7n979w298f1q3nm40000gq/T/pip-req-build-f3a6j20h
      Running command git clone -q https://github.com/DeepMReye/DeepMReye.git /private/var/folders/4s/b4y96bxd7n979w298f1q3nm40000gq/T/pip-req-build-f3a6j20h
      Resolved https://github.com/DeepMReye/DeepMReye.git to commit 1c349e1c8477e4261f0de685c2d5ccb180dd7266
    Collecting numpy
      Using cached numpy-1.23.3-cp39-cp39-macosx_11_0_arm64.whl (13.4 MB)
    Collecting pandas
      Using cached pandas-1.5.0-cp39-cp39-macosx_11_0_arm64.whl (10.9 MB)
    Collecting matplotlib
      Using cached matplotlib-3.6.0-cp39-cp39-macosx_11_0_arm64.whl (7.2 MB)
    Collecting scipy
      Using cached scipy-1.9.1-cp39-cp39-macosx_12_0_arm64.whl (29.9 MB)
    Collecting ipython
      Using cached ipython-8.5.0-py3-none-any.whl (752 kB)
    Collecting plotly
      Using cached plotly-5.10.0-py2.py3-none-any.whl (15.2 MB)
    Collecting scikit-learn
      Using cached scikit_learn-1.1.2-cp39-cp39-macosx_12_0_arm64.whl (7.7 MB)
    Collecting antspyx
      Using cached antspyx-0.3.3.tar.gz (7.0 MB)
    ERROR: Could not find a version that satisfies the requirement tensorflow<2.7 (from deepmreye) (from versions: none)
    ERROR: No matching distribution found for tensorflow<2.7
    
    opened by yyang1234 4
  • Error of Segmentation fault (core dumped) when running deepmreye_example_usage.ipynb

    Error of Segmentation fault (core dumped) when running deepmreye_example_usage.ipynb

    Hi,

    I have installed the packages and the conda environment (python 3.7) according to your instruction. I downloaded your example data and tried to run deepmreye_example_usage.ipynb, but I kept getting the error of Segmentation fault when reaching preprocess.run_participant(fp_func, dme_template, eyemask_big, eyemask_small, x_edges, y_edges, z_edges) (there is no other error):

    Running participant sub-NDARAA948VFH
    Segmentation fault (core dumped)
    

    I have a GPU under this environment. Do you have any ideas or solutions about it? Thank you.

    opened by yqsong96 4
  • Missing library libpng16.16.dylib on Mac

    Missing library libpng16.16.dylib on Mac

    I've tried to run the demo but many bugs appeared: Code test:

    # Import modules and add library to path
    import sys
    sys.path.insert(0, "/Users/admin/Desktop/Github")
    import os
    os.environ["CUDA_VISIBLE_DEVICES"] = "0" # Change to os.environ["CUDA_VISIBLE_DEVICES"] = "" if you dont have access to a GPU
    import numpy as np 
    import pandas as pd
    import pickle
    
    # DeepMReye imports
    from deepmreye import architecture, train, analyse, preprocess
    from deepmreye.util import util, data_generator, model_opts
    
    # Initialize plotly figures
    from plotly.offline import init_notebook_mode 
    init_notebook_mode(connected = True)
    
    # Make sure the output width is adjusted for better export as HTML
    from IPython.core.display import display, HTML
    display(HTML("<style>.container { width:70% !important; }</style>"))
    display(HTML("<style>.output_result { max-width:70% !important; }</style>"))
    
    # Autoreload modules
    %load_ext autoreload
    %autoreload 2
    

    Bug report:

    ImportError                               Traceback (most recent call last)
    /var/folders/w3/l91cnb1d7dvdh581bynsm_n80000gp/T/ipykernel_22011/2184068996.py in <module>
          9 
         10 # DeepMReye imports
    ---> 11 from deepmreye import architecture, train, analyse, preprocess
         12 from deepmreye.util import util, data_generator, model_opts
         13 
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/deepmreye/__init__.py in <module>
          1 from . import util
    ----> 2 from . import preprocess
          3 from . import architecture
          4 from . import train
          5 from . import analyse
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/deepmreye/preprocess.py in <module>
          2 import pickle
          3 import numpy as np
    ----> 4 import ants
          5 import matplotlib.pyplot as plt
          6 from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/__init__.py in <module>
          5     pass
          6 
    ----> 7 from .core import *
          8 from .utils import *
          9 from .segmentation import *
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/core/__init__.py in <module>
          1 
    ----> 2 from .ants_image import *
          3 from .ants_image_io import *
          4 
          5 from .ants_transform import *
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/core/ants_image.py in <module>
         27 import inspect
         28 
    ---> 29 from .. import registration, segmentation, utils, viz
         30 from . import ants_image_io as iio2
         31 
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/registration/__init__.py in <module>
          1 
          2 
    ----> 3 from .affine_initializer import *
          4 from .apply_transforms import *
          5 from .create_jacobian_determinant_image import *
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/registration/affine_initializer.py in <module>
          5 from tempfile import mktemp
          6 
    ----> 7 from .. import utils
          8 
          9 def affine_initializer(fixed_image, moving_image, search_factor=20,
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/utils/__init__.py in <module>
          1 from .add_noise_to_image import *
    ----> 2 from .bias_correction import *
          3 from .channels import *
          4 from .convert_nibabel import *
          5 from .crop_image import *
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/utils/bias_correction.py in <module>
          2 
          3 
    ----> 4 from . import process_args as pargs
          5 from .get_mask import get_mask
          6 from .iMath import iMath
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/utils/process_args.py in <module>
         12 
         13 from ..core import ants_image as iio
    ---> 14 from .. import lib
         15 
         16 _short_ptype_map = {
    
    ~/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/lib/__init__.py in <module>
          1 ## LOCAL ##
    ----> 2 from .addNoiseToImage import *
          3 from .antiAlias import *
          4 from .antsImage import *
          5 from .antsImageClone import *
    
    ImportError: dlopen(/Users/admin/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/lib/addNoiseToImage.cpython-37m-darwin.so, 0x0002): Library not loaded: /opt/X11/lib/libpng16.16.dylib
      Referenced from: /Users/admin/miniconda3/envs/deepmreye/lib/python3.7/site-packages/ants/lib/addNoiseToImage.cpython-37m-darwin.so
      Reason: tried: '/opt/X11/lib/libpng16.16.dylib' (no such file), '/usr/local/lib/libpng16.16.dylib' (no such file), '/usr/lib/libpng16.16.dylib' (no such file)
    

    Python version: Python 3.7.13

    Pip list:

    Package Version


    absl-py 0.15.0 antspyx 0.3.3 appnope 0.1.3 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 astunparse 1.6.3 attrs 21.4.0 backcall 0.2.0 bcrypt 3.2.2 beautifulsoup4 4.11.1 bleach 5.0.1 cached-property 1.5.2 cachetools 4.2.4 certifi 2022.6.15 cffi 1.15.1 charset-normalizer 2.1.0 chart-studio 1.1.0 clang 5.0 cmake 3.22.5 cryptography 37.0.2 cycler 0.11.0 debugpy 1.6.0 decorator 5.1.1 deepmreye 0.1.dev147+gcabc3f0 defusedxml 0.7.1 entrypoints 0.4 fastjsonschema 2.15.3 flatbuffers 1.12 fonttools 4.33.3 gast 0.4.0 google-auth 1.35.0 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 grpcio 1.47.0 h5py 3.1.0 idna 3.3 imageio 2.19.3 importlib-metadata 4.12.0 importlib-resources 5.8.0 ipykernel 6.15.0 ipython 7.34.0 ipython-genutils 0.2.0 ipywidgets 7.7.1 jedi 0.18.1 Jinja2 3.1.2 joblib 1.1.0 jsonschema 4.6.1 jupyter 1.0.0 jupyter-client 7.3.4 jupyter-console 6.4.4 jupyter-core 4.10.0 jupyterlab-pygments 0.2.2 jupyterlab-widgets 1.1.1 keras 2.6.0 Keras-Preprocessing 1.1.2 kiwisolver 1.4.3 Markdown 3.3.7 MarkupSafe 2.1.1 matplotlib 3.5.2 matplotlib-inline 0.1.3 mistune 0.8.4 nbclient 0.6.6 nbconvert 6.5.0 nbformat 5.4.0 nest-asyncio 1.5.5 networkx 2.6.3 nibabel 4.0.1 notebook 6.4.12 numpy 1.19.5 oauthlib 3.2.0 opt-einsum 3.3.0 packaging 21.3 pandas 1.3.5 pandocfilters 1.5.0 paramiko 2.11.0 parso 0.8.3 patsy 0.5.2 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.2.0 pip 21.2.2 plotly 5.9.0 prometheus-client 0.14.1 prompt-toolkit 3.0.30 protobuf 3.19.4 psutil 5.9.1 ptyprocess 0.7.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycparser 2.21 Pygments 2.12.0 PyNaCl 1.5.0 pyparsing 3.0.9 pyrsistent 0.18.1 python-dateutil 2.8.2 pytz 2022.1 PyWavelets 1.3.0 PyYAML 6.0 pyzmq 23.2.0 qtconsole 5.3.1 QtPy 2.1.0 requests 2.28.1 requests-oauthlib 1.3.1 retrying 1.3.3 rsa 4.8 scikit-image 0.19.3 scikit-learn 1.0.2 scipy 1.7.3 Send2Trash 1.8.0 setuptools 61.2.0 six 1.15.0 soupsieve 2.3.2.post1 statsmodels 0.13.2 tenacity 8.0.1 tensorboard 2.6.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorflow 2.6.5 tensorflow-estimator 2.6.0 termcolor 1.1.0 terminado 0.15.0 threadpoolctl 3.1.0 tifffile 2021.11.2 tinycss2 1.1.1 tornado 6.2 traitlets 5.3.0 typing-extensions 3.10.0.2 urllib3 1.26.9 wcwidth 0.2.5 webcolors 1.12 webencodings 0.5.1 Werkzeug 2.1.2 wheel 0.37.1 widgetsnbextension 3.6.1 wrapt 1.12.1 zipp 3.8.0

    Links I've tried:

    Stackoverflow Homebrew

    opened by WeeXee 3
  • [FIX] use path relative to location of get_mask file to get the masks

    [FIX] use path relative to location of get_mask file to get the masks

    ok this should be the actual fix for #6

    To test it, make sure that the MWE mention in here does produce the bug.

    Then uninstall deepmreye install the version from my branch with the fix:

    pip uninstall deepmreye
    pip install git+https://github.com/Remi-Gau/DeepMReye.git@remi-6-get_masks
    

    And rerun the "test" to make sure the masks and templates are found.

    launch iPython

    ipython
    

    try to get the masks

    from deepmreye import preprocess
    (
        eyemask_small,
        eyemask_big,
        dme_template,
        mask,
        x_edges,
        y_edges,
        z_edges,
    ) = preprocess.get_masks()
    
    opened by Remi-Gau 3
  • [FIX] fix dockerfile and add a dockerfile for dev

    [FIX] fix dockerfile and add a dockerfile for dev

    related to #31

    • Dockerfile will use deepmreye from pypi
    • dev Dockerfile will use the local version of pypi
    • update default model to use in docker and notebooks
    opened by Remi-Gau 0
  • Issues on MacBooks with M1/M2 chips

    Issues on MacBooks with M1/M2 chips

    We are aware of installation issues of DeepMReye on Apple MacBooks with M1 or M2 chips.

    The DeepLabCut team made us aware of a potential solution: Use miniconda3 instead of anaconda. https://github.com/DeepLabCut/DeepLabCut/blob/main/docs/installation.md#install-anaconda-or-use-miniconda3-ideal-for-macos-users

    We will be testing this soon, but if you have an M1/M2 MacBook and run into issues, please try out the suggested solution and let us know if it works.

    opened by Naubody 6
  • Implementing calibration data to Deepmreye

    Implementing calibration data to Deepmreye

    Hello, first of all thank you for this library! I wanted to ask how would you recommend to implement the calibration data acquired through the Matlab code you published. Should I need to turn the edf Eyelink files to a specificformat (asc, csvor other)? I am also confused at which part of the main Python script (as shown on the exemplary data on the Jupyter Notebook) the calibration data is used?

    opened by jonathantol 8
  • properly set up matrix os for CI

    properly set up matrix os for CI

    Tackle after merging of #18

    the other errors came either from changing v2->v3 or from matrix os (I guess the latter).

    yes most likely the latter because the way it is set up now, all tests are only run on Linux

    See this workflow that is supposed to run on MacOS but actually ran on Ubuntu https://github.com/DeepMReye/DeepMReye/runs/7576038296?check_suite_focus=true#step:1:3

    So I would suggest merging this and fixing the CI for OS in another PR after that.

    Originally posted by @Remi-Gau in https://github.com/DeepMReye/DeepMReye/issues/18#issuecomment-1200162849

    opened by Remi-Gau 0
  • Create code tutorial and model weights for eyes open vs eyes closed

    Create code tutorial and model weights for eyes open vs eyes closed

    Current notebooks only support regression targets, therefore we should:

    • Add Jupyter notebook with code for model training eyes open vs. eyes closed (or other classification targets)
    • Add model weights to OSF
    opened by CYHSM 4
Releases(0.1.1)
Owner
DeepMReye is a software package for magnetic resonance-based eye tracking for (f)MRI experiments. Contact: matthias.nau[at]nih.gov & markus.frey[at]ntnu.no
null
Towards End-to-end Video-based Eye Tracking

Towards End-to-end Video-based Eye Tracking The code accompanying our ECCV 2020 publication and dataset, EVE. Authors: Seonwook Park, Emre Aksan, Xuco

Seonwook Park 76 Dec 12, 2022
CenterPoint 3D Object Detection and Tracking using center points in the bird-eye view.

CenterPoint 3D Object Detection and Tracking using center points in the bird-eye view. Center-based 3D Object Detection and Tracking, Tianwei Yin, Xin

Tianwei Yin 134 Dec 23, 2022
Neural Magic Eye: Learning to See and Understand the Scene Behind an Autostereogram, arXiv:2012.15692.

Neural Magic Eye Preprint | Project Page | Colab Runtime Official PyTorch implementation of the preprint paper "NeuralMagicEye: Learning to See and Un

Zhengxia Zou 56 Jul 15, 2022
Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

OpenGaze: Web Service for OpenFace Facial Behaviour Analysis Toolkit Overview OpenFace is a fantastic tool intended for computer vision and machine le

Sayom Shakib 4 Nov 3, 2022
Rohit Ingole 2 Mar 24, 2022
PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

null 63 Dec 16, 2022
Joint detection and tracking model named DEFT, or ``Detection Embeddings for Tracking.

DEFT: Detection Embeddings for Tracking DEFT: Detection Embeddings for Tracking, Mohamed Chaabane, Peter Zhang, J. Ross Beveridge, Stephen O'Hara

Mohamed Chaabane 253 Dec 18, 2022
Tracking code for the winner of track 1 in the MMP-Tracking Challenge at ICCV 2021 Workshop.

Tracking Code for the winner of track1 in MMP-Trakcing challenge This repository contains our tracking code for the Multi-camera Multiple People Track

DamoCV 29 Nov 13, 2022
Tracking Pipeline helps you to solve the tracking problem more easily

Tracking_Pipeline Tracking_Pipeline helps you to solve the tracking problem more easily I integrate detection algorithms like: Yolov5, Yolov4, YoloX,

VNOpenAI 32 Dec 21, 2022
Quadruped-command-tracking-controller - Quadruped command tracking controller (flat terrain)

Quadruped command tracking controller (flat terrain) Prepare Install RAISIM link

Yunho Kim 4 Oct 20, 2022
Python package for multiple object tracking research with focus on laboratory animals tracking.

motutils is a Python package for multiple object tracking research with focus on laboratory animals tracking. Features loads: MOTChallenge CSV, sleap

Matěj Šmíd 2 Sep 5, 2022
Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here.

Richard Wang 443 Dec 6, 2022
PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"

FIERY This is the PyTorch implementation for inference and training of the future prediction bird's-eye view network as described in: FIERY: Future In

Wayve 406 Dec 24, 2022
Code + pre-trained models for the paper Keeping Your Eye on the Ball Trajectory Attention in Video Transformers

Motionformer This is an official pytorch implementation of paper Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers. In this rep

Facebook Research 192 Dec 23, 2022
OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.

OpenFace 2.2.0: a facial behavior analysis toolkit Over the past few years, there has been an increased interest in automatic facial behavior analysis

Tadas Baltrusaitis 5.8k Dec 31, 2022
Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

Complex-Valued Neural Networks (CVNN) Done by @NEGU93 - J. Agustin Barrachina Using this library, the only difference with a Tensorflow code is that y

youceF 1 Nov 12, 2021
A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks without the use of any outside machine learning libraries - all from scratch.

Kordel K. France 2 Nov 14, 2022
Cancer-and-Tumor-Detection-Using-Inception-model - In this repo i am gonna show you how i did cancer/tumor detection in lungs using deep neural networks, specifically here the Inception model by google.

Cancer-and-Tumor-Detection-Using-Inception-model In this repo i am gonna show you how i did cancer/tumor detection in lungs using deep neural networks

Deepak Nandwani 1 Jan 1, 2022
Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks. Bayesian-Torch is designed to be flexible and seamless in extending a deterministic deep neural network architecture to corresponding Bayesian form by simply replacing the deterministic layers with Bayesian layers.

Intel Labs 210 Jan 4, 2023