Motivation and Context

PyTorch DataLoader for loading data from hangar directly into PyTorch

If it fixes an open issue, please link to the issue here:

#13

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [ ] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [x] Ready for review
• [ ] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [ ] Current tests cover modifications made
• [ ] New tests have been added to the test suite
• [ ] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have signed (or will sign when prompted) the tensorwork CLA.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.

Motivation and Context

Why is this change required? What problem does it solve?:

To simplify user interface with arraysets and provide some concept of a dataset as a view across arraysets.

NOTE: This initial PR is a proof of concept only, and will require extensive discussion before the final design is agreed upon

If it fixes an open issue, please link to the issue here:

related to #79 and many conversations on the Hangar Users Slack Channel

Description

Describe your changes in detail:

Added CheckoutIndexer class which is inhereted in ReaderCheckout and WriterCheckout to enable the following API. (originally proposed by @lantiga and @elistevens)

dset = repo.checkout(write=True)
# get an arrayset of the dataset (i.e. a "column" of the dataset?)
aset = dset['foo']

# get a specific array from 'foo' (returns a named tuple)
arr = dset['foo', '1']
# set it too
dset['foo', '1'] = arr

# get data from dset (returns a named tuple)
subarr = dset['foo', '1']
# and set into it
dset['foo', '1'] = subarr + 1

# get a sample of a dataset across 'foo' and 'bar' (returns a named tuple)
sample = dset[('foo', 'bar'), '1']

# get a sample of all arraysets in the checkout (returns a named tuple)
sample = dset[:, '1']
sample = dset[..., '1']

# get multiple samples
sample_ids = ['1', '2', '3']
batch = dset[('foo', 'bar'), sample_ids]
batch = dset[:, sample_ids]
batch = dset[..., sample_ids]

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [ ] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [x] Ready for review
• [ ] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [ ] Current tests cover modifications made
• [x] New tests have been added to the test suite
• [ ] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have signed (or will sign when prompted) the tensorwork CLA.
• [x] I have added tests to cover my changes.
• [x] All new and existing tests passed.

Motivation and Context

Why is this change required? What problem does it solve?:

The name Datasets no longer fit the appropriate description as a container of tensor/array data.

In order to make this clearer, datasets has been replaced by the term datacells. Since this more accuratly describes the ability for a single sample in a dataset to be made up of individual pieces spread across datacells.

Description

Describe your changes in detail:

The new explantation is in full HERE, but for the sake of brevity, this diagram illustrates the "unique" relationship that a dataset has to samples and datacells:

   A Dataset is thought of as containing Samples, but is actually defined by
    Datacells, which store parts of fully defined Samples in structures
       common across the full aggregation of Samples in the Dataset

   _____________________________________
         S1     |    S2    |     S3     |  <------------------------|
   --------------------------------------                           |
       image    |  image   |   image    |  <- Datacell 1  <--|      |
     filename   | filename |  filename  |  <- Datacell 2  <--|-- Dataset
       label    |  label   |   label    |  <- Datacell 3  <--|
     annotation |    -     | annotation |  <- Datacell 4  <--|


   If a sample does not have a piece of data, lack of info in the Datacell
         makes no difference in any way to the larger picture.

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [ ] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [x] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [x] Ready for review
• [ ] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [x] Current tests cover modifications made
• [ ] New tests have been added to the test suite
• [x] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have signed (or will sign when prompted) the tensorwork CLA.
• [x] I have added tests to cover my changes.
• [x] All new and existing tests passed.

Please give this a review @lantiga @hhsecond

Motivation and Context

Why is this change required? What problem does it solve?:

This is a large PR which started with the motivation of allowing arraysets to contain subsamples under a common key. Though minimal work was needed for the technical implementation (with esentially no changes made to the hangar core record parsing, history traversal, or tensor storage backends), the integration of the API into the current model proved difficult, which required some major refactoring of what was previously known as ArraysetDataReader and ArraysetDataWriter classes.

Description

Describe your changes in detail:

Rather than try to combine every possible API method needed by flat and nested arrayset access into a frankenstein monster class, each access convention implements it's own API class methods (fully independent from one another). The appropriate constructors are selected based on the constains_subsamples argument in init_arrayset(). The argument is recorded in the schema so the correct type can be identified in subsequent checkouts.

I'm working on putting together a summary of the API. That will follow shortly.

At the moment, about half the tests for the new nested sample container are missing, and I need to re-evaluate some implementation details for how backend file handles are dealt with.

Screenshots (if appropriate):

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [ ] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [ ] Ready for review
• [x] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [ ] Current tests cover modifications made
• [x] New tests have been added to the test suite
• [x] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have signed (or will sign when prompted) the tensorwork CLA.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.

Motivation and Context

Why is this change required? What problem does it solve?:

New tutorial covering only the basic stuff for version 0.5 release, such as Repository creation and initialization, adding data to columns and committing changes.

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [x] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [x] Ready for review
• [ ] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [x] Current tests cover modifications made
• [ ] New tests have been added to the test suite
• [ ] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [x] My code follows the code style of this project.
• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have signed (or will sign when prompted) the tensorwork CLA.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.

Describe the bug A clear and concise description of what the bug is.

The pytorch dataloader cannot currently be run with multiprocess workers:

>>> torch_dset = make_torch_dataset(aset, index_range=slice(1, 100))
>>> loader = DataLoader(torch_dset, batch_size=16, num_workers=2)
>>> for batch in loader:
...     train_model(batch)
Exception: Cannot pickle `hangar.dataloaders.TorchDataset.BatchTuple`

This is because the the BatchTuple wrappers passed to hangar.dataloaders.TorchDataset are dynamically defined namedtuple classes, whose definition is not appropriately scoped for a forked subprocess to introspect it's name/contents upon pickling.

https://github.com/tensorwerk/hangar-py/blob/e2c7a89ccb9ddb379e8a3fa8f20dae20fcfb6345/src/hangar/dataloaders/torchloader.py#L63-L74

As the structure needs to be dynamically defined based on user arguments, we cannot just place the BatchTuple definition in the main body of the module.

Two possible solutions:

@lantiga and @hhsecond, let me know what you prefer, or any other solutions you might have.

1) keep the current return type exactally the same, but add the definition of BatchTuple to globals() before it is passed to TorchLoader

        wrapper = namedtuple('BatchTuple', field_names=field_names)
    else:
        wrapper = namedtuple('BatchTuple', field_names=gasets.arrayset_names, rename=True)

    globals()[`BatchTuple`] = wrapper
    return TorchDataset(gasets.arrayset_array, gasets.sample_names, wrapper)

This works, but is generally bad practice to manually modify global scope.

2) return a dict of field_names and tensors instead of a namedtuple

        wrapper = tuple(field_names)
    else:
        wrapper = tuple(gasets.arrayset_names)

    return TorchDataset(gasets.arrayset_array, gasets.sample_names, wrapper)

And in TorchDataset replace: https://github.com/tensorwerk/hangar-py/blob/e2c7a89ccb9ddb379e8a3fa8f20dae20fcfb6345/src/hangar/dataloaders/torchloader.py#L135 with

    return dict(zip(self.wrapper, out))

Which still works, and does not modify globals(), but changes the output o the function to something "not quite as nice" as a namedtuple.

Severity

Select an option:

• [ ] Data Corruption / Loss of Any Kind
• [ ] Unexpected Behavior, Exceptions or Error Thrown
• [x] Performance Bottleneck

Motivation and Context

Revamping plugin system to make at actually pluggable for different data types etc.

Description

• Introducing new io module. This will help users to use the import/export functionality of plugins through the program if they don't wan't to interact with the low-level hangar APIs
• We potentially can move other modules like dataset inside io module
• Test cases are work in progress
• Docs are work in progress

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [x] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [ ] Ready for review
• [x] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [x] Current tests cover modifications made
• [x] New tests have been added to the test suite
• [x] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have signed (or will sign when prompted) the tensorwork CLA.
• [x] I have added tests to cover my changes.
• [x] All new and existing tests passed.

Describe the bug If we try to commit inside the context manager (before __exit__()), hangar throws RuntimeError saying No changes made in the staging area. Cannot commit.. We should allow the user to do commits inside the context manager IMO but probably with a warning about the performance hit

Severity

Select an option:

• [ ] Data Corruption / Loss of Any Kind
• [x] Unexpected Behavior, Exceptions or Error Thrown
• [ ] Performance Bottleneck

To Reproduce

import numpy as np

from hangar import Repository
repo = Repository(path='myhangarrepo')
repo.init(user_name='Sherin Thomas', user_email='[email protected]', remove_old=True)

# generate data
data = []
for i in range(1000):
    data.append(np.random.rand(28, 28))
data = np.array(data)

co = repo.checkout(write=True)
data_dset = co.datasets.init_dataset('mnist_data', prototype=data[0])
co.commit('datasets init')
co.close()
co = repo.checkout(write=True)
data_dset = co.datasets['mnist_data']

with data_dset:
    for i in range(len(data)):
        sample_name = str(i)
        data_dset[sample_name] = data[i]
        co.commit('dataset curation: stage 1')  # this throws error
co.close()

Expected behavior It should not break the program instead raise a warning about the performance hit

@rlizzo I was thinking an API like hangar.dataloaders.pytorch (let me know if you have another structuring in your mind). Basically, the idea is to load data in batches synchronously or asynchronously and enable the features of PyTorch DataLoader. My plan is to have a Dataset class and a DataLoader class which is essentially the way PyTorch's data loader work.

Describe the bug

The message HANGAR RUNTIME WARNING: no repository exists at /some/path/__hangar, please use init_repo function makes me think my script is doing something wrong, even though the next thing that I do is call repo.init().

Additionally, if the path does not exist, there should be an option to have it be created.

A single default param exists=True flag could handle both cases, creating the directory and suppressing the init warning when set to False

Severity

Select an option:

• [ ] Data Corruption / Loss of Any Kind
• [x] Unexpected Behavior, Exceptions or Error Thrown
• [ ] Performance Bottleneck

Motivation and Context

Why is this change required? What problem does it solve?:

Introducing import-export utility over the command line

If it fixes an open issue, please link to the issue here:

This PR won't solve issue #72 completely but starting with image import and export. Also, the release is going to be experimental and the APIs are subjective to change

Description

Describe your changes in detail:

• Moving CLI as a module
• Introducing import option using click
• Introducing export option using click
• Base class for the introduction of plugin system

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [x] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [ ] Ready for review
• [x] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [ ] Current tests cover modifications made
• [x] New tests have been added to the test suite
• [ ] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have signed (or will sign when prompted) the tensorwork CLA.
• [ ] I have added tests to cover my changes.
• [x] All new and existing tests passed.

Motivation and Context

Why is this change required? What problem does it solve?:

better documentation generator.

Does not work at the current moment (unhappy with API plugin for numpy style docstrings).

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

• [x] Documentation update
• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

• [ ] Ready for review
• [x] Work in progress

How Has This Been Tested?

Put an x in the boxes that apply:

• [ ] Current tests cover modifications made
• [ ] New tests have been added to the test suite
• [ ] Modifications were made to existing tests to support these changes
• [ ] Tests may be needed, but they are not included when the PR was proposed
• [ ] I don't know. Help!

Checklist:

• [ ] My code follows the code style of this project.
• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [ ] I have read the CONTRIBUTING document.
• [ ] I have signed (or will sign when prompted) the tensorwork CLA.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.

Is your feature request related to a problem? Please describe. Currently, read checkout won't be able to read data from the staging area. I was wondering what would be the technical difficulties for reading data from staging area

Describe the solution you'd like repo.checkout(stage=True) or something similar could give access to the staging area. I think the major bottleneck from implementing this is the fact that the staging area could change and might even make the data unavailable. Maybe we could have a flag changes when the data changes and on read from a stage checkout, it could let the reading process know that the data changed (not the specific information but just that the data changed) and it can invalidate the checkout?

Executive Summary how does the approach of hangar compare with DVID? I am looking at how solutions to managing really large datasets, and stored ML models. The project dvid seems like its doing something similar to hangar?

I don't know enough about devops to be able to determine what kind of solution I could or should choose or what one is buying into when they choose one.

Describe the bug Diff status always returns CLEAN inside CM

Severity

Select an option:

• [ ] Data Corruption / Loss of Any Kind
• [x] Unexpected Behavior, Exceptions or Error Thrown
• [ ] Performance Bottleneck

To Reproduce

from hangar import Repository
import numpy as np


repo = Repository('.')
repo.init(user_name='me', user_email='[email protected]', remove_old=True)
co = repo.checkout(write=True)
co.add_ndarray_column('x', prototype=np.array([1]))
co.commit('added columns')
co.close()

co = repo.checkout(write=True)
x = co.columns['x']


with x:
    for i in range(10):
        x[i] = np.array([i])
        print(co.diff.status())  # this should return DIRTY but returns CLEAN
print(co.diff.status())  # this returns DIRTY as expected
co.commit('adding file')

Describe the bug In case of multiple columns, if we open only few of such columns in context manager and still tries to write on other columns inside the context manager, transactions get False. A sample script to reproduce is given below

Severity

Select an option:

• [ ] Data Corruption / Loss of Any Kind
• [x] Unexpected Behavior, Exceptions or Error Thrown
• [ ] Performance Bottleneck

To Reproduce

from hangar import Repository
import numpy as np


repo = Repository('.')
repo.init(user_name='me', user_email='[email protected]', remove_old=True)
co = repo.checkout(write=True)
co.add_ndarray_column('x', prototype=np.array([1]))
co.add_ndarray_column('y', prototype=np.array([1]))
co.commit('added columns')
co.close()

co = repo.checkout(write=True)
x = co.columns['x']
y = co.columns['y']


with x:  # note that we are opening only `x` in the CM
    for i in range(10):
        y[i] = np.array([i])  # but we are trying to update `y` column
        x[i] = np.array([i])
co.commit('adding file')
co.close()

Desktop (please complete the following information):

• OS: Ubuntu 19.10
• Python: 3.7
• Hangar: 0.5.1.dev0 (master, at the time of writing)

May be mention that this metadata is commit level and will not be part of the history.

Originally posted by @hhsecond in https://github.com/tensorwerk/hangar-py/pull/180