Description

Added buffering feature for SyncedCollection. The buffering will be provided by signac.buffered and SyncedCollection.buffered.

Motivation and Context

Related to #249. This is continuation of work in PR

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog and added all related issue and pull request numbers for future reference (if applicable). See example below.

Example for a changelog entry: Fix issue with launching rockets to the moon (#101, #212).

This PR switched from a bug fix to a refactor. See #666 for the bug fix only.

Original Title: Don't generate views with underspecified path provided by user

Signac has unexpected behavior when generating a view if the user specifies a custom path that doesn't uniquely specifiy jobs.

I would expect signac to link to all jobs matching the description in the view folder.

What @Nipuli-Gunaratne and I found was that signac just picks one job.

Description

We check that the mapping of source-> link is 1-1 in other parts of import_export.py but don't check user's provided path function.

Two posssible solutions I see are:

1. Generate an error and exit. Suggest the fix in the error message.
2. Try to fix the problem by adding jobs ids to the path.

In this draft PR, I find the places in the code for adding the error checking code in two places it might fit: import_export.py::_make_path_function and in linked_view.py::create_linked_view.

I think it works best in _make_path_function.

Motivation and Context

If you make this test project

#init.py
import signac

project = signac.init_project('Test-view')

jobs = [dict(a=1,b=1),
        dict(a=1,b=2),
        dict(a=2,b=1),
        dict(a=2,b=2)
        ]


for j in jobs:
    job = project.open_job(j)
    job.init()
    print(j, job.id)

and generate a view with a user-specified custom path that does not uniquely identify jobs

signac view test_error "a/{a}"

Signac just picks one of the jobs to link. a=1 gets job 8aacdb17187e6acf2b175d4aa08d7213 (b=2) and not 386b19932c82f3f9749dd6611e846293 (b=1)
a=2 gets job 5e4d14d82c320bafb2f1286fe486d1f8 (b=1) and not d48f81ad571306570e2eb9fe7920cd3c (b=2)

Fix that we'd suggest to users OR try to do automatically:

Remake the path specification as "a/{a}/id/{id}"

Types of Changes

• [ ] Documentation update
• [x] Bug fix
• [ ] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [ ] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog and added all related issue and pull request numbers for future reference (if applicable). See example below.

Tl;dr: We need to improve synchronization and caching logic, and I think the first step is to combine the _SyncedDict, SyncedAttrDict, and JSONDict classes.

I apologize in advance for the lengthy nature of this issue. This issue will serve as a pseudo-signac Enhancement Proposal, I'll try and document very thoroughly and it can be a test case for the utility of such proposals :)

In view of our recent push for deprecations and our discussion of reorganizing namespaces and subpackages to prepare for signac 2.0, I'd like to also revisit discussion of the different dict classes. We have various open bugs and features (#234, #196, #239, #238, #198) that are related to improving our synchronization and caching processes. Our synchronization clearly has some holes in it, and in the process of making #239 @bdice has raised concerns about inconsistencies with respect to cache correctness and cache coherence, e.g. the fact that a job that exists and is cached will still exist in the cache after it is deleted (Bradley, feel free to add more information).

Fixing all of these is a complex problem, in part due to fragmentation in our implementation of various parts of the logic. I'd like to use this issue to broadly discuss the various problems that we need to fix, and we can spawn off related issues as needed once we have more of a plan of attack to address our problems. Planning this development more thoroughly is critical since the bugs that may arise touch on pretty much all critical code paths in signac. I think that a good first step is looking into simplifying the logic associated with our various dictionary classes. That change should make it easier to improve #198 since synchronization will be in one place. After that, I think it will be easier to consider the various levels of caching and properly define the invariants we want to preserve.

With respect to the various dictionary classes, I think we need to reassess and simplify our hierarchy:

• The core class is the _SyncedDict class in core/synceddict.py, and I think it should exist in more or less its current form.
• I understand the logic of separating out the SyncedAttrDict in core/attrdict.py since attribute-based access to a dictionary is technically a feature unrelated to synchronization. However, the class is very minimal, and I think that the benefits of maintaining this level of purity in distinction is outweighed by the increased difficulty users and newer developers have in finding code in the code base. I would like to merge these classes.
• The JSONDict class in core/jsondict.py is, in my opinion, harder to justify separating from _SyncedDict on a conceptual level. Although in principle one could argue for different types of file backends, in practice we're very tied to JSON. The bigger problem, though, is that in my understanding (please correct me if I'm wrong here) the primary distinction between the two classes is less about the file backend and more about buffering. We always set the parent of SyncedAttrDict, which is what we use for job statepoints, and this ensures that statepoints changes are immediately synced to disk. Conversely, job documents are JSONDict objects, which use buffering. The fact that the _SyncedDict has _load and _save methods that essentially must be implemented by a child class when parent is not set, and that the JSONDict is the only example we have of such a class, suggests that this is a level of abstraction that isn't very helpful and mainly complicates management of the code. At least for now, I would prefer to unify the JSONDict with _SyncedDict; the logic for when we buffer is already governed by the parent, but the logic for how we buffer is governed by the various other functions in jsondict.py. Afterwards, if we see a benefit to separating the choice of file backend, we could recreate JSONDict where the new version of the class would really only implement JSON-specific logic. This change would have the added benefit of unifying statepoints and documents: I don't think it is intuitive design to have the document and the statepoint be two different classes for reasons of buffering, and it makes it substantially more difficult to follow the logic of how the _sp_save_hook works and why it's necessary. Longer term, I would like to refactor the logic for persistence vs. buffering so that the roles of Job and _SyncedDict are more disjoint, but I recognize that there may not be a way to completely decouple the bidirectional link.

@csadorf @bdice @mikemhenry any commentary on this is welcome, also please tag any other devs who might have enough knowledge of these topics to provide useful feedback.

Description

Changes behavior of Job to load its statepoint lazily, when opened by id.

Motivation and Context

Implementation of #238.

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [x] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [x] I have updated the changelog.

Add id property to Job

Description

Adds id as a property of the Job class. I also added a test to ensure that it produces the correct string.

Motivation and Context

Just follows the python trend of using properties instead of getters and setters.

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog.

Description

I created this PR to start addressing this issue: https://github.com/glotzerlab/signac-docs/issues/50

@klywang Do you have specific suggestions on how to improve this? (I don't think I've "solved" the issue yet, since I haven't provided explicit examples for job.data as requested.) Feel free to edit this PR directly.

Motivation and Context

https://github.com/glotzerlab/signac-docs/issues/50

Types of Changes

• [x] Documentation update
• [ ] Bug fix
• [ ] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [ ] My name is on the list of contributors.
• [ ] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog.

Description

This PR is related to #249 . In this PR, we are implementing SyncedCollection, SyncedAttrDict, SyncedList, JSONCollection, JSONDict, JSONList.

Motivation and Context

This refractor is to provide support for the multiple backends and resolve #196.

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [x] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog and added all related issue and pull request numbers for future reference (if applicable). See example below.

Example for a changelog entry: Fix issue with launching rockets to the moon (#101, #212).

A possible optimization for signac 2.0 (or later) would be to reduce the amount of code we use in the Collection class when calling Project.find_jobs. See: https://github.com/glotzerlab/signac/blob/b29e0485c7998ba0c6d041e9ec15b533334d9b64/signac/contrib/project.py#L717

We copy a large amount of data used from the Project's internal caches during the construction of a Collection and calling its find method. At the very least, we never use the Collection's ability to interact with data on disk or its ability to automatically generate ids (primary keys) for new records in the context of a signac Project, so we could eliminate some logic there in a cut-down class.

Originally posted by @bdice in https://github.com/glotzerlab/signac/issues/652#issuecomment-1002303783

I attempted an optimization in 13f2a8fb205f65d49421f4f4009ee3d78d00f9bf but it was unclear if copy/reference semantics would be correct in the resulting indices outside the context of a signac Project's limited usage of Collection. A smaller class that is designed for the actual use case of signac's Project.find_jobs could act as an internal cache with only the necessary logic (e.g. no file I/O or id generation).

As part of our overall docs overhaul (see glotzerlab/signac-docs#64), we want to convert our docstrings to numpy style (as decided in glotzerlab/signac-docs#74). The best automated tool I'm familiar with for this task is pyment. In addition to converting docstrings, it will also generate docstrings for functions, classes, etc that are missing docstrings entirely. However, the conversion will require significant manual review to ensure that all docstrings are converted correctly.

Original report by Bradley Dice (Bitbucket: bdice, GitHub: bdice).

I would like to use nested keys in a job document. Presently, this does not work as one would expect from "normal" dictionaries. See code snippet below.

#!python
>>> job.document # We start from an empty job document
{}
>>> job.document['a']['b'] = 'c' # This will error as expected, since job.document['a'] is unassigned
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bdice/env/glotzer-software/signac/0.8.5/lib/python/signac-0.8.5-py3.5.egg/signac/core/jsondict.py", line 48, in __getitem__
KeyError: 'a'
>>> job.document['a'] = dict() # We create an empty dictionary as the value of key 'a'
>>> job.document['a']['b'] = 'c' # We attempt to assign a nested key
>>> job.document # This nested key does not appear to be set
{'a': {}}
>>> job.document['a'] = dict(b = 'c') # However, it is possible to set the value to be another dictionary, and this works
>>> job.document
{'a': {'b': 'c'}}

After looking through the source code of the JSonDict with @vyasr, it is not immediately clear how to fix this in the source. It would probably involve some customization of the __getitem__ or __setitem__ functions. However, this workaround will function:

#!python
>>> temp_dict = job.document['a']
>>> temp_dict['b'] = 'c'
>>> job.document['a'] = temp_dict

Version:

$ signac --version
signac 0.8.5

Based on feature usage, the project root directory currently contains a number of special files. Some of them are hidden, some are not.

• signac_project_document.json
• signac.rc
• .signac_sp_cache.json.gz
• .signac_history.txt

As suggested by @vyasr we may want to switch to a more compact storage format, for example by bundling all files within a .signac folder.

Bumps coverage from 6.5.0 to 7.0.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps pytest-xdist from 3.0.2 to 3.1.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps redis from 4.3.5 to 4.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps numpy from 1.23.5 to 1.24.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps tables from 3.7.0 to 3.8.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Description

This PR removes setup.py and setup.cfg entirely, migrating all project and build configuration information into pyproject.toml. In the process, all linter configs have also been moved into pyproject.toml. The exception is flake8, which does not (and will not) support pyproject.toml, so the flake8 configuration is now stored in the .flake8 file, which is specific to this linter. Additionally, bump2version also does not support pyproject.toml (although unlike flake8 the proposal has not been entirely rejected, so it may eventually), so that configuration has also been moved to a project-specific .bumpversion file.

Motivation and Context

Various changes to Python packaging over the last 6 or 7 years have moved towards more static packaging and towards storing data in a backend-agnostic format. These changes allow these of setuptools alternatives (like flit) as well as more reproducible builds based on build isolation into virtual environments that provide all necessary build dependencies. Direct invocation of setup.py has been deprecated in the process. The changes in this PR modernize signac's build system for compatibility with these new approaches.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.
• [x] The package documentation and framework documentation in signac-docs are up to date with these changes.
• [x] I have updated the changelog and added any related issue and pull request numbers for future reference.