Description

Goal of this branch is to create a grouping feature that will allow for operations in FlowProject to be organized into "metaoperations." Currently to impliment this behavior either a new FlowProject has to operate on the original set of operations or user commenting has to be done to prevent the group operations from running individually.

An example snippet of expected functional shows in more detail a potential API and use of the feature,

a_group = Project.make_group('a')

@Pr.operation
@a_group
def foo(job):
    pass

@Pr.operation
@a_group
def bar(job):
    pass

Then to use the grouping a command like, submit --group a_group would output a job for the scheduler with the final command being something akin to python project.py run -j abc123 -o foo bar

Motivation and Context

This change is a feature change that seeks to give the user more control over the workflow of their project. The group concept will allow for operations to be tagged to be run at once in a single job which can work even if the pre and post conditions are not met initially but will be once other operations have run.

This abstraction of a workflow is useful for multiple scenerios such as automatically restricting operations to particular environements, running a job repeatedly in a single submit #4 , and moving workflow logic into the FlowProject and not the individual operations for idiomatic code. Another use case would to be mark operations as submit only #33 . Another issue tangentially related would be #41 as groups could be considered to create graphs themselves or groups might make up higher level graphs.

Types of Changes

• [x] Documentation update
• [ ] Bug fix
• [x] New feature
• [x] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [x] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog.

This pull request refactors the current job-operation model into jobs-operations, that means each operation is a function of one or more jobs.

Prior to merging we need to tackle the following items:

• [ ] Implement parallelized status update
• [ ] Update changelog
• [ ] Deprecate JobOperation
• [ ] Ensure that the scheduler status is updated prior to submission.

Description

Adopting pytest as testing framework. This version of my code outputs an assertion error.

Motivation and Context

After this change, We'll be using pytest instead of unittest to perform the unit tests.

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [ ] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [x] I have updated the changelog.

Description

Added eligibility check for operations generated when cmd is provided in main_script

Motivation and Context

Fixes #218

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [ ] New feature
• [x] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [ ] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog.

Description

Motivation and Context

see issue #113

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [x] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog.

Description

This pull request implements an execution hooks framework that enables users to execute certain functions on start, on finish, on success, and on fail of the execution of an operation.

Resolves #28, resolves #14.

Motivation and Context

The motivation for this framework is to enable users to automatically execute certain functions with operations for example for logging and tracking purposes. For example, to log when a specific operation is executed, a user can provide the following function:

@Project.operation
@Project.hooks.on_start(lambda op: logger.info(f"Executing {op}..."))
def foo(job):
    pass

The framework comes with a selection of pre-implemented hook systems to cover all the currently expected use cases, roughly in order of expensiveness:

1. LogOperation - Log basic information about the execution of operations to a log file.
2. TrackOperations - Keep detailed metadata records about the state of the project root directory and the operation, including directives, to a log file, optionally in conjunction with git.
3. SnapshotProject - Create a snapshot of the project root directory to keep track of the code used for the execution of an operation.
4. TrackWorkspaceWithGit - The workspace is treated as a git repository and automatically committed to before and after the execution of an operation. The can be done either on a per-job basis or workspace global.

A hook system is meant to describe a collection of hook functions that together achieve a specific purpose. The shipped hook systems are implemented as classes that can be installed project-wide via the respective install_hooks function.

High-level API

Hooks can be installed either on a per-operation or a per-project basis. In this way users have the option to execute hook functions either with specific operations or with all operations of a project.

Furthermore, there are two ways that hooks can be installed. One way is directly in Python, for example within the __main__ clause of the project.py file:

# ...

if __name__ == '__main__':
    from flow.hooks import LogOperations
    LogOperations().install_hooks(Project()).main()

Alternatively, hooks can also be installed through the (project) configuration file:

project = my_project
[flow]
hooks=flow.hooks.LogOperations,flow.hooks.SnapshotProject(compress=True)

It is assumed that the entities provided here, are callable, i.e., are either a function or a functor. The first argument must be the instance of FlowProject.

Low-level API

The FlowProject class has a hooks attribute with four lists: on_start, on_finish, on_success, and on_fail, which can be appended to. Hook functions installed in this way are executed for all operations.

Furthermore, hook functions can be installed for individual operations either with a decorator:

@Project.operation
@Project.hook.on_start(my_hook_function)
def op(job):
    pass

or by passing a dictionary to the add_operation() function: project.add_operation(..., hooks=dict(on_start=my_hook_function)).

Custom hook systems

Users can very easily implement their own hook systems and install them in similar manner. For example:

# myhooks.py
def log_op(operation):
    with open(operation.job.fn('operations.log'), 'a') as logfile:
        logfile.write(f"Executed operation '{operation}'.")

def install(project):
    project.hooks.on_success.append(log_op)
    return project

This could then be installed either in the project module:

# ...

if __name__ == '__main__':
    import myhooks
    myhooks.install(Project()).main()

or via the configuration file:

project = myproject
[flow]
hooks=myhooks.install

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [x] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [x] I have updated the changelog.

Feature description

Related to #61 and #144, I think it would be good if show_traceback = on by default.

Proposed solution

Change default behavior to enable traceback.

@glotzerlab/signac-developers What are your thoughts about this potential change? The tracebacks are sometimes a little complicated, but I think it's reasonable to expect that Python scripts will show a traceback on error.

Description

This pull request adds the support of aggregator class in project.py

Motivation and Context

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog.

We can now use Pool or ThreadPool conditionally.

Description

This pull request is a consequence of #269, which was accidentally closed by me.

Motivation and Context

Fixes #264

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [ ] I have updated the changelog.

Description

I am trying to integrate fbpic, a well-known CUDA code (based on Python + Numba) for laser-plasma simulation with signac. The integration repo is signac-driven-fbpic.

I managed to succesfully run on a single GPU, via python3 src/project.py run from inside the signac folder, but if I add --parallel I get

numba.cuda.cudadrv.error.CudaDriverError: CUDA initialized before forking

The goal is to get 8 (independent) copies of fbpic (with different input params) running in parallel on the 8 NVIDIA P100 GPUs that are on the same machine.

To reproduce

Clone the signac-driven-fbpic repo and follow the install instructions. Then go to the signac subfolder, and do

conda activate signac-driven-fbpic
python3 src/init.py
python3 src/project.py run --parallel

Error output

(signac-driven-fbpic) andrei@ServerS:~/Development/signac-driven-fbpic/signac$ python3 src/project.py run --parallel --show-traceback
Using environment configuration: UnknownEnvironment
Serialize tasks|----------------------------------------------------------------------------------Serialize tasks|#####-----------------------------------------------------------------------------Serialize tasks|##########------------------------------------------------------------------------Serialize tasks|###############-------------------------------------------------------------------Serialize tasks|####################--------------------------------------------------------------Serialize tasks|##########################--------------------------------------------------------Serialize tasks|###############################---------------------------------------------------Serialize tasks|####################################----------------------------------------------Serialize tasks|#########################################-----------------------------------------Serialize tasks|###############################################-----------------------------------Serialize tasks|####################################################------------------------------Serialize tasks|#########################################################-------------------------Serialize tasks|##############################################################--------------------Serialize tasks|###################################################################---------------Serialize tasks|#########################################################################---------Serialize tasks|##############################################################################----Serialize tasks|##################################################################################Serialize tasks|##################################################################################Serialize tasks|##############################################################################################|100%
ERROR: Encountered error during program execution: 'CUDA initialized before forking'
Execute with '--show-traceback' or '--debug' to get more information.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 2727, in _fork_with_serialization
    project._fork(project._loads_op(operation))
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 1467, in _fork
    self._operation_functions[operation.name](operation.job)
  File "src/project.py", line 172, in run_fbpic
    verbose_level=2,
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/fbpic/main.py", line 232, in __init__
    n_guard, n_damp, None, exchange_period, use_all_mpi_ranks )
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/fbpic/boundaries/boundary_communicator.py", line 267, in __init__
    self.d_left_damp = cuda.to_device( self.left_damp )
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context
    return fn(*args, **kws)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/api.py", line 103, in to_device
    to, new = devicearray.auto_device(obj, stream=stream, copy=copy)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 683, in auto_device
    devobj = from_array_like(obj, stream=stream)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 621, in from_array_like
    writeback=ary, stream=stream, gpu_data=gpu_data)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 102, in __init__
    gpu_data = devices.get_context().memalloc(self.alloc_size)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 697, in memalloc
    self._attempt_allocation(allocator)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 680, in _attempt_allocation
    allocator()
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 695, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 290, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 324, in _check_error
    raise CudaDriverError("CUDA initialized before forking")
numba.cuda.cudadrv.error.CudaDriverError: CUDA initialized before forking
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "src/project.py", line 238, in <module>
    Project().main()
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 2721, in main
    _exit_or_raise()
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 2689, in main
    args.func(args)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 2414, in _main_run
    run()
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/legacy.py", line 193, in wrapper
    return func(self, jobs=jobs, names=names, *args, **kwargs)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 1597, in run
    np=np, timeout=timeout, progress=progress)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 1421, in run_operations
    pool, cloudpickle, operations, progress, timeout)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/site-packages/flow/project.py", line 1458, in _run_operations_in_parallel
    result.get(timeout=timeout)
  File "/home/andrei/anaconda3/envs/signac-driven-fbpic/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
numba.cuda.cudadrv.error.CudaDriverError: CUDA initialized before forking

Relevant numba link.

System configuration

• Operating System: Ubuntu 16.04
• Version of Python: 3.6.8
• Version of signac: 1.1.0
• Version of signac-flow: 0.7.1
• NVIDIA Driver Version: 410.72

By - @b-butler

Adds two classes Directives and DirectivesItem that serve as a smart mapping for the environment and user-specified directives and a specification for environmental directives respectively. FlowProject and FlowGroup have been changed accordingly.

Description

Motivation and Context

This resolves #265 and helps to centralize logic for directives. This pull request is a necessary follow-up for #282. Also resolves #240.

Types of Changes

• [ ] Documentation update
• [ ] Bug fix
• [x] New feature
• [ ] Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Tasks to accomplish:

• [x] Test the individual DirectivesItems.
• [x] Test the Directives class.
• [x] Determine and add environment specific DirectivesItem and correct their get_default_directives function (this at least at the decision level will likely involve a discussion between multiple people).
• [x] Some probably large degree of code refactoring (I was just initially trying to get the outline working and there).
• [ ] Opening a PR in signac-docs to update documentation.
• [x] Going through docstrings and ensuring they are complete, grammatically correct, and helpful. Multiple methods will need these as well.
• [x] Add tracking of user specified directives. Previously TrackGetItemDict found in flow/util/misc.py was used. We can use this internally or create our own fix.
• [x] Fix code so tests pass

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] My code follows the code style guideline of this project.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• [ ] I have updated the API documentation as part of the package doc-strings.
• [ ] I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• [x] I have updated the changelog.

Description

This PR removes setup.py and setup.cfg entirely, migrating all project and build configuration information into pyproject.toml. In the process, all linter configs have also been moved into pyproject.toml. The exception is flake8, which does not (and will not) support pyproject.toml, so the flake8 configuration is now stored in the .flake8 file, which is specific to this linter. Additionally, bump2version also does not support pyproject.toml (although unlike flake8 the proposal has not been entirely rejected, so it may eventually), so that configuration has also been moved to a project-specific .bumpversion file.

Motivation and Context

Various changes to Python packaging over the last 6 or 7 years have moved towards more static packaging and towards storing data in a backend-agnostic format. These changes allow these of setuptools alternatives (like flit) as well as more reproducible builds based on build isolation into virtual environments that provide all necessary build dependencies. Direct invocation of setup.py has been deprecated in the process. The changes in this PR modernize flow's build system for compatibility with these new approaches.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.
• [ ] The package documentation and framework documentation in signac-docs are up to date with these changes.
• [ ] I have updated the changelog and added any related issue and pull request numbers for future reference.

Feature description

Discussed with @joaander today.

I would like to be able to set the account on the command line when submitting jobs through flow.

I'm setting it via environment variable, so signac-flow gives me this warning. I would like to be able to silence this warning.

Environment 'DeltaEnvironment' allows the specification of an account name that will be charged for jobs' compute time.
Set the account name with the command:

  $ signac config --global set 'flow.DeltaEnvironment.account' ACCOUNT_NAME

Proposed solution

This may be a good first issue.

Additional context

Other ways to set account:

• environment variable
• flow config
• custom template

Description

Fixes logic where the --ntasks-per-node would not normalize based on number of nodes for GPU submissions where the number of tasks is often the number of GPUs.

Motivation and Context

Resolved: #566

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.
• [x] The package documentation and framework documentation in signac-docs are up to date with these changes.
• [ ] I have updated the changelog and added any related issue and pull request numbers for future reference.

Description

Currently we execute operation decorators from outside in or top bottom. This is confusing when using decorators as functions directly.

@FlowProject.operation
def op(job):
    pass


FlowProject.pre(expesive_cond)(FlowProject.pre.true("foo")(op))

This actually runs the expensive computation first. This is to make this

@FlowProject.pre.true("foo")
@FlowProject.pre(expensive_cond)
@FlowProject.operation
def op(job):
    pass

more intuitive. However, I disagree that this is more intuitive. As someone learns Python in fact this begins to become less and less intuitive to the point that without our documentation suggesting the correct ordering, a Python expert would write,

@FlowProject.pre(expensive_cond)
@FlowProject.pre.true("foo")
@FlowProject.operation
def op(job):
    pass

Given our recent decorator ordering requirements we are already making users come to understand decorators apply bottom to top.

Suggestion

I think we should apply conditions in the order they come in the execution (i.e. bottom to top). As an irrelevant side note this would make the project definition run faster.

Description

Migrates signac-flow's CI to use GitHub Actions.

This PR drops coverage for a few things that I think are non-essential:

• Nightly/weekly testing of pip install signac signac-flow and conda install signac signac-flow
• Testing against the latest commit of signac (which is currently disabled because it's broken until we release 2.0)
• Checking Zenodo metadata on release/.* branches.

If another contributor wishes to add these to GitHub Actions, that would be fine.

Before merging, a repo administrator will need to update the CI checks used for branch protections.

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.
• [ ] The package documentation and framework documentation in signac-docs are up to date with these changes.
• [ ] I have updated the changelog and added any related issue and pull request numbers for future reference.

Description

Added the no-progress flag for FlowProject.print_status. This hides the progress bar output generated when FlowProject.print_status, and adds the options to hide the progress bar when desired.

Motivation and Context

The progress bar displays a significant amount of the output for small workspaces and does not give any benefit. Additionally, using FlowProject.print_status in Jupyter notebooks requires special configurations or else it does not work. This addresses issue #602

Checklist:

• [x] I am familiar with the Contributing Guidelines.
• [x] I agree with the terms of the Contributor Agreement.
• [x] My name is on the list of contributors.
• [x] The changes introduced by this pull request are covered by existing or newly introduced tests.
• [x] The package documentation and framework documentation in signac-docs are up to date with these changes.
• [x] I have updated the changelog and added any related issue and pull request numbers for future reference.