Cylc: a workflow engine for cycling systems

Overview

PyPI Anaconda-Server Badge chat forum Documentation

Cylc (pronounced silk) is a general purpose workflow engine that specialises in cycling workflows and has strong scaling characteristics.

Cylc was originally developed to meet the challenges of production weather forecasting - which is notorious for the size and complexity of its workflows.

Citations & Publications

DOI JOSS CISE

Cylc 7 (production)

python Documentation

  • Production ready.
  • HTTPS network layer.
  • PyGTK GUI.
  • On the 7.8.x branch in the source code.
  • 7.8 - Python 2.6
  • 7.9 - Python 2.7

Quick Installation | Website | Documentation

Cylc 8 (pre-release)

PyPI PyPI Anaconda-Server Badge Documentation

  • Beta pre-release.
  • ZMQ (TCP) network layer.
  • In-browser GUI
  • On the master branch in the source code.

Cylc-8.0.0 will be released in 2021. Until then we recommend the latest Cylc 7 versions for production use.

Installation | Documentation - in development

Copyright and Terms of Use

License

Copyright (C) 2008-2021 NIWA & British Crown (Met Office) & Contributors.

Cylc is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Cylc is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Cylc. If not, see GNU licenses.

Contributing

Contributors Commit activity Last commit

Contributions welcome:

Comments
  • Use ISO standard date-time formats

    Use ISO standard date-time formats

    This is a completely general notation for specifying date/times and time ranges. It would presumably allow us to replace all the current cycling modules (daily, monthly, etc.) with a single one that could handle any cycle time sequence. And cycling for paleo-climate simulations?

    MPI is developing an ISO time library for Python.

    opened by cylc 71
  • trigger: generalisation of triggering approaches

    trigger: generalisation of triggering approaches

    Related Issues:

    If agreed this issue should supersede:

    • https://github.com/cylc/cylc-flow/issues/4657
    • https://github.com/cylc/cylc-flow/issues/4653

    After a long chat with @dpmatthews (who proposed yet another triggering approach 😁) I think we can generalise the trigger problem into two dimensions:

    • Continue (yes/no).
      • After I trigger the task will the flow continue from that point immediately.
      • Or does it only continue if/when a flow front catches up with it.
      • I.E. Should the triggered tasks spawn children on completion or after "merge".
    • Overrun (yes/no).
      • Should the "merge" [1] condition be based on the pool or the DB?
      • I.E. Should triggered tasks overrun previous runs of tasks?
      • I.E. Should the following flow overrun the triggered tasks?

    Note: From the internal implementation these two dimensions may appear flip-sides of the same coin since they both boil down to the flow_nums, however, considering them from a user standpoint I think it's fair to prise them apart.

    Note: Purposefully using new terminology to avoid conflation with existing terms, we may want to workshop "continue" and "overrun" a touch.

    [1]: The quoted "merge" above relates to the interaction between two tasks with different flow_nums in general and not to the more specific concept of "flow merging" in the pool exclusively.

    Combing these we get four spaces:

    | | Continue | Don't Continue | |------------|-----------------------|---------------------------------------------------| | Overrun | (1) Reflow (as currently implemented) | (3) No Flow (current default trigger behaviour) | | No Overrun | (2) Continue (@dpmatthews new proposed implementation) | (4) No Flow (@oliver-sanders proposed implementation) |

    • The bad news is it looks like we have use cases for all four.
    • Dave & I think the no-overrun cases are more important than the overrun ones.
    • The good news is that they can coexist and the mechanism for supporting all four is currently implemented, it's mostly an interface problem.

    Going through the four spaces in detail:

    1) Reflow (implemented)

    Equivalent to cylc trigger --flow=<new-flow-number>.

    Continue: Yes Overrun: Yes

    • Tasks are triggered with a new flow number.
    • The reflow can overrun previous flows.
    • The reflow will merge if it collides with another flow in the pool (and only in the pool i.e. overrun).

    The use case is for re-running over tasks which have been previously run e.g. change configuration and re-run a sub-graph.

    2) Continue (proposed)

    Equivalent to cylc trigger --flow=<all-flow-numbers>,<new-flow-number>.

    Continue: Yes Overrun: No

    • A new trigger approach proposed by @dpmatthews.
    • Tasks are triggered with all existing flow numbers plus a new flow number (which we added purely so the new flow can still be targeted by CLI tools).
    • Because this flow contains all existing flow numbers it will not be overrun by any of the flows which exist at the time of the trigger.
    • This is intended for the sort of use cases we would expect --flow=1 to be used for, but has been generalised to be reflow compatible.

    This approach feels quite "natural". The use cases are setting off another bit of the same flow where you don't want tasks to be overrun.

    3) No Flow (implemented)

    Equivalent to cylc trigger --flow -1.

    I am using a negative flow number rather than None to distinguish the two no-flow approaches. Internally we can still maintain the same no-flow logic as present but would need to change the marker.

    Continue: No Overrun: Yes

    Useful for running one-off tasks that you do not want to impact the workflow in any way (i.e. cylc submit type uses).

    4) No Flow (proposed)

    Equivalent to cylc trigger --flow -2.

    I am using a negative flow number rather than None to distinguish the two no-flow approaches. Internally we can still maintain the same no-flow logic as present.

    Continue: No Overrun: No

    Use case is for manually intervening in graph execution by ignoring dependencies or runahead limit and skipping ahead to a task which you want to be considered a part of the approaching flow front.

    Interface

    The internals to handle the four cases are already in-place, flow_nums, DB lookups etc, so it mostly boils down to an interface / documentation issue.

    I think all four methods could be exposed via a single --flow argument, however, it is sensible to provide defaults for the different behaviours. I think it would be good to document the --flow equivalents as they may help users to understand their function.

    Note that --reflow currently determines the new flow number server rather than client side which is sensible.

    1) Enable behaviours explicitly

    If we are happy with the continue/overrun model (after workshopping the terms) we could expose it directly something like:

    # 1) reflow
    cylc trigger --continue --overrun
    
    # 2) continue
    cylc trigger --continue
    
    # 3) no-flow (implemented)
    cylc trigger --overrun
    
    # 4) no-flow (proposed)
    cylc trigger
    

    This is quite nice as you have to explicitly opt in to each behaviour separately reducing the scope for unintended results and accidents.

    2) Single --flow argument

    if we don't like the continue/overrun model we could move the presets into the flow argument something like:

    # 1) reflow
    cylc trigger --flow=new
    
    # 2) continue
    cylc trigger --flow=any
    
    # 3) no-flow (implemented)
    cylc trigger --flow=none
    
    # 4) no-flow (proposed)
    cylc trigger --flow=next
    

    It's less behaviour driven so we would need to explain each option separately.

    3) Separate flag for each approach

    An alternative to (2) would be to could come up with three/four different flags:

    # 1) reflow
    cylc trigger --reflow
    
    # 2) continue
    cylc trigger --flow
    
    # 3) no-flow (implemented)
    cylc trigger --rerun
    
    # 4) no-flow (proposed)
    cylc trigger  # --run
    

    Default

    I think no-continue & no-overrun is the safest, sanest default because:

    • The minimum set of behaviours is the simplest.
    • The "Continue" cases have a dramatic impact on the workflow execution and are hard to revoke.
    • The "Re-run" cases are quite advanced and require additional knowledge to operate.

    But I'm biased. I think the default is less important than the clear separation of behaviours.

    opened by oliver-sanders 70
  • Fix param graph when mixing offset and conditional

    Fix param graph when mixing offset and conditional

    These changes close #2608

    Requirements check-list

    • [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
    • [x] Contains logically grouped changes (else tidy your branch by rebase).
    • [x] Does not contain off-topic changes (use other PRs for other changes).
    • [x] Appropriate tests are included (unit and/or functional).
    • [x] Appropriate change log entry included.
    • [x] No documentation required.
    opened by kinow 68
  • Add coverage to the project reports.

    Add coverage to the project reports.

    Travis-CI configuration was updated to use build stages. The build matrix was moved to the test stage, and extra configuration for coverage.py and codacy were added.

    The tool used to collect coverage from the tests in Coverage.py, which is supported by multiple build and reporting systems.

    Due to the nature of the tests in cylc, and to how they run in parallel in different containers in Travis-CI, we had to use a sitecustomize.py script.

    As Codacy was already used for code analysis, this pull request includes settings to export the coverage report to Codacy. I assume the Codacy token is already set up. So maybe just merging the pull request would already make the coverage report available in the cylc dashboard.

    It is also possible to get the coverage of pull requests.

    Cheers Bruno

    opened by kinow 65
  • Incremental data-store generation

    Incremental data-store generation

    The current data-store generation is inefficient (as was the old state_summary_mgr.py), in that it generates an entirely new data-store on every loop where any change has flagged... The update/creation has been granulated into the following actions/methods:

    1. Creation of workflow/taskdef/familydef data elements.
    2. Generation of graph edges and ghost-nodes/ghost-instances/ghost-proxies (containing def data and tree relationships) for each new cycle point in the pool.
    3. Update/population of [ghost] task instance fields with dynamic data (i.e. status info) for given pool tasks.
    4. Update/population of [ghost] family instance fields with dynamic data (i.e. status info) according to given cycle points (i.e. those cycle points of the updated tasks).
    5. Update/population of workflow (data element) fields with dynamic data (i.e. status info).
    6. Prune data-store of elements (edges/nodes ...etc) and items in fields on or in reference to given list of point_strings.

    These are used within the scheduler in implementation with:

    • Initiation of data-store on run/restart/reload using (1)-(5)
    • Increment of graph elements and prune old elements using (2), (5), and (6) worked out from the head and tail of the task pool respectively (workflow updated after with new state totals etc).
    • Populate/Update of dynamic fields of task/family instances and then workflow element ((3), (4), then (5)), from tasks flagged as updated by the workflow.

    There was a bug in the job pool preventing job element deletion, this has also been fixed.

    This change has the following impacts:

    • Significant reduction of load on the workflow, mainly from not having to generate entire graph on every change (also that job deletion bug).
    • The data-store is used both in the WS and UI Server as driving data for GraphQL, so this change sets up the ground work for implementing an incremental data sync between WS & UIS data store.

    Closes https://github.com/cylc/cylc-flow/issues/3261 Closes https://github.com/cylc/cylc-flow/issues/3212

    Requirements check-list

    • [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
    • [x] Contains logically grouped changes (else tidy your branch by rebase).
    • [x] Does not contain off-topic changes (use other PRs for other changes).
    • [x] Appropriate tests are included (unit and/or functional).
    • [x] No change log entry required (e.g. change is small or internal only).
    • [x] No documentation update required for this change.
    efficiency 
    opened by dwsutherland 63
  • Task proxy spawn on demand.

    Task proxy spawn on demand.

    Spawn on Demand implementation as per proposal https://cylc.github.io/cylc-admin/proposal-spawn-on-d.html

    See #3474 (superseded) for earlier discussion and some proof of efficiency.

    • partially addresses #3304
    • close #774
    • close #2143
    • close #3226
    • (probably closes other issues too...):
    • close #987
    • close #2643
    • close #1314
    • close #2654

    This branch:

    • correctly runs the "complex" test workflow for multiple cycles (last I checked, recently)
    • handles reflow properly as described in the proposal
    • seems to work for all manner of manual test cases

    What's still to do:

    • [x] restart: need to record upstream output status for each task (for prerequisites) in the DB, not just task status
    • ~~refine the cycle point based housekeeping to account for inter-cycle dependence~~ (no longer needed)
    • [x] unit and functional tests (many mods needed)
    • [x] implement reflow stop mechanisms as described in the proposal
    • [x] implement cylc spawn for specific individual outputs

    Requirements check-list

    • [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
    • [x] Contains logically grouped changes (else tidy your branch by rebase).
    • [x] Does not contain off-topic changes (use other PRs for other changes).
    • [x] Appropriate tests are included (unit and/or functional).
    • [x] Already covered by existing tests.
    • [x] Appropriate change log entry included.
    efficiency db change 
    opened by hjoliver 56
  • cylc monitor rewrite

    cylc monitor rewrite

    Closes #1685

    cylc-monitor-2

    Jet-lag washup - re-implement cylc monitor using urwid.

    • A direct replacement for the old cylc monitor.
      • Gets data direct from the suite.
        • Uses GraphQL so we can add the option to go via the UI Server later.
        • One second (non-adaptive) polling.
      • Performs global updating (there is scope for partial updating).
    • Follows the new Cylc UI - TreeView interface which is easily ported to a terminal interface.
    • Uses unicode to replicate task icons.
      • Note font size can be changed with ctrl +.

    What's supported:

    • The full Workflow-Cycle-Family-Task-Job tree.
    • A predefined set of job information.
    • Task and job icons (including 25,50,75% progress indicators).
    • Family grouping with consolidated task icons.

    What's not supported:

    • Multiple workflows (to be supported via GScan-esque panel at a future date).
    • Cycle consolidated task icons (computationally expensive).
    • Mutations (attempt after/with https://github.com/cylc/cylc-flow/issues/2123).
    • ctrl c exit, use q or ctrl d.

    TODO

    • [x] Fix tests which depend on the old cylc monitor.
    • [X] Unit test the new code where possible quite difficult.
    • [x] Write visual test via HTML translation.
    • [x] Add suite state summary to toolbar.
    • [x] Add task state filtering.
    • [ ] ~Investigate performance with larger suites.~ attempt after delta updates

    Requirements check-list

    • [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
    • [x] Contains logically grouped changes (else tidy your branch by rebase).
    • [x] Does not contain off-topic changes (use other PRs for other changes).
    • [x] Appropriate tests are included (unit and/or functional).
    • [x] Appropriate change log entry included.
    • [ ] (master branch) I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.
    opened by oliver-sanders 55
  • #1475 user auth (steps 1 & 2) - configurable public access

    #1475 user auth (steps 1 & 2) - configurable public access

    First part of #1475 - use a custom connection validator to authorize a client for free access to scan data (no passphrase required) or full control access (with the standard suite passphrase).

    For the next step (not in this pull request) this new connection validator can easily be made to work with a user+password file for control or read-only authorization of individual users, instead of a single passphrase required by all authorized users - we just need to decide on details (e.g. do we want to allow authorization to multiple suites at once...)

    As a side effect, this change greatly simplifies port scanning - no need to try all passphrases on each port (except in a much cleaned-up back compat mode for existing older suite daemons).

    Also, cylc scan can now see all suites (not just the user's) and I've allowed it to retrieve minimal task state totals as well as name and owner, which makes it a CLI version of cylc gsummary - hence this can close #578. Do we agree that this is an appropriate extension to cylc scan?

    [update: global config scan groups taken out in favour of simple suite name and owner options] [update: outdated image removed; see new version below] [update: gsummary comments shifted to a new issue #1488]

    Close #577

    opened by hjoliver 55
  • suite http(s) server improvements

    suite http(s) server improvements

    • [ ] Find a replacement for cherrypy:
      • See https://github.com/cylc/cylc/issues/2113#issuecomment-275082942 for reason. Situation does not appear to have improved.
      • Tornado looks like the way forward.
    • [ ] Support running multiple suites on a single http(s) server.E.g.: one server per user@host or even one server per host. (Imagine talking to your suites and launching new ones via a single URL in your browser.)
    • [ ] Authentication and authorization improvements. #1901
    superseded 
    opened by matthewrmshin 54
  • UIS Delta subscriptions & GraphQL null stripping back-end

    UIS Delta subscriptions & GraphQL null stripping back-end

    Closes #2684

    These changes partially address: https://github.com/cylc/cylc-admin/blob/master/docs/proposal-subscriptions.md As a back-end solution required to bring the proposal to fruition.

    This PR is the prerequisite to the UI Server implementation: https://github.com/cylc/cylc-uiserver/pull/118 (They should be merge together, as there is one change that is breaking to the BaseResolvers)

    Labeled POC, but is functional and probably good to go if deemed desirable.

    Features:

    • Deltas of a single main loop iteration grouped into a single message, and published as topic all. This grouping is somewhat artificial, but ensures that any related deltas (i.e. new edges and their nodes) do not arrive at the UIS (and hence WebUI) in separate pushes (helps WUI in reducing incomplete/frequency-of view updates).
    • The schema definition of the GraphQL delta subscription and resolving async generator. The resolver creates and monitors a queue, populated during delta application, and maps the Protobuf delta to the yielded GraphQL subscription ObjectType.
    • Bespoke GraphQL backend and middleware to set undesired/empty field values to null, and strip these out of the execution result.

    Future:

    • Workflow Scheduler ZMQ GraphQL subscriptions. As far as I know, there are no supported packages for ZMQ+GraphQL and the current REQ/RES GraphQL end-point is something I/we created for Cylc, So we need to write a module combining our currently implemented PUB/SUB and the execution of our schema, possibly in the same way as Tornado does with graphql-ws or likely something more bespoke.
    • Make strip_null/strip_values an argument of the query.

    Requirements check-list

    • [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
    • [x] Contains logically grouped changes.
    • [x] Does not contain off-topic changes (use other PRs for other changes).
    • [x] Already covered by or modifies existing tests (and/or belongs to future work).
    • [x] No change log entry required as invisible to users.
    • [x] No documentation update required.
    efficiency 
    opened by dwsutherland 53
  • Move the logic from Cylc commands/scripts to the cylc package

    Move the logic from Cylc commands/scripts to the cylc package

    For reference: python packaging - Command Line Scripts

    I have a branch with a work-in-progress setup.py for Cylc (if you have a look at the branch, skip the deleted files and don't mind the number of commits).

    In the setup.py, we have the new list of dependencies for Cylc, like isodatetime, and jinja2, matching versions required. And there is also an entry for scripts. The scripts option is configured to list all files under the bin/ folder.

    What it means, is that running pip install cylc (not exactly as we haven't published anything there... but just for example) would install the module cylc (i.e. contents of lib/cylc), its dependencies (e.g. versions needed for jinja2, and isodatetime), and also put the Cylc scripts in the $PATH.

    Right now users are asked to do that as a part of the installation process, but with this approach that won't be necessary anymore. The scripts would be installed with the new module in the user environment.

    Furthermore, users would be able to create virtualenv's, conda environments, or if the deb/rpm packaging works, they could use that option too. And that way they would be able to manage multiple versions of Cylc.

    It is also possible to achieve the same with containers, but containers are at a higher level. The option with pip requires simply a Python installation, pip, and virtualenv to support multiple versions (or in the supercomputer perhaps that load module trick...).

    One problem with this approach, however, is that it becomes clear that the scripts under the bin/ folder contain too much logic.

    This ticket suggests to move that logic to the Cylc module (i.e. under lib/cylc), within appropriate objects for encapsulation. Write tests (which is harder right now), and then later replace the scripts by the Python entry point: console_scripts.

    The console_scripts allow to expose functions as scripts.

    In other words, cylc scan would be a thin shim, calling actually a function like SomeCylcObject.scan(*args).

    The clear downside is the complexity and risk of this change.

    But here's a list of benefits:

    • We can test the scripts
    • IMO, it would be easier to maintain
    • Easier to expose, deprecate, replace scripts, as we would change simply the function exposed in the entry point
    • Users that did a import cylc in a Jupyter Notebook, for example, would be able to write code that calls that same function, without having to fork a process
    • The new Web GUI wouldn't have to fork processes (i.e. start a new process, allocate memory, file handlers, I/O, etc), it would simply import files from the syspath, within the Python world, and execute exactly what a user does with cylc scan or any other command (Probably not entirely true... we could still re-use the main() method in most cases?...)

    This can be done after the new Web GUI. I would prefer to try to implement this before, as I think the code would look simpler... but on the other hand, if we are not able to implement it right now, I could benchmark the current approach with the scripts, and compare against this approach and validate the last item.

    Cheers Bruno

    opened by kinow 53
  • cat-log: consider configuring the sleep interval for tail commands

    cat-log: consider configuring the sleep interval for tail commands

    The tail command takes a --sleep-interval option which allows for more efficient polling:

           -s, --sleep-interval=N
                  with  -f,  sleep  for  approximately N seconds (default 1.0) between
                  iterations; with inotify and --pid=P, check process P at least  once
                  every N seconds
    

    The Cylc default could be configured by the global.cylc file.

    Will need to check when this feature was introduced to ensure that this wouldn't break compatibility with current systems.

    Pull requests welcome!

    small 
    opened by oliver-sanders 0
  • Use scheduler default permissions for log retrieval

    Use scheduler default permissions for log retrieval

    The default for retrieve job logs command is currently rsync -a. -a implies -p which means that the files get the same permissions as on the remote host: https://explainshell.com/explain?cmd=rsync+-p This can cause problems. In my case, the default permissions on my scheduler host give world read access. However, my remote host is setup to give no access to world. As a result, when the log files are retrieved to the scheduler host they have no world access which is not what I expect on this host. Also, the way the rsync is constructed means that the permissions of the log/job directory get changed to match the remote host so world access to the entire job/log directory gets disabled after the first log retrieval. Our cylc review relies on world read access so it was very confused as to why I couldn't see any job logs (even for local jobs).

    In order for the retrieved log files to get the default permissions used on the scheduler host you can use: retrieve job logs command = rsync -a --no-p --no-g --chmod=ugo=rwX Should we set this as the default?

    question 
    opened by dpmatthews 3
  • Fix clock trigger execution retry delay bug

    Fix clock trigger execution retry delay bug

    Closes #5217

    Fix bug where old-style clock triggered tasks would skip execution retry delays and just retry immediately.

    Check List

    • [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
    • [x] Contains logically grouped changes (else tidy your branch by rebase).
    • [x] Does not contain off-topic changes (use other PRs for other changes).
    • [x] No dependency changes
    • [x] Tests are included
    • [x] CHANGES.md entry included if this is a change that can affect users
    • [x] No docs needed
    bug 
    opened by MetRonnie 1
  • clock-triggers: re-implement as xtriggers

    clock-triggers: re-implement as xtriggers

    Cylc currently provides two mechanisms for clock-limiting tasks:

    • [xtriggers]wall_clock The new approach, the @clock trigger is used in the graph like regular dependencies. This has the advantage of transparency.
    • [special tasks]clock-trigger The old approach. marked as deprecated but still widely used. Allows listing of tasks which should be clock-limited.

    We will presumably look to retire the old clock-triggers at some point after Cylc 7 back-compat mode is dropped, but this is still a way off. This means we need to keep these triggers working for the near future (which is something else to go wrong #5217).

    Suggest converting clock-triggers to xtriggers at configure time. I think we can do this fairly easily at configure time something along the lines of:

    # config.py
    def convert_clock_triggers(self):
        for string in self.cfg['scheduling']['special tasks']['clock-trigger']:
            task_name, offset  = do_something(string)
            xtrig_name = f'_cylc_clock_trigger_{task_name}'
            self.cfg['scheduling']['xtriggers'][xtrig_name] = f'wall_clock({offset})'
            taskdef = self.taskdefs[task_name]
            taskdef.add_xtrig_label(xtrig_name)
    

    This way the old triggers will continue working under the new logic allowing us to retire this feature early. The old triggers will also become visible in the UI once xtrigger support is added.

    Pull requests welcome!

    opened by oliver-sanders 1
  • Update to Isodatetime with cached methods

    Update to Isodatetime with cached methods

    Describe exactly what you would like to see in an upcoming release

    • Cache method calls in Isodatetime - https://github.com/metomi/isodatetime/issues/176
    • Update the dependency pin here once that's released
    efficiency 
    opened by MetRonnie 0
  • poll/kill: don't retry on NoHosts error

    poll/kill: don't retry on NoHosts error

    • Closes #5276
    • Fixes a bug where poll/kill commands were called in an infinite-recursion loop if a NoHostsError exception was raised for the platform.
    • Bug introduced by this diff https://github.com/cylc/cylc-flow/commit/dc15ce4b89f801ee472e837cb532be5c6e3c50b6#diff-0b970d88e6ba8fe34ed659c20689046b5366912171e5157932ad5175c0468460R944-R960
      • Before this diff (8.0.2) the NoHostsError was raised and the scheduler shut down.
      • After this diff (8.0.3) the poll/kill would be retried until Python's recursion limit caused an error to be raised and the scheduler to shut down.
    • This diff reverts to NOT retrying the poll/kill on another host if all hosts in the platform are un-available.

    Aside:

    This makes me more convinced that #5017 is necessary and important. The call-callback code is extremely difficult to follow. This has become a larger problem now that we have several layers of callback code (remote-init, remote-file-install, job-submit and a regular and 255 callbacks for everything).

    Check List

    • [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
    • [x] Contains logically grouped changes (else tidy your branch by rebase).
    • [x] Does not contain off-topic changes (use other PRs for other changes).
    • [x] Applied any dependency changes to both setup.cfg and conda-environment.yml.
    • [ ] Tests are included (or explain why tests are not needed).
    • [ ] CHANGES.md entry included if this is a change that can affect users
    • [x] Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
    • [x] If this is a bug fix, PR should be raised against the relevant ?.?.x branch.
    bug 
    opened by oliver-sanders 0
Releases(8.0.4)
  • 8.0.4(Dec 14, 2022)

  • 8.0.3(Oct 17, 2022)

  • 7.9.7(Sep 30, 2022)

    Final 7.8 maintenance release, for cylc review compatibility with Cylc 8.

    See CHANGES.md for details.

    NOTE: Use Python 3 based Cylc 8 if possible.

    This is cylc-7.8.12 with the bundled Jinja2 upgraded to version 2.11.1. If stuck with Python 2.7+, use 7.9.7. If stuck with Python 2.6, use 7.8.12.

    Source code(tar.gz)
    Source code(zip)
  • 7.8.12(Sep 30, 2022)

    Final 7.8 maintenance release, for cylc review compatibility with Cylc 8.

    See CHANGES.md for details.

    NOTE: Use Python 3 based Cylc 8 if possible.

    This is cylc-7.9.7 bundled with an older version of Jinja2. If stuck with Python 2.7+, use 7.9.7. If stuck with Python 2.6, use 7.8.12.

    Source code(tar.gz)
    Source code(zip)
  • 8.0.2(Sep 12, 2022)

  • 8.0.1(Aug 16, 2022)

  • 8.0.0(Jul 28, 2022)

  • 8.0rc3(May 19, 2022)

  • 8.0rc2(Mar 23, 2022)

  • 7.9.6(Mar 7, 2022)

    Another final Cylc 7 release. Contains one bug fix for task messaging, and several more cylc review changes for Cylc 8 compatibility.

    See CHANGES.md for details.

    This is cylc-7.8.11 bundled with a newer version of Jinja2. Use cylc-7.9.6 unless you're stuck with Python 2.6. NOTE: Python 3 based Cylc 8 releases are now available for testing.

    Source code(tar.gz)
    Source code(zip)
  • 7.8.11(Mar 7, 2022)

    Another final Cylc 7 release. Contains one bug fix for task messaging, and several more cylc review changes for Cylc 8 compatibility.

    See CHANGES.md for details.

    This is cylc-7.9.6 bundled with an older version of Jinja2. Use cylc-7.9.6 unless you're stuck with Python 2.6. NOTE: Python 3 based Cylc 8 releases are now available for testing.

    Source code(tar.gz)
    Source code(zip)
  • 8.0rc1(Feb 17, 2022)

    See CHANGES.md for detail.

    Cylc 8 can be installed via pypi or Conda - you don't need to download this release directly. See the installation section of the documentation.

    Note: If using pip to install Cylc versions 8.0rc1 and below, a bug in Jinja2 means you will have to manually install markupsafe at less than version 2.1:

    pip install 'markupsafe<2.1' cylc-flow
    
    Source code(tar.gz)
    Source code(zip)
  • 8.0b3(Nov 10, 2021)

  • 7.9.5(Sep 30, 2021)

    Final Cylc 7 release. Fixes a bug where job polling could in rare circumstances misidentify a finished job as submit-failed, and allow cylc review to see Cylc 8 log files with changed names. (Cylc 7's cylc review will be needed for some time with Cylc 8).

    See CHANGES.md for details.

    This is cylc-7.8.10 bundled with a newer version of Jinja2. Use cylc-7.9.5 unless you're stuck with Python 2.6.

    Source code(tar.gz)
    Source code(zip)
  • 7.8.10(Sep 30, 2021)

    Final Cylc 7 release. Fixes a bug where job polling could in rare circumstances misidentify a finished job as submit-failed, and allow cylc review to see Cylc 8 log files with changed names. (Cylc 7's cylc review will be needed for some time with Cylc 8).

    See CHANGES.md for details.

    This is cylc-7.9.5 bundled with an older version of Jinja2. Use cylc-7.9.5 unless you're stuck with Python 2.6.

    Source code(tar.gz)
    Source code(zip)
  • 7.9.4(Sep 23, 2021)

    Almost-final Cylc 7 release containing several minor bug fixes, and enhanced to prevent Cylc 7 messing with Cylc 8 workflows for those who need both installed at once for a while. (And note the Cylc 7 cylc review will be needed for some time with Cylc 8).

    See CHANGES.md for details.

    This is cylc-7.8.9 bundled with the newer 2.11.1 version of Jinja2., which allows us to correctly report the line number (context) for all Jinja2 errors. cylc-7.9.x requires Python 2.7, whereas cylc-7.8.x can be used with Python 2.6 or 2.7.

    Source code(tar.gz)
    Source code(zip)
  • 7.8.9(Sep 23, 2021)

    Almost-final Cylc 7 release containing several minor bug fixes, and enhanced to prevent Cylc 7 messing with Cylc 8 workflows for those who need both installed at once for a while. (And note the Cylc 7 cylc review will be needed for some time with Cylc 8).

    See CHANGES.md for details.

    This is cylc-7.9.4 bundled with an older version of Jinja2. Use cylc-7.9.4 unless you're stuck with Python 2.6.

    Source code(tar.gz)
    Source code(zip)
  • 8.0b2(Jul 28, 2021)

  • 8.0b1(Apr 21, 2021)

  • 8.0b0(Mar 28, 2021)

  • 7.9.3(Mar 26, 2021)

    Cylc 7 maintenance release: allow "cylc review" to see Cylc 8 workflows.

    This is just cylc-7.8.8 with the bundled Jinja2 version upgraded to 2.11.1, which allows us to correctly report the line number (context) for all Jinja2 errors. Cylc-7.9.x requires Python 2.7, whereas cylc-7.8.x can be used with Python 2.6 or 2.7.

    Source code(tar.gz)
    Source code(zip)
  • 7.8.8(Mar 26, 2021)

    Cylc 7 maintenance release: allow "cylc review" to see Cylc 8 workflows.

    This is cylc-7.9.3 bundled with an older version of Jinja2. Please use cylc-7.9.3 unless you need Python 2.6

    Source code(tar.gz)
    Source code(zip)
  • 7.9.2(Dec 4, 2020)

    cylc-7.9.2 maintenance release.

    This is just cylc-7.8.7 with the bundled Jinja2 version upgraded to 2.11.1, which allows us to correctly report the line number (context) for all Jinja2 errors. Cylc-7.9.x requires Python 2.7, whereas cylc-7.8.x can be used with Python 2.6 or 2.7.

    Selected changes:

    • support Slurm heterogeneous jobs
    • fix frequent writes to a DB table, to correct last-updated timestamps in cylc review and reduce filesystem load
    • fix a bug in auto-restart (scheduler migration) which caused suites to wait unnecessarily for local jobs on other hosts before restarting
    • fix a bug in the GUI tree view that could cause tasks to be sorted in the wrong order

    Change log

    Source code(tar.gz)
    Source code(zip)
  • 7.8.7(Dec 4, 2020)

    cylc-7.8.7 maintenance release.

    Selected changes:

    • support Slurm heterogeneous jobs
    • fix frequent writes to a DB table, to correct last-updated timestamps in cylc review and reduce filesystem load
    • fix a bug in auto-restart (scheduler migration) which caused suites to wait unnecessarily for local jobs on other hosts before restarting
    • fix a bug in the GUI tree view that could cause tasks to be sorted in the wrong order

    Change log

    Source code(tar.gz)
    Source code(zip)
  • 8.0a2(Jul 3, 2020)

  • 7.9.1(May 14, 2020)

    The cylc-7.9.1 release is just cylc-7.8.6 with the bundled Jinja2 version upgraded to 2.11.1, which allows us to correctly report the line number (context) for all Jinja2 errors. Cylc-7.9.x requires Python 2.7, whereas cylc-7.8.x can be used with Python 2.6 or 2.7.

    See CHANGES.md for details.

    Source code(tar.gz)
    Source code(zip)
  • 7.8.6(May 14, 2020)

    cylc-7.8.6 maintenance release:

    • Fix a bug that could prevent housekeeping of the task_action_timers DB table and cause many warnings at restart.
    • Fix a bug that prevented cycle point format conversion by the cylc suite-state command and the suite_state xtrigger function, if the target suite used the default format but downstream command or suite did not.

    See CHANGES.md for details.

    Source code(tar.gz)
    Source code(zip)
  • 7.9.0(Apr 24, 2020)

    The cylc-7.9.0 release is just cylc-7.8.5 with the bundled Jinja2 version upgraded to 2.11.1, which allows us to correctly report the line number (context) for all Jinja2 errors. Cylc-7.9.0 requires Python 2.7, whereas cylc-7.8.5 can be used with Python 2.6 or 2.7.

    See CHANGES.md for details.

    Source code(tar.gz)
    Source code(zip)
  • 7.8.5(Apr 22, 2020)

    cylc-7.8.5 maintenance release with several fixes:

    • don't expand all suites in cylc gscan if one is expanded
    • use natural sort order for integer cycles in the cylc gui tree view
    • fix job submission to SLURM when task name has a percent % character
    • don't warn that a task was re-added to the same internal queue
    • fix list-valued global config items not defaulting to localhost values.

    See CHANGES.md for details.

    Source code(tar.gz)
    Source code(zip)
  • 8.0a1(Sep 19, 2019)

    Cylc 8 Alpha-1 Preview Release

    cylc-flow-8.0a1; companion of:

    • cylc-uiserver 0.1
    • cylc-ui 0.1
    • cylc-conda 8.0a1

    Cylc-8 is installed by conda - you don't need to download this cylc-flow release directly. https://github.com/cylc/cylc-conda#cylc-8-conda-installation-instructions

    Source code(tar.gz)
    Source code(zip)
Owner
The Cylc Workflow Engine
A workflow engine for cycling systems.
The Cylc Workflow Engine
PyWorkflow(PyWF) - A Python Binding of C++ Workflow

PyWorkflow(PyWF) - A Python Binding of C++ Workflow 概览 C++ Workflow是一个高性能的异步引擎,本项目着力于实现一个Python版的Workflow,让Python用户也能享受Workflow带来的绝佳体验。

Sogou-inc 108 Dec 1, 2022
LOC-FLOW is an “hands-free” earthquake location workflow to process continuous seismic records

LOC-FLOW is an “hands-free” earthquake location workflow to process continuous seismic records: from raw waveforms to well located earthquakes with magnitude calculations. The package assembles several popular routines for sequential earthquake location refinements, suitable for catalog building ranging from local to regional scales.

Miao Zhang 71 Jan 9, 2023
A frontend to ease the use of pulseaudio's routing capabilities, mimicking voicemeeter's workflow

Pulsemeeter A frontend to ease the use of pulseaudio's routing capabilities, mimicking voicemeeter's workflow Features Create virtual inputs and outpu

Gabriel Carneiro 164 Jan 4, 2023
A python based app to improve your presentation workflow

Presentation Remote A remote made for making presentations easier by enabling all the members to have access to change the slide and control the flow

Parnav 1 Oct 28, 2021
A Snakemake workflow for standardised sc/snRNAseq analysis

single_snake_sequencing - sc/snRNAseq Snakemake Workflow A Snakemake workflow for standardised sc/snRNAseq analysis. Every single cell analysis is sli

IMS Bio2Core Facility 1 Nov 2, 2021
redun aims to be a more expressive and efficient workflow framework

redun yet another redundant workflow engine redun aims to be a more expressive and efficient workflow framework, built on top of the popular Python pr

insitro 372 Jan 4, 2023
A basic animation modding workflow for FFXIV

AnimAssist Provides a quick and easy way to mod animations in FFXIV. You will need: Before anything, the VC++2012 32-bit Redist from here. Havok will

liam 37 Dec 16, 2022
A cookiecutter to start a Python package with flawless practices and a magical workflow 🧙🏼‍♂️

PyPackage Cookiecutter This repository is a cookiecutter to quickly start a Python package. It contains a ton of very useful features ?? : Package man

Daniel Leal 16 Dec 13, 2021
This is a fork of the BakeTool with some improvements that I did to have better workflow.

blender-bake-tool This is a fork of the BakeTool with some improvements that I did to have better workflow. 99.99% of work was done by BakeTool team.

Acvarium 3 Oct 4, 2022
Alfred 4 Workflow to search through your maintained/watched/starred GitHub repositories.

Alfred 4 Workflow to search through your maintained/watched/starred GitHub repositories. Setup This workflow requires a number of Python modules. Thes

Bᴇʀɴᴅ Sᴄʜᴏʀɢᴇʀs 1 Oct 14, 2022
Structural basis for solubility in protein expression systems

Structural basis for solubility in protein expression systems Large-scale protein production for biotechnology and biopharmaceutical applications rely

ProteinQure 16 Aug 18, 2022
LSO, also known as Linux Swap Operator, is a software with both GUI and terminal versions that you can manage the Swap area for Linux operating systems.

LSO - Linux Swap Operator Türkçe - LSO Nedir? LSO, diğer adıyla Linux Swap Operator Linux işletim sistemleri için Swap alanını yönetebileceğiniz hem G

Eren İnce 4 Feb 9, 2022
ASVspoof 2021 Baseline Systems

ASVspoof 2021 Baseline Systems Baseline systems are grouped by task: Speech Deepfake (DF) Logical Access (LA) Physical Access (PA) Please find more de

null 91 Dec 28, 2022
Control System Packer is a lightweight, low-level program to transform energy equations into the compact libraries for control systems.

Control System Packer is a lightweight, low-level program to transform energy equations into the compact libraries for control systems. Packer supports Python ?? , C ?? and C++ ?? libraries.

mirnanoukari 31 Sep 15, 2022
Test reproducibility of leiden/umap on different systems

Demonstrate that UMAP and Leiden analysis is not reproducible between different cpu architectures.

Gregor Sturm 2 Oct 16, 2021
EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

EasyBuild community 87 Dec 27, 2022
Intelligent Systems Project In Python

Intelligent Systems Project In Python

RLLAB 3 May 16, 2022
Using Python to parse through email logs received through several backup systems.

outlook-automated-backup-control Backup monitoring on a mailbox: In this mailbox there will be backup logs. The identification will based on the follo

Connor 2 Sep 28, 2022
Demo code for "Logs in distributed systems" webinar

Hexlet Logs Demo Пререквизиты docker-compose python3 Учетка в DataDog Базовое понимание, что такое логи (можно почитать гайд

Anton Markelov 1 Dec 1, 2021