Cylc: a workflow engine for cycling systems

This is a completely general notation for specifying date/times and time ranges. It would presumably allow us to replace all the current cycling modules (daily, monthly, etc.) with a single one that could handle any cycle time sequence. And cycling for paleo-climate simulations?

MPI is developing an ISO time library for Python.

Related Issues:

If agreed this issue should supersede:

• https://github.com/cylc/cylc-flow/issues/4657
• https://github.com/cylc/cylc-flow/issues/4653

After a long chat with @dpmatthews (who proposed yet another triggering approach 😁) I think we can generalise the trigger problem into two dimensions:

• Continue (yes/no).
  • After I trigger the task will the flow continue from that point immediately.
  • Or does it only continue if/when a flow front catches up with it.
  • I.E. Should the triggered tasks spawn children on completion or after "merge".
• Overrun (yes/no).
  • Should the "merge" [1] condition be based on the pool or the DB?
  • I.E. Should triggered tasks overrun previous runs of tasks?
  • I.E. Should the following flow overrun the triggered tasks?

Note: From the internal implementation these two dimensions may appear flip-sides of the same coin since they both boil down to the flow_nums, however, considering them from a user standpoint I think it's fair to prise them apart.

Note: Purposefully using new terminology to avoid conflation with existing terms, we may want to workshop "continue" and "overrun" a touch.

[1]: The quoted "merge" above relates to the interaction between two tasks with different flow_nums in general and not to the more specific concept of "flow merging" in the pool exclusively.

Combing these we get four spaces:

| | Continue | Don't Continue | |------------|-----------------------|---------------------------------------------------| | Overrun | (1) Reflow (as currently implemented) | (3) No Flow (current default trigger behaviour) | | No Overrun | (2) Continue (@dpmatthews new proposed implementation) | (4) No Flow (@oliver-sanders proposed implementation) |

• The bad news is it looks like we have use cases for all four.
• Dave & I think the no-overrun cases are more important than the overrun ones.
• The good news is that they can coexist and the mechanism for supporting all four is currently implemented, it's mostly an interface problem.

Going through the four spaces in detail:

1) Reflow (implemented)

Equivalent to cylc trigger --flow=<new-flow-number>.

Continue: Yes Overrun: Yes

• Tasks are triggered with a new flow number.
• The reflow can overrun previous flows.
• The reflow will merge if it collides with another flow in the pool (and only in the pool i.e. overrun).

The use case is for re-running over tasks which have been previously run e.g. change configuration and re-run a sub-graph.

2) Continue (proposed)

Equivalent to cylc trigger --flow=<all-flow-numbers>,<new-flow-number>.

Continue: Yes Overrun: No

• A new trigger approach proposed by @dpmatthews.
• Tasks are triggered with all existing flow numbers plus a new flow number (which we added purely so the new flow can still be targeted by CLI tools).
• Because this flow contains all existing flow numbers it will not be overrun by any of the flows which exist at the time of the trigger.
• This is intended for the sort of use cases we would expect --flow=1 to be used for, but has been generalised to be reflow compatible.

This approach feels quite "natural". The use cases are setting off another bit of the same flow where you don't want tasks to be overrun.

3) No Flow (implemented)

Equivalent to cylc trigger --flow -1.

I am using a negative flow number rather than None to distinguish the two no-flow approaches. Internally we can still maintain the same no-flow logic as present but would need to change the marker.

Continue: No Overrun: Yes

Useful for running one-off tasks that you do not want to impact the workflow in any way (i.e. cylc submit type uses).

4) No Flow (proposed)

Equivalent to cylc trigger --flow -2.

I am using a negative flow number rather than None to distinguish the two no-flow approaches. Internally we can still maintain the same no-flow logic as present.

Continue: No Overrun: No

Use case is for manually intervening in graph execution by ignoring dependencies or runahead limit and skipping ahead to a task which you want to be considered a part of the approaching flow front.

Interface

The internals to handle the four cases are already in-place, flow_nums, DB lookups etc, so it mostly boils down to an interface / documentation issue.

I think all four methods could be exposed via a single --flow argument, however, it is sensible to provide defaults for the different behaviours. I think it would be good to document the --flow equivalents as they may help users to understand their function.

Note that --reflow currently determines the new flow number server rather than client side which is sensible.

1) Enable behaviours explicitly

If we are happy with the continue/overrun model (after workshopping the terms) we could expose it directly something like:

# 1) reflow
cylc trigger --continue --overrun

# 2) continue
cylc trigger --continue

# 3) no-flow (implemented)
cylc trigger --overrun

# 4) no-flow (proposed)
cylc trigger

This is quite nice as you have to explicitly opt in to each behaviour separately reducing the scope for unintended results and accidents.

2) Single --flow argument

if we don't like the continue/overrun model we could move the presets into the flow argument something like:

# 1) reflow
cylc trigger --flow=new

# 2) continue
cylc trigger --flow=any

# 3) no-flow (implemented)
cylc trigger --flow=none

# 4) no-flow (proposed)
cylc trigger --flow=next

It's less behaviour driven so we would need to explain each option separately.

3) Separate flag for each approach

An alternative to (2) would be to could come up with three/four different flags:

# 1) reflow
cylc trigger --reflow

# 2) continue
cylc trigger --flow

# 3) no-flow (implemented)
cylc trigger --rerun

# 4) no-flow (proposed)
cylc trigger  # --run

Default

I think no-continue & no-overrun is the safest, sanest default because:

• The minimum set of behaviours is the simplest.
• The "Continue" cases have a dramatic impact on the workflow execution and are hard to revoke.
• The "Re-run" cases are quite advanced and require additional knowledge to operate.

But I'm biased. I think the default is less important than the clear separation of behaviours.

These changes close #2608

Requirements check-list

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] Appropriate tests are included (unit and/or functional).
• [x] Appropriate change log entry included.
• [x] No documentation required.

Travis-CI configuration was updated to use build stages. The build matrix was moved to the test stage, and extra configuration for coverage.py and codacy were added.

The tool used to collect coverage from the tests in Coverage.py, which is supported by multiple build and reporting systems.

Due to the nature of the tests in cylc, and to how they run in parallel in different containers in Travis-CI, we had to use a sitecustomize.py script.

As Codacy was already used for code analysis, this pull request includes settings to export the coverage report to Codacy. I assume the Codacy token is already set up. So maybe just merging the pull request would already make the coverage report available in the cylc dashboard.

It is also possible to get the coverage of pull requests.

Cheers Bruno

The current data-store generation is inefficient (as was the old state_summary_mgr.py), in that it generates an entirely new data-store on every loop where any change has flagged... The update/creation has been granulated into the following actions/methods:

1. Creation of workflow/taskdef/familydef data elements.
2. Generation of graph edges and ghost-nodes/ghost-instances/ghost-proxies (containing def data and tree relationships) for each new cycle point in the pool.
3. Update/population of [ghost] task instance fields with dynamic data (i.e. status info) for given pool tasks.
4. Update/population of [ghost] family instance fields with dynamic data (i.e. status info) according to given cycle points (i.e. those cycle points of the updated tasks).
5. Update/population of workflow (data element) fields with dynamic data (i.e. status info).
6. Prune data-store of elements (edges/nodes ...etc) and items in fields on or in reference to given list of point_strings.

These are used within the scheduler in implementation with:

• Initiation of data-store on run/restart/reload using (1)-(5)
• Increment of graph elements and prune old elements using (2), (5), and (6) worked out from the head and tail of the task pool respectively (workflow updated after with new state totals etc).
• Populate/Update of dynamic fields of task/family instances and then workflow element ((3), (4), then (5)), from tasks flagged as updated by the workflow.

There was a bug in the job pool preventing job element deletion, this has also been fixed.

This change has the following impacts:

• Significant reduction of load on the workflow, mainly from not having to generate entire graph on every change (also that job deletion bug).
• The data-store is used both in the WS and UI Server as driving data for GraphQL, so this change sets up the ground work for implementing an incremental data sync between WS & UIS data store.

Closes https://github.com/cylc/cylc-flow/issues/3261 Closes https://github.com/cylc/cylc-flow/issues/3212

Requirements check-list

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] Appropriate tests are included (unit and/or functional).
• [x] No change log entry required (e.g. change is small or internal only).
• [x] No documentation update required for this change.

Spawn on Demand implementation as per proposal https://cylc.github.io/cylc-admin/proposal-spawn-on-d.html

See #3474 (superseded) for earlier discussion and some proof of efficiency.

• partially addresses #3304
• close #774
• close #2143
• close #3226
• (probably closes other issues too...):
• close #987
• close #2643
• close #1314
• close #2654

This branch:

• correctly runs the "complex" test workflow for multiple cycles (last I checked, recently)
• handles reflow properly as described in the proposal
• seems to work for all manner of manual test cases

What's still to do:

• [x] restart: need to record upstream output status for each task (for prerequisites) in the DB, not just task status
• ~~refine the cycle point based housekeeping to account for inter-cycle dependence~~ (no longer needed)
• [x] unit and functional tests (many mods needed)
• [x] implement reflow stop mechanisms as described in the proposal
• [x] implement cylc spawn for specific individual outputs

Requirements check-list

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).

• [x] Appropriate tests are included (unit and/or functional).
• [x] Already covered by existing tests.

• [x] Appropriate change log entry included.

• [x] (master branch) I have opened a documentation Issue at cylc/cylc-doc#132

Closes #1685

Jet-lag washup - re-implement cylc monitor using urwid.

• A direct replacement for the old cylc monitor.
  • Gets data direct from the suite.
    • Uses GraphQL so we can add the option to go via the UI Server later.
    • One second (non-adaptive) polling.
  • Performs global updating (there is scope for partial updating).
• Follows the new Cylc UI - TreeView interface which is easily ported to a terminal interface.
• Uses unicode to replicate task icons.
  • Note font size can be changed with ctrl +.

What's supported:

• The full Workflow-Cycle-Family-Task-Job tree.
• A predefined set of job information.
• Task and job icons (including 25,50,75% progress indicators).
• Family grouping with consolidated task icons.

What's not supported:

• Multiple workflows (to be supported via GScan-esque panel at a future date).
• Cycle consolidated task icons (computationally expensive).
• Mutations (attempt after/with https://github.com/cylc/cylc-flow/issues/2123).
• ctrl c exit, use q or ctrl d.

TODO

• [x] Fix tests which depend on the old cylc monitor.
• [X] Unit test the new code where possible quite difficult.
• [x] Write visual test via HTML translation.
• [x] Add suite state summary to toolbar.
• [x] Add task state filtering.
• [ ] ~Investigate performance with larger suites.~ attempt after delta updates

Requirements check-list

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] Appropriate tests are included (unit and/or functional).
• [x] Appropriate change log entry included.
• [ ] (master branch) I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.

First part of #1475 - use a custom connection validator to authorize a client for free access to scan data (no passphrase required) or full control access (with the standard suite passphrase).

For the next step (not in this pull request) this new connection validator can easily be made to work with a user+password file for control or read-only authorization of individual users, instead of a single passphrase required by all authorized users - we just need to decide on details (e.g. do we want to allow authorization to multiple suites at once...)

As a side effect, this change greatly simplifies port scanning - no need to try all passphrases on each port (except in a much cleaned-up back compat mode for existing older suite daemons).

Also, cylc scan can now see all suites (not just the user's) and I've allowed it to retrieve minimal task state totals as well as name and owner, which makes it a CLI version of cylc gsummary - hence this can close #578. Do we agree that this is an appropriate extension to cylc scan?

[update: global config scan groups taken out in favour of simple suite name and owner options] [update: outdated image removed; see new version below] [update: gsummary comments shifted to a new issue #1488]

Close #577

• [ ] Find a replacement for cherrypy:
  • See https://github.com/cylc/cylc/issues/2113#issuecomment-275082942 for reason. Situation does not appear to have improved.
  • Tornado looks like the way forward.
• [ ] Support running multiple suites on a single http(s) server.E.g.: one server per user@host or even one server per host. (Imagine talking to your suites and launching new ones via a single URL in your browser.)
• [ ] Authentication and authorization improvements. #1901

Closes #2684

These changes partially address: https://github.com/cylc/cylc-admin/blob/master/docs/proposal-subscriptions.md As a back-end solution required to bring the proposal to fruition.

This PR is the prerequisite to the UI Server implementation: https://github.com/cylc/cylc-uiserver/pull/118 (They should be merge together, as there is one change that is breaking to the BaseResolvers)

Labeled POC, but is functional and probably good to go if deemed desirable.

Features:

• Deltas of a single main loop iteration grouped into a single message, and published as topic all. This grouping is somewhat artificial, but ensures that any related deltas (i.e. new edges and their nodes) do not arrive at the UIS (and hence WebUI) in separate pushes (helps WUI in reducing incomplete/frequency-of view updates).
• The schema definition of the GraphQL delta subscription and resolving async generator. The resolver creates and monitors a queue, populated during delta application, and maps the Protobuf delta to the yielded GraphQL subscription ObjectType.
• Bespoke GraphQL backend and middleware to set undesired/empty field values to null, and strip these out of the execution result.

Future:

• Workflow Scheduler ZMQ GraphQL subscriptions. As far as I know, there are no supported packages for ZMQ+GraphQL and the current REQ/RES GraphQL end-point is something I/we created for Cylc, So we need to write a module combining our currently implemented PUB/SUB and the execution of our schema, possibly in the same way as Tornado does with graphql-ws or likely something more bespoke.
• Make strip_null/strip_values an argument of the query.

Requirements check-list

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes.
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] Already covered by or modifies existing tests (and/or belongs to future work).
• [x] No change log entry required as invisible to users.
• [x] No documentation update required.

For reference: python packaging - Command Line Scripts

I have a branch with a work-in-progress setup.py for Cylc (if you have a look at the branch, skip the deleted files and don't mind the number of commits).

In the setup.py, we have the new list of dependencies for Cylc, like isodatetime, and jinja2, matching versions required. And there is also an entry for scripts. The scripts option is configured to list all files under the bin/ folder.

What it means, is that running pip install cylc (not exactly as we haven't published anything there... but just for example) would install the module cylc (i.e. contents of lib/cylc), its dependencies (e.g. versions needed for jinja2, and isodatetime), and also put the Cylc scripts in the $PATH.

Right now users are asked to do that as a part of the installation process, but with this approach that won't be necessary anymore. The scripts would be installed with the new module in the user environment.

Furthermore, users would be able to create virtualenv's, conda environments, or if the deb/rpm packaging works, they could use that option too. And that way they would be able to manage multiple versions of Cylc.

It is also possible to achieve the same with containers, but containers are at a higher level. The option with pip requires simply a Python installation, pip, and virtualenv to support multiple versions (or in the supercomputer perhaps that load module trick...).

One problem with this approach, however, is that it becomes clear that the scripts under the bin/ folder contain too much logic.

This ticket suggests to move that logic to the Cylc module (i.e. under lib/cylc), within appropriate objects for encapsulation. Write tests (which is harder right now), and then later replace the scripts by the Python entry point: console_scripts.

The console_scripts allow to expose functions as scripts.

In other words, cylc scan would be a thin shim, calling actually a function like SomeCylcObject.scan(*args).

The clear downside is the complexity and risk of this change.

But here's a list of benefits:

• We can test the scripts
• IMO, it would be easier to maintain
• Easier to expose, deprecate, replace scripts, as we would change simply the function exposed in the entry point
• Users that did a import cylc in a Jupyter Notebook, for example, would be able to write code that calls that same function, without having to fork a process
• The new Web GUI wouldn't have to fork processes (i.e. start a new process, allocate memory, file handlers, I/O, etc), it would simply import files from the syspath, within the Python world, and execute exactly what a user does with cylc scan or any other command (Probably not entirely true... we could still re-use the main() method in most cases?...)

This can be done after the new Web GUI. I would prefer to try to implement this before, as I think the code would look simpler... but on the other hand, if we are not able to implement it right now, I could benchmark the current approach with the scripts, and compare against this approach and validate the last item.

Cheers Bruno

The tail command takes a --sleep-interval option which allows for more efficient polling:

       -s, --sleep-interval=N
              with  -f,  sleep  for  approximately N seconds (default 1.0) between
              iterations; with inotify and --pid=P, check process P at least  once
              every N seconds

The Cylc default could be configured by the global.cylc file.

Will need to check when this feature was introduced to ensure that this wouldn't break compatibility with current systems.

Pull requests welcome!

The default for retrieve job logs command is currently rsync -a. -a implies -p which means that the files get the same permissions as on the remote host: https://explainshell.com/explain?cmd=rsync+-p This can cause problems. In my case, the default permissions on my scheduler host give world read access. However, my remote host is setup to give no access to world. As a result, when the log files are retrieved to the scheduler host they have no world access which is not what I expect on this host. Also, the way the rsync is constructed means that the permissions of the log/job directory get changed to match the remote host so world access to the entire job/log directory gets disabled after the first log retrieval. Our cylc review relies on world read access so it was very confused as to why I couldn't see any job logs (even for local jobs).

In order for the retrieved log files to get the default permissions used on the scheduler host you can use: retrieve job logs command = rsync -a --no-p --no-g --chmod=ugo=rwX Should we set this as the default?

Closes #5217

Fix bug where old-style clock triggered tasks would skip execution retry delays and just retry immediately.

Check List

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] No dependency changes
• [x] Tests are included
• [x] CHANGES.md entry included if this is a change that can affect users
• [x] No docs needed

Cylc currently provides two mechanisms for clock-limiting tasks:

• [xtriggers]wall_clock The new approach, the @clock trigger is used in the graph like regular dependencies. This has the advantage of transparency.
• [special tasks]clock-trigger The old approach. marked as deprecated but still widely used. Allows listing of tasks which should be clock-limited.

We will presumably look to retire the old clock-triggers at some point after Cylc 7 back-compat mode is dropped, but this is still a way off. This means we need to keep these triggers working for the near future (which is something else to go wrong #5217).

Suggest converting clock-triggers to xtriggers at configure time. I think we can do this fairly easily at configure time something along the lines of:

# config.py
def convert_clock_triggers(self):
    for string in self.cfg['scheduling']['special tasks']['clock-trigger']:
        task_name, offset  = do_something(string)
        xtrig_name = f'_cylc_clock_trigger_{task_name}'
        self.cfg['scheduling']['xtriggers'][xtrig_name] = f'wall_clock({offset})'
        taskdef = self.taskdefs[task_name]
        taskdef.add_xtrig_label(xtrig_name)

This way the old triggers will continue working under the new logic allowing us to retire this feature early. The old triggers will also become visible in the UI once xtrigger support is added.

Pull requests welcome!

Describe exactly what you would like to see in an upcoming release

• Cache method calls in Isodatetime - https://github.com/metomi/isodatetime/issues/176
• Update the dependency pin here once that's released

• Closes #5276
• Fixes a bug where poll/kill commands were called in an infinite-recursion loop if a NoHostsError exception was raised for the platform.
• Bug introduced by this diff https://github.com/cylc/cylc-flow/commit/dc15ce4b89f801ee472e837cb532be5c6e3c50b6#diff-0b970d88e6ba8fe34ed659c20689046b5366912171e5157932ad5175c0468460R944-R960
  • Before this diff (8.0.2) the NoHostsError was raised and the scheduler shut down.
  • After this diff (8.0.3) the poll/kill would be retried until Python's recursion limit caused an error to be raised and the scheduler to shut down.
• This diff reverts to NOT retrying the poll/kill on another host if all hosts in the platform are un-available.

Aside:

This makes me more convinced that #5017 is necessary and important. The call-callback code is extremely difficult to follow. This has become a larger problem now that we have several layers of callback code (remote-init, remote-file-install, job-submit and a regular and 255 callbacks for everything).

Check List

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] Applied any dependency changes to both setup.cfg and conda-environment.yml.
• [ ] Tests are included (or explain why tests are not needed).
• [ ] CHANGES.md entry included if this is a change that can affect users
• [x] Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• [x] If this is a bug fix, PR should be raised against the relevant ?.?.x branch.