pypyr task-runner cli & api for automation pipelines.

pypyr

Last update: Dec 15, 2022

Related tags

CLI Tools python devops automation pipeline script tool continuous-integration ci cd continuous-deployment pipelines ci-cd task-runner task-manager script-loader pipeline-processor pipelines-yaml taskrunner pipeline-runner

Overview

pypyr automation task runner

All documentation is here: https://pypyr.io/

pypyr

pronounce how you like, but I generally say piper as in "piping down the valleys wild"

pypyr is a free & open-source task-runner that lets you define and run sequential steps in a pipeline.

Like a turbo-charged shell script, but less finicky. Less annoying than a makefile.

pypyr runs pipelines defined in yaml. A pipeline is pretty much anything you want to automate with a sequence of steps.

Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

You can run loops, conditionally execute steps based on conditions you specify, wait for status changes before continuing, break on failure conditions or swallow errors. Pretty useful for orchestrating continuous integration, continuous deployment & devops operations.

pypyr gives you simple variable substitution & configuration file management so you can read, merge and write configuration files to and from yaml, json or just text.

Installation

$ pip install --upgrade pypyr

Tested against Python >=3.7

Stuck with an older version of python? Want to run pypyr in an environment that you don't control, like a CI server somewhere?

You can use the official pypyr docker image as a drop-in replacement for the pypyr executable.

$ docker run pypyr/pypyr echo "Ceci n'est pas une pipe"

Usage

This is a pipeline

Example pipeline that runs a sequence of steps and takes an optional custom cli input argument:

# ./show-me-what-you-got.yaml
context_parser: pypyr.parser.keyvaluepairs
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: o hai!
  - name: pypyr.steps.cmd
    in:
      cmd: echo any cmd you like
  - name: pypyr.steps.shell
    in:
      cmd: echo ninja shell power | grep '^ninja.*r$' 
  - name: pypyr.steps.py
    in:
      py: print('any python you like')
  - name: pypyr.steps.cmd
    while:
      max: 3
    in:
      cmd: echo gimme a {whileCounter}
  - name: pypyr.steps.cmd
    foreach: [once, twice, thrice]
    in:
      cmd: echo say {i}
  - name: pypyr.steps.default
    in:
      defaults:
        sayBye: False
  - name: pypyr.steps.echo
    run: '{sayBye}'
    in:
      echoMe: k bye!

This is how you run a pipeline

This is what happens when you run this pipeline:

$ pypyr show-me-what-you-got
o hai!
any cmd you like
ninja shell power
any python you like
gimme a 1
gimme a 2
gimme a 3
say once
say twice
say thrice

$ pypyr show-me-what-you-got sayBye=true  
o hai!
any cmd you like
ninja shell power
any python you like
gimme a 1
gimme a 2
gimme a 3
say once
say twice
say thrice
k bye!

Help!

Don't Panic! Check the pypyr technical docs to begin. For help, community & talk, check pypyr twitter, or join the chat at the pypyr community discussion forum!

Contribute

Developers

For information on how to help with pypyr, run tests and coverage, please do check out the contribution guide.

Bugs

Well, you know. No one's perfect. Feel free to create an issue.

License

pypyr is free & open-source software distributed under the Apache 2.0 License.

Please see LICENSE file in the root of the repo..

Comments

save error

save error information from both the swallow and the retry decorators to context, so that subsequent steps can use them.

also make err available in the on_failure handler.
enhancement type: decorator

opened by yaythomas 9
Execute pypyr from python code
Hello,

First of all let me thank you, for a nice project.

I plan to use pypyr not as cli, but as remote command executor, using template dynamically loaded from config service like zookeeper.

So my questions is:

btw is this totally a good idea to use pypyr not as cli, but as single runner service, or I am going a wrong way?

if so, is any plan to support for step restrictions or sandboxing, due to I do not want to allow my user to run custom shell script?

Is any plan to get asyncio support? I want to run the same script multiple times separately for different users, and my steps will have a lot of i/o operations.

Thanks
question
opened by Reskov 7
Allow Substitutions in pypyr.steps.pype

Allow Substitutions in pype steps. Rationale; Just like having the possibility to have Run/Skip being evaluated at runtime it could be useful to have be able to pick the pipe to run at runtime.

A toy example is that an initial step is to classify an object into a shape (square, circle, hexagon). That could be followed by a pipe that computes data (area etc) for that shape. Back in the parent pipe a report can be conducted regardless of which child pipe generated the data.
enhancement type: step

opened by rickardcronholm 6
run_failure_step_group: Object of type Exception is not JSON serializable
when I raise a ValueError or any other Exception in a python module, the following error appears in the on_failure handler:

ERROR:pypyr.stepsrunner:run_failure_step_group: Failure handler also failed. Swallowing.

ERROR:pypyr.stepsrunner:run_failure_step_group: Object of type Exception is not JSON serializable

How can raise an error in python modules that works in the failure handler?
opened by joenatan 5
DO NOT MERGE: Experimental rewrite of ops to invoke
To learn pypyr, I rewrote existing ops/* content to invoke as I am using invoke often and feel quite familiar with it.

This pull requests is not supposed to be accepted, I put it here to notify about possible comparison between pypyr and invoke implementation only.

For quick review, read the ops/README.rst file.

All stuff from ops/*.yaml is implemented in tasks.py and ops/*.py files.

GitHub actions are not modified and I am not planning to change that.

Comparing amount of code:

pypyr ops/*.yml files: 259 lines

tasks.py + ops/*.py + invoke.yml: 399 lines.

If I would try to have less lines, it seems feasible to get the invoke part down to 300 lines This makes pypyr 259 lines rewritten to invoke 300 lines what seems a bit more, but still pretty close.

Lessons learned

pypyr turns to be a bit more succinct comparing to invoke code.

some people might prefer coding in python as this does not require to learn any new syntax

invoke solution allows more granular set of actions to call and is quite self-explanatory

learning invoke in depth also requires some time

invoke required some customization to report called tasks and steps and their results in more readable way

the most daunting task is to design actual actions to do for build, tests and other actions and this applies to both pypyr and invoke implementation.

pypyr surprised me by wide range of constructs usable for given purpose. First impression was it is not so easy to read but after this exercise I like the pypyr syntax more
opened by vlcinsky 5
fix ruamel.yaml on 0.14.X for now

Thank you for using ruamel.yaml.

There will be API changes in the 0.15+ versions, that might lead to warnings that your users could see. Therefore please release a version of your package with this change, so that it will not automatically take the latest ruamel.yaml release.

opened by AvdN 5

fetchFile step

Add step fetchFile to fetch file from remote URL. supported protocols: http, https, ftp and sftp. Also supported http basic auth, ftp and sftp authentication.

Example:

steps:
  - name: pypyr.steps.fetchfile
    comment: fetch file from URL and save.
    in:
      fetchFile:
          url: https://www.example.com/files/file.tar.gz
          path: /home/user/Downloads

opened by mofm 4

Please evaluate OmegaConf and Hydra
Hi, interesting project! I am the author of OmegaConf and of Hydra.

Take a look at OmegaConf has some powerful features that can help your project:

Config composition

Interpolations

Optional strong runtime type safety (via dataclasses). And more.

Hydra could also be of interest, although some of it's default behaviors may be inappropriate for your proejct.
opened by omry 4

Run/skip evaluation ordering

Currently, pypyr always evaluates all of run_me, skip_me, swallow_me parameters. This is not an issue, but may produce some unexpected behavior, (or expected I am not pretty sure 😄 )

For example:


steps:
   - name: pypyr.steps.echo
     run: False
     skip: !py "invalid"
     in:
       echoMe: "hello"

This example will fail with NameError

Something went wrong. Will now try to run on_failure.

NameError: name 'invalid' is not defined

But it should be clear without evaluation of skip_me that this step will never be executed.

https://github.com/pypyr/pypyr/blob/67bc1795493c19e648e12f776a644f92e3bd2fc8/pypyr/dsl.py#L564

@yaythomas What do you think about lazy/strict evaluation of these params?

Something like:

         run_me = context.get_formatted_as_type(self.run_me, out_type=bool)
-        skip_me = context.get_formatted_as_type(self.skip_me, out_type=bool)
-        swallow_me = context.get_formatted_as_type(self.swallow_me,
-                                                   out_type=bool)
 
         if run_me:
+            skip_me = context.get_formatted_as_type(self.skip_me, out_type=bool)
+
             if not skip_me:
                 try:
                     if self.retry_decorator:
@@ -579,6 +578,9 @@ def run_conditional_decorators(self, context):
                     # else, not errors per se.
                     raise
                 except Exception as exc_info:
+                    swallow_me = context.get_formatted_as_type(self.swallow_me,
+                                                               out_type=bool)
+
                     if isinstance(exc_info, HandledError):

In addition, it will slightly optimize performance

enhancement type: core

opened by Reskov 3

The nested group call is unsupported when the parent section is in a loop.
I am using a lot of call groups in my project, they helped me to structure better my pipeline. Meanwhile, I face a little problem with calling subgroups. I had prepared a little bit tricky example of pypyr usage to show the problem.

steps: - name: pypyr.steps.default in: defaults: index: 0 - name: pypyr.steps.call in: call: groups: - main while: stop: !py 'index > 2' main: - name: pypyr.steps.contextmerge in: contextMerge: index: !py 'index + 1' - name: pypyr.steps.call in: call: groups: - subgroup subgroup: - pypyr.steps.debug

This pipeline will stack in the infinity loop, due to the call section is overridden after the first iteration.

It seems that the nested group call is unsupported when the parent section is in a loop. The problem is that the nested group is resetting 'call' key in the context, and unfortunately while from the main will call subgroup section instead of the main How can we deal with this?
bug type: core
opened by Reskov 3
helpers for py strings
Create helper functions that are directly available to py strings.

Specifically:

hasvalue(thing)

exists(thing)

Add new module for eval expression helpers. Add this module's vars to the eval global namespace.

this is useful because currently pipers have to jump through hoops on run bool conditions on existence checks(you get an error if the var doesn't exist in context) and on truthy style checks on strs (str that exists but is not literal "true" will be resolve false because the pypyr core tries to be too helpful with converting strings to bools by evading True/True/1 et as bool True and everything else False).
enhancement up next type: core type: step
opened by yaythomas 3
cmd step: support ~ & env
pypyr.steps.cmd doesn't support ~ expansion or $ENV vars in its cmd input.

Easy enough to add. Then operators won't have to go and invoke the more inefficient pypyr.steps.shell just for doing something like:

- name: pypyr.steps.cmd in: cmd: cp ~/mypath ~/myoutpath

Note that $ENV expansion supports formats $ENV and ${ENV}. It leaves unrecognised tokens alone, meaning pypyr can do a {substitution} format afterwards without breaking anything.
enhancement type: step
opened by yaythomas 1
integration tests use out list instead of results file + patching echo

Some of the integration tests are fairly slow & this gets annoying during dev cycles.

Especially the integration tests that read expected output from a _expected file can rewire instead to check a list from the result context, assuming the test pipelines replace pypyr.steps.echo with pypyr.steps.append. This saves at least one file PER TEST being read from disk during test runs.

This can be an incremental task. For starters, will just focus on the slowest offenders - which is tests/integrations/pypyr/call/loopcallnested_int_test.py. These are the only tests that use assert_pipeline_raises_match_file.

Other integration tests that just intercepts echo can also use the pypyr.steps.append technique - these will be updated incrementally over time.
enhancement

opened by yaythomas 0
JSON Schema for pypyr YAML files

Having JSON schema for pypyr YAML configuration files would be handy. Many editors (at least my neovim) allow to configure runtime validation of authored file if JSON schema is available.

My question is - is there any JSON schema for pypyr already in place? My quick research did not reveal any.

If it does not exist, I am considering authoring one (using pydantic allows to create such a schema very simply).
enhancement type: env on hold

opened by vlcinsky 5
step: pypyr.steps.sleep
step that sleeps/pauses execution for given number of seconds.

step context input:

sleep: 5 # input is seconds, decimals allowed.

Supports substitutions.

Defaults to 5.
enhancement type: step
opened by yaythomas 0
pypyr.steps.find

Step to find a specified string, string expression (or maybe regex) in a file.

If not file, specify path to recurse all files in that path looking for file contents that match the pattern.

Return True, filename/position. Have raise/throw attribute to raise error if not found.
enhancement type: step

opened by yaythomas 0

Releases(v5.7.1)

v5.7.1(Oct 25, 2022)
summary

maintenance release, no new features

bug fix for pypyr.steps.filereplace to honor flat format :ff directive

python 3.11 compatibility confirmed

on python 3.11 runtimes, the tomli dependency won't install separately anymore since it's part of stdlib now

what's changed

pypyr.steps.filereplace get_formatted X2 by @yaythomas in https://github.com/pypyr/pypyr/pull/304

new codecov uploader by @yaythomas in https://github.com/pypyr/pypyr/pull/305

py 3.11 & conditional tomli dep by @yaythomas in https://github.com/pypyr/pypyr/pull/306

Bump version: 5.7.0 → 5.7.1 by @yaythomas in https://github.com/pypyr/pypyr/pull/307

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.7.0...v5.7.1
Source code(tar.gz)
Source code(zip)
v5.7.0(Oct 20, 2022)
summary

New switch step for IF-ELSE style branching in your pipelines!

New argskwargs parser to combine plain args and key-value pairs (key=value) from the cli

keyvaluepairs and dict parsers now support having = in the value, so you can have key=one+one=two parse to {'key': 'one+one=two'}

these parsers now also support passing args with no =, in which case arg1 becomes {'arg1': ''}

what's changed

switch by @yaythomas in https://github.com/pypyr/pypyr/pull/299

typing on dsl.cof by @yaythomas in https://github.com/pypyr/pypyr/pull/300

gh action checkout v3 by @yaythomas in https://github.com/pypyr/pypyr/pull/301

argskwargs parser & partitioning on = for kvp by @yaythomas in https://github.com/pypyr/pypyr/pull/302

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.6.0...v5.7.0
Source code(tar.gz)
Source code(zip)
v5.6.0(Oct 5, 2022)
summary

you can now use a custom error message when pypyr.steps.assert raises an exception.

new pypyr.steps.venv step to create venvs in parallel from yaml or toml config.

new venv-create pipeline so you can provision venvs concurrently from declarative config without having to write your own pipeline or script.

pypyr is now built & packaged by the excellent PEP517 compliant flit.

what's changed

flake8 new version line length by @yaythomas in https://github.com/pypyr/pypyr/pull/286

from setup.py to flit by @yaythomas in https://github.com/pypyr/pypyr/pull/290

remove redundant dict on parser context init by @yaythomas in https://github.com/pypyr/pypyr/pull/292

venv creator & log to stdout by @yaythomas in https://github.com/pypyr/pypyr/pull/296

ops/build uses venv for publish verify by @yaythomas in https://github.com/pypyr/pypyr/pull/297

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.5.0...v5.6.0
Source code(tar.gz)
Source code(zip)
v5.5.0(May 26, 2022)
summary

cmdOut for pypyr.steps.cmd & pypyr.steps.shell gets dotted attribute access.

instead of cmdOut[returncode] access value like this: cmdOut.returncode

backwards compatible - the old-style dict-like accessors will still work.

closes #272

Introduce pypyr.steps.cmds & pypyr.steps.shells to run programs or shell statements asynchronously as parallel (concurrent) sub-processes.

cmdOut for these works as with the serial versions of the steps using the new dotted attribute access introduced in this release.

closes #273

Set default encoding for all the cmd/shell steps using default_cmd_encoding in config. You can use the $env variable PYPYR_CMD_ENCODING to initialize this value, in addition to the usual config files.

closes #275

what's changed

async cmds + shells by @yaythomas in https://github.com/pypyr/pypyr/pull/274

release 5.5 & default_cmd_encoding for serial by @yaythomas in https://github.com/pypyr/pypyr/pull/276

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.4.0...v5.5.0
Source code(tar.gz)
Source code(zip)

v5.4.0(Apr 14, 2022)

Summary

pypyr.steps.cmd & pypyr.steps.shell now also takes a list input to run multiple commands/shell statements in the same step!

- name: pypyr.steps.cmd
  in:
    cmd:
      - echo 1
      - echo 2

- name: pypyr.steps.cmd
  in:
    cmd:
      run:
        - echo 3
        - echo 4
      save: False
      cwd: mydir/subdir

Both the cmd and shell steps expanded to allow:
- decode output in different encodings
- save output as raw bytes or as encoded text.
- File output for stdout/stderr.
  - File create mode of append or overwrite.
- stderr can redirect to /dev/stdout
- Both stdout & stderr can redirect to /dev/null

#  when save: True
- name: pypyr.steps.cmd
  comment: when save is True
  in:
    cmd:
      run: curl https://myurl.blah/diblah
      save: True
      cwd: .
      bytes: False
      encoding: utf-8

#  when save: False (this is default)
- name: pypyr.steps.cmd
  comment: when save is False (the default when `save` not set)
  in:
    cmd:
      run: curl --cert certfile --key keyfile https://myurl.blah/diblah
      cwd: ..
      stdout: ./path/out.txt
      stderr: ./path/err.txt
      append: False

What's Changed

cmd & shell list input by @yaythomas in https://github.com/pypyr/pypyr/pull/271

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.3.0...v5.4.0

Source code(tar.gz)
Source code(zip)

v5.3.0(Mar 9, 2022)
Summary

Create shortcuts to your pypyr run commands & their input args. This is handy for creating short & sweet aliases for longer pipeline run commands.

So if you have a pipeline you normally run like this:

$ pypyr arb/my-pipeline arg1=1234 arg2="/path/to long/annoying path to type/x" arg3="arb'hello"

You can create a shortcut alias for this like so:

shortcuts: my-shortcut: pipeline: arb/my-pipeline args: arg1: 1234 arg2: /path/to long/annoying path to type/x arg3: "arb'hello"

And now for the same pipeline + inputs you can just run:

$ pypyr my-shortcut

You can create your shortcut definition in pyproject.toml or in the pypyr yaml config file.

You can make your shortcuts project specific, or for the entire user profile, or globally for your entire system.

What's Changed

Shortcuts by @yaythomas in https://github.com/pypyr/pypyr/pull/268

Bump version: 5.2.0 → 5.3.0 by @yaythomas in https://github.com/pypyr/pypyr/pull/269

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.2.0...v5.3.0
Source code(tar.gz)
Source code(zip)
v5.2.0(Feb 19, 2022)
Summary

pypyr is now configurable with yaml or pyproject.toml!

Explicitly over-ride encoding on any filesystem steps

What's Changed

friendly err on malformed pipeline. by @yaythomas in https://github.com/pypyr/pypyr/pull/250

config & encoding by @yaythomas in https://github.com/pypyr/pypyr/pull/255

file encoding from config by @yaythomas in https://github.com/pypyr/pypyr/pull/259

windows tests & config platform indicators by @yaythomas in https://github.com/pypyr/pypyr/pull/261

typings for config, platform by @yaythomas in https://github.com/pypyr/pypyr/pull/262

FakePath PathLike & parent also fake by @yaythomas in https://github.com/pypyr/pypyr/pull/264

pypyr.steps.python & version bump 5.2.0 by @yaythomas in https://github.com/pypyr/pypyr/pull/266

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.1.0...v5.2.0
Source code(tar.gz)
Source code(zip)
v5.1.0(Dec 12, 2021)
Finally, toml, has come, to pypyr! 🎆

pypyr.parser.tomlfile to initialise pipeline context with a toml file

pypyr.steps.fetchtoml to read toml file into context as an object with typing

pypyr.steps.filewritetoml to write specified context to output toml file - lets you create toml files on-the-fly programmatically.

pypyr.steps.fileformattoml to read input tomls specified by path, glob or list of paths, replace {formatting expressions} and write to output file(s).

Working with files:

pypyr.steps.fileread to read files in text or binary mode into context.

pypyr.steps.filewrite to write or append to file in text or binary modes.

What's Changed

pypyr.steps.fileread by @yaythomas in https://github.com/pypyr/pypyr/pull/244

pypyr.steps.filewrite by @yaythomas in https://github.com/pypyr/pypyr/pull/245

toml by @yaythomas in https://github.com/pypyr/pypyr/pull/246

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.0.0...v5.1.0
Source code(tar.gz)
Source code(zip)
v5.0.0(Nov 20, 2021)
Implement adr2 relative pipelines + api changes.

In brief, this release lets pipelines reference custom modules & child pipelines relative to the pipeline itself, rather than the current directory. This lets you create portable, re-usable & composable pipeline libraries.

Breaking Changes

This is a major version increment because it comes with BREAKING CHANGES:

API: pipelinerunner.run() replaces both pipelinerunner.main() and pipelinerunner.main_with_context()

API: def get_pipeline_definition(pipeline_name, working_directory) signature for custom pype loaders changes to def get_pipeline_definition(pipeline_name, parent)

CLI: the —dir flag now only sets the directory for ad hoc custom Python modules, it does NOT also set the directory for pipelines anymore

Final removal of deprecated get_formatted_iterable, get_formatted_string #195 & pypyr.steps.contextset #184. Where previously these would just give deprecation warnings, they are now completely removed.

pypyr.pypeloaders.fileloader renamed pypyr.loaders.file

Non-Breaking Changes

You can now access the current pipeline’s metadata & loader information from within a pipeline with context.current_pipeline

Improve handling of absolute paths in file loader only to search path once, rather than unnecessarily go through the same relative path lookup sequence with the same path.

Typing support added for the pypyr API entrypoint.

Detailed Technical Breakdown:

Introduce new classes to model pipeline payload, rather than just using the bare dict-like yaml directly.

PipelineInfo - pipeline metadata set by loader. This maintains a pipeline’s parent/path info so that child pipelines can load relative to the parent.

PipelineDefinition - this wraps the pipeline payload and its metadata (PipelineInfo) to allow pypyr to cache it all with one reference

Add new Pipeline class for the run-time properties of a single run.

The Pipeline references the shared cached PipelineDefinition.

Move run + load_and_run logic from pypyr.pipelinerunner to the new Pipeline class. This massively streamlines the pipeline invocation process, since run/load_and_run can just operate on the shared Pipeline state rather than sling a bunch of args between different functions as before.

Add a call-stack of running Pipeline instances on Context. i.e Parent -> child1 -> child2 where the root pipeline calls child pipelines via pype

Add current_pipeline attribute to Context, controlled with a context manager to scope itself to an individual pipeline run’s lifespan.

This means that steps can access current pipeline’s properties.

This allows pypyr.steps.pype to find the current (i.e parent) pipeline’s metadata such as path, to load child pipeline relative to the calling pipeline’s location.

When a child pipeline completes, the calling pipeline (i.e the previous Pipeline in the call-stack) becomes the current pipeline

Amend pypyr.steps.pype to instantiate Pipeline object to load_and_run() child.

pype now deals the context.current_pipeline.pipeline_definition.info metadata to work out whether to cascade parent path down to child, so child can load relative to the parent.

Notably, the parent loader now cascades to the child, so pipeline authors don’t need explicitly to set the same custom loader repeatedly for each child.

Given the context manager controlling current pipeline scope in Pipeline.load_and_run_pipeline remove the clumsy side-shuffle for pipeline_name, working_dir to swap out these values as child pipe runs and swap these back when it completes/errors.

Remove global PipelineCache. Replace with distinct pipeline cache per loader.

This resolves a long standing limitation where pypyr assumed unique pipeline names across all loaders.

The per-loader pipeline cache stores PipelineDefinition objects.

Introduce Loader class, which wraps loader & its pipeline cache. Loader is what the LoaderCache caches.

Thus LoaderCache -> Loader -> _pipelineCache -> PipelineDefinition

File loader has a private file cache keyed on absolute path of file

This is to prevent >1 load+parse where the same underlying pipeline.yaml file has different names in the loader’s pipeline cache

e.g (name=‘dir/mypipe’, parent=None) and (name=‘mypipe’, parent=‘dir’) both resolve to dir/mypipe.yaml

Caching a reference to the PipelineDefinition object, so not duplicating memory

Improve handling of absolute paths in file loader only to search path X1, rather than unecessarilly go through the same relative path lookup sequence with the same path.

File loader get_pipeline_definition now returns a PipelineDefinition with PipelineFileInfo to store file-system specific metadata for the loader pipeline

Remove working_dir global. The py_dir input on run() now refers ONLY to module paths, NOT pipeline locations.

The CLI —dir flag, or py_dir input on run() basically adds the specified directory to sys.path.

Add current pipeline’s parent directory to sys.path on load. This allows child pipelines to resolve custom modules relative to itself.

pypyr.dsl.Step does not need StepsRunner anymore, because it can get it from the context.current_pipeline instead.

Recode (some) integration tests to take advantage of list pypyr.steps.append step and checking that for output on return context rather than intercepting logger.NOTIFY.

Rename master branch to main in CI/CD GitHub actions

Add typing annotations to the public run() function and the Pipeline class public accessors. The idea is NOT to type pypyr exhaustively, just to provide annotations for the sensible/likely entrypoint to enhance API user experience. Include py.typed in pypyr package.

Remedy packaging snafu where tests.common was deploying alongside pypyr because exclude condition in find_packages didn't include wildcard for sub packages.

What's Changed

Relative pipelines & API run() replaces main/main_with_context by @yaythomas in https://github.com/pypyr/pypyr/pull/243

Full Changelog: https://github.com/pypyr/pypyr/compare/v4.6.0...v5.0.0
Source code(tar.gz)
Source code(zip)
v4.6.0(Oct 6, 2021)
What's Changed

Fix failing ops/build (failing on linting issues and locale dependent failure) by @vlcinsky in https://github.com/pypyr/pypyr/pull/226

Evaluate step skip only if the run check returned true by @Reskov in https://github.com/pypyr/pypyr/pull/234

pypyr.steps.append & pypyr.steps.add by @yaythomas in https://github.com/pypyr/pypyr/pull/235

shorter alias set for contextsetf. py 3.10. by @yaythomas in https://github.com/pypyr/pypyr/pull/238

New Contributors

@vlcinsky made their first contribution in https://github.com/pypyr/pypyr/pull/226

Full Changelog: https://github.com/pypyr/pypyr/compare/v4.5.0...v4.6.0
Source code(tar.gz)
Source code(zip)
v4.5.0(Feb 2, 2021)
add retry backoff strategies. #216.

prevent duplicates in sys.path (#218) on main* api entry points. thread-safe existence check on sys.path list.

prevent redundant multiple addition of notify log-level on main* api entry. Move initialization code to package __init__ (#219).

Source code(tar.gz)
Source code(zip)
v4.4.1(Jan 1, 2021)
Context parsers that create an entry in context now initialise to empty rather than None. This means you can directly use something like {argList}, {argDict} and argString (initialising respectively to [], {}, '') directly for things like foreach loops without having to worry about None checks. Your existing truthy checks for these values will work as before. closes #214.

Source code(tar.gz)
Source code(zip)
v4.4.0(Dec 26, 2020)
New step pypyr.steps.pyimport to import references to the py-string namespace.

This includes an underlying api signature change by removal of pypyr.utils.expressions.eval_string(), but this is sufficiently far down the call-chain that it shouldn’t affect any normal pipeline operator or api consumer.

See #110 for details.

pypyr.steps.contextclearall wipes pyimport imported references in addition to the key/values inside context.

Simplify pypyr.steps.py syntax by allowing a new py (rather than pycode) input. This allows pipeline authors to use context key names directly as variables, rather than have to specify them as keys in context (my_var vs context[‘my_var’]).

see #204 for details on simplifying the pypyr.steps.py syntax.

the old pycode will keep on working in the same way, so no need to worry about backwards compatibility for your existing pipelines.

Allow substitutions on Retry max. Resolves #207. Excellent bug find & fix by @Reskov, much thanks 🙌 🙌 🙌 as ever for a superb contribution! 🔥 🔥 🔥

foreach can now use any iterable, including generators. Closes #209

Source code(tar.gz)
Source code(zip)
v4.3.0(Nov 12, 2020)
Streamline main entrypoint API. close #201.

main() allows consumer to set pype loader, rather than having to drop further down into api to load_and_run_pipeline()

new main_with_context() to input dict to initialise context, and bypass context_parser entirely. Also returns the Context object after pipeline run completes.

make all non-essential args optional to allow minimal calls to main entrypoint without having to add optional=None style inputs.

This is fully backwards compatible.

pypyr.steps.pype

defaults useParentContext to False is pipeArgs specified.

pipeArgs shlex-es input string

set pipeline_name on child pipeline rather than use parent pipeline name

working_dir uses Path object rather than string

pypyr.steps.echo remove redundant string check.

Source code(tar.gz)
Source code(zip)
v4.2.0(Oct 30, 2020)
python 3.9 compatibility officially confirmed. All good. Woo!

New Flat ff and Recursive rf formatting specifiers. #195

Maintain backwards compatibility for all current formatting functionality.

The get_formatted_string, get_formatted_iterable & get_processed_string methods on Context() are now all deprecated. Use get_formatted_value instead. All of the deprecated functions will keep on working as before, but will print a WARN to the output.

get_formatted and get_formatted_value remain the official best ways of getting formatted values.

New !jsonify yaml special tag directive. This converts context object to a json string. #197.

New pypyr.steps.jsonparse step. Parse and deserialise a json string into Context. #199

Bare pypyr.steps.assert syntax. #196

Add new utils.asserts for extra flexibility with nested key validation.

PyStrings optimised. Not so that you'd notice, but still, it'll be a smidge faster.

Friendlier error messages when yaml/json context parsers reject documents that do NOT have a mapping at root.

Source code(tar.gz)
Source code(zip)
v4.1.0(Aug 30, 2020)
Better description output - add (skipping) to output if the step is not running because run is False or skip is True. Ref #158.

First release published from shiny new GitHub Action CI/CD!

Add License to wheel published to pypi

Source code(tar.gz)
Source code(zip)
v4.0.0(Aug 17, 2020)
BREAKING CHANGE: final deprecation of in args persisting after step execution. For full discussion please see #177. Preview functionality where setting environment variable $PYPYR_IN_CLEAN = 1 is now the default. You can remove the $PYPYR_IN_CLEAN variable from this release onwards. Henceforth pypyr removes in args from context after step completes. If you want to persist values in context, use pypyr.steps.contextsetf, pypyr.steps.contextcopy or pypyr.steps.default.

BREAKING CHANGE: pypyr.steps.assert raises an AssertionError rather than a ContextError. Ref #188.

The cli now defaults to a simplified log format that is way less noisy. If you still want the full, old style logs, run pypyr with --log 25 set explicitly. pypyr my pipe --log 25. Ref #187

description decorator output also simplified to echo only your actual custom text.

Failure Groups now support Stop instructions. This allows you to exit the failure handler reporting success, so pypyr can report success if you consider the error condition handled. Ref #175.

pypyr.steps.contextset renamed pypyr.steps.contextcopy. The old name will keep on working, but with a deprecation warning. Ref #184.

internal: improvements to integration testing error reporting.

Source code(tar.gz)
Source code(zip)
v3.2.2(Aug 13, 2020)
Line + Column Number count from 1 not 0. fix #151

update README to point at new website pypyr.io.

update CONTRIBUTING to point at new website.

update documentation and pypi setup with new repo name.

Source code(tar.gz)
Source code(zip)
v3.2.1(Apr 14, 2020)
Allow non-string keys on context.get_formatted_iterable and context merge (including the step pypyr.steps.contextmerge). Resolves #179. Much thanks to @Reskov for a detailed & helpful bug report!

Source code(tar.gz)
Source code(zip)
v3.2.0(Apr 4, 2020)
ATTENTION: Please start preparing all your pipelines to be ready for the next major release where in context arguments will be removed from context on step completion. This shouldn't be too much of an issue. For any steps where you need to set enduring context, use contextsetf or contextset.

Allow using a call control-of-flow instruction nested inside while/for/retry loops. Previously a nested call instruction would over-write the parent call's configuration and thus create confusion on the next loop round the parent. Much thanks 🙏 to @Reskov for identifying this tricky little recursion issue & assistance in resolution. Close #176

in context parameters only available for the duration of the step they decorate. Previously in items would be added to context and stay in context after the step finishes. This is a potentially breaking change for existing pipelines, but worth it, full discussion of reasons here #177.

For this release this change in functionality is not the default, but enabled by setting the following environment variable PYPYR_IN_CLEAN = 1. You can set it selectively on a single pypyr run using standard shell syntax like: $ PYPYR_IN_CLEAN=1 pypyr mypipeline arg1 arg2

README updates to document #177 resolution as the default behaviour - in context parameters cleaned from context on step completion to prepare for next major release with the breaking change..

Source code(tar.gz)
Source code(zip)
v3.1.0(Feb 11, 2020)

pypyr.log.logger.set_root_logger(log_level, log_path) call moved from pipelinerunner.main() to cli.main()

this has the side-effect that pipelinerunner.main() signature has changed, with log_level and log_path removed. API consumers should update.

Reason being API consumers should set their own log handlers, since handler configuration should be the prerogative of the calling application, not the invoked library.

Re #172, #173
Source code(tar.gz)
Source code(zip)
v3.0.3(Jan 29, 2020)
Retry Decorator correctly handles errors raised from .call or .jump steps when checking error type with retryOn and stopOn. Close #170 - thanks to @Reskov for reporting!

add tarball to PyPI deploy artefacts.

Source code(tar.gz)
Source code(zip)
v3.0.2(Nov 18, 2019)
When importing a step that does exist (mystep.blah), a failing import mymodule.blah inside mystep.blah would give a misleading error and hide the actual source of the problem. Fixes #166. Eternal thanks to @irancati for raising the issue!

Source code(tar.gz)
Source code(zip)
v3.0.1(Sep 13, 2019)
With a million thank yous 🙏 to @Reskov for these much needed fixes/enhancements! 🎆

Resolves issue #149. pypyr.steps.debug now supports outputting any type, including PyString & SicString

Resolve pypyr.steps.contextmerge issue to support merging any type, including PyString & SicString.

Source code(tar.gz)
Source code(zip)
v3.0.0(Aug 22, 2019)
BREAKING CHANGE: Remove deprecated step input context for (ref #118)

assert

env

fetchjson

fetchyaml

fileformat

fileformatjson

fileformatyaml

filereplace

tar

BREAKING CHANGE: cli context input now uses standard cli spacing rather than needing to str close everything. Closes #94

So where you had:

pypyr mypipeline "key1=value1,key2=value2”

Now instead:

pypyr mypipeline key1=value1 key2=value2

BREAKING CHANGE: Big API changes to pipelinerunner, stepsrunner and Step

BREAKING CHANGE: pypyr.parser.commas replaced by pypyr.parser.keys

Caching pipelines, context parsers, loaders, step run functions. Basically cache anything that loads from disk. Closes #126

Custom step groups. Closes #126

Jump, Call, and Stop ref #128 #127 #123

The ./pipelines default directory location no longer necessary. To run a pipeline ./dir/mypipe.yaml, just go pypyr dir/mypipe

README+help updates for all of the above

Source code(tar.gz)
Source code(zip)
v2.11.0(Aug 6, 2019)
This one is all @Reskov 🎉 eternal gratitude!

new NOTIFY logging level added. this level is much less verbose than even INFO, and only displays explicits echos that you specify in your pipeline, and also the description from the step decorator. This should make it much easier to see what your pipeline is doing without the extra detail that can make things confusing and harder to follow. #132

under-the-hood api improvement for more performant logging #152

Source code(tar.gz)
Source code(zip)
v2.10.0(Jul 10, 2019)
With eternal thanks to @Reskov 🎉 for next level super useful improved error handling! 🙇

#139 - pypyr now saves all run-time errors to context under runErrors. This means subsequent steps can use error information from previous steps.

#135 - error information now include the pipeline source yaml line+col numbers

Source code(tar.gz)
Source code(zip)
v2.9.0(Jun 14, 2019)
pypyr.steps.glob lets you get a list of paths from a glob, and combine different globs into a combined output result list. Issue #145

with much thanks to @Reskov 🎉 , resolves issue #143 - merge step now supports special tag directives

Source code(tar.gz)
Source code(zip)
v2.8.0(Mar 24, 2019)
new Retry decorator allows steps to retry automatically when step encounters an error. ref #130

fix run_step AttributeError might have caught or hidden AttributeErrors in the step code itself, rather than just when the step module didn't contain a run_step function. ref #129

while loop checks stop condition only at end of each iteration. previously if a stop condition evaluated True even before the loop started the loop wouldn't run at all. now the loop will run once. If you don't want the step to run, use the skip: True decorator instead.

it's now easier to write infinite While loops with an explicit stop: False condition

Source code(tar.gz)
Source code(zip)
v2.7.0(Mar 21, 2019)
add string interpolation to pypyr.steps.pype input.

Source code(tar.gz)
Source code(zip)