pypyr task-runner cli & api for automation pipelines.

Overview

pypyr task runner for automation pipelines

pypyr automation task runner

All documentation is here: https://pypyr.io/

build status coverage status pypi version apache 2.0 license

pypyr

pronounce how you like, but I generally say piper as in "piping down the valleys wild"

pypyr is a free & open-source task-runner that lets you define and run sequential steps in a pipeline.

Like a turbo-charged shell script, but less finicky. Less annoying than a makefile.

pypyr runs pipelines defined in yaml. A pipeline is pretty much anything you want to automate with a sequence of steps.

Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

You can run loops, conditionally execute steps based on conditions you specify, wait for status changes before continuing, break on failure conditions or swallow errors. Pretty useful for orchestrating continuous integration, continuous deployment & devops operations.

pypyr gives you simple variable substitution & configuration file management so you can read, merge and write configuration files to and from yaml, json or just text.

Installation

$ pip install --upgrade pypyr

Tested against Python >=3.7

Stuck with an older version of python? Want to run pypyr in an environment that you don't control, like a CI server somewhere?

You can use the official pypyr docker image as a drop-in replacement for the pypyr executable.

$ docker run pypyr/pypyr echo "Ceci n'est pas une pipe"

Usage

This is a pipeline

Example pipeline that runs a sequence of steps and takes an optional custom cli input argument:

# ./show-me-what-you-got.yaml
context_parser: pypyr.parser.keyvaluepairs
steps:
  - name: pypyr.steps.echo
    in:
      echoMe: o hai!
  - name: pypyr.steps.cmd
    in:
      cmd: echo any cmd you like
  - name: pypyr.steps.shell
    in:
      cmd: echo ninja shell power | grep '^ninja.*r$' 
  - name: pypyr.steps.py
    in:
      py: print('any python you like')
  - name: pypyr.steps.cmd
    while:
      max: 3
    in:
      cmd: echo gimme a {whileCounter}
  - name: pypyr.steps.cmd
    foreach: [once, twice, thrice]
    in:
      cmd: echo say {i}
  - name: pypyr.steps.default
    in:
      defaults:
        sayBye: False
  - name: pypyr.steps.echo
    run: '{sayBye}'
    in:
      echoMe: k bye!

This is how you run a pipeline

This is what happens when you run this pipeline:

$ pypyr show-me-what-you-got
o hai!
any cmd you like
ninja shell power
any python you like
gimme a 1
gimme a 2
gimme a 3
say once
say twice
say thrice

$ pypyr show-me-what-you-got sayBye=true  
o hai!
any cmd you like
ninja shell power
any python you like
gimme a 1
gimme a 2
gimme a 3
say once
say twice
say thrice
k bye!

Help!

Don't Panic! Check the pypyr technical docs to begin. For help, community & talk, check pypyr twitter, or join the chat at the pypyr community discussion forum!

Contribute

Developers

For information on how to help with pypyr, run tests and coverage, please do check out the contribution guide.

Bugs

Well, you know. No one's perfect. Feel free to create an issue.

License

pypyr is free & open-source software distributed under the Apache 2.0 License.

Please see LICENSE file in the root of the repo..

Copyright 2017 the pypyr contributors.

Comments
  • save error

    save error

    save error information from both the swallow and the retry decorators to context, so that subsequent steps can use them.

    also make err available in the on_failure handler.

    enhancement type: decorator 
    opened by yaythomas 9
  • Execute pypyr from python code

    Execute pypyr from python code

    Hello,

    First of all let me thank you, for a nice project.

    I plan to use pypyr not as cli, but as remote command executor, using template dynamically loaded from config service like zookeeper.

    So my questions is:

    1. btw is this totally a good idea to use pypyr not as cli, but as single runner service, or I am going a wrong way?

    2. if so, is any plan to support for step restrictions or sandboxing, due to I do not want to allow my user to run custom shell script?

    3. Is any plan to get asyncio support? I want to run the same script multiple times separately for different users, and my steps will have a lot of i/o operations.

    Thanks

    question 
    opened by Reskov 7
  • Allow Substitutions in pypyr.steps.pype

    Allow Substitutions in pypyr.steps.pype

    Allow Substitutions in pype steps. Rationale; Just like having the possibility to have Run/Skip being evaluated at runtime it could be useful to have be able to pick the pipe to run at runtime.

    A toy example is that an initial step is to classify an object into a shape (square, circle, hexagon). That could be followed by a pipe that computes data (area etc) for that shape. Back in the parent pipe a report can be conducted regardless of which child pipe generated the data.

    enhancement type: step 
    opened by rickardcronholm 6
  • run_failure_step_group: Object of type Exception is not JSON serializable

    run_failure_step_group: Object of type Exception is not JSON serializable

    when I raise a ValueError or any other Exception in a python module, the following error appears in the on_failure handler:

    • ERROR:pypyr.stepsrunner:run_failure_step_group: Failure handler also failed. Swallowing.
    • ERROR:pypyr.stepsrunner:run_failure_step_group: Object of type Exception is not JSON serializable

    How can raise an error in python modules that works in the failure handler?

    opened by joenatan 5
  • DO NOT MERGE: Experimental rewrite of ops to invoke

    DO NOT MERGE: Experimental rewrite of ops to invoke

    To learn pypyr, I rewrote existing ops/* content to invoke as I am using invoke often and feel quite familiar with it.

    This pull requests is not supposed to be accepted, I put it here to notify about possible comparison between pypyr and invoke implementation only.

    For quick review, read the ops/README.rst file.

    All stuff from ops/*.yaml is implemented in tasks.py and ops/*.py files.

    GitHub actions are not modified and I am not planning to change that.

    Comparing amount of code:

    • pypyr ops/*.yml files: 259 lines
    • tasks.py + ops/*.py + invoke.yml: 399 lines.

    If I would try to have less lines, it seems feasible to get the invoke part down to 300 lines This makes pypyr 259 lines rewritten to invoke 300 lines what seems a bit more, but still pretty close.

    Lessons learned

    • pypyr turns to be a bit more succinct comparing to invoke code.
    • some people might prefer coding in python as this does not require to learn any new syntax
    • invoke solution allows more granular set of actions to call and is quite self-explanatory
    • learning invoke in depth also requires some time
    • invoke required some customization to report called tasks and steps and their results in more readable way
    • the most daunting task is to design actual actions to do for build, tests and other actions and this applies to both pypyr and invoke implementation.
    • pypyr surprised me by wide range of constructs usable for given purpose. First impression was it is not so easy to read but after this exercise I like the pypyr syntax more
    opened by vlcinsky 5
  • fix ruamel.yaml on 0.14.X for now

    fix ruamel.yaml on 0.14.X for now

    Thank you for using ruamel.yaml.

    There will be API changes in the 0.15+ versions, that might lead to warnings that your users could see. Therefore please release a version of your package with this change, so that it will not automatically take the latest ruamel.yaml release.

    opened by AvdN 5
  • fetchFile step

    fetchFile step

    Add step fetchFile to fetch file from remote URL. supported protocols: http, https, ftp and sftp. Also supported http basic auth, ftp and sftp authentication.

    Example:

    steps:
      - name: pypyr.steps.fetchfile
        comment: fetch file from URL and save.
        in:
          fetchFile:
              url: https://www.example.com/files/file.tar.gz
              path: /home/user/Downloads
    
    opened by mofm 4
  • Please evaluate OmegaConf and Hydra

    Please evaluate OmegaConf and Hydra

    Hi, interesting project! I am the author of OmegaConf and of Hydra.

    Take a look at OmegaConf has some powerful features that can help your project:

    • Config composition
    • Interpolations
    • Optional strong runtime type safety (via dataclasses). And more.

    Hydra could also be of interest, although some of it's default behaviors may be inappropriate for your proejct.

    opened by omry 4
  • Run/skip evaluation ordering

    Run/skip evaluation ordering

    Currently, pypyr always evaluates all of run_me, skip_me, swallow_me parameters. This is not an issue, but may produce some unexpected behavior, (or expected I am not pretty sure 😄 )

    For example:

    
    steps:
       - name: pypyr.steps.echo
         run: False
         skip: !py "invalid"
         in:
           echoMe: "hello"
    

    This example will fail with NameError

    Something went wrong. Will now try to run on_failure.
    
    NameError: name 'invalid' is not defined
    
    

    But it should be clear without evaluation of skip_me that this step will never be executed.

    https://github.com/pypyr/pypyr/blob/67bc1795493c19e648e12f776a644f92e3bd2fc8/pypyr/dsl.py#L564

    @yaythomas What do you think about lazy/strict evaluation of these params?

    Something like:

             run_me = context.get_formatted_as_type(self.run_me, out_type=bool)
    -        skip_me = context.get_formatted_as_type(self.skip_me, out_type=bool)
    -        swallow_me = context.get_formatted_as_type(self.swallow_me,
    -                                                   out_type=bool)
     
             if run_me:
    +            skip_me = context.get_formatted_as_type(self.skip_me, out_type=bool)
    +
                 if not skip_me:
                     try:
                         if self.retry_decorator:
    @@ -579,6 +578,9 @@ def run_conditional_decorators(self, context):
                         # else, not errors per se.
                         raise
                     except Exception as exc_info:
    +                    swallow_me = context.get_formatted_as_type(self.swallow_me,
    +                                                               out_type=bool)
    +
                         if isinstance(exc_info, HandledError):
    
    

    In addition, it will slightly optimize performance

    enhancement type: core 
    opened by Reskov 3
  • The nested group call is unsupported when the parent section is in a loop.

    The nested group call is unsupported when the parent section is in a loop.

    I am using a lot of call groups in my project, they helped me to structure better my pipeline. Meanwhile, I face a little problem with calling subgroups. I had prepared a little bit tricky example of pypyr usage to show the problem.

    
    steps:
      - name: pypyr.steps.default
        in:
          defaults:
            index: 0
      - name: pypyr.steps.call
        in:
          call:
            groups:
                - main
        while:
            stop: !py 'index > 2'
    
    main:
      - name: pypyr.steps.contextmerge
        in:
          contextMerge:
            index: !py 'index + 1'
    
      - name: pypyr.steps.call
        in:
          call:
            groups:
              - subgroup
    
    subgroup:
      - pypyr.steps.debug
    

    This pipeline will stack in the infinity loop, due to the call section is overridden after the first iteration.

    It seems that the nested group call is unsupported when the parent section is in a loop. The problem is that the nested group is resetting 'call' key in the context, and unfortunately while from the main will call subgroup section instead of the main How can we deal with this?

    bug type: core 
    opened by Reskov 3
  • helpers for py strings

    helpers for py strings

    Create helper functions that are directly available to py strings.

    Specifically:

    • hasvalue(thing)
    • exists(thing)

    Add new module for eval expression helpers. Add this module's vars to the eval global namespace.

    this is useful because currently pipers have to jump through hoops on run bool conditions on existence checks(you get an error if the var doesn't exist in context) and on truthy style checks on strs (str that exists but is not literal "true" will be resolve false because the pypyr core tries to be too helpful with converting strings to bools by evading True/True/1 et as bool True and everything else False).

    enhancement up next type: core type: step 
    opened by yaythomas 3
  • cmd step: support ~ & env

    cmd step: support ~ & env

    pypyr.steps.cmd doesn't support ~ expansion or $ENV vars in its cmd input.

    Easy enough to add. Then operators won't have to go and invoke the more inefficient pypyr.steps.shell just for doing something like:

    - name: pypyr.steps.cmd
       in:
         cmd: cp ~/mypath ~/myoutpath
    

    Note that $ENV expansion supports formats $ENV and ${ENV}. It leaves unrecognised tokens alone, meaning pypyr can do a {substitution} format afterwards without breaking anything.

    enhancement type: step 
    opened by yaythomas 1
  • integration tests use out list instead of results file + patching echo

    integration tests use out list instead of results file + patching echo

    Some of the integration tests are fairly slow & this gets annoying during dev cycles.

    Especially the integration tests that read expected output from a _expected file can rewire instead to check a list from the result context, assuming the test pipelines replace pypyr.steps.echo with pypyr.steps.append. This saves at least one file PER TEST being read from disk during test runs.

    This can be an incremental task. For starters, will just focus on the slowest offenders - which is tests/integrations/pypyr/call/loopcallnested_int_test.py. These are the only tests that use assert_pipeline_raises_match_file.

    Other integration tests that just intercepts echo can also use the pypyr.steps.append technique - these will be updated incrementally over time.

    enhancement 
    opened by yaythomas 0
  • JSON Schema for pypyr YAML files

    JSON Schema for pypyr YAML files

    Having JSON schema for pypyr YAML configuration files would be handy. Many editors (at least my neovim) allow to configure runtime validation of authored file if JSON schema is available.

    My question is - is there any JSON schema for pypyr already in place? My quick research did not reveal any.

    If it does not exist, I am considering authoring one (using pydantic allows to create such a schema very simply).

    enhancement type: env on hold 
    opened by vlcinsky 5
  • step: pypyr.steps.sleep

    step: pypyr.steps.sleep

    step that sleeps/pauses execution for given number of seconds.

    step context input:

        sleep: 5 # input is seconds, decimals allowed.
    

    Supports substitutions.

    Defaults to 5.

    enhancement type: step 
    opened by yaythomas 0
  • pypyr.steps.find

    pypyr.steps.find

    Step to find a specified string, string expression (or maybe regex) in a file.

    If not file, specify path to recurse all files in that path looking for file contents that match the pattern.

    Return True, filename/position. Have raise/throw attribute to raise error if not found.

    enhancement type: step 
    opened by yaythomas 0
Releases(v5.7.1)
  • v5.7.1(Oct 25, 2022)

    summary

    • maintenance release, no new features
    • bug fix for pypyr.steps.filereplace to honor flat format :ff directive
    • python 3.11 compatibility confirmed
      • on python 3.11 runtimes, the tomli dependency won't install separately anymore since it's part of stdlib now

    what's changed

    • pypyr.steps.filereplace get_formatted X2 by @yaythomas in https://github.com/pypyr/pypyr/pull/304
    • new codecov uploader by @yaythomas in https://github.com/pypyr/pypyr/pull/305
    • py 3.11 & conditional tomli dep by @yaythomas in https://github.com/pypyr/pypyr/pull/306
    • Bump version: 5.7.0 → 5.7.1 by @yaythomas in https://github.com/pypyr/pypyr/pull/307

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.7.0...v5.7.1

    Source code(tar.gz)
    Source code(zip)
  • v5.7.0(Oct 20, 2022)

    summary

    • New switch step for IF-ELSE style branching in your pipelines!
    • New argskwargs parser to combine plain args and key-value pairs (key=value) from the cli
    • keyvaluepairs and dict parsers now support having = in the value, so you can have key=one+one=two parse to {'key': 'one+one=two'}
      • these parsers now also support passing args with no =, in which case arg1 becomes {'arg1': ''}

    what's changed

    • switch by @yaythomas in https://github.com/pypyr/pypyr/pull/299
    • typing on dsl.cof by @yaythomas in https://github.com/pypyr/pypyr/pull/300
    • gh action checkout v3 by @yaythomas in https://github.com/pypyr/pypyr/pull/301
    • argskwargs parser & partitioning on = for kvp by @yaythomas in https://github.com/pypyr/pypyr/pull/302

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.6.0...v5.7.0

    Source code(tar.gz)
    Source code(zip)
  • v5.6.0(Oct 5, 2022)

    summary

    • you can now use a custom error message when pypyr.steps.assert raises an exception.
    • new pypyr.steps.venv step to create venvs in parallel from yaml or toml config.
    • new venv-create pipeline so you can provision venvs concurrently from declarative config without having to write your own pipeline or script.
    • pypyr is now built & packaged by the excellent PEP517 compliant flit.

    what's changed

    • flake8 new version line length by @yaythomas in https://github.com/pypyr/pypyr/pull/286
    • from setup.py to flit by @yaythomas in https://github.com/pypyr/pypyr/pull/290
    • remove redundant dict on parser context init by @yaythomas in https://github.com/pypyr/pypyr/pull/292
    • venv creator & log to stdout by @yaythomas in https://github.com/pypyr/pypyr/pull/296
    • ops/build uses venv for publish verify by @yaythomas in https://github.com/pypyr/pypyr/pull/297

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.5.0...v5.6.0

    Source code(tar.gz)
    Source code(zip)
  • v5.5.0(May 26, 2022)

    summary

    • cmdOut for pypyr.steps.cmd & pypyr.steps.shell gets dotted attribute access.
      • instead of cmdOut[returncode] access value like this: cmdOut.returncode
      • backwards compatible - the old-style dict-like accessors will still work.
      • closes #272
    • Introduce pypyr.steps.cmds & pypyr.steps.shells to run programs or shell statements asynchronously as parallel (concurrent) sub-processes.
      • cmdOut for these works as with the serial versions of the steps using the new dotted attribute access introduced in this release.
      • closes #273
    • Set default encoding for all the cmd/shell steps using default_cmd_encoding in config. You can use the $env variable PYPYR_CMD_ENCODING to initialize this value, in addition to the usual config files.
      • closes #275

    what's changed

    • async cmds + shells by @yaythomas in https://github.com/pypyr/pypyr/pull/274
    • release 5.5 & default_cmd_encoding for serial by @yaythomas in https://github.com/pypyr/pypyr/pull/276

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.4.0...v5.5.0

    Source code(tar.gz)
    Source code(zip)
  • v5.4.0(Apr 14, 2022)

    Summary

    • pypyr.steps.cmd & pypyr.steps.shell now also takes a list input to run multiple commands/shell statements in the same step!
    - name: pypyr.steps.cmd
      in:
        cmd:
          - echo 1
          - echo 2
    
    - name: pypyr.steps.cmd
      in:
        cmd:
          run:
            - echo 3
            - echo 4
          save: False
          cwd: mydir/subdir
    
    • Both the cmd and shell steps expanded to allow:
      • decode output in different encodings
      • save output as raw bytes or as encoded text.
      • File output for stdout/stderr.
        • File create mode of append or overwrite.
      • stderr can redirect to /dev/stdout
      • Both stdout & stderr can redirect to /dev/null
    #  when save: True
    - name: pypyr.steps.cmd
      comment: when save is True
      in:
        cmd:
          run: curl https://myurl.blah/diblah
          save: True
          cwd: .
          bytes: False
          encoding: utf-8
    
    #  when save: False (this is default)
    - name: pypyr.steps.cmd
      comment: when save is False (the default when `save` not set)
      in:
        cmd:
          run: curl --cert certfile --key keyfile https://myurl.blah/diblah
          cwd: ..
          stdout: ./path/out.txt
          stderr: ./path/err.txt
          append: False
    

    What's Changed

    • cmd & shell list input by @yaythomas in https://github.com/pypyr/pypyr/pull/271

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.3.0...v5.4.0

    Source code(tar.gz)
    Source code(zip)
  • v5.3.0(Mar 9, 2022)

    Summary

    Create shortcuts to your pypyr run commands & their input args. This is handy for creating short & sweet aliases for longer pipeline run commands.

    So if you have a pipeline you normally run like this:

    $ pypyr arb/my-pipeline arg1=1234 arg2="/path/to long/annoying path to type/x" arg3="arb'hello"
    

    You can create a shortcut alias for this like so:

    shortcuts:
      my-shortcut:
        pipeline: arb/my-pipeline
        args:
          arg1: 1234
          arg2: /path/to long/annoying path to type/x
          arg3: "arb'hello"
    

    And now for the same pipeline + inputs you can just run:

    $ pypyr my-shortcut
    

    You can create your shortcut definition in pyproject.toml or in the pypyr yaml config file.

    You can make your shortcuts project specific, or for the entire user profile, or globally for your entire system.

    What's Changed

    • Shortcuts by @yaythomas in https://github.com/pypyr/pypyr/pull/268
    • Bump version: 5.2.0 → 5.3.0 by @yaythomas in https://github.com/pypyr/pypyr/pull/269

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.2.0...v5.3.0

    Source code(tar.gz)
    Source code(zip)
  • v5.2.0(Feb 19, 2022)

    Summary

    • pypyr is now configurable with yaml or pyproject.toml!
    • Explicitly over-ride encoding on any filesystem steps

    What's Changed

    • friendly err on malformed pipeline. by @yaythomas in https://github.com/pypyr/pypyr/pull/250
    • config & encoding by @yaythomas in https://github.com/pypyr/pypyr/pull/255
    • file encoding from config by @yaythomas in https://github.com/pypyr/pypyr/pull/259
    • windows tests & config platform indicators by @yaythomas in https://github.com/pypyr/pypyr/pull/261
    • typings for config, platform by @yaythomas in https://github.com/pypyr/pypyr/pull/262
    • FakePath PathLike & parent also fake by @yaythomas in https://github.com/pypyr/pypyr/pull/264
    • pypyr.steps.python & version bump 5.2.0 by @yaythomas in https://github.com/pypyr/pypyr/pull/266

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.1.0...v5.2.0

    Source code(tar.gz)
    Source code(zip)
  • v5.1.0(Dec 12, 2021)

    Finally, toml, has come, to pypyr! 🎆

    • pypyr.parser.tomlfile to initialise pipeline context with a toml file
    • pypyr.steps.fetchtoml to read toml file into context as an object with typing
    • pypyr.steps.filewritetoml to write specified context to output toml file - lets you create toml files on-the-fly programmatically.
    • pypyr.steps.fileformattoml to read input tomls specified by path, glob or list of paths, replace {formatting expressions} and write to output file(s).

    Working with files:

    • pypyr.steps.fileread to read files in text or binary mode into context.
    • pypyr.steps.filewrite to write or append to file in text or binary modes.

    What's Changed

    • pypyr.steps.fileread by @yaythomas in https://github.com/pypyr/pypyr/pull/244
    • pypyr.steps.filewrite by @yaythomas in https://github.com/pypyr/pypyr/pull/245
    • toml by @yaythomas in https://github.com/pypyr/pypyr/pull/246

    Full Changelog: https://github.com/pypyr/pypyr/compare/v5.0.0...v5.1.0

    Source code(tar.gz)
    Source code(zip)
  • v5.0.0(Nov 20, 2021)

    Implement adr2 relative pipelines + api changes.

    In brief, this release lets pipelines reference custom modules & child pipelines relative to the pipeline itself, rather than the current directory. This lets you create portable, re-usable & composable pipeline libraries.

    Breaking Changes

    This is a major version increment because it comes with BREAKING CHANGES:

    1. API: pipelinerunner.run() replaces both pipelinerunner.main() and pipelinerunner.main_with_context()
    2. API: def get_pipeline_definition(pipeline_name, working_directory) signature for custom pype loaders changes to def get_pipeline_definition(pipeline_name, parent)
    3. CLI: the —dir flag now only sets the directory for ad hoc custom Python modules, it does NOT also set the directory for pipelines anymore
    4. Final removal of deprecated get_formatted_iterable, get_formatted_string #195 & pypyr.steps.contextset #184. Where previously these would just give deprecation warnings, they are now completely removed.
    5. pypyr.pypeloaders.fileloader renamed pypyr.loaders.file

    Non-Breaking Changes

    • You can now access the current pipeline’s metadata & loader information from within a pipeline with context.current_pipeline
    • Improve handling of absolute paths in file loader only to search path once, rather than unnecessarily go through the same relative path lookup sequence with the same path.
    • Typing support added for the pypyr API entrypoint.

    Detailed Technical Breakdown:

    • Introduce new classes to model pipeline payload, rather than just using the bare dict-like yaml directly.
      • PipelineInfo - pipeline metadata set by loader. This maintains a pipeline’s parent/path info so that child pipelines can load relative to the parent.
      • PipelineDefinition - this wraps the pipeline payload and its metadata (PipelineInfo) to allow pypyr to cache it all with one reference
    • Add new Pipeline class for the run-time properties of a single run.
      • The Pipeline references the shared cached PipelineDefinition.
      • Move run + load_and_run logic from pypyr.pipelinerunner to the new Pipeline class. This massively streamlines the pipeline invocation process, since run/load_and_run can just operate on the shared Pipeline state rather than sling a bunch of args between different functions as before.
    • Add a call-stack of running Pipeline instances on Context. i.e Parent -> child1 -> child2 where the root pipeline calls child pipelines via pype
      • Add current_pipeline attribute to Context, controlled with a context manager to scope itself to an individual pipeline run’s lifespan.
        • This means that steps can access current pipeline’s properties.
        • This allows pypyr.steps.pype to find the current (i.e parent) pipeline’s metadata such as path, to load child pipeline relative to the calling pipeline’s location.
        • When a child pipeline completes, the calling pipeline (i.e the previous Pipeline in the call-stack) becomes the current pipeline
    • Amend pypyr.steps.pype to instantiate Pipeline object to load_and_run() child.
      • pype now deals the context.current_pipeline.pipeline_definition.info metadata to work out whether to cascade parent path down to child, so child can load relative to the parent.
      • Notably, the parent loader now cascades to the child, so pipeline authors don’t need explicitly to set the same custom loader repeatedly for each child.
      • Given the context manager controlling current pipeline scope in Pipeline.load_and_run_pipeline remove the clumsy side-shuffle for pipeline_name, working_dir to swap out these values as child pipe runs and swap these back when it completes/errors.
    • Remove global PipelineCache. Replace with distinct pipeline cache per loader.
      • This resolves a long standing limitation where pypyr assumed unique pipeline names across all loaders.
      • The per-loader pipeline cache stores PipelineDefinition objects.
      • Introduce Loader class, which wraps loader & its pipeline cache. Loader is what the LoaderCache caches.
      • Thus LoaderCache -> Loader -> _pipelineCache -> PipelineDefinition
    • File loader has a private file cache keyed on absolute path of file
      • This is to prevent >1 load+parse where the same underlying pipeline.yaml file has different names in the loader’s pipeline cache
        • e.g (name=‘dir/mypipe’, parent=None) and (name=‘mypipe’, parent=‘dir’) both resolve to dir/mypipe.yaml
      • Caching a reference to the PipelineDefinition object, so not duplicating memory
    • Improve handling of absolute paths in file loader only to search path X1, rather than unecessarilly go through the same relative path lookup sequence with the same path.
    • File loader get_pipeline_definition now returns a PipelineDefinition with PipelineFileInfo to store file-system specific metadata for the loader pipeline
    • Remove working_dir global. The py_dir input on run() now refers ONLY to module paths, NOT pipeline locations.
      • The CLI —dir flag, or py_dir input on run() basically adds the specified directory to sys.path.
    • Add current pipeline’s parent directory to sys.path on load. This allows child pipelines to resolve custom modules relative to itself.
    • pypyr.dsl.Step does not need StepsRunner anymore, because it can get it from the context.current_pipeline instead.
    • Recode (some) integration tests to take advantage of list pypyr.steps.append step and checking that for output on return context rather than intercepting logger.NOTIFY.
    • Rename master branch to main in CI/CD GitHub actions
    • Add typing annotations to the public run() function and the Pipeline class public accessors. The idea is NOT to type pypyr exhaustively, just to provide annotations for the sensible/likely entrypoint to enhance API user experience. Include py.typed in pypyr package.
    • Remedy packaging snafu where tests.common was deploying alongside pypyr because exclude condition in find_packages didn't include wildcard for sub packages.

    What's Changed

    • Relative pipelines & API run() replaces main/main_with_context by @yaythomas in https://github.com/pypyr/pypyr/pull/243

    Full Changelog: https://github.com/pypyr/pypyr/compare/v4.6.0...v5.0.0

    Source code(tar.gz)
    Source code(zip)
  • v4.6.0(Oct 6, 2021)

    What's Changed

    • Fix failing ops/build (failing on linting issues and locale dependent failure) by @vlcinsky in https://github.com/pypyr/pypyr/pull/226
    • Evaluate step skip only if the run check returned true by @Reskov in https://github.com/pypyr/pypyr/pull/234
    • pypyr.steps.append & pypyr.steps.add by @yaythomas in https://github.com/pypyr/pypyr/pull/235
    • shorter alias set for contextsetf. py 3.10. by @yaythomas in https://github.com/pypyr/pypyr/pull/238

    New Contributors

    • @vlcinsky made their first contribution in https://github.com/pypyr/pypyr/pull/226

    Full Changelog: https://github.com/pypyr/pypyr/compare/v4.5.0...v4.6.0

    Source code(tar.gz)
    Source code(zip)
  • v4.5.0(Feb 2, 2021)

    • add retry backoff strategies. #216.
    • prevent duplicates in sys.path (#218) on main* api entry points. thread-safe existence check on sys.path list.
    • prevent redundant multiple addition of notify log-level on main* api entry. Move initialization code to package __init__ (#219).
    Source code(tar.gz)
    Source code(zip)
  • v4.4.1(Jan 1, 2021)

    • Context parsers that create an entry in context now initialise to empty rather than None. This means you can directly use something like {argList}, {argDict} and argString (initialising respectively to [], {}, '') directly for things like foreach loops without having to worry about None checks. Your existing truthy checks for these values will work as before. closes #214.
    Source code(tar.gz)
    Source code(zip)
  • v4.4.0(Dec 26, 2020)

    • New step pypyr.steps.pyimport to import references to the py-string namespace.
      • This includes an underlying api signature change by removal of pypyr.utils.expressions.eval_string(), but this is sufficiently far down the call-chain that it shouldn’t affect any normal pipeline operator or api consumer.
      • See #110 for details.
    • pypyr.steps.contextclearall wipes pyimport imported references in addition to the key/values inside context.
    • Simplify pypyr.steps.py syntax by allowing a new py (rather than pycode) input. This allows pipeline authors to use context key names directly as variables, rather than have to specify them as keys in context (my_var vs context[‘my_var’]).
      • see #204 for details on simplifying the pypyr.steps.py syntax.
      • the old pycode will keep on working in the same way, so no need to worry about backwards compatibility for your existing pipelines.
    • Allow substitutions on Retry max. Resolves #207. Excellent bug find & fix by @Reskov, much thanks 🙌 🙌 🙌 as ever for a superb contribution! 🔥 🔥 🔥
    • foreach can now use any iterable, including generators. Closes #209
    Source code(tar.gz)
    Source code(zip)
  • v4.3.0(Nov 12, 2020)

    • Streamline main entrypoint API. close #201.
      • main() allows consumer to set pype loader, rather than having to drop further down into api to load_and_run_pipeline()
      • new main_with_context() to input dict to initialise context, and bypass context_parser entirely. Also returns the Context object after pipeline run completes.
      • make all non-essential args optional to allow minimal calls to main entrypoint without having to add optional=None style inputs.
      • This is fully backwards compatible.
    • pypyr.steps.pype
      • defaults useParentContext to False is pipeArgs specified.
      • pipeArgs shlex-es input string
      • set pipeline_name on child pipeline rather than use parent pipeline name
    • working_dir uses Path object rather than string
    • pypyr.steps.echo remove redundant string check.
    Source code(tar.gz)
    Source code(zip)
  • v4.2.0(Oct 30, 2020)

    • python 3.9 compatibility officially confirmed. All good. Woo!
    • New Flat ff and Recursive rf formatting specifiers. #195
      • Maintain backwards compatibility for all current formatting functionality.
      • The get_formatted_string, get_formatted_iterable & get_processed_string methods on Context() are now all deprecated. Use get_formatted_value instead. All of the deprecated functions will keep on working as before, but will print a WARN to the output.
      • get_formatted and get_formatted_value remain the official best ways of getting formatted values.
    • New !jsonify yaml special tag directive. This converts context object to a json string. #197.
    • New pypyr.steps.jsonparse step. Parse and deserialise a json string into Context. #199
    • Bare pypyr.steps.assert syntax. #196
    • Add new utils.asserts for extra flexibility with nested key validation.
    • PyStrings optimised. Not so that you'd notice, but still, it'll be a smidge faster.
    • Friendlier error messages when yaml/json context parsers reject documents that do NOT have a mapping at root.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.0(Aug 30, 2020)

    • Better description output - add (skipping) to output if the step is not running because run is False or skip is True. Ref #158.
    • First release published from shiny new GitHub Action CI/CD!
    • Add License to wheel published to pypi
    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(Aug 17, 2020)

    • BREAKING CHANGE: final deprecation of in args persisting after step execution. For full discussion please see #177. Preview functionality where setting environment variable $PYPYR_IN_CLEAN = 1 is now the default. You can remove the $PYPYR_IN_CLEAN variable from this release onwards. Henceforth pypyr removes in args from context after step completes. If you want to persist values in context, use pypyr.steps.contextsetf, pypyr.steps.contextcopy or pypyr.steps.default.
    • BREAKING CHANGE: pypyr.steps.assert raises an AssertionError rather than a ContextError. Ref #188.
    • The cli now defaults to a simplified log format that is way less noisy. If you still want the full, old style logs, run pypyr with --log 25 set explicitly. pypyr my pipe --log 25. Ref #187
      • description decorator output also simplified to echo only your actual custom text.
    • Failure Groups now support Stop instructions. This allows you to exit the failure handler reporting success, so pypyr can report success if you consider the error condition handled. Ref #175.
    • pypyr.steps.contextset renamed pypyr.steps.contextcopy. The old name will keep on working, but with a deprecation warning. Ref #184.
    • internal: improvements to integration testing error reporting.
    Source code(tar.gz)
    Source code(zip)
  • v3.2.2(Aug 13, 2020)

    • Line + Column Number count from 1 not 0. fix #151
    • update README to point at new website pypyr.io.
    • update CONTRIBUTING to point at new website.
    • update documentation and pypi setup with new repo name.
    Source code(tar.gz)
    Source code(zip)
  • v3.2.1(Apr 14, 2020)

    • Allow non-string keys on context.get_formatted_iterable and context merge (including the step pypyr.steps.contextmerge). Resolves #179. Much thanks to @Reskov for a detailed & helpful bug report!
    Source code(tar.gz)
    Source code(zip)
  • v3.2.0(Apr 4, 2020)

    ATTENTION: Please start preparing all your pipelines to be ready for the next major release where in context arguments will be removed from context on step completion. This shouldn't be too much of an issue. For any steps where you need to set enduring context, use contextsetf or contextset.

    • Allow using a call control-of-flow instruction nested inside while/for/retry loops. Previously a nested call instruction would over-write the parent call's configuration and thus create confusion on the next loop round the parent. Much thanks 🙏 to @Reskov for identifying this tricky little recursion issue & assistance in resolution. Close #176
    • in context parameters only available for the duration of the step they decorate. Previously in items would be added to context and stay in context after the step finishes. This is a potentially breaking change for existing pipelines, but worth it, full discussion of reasons here #177.
      • For this release this change in functionality is not the default, but enabled by setting the following environment variable PYPYR_IN_CLEAN = 1. You can set it selectively on a single pypyr run using standard shell syntax like: $ PYPYR_IN_CLEAN=1 pypyr mypipeline arg1 arg2
      • README updates to document #177 resolution as the default behaviour - in context parameters cleaned from context on step completion to prepare for next major release with the breaking change..
    Source code(tar.gz)
    Source code(zip)
  • v3.1.0(Feb 11, 2020)

    pypyr.log.logger.set_root_logger(log_level, log_path) call moved from pipelinerunner.main() to cli.main()

    this has the side-effect that pipelinerunner.main() signature has changed, with log_level and log_path removed. API consumers should update.

    Reason being API consumers should set their own log handlers, since handler configuration should be the prerogative of the calling application, not the invoked library.

    Re #172, #173

    Source code(tar.gz)
    Source code(zip)
  • v3.0.3(Jan 29, 2020)

    • Retry Decorator correctly handles errors raised from .call or .jump steps when checking error type with retryOn and stopOn. Close #170 - thanks to @Reskov for reporting!
    • add tarball to PyPI deploy artefacts.
    Source code(tar.gz)
    Source code(zip)
  • v3.0.2(Nov 18, 2019)

    • When importing a step that does exist (mystep.blah), a failing import mymodule.blah inside mystep.blah would give a misleading error and hide the actual source of the problem. Fixes #166. Eternal thanks to @irancati for raising the issue!
    Source code(tar.gz)
    Source code(zip)
  • v3.0.1(Sep 13, 2019)

    With a million thank yous 🙏 to @Reskov for these much needed fixes/enhancements! 🎆

    • Resolves issue #149. pypyr.steps.debug now supports outputting any type, including PyString & SicString
    • Resolve pypyr.steps.contextmerge issue to support merging any type, including PyString & SicString.
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Aug 22, 2019)

    • BREAKING CHANGE: Remove deprecated step input context for (ref #118)
      • assert
      • env
      • fetchjson
      • fetchyaml
      • fileformat
      • fileformatjson
      • fileformatyaml
      • filereplace
      • tar
    • BREAKING CHANGE: cli context input now uses standard cli spacing rather than needing to str close everything. Closes #94
      • So where you had:
      • pypyr mypipeline "key1=value1,key2=value2”
      • Now instead:
      • pypyr mypipeline key1=value1 key2=value2
    • BREAKING CHANGE: Big API changes to pipelinerunner, stepsrunner and Step
    • BREAKING CHANGE: pypyr.parser.commas replaced by pypyr.parser.keys
    • Caching pipelines, context parsers, loaders, step run functions. Basically cache anything that loads from disk. Closes #126
    • Custom step groups. Closes #126
    • Jump, Call, and Stop ref #128 #127 #123
    • The ./pipelines default directory location no longer necessary. To run a pipeline ./dir/mypipe.yaml, just go pypyr dir/mypipe
    • README+help updates for all of the above
    Source code(tar.gz)
    Source code(zip)
  • v2.11.0(Aug 6, 2019)

    This one is all @Reskov 🎉 eternal gratitude!

    • new NOTIFY logging level added. this level is much less verbose than even INFO, and only displays explicits echos that you specify in your pipeline, and also the description from the step decorator. This should make it much easier to see what your pipeline is doing without the extra detail that can make things confusing and harder to follow. #132
    • under-the-hood api improvement for more performant logging #152
    Source code(tar.gz)
    Source code(zip)
  • v2.10.0(Jul 10, 2019)

    With eternal thanks to @Reskov 🎉 for next level super useful improved error handling! 🙇

    • #139 - pypyr now saves all run-time errors to context under runErrors. This means subsequent steps can use error information from previous steps.
    • #135 - error information now include the pipeline source yaml line+col numbers
    Source code(tar.gz)
    Source code(zip)
  • v2.9.0(Jun 14, 2019)

    • pypyr.steps.glob lets you get a list of paths from a glob, and combine different globs into a combined output result list. Issue #145
    • with much thanks to @Reskov 🎉 , resolves issue #143 - merge step now supports special tag directives
    Source code(tar.gz)
    Source code(zip)
  • v2.8.0(Mar 24, 2019)

    • new Retry decorator allows steps to retry automatically when step encounters an error. ref #130
    • fix run_step AttributeError might have caught or hidden AttributeErrors in the step code itself, rather than just when the step module didn't contain a run_step function. ref #129
    • while loop checks stop condition only at end of each iteration. previously if a stop condition evaluated True even before the loop started the loop wouldn't run at all. now the loop will run once. If you don't want the step to run, use the skip: True decorator instead.
    • it's now easier to write infinite While loops with an explicit stop: False condition
    Source code(tar.gz)
    Source code(zip)
  • v2.7.0(Mar 21, 2019)

Owner
pypyr
pypyr is an open-source lightweight task-runner for automation pipelines in easy yaml
pypyr
Basic python tools to generate shellcode runner in vba

vba_bin_runner Basic python tools to generate shellcode runner in vba. The stub use ZwAllocateVirtualMemory to allocate memory, RtlMoveMemory to write

null 4 Aug 24, 2021
Shellcode runner to execute malicious payload and bypass AV

buffshark-shellcode-runner Python Shellcode Runner to execute malicious payload and bypass AV This script utilizes mmap(for linux) and win api wrapper

Momo Lenard 9 Dec 29, 2022
Yts-cli-streamer - A CLI movie streaming client which works on yts.mx API written in python

YTSP It is a CLI movie streaming client which works on yts.mx API written in pyt

null 1 Feb 5, 2022
a CLI that provides a generic automation layer for assessing the security of ML models

a CLI that provides a generic automation layer for assessing the security of ML models

Microsoft Azure 268 May 9, 2021
Simple CLI interface for linear task manager

Linear CLI (Unmaintained) Simple CLI interface for linear task manager Usage Install: pip install linearcli Setup: Generate a pe

Mike Lyons 1 Jan 7, 2022
Task-manager-CLI with Priority Modification

Task-manager-CLI with Priority Modification The functions for the app have been written in task.py file. 1. Install Node.js This project requires Node

null 1 Jan 21, 2022
Sink is a CLI tool that allows users to synchronize their local folders to their Google Drives. It is similar to the Git CLI and allows fast and reliable syncs with the drive.

Sink is a CLI synchronisation tool that enables a user to synchronise local system files and folders with their Google Drives. It follows a git C

Yash Thakre 16 May 29, 2022
flora-dev-cli (fd-cli) is command line interface software to interact with flora blockchain.

Install git clone https://github.com/Flora-Network/fd-cli.git cd fd-cli python3 -m venv venv source venv/bin/activate pip install -e . --extra-index-u

null 14 Sep 11, 2022
AWS Interactive CLI - Allows you to execute a complex AWS commands by chaining one or more other AWS CLI dependency

AWS Interactive CLI - Allows you to execute a complex AWS commands by chaining one or more other AWS CLI dependency

Rafael Torres 2 Dec 10, 2021
Python-Stock-Info-CLI: Get stock info through CLI by passing stock ticker.

Python-Stock-Info-CLI Get stock info through CLI by passing stock ticker. Installation Use the following command to install the required modules at on

Ayush Soni 1 Nov 5, 2021
A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool

Privateer A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool How

Shreyash Chavan 2 Apr 4, 2022
[WIP]An ani-cli like cli tool for movies and webseries

mov-cli A cli to browse and watch movies. Installation This project is a work in progress. However, you can try it out python git clone https://github

null 166 Dec 30, 2022
This CLI give the possibility to do a queries in Star Wars API and returns a JSON in a terminal.

Star Wars CLI (swcli) This CLI give the possibility to do a queries in Star Wars API and returns a JSON in a terminal. Install $ pip install swcli Qu

Pery Lemke 5 Nov 5, 2021
Simple CLI prompt for easy I/O with OpenAI's API

openai-cli-prompt Simple CLI prompt for easy I/O with OpenAI's API Quickstart Create a .env file with: OPENAI_API_KEY=<Your OpenAI API Key> Configure

Erik Nomitch 1 Oct 12, 2021
Python API and CLI for the ikea IDÅSEN desk.

idasen This is a heavily modified fork of rhyst/idasen-controller. The IDÅSEN is an electric sitting standing desk with a Linak controller sold by ike

Alex 79 Dec 14, 2022
CLI translator based on Google translate API

Translate-CLI CLI переводчик основанный на Google translate API как пользоваться ? запустить в консоли скомпилированный скрипт (exe - windows, bin - l

null 7 Aug 13, 2022
A simple cli utility for importing or exporting dashboard json definitions using the Grafana HTTP API.

grafana-dashboard-manager A simple cli utility for importing or exporting dashboard json definitions using the Grafana HTTP API. This may be useful fo

Beam Connectivity 31 Jan 6, 2023