A software manager for easy development and distribution of Python code

Overview

Piper

A software manager for easy development and distribution of Python code.

The main features that Piper adds to Python are:

  • Support for large-scale, multi-package projects
  • Reproducibility (clear, transparent dependency management)
  • Robust development-lifecycle, from blueprinting to distribution

Piper is inspired by what Maven is for Java and uses Pip and Virtual Environments.

Why Piper

Python is great in many things, particularly for scripting. But it is powerful enough to create complex software too. Still, when doing so, it lacks some robustness and quickness.

Instead:

  • Piper lets you forget about repetitive setup of Python projects, with creation of Virtual Environments, issues with imports, PYTHONPATH, folder structures. It does all of this for you.

  • Piper ensures that when you run a Python script, all its requirements are implicitly installed. Along with reproducibility, this makes room for easy collaboration between developers.

  • Also, while with standard tools it's hard to have multiple packages, one requiring the other, with Piper is a matter of few YAML files. You can split your project in several packages instead of having tons of requirements, code replication and/or single package-monoliths.

All in all, Piper lets you design proper structure and modularity for your code. No worries about imports and low-level stuff.

Example of Piper Project

You can find an example of Piper project here.

Installation

With Python >= 3.7, simply run

pip install piper-tools

From now on, you can use piper from command line.

Basic Usage

Once you have defined your structure of Pipes (submodules of your project), for instance like in the example or like this

complexproject/
    common/
        package/
        pipe.yml
    microservices/
        first/
            package/
            pipe.yml
        second/
            package/
            pipe.yml
        pipe.yml
    pipe.yml

you can run

piper setup

in the main folder, where the first pipe.yml file is located.

This will:

  • Understand the dependencies
  • Create the Virtual Environments
  • Install the requirements
  • Install the packages themselves in development mode (so that you can import them in their Virtual Environments)

You can also create a distribution package with piper pack [commands], which is the equivalent of python setup.py [commands], for instance:

piper pack sdist

in the folder of the Pipe you want to pack. This will create the distribution in the ./build/package/dist folder.

In any moment, you can run

piper clean

to destroy the Virtual Environments and other build objects.

Comments
  • travel setup: faulty handling of whitespaces in filepath

    travel setup: faulty handling of whitespaces in filepath

    Bug affecting travel setup happening when the filepath of the Python installation contains whitespaces.

    C:\Users\a.migotto\Repositories\travel_test\src>travel setup
    
    ============================================================
    ===                       backend                        ===
    ============================================================
    Creating the virtual environment for venv-backend...
    'C:\Program' is not recognized as an internal or external command,
    operable program or batch file.
    
    bug 
    opened by amigotto 0
  • Download of new Python versions

    Download of new Python versions

    After experimenting a bit with adding Python versions on Travel, it probably should be tweaked a bit, mainly because it forces you to always have that version of Python in the system, which can be somewhat limiting (e.g., we need a specific image, say in a step of a Bitbucket pipeline, but it doesn't have the right Python version)

    I suggest using python-build, a Pyenv plugin that takes care of the whole installation procedure. As you can see from the README, it does not need Pyenv to be installed, but can operate standalone like this:

    python-build <python-version> <output-directory>
    

    where <python-version> is the version of Python to install (e.g., 3.7.15) and <output-directory> is where to install it.

    It should be quite easy (maybe with Git submodules?) to go and reference Pyenv's plugin directly, so you can keep it up to date, and at the same time change the Travel command to add Python versions to something like:

    1. ​travel config add python <python-version> [<python-location>]​
    2. If the Python version has already been added, ignore the command (or add a flag to force the override)
    3. Is python-location available?
      • If yes, look for the Python binary from there and:
        • if it is not present, raise a FileNotFoundError
        • if it is present, but the binary does not match <python-version>, raise a ValueError
      • If no, run python-build on that version to download the binary to some known internal path
    4. Add that (version, binary) pair to Travel config.yml

    N.B.: The above flow chart works only on UNIX-like systems. For Windows systems, we should look for an equivalent standalone plugin in Pyenv-win.

    opened by DiTo97 0
  • Support for non-PyPI packages

    Support for non-PyPI packages

    Currently, Travel does not allow requirements not hosted on PyPI (e.g., Git wheels).

    For instance, it is not possible to install the detectron2 package by Meta Research by following the installation instructions, whereas it would be regularly installed with pip using the following command:

    pip install git+https://github.com/facebookresearch/detectron2.git
    

    It is a bit of a corner case, but it has popped up a few times in my experience. The only workaround is a bit clumsy:

    1. Issue travel setup without any of these packages involved
    2. Install all non-PyPI packages manually with pip in all environments
    3. Issue pip freeze in all environments to the corresponding requirements*.txt file

    N.B.: After step 3. you have to delete the -e hook to the Travel bag from all requirements*.txt files.

    I suspect the issue lies in what kinds of package names are allowed in pip_sanitizer.py, since adding a package like the one above (git+https://github.com/facebookresearch/detectron2.git) to the requirements of a Travel bag leads to:

    Traceback (most recent call last):
        ...
    AssertionError
    

    which is likely caused by asserter.py

    opened by DiTo97 0
  • Travel tasks only accept `==` modifier

    Travel tasks only accept `==` modifier

    Generally, the definition of a Travel task has the following form:

    tasks:
      <phase>:
        <step>:
          - task: <travel-task-package><modifier><version>
            name: <travel-task-name>
            config:
              .
              .
              .
    

    where <travel-task-package> can either be a published package (e.g., on PyPI) or a local bag.

    The problem is that, unless the optional arg python_module is provided, the only accepted <modifier> is ==, whereas all of them should be supported as in the function sanitize_versioned_package.

    At the moment, a Travel task like the following crashes:

    tasks:
      <phase>:
        <step>:
          - task: traveltask_google_artifact_registry~=0.0.1
            name: <travel-task-name>
            config:
              .
              .
              .
    

    with the following error:

    Traceback (most recent call last):
        ...
    ModuleNotFoundError: No module named 'traveltask_google_artifact_registry~=0'
    

    Implementing a fix should be quite easy since the file pip_sanitizers.py already has all the logic to check for different modifiers, and assert that any resulting split is a valid package.

    opened by DiTo97 0
  • spaces in project path problem

    spaces in project path problem

    When running 'travel setup' in a project and its path contains a directory with spaces in the name, an error occurs. Path Example: 'C/users/gianni/My cool project/src/'

    It's a very clear problem with an easy workaround but worth noticing for further development. The quick workaround: do not use spaces in directory names (a good pratice in every code)

    opened by Gogo693 0
  • debug option not working with setup command

    debug option not working with setup command

    Using the --debug option with travel setup seems to have no effect, no further detail about the error is displayed.

    C:\Users\a.migotto\Repositories\travel_test\src>travel setup
    
    ============================================================
    ===                       backend                        ===
    ============================================================
    Creating the virtual environment for venv-backend...
    'C:\Program' is not recognized as an internal or external command,
    operable program or batch file.
    
    C:\Users\a.migotto\Repositories\travel_test\src>travel --debug setup
    
    ============================================================
    ===                       backend                        ===
    ============================================================
    Creating the virtual environment for venv-backend...
    'C:\Program' is not recognized as an internal or external command,
    operable program or batch file.
    
    bug 
    opened by amigotto 0
  • Bug in: travel plan

    Bug in: travel plan

    Using the following travel.yml the travel plan projectName returns the following error:

    C:\Users\g.maccagni\Desktop\Workspace>travel --debug plan projectName
    c:\users\g.maccagni\appdata\local\programs\python\python39\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
      warnings.warn(
    
    Traceback (most recent call last):
      File "c:\users\g.maccagni\appdata\local\programs\python\python39\lib\site-packages\travel\__main__.py", line 65, in main
        arguments.action(arguments, remainder)
      File "c:\users\g.maccagni\appdata\local\programs\python\python39\lib\site-packages\travel\__main__.py", line 28, in <lambda>
        plan.set_defaults(action=lambda args, rest: planner.run(args.context, args.name))
      File "c:\users\g.maccagni\appdata\local\programs\python\python39\lib\site-packages\travel\cli\planner.py", line 79, in run
        raise e
      File "c:\users\g.maccagni\appdata\local\programs\python\python39\lib\site-packages\travel\cli\planner.py", line 71, in run
        _generate_breath_first(folder, yml)
      File "c:\users\g.maccagni\appdata\local\programs\python\python39\lib\site-packages\travel\cli\planner.py", line 50, in _generate_breath_first
        Path(bag_file).touch()
      File "c:\users\g.maccagni\appdata\local\programs\python\python39\lib\pathlib.py", line 1304, in touch
        fd = self._raw_open(flags, mode)
      File "c:\users\g.maccagni\appdata\local\programs\python\python39\lib\pathlib.py", line 1116, in _raw_open
        return self._accessor.open(self, flags, mode)
    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\g.maccagni\\Desktop\\Workspace\\projectName\\bag.yml'
    
    

    The used travel.yml is here:

    src:
      common:
        plan: https://bitbucket.org/mlreply/project-travelplan.git
        checkout: plan/library
        config:
          name: common  # The Bag name, i.e. name of the component/folder
          package_name: common  # Optional and optionally different from 'name' of Bag
          description: "The common module"  # Description of the package
    
      microservices:
        group1:
          first:
            plan: https://bitbucket.org/mlreply/project-travelplan.git
            checkout: plan/flask
            config:
              name: first  # The Bag name, i.e. name of the component/folder
              package_name: firstapi  # Optional and optionally different from 'name' of Bag
              description: "The common module"  # Description of the package
              api_prefix: "/api"  # Prefix after the hostname for the APIs
    
          second:
            plan: https://bitbucket.org/mlreply/project-travelplan.git
            checkout: plan/flask
            config:
              name: second  # The Bag name, i.e. name of the component/folder
              package_name: secondapi  # Optional and optionally different from 'name' of Bag
              description: "The common module"  # Description of the package
              api_prefix: "/api"  # Prefix after the hostname for the APIs
    
    bug 
    opened by gmaccagnireply 0
Owner
null
Python Development Workflow for Humans.

Pipenv: Python Development Workflow for Humans [ ~ Dependency Scanning by PyUp.io ~ ] Pipenv is a tool that aims to bring the best of all packaging wo

Python Packaging Authority 23.5k Jan 6, 2023
OS-agnostic, system-level binary package manager and ecosystem

Conda is a cross-platform, language-agnostic binary package manager. It is the package manager used by Anaconda installations, but it may be used for

Conda 5.1k Dec 30, 2022
OS-agnostic, system-level binary package manager and ecosystem

Conda is a cross-platform, language-agnostic binary package manager. It is the package manager used by Anaconda installations, but it may be used for

Conda 5.1k Jan 7, 2023
Package manager based on libdnf and libsolv. Replaces YUM.

Dandified YUM Dandified YUM (DNF) is the next upcoming major version of YUM. It does package management using RPM, libsolv and hawkey libraries. For m

null 1.1k Dec 26, 2022
A flexible package manager that supports multiple versions, configurations, platforms, and compilers.

Spack Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, a

Spack 3.1k Jan 9, 2023
Workon - A simple project manager for conda, windows 10 and vscode

WORK ON A simple project manager for conda, windows 10 and vscode Installation p

Jesus Alan Hernandez Galvan 1 Jan 16, 2022
The Fast Cross-Platform Package Manager

The Fast Cross-Platform Package Manager part of mamba-org Package Manager mamba Package Server quetz Package Builder boa mamba Mamba is a reimplementa

Mamba 4k Dec 30, 2022
Conan - The open-source C/C++ package manager

Conan Decentralized, open-source (MIT), C/C++ package manager. Homepage: https://conan.io/ Github: https://github.com/conan-io/conan Docs: https://doc

Conan.io 6.5k Jan 5, 2023
[DEPRECATED] YUM package manager

⛔ This project is deprecated. Please use DNF, the successor of YUM. YUM Yum is an automatic updater and installer for rpm-based systems. Included prog

null 111 Dec 20, 2022
The delightful package manager for AppImages

⚡️ Zap The delightful package manager for AppImages Report bug · Request feature Looking for the older Zap v1 (Python) implementation? Head over to v1

Srevin Saju 368 Jan 4, 2023
Dotpkg - Package manager for your dotfiles

Dotpkg A package manager for your dotfiles. Usage First make sure to have Python

FW 4 Mar 18, 2022
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 23.1k Jan 1, 2023
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 23.2k Jan 5, 2023
Python PyPi staging server and packaging, testing, release tool

devpi: PyPI server and packaging/testing/release tool This repository contains three packages comprising the core devpi system on the server and clien

null 629 Jan 1, 2023
Install and Run Python Applications in Isolated Environments

pipx — Install and Run Python Applications in Isolated Environments Documentation: https://pipxproject.github.io/pipx/ Source Code: https://github.com

null 5.8k Dec 31, 2022
An installation and dependency system for Python

Pyflow Simple is better than complex - The Zen of Python Pyflow streamlines working with Python projects and files. It's an easy-to-use CLI app with a

David O'Connor 1.2k Dec 23, 2022
A set of tools to keep your pinned Python dependencies fresh.

pip-tools = pip-compile + pip-sync A set of command line tools to help you keep your pip-based packages fresh, even when you've pinned them. You do pi

Jazzband 6.5k Dec 29, 2022
A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

This is a PyPI mirror client according to PEP 381 + PEP 503 http://www.python.org/dev/peps/pep-0381/. bandersnatch >=4.0 supports Linux, MacOSX + Wind

Python Packaging Authority 345 Dec 28, 2022
The Python Package Index

Warehouse Warehouse is the software that powers PyPI. See our development roadmap, documentation, and architectural overview. Getting Started You can

Python Packaging Authority 3.1k Jan 1, 2023