A simple but powerful Python packer to run any project with any virtualenv dependencies anywhwere.

Overview

PyEmpaq

A simple but powerful Python packer to run any project with any virtualenv dependencies anywhwere.

With PyEmpaq you can convert any Python project (see limitations below) into a single .pyz file with all the project's content packed inside.

Packing is super simple, see this demo:

pack-demo

That single file is everything that needs to be distributed. When the final user executes it, the original project will be expanded, its dependencies installed in a virtualenv, and then executed. Note that no special permissions or privileges are required, as everything happens in the user environment.

See that in action:

run-demo

Both the packaging and the execution are fully multiplatorm. This means that you can pack a project in Linux, Windows, Mac or whatever, and it will run ok in Linux, Windows, Mac or whatever. The only requirement is Python to be already installed.

You can try yourself some packed with PyEmpaq examples, very easy, just download any of these files and run it with Python:

logo

You can install pyempaq directly from PyPI; see instructions below.

PyEmpaq is security friendly, there is nothing obscure or secretly shipped when you distribute your project with it: anybody can just open the pyz file (it's just a ZIP) and inspect it.

Limitations:

There are some limitations:

  • Only Python >= 3.7 is supported

  • Only Linux, Windows and Mac is supported

  • Only pip-installable dependencies are supported (from PyPI or whatever).

  • Only dependencies that are pure Python or provide wheels are supported.

All this means that the most majority of the projects could be packed and run by PyEmpaq. If you have any ideas on how to overcome any of these limitations, let's talk!

How does this work?

There are two phases: packing and execution.

The packing phase is executed by the project developer, only once, before distribution. It's a simple step where the developer runs PyEmpaq indicating all needed info, and PyEmpaq will produce a single .pyz file. That's all, and that only file is what is needed for distribution.

In this packing phase, PyEmpaq builds the indicated packed file, putting inside:

  • the payload project, with all the indicated modules and binary files (currently everything from the project, but this will be improved in the future)

  • an unpacker script from PyEmpaq, which will be run during the execution phase

  • few more stuff: some needed infrastructure details for the .pyz to run correctly

After packing, the developer will distribute the packed file, final users will download/receive/get it, and execute it.

In the execution phase all that needs to be done by the final user is to run it using Python, which can be done from the command line (e.g. python3 supergame.pyz) or by doing double click from the file explorer in those systems that relate the .pyz extension to Python (e.g. Windows).

In this execution phase, the unpacker script put by PyEmpaq inside the packed file will be run, running the following steps:

  • check if the needed setup exists from a previous run; if yes, it will just run the payload project with almost no extra work; otherwise...

  • create a directory in the user data dir, and expand the .pyz file there

  • create a virtualenv in that new directory, and install all the payload's project dependencies

  • run the payload's project inside that virtualenv

The verification that the unpacker does to see if has a reusable setup from the past is based on the .pyz timestamp; if it changed (a new file was distributed), a new setup will be created and used.

Command line options

There is one mandatory parameter:

  • source: it needs to point the configuration file or to the directory where the configuration will be located (it will be searched by its default name pyempaq.yaml).

Examples:

  • pyempaq .

  • pyempaq /data/project/

  • pyempaq ~/repo/proj/config.yaml

Note: in the future we will be able to control verbosity, we're not there yet.

The configuration file

All the information that PyEmpaq needs to pack a project comes from a configuration file, which the developer would normally have in the project itself.

The following is the structure of the pyempaq.yaml configuration file:

  • name [mandatory]: the name of the project.

  • basedir [optional, defaults to where the configuration file is located]: the project's base directory.

  • exec [mandatory]: the section where is defined the information to execute the project once unpacked; it holds different subkeys (some subkeys are marked with †: ONE of those keys must be present, but only ONE):

    • script [†]: the filepath of the python script to run; when unpacking PyEmpaq will do python3 SCRIPT.

    • module [†]: the name of the module to invoke; when unpacking PyEmpaq will do python3 -m MODULE.

    • entrypoint [†]: freeform, as a list of strings; when unpacking PyEmpaq will only insert the proper python3 at the beginning: python3 STR1 STR2 ....

    • default-args [optional]: the default arguments to be passed to the script/module/entrypoint (if not overriden when the distributed .pyz is executed); note that this option is not available yet.

  • requirements: a list of filepaths pointing to the requirement files with pip-installable dependencies.

  • dependencies: a list of strings to directly specify packages to be installed by pip without needing to have a requirement file.

All specified filepaths must exist inside the project and must be relative (to the project's base directory), with the exception of basedir itself which can be absolute or relative (to the configuration file location).

The following are examples of different configuration files (which were the ones used to build the packed examples mentioned before):

How to install PyEmpaq

Directly from PyPI:

pip install --user --upgrade --ignore-installed pyempaq

It's handy to install it using pipx, if you have it:

pipx install pyempaq

If you have fades you don't even need to install pyempaq, just run it:

fades -d pyempaq -x pyempaq

In the future there will be more ways to install it. Please open an issue if you desire an installation method (extra points if you specify how), thanks!

Try packing an example project

PyEmpaq sources come with a small example project if you want to play a little packing it. Just a couple of dir/files under examples/srcproject:

  • a src and media, with stuff to be imported and accessed.

  • a pyempaq.yaml with the configuration for PyEmpaq.

  • a ep.py file which is the project's entrypoint; all it does is to inform it started, import the internal module, access the media files, and use the declared dependency, reporting every step.

This explores most of the needs of any project. You can try this example, and surely after you will be ready to actually pack any other project you want.

So, let's pack the example source project. As you're working with the PyEmpaq project itself (as you're packing its example), you don't really need to have it installed yet. In that case, if you have fades installed is super easy:

fades -r requirements.txt -m pyempaq examples/srcproject/

Otherwise, you would need to create and use a virtual environment:

python3 -m venv env
source env/bin/activate
pip install -r requirements
python3 -m pyempaq examples/srcproject/

After running that command, you will see a pyempaq-example.pyz file. That is the whole project encoded in a single file.

At this point you may move that pyempaq-example.pyz to another directory, or to another machine, even that other machine having another operating system.

Then, try it:

python3 pyempaq-example.pyz

You should see the project's reportings that we mentioned above (note: these lines will be surrounded by debug ones that will be hidden by default in the future):

Hello world
Code access ok .../pyempaq/projectname-20210722013526/orig/src/foo.py
Media access ok
Module requests imported .../pyempaq/projectname-20210722013526/venv/lib/python3.8/site-packages/requests/__init__.py

This shows that what you've run actually started, accessed the internal modules and other files, and imported correctly a custom-installed dependency.

Comments
  • Logged exec in unpacker

    Logged exec in unpacker

    #12 Cree la rama a partir de #40 ya que están vinculados (si bien todavía no mergeamos, restaba ajustar algo de los tests nomás). Lo dejo en draft porque falla test_unpacker(). Todavía no sé muy bien dónde está el inconveniente.

    opened by cacrespo 4
  • moves logged_exec from main.py to common.py

    moves logged_exec from main.py to common.py

    #11 Además de la función logged_exec() moví la clase ExecutionError porque si la quiero importar desde common.py explota por la referencia circular (main --> common --> main ...).

    Entiendo que el formateo del logger ya queda establecido al principio de main.py. Es decir cuando lo traigo desde common.py no es necesario volver a setear eso ¿cierto?

    Por último, en relación al test debería chequear tres situaciones:

    • se ejecutó un comando cualquiera y fue todo OK.
    • se ejecutó un comando y lanzó error
    • se ejecutó un comando ~y se quedó esperando una respuesta (y lo tomamos como un error)~, finalizó y retornó "distinto a 0".

    ¿es así?

    opened by cacrespo 3
  • Make `find_venv_bin` a common function and test it

    Make `find_venv_bin` a common function and test it

    Note that as will be used from the unpacker, the new common module needs to be imported also from there, so needs to be "packed" in the final project.

    good first issue 
    opened by facundobatista 3
  • added pipfile for easier installation, added version flag

    added pipfile for easier installation, added version flag

    With this PR, I added the following:

    • A Pipfile for easier installation, or at least an additional way to setup an isolated environment.
    • black formatter.
    • -V/--version flag by simply parsing the setup.py for the version of the module. This should fix #25
    opened by arthtyagi 2
  • Make `logged_exec` a common function and test it

    Make `logged_exec` a common function and test it

    Note that as will be used from the unpacker, the new common module needs to be imported also from there, so needs to be "packed" in the final project.

    good first issue 
    opened by facundobatista 2
  • control default verbosity in unpacker

    control default verbosity in unpacker

    closes #9

    Lo probé previo

     export PYEMPAQ_DEBUG=1
    

    y funciona todo OK. Aparecen (o no) los prints() en pantalla según corresponda.

    ¿deberíamos agregar algún test al respecto?

    opened by cacrespo 1
  • find_venv_bin function to common.py

    find_venv_bin function to common.py

    #10

    En pack() del file main.py creamos la carpeta tmpdir/pyempaq/ que nos habilita poder hacer

    from pyempaq.common import find_venv_bin 
    

    tanto en main como en el unpacker.

    De este modo pasan OK los test integration y unpacker

    opened by cacrespo 1
  • Use Python's logging in `main`

    Use Python's logging in `main`

    Use a simple format:

    • timestamps (with microseconds), but no date
    • timestamp, the level, and the message

    Stop using print:

    • all DEBUG prints should be logger.debug
    • stuff inside logged_exec also .debug
    • the rest should be .info
    good first issue 
    opened by facundobatista 1
  • add test for function find_venv_bin

    add test for function find_venv_bin

    Sigo con #10 y añado test pendiente. Chequea que la función retorne un Path con el nombre del archivo correcto según SO.

    @marcorichetta esto es lo que te comentaba.

    opened by cacrespo 0
  • The unpacker should execute the entrypoint with parameters, if available

    The unpacker should execute the entrypoint with parameters, if available

    The parameters may come from two different sources:

    • a default-args key in the metadata (under exec)
    • the .pyz execution itself (which overrides default-args, if any)
    opened by facundobatista 0
  • Refactor source of information

    Refactor source of information

    We need to stop using command line arguments to gather the needed information to pack a project, it does not scale.

    We will use a YAML file, pyempaq.yaml, that will hold all the required information:

    • basedir: the project's base dir (absolute or relative to the YAML; the rest of the paths can be absolute or relative to this basedir)
    • exec, which hold different subkeys
      • an XOR of the following ones:
        • script: the filepath of the python script to run; when unpacking PyEmpaq will do python3 SCRIPT
        • module: the name of the module to invoke; when unpacking PyEmpaq will do python3 -m MODULE
        • entrypoint: freeform, as a list of strings, when unpacking PyEmpaq will only insert the proper python3 at the beginning
      • also, default-args (not now, see this issue)
    • requirements: a list of filepaths pointing to the requirement files with pip-installable dependencies

    Note that we would need to heavily adapt/fix the README after this change. In particular, we need a section explaining all the pyempaq.yaml keys.

    The command line will end up being VERY simple:

    • the --verbose and/or --quiet options to control verbosity (see this issue)
    • a source mandatory parameter: the path to a file or a directory; if it's a file, it will be source of information, if it's a directory PyEmpaq will search for the source there (sequentially pyempaq.yaml, setup.py, etc.)

    Properly explain i the README how the command line works.

    opened by facundobatista 0
  • Allow to specify min python version

    Allow to specify min python version

    Fixes #29

    Se agrega opción para especificar versión de Python miníma para ejecutar el .pyz

    TODO:

    • [ ] Agregar este minimum_version a la metadata para ser usada en el unpacking
    • [ ] Agregar lógica para el unpacking
    • [ ] Documentar
    opened by marcorichetta 1
  • At unpacking, flag the successful installation of dependencies and use that to check the reusing

    At unpacking, flag the successful installation of dependencies and use that to check the reusing

    Currently, when the unpacker starts, it checks that the project dir exists to reuse it:

        if project_dir.exists():
    

    If not, it's created, and dependencies are installed. The problem is that because of underlying virtualenv magic (which varies from system to system, and we don't want to get into that, just be robust) the venv.create() call may end up in the process being exited, without properly finishing the whole directory setup.

    So, we need to:

    • create a flag file inside project_dir, something like project-setup-done.flag (empty is enough)
    • when verifying that "the previous process did all fine", use this new file, like:
        if project_dir.exists() and (project_dir / project-setup-done.flag).exists() :
    
    bug 
    opened by facundobatista 1
  • Allow to specify the minimum required Python version to run the packed content

    Allow to specify the minimum required Python version to run the packed content

    The configuration should have a key called unpack_restrictions. It's a "dict", currently will have only one key: `minimum_python_version". Example:

    unpack_restrictions:
        minimum_python_version: "3.8"
    

    When packing, the code should:

    • validate that the value is a string, not a float (a common mistake is writing the YAML like minimum_python_version: 3.8, which ends up being a float).
    • annotate this inside the metadata, for the unpacker

    When unpacking, it should:

    • very early in the process verify if that metadata is present, and if yes, it should validate that the Python used to run the .pyz is fine.
    • the comparison should be done using parse_version (from pkg_resources)
    • if fails, it should error out saying "Unsupported Python version, the minimum required is XXX" (where XXX is the repr() of the string that the developer configured)
    • however, if the PYEMPAQ_IGNORE_RESTRICTIONS environment variable is set, it should NOT fail (but still show a warning with the same text)
    opened by facundobatista 1
  • Added option to check current package version

    Added option to check current package version

    This PR makes the following changes:-

    • adds command line option to check current python version
    • black formatted pyempaq/main.py

    Thus, this PR closes #25.

    Code Proof

    >>pyempaq -V
    pyempaq 0.2.2
    
    >>pyempaq --version
    pyempaq 0.2.2
    

    Signed-off-by: Shivam Shandilya [email protected]

    opened by shivamshan 0
  • Support different forms of installation, if source project is prepared

    Support different forms of installation, if source project is prepared

    If the source project is prepared and opts-in, the first run of a project and everytime the packed project changes, the unpacker should do stuff for the unpacked project remains "installed" in the final system.

    • Linux: the project could have a .desktop file, and the unpacker could put it in the proper place (if exists)

    • Windows: maybe add a "symlink" ("direct access", are they called?) in the user's desktop, pointing to the "expanded" location

      • as this dirties the user environment, maybe the project should "opt in"
      • how can the project be registered so it appears "in the menu"?
    • Mac OS: no clue on what is needed here

    opened by facundobatista 0
Releases(0.2.2)
Owner
Facundo Batista
Facundo Batista
Run a command in the named virtualenv.

Vex Run a command in the named virtualenv. vex is an alternative to virtualenv's source wherever/bin/activate and deactivate, and virtualenvwrapper's

Sasha Hart 374 Dec 21, 2022
Ready-to-run Docker images containing Jupyter applications

Jupyter Docker Stacks are a set of ready-to-run Docker images containing Jupyter applications and interactive computing tools.

Project Jupyter 7k Jan 3, 2023
Simple Python version management

Simple Python Version Management: pyenv pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UN

pyenv 30.1k Jan 4, 2023
Virtual Python Environment builder

virtualenv A tool for creating isolated virtual python environments. Installation Documentation Changelog Issues PyPI Github Code of Conduct Everyone

Python Packaging Authority 4.3k Dec 30, 2022
Python Development Workflow for Humans.

Pipenv: Python Development Workflow for Humans [ ~ Dependency Scanning by PyUp.io ~ ] Pipenv is a tool that aims to bring the best of all packaging wo

Python Packaging Authority 23.5k Jan 1, 2023
Python virtualenvs in Debian packages

dh-virtualenv Contents Overview Presentations, Blogs & Other Resources Using dh-virtualenv How does it work? Running tests Building the package in a D

Spotify 1.5k Jan 2, 2023
Manage python virtual environments on the working notebook server

notebook-environments Manage python virtual environments on the working notebook server. Installation It is recommended to use this package together w

Vladislav Punko 44 Nov 2, 2022
Fish shell tool for managing Python virtual environments

VirtualFish VirtualFish is a Python virtual environment manager for the Fish shell. You can get started by reading the documentation. (It’s quite shor

Justin Mayer 968 Dec 24, 2022
Define requirements inside your python code and scriptenv makes them ready to import.

scriptenv Define requirements inside your python code and scriptenv makes them ready to import. Getting Started Install scriptenv $ pip install script

Stefan Hoelzl 6 Nov 4, 2022
A fast and easy python virtual environment creator for linux with some pre-installed libraries.

python-venv-creator A fast and easy python virtual environment created for linux with some optional pre-installed libraries. Dependencies: The followi

null 2 Apr 19, 2022
a pyenv plugin to manage virtualenv (a.k.a. python-virtualenv)

pyenv-virtualenv pyenv-virtualenv is a pyenv plugin that provides features to manage virtualenvs and conda environments for Python on UNIX-like system

pyenv 5.3k Jan 8, 2023
Run a command in the named virtualenv.

Vex Run a command in the named virtualenv. vex is an alternative to virtualenv's source wherever/bin/activate and deactivate, and virtualenvwrapper's

Sasha Hart 374 Dec 21, 2022
Fully Automated YouTube Channel ▶️with Added Extra Features.

Fully Automated Youtube Channel ▒█▀▀█ █▀▀█ ▀▀█▀▀ ▀▀█▀▀ █░░█ █▀▀▄ █▀▀ █▀▀█ ▒█▀▀▄ █░░█ ░░█░░ ░▒█░░ █░░█ █▀▀▄ █▀▀ █▄▄▀ ▒█▄▄█ ▀▀▀▀ ░░▀░░ ░▒█░░ ░▀▀▀ ▀▀▀░

sam-sepiol 249 Jan 2, 2023
Python 3 script unpacking statically x86 CryptOne packer.

Python 3 script unpacking statically x86 CryptOne packer. CryptOne versions: 2021/08 until now (2021/12)

null 5 Feb 23, 2022
Run your jupyter notebooks as a REST API endpoint. This isn't a jupyter server but rather just a way to run your notebooks as a REST API Endpoint.

Jupter Notebook REST API Run your jupyter notebooks as a REST API endpoint. This isn't a jupyter server but rather just a way to run your notebooks as

Invictify 54 Nov 4, 2022
Control System Packer is a lightweight, low-level program to transform energy equations into the compact libraries for control systems.

Control System Packer is a lightweight, low-level program to transform energy equations into the compact libraries for control systems. Packer supports Python ?? , C ?? and C++ ?? libraries.

mirnanoukari 31 Sep 15, 2022
Metal Gear Rising: Revengeance's DAT archive (un)packer

DOOMP Metal Gear Rising: Revengeance's DAT archive (un)packer

Christopher Holzmann Pérez 5 Sep 2, 2022
shiv is a command line utility for building fully self contained Python zipapps as outlined in PEP 441, but with all their dependencies included.

shiv shiv is a command line utility for building fully self-contained Python zipapps as outlined in PEP 441, but with all their dependencies included!

LinkedIn 1.5k Dec 28, 2022
Dear PyGui: A fast and powerful Graphical User Interface Toolkit for Python with minimal dependencies

(This library is available under a free and permissive license however, if you Enjoy Dear PyGui please consider becoming a Sponsor) Dear PyGui is a si

Jonathan Hoffstadt 9.4k Jan 4, 2023
Dear PyGui: A fast and powerful Graphical User Interface Toolkit for Python with minimal dependencies

(This library is available under a free and permissive license however, if you Enjoy Dear PyGui please consider becoming a Sponsor) Dear PyGui is a si

Jonathan Hoffstadt 9.4k Jan 7, 2023