A PDM plugin that packs your packages into a zipapp

Frost Ming

Last update: Dec 29, 2022

Related tags

Overview

pdm-packer

A PDM plugin that packs your packages into a zipapp

Requirements

pdm-packer requires Python >=3.7

Installation

If you have installed PDM with the recommended tool pipx, add this plugin by:

$ pipx inject pdm pdm-packer

Or if you have installed PDM with pip install --user pdm, install with pip to the user site:

$ python -m pip install --user pdm-packer

Otherwise, install pdm-packer to the same place where PDM is located.

Usage

$ pdm pack [common-options] [pack-options]

Common Options:

-h, --help

show this help message and exit

-v, --verbose

-v for detailed output and -vv for more detailed

-g, --global

Use the global project, supply the project root with -p option

-p PROJECT_PATH, --project PROJECT_PATH

Specify another path as the project root, which changes the base of pyproject.toml and __pypackages__

Pack Options:

-m MAIN, --main MAIN

Specify the console script entry point for the zipapp

-o OUTPUT, --output OUTPUT

Specify the output filename. By default the file name will be inferred from the project name.

-c, --compress

Compress files with the deflate method, no compress by default

-i INTERPRETER, --interpreter INTERPRETER

The Python interpreter path, default: the project interpreter

--exe

Create an executable file. If the output file isn't given, the file name will end with .exe(Windows) or no suffix(Posix)

Examples

# Create with default name(<project_name>.pyz) and console_script as the __main__.py
pdm pack
# Create an executable file
pdm pack --exe
# Create with custom __main__.py and filename
pdm pack -o app.pyz -m app:main

Caveats

If the result zipapp contains binaries, it can only be deployed to the platforms with the same abi, any cross-abi usage of that app might expect a failure.
Any console scripts except for what is given to --main will be lost.
The .exe file is different from what is produced by pyinstaller in the way that it doesn't embed a Python interpreter. This means you have to install a Python with exactly the same version on the deployment platform.
If you have code to run in your project, the project itself should be installed into __pypackages__ as well. Make sure you have set a project name in pyproject.toml.

About executable zipapp

By default, zipapp is created with .pyz suffix. On Windows, if you have associted .pyz files with Python program, you can run the app by double-clicking the file in the explorer. But if you create the app with --exe turn on, you can have a .exe file on Windows and an executable file on Unix-like systems, so that the app can be executed without a python command prefixing it and no matter you assoicated the file exensition properly or not.

Changelog

See CHANGELOG.md

Comments

[QUESTION]. how to use it?

I created a test project

pdm init

add run.py

def run():
    print('hello')

pack

pdm pack -m run:run -o app.zip -v
Packing packages...
All packages are synced to date, nothing to do.

  All complete!
Packages are prepared at U:\Temp\pdm-pack-a64tsc6l\lib
Creating zipapp...
Traceback (most recent call last):
  File "C:\Python\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "c:\users\user\.local\bin\pdm.exe\__main__.py", line 7, in <module>
  File "C:\Users\user\.local\pipx\venvs\pdm\lib\site-packages\pdm\core.py", line 254, in main
    return Core().main(args)
  File "C:\Users\user\.local\pipx\venvs\pdm\lib\site-packages\pdm\core.py", line 187, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "C:\Users\user\.local\pipx\venvs\pdm\lib\site-packages\pdm\core.py", line 182, in main
    f(options.project, options)
  File "C:\Users\user\.local\pipx\venvs\pdm\lib\site-packages\pdm_packer\command.py", line 120, in handle
    zipapp.create_archive(
  File "C:\Python\lib\zipapp.py", line 111, in create_archive
    raise ZipAppError("Source does not exist")
zipapp.ZipAppError: Source does not exist

Cannot pack if there are no packages? - pdm add colorama (why not use without packages) and try again

pdm pack -m run:run -o app.zip -v
Packing packages...
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

unearth: Downloading <Link https://files.pythonhosted.org/packages/77/8b/7550e87b2d308a1b711725dfaddc19c695f8c5fa413c640b2be01662f4e6/colorama-0.4.5-py2.py3-none-any.whl (from None)> to U:\Temp\pdm-build-42ueqwgg\colorama-0.4.5-py2.py3-none-any.whl
  v Install colorama 0.4.5 successful

  All complete!
Packages are prepared at U:\Temp\pdm-pack-1trs0fvn\lib
Creating zipapp...
Zipapp is generated at app.zip

app.zip structure

        colorama
        __main__.py

no run.py file

pdm pack --help
Usage: pdm pack [-h] [-v] [-g] [-p PROJECT_PATH] [-m MAIN] [-o OUTPUT] [-c] [--pyc] [--no-py] [-i INTERPRETER] [--exe]

Pack the packages into a zipapp

only packages without project files? then why need -m run:run?

pdm info && pdm info --env

pdm info && pdm info --env
PDM version:
  2.0.0
Python Interpreter:
  C:\Python\python.exe (3.10)
Project Root:
  z:/testapp
Project Packages:
  z:\testapp\__pypackages__\3.10
{
  "implementation_name": "cpython",
  "implementation_version": "3.10.5",
  "os_name": "nt",
  "platform_machine": "AMD64",
  "platform_release": "10",
  "platform_system": "Windows",
  "platform_version": "10.0.19044",
  "python_full_version": "3.10.5",
  "platform_python_implementation": "CPython",
  "python_version": "3.10",
  "sys_platform": "win32"
}

opened by vitidev 6

The project files were not packaged in when packaging

This will cause python app.pyz to not run correctly when I only give the pyz package.

I checked the source code and this should happen here: https://github.com/frostming/pdm-packer/blob/main/src/pdm_packer/command.py#L100

opened by abersheeran 4
when packing pdm the version given at the stdout is not the same as the one given when doing --version

I can see the version of pdm used listed on stdout when creating the zipapp, but when running the zip app with --version, I get a different version.

To Reproduce git checkout 2.3.3 pdm add pdm-packer pdm pack ---------8<---------8<---------8<---------8<---------8<--------- ... [2022-12-15T20:38:56.771Z] v Install pdm 2.3.3+d20221215 successful [2022-12-15T20:38:56.771Z] [2022-12-15T20:38:56.771Z] All complete! [2022-12-15T20:38:56.771Z] [2022-12-15T20:38:56.771Z] Packages are prepared at xx [2022-12-15T20:38:56.771Z] Creating zipapp... [2022-12-15T20:38:57.026Z] Zipapp is generated at pdm.pyz [2022-12-15T20:38:57.586Z] pdm.pyz --version [2022-12-15T20:38:58.947Z] PDM, version 0.0.0+local ---------8<---------8<---------8<---------8<---------8<---------

I would have expected the version to be 2.3.3+d20221215, or 2.3.3 but not 0.0.0+local

Is this expected? What would I need to do to get the version to be 2.3.3+packedbymyorgnamehere?

Thanks in advance, Franz

opened by franzhaas 3
The problem with git-lfs....
Unless git-lfs is installed (and configured possibly requiring extra paid services from GitHub) a plain git clone of this repo fails. This is creating problems like @kloczek reported in https://github.com/pdm-project/pdm/issues/1084

And I reckon that your position is https://github.com/pdm-project/pdm/issues/1084#issuecomment-1131407250

Checking in binary files in a git repository is not a good practice. Because Git will store all histories in the .git directory, if the file is a binary the storage is not efficient: it can't store the delta changes like a plain text file.

I understand your point, but here this is eventually creating difficulties for any adopter and aspiring contributor. IMHO this may not be the effect you are looking for? The whole set of wheels and archives is about 3.1MB total and this are binaries but not what I would call large files.

I would suggest to consider using something else:

just copy the test wheels inside the repo... these are not super big and not evolving quickly

or use a separate repo just for the test wheels and use git modules to get it

or put these in a tarball under a release of this repo and fetch these from there
opened by pombredanne 3
pdm_packer:plugin: cannot import name 'cached_property' from 'pdm.utils' .

After installing packer, i get the following error message.:

Failed to load plugin pdm-packer=pdm_packer:plugin: cannot import name 'cached_property' from 'pdm.utils' .

pdm version is 2.3.2 pdm packer version is 0.4.0

any thoughts?

opened by franzhaas 1

How do you pack all files within project and access them?

Probably just some examples I'm missing but how do you pack and access resource files? For example my project consists of:

lib/python/.... (python modules)
run.py (run function as entrypoint)
.env
README.md
etc.

How do I access .env file for example? because now my app is failing due to dotenv.load_dotenv function

Traceback (most recent call last):
  File "/home/mariusg/.pyenv/versions/3.9.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/mariusg/.pyenv/versions/3.9.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mariusg/Documents/Projects/infra-terraform/./app.pyz/__main__.py", line 3, in <module>
  File "/tmp/pdm-pack-rpnrz65t/lib/run.py", line 10, in run
    load_dotenv()
  File "/tmp/pdm-pack-rpnrz65t/lib/dotenv/main.py", line 331, in load_dotenv
  File "/tmp/pdm-pack-rpnrz65t/lib/dotenv/main.py", line 298, in find_dotenv
  File "/tmp/pdm-pack-rpnrz65t/lib/dotenv/main.py", line 255, in _walk_to_root
OSError: Starting path not found

run.py:

#!/usr/bin/env python3

from dotenv import load_dotenv

def run():
    # Load up .env files
    load_dotenv()

Package command:

pdm pack -m run:run -c --pyc --no-py --exe -o app.pyz

opened by mgzenitech 1

Cannot find psycopg2-binary

Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "./nina_database_manager.pyz/__main__.py", line 2, in <module>
  File "/usr/local/lib/python3.8/site-packages/manager.py", line 9, in <module>
    import psycopg2
  File "<frozen zipimport>", line 259, in load_module
  File "./nina_database_manager.pyz/psycopg2/__init__.py", line 51, in <module>
ModuleNotFoundError: No module named 'psycopg2._psycopg'

Describe the bug psycopg2-binary is a dependency of the project, but it is not found when I try to execute the .pyz zip file.

To Reproduce Steps to reproduce the behavior:

Add psycopg2-binary
Import psycopg2 in your app
Pack your app
Run it
See error

Expected behavior psycopg2 should be imported as usual.

System (please complete the following information):

pdm-packer version: 0.2.1
Python version: 3.8
OS: Linux

Additional context Other dependencies, such as ldap, seems to work as usual.

opened by frafra 1

Latest release is incompatible with latest pdm release
Describe the bug The latest released version of pdm-packer is not compatible with the latest version of pdm (currently 1.14.1). This is indicated by an error message when calling pdm as well as the pdm pack not working. Thankfully, the underlying problem is already fixed on main :-)

To Reproduce

Run pdm plugin add pdm-packer and see the installation succeed

Run pdm plugin list and see this error:
Failed to load plugin pdm-packer=pdm_packer:plugin: cannot import name 'get_architecture' from 'pdm.models.in_process' (/some/venv/pdm/lib/python3.10/site-packages/pdm/models/in_process/__init__.py)

Install pdm-packer from the latest main branch in Git instead, e.g. with pipx: pipx inject pdm "git+https://github.com/frostming/pdm-packer.git@main"

Run pdm plugin list: No error shows up

Expected behavior The latest version of pdm-packer in PyPI should work with the latest version of pdm.

System

pdm-packer version: 0.3

pdm version: 1.14.1

Python version: 3.8, 3.10

OS: Ubuntu 20.04, macOS 12.3
opened by sleiner 0
[pre-commit.ci] pre-commit autoupdate
updates:

github.com/pre-commit/pre-commit-hooks: v3.2.0 → v4.0.1

github.com/psf/black: 20.8b1 → 21.5b2

https://gitlab.com/pycqa/flake8 → https://github.com/PyCQA/flake8

github.com/PyCQA/flake8: 3.8.3 → 3.9.2

github.com/pycqa/isort: 5.6.3 → 5.8.0
opened by pre-commit-ci[bot] 0
packing pdm itself apears only to work with python 3.11, but neither 3.9 nor 3.10 on win10

Hi,

I was succesfully able to pack pdm into a zipapp. This works nicely when I use python 3.11 to execute the zipapp.

I encountered 2 problems with the exe target. 1.) I needed to use the -i command to specify the interpreter "py" which is to my understanding the recommended way to find python on windows. 2.) on the target machine python 3.11 needs to be the default selected by "py".

The issue appears to only affect sync, not lock or search.

I would like to know if there is a possibility to specify the python version which should've used, like "py -3.11" when building an exe

This is the way I create the zipapp.:
git clone https://github.com/pdm-project/pdm.git cd pdm git checkout 2.3.4 pdm add pdm-packer pdm pack pdm pack --exe -i py.exe
Any hints?

Thanks in advance, Franz

opened by franzhaas 3

Releases(0.4.0)

0.4.0(Jul 27, 2022)
What's Changed

[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/frostming/pdm-packer/pull/17

feat: Update import paths by @frostming in https://github.com/frostming/pdm-packer/pull/20

Full Changelog: https://github.com/frostming/pdm-packer/compare/0.3.2...0.4.0
Source code(tar.gz)
Source code(zip)