Reproduction
- Clone https://github.com/zmanji/git-squash
- Checkout
505cf06c598409db9d119a521480e43de3fa8908
- Run
pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex
With latest pex on macOS.
Expectation
Because of --no-build
, --wheel
, --no-transitive
and a fully resolved requirements.txt
from pip-compile
building a pex should be very fast. The wheels can be fetched from pypi if they are not cached locally, the package it self a single file, so building the wheel should be very fast and then building the pex should be straight forward
For reference building the wheel for git-squash
via python ./setup.py bdist_wheel
takes about 0.3s on my computer.
Reality
I see this output
pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Resolving for:
DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macpex: Hashing pex pex: Hashing pex: 67.4ms
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Building distributions for:
BuildRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', pex: Building /Users/zmanji/code/git-squash to /Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
LocalProjectRequirement(line=LogicalLine(raw_text='.', processed_text='.', source='<string>', start_line=1, end_line=1), path='/Users/zmanji/code/git-squash', extras=(), marker=None, editable=False)
PyPIRequirement(line=LogicalLine(raw_text='gitdb==4.0.7\n', processed_text='gitdb==4.0.7', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=7, end_line=7), requirement=Requirement.parse('gitdb==4.0.7'), editable=False)
PyPIRequirement(line=LogicalLine(raw_text='gitpython==3.1.17\n', processed_text='gitpython==3.1.17', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=9, end_line=9), requirement=Requirement.parse('gitpython==3.1.17'), editable=False)
PyPIRequirement(line=LogicalLine(raw_text='smmap==4.0.0\n', processed_text='smmap==4.0.0', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=11, end_line=11), requirement=Rpex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Installing:
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/GitPython-3.1.17-py3-none-any.whl', fingerprint='59d63e79b53503f5403220d3159d8e409c46e107')
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/smmap-4.0.0-py2.py3-none-any.whl', fingerprint='5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f')
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/gitdb-4.0.7-py3-none-any.whl', fingerprint='9dacbc989a247dabdad014d6397d8482fd55f26c')
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-pex: Using cached installation of GitPython-3.1.17-py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/59d63e79b53503f5403220d3159d8e409c46e107/GitPython-3.1.17-py3-none-any.whl
pex: Using cached installation of smmap-4.0.0-py2.py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f/smmap-4.0.0-py2.py3-none-any.whl
pex: Using cached installation of gitdb-4.0.7-py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/9dacbc989a247dabdad014d6397d8482fd55f26c/gitdb-4.0.7-py3-none-any.whl
pex: Installing /Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-py3-none-any.whl in /Users/zmanji/.pex/installed_wheels/c355002f497d1f36a0bae44b05e55b4338da8a0f/git_squash-0.0.1-py3-none-any.whl
pex: Set entrypoint to console_script 'git_squash.git_squash:main' in git-squash 0.0.1 (/Users/zmanji/.pex/installed_wheels/c355002f497d1f36a0bae44b05e55b4338da8a0f/git_squash-0.0.1-py3-none-any.whl)
pex: Building pex: 13722.4ms
pex: Resolving distributions (['.', 'requirements.txt']): 13720.7ms
pex: Resolving requirements.: 13709.1ms
pex: Resolving for:
DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))): 6592.5ms
pex: Building distributions for:
BuildRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), source_path='/Users/zmanji/code/git-squash', fingerprint='fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e'): 6475.2ms
pex: Calculating project names for direct requirements:
LocalProjectRequirement(line=LogicalLine(raw_text='.', processed_text='.', source='<string>', start_line=1, end_line=1), path='/Users/zmanji/code/git-squash', extras=(), marker=None, editable=False)
PyPIRequirement(line=LogicalLine(raw_text='gitdb==4.0.7\n', processed_text='gitdb==4.0.7', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=7, end_line=7), requirement=Requirement.parse('gitdb==4.0.7'), editable=False)
PyPIRequirement(line=LogicalLine(raw_text='gitpython==3.1.17\n', processed_text='gitpython==3.1.17', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=9, end_line=9), requirement=Requirement.parse('gitpython==3.1.17'), editable=False)
PyPIRequirement(line=LogicalLine(raw_text='smmap==4.0.0\n', processed_text='smmap==4.0.0', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=11, end_line=11), requirement=Requirement.parse('smmap==4.0.0'), editable=False): 4.3ms
pex: Installing:
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/GitPython-3.1.17-py3-none-any.whl', fingerprint='59d63e79b53503f5403220d3159d8e409c46e107')
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/smmap-4.0.0-py2.py3-none-any.whl', fingerprint='5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f')
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/gitdb-4.0.7-py3-none-any.whl', fingerprint='9dacbc989a247dabdad014d6397d8482fd55f26c')
InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-py3-none-any.whl', fingerprint='c355002f497d1f36a0bae44b05e55b4338da8a0f'): 430.6ms
Saving PEX file to dist/out.pex
pex: Zipping PEX file.: 148.4ms
pex . -vv -r requirements.txt --no-build --wheel --no-transitive --script 9.65s user 4.27s system 96% cpu 14.348 total
Notice:
pex: Resolving requirements.: 13709.1ms
This is very unexpected and there is no information in the output as to why it takes so long. With all of the caching PEX uses, I would expect this to be very fast.
answered