Support --sh-boot for --layout {loose,packed}.
#2645
Conversation
|
Reviewers, you're on here because this should allow eliminating the hackery I added to Pants long ago to reduce PEX venv latency. You should find the same numbers you have today, but with less code and less brittleness. The benchmark: #!/usr/bin/env bash
set -euo pipefail
EXECUTION_MODE_ARGS=(
"ZIPAPP|"
"ZIPAPP-sh-boot|--sh-boot"
"VENV|--venv prepend"
"VENV-sh-boot|--venv prepend --sh-boot"
)
LAYOUTS=(
loose
packed
zipapp
)
PEXES=()
for value in "${EXECUTION_MODE_ARGS[@]}"; do
em_label="${value/\|*/}"
em_args="${value/*\|/}"
for layout in "${LAYOUTS[@]}"; do
mkdir -p /tmp/test
PEX="/tmp/test/${em_label}-${layout}.pex"
python3.13 -mpex cowsay -ccowsay ${em_args} --layout ${layout} -o "${PEX}"
if [[ -d ${PEX} ]]; then
PEXES+=("${PEX}/pex")
else
PEXES+=("${PEX}")
fi
done
done
params="$(echo "${PEXES[@]}" | xargs | tr ' ' ,)"
hyperfine -w2 -L pex ${params} "{pex} -t Moo!"And the results: |
| def create_sh_python_redirector_shebang(sh_script_content): | ||
| # type: (str) -> Tuple[str, str] | ||
| """Create a shebang block for a Python file that uses /bin/sh to find an appropriate Python. | ||
|
|
||
| The script should be POSIX compliant sh and terminate on all execution paths with an | ||
| explicit exit or exec. | ||
|
|
||
| The returned shebang block will include the leading `#!` but will not include a trailing new | ||
| line character. | ||
|
|
||
| :param sh_script_content: A POSIX compliant sh script that always explicitly terminates. | ||
| :return: A shebang line and trailing block of text that can be combined for use as a shebang | ||
| header for a Python file. | ||
| """ | ||
| # This trick relies on /bin/sh being ubiquitous and the concordance of: | ||
| # | ||
| # 1. Python: Has triple quoted strings plus allowance for free-floating string values in | ||
| # python files. | ||
| # 2. sh: Any number of pairs of `'` evaluating away when followed immediately by a | ||
| # command string (`''command` -> `command`). | ||
| # 3. sh: The `:` noop command which accepts and discards arbitrary args. | ||
| # See: https://pubs.opengroup.org/onlinepubs/009604599/utilities/colon.html | ||
| # 4. sh: Lazy parsing allowing for invalid sh content immediately following an exit or exec | ||
| # line. | ||
| # | ||
| # The end result is a file that is both a valid sh script with a short shebang and a valid | ||
| # Python program. | ||
| return "#!/bin/sh", ( | ||
| dedent( | ||
| """\ | ||
| '''': pshprs | ||
| {sh_script_content} | ||
| ''' | ||
| """ | ||
| ) | ||
| .format(sh_script_content=sh_script_content.rstrip()) | ||
| .strip() | ||
| ) |
There was a problem hiding this comment.
This (and its use in pex/pex_builder.py) is the only bit of real interest in this PR. An old trick, but formalized and improved with the : pshprs sh comment allowing for efficient is_python_script checks of these otherwise not apparently Python scripts.
This trick is what allows using the existing --sh-boot PEX zip header script wholesale as a header to the existing PEX __main__.py launcher in loose and packed PEX directories.
There was a problem hiding this comment.
Awesome!
(Cc'ing the relevant pants issue: pantsbuild/pants#21177 )
| @@ -660,6 +660,19 @@ def build( | |||
| compress=compress, | |||
| bytecode_compile=bytecode_compile, | |||
| ) | |||
| if layout in (Layout.LOOSE, Layout.PACKED): | |||
| pex_script = os.path.join(tmp_pex, "pex") | |||
There was a problem hiding this comment.
Am I inferring correctly that this pex script is a new top-level file included in loose and packed PEXes? And is designed to be the "best" way to execute such a pex?
There was a problem hiding this comment.
It's not really new, notice at the end of the block pex just becomes a symlink to __main__.py, but running either file directly is the best way to run the PEX. This is because the file has new content, the sh boot header.
You used to be able to run the PEX in 3 ways:
python the/dir-layout-pex/
python the/dir-layout-pex/__main__.py
the/dir-layout-pex/__main__.py
You still can, Pex will never break you, but whereas all three booted with the same latency before, the third has lower latency now. Added to that are two more ways to launch equivalent to the second and third since pex is symlink to __main__.py:
python the/dir-layout-pex/pex
the/dir-layout-pex/pex
Here the second form has the low latency boot. These forms parallel the root of a PEX venv and have the mildly pleasing aesthetic of -o app.pex vs --layout {loose,packed} -o app leading to execution via ./app.pex vs ./app/pex.
No description provided.