Added support for armv7l #2620

Lauszus · 2024-12-11T20:19:08Z

No description provided.

Lauszus · 2024-12-11T21:30:54Z

@jsirois this one is ready. I just tested this by compiling my lift branch (a-scie/lift#106) and then using that using the --scie-science-binary argument.

Lauszus · 2024-12-11T21:31:35Z

I'm a bit unsure about all the tests, but I tried to add armv7l a few places I thought made sense.

Lauszus · 2024-12-12T11:07:27Z

pex/version.py

@@ -1,4 +1,4 @@
 # Copyright 2015 Pex project contributors.
 # Licensed under the Apache License, Version 2.0 (see LICENSE).

-__version__ = "2.25.2"
+__version__ = "2.26.0"


@jsirois I have prepared for a release here as well. Let me know if you want me to do any other changes.

Lauszus · 2024-12-12T16:38:41Z

@jsirois I'm struggling a bit to get the get-scie-platform script working. At least it is running now, but I am getting:

$ docker run --rm -v "/tmp/pex:/tmp/pex" -v $PWD:/code -w /code --platform linux/arm/v7 python:3.11-slim-bookworm bash ./gen-scie-platform.sh
ROOT: provision> .tox/.tox/bin/python -m tox -e gen-scie-platform -- -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0
gen-scie-platform create: /code/.tox/gen-scie-platform
gen-scie-platform installdeps: httpx==0.23.0, toml==0.10.2, PyGithub==2.4.0, cryptography==43.0.3
gen-scie-platform installed: anyio==3.7.1,certifi==2024.8.30,cffi==1.17.1,charset-normalizer==3.4.0,cryptography==43.0.3,Deprecated==1.2.15,h11==0.12.0,httpcore==0.15.0,httpx==0.23.0,idna==3.10,pycparser==2.22,PyGithub==2.4.0,PyJWT==2.10.1,PyNaCl==1.5.0,requests==2.32.3,rfc3986==1.5.0,sniffio==1.3.1,toml==0.10.2,typing_extensions==4.12.2,urllib3==2.2.3,wrapt==1.17.0
gen-scie-platform run-test-pre: PYTHONHASHSEED='707027689'
gen-scie-platform run-test: commands[0] | python scripts/gen-scie-platform.py -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0
HTTP Error 404: Not Found
Command '['/code/.tox/gen-scie-platform/bin/python', '-m', 'pex', '.', '-c', 'pex3', '--scie', 'lazy', '--scie-pbs-release', '20241016', '--scie-python-version', '3.13.0', '-o', '/tmp/tmp8qw_ylpz/pex3']' returned non-zero exit status 1.
ERROR: InvocationError for command /code/.tox/gen-scie-platform/bin/python scripts/gen-scie-platform.py -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0 (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   gen-scie-platform: commands failed

Same command work when using --platform linux/amd64

Lauszus · 2024-12-12T16:51:50Z

@jsirois I'm struggling a bit to get the get-scie-platform script working. At least it is running now, but I am getting:

$ docker run --rm -v "/tmp/pex:/tmp/pex" -v $PWD:/code -w /code --platform linux/arm/v7 python:3.11-slim-bookworm bash ./gen-scie-platform.sh
ROOT: provision> .tox/.tox/bin/python -m tox -e gen-scie-platform -- -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0
gen-scie-platform create: /code/.tox/gen-scie-platform
gen-scie-platform installdeps: httpx==0.23.0, toml==0.10.2, PyGithub==2.4.0, cryptography==43.0.3
gen-scie-platform installed: anyio==3.7.1,certifi==2024.8.30,cffi==1.17.1,charset-normalizer==3.4.0,cryptography==43.0.3,Deprecated==1.2.15,h11==0.12.0,httpcore==0.15.0,httpx==0.23.0,idna==3.10,pycparser==2.22,PyGithub==2.4.0,PyJWT==2.10.1,PyNaCl==1.5.0,requests==2.32.3,rfc3986==1.5.0,sniffio==1.3.1,toml==0.10.2,typing_extensions==4.12.2,urllib3==2.2.3,wrapt==1.17.0
gen-scie-platform run-test-pre: PYTHONHASHSEED='707027689'
gen-scie-platform run-test: commands[0] | python scripts/gen-scie-platform.py -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0
HTTP Error 404: Not Found
Command '['/code/.tox/gen-scie-platform/bin/python', '-m', 'pex', '.', '-c', 'pex3', '--scie', 'lazy', '--scie-pbs-release', '20241016', '--scie-python-version', '3.13.0', '-o', '/tmp/tmp8qw_ylpz/pex3']' returned non-zero exit status 1.
ERROR: InvocationError for command /code/.tox/gen-scie-platform/bin/python scripts/gen-scie-platform.py -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0 (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   gen-scie-platform: commands failed

Same command work when using --platform linux/amd64

Turning on verbose did not give more outout:

gen-scie-platform run-test: commands[0] | python scripts/gen-scie-platform.py -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0 -v
HTTP Error 404: Not Found
Command '['/code/.tox/gen-scie-platform/bin/python', '-m', 'pex', '.', '-c', 'pex3', '--scie', 'lazy', '--scie-pbs-release', '20241016', '--scie-python-version', '3.13.0', '-o', '/tmp/tmpzagyb9oh/pex3']' returned non-zero exit status 1.
ERROR: InvocationError for command /code/.tox/gen-scie-platform/bin/python scripts/gen-scie-platform.py -d /tmp/pex-tmp --pbs-release 20241016 --python-version 3.13.0 -v (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   gen-scie-platform: commands failed

jsirois · 2024-12-12T17:10:17Z

@Lauszus is CI running gen-scie-platform or are you running it to test it? If the latter, what command are you using exactly?

Lauszus · 2024-12-12T17:20:48Z

@Lauszus is CI running gen-scie-platform or are you running it to test it? If the latter, what command are you using exactly?

I'm running it locally since I can not trigger the CI.

I'm running:

docker run --rm -v "/tmp/pex:/tmp/pex" -v $PWD:/code -w /code --platform linux/arm/v7 python:3.11-slim-bookworm bash ./gen-scie-platform.sh

My install scripts looks like this:

#!/usr/bin/env bash

python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -U tox
python -V

tox -e gen-scie-platform -- \
  -d /tmp/pex \
  --pbs-release 20241016 \
  --python-version 3.13.0 -v

.github/workflows/gen-scie-platforms.yml

jsirois · 2024-12-12T19:44:42Z

I'm running it locally since I can not trigger the CI.

Ok, thanks for those details. There are a few issues here:

The 404: This is due to the default science version remaining at 0.8.0 which has no armv7l release available to download; so that needs to be bumped to 0.9.0:

pex/pex/scie/science.py

Line 67 in 74f34aa

MIN_SCIENCE_VERSION = Version("0.8.0")
After that, no wheels for psutil: The Pex PEX scies are all built from locks that do not allow sdists, only wheels. The current locking code just uses PyPI; so does not find a wheel for psutil for armv7l and dies. This will need plumbing of --use-pip-config to pick up your tox.ini setup of PIP_EXTRA_INDEX_URL, which currently only applies to bare use of Pip and not Pex internal uses of Pip (locking / resolving) which are hermetic by default. But even then, psutil 6.0 is only pre-built for 3.11 and the Pex PEX scies use 3.13 currently: https://www.piwheels.org/project/psutil/

I have no clue what timezone you're actually in right now - we're both working odd hours it seems. Combined with the fact the complete-platform gen will need to happen before release, but that this PR needs to land before complete-platform gen can be run means the CHANGES.md / pex/version.py changes will need to be backed out. I'm sorry about that - I should have seen that coming. Unless you say otherwise, I'm going to get involved in this PR with a few edits towards these ends to move this along and get out a release today.

Lauszus · 2024-12-12T22:34:26Z

I'm running it locally since I can not trigger the CI.

Ok, thanks for those details. There are a few issues here:

The 404: This is due to the default science version remaining at 0.8.0 which has no armv7l release available to download; so that needs to be bumped to 0.9.0:

pex/pex/scie/science.py

Line 67 in 74f34aa

MIN_SCIENCE_VERSION = Version("0.8.0")

After that, no wheels for psutil: The Pex PEX scies are all built from locks that do not allow sdists, only wheels. The current locking code just uses PyPI; so does not find a wheel for psutil for armv7l and dies. This will need plumbing of --use-pip-config to pick up your tox.ini setup of PIP_EXTRA_INDEX_URL, which currently only applies to bare use of Pip and not Pex internal uses of Pip (locking / resolving) which are hermetic by default. But even then, psutil 6.0 is only pre-built for 3.11 and the Pex PEX scies use 3.13 currently: https://www.piwheels.org/project/psutil/

I have no clue what timezone you're actually in right now - we're both working odd hours it seems. Combined with the fact the complete-platform gen will need to happen before release, but that this PR needs to land before complete-platform gen can be run means the CHANGES.md / pex/version.py changes will need to be backed out. I'm sorry about that - I should have seen that coming. Unless you say otherwise, I'm going to get involved in this PR with a few edits towards these ends to move this along and get out a release today.

No problem. Just push to this PR.

FYI I'm based in Sweden, so just go ahead with the release.

Also roll back the release prep changes. The Pex PEX scie complete platforms and lock will need to be re-generated before a release.

jsirois

We'll see how this goes. Your stuff looks good @Lauszus but my Pex PEX scie complete platform changes are a bit more doubtful.

jsirois · 2024-12-13T05:44:06Z

Alright, I was not able to get the Pex PEX scie working for Linux armv7l for the 2.26.0 release @Lauszus but you can still use Pex on and for armv7l; it's just that there is no native Pex PEX scie binary for the platform yet. I'm not even sure if you care about that. At any rate, I'll be adding one, if possible, in a follow up release.

Lauszus · 2024-12-13T09:00:20Z

Alright, I was not able to get the Pex PEX scie working for Linux armv7l for the 2.26.0 release @Lauszus but you can still use Pex on and for armv7l; it's just that there is no native Pex PEX scie binary for the platform yet. I'm not even sure if you care about that. At any rate, I'll be adding one, if possible, in a follow up release.

Thanks. No I don't really need a Pex PEX scie binary. My use cases is that I want to create a PEX for a Python application for some embedded devices on a cellular connection and I can just do that on a x86-64 machine.

Lauszus mentioned this pull request Dec 11, 2024

Add support for armv7l. a-scie/ptex#253

Merged

Lauszus force-pushed the feature/armv7l branch 4 times, most recently from d3b6a7b to 04d8687 Compare December 11, 2024 21:07

Lauszus marked this pull request as ready for review December 11, 2024 21:29

Lauszus force-pushed the feature/armv7l branch from 04d8687 to 7f75877 Compare December 12, 2024 10:43

Lauszus mentioned this pull request Dec 12, 2024

Added support for armv7l a-scie/lift#106

Merged

Lauszus force-pushed the feature/armv7l branch from 7f75877 to d3bffab Compare December 12, 2024 11:06

Lauszus commented Dec 12, 2024

View reviewed changes

Added support for armv7l

b0a473f

Lauszus force-pushed the feature/armv7l branch from d3bffab to b0a473f Compare December 12, 2024 11:08

Trying to get "gen-scie-platform" working

a600657

Lauszus commented Dec 12, 2024

View reviewed changes

.github/workflows/gen-scie-platforms.yml Show resolved Hide resolved

jsirois added 4 commits December 12, 2024 16:25

Fixup Pex PEX scie gen for armv7l.

f925872

Also roll back the release prep changes. The Pex PEX scie complete platforms and lock will need to be re-generated before a release.

Allow for the armv7l scie to be optional for now.

93a030c

Encode requiredness in ScieConfig for completeness.

43c3e97

Fixup test_pex_scie science version.

6fe6af1

jsirois approved these changes Dec 13, 2024

View reviewed changes

jsirois merged commit 195d8fd into pex-tool:main Dec 13, 2024
23 checks passed

Lauszus deleted the feature/armv7l branch December 13, 2024 08:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for armv7l #2620

Added support for armv7l #2620

Lauszus commented Dec 11, 2024

Lauszus commented Dec 11, 2024 •

edited

Loading

Lauszus commented Dec 11, 2024 •

edited

Loading

Lauszus Dec 12, 2024

Lauszus commented Dec 12, 2024 •

edited

Loading

Lauszus commented Dec 12, 2024

jsirois commented Dec 12, 2024

Lauszus commented Dec 12, 2024 •

edited

Loading

jsirois commented Dec 12, 2024 •

edited

Loading

Lauszus commented Dec 12, 2024

jsirois left a comment

jsirois commented Dec 13, 2024

Lauszus commented Dec 13, 2024 •

edited

Loading

Added support for armv7l #2620

Added support for armv7l #2620

Conversation

Lauszus commented Dec 11, 2024

Lauszus commented Dec 11, 2024 • edited Loading

Lauszus commented Dec 11, 2024 • edited Loading

Lauszus Dec 12, 2024

Choose a reason for hiding this comment

Lauszus commented Dec 12, 2024 • edited Loading

Lauszus commented Dec 12, 2024

jsirois commented Dec 12, 2024

Lauszus commented Dec 12, 2024 • edited Loading

jsirois commented Dec 12, 2024 • edited Loading

Lauszus commented Dec 12, 2024

jsirois left a comment

Choose a reason for hiding this comment

jsirois commented Dec 13, 2024

Lauszus commented Dec 13, 2024 • edited Loading

Lauszus commented Dec 11, 2024 •

edited

Loading

Lauszus commented Dec 11, 2024 •

edited

Loading

Lauszus commented Dec 12, 2024 •

edited

Loading

Lauszus commented Dec 12, 2024 •

edited

Loading

jsirois commented Dec 12, 2024 •

edited

Loading

Lauszus commented Dec 13, 2024 •

edited

Loading