add `--keyring-provider` flag to configure keyring-based authentication #2592

tdyas · 2024-11-13T00:44:57Z

Add a --keyring-provider flag to allow Pex callers to configure Pip to use a keyring provider without the need to loosen the Pip isolated mode entirely as would be done if --use-pip-config were passed.

pex/pip/tool.py

jsirois · 2024-11-13T17:56:25Z

The "./dtox.sh -e py38-pip22_3_1-integration --shard 2/2" shard failure is bitrot from 11/11 addressed here: #2595

tdyas · 2024-11-13T19:45:11Z

The "./dtox.sh -e py38-pip22_3_1-integration --shard 2/2" shard failure is bitrot from 11/11 addressed here: #2595

With dtox.sh, my unit test appears to pass. I'll leave this PR in WIP state though while I work on the Pants-side of the ultimate change.

jsirois · 2024-11-15T03:31:14Z

tests/test_pip.py

@@ -415,16 +415,21 @@ def test_keyring_provider(
    download_dir = os.path.join(str(tmpdir), "downloads")
    assert not os.path.exists(download_dir)

-    with ENV.patch(PIP_KEYCHAIN_PROVIDER="invalid") as env, environment_as(**env):
-        assert "invalid" == os.environ["PIP_KEYCHAIN_PROVIDER"]
+    with ENV.patch(PIP_KEYRING_PROVIDER="invalid") as env, environment_as(**env):


I'm not sure if you've seen /home/jsirois/dev/pex-tool/pex/tests/integration/test_keyring_support.py, but its probably what you eventually want to amend to test the "import" + --extra-pip-requirement and "subprocess" + PATH means of hooking the keyring provider.

Couldn't I just update that test to use the new --keyring-provider option instead of the existing --use-pip-config option? (Or paramterize the test to try both methods?)

~the latter is exactly what I was suggesting, yes.

Hopefully you've noticed though how tortured the very concept of a keyring provider is. There is a nasty bootstrap problem for any locked down org.

Yeah, the whole shebang is a marvelous house of cards. The work in Pants using this support focuses on --keyring-provider=subprocess with a trampoline keyring script so Pants won't need to bootstrap keyring PyPi at all.

Absolutely. I'll hint you 1st: how do you think Pex implements --pip-version any-not-vendored-version?

The answer to that is fairly obvious and I'm sure you've already forehead smacked. That said, solving is thornier. I already had to deal with how to bootstrap --pip-version any-not-vendored-version here:

pex/pex/pip/installation.py

Lines 290 to 305 in 3361e79

bootstrap_pip_version = try_(

compatible_version(

targets,

PipVersion.VENDORED,

context="Bootstrapping Pip {version}".format(version=version),

warn=False,

)

)

if bootstrap_pip_version is not PipVersion.VENDORED and not extra_requirements:

return _pip_installation(

version=version,

iter_distribution_locations=_bootstrap_pip(version, interpreter=interpreter),

interpreter=interpreter,

fingerprint=_fingerprint(extra_requirements),

use_system_time=use_system_time,

)

The context there is using a new enough Python (3.12+) where distutils is gone from the stdlib and vendored Pip fails as a result. In that situation I had to find another way to bootstrap a specific Pip version without using vendored Pip. So, basically, iff using the current python to python -mvenv nets a venv with a Pip new enough to support --keyring-provider, the thorny bootstrap can be just the thorny problem you described, solved with --keyring-provider subprocess and some pre-arranged provider on the PATH. If not, you have an even thornier bootstrap problem than you expected.

Following up a little more, --keyring-provider was added for Pip 23.1. Python 3.10.15 and older come with ensurepip that yield at most Pip 23.0.1; so you can only count on Python>=3.11.4 (3.11 only hit bundled Pip>=23.1 here: python/cpython#103752) for bootstrapping any --pip-version >= 23.1 with --keyring-provider being passed as part of the bootstrap process.

I think the intended user of my Pants change will be just fine with Python >= 3.11.4. Is that enough to ensure bootstrap of a pip supporting --keyring-provider though? I assume the pip bootstrap happens with the user Python chosen by Pants and not the Pants-specific PBS Python used by scie-pants?

I think the intended user of my Pants change will be just fine with Python >= 3.11.4. Is that enough to ensure bootstrap of a pip supporting --keyring-provider though?

Not with the code as-it-is, since the python -mvenv ... bootstrap trick is not used as aggressively as it could be. That code pointer I gave does highlight the existing mechanism though which should point the way to using it more aggressively.

In short though, knowing which version of Python you have is clearly trivial; then using that to fail fast or move forward with bootstrapping Pip via -mvenv and passing --keyring-provider after that is definitely in the realm of just putting in the work.

As I mentioned below though, I'll think on this whole situation a bit and report back. There may be some better way than either this or picking a random new Pip to vendor (then can't live without it feature with bootstrapping issue comes up in a newer Pip and surely vendoring a 3rd version is a no go, etc...).

tdyas · 2024-11-26T19:05:03Z

@jsirois: Would you be amenable if I submitted an upgrade of the vendored pip in Pex? Is it feasible or desirable to do so? I.e., what should I know about that solution path which I probably do not currently know?

jsirois · 2024-11-26T19:34:22Z

I would not. Hard rules are not breaking people and the current vendored Pip is the last version to support 2.7; so it has to stay. I'll think on a solution a bit more over the next few days and report if I can concoct anything.

tdyas · 2024-11-26T19:50:41Z

I would not. Hard rules are not breaking people and the current vendored Pip is the last version to support 2.7; so it has to stay. I'll think on a solution a bit more over the next few days and report if I can concoct anything.

What about vendoring two versions of pip? Keep the existing vendored pip for Python 2.7 compatibility and then vendor a newer pip (with for example --keyring-provider) support. The second vendored copy will be documented as something Pex may update from time to time.

Pex can then (1) offer the choice of the newer vendored pip via an option and/or (2) use the newer vendored copy by default instead of the older vendored copy except where Python 2.7 is involved (in which case the existing vendored copy would be the default).

Thoughts?

jsirois · 2024-11-26T21:16:34Z

What about vendoring two versions of pip?

I'm definitely not a fan. I thought I was going to have to do this to support Python 3.12 a year or two ago and worked up the python -mvenv ... (~python -mensurepip) trick at that time specifically to avoid this.

Like I said, I'm going to think on this a bit and report back.

jsirois · 2024-11-26T23:30:25Z

Ok, so @tdyas, I think your particular problem is solved by existing means if you just plumb fail fast code here to reject --keyring-provider with a useful message when the active interpreter is <3.11.4. That combined with appropriate warning verbage in the option help would allow Pants to just switch to the Pex PEX scies which embed Python 3.12.7 today (and that will only climb in future releases). That Pex will default to Pip 23.2 instead of vendored (Python 3.12 can't run vendored):

pex/pex/pip/version.py

Lines 244 to 249 in 50ff75a

    
           v23_2 = PipVersionValue( 
        
               version="23.2", 
        
               setuptools_version="68.0.0", 
        
               wheel_version="0.40.0", 
        
               requires_python=">=3.7,<3.13", 
        
           )

pex/pex/pip/version.py

Line 322 in 50ff75a

DEFAULT = DefaultPipVersion(preferred=(VENDORED, v23_2, v24_1))

The bootstrap Pip will be 24.2, the version of Pip that ships with PBS Python 3.12.7. Since Pip 23.1 is the magic version sprouting --keyring-provider support, all is good. (EDIT: This just got bumped to PBS Python 3.13.0 which also includes Pip 24.2 for the next release in #2604).

I've already lightly pushed Pants to switch away from the Pex PEX to the Pex PEX scies since those scies embed psutil which makes the new pex3 cache prune command useful in the face of concurrent runs (see: #2586 (comment)). This would be another reason to do so.

AFAICT that's a basic solution that leaves open even better solutions going forward that cover more than just Pants' use case (i.e.: making this all tractable for Pex installed in a venv users too). For example, I'm much less worried about binary size with the Pex PEX scies since they are already large - dominated by the embedded CPython distribution size - and those scies might be able to embed keyring wheels in the future such that keyring comes pre-installed when Pip 1st runs. Then, at least, the default keyring providers would be ready to go. Only users of custom keyring providers would still have bootstrap issues to wrestle with.

tdyas · 2024-12-11T22:13:49Z

I tried the macOS/x86 SCIE. Apple really does not like unsigned binaries. Opened #2621 to discuss.

tdyas · 2025-01-11T04:15:42Z

@jsirois: Any advice on the CI failure at https://github.com/pex-tool/pex/actions/runs/12720783453/job/35462921525?pr=2592#step:5:5585 for pip: no such option: --keyring-provider? The test was of Pip v23.3.2 which should support the --keyring-provider option.

jsirois · 2025-01-11T04:21:10Z

Before looking deeply, is this question coming with the fact Pex generally bootstraps newer Pips using vendored Pip 20.3.4 understood / remembered from earlier conversations?

tdyas · 2025-01-11T04:23:51Z

Before looking deeply, is this question coming with the fact Pex generally bootstraps newer Pips using vendored Pip 20.3.4 understood / remembered from earlier conversations?

Potentially. Then my question becomes, how do I exclude use of the option under test (i.e., --keyring-provider) when bootstrapping the newer Pip in the test?

jsirois · 2025-01-11T04:25:42Z

1st, proof:

Failed to spawn a job for /development/pex/.tox/py311-pip23_3_2-integration/bin/python: pid 13544 -> /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/venvs/1/29f65b504ee0cb8e829c458f76fa5bc8a2c798b8/30cada8422d156b16f1bae989acbfba03c485755/bin/python /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/venvs/1/29f65b504ee0cb8e829c458f76fa5bc8a2c798b8/30cada8422d156b16f1bae989acbfba03c485755/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated --keyring-provider import --log /tmp/pex-pip-log.hjvi9c3z/pip.log -v --cache-dir /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/pip/1/20.3.4-patched/pip_cache download --dest /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/downloads/1/.tmp/resolver_download.bdwjtmj_/development.pex..tox.py311-pip23_3_2-integration.bin.python pip==23.1.2 setuptools==67.7.2 wheel==0.40.0 pex-test-backend --index-url http://localhost:59775/root/pypi/+simple/ --find-links /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/find-links --trusted-host localhost:59775 --retries 5 --timeout 15 exited with 2 and STDERR:

Search that line for 20.3.4.

tdyas · 2025-01-11T04:26:21Z

Failed to spawn a job for /development/pex/.tox/py311-pip23_3_2-integration/bin/python: pid 13544 -> /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/venvs/1/29f65b504ee0cb8e829c458f76fa5bc8a2c798b8/30cada8422d156b16f1bae989acbfba03c485755/bin/python /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/venvs/1/29f65b504ee0cb8e829c458f76fa5bc8a2c798b8/30cada8422d156b16f1bae989acbfba03c485755/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated --keyring-provider import --log /tmp/pex-pip-log.hjvi9c3z/pip.log -v --cache-dir /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/pip/1/20.3.4-patched/pip_cache download --dest /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/pex-root/downloads/1/.tmp/resolver_download.bdwjtmj_/development.pex..tox.py311-pip23_3_2-integration.bin.python pip==23.1.2 setuptools==67.7.2 wheel==0.40.0 pex-test-backend --index-url http://localhost:59775/root/pypi/+simple/ --find-links /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_provider_Tru-fbc100/find-links --trusted-host localhost:59775 --retries 5 --timeout 15 exited with 2 and STDERR:

So yes, it was installing the newer pip since it has pip==23.1.2.

jsirois · 2025-01-11T04:29:04Z

Then my question becomes, how do I exclude use of the option under test (i.e., --keyring-provider) when bootstrapping the newer Pip in the test?

IIUC this is not a test issue to be fixed but a production issue to be fixed. Fix prod and test just works. So, in prod, that option can only be emitted for versions of Pip that support it. Presumably you're not conditionally emitting the option. That said, the fact the option is conditional means the bootstrapping Pip will not be using a keyring provider, which is an issue. I feel like we've been around this loop before?

tdyas · 2025-01-13T18:16:37Z

Given this test failure, seems like this test will either need a manually-bootstrapped Pip version or (most likely) just restrict to running with Pip v23.1 and higher?

jsirois · 2025-01-13T18:24:37Z

Given [this test failure], seems like this test will either need a manually-bootstrapped Pip version or (most likely) just restrict to running with Pip v23.1 and higher?

Tom, can you fill in [this test failure] with a link?

tdyas · 2025-01-13T18:48:46Z

Given [this test failure], seems like this test will either need a manually-bootstrapped Pip version or (most likely) just restrict to running with Pip v23.1 and higher?

Tom, can you fill in [this test failure] with a link?

Filled in. https://github.com/pex-tool/pex/actions/runs/12752942699/job/35543486671?pr=2592#step:5:1947

jsirois · 2025-01-13T19:00:30Z

@tdyas I am super duper confused.

test_keyring_provider[20.3.4-patched] is https://github.com/pex-tool/pex/actions/runs/12752942699/job/35543486671?pr=2592#step:5:1947

Your "this test" link is line 405 here:

pex/tests/test_pip.py

Lines 398 to 417 in 30c2ec8

    
           @applicable_pip_versions 
        
           def test_extra_pip_requirements_pip_not_allowed( 
        
               create_pip,  # type: CreatePip 
        
               version,  # type: PipVersionValue 
        
               current_interpreter,  # type: PythonInterpreter 
        
           ): 
        
               # type: (...) -> None 
        
               with pytest.raises( 
        
                   ValueError, 
        
                   match=re.escape( 
        
                       "An `--extra-pip-requirement` cannot be used to override the Pip version; use " 
        
                       "`--pip-version` to select a supported Pip version instead. Given: pip~=24.0" 
        
                   ), 
        
               ): 
        
                   create_pip( 
        
                       current_interpreter, 
        
                       version=version, 
        
                       extra_requirements=[Requirement.parse("pip~=24.0")], 
        
                   )

Those two are unrelated. Is one of your two links wrong?

Skipping past 20 questions perhaps:
The failing test is yours. It fails because it has not adapted to the fact that you now conditionally inject a --keyring-provider arg to the Pip arglist:

>           keyring_arg = job._command.index("--keyring-provider")
E           ValueError: tuple.index(x): x not in tuple

And so the corresponding question makes no sense. Presumably you want to test both branches of your new code - the branch that injects the arg and the branch that does not and warns.

tdyas · 2025-01-13T19:36:03Z

@tdyas I am super duper confused.

test_keyring_provider[20.3.4-patched] is https://github.com/pex-tool/pex/actions/runs/12752942699/job/35543486671?pr=2592#step:5:1947

Your "this test" link is line 405 here:

pex/tests/test_pip.py

Lines 398 to 417 in 30c2ec8

@applicable_pip_versions

def test_extra_pip_requirements_pip_not_allowed(

create_pip, # type: CreatePip

version, # type: PipVersionValue

current_interpreter, # type: PythonInterpreter

):

# type: (...) -> None

with pytest.raises(

ValueError,

match=re.escape(

"An `--extra-pip-requirement` cannot be used to override the Pip version; use "

"`--pip-version` to select a supported Pip version instead. Given: pip~=24.0"

),

):

create_pip(

current_interpreter,

version=version,

extra_requirements=[Requirement.parse("pip~=24.0")],

)

Those two are unrelated. Is one of your two links wrong?

Skipping past 20 questions perhaps:

The failing test is yours. It fails because it has not adapted to the fact that you now conditionally inject a --keyring-provider arg to the Pip arglist:
>           keyring_arg = job._command.index("--keyring-provider")

E           ValueError: tuple.index(x): x not in tuple
And so the corresponding question makes no sense. Presumably you want to test both branches of your new code - the branch that injects the arg and the branch that does not and warns.

Sorry will fix link after I get back from my walk to clear my head of these very code weeds.

tdyas · 2025-01-13T20:37:55Z

The failing test is yours. It fails because it has not adapted to the fact that you now conditionally inject a --keyring-provider arg to the Pip arglist:

Good point. And that is the failing test. Will update. (I running both macOS and Linux laptops on this PR and pasted link from wrong machine.)

pex/pip/tool.py

pex/resolve/resolver_options.py

tdyas · 2025-01-15T14:42:55Z

CI tests are passing now. Marked ready for review. I added a few questions around what sort of documentation this needs to have.

tdyas · 2025-01-15T15:45:11Z

CI failed due to an infrastructure error in downloading. https://github.com/pex-tool/pex/actions/runs/12790796085/job/35657416561#step:5:2870

Weirdly, I don't see an option in the GitHub Actions UI for the build to retry just the failed job.

pex/resolve/resolver_options.py

jsirois · 2025-01-15T20:07:38Z

Thanks @tdyas. I'm going to get a PEP-723 locking feature banged out this afternoon which will also include the release prep for 2.29.0 which will include this change.

jsirois · 2025-01-17T02:53:51Z

Ok, now released in 2.29.0: https://github.com/pex-tool/pex/releases/tag/v2.29.0

tdyas · 2025-01-17T03:20:11Z

Ok, now released in 2.29.0: https://github.com/pex-tool/pex/releases/tag/v2.29.0

Thanks!

add --with-extra-pip-arg flag

3207adc

jsirois reviewed Nov 13, 2024

View reviewed changes

pex/pip/tool.py Outdated Show resolved Hide resolved

tdyas changed the title ~~add --with-extra-pip-arg flag~~ [WIP] add --with-extra-pip-arg flag Nov 13, 2024

tdyas added 2 commits November 12, 2024 23:05

revert setuptools addition

c74dc81

--keychain-provider

1fc4eb7

tdyas added 4 commits November 14, 2024 09:14

more spots

2f21708

fix incorrect variable reference

ef3a8f2

name the option correctly

ccc213a

fix test and comments

e2d0c26

jsirois reviewed Nov 15, 2024

View reviewed changes

tdyas changed the title ~~[WIP] add --with-extra-pip-arg flag~~ [WIP] add --keyring-provider flag Nov 19, 2024

tdyas added 2 commits November 19, 2024 11:47

Merge branch 'main' into extra_pip_args

ba6d06d

add integration test

31662b6

tdyas added 3 commits January 10, 2025 16:54

Merge branch 'main' into extra_pip_args

0963e08

selected wrong side of the merge

755657e

more post-merge cleanup for lock_downloader

5b8894f

tdyas added 5 commits January 13, 2025 16:55

improve pip test

a070c68

redo warning message

5bacd6b

soften language and fix import

8dd5098

remove unused import

f903bf8

more cleanups

11c38c9

tdyas marked this pull request as ready for review January 15, 2025 14:27

tdyas requested a review from jsirois January 15, 2025 14:27

tdyas commented Jan 15, 2025

View reviewed changes

pex/pip/tool.py Outdated Show resolved Hide resolved

better help message

352eb4e

tdyas commented Jan 15, 2025

View reviewed changes

pex/resolve/resolver_options.py Show resolved Hide resolved

little edit

69f822d

fmt

9ae677d

tdyas changed the title ~~[WIP] add --keyring-provider flag~~ add --keyring-provider flag to configure keyring-based authentication Jan 15, 2025

jsirois reviewed Jan 15, 2025

View reviewed changes

pex/resolve/resolver_options.py Outdated Show resolved Hide resolved

tdyas added 2 commits January 15, 2025 12:39

edit version language

8be4043

update warning to refer to --keyring-provider

668caa6

jsirois approved these changes Jan 15, 2025

View reviewed changes

jsirois merged commit 3dfda00 into pex-tool:main Jan 15, 2025
23 checks passed

tdyas deleted the extra_pip_args branch January 17, 2025 03:20

	bootstrap_pip_version = try_(
	compatible_version(
	targets,
	PipVersion.VENDORED,
	context="Bootstrapping Pip {version}".format(version=version),
	warn=False,
	)
	)
	if bootstrap_pip_version is not PipVersion.VENDORED and not extra_requirements:
	return _pip_installation(
	version=version,
	iter_distribution_locations=_bootstrap_pip(version, interpreter=interpreter),
	interpreter=interpreter,
	fingerprint=_fingerprint(extra_requirements),
	use_system_time=use_system_time,
	)

add --keyring-provider flag to configure keyring-based authentication #2592

add --keyring-provider flag to configure keyring-based authentication #2592

Conversation

tdyas commented Nov 13, 2024 • edited Loading

jsirois commented Nov 13, 2024

tdyas commented Nov 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdyas Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdyas commented Nov 26, 2024

jsirois commented Nov 26, 2024 • edited Loading

tdyas commented Nov 26, 2024 • edited Loading

jsirois commented Nov 26, 2024

jsirois commented Nov 26, 2024 • edited Loading

tdyas commented Dec 11, 2024

tdyas commented Jan 11, 2025

jsirois commented Jan 11, 2025 • edited Loading

tdyas commented Jan 11, 2025

jsirois commented Jan 11, 2025

tdyas commented Jan 11, 2025 • edited Loading

jsirois commented Jan 11, 2025

tdyas commented Jan 13, 2025 • edited Loading

jsirois commented Jan 13, 2025

tdyas commented Jan 13, 2025

jsirois commented Jan 13, 2025 • edited Loading

tdyas commented Jan 13, 2025

tdyas commented Jan 13, 2025

tdyas commented Jan 15, 2025

tdyas commented Jan 15, 2025

jsirois commented Jan 15, 2025

jsirois commented Jan 17, 2025

tdyas commented Jan 17, 2025

add `--keyring-provider` flag to configure keyring-based authentication #2592

add `--keyring-provider` flag to configure keyring-based authentication #2592

tdyas commented Nov 13, 2024 •

edited

Loading

tdyas Nov 19, 2024 •

edited

Loading

jsirois Nov 19, 2024 •

edited

Loading

jsirois commented Nov 26, 2024 •

edited

Loading

tdyas commented Nov 26, 2024 •

edited

Loading

jsirois commented Nov 26, 2024 •

edited

Loading

jsirois commented Jan 11, 2025 •

edited

Loading

tdyas commented Jan 11, 2025 •

edited

Loading

tdyas commented Jan 13, 2025 •

edited

Loading

jsirois commented Jan 13, 2025 •

edited

Loading