Add initial support for license expression (PEP 639)

[cdce8p]

Summary of changes

• Update validate_pyproject to 0.23
  https://github.com/abravalheri/validate-pyproject/blob/v0.23/CHANGELOG.rst
  https://github.com/abravalheri/validate-pyproject/releases/tag/v0.23
• Initial support for license expression in project metadata

 [project]
 name = "spam"
 ...
-license = {text = "MIT"}
+license = "MIT"

If present, the normalized license expression is written to the License field until metadata version 2.4 is supported.

Refs: #4629

[abravalheri]

Hi @cdce8p thank you very much for having a look on this.

We probably have to organise so that PEP 639 lands after PEP 685 implementation (ideally we want a couple of days/weeks between the two so that users have time to report potential bugs).

[cdce8p]

We probably have to organise so that PEP 639 lands after PEP 685 implementation (ideally we want a couple of days/weeks between the two so that users have time to report potential bugs).

Maybe I'm missing something obvious but I'm not quite sure how PEP 685 is related. Would you mind explaining it a bit more?

The way I've designed the PR, it shouldn't depend on anything. It just allows users to specify license = MIT in pyproject.toml which will be used for the License-Expression field. Note, the metadata version is still 2.2 (not 2.4 which is the first to "require" support for PEP 639).

[cdce8p]

Opened #4734 with just the validate-pyproject update. This will make the changes here easier to review.

[abravalheri]

Maybe I'm missing something obvious but I'm not quite sure how PEP 685 is related. Would you mind explaining it a bit more?

The problem is that the metadata version of the core metadata is incremental:

• PEP 685 stablishes 2.2
• PEP 639 stablishes 2.3

So we cannot implement PEP 639 before 685, otherwise metadata validation may fail.

Even if the PR is independent we cannot merge it now because of the nature of the releases of setuptools (we cannot risk PEP 639 support being released before PEP 685 support).

[cdce8p]

So we cannot implement PEP 639 before 685, otherwise metadata validation may fail.

The validation will only happen once we update the metadata version.
https://github.com/pypa/packaging/blob/24.2/src/packaging/metadata.py#L826-L831

If not, a lot of wheel uploads would probably fail as we currently include License-File metadata that's not up to spec. #3596 (Fix: #4728).

What I'm proposing here is simply to add the basic support to setuptools. It won't get validated by PyPI until we're ready to move to 2.4 but then all projects using the "new" syntax for pyproject.license will already be ready to go.

[abravalheri]

For sure the existence of the License-File in the setuptools-produced metadata with version 2.1 is violating the specs. However, this is a well known violation that the Python ecosystem has grown acquainted to since the early prototype implementation of PEP 639, and for better or worse, learned how to deal with.

By accepting only some aspects of PEP 639, we add other kinds of spec violations that some tools may not be prepared to deal with.

So for the sake of reducing the unknown risks, I propose we pipeline the changes you kindly contributed after the implementation of 685. Unfortunately, it means that people cannot start modifying their pyproject.toml immediately, but it feels safer.

[abravalheri]

Sorry, I have been using the wrong PEP number, sorry for the confusion.

PEP 685 is important, but the really big one for us to implement before 639 is PEP 643 (the one with Dynamic).

[cdce8p]

So for the sake of reducing the unknown risks, I propose we pipeline the changes you kindly contributed after the implementation of 685. Unfortunately, it means that people cannot start modifying their pyproject.toml immediately, but it feels safer.

I understand your reasoning. One last suggestion. The last days I've been looking into how hatchling does it as I noticed it was one of the earliest adopters of License-Expression in the metadata. Turns out the packaging update caused upload failures pypa/hatch#1786. I can't reproduce it with PyPI at the moment. However, packaging==24.2 rejects any License-Expression and License-File keys with metadata <2.4. That would be relevant for us too. After the issue report, they removed the PEP 639 keys from the metadata and now include the project.license as part of the License key.

My suggestion

• Remove the License-File keys from the generate metadata as this might cause upload failures once the new packaging version is rolled out to PyPI.
• Keep the license files in the .dist-info folder but move them to the licenses subfolder in according to the spec. Essentially Store license-files in licenses subfolder #4728
• Add support for project.license by back-populating the License key, similar to Back-populate string license fields hatch#1805 Together with Update validate-pyproject to 0.23.0 #4734, this would allow users to use the new syntax while keeping the old metadata format.

What do you think?

[cdce8p]

@abravalheri With the last updates, I removed the License-Expression field and instead back-populate the License field with the validated license expression. That's the approach currently taken by hatchling. Since it doesn't add any new fields, would it be ok to do now?

[abravalheri]

@cdce8p I updated this part of the code to solve the merge conflict. I hope that is OK.

[cdce8p]

Thanks. Feel free to ping me, especially if one of my PRs need attention. I'm usually fairly quick to respond.

Small side note. If I review PRs from other people, I only update them with merge commits. The rebase one makes it unnecessarily difficult for the OP to follow and understand what's changed on their own PR. I myself only do force pushes if my PR hasn't been reviewed yet, afterwards only merge commits.

If that's ok with you, I'll force push the rebase once #4796 is done. Just so I'm aware what has changed.

[abravalheri]

If that's ok with you, I'll force push the rebase once #4796 is done. Just so I'm aware what has changed.

Thanks @cdce8p. That would be helpful.
I usually try to avoid merging the full main branch back to a PR, because it messes up the graph visualisation (which becomes difficult to follow), that is why I went with a rebase, sorry if that disrupted your workflow.

[cdce8p]

I usually try to avoid merging the full main branch back to a PR, because it messes up the graph visualisation (which becomes difficult to follow), that is why I went with a rebase, sorry if that disrupted your workflow.

Don't worry about it. Guess everyone has their own preferred workflow. Just wrote that as it took me a bit to actually see what you've changed during the rebase.

Thanks @cdce8p. That would be helpful.

👍🏻