[python/ci] Add CI job to test building wheel from sdist

[jdblischak]

Issue and/or context:

TileDB-SOMA-Py is built and distributed in many different ways (from source, PyPI wheels, conda binaries). We want to identify any packaging-related problems immediately in the PR that generates them instead of in a nightly build or worse after the release has been made.

Changes:

This PR includes 2 main changes:

• The first is rebranding my existing CI pipeline to python-ci-packaging.yml. The original name was from my initial PR to test the shared object copying behavior ([python] Do not copy libtiledbsoma.so into package_data if it already exists #1937, [ci] Add MacOS job to test shared object copying #2220). However, I had already previously expanded the pipeline to test the pure setuptools installation setup (which we use in tiledbsoma-feedstock) ([python] Fix and test setup.py flags to external shared objects #2221). With the addition of a 3rd job (below), I wanted to update the CI to better match its purpose

• The 1.9.0 release was delayed because the PyPI wheels could not be built ([python/ci] Unbreak PyPI wheel builds #2337, [python/ci] Unbreak wheel builds #2339). While we build wheels in many of the existing CI pipelines, none build the wheel from a source tarball (sdist) as is done in python-packaging.yml, so this error wasn't caught until post release. With this new CI job, we will immediately detect if a crucial file is absent from the source tarball

Notes for Reviewer:

See the wiki page Debugging wheel‐build issues for more details

[johnkerl]

@jdblischak is this partially duplicating
https://github.com/single-cell-data/TileDB-SOMA/blob/main/.github/workflows/python-packaging.yml
?

[jdblischak]

is this partially duplicating https://github.com/single-cell-data/TileDB-SOMA/blob/main/.github/workflows/python-packaging.yml?

I wouldn't describe it as duplication. I would describe it more as preparation for the official building of the PyPI wheels.

Differences:

• Uses transparent python commands instead of the opaque action pypa/cibuildwheel, and therefore easier to debug
• Only builds two wheels total (python 3.11 on Ubuntu and macOS), so it is much faster. It's not creating the wheels for distribution, but instead just providing a quick check that all the required files are distributed in the source tarball

In summary, the goal is to identify any potential problems with building the PyPI wheels when a PR is submitted, and not at release time.

[johnkerl]

@jdblischak can we please get this verbatim into the affected YAML file as a block comment?

Differences:

Uses transparent python commands instead of the opaque action pypa/cibuildwheel, and therefore easier to debug
Only builds two wheels total (python 3.11 on Ubuntu and macOS), so it is much faster. It's not creating the wheels for distribution, but instead just providing a quick check that all the required files are distributed in the source tarball
In summary, the goal is to identify any potential problems with building the PyPI wheels when a PR is submitted, and not at release time.

[johnkerl]

@jdblischak SGTM as long as CI resource consumption on every PR doesn't become excessive ... main issue being macos runners: choices are 5, 5, 5, or enterprise with no "step up" to 10 or 20 say ...

[johnkerl]

@jdblischak SGTM as long as CI resource consumption on every PR doesn't become excessive ... main issue being macos runners: choices are 5, 5, 5, or enterprise with no "step up" to 10 or 20 say ...

[jdblischak]

as long as CI resource consumption on every PR doesn't become excessive

This pipeline does spawn a lot of jobs. Two reasons why I think it avoids being excessive:

1. Aggressive use of path-based filters. This pipeline is only triggered if a file that could affect the Python installation process is affected (eg setup.py, pyproject.toml, etc). A PR that edits the C++ code, R code, or even the non-installation Python files will never trigger these jobs
2. The macOS jobs all finish in under 10 minutes. So even if it hogs our parallel macOS jobs, it will be for a brief period of time

If it does start to become excessive, the first macOS job I would drop is the one added in this PR, sdist. It is mainly to catch files missing from the source tarball, and that behavior should be consistent across Linux and macOS.

[johnkerl]

Cool and thanks @jdblischak ! I indeed just looked for paths: in your affected YAML and we have protection there against over-running so my concern was premature (I should have checked there first). But thanks for looking, and thanks for the prioritization if/when "something has to go" someday -- really appreciate it! :)