Is eleven backends too many?

[jpivarski]

When I was last here, there were (I think) four backends, covering a basis set that included pure Python, pybind11, classic setuptools, new workflows like poetry. Starting a repo, which might or might not be the start of a project (because a common pattern is to do some offline development and then copy the results of that into a cookie-cut shell), I'm going to know basic things like whether I'll be needing pybind11 or a Rust compiler, or whether I want to go old-school setuptools or use a new all-in-one manager, but I'm not going to know whether I need

• poetry
• flit
• pdm
• trampolim
• whey, or
• hatch!

I had been thinking that this cookiecutter was for simplifying the process of getting a shell that I can put my code into, so that I don't need to set up pre-commit.ci, GitHub Actions, PyPI deployment, and all that. Because that set of DevOps tools (black, flake8, pylint, mypy, isort, ...) has more configuration options than I want to think about; I just want a set of good defaults. And when I run into issues, I'll modify my copy of the configurations, as needed.

But the number of simplifying tools is starting to rival the number of things that they simplify!

I guess it's great that scikit-hep/cookiecutter can produce hatch repos, if a user knows what hatch is and wants to use it, but do we have a recommended manager tool for pure Python repos? Is it poetry?

I'm about to make a repo for Awkward Pandas, to start on this roadmap item. It will be pure Python, but depend on awkward and pandas.

[jpivarski]

(I didn't intend for this to come across as pugnacious, but perhaps bewildered. It's astonishing to me that this space would fill up with so many alternatives, particularly when tool-discovery and choice-paralysis are exactly what they're all trying to address. I'm sure there's an XKCD for that.)

[henryiii]

For pure Python, we should follow the packaging.python.org recommendation (which I wrote) and use hatchling. It is absolutely not Poetry, which does not follow standards, and has some bad shortcomings. If you want to use hatch with hatchling, I guess you can; I'm waiting to try hatch until it has some locking support; I'll continue to use PDM until then. But Hatch (the CLI tool) and PDM can use any PEP 621 backend¹.

The 11 backends in a single tools has been fantastic in helping backends (including setuptools) improve their support for standards. Currently it's really classic setuptools, poetry, and then all the PEP 621 backends are nearly the same.

choice-paralysis are exactly what they're all trying to address

This is sort of the opposite of what we are trying to address. Up until very recently, there was one and only one choice. Setuptools. Nothing else. Even if coding anything even slightly complex for it was horrific. Even when it didn't support anything we need, like selecting the C++ standard. This is where all the "hack" packages came out, like scikit-build. Now we finally can build a better tool, and it's okay to pick the one that best matches your problem. I'm hopeing to make scikit-build a fantastic choice for CMake. I'm hoping mesonpy becomes a fantastic choice for Meson.

Great packages come about because there is choice. Jax & PyTorch took a lot of inspiration from TensorFlow and the other packages that came before them. I'm all for making an existing package better, but choice is needed to allow ideas to develop. The packaging space in Python had no choice at all before.

---

Choice paralysis of course can be a problem. The problem really is that I can't put nice recommendations into the choice list. If you don't know what to select:

• Pure Python: hatchling.
• C++: scikit-build (unless you have a very simple build, then pybind11 setuptools)
• Rust: maturin

I'd love to be able to add a little description or simi-hide some choices, but last I checked cookiecutter didn't let you do that.

PS: The good news is as long as you don't pick Poetry or classic setuptools, you can switch between backends by changing two lines. Differences come to speed, how source files are included, if you can use VCS versioning, better/worse error messages, and what plugins are supported (if any are). Hatchling is dominating in all those categories right now (except maybe plugins, but it's coming).

Footnotes

1. I'm sure PDM can, because I use Flit with it, I think Hatch can use any PEP 621 backend too because @ofek tends to follow standards. ↩

[henryiii]

PS: I also literally started moving plumbum to hatchling just before this post. tomerfiliba/plumbum#607 - Flit's file inclusion is rather broken, while Hatchling's .gitignore based inclusion is fantastic.

[jpivarski]

For pure Python, we should follow the packaging.python.org recommendation (which I wrote) and use hatchling.

Great, then that's what I'll use. However, I didn't see a recommendation like that in the scikit-hep.org developer pages (that's where I started) and I don't see where it says in packaging.python.org that we should use hatchling. The names of several options are given there.

choice-paralysis are exactly what they're all trying to address

This is sort of the opposite of what we are trying to address.

When I've advocated scikit-hep/cookie as the solution to the "packaging is hard" problem, it was from the point of view of a scientist who has just written a function or algorithm they want to share, and who might otherwise be inclined to put it in a GitHub repo without any tests or distribution mechanism, just some site-specific instructions in the README (if that). They're not going to have the background to know which packaging system to pick. (Myself included; the only one I've heard of outside the context of scikit-hep/cookie is poetry.)

Of course it's good that this space is being developed. I want packaging to get better, and it can probably only do that by eliminating cruft. Sure, we'll have to learn some new names and new best practices.

But from the perspective of a user of packaging tools, we need to be told, "start with this."

Choice paralysis of course can be a problem. The problem really is that I can't put nice recommendations into the choice list. If you don't know what to select:

• Pure Python: hatchling.
• C++: scikit-build (unless you have a very simple build, then pybind11 setuptools)
• Rust: maturin

Can the first 6 inches of the README be dedicated to this (and the cookiecutter gh:scikit-hep/cookie command)? Surrounded by thick whitespace?

Maybe this is a documentation problem. We can have all the details further down the page, but I still think that the most important target audience for this particular tool is a clueless user (like me) who needs to find the recommendation box. That's not true of all tools; this one, I feel, is especially for people who want to use it as an appliance, rather than a Swiss Army knife.

[ofek]

I think Hatch can use any PEP 621 backend too

Yup! https://hatch.pypa.io/1.2/meta/faq/#interoperability

[henryiii]

I'm not surprised there's no recommendation here. Hatch didn't hit 1.0 till PyCon this year, and get moved to PyPA until after that. It didn't become the recommended backend for the packaging tutorial until the beginning of this week. And I haven't really had a chance to use it properly yet - I've mostly used Flit as a backend so far on my project. Though in helping with the transition to Flit in the PyPA (we need Flit because it has 0 dependencies for bootstrappable libraries, and is designed specifically to be easy to bootstrap), I've realized one of the reasons the author of Flit isn't interested in having it recommended is because it is intentionally crippled in PEP 517 mode. Hatchling correctly handles file exclusion in the SDist via .gitignore, which works great (even if git is not installed).

So, to be clear, I agree a recommendation is needed, but probably should wait a little more until I move a couple of packages to hatchling too. Though others that I trust have already moved to it and are very happy (and I've liked everything I've seen so far). Pipx moved, Tox moved, Nox either moved or was planning on it, something like that. Etc.

The PyPA tutorial states: "this tutorial uses Hatchling by default". Okay, seems like in review edits this was toned down quite a lot, but it's the default selected backends; we expect most users to not bother to toggle. It is the first backend now on scikit-hep, too.

The goal of having a few backends on that page was to provide a visual example of what changes (just two lines) between backends, and to help keep the tutorial backend agnostic (I fought long and hard to keep a tabbed toggle here!).

---

I think the recommendation should be make clear in the scikit-hep developer pages. The cookie cutter explicitly states you should read the pages first, so I think it can get away with being a little less helpful. Though if I can add a heading that states recommended choices, that would be a good idea too. (cookiecutter's CLI is terrible, and they know it and only will improve it with funding. Which it deserves, IMO).

[henryiii]

(I've been really busy the last weeks, ended up off a lot the last couple of weeks, and a month of travel including SciPy and Snowmass is coming up. And I need to help with extensionlib which I"m excited about ;) ).