Reduce complexity of Helm chart's Dockerfiles and define expectations

[consideRatio]

We have two Dockerfiles that we build and use within the Helm chart:

• dask-gateway-server/Dockerfile: used by the controller and api pods.
• dask-gateway/Dockerfile: used by default for scheduler and worker pods.

Reduce complexity - respect setup.py's install_requires

I'd like to make these images install dask-gateway and dask-gateway-server in a more straight forward manner. Currently we first install conda-forge provided dependencies matching those we know we need by a hardcoded list and ignore the dependency details specified in setup.py.

This adds complexity to this project! For example in #522 (comment) I thought that setup.py defined restrictions on a dependency that caused test failures for us would be respected also in the images we build for our k8s tests - but it wasn't and I kept looking for other reasons of failing k8s tests.

Document expectations - how they intended to be used?

These images are around, but it isn't clear for us maintainers or end users how they are to be used. My perception is that end users are not to derive their own images from these, but instead install dask-gateway with a package manager in their own image whenever they want to customize a scheduler or worker image.

In practice, I think we should declare that the images are not meant to be used as base images, nor are the Dockerfile's, which instead describe that users are supposed to do a conda-forge installation in their custom image to acquire a scheduler and a worker suitable image.

Related

• I've have tried to avoid duplicated specification of Python dependencies as the maintenance complexity that follows spreading them out is high
  De-duplicate package dependencies #516
  ci: de-duplicate deps by docs/requirements.txt and tests/requirements.txt and update CI images #519
• I failed to realize that dependencies in setup.py wasn't respected in these images as --no-deps is specified, and a set of hardcoded conda packages are installed ahead of time as a separate set of dependencies.
  CI failures for main/k8s tests has been introduced without changes in this repo #522
• There have been discussion related to end users use of the repo's dask-gateway/Dockerfile
  Make conda/mamba flavor a build arg #489

Action points

• Discuss: is it acceptable to lower the expectations and asking users to not rely on these images?
• Create a PR related to Reduce complexity of Helm chart's Dockerfiles and define expectations #524 (comment) including documentation updates, a disclaimer in the Dockerfiles, etc.
  Helm chart images: conda removed -> pip only, usage disclaimer added, minimized Dockerfile complexity #533

[consideRatio]

In a meeting with @jcrist and @jacobtomlinson we discussed this. My conclusions is that it was found acceptable if we would take the following actions regarding various images we have.

1. dask/dask-gateway-server, an image used by the Helm chart's controller and api pod, should not be something end users derive their own image from. If they do for some unknown reason though, they shouldn't expect this project to respect backward compatebility or changelogs etc.
2. dask/dask-gateway, an image used by the Helm chart as a sample image for a clusters scheduler/worker pods, should be treated as a sample image that people shouldn't derive their own scheduler/worker images from. Instead, they should build their scheduler/worker images in any way according to documentation even though that documentation may require an update.
3. The example Dockerfile, just an example and not an image we build, is probably a bit too complicated to maintain and keep updated. As outdated examples are sometimes problematic I think the best course of action is to rely on the documentation on how to create your own image, and suggest asking users to look at the dask/dask-gateway image for practical inspiration on how you may do it.