Deploying a MVP BinderHub #1404

sgibson91 · 2022-06-08T14:05:08Z

related #1280

This PR deploys a minimal working BinderHub to the 2i2c cluster. It can be accessed at https://binder-staging.2i2c.cloud

Basing Binderhub's values from the original Pangeo Binder's values: https://github.com/pangeo-data/pangeo-binder/blob/staging/pangeo-binder/values.yaml
Config for the BinderHub has been added
- For now the BinderHub pushes images to my personal Docker Hub account
Deployer has been updated to handle a BinderHub deployment
All extra config entries have been disabled in the BinderHub helm chart for now, until we migrate over the templates from basehub

Compared to basehub or daskhub

This is Sarah's Docker Hub account for now

for more information, see https://pre-commit.ci

sgibson91 · 2022-06-08T15:22:02Z

This config is currently failing with the following messages:

From Helm CLI:

I0608 16:18:36.603860    2220 request.go:665] Waited for 1.098502108s due to client-side throttling, not priority and fairness, request: GET:https://104.198.26.247/apis/cert-manager.io/v1alpha2?timeout=32s

Events from hub pod which is in CrashLoopBackOff

Events:
  Type     Reason     Age                   From                                   Message
  ----     ------     ----                  ----                                   -------
  Normal   Scheduled  2m23s                 gke.io/optimize-utilization-scheduler  Successfully assigned binder-staging/hub-66958dc964-mtd6s to gke-pilot-hubs-cluster-core-pool-d184c825-vhck
  Warning  Unhealthy  107s (x3 over 2m13s)  kubelet                                Readiness probe failed: Get "http://10.0.1.33:8081/hub/health": dial tcp 10.0.1.33:8081: connect: connection refused
  Normal   Created    75s (x4 over 2m13s)   kubelet                                Created container hub
  Normal   Started    74s (x4 over 2m13s)   kubelet                                Started container hub
  Warning  BackOff    48s (x9 over 2m8s)    kubelet                                Back-off restarting failed container
  Normal   Pulled     33s (x5 over 2m14s)   kubelet                                Container image "jupyterhub/k8s-hub:1.1.2" already present on machine

sgibson91 · 2022-06-08T15:24:04Z

The above looks like the following PR but we have not enabled the prePuller or userPlaceholders

Enable 0 user placeholders when using continuous prePuller #1390

sgibson91 · 2022-06-08T16:04:49Z

From the hub logs

k logs -c hub hub-66958dc964-mtd6s --previous
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: 0-binderspawnermixin
Loading extra config: 00-binder
Loading extra config: 01-custom-theme
Loading extra config: 02-custom-admin
[E 2022-06-08 16:01:34.455 JupyterHub app:2969]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2966, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2457, in initialize
        self.load_config_file(self.config_file)
      File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 87, in inner
        return method(app, *args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 775, in load_config_file
        for (config, filename) in self._load_config_files(filename, path=path, log=self.log,
      File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 737, in _load_config_files
        config = loader.load_config()
      File "/usr/local/lib/python3.8/dist-packages/traitlets/config/loader.py", line 616, in load_config
        self._read_file_as_dict()
      File "/usr/local/lib/python3.8/dist-packages/traitlets/config/loader.py", line 648, in _read_file_as_dict
        exec(compile(f.read(), conf_filename, 'exec'), namespace, namespace)
      File "/usr/local/etc/jupyterhub/jupyterhub_config.py", line 446, in <module>
        exec(config_py)
      File "<string>", line 3, in <module>
    ModuleNotFoundError: No module named 'jupyterhub_configurator'

for more information, see https://pre-commit.ci

sgibson91 · 2022-06-08T16:20:59Z

Commenting out all the custom scripts made the hub pod get out of crashloopbackoff

sgibson91 · 2022-06-08T16:34:47Z

Binder exists at https://binder-staging.2i2c.cloud/ Image build and push succeeded, but spawn failed with:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"jupyter-binder-2dexamples-2drequirements-2dat9cvbbg\" is forbidden: error looking up service account binder-staging/user-sa: serviceaccount \"user-sa\" not found","reason":"Forbidden","details":{"name":"jupyter-binder-2dexamples-2drequirements-2dat9cvbbg","kind":"pods"},"code":403}


Spawn failed: (403)
Reason: error
HTTP response headers: HTTPHeaderDict({'Audit-Id': '9b2b8bc3-ab2b-4fed-8d75-0fcacf2b86a4', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '6076e3d8-fb9c-4af6-bc16-1828eab8bf98', 'X-Kubernetes-Pf-Prioritylevel-Uid': '3b5efc35-8cfb-4eeb-8300-d32a394811b8', 'Date': 'Wed, 08 Jun 2022 16:33:06 GMT', 'Content-Length': '369'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"jupyter-binder-2dexamples-2drequirements-2dat9cvbbg\" is forbidden: error looking up service account binder-staging/user-sa: serviceaccount \"user-sa\" not found","reason":"Forbidden","details":{"name":"jupyter-binder-2dexamples-2drequirements-2dat9cvbbg","kind":"pods"},"code":403}

sgibson91 · 2022-06-08T16:58:18Z

Removed singleuser.serviceAccountName to circumvent the above issue

This reverts commit 69f712d.

https://github.com/pangeo-data/pangeo-binder/blob/staging/pangeo-binder/values.yaml

sgibson91 · 2022-06-15T13:38:39Z

Current problems

Dask workers are not scaling up
- I have been testing using this repo: https://github.com/TomAugspurger/pangeo-dask-gateway
- Launch using this link: https://binder-staging.2i2c.cloud/v2/gh/pangeo-gallery/default-binder/master?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FTomAugspurger%252Fpangeo-dask-gateway%26urlpath%3Dtree%252Fpangeo-dask-gateway%252F%26branch%3Dmaster
- However, a dask cluster is never successfully created. I can't remember if we need a service account or not for that to work - can anyone remind me?

~~Validation failure~~ Solved by 2d46bc2 (#1404)

I am not sure how to resolve this:

Error: execution error at (binderhub/charts/dask-gateway/templates/gateway/secret.yaml:10:27): gateway.auth.jupyterhub.apiToken must be defined when using jupyterhub auth

GeorgianaElena · 2022-06-15T14:14:07Z

However, a dask cluster is never successfully created. I can't remember if we need a service account or not for that to work - can anyone remind me?

hmm, I just tried and it seems like it's starting a cluster for me?

GeorgianaElena · 2022-06-15T14:20:30Z

Error: execution error at (binderhub/charts/dask-gateway/templates/gateway/secret.yaml:10:27): gateway.auth.jupyterhub.apiToken must be defined when using jupyterhub auth

I believe that until dask/dask-gateway#473 is solved, a token must be manually generated and explicitly set for the jupyterhub gateway service. The code here probably makes more sense than what I'm saying.

infrastructure/deployer/hub.py

Lines 169 to 186 in 7b6d316

    
           # FIXME: This section can be removed upon resolution of the below linked issue, where we would 
        
           #        instead just define a JupyterHub service under hub.services and 
        
           #        rely on the JupyterHub Helm chart to generate an api token if 
        
           #        needed. 
        
           # 
        
           #        Blocked by https://github.com/dask/dask-gateway/issues/473 and a 
        
           #        release including it. 
        
           # 
        
           if hub_helm_chart == "daskhub": 
        
               gateway_token = hmac.new( 
        
                   secret_key, b"gateway-" + self.spec["name"].encode(), hashlib.sha256 
        
               ).hexdigest() 
        
               generated_config["dask-gateway"] = { 
        
                   "gateway": {"auth": {"jupyterhub": {"apiToken": gateway_token}}} 
        
               } 
        
               generated_config["basehub"].setdefault("jupyterhub", {}).setdefault( 
        
                   "hub", {} 
        
               ).setdefault("services", {})["dask-gateway"] = {"apiToken": gateway_token}

infrastructure/deployer/hub.py

Lines 177 to 186 in 7b6d316

    
           if hub_helm_chart == "daskhub": 
        
               gateway_token = hmac.new( 
        
                   secret_key, b"gateway-" + self.spec["name"].encode(), hashlib.sha256 
        
               ).hexdigest() 
        
               generated_config["dask-gateway"] = { 
        
                   "gateway": {"auth": {"jupyterhub": {"apiToken": gateway_token}}} 
        
               } 
        
               generated_config["basehub"].setdefault("jupyterhub", {}).setdefault( 
        
                   "hub", {} 
        
               ).setdefault("services", {})["dask-gateway"] = {"apiToken": gateway_token}

sgibson91 · 2022-06-15T14:30:50Z

hmm, I just tried and it seems like it's starting a cluster for me?

Ah that's because I pasted the wrong link! The original (now updated) link points to dask-staging instead of binder-staging. Try this one?

https://binder-staging.2i2c.cloud/v2/gh/pangeo-gallery/default-binder/master?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FTomAugspurger%252Fpangeo-dask-gateway%26urlpath%3Dtree%252Fpangeo-dask-gateway%252F%26branch%3Dmaster

sgibson91 · 2022-06-15T14:33:40Z

I believe that until dask/dask-gateway#473 is solved, a token must be manually generated and explicitly set for the jupyterhub gateway service.

So do we actually manually set this for other dask hubs in their values files? I feel like we don't, we let the deployer generate one. Therefore, if we wait for the deployer to generate one, how do those hubs pass the validation check - because the validation step runs before the deployer generates and inserts the token?

Am I misunderstanding something?

I added code to generate/inject the API token for binderhub here:

https://github.com/2i2c-org/infrastructure/pull/1404/files#diff-eedaf02b81cd907a3feb5e4389e9825226bf7dc82a0fb582f9ad367c00ba6651R217

And I can deploy the hub fine so long as I skip the validation check. See 69f712d

GeorgianaElena · 2022-06-15T15:10:59Z

@sgibson91, validation works for other daskhubs because of this (had no idea about it and never noticed it)...

infrastructure/deployer/config_validation.py

Line 113 in 7b6d316

cmd.append("--set=dask-gateway.gateway.auth.jupyterhub.apiToken=dummy")

sgibson91 · 2022-06-15T15:14:09Z

Ahhhh, so I need a conditional there for BinderHub too - thank you @GeorgianaElena! <3

yuvipanda

This looks great to me!

helm-charts/binderhub/values.yaml

deployer/config_validation.py

Co-authored-by: Yuvi Panda <[email protected]>

for more information, see https://pre-commit.ci

helm-charts/binderhub/values.yaml

upstream has a much more recent tag for the dind image

Used to provide access to cloud resources

Avoids extra dind deployments on core nodes and dask-gateway nodes

Add userServiceAccount to binderhub

sgibson91 · 2022-06-22T09:12:07Z

Got a warning in the logs about calico

2022-06-22T09:11:06Z [Warning] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1b7ca330c53fd745eeae4e2c6d6145e4ec90293d784b615366dfa70466ece06b": stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

sgibson91 · 2022-06-22T09:19:10Z

Other than the above comment about calico, this is deployed and ready for review! 🎉

yuvipanda

\o/ Let's go! THANKS @sgibson91!

yuvipanda · 2022-06-22T16:02:57Z

The calico warning is harmless for the most part, means the pod got scheduled on the node before networkpolicy had a chance to start up.

github-actions · 2022-06-22T16:35:21Z

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/workflows/deploy-hubs.yaml?query=branch%3Amaster

sgibson91 and others added 6 commits June 8, 2022 14:58

Add binderhub helm chart to deployer's validation step

2656cc7

Generate different config for binderhub

ec6f0ae

Compared to basehub or daskhub

Create basic binderhub config

a79a58e

Setup secret binderhub config with registry credentials

45dabd1

This is Sarah's Docker Hub account for now

Add binder-staging deployment to 2i2c's cluster.yaml file

ec6a0f8

[pre-commit.ci] auto fixes from pre-commit.com hooks

6eb401d

for more information, see https://pre-commit.ci

sgibson91 added 2 commits June 8, 2022 16:26

Set jupyterhub API token for dask-gateway in BinderHub chart

3a266f8

Temporarily skip validation when deploying

69f712d

sgibson91 and others added 2 commits June 8, 2022 17:20

Comment out all the custom stuff

694b21f

[pre-commit.ci] auto fixes from pre-commit.com hooks

00da27e

for more information, see https://pre-commit.ci

Remove singleuser.serviceAccountName

6543a9c

damianavila assigned sgibson91 Jun 13, 2022

sgibson91 mentioned this pull request Jun 14, 2022

Create a BinderHub helm chart in the infrastructure repo #1280

Closed

1 task

sgibson91 force-pushed the deploy-test-binder branch from b384c5c to 6543a9c Compare June 15, 2022 08:49

sgibson91 changed the title ~~Deploying a test BinderHub~~ Deploying a MVP BinderHub Jun 15, 2022

sgibson91 added 2 commits June 15, 2022 09:53

Revert "Temporarily skip validation when deploying"

54bfd7b

This reverts commit 69f712d.

Update values to resemble the original Pangeo Binder's

bbc5a55

https://github.com/pangeo-data/pangeo-binder/blob/staging/pangeo-binder/values.yaml

Merge branch 'master' into deploy-test-binder

4d9f26d

Set dummy dask api token for binderhub too

2d46bc2

yuvipanda reviewed Jun 16, 2022

View reviewed changes

helm-charts/binderhub/values.yaml Outdated Show resolved Hide resolved

deployer/config_validation.py Outdated Show resolved Hide resolved

sgibson91 and others added 2 commits June 16, 2022 10:39

Update deployer/config_validation.py

a31a2d5

Co-authored-by: Yuvi Panda <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

6c86fc6

for more information, see https://pre-commit.ci

sgibson91 commented Jun 16, 2022

View reviewed changes

helm-charts/binderhub/values.yaml Outdated Show resolved Hide resolved

sgibson91 and others added 7 commits June 16, 2022 10:40

remove dind config, inherit from upstream binderhub chart

0ffba14

upstream has a much more recent tag for the dind image

Don't pin traefik image

fe9cd96

Merge branch 'master' into deploy-test-binder

fb45859

Add userServiceAccount to binderhub

c0eefeb

Used to provide access to cloud resources

Build user images only on user nodes

461f08a

Avoids extra dind deployments on core nodes and dask-gateway nodes

Merge pull request #197 from yuvipanda/no-user

a87bd38

Merge pull request #196 from yuvipanda/deploy-test-binder

cdcdd0e

Add userServiceAccount to binderhub

sgibson91 marked this pull request as ready for review June 22, 2022 09:18

sgibson91 requested a review from yuvipanda June 22, 2022 09:18

sgibson91 mentioned this pull request Jun 22, 2022

Increase memory for meom-ige prometheus #1448

Merged

yuvipanda approved these changes Jun 22, 2022

View reviewed changes

sgibson91 merged commit 1f76a48 into 2i2c-org:master Jun 22, 2022

sgibson91 deleted the deploy-test-binder branch June 22, 2022 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploying a MVP BinderHub #1404

Deploying a MVP BinderHub #1404

sgibson91 commented Jun 8, 2022 •

edited

Loading

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022 •

edited

Loading

sgibson91 commented Jun 15, 2022 •

edited

Loading

GeorgianaElena commented Jun 15, 2022

GeorgianaElena commented Jun 15, 2022

sgibson91 commented Jun 15, 2022 •

edited

Loading

sgibson91 commented Jun 15, 2022 •

edited

Loading

GeorgianaElena commented Jun 15, 2022

sgibson91 commented Jun 15, 2022

yuvipanda left a comment

sgibson91 commented Jun 22, 2022

sgibson91 commented Jun 22, 2022 •

edited

Loading

yuvipanda left a comment

yuvipanda commented Jun 22, 2022

github-actions bot commented Jun 22, 2022

Deploying a MVP BinderHub #1404

Deploying a MVP BinderHub #1404

Conversation

sgibson91 commented Jun 8, 2022 • edited Loading

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022

sgibson91 commented Jun 8, 2022 • edited Loading

sgibson91 commented Jun 15, 2022 • edited Loading

Current problems

GeorgianaElena commented Jun 15, 2022

GeorgianaElena commented Jun 15, 2022

sgibson91 commented Jun 15, 2022 • edited Loading

sgibson91 commented Jun 15, 2022 • edited Loading

GeorgianaElena commented Jun 15, 2022

sgibson91 commented Jun 15, 2022

yuvipanda left a comment

Choose a reason for hiding this comment

sgibson91 commented Jun 22, 2022

sgibson91 commented Jun 22, 2022 • edited Loading

yuvipanda left a comment

Choose a reason for hiding this comment

yuvipanda commented Jun 22, 2022

github-actions bot commented Jun 22, 2022

sgibson91 commented Jun 8, 2022 •

edited

Loading

sgibson91 commented Jun 8, 2022 •

edited

Loading

sgibson91 commented Jun 15, 2022 •

edited

Loading

sgibson91 commented Jun 15, 2022 •

edited

Loading

sgibson91 commented Jun 15, 2022 •

edited

Loading

sgibson91 commented Jun 22, 2022 •

edited

Loading