Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: (IAC-1379) Add Support for K8s 1.29 #117

Merged
merged 8 commits into from
Apr 4, 2024
Merged

feat: (IAC-1379) Add Support for K8s 1.29 #117

merged 8 commits into from
Apr 4, 2024

Conversation

jarpat
Copy link
Contributor

@jarpat jarpat commented Apr 4, 2024

Changes

  • Updated the default kubernetes_version/cluster_version in the example files and doc to 1.28.7
  • Updated the default kubectl version in the Dockerfile to 1.28.7 (currently the latest)

Notes:

There was an issue discovered with kube-vip and K8s 1.29+. In short kube-vip requires super-admin.conf permissions with Kubernetes 1.29 and without it, we run into issues setting up a new cluster with kubeadm init.

super-admin.conf was introduced Kubernetes 1.29, and the user within that file is bound to the system:masters RBAC group. In previous kubernetes versions the admin.conf user was bound to this RBAC group, but now in 1.29 this user is bound to a new group called kubeadm:cluster-admins that has cluster-admin ClusterRole access.

If you take a look at the 1.29 Urgent Upgrade Notes
from the kubernetes repo this change is described in more detail:

kubeadm: a separate "super-admin.conf" file is now deployed. The User in admin.conf is now bound to a new RBAC Group kubeadm:cluster-admins that has cluster-admin ClusterRole access. The User in super-admin.conf is now bound to the system:masters built-in super-powers / break-glass Group that can bypass RBAC. Before this change, the default admin.conf was bound to system:masters Group, which was undesired. Executing kubeadm init phase kubeconfig all or just kubeadm init will now generate the new super-admin.conf file. The cluster admin can then decide to keep the file present on a node host or move it to a safe location. kubadm certs renew will renew the certificate in super-admin.conf to one year if the file exists; if it does not exist a "MISSING" note will be printed. kubeadm upgrade apply for this release will migrate this particular node to the two file setup. Subsequent kubeadm releases will continue to optionally renew the certificate in super-admin.conf if the file exists on disk and if renew on upgrade is not disabled. kubeadm join --control-plane will now generate only an admin.conf file that has the less privileged User.

At this point in time, kube-vip (even the latest versions) requires super-admin.conf with Kubernetes 1.29 during the initial kubeadm init phase and will fail without it as described in this GitHub issue here: kube-vip/kube-vip#684. Our PR makes use of a workaround recommended in that GitHub issue where we're temporarily replacing the mounted kube conf file in the kube-vip.yaml manifest with super-admin.conf manifest before running kubeadm init and then immediately replacing it with admin.conf after the command is run.

We will have to keep using the workaround for 1.29+ until a version of kube-vip is released that resolves this issue. After the fix is in place we can remove the workaround and point users to select a version of kube-vip with that particular fix for K8s 1.29+ installs.

Tests

Scenario Provider K8s Version kubectl cluster_lb_type Order Cadence Notes
1 OSS 1.27.11 1.28.7 kube_vip fast:2020 OOTB
2 OSS 1.28.7 1.28.7 kube_vip fast:2020 OOTB
3 OSS 1.29.2 1.28.7 kube_vip fast:2020 OOTB - overwrite kubectl version in the DO
4 OSS 1.29.2 1.28.7 metallb fast:2020 OOTB - overwrite kubectl version in the DO

@jarpat jarpat added documentation Improvements or additions to documentation enhancement New feature or request labels Apr 4, 2024
@jarpat jarpat self-assigned this Apr 4, 2024
@jarpat jarpat merged commit 2a864e9 into staging Apr 4, 2024
4 checks passed
@jarpat jarpat deleted the IAC-1379 branch April 4, 2024 17:34
@jarpat jarpat mentioned this pull request Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants