sig-node: Kubelet-in-UserNS, aka Rootless mode

[AkihiroSuda]

Allow running the entire Kubernetes components (kubelet, CRI, OCI, CNI, and all kube-*) as a non-root user on the host, by using a user namespace and cgroup v2 (#1370).

Rootless mode has been already adopted by k3s.

Also, kind already supports Rootless Docker/Podman, with unmodified Kubernetes, but it uses very dirty hack to avoid sysctl errors, so this KEP still has to be accepted.

Replaces #1084

---

POC: https://github.com/rootless-containers/usernetes
Kubernetes PR: kubernetes/kubernetes#92863
Tracking issue: #2033

[k8s-ci-robot]

Hi @AkihiroSuda. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[AkihiroSuda]

cc @giuseppe @ibuildthecloud @derekwaynecarr @dchen1107 @yujuhong

[BenTheElder]

/ok-to-test

[dims]

/assign @dchen1107 @derekwaynecarr

[AkihiroSuda]

ping

[dims]

@AkihiroSuda i've added this to the weekly sig-node meeting. let's see if we can get some eyes there

[AkihiroSuda]

Thanks, but I'm not likely to be able to attend the meeting due to the timezone, sorry.

[dims]

@AkihiroSuda no worries, i will raise it on your behalf :)

[giuseppe]

isn't it blocked on #1370?

[AkihiroSuda]

isn't it blocked on #1370?

Yes, w.r.t. the cgroup part, but other parts can be discussed / merged ?
(Anyway, #1370 seems ready to be merged?)

[derekwaynecarr]

I think we should tackle cgroups v2 first which has a related KEP.

I would prefer we track rootless behaviors in the cgroups v2 KEP which still needed further iteration.

[AkihiroSuda]

Updated PR to address comments, thanks all for reviewing!

[utkarsh2102]

Hi, I was just going through this and found some super-trivial things that you might be interested in? 😅

[thockin]

This seems like an incredibly expensive way (from a network POV) to get rid of root-components and the resulting semantics seem dodgy (those rlimits are configured for a reason).

That said, the request, as far as I can see, is pretty non-invasive, so who am I to tell you what (not) to do?

[jayunit100]

... windows has these 'kernel silos' or something like that, need @jsturtevant to weigh in here, on what this would mean for hostProcess containers....

[thockin]

I guess that nodeports are generally high numbers, so non-root is ok. As I write that, I assumed iptables (which would suck less than userspace).

…

On Mon, May 17, 2021, 6:21 PM Akihiro Suda ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In keps/sig-node/2033-rootless/README.md <#1371 (comment)> : > +$ nsenter -t $ROOTLESSKIT_CHILD_PID -a kube-proxy ... & +$ nsenter -t $ROOTLESSKIT_CHILD_PID -a flanneld ... & +``` + +#### Paths + +Some paths like `/var/log/pods` are hardcoded in Kubernetes and hard to change. + +Although these directories are not writable by unprivileged users, Kubernetes does NOT need to be changed to use unprivileged home directories, +because RootlessKit can bind-mount writable directories on these paths without the root privileges. (`rootlesskit --copy-up=/var`) + +#### Network +The node components need to be executed in RootlessKit's network namespace, because an unprivileged user cannot do privileged operations in the host network namespace. +As the components are executed inside a network namespace, `NodePorts` are not directly accessible from other hosts. + +An external controller should watch changes on `corev1.Service` resources and call [RootlessKit API](https://github.com/rootless-containers/rootlesskit/blob/v0.11.1/pkg/api/openapi.yaml) to set up port forwarding for the node ports. No, the port forwarder is rootless too. This port forwarder is needed because kube-proxy cannot listen on ports outside the namespace. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1371 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVB2GZYUPGMPTPNYXYDTOG6JVANCNFSM4JOY5PKQ> .

[AkihiroSuda]

This seems like an incredibly expensive way (from a network POV) to get rid of root-components and the resulting semantics seem dodgy (those rlimits are configured for a reason).

There is an experiment to remove slirp overhead by using SECCOMP_IOCTL_NOTIF_ADDFD (Kernel 5.9) https://github.com/rootless-containers/bypass4netns .
It is also possible to use LXC's SETUID binary to use the native vEth.

[derekwaynecarr]

thank you @thockin @gnufied @jsafrane for sharing your insights as well.

it looks like there is no disagreement on exploring further iteration of the concept with the caveats enumerated for kubelet and kube-proxy, but definitely we have a lot to iron out as this evolves.

/approve
/lgtm

[dims]

@ehashman please approve!

[ehashman]

Feature gate has been added, addressing the last of my PRR comments.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AkihiroSuda, derekwaynecarr, ehashman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [ehashman]
• keps/sig-node/OWNERS [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[AkihiroSuda]

@ehashman Thanks for approval, can we get this merged?

SIG-release approved this in v1.22 milestone, with the merge deadline May 25.
https://groups.google.com/g/kubernetes-sig-release/c/eggXiBwlzw8/m/ODnL8DYbAQAJ

[dims]

/hold cancel

[AkihiroSuda]

Thanks @dims!