YugabyteDB ulimit open files too low in containerized k3s cluster

[cherusk]

Environmental Info:
K3s Version:

~ # k3s -v
k3s version v1.31.3+k3s1 (6e6af98)
go version go1.22.8

Node(s) CPU architecture, OS, and Version:
Containerized, but underlying system is a nixos runner on osuosl infrastructure:

Linux nixos 5.10.229 #1-NixOS SMP Fri Nov 8 15:22:04 UTC 2024 x86_64 GNU/Linux

Cluster Configuration:

Essentially I use the template docker compose file for the cluster:

https://github.com/godon-dev/godon-charts/blob/main/k3s-docker-compose.yml

Describe the bug:

While deploying yugabyte db as part of the godon stack, I can perceive in the logs and the deployment fails:

[root@nixos:~]# kubectl --kubeconfig=/run/kubeconfig.yaml -n godon logs yb-tserver-0
Defaulted container "yb-tserver" out of: yb-tserver, yb-cleanup
Required ulimit values too low, see below. In kubernetesset the helm override 'preflight.skipUlimit: false'to override this check
open files | Expected Value : 1048576 | Current Value : 1024

That's happening as part of this PR: godon-dev/godon-charts#22

I've attempted to apply this previous patch suggestion #3934 and raised the no open files ulimit config of the underlying docker-compose containers.

Though the issue persists.

Expected behavior:
YugaByteDB pods are scheduled like the others. Pod containers do not get a ridiculous low ulimit

Actual behavior:

K3S engine seemingly sets ridiculously low open files ulimit for pod containers.

[dereknola]

We aren't going to change our existing default limits for a containerized version of K3s on a platform we don't support. You should be able to modify the docker compose file and/or the k3s.service file yourself to achieve what you want.

[cherusk]

For others that might come across this.

The solution was indeed to adjust the nofile ulimit on the host layer.

The setting in the os of the agent container takes no effect. Not sure if it is sensible to set those limits at container level at all then. Might causes more confusion than it provides value.