Deploy Loadbalancer using combo of NGINX IC (not proxy) & metallb; BOTH as Cluster resources, not externally

[Daemon-Fadi]

HI everyone,

Goal

Deploy Talos-based baremetal (ISO) HA production cluster (3 control-plane + X worker nodes) using NGINX Ingress controller (IC) & MetalLB as Cluster managed Resources to deliver the LoadBalancer service with external IP assignment (via metallb). Why this config? For security: Want SSL termination to happen inside the cluster, not outside. A proxy outside means traffic between proxy and cluster is unencrypted.

Problem: Configs I see in production guides require pointing to a loadbalancer BEFORE setting up the HA cluster. And to do that, a reverse proxy is needed first. But I don't want the proxy to be sitting outside the cluster. I want it to be a cluster managed resource (this is a common setup in k8s, and both metallb (CRD) and NGINX IC end up being deployed via manifest. Note: Setting up vIPs doesn't seem to work as an interim solution as I get an error when I deploy metallb later to be the loadbalancer (I cannot release the vIPs). In fact, I cannot even get the cluster to setup this way (vIP) at all.

Question
Has someone been successful at doing this? I would appreciate any help in this regard?

Note: In the above target GOAL, NGINX IC actually ends up doing the service load balancing while metallb provides external IP visibility (to satisfy LoadBalancer service requirements); but perhaps this is not the way it is done in Talos.

Your help is appreciated.

Thanks!

[smira]

I guess you want to run Kubernetes API loadbalancer for the cluster inside the cluster?

This is not recommended, as it has some obvious downsides: if Kubernetes goes down, you might lose access to the loadbalancer (and Kubernetes API).

If you still want to do it, enable KubePrism to make sure that cluster itself doesn't depend on the loadbalancer being up. This way loadbalancer will be used mostly for external access, so if it goes down, the cluster itself is operational, but external access is limited.

Want SSL termination to happen inside the cluster, not outside. A proxy outside means traffic between proxy and cluster is unencrypted.

Kubernetes API requires TCP loadbalancer, so TLS termination always happens at the kube-apiserver, so traffic flows encrypted.

[Daemon-Fadi]

Thanks Andrey @smira
I will move this topic to Slack and close it here. Learned Slack existed after I opened this topic.

[smira]

This place is better than Slack actually, as it gets properly indexed and stays forever.