node not finish boot sequence [AWS]

[paucampana]

Hello,

I am trying to create a simple single node control plane in AWS following your documentation.

The problem that I have is that the boot sequence never finish:

talosctl health
discovered nodes: ["3.122.228.84"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: nodes ["3.122.228.84"] are still in boot sequence

If I check the services, everything looks okay:

talosctl service
NODE           SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT
3.122.228.84   apid         Running   OK       7m22s ago     Health check successful
3.122.228.84   containerd   Running   OK       7m28s ago     Health check successful
3.122.228.84   cri          Running   OK       7m27s ago     Health check successful
3.122.228.84   etcd         Running   OK       7m22s ago     Health check successful
3.122.228.84   kubelet      Running   OK       7m20s ago     Health check successful
3.122.228.84   machined     Running   ?        7m31s ago     Service started as goroutine
3.122.228.84   trustd       Running   OK       7m27s ago     Health check successful
3.122.228.84   udevd        Running   OK       7m28s ago     Health check successful

But checking kubelet logs I see the error: 3.122.228.84: {"ts":1662470339261.1562,"caller":"kubelet/kubelet.go:2466","msg":"Error getting node","err":"node \"ip-10-0-0-181\" not found"}

The public IP of the ec2 is 3.122.228.84 and the private one is 10.0.0.181.

I do not know what is happening. I am quite new using Talos, so I imagine I am doing something wrong..

ENVIRONMENT:

I have the following network configuration:
VPC CIDR: 10.0.0.0/16
SUBNET CIDR: 10.0.0.0/24

route table:

10.0.0.0/16 | local
0.0.0.0/0 |  INTERNET-GATEWAY-ID

talos version:

 talosctl version
Client:
        Tag:         v1.2.0
        SHA:         c568770c
        Built:
        Go version:  go1.19
        OS/Arch:     linux/amd64
Server:
        NODE:        3.122.228.84
        Tag:         v1.2.0
        SHA:         c568770c
        Built:
        Go version:  go1.19
        OS/Arch:     linux/amd64
        Enabled:     RBAC

talosctl get nodeaddresses:

NODE           NAMESPACE   TYPE          ID                      VERSION   ADDRESSES
3.122.228.84   network     NodeAddress   accumulative            3         ["3.122.228.84/32","10.0.0.181/24"]
3.122.228.84   network     NodeAddress   accumulative-no-k8s     1         ["3.122.228.84/32","10.0.0.181/24"]
3.122.228.84   network     NodeAddress   accumulative-only-k8s   1         []
3.122.228.84   network     NodeAddress   current                 3         ["3.122.228.84/32","10.0.0.181/24"]
3.122.228.84   network     NodeAddress   current-no-k8s          1         ["3.122.228.84/32","10.0.0.181/24"]
3.122.228.84   network     NodeAddress   current-only-k8s        1         []
3.122.228.84   network     NodeAddress   default                 1         ["10.0.0.181/24"]
3.122.228.84   network     NodeAddress   routed                  2         ["10.0.0.181/24"]
3.122.228.84   network     NodeAddress   routed-no-k8s           1         ["10.0.0.181/24"]
3.122.228.84   network     NodeAddress   routed-only-k8s         1         []

[paucampana]

Some more information:

• Platform: AWS.
  Using:
• instant type: m6a.xlarge
• AMI ID: ami-0a567162d9680e404

[paucampana]

[smira]

It's hard to say what might be wrong, it might be helpful to attach the support.zip package produced by talosctl support.

btw, it might not be the best idea to share Talos machineconfig, as it contains PKI information to access the cluster.

[paucampana]

They are test cluster. I am deploying and deleting clusters all the time. But thanks for the advise. I have removed the machineconfig.

[paucampana]

support.zip

[smira]

Looks like probably your AWS loadbalancer is misconfigured or something to that effect, Talos can't talk to it:

{"ts":1662472868072.9333,"caller":"lease/controller.go:144","msg":"failed to ensure lease exists, will retry in 7s, error: Get \"https://talos-aws-tutorial-lb-c420de797e0c3605.elb.eu-central-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-0-181?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n"}

[paucampana]

Thanks for your time. It was an issue with the security group..nothing related with Talos. 🙇‍♂️