cannot get bare metal nodes to provision using flavor hetzner-hcloud-control-planes

[Privatecoder]

Hi @batistein,

I have tried a lot of things but unfortunately to no avail.

What works: Create a bootstrap cluster using kind, generating a new cluster config with the hcloud-network flavor (for example), installing ccm (syself/ccm-hcloud), provision this cluster to the cloud provider and deleting the bootstrap cluster afterwards. All up and running, all worker nodes ready.

What also works is using this cluster as management-cluster to create another workload-cluster on Hetzner's HCloud without the need for a local kind cluster.

However trying to set-up a cluster with the hetzner-hcloud-control-planes flavor (using the bm-ccm syself/ccm-hetzner) the bm-nodes never become available. I did create HetznerBareMetalHost objects (with and without raid-settings, with and without wwn-settings) for my three existing bm-nodes and of course created a ssh-key in the Robot->Server section along with a "Webservice- and App-Settings" user-account account (without IP-restrictions) following this, this and this guides.

• I added the three HetznerBareMetalHosts to the bottom of my-cluster.yaml and according to the log-output of kubectl apply, they get created.
• I can query them by running kubectl describe hetznerbaremetalhost however they are always stuck in in Provisioning State: preparing
• I cannot see any updates, errors or other hints that led me to a possible solution yet.
• It seems as if the ccm does not even try to start provisioning the bm-hosts.
• The attached screenshot shows the output of clusterctl describe cluster with a mixed-setup; md-0, i.e. HCloud worker-nodes are up and running, bm-nodes are not.
• However after a while (seemingly forever) all bm-hosts get a new name in the Robot, i.e. test-md-1-qsdfbr etc. so something does indeed happen but I cannot see any logs referring to this in the ccm as well..

Questions:

1. With a mixed cluster using the hetzner-hcloud-control-planes using kind as local bootstrap cluster, after adding HetznerBareMetalHosts objects, will they be moved to the target cluster by clusterctl move as well?
2. What logs do I need to check / how can I debug this as I could not see anything in the ccm-pod-logs.
3. Do I need to set special names for the bm-hosts inside the Robot, i.e. prefixing them with bm- (I've done that)?
4. General question: Do I need to have one management- (kind/bootstrap-cluster->management-cluster@Hetzner) and one workload-cluster (management-cluster@Hetzner->workload-cluster@hetzner)?

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HetznerBareMetalHost
metadata:
  name: "test1" #change
spec:
  serverID: 1234567 #change
  rootDeviceHints:
    raid:
      wwn:
        - "eui.8324fa988998a89" #change
        - "eui.8324fa988998a90" #change
  maintenanceMode: false
  description: test1 #change

[batistein]

1. Only if they are labeled with clusterctl.cluster.x-k8s.io/move":""
2. You could check the logs of the caph-controller, capi-controller, bootstrap-controller. Also you could check the full status of the Host Object by doing a kubectl get HetznerBareMetalHost <your-machine> -o yaml. This should give you a good starting point.
3. The prefix in robot comes from the caph controller if a machine is successfully ready. Usually you don't give any special name to the nodes.
4. No you can selfhost the cluster also. workload=bootstrap. From technical perspective there is no such difference. The controller needs only a kubernetes cluster to run.

[Privatecoder]

Thank you, will give that a try. Two more questions:

1. Do I need / have to move the HetznerBareMetalHosts objects, i.e. patch them?
2. Does the metadata->name of the HetznerBareMetalHosts objects have to match the name in the Robot or are there other requirements?

[batistein]

1. They need to be in the same cluster and namespace as the machinedeployment, cluster and all the other resources for the one workload cluster.
2. no that's completly independent. The name of the hosts are only to name the k8s object and acts as an inventory

[Privatecoder]

@batistein well... I will update the docs and submit a PR.

From the caph-controller-manager-log:

"error":"failed to reconcile HetznerBareMetalHost default/bm-02: action \"preparing\" failed: failed to set ssh key: The key already exists (KEY_ALREADY_EXISTS)","errorVerbose":"The key already exists (KEY_ALREADY_EXISTS)\nfailed to set ssh key\ngithub.com/syself/cluster-api-provider-hetzner/pkg/services/baremetal/host.(*Service).ensureSSHKey\n\t/workspace/pkg/services/baremetal/host/host.go:370\ngithub.com/syself/cluster-api-provider-hetzner/pkg/services/baremetal/host.(*Service).actionPreparing\n\t/workspace/pkg/services/baremetal/host/host.go:231\ngithub.com/syself/cluster-api-provider-hetzner/pkg/services/baremetal/hos

For anyone running into this issue before the docs have been updated: You must not have your ssh-key set in the Robot as it will be set by the caph-controller-manager (with the name cluster).

[batistein]

Happy to hear that you worked it out! Looking forward to your PR ;)

[Privatecoder]

@batistein will do ;)

But:
=> What if I am going to add another cluster with different bm-nodes and a different ssh-key, what name will be chosen?
..as the controller adds the robot-ssh key under the unspecific/non-unique name cluster would this become an issue with the scenario above?

[janiskemper]

You chose the name "cluster" in your secret. You can easily change that name, which will then be reflected in the name of the ssh secret in the Robot UI

[batistein]

You change the value of sshkey-name to something different kubectl create secret generic robot-ssh --from-literal=sshkey-name=cluster --from-file=ssh-privatekey=$HETZNER_SSH_PRIV_PATH --from-file=ssh-publickey=$HETZNER_SSH_PUB_PATH

this command is from https://github.com/syself/cluster-api-provider-hetzner/blob/main/docs/topics/preparation.md

[Privatecoder]

you are right, I did not see that..