How to get etcd metrics?

[CRASH-Tech]

It's not a pod, and I can't scape metrics from the cluster.
Where is metrics endpoint?

[frezbo]

You'd set the extraArgs for etcd here: https://www.talos.dev/v1.4/reference/configuration/#etcdconfig

and add a listen-metric-urls

[CRASH-Tech]

Set to 0.0.0.0 and scrape from cp node ip's?

[frezbo]

yes, you can also set it to the node ip

[CRASH-Tech]

Thanks!

[gerhard]

So what is the recommended way of configuring etcd metrics scraping again?

I was not able to get the above recommendation to work. I am running:

• Talos v1.5.3 managed by Omni v0.18.1
• K8s v1.28.2
• kube-prometheus-stack v48.3.6

After applying the following cluster patch:

cluster:
  etcd:
    extraArgs:
      listen-metrics-url: 0.0.0.0

etcd continues to listen on IPv6 only:

# netstat -ltnp | grep etcd
tcp6       0      0 :::2380                 :::*                    LISTEN      2061/etcd
tcp6       0      0 :::2379                 :::*                    LISTEN      2061/etcd

---

FWIW:

# curl -6 "http://[::1]:2380/metrics" -v
*   Trying [::1]:2380...
* Connected to ::1 (::1) port 2380 (#0)
> GET /metrics HTTP/1.1
> Host: [::1]:2380
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

# curl -6 "http://[::1]:2379/metrics" -v
*   Trying [::1]:2379...
* Connected to ::1 (::1) port 2379 (#0)
> GET /metrics HTTP/1.1
> Host: [::1]:2379
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

[gerhard]

I was able to reboot it from Omni after the manual reboot.

[gerhard]

All working, thanks everyone!

kube-prometheus-stack.yaml

# ↑ 1557 rows of config before

kubeEtcd:
  enabled: true

  ## If your etcd is not deployed as a pod, specify IPs it can be found on
  ##
  endpoints:
  - 192.168.88.12

  ## Etcd service. If using kubeEtcd.endpoints only the port and targetPort are used
  ##
  service:
    enabled: true
    port: 2381
    targetPort: 2381

# ↓ 2482 lines of config after

patches/etcd-metrics.yaml (for Omni)

# ⚠️ Requires reboot after changing!
cluster:
  etcd:
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381

[R-Studio]

Is there no way to automate this (prevent hardcoded IP's in kube-prometheus-stack.yaml)?

[abueschel]

@R-Studio I found a (admittedly bit ugly) workaround to this. Under the assumption that every controlplane node runs the same controlplane services (etcd, kube-controller-manager etc.) you can modify the selector of the headless servicekube-prometheus-stack-kube-etcd that kube-prometheus-stack uses for the serviceMonitor resource:

kube-prometheus-stack.yaml:

kubeEtcd:
  service:
    selector:
      k8s-app: kube-controller-manager # or any other service like kube-proxy or kube-scheduler

This populates the endpoints of the service with the controlplane server IPs, because kube-controller-manager pods are running in host network mode. You might want to do some additional relabeling, as the serviceMonitor will add some wrong labels for e.g. pod (etcd isn't running in a pod...)

Alternatively you could also add a simple deployment manifest to your Talos machine configuration for the controlplane. Just run a "pause" container in host network mode and you could pick a more fitting name.

[R-Studio]

@abueschel nice this workaround works perfectly (all of our controlplane nodes runs the same controlplane services).
My kube-prometheus-stack Helm values for etcd looks like this:

kubeEtcd:
  enabled: true
  service:
    selector:
      k8s-app: kube-controller-manager # Fix selector for kube-etcd for Talos (set itentionally to kube-controller-manager because all master nodes has the same roles)
  serviceMonitor:
    relabelings:   # Add nodename label
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        separator: ;
        regex: ^(.*)$
        targetLabel: nodename
        replacement: $1
        action: replace
    metricRelabelings:   # Remove pod label
      - action: labeldrop
        regex: pod

[Charlotte-br560]

You can retrieve etcd metrics directly from the etcd server using its HTTP API. The metrics endpoint typically resides at http://<etcd-server>:<port>/metrics. Ensure you have access to the etcd server and its port to fetch metrics.

[maxpain]

But this does require a client certificate.

[Hr46ph]

For those coming here using kube-prometheus-stack and looking for a way to make scraping etcd, controller manager and scheduler work, here is how I did it.

Add mc patch on all control planes:

- op: add
  path: /cluster/etcd/extraArgs
  value:
    listen-metrics-urls: https://0.0.0.0:2379
- op: add
  path: /cluster/controllerManager/extraArgs
  value:
    bind-address: 0.0.0.0
- op: add
  path: /cluster/scheduler/extraArgs
  value:
    bind-address: 0.0.0.0

Patch:
talosctl patch mc -n 10.0.0.1 --patch @etcd_metrics_patch.yaml

And repeat for all control planes.

Get the certificates from etcd by running:

talosctl get etcdrootsecret -o yaml
talosctl get etcdsecret  -o yaml

The output should look like:

spec:
    etcdCA:
        crt

spec:
    etcd:
        crt:
        key:

The strings are already base64 encoded so straight copy/paste into a new secret.

Create a new secret:

---
apiVersion: v1
kind: Secret
metadata:
  name: etcd-client-cert
  namespace: monitoring
type: Opaque
data:
  etcd-ca.crt:
    LS0t....LS0K
  etcd-client.crt:
    LS0t....LS0K
  etcd-client-key.key:
    LS0t....=

kubectl apply -f etcd-secret.yaml.

In your custom-values.yaml for the helm deployment, add / change the following and replace the IP's with your control planes:

kubeControllerManager:
  endpoints:
    - 10.0.0.1
    - 10.0.0.2
    - 10.0.0.3

kubeEtcd:
  endpoints:
    - 10.0.0.1
    - 10.0.0.2
    - 10.0.0.3
  service:
    selector:
      component: etcd
  serviceMonitor:
    scheme: https
    insecureSkipVerify: false
    serverName: "localhost"
    caFile: "/etc/prometheus/secrets/etcd-client-cert/etcd-ca.crt"
    certFile: "/etc/prometheus/secrets/etcd-client-cert/etcd-client.crt"
    keyFile: "/etc/prometheus/secrets/etcd-client-cert/etcd-client-key.key"

kubeScheduler:
  endpoints:
    - 10.0.0.1
    - 10.0.0.2
    - 10.0.0.3

prometheus:
  prometheusSpec:
    secrets:
      - etcd-client-cert

Note stating the obvious: the values above are not enough for a complete and succesful deployment of kube-prometheus-stack. These are only the additional changes that you need to make this particular scraping work.

[pl4nty]

thanks for that trick with etcdsecret, helped me fix my very broken cluster

[rothgar]

Would you mind adding this information to the talos docs? It would be helpful for people in the future

[Hr46ph]

I made a pull request #10094 you should be able to see it.

[Hr46ph]

@rothgar are you able to see it? I don't make pr's very often so I might have done something wrong. let me know if you need me to change anything. Thanks.