Attach Labels to K8s Pod Metrics & Logs Based on Pod Annotations

[gabriel-yahav]

I’m using the Grafana k8s-monitoring Helm chart to collect Kubernetes pod metrics and logs, and I’d like to attach
labels to these data points based on the value of a specific annotation attached to each pod.

For example, if a pod has an annotation like this:

metadata:
  annotations:
    mycustom/label: "team-A"

I want the collected metrics and logs to have a label like: mycustom_label="team-A"

This would help in filtering, grouping, and querying logs/metrics in Grafana more effectively.
Is there an existing way in the k8s-monitoring Helm chart to achieve this? If not, would you consider adding support for this feature?

Thanks for your help ! 🚀

[petewall]

Hi Gabriel!

Settings labels on pod logs from the pod annotation can be done with:

podLogs:
  annotations:
    mycustom_label: "mycustom/label"

[petewall]

Setting that label on metrics is a little trickier. Give me a minute and I'll type up some ideas there.

[petewall]

As for metrics, it all depends on where the metrics are coming from.

If you're scraping metrics from the pod itself, you can use a relabeling rule to find the annotation and add it to the pod:

      rule {
        source_labels = ["__meta_kubernetes_pod_annotation_ mycustom_label"]
        target_label = "mycustom_label"
      }

However, if you want metrics about the pod, but from some other source, like kube_pod_info metrics from kube-state-metrics, you cannot do the above relabeling rule, because it will give you the annotations for the kube-state-metrics pod, not the pod you care about. Same applies to metrics from the kubelet, cadvisor, node-exporter, etc...

To do that, what you can do is configure kube-state-metrics to capture pod annotations and report them as a standalone metric: kube_pod_annotations. Then you can use that to join against other metrics in your visualization layer.

To get those, you'll need to do two changes: Tell ksm to capture those metrics, and tell alloy to include them:

clusterMetrics:
  kube-state-metrics:
    metricsTuning:
      includeMetrics: [kube_pod_annotations]
    metricAnnotationsAllowlist:
    - pods=[*]

[gabriel-yahav]

@petewall
Thanks for the detailed answer - It’s exactly what I was looking for.
Appreciate your help! 🙌

[petewall]

Of course! Thanks for the question!