Is there a way to have performance metrics when running kubeai?

[strus38]

Hello
I am looking at a way to have performance metrics show-up while using kubeai.
Is this provided already or is there a way to do it, even if it means deploying more containers alongside?
Thanks

[nstogner]

KubeAI exposes some very basic metrics. Some of the backend engines like vLLM expose more granular metrics about their performance. I think it makes sense for us to add a guide on how to scrape all of these metrics. Do you have a monitoring stack that you already use?

[samos123]

Here is an example pod scraping metric that you can use with Google Managed Prometheus:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: vllm-pods
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: vllm
  endpoints:
  - port: http
    interval: 30s

I plan to work on optional config to deploy prometheus + grafana together with KubeAI so you get everything working out of the box. Would that be of interest?

[samos123]

We should include the following grafana dashboard: https://github.com/vllm-project/vllm/tree/main/examples/production_monitoring