[Bug] Loading dashboards from GrafanaDashboard - "context deadline exceeded"

[Mart-Kuc]

Describe the bug
We are using K8S object GrafanaDashboard for importing json dashboard to Grafana from ConfigMap.
When GrafanaOperator try to "import" Dashboards gets error context deadline exceeded

We have approximately 22 GrafanaDashboards object with 22 ConfigMaps, so 22 dashboards in the result.

Log:
ERROR GrafanaDashboardReconciler error reconciling dashboard {"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"qm--frontend-grafanadashboard","namespace":"monitoring"}, "namespace": "monitoring", "name": "qm--frontend-grafanadashboard", "reconcileID": "1be2129b-189b-46ca-5713-1c2b50319a91", "dashboard": "qm--frontend-grafanadashboard", "grafana": "grafana-name", "error": "Internal error occurred: Get \"http://grafana-name-service.monitoring:3000/api/folders?limit=1000&page=1\": context deadline exceeded"}

In version 5.6.3 we didn't any problem with this one

Version
Grafana-operator: v5.8.0
Grafana: 10.4.0
Helm-chart: bitnami-grafana-operator: 3.12.3

To Reproduce
Steps to reproduce the behavior:

1. Deploy Grafana/Grafana-operator
2. Create ConfigMap with Json dashboard
3. Create K8S object GrafanaDashboards with referencing on ConfigMap and proper "instanceSelector"
4. Go to logs of Grafana-operator

Expected behavior
Importing all Dashboards to Grafana

Suspect component/Location where the bug might be occurring
unknown

Runtime (please complete the following information):

• Grafana Operator: v5.8.0
• Environment: Kubernetes v1.27.11
• Deployment type: helm-chart: 3.12.3
• Other: Cilium: 15.1

Best regard: Mart-Kuc

[theSuess]

This and #1465 both seem to be a result of switching to the new openapi client. I'll take a look at it this week

[theSuess]

We just merged #1475 which fixes an issue with timeouts in the new client.

To verify if this was your issue - do you have a custom timeout set in your Grafana instance?

[occelebi]

Bitnami's grafana operator chart deploys it as default with timeout, unfortunately :/

  client:
    timeout: 5

https://github.com/bitnami/charts/blob/0bd4fe03a2985297c4791e392093b942fbeb54f3/bitnami/grafana-operator/values.yaml#L658

[theSuess]

Got it - this is now fixed in 5.8.1! Feel free to reopen the issue if this still occurs in the newest version