Alert rule group failed to be applied - context deadline exceeded

[ErikRosic]

Describe the bug

After deploying grafana-operator-grafana-deployment with CR alertrulegroup i'm getting following error:

Alert Rule Group failed to be applied for 1 out of 1 instances. Errors: │
│ - grafana-operator/grafana-operator-grafana: fetching folder: Get "http://grafana-operator-grafana-service.grafana-operator:3000/api/folders/670f1565-8f76-49f6-b78a-51469d413791": context deadline exceeded

Version

Helm chart version 3.12.2

To Reproduce

Deploy following CRs:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: erik-folder
spec:
  instanceSelector:
    matchLabels:
      app: grafana-operator-grafana
  permissions: |
    {
      "items": [
        {
          "role": "Admin",
          "permission": 4
        }
      ]
    }

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: zookeeper-elastic
spec:
  folderRef: erik-folder
  instanceSelector:
    matchLabels:
      app: grafana-operator-grafana
  interval: 1m
  rules:
    - uid: a55303dc-81ba-4d92-8d71-82de4dd6541f
      title: zookeeper-elastic
      condition: A
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: 51226607-32f3-4822-b9e4-d81ce739e094
          model:
            alias: ""
            bucketAggs:
              - field: '@timestamp'
                id: "2"
                settings:
                  interval: auto
                type: date_histogram
            datasource:
              type: elasticsearch
              uid: 51226607-32f3-4822-b9e4-d81ce739e094
            hide: false
            intervalMs: 1000
            maxDataPoints: 43200
            metrics:
              - id: "1"
                type: count
            query: 'kubernetes.labels.app_kubernetes_io/name: zookeeper'
            refId: A
            timeField: '@timestamp'
      noDataState: NoData
      execErrState: Error
      for: 1m
      annotations: { }
      labels: { }
      isPaused: false`

Expected behavior
Deployed alert in grafana-operator-grafana-deployment

Suspect component/Location where the bug might be occurring
Status in CR grafanaalertrulegroup

Runtime (please complete the following information):

k8s version v1.26.10+rke2r2

[theSuess]

We just merged #1475 which fixes an issue with timeouts in the new client.

To verify if this was your issue - do you have a custom timeout set in your Grafana instance?

[theSuess]

This is now fixed in 5.8.1! Feel free to reopen the issue if this still occurs in the newest version