Alert rules are constantly updated - causing alert state reset

[lupa95]

Describe the bug
I am using grafana-operator to deploy Grafana alert rules.

The alert rule groups are synced to the Grafana instances every 10 minutes (default). Sometimes, the alert rules are updated in a manner, that their fingerprint changes. This causes the alert rule state to reset, which again causes a lot of noise by triggering new notifications after the pending period for already firing alerts.

I tried to debug the issue by checking if there is constant drift between the alert rules in the CRs of the Grafana operator and the Grafana instances itself. But i can not find anything wrong. I did also check the alert_rule_version table in the Grafana DB. The only columns that are different between the alert rule versions are:

• id (makes sense)
• parent_version (makes sense)
• version (makes sense)
• created (makes sense)
• rule_group_idx (not sure what that is, some ID for the whole rule group that changes?)

How can i debug this further? Any ideas to why the alert rules are updated constantly?

Version
Grafana operator: v5.17.0
Grafana: v11.5.2

To Reproduce
Steps to reproduce the behavior:

1. Create alert rule groups with multiple alert rules with grafana operator
2. Check alert rule history in Grafana to see them being updated constantly

Expected behavior
Alert rules only update, when they actually change.

Additional context
Output of the Grafana alert_rule_version table for one specific alert rule, that has this problem:

   id   | rule_org_id |          rule_uid            |          rule_namespace_uid          | rule_group   | parent_version | restored_from | version |       created       |            title             | condition |                                                data                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | interval_seconds | no_data_state | exec_err_state |     for      |                                          annotations                                                                                                                                  |                                  labels                   | rule_group_idx | is_paused |     notification_settings | record |                                                    metadata
--------+-------------+------------------------------+--------------------------------------+--------------+----------------+---------------+---------+---------------------+------------------------------+-----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+---------------+----------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+----------------+-----------+---------------------------+--------+-----------------------------------------------------------------------------------------------------------------
 664168 |           1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev  |             21 |             0 |      22 | 2025-03-14 12:54:15 | KubernetesNodeMemoryPressure | B         | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] |              600 | NoData        | Error          | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} |              1 | f         | [{"receiver":"opsgenie"}] |        | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
 664230 |           1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev  |             22 |             0 |      23 | 2025-03-14 13:04:15 | KubernetesNodeMemoryPressure | B         | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] |              600 | NoData        | Error          | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} |              0 | f         | [{"receiver":"opsgenie"}] |        | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
 664238 |           1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev  |             23 |             0 |      24 | 2025-03-14 13:04:16 | KubernetesNodeMemoryPressure | B         | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] |              600 | NoData        | Error          | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} |              1 | f         | [{"receiver":"opsgenie"}] |        | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
 664300 |           1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev  |             24 |             0 |      25 | 2025-03-14 13:14:16 | KubernetesNodeMemoryPressure | B         | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] |              600 | NoData        | Error          | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} |              0 | f         | [{"receiver":"opsgenie"}] |        | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
 664308 |           1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev  |             25 |             0 |      26 | 2025-03-14 13:14:17 | KubernetesNodeMemoryPressure | B         | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] |              600 | NoData        | Error          | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} |              1 | f         | [{"receiver":"opsgenie"}] |        | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
(5 rows)

[pb82]

It could be that Grafana updates the rule_group_idx internally and then the Operator applying the old value back. But we need to look further into this.

[lupa95]

Let me know if i can provide any more information. The alert rule group in question has multiple alert rules. I will try to split everything up (1 alert rule per group) to see if this changes anything.

[weisdd]

@lupa95 would be nice if you could prepare a set of manifests (including grafana instance) that would show the issue in action.

Also, how did the experiment go?

[lupa95]

@weisdd sorry for the delay, i was out of office last week. I have split the alert rules up and will post the results tomorrow. The updates only occour every few hours, so it takes a while to test it.

I have also seen that Grafana 11.6 released. There are a bunch of updates related to the alerting. I will also try the new version, maybe there is a fix for my issue in there.

[lupa95]

After splitting the alert rules into one per group, the issue is gone.

I am using these rules to test (though different sets of rules have the same issue):

Combined into one rule group

---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: hub-shared
  namespace: mla-hub-shared
spec:
  uid: hub-shared
  instanceSelector:
    matchLabels:
      instance: grafana-mla-hub-shared
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: hub-shared
  namespace: mla-hub-shared
spec:
  folderUID: hub-shared
  instanceSelector:
    matchLabels:
      instance: grafana-mla-hub-shared
  interval: 10m
  rules:
    - uid: de18z4kexxedcd
      title: KubernetesNodeNotReady
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: kube_node_status_condition{condition="Ready",status="true"}
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 1
                    - 1
                  type: lt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: NoData
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Node {{ $labels.node }} has been unready for a long time
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes Node ready (node {{ $labels.node }})
      labels:
        severity: critical
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example
    - uid: ee18zv2xe445cb
      title: KubernetesNodeMemoryPressure
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: kube_node_status_condition{condition="MemoryPressure",status="true"}
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 0
                  type: gt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: NoData
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Node {{ $labels.node }} has MemoryPressure condition
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes memory pressure (node {{ $labels.node }})
      labels:
        severity: critical
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example
    - uid: ae19107o584xsd
      title: KubernetesContainerOomKiller
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m])
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 0
                  type: gt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: OK
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes container oom killer ({{ $labels.namespace }}/{{ $labels.pod }}:{{ $labels.container }})
      labels:
        severity: warning
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example
    - uid: fe191f431qyv4c
      title: KubernetesNodeDiskPressure
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: kube_node_status_condition{condition="DiskPressure",status="true"}
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 0
                  type: gt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: NoData
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Node {{ $labels.node }} has DiskPressure condition
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes disk pressure (node {{ $labels.node }})
      labels:
        severity: critical
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example

Rules split

---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: hub-shared
  namespace: mla-hub-shared
spec:
  uid: hub-shared
  instanceSelector:
    matchLabels:
      instance: grafana-mla-hub-shared
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: hub-shared
  namespace: mla-hub-shared
spec:
  folderUID: hub-shared
  instanceSelector:
    matchLabels:
      instance: grafana-mla-hub-shared
  interval: 10m
  rules:
    - uid: de18z4kexxedcd
      title: KubernetesNodeNotReady
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: kube_node_status_condition{condition="Ready",status="true"}
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 1
                    - 1
                  type: lt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: NoData
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Node {{ $labels.node }} has been unready for a long time
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes Node ready (node {{ $labels.node }})
      labels:
        severity: critical
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: hub-shared-1
  namespace: mla-hub-shared
spec:
  folderUID: hub-shared
  instanceSelector:
    matchLabels:
      instance: grafana-mla-hub-shared
  interval: 10m
  rules:
    - uid: ee18zv2xe445cb
      title: KubernetesNodeMemoryPressure
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: kube_node_status_condition{condition="MemoryPressure",status="true"}
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 0
                  type: gt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: NoData
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Node {{ $labels.node }} has MemoryPressure condition
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes memory pressure (node {{ $labels.node }})
      labels:
        severity: critical
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: hub-shared-2
  namespace: mla-hub-shared
spec:
  folderUID: hub-shared
  instanceSelector:
    matchLabels:
      instance: grafana-mla-hub-shared
  interval: 10m
  rules:
    - uid: ae19107o584xsd
      title: KubernetesContainerOomKiller
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m])
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 0
                  type: gt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: OK
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes container oom killer ({{ $labels.namespace }}/{{ $labels.pod }}:{{ $labels.container }})
      labels:
        severity: warning
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: hub-shared-3
  namespace: mla-hub-shared
spec:
  folderUID: hub-shared
  instanceSelector:
    matchLabels:
      instance: grafana-mla-hub-shared
  interval: 10m
  rules:
    - uid: fe191f431qyv4c
      title: KubernetesNodeDiskPressure
      condition: B
      data:
        - refId: A
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: prometheus-hub-shared
          model:
            editorMode: code
            expr: kube_node_status_condition{condition="DiskPressure",status="true"}
            instant: true
            intervalMs: 1000
            legendFormat: __auto
            maxDataPoints: 43200
            range: false
            refId: A
        - refId: B
          relativeTimeRange:
            from: 600
            to: 0
          datasourceUid: __expr__
          model:
            conditions:
              - evaluator:
                  params:
                    - 0
                  type: gt
                operator:
                  type: and
                query:
                  params:
                    - C
                reducer:
                  params: []
                  type: last
                type: query
            datasource:
              type: __expr__
              uid: __expr__
            expression: A
            intervalMs: 1000
            maxDataPoints: 43200
            refId: B
            type: threshold
      noDataState: NoData
      execErrState: Error
      for: 10m
      annotations:
        description: |-
          Node {{ $labels.node }} has DiskPressure condition
          VALUE = {{ $value }}
          LABELS = {{ $labels }}
        summary: Kubernetes disk pressure (node {{ $labels.node }})
      labels:
        severity: critical
        customer: example
        project: hub
        stage: shared
      isPaused: false
      notificationSettings:
        receiver: example

Grafana Instance

---
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana-mla-hub-shared
  namespace: mla-hub-shared
  labels:
    instance: grafana-mla-hub-shared
spec:
  config:
    security:
      disable_gravatar: "true"
    analytics:
      reporting_enabled: "false"
      check_for_updates: "false"
      check_for_plugin_updates: "false"
      feedback_links_enabled: "false"
    news:
      news_feed_enabled: "false"
    database:
      type: postgres
      host: postgres-headless-svc
      name: grafana
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
    server:
      root_url: <sanitized>
    users:
      auto_assign_org_role: None
    auth:
      disable_login_form: "true"
      oauth_auto_login: "true"
    auth.generic_oauth:
      enabled: "true"
      name: Keycloak
      allow_sign_up: "true"
      client_id: grafana-mla-hub-shared
      client_secret: ${KEYCLOAK_CLIENT_SECRET}
      scopes: openid email profile offline_access roles
      email_attribute_path: email
      login_attribute_path: username
      name_attribute_path: full_name
      auth_url: <sanitized>
      token_url: <sanitized>
      api_url: <sanitized>
      role_attribute_path: <sanitized>
      allow_assign_grafana_admin: "true"
      use_refresh_token: "true"
      signout_redirect_url: <sanitized>
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: grafana
              image: grafana/grafana:11.5.2
              env:
                - name: GF_SECURITY_ADMIN_USER
                  valueFrom:
                    secretKeyRef:
                      key: GF_SECURITY_ADMIN_USER
                      name: grafana-admin-secret
                - name: GF_SECURITY_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: GF_SECURITY_ADMIN_PASSWORD
                      name: grafana-admin-secret
                - name: POSTGRES_USER
                  valueFrom:
                    secretKeyRef:
                      key: POSTGRES_USER
                      name: postgres-secret
                - name: POSTGRES_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: POSTGRES_PASSWORD
                      name: postgres-secret
                - name: KEYCLOAK_CLIENT_SECRET
                  valueFrom:
                    secretKeyRef:
                      key: KEYCLOAK_CLIENT_SECRET
                      name: keycloak-secret
  ingress:
    metadata:
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt
    spec:
      ingressClassName: nginx
      rules:
        - host: <sanitized>
          http:
            paths:
              - backend:
                  service:
                    name: grafana-mla-hub-shared-service
                    port:
                      number: 3000
                path: /
                pathType: Prefix
        - host: <sanitized>
          http:
            paths:
              - backend:
                  service:
                    name: grafana-mla-hub-shared-service
                    port:
                      number: 3000
                path: /
                pathType: Prefix
      tls:
        - hosts:
            - <sanitized>
            - <sanitized>
          secretName: grafana-tls-secret

I will test Grafana 11.6 with the combined set of groups and check if that helps.

[lupa95]

Upgrading to Grafana 11.6 did not solve my issue.

[lupa95]

Any ideas what causes this, or what i can troubleshoot?

[weisdd]

Most likely, it's an upstream issue, so can be fully resolved only on Grafana side.

[github-actions]

This issue hasn't been updated for a while, marking as stale, please respond within the next 7 days to remove this label

[sachinjkattoor]

I also ran into this issue.

It could be that Grafana updates the rule_group_idx internally and then the Operator applying the old value back. But we need to look further into this.

@pb82 Did you find anything on this?