Notifications have became useless due to high volumes

[sylr]

Hi,

First, let me wish you all the best for 2021 and congratulate for the work you've done on flux2 in 2020 👍

However I'd like to report that the volume of notification generated by the the different flux controllers in case of error is such that the configured target channels have became trash in the eyes of my colleagues.

We have one kustomization per namespace and dozens of namespaces. Each time their is communication errors between components we get as many notifications as we have kustomizations/helmreleases/repositories ... etc.

Also when the error is limited to the reconciliation of one flux object, we get a repetition of notifications each time the components perform retries.

I think that either the notification-controller for all the other controllers, or each controllers, should keep error states in order to limit the volume of notifications down to what is really informative.

Regards.

[stefanprodan]

flux2 v0.9.1 release comes with improvements to the notification system. The kustomize-controller and helm-controller are now performing retries with exponential backoff when fetching artifacts. This prevents spamming events and alerts when source-controller becomes unavailable for a short period of time (e.g. upgrades, pod rescheduling, leader election changes, etc).

@sylr please give it a try and let us know if the situation has improved for you.