notify: always retry with a back-off (prometheus#2290)

By default the library implementing the back-off timer stops the timer
after 15 minutes. Since the code never checked the value returned by the
ticker, notification retries were executed without delay after the 15
minutes had elapsed (e.g. for `group_interval` greater than 15m).

This change ensures that the back-off timer never expires.

Signed-off-by: Simon Pasquier <[email protected]>

Author: simonpasquier

CHANGELOG.md

@@ -13,6 +13,7 @@
 * [BUGFIX] Fix a potential race condition in dispatcher. #2208
 * [BUGFIX] [API v2] Return an empty array of peers when the clustering is disabled. #2203
 * [BUGFIX] Fix the registration of `alertmanager_dispatcher_aggregation_groups` and `alertmanager_dispatcher_alert_processing_duration_seconds` metrics. #2200
+* [BUGFIX] Always retry notifications with back-off. #2290
 
 ## 0.20.0 / 2019-12-11
 

notify/notify.go

@@ -633,13 +633,16 @@ func (r RetryStage) Exec(ctx context.Context, l log.Logger, alerts ...*types.Ale
 		sent = alerts
 	}
 
+	b := backoff.NewExponentialBackOff()
+	b.MaxElapsedTime = 0 // Always retry.
+
+	tick := backoff.NewTicker(b)
+	defer tick.Stop()
+
 	var (
 		i    = 0
-		b    = backoff.NewExponentialBackOff()
-		tick = backoff.NewTicker(b)
 		iErr error
 	)
-	defer tick.Stop()
 	l = log.With(l, "receiver", r.groupName, "integration", r.integration.String())
 
 	for {