Skip to content

Commit 56f09a6

Browse files
notify: always retry with a back-off (prometheus#2290)
By default the library implementing the back-off timer stops the timer after 15 minutes. Since the code never checked the value returned by the ticker, notification retries were executed without delay after the 15 minutes had elapsed (e.g. for `group_interval` greater than 15m). This change ensures that the back-off timer never expires. Signed-off-by: Simon Pasquier <[email protected]>
1 parent 2f74a34 commit 56f09a6

File tree

2 files changed

+7
-3
lines changed

2 files changed

+7
-3
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
* [BUGFIX] Fix a potential race condition in dispatcher. #2208
1414
* [BUGFIX] [API v2] Return an empty array of peers when the clustering is disabled. #2203
1515
* [BUGFIX] Fix the registration of `alertmanager_dispatcher_aggregation_groups` and `alertmanager_dispatcher_alert_processing_duration_seconds` metrics. #2200
16+
* [BUGFIX] Always retry notifications with back-off. #2290
1617

1718
## 0.20.0 / 2019-12-11
1819

notify/notify.go

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -633,13 +633,16 @@ func (r RetryStage) Exec(ctx context.Context, l log.Logger, alerts ...*types.Ale
633633
sent = alerts
634634
}
635635

636+
b := backoff.NewExponentialBackOff()
637+
b.MaxElapsedTime = 0 // Always retry.
638+
639+
tick := backoff.NewTicker(b)
640+
defer tick.Stop()
641+
636642
var (
637643
i = 0
638-
b = backoff.NewExponentialBackOff()
639-
tick = backoff.NewTicker(b)
640644
iErr error
641645
)
642-
defer tick.Stop()
643646
l = log.With(l, "receiver", r.groupName, "integration", r.integration.String())
644647

645648
for {

0 commit comments

Comments
 (0)