Enable failure store to APM datastreams

[rubvs]

Closes https://github.com/elastic/apm-managed-service/issues/1541

Enable failure store to APM datastreams

[jbaiera]

LGTM, though this will likely need to wait until after the feature is out from behind the feature flag to be merged.

[jbaiera]

This won't handle APM data streams that already exist when applied, is that something that needs to be considered as well for this rollout?

[rubvs]

This won't handle APM data streams that already exist when applied, is that something that needs to be considered as well for this rollout?

@1pkg and @simitt can you please give your input on this?

[simitt]

@jbaiera is there a different way to also apply it to existing data streams?

[jbaiera]

is there a different way to also apply it to existing data streams?

@simitt This is what the data_streams.failure_store.enabled cluster setting is for. It takes a list of index wildcard patterns as a value, and any data streams that match the patterns will have failure store enabled, as long as those data streams do not have failure store explicitly enabled/disabled. It's effectively selecting the data stream names that will have the feature enabled by default if not currently specified.

We'll need to use both the template and the cluster setting as part of the rollout. The template to enable the failure store on all data streams going forward, and the cluster setting to pick up any existing data streams from before the template update.