[Failure Store] Expose failure store lifecycle information via the `GET` data stream API #126668

gmarouli · 2025-04-11T09:11:16Z

To retrieve the effective configuration you need to use the GET data streams API, for example, if a data stream has empty data stream options, it might still have failure store enabled from a cluster setting. The failure store is managed by default with a lifecycle with infinite (for now) retention, so the response will look like this:

GET _data_stream/*
{
  "data_streams": [
    {
      "name": "my-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      .....
      "failure_store": {
        "enabled": true,
        "lifecycle": {
          "enabled": true
        },
        "rollover_on_write": false,
        "indices": [
           {
            "index_name": ".fs-my-data-stream-2099.03.08-000003",
            "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
            "managed_by": "Data stream lifecycle"
          }
        ]
      }
    },...
]

In case there is a failure indexed managed by ILM the failure index info will be displayed as follows.

      {
          "index_name": ".fs-my-data-stream-2099.03.08-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "my-lifecycle-policy",
          "managed_by": "Index Lifecycle Management"
        }

elasticsearchmachine · 2025-04-11T09:16:47Z

Pinging @elastic/es-data-management (Team:Data Management)

jbaiera

LGTM, left a couple small comments. I think also now that we've removed the feature flag some of the code can be simplified.

jbaiera · 2025-04-22T06:26:46Z

server/src/main/java/org/elasticsearch/cluster/metadata/DataStreamFailureStore.java

@@ -29,6 +29,7 @@
 * supports the following configurations only explicitly enabling or disabling the failure store
 */
 public record DataStreamFailureStore(Boolean enabled) implements SimpleDiffable<DataStreamFailureStore>, ToXContentObject {
+    public static final String FAILURES_LIFECYCLE_API_CAPABILITY = "failures_lifecycle";


Nit: It looks like this is only used in the rest action. Does it make sense to move this there? Alternatively, it's describing something about the structure of the response, should it be moved to the response object/action class?

Also, name wise, should we make this something like failure_store.lifecycle to denote specifically what part of the request is new? I see we have other changes to the response as well, so this is a soft suggestion. If you think this should be more generic to describe those changes that's also fine.

+1 on the naming, about placement, I put it here because it will be used also by the data stream options later. But we can always move it then.

elasticsearchmachine · 2025-04-23T13:46:01Z

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 126668

gmarouli · 2025-04-23T14:36:29Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

@timestamp

…ET` data stream API (elastic#126668) To retrieve the effective configuration you need to use the `GET` data streams API, for example, if a data stream has empty data stream options, it might still have failure store enabled from a cluster setting. The failure store is managed by default with a lifecycle with infinite (for now) retention, so the response will look like this: ``` GET _data_stream/* { "data_streams": [ { "name": "my-data-stream", "timestamp_field": { "name": "@timestamp" }, ..... "failure_store": { "enabled": true, "lifecycle": { "enabled": true }, "rollover_on_write": false, "indices": [ { "index_name": ".fs-my-data-stream-2099.03.08-000003", "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw", "managed_by": "Data stream lifecycle" } ] } },... ] ``` In case there is a failure indexed managed by ILM the failure index info will be displayed as follows. ``` { "index_name": ".fs-my-data-stream-2099.03.08-000002", "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw", "prefer_ilm": true, "ilm_policy": "my-lifecycle-policy", "managed_by": "Index Lifecycle Management" } ``` (cherry picked from commit db2992f) # Conflicts: # modules/data-streams/src/test/java/org/elasticsearch/datastreams/action/GetDataStreamsResponseTests.java # server/src/main/java/org/elasticsearch/action/datastreams/GetDataStreamAction.java

@timestamp

…the `GET` data stream API (#126668) (#127255) * [Failure Store] Expose failure store lifecycle information via the `GET` data stream API (#126668) To retrieve the effective configuration you need to use the `GET` data streams API, for example, if a data stream has empty data stream options, it might still have failure store enabled from a cluster setting. The failure store is managed by default with a lifecycle with infinite (for now) retention, so the response will look like this: ``` GET _data_stream/* { "data_streams": [ { "name": "my-data-stream", "timestamp_field": { "name": "@timestamp" }, ..... "failure_store": { "enabled": true, "lifecycle": { "enabled": true }, "rollover_on_write": false, "indices": [ { "index_name": ".fs-my-data-stream-2099.03.08-000003", "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw", "managed_by": "Data stream lifecycle" } ] } },... ] ``` In case there is a failure indexed managed by ILM the failure index info will be displayed as follows. ``` { "index_name": ".fs-my-data-stream-2099.03.08-000002", "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw", "prefer_ilm": true, "ilm_policy": "my-lifecycle-policy", "managed_by": "Index Lifecycle Management" } ``` (cherry picked from commit db2992f) # Conflicts: # modules/data-streams/src/test/java/org/elasticsearch/datastreams/action/GetDataStreamsResponseTests.java # server/src/main/java/org/elasticsearch/action/datastreams/GetDataStreamAction.java * Ignore correctly the failure store response

…tion (#127314) The failure store is a set of data stream indices that are used to store certain type of ingestion failures. Until this moment they were sharing the configuration of the backing indices. We understand that the two data sets have different lifecycle needs. We believe that typically the failures will need to be retained much less than the data. Considering this we believe the lifecycle needs of the failures also more limited and they fit better the simplicity of the data stream lifecycle feature. This allows the user to only set the desired retention and we will perform the rollover and other maintenance tasks without the user having to think about them. Furthermore, having only one lifecycle management feature allows us to ensure that these data is managed by default. This PR introduces the following: Configuration We extend the failure store configuration to allow lifecycle configuration too, this configuration reflects the user's configuration only as shown below: PUT _data_stream/*/options { "failure_store": { "lifecycle": { "data_retention": "5d" } } } GET _data_stream/*/options { "data_streams": [ { "name": "my-ds", "options": { "failure_store": { "lifecycle": { "data_retention": "5d" } } } } ] } To retrieve the effective configuration you need to use the GET data streams API, see #126668 Functionality The data stream lifecycle (DLM) will manage the failure indices regardless if the failure store is enabled or not. This will ensure that if the failure store gets disabled we will not have stagnant data. The data stream options APIs reflect only the user's configuration. The GET data stream API should be used to check the current state of the effective failure store configuration. Telemetry We extend the data stream failure store telemetry to also include the lifecycle telemetry. { "data_streams": { "available": true, "enabled": true, "data_streams": 10, "indices_count": 50, "failure_store": { "explicitly_enabled_count": 1, "effectively_enabled_count": 15, "failure_indices_count": 30 "lifecycle": { "explicitly_enabled_count": 5, "effectively_enabled_count": 20, "data_retention": { "configured_data_streams": 5, "minimum_millis": X, "maximum_millis": Y, "average_millis": Z, }, "effective_retention": { "retained_data_streams": 20, "minimum_millis": X, "maximum_millis": Y, "average_millis": Z }, "global_retention": { "max": { "defined": false }, "default": { "defined": true, <------ this is the default value applicable for the failure store "millis": X } } } } } Implementation details We ensure that partially reset failure store will create valid failure store configuration. We ensure that when a node communicates with a note with a previous version it will ensure it will not send an invalid failure store configuration enabled: null.

…tion (elastic#127314) The failure store is a set of data stream indices that are used to store certain type of ingestion failures. Until this moment they were sharing the configuration of the backing indices. We understand that the two data sets have different lifecycle needs. We believe that typically the failures will need to be retained much less than the data. Considering this we believe the lifecycle needs of the failures also more limited and they fit better the simplicity of the data stream lifecycle feature. This allows the user to only set the desired retention and we will perform the rollover and other maintenance tasks without the user having to think about them. Furthermore, having only one lifecycle management feature allows us to ensure that these data is managed by default. This PR introduces the following: Configuration We extend the failure store configuration to allow lifecycle configuration too, this configuration reflects the user's configuration only as shown below: PUT _data_stream/*/options { "failure_store": { "lifecycle": { "data_retention": "5d" } } } GET _data_stream/*/options { "data_streams": [ { "name": "my-ds", "options": { "failure_store": { "lifecycle": { "data_retention": "5d" } } } } ] } To retrieve the effective configuration you need to use the GET data streams API, see elastic#126668 Functionality The data stream lifecycle (DLM) will manage the failure indices regardless if the failure store is enabled or not. This will ensure that if the failure store gets disabled we will not have stagnant data. The data stream options APIs reflect only the user's configuration. The GET data stream API should be used to check the current state of the effective failure store configuration. Telemetry We extend the data stream failure store telemetry to also include the lifecycle telemetry. { "data_streams": { "available": true, "enabled": true, "data_streams": 10, "indices_count": 50, "failure_store": { "explicitly_enabled_count": 1, "effectively_enabled_count": 15, "failure_indices_count": 30 "lifecycle": { "explicitly_enabled_count": 5, "effectively_enabled_count": 20, "data_retention": { "configured_data_streams": 5, "minimum_millis": X, "maximum_millis": Y, "average_millis": Z, }, "effective_retention": { "retained_data_streams": 20, "minimum_millis": X, "maximum_millis": Y, "average_millis": Z }, "global_retention": { "max": { "defined": false }, "default": { "defined": true, <------ this is the default value applicable for the failure store "millis": X } } } } } Implementation details We ensure that partially reset failure store will create valid failure store configuration. We ensure that when a node communicates with a note with a previous version it will ensure it will not send an invalid failure store configuration enabled: null. (cherry picked from commit 03d7781) # Conflicts: # modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamsPlugin.java # modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleService.java # modules/data-streams/src/test/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleServiceTests.java # server/src/main/java/org/elasticsearch/TransportVersions.java # server/src/main/java/org/elasticsearch/cluster/metadata/MetadataDataStreamsService.java # server/src/test/java/org/elasticsearch/cluster/metadata/DataStreamTests.java

…nfiguration (#127314) (#127577) * [Failure store] Introduce dedicated failure store lifecycle configuration (#127314) The failure store is a set of data stream indices that are used to store certain type of ingestion failures. Until this moment they were sharing the configuration of the backing indices. We understand that the two data sets have different lifecycle needs. We believe that typically the failures will need to be retained much less than the data. Considering this we believe the lifecycle needs of the failures also more limited and they fit better the simplicity of the data stream lifecycle feature. This allows the user to only set the desired retention and we will perform the rollover and other maintenance tasks without the user having to think about them. Furthermore, having only one lifecycle management feature allows us to ensure that these data is managed by default. This PR introduces the following: Configuration We extend the failure store configuration to allow lifecycle configuration too, this configuration reflects the user's configuration only as shown below: PUT _data_stream/*/options { "failure_store": { "lifecycle": { "data_retention": "5d" } } } GET _data_stream/*/options { "data_streams": [ { "name": "my-ds", "options": { "failure_store": { "lifecycle": { "data_retention": "5d" } } } } ] } To retrieve the effective configuration you need to use the GET data streams API, see #126668 Functionality The data stream lifecycle (DLM) will manage the failure indices regardless if the failure store is enabled or not. This will ensure that if the failure store gets disabled we will not have stagnant data. The data stream options APIs reflect only the user's configuration. The GET data stream API should be used to check the current state of the effective failure store configuration. Telemetry We extend the data stream failure store telemetry to also include the lifecycle telemetry. { "data_streams": { "available": true, "enabled": true, "data_streams": 10, "indices_count": 50, "failure_store": { "explicitly_enabled_count": 1, "effectively_enabled_count": 15, "failure_indices_count": 30 "lifecycle": { "explicitly_enabled_count": 5, "effectively_enabled_count": 20, "data_retention": { "configured_data_streams": 5, "minimum_millis": X, "maximum_millis": Y, "average_millis": Z, }, "effective_retention": { "retained_data_streams": 20, "minimum_millis": X, "maximum_millis": Y, "average_millis": Z }, "global_retention": { "max": { "defined": false }, "default": { "defined": true, <------ this is the default value applicable for the failure store "millis": X } } } } } Implementation details We ensure that partially reset failure store will create valid failure store configuration. We ensure that when a node communicates with a note with a previous version it will ensure it will not send an invalid failure store configuration enabled: null. (cherry picked from commit 03d7781) # Conflicts: # modules/data-streams/src/main/java/org/elasticsearch/datastreams/DataStreamsPlugin.java # modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleService.java # modules/data-streams/src/test/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleServiceTests.java # server/src/main/java/org/elasticsearch/TransportVersions.java # server/src/main/java/org/elasticsearch/cluster/metadata/MetadataDataStreamsService.java # server/src/test/java/org/elasticsearch/cluster/metadata/DataStreamTests.java * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <[email protected]>

Expose failure store lifecycle information via the GET data stream API

dc5f3cd

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.1.0 labels Apr 11, 2025

gmarouli added >non-issue :Data Management/Data streams Data streams and their lifecycles auto-backport Automatically create backport pull requests when merged v8.19.0 labels Apr 11, 2025

elasticsearchmachine added Team:Data Management Meta label for data/management team and removed needs:triage Requires assignment of a team area label labels Apr 11, 2025

Add test case for when we display ilm

cd4e3fa

gmarouli requested a review from jbaiera April 11, 2025 10:50

gmarouli mentioned this pull request Apr 11, 2025

[WIP] Failure store - Lifecycle Management #125658

Closed

gmarouli added 2 commits April 11, 2025 14:10

Move test to a more appropriate place

3fa4335

Merge branch 'main' into failure-lifecycle/expose-in-get-data-streams

612628f

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Apr 14, 2025

gmarouli added 4 commits April 14, 2025 14:54

Merge branch 'main' into failure-lifecycle/expose-in-get-data-streams

8586b4b

Merge branch 'main' into failure-lifecycle/expose-in-get-data-streams

5ecc07a

Add test case that covers failure store enabled by cluster setting

71eb3ed

Merge branch 'main' into failure-lifecycle/expose-in-get-data-streams

705fb99

jbaiera approved these changes Apr 22, 2025

View reviewed changes

gmarouli added 2 commits April 23, 2025 13:40

Merge branch 'main' into failure-lifecycle/expose-in-get-data-streams

09ac0e4

Address review comments

76fc076

gmarouli added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Apr 23, 2025

Fix test

5e304cd

elasticsearchmachine merged commit db2992f into elastic:main Apr 23, 2025
19 checks passed

gmarouli deleted the failure-lifecycle/expose-in-get-data-streams branch April 23, 2025 13:45

elasticsearchmachine added the backport pending label Apr 23, 2025

gmarouli mentioned this pull request Apr 23, 2025

[8.x] [Failure Store] Expose failure store lifecycle information via the GET data stream API (#126668) #127255

Merged

gmarouli mentioned this pull request Apr 24, 2025

[Failure store] Introduce dedicated failure store lifecycle configuration #127314

Merged

gmarouli removed the backport pending label Apr 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Failure Store] Expose failure store lifecycle information via the `GET` data stream API #126668

[Failure Store] Expose failure store lifecycle information via the `GET` data stream API #126668

gmarouli commented Apr 11, 2025 •

edited

Loading

elasticsearchmachine commented Apr 11, 2025

jbaiera left a comment

jbaiera Apr 22, 2025

jbaiera Apr 22, 2025

gmarouli Apr 23, 2025

elasticsearchmachine commented Apr 23, 2025

gmarouli commented Apr 23, 2025

[Failure Store] Expose failure store lifecycle information via the GET data stream API #126668

[Failure Store] Expose failure store lifecycle information via the GET data stream API #126668

Conversation

gmarouli commented Apr 11, 2025 • edited Loading

elasticsearchmachine commented Apr 11, 2025

jbaiera left a comment

Choose a reason for hiding this comment

jbaiera Apr 22, 2025

Choose a reason for hiding this comment

jbaiera Apr 22, 2025

Choose a reason for hiding this comment

gmarouli Apr 23, 2025

Choose a reason for hiding this comment

elasticsearchmachine commented Apr 23, 2025

💔 Backport failed

gmarouli commented Apr 23, 2025

💚 All backports created successfully

Questions ?

[Failure Store] Expose failure store lifecycle information via the `GET` data stream API #126668

[Failure Store] Expose failure store lifecycle information via the `GET` data stream API #126668

gmarouli commented Apr 11, 2025 •

edited

Loading