Add ability to redirect ingestion failures on data streams to a failure store

[jbaiera]

Documents that encountered ingest pipeline failures or mapping conflicts would previously be returned to the client as errors in the bulk and index operations. Many client applications are not equipped to respond to these failures. This leads to the failed documents often being dropped by the client which cannot hold the broken documents indefinitely. In many end user workloads, these failed documents represent events that could be critical signals for observability or security use cases.

To help mitigate this problem, data streams now maintain a "failure store" which is used to accept and hold documents that fail to be ingested due to preventable configuration errors. The data stream's failure store operates like a separate set of backing indices with their own mappings and access patterns that allow Elasticsearch to accept documents that would otherwise be rejected due to unhandled ingest pipeline exceptions or mapping conflicts.

Users can enable redirection of ingest failures to the failure store on new data streams by specifying it in the new data_stream_options field inside of a component or index template:

PUT _index_template/my-template
{
  "index_patterns": ["logs-test-*"],
  "data_stream": {},
  "template": {
    "data_stream_options": {
      "failure_store": {
        "enabled": true
      }
    }
  }
}'

Existing data streams can be configured with the new data stream _options endpoint:

PUT _data_stream/logs-test-apache/_options
{
  "failure_store": {
    "enabled": "true"
  }
}

When redirection is enabled, any ingestion related failures will be captured in the failure store if the cluster is able to, along with the timestamp that the failure occurred, details about the error encountered, and the document that could not be ingested. Since failure stores are a kind of Elasticsearch index, we can search the data stream for the failures that it has collected. The failures are not shown by default as they are stored in different indices than the normal data stream data. In order to retrieve the failures, we use the _search API along with a new bit of index pattern syntax, the :: selector.

POST logs-test-apache::failures/_search

This index syntax informs the search operation to target the indices in its failure store instead of its backing indices. It can be mixed in a number of ways with other index patterns to include their failure store indices in the search operation:

POST logs-*::failures/_search
POST logs-*,logs-*::failures/_search
POST *::failures/_search
POST _query
{
  "query": "FROM my_data_stream*::failures"
}

---

This PR removes the feature flags and guards that prevent the new failure store functionality from operating in production runtimes.

[elasticsearchmachine]

Hi @jbaiera, I've created a changelog YAML for you.

[elasticsearchmachine]

Hi @jbaiera, I've updated the changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

[jbaiera]

@elasticmachine update branch

[jbaiera]

@elasticmachine update branch

[jbaiera]

@elasticmachine update branch

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

LGTM, I left a few minor comments

[dakrone]

Why should this be removed after backport?

[jbaiera]

This is an old field - we use the data stream options to determine if failure store is enabled now. Currently, the field is read and used as a fall back if data stream options are not present, but I think that's mostly for BWC testing during development. I think logic-wise we could simply ignore the field if present because it would always be overridden by data stream options. The only situation where this field is relevant is mixed clusters with 8.19 nodes and very old nodes running with the feature flag on which we would not support.

I opened #127071 for this. I also cleaned up the serialization logic a little.