Skip to content

Commit d1e4be8

Browse files
mridula-s109mridula-s109
authored
Merge branch 'main' into pinned-retriever
2 parents 43d4daa + 4494fdc commit d1e4be8

File tree

99 files changed

+2580
-1011
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

99 files changed

+2580
-1011
lines changed

‎docs/changelog/127314.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 127314
2+
summary: "[Failure store] Introduce dedicated failure store lifecycle configuration"
3+
area: Data streams
4+
type: enhancement
5+
issues: []

‎docs/changelog/127532.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 127532
2+
summary: Fix case insensitive comparisons to ""
3+
area: ES|QL
4+
type: bug
5+
issues:
6+
- 127431

‎docs/reference/query-languages/sql/sql-commands.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-commands.html
44
---
55

6-
# SQL Commands [sql-commands]
6+
# SQL commands [sql-commands]
77

88
This section contains the list of SQL commands supported by Elasticsearch SQL along with their syntax:
99

‎docs/reference/query-languages/sql/sql-data-types.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-data-types.html
44
---
55

6-
# Data Types [sql-data-types]
6+
# Data types [sql-data-types]
77

88
**Core types**
99

‎docs/reference/query-languages/sql/sql-functions-aggs.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-aggs.html
44
---
55

6-
# Aggregate Functions [sql-functions-aggs]
6+
# Aggregate functions [sql-functions-aggs]
77

88
Functions for computing a *single* result from a set of input values. Elasticsearch SQL supports aggregate functions only alongside [grouping](/reference/query-languages/sql/sql-syntax-select.md#sql-syntax-group-by) (implicit or explicit).
99

1010

11-
## General Purpose [sql-functions-aggs-general]
11+
## General purpose [sql-functions-aggs-general]
1212

1313
## `AVG` [sql-functions-aggs-avg]
1414

‎docs/reference/query-languages/sql/sql-functions-conditional.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-conditional.html
44
---
55

6-
# Conditional Functions And Expressions [sql-functions-conditional]
6+
# Conditional functions and expressions [sql-functions-conditional]
77

88
Functions that return one of their arguments by evaluating in an if-else manner.
99

‎docs/reference/query-languages/sql/sql-functions-datetime.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-datetime.html
44
---
55

6-
# Date/Time and Interval Functions and Operators [sql-functions-datetime]
6+
# Date/time and interval functions and operators [sql-functions-datetime]
77

88
Elasticsearch SQL offers a wide range of facilities for performing date/time manipulations.
99

‎docs/reference/query-languages/sql/sql-functions-geo.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-geo.html
44
---
55

6-
# Geo Functions [sql-functions-geo]
6+
# Geo functions [sql-functions-geo]
77

88
::::{warning}
99
This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

‎docs/reference/query-languages/sql/sql-functions-grouping.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-grouping.html
44
---
55

6-
# Grouping Functions [sql-functions-grouping]
6+
# Grouping functions [sql-functions-grouping]
77

88
Functions for creating special *grouping*s (also known as *bucketing*); as such these need to be used as part of the [grouping](/reference/query-languages/sql/sql-syntax-select.md#sql-syntax-group-by).
99

‎docs/reference/query-languages/sql/sql-functions-math.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-math.html
44
---
55

6-
# Mathematical Functions [sql-functions-math]
6+
# Mathematical functions [sql-functions-math]
77

88
All math and trigonometric functions require their input (where applicable) to be numeric.
99

‎docs/reference/query-languages/sql/sql-functions-search.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-search.html
44
---
55

6-
# Full-Text Search Functions [sql-functions-search]
6+
# Full-text search functions [sql-functions-search]
77

88
Search functions should be used when performing full-text search, namely when the `MATCH` or `QUERY` predicates are being used. Outside a, so-called, search context, these functions will return default values such as `0` or `NULL`.
99

‎docs/reference/query-languages/sql/sql-functions-string.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-string.html
44
---
55

6-
# String Functions [sql-functions-string]
6+
# String functions [sql-functions-string]
77

88
Functions for performing string manipulation.
99

‎docs/reference/query-languages/sql/sql-functions-system.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-system.html
44
---
55

6-
# System Functions [sql-functions-system]
6+
# System functions [sql-functions-system]
77

88
These functions return metadata type of information about the system being queried.
99

‎docs/reference/query-languages/sql/sql-functions-type-conversion.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions-type-conversion.html
44
---
55

6-
# Type Conversion Functions [sql-functions-type-conversion]
6+
# Type conversion functions [sql-functions-type-conversion]
77

88
Functions for converting an expression of one data type to another.
99

‎docs/reference/query-languages/sql/sql-functions.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-functions.html
44
---
55

6-
# Functions and Operators [sql-functions]
6+
# Functions and operators [sql-functions]
77

88
Elasticsearch SQL provides a comprehensive set of built-in operators and functions:
99

‎docs/reference/query-languages/sql/sql-index-frozen.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-index-frozen.html
44
---
55

6-
# Frozen Indices [sql-index-frozen]
6+
# Frozen indices [sql-index-frozen]
77

8-
By default, Elasticsearch SQL doesn’t search [frozen indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). To search frozen indices, use one of the following features:
8+
By default, Elasticsearch SQL doesn't search frozen indices. To search frozen indices, use one of the following features:
99

1010
dedicated configuration parameter
1111
: Set to `true` properties `index_include_frozen` in the [SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query) or `index.include.frozen` in the drivers to include frozen indices.

‎docs/reference/query-languages/sql/sql-lexical-structure.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-lexical-structure.html
44
---
55

6-
# Lexical Structure [sql-lexical-structure]
6+
# Lexical structure [sql-lexical-structure]
77

88
This section covers the major lexical structure of SQL, which for the most part, is going to resemble that of ANSI SQL itself hence why low-levels details are not discussed in depth.
99

‎docs/reference/query-languages/sql/sql-like-rlike-operators.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-like-rlike-operators.html
44
---
55

6-
# LIKE and RLIKE Operators [sql-like-rlike-operators]
6+
# LIKE and RLIKE operators [sql-like-rlike-operators]
77

88
`LIKE` and `RLIKE` operators are commonly used to filter data based on string patterns. They usually act on a field placed on the left-hand side of the operator, but can also act on a constant (literal) expression. The right-hand side of the operator represents the pattern. Both can be used in the `WHERE` clause of the `SELECT` statement, but `LIKE` can also be used in other places, such as defining an [index pattern](/reference/query-languages/sql/sql-index-patterns.md) or across various [SHOW commands](/reference/query-languages/sql/sql-commands.md). This section covers only the `SELECT ... WHERE ...` usage.
99

‎docs/reference/query-languages/sql/sql-limitations.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# SQL Limitations [sql-limitations]
1+
# SQL limitations [sql-limitations]
22

33

44
## Large queries may throw `ParsingException` [large-parsing-trees]

‎docs/reference/query-languages/sql/sql-operators-cast.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-operators-cast.html
44
---
55

6-
# Cast Operators [sql-operators-cast]
6+
# Cast operators [sql-operators-cast]
77

88
## `Cast (::)` [sql-operators-cast-cast]
99

‎docs/reference/query-languages/sql/sql-operators-logical.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-operators-logical.html
44
---
55

6-
# Logical Operators [sql-operators-logical]
6+
# Logical operators [sql-operators-logical]
77

88
Boolean operator for evaluating one or two expressions.
99

‎docs/reference/query-languages/sql/sql-operators-math.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-operators-math.html
44
---
55

6-
# Math Operators [sql-operators-math]
6+
# Math operators [sql-operators-math]
77

88
Perform mathematical operations affecting one or two values. The result is a value of numeric type.
99

‎docs/reference/query-languages/sql/sql-operators.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-operators.html
44
---
55

6-
# Comparison Operators [sql-operators]
6+
# Comparison operators [sql-operators]
77

88
Boolean operator for comparing against one or multiple expressions.
99

‎docs/reference/query-languages/sql/sql-spec.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,17 @@ mapped_pages:
33
- https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-spec.html
44
---
55

6-
# SQL Language [sql-spec]
6+
# SQL language [sql-spec]
77

8-
This chapter describes the SQL syntax and semantics supported namely:
8+
This section describes the SQL syntax and semantics supported namely:
99

10-
[Lexical Structure](/reference/query-languages/sql/sql-lexical-structure.md)
10+
[Lexical structure](/reference/query-languages/sql/sql-lexical-structure.md)
1111
: Lexical structure
1212

13-
[SQL Commands](/reference/query-languages/sql/sql-commands.md)
13+
[SQL commands](/reference/query-languages/sql/sql-commands.md)
1414
: Commands
1515

16-
[Data Types](/reference/query-languages/sql/sql-data-types.md)
16+
[Data types](/reference/query-languages/sql/sql-data-types.md)
1717
: Data types
1818

1919
[Index patterns](/reference/query-languages/sql/sql-index-patterns.md)

‎modules/data-streams/build.gradle

+1
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ tasks.named("yamlRestCompatTestTransform").configure({ task ->
7878
task.skipTest("data_stream/210_rollover_failure_store/Lazily roll over a data stream's failure store after an ingest failure", "Rolling over a data stream using target_failure_store is no longer supported.")
7979
task.skipTest("data_stream/210_rollover_failure_store/Lazily roll over a data stream's failure store after a shard failure", "Rolling over a data stream using target_failure_store is no longer supported.")
8080
task.skipTest("data_stream/210_rollover_failure_store/Roll over a data stream's failure store without conditions", "Rolling over a data stream using target_failure_store is no longer supported.")
81+
task.skipTest("data_stream/240_failure_store_info/Get failure store info from explicitly enabled failure store and disabled lifecycle", "failure store lifecycle is not using anymore the data stream lifecycle")
8182
})
8283

8384
configurations {

‎modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/lifecycle/DataStreamLifecycleServiceIT.java

+34-18
Original file line numberDiff line numberDiff line change
@@ -28,19 +28,20 @@
2828
import org.elasticsearch.action.datastreams.DeleteDataStreamAction;
2929
import org.elasticsearch.action.datastreams.GetDataStreamAction;
3030
import org.elasticsearch.action.datastreams.ModifyDataStreamsAction;
31+
import org.elasticsearch.action.datastreams.PutDataStreamOptionsAction;
3132
import org.elasticsearch.action.datastreams.lifecycle.ErrorEntry;
3233
import org.elasticsearch.action.datastreams.lifecycle.ExplainDataStreamLifecycleAction;
3334
import org.elasticsearch.action.datastreams.lifecycle.ExplainIndexDataStreamLifecycle;
3435
import org.elasticsearch.action.datastreams.lifecycle.PutDataStreamLifecycleAction;
35-
import org.elasticsearch.action.downsample.DownsampleConfig;
3636
import org.elasticsearch.action.index.IndexRequest;
3737
import org.elasticsearch.action.support.broadcast.BroadcastResponse;
3838
import org.elasticsearch.cluster.coordination.StableMasterHealthIndicatorService;
3939
import org.elasticsearch.cluster.metadata.ComposableIndexTemplate;
4040
import org.elasticsearch.cluster.metadata.DataStream;
4141
import org.elasticsearch.cluster.metadata.DataStreamAction;
42+
import org.elasticsearch.cluster.metadata.DataStreamFailureStore;
4243
import org.elasticsearch.cluster.metadata.DataStreamLifecycle;
43-
import org.elasticsearch.cluster.metadata.DataStreamTestHelper;
44+
import org.elasticsearch.cluster.metadata.DataStreamOptions;
4445
import org.elasticsearch.cluster.metadata.IndexMetadata;
4546
import org.elasticsearch.cluster.metadata.Metadata;
4647
import org.elasticsearch.cluster.metadata.Template;
@@ -71,7 +72,6 @@
7172
import org.elasticsearch.plugins.Plugin;
7273
import org.elasticsearch.plugins.SystemIndexPlugin;
7374
import org.elasticsearch.rest.RestStatus;
74-
import org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramInterval;
7575
import org.elasticsearch.test.ESIntegTestCase;
7676
import org.elasticsearch.test.transport.MockTransportService;
7777
import org.elasticsearch.xcontent.ToXContent;
@@ -893,17 +893,8 @@ public void testReenableDataStreamLifecycle() throws Exception {
893893
}
894894

895895
public void testLifecycleAppliedToFailureStore() throws Exception {
896-
// We configure a lifecycle with downsampling to ensure it doesn't fail
897-
DataStreamLifecycle.Template lifecycle = DataStreamLifecycle.dataLifecycleBuilder()
896+
DataStreamLifecycle.Template lifecycle = DataStreamLifecycle.failuresLifecycleBuilder()
898897
.dataRetention(TimeValue.timeValueSeconds(20))
899-
.downsampling(
900-
List.of(
901-
new DataStreamLifecycle.DownsamplingRound(
902-
TimeValue.timeValueMillis(10),
903-
new DownsampleConfig(new DateHistogramInterval("10m"))
904-
)
905-
)
906-
)
907898
.buildTemplate();
908899

909900
putComposableIndexTemplate("id1", """
@@ -917,7 +908,7 @@ public void testLifecycleAppliedToFailureStore() throws Exception {
917908
"type": "boolean"
918909
}
919910
}
920-
}""", List.of("metrics-fs*"), Settings.builder().put("index.number_of_replicas", 0).build(), null, lifecycle, true);
911+
}""", List.of("metrics-fs*"), Settings.builder().put("index.number_of_replicas", 0).build(), null, null, lifecycle, true);
921912

922913
String dataStreamName = "metrics-fs";
923914
CreateDataStreamAction.Request createDataStreamRequest = new CreateDataStreamAction.Request(
@@ -958,7 +949,7 @@ public void testLifecycleAppliedToFailureStore() throws Exception {
958949
);
959950
});
960951

961-
updateLifecycle(dataStreamName, TimeValue.timeValueSeconds(1));
952+
updateFailureStoreConfiguration(dataStreamName, true, TimeValue.timeValueSeconds(1));
962953

963954
// And finally apply retention
964955
assertBusy(() -> {
@@ -1027,7 +1018,20 @@ static void putComposableIndexTemplate(
10271018
List<String> patterns,
10281019
@Nullable Settings settings,
10291020
@Nullable Map<String, Object> metadata,
1030-
@Nullable DataStreamLifecycle.Template lifecycle,
1021+
@Nullable DataStreamLifecycle.Template dataLifecycle,
1022+
boolean withFailureStore
1023+
) throws IOException {
1024+
putComposableIndexTemplate(id, mappings, patterns, settings, metadata, dataLifecycle, null, withFailureStore);
1025+
}
1026+
1027+
static void putComposableIndexTemplate(
1028+
String id,
1029+
@Nullable String mappings,
1030+
List<String> patterns,
1031+
@Nullable Settings settings,
1032+
@Nullable Map<String, Object> metadata,
1033+
@Nullable DataStreamLifecycle.Template dataLifecycle,
1034+
@Nullable DataStreamLifecycle.Template failuresLifecycle,
10311035
boolean withFailureStore
10321036
) throws IOException {
10331037
TransportPutComposableIndexTemplateAction.Request request = new TransportPutComposableIndexTemplateAction.Request(id);
@@ -1038,8 +1042,10 @@ static void putComposableIndexTemplate(
10381042
Template.builder()
10391043
.settings(settings)
10401044
.mappings(mappings == null ? null : CompressedXContent.fromJSON(mappings))
1041-
.lifecycle(lifecycle)
1042-
.dataStreamOptions(DataStreamTestHelper.createDataStreamOptionsTemplate(withFailureStore))
1045+
.lifecycle(dataLifecycle)
1046+
.dataStreamOptions(
1047+
new DataStreamOptions.Template(new DataStreamFailureStore.Template(withFailureStore, failuresLifecycle))
1048+
)
10431049
)
10441050
.metadata(metadata)
10451051
.dataStreamTemplate(new ComposableIndexTemplate.DataStreamTemplate())
@@ -1058,6 +1064,16 @@ static void updateLifecycle(String dataStreamName, TimeValue dataRetention) {
10581064
assertAcked(client().execute(PutDataStreamLifecycleAction.INSTANCE, putDataLifecycleRequest));
10591065
}
10601066

1067+
static void updateFailureStoreConfiguration(String dataStreamName, boolean enabled, TimeValue retention) {
1068+
PutDataStreamOptionsAction.Request putDataOptionsRequest = new PutDataStreamOptionsAction.Request(
1069+
TEST_REQUEST_TIMEOUT,
1070+
TEST_REQUEST_TIMEOUT,
1071+
new String[] { dataStreamName },
1072+
new DataStreamFailureStore(enabled, DataStreamLifecycle.failuresLifecycleBuilder().dataRetention(retention).build())
1073+
);
1074+
assertAcked(client().execute(PutDataStreamOptionsAction.INSTANCE, putDataOptionsRequest));
1075+
}
1076+
10611077
/*
10621078
* This test plugin adds `.system-test` as a known system data stream. The data stream is not created by this plugin. But if it is
10631079
* created, it will be a system data stream.

0 commit comments

Comments
 (0)