diff --git a/docs/reference/data-analysis/aggregations/pipeline.md b/docs/reference/data-analysis/aggregations/pipeline.md index a6c16b504e5a2..ecdf9b25de5b1 100644 --- a/docs/reference/data-analysis/aggregations/pipeline.md +++ b/docs/reference/data-analysis/aggregations/pipeline.md @@ -102,6 +102,7 @@ POST /_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this max_bucket aggregation that we want the maximum value of the `sales` aggregation in the `sales_per_month` date histogram. @@ -146,6 +147,7 @@ POST /_search } } ``` +% TEST[setup:sales] 1. `buckets_path` selects the hats and bags buckets (via `['hat']`/`['bag']``) to use in the script specifically, instead of fetching all the buckets from `sale_type` aggregation @@ -214,6 +216,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. By using `_bucket_count` instead of a metric name, we can filter out `histo` buckets where they contain no buckets for the `categories` aggregation @@ -226,6 +229,7 @@ An alternate syntax is supported to cope with aggregations or metrics which have ```js "buckets_path": "my_percentile[99.9]" ``` +% NOTCONSOLE ## Dealing with gaps in the data [gap-policy] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-adjacency-matrix-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-adjacency-matrix-aggregation.md index eb1bc60441714..70868682a5e0f 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-adjacency-matrix-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-adjacency-matrix-aggregation.md @@ -87,6 +87,9 @@ The response contains buckets with document counts for each filter and combinati } } ``` +% TESTRESPONSE[s/"took": 9/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] ## Parameters [adjacency-matrix-agg-params] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-autodatehistogram-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-autodatehistogram-aggregation.md index 67ffd7b0d6291..9a42b1650fb07 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-autodatehistogram-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-autodatehistogram-aggregation.md @@ -28,6 +28,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] ## Keys [_keys] @@ -54,6 +55,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. Supports expressive date [format pattern](/reference/data-analysis/aggregations/search-aggregations-bucket-daterange-aggregation.md#date-format-pattern) @@ -87,6 +89,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Intervals [_intervals] @@ -182,6 +185,7 @@ UTC is used if no time zone is specified, three 1-hour buckets are returned star } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] If a `time_zone` of `-01:00` is specified, then midnight starts at one hour before midnight UTC: @@ -199,6 +203,7 @@ GET my-index-000001/_search?size=0 } } ``` +% TEST[continued] Now three 1-hour buckets are still returned but the first bucket starts at 11:00pm on 30 September 2015 since that is the local time for the bucket in the specified time zone. @@ -229,6 +234,7 @@ Now three 1-hour buckets are still returned but the first bucket starts at 11:00 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] 1. The `key_as_string` value represents midnight on each day in the specified time zone. @@ -268,6 +274,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] ## Missing value [_missing_value] @@ -290,6 +297,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. Documents without a value in the `publish_date` field will fall into the same bucket as documents that have the value `2000-01-01`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-categorize-text-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-categorize-text-aggregation.md index 4f785a342da34..af807c86f906a 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-categorize-text-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-categorize-text-aggregation.md @@ -108,6 +108,7 @@ POST log-messages/_search?filter_path=aggregations } } ``` +% TEST[setup:categorize_text] Response: @@ -161,6 +162,7 @@ POST log-messages/_search?filter_path=aggregations } } ``` +% TEST[setup:categorize_text] 1. The filters to apply to the analyzed tokens. It filters out tokens like `bar_123`. @@ -218,6 +220,7 @@ POST log-messages/_search?filter_path=aggregations } } ``` +% TEST[setup:categorize_text] 1. The filters to apply to the analyzed tokens. It filters out tokens like `bar_123`. 2. Require 11% of token weight to match before adding a message to an existing category rather than creating a new one. @@ -280,6 +283,7 @@ POST log-messages/_search?filter_path=aggregations } } ``` +% TEST[setup:categorize_text] ```console-result { diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-children-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-children-aggregation.md index 9d860fa8873e2..1adcb92cdd4b6 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-children-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-children-aggregation.md @@ -52,6 +52,7 @@ PUT child_example/_doc/1 ] } ``` +% TEST[continued] Examples of `answer` documents: @@ -86,6 +87,7 @@ PUT child_example/_doc/3?routing=1&refresh "creation_date": "2009-05-05T13:45:37.030" } ``` +% TEST[continued] The following request can be built that connects the two together: @@ -117,6 +119,7 @@ POST child_example/_search?size=0 } } ``` +% TEST[continued] 1. The `type` points to type / mapping with the name `answer`. @@ -216,6 +219,7 @@ Possible response: } } ``` +% TESTRESPONSE[s/"took": 25/"took": $body.took/] 1. The number of question documents with the tag `file-transfer`, `windows-server-2003`, etc. 2. The number of answer documents that are related to question documents with the tag `file-transfer`, `windows-server-2003`, etc. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md index cd477c51603c7..dd35b97bfee53 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md @@ -26,6 +26,7 @@ For example, consider the following document: "number": [23, 65, 76] } ``` +% NOTCONSOLE Using `keyword` and `number` as source fields for the aggregation results in the following composite buckets: @@ -37,6 +38,7 @@ Using `keyword` and `number` as source fields for the aggregation results in the { "keyword": "bar", "number": 65 } { "keyword": "bar", "number": 76 } ``` +% NOTCONSOLE ## Value sources [_value_sources] @@ -310,6 +312,7 @@ Instead of a single bucket starting at midnight, the above request groups the do } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ::::{note} The start `offset` of each bucket is calculated after `time_zone` adjustments have been made. @@ -513,6 +516,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] ... returns: @@ -545,6 +549,7 @@ GET /_search } } ``` +% TESTRESPONSE[s/...//] To get the next set of buckets, resend the same aggregation with the `after` parameter set to the `after_key` value returned in the response. For example, this request uses the `after_key` value provided in the previous response: @@ -709,6 +714,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] ... returns: @@ -767,6 +773,7 @@ GET /_search } } ``` +% TESTRESPONSE[s/...//] ## Pipeline aggregations [search-aggregations-bucket-composite-aggregation-pipeline-aggregations] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-correlation-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-correlation-aggregation.md index 33b71364f174d..80b36349a87d0 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-correlation-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-correlation-aggregation.md @@ -68,6 +68,7 @@ A `bucket_correlation` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE 1. The buckets containing the values to correlate against. 2. The correlation function definition. @@ -126,6 +127,7 @@ POST correlate_latency/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:correlate_latency] 1. The term buckets containing a range aggregation and the bucket correlation aggregation. Both are utilized to calculate the correlation of the term values with the latency. 2. The range aggregation on the latency field. The ranges were created referencing the percentiles of the latency field. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-count-ks-test-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-count-ks-test-aggregation.md index 92f3e60dd10ae..c81e54fd2b209 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-count-ks-test-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-count-ks-test-aggregation.md @@ -37,6 +37,7 @@ A `bucket_count_ks_test` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE 1. The buckets containing the values to test against. 2. The alternatives to calculate. @@ -89,6 +90,7 @@ POST correlate_latency/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:correlate_latency] 1. The term buckets containing a range aggregation and the bucket correlation aggregation. Both are utilized to calculate the correlation of the term values with the latency. 2. The range aggregation on the latency field. The ranges were created referencing the percentiles of the latency field. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md index 590502c5ce3a0..4918aef7a0977 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md @@ -70,6 +70,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] If you attempt to use multiples of calendar units, the aggregation will fail because only singular calendar units are supported: @@ -88,6 +89,8 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] +% TEST[catch:bad_request] ```js { @@ -103,6 +106,7 @@ POST /sales/_search?size=0 } } ``` +% NOTCONSOLE @@ -150,6 +154,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] But if we try to use a calendar unit that is not supported, such as weeks, we’ll get an exception: @@ -168,6 +173,8 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] +% TEST[catch:bad_request] ```js { @@ -183,6 +190,7 @@ POST /sales/_search?size=0 } } ``` +% NOTCONSOLE @@ -235,6 +243,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. Supports expressive date [format pattern](/reference/data-analysis/aggregations/search-aggregations-bucket-daterange-aggregation.md#date-format-pattern) @@ -267,6 +276,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Time zone [datehistogram-aggregation-time-zone] @@ -335,6 +345,7 @@ If you don’t specify a time zone, UTC is used. This would result in both of th } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] If you specify a `time_zone` of `-01:00`, midnight in that time zone is one hour before midnight UTC: @@ -352,6 +363,7 @@ GET my-index-000001/_search?size=0 } } ``` +% TEST[continued] Now the first document falls into the bucket for 30 September 2015, while the second document falls into the bucket for 1 October 2015: @@ -376,6 +388,7 @@ Now the first document falls into the bucket for 30 September 2015, while the se } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] 1. The `key_as_string` value represents midnight on each day in the specified time zone. @@ -442,6 +455,7 @@ Instead of a single bucket starting at midnight, the above request groups the do } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ::::{note} The start `offset` of each bucket is calculated after `time_zone` adjustments have been made. @@ -470,6 +484,7 @@ $$$datehistogram-aggregation-offset-example-19d$$$ { "key_as_string": "2022-08-20", "key": 1660953600000, "doc_count": 1 } ] ``` +% TESTRESPONSE[skip:no setup made for this example yet] Increasing the offset to `+20d`, each document will appear in a bucket for the previous month, with all bucket keys ending with the same day of the month, as normal. However, further increasing to `+28d`, what used to be a February bucket has now become `"2022-03-01"`. @@ -487,6 +502,7 @@ $$$datehistogram-aggregation-offset-example-28d$$$ { "key_as_string": "2022-07-29", "key": 1659052800000, "doc_count": 1 } ] ``` +% TESTRESPONSE[skip:no setup made for this example yet] If we continue to increase the offset, the 30-day months will also shift into the next month, so that 3 of the 8 buckets have different days than the other five. In fact if we keep going, we will find cases where two documents appear in the same month. Documents that were originally 30 days apart can be shifted into the same 31-day month bucket. @@ -503,6 +519,7 @@ $$$datehistogram-aggregation-offset-example-50d$$$ { "key_as_string": "2022-08-20", "key": 1660953600000, "doc_count": 1 } ] ``` +% TESTRESPONSE[skip:no setup made for this example yet] It is therefore always important when using `offset` with `calendar_interval` bucket sizes to understand the consequences of using offsets larger than the interval size. @@ -534,6 +551,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -563,6 +581,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Scripts [date-histogram-scripts] @@ -596,6 +615,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] ## Parameters [date-histogram-params] @@ -622,6 +642,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. Documents without a value in the `date` field will fall into the same bucket as documents that have the value `2000-01-01`. @@ -654,6 +675,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -678,6 +700,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The response will contain all the buckets having the relative day of the week as key : 1 for Monday, 2 for Tuesday…​ 7 for Sunday. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-daterange-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-daterange-aggregation.md index 5cb2485903f43..b313e905247b1 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-daterange-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-daterange-aggregation.md @@ -30,6 +30,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales s/now-10M\/M/10-2015/] 1. < now minus 10 months, rounded down to the start of the month. 2. >= now minus 10 months, rounded down to the start of the month. @@ -62,6 +63,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ::::{warning} If a format or date value is incomplete, the date range aggregation replaces any missing components with default values. See [Missing date components](/reference/query-languages/query-dsl-range-query.md#missing-date-components). @@ -98,6 +100,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. Documents without a value in the `date` field will be added to the "Older" bucket, as if they had a date value of "1976-11-30". @@ -221,6 +224,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. This date will be converted to `2016-02-01T00:00:00.000+01:00`. 2. `now/d` will be rounded to the beginning of the day in the CET time zone. @@ -251,6 +255,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales s/now-10M\/M/10-2015/] Response: @@ -275,6 +280,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] It is also possible to customize the key for each range: @@ -298,6 +304,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -326,5 +333,6 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-diversified-sampler-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-diversified-sampler-aggregation.md index e8483a65571af..24f4f92274a83 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-diversified-sampler-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-diversified-sampler-aggregation.md @@ -54,6 +54,7 @@ POST /stackoverflow/_search?size=0 } } ``` +% TEST[setup:stackoverflow] Response: @@ -79,11 +80,12 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/2.213/$body.aggregations.my_unbiased_sample.keywords.buckets.0.score/] 1. 151 documents were sampled in total. 2. The results of the significant_terms aggregation are not skewed by any single author’s quirks because we asked for a maximum of one post from any one author in our sample. - ## Scripted example [_scripted_example] In this scenario we might want to diversify on a combination of field values. We can use a [runtime field](docs-content://manage-data/data-store/mapping/runtime-fields.md) to produce a hash of the multiple values in a tags field to ensure we don’t have a sample that consists of the same repeated combinations of tags. @@ -123,6 +125,7 @@ POST /stackoverflow/_search?size=0 } } ``` +% TEST[setup:stackoverflow] Response: @@ -154,6 +157,9 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/2.213/$body.aggregations.my_unbiased_sample.keywords.buckets.0.score/] +% TESTRESPONSE[s/1.34/$body.aggregations.my_unbiased_sample.keywords.buckets.1.score/] ## shard_size [_shard_size] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filter-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filter-aggregation.md index 565f58cdba443..d461d6f68e4c0 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filter-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filter-aggregation.md @@ -27,6 +27,7 @@ POST /sales/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:sales] The previous example calculates the average price of all sales as well as the average price of all T-shirt sales. @@ -61,6 +62,7 @@ POST /sales/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:sales] Instead of this: @@ -79,6 +81,7 @@ POST /sales/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:sales] ## Use the `filters` aggregation for multiple filters [use-filters-agg-for-multiple-filters] @@ -107,6 +110,7 @@ POST /sales/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:sales] Instead of this: @@ -131,5 +135,6 @@ POST /sales/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:sales] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filters-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filters-aggregation.md index 03311dc5d8ac3..95fec2aa2bab9 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filters-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filters-aggregation.md @@ -62,6 +62,9 @@ Response: } } ``` +% TESTRESPONSE[s/"took": 9/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] ## Anonymous filters [anonymous-filters] @@ -85,6 +88,7 @@ GET logs/_search } } ``` +% TEST[continued] The filtered buckets are returned in the same order as provided in the request. The response for this example would be: @@ -108,6 +112,9 @@ The filtered buckets are returned in the same order as provided in the request. } } ``` +% TESTRESPONSE[s/"took": 4/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] ## `Other` Bucket [other-bucket] @@ -148,6 +155,7 @@ GET logs/_search } } ``` +% TEST[continued] The response would be something like the following: @@ -174,6 +182,9 @@ The response would be something like the following: } } ``` +% TESTRESPONSE[s/"took": 3/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] ## Non-keyed Response [non-keyed-response] @@ -217,6 +228,7 @@ POST /sales/_search?size=0&filter_path=aggregations } } ``` +% TEST[setup:sales] Response: diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-frequent-item-sets-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-frequent-item-sets-aggregation.md index 39e37850bc067..38fd9a8a39d87 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-frequent-item-sets-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-frequent-item-sets-aggregation.md @@ -26,6 +26,7 @@ A `frequent_item_sets` aggregation looks like this in isolation: ] } ``` +% NOTCONSOLE | | | | | | --- | --- | --- | --- | @@ -104,12 +105,14 @@ POST /kibana_sample_data_ecommerce/_async_search } } ``` +% TEST[skip:setup kibana sample data] The response of the API call above contains an identifier (`id`) of the async search request. You can use the identifier to retrieve the search results: ```console GET /_async_search/ ``` +% TEST[skip:setup kibana sample data] The API returns a response similar to the following one: @@ -162,6 +165,7 @@ The API returns a response similar to the following one: } } ``` +% TEST[skip:setup kibana sample data] 1. The array of returned item sets. 2. The `key` object contains one item set. In this case, it consists of two values of the `category.keyword` field and one value of the `geoip.city_name`. @@ -199,6 +203,7 @@ POST /kibana_sample_data_ecommerce/_async_search } } ``` +% TEST[skip:setup kibana sample data] The result will only show item sets that created from documents matching the filter, namely purchases in Europe. Using `filter`, the calculated `support` still takes all purchases into acount. That’s different than specifying a query at the top-level, in which case `support` gets calculated only from purchases in Europe. @@ -244,6 +249,7 @@ GET kibana_sample_data_ecommerce/_search } } ``` +% TEST[skip:setup kibana sample data] The API returns a response similar to the following one: @@ -305,6 +311,7 @@ The API returns a response similar to the following one: } } ``` +% TEST[skip:setup kibana sample data] The response shows the categories that customers purchase from most frequently together, the location of the customers who tend to buy items from these categories, and the most frequent price ranges of these purchases. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geodistance-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geodistance-aggregation.md index f34919ec75909..211d09528ea68 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geodistance-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geodistance-aggregation.md @@ -85,6 +85,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] The specified field must be of type `geo_point` (which can only be set explicitly in the mappings). And it can also hold an array of `geo_point` fields, in which case all will be taken into account during aggregation. The origin point can accept all formats supported by the [`geo_point` type](/reference/elasticsearch/mapping-reference/geo-point.md): @@ -113,6 +114,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] 1. The distances will be computed in kilometers @@ -139,6 +141,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] ## Keyed Response [_keyed_response_2] @@ -163,6 +166,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] Response: @@ -191,6 +195,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] It is also possible to customize the key for each range: @@ -213,6 +218,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] Response: @@ -241,5 +247,6 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohashgrid-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohashgrid-aggregation.md index 10b3f689e6a6e..2defd93a0f2a0 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohashgrid-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohashgrid-aggregation.md @@ -89,6 +89,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## High-precision requests [_high_precision_requests] @@ -122,6 +123,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] The geohashes returned by the `geohash_grid` aggregation can be also used for zooming in. To zoom into the first geohash `u17` returned in the previous example, it should be specified as both `top_left` and `bottom_right` corner: @@ -150,6 +152,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] ```console-result { @@ -177,6 +180,7 @@ POST /museums/_search?size=0 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] For "zooming in" on the system that don’t support geohashes, the bucket keys should be translated into bounding boxes using one of available geohash libraries. For example, for javascript the [node-geohash](https://github.com/sunng87/node-geohash) library can be used: @@ -212,6 +216,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] ```console-result { @@ -236,6 +241,7 @@ POST /museums/_search?size=0 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## Cell dimensions at the equator [_cell_dimensions_at_the_equator] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md index 1e53ffc901ec1..73279f1bee535 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md @@ -84,6 +84,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## High-precision requests [geohexgrid-high-precision] @@ -117,6 +118,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] Response: @@ -146,6 +148,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## Requests with additional bounding box filtering [geohexgrid-addtl-bounding-box-filtering] @@ -171,6 +174,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] Response: @@ -197,6 +201,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ### Aggregating `geo_shape` fields [geohexgrid-aggregating-geo-shape] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md index 944f4ab40ef53..a3643c92dfbd9 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md @@ -89,6 +89,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## High-precision requests [geotilegrid-high-precision] @@ -122,6 +123,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] Response: @@ -151,6 +153,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## Requests with additional bounding box filtering [geotilegrid-addtl-bounding-box-filtering] @@ -176,6 +179,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] Response: @@ -202,6 +206,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ### Aggregating `geo_shape` fields [geotilegrid-aggregating-geo-shape] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-global-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-global-aggregation.md index 85ce812ea0502..c9b96582b46d1 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-global-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-global-aggregation.md @@ -35,6 +35,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. The `global` aggregation has an empty body 2. The sub-aggregations that are registered for this `global` aggregation @@ -60,6 +61,7 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] 1. The number of documents that were aggregated (in our case, all documents within the search context) 2. The average price of all products in the index diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-histogram-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-histogram-aggregation.md index 198ab127e783b..60aa15c0571cd 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-histogram-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-histogram-aggregation.md @@ -34,6 +34,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] And the following may be the response: @@ -68,6 +69,7 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Minimum document count [_minimum_document_count] @@ -89,6 +91,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -119,6 +122,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] $$$search-aggregations-bucket-histogram-aggregation-extended-bounds$$$ By default the `histogram` returns all the buckets within the range of the data itself, that is, the documents with the smallest values (on which with histogram) will determine the min bucket (the bucket with the smallest key) and the documents with the highest values will determine the max bucket (the bucket with the highest key). Often, when requesting empty buckets, this causes a confusion, specifically, when the data is also filtered. @@ -155,6 +159,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] When aggregating ranges, buckets are based on the values of the returned documents. This means the response may include buckets outside of a query’s range. For example, if your query looks for values greater than 100, and you have a range covering 50 to 150, and an interval of 50, that document will land in 3 buckets - 50, 100, and 150. In general, it’s best to think of the query and aggregation steps as independent - the query selects a set of documents, and then the aggregation buckets those documents without regard to how they were selected. See [note on bucketing range fields](/reference/data-analysis/aggregations/search-aggregations-bucket-range-field-note.md) for more information and an example. @@ -185,6 +190,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] In this example even though the range specified in the query is up to 500, the histogram will only have 2 buckets starting at 100 and 150. All other buckets will be omitted even if documents that should go to this buckets are present in the results. @@ -221,6 +227,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -255,6 +262,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Missing value [_missing_value_2] @@ -277,6 +285,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. Documents without a value in the `quantity` field will fall into the same bucket as documents that have the value `0`. @@ -367,6 +376,7 @@ The `histogram` aggregation will sum the counts of each interval computed based } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ::::{important} Histogram aggregation is a bucket aggregation, which partitions documents into buckets rather than calculating metrics over fields like metrics aggregations do. Each bucket represents a collection of documents which sub-aggregations can run on. On the other hand, a histogram field is a pre-aggregated field representing multiple values inside a single field: buckets of numerical data and a count of items/documents for each bucket. This mismatch between the histogram aggregations expected input (expecting raw documents) and the histogram field (that provides summary information) limits the outcome of the aggregation to only the doc counts for each bucket. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-ipprefix-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-ipprefix-aggregation.md index f15f177f6d5e9..40f0736ea0a2c 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-ipprefix-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-ipprefix-aggregation.md @@ -44,6 +44,7 @@ POST /network-traffic/_bulk?refresh {"index":{"_id":8}} {"ipv4":"192.168.3.107","ipv6":"2001:db8:a4f8:114f:6001:0:12:7307"} ``` +% TESTSETUP The following aggregation groups documents into buckets. Each bucket identifies a different sub-network. The sub-network is calculated by applying a netmask with prefix length of `24` to each IP address in the `ipv4` field: @@ -63,6 +64,7 @@ GET /network-traffic/_search } } ``` +% TEST Response: @@ -99,6 +101,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] To aggregate IPv6 addresses, set `is_ipv6` to `true`. @@ -119,6 +122,7 @@ GET /network-traffic/_search } } ``` +% TEST If `is_ipv6` is `true`, the response doesn’t include a `netmask` for each bucket. @@ -152,6 +156,7 @@ If `is_ipv6` is `true`, the response doesn’t include a `netmask` for each buck } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Parameters [ip-prefix-agg-params] @@ -216,6 +221,7 @@ GET /network-traffic/_search } } ``` +% TEST Response: @@ -249,6 +255,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Append the prefix length to the IP address key [ipprefix-agg-append-prefix-length] @@ -274,6 +281,7 @@ GET /network-traffic/_search } } ``` +% TEST Response: @@ -310,6 +318,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Minimum document count [ipprefix-agg-min-doc-count] @@ -333,6 +342,7 @@ GET /network-traffic/_search } } ``` +% TEST Response: @@ -362,5 +372,6 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-iprange-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-iprange-aggregation.md index b8bbd060c117a..b3ff34fc37bf5 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-iprange-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-iprange-aggregation.md @@ -30,6 +30,7 @@ GET /ip_addresses/_search } } ``` +% TEST[setup:iprange] Response: @@ -55,6 +56,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] IP ranges can also be defined as CIDR masks: @@ -77,6 +79,7 @@ GET /ip_addresses/_search } } ``` +% TEST[setup:iprange] Response: @@ -104,6 +107,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Keyed Response [_keyed_response_3] @@ -129,6 +133,7 @@ GET /ip_addresses/_search } } ``` +% TEST[setup:iprange] Response: @@ -152,6 +157,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] It is also possible to customize the key for each range: @@ -175,6 +181,7 @@ GET /ip_addresses/_search } } ``` +% TEST[setup:iprange] Response: @@ -198,5 +205,6 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-missing-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-missing-aggregation.md index b2e02b12f1ef7..49e30429e4e0a 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-missing-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-missing-aggregation.md @@ -23,6 +23,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] In the above example, we get the total number of products that do not have a price. @@ -38,4 +39,5 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-multi-terms-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-multi-terms-aggregation.md index 3b0e825186a66..5a9fea46a5e24 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-multi-terms-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-multi-terms-aggregation.md @@ -31,6 +31,7 @@ GET /products/_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] 1. `multi_terms` aggregation can work with the same field types as a [`terms aggregation`](/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md#search-aggregations-bucket-terms-aggregation-order) and supports most of the terms aggregation parameters. @@ -82,6 +83,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] 1. an upper bound of the error on the document counts for each term, see < 2. when there are lots of unique terms, Elasticsearch only returns the top terms; this number is the sum of the document counts for all buckets that are not part of the response @@ -148,6 +150,7 @@ GET /products/_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -188,6 +191,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] ## Missing value [_missing_value_3] @@ -216,6 +220,7 @@ GET /products/_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -272,6 +277,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] 1. Documents without a value in the `product` field will fall into the same bucket as documents that have the value `Product Z`. @@ -320,6 +326,7 @@ GET /products/_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] ```console-result { @@ -378,5 +385,6 @@ GET /products/_search } } ``` +% TESTRESPONSE[s/...//] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-nested-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-nested-aggregation.md index c8407fd53e2e3..b160bd8f58ed8 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-nested-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-nested-aggregation.md @@ -55,6 +55,7 @@ PUT /products/_doc/0?refresh ] } ``` +% TEST[continued] 1. We are using a dynamic mapping for the `name` attribute. @@ -85,6 +86,8 @@ GET /products/_search?size=0 } } ``` +% TEST[s/size=0/size=0&filter_path=aggregations/] +% TEST[continued] As you can see above, the nested aggregation requires the `path` of the nested documents within the top level documents. Then one can define any type of aggregation over these nested documents. @@ -103,6 +106,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] You can use a [`filter`](/reference/data-analysis/aggregations/search-aggregations-bucket-filter-aggregation.md) sub-aggregation to return results for a specific reseller. @@ -145,6 +149,8 @@ GET /products/_search?size=0 } } ``` +% TEST[s/size=0/size=0&filter_path=aggregations/] +% TEST[continued] The search returns: @@ -164,4 +170,5 @@ The search returns: } } ``` +% TESTRESPONSE[s/...//] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-parent-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-parent-aggregation.md index 2186ef734b9c6..9a37668745a16 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-parent-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-parent-aggregation.md @@ -52,6 +52,7 @@ PUT parent_example/_doc/1 ] } ``` +% TEST[continued] Examples of `answer` documents: @@ -86,6 +87,7 @@ PUT parent_example/_doc/3?routing=1&refresh "creation_date": "2009-05-05T13:45:37.030" } ``` +% TEST[continued] The following request can be built that connects the two together: @@ -117,6 +119,7 @@ POST parent_example/_search?size=0 } } ``` +% TEST[continued] 1. The `type` points to type / mapping with the name `answer`. @@ -203,6 +206,7 @@ Possible response: } } ``` +% TESTRESPONSE[s/"took": 9/"took": $body.took/] 1. The number of answer documents with the tag `Sam`, `Troll`, etc. 2. The number of question documents that are related to answer documents with the tag `Sam`, `Troll`, etc. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-aggregation.md index c28747d411b89..ab0073b9faa6e 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-aggregation.md @@ -30,6 +30,8 @@ GET sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -60,6 +62,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] ## Keyed Response [_keyed_response_4] @@ -85,6 +88,8 @@ GET sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -112,6 +117,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] It is also possible to customize the key for each range: @@ -135,6 +141,8 @@ GET sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -162,6 +170,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] ## Script [_script] @@ -200,6 +209,8 @@ GET sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] ## Sub Aggregations [_sub_aggregations_2] @@ -230,6 +241,8 @@ GET sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -281,6 +294,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] ## Histogram fields [search-aggregations-bucket-range-aggregation-histogram-fields] @@ -380,6 +394,7 @@ The `range` aggregation will sum the counts of each range computed based on the } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ::::{important} Range aggregation is a bucket aggregation, which partitions documents into buckets rather than calculating metrics over fields like metrics aggregations do. Each bucket represents a collection of documents which sub-aggregations can run on. On the other hand, a histogram field is a pre-aggregated field representing multiple values inside a single field: buckets of numerical data and a count of items/documents for each bucket. This mismatch between the range aggregations expected input (expecting raw documents) and the histogram field (that provides summary information) limits the outcome of the aggregation to only the doc counts for each bucket. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-field-note.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-field-note.md index 9541b77c55ed7..6375987657443 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-field-note.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-field-note.md @@ -40,6 +40,7 @@ PUT range_index/_doc/1?refresh } } ``` +% TESTSETUP The range is wider than the interval in the following aggregation, and thus the document will land in multiple buckets. @@ -84,6 +85,7 @@ Since the interval is `5` (and the offset is `0` by default), we expect buckets } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] A document cannot exist partially in a bucket; For example, the above document cannot count as one-third in each of the above three buckets. In this example, since the document’s range landed in multiple buckets, the full value of that document would also be counted in any sub-aggregations for each bucket as well. @@ -170,6 +172,7 @@ Even though the query only considers days in November, the aggregation generates } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] Depending on the use case, a `CONTAINS` query could limit the documents to only those that fall entirely in the queried range. In this example, the one document would not be included and the aggregation would be empty. Filtering the buckets after the aggregation is also an option, for use cases where the document should be counted but the out of bounds data can be safely ignored. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-rare-terms-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-rare-terms-aggregation.md index 480747123dcc2..f96cc3213f736 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-rare-terms-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-rare-terms-aggregation.md @@ -21,6 +21,7 @@ A `rare_terms` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE | | | | | | --- | --- | --- | --- | @@ -48,6 +49,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -66,6 +68,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] In this example, the only bucket that we see is the "swing" bucket, because it is the only term that appears in one document. If we increase the `max_doc_count` to `2`, we’ll see some more buckets: @@ -84,6 +87,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] This now shows the "jazz" term which has a `doc_count` of 2": @@ -106,6 +110,7 @@ This now shows the "jazz" term which has a `doc_count` of 2": } } ``` +% TESTRESPONSE[s/...//] ## Maximum document count [search-aggregations-bucket-rare-terms-aggregation-max-doc-count] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md index 5282f0d491723..98855b6517e04 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md @@ -74,6 +74,8 @@ GET /issues/_search } } ``` +% TEST[continued] +% TEST[s/_search/_search?filter_path=aggregations/] As you can see above, the `reverse_nested` aggregation is put in to a `nested` aggregation as this is the only place in the dsl where the `reverse_nested` aggregation can be used. Its sole purpose is to join back to a parent doc higher up in the nested structure. @@ -116,4 +118,5 @@ Possible response snippet: } } ``` +% TESTRESPONSE[s/...//] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-sampler-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-sampler-aggregation.md index 4fa9ef8b27916..5572c8651d65f 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-sampler-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-sampler-aggregation.md @@ -43,6 +43,7 @@ POST /stackoverflow/_search?size=0 } } ``` +% TEST[setup:stackoverflow] Response: @@ -74,6 +75,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] 1. 200 documents were sampled in total. The cost of performing the nested significant_terms aggregation was therefore limited rather than unbounded. @@ -101,6 +103,7 @@ POST /stackoverflow/_search?size=0 } } ``` +% TEST[setup:stackoverflow] Response: @@ -135,6 +138,9 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/0.02777/$body.aggregations.low_quality_keywords.buckets.0.score/] +% TESTRESPONSE[s/0.0069/$body.aggregations.low_quality_keywords.buckets.2.score/] ## shard_size [_shard_size_2] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significantterms-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significantterms-aggregation.md index 6a6a817d94a16..c9a15bbb8639c 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significantterms-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significantterms-aggregation.md @@ -38,6 +38,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -61,6 +62,8 @@ Response: } } ``` +% TESTRESPONSE[s/...//] +% TESTRESPONSE[s/: (0.)?[0-9]+/: $body.$_path/] When querying an index of all crimes from all police forces, what these results show is that the British Transport Police force stand out as a force dealing with a disproportionately large number of bicycle thefts. Ordinarily, bicycle thefts represent only 1% of crimes (66799/5064554) but for the British Transport Police, who handle crime on railways and stations, 7% of crimes (3640/47347) is a bike theft. This is a significant seven-fold increase in frequency and so this anomaly was highlighted as the top crime type. @@ -92,6 +95,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -142,6 +146,9 @@ Response: } } ``` +% TESTRESPONSE[s/...//] +% TESTRESPONSE[s/: (0.)?[0-9]+/: $body.$_path/] +% TESTRESPONSE[s/: "[^"]*"/: $body.$_path/] Now we have anomaly detection for each of the police forces using a single request. @@ -256,6 +263,7 @@ The JLH score can be used as a significance score by adding the parameter "jlh": { } ``` +% NOTCONSOLE The scores are derived from the doc frequencies in *foreground* and *background* sets. The *absolute* change in popularity (foregroundPercent - backgroundPercent) would favor common terms whereas the *relative* change in popularity (foregroundPercent/ backgroundPercent) would favor rare terms. Rare vs common is essentially a precision vs recall balance and so the absolute and relative changes are multiplied to provide a sweet spot between precision and recall. @@ -269,6 +277,7 @@ Mutual information as described in "Information Retrieval", Manning et al., Chap "include_negatives": true } ``` +% NOTCONSOLE Mutual information does not differentiate between terms that are descriptive for the subset or for documents outside the subset. The significant terms therefore can contain terms that appear more or less frequent in the subset than outside the subset. To filter out the terms that appear less often in the subset than in documents outside the subset, `include_negatives` can be set to `false`. @@ -277,6 +286,7 @@ Per default, the assumption is that the documents in the bucket are also contain ```js "background_is_superset": false ``` +% NOTCONSOLE ### Chi square [_chi_square] @@ -287,6 +297,7 @@ Chi square as described in "Information Retrieval", Manning et al., Chapter 13.5 "chi_square": { } ``` +% NOTCONSOLE Chi square behaves like mutual information and can be configured with the same parameters `include_negatives` and `background_is_superset`. @@ -299,6 +310,7 @@ Google normalized distance as described in ["The Google Similarity Distance", Ci "gnd": { } ``` +% NOTCONSOLE `gnd` also accepts the `background_is_superset` parameter. @@ -382,6 +394,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?size=0/] @@ -397,6 +410,7 @@ It would be hard for a seasoned boxer to win a championship if the prize was awa "percentage": { } ``` +% NOTCONSOLE ### Which one is best? [_which_one_is_best] @@ -420,6 +434,7 @@ Customized scores can be implemented via a script: } } ``` +% NOTCONSOLE Scripts can be inline (as in above example), indexed or stored on disk. For details on the options, see [script documentation](docs-content://explore-analyze/scripting.md). diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significanttext-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significanttext-aggregation.md index 845e68d95501e..36d918b6bf019 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significanttext-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significanttext-aggregation.md @@ -51,6 +51,7 @@ GET news/_search } } ``` +% TEST[setup:news] Response: @@ -79,6 +80,7 @@ Response: } } ``` +% TESTRESPONSE[skip:historically skipped] The results show that "h5n1" is one of several terms strongly associated with bird flu. It only occurs 5 times in our index as a whole (see the `bg_count`) and yet 4 of these were lucky enough to appear in our 100 document sample of "bird flu" results. That suggests a significant word and one which the user can potentially add to their search. @@ -124,6 +126,7 @@ First let’s look at an unfiltered real-world example using the [Signal media d } ``` +% NOTCONSOLE The uncleansed documents have thrown up some odd-looking terms that are, on the face of it, statistically correlated with appearances of our search term "elasticsearch" e.g. "pozmantier". We can drill down into examples of these documents to see why pozmantier is connected using this query: @@ -148,6 +151,7 @@ GET news/_search } } ``` +% TEST[setup:news] The results show a series of very similar news articles about a judging panel for a number of tech projects: @@ -184,6 +188,7 @@ The results show a series of very similar news articles about a judging panel fo }, ... ``` +% NOTCONSOLE Mike Pozmantier was one of many judges on a panel and elasticsearch was used in one of many projects being judged. @@ -218,6 +223,7 @@ GET news/_search } } ``` +% TEST[setup:news] The results from analysing our deduplicated text are obviously of higher quality to anyone familiar with the elastic stack: @@ -254,6 +260,7 @@ The results from analysing our deduplicated text are obviously of higher quality } } ``` +% NOTCONSOLE Mr Pozmantier and other one-off associations with elasticsearch no longer appear in the aggregation results as a consequence of copy-and-paste operations or other forms of mechanical repetition. @@ -366,6 +373,7 @@ GET news/_search } } ``` +% TEST[setup:news] The above filter would help focus in on terms that were peculiar to the city of Madrid rather than revealing terms like "Spanish" that are unusual in the full index’s worldwide context but commonplace in the subset of documents containing the word "Spain". @@ -399,6 +407,7 @@ GET news/_search } } ``` +% TEST[setup:news] ### Filtering Values [_filtering_values_3] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md index c2112aafca0c2..e85290461e4c1 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md @@ -23,6 +23,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] Response: @@ -51,6 +52,7 @@ Response: } } ``` +% TESTRESPONSE[s/...//] 1. an upper bound of the error on the document counts for each term, see [below](#terms-agg-doc-count-error) 2. when there are lots of unique terms, Elasticsearch only returns the top terms; this number is the sum of the document counts for all buckets that are not part of the response @@ -122,6 +124,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] These errors can only be calculated in this way when the terms are ordered by descending document count. When the aggregation is ordered by the terms values themselves (either ascending or descending) there is no error in the document count since if a shard does not return a particular term which appears in the results from another shard, it must not have that term in its index. When the aggregation is either sorted by a sub aggregation or in order of ascending document count, the error in the document counts cannot be determined and is given a value of -1 to indicate this. @@ -419,6 +422,7 @@ Which will look like: ... } ``` +% TESTRESPONSE[s/.../"took": "$body.took", "timed_out": false, "_shards": "$body._shards", "hits": "$body.hits"/] This is a little slower because the runtime field has to access two fields instead of one and because there are some optimizations that work on non-runtime `keyword` fields that we have to give up for for runtime `keyword` fields. If you need the speed, you can index the `normalized_genre` field. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-time-series-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-time-series-aggregation.md index 24bce3368a67f..176a995288799 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-time-series-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-time-series-aggregation.md @@ -33,6 +33,7 @@ PUT /my-time-series-index-0/_bulk { "index": {}} { "key": "b", "val": 3, "@timestamp": "2022-01-02T00:00:00Z" } ``` +% NOTCONSOLE To perform a time series aggregation, specify "time_series" as the aggregation type. When the boolean "keyed" is true, each bucket is given a unique key. @@ -48,6 +49,7 @@ GET /_search } } ``` +% NOTCONSOLE This will return all results in the time series, however a more typical query will use sub aggregations to reduce the date returned to something more relevant. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-variablewidthhistogram-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-variablewidthhistogram-aggregation.md index ba44b9fff573a..d06129ae31c07 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-bucket-variablewidthhistogram-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-bucket-variablewidthhistogram-aggregation.md @@ -29,6 +29,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -55,6 +56,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ::::{important} This aggregation cannot currently be nested under any aggregation that collects from more than a single bucket. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-change-point-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-change-point-aggregation.md index 56960c32b5565..f8d04315548f4 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-change-point-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-change-point-aggregation.md @@ -36,6 +36,7 @@ A `change_point` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE 1. The buckets containing the values to test against. @@ -98,6 +99,7 @@ GET kibana_sample_data_logs/_search } } ``` +% NOTCONSOLE 1. A date histogram aggregation that creates buckets with one day long interval. 2. A sibling aggregation of the `date` aggregation that calculates the average value of the `bytes` field within every bucket. @@ -124,6 +126,7 @@ The request returns a response that is similar to the following: } } ``` +% NOTCONSOLE 1. The bucket key that is the change point. 2. The number of documents in that bucket. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-matrix-stats-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-matrix-stats-aggregation.md index a3939ae956d4f..5bc75c890b397 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-matrix-stats-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-matrix-stats-aggregation.md @@ -51,6 +51,7 @@ GET /_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations/] The aggregation type is `matrix_stats` and the `fields` setting defines the set of fields (as an array) for computing the statistics. The above request returns the following response: @@ -95,6 +96,8 @@ The aggregation type is `matrix_stats` and the `fields` setting defines the set } } ``` +% TESTRESPONSE[s/...//] +% TESTRESPONSE[s/: (-)?[0-9.E]+/: $body.$_path/] The `doc_count` field indicates the number of documents involved in the computation of the statistics. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md index 2c0c0a0a58dbb..c628e6aae3c79 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md @@ -19,6 +19,7 @@ POST /exams/_search?size=0 } } ``` +% TEST[setup:exams] The above aggregation computes the average grade over all documents. The aggregation type is `avg` and the `field` setting defines the numeric field of the documents the average will be computed on. The above will return the following: @@ -32,6 +33,7 @@ The above aggregation computes the average grade over all documents. The aggrega } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The name of the aggregation (`avg_grade` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -62,6 +64,8 @@ POST /exams/_search?size=0 } } ``` +% TEST[setup:exams] +% TEST[s/size=0/size=0&filter_path=aggregations/] ## Missing value [_missing_value_6] @@ -81,6 +85,7 @@ POST /exams/_search?size=0 } } ``` +% TEST[setup:exams] 1. Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`. @@ -133,5 +138,6 @@ For each histogram field the `avg` aggregation adds each number in the `values` } } ``` +% TESTRESPONSE[skip:test not setup] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-boxplot-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-boxplot-aggregation.md index d8997339a9541..d65e91702adc3 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-boxplot-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-boxplot-aggregation.md @@ -22,6 +22,7 @@ A `boxplot` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE Let’s look at a boxplot representing load time: @@ -38,6 +39,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. The field `load_time` must be a numeric field @@ -61,6 +63,7 @@ The response will look like this: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] In this case, the lower and upper whisker values are equal to the min and max. In general, these values are the 1.5 * IQR range, which is to say the nearest values to `q1 - (1.5 * IQR)` and `q3 + (1.5 * IQR)`. Since this is an approximation, the given values may not actually be observed values from the data, but should be within a reasonable error bound of them. While the Boxplot aggregation doesn’t directly return outlier points, you can check if `lower > min` or `upper < max` to see if outliers exist on either side, and then query for them directly. @@ -91,6 +94,9 @@ GET latency/_search } } ``` +% TEST[setup:latency] +% TEST[s/_search/_search?filter_path=aggregations/] +% TEST[s/"timeUnit": 1000/"timeUnit": 10/] ## Boxplot values are (usually) approximate [search-aggregations-metrics-boxplot-aggregation-approximation] @@ -122,6 +128,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Compression controls memory usage and approximation error @@ -151,6 +158,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Optimize TDigest for accuracy, at the expense of performance @@ -176,6 +184,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md index 0c7711824a357..26dca491e340d 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md @@ -23,6 +23,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -36,6 +37,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Precision control [_precision_control] @@ -54,6 +56,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. The `precision_threshold` options allows to trade memory for accuracy, and defines a unique count below which counts are expected to be close to accurate. Above this value, counts might become a bit more fuzzy. The maximum supported value is 40000, thresholds above this number will have the same effect as a threshold of 40000. The default value is `3000`. @@ -112,6 +115,8 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] +% TEST[s/size=0/size=0&filter_path=aggregations/] ## Missing value [_missing_value_8] @@ -131,6 +136,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] 1. Documents without a value in the `tag` field will fall into the same bucket as documents that have the value `N/A`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-bounds-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-bounds-aggregation.md index 832014b936d56..7c7167f240025 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-bounds-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-bounds-aggregation.md @@ -83,6 +83,7 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## Cartesian Bounds Aggregation on `shape` fields [cartesian-bounds-aggregation-shape] @@ -120,6 +121,7 @@ POST /places/_search?size=0 } } ``` +% TEST ```console-result { @@ -140,4 +142,5 @@ POST /places/_search?size=0 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-centroid-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-centroid-aggregation.md index cc9b820897496..efddecc8115d7 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-centroid-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cartesian-centroid-aggregation.md @@ -70,6 +70,7 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] The `cartesian_centroid` aggregation is more interesting when combined as a sub-aggregation to other bucket aggregations. @@ -90,6 +91,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] The above example uses `cartesian_centroid` as a sub-aggregation to a [terms](/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) bucket aggregation for finding the central location for museums in each city. @@ -141,6 +143,7 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## Cartesian Centroid Aggregation on `shape` fields [cartesian-centroid-aggregation-geo-shape] @@ -185,6 +188,7 @@ POST /places/_search?size=0 } } ``` +% TEST ```console-result { @@ -200,4 +204,5 @@ POST /places/_search?size=0 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-extendedstats-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-extendedstats-aggregation.md index ff573b08dd54a..8f919521e9157 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-extendedstats-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-extendedstats-aggregation.md @@ -22,6 +22,7 @@ GET /exams/_search } } ``` +% TEST[setup:exams] The above aggregation computes the grades statistics over all documents. The aggregation type is `extended_stats` and the `field` setting defines the numeric field of the documents the stats will be computed on. The above will return the following: @@ -57,6 +58,7 @@ The `std_deviation` and `variance` are calculated as population metrics so they } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The name of the aggregation (`grades_stats` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -78,6 +80,7 @@ GET /exams/_search } } ``` +% TEST[setup:exams] 1. `sigma` controls how many standard deviations +/- from the mean should be displayed @@ -121,6 +124,8 @@ GET /exams/_search } } ``` +% TEST[setup:exams] +% TEST[s/_search/_search?filter_path=aggregations/] ## Missing value [_missing_value_9] @@ -141,6 +146,7 @@ GET /exams/_search } } ``` +% TEST[setup:exams] 1. Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `0`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geo-line.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geo-line.md index ca57725ff13de..6b1d9b6f0b0ce 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geo-line.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geo-line.md @@ -72,6 +72,7 @@ Which returns: } } ``` +% TESTRESPONSE The resulting [GeoJSON Feature](https://tools.ietf.org/html/rfc7946#section-3.2) contains both a `LineString` geometry for the path generated by the aggregation, as well as a map of `properties`. The property `complete` informs of whether all documents matched were used to generate the geometry. The [`size` option](#search-aggregations-metrics-geo-line-size) can be used to limit the number of documents included in the aggregation, leading to results with `complete: false`. Exactly which documents are dropped from results [depends on whether the aggregation is based on `time_series` or not](#search-aggregations-metrics-geo-line-grouping-time-series-advantages). @@ -93,6 +94,7 @@ Example usage configuring `my_location` as the point field: "field": "my_location" } ``` +% NOTCONSOLE `sort` : (Required outside [`time_series`](#search-aggregations-metrics-geo-line-grouping-time-series) aggregations) @@ -106,6 +108,7 @@ Example usage configuring `@timestamp` as the sort key: "field": "@timestamp" } ``` +% NOTCONSOLE `include_sort` : (Optional, boolean, default: `false`) This option includes, when true, an additional array of the sort values in the feature properties. @@ -205,6 +208,7 @@ POST /tour/_search?filter_path=aggregations } } ``` +% TEST[continued] Which returns: @@ -262,6 +266,7 @@ Which returns: } } ``` +% TESTRESPONSE These results contain an array of buckets, where each bucket is a JSON object with the `key` showing the name of the `city` field, and an inner aggregation result called `museum_tour` containing a [GeoJSON Feature](https://tools.ietf.org/html/rfc7946#section-3.2) describing the actual route between the various attractions in that city. Each result also includes a `properties` object with a `complete` value which will be `false` if the geometry was truncated to the limits specified in the `size` parameter. Note that when we use `time_series` in the next example, we will get the same results structured a little differently. @@ -294,6 +299,7 @@ POST /tour/_search?filter_path=aggregations } } ``` +% TEST[continued] ::::{note} The `geo_line` aggregation no longer requires the `sort` field when nested within a [`time_series` aggregation](/reference/data-analysis/aggregations/search-aggregations-bucket-time-series-aggregation.md). This is because the sort field is set to `@timestamp`, which all time-series indexes are pre-sorted by. If you do set this parameter, and set it to something other than `@timestamp` you will get an error. @@ -360,6 +366,7 @@ This query will result in: } } ``` +% TESTRESPONSE These results are essentially the same as with the previous `terms` aggregation example, but structured differently. Here we see the buckets returned as a map, where the key is an internal description of the TSID. This TSID is unique for each unique combination of fields with `time_series_dimension: true`. Each bucket contains a `key` field which is also a map of all dimension values for the TSID, in this case only the city name is used for grouping. In addition, there is an inner aggregation result called `museum_tour` containing a [GeoJSON Feature](https://tools.ietf.org/html/rfc7946#section-3.2) describing the actual route between the various attractions in that city. Each result also includes a `properties` object with a `complete` value which will be false if the geometry was simplified to the limits specified in the `size` parameter. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geobounds-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geobounds-aggregation.md index ff6cfe8a8173c..d73cd2e404dea 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geobounds-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geobounds-aggregation.md @@ -80,6 +80,7 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## Geo Bounds Aggregation on `geo_shape` fields [geobounds-aggregation-geo-shape] @@ -121,6 +122,7 @@ POST /places/_search?size=0 } } ``` +% TEST ```console-result { @@ -141,4 +143,5 @@ POST /places/_search?size=0 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md index eb8ad47acbeea..d46c2f2672e36 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md @@ -70,6 +70,7 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] The `geo_centroid` aggregation is more interesting when combined as a sub-aggregation to other bucket aggregations. @@ -90,6 +91,7 @@ POST /museums/_search?size=0 } } ``` +% TEST[continued] The above example uses `geo_centroid` as a sub-aggregation to a [terms](/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) bucket aggregation for finding the central location for museums in each city. @@ -141,6 +143,7 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ## Geo Centroid Aggregation on `geo_shape` fields [geocentroid-aggregation-geo-shape] @@ -185,6 +188,7 @@ POST /places/_search?size=0 } } ``` +% TEST ```console-result { @@ -200,6 +204,7 @@ POST /places/_search?size=0 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"_shards": $body._shards,"hits":$body.hits,"timed_out":false,/] ::::{admonition} Using `geo_centroid` as a sub-aggregation of `geohash_grid` :class: warning diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md index 9c2776976b973..635998a070d15 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md @@ -24,6 +24,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -37,6 +38,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] As can be seen, the name of the aggregation (`max_price` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -67,6 +69,8 @@ POST /sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] ## Missing value [_missing_value_10] @@ -86,6 +90,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-median-absolute-deviation-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-median-absolute-deviation-aggregation.md index 3926373e337cc..005c7ba2e0465 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-median-absolute-deviation-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-median-absolute-deviation-aggregation.md @@ -37,6 +37,7 @@ GET reviews/_search } } ``` +% TEST[setup:reviews] 1. `rating` must be a numeric field @@ -56,6 +57,7 @@ The resulting median absolute deviation of `2` tells us that there is a fair amo } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] ## Approximation [_approximation] @@ -78,6 +80,7 @@ GET reviews/_search } } ``` +% TEST[setup:reviews] The default `compression` value for this aggregation is `1000`. At this compression level this aggregation is usually within 5% of the exact result, but observed performance will depend on the sample data. @@ -115,6 +118,7 @@ GET reviews/_search?filter_path=aggregations } } ``` +% TEST[setup:reviews] Which will result in: @@ -152,5 +156,6 @@ GET reviews/_search } } ``` +% TEST[setup:reviews] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-min-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-min-aggregation.md index 2c313a3d2dbef..f4f943c278a04 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-min-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-min-aggregation.md @@ -24,6 +24,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -38,6 +39,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] As can be seen, the name of the aggregation (`min_price` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -68,6 +70,8 @@ POST /sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] ## Missing value [_missing_value_12] @@ -87,6 +91,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md index 1d9c78d0991a1..c9e301f5e41bc 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md @@ -32,6 +32,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. The field `load_time` must be a numeric field @@ -57,6 +58,14 @@ By default, the `percentile` metric will generate a range of percentiles: `[ 1, } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/"1.0": 10.0/"1.0": 9.9/] +% TESTRESPONSE[s/"5.0": 30.0/"5.0": 29.5/] +% TESTRESPONSE[s/"25.0": 170.0/"25.0": 167.5/] +% TESTRESPONSE[s/"50.0": 445.0/"50.0": 445.0/] +% TESTRESPONSE[s/"75.0": 720.0/"75.0": 722.5/] +% TESTRESPONSE[s/"95.0": 940.0/"95.0": 940.5/] +% TESTRESPONSE[s/"99.0": 980.0/"99.0": 980.1/] As you can see, the aggregation will return a calculated value for each percentile in the default range. If we assume response times are in milliseconds, it is immediately obvious that the webpage normally loads in 10-720ms, but occasionally spikes to 940-980ms. @@ -76,6 +85,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Use the `percents` parameter to specify particular percentiles to calculate @@ -98,6 +108,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] Response: @@ -141,6 +152,14 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/"value": 10.0/"value": 9.9/] +% TESTRESPONSE[s/"value": 30.0/"value": 29.5/] +% TESTRESPONSE[s/"value": 170.0/"value": 167.5/] +% TESTRESPONSE[s/"value": 445.0/"value": 445.0/] +% TESTRESPONSE[s/"value": 720.0/"value": 722.5/] +% TESTRESPONSE[s/"value": 940.0/"value": 940.5/] +% TESTRESPONSE[s/"value": 980.0/"value": 980.1/] ## Script [_script_10] @@ -171,6 +190,9 @@ GET latency/_search } } ``` +% TEST[setup:latency] +% TEST[s/_search/_search?filter_path=aggregations/] +% TEST[s/"timeUnit": 1000/"timeUnit": 10/] ## Percentiles are (usually) approximate [search-aggregations-metrics-percentile-aggregation-approximation] @@ -220,6 +242,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Compression controls memory usage and approximation error @@ -251,6 +274,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Optimize TDigest for accuracy, at the expense of performance @@ -281,6 +305,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. `hdr` object indicates that HDR Histogram should be used to calculate the percentiles and specific settings for this algorithm can be specified inside the object 2. `number_of_significant_value_digits` specifies the resolution of values for the histogram in number of significant digits @@ -307,6 +332,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-rank-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-rank-aggregation.md index 83b8897875b01..5d42d458693d9 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-rank-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-rank-aggregation.md @@ -35,6 +35,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. The field `load_time` must be a numeric field @@ -55,6 +56,9 @@ The response will look like this: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/"500.0": 55.0/"500.0": 55.00000000000001/] +% TESTRESPONSE[s/"600.0": 64.0/"600.0": 64.0/] From this information you can determine you are hitting the 99% load time target but not quite hitting the 95% load time target @@ -77,6 +81,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] Response: @@ -100,6 +105,9 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/"value": 55.0/"value": 55.00000000000001/] +% TESTRESPONSE[s/"value": 64.0/"value": 64.0/] ## Script [_script_9] @@ -131,6 +139,8 @@ GET latency/_search } } ``` +% TEST[setup:latency] +% TEST[s/_search/_search?filter_path=aggregations/] ## HDR Histogram [_hdr_histogram] @@ -156,6 +166,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. `hdr` object indicates that HDR Histogram should be used to calculate the percentiles and specific settings for this algorithm can be specified inside the object 2. `number_of_significant_value_digits` specifies the resolution of values for the histogram in number of significant digits @@ -183,6 +194,7 @@ GET latency/_search } } ``` +% TEST[setup:latency] 1. Documents without a value in the `load_time` field will fall into the same bucket as documents that have the value `10`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-rate-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-rate-aggregation.md index 15479da75465e..cccd1d9c9f535 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-rate-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-rate-aggregation.md @@ -26,6 +26,7 @@ A `rate` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE The following request will group all sales records into monthly buckets and then convert the number of sales transactions in each bucket into per annual sales rate. @@ -50,6 +51,7 @@ GET sales/_search } } ``` +% TEST[setup:sales] 1. Histogram is grouped by month. 2. But the rate is converted into annual rate. @@ -92,6 +94,7 @@ The response will return the annual rate of transactions in each bucket. Since t } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] Instead of counting the number of documents, it is also possible to calculate a sum of all values of the fields in the documents in each bucket or the number of values in each bucket. The following request will group all sales records into monthly bucket and than calculate the total monthly sales and convert them into average daily sales. @@ -117,6 +120,7 @@ GET sales/_search } } ``` +% TEST[setup:sales] 1. Histogram is grouped by month. 2. Calculate sum of all sale prices @@ -160,6 +164,7 @@ The response will contain the average daily sale prices for each month. } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] You can also take advantage of `composite` aggregations to calculate the average daily sale price for each item in your inventory @@ -199,6 +204,7 @@ GET sales/_search?filter_path=aggregations&size=0 } } ``` +% TEST[setup:sales] 1. Composite aggregation with a date histogram source and a source for the item type. 2. The date histogram source grouping monthly @@ -319,6 +325,7 @@ GET sales/_search } } ``` +% TEST[setup:sales] 1. Histogram is grouped by month. 2. Calculate number of all sale prices @@ -363,6 +370,7 @@ The response will contain the average daily sale prices for each month. } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] By default `sum` mode is used. @@ -440,6 +448,7 @@ GET sales/_search } } ``` +% TEST[setup:sales] ```console-result { @@ -476,5 +485,6 @@ GET sales/_search } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md index 9defeda72a325..f3949168a1ade 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md @@ -39,6 +39,7 @@ POST ledger/_search?size=0 } } ``` +% TEST[setup:ledger] 1. `init_script` is an optional parameter, all other scripts are required. @@ -58,6 +59,8 @@ The response for the above aggregation: } } ``` +% TESTRESPONSE[s/"took": 218/"took": $body.took/] +% TESTRESPONSE[s/.../"_shards": $body._shards, "hits": $body.hits, "timed_out": false,/] The above example can also be specified using stored scripts as follows: @@ -87,6 +90,7 @@ POST ledger/_search?size=0 } } ``` +% TEST[setup:ledger,stored_scripted_metric_script] 1. script parameters for `init`, `map` and `combine` scripts must be specified in a global `params` object so that it can be shared between the scripts. @@ -157,6 +161,7 @@ Lets say that documents 1 and 3 end up on shard A and documents 2 and 4 end up o ```js "state" : {} ``` +% NOTCONSOLE ### After init_script [_after_init_script] @@ -222,6 +227,7 @@ The reduce_script receives a `states` array containing the result of the combine 120 ] ``` +% NOTCONSOLE It reduces the responses for the shards down to a final overall profit figure (by summing the values) and returns this as the result of the aggregation to produce the response: @@ -236,6 +242,7 @@ It reduces the responses for the shards down to a final overall profit figure (b } } ``` +% NOTCONSOLE diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md index 1fe99e261d240..7aee6a2361680 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md @@ -21,6 +21,7 @@ POST /exams/_search?size=0 } } ``` +% TEST[setup:exams] The above aggregation computes the grades statistics over all documents. The aggregation type is `stats` and the `field` setting defines the numeric field of the documents the stats will be computed on. The above will return the following: @@ -39,6 +40,7 @@ The above aggregation computes the grades statistics over all documents. The agg } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The name of the aggregation (`grades_stats` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -67,6 +69,8 @@ POST /exams/_search } } ``` +% TEST[setup:exams] +% TEST[s/_search/_search?filter_path=aggregations/] ## Missing value [_missing_value_15] @@ -86,6 +90,7 @@ POST /exams/_search?size=0 } } ``` +% TEST[setup:exams] 1. Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `0`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-string-stats-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-string-stats-aggregation.md index 04d37af374c00..74c355db13b38 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-string-stats-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-string-stats-aggregation.md @@ -27,6 +27,7 @@ POST /my-index-000001/_search?size=0 } } ``` +% TEST[setup:messages] The above aggregation computes the string statistics for the `message` field in all documents. The aggregation type is `string_stats` and the `field` parameter defines the field of the documents the stats will be computed on. The above will return the following: @@ -45,6 +46,7 @@ The above aggregation computes the string statistics for the `message` field in } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The name of the aggregation (`message_stats` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -65,6 +67,7 @@ POST /my-index-000001/_search?size=0 } } ``` +% TEST[setup:messages] 1. Set the `show_distribution` parameter to `true`, so that probability distribution for all characters is returned in the results. @@ -109,6 +112,7 @@ POST /my-index-000001/_search?size=0 } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The `distribution` object shows the probability of each character appearing in all terms. The characters are sorted by descending probability. @@ -136,6 +140,8 @@ POST /my-index-000001/_search } } ``` +% TEST[setup:messages] +% TEST[s/_search/_search?filter_path=aggregations/] ## Missing value [_missing_value_16] @@ -155,6 +161,7 @@ POST /my-index-000001/_search?size=0 } } ``` +% TEST[setup:messages] 1. Documents without a value in the `message` field will be treated as documents that have the value `[empty message]`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-sum-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-sum-aggregation.md index cf97a4697b330..015133993207e 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-sum-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-sum-aggregation.md @@ -26,6 +26,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Resulting in: @@ -39,6 +40,7 @@ Resulting in: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The name of the aggregation (`hat_prices` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -77,6 +79,8 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] +% TEST[s/size=0/size=0&filter_path=aggregations/] ## Missing value [_missing_value_17] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md index 57ed6f15bb52b..8428710e9b8b5 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md @@ -80,6 +80,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Possible response: @@ -177,6 +178,10 @@ Possible response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] +% TESTRESPONSE[s/AVnNBmauCQpcRyxw6ChK/$body.aggregations.top_tags.buckets.0.top_sales_hits.hits.hits.0._id/] +% TESTRESPONSE[s/AVnNBmauCQpcRyxw6ChL/$body.aggregations.top_tags.buckets.1.top_sales_hits.hits.hits.0._id/] +% TESTRESPONSE[s/AVnNBmatCQpcRyxw6ChH/$body.aggregations.top_tags.buckets.2.top_sales_hits.hits.hits.0._id/] ## Field collapse example [_field_collapse_example] @@ -219,6 +224,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] At the moment the `max` (or `min`) aggregator is needed to make sure the buckets from the `terms` aggregator are ordered according to the score of the most relevant webpage per domain. Unfortunately the `top_hits` aggregator can’t be used in the `order` option of the `terms` aggregator yet. @@ -265,6 +271,7 @@ PUT /sales/_doc/1?refresh ] } ``` +% TEST[continued] It’s now possible to execute the following `top_hits` aggregation (wrapped in a `nested` aggregation): @@ -296,6 +303,8 @@ POST /sales/_search } } ``` +% TEST[continued] +% TEST[s/_search/_search?filter_path=aggregations.by_sale.by_user.buckets/] Top hits response snippet with a nested hit, which resides in the first slot of array field `comments`: @@ -341,6 +350,7 @@ Top hits response snippet with a nested hit, which resides in the first slot of } } ``` +% TESTRESPONSE[s/...//] 1. Name of the array field containing the nested hit 2. Position if the nested hit in the containing array @@ -385,6 +395,7 @@ In the example below a nested hit resides in the first slot of the field `nested } ... ``` +% NOTCONSOLE ## Use in pipeline aggregations [_use_in_pipeline_aggregations] @@ -429,6 +440,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] The `bucket_path` uses the `top_hits` name `top_sales_hits` and a keyword for the field providing the aggregate value, namely `_source` field `price` in the example above. Other options include `top_sales_hits[_sort]`, for filtering on the sort value `date` above, and `top_sales_hits[_score]`, for filtering on the score of the top hit. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-metrics.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-metrics.md index f49eebba715a5..d0f13fc567709 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-metrics.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-metrics.md @@ -43,6 +43,7 @@ Which returns: } } ``` +% TESTRESPONSE `top_metrics` is fairly similar to [`top_hits`](/reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md) in spirit but because it is more limited it is able to do its job using less memory and is often faster. @@ -138,6 +139,7 @@ Which returns: } } ``` +% TESTRESPONSE ## `missing` [_missing] @@ -248,6 +250,7 @@ Which returns: } } ``` +% TESTRESPONSE The default `size` is 1. The maximum default size is `10` because the aggregation’s working storage is "dense", meaning we allocate `size` slots for every bucket. `10` is a **very** conservative default maximum and you can raise it if you need to by changing the `top_metrics_max_size` index setting. But know that large sizes can take a fair bit of memory, especially if they are inside of an aggregation which makes many buckes like a large [terms aggregation](#search-aggregations-metrics-top-metrics-example-terms). If you till want to raise it, use something like: @@ -257,6 +260,7 @@ PUT /test/_settings "top_metrics_max_size": 100 } ``` +% TEST[continued] ::::{note} If `size` is more than `1` the `top_metrics` aggregation can’t be the **target** of a sort. @@ -337,6 +341,7 @@ Which returns: } } ``` +% TESTRESPONSE Unlike `top_hits`, you can sort buckets by the results of this metric: @@ -361,6 +366,7 @@ POST /node/_search?filter_path=aggregations } } ``` +% TEST[continued] Which returns: @@ -390,6 +396,7 @@ Which returns: } } ``` +% TESTRESPONSE ### Mixed sort types [_mixed_sort_types] @@ -430,6 +437,7 @@ Which returns: } } ``` +% TESTRESPONSE While this is better than an error it **probably** isn’t what you were going for. While it does lose some precision, you can explicitly cast the whole number fields to floating points with something like: @@ -446,6 +454,7 @@ POST /test*/_search?filter_path=aggregations } } ``` +% TEST[continued] Which returns the much more expected: @@ -458,6 +467,7 @@ Which returns the much more expected: } } ``` +% TESTRESPONSE ### Use in pipeline aggregations [_use_in_pipeline_aggregations_2] @@ -493,6 +503,7 @@ POST /test*/_search?filter_path=aggregations } } ``` +% TEST[continued] The `bucket_path` uses the `top_metrics` name `tm` and a keyword for the metric providing the aggregate value, namely `m`. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-ttest-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-ttest-aggregation.md index 95970a6a3584f..d46d82778f932 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-ttest-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-ttest-aggregation.md @@ -22,6 +22,7 @@ A `t_test` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE Assuming that we have a record of node start up times before and after upgrade, let’s look at a t-test to see if upgrade affected the node start up time in a meaningful way. @@ -40,6 +41,7 @@ GET node_upgrade/_search } } ``` +% TEST[setup:node_upgrade] 1. The field `startup_time_before` must be a numeric field. 2. The field `startup_time_after` must be a numeric field. @@ -59,6 +61,7 @@ The response will return the p-value or probability value for the test. It is th } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] 1. The p-value. @@ -111,6 +114,7 @@ GET node_upgrade/_search } } ``` +% TEST[setup:node_upgrade] 1. The field `startup_time_before` must be a numeric field. 2. Any query that separates two groups can be used here. @@ -130,6 +134,7 @@ GET node_upgrade/_search } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] 1. The p-value. @@ -171,5 +176,7 @@ GET node_upgrade/_search } } ``` +% TEST[setup:node_upgrade] +% TEST[s/_search/_search?filter_path=aggregations/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-valuecount-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-valuecount-aggregation.md index 318d131006299..0f35f8e7858d0 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-valuecount-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-valuecount-aggregation.md @@ -19,6 +19,7 @@ POST /sales/_search?size=0 } } ``` +% TEST[setup:sales] Response: @@ -32,6 +33,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The name of the aggregation (`types_count` above) also serves as the key by which the aggregation result can be retrieved from the returned response. @@ -63,6 +65,8 @@ POST /sales/_search } } ``` +% TEST[setup:sales] +% TEST[s/_search/_search?filter_path=aggregations/] ## Histogram fields [search-aggregations-metrics-valuecount-aggregation-histogram-fields] @@ -114,5 +118,6 @@ Eventually, the aggregation will add all values for all histograms and return th } } ``` +% TESTRESPONSE[skip:test not setup] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-weight-avg-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-weight-avg-aggregation.md index ea83567f7fdff..d061d391d597e 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-metrics-weight-avg-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-metrics-weight-avg-aggregation.md @@ -61,6 +61,7 @@ POST /exams/_search } } ``` +% TEST[setup:exams] Which yields a response like: @@ -74,6 +75,7 @@ Which yields a response like: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] While multiple values-per-field are allowed, only one weight is allowed. If the aggregation encounters a document that has more than one weight (e.g. the weight field is a multi-valued field) it will abort the search. If you have this situation, you should build a [Runtime field](#search-aggregations-metrics-weight-avg-aggregation-runtime-field) to combine those values into a single weight. @@ -105,6 +107,7 @@ POST /exams/_search } } ``` +% TEST The three values (`1`, `2`, and `3`) will be included as independent values, all with the weight of `2`: @@ -118,6 +121,7 @@ The three values (`1`, `2`, and `3`) will be included as independent values, all } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] The aggregation returns `2.0` as the result, which matches what we would expect when calculating by hand: `((1*2) + (2*2) + (3*2)) / (2+2+2) == 2` @@ -205,5 +209,6 @@ POST /exams/_search } } ``` +% TEST[setup:exams] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-avg-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-avg-bucket-aggregation.md index 4c2f373fe33a2..97ed5d699df98 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-avg-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-avg-bucket-aggregation.md @@ -18,6 +18,7 @@ A sibling pipeline aggregation which calculates the mean value of a specified me "format": "#,##0.00;(#,##0.00)" } ``` +% NOTCONSOLE ## Parameters [avg-bucket-params] @@ -124,5 +125,8 @@ The request returns the following response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md index 2f684705d4725..090597b5e5f7a 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md @@ -24,6 +24,7 @@ A `bucket_script` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE 1. Here, `my_var1` is the name of the variable for this buckets path to use in the script, `the_sum` is the path to the metrics to use for that variable. @@ -83,6 +84,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] And the following may be the response: @@ -151,5 +153,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-selector-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-selector-aggregation.md index 04e39708f048b..af372ba5b6395 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-selector-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-selector-aggregation.md @@ -29,6 +29,7 @@ A `bucket_selector` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE 1. Here, `my_var1` is the name of the variable for this buckets path to use in the script, `the_sum` is the path to the metrics to use for that variable. @@ -72,6 +73,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] And the following may be the response: @@ -105,6 +107,9 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] 1. Bucket for `2015/02/01 00:00:00` has been removed as its total sales was less than 200 diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-sort-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-sort-aggregation.md index 5d702cbc9aaab..69974df06174b 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-sort-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-sort-aggregation.md @@ -31,6 +31,7 @@ A `bucket_sort` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE 1. Here, `sort_field_1` is the bucket path to the variable to be used as the primary sort and its order is ascending. @@ -75,6 +76,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `sort` is set to use the values of `total_sales` in descending order 2. `size` is set to `3` meaning only the top 3 months in `total_sales` will be returned @@ -120,6 +122,9 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 82/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] ## Truncating without sorting [_truncating_without_sorting] @@ -150,6 +155,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] Response: @@ -172,5 +178,8 @@ Response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-cardinality-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-cardinality-aggregation.md index 880adfda21642..e81866139b5d2 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-cardinality-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-cardinality-aggregation.md @@ -22,6 +22,7 @@ A `cumulative_cardinality` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$cumulative-cardinality-params$$$ @@ -58,6 +59,7 @@ GET /user_hits/_search } } ``` +% TEST[setup:user_hits] 1. `buckets_path` instructs this aggregation to use the output of the `distinct_users` aggregation for the cumulative cardinality @@ -111,6 +113,9 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] Note how the second day, `2019-01-02`, has two distinct users but the `total_new_users` metric generated by the cumulative pipeline agg only increments to three. This means that only one of the two users that day were new, the other had already been seen in the previous day. This happens again on the third day, where only one of three users is completely new. @@ -152,6 +157,7 @@ GET /user_hits/_search } } ``` +% TEST[setup:user_hits] And the following may be the response: @@ -208,5 +214,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-sum-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-sum-aggregation.md index a0c08e7ffb590..6077430ca6d21 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-sum-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-sum-aggregation.md @@ -20,6 +20,7 @@ A `cumulative_sum` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$cumulative-sum-params$$$ @@ -56,6 +57,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this cumulative sum aggregation to use the output of the `sales` aggregation for the cumulative sum @@ -109,5 +111,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-derivative-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-derivative-aggregation.md index e04e0848d2264..407fc8fd72112 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-derivative-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-derivative-aggregation.md @@ -18,6 +18,7 @@ A `derivative` aggregation looks like this in isolation: "buckets_path": "the_sum" } ``` +% NOTCONSOLE $$$derivative-params$$$ @@ -58,10 +59,10 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this derivative aggregation to use the output of the `sales` aggregation for the derivative - And the following may be the response: ```console-result @@ -108,13 +109,14 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] 1. No derivative for the first bucket since we need at least 2 data points to calculate the derivative 2. Derivative value units are implicitly defined by the `sales` aggregation and the parent histogram so in this case the units would be $/month assuming the `price` field has units of $. 3. The number of documents in the bucket are represented by the `doc_count` - - ## Second Order Derivative [_second_order_derivative] A second order derivative can be calculated by chaining the derivative pipeline aggregation onto the result of another derivative pipeline aggregation as in the following example which will calculate both the first and the second order derivative of the total monthly sales: @@ -150,10 +152,10 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` for the second derivative points to the name of the first derivative - And the following may be the response: ```console-result @@ -203,11 +205,12 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 50/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] 1. No second derivative for the first two buckets since we need at least 2 data points from the first derivative to calculate the second derivative - - ## Units [_units] The derivative aggregation allows the units of the derivative values to be specified. This returns an extra field in the response `normalized_value` which reports the derivative value in the desired x-axis units. In the below example we calculate the derivative of the total sales per month but ask for the derivative of the sales as in the units of sales per day: @@ -239,10 +242,10 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `unit` specifies what unit to use for the x-axis of the derivative calculation - And the following may be the response: ```console-result @@ -291,9 +294,9 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 50/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] 1. `value` is reported in the original units of *per month* 2. `normalized_value` is reported in the desired units of *per day* - - - diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-extended-stats-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-extended-stats-bucket-aggregation.md index facf92a6e82d7..42ccd070e63f4 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-extended-stats-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-extended-stats-bucket-aggregation.md @@ -22,6 +22,7 @@ A `extended_stats_bucket` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$extended-stats-bucket-params$$$ @@ -60,6 +61,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `bucket_paths` instructs this `extended_stats_bucket` aggregation that we want the calculate stats for the `sales` aggregation in the `sales_per_month` date histogram. @@ -126,5 +128,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md index 28cae9b986f58..697686d091a98 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md @@ -31,6 +31,7 @@ A `inference` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE 1. The unique identifier or alias for the trained model. 2. The optional inference config which overrides the model’s default settings @@ -157,6 +158,7 @@ GET kibana_sample_data_logs/_search } } ``` +% TEST[skip:setup kibana sample data] 1. A composite bucket aggregation that aggregates the data by `client_ip`. 2. A series of metrics and bucket sub-aggregations. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-max-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-max-bucket-aggregation.md index c78bb02de6c4b..a4ce6b8043c36 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-max-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-max-bucket-aggregation.md @@ -20,6 +20,7 @@ A `max_bucket` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$max-bucket-params$$$ @@ -57,10 +58,10 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this max_bucket aggregation that we want the maximum value of the `sales` aggregation in the `sales_per_month` date histogram. - And the following may be the response: ```console-result @@ -105,8 +106,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] 1. `keys` is an array of strings since the maximum value may be present in multiple buckets - - - diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-min-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-min-bucket-aggregation.md index 7d408cba067a9..0fac6c9ccfca9 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-min-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-min-bucket-aggregation.md @@ -20,6 +20,7 @@ A `min_bucket` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$min-bucket-params$$$ @@ -57,10 +58,10 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this min_bucket aggregation that we want the minimum value of the `sales` aggregation in the `sales_per_month` date histogram. - And the following may be the response: ```console-result @@ -105,8 +106,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] 1. `keys` is an array of strings since the minimum value may be present in multiple buckets - - - diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-movfn-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-movfn-aggregation.md index a04ee179282a9..e1d40a6dec2b8 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-movfn-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-movfn-aggregation.md @@ -22,6 +22,7 @@ A `moving_fn` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$moving-fn-params$$$ @@ -61,6 +62,7 @@ POST /_search } } ``` +% TEST[setup:sales] 1. A `date_histogram` named "my_date_histo" is constructed on the "timestamp" field, with one-month intervals 2. A `sum` metric is used to calculate the sum of a field. This could be any numeric metric (sum, min, max, etc) @@ -118,6 +120,9 @@ An example response from the above aggregation may look like: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] ## Custom user scripting [_custom_user_scripting] @@ -152,6 +157,7 @@ POST /_search } } ``` +% TEST[setup:sales] ## shift parameter [shift-parameter] @@ -217,6 +223,7 @@ POST /_search } } ``` +% TEST[setup:sales] ### min Function [_min_function] @@ -255,6 +262,7 @@ POST /_search } } ``` +% TEST[setup:sales] ### sum Function [_sum_function] @@ -293,6 +301,7 @@ POST /_search } } ``` +% TEST[setup:sales] ### stdDev Function [_stddev_function] @@ -332,6 +341,7 @@ POST /_search } } ``` +% TEST[setup:sales] The `avg` parameter must be provided to the standard deviation function because different styles of averages can be computed on the window (simple, linearly weighted, etc). The various moving averages that are detailed below can be used to calculate the average for the standard deviation function. @@ -374,6 +384,7 @@ POST /_search } } ``` +% TEST[setup:sales] @@ -415,6 +426,7 @@ POST /_search } } ``` +% TEST[setup:sales] ## ewma Function [_ewma_function] @@ -456,6 +468,7 @@ POST /_search } } ``` +% TEST[setup:sales] ## holt Function [_holt_function] @@ -502,6 +515,7 @@ POST /_search } } ``` +% TEST[setup:sales] In practice, the `alpha` value behaves very similarly in `holtMovAvg` as `ewmaMovAvg`: small values produce more smoothing and more lag, while larger values produce closer tracking and less lag. The value of `beta` is often difficult to see. Small values emphasize long-term trends (such as a constant linear trend in the whole series), while larger values emphasize short-term trends. @@ -553,6 +567,7 @@ POST /_search } } ``` +% TEST[setup:sales] ::::{warning} Multiplicative Holt-Winters works by dividing each data point by the seasonal value. This is problematic if any of your data is zero, or if there are gaps in the data (since this results in a divid-by-zero). To combat this, the `mult` Holt-Winters pads all values by a very small amount (1*10-10) so that all values are non-zero. This affects the result, but only minimally. If your data is non-zero, or you prefer to see `NaN` when zero’s are encountered, you can disable this behavior with `pad: false` diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-moving-percentiles-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-moving-percentiles-aggregation.md index 1cab3ef611c66..df0376e024179 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-moving-percentiles-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-moving-percentiles-aggregation.md @@ -23,6 +23,7 @@ A `moving_percentiles` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$moving-percentiles-params$$$ @@ -62,6 +63,7 @@ POST /_search } } ``` +% TEST[setup:sales] 1. A `date_histogram` named "my_date_histo" is constructed on the "timestamp" field, with one-day intervals 2. A `percentile` metric is used to calculate the percentiles of a field. @@ -131,6 +133,9 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] The output format of the `moving_percentiles` aggregation is inherited from the format of the referenced [`percentiles`](/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md) aggregation. diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-normalize-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-normalize-aggregation.md index cdec3c4a59579..433abdbe8fa52 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-normalize-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-normalize-aggregation.md @@ -21,6 +21,7 @@ A `normalize` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$normalize_pipeline-params$$$ @@ -135,6 +136,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this normalize aggregation to use the output of the `sales` aggregation for rescaling 2. `method` sets which rescaling to apply. In this case, `percent_of_sum` will calculate the sales value as a percent of all sales in the parent bucket @@ -193,5 +195,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-percentiles-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-percentiles-bucket-aggregation.md index 54e6b80ab2bf2..610c7201540eb 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-percentiles-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-percentiles-bucket-aggregation.md @@ -20,6 +20,7 @@ A `percentiles_bucket` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$percentiles-bucket-params$$$ @@ -60,6 +61,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this percentiles_bucket aggregation that we want to calculate percentiles for the `sales` aggregation in the `sales_per_month` date histogram. 2. `percents` specifies which percentiles we wish to calculate, in this case, the 25th, 50th and 75th percentiles. @@ -112,6 +114,9 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] ## Percentiles_bucket implementation [_percentiles_bucket_implementation] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-serialdiff-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-serialdiff-aggregation.md index cd246d6217ce3..597a4ffb1eb50 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-serialdiff-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-serialdiff-aggregation.md @@ -43,6 +43,7 @@ A `serial_diff` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$serial-diff-params$$$ diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-stats-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-stats-bucket-aggregation.md index e816feaa1bbe0..594b89031c646 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-stats-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-stats-bucket-aggregation.md @@ -20,6 +20,7 @@ A `stats_bucket` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$stats-bucket-params$$$ @@ -57,6 +58,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `bucket_paths` instructs this `stats_bucket` aggregation that we want the calculate stats for the `sales` aggregation in the `sales_per_month` date histogram. @@ -108,5 +110,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-sum-bucket-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-sum-bucket-aggregation.md index f2c0e1597726f..0c6002bf0ccb8 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-sum-bucket-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-sum-bucket-aggregation.md @@ -20,6 +20,7 @@ A `sum_bucket` aggregation looks like this in isolation: } } ``` +% NOTCONSOLE $$$sum-bucket-params$$$ @@ -57,6 +58,7 @@ POST /sales/_search } } ``` +% TEST[setup:sales] 1. `buckets_path` instructs this sum_bucket aggregation that we want the sum of the `sales` aggregation in the `sales_per_month` date histogram. @@ -104,5 +106,8 @@ And the following may be the response: } } ``` +% TESTRESPONSE[s/"took": 11/"took": $body.took/] +% TESTRESPONSE[s/"_shards": .../"_shards": $body._shards/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] diff --git a/docs/reference/data-analysis/aggregations/search-aggregations-random-sampler-aggregation.md b/docs/reference/data-analysis/aggregations/search-aggregations-random-sampler-aggregation.md index 0c68bb40b1ce8..3e19718b64156 100644 --- a/docs/reference/data-analysis/aggregations/search-aggregations-random-sampler-aggregation.md +++ b/docs/reference/data-analysis/aggregations/search-aggregations-random-sampler-aggregation.md @@ -40,6 +40,7 @@ GET kibana_sample_data_ecommerce/_search?size=0&track_total_hits=false } } ``` +% TEST[setup:kibana_sample_data_ecommerce] ## Top-level parameters for random_sampler [random-sampler-top-level-params] diff --git a/docs/reference/data-analysis/text-analysis/analysis-delimited-payload-tokenfilter.md b/docs/reference/data-analysis/text-analysis/analysis-delimited-payload-tokenfilter.md index f705206e046a6..e5621fe1a7dc8 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-delimited-payload-tokenfilter.md +++ b/docs/reference/data-analysis/text-analysis/analysis-delimited-payload-tokenfilter.md @@ -166,6 +166,7 @@ POST text_payloads/_doc/1 "text": "the|0 brown|3 fox|4 is|0 quick|10" } ``` +% TEST[continued] Use the [term vectors API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-termvectors) to return the document’s tokens and base64-encoded payloads. @@ -176,6 +177,7 @@ GET text_payloads/_termvectors/1 "payloads": true } ``` +% TEST[continued] The API returns the following response: @@ -244,5 +246,6 @@ The API returns the following response: } } ``` +% TESTRESPONSE[s/"took": 8/"took": "$body.took"/] diff --git a/docs/reference/data-analysis/text-analysis/analysis-fingerprint-analyzer.md b/docs/reference/data-analysis/text-analysis/analysis-fingerprint-analyzer.md index d00f108c93f82..b5a602b25a915 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-fingerprint-analyzer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-fingerprint-analyzer.md @@ -117,4 +117,5 @@ PUT /fingerprint_example } } ``` +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: fingerprint_example, first: fingerprint, second: rebuilt_fingerprint}\nendyaml\n/] diff --git a/docs/reference/data-analysis/text-analysis/analysis-hyp-decomp-tokenfilter.md b/docs/reference/data-analysis/text-analysis/analysis-hyp-decomp-tokenfilter.md index cb2ae226f6eb9..06c6679a2d080 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-hyp-decomp-tokenfilter.md +++ b/docs/reference/data-analysis/text-analysis/analysis-hyp-decomp-tokenfilter.md @@ -29,6 +29,7 @@ GET _analyze "text": "Kaffeetasse" } ``` +% TEST[skip: requires a valid hyphenation_patterns.xml file for DE-DR] The filter produces the following tokens: diff --git a/docs/reference/data-analysis/text-analysis/analysis-keyword-analyzer.md b/docs/reference/data-analysis/text-analysis/analysis-keyword-analyzer.md index 0e674a9477282..51a2c3169b54e 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-keyword-analyzer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-keyword-analyzer.md @@ -58,6 +58,7 @@ PUT /keyword_example } } ``` +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: keyword_example, first: keyword, second: rebuilt_keyword}\nendyaml\n/] 1. You’d add any token filters here. diff --git a/docs/reference/data-analysis/text-analysis/analysis-keyword-repeat-tokenfilter.md b/docs/reference/data-analysis/text-analysis/analysis-keyword-repeat-tokenfilter.md index fd42a15a5bdff..ebbce3b508b75 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-keyword-repeat-tokenfilter.md +++ b/docs/reference/data-analysis/text-analysis/analysis-keyword-repeat-tokenfilter.md @@ -130,6 +130,7 @@ The API returns the following response. Note that one version of each token has } } ``` +% TESTRESPONSE[s/"tokenizer": .../"tokenizer": $body.detail.tokenizer/] :::: @@ -240,6 +241,8 @@ The API returns the following response. Note the following changes: } } ``` +% TESTRESPONSE[s/"tokenizer": .../"tokenizer": $body.detail.tokenizer/] +% TESTRESPONSE[s/"tokens": …​/"tokens": $body.$_path/] :::: @@ -338,6 +341,8 @@ The API returns the following response. Note that the duplicate tokens for `fox` } } ``` +% TESTRESPONSE[s/"tokenizer": .../"tokenizer": $body.detail.tokenizer/] +% TESTRESPONSE[s/"tokens": …​/"tokens": $body.$_path/] :::: diff --git a/docs/reference/data-analysis/text-analysis/analysis-lang-analyzer.md b/docs/reference/data-analysis/text-analysis/analysis-lang-analyzer.md index 3e1c651805a73..d8140a00ad6eb 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-lang-analyzer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-lang-analyzer.md @@ -73,7 +73,8 @@ PUT /arabic_example } } ``` - +% TEST[s/"arabic_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: arabic_example, first: arabic, second: rebuilt_arabic}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -117,7 +118,8 @@ PUT /armenian_example } } ``` - +% TEST[s/"armenian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: armenian_example, first: armenian, second: rebuilt_armenian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -161,7 +163,8 @@ PUT /basque_example } } ``` - +% TEST[s/"basque_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: basque_example, first: basque, second: rebuilt_basque}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -208,7 +211,8 @@ PUT /bengali_example } } ``` - +% TEST[s/"bengali_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: bengali_example, first: bengali, second: rebuilt_bengali}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -252,7 +256,8 @@ PUT /brazilian_example } } ``` - +% TEST[s/"brazilian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: brazilian_example, first: brazilian, second: rebuilt_brazilian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -296,7 +301,8 @@ PUT /bulgarian_example } } ``` - +% TEST[s/"bulgarian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: bulgarian_example, first: bulgarian, second: rebuilt_bulgarian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -346,7 +352,8 @@ PUT /catalan_example } } ``` - +% TEST[s/"catalan_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: catalan_example, first: catalan, second: rebuilt_catalan}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -393,7 +400,8 @@ PUT /cjk_example } } ``` - +% TEST[s/"cjk_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: cjk_example, first: cjk, second: rebuilt_cjk}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. The default stop words are **almost** the same as the `_english_` set, but not exactly the same. @@ -436,7 +444,8 @@ PUT /czech_example } } ``` - +% TEST[s/"czech_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: czech_example, first: czech, second: rebuilt_czech}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -480,7 +489,8 @@ PUT /danish_example } } ``` - +% TEST[s/"danish_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: danish_example, first: danish, second: rebuilt_danish}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -534,7 +544,8 @@ PUT /dutch_example } } ``` - +% TEST[s/"dutch_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: dutch_example, first: dutch, second: rebuilt_dutch}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -583,7 +594,8 @@ PUT /english_example } } ``` - +% TEST[s/"english_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: english_example, first: english, second: rebuilt_english}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -627,7 +639,8 @@ PUT /estonian_example } } ``` - +% TEST[s/"estonian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: estonian_example, first: estonian, second: rebuilt_estonian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -671,7 +684,8 @@ PUT /finnish_example } } ``` - +% TEST[s/"finnish_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: finnish_example, first: finnish, second: rebuilt_finnish}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -725,7 +739,8 @@ PUT /french_example } } ``` - +% TEST[s/"french_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: french_example, first: french, second: rebuilt_french}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -769,7 +784,8 @@ PUT /galician_example } } ``` - +% TEST[s/"galician_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: galician_example, first: galician, second: rebuilt_galician}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -814,7 +830,8 @@ PUT /german_example } } ``` - +% TEST[s/"german_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: german_example, first: german, second: rebuilt_german}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -862,7 +879,8 @@ PUT /greek_example } } ``` - +% TEST[s/"greek_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: greek_example, first: greek, second: rebuilt_greek}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -909,7 +927,8 @@ PUT /hindi_example } } ``` - +% TEST[s/"hindi_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: hindi_example, first: hindi, second: rebuilt_hindi}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -953,7 +972,8 @@ PUT /hungarian_example } } ``` - +% TEST[s/"hungarian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: hungarian_example, first: hungarian, second: rebuilt_hungarian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -997,7 +1017,8 @@ PUT /indonesian_example } } ``` - +% TEST[s/"indonesian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: indonesian_example, first: indonesian, second: rebuilt_indonesian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1057,7 +1078,8 @@ PUT /irish_example } } ``` - +% TEST[s/"irish_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: irish_example, first: irish, second: rebuilt_irish}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1112,7 +1134,8 @@ PUT /italian_example } } ``` - +% TEST[s/"italian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: italian_example, first: italian, second: rebuilt_italian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1156,7 +1179,8 @@ PUT /latvian_example } } ``` - +% TEST[s/"latvian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: latvian_example, first: latvian, second: rebuilt_latvian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1200,7 +1224,8 @@ PUT /lithuanian_example } } ``` - +% TEST[s/"lithuanian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: lithuanian_example, first: lithuanian, second: rebuilt_lithuanian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1244,7 +1269,8 @@ PUT /norwegian_example } } ``` - +% TEST[s/"norwegian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: norwegian_example, first: norwegian, second: rebuilt_norwegian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1289,7 +1315,7 @@ PUT /persian_example } } ``` - +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: persian_example, first: persian, second: rebuilt_persian}\nendyaml\n/] 1. Replaces zero-width non-joiners with an ASCII space. 2. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. @@ -1333,7 +1359,8 @@ PUT /portuguese_example } } ``` - +% TEST[s/"portuguese_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: portuguese_example, first: portuguese, second: rebuilt_portuguese}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1377,7 +1404,8 @@ PUT /romanian_example } } ``` - +% TEST[s/"romanian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: romanian_example, first: romanian, second: rebuilt_romanian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1421,7 +1449,8 @@ PUT /russian_example } } ``` - +% TEST[s/"russian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: russian_example, first: russian, second: rebuilt_russian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1466,7 +1495,8 @@ PUT /serbian_example } } ``` - +% TEST[s/"serbian_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: serbian_example, first: serbian, second: rebuilt_serbian}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1512,7 +1542,8 @@ PUT /sorani_example } } ``` - +% TEST[s/"sorani_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: sorani_example, first: sorani, second: rebuilt_sorani}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1556,7 +1587,8 @@ PUT /spanish_example } } ``` - +% TEST[s/"spanish_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: spanish_example, first: spanish, second: rebuilt_spanish}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1600,7 +1632,8 @@ PUT /swedish_example } } ``` - +% TEST[s/"swedish_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: swedish_example, first: swedish, second: rebuilt_swedish}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1649,7 +1682,8 @@ PUT /turkish_example } } ``` - +% TEST[s/"turkish_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: turkish_example, first: turkish, second: rebuilt_turkish}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. This filter should be removed unless there are words which should be excluded from stemming. @@ -1684,9 +1718,6 @@ PUT /thai_example } } ``` - +% TEST[s/"thai_keywords",//] +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: thai_example, first: thai, second: rebuilt_thai}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. - - - - diff --git a/docs/reference/data-analysis/text-analysis/analysis-mapping-charfilter.md b/docs/reference/data-analysis/text-analysis/analysis-mapping-charfilter.md index d418ff09ae649..b99d45f7a78b0 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-mapping-charfilter.md +++ b/docs/reference/data-analysis/text-analysis/analysis-mapping-charfilter.md @@ -111,6 +111,7 @@ GET /my-index-000001/_analyze "text": "I'm delighted about it :(" } ``` +% TEST[continued] The filter produces the following text: diff --git a/docs/reference/data-analysis/text-analysis/analysis-multiplexer-tokenfilter.md b/docs/reference/data-analysis/text-analysis/analysis-multiplexer-tokenfilter.md index 2a4f6c7d39608..b3e6c8d067323 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-multiplexer-tokenfilter.md +++ b/docs/reference/data-analysis/text-analysis/analysis-multiplexer-tokenfilter.md @@ -64,6 +64,7 @@ POST /multiplexer_example/_analyze "text" : "Going HOME" } ``` +% TEST[continued] And it’d respond: diff --git a/docs/reference/data-analysis/text-analysis/analysis-pathhierarchy-tokenizer.md b/docs/reference/data-analysis/text-analysis/analysis-pathhierarchy-tokenizer.md index 139d7ea7a613e..c93dceac555c2 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-pathhierarchy-tokenizer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-pathhierarchy-tokenizer.md @@ -182,6 +182,7 @@ GET file-path-test/_search } } ``` +% TEST[continued] It’s simple to match or filter documents with file paths that exist within a particular directory using the `file_path.tree` field. @@ -195,6 +196,7 @@ GET file-path-test/_search } } ``` +% TEST[continued] With the reverse parameter for this tokenizer, it’s also possible to match from the other end of the file path, such as individual file names or a deep level subdirectory. The following example shows a search for all files named `my_photo1.jpg` within any directory via the `file_path.tree_reversed` field configured to use the reverse parameter in the mapping. @@ -210,6 +212,7 @@ GET file-path-test/_search } } ``` +% TEST[continued] Viewing the tokens generated with both forward and reverse is instructive in showing the tokens created for the same file path value. @@ -226,6 +229,7 @@ POST file-path-test/_analyze "text": "/User/alice/photos/2017/05/16/my_photo1.jpg" } ``` +% TEST[continued] It’s also useful to be able to filter with file paths when combined with other types of searches, such as this example looking for any files paths with `16` that also must be in Alice’s photo directory. @@ -244,4 +248,5 @@ GET file-path-test/_search } } ``` +% TEST[continued] diff --git a/docs/reference/data-analysis/text-analysis/analysis-pattern-analyzer.md b/docs/reference/data-analysis/text-analysis/analysis-pattern-analyzer.md index 3e0e6170df319..567f9b859858d 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-pattern-analyzer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-pattern-analyzer.md @@ -183,6 +183,7 @@ PUT /pattern_example } } ``` +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: pattern_example, first: pattern, second: rebuilt_pattern}\nendyaml\n/] 1. The default pattern is `\W+` which splits on non-word characters and this is where you’d change it. 2. You’d add other token filters after `lowercase`. diff --git a/docs/reference/data-analysis/text-analysis/analysis-pattern-replace-charfilter.md b/docs/reference/data-analysis/text-analysis/analysis-pattern-replace-charfilter.md index 3fbdd968e3d71..75d4c626bb701 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-pattern-replace-charfilter.md +++ b/docs/reference/data-analysis/text-analysis/analysis-pattern-replace-charfilter.md @@ -70,6 +70,7 @@ POST my-index-000001/_analyze "text": "My credit card is 123-456-789" } ``` +% TEST[s/$1//] The above example produces the following terms: @@ -154,6 +155,7 @@ GET my-index-000001/_search } } ``` +% TEST[continued] The output from the above is: @@ -191,6 +193,7 @@ The output from the above is: } } ``` +% TESTRESPONSE[s/"took".*/"took": "$body.took",/] 1. Note the incorrect highlight. diff --git a/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md b/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md index 1c648d55998e7..6b6b495a4a3f5 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md @@ -109,6 +109,7 @@ PUT /standard_example } } ``` +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: standard_example, first: standard, second: rebuilt_standard}\nendyaml\n/] 1. You’d add any token filters after `lowercase`. diff --git a/docs/reference/data-analysis/text-analysis/analysis-stop-analyzer.md b/docs/reference/data-analysis/text-analysis/analysis-stop-analyzer.md index bf250da6e831c..a7313ba7ddb76 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-stop-analyzer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-stop-analyzer.md @@ -110,6 +110,7 @@ PUT /stop_example } } ``` +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: stop_example, first: stop, second: rebuilt_stop}\nendyaml\n/] 1. The default stopwords can be overridden with the `stopwords` or `stopwords_path` parameters. 2. You’d add any token filters after `english_stop`. diff --git a/docs/reference/data-analysis/text-analysis/analysis-whitespace-analyzer.md b/docs/reference/data-analysis/text-analysis/analysis-whitespace-analyzer.md index 361ec7ba003d4..b07589871914a 100644 --- a/docs/reference/data-analysis/text-analysis/analysis-whitespace-analyzer.md +++ b/docs/reference/data-analysis/text-analysis/analysis-whitespace-analyzer.md @@ -58,6 +58,7 @@ PUT /whitespace_example } } ``` +% TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: whitespace_example, first: whitespace, second: rebuilt_whitespace}\nendyaml\n/] 1. You’d add any token filters here. diff --git a/docs/reference/elasticsearch/command-line-tools/service-tokens-command.md b/docs/reference/elasticsearch/command-line-tools/service-tokens-command.md index 0bd6b4cc9643f..65280783b0233 100644 --- a/docs/reference/elasticsearch/command-line-tools/service-tokens-command.md +++ b/docs/reference/elasticsearch/command-line-tools/service-tokens-command.md @@ -112,6 +112,7 @@ Use this bearer token to authenticate with your {{es}} cluster. ```shell curl -H "Authorization: Bearer AAEAAWVsYXN0aWM...vZmxlZXQtc2VydmVyL3Rva2VuMTo3TFdaSDZ" http://localhost:9200/_cluster/health ``` +% NOTCONSOLE ::::{note} If your node has `xpack.security.http.ssl.enabled` set to `true`, then you must specify `https` in the request URL. diff --git a/docs/reference/elasticsearch/index-settings/index-block.md b/docs/reference/elasticsearch/index-settings/index-block.md index e292f3f0e303c..211b1dbd22638 100644 --- a/docs/reference/elasticsearch/index-settings/index-block.md +++ b/docs/reference/elasticsearch/index-settings/index-block.md @@ -47,6 +47,8 @@ Adds an index block to an index. ```console PUT /my-index-000001/_block/write ``` +% TEST[s/^/PUT my-index-000001\n/] +% TEST[setup:my_index] ### {{api-request-title}} [add-index-block-api-request] @@ -128,6 +130,7 @@ The following example shows how to add an index block: ```console PUT /my-index-000001/_block/write ``` +% TEST[setup:my_index] The API returns following response: diff --git a/docs/reference/elasticsearch/index-settings/recovery-prioritization.md b/docs/reference/elasticsearch/index-settings/recovery-prioritization.md index fe28c991552c0..679ee86cbff39 100644 --- a/docs/reference/elasticsearch/index-settings/recovery-prioritization.md +++ b/docs/reference/elasticsearch/index-settings/recovery-prioritization.md @@ -51,4 +51,5 @@ PUT index_4/_settings "index.priority": 1 } ``` +% TEST[continued] diff --git a/docs/reference/elasticsearch/index-settings/shard-allocation.md b/docs/reference/elasticsearch/index-settings/shard-allocation.md index 8403e8d7256e8..35795923b5a31 100644 --- a/docs/reference/elasticsearch/index-settings/shard-allocation.md +++ b/docs/reference/elasticsearch/index-settings/shard-allocation.md @@ -105,4 +105,5 @@ PUT test/_settings "index.routing.allocation.include._ip": "192.168.2.*" } ``` +% TEST[skip:indexes don’t assign] diff --git a/docs/reference/elasticsearch/index-settings/similarity.md b/docs/reference/elasticsearch/index-settings/similarity.md index b669a7d09ac0b..bbff9b3f6922e 100644 --- a/docs/reference/elasticsearch/index-settings/similarity.md +++ b/docs/reference/elasticsearch/index-settings/similarity.md @@ -46,6 +46,7 @@ PUT /index/_mapping } } ``` +% TEST[continued] ## Available similarities [_available_similarities] @@ -276,6 +277,8 @@ Which yields: } } ``` +% TESTRESPONSE[s/"took": 12/"took" : $body.took/] +% TESTRESPONSE[s/OzrdjxNtQGaqs4DmioFw9A/$body.hits.hits.0._node/] ::::{warning} While scripted similarities provide a lot of flexibility, there is a set of rules that they need to satisfy. Failing to do so could make Elasticsearch silently return wrong top hits or fail with internal errors at search time: @@ -360,4 +363,5 @@ PUT /index/_settings POST /index/_open ``` +% TEST[continued] diff --git a/docs/reference/elasticsearch/index-settings/slow-log.md b/docs/reference/elasticsearch/index-settings/slow-log.md index b5a5ef13bd41c..b8c87f0f50a19 100644 --- a/docs/reference/elasticsearch/index-settings/slow-log.md +++ b/docs/reference/elasticsearch/index-settings/slow-log.md @@ -53,6 +53,7 @@ If a call was initiated with an `X-Opaque-ID` header, then the ID is automatical "user.realm": "reserved" } ``` +% NOTCONSOLE The following is an example of an indexing event in the slow log: @@ -79,6 +80,7 @@ The following is an example of an indexing event in the slow log: "user.realm": "reserved" } ``` +% NOTCONSOLE ## Enable slow logging [enable-slow-log] @@ -137,6 +139,7 @@ PUT /my-index-000001/_settings "index.search.slowlog.include.user": true } ``` +% TEST[setup:my_index] ### Enable slow logging for indexing events [index-slow-log] @@ -173,6 +176,7 @@ PUT /my-index-000001/_settings "index.indexing.slowlog.include.user": true } ``` +% TEST[setup:my_index] #### Logging the `_source` field [_logging_the_source_field] diff --git a/docs/reference/elasticsearch/index-settings/sorting.md b/docs/reference/elasticsearch/index-settings/sorting.md index 567aa876d2260..feaecd078bd81 100644 --- a/docs/reference/elasticsearch/index-settings/sorting.md +++ b/docs/reference/elasticsearch/index-settings/sorting.md @@ -136,6 +136,7 @@ GET /events/_search ] } ``` +% TEST[continued] Elasticsearch will detect that the top docs of each segment are already sorted in the index and will only compare the first N documents per segment. The rest of the documents matching the query are collected to count the total number of results and to build aggregations. @@ -151,10 +152,10 @@ GET /events/_search "track_total_hits": false } ``` +% TEST[continued] 1. The index sort will be used to rank the top documents and each segment will early terminate the collection after the first 10 matches. - This time, Elasticsearch will not try to count the number of documents and will be able to terminate the query as soon as N documents have been collected per segment. ```console-result @@ -168,10 +169,11 @@ This time, Elasticsearch will not try to count the number of documents and will "timed_out": false } ``` +% TESTRESPONSE[s/"_shards": .../"_shards": "$body._shards",/] +% TESTRESPONSE[s/"took": 20,/"took": "$body.took",/] 1. The total number of hits matching the query is unknown because of early termination. - ::::{note} Aggregations will collect all documents that match the query regardless of the value of `track_total_hits` :::: diff --git a/docs/reference/elasticsearch/mapping-reference/aggregate-metric-double.md b/docs/reference/elasticsearch/mapping-reference/aggregate-metric-double.md index 643f6d0b44f25..297e7ada41740 100644 --- a/docs/reference/elasticsearch/mapping-reference/aggregate-metric-double.md +++ b/docs/reference/elasticsearch/mapping-reference/aggregate-metric-double.md @@ -115,6 +115,8 @@ PUT stats-index/_doc/2 } } ``` +% TEST[continued] +% TEST[s/_doc/2/_doc/2?refresh=wait_for/] You can run `min`, `max`, `sum`, `value_count`, and `avg` aggregations on a `agg_metric` field. @@ -130,6 +132,7 @@ POST stats-index/_search?size=0 } } ``` +% TEST[continued] The aggregation results are based on related metric sub-field values. @@ -155,6 +158,7 @@ The aggregation results are based on related metric sub-field values. } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/] Queries on a `aggregate_metric_double` field use the `default_metric` value. @@ -170,6 +174,7 @@ GET stats-index/_search } } ``` +% TEST[continued] The search returns the following hit. The value of the `default_metric` field, `max`, matches the query value. @@ -200,6 +205,7 @@ The search returns the following hit. The value of the `default_metric` field, ` } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,/] ## Synthetic `_source` [aggregate-metric-double-synthetic-source] @@ -246,6 +252,7 @@ PUT idx/_doc/1 } } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: diff --git a/docs/reference/elasticsearch/mapping-reference/binary.md b/docs/reference/elasticsearch/mapping-reference/binary.md index cd2649ba5c9ed..b6d4f83f6c946 100644 --- a/docs/reference/elasticsearch/mapping-reference/binary.md +++ b/docs/reference/elasticsearch/mapping-reference/binary.md @@ -79,6 +79,7 @@ PUT idx/_doc/1 "binary": ["IAA=", "EAA="] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -87,5 +88,6 @@ Will become: "binary": ["EAA=", "IAA="] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/boolean.md b/docs/reference/elasticsearch/mapping-reference/boolean.md index 3b89832213084..5227d1c687c2b 100644 --- a/docs/reference/elasticsearch/mapping-reference/boolean.md +++ b/docs/reference/elasticsearch/mapping-reference/boolean.md @@ -43,6 +43,7 @@ GET my-index-000001/_search } } ``` +% TEST[s/_search/_search?filter_path=hits.hits/] 1. Indexing a document with `"true"`, which is interpreted as `true`. 2. Searching for documents with a JSON `true`. @@ -82,6 +83,7 @@ GET my-index-000001/_search } } ``` +% TEST[s/_search/_search?filter_path=aggregations,hits.hits/] ## Parameters for `boolean` fields [boolean-params] @@ -160,6 +162,7 @@ PUT idx/_doc/1 "bool": [true, false, true, false] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -168,5 +171,6 @@ Will become: "bool": [false, false, true, true] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/coerce.md b/docs/reference/elasticsearch/mapping-reference/coerce.md index 673be8656bc7a..f8aa385b54caa 100644 --- a/docs/reference/elasticsearch/mapping-reference/coerce.md +++ b/docs/reference/elasticsearch/mapping-reference/coerce.md @@ -40,6 +40,7 @@ PUT my-index-000001/_doc/2 "number_two": "10" <2> } ``` +% TEST[catch:bad_request] 1. The `number_one` field will contain the integer `10`. 2. This document will be rejected because coercion is disabled. @@ -79,6 +80,7 @@ PUT my-index-000001/_doc/1 PUT my-index-000001/_doc/2 { "number_two": "10" } <2> ``` +% TEST[catch:bad_request] 1. The `number_one` field overrides the index level setting to enable coercion. 2. This document will be rejected because the `number_two` field inherits the index-level coercion setting. diff --git a/docs/reference/elasticsearch/mapping-reference/date.md b/docs/reference/elasticsearch/mapping-reference/date.md index 2a828ae98e38f..5c817ecb64bcb 100644 --- a/docs/reference/elasticsearch/mapping-reference/date.md +++ b/docs/reference/elasticsearch/mapping-reference/date.md @@ -32,6 +32,7 @@ Date formats can be customised, but if no `format` is specified then it uses the ```js "strict_date_optional_time||epoch_millis" ``` +% NOTCONSOLE This means that it will accept dates with optional timestamps, which conform to the formats supported by [`strict_date_optional_time`](/reference/elasticsearch/mapping-reference/mapping-date-format.md#strict-date-time) or milliseconds-since-the-epoch. @@ -163,6 +164,7 @@ POST my-index-000001/_search "_source": false } ``` +% TEST[s/_search/_search?filter_path=hits.hits/] Which will reply with a date like: @@ -218,6 +220,7 @@ PUT idx/_doc/1 "date": ["2015-01-01T12:10:30Z", "2014-01-01T12:10:30Z"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -226,5 +229,6 @@ Will become: "date": ["2014-01-01T12:10:30.000Z", "2015-01-01T12:10:30.000Z"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/date_nanos.md b/docs/reference/elasticsearch/mapping-reference/date_nanos.md index b5db585def0e0..13c4d8f5fb46d 100644 --- a/docs/reference/elasticsearch/mapping-reference/date_nanos.md +++ b/docs/reference/elasticsearch/mapping-reference/date_nanos.md @@ -16,6 +16,7 @@ Date formats can be customised, but if no `format` is specified then it uses the ```js "strict_date_optional_time_nanos||epoch_millis" ``` +% NOTCONSOLE For instance: @@ -59,6 +60,7 @@ GET my-index-000001/_search ] } ``` +% TEST[s/_search/_search?filter_path=hits.hits/] 1. The `date` field uses the default `format`. 2. This document uses a plain date. @@ -117,6 +119,7 @@ PUT idx/_doc/1 "date": ["2015-01-01T12:10:30.000Z", "2014-01-01T12:10:30.000Z"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -125,5 +128,6 @@ Will become: "date": ["2014-01-01T12:10:30.000Z", "2015-01-01T12:10:30.000Z"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/dense-vector.md b/docs/reference/elasticsearch/mapping-reference/dense-vector.md index 13e8706b3c120..4082e15d3540c 100644 --- a/docs/reference/elasticsearch/mapping-reference/dense-vector.md +++ b/docs/reference/elasticsearch/mapping-reference/dense-vector.md @@ -352,6 +352,7 @@ POST /my-bit-vectors/_bulk?refresh {"index": {"_id" : "2"}} {"my_vector": "8100012a7f"} <2> ``` +% TEST[continued] 1. 5 bytes representing the 40 bit dimensioned vector 2. A hexidecimal string representing the 40 bit dimensioned vector @@ -370,6 +371,7 @@ POST /my-bit-vectors/_search?filter_path=hits.hits } } ``` +% TEST[continued] ```console-result { @@ -452,6 +454,7 @@ PUT /my-index-000001/_mapping } } ``` +% TEST[setup:my_index] Vectors indexed before this change will keep using the `flat` type (raw float32 representation and brute force search for KNN queries). diff --git a/docs/reference/elasticsearch/mapping-reference/eager-global-ordinals.md b/docs/reference/elasticsearch/mapping-reference/eager-global-ordinals.md index 47da87c7f5719..ba5a4274b8e19 100644 --- a/docs/reference/elasticsearch/mapping-reference/eager-global-ordinals.md +++ b/docs/reference/elasticsearch/mapping-reference/eager-global-ordinals.md @@ -40,6 +40,7 @@ PUT my-index-000001/_mapping } } ``` +% TEST[s/^/PUT my-index-000001\n/] When `eager_global_ordinals` is enabled, global ordinals are built when a shard is [refreshed](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-refresh) — Elasticsearch always loads them before exposing changes to the content of the index. This shifts the cost of building global ordinals from search to index-time. Elasticsearch will also eagerly build global ordinals when creating a new copy of a shard, as can occur when increasing the number of replicas or relocating a shard onto a new node. @@ -56,6 +57,7 @@ PUT my-index-000001/_mapping } } ``` +% TEST[continued] ## Avoiding global ordinal loading [_avoiding_global_ordinal_loading] diff --git a/docs/reference/elasticsearch/mapping-reference/field-alias.md b/docs/reference/elasticsearch/mapping-reference/field-alias.md index 8e3f49cc8b9de..18e360f30a8f5 100644 --- a/docs/reference/elasticsearch/mapping-reference/field-alias.md +++ b/docs/reference/elasticsearch/mapping-reference/field-alias.md @@ -50,6 +50,7 @@ In some parts of the search request and when requesting field capabilities, fiel ```console GET trips/_field_caps?fields=route_*,transit_mode ``` +% TEST[continued] ## Alias targets [alias-targets] @@ -79,6 +80,7 @@ GET /_search "_source": "route_length_miles" } ``` +% TEST[continued] Currently only the search and field capabilities APIs will accept and resolve field aliases. Other APIs that accept field names, such as [term vectors](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-termvectors), cannot be used with field aliases. diff --git a/docs/reference/elasticsearch/mapping-reference/flattened.md b/docs/reference/elasticsearch/mapping-reference/flattened.md index 9c0002d236017..05d68d8139101 100644 --- a/docs/reference/elasticsearch/mapping-reference/flattened.md +++ b/docs/reference/elasticsearch/mapping-reference/flattened.md @@ -50,6 +50,7 @@ POST bug_reports/_doc/1 } } ``` +% TESTSETUP During indexing, tokens are created for each leaf value in the JSON object. The values are indexed as string keywords, without analysis or special handling for numbers or dates. @@ -126,6 +127,9 @@ POST my-index-000001/_search "_source": false } ``` +% TESTRESPONSE[s/"took": 2/"took": $body.took/] +% TESTRESPONSE[s/"max_score" : 1.0/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 1.0/"_score" : $body.hits.hits.0._score/] ```console-result { @@ -154,6 +158,9 @@ POST my-index-000001/_search } } ``` +% TESTRESPONSE[s/"took": 2/"took": $body.took/] +% TESTRESPONSE[s/"max_score" : 1.0/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 1.0/"_score" : $body.hits.hits.0._score/] You can also use a [Painless script](docs-content://explore-analyze/scripting/modules-scripting-painless.md) to retrieve values from sub-fields of flattened fields. Instead of including `doc[''].value` in your Painless script, use `doc['.'].value`. For example, if you have a flattened field called `label` with a `release` sub-field, your Painless script would be `doc['labels.release'].value`. @@ -186,6 +193,7 @@ POST /my-index-000001/_bulk?refresh {"index":{}} {"title":"Not urgent","labels":{"priority":"low","release":["v1.2.0"],"timestamp":{"created":1541458026,"closed":1541457010}}} ``` +% TEST[continued] Because `labels` is a `flattened` field type, the entire object is mapped as a single field. To retrieve values from this sub-field in a Painless script, use the `doc['.'].value` format. @@ -272,6 +280,7 @@ PUT idx/_doc/1 } } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -282,6 +291,7 @@ Will become: } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] Synthetic source always uses nested objects instead of array of objects. For example: @@ -316,6 +326,7 @@ PUT idx/_doc/1 } } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become (note the nested objects instead of the "flattened" array): @@ -329,6 +340,7 @@ Will become (note the nested objects instead of the "flattened" array): } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] Synthetic source always uses single-valued fields for one-element arrays. For example: @@ -359,6 +371,7 @@ PUT idx/_doc/1 } } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become (note the nested objects instead of the "flattened" array): @@ -369,3 +382,4 @@ Will become (note the nested objects instead of the "flattened" array): } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/geo-point.md b/docs/reference/elasticsearch/mapping-reference/geo-point.md index f683eb55d98a7..570cd31c18489 100644 --- a/docs/reference/elasticsearch/mapping-reference/geo-point.md +++ b/docs/reference/elasticsearch/mapping-reference/geo-point.md @@ -193,6 +193,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -204,5 +205,6 @@ Will become: ] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/geo-shape.md b/docs/reference/elasticsearch/mapping-reference/geo-shape.md index 11301d518f183..1ca759f26203f 100644 --- a/docs/reference/elasticsearch/mapping-reference/geo-shape.md +++ b/docs/reference/elasticsearch/mapping-reference/geo-shape.md @@ -64,6 +64,7 @@ PUT /example } } ``` +% TESTSETUP ### Input Structure [input-structure] diff --git a/docs/reference/elasticsearch/mapping-reference/ignore-malformed.md b/docs/reference/elasticsearch/mapping-reference/ignore-malformed.md index b5ae850adfce4..2fdef3e7f2e63 100644 --- a/docs/reference/elasticsearch/mapping-reference/ignore-malformed.md +++ b/docs/reference/elasticsearch/mapping-reference/ignore-malformed.md @@ -39,6 +39,7 @@ PUT my-index-000001/_doc/2 "number_two": "foo" <2> } ``` +% TEST[catch:bad_request] 1. This document will have the `text` field indexed, but not the `number_one` field. 2. This document will be rejected because `number_two` does not allow malformed values. diff --git a/docs/reference/elasticsearch/mapping-reference/index-prefixes.md b/docs/reference/elasticsearch/mapping-reference/index-prefixes.md index 3bf1e2133bc0c..c8fec42bdc360 100644 --- a/docs/reference/elasticsearch/mapping-reference/index-prefixes.md +++ b/docs/reference/elasticsearch/mapping-reference/index-prefixes.md @@ -72,4 +72,5 @@ GET my-index-000001/_search } } ``` +% TEST[continued] diff --git a/docs/reference/elasticsearch/mapping-reference/ip.md b/docs/reference/elasticsearch/mapping-reference/ip.md index 68c5e57e1ef25..db8c13c0c90f6 100644 --- a/docs/reference/elasticsearch/mapping-reference/ip.md +++ b/docs/reference/elasticsearch/mapping-reference/ip.md @@ -35,6 +35,7 @@ GET my-index-000001/_search } } ``` +% TESTSETUP ::::{note} You can also store ip ranges in a single field using an [ip_range data type](/reference/elasticsearch/mapping-reference/range.md). @@ -156,6 +157,7 @@ PUT idx/_doc/1 "2001:db8::1:0:0:1", "::afff:4567:890a"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -164,6 +166,7 @@ Will become: "ip": ["::afff:4567:890a", "10.10.12.123", "192.168.0.1", "2001:db8::1:0:0:1"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] ::::{note} IPv4 addresses are sorted as though they were IPv6 addresses prefixed by `::ffff:0:0:0/96` as specified by [rfc6144](https://datatracker.ietf.org/doc/html/rfc6144). diff --git a/docs/reference/elasticsearch/mapping-reference/keyword.md b/docs/reference/elasticsearch/mapping-reference/keyword.md index 7003c0fd784e8..b3a2abcb3d6f6 100644 --- a/docs/reference/elasticsearch/mapping-reference/keyword.md +++ b/docs/reference/elasticsearch/mapping-reference/keyword.md @@ -154,6 +154,7 @@ PUT idx/_doc/1 "kwd": ["foo", "foo", "bar", "baz"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -162,6 +163,7 @@ Will become: "kwd": ["bar", "baz", "foo"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] If a `keyword` field sets `store` to `true` then order and duplicates are preserved. For example: @@ -190,6 +192,7 @@ PUT idx/_doc/1 "kwd": ["foo", "foo", "bar", "baz"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -198,6 +201,7 @@ Will become: "kwd": ["foo", "foo", "bar", "baz"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] Values longer than `ignore_above` are preserved but sorted to the end. For example: @@ -226,6 +230,7 @@ PUT idx/_doc/1 "kwd": ["foo", "foo", "bang", "bar", "baz"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -234,6 +239,7 @@ Will become: "kwd": ["bar", "baz", "foo", "bang"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] ## Constant keyword field type [constant-keyword-field-type] @@ -278,6 +284,7 @@ POST logs-debug/_doc "message": "Starting up Elasticsearch" } ``` +% TEST[continued] However providing a value that is different from the one configured in the mapping is disallowed. @@ -414,6 +421,7 @@ PUT idx/_doc/1 "card": ["king", "ace", "ace", "jack"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -422,3 +430,4 @@ Will become: "card": ["ace", "jack", "king"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/mapping-doc-count-field.md b/docs/reference/elasticsearch/mapping-reference/mapping-doc-count-field.md index 99896d2765783..9efce653b5309 100644 --- a/docs/reference/elasticsearch/mapping-reference/mapping-doc-count-field.md +++ b/docs/reference/elasticsearch/mapping-reference/mapping-doc-count-field.md @@ -106,5 +106,6 @@ We will get the following response: } } ``` +% TESTRESPONSE[skip:test not setup] diff --git a/docs/reference/elasticsearch/mapping-reference/mapping-field-meta.md b/docs/reference/elasticsearch/mapping-reference/mapping-field-meta.md index 33a467e6fbc43..d11189f43e7b1 100644 --- a/docs/reference/elasticsearch/mapping-reference/mapping-field-meta.md +++ b/docs/reference/elasticsearch/mapping-reference/mapping-field-meta.md @@ -22,6 +22,7 @@ PUT my-index-000001 } } ``` +% TEST ::::{note} Field metadata enforces at most 5 entries, that keys have a length that is less than or equal to 20, and that values are strings whose length is less than or equal to 50. diff --git a/docs/reference/elasticsearch/mapping-reference/mapping-meta-field.md b/docs/reference/elasticsearch/mapping-reference/mapping-meta-field.md index 925f2ee108cb5..9d6dc572292e4 100644 --- a/docs/reference/elasticsearch/mapping-reference/mapping-meta-field.md +++ b/docs/reference/elasticsearch/mapping-reference/mapping-meta-field.md @@ -39,4 +39,5 @@ PUT my-index-000001/_mapping } } ``` +% TEST[continued] diff --git a/docs/reference/elasticsearch/mapping-reference/mapping-routing-field.md b/docs/reference/elasticsearch/mapping-reference/mapping-routing-field.md index 0f44ef5a279bc..5eb999f93b10a 100644 --- a/docs/reference/elasticsearch/mapping-reference/mapping-routing-field.md +++ b/docs/reference/elasticsearch/mapping-reference/mapping-routing-field.md @@ -23,6 +23,7 @@ PUT my-index-000001/_doc/1?routing=user1&refresh=true <1> GET my-index-000001/_doc/1?routing=user1 <2> ``` +% TESTSETUP 1. This document uses `user1` as its routing value, instead of its ID. 2. The same `routing` value needs to be provided when [getting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get), [deleting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-delete), or [updating](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update) the document. @@ -89,6 +90,7 @@ PUT my-index-000002/_doc/1 <2> "text": "No routing value provided" } ``` +% TEST[catch:bad_request] 1. Routing is required for all documents. 2. This index request throws a `routing_missing_exception`. diff --git a/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md b/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md index a5e0b95af48e3..299d6dfdf8a03 100644 --- a/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md +++ b/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md @@ -32,6 +32,7 @@ PUT idx } } ``` +% TESTSETUP While this on-the-fly reconstruction is *generally* slower than saving the source documents verbatim and loading them at query time, it saves a lot of storage space. Additional latency can be avoided by not loading `_source` field in queries when it is not needed. @@ -73,6 +74,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -83,6 +85,7 @@ Will become: } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] This can cause some arrays to vanish: @@ -101,6 +104,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -112,6 +116,7 @@ Will become: } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] #### Fields named as they are mapped [synthetic-source-modifications-field-names] @@ -126,6 +131,7 @@ PUT idx/_doc/1 "foo.bar.baz": 1 } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -138,24 +144,28 @@ Will become: } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] This impacts how source contents can be referenced in [scripts](docs-content://explore-analyze/scripting/modules-scripting-using.md). For instance, referencing a script in its original source form will return null: ```js "script": { "source": """ emit(params._source['foo.bar.baz']) """ } ``` +% NOTCONSOLE Instead, source references need to be in line with the mapping structure: ```js "script": { "source": """ emit(params._source['foo']['bar']['baz']) """ } ``` +% NOTCONSOLE or simply ```js "script": { "source": """ emit(params._source.foo.bar.baz) """ } ``` +% NOTCONSOLE The following [field APIs](docs-content://explore-analyze/scripting/modules-scripting-fields.md) are preferable as, in addition to being agnostic to the mapping structure, they make use of docvalues if available and fall back to synthetic source only when needed. This reduces source synthesizing, a slow and costly operation. @@ -163,6 +173,7 @@ The following [field APIs](docs-content://explore-analyze/scripting/modules-scri "script": { "source": """ emit(field('foo.bar.baz').get(null)) """ } "script": { "source": """ emit($('foo.bar.baz', null)) """ } ``` +% NOTCONSOLE #### Alphabetical sorting [synthetic-source-modifications-alphabetical] @@ -218,6 +229,7 @@ PUT idx_keep } } ``` +% TEST $$$synthetic-source-keep-example$$$ @@ -234,6 +246,7 @@ PUT idx_keep/_doc/1 "ids": [ 200, 100, 300, 100 ] } ``` +% TEST[s/$/\nGET idx_keep/_doc/1?filter_path=_source\n/] returns the original source, with no array deduplication and sorting: @@ -249,6 +262,7 @@ returns the original source, with no array deduplication and sorting: "ids": [ 200, 100, 300, 100 ] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] The option for capturing the source of arrays can be applied at index level, by setting `index.mapping.synthetic_source_keep` to `arrays`. This applies to all objects and fields in the index, except for the ones with explicit overrides of `synthetic_source_keep` set to `none`. In this case, the storage overhead grows with the number and sizes of arrays present in source of each document, naturally. diff --git a/docs/reference/elasticsearch/mapping-reference/nested.md b/docs/reference/elasticsearch/mapping-reference/nested.md index 1d68765b7ebe3..6e2ec4b36b450 100644 --- a/docs/reference/elasticsearch/mapping-reference/nested.md +++ b/docs/reference/elasticsearch/mapping-reference/nested.md @@ -52,6 +52,7 @@ The previous document would be transformed internally into a document that looks "user.last" : [ "smith", "white" ] } ``` +% NOTCONSOLE The `user.first` and `user.last` fields are flattened into multi-value fields, and the association between `alice` and `white` is lost. This document would incorrectly match a query for `alice AND smith`: @@ -68,6 +69,7 @@ GET my-index-000001/_search } } ``` +% TEST[continued] ## Using `nested` fields for arrays of objects [nested-fields-array-objects] diff --git a/docs/reference/elasticsearch/mapping-reference/normalizer.md b/docs/reference/elasticsearch/mapping-reference/normalizer.md index c231ea31ac6ee..3054b305a5dbc 100644 --- a/docs/reference/elasticsearch/mapping-reference/normalizer.md +++ b/docs/reference/elasticsearch/mapping-reference/normalizer.md @@ -110,6 +110,7 @@ The above queries match documents 1 and 2 since `BÀR` is converted to `bar` at } } ``` +% TESTRESPONSE[s/"took".*/"took": "$body.took",/] Also, the fact that keywords are converted prior to indexing also means that aggregations return normalized values: @@ -126,6 +127,7 @@ GET index/_search } } ``` +% TEST[continued] returns @@ -165,4 +167,5 @@ returns } } ``` +% TESTRESPONSE[s/"took".*/"took": "$body.took",/] diff --git a/docs/reference/elasticsearch/mapping-reference/norms.md b/docs/reference/elasticsearch/mapping-reference/norms.md index aa004d6716dfa..bcdc82a136249 100644 --- a/docs/reference/elasticsearch/mapping-reference/norms.md +++ b/docs/reference/elasticsearch/mapping-reference/norms.md @@ -27,6 +27,7 @@ PUT my-index-000001/_mapping } } ``` +% TEST[s/^/PUT my-index-000001\n/] ::::{note} Norms will not be removed instantly, but will be removed as old segments are merged into new segments as you continue indexing new documents. Any score computation on a field that has had norms removed might return inconsistent results since some documents won’t have norms anymore while other documents might still have norms. diff --git a/docs/reference/elasticsearch/mapping-reference/number.md b/docs/reference/elasticsearch/mapping-reference/number.md index d9ef86af21a84..805ecfa61ae2f 100644 --- a/docs/reference/elasticsearch/mapping-reference/number.md +++ b/docs/reference/elasticsearch/mapping-reference/number.md @@ -215,6 +215,7 @@ PUT idx/_doc/1 "long": [0, 0, -123466, 87612] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -223,6 +224,7 @@ Will become: "long": [-123466, 0, 0, 87612] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] Scaled floats will always apply their scaling factor so: @@ -251,6 +253,7 @@ PUT idx/_doc/1 "f": 123 } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -259,5 +262,6 @@ Will become: "f": 100.0 } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/object.md b/docs/reference/elasticsearch/mapping-reference/object.md index f989c834869fe..6cf0264411f67 100644 --- a/docs/reference/elasticsearch/mapping-reference/object.md +++ b/docs/reference/elasticsearch/mapping-reference/object.md @@ -38,6 +38,7 @@ Internally, this document is indexed as a simple, flat list of key-value pairs, "manager.name.last": "Smith" } ``` +% NOTCONSOLE An explicit mapping for the above document could look like this: diff --git a/docs/reference/elasticsearch/mapping-reference/parent-join.md b/docs/reference/elasticsearch/mapping-reference/parent-join.md index 11f71147eeb2e..ed246a5a3d595 100644 --- a/docs/reference/elasticsearch/mapping-reference/parent-join.md +++ b/docs/reference/elasticsearch/mapping-reference/parent-join.md @@ -60,10 +60,10 @@ PUT my-index-000001/_doc/2?refresh } } ``` +% TEST[continued] 1. This document is a `question` document. - When indexing parent documents, you can choose to specify just the name of the relation as a shortcut instead of encapsulating it in the normal object notation: ```console @@ -81,10 +81,10 @@ PUT my-index-000001/_doc/2?refresh "my_join_field": "question" } ``` +% TEST[continued] 1. Simpler notation for a parent document just uses the relation name. - When indexing a child, the name of the relation as well as the parent id of the document must be added in the `_source`. ::::{warning} @@ -115,12 +115,12 @@ PUT my-index-000001/_doc/4?routing=1&refresh } } ``` +% TEST[continued] 1. The routing value is mandatory because parent and child documents must be indexed on the same shard 2. `answer` is the name of the join for this document 3. The parent id of this child document - ## Parent-join and performance [_parent_join_and_performance] The join field shouldn’t be used like joins in a relation database. In Elasticsearch the key to good performance is to de-normalize your data into documents. Each join field, `has_child` or `has_parent` query adds a significant tax to your query performance. It can also trigger [global ordinals](/reference/elasticsearch/mapping-reference/eager-global-ordinals.md) to be built. @@ -156,6 +156,7 @@ GET my-index-000001/_search "sort": ["my_id"] } ``` +% TEST[continued] Will return: @@ -233,14 +234,13 @@ Will return: } } ``` +% TESTRESPONSE[s/.../"timed_out": false, "took": $body.took, "_shards": $body._shards/] 1. This document belongs to the `question` join 2. This document belongs to the `question` join 3. This document belongs to the `answer` join 4. The linked parent id for the child document - - ## Parent-join queries and aggregations [_parent_join_queries_and_aggregations] See the [`has_child`](/reference/query-languages/query-dsl-has-child-query.md) and [`has_parent`](/reference/query-languages/query-dsl-has-parent-query.md) queries, the [`children`](/reference/data-analysis/aggregations/search-aggregations-bucket-children-aggregation.md) aggregation, and [inner hits](/reference/elasticsearch/rest-apis/retrieve-inner-hits.md#parent-child-inner-hits) for more information. @@ -277,13 +277,13 @@ GET my-index-000001/_search ] } ``` +% TEST[continued] +% TEST[s/_search/_search?filter_path=aggregations,hits.hits&sort=my_id/] 1. Querying the `parent id` field (also see the [`has_parent` query](/reference/query-languages/query-dsl-has-parent-query.md) and the [`has_child` query](/reference/query-languages/query-dsl-has-child-query.md)) 2. Aggregating on the `parent id` field (also see the [`children`](/reference/data-analysis/aggregations/search-aggregations-bucket-children-aggregation.md) aggregation) 3. Accessing the `parent id` field in scripts. - - ## Global ordinals [_global_ordinals] The `join` field uses [global ordinals](/reference/elasticsearch/mapping-reference/eager-global-ordinals.md) to speed up joins. Global ordinals need to be rebuilt after any change to a shard. The more parent id values are stored in a shard, the longer it takes to rebuild the global ordinals for the `join` field. @@ -318,6 +318,7 @@ GET _stats/fielddata?human&fields=my_join_field#question # Per-node per-index GET _nodes/stats/indices/fielddata?human&fields=my_join_field#question ``` +% TEST[continued] ## Multiple children per parent [_multiple_children_per_parent] @@ -397,9 +398,11 @@ PUT my-index-000001/_doc/3?routing=1&refresh <1> } } ``` +% TEST[continued] 1. This child document must be on the same shard than its grand-parent and parent 2. The parent id of this document (must points to an `answer` document) + diff --git a/docs/reference/elasticsearch/mapping-reference/passthrough.md b/docs/reference/elasticsearch/mapping-reference/passthrough.md index 0d30a7b2f4984..0272583a21ec5 100644 --- a/docs/reference/elasticsearch/mapping-reference/passthrough.md +++ b/docs/reference/elasticsearch/mapping-reference/passthrough.md @@ -163,6 +163,7 @@ POST metrics-mymetrics-test/_doc "cpu": 10 } ``` +% TEST[skip: The @timestamp value won’t match an accepted range in the TSDS] In the example above, `attributes` is defined as a dimension container. Its sub-fields `host.name` (static) and `zone` (dynamic) get included in the routing path and tsid, and can be referenced in queries without the `attributes.` prefix. diff --git a/docs/reference/elasticsearch/mapping-reference/percolator.md b/docs/reference/elasticsearch/mapping-reference/percolator.md index 0756c9b999cb0..1a3cfdb259501 100644 --- a/docs/reference/elasticsearch/mapping-reference/percolator.md +++ b/docs/reference/elasticsearch/mapping-reference/percolator.md @@ -28,6 +28,7 @@ PUT my-index-000001 } } ``` +% TESTSETUP Then you can index a query: @@ -91,6 +92,7 @@ PUT queries/_doc/1?refresh } } ``` +% TEST[continued] 1. It is always recommended to define an alias for your index, so that in case of a reindex systems / applications don’t need to be changed to know that the percolator queries are now in a different index. @@ -140,6 +142,7 @@ POST _aliases ] } ``` +% TEST[continued] 1. If you have an alias don’t forget to point it to the new index. @@ -159,6 +162,7 @@ GET /queries/_search } } ``` +% TEST[continued] now returns matches from the new index: @@ -198,6 +202,7 @@ now returns matches from the new index: } } ``` +% TESTRESPONSE[s/"took": 3,/"took": "$body.took",/] 1. Percolator query hit is now being presented from the new index. @@ -224,6 +229,7 @@ Lets say we want to index the following percolator query: } } ``` +% NOTCONSOLE with these settings and mapping: @@ -253,6 +259,7 @@ PUT /test_index } } ``` +% TEST[continued] 1. For the purpose of this example, this analyzer is considered expensive. @@ -266,6 +273,7 @@ POST /test_index/_analyze "text" : "missing bicycles" } ``` +% TEST[continued] This results the following response: @@ -305,6 +313,7 @@ PUT /test_index/_doc/1?refresh } } ``` +% TEST[continued] 1. It is important to select a whitespace analyzer here, otherwise the analyzer defined in the mapping will be used, which defeats the point of using this workflow. Note that `whitespace` is a built-in analyzer, if a different analyzer needs to be used, it needs to be configured first in the index’s settings. @@ -326,6 +335,7 @@ GET /test_index/_search } } ``` +% TEST[continued] This results in a response like this: @@ -368,6 +378,7 @@ This results in a response like this: } } ``` +% TESTRESPONSE[s/"took": 6,/"took": "$body.took",/] ## Optimizing wildcard queries. [_optimizing_wildcard_queries] @@ -421,6 +432,7 @@ PUT my_queries1 } } ``` +% TEST[continued] 1. The analyzer that generates the prefix tokens to be used at index time only. 2. Increase the `min_gram` and decrease `max_gram` settings based on your prefix search needs. @@ -438,6 +450,7 @@ Then instead of indexing the following query: } } ``` +% NOTCONSOLE this query below should be indexed: @@ -451,6 +464,7 @@ PUT /my_queries1/_doc/1?refresh } } ``` +% TEST[continued] This way can handle the second query more efficiently than the first query. @@ -469,6 +483,7 @@ GET /my_queries1/_search } } ``` +% TEST[continued] ```console-result { @@ -508,6 +523,7 @@ GET /my_queries1/_search } } ``` +% TESTRESPONSE[s/"took": 6,/"took": "$body.took",/] The same technique can also be used to speed up suffix wildcard searches. By using the `reverse` token filter before the `edge_ngram` token filter. @@ -563,6 +579,7 @@ PUT my_queries2 } } ``` +% TEST[continued] 1. A custom analyzer is needed at search time too, because otherwise the query terms are not being reversed and would otherwise not match with the reserved suffix tokens. @@ -578,6 +595,7 @@ Then instead of indexing the following query: } } ``` +% NOTCONSOLE the following query below should be indexed: @@ -591,6 +609,7 @@ PUT /my_queries2/_doc/2?refresh } } ``` +% TEST[continued] 1. The `match` query should be used instead of the `term` query, because text analysis needs to reverse the query terms. @@ -610,6 +629,7 @@ GET /my_queries2/_search } } ``` +% TEST[continued] ## Dedicated Percolator Index [_dedicated_percolator_index] diff --git a/docs/reference/elasticsearch/mapping-reference/properties.md b/docs/reference/elasticsearch/mapping-reference/properties.md index 4d8a6a6c5bfe8..07dce9352ae19 100644 --- a/docs/reference/elasticsearch/mapping-reference/properties.md +++ b/docs/reference/elasticsearch/mapping-reference/properties.md @@ -95,6 +95,7 @@ GET my-index-000001/_search } } ``` +% TEST[continued] ::::{important} The full path to the inner field must be specified. diff --git a/docs/reference/elasticsearch/mapping-reference/range.md b/docs/reference/elasticsearch/mapping-reference/range.md index 7fbda52ea3edc..8227268d88c88 100644 --- a/docs/reference/elasticsearch/mapping-reference/range.md +++ b/docs/reference/elasticsearch/mapping-reference/range.md @@ -62,6 +62,7 @@ PUT range_index/_doc/1?refresh } } ``` +% TESTSETUP 1. `date_range` types accept the same field parameters defined by the [`date`](/reference/elasticsearch/mapping-reference/date.md) type. 2. Example indexing a meeting with 10 to 20 attendees, not including 20. @@ -119,6 +120,7 @@ The result produced by the above query. } } ``` +% TESTRESPONSE[s/"took": 13/"took" : $body.took/] The following is an example of a `date_range` query over the `date_range` field named "time_frame". @@ -177,6 +179,7 @@ This query produces a similar result: } } ``` +% TESTRESPONSE[s/"took": 13/"took" : $body.took/] ## IP Range [ip-range] @@ -270,6 +273,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -291,6 +295,7 @@ Will become: ] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] Values of `ip_range` fields are not sorted but original order is not preserved. Duplicate ranges are removed. If `ip_range` field value is provided as a CIDR, it will be represented as a range of IP addresses in synthetic source. @@ -328,6 +333,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -340,6 +346,7 @@ Will become: } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] $$$range-synthetic-source-inclusive$$$ Range field values are always represented as inclusive on both sides with bounds adjusted accordingly. Default values for range bounds are represented as `null`. This is true even if range bound was explicitly provided. For example: @@ -373,6 +380,7 @@ PUT idx/_doc/1 } } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -384,6 +392,7 @@ Will become: } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] $$$range-synthetic-source-default-bounds$$$ Default values for range bounds are represented as `null` in synthetic source. This is true even if range bound was explicitly provided with default value. For example: @@ -416,6 +425,7 @@ PUT idx/_doc/1 } } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -427,6 +437,7 @@ Will become: } } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] `date` ranges are formatted using provided `format` or by default using `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format. For example: @@ -465,6 +476,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -482,5 +494,6 @@ Will become: ] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/mapping-reference/rank-vectors.md b/docs/reference/elasticsearch/mapping-reference/rank-vectors.md index 86e760d859db3..3c8dd29a87a19 100644 --- a/docs/reference/elasticsearch/mapping-reference/rank-vectors.md +++ b/docs/reference/elasticsearch/mapping-reference/rank-vectors.md @@ -35,6 +35,7 @@ PUT my-rank-vectors-float/_doc/1 "my_vector" : [[0.5, 10, 6], [-0.5, 10, 10]] } ``` +% TESTSETUP In addition to the `float` element type, `byte` and `bit` element types are also supported. diff --git a/docs/reference/elasticsearch/mapping-reference/search-as-you-type.md b/docs/reference/elasticsearch/mapping-reference/search-as-you-type.md index 0a6e5efadb664..8e750895d2862 100644 --- a/docs/reference/elasticsearch/mapping-reference/search-as-you-type.md +++ b/docs/reference/elasticsearch/mapping-reference/search-as-you-type.md @@ -50,6 +50,7 @@ PUT my-index-000001/_doc/1?refresh "my_field": "quick brown fox jump lazy dog" } ``` +% TEST[continued] The most efficient way of querying to serve a search-as-you-type use case is usually a [`multi_match`](/reference/query-languages/query-dsl-multi-match-query.md) query of type [`bool_prefix`](/reference/query-languages/query-dsl-match-bool-prefix-query.md) that targets the root `search_as_you_type` field and its shingle subfields. This can match the query terms in any order, but will score documents higher if they contain the terms in order in a shingle subfield. @@ -76,6 +77,7 @@ GET my-index-000001/_search } } ``` +% TEST[continued] 1. Adding "my_field._index_prefix" to the `matched_fields` allows to highlight "my_field" also based on matches from "my_field._index_prefix" field. @@ -114,6 +116,9 @@ GET my-index-000001/_search } } ``` +% TESTRESPONSE[s/"took" : 44/"took" : $body.took/] +% TESTRESPONSE[s/"max_score" : 0.8630463/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 0.8630463/"_score" : $body.hits.hits.0._score/] To search for documents that strictly match the query terms in order, or to search using other properties of phrase queries, use a [`match_phrase_prefix` query](/reference/query-languages/query-dsl-match-query-phrase-prefix.md) on the root field. A [`match_phrase` query](/reference/query-languages/query-dsl-match-query-phrase.md) can also be used if the last term should be matched exactly, and not as a prefix. Using phrase queries may be less efficient than using the `match_bool_prefix` query. @@ -127,6 +132,7 @@ GET my-index-000001/_search } } ``` +% TEST[continued] ## Parameters specific to the `search_as_you_type` field [specific-params] diff --git a/docs/reference/elasticsearch/mapping-reference/semantic-text.md b/docs/reference/elasticsearch/mapping-reference/semantic-text.md index 5e38f26029358..961db5248d503 100644 --- a/docs/reference/elasticsearch/mapping-reference/semantic-text.md +++ b/docs/reference/elasticsearch/mapping-reference/semantic-text.md @@ -50,6 +50,7 @@ PUT my-index-000002 } } ``` +% TEST[skip:Requires inference endpoint] 1. The `inference_id` of the {{infer}} endpoint to use to generate embeddings. @@ -70,6 +71,7 @@ PUT my-index-000003 } } ``` +% TEST[skip:Requires inference endpoint] ## Parameters for `semantic_text` fields [semantic-text-params] @@ -124,6 +126,7 @@ POST test-index/_search } } ``` +% TEST[skip:Requires inference endpoint] 1. Specifies the maximum number of fragments to return. 2. Sorts highlighted fragments by score when set to `score`. By default, fragments will be output in the order they appear in the field (order: none). @@ -150,6 +153,7 @@ PUT test-index } } ``` +% TEST[skip:Requires inference endpoint] 1. Ensures that highlighting is applied exclusively to semantic_text fields. @@ -190,6 +194,7 @@ PUT test-index } } ``` +% TEST[skip:TBD] can also be declared as multi-fields: @@ -211,6 +216,7 @@ PUT test-index } } ``` +% TEST[skip:TBD] ## Limitations [limitations] diff --git a/docs/reference/elasticsearch/mapping-reference/shape.md b/docs/reference/elasticsearch/mapping-reference/shape.md index 3f209ac8c0855..e24d05ef3fb38 100644 --- a/docs/reference/elasticsearch/mapping-reference/shape.md +++ b/docs/reference/elasticsearch/mapping-reference/shape.md @@ -47,6 +47,7 @@ PUT /example } } ``` +% TESTSETUP This mapping definition maps the geometry field to the shape type. The indexer uses single precision floats for the vertex values so accuracy is guaranteed to the same precision as `float` values provided by the java virtual machine approximately (typically 1E-38). diff --git a/docs/reference/elasticsearch/mapping-reference/text.md b/docs/reference/elasticsearch/mapping-reference/text.md index d396c6c26af44..33ab520abde91 100644 --- a/docs/reference/elasticsearch/mapping-reference/text.md +++ b/docs/reference/elasticsearch/mapping-reference/text.md @@ -144,6 +144,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -155,6 +156,7 @@ Will become: ] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] ::::{note} Reordering text fields can have an effect on [phrase](/reference/query-languages/query-dsl-match-query-phrase.md) and [span](/reference/query-languages/span-queries.md) queries. See the discussion about [`position_increment_gap`](/reference/elasticsearch/mapping-reference/position-increment-gap.md) for more detail. You can avoid this by making sure the `slop` parameter on the phrase queries is lower than the `position_increment_gap`. This is the default. @@ -192,6 +194,7 @@ PUT idx/_doc/1 ] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -262,6 +265,7 @@ PUT my-index-000001/_mapping } } ``` +% TEST[continued] 1. The mapping that you specify for `my_field` should consist of the existing mapping for that field, plus the `fielddata` parameter. diff --git a/docs/reference/elasticsearch/mapping-reference/unsigned-long.md b/docs/reference/elasticsearch/mapping-reference/unsigned-long.md index 7b8b89b3f4908..40fe6f8768526 100644 --- a/docs/reference/elasticsearch/mapping-reference/unsigned-long.md +++ b/docs/reference/elasticsearch/mapping-reference/unsigned-long.md @@ -35,6 +35,7 @@ POST /my_index/_bulk?refresh {"index":{"_id":4}} {"my_counter": 18446744073709551615} ``` +% TEST[continued] Term queries accept any numbers in a numeric or string form. @@ -48,6 +49,7 @@ GET /my_index/_search } } ``` +% TEST[continued] Range query terms can contain values with decimal parts. In this case {{es}} converts them to integer values: `gte` and `gt` terms are converted to the nearest integer up inclusive, and `lt` and `lte` ranges are converted to the nearest integer down inclusive. @@ -66,6 +68,7 @@ GET /my_index/_search } } ``` +% TEST[continued] ## Sort values [_sort_values_2] @@ -85,6 +88,7 @@ GET /my_index/_search "sort" : {"my_counter" : "desc"} } ``` +% TEST[continued] ## Stored fields [_stored_fields] @@ -118,6 +122,7 @@ GET /my_index/_search } } ``` +% TEST[continued] Alternatively, you can treat the unsigned long type as `BigInteger` in your scripts by using the field API. For example, this script treats `my_counter` as `BigInteger` with a default value of `BigInteger.ZERO`: @@ -126,6 +131,7 @@ Alternatively, you can treat the unsigned long type as `BigInteger` in your scri "source": "field('my_counter').asBigInteger(BigInteger.ZERO)" } ``` +% NOTCONSOLE For scripts that need to return float or double values, you can further convert `BigInteger` values to double or float: @@ -142,6 +148,7 @@ GET /my_index/_search } } ``` +% TEST[continued] ## Queries with mixed numeric types [_queries_with_mixed_numeric_types] diff --git a/docs/reference/elasticsearch/mapping-reference/version.md b/docs/reference/elasticsearch/mapping-reference/version.md index fc9ba89f55588..8209ed9786fc2 100644 --- a/docs/reference/elasticsearch/mapping-reference/version.md +++ b/docs/reference/elasticsearch/mapping-reference/version.md @@ -77,6 +77,7 @@ PUT idx/_doc/1 "versions": ["8.0.0-beta1", "8.5.0", "0.90.12", "2.6.1", "1.3.4", "1.3.4"] } ``` +% TEST[s/$/\nGET idx/_doc/1?filter_path=_source\n/] Will become: @@ -85,5 +86,6 @@ Will become: "versions": ["0.90.12", "1.3.4", "2.6.1", "8.0.0-beta1", "8.5.0"] } ``` +% TEST[s/^/{"_source":/ s/\n$/}/] diff --git a/docs/reference/elasticsearch/rest-apis/api-conventions.md b/docs/reference/elasticsearch/rest-apis/api-conventions.md index df8a58b845099..e521b039c95cf 100644 --- a/docs/reference/elasticsearch/rest-apis/api-conventions.md +++ b/docs/reference/elasticsearch/rest-apis/api-conventions.md @@ -288,6 +288,8 @@ GET /%3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogs } } ``` +% TEST[s/^/PUT logstash-2016.09.20\nPUT logstash-2016.09.19\nPUT logstash-2016.09.18\n/] +% TEST[s/now/2016.09.20%7C%7C/] ## Multi-target syntax [api-multi-index] diff --git a/docs/reference/elasticsearch/rest-apis/collapse-search-results.md b/docs/reference/elasticsearch/rest-apis/collapse-search-results.md index a1939da075e2f..6b645ec544589 100644 --- a/docs/reference/elasticsearch/rest-apis/collapse-search-results.md +++ b/docs/reference/elasticsearch/rest-apis/collapse-search-results.md @@ -33,6 +33,7 @@ GET my-index-000001/_search "from": 0 <3> } ``` +% TEST[setup:my_index] 1. Collapse the result set using the `user.id` field 2. Sort the results by `http.response.bytes` @@ -82,6 +83,7 @@ GET /my-index-000001/_search ] } ``` +% TEST[setup:my_index] 1. Collapse the result set using the `user.id` field 2. The name used for the inner hit section in the response @@ -134,6 +136,7 @@ GET /my-index-000001/_search ] } ``` +% TEST[setup:my_index] 1. Collapse the result set using the `user.id` field 2. Return the three largest HTTP responses for the user @@ -169,6 +172,7 @@ GET /my-index-000001/_search "search_after": ["dd5ce1ad"] } ``` +% TEST[setup:my_index] ## Rescore collapse results [rescore-collapse-results] @@ -183,6 +187,7 @@ POST /my-index-000001/_doc?routing=xyz <1> "user.id": "xyz" } ``` +% TEST[setup:my_index] 1. Assign routing with the collapse field value (`user.id`). @@ -216,6 +221,7 @@ GET /my-index-000001/_search } } ``` +% TEST[setup:my_index] ::::{warning} Rescorers are not applied to [`inner hits`](/reference/elasticsearch/rest-apis/retrieve-inner-hits.md). @@ -252,6 +258,8 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] +% TEST[s/_search/_search?filter_path=hits.hits/] ```console-result { diff --git a/docs/reference/elasticsearch/rest-apis/common-options.md b/docs/reference/elasticsearch/rest-apis/common-options.md index 2781cd202a70d..f35eaccad44fe 100644 --- a/docs/reference/elasticsearch/rest-apis/common-options.md +++ b/docs/reference/elasticsearch/rest-apis/common-options.md @@ -79,6 +79,7 @@ All REST APIs accept a `filter_path` parameter that can be used to reduce the re ```console GET /_search?q=kimchy&filter_path=took,hits.hits._id,hits.hits._score ``` +% TEST[setup:my_index] Responds: @@ -95,12 +96,15 @@ Responds: } } ``` +% TESTRESPONSE[s/"took" : 3/"took" : $body.took/] +% TESTRESPONSE[s/1.6375021/$body.hits.hits.0._score/] It also supports the `*` wildcard character to match any field or part of a field’s name: ```console GET /_cluster/state?filter_path=metadata.indices.*.stat* ``` +% TEST[s/^/PUT my-index-000001\n/] Responds: @@ -119,6 +123,7 @@ And the `**` wildcard can be used to include fields without knowing the exact pa ```console GET /_cluster/state?filter_path=routing_table.indices.**.state ``` +% TEST[s/^/PUT my-index-000001\n/] Responds: @@ -141,6 +146,7 @@ It is also possible to exclude one or more fields by prefixing the filter with t ```console GET /_count?filter_path=-_shards ``` +% TEST[setup:my_index] Responds: @@ -155,6 +161,7 @@ And for more control, both inclusive and exclusive filters can be combined in th ```console GET /_cluster/state?filter_path=metadata.indices.*.state,-metadata.indices.logstash-* ``` +% TEST[s/^/PUT my-index-000001\nPUT my-index-000002\nPUT my-index-000003\nPUT logstash-2016.01\n/] Responds: @@ -204,6 +211,7 @@ The `flat_settings` flag affects rendering of the lists of settings. When the `f ```console GET my-index-000001/_settings?flat_settings=true ``` +% TEST[setup:my_index] Returns: @@ -222,12 +230,16 @@ Returns: } } ``` +% TESTRESPONSE[s/1474389951325/$body.my-index-000001.settings.index\\.creation_date/] +% TESTRESPONSE[s/n6gzFZTgS664GUfx0Xrpjw/$body.my-index-000001.settings.index\\.uuid/] +% TESTRESPONSE[s/"index.version.created": .../"index.version.created": $body.my-index-000001.settings.index\\.version\\.created/] When the `flat_settings` flag is `false`, settings are returned in a more human readable structured format: ```console GET my-index-000001/_settings?flat_settings=false ``` +% TEST[setup:my_index] Returns: @@ -256,6 +268,9 @@ Returns: } } ``` +% TESTRESPONSE[s/1474389951325/$body.my-index-000001.settings.index.creation_date/] +% TESTRESPONSE[s/n6gzFZTgS664GUfx0Xrpjw/$body.my-index-000001.settings.index.uuid/] +% TESTRESPONSE[s/"created": .../"created": $body.my-index-000001.settings.index.version.created/] By default `flat_settings` is set to `false`. @@ -294,6 +309,7 @@ By default when a request returns an error Elasticsearch doesn’t include the s ```console POST /my-index-000001/_search?size=surprise_me ``` +% TEST[s/surprise_me/surprise_me&error_trace=false/ catch:bad_request] The response looks like: @@ -322,6 +338,7 @@ But if you set `error_trace=true`: ```console POST /my-index-000001/_search?size=surprise_me&error_trace=true ``` +% TEST[catch:bad_request] The response looks like: @@ -347,4 +364,7 @@ The response looks like: "status": 400 } ``` +% TESTRESPONSE[s/"stack_trace": "Failed to parse int parameter.+..."/"stack_trace": $body.error.root_cause.0.stack_trace/] +% TESTRESPONSE[s/"stack_trace": "java.lang.IllegalArgum.+..."/"stack_trace": $body.error.stack_trace/] +% TESTRESPONSE[s/"stack_trace": "java.lang.Number.+..."/"stack_trace": $body.error.caused_by.stack_trace/] diff --git a/docs/reference/elasticsearch/rest-apis/create-index-from-source.md b/docs/reference/elasticsearch/rest-apis/create-index-from-source.md index 79524341a0592..c7149e0f3c2b3 100644 --- a/docs/reference/elasticsearch/rest-apis/create-index-from-source.md +++ b/docs/reference/elasticsearch/rest-apis/create-index-from-source.md @@ -73,6 +73,7 @@ PUT /my-index } } ``` +% TESTSETUP Now we create a destination index from the source index. This new index will have the same mappings and settings as the source index. diff --git a/docs/reference/elasticsearch/rest-apis/filter-search-results.md b/docs/reference/elasticsearch/rest-apis/filter-search-results.md index da71e9942f106..66cd269e977ee 100644 --- a/docs/reference/elasticsearch/rest-apis/filter-search-results.md +++ b/docs/reference/elasticsearch/rest-apis/filter-search-results.md @@ -41,6 +41,7 @@ PUT /shirts/_doc/1?refresh "model": "slim" } ``` +% TESTSETUP Imagine a user has specified two filters: @@ -183,6 +184,7 @@ POST /_search } } ``` +% TEST[setup:my_index] The way the scores are combined can be controlled with the `score_mode`: @@ -241,6 +243,7 @@ POST /_search } ] } ``` +% TEST[setup:my_index] The first one gets the results of the query then the second one gets the results of the first, etc. The second rescore will "see" the sorting done by the first rescore so it is possible to use a large window on the first rescore to pull documents into a smaller window for the second rescore. diff --git a/docs/reference/elasticsearch/rest-apis/highlighting.md b/docs/reference/elasticsearch/rest-apis/highlighting.md index 96f0481fa47bc..340a8a1a718f7 100644 --- a/docs/reference/elasticsearch/rest-apis/highlighting.md +++ b/docs/reference/elasticsearch/rest-apis/highlighting.md @@ -31,6 +31,7 @@ GET /_search } } ``` +% TEST[setup:my_index] {{es}} supports three highlighters: `unified`, `plain`, and `fvh` (fast vector highlighter) for `text` and `keyword` fields and the `semantic` highlighter for `semantic_text` fields. You can specify the highlighter `type` you want to use for each field or rely on the field type’s default highlighter. @@ -251,6 +252,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ## Specify a highlight query [specify-highlight-query] @@ -314,6 +316,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ## Set highlighter type [set-highlighter-type] @@ -333,6 +336,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ## Configure highlighting tags [configure-tags] @@ -354,6 +358,7 @@ GET /_search } } ``` +% TEST[setup:my_index] When using the fast vector highlighter, you can specify additional tags and the "importance" is ordered. @@ -372,6 +377,7 @@ GET /_search } } ``` +% TEST[setup:my_index] You can also use the built-in `styled` tag schema: @@ -389,6 +395,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ## Highlight in all fields [highlight-all] @@ -409,6 +416,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ## Combine matches on multiple fields [matched-fields] @@ -444,6 +452,7 @@ PUT index1 } } ``` +% TEST[continued] ```console PUT index1/_bulk?refresh=true @@ -452,6 +461,7 @@ PUT index1/_bulk?refresh=true { "index" : {"_id": "doc2"} } {"comment": "running with scissors"} ``` +% TEST[continued] ```console GET index1/_search @@ -470,6 +480,7 @@ GET index1/_search } } ``` +% TEST[continued] The above request matches both "run with scissors" and "running with scissors" and would highlight "running" and "scissors" but not "run". If both phrases appear in a large document then "running with scissors" is sorted above "run with scissors" in the fragments list because there are more matches in that fragment. @@ -513,6 +524,7 @@ The above request matches both "run with scissors" and "running with scissors" a } } ``` +% TESTRESPONSE[s/.../"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] The below request highlights "run" as well as "running" and "scissors", because the `matched_fields` parameter instructs that for highlighting we need to combine matches from the `comment.english` field with the matches from the original `comment` field. @@ -535,6 +547,7 @@ GET index1/_search } } ``` +% TEST[continued] ```console-result { @@ -576,6 +589,8 @@ GET index1/_search } } ``` +% TESTRESPONSE[s/.../"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] + :::::: ::::::{tab-item} FVH @@ -602,6 +617,7 @@ PUT index2 } } ``` +% TEST[continued] ```console PUT index2/_bulk?refresh=true @@ -610,6 +626,7 @@ PUT index2/_bulk?refresh=true { "index" : {"_id": "doc2"} } {"comment": "running with scissors"} ``` +% TEST[continued] ```console GET index2/_search @@ -630,6 +647,7 @@ GET index2/_search } } ``` +% TEST[continued] The above request matches both "run with scissors" and "running with scissors" and would highlight "running" and "scissors" but not "run". If both phrases appear in a large document then "running with scissors" is sorted above "run with scissors" in the fragments list because there are more matches in that fragment. @@ -673,6 +691,7 @@ The above request matches both "run with scissors" and "running with scissors" a } } ``` +% TESTRESPONSE[s/.../"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] The below request highlights "run" as well as "running" and "scissors", because the `matched_fields` parameter instructs that for highlighting we need to combine matches from the `comment` and `comment.english` fields. @@ -696,6 +715,7 @@ GET index2/_search } } ``` +% TEST[continued] ```console-result { @@ -737,6 +757,7 @@ GET index2/_search } } ``` +% TESTRESPONSE[s/.../"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] The below request wouldn’t highlight "run" or "scissor" but shows that it is just fine not to list the field to which the matches are combined (`comment.english`) in the matched fields. @@ -760,6 +781,7 @@ GET index2/_search } } ``` +% TEST[continued] ```console-result { @@ -801,6 +823,7 @@ GET index2/_search } } ``` +% TESTRESPONSE[s/.../"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] :::::{note} There is a small amount of overhead involved with setting `matched_fields` to a non-empty array so always prefer @@ -812,6 +835,7 @@ There is a small amount of overhead involved with setting `matched_fields` to a } } ``` +% NOTCONSOLE to @@ -825,6 +849,8 @@ to } } ``` +% NOTCONSOLE + :::::: ::::::: @@ -852,6 +878,7 @@ GET /_search } } ``` +% TEST[setup:my_index] None of the highlighters built into Elasticsearch care about the order that the fields are highlighted but a plugin might. @@ -873,6 +900,7 @@ GET /_search } } ``` +% TEST[setup:my_index] On top of this it is possible to specify that highlighted fragments need to be sorted by score: @@ -890,6 +918,7 @@ GET /_search } } ``` +% TEST[setup:my_index] If the `number_of_fragments` value is set to `0` then no fragments are produced, instead the whole content of the field is returned, and of course it is highlighted. This can be very handy if short texts (like document title or address) need to be highlighted but no fragmentation is required. Note that `fragment_size` is ignored in this case. @@ -907,6 +936,7 @@ GET /_search } } ``` +% TEST[setup:my_index] When using `fvh` one can use `fragment_offset` parameter to control the margin to start highlighting from. @@ -929,6 +959,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ## Highlight using the postings list [highlight-postings-list] @@ -988,6 +1019,7 @@ GET my-index-000001/_search } } ``` +% TEST[setup:messages] Response: @@ -1020,6 +1052,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,/] ```console GET my-index-000001/_search @@ -1039,6 +1072,7 @@ GET my-index-000001/_search } } ``` +% TEST[setup:messages] Response: @@ -1070,6 +1104,7 @@ Response: } } ``` +% TESTRESPONSE[s/.../"took": $body.took,"timed_out": false,"_shards": $body._shards,/] If the `number_of_fragments` option is set to `0`, `NullFragmenter` is used which does not fragment the text at all. This is useful for highlighting the entire contents of a document or field. @@ -1137,6 +1172,7 @@ PUT test_index } } ``` +% NOTCONSOLE We put the following document into the index: @@ -1146,6 +1182,7 @@ PUT test_index/_doc/doc1 "content" : "For you I'm only a fox like a hundred thousand other foxes. But if you tame me, we'll need each other. You'll be the only boy in the world for me. I'll be the only fox in the world for you." } ``` +% NOTCONSOLE And we ran the following query with a highlight request: @@ -1164,6 +1201,7 @@ GET test_index/_search } } ``` +% NOTCONSOLE After `doc1` is found as a hit for this query, this hit will be passed to the unified highlighter for highlighting the field `content` of the document. Since the field `content` was not indexed either with offsets or term vectors, its raw field value will be analyzed, and in-memory index will be built from the terms that match the query: diff --git a/docs/reference/elasticsearch/rest-apis/optimistic-concurrency-control.md b/docs/reference/elasticsearch/rest-apis/optimistic-concurrency-control.md index ff6bf5e6ab12f..28bdde4b5b34a 100644 --- a/docs/reference/elasticsearch/rest-apis/optimistic-concurrency-control.md +++ b/docs/reference/elasticsearch/rest-apis/optimistic-concurrency-control.md @@ -38,12 +38,15 @@ You can see the assigned sequence number and primary term in the `_seq_no` and ` "result": "created" } ``` +% TESTRESPONSE[s/"_seq_no": 362/"_seq_no": $body._seq_no/] +% TESTRESPONSE[s/"_primary_term": 2/"_primary_term": $body._primary_term/] Elasticsearch keeps tracks of the sequence number and primary term of the last operation to have changed each of the documents it stores. The sequence number and primary term are returned in the `_seq_no` and `_primary_term` fields in the response of the [GET API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get): ```console GET products/_doc/1567 ``` +% TEST[continued] returns: @@ -61,6 +64,8 @@ returns: } } ``` +% TESTRESPONSE[s/"_seq_no": 362/"_seq_no": $body._seq_no/] +% TESTRESPONSE[s/"_primary_term": 2/"_primary_term": $body._primary_term/] Note: The [Search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) can return the `_seq_no` and `_primary_term` for each search hit by setting [`seq_no_primary_term` parameter](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#request-body-search-seq-no-primary-term). @@ -76,4 +81,6 @@ PUT products/_doc/1567?if_seq_no=362&if_primary_term=2 "tags": [ "droid" ] } ``` +% TEST[continued] +% TEST[catch: conflict] diff --git a/docs/reference/elasticsearch/rest-apis/paginate-search-results.md b/docs/reference/elasticsearch/rest-apis/paginate-search-results.md index fedb734d5ee35..3c13818d3cece 100644 --- a/docs/reference/elasticsearch/rest-apis/paginate-search-results.md +++ b/docs/reference/elasticsearch/rest-apis/paginate-search-results.md @@ -52,6 +52,7 @@ GET twitter/_search ] } ``` +% TEST[continued] 1. A copy of the `_id` field with `doc_values` enabled @@ -92,6 +93,7 @@ The search response includes an array of `sort` values for each hit: } } ``` +% TESTRESPONSE[skip: demo of where the sort values are] 1. Sort values for the last returned hit. @@ -113,12 +115,14 @@ GET twitter/_search ] } ``` +% TEST[continued] Repeat this process by updating the `search_after` array every time you retrieve a new page of results. If a [refresh](docs-content://solutions/search/search-approaches/near-real-time-search.md) occurs between these requests, the order of your results may change, causing inconsistent results across pages. To prevent this, you can create a [point in time (PIT)](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time) to preserve the current index state over your searches. ```console POST /my-index-000001/_pit?keep_alive=1m ``` +% TEST[setup:my_index] The API returns a PIT ID. @@ -128,6 +132,8 @@ The API returns a PIT ID. "_shards": ... } ``` +% TESTRESPONSE[s/"id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA=="/"id": $body.id/] +% TESTRESPONSE[s/"_shards": .../"_shards": "$body._shards"/] To get the first page of results, submit a search request with a `sort` argument. If using a PIT, specify the PIT ID in the `pit.id` parameter and omit the target data stream or index from the request path. @@ -164,6 +170,7 @@ GET /_search ] } ``` +% TEST[catch:unavailable] 1. PIT ID for the search. 2. Sorts hits for the search with an implicit tiebreak on `_shard_doc` ascending. @@ -190,6 +197,7 @@ GET /_search ] } ``` +% TEST[catch:unavailable] 1. PIT ID for the search. 2. Sorts hits for the search with an explicit tiebreak on `_shard_doc` descending. @@ -220,6 +228,7 @@ GET /_search } } ``` +% TESTRESPONSE[skip: unable to access PIT ID] 1. Updated `id` for the point in time. 2. Sort values for the last returned hit. @@ -251,6 +260,7 @@ GET /_search "track_total_hits": false <3> } ``` +% TEST[catch:unavailable] 1. PIT ID returned by the previous search. 2. Sort values from the previous search’s last hit. @@ -267,6 +277,7 @@ DELETE /_pit "id" : "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==" } ``` +% TEST[catch:missing] ## Scroll search results [scroll-search-results] @@ -313,6 +324,7 @@ POST /my-index-000001/_search?scroll=1m } } ``` +% TEST[setup:my_index] The result from the above request includes a `_scroll_id`, which should be passed to the `scroll` API in order to retrieve the next batch of results. @@ -323,6 +335,7 @@ POST /_search/scroll "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==" <3> } ``` +% TEST[continued s/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==/$body._scroll_id/] 1. `GET` or `POST` can be used and the URL should not include the `index` name — this is specified in the original `search` request instead. 2. The `scroll` parameter tells Elasticsearch to keep the search context open for another `1m`. @@ -354,6 +367,7 @@ GET /_search?scroll=1m ] } ``` +% TEST[setup:my_index] ### Keeping the search context alive [scroll-search-context] @@ -393,6 +407,7 @@ DELETE /_search/scroll "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==" } ``` +% TEST[catch:missing] Multiple scroll IDs can be passed as array: @@ -405,6 +420,7 @@ DELETE /_search/scroll ] } ``` +% TEST[catch:missing] All search contexts can be cleared with the `_all` parameter: @@ -417,6 +433,7 @@ The `scroll_id` can also be passed as a query string parameter or in the request ```console DELETE /_search/scroll/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==,DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAABFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAAAxZrUllkUVlCa1NqNmRMaUhiQlZkMWFBAAAAAAAAAAIWa1JZZFFZQmtTajZkTGlIYkJWZDFhQQAAAAAAAAAFFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAABBZrUllkUVlCa1NqNmRMaUhiQlZkMWFB ``` +% TEST[catch:missing] ### Sliced scroll [slice-scroll] @@ -449,6 +466,7 @@ GET /my-index-000001/_search?scroll=1m } } ``` +% TEST[setup:my_index_big] 1. The id of the slice 2. The maximum number of slices @@ -488,6 +506,7 @@ GET /my-index-000001/_search?scroll=1m } } ``` +% TEST[setup:my_index_big] For append only time-based indices, the `timestamp` field can be used safely. diff --git a/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md b/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md index 492317090e411..caabb00a03f89 100644 --- a/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md +++ b/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md @@ -25,6 +25,7 @@ return score # result(q) is the result set of q # rank( result(q), d ) is d's rank within the result(q) starting from 1 ``` +% NOTCONSOLE ## Reciprocal rank fusion API [rrf-api] @@ -88,6 +89,7 @@ GET example-index/_search } } ``` +% TEST[skip:example fragment] In the above example, we execute the `knn` and `standard` retrievers independently of each other. Then we use the `rrf` retriever to combine the results. @@ -163,6 +165,7 @@ GET example-index/_search } } ``` +% TEST[skip:example fragment] In the above example, we execute each of the two `standard` retrievers independently of each other. Then we use the `rrf` retriever to combine the results. @@ -242,6 +245,7 @@ PUT example-index/_doc/5 POST example-index/_refresh ``` +% TEST We now execute a search using an `rrf` retriever with a `standard` retriever specifying a BM25 query, a `knn` retriever specifying a kNN search, and a terms aggregation. @@ -283,6 +287,7 @@ GET example-index/_search } } ``` +% TEST[continued] And we receive the response with ranked `hits` and the terms aggregation result. We have both the ranker’s `score` and the `_rank` option to show our top-ranked documents. @@ -356,6 +361,7 @@ And we receive the response with ranked `hits` and the terms aggregation result. } } ``` +% TESTRESPONSE[s/: .../: $body.$_path/] Let’s break down how these hits were ranked. We start by running the `standard` retriever specifying a query and the `knn` retriever specifying a kNN search separately to collect what their individual hits are. @@ -404,13 +410,13 @@ First, we look at the hits for the query from the `standard` retriever. } ] ``` +% TEST[skip:example fragment] 1. rank 1, `_id` 4 2. rank 2, `_id` 3 3. rank 3, `_id` 2 4. rank 4, `_id` 1 - Note that our first hit doesn’t have a value for the `vector` field. Now, we look at the results for the kNN search from the `knn` retriever. ```console-result @@ -456,13 +462,13 @@ Note that our first hit doesn’t have a value for the `vector` field. Now, we l } ] ``` +% TEST[skip:example fragment] 1. rank 1, `_id` 3 2. rank 2, `_id` 2 3. rank 3, `_id` 1 4. rank 4, `_id` 5 - We can now take the two individually ranked result sets and apply the RRF formula to them using parameters from the `rrf` retriever to get our final ranking. ```python @@ -473,6 +479,7 @@ _id: 3 = 1.0/(1+2) + 1.0/(1+1) = 0.8333 _id: 4 = 1.0/(1+1) = 0.5000 _id: 5 = 1.0/(1+4) = 0.2000 ``` +% NOTCONSOLE We rank the documents based on the RRF formula with a `rank_window_size` of `5` truncating the bottom `2` docs in our RRF result set with a `size` of `3`. We end with `_id: 3` as `_rank: 1`, `_id: 2` as `_rank: 2`, and `_id: 4` as `_rank: 3`. This ranking matches the result set from the original RRF search as expected. @@ -530,6 +537,7 @@ In addition to individual query scoring details, we can make use of the `explain ] } ``` +% NOTCONSOLE 1. the final RRF score for document with `_id=3` 2. a description on how this score was computed based on the ranks of this document in each individual query @@ -580,10 +588,10 @@ GET example-index/_search } } ``` +% NOTCONSOLE 1. Here we specify a `_name` for the `knn` retriever - The response would now include the named query in the explanation: ```js @@ -623,11 +631,10 @@ The response would now include the named query in the explanation: ] } ``` +% NOTCONSOLE 1. Instead of the anonymous `at index n` , we now have a reference to the named query `my_knn_query`. - - ## Pagination in RRF [_pagination_in_rrf] When using `rrf` you can paginate through the results using the `from` parameter. As the final ranking is solely dependent on the original query ranks, to ensure consistency when paginating, we have to make sure that while `from` changes, the order of what we have already seen remains intact. To that end, we’re using a fixed `rank_window_size` as the whole available result set upon which we can paginate. This essentially means that if: @@ -647,6 +654,7 @@ _id: | 3 | 3 | _id: | 4 | 1 | _id: | | 2 | ``` +% NOTCONSOLE For `rank_window_size=5` we would get to see all documents from both `queryA` and `queryB`. Assuming a `rank_constant=1`, the `rrf` scores would be: @@ -658,6 +666,7 @@ _id: 3 = 1.0/(1+3) + 1.0/(1+3) = 0.5 _id: 4 = 1.0/(1+4) + 1.0/(1+2) = 0.533 _id: 5 = 0 + 1.0/(1+1) = 0.5 ``` +% NOTCONSOLE So the final ranked result set would be [`1`, `4`, `2`, `3`, `5`] and we would paginate over that, since `rank_window_size == len(results)`. In this scenario, we would have: @@ -675,6 +684,7 @@ _id: 2 = 1.0/(1+2) + 0 = 0.33 _id: 4 = 0 + 1.0/(1+2) = 0.33 _id: 5 = 0 + 1.0/(1+1) = 0.5 ``` +% NOTCONSOLE The final ranked result set would be [`1`, `5`, `2`, `4`], and we would be able to paginate on the top `rank_window_size` results, i.e. [`1`, `5`]. So for the same params as above, we would now have: @@ -700,6 +710,7 @@ For example, consider the following document set: "_id": 4, "termA": "foo", "termB": "bar" } ``` +% NOTCONSOLE Perform a term aggregation on the `termA` field using an `rrf` retriever: @@ -738,6 +749,7 @@ Perform a term aggregation on the `termA` field using an `rrf` retriever: } } ``` +% NOTCONSOLE The aggregation results will include **all** matching documents, regardless of `rank_window_size`. @@ -747,6 +759,7 @@ The aggregation results will include **all** matching documents, regardless of ` "aardvark": 1 } ``` +% NOTCONSOLE ## Highlighting in RRF [_highlighting_in_rrf] diff --git a/docs/reference/elasticsearch/rest-apis/reindex-data-stream.md b/docs/reference/elasticsearch/rest-apis/reindex-data-stream.md index d81b4dba6bd17..9a8053f184601 100644 --- a/docs/reference/elasticsearch/rest-apis/reindex-data-stream.md +++ b/docs/reference/elasticsearch/rest-apis/reindex-data-stream.md @@ -79,6 +79,7 @@ POST _migration/reindex "mode": "upgrade" } ``` +% TEST[setup:my_data_stream] As this task runs in the background this API will return immediately. The task will do the following. @@ -102,6 +103,7 @@ While the reindex data stream task is running, we can inspect the current status ```console GET /_migration/reindex/my-data-stream/_status ``` +% TEST[continued] For the above example, the following would be a possible status: @@ -123,6 +125,7 @@ For the above example, the following would be a possible status: "errors": [] } ``` +% TEST[skip:specific value is part of explanation] This output means that the first backing index, `.ds-my-data-stream-2025.01.23-000001`, is currently being processed, and none of the backing indices have yet completed. Notice that `total_indices_in_data_stream` has a value of `4`, because after the rollover, there are 4 indices in the data stream. But the new write index has an 8.x version, and thus doesn’t need to be reindexed, so `total_indices_requiring_upgrade` is only 3. @@ -150,6 +153,7 @@ Continuing with the above example, assume the reindexing task has not yet comple "errors": [] } ``` +% TEST[skip:specific value is part of explanation] Let’s assume the task has been running for a long time. By default, we throttle how many requests the reindex operation can execute per second. This keeps the reindex process from consuming too many resources. But the default value of `1000` request per second will not be correct for all use cases. The [`migrate.data_stream_reindex_max_request_per_second` setting](#migrate_data_stream_reindex_max_request_per_second-setting) can be used to increase or decrease the number of requests per second, or to remove the throttle entirely. @@ -160,6 +164,7 @@ But in the above status, `.ds-my-data-stream-2025.01.23-000002` has values of 10 ```console POST /_migration/reindex/my-data-stream/_cancel ``` +% TEST[skip:task will not be present] Now we can use the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) to increase the throttle: @@ -171,6 +176,7 @@ PUT /_cluster/settings } } ``` +% TEST[continued] The [original reindex command](#reindex-data-stream-start) can now be used to restart reindexing. Because the first backing index, `.ds-my-data-stream-2025.01.23-000001`, has already been reindexed and thus is already version 8.x, it will be skipped. The task will start by reindexing `.ds-my-data-stream-2025.01.23-000002` again from the beginning. @@ -188,6 +194,7 @@ Later, once all the backing indices have finished, the [reindex status API](http "errors": [] } ``` +% TEST[skip:specific value is part of explanation] Notice that the value of `total_indices_requiring_upgrade` is `2`, unlike the previous status, which had a value of `3`. This is because `.ds-my-data-stream-2025.01.23-000001` was upgraded before the task cancellation. After the restart, the API sees that it does not need to be upgraded, thus does not include it in `total_indices_requiring_upgrade` or `successes`, despite the fact that it upgraded successfully. @@ -198,6 +205,7 @@ We can now check the data stream to verify that indices were upgraded: ```console GET _data_stream/my-data-stream?filter_path=data_streams.indices.index_name ``` +% TEST[continued] which returns: @@ -223,6 +231,7 @@ which returns: ] } ``` +% TEST[skip:did not actually run reindex] Index `.ds-my-data-stream-2025.01.23-000004` is the write index and didn’t need to be upgraded because it was created with version 8.x. The other three backing indices are now prefixed with `.migrated` because they have been upgraded. @@ -231,6 +240,7 @@ We can now check the indices and verify that they have version 8.x: ```console GET .migrated-ds-my-data-stream-2025.01.23-000001?human&filter_path=*.settings.index.version.created_string ``` +% TEST[skip:migrated index does not exist] which returns: @@ -247,6 +257,7 @@ which returns: } } ``` +% TEST[skip:migrated index does not exist] ### Handling Failures [reindex-data-stream-handling-failure] @@ -270,5 +281,6 @@ Since the reindex data stream API runs in the background, failure information ca ] } ``` +% TEST[skip:result just part of explanation] Once the issue has been fixed, the failed reindex task can be re-run. First, the failed run’s status must be cleared using the [reindex cancel API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-cancel-migrate-reindex). Then the [original reindex command](#reindex-data-stream-start) can be called to pick up where it left off. diff --git a/docs/reference/elasticsearch/rest-apis/retrieve-inner-hits.md b/docs/reference/elasticsearch/rest-apis/retrieve-inner-hits.md index 62c521840f5f3..4ea44577a6032 100644 --- a/docs/reference/elasticsearch/rest-apis/retrieve-inner-hits.md +++ b/docs/reference/elasticsearch/rest-apis/retrieve-inner-hits.md @@ -20,6 +20,7 @@ Inner hits can be used by defining an `inner_hits` definition on a `nested`, `ha } } ``` +% NOTCONSOLE If `inner_hits` is defined on a query that supports it then each search hit will contain an `inner_hits` json object with the following structure: @@ -48,6 +49,7 @@ If `inner_hits` is defined on a query that supports it then each search hit will ... ] ``` +% NOTCONSOLE ## Options [inner-hits-options] @@ -174,6 +176,8 @@ An example of a response snippet that could be generated from the above search r } } ``` +% TESTRESPONSE[s/"_source": \.\.\./"_source": $body.hits.hits.0._source/] +% TESTRESPONSE[s/\.\.\./"timed_out": false, "took": $body.took, "_shards": $body._shards/] 1. The name used in the inner hit definition in the search request. A custom key can be used via the `name` option. @@ -234,7 +238,8 @@ POST test/_search } } ``` - +% TESTRESPONSE[s/"_source": \.\.\./"_source": $body.hits.hits.0._source/] +% TESTRESPONSE[s/\.\.\./"timed_out": false, "took": $body.took, "_shards": $body._shards/] ## Hierarchical levels of nested object fields and inner hits. [hierarchical-nested-inner-hits] @@ -345,6 +350,8 @@ Which would look like: } } ``` +% TESTRESPONSE[s/"_source": \.\.\./"_source": $body.hits.hits.0._source/] +% TESTRESPONSE[s/\.\.\./"timed_out": false, "took": $body.took, "_shards": $body._shards/] This indirect referencing is only supported for nested inner hits. @@ -453,4 +460,5 @@ An example of a response snippet that could be generated from the above search r } } ``` - +% TESTRESPONSE[s/"_source": \.\.\./"_source": $body.hits.hits.0._source/] +% TESTRESPONSE[s/\.\.\./"timed_out": false, "took": $body.took, "_shards": $body._shards/] diff --git a/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md b/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md index fb429337e3d63..6b7a02a5d1301 100644 --- a/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md +++ b/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md @@ -58,11 +58,12 @@ POST my-index-000001/_search "_source": false } ``` +% TEST[setup:my_index] +% TEST[s/_search/_search\?filter_path=hits/] 1. Both full field names and wildcard patterns are accepted. 2. Use the `format` parameter to apply a custom format for the field’s values. - ::::{note} By default, document metadata fields like `_id` or `_index` are not returned when the requested `fields` option uses wildcard patterns like `*`. However, when explicitly requested using the field name, the `_id`, `_routing`, `_ignored`, `_index` and `_version` metadata fields can be retrieved. :::: @@ -107,6 +108,8 @@ The response includes values as a flat list in the `fields` section for each hit } } ``` +% TESTRESPONSE[s/"max_score" : 1.0/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 1.0/"_score" : $body.hits.hits.0._score/] #### Retrieve nested fields [search-fields-nested] @@ -193,6 +196,9 @@ The response will group `first` and `last` name instead of returning them as a f } } ``` +% TESTRESPONSE[s/"took": 2/"took": $body.took/] +% TESTRESPONSE[s/"max_score" : 1.0/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 1.0/"_score" : $body.hits.hits.0._score/] Nested fields will be grouped by their nested paths, no matter the pattern used to retrieve them. For example, if you query only for the `user.first` field from the previous example: @@ -203,6 +209,7 @@ POST my-index-000001/_search "_source": false } ``` +% TEST[continued] The response returns only the user’s first name, but still maintains the structure of the nested `user` array: @@ -239,6 +246,9 @@ The response returns only the user’s first name, but still maintains the struc } } ``` +% TESTRESPONSE[s/"took": 2/"took": $body.took/] +% TESTRESPONSE[s/"max_score" : 1.0/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 1.0/"_score" : $body.hits.hits.0._score/] However, when the `fields` pattern targets the nested `user` field directly, no values will be returned because the pattern doesn’t match any leaf fields. @@ -321,6 +331,9 @@ The response will contain field results under the `session_data.object.*` path, } } ``` +% TESTRESPONSE[s/"took" : 2/"took": $body.took/] +% TESTRESPONSE[s/"max_score" : 1.0/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 1.0/"_score" : $body.hits.hits.0._score/] :::: @@ -403,6 +416,9 @@ The response will contain ignored field values under the `ignored_field_values` } } ``` +% TESTRESPONSE[s/"took" : 2/"took": $body.took/] +% TESTRESPONSE[s/"max_score" : 1.0/"max_score" : $body.hits.max_score/] +% TESTRESPONSE[s/"_score" : 1.0/"_score" : $body.hits.hits.0._score/] :::: @@ -522,11 +538,11 @@ GET my-index-000001/_search ] } ``` +% TEST[setup:my_index] 1. Both full field names and wildcard patterns are accepted. 2. Using object notation, you can pass a `format` parameter to apply a custom format for the field’s doc values. [Date fields](/reference/elasticsearch/mapping-reference/date.md) support a [date `format`](/reference/elasticsearch/mapping-reference/mapping-date-format.md). [Numeric fields](/reference/elasticsearch/mapping-reference/number.md) support a [DecimalFormat pattern](https://docs.oracle.com/javase/8/docs/api/java/text/DecimalFormat.md). Other field datatypes do not support the `format` parameter. - ::::{tip} You cannot use the `docvalue_fields` parameter to retrieve doc values for nested objects. If you specify a nested object, the search returns an empty array (`[ ]`) for the field. To access nested fields, use the [`inner_hits`](/reference/elasticsearch/rest-apis/retrieve-inner-hits.md) parameter’s `docvalue_fields` property. :::: @@ -629,6 +645,7 @@ GET /_search } } ``` +% TEST[setup:sales] Script fields can work on fields that are not stored (`price` in the above case), and allow to return custom values to be returned (the evaluated value of the script). @@ -647,6 +664,7 @@ GET /_search } } ``` +% TEST[setup:my_index] Note the `_source` keyword here to navigate the json-like model. diff --git a/docs/reference/elasticsearch/rest-apis/retrievers.md b/docs/reference/elasticsearch/rest-apis/retrievers.md index 14c413a7832ed..baf2176a18597 100644 --- a/docs/reference/elasticsearch/rest-apis/retrievers.md +++ b/docs/reference/elasticsearch/rest-apis/retrievers.md @@ -238,6 +238,7 @@ GET /restaurants/_search } } ``` +% TEST[continued] 1. Configuration for k-nearest neighbor (knn) search, which is based on vector similarity. 2. Specifies the field name that contains the vectors. @@ -368,6 +369,7 @@ GET /restaurants/_search } } ``` +% TEST[continued] 1. Defines a retriever tree with an RRF retriever. 2. The sub-retriever array. @@ -425,6 +427,7 @@ GET movies/_search } } ``` +% TEST[skip:uses ELSER] ## Rescorer Retriever [rescorer-retriever] @@ -538,6 +541,7 @@ GET movies/_search } } ``` +% TEST[skip:uses ELSER] 1. Specifies the number of top documents to return in the final response. 2. A `rescorer` retriever applied as the final step. @@ -723,6 +727,7 @@ GET /index/_search } } ``` +% TEST[skip:uses ML] ### Example: Semantic re-ranking with a Hugging Face model [text-similarity-reranker-retriever-example-eland] diff --git a/docs/reference/elasticsearch/rest-apis/search-multiple-data-streams-indices.md b/docs/reference/elasticsearch/rest-apis/search-multiple-data-streams-indices.md index 0c8f799efd91c..b1decfef78b27 100644 --- a/docs/reference/elasticsearch/rest-apis/search-multiple-data-streams-indices.md +++ b/docs/reference/elasticsearch/rest-apis/search-multiple-data-streams-indices.md @@ -26,6 +26,8 @@ GET /my-index-000001,my-index-000002/_search } } ``` +% TEST[setup:my_index] +% TEST[s/^/PUT my-index-000002\n/] You can also search multiple data streams and indices using an index pattern. @@ -41,6 +43,7 @@ GET /my-index-*/_search } } ``` +% TEST[setup:my_index] You can exclude specific indices from a search. The request will retrieve data from all indices starting with `my-index-`, except for `my-index-01`. @@ -67,6 +70,7 @@ GET /my-index-*/_search } } ``` +% TEST[setup:my_index] To search all data streams and indices in a cluster, omit the target from the request path. Alternatively, you can use `_all` or `*`. @@ -100,6 +104,7 @@ GET /*/_search } } ``` +% TEST[setup:my_index] ## Index boost [index-boost] @@ -120,6 +125,7 @@ GET /_search ] } ``` +% TEST[s/^/PUT my-index-000001\nPUT my-index-000002\n/] Aliases and index patterns can also be used: @@ -132,6 +138,7 @@ GET /_search ] } ``` +% TEST[s/^/PUT my-index-000001\nPUT my-index-000001/_alias/my-alias\n/] If multiple matches are found, the first match will be used. For example, if an index is included in `alias1` and matches the `my-index*` pattern, a boost value of `1.4` is applied. diff --git a/docs/reference/elasticsearch/rest-apis/search-shard-routing.md b/docs/reference/elasticsearch/rest-apis/search-shard-routing.md index 61fb63fac8630..e240fda901778 100644 --- a/docs/reference/elasticsearch/rest-apis/search-shard-routing.md +++ b/docs/reference/elasticsearch/rest-apis/search-shard-routing.md @@ -39,6 +39,7 @@ GET /my-index-000001/_search?preference=_local } } ``` +% TEST[setup:my_index] You can also use the `preference` parameter to route searches to specific shards based on a provided string. If the cluster state and selected shards do not change, searches using the same `preference` string are routed to the same shards in the same order. @@ -61,6 +62,7 @@ GET /my-index-000001/_search?preference=my-custom-shard-string } } ``` +% TEST[setup:my_index] ::::{note} If the cluster state or selected shards change, the same `preference` string may not route searches to the same shards in the same order. This can occur for a number of reasons, including shard relocations and shard failures. A node can also reject a search request, which {{es}} would re-route to another node. @@ -97,6 +99,7 @@ GET /my-index-000001/_search?routing=my-routing-value } } ``` +% TEST[setup:my_index] You can also provide multiple comma-separated routing values: @@ -110,6 +113,7 @@ GET /my-index-000001/_search?routing=my-routing-value,my-routing-value-2 } } ``` +% TEST[setup:my_index] ## Search concurrency and parallelism [search-concurrency-and-parallelism] @@ -133,6 +137,7 @@ GET /my-index-000001/_search?max_concurrent_shard_requests=3 } } ``` +% TEST[setup:my_index] You can also use the `action.search.shard_count.limit` cluster setting to set a search shard limit and reject requests that hit too many shards. You can configure `action.search.shard_count.limit` using the [cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). diff --git a/docs/reference/elasticsearch/rest-apis/search-suggesters.md b/docs/reference/elasticsearch/rest-apis/search-suggesters.md index e3dd525443fb2..38147b5aea791 100644 --- a/docs/reference/elasticsearch/rest-apis/search-suggesters.md +++ b/docs/reference/elasticsearch/rest-apis/search-suggesters.md @@ -34,6 +34,8 @@ POST _search } } ``` +% TEST[setup:messages] +% TEST[s/^/PUT my-index-000001/_mapping\n{"properties":{"user":{"properties":{"id":{"type":"keyword"}}}}}\n/] The following suggest response example includes the suggestion response for `my-suggest-1` and `my-suggest-2`. Each suggestion part contains entries. Each entry is effectively a token from the suggest text and contains the suggestion entry text, the original start offset and length in the suggest text and if found an arbitrary number of options. @@ -64,6 +66,10 @@ The following suggest response example includes the suggestion response for `my- } } ``` +% TESTRESPONSE[s/"_shards": .../"_shards": "$body._shards",/] +% TESTRESPONSE[s/"hits": …​/"hits": "$body.hits",/] +% TESTRESPONSE[s/"took": 2,/"took": "$body.took",/] +% TESTRESPONSE[s/"my-suggest-2": .../"my-suggest-2": "$body.suggest.my-suggest-2"/] Each options array contains an option object that includes the suggested text, its document frequency and score compared to the suggest entry text. The meaning of the score depends on the used suggester. The term suggester's score is based on the edit distance. @@ -267,6 +273,9 @@ The response contains suggestions scored by the most likely spelling correction } } ``` +% TESTRESPONSE[s/"_shards": …​/"_shards": "$body._shards",/] +% TESTRESPONSE[s/"hits": …​/"hits": "$body.hits",/] +% TESTRESPONSE[s/"took": 3,/"took": "$body.took",/] $$$_basic_phrase_suggest_api_parameters$$$ Basic phrase suggest API parameters include: @@ -507,6 +516,7 @@ PUT music/_doc/1?refresh } } ``` +% TEST The supported parameters include: @@ -543,6 +553,7 @@ PUT music/_doc/1?refresh ] } ``` +% TEST[continued] You can use the following shorthand form. Note that you can not specify a weight with suggestion(s) in the shorthand form. @@ -552,6 +563,7 @@ PUT music/_doc/1?refresh "suggest" : [ "Nevermind", "Nirvana" ] } ``` +% TEST[continued] ### Querying [querying] @@ -575,6 +587,7 @@ POST music/_search?pretty 1. Prefix used to search for suggestions 2. Type of suggestions 3. Name of the field to search for suggestions in +% TEST[continued] It returns this response: @@ -607,6 +620,8 @@ It returns this response: } } ``` +% TESTRESPONSE[s/"hits": …​/"hits": "$body.hits",/] +% TESTRESPONSE[s/"took": 2,/"took": "$body.took",/] ::::{important} `_source` metadata field must be enabled, which is the default behavior, to enable returning `_source` with suggestions. @@ -629,12 +644,12 @@ POST music/_search } } ``` +% TEST[continued] 1. Filter the source to return only the `suggest` field 2. Name of the field to search for suggestions in 3. Number of suggestions to return - Which should look like: ```console-result @@ -673,6 +688,7 @@ Which should look like: } } ``` +% TESTRESPONSE[s/"took": 6,/"took": $body.took,/] The supported parameters for a basic completion suggester query include: @@ -918,6 +934,7 @@ POST place/_search?pretty } } ``` +% TEST[continued] ::::{note} If multiple categories or category contexts are set on the query they are merged as a disjunction. This means that suggestions match if they contain at least one of the provided context values. @@ -945,9 +962,11 @@ POST place/_search?pretty } } ``` +% TEST[continued] 1. The context query filter suggestions associated with categories *cafe* and *restaurants* and boosts the suggestions associated with *restaurants* by a factor of `2` + In addition to accepting category values, a context query can be composed of multiple category context clauses. The parameters that are supported for a `category` context clause include: @@ -1030,6 +1049,7 @@ POST place/_search } } ``` +% TEST[continued] ::::{note} When a location with a lower precision at query time is specified, all suggestions that fall within the area will be considered. @@ -1069,9 +1089,11 @@ POST place/_search?pretty } } ``` +% TEST[continued] 1. The context query filters for suggestions that fall under the geo location represented by a geohash of *(43.662, -79.380)* with a precision of *2* and boosts suggestions that fall under the geohash representation of *(43.6624803, -79.3863353)* with a default precision of *6* by a factor of `2` + ::::{note} If a suggestion entry matches multiple contexts the final score is computed as the maximum score produced by any matching contexts. :::: @@ -1119,6 +1141,7 @@ POST _search?typed_keys } } ``` +% TEST[setup:messages] In the response, the suggester names will be changed to respectively `term#my-first-suggester` and `phrase#my-second-suggester`, reflecting the types of each suggestion: @@ -1168,6 +1191,9 @@ In the response, the suggester names will be changed to respectively `term#my-fi ... } ``` +% TESTRESPONSE[s/.../"took": "$body.took", "timed_out": false, "_shards": "$body._shards", "hits": "$body.hits"/] +% TESTRESPONSE[s/"score": 0.8333333/"score": $body.suggest.term#my-first-suggester.2.options.0.score/] +% TESTRESPONSE[s/"score": 0.030227963/"score": $body.suggest.phrase#my-second-suggester.0.options.0.score/] 1. The name `my-first-suggester` now contains the `term` prefix. 2. The name `my-second-suggester` now contains the `phrase` prefix. diff --git a/docs/reference/elasticsearch/rest-apis/searching-with-query-rules.md b/docs/reference/elasticsearch/rest-apis/searching-with-query-rules.md index 47fb38c513c48..39589bc6e110d 100644 --- a/docs/reference/elasticsearch/rest-apis/searching-with-query-rules.md +++ b/docs/reference/elasticsearch/rest-apis/searching-with-query-rules.md @@ -142,6 +142,7 @@ There is a limit of 100 rules per ruleset. This can be increased up to 1000 usin "result": "created" } ``` +% TEST[continued] You can use the [Get query ruleset](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-query-rules-get-ruleset) call to retrieve the ruleset you just created, the [List query rulesets](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-query-rules-list-rulesets) call to retrieve a summary of all query rulesets, and the [Delete query ruleset](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-query-rules-delete-ruleset) call to delete a query ruleset. @@ -177,6 +178,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] This rule query will match against `rule1` in the defined query ruleset, and will convert the organic query into a pinned query with `id1` and `id2` pinned as the top hits. Any other matches from the organic query will be returned below the pinned results. @@ -228,6 +230,7 @@ GET my-index-000001/_search } } ``` +% TEST[continued] This will apply pinned and excluded query rules on top of the content that was reranked by RRF. diff --git a/docs/reference/elasticsearch/rest-apis/shard-request-cache.md b/docs/reference/elasticsearch/rest-apis/shard-request-cache.md index e8448c1579345..8c51039ef74bf 100644 --- a/docs/reference/elasticsearch/rest-apis/shard-request-cache.md +++ b/docs/reference/elasticsearch/rest-apis/shard-request-cache.md @@ -31,6 +31,7 @@ The cache can be expired manually with the [clear cache API](https://www.elastic ```console POST /my-index-000001,my-index-000002/_cache/clear?request=true ``` +% TEST[s/^/PUT my-index-000001\nPUT my-index-000002\n/] ## Enabling and disabling caching [_enabling_and_disabling_caching] @@ -71,6 +72,7 @@ GET /my-index-000001/_search?request_cache=true } } ``` +% TEST[continued] Requests where `size` is greater than `0` will not be cached even if the request cache is enabled in the index settings. To cache these requests you will need to use the query parameter. diff --git a/docs/reference/elasticsearch/rest-apis/sort-search-results.md b/docs/reference/elasticsearch/rest-apis/sort-search-results.md index 4d76986a6beb2..aa8abc24f1063 100644 --- a/docs/reference/elasticsearch/rest-apis/sort-search-results.md +++ b/docs/reference/elasticsearch/rest-apis/sort-search-results.md @@ -30,6 +30,7 @@ PUT /my-index-000001 } } ``` +% TEST[continued] ```console GET /my-index-000001/_search @@ -46,6 +47,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] ::::{note} `_doc` has no real use-case besides being the most efficient sort order. So if you don’t care about the order in which documents are returned, then you should sort by `_doc`. This especially helps when [scrolling](/reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results). @@ -68,6 +70,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] ## Sort order [_sort_order] @@ -144,6 +147,7 @@ PUT /index_double } } ``` +% TEST[continued] ```console PUT /index_long @@ -155,6 +159,7 @@ PUT /index_long } } ``` +% TEST[continued] Since `field` is mapped as a `double` in the first index and as a `long` in the second index, it is not possible to use this field to sort requests that query both indices by default. However you can force the type to one or the other with the `numeric_type` option in order to force a specific type for all indices: @@ -170,6 +175,7 @@ POST /index_long,index_double/_search ] } ``` +% TEST[continued] In the example above, values for the `index_long` index are casted to a double in order to be compatible with the values produced by the `index_double` index. It is also possible to transform a floating point field into a `long` but note that in this case floating points are replaced by the largest value that is less than or equal (greater than or equal if the value is negative) to the argument and is equal to a mathematical integer. @@ -185,6 +191,7 @@ PUT /index_double } } ``` +% TEST[continued] ```console PUT /index_long @@ -196,6 +203,7 @@ PUT /index_long } } ``` +% TEST[continued] Values in these indices are stored with different resolutions so sorting on these fields will always sort the `date` before the `date_nanos` (ascending order). With the `numeric_type` type option it is possible to set a single resolution for the sort, setting to `date` will convert the `date_nanos` to the millisecond resolution while `date_nanos` will convert the values in the `date` field to the nanoseconds resolution: @@ -211,6 +219,7 @@ POST /index_long,index_double/_search ] } ``` +% TEST[continued] ::::{warning} To avoid overflow, the conversion to `date_nanos` cannot be applied on dates before 1970 and after 2262 as nanoseconds are represented as longs. diff --git a/docs/reference/ingestion-tools/enrich-processor/append-processor.md b/docs/reference/ingestion-tools/enrich-processor/append-processor.md index 1059dfee92162..82b95a57e58d6 100644 --- a/docs/reference/ingestion-tools/enrich-processor/append-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/append-processor.md @@ -31,4 +31,5 @@ $$$append-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/attachment.md b/docs/reference/ingestion-tools/enrich-processor/attachment.md index 9826b461f28e5..1edec4927de97 100644 --- a/docs/reference/ingestion-tools/enrich-processor/attachment.md +++ b/docs/reference/ingestion-tools/enrich-processor/attachment.md @@ -79,6 +79,7 @@ The document’s `attachment` object contains extracted properties for the file: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] ## Exported fields [attachment-fields] @@ -183,6 +184,7 @@ The document’s `_source` object includes the original binary field: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] ## Use the attachment processor with CBOR [attachment-cbor] @@ -280,6 +282,7 @@ Returns this: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] ```console PUT _ingest/pipeline/attachment @@ -325,6 +328,7 @@ Returns this: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] ## Using the attachment processor with arrays [attachment-with-arrays] @@ -347,6 +351,7 @@ For example, given the following source: ] } ``` +% NOTCONSOLE In this case, we want to process the data field in each element of the attachments field and insert the properties into the document so the following `foreach` processor is used: @@ -419,6 +424,7 @@ Returns this: } } ``` +% TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] Note that the `target_field` needs to be set, otherwise the default value is used which is a top level field `attachment`. The properties on this top level field will contain the value of the first attachment only. However, by specifying the `target_field` on to a value on `_ingest._value` it will correctly associate the properties with the correct attachment. diff --git a/docs/reference/ingestion-tools/enrich-processor/bytes-processor.md b/docs/reference/ingestion-tools/enrich-processor/bytes-processor.md index 18fabdd3b9911..f14d1f26fd750 100644 --- a/docs/reference/ingestion-tools/enrich-processor/bytes-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/bytes-processor.md @@ -31,4 +31,5 @@ $$$bytes-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/community-id-processor.md b/docs/reference/ingestion-tools/enrich-processor/community-id-processor.md index c0114eca62c0f..d582027fbc8c6 100644 --- a/docs/reference/ingestion-tools/enrich-processor/community-id-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/community-id-processor.md @@ -45,6 +45,7 @@ Here is an example definition of the community ID processor: ] } ``` +% NOTCONSOLE When the above processor executes on the following document: @@ -65,6 +66,7 @@ When the above processor executes on the following document: } } ``` +% NOTCONSOLE It produces this result: @@ -84,4 +86,5 @@ It produces this result: } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/convert-processor.md b/docs/reference/ingestion-tools/enrich-processor/convert-processor.md index 9145a267fbbf5..5bd7bc65a7698 100644 --- a/docs/reference/ingestion-tools/enrich-processor/convert-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/convert-processor.md @@ -45,4 +45,5 @@ PUT _ingest/pipeline/my-pipeline-id ] } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/csv-processor.md b/docs/reference/ingestion-tools/enrich-processor/csv-processor.md index cc0ce9757acd4..73374e7df3e9c 100644 --- a/docs/reference/ingestion-tools/enrich-processor/csv-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/csv-processor.md @@ -34,6 +34,7 @@ $$$csv-options$$$ } } ``` +% NOTCONSOLE If the `trim` option is enabled then any whitespace in the beginning and in the end of each unquoted field will be trimmed. For example with configuration above, a value of `A, B` will result in field `field2` having value `{{nbsp}}B` (with space at the beginning). If `trim` is enabled `A, B` will result in field `field2` having value `B` (no whitespace). Quoted fields will be left untouched. diff --git a/docs/reference/ingestion-tools/enrich-processor/date-index-name-processor.md b/docs/reference/ingestion-tools/enrich-processor/date-index-name-processor.md index 8e650e1aeb9e1..51cb806e4015e 100644 --- a/docs/reference/ingestion-tools/enrich-processor/date-index-name-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/date-index-name-processor.md @@ -39,6 +39,7 @@ PUT /my-index/_doc/1?pipeline=monthlyindex "date1" : "2016-04-25T12:02:01.789Z" } ``` +% TEST[continued] ```console-result { @@ -55,6 +56,7 @@ PUT /my-index/_doc/1?pipeline=monthlyindex "_primary_term" : 1 } ``` +% TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] The above request will not index this document into the `my-index` index, but into the `my-index-2016-04-01` index because it was rounded by month. This is because the date-index-name-processor overrides the `_index` property of the document. @@ -107,6 +109,7 @@ and the result: ] } ``` +% TESTRESPONSE[s/2016-11-08T19:43:03.850+0000/$body.docs.0.doc._ingest.timestamp/] The above example shows that `_index` was set to ``. Elasticsearch understands this to mean `2016-04-01` as is explained in the [date math index name documentation](/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) diff --git a/docs/reference/ingestion-tools/enrich-processor/date-processor.md b/docs/reference/ingestion-tools/enrich-processor/date-processor.md index c80620620b3a2..54f7b2e1e7cc2 100644 --- a/docs/reference/ingestion-tools/enrich-processor/date-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/date-processor.md @@ -42,6 +42,7 @@ Here is an example that adds the parsed date to the `timestamp` field based on t ] } ``` +% NOTCONSOLE The `timezone` and `locale` processor parameters are templated. This means that their values can be extracted from fields within documents. The example below shows how to extract the locale/timezone details from existing fields, `my_timezone` and `my_locale`, in the ingested document that contain the timezone and locale values. @@ -61,4 +62,5 @@ The `timezone` and `locale` processor parameters are templated. This means that ] } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/dissect-processor.md b/docs/reference/ingestion-tools/enrich-processor/dissect-processor.md index 51bed2db88726..425ef9981af81 100644 --- a/docs/reference/ingestion-tools/enrich-processor/dissect-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/dissect-processor.md @@ -43,6 +43,7 @@ and result in a document with the following fields: } } ``` +% NOTCONSOLE A dissect pattern is defined by the parts of the string that will be discarded. In the previous example, the first part to be discarded is a single space. Dissect finds this space, then assigns the value of `clientip` everything up until that space. Next, dissect matches the `[` and then `]` and then assigns `@timestamp` to everything in-between `[` and `]`. Paying special attention to the parts of the string to discard will help build successful dissect patterns. @@ -72,6 +73,7 @@ $$$dissect-options$$$ } } ``` +% NOTCONSOLE ## Dissect key modifiers [dissect-key-modifiers] diff --git a/docs/reference/ingestion-tools/enrich-processor/dot-expand-processor.md b/docs/reference/ingestion-tools/enrich-processor/dot-expand-processor.md index 92eed478491eb..65abb6c6002fa 100644 --- a/docs/reference/ingestion-tools/enrich-processor/dot-expand-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/dot-expand-processor.md @@ -29,6 +29,7 @@ $$$dot-expander-options$$$ } } ``` +% NOTCONSOLE For example the dot expand processor would turn this document: @@ -37,6 +38,7 @@ For example the dot expand processor would turn this document: "foo.bar" : "value" } ``` +% NOTCONSOLE into: @@ -47,6 +49,7 @@ into: } } ``` +% NOTCONSOLE If there is already a `bar` field nested under `foo` then this processor merges the `foo.bar` field into it. If the field is a scalar value then it will turn that field into an array field. @@ -60,6 +63,7 @@ For example, the following document: } } ``` +% NOTCONSOLE is transformed by the `dot_expander` processor into: @@ -70,6 +74,7 @@ is transformed by the `dot_expander` processor into: } } ``` +% NOTCONSOLE Contrast that with when the `override` option is set to `true`. @@ -81,6 +86,7 @@ Contrast that with when the `override` option is set to `true`. } } ``` +% NOTCONSOLE In that case, the value of the expanded field overrides the value of the nested object. @@ -91,6 +97,7 @@ In that case, the value of the expanded field overrides the value of the nested } } ``` +% NOTCONSOLE
The value of `field` can also be set to a `*` to expand all top-level dotted field names: @@ -102,6 +109,7 @@ The value of `field` can also be set to a `*` to expand all top-level dotted fie } } ``` +% NOTCONSOLE The dot expand processor would turn this document: @@ -111,6 +119,7 @@ The dot expand processor would turn this document: "baz.qux" : "value" } ``` +% NOTCONSOLE into: @@ -124,6 +133,7 @@ into: } } ``` +% NOTCONSOLE
If the dotted field is nested within a non-dotted structure, then use the `path` option to navigate the non-dotted structure: @@ -136,6 +146,7 @@ If the dotted field is nested within a non-dotted structure, then use the `path` } } ``` +% NOTCONSOLE The dot expand processor would turn this document: @@ -147,6 +158,7 @@ The dot expand processor would turn this document: } } ``` +% NOTCONSOLE into: @@ -160,6 +172,7 @@ into: } } ``` +% NOTCONSOLE
If any field outside of the leaf field conflicts with a pre-existing field of the same name, then that field needs to be renamed first. @@ -172,6 +185,7 @@ Consider the following document: "foo.bar": "value2" } ``` +% NOTCONSOLE Then the `foo` needs to be renamed first before the `dot_expander` processor is applied. So in order for the `foo.bar` field to properly be expanded into the `bar` field under the `foo` field the following pipeline should be used: @@ -192,6 +206,7 @@ Then the `foo` needs to be renamed first before the `dot_expander` processor is ] } ``` +% NOTCONSOLE The reason for this is that Ingest doesn’t know how to automatically cast a scalar field to an object field. diff --git a/docs/reference/ingestion-tools/enrich-processor/drop-processor.md b/docs/reference/ingestion-tools/enrich-processor/drop-processor.md index b6b5fcb1baf57..d8dac8fa96e07 100644 --- a/docs/reference/ingestion-tools/enrich-processor/drop-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/drop-processor.md @@ -26,4 +26,5 @@ $$$drop-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/fail-processor.md b/docs/reference/ingestion-tools/enrich-processor/fail-processor.md index 514971caae23f..51807b75c6da4 100644 --- a/docs/reference/ingestion-tools/enrich-processor/fail-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/fail-processor.md @@ -28,4 +28,5 @@ $$$fail-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/fingerprint-processor.md b/docs/reference/ingestion-tools/enrich-processor/fingerprint-processor.md index 7c978c83ce429..82628075d6930 100644 --- a/docs/reference/ingestion-tools/enrich-processor/fingerprint-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/fingerprint-processor.md @@ -78,4 +78,5 @@ Which produces the following result: ] } ``` +% TESTRESPONSE[s/.../"_index":"_index","_id":"_id","_version":"-3","_ingest":{"timestamp":$body.docs.0.doc._ingest.timestamp},/] diff --git a/docs/reference/ingestion-tools/enrich-processor/foreach-processor.md b/docs/reference/ingestion-tools/enrich-processor/foreach-processor.md index a1d6e0a63bc85..869fc8a5d0cc0 100644 --- a/docs/reference/ingestion-tools/enrich-processor/foreach-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/foreach-processor.md @@ -60,6 +60,7 @@ Assume the following document: "values" : ["foo", "bar", "baz"] } ``` +% NOTCONSOLE When this `foreach` processor operates on this sample document: @@ -75,6 +76,7 @@ When this `foreach` processor operates on this sample document: } } ``` +% NOTCONSOLE Then the document will look like this after processing: @@ -83,6 +85,7 @@ Then the document will look like this after processing: "values" : ["FOO", "BAR", "BAZ"] } ``` +% NOTCONSOLE ### Array of objects [foreach-array-objects-ex] @@ -103,6 +106,7 @@ Assume the following document: ] } ``` +% NOTCONSOLE In this case, the `id` field needs to be removed, so the following `foreach` processor is used: @@ -118,6 +122,7 @@ In this case, the `id` field needs to be removed, so the following `foreach` pro } } ``` +% NOTCONSOLE After processing the result is: @@ -133,6 +138,7 @@ After processing the result is: ] } ``` +% NOTCONSOLE For another array of objects example, refer to the [attachment processor documentation](/reference/ingestion-tools/enrich-processor/attachment.md#attachment-with-arrays). @@ -162,6 +168,7 @@ You can also use the `foreach` processor on object fields. For example, the foll } } ``` +% NOTCONSOLE The following `foreach` processor changes the value of `products.display_name` to uppercase. @@ -177,6 +184,7 @@ The following `foreach` processor changes the value of `products.display_name` t } } ``` +% NOTCONSOLE When run on the document, the `foreach` processor returns: @@ -201,6 +209,7 @@ When run on the document, the `foreach` processor returns: } } ``` +% NOTCONSOLE The following `foreach` processor sets each element’s key to the value of `products.display_name`. If `products.display_name` contains an empty string, the processor deletes the element. @@ -217,6 +226,7 @@ The following `foreach` processor sets each element’s key to the value of `pro } } ``` +% NOTCONSOLE When run on the previous document, the `foreach` processor returns: @@ -236,6 +246,7 @@ When run on the previous document, the `foreach` processor returns: } } ``` +% NOTCONSOLE ### Failure handling [failure-handling-ex] @@ -262,6 +273,7 @@ The wrapped processor can have a `on_failure` definition. For example, the `id` } } ``` +% NOTCONSOLE In this example, if the `remove` processor does fail, then the array elements that have been processed thus far will be updated. diff --git a/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md b/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md index a904942aa12b7..a45f0f7b061b5 100644 --- a/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md @@ -93,6 +93,7 @@ Which returns: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term":1/"_primary_term" : $body._primary_term/] Here is an example that uses the default country database and adds the geographical information to the `geo` field based on the `ip` field. Note that this database is downloaded automatically. So this: @@ -137,6 +138,7 @@ returns this: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] Not all IP addresses find geo information from the database, When this occurs, no `target_field` is inserted into the document. @@ -178,6 +180,7 @@ Which returns: } } ``` +% TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] ### Recognizing Location as a Geopoint [ingest-geoip-mappings-note] diff --git a/docs/reference/ingestion-tools/enrich-processor/grok-processor.md b/docs/reference/ingestion-tools/enrich-processor/grok-processor.md index d1d754e099725..ff26813a333a0 100644 --- a/docs/reference/ingestion-tools/enrich-processor/grok-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/grok-processor.md @@ -83,6 +83,7 @@ This pipeline will insert these named captures as new fields within the document ] } ``` +% TESTRESPONSE[s/2016-11-08T19:43:03.850+0000/$body.docs.0.doc._ingest.timestamp/] ## Custom Patterns [custom-patterns] @@ -108,6 +109,7 @@ You can add your own patterns to a processor definition under the `pattern_defin ] } ``` +% NOTCONSOLE ## Providing Multiple Match Patterns [trace-match] @@ -166,6 +168,7 @@ response: ] } ``` +% TESTRESPONSE[s/2016-11-08T19:43:03.850+0000/$body.docs.0.doc._ingest.timestamp/] Both patterns will set the field `pet` with the appropriate match, but what if we want to trace which of our patterns matched and populated our fields? We can do this with the `trace_match` parameter. Here is the output of that same pipeline, but with `"trace_match": true` configured: @@ -190,6 +193,7 @@ Both patterns will set the field `pet` with the appropriate match, but what if w ] } ``` +% TESTRESPONSE[s/2016-11-08T19:43:03.850+0000/$body.docs.0.doc._ingest.timestamp/] In the above response, you can see that the index of the pattern that matched was `"1"`. This is to say that it was the second (index starts at zero) pattern in `patterns` to match. @@ -214,6 +218,7 @@ The above request will return a response body containing a key-value representat ... } ``` +% NOTCONSOLE By default, the API returns a list of legacy Grok patterns. These legacy patterns predate the [Elastic Common Schema (ECS)](ecs://reference/ecs-field-reference.md) and don’t use ECS field names. To return patterns that extract ECS field names, specify `v1` in the optional `ecs_compatibility` query parameter. @@ -240,6 +245,7 @@ The API returns the following response. ... } ``` +% NOTCONSOLE This can be useful to reference as the built-in patterns change across versions. @@ -270,5 +276,6 @@ PUT _cluster/settings } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/gsub-processor.md b/docs/reference/ingestion-tools/enrich-processor/gsub-processor.md index 4d6c6939f686d..84fe7eb0acd0f 100644 --- a/docs/reference/ingestion-tools/enrich-processor/gsub-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/gsub-processor.md @@ -33,4 +33,5 @@ $$$gsub-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/htmlstrip-processor.md b/docs/reference/ingestion-tools/enrich-processor/htmlstrip-processor.md index 9adae0eb17224..e4a9880c60efb 100644 --- a/docs/reference/ingestion-tools/enrich-processor/htmlstrip-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/htmlstrip-processor.md @@ -34,4 +34,5 @@ $$$htmlstrip-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/inference-processor.md b/docs/reference/ingestion-tools/enrich-processor/inference-processor.md index ca83de627e3e8..c4cbe705dd57c 100644 --- a/docs/reference/ingestion-tools/enrich-processor/inference-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/inference-processor.md @@ -54,6 +54,7 @@ If the specified `output_field` already exists in the ingest document, it won’ } } ``` +% NOTCONSOLE ## Configuring multiple inputs [_configuring_multiple_inputs] @@ -76,6 +77,7 @@ The `content` and `title` fields will be read from the incoming document and sen } } ``` +% NOTCONSOLE Selecting the input fields with `input_output` is incompatible with the `target_field` and `field_map` options. @@ -93,6 +95,7 @@ Selecting the input fields with `input_output` is incompatible with the `target_ } } ``` +% NOTCONSOLE ## {{classification-cap}} configuration options [inference-processor-classification-opt] @@ -593,6 +596,7 @@ Regression configuration for inference. } } ``` +% NOTCONSOLE This configuration specifies a `regression` inference and the results are written to the `my_regression` field contained in the `target_field` results object. The `field_map` configuration maps the field `original_fieldname` from the source document to the field expected by the model. @@ -608,6 +612,7 @@ This configuration specifies a `regression` inference and the results are writte } } ``` +% NOTCONSOLE This configuration specifies a `classification` inference. The number of categories for which the predicted probabilities are reported is 2 (`num_top_classes`). The result is written to the `prediction` field and the top classes to the `probabilities` field. Both fields are contained in the `target_field` results object. @@ -632,6 +637,7 @@ To get the full benefit of aggregating and searching for [{{feat-imp}}](docs-con } } ``` +% NOTCONSOLE The mapping field name for {{feat-imp}} (in the example above, it is `ml.inference.feature_importance`) is compounded as follows: @@ -648,6 +654,7 @@ For example, if you provide a tag `foo` in the definition as you can see below: ... } ``` +% NOTCONSOLE Then, the {{feat-imp}} value is written to the `ml.inference.foo.feature_importance` field. @@ -659,6 +666,7 @@ You can also specify the target field as follows: "target_field": "my_field" } ``` +% NOTCONSOLE In this case, {{feat-imp}} is exposed in the `my_field.foo.feature_importance` field. @@ -693,6 +701,7 @@ PUT _ingest/pipeline/query_helper_pipeline ] } ``` +% TEST[skip: An inference endpoint is required.] 1. The `prompt` field contains the prompt used for the completion task, created with [Painless](docs-content://explore-analyze/scripting/modules-scripting-painless.md). `+ ctx.content` appends the natural language input to the prompt. 2. The ID of the pre-configured {{infer}} endpoint, which utilizes the [`openai` service](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the `completion` task type. @@ -712,6 +721,7 @@ POST _ingest/pipeline/query_helper_pipeline/_simulate ] } ``` +% TEST[skip: An inference processor with an inference endpoint is required.] 1. The natural language query used to generate an {{es}} query within the prompt created by the {{infer}} processor. diff --git a/docs/reference/ingestion-tools/enrich-processor/ingest-circle-processor.md b/docs/reference/ingestion-tools/enrich-processor/ingest-circle-processor.md index 79a7de3100704..03024207448f4 100644 --- a/docs/reference/ingestion-tools/enrich-processor/ingest-circle-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/ingest-circle-processor.md @@ -72,6 +72,7 @@ PUT circles/_doc/1?pipeline=polygonize_circles GET circles/_doc/1 ``` +% TEST[continued] The response from the above index request: @@ -88,6 +89,7 @@ The response from the above index request: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/] ## Example: Circle defined in GeoJSON [_example_circle_defined_in_geojson] @@ -106,6 +108,7 @@ PUT circles/_doc/2?pipeline=polygonize_circles GET circles/_doc/2 ``` +% TEST[continued] The response from the above index request: @@ -134,6 +137,7 @@ The response from the above index request: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/] ## Notes on Accuracy [circle-processor-notes] diff --git a/docs/reference/ingestion-tools/enrich-processor/ingest-geo-grid-processor.md b/docs/reference/ingestion-tools/enrich-processor/ingest-geo-grid-processor.md index 95556f5808310..286750f7c7331 100644 --- a/docs/reference/ingestion-tools/enrich-processor/ingest-geo-grid-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/ingest-geo-grid-processor.md @@ -84,6 +84,7 @@ PUT geocells/_doc/1?pipeline=geotile2shape GET geocells/_doc/1 ``` +% TEST[continued] The response shows how the ingest-processor has replaced the `geocell` field with an indexable `geo_shape`: @@ -106,6 +107,7 @@ The response shows how the ingest-processor has replaced the `geocell` field wit } } ``` +% TESTRESPONSE[s/"_version": \d+/"_version": $body._version/ s/"_seq_no": \d+/"_seq_no": $body._seq_no/ s/"_primary_term": 1/"_primary_term": $body._primary_term/] ![Kibana map with showing the geotile at 4/8/5 and its four child cells](../../../images/geogrid_tile.png "") @@ -122,6 +124,7 @@ PUT geocells/_doc/1?pipeline=geohex2shape GET geocells/_doc/1 ``` +% TEST[continued] The response shows how the ingest-processor has replaced the `geocell` field with an indexable `geo_shape`: @@ -138,6 +141,7 @@ The response shows how the ingest-processor has replaced the `geocell` field wit } } ``` +% TESTRESPONSE[s/"_version": \d+/"_version": $body._version/ s/"_seq_no": \d+/"_seq_no": $body._seq_no/ s/"_primary_term": 1/"_primary_term": $body._primary_term/] ![Kibana map with showing an H3 cell](../../../images/geogrid_h3.png "") @@ -178,6 +182,7 @@ PUT geocells/_doc/1?pipeline=geohex2shape GET geocells/_doc/1 ``` +% TEST[continued] The response from this index request: @@ -214,6 +219,7 @@ The response from this index request: } } ``` +% TESTRESPONSE[s/"_version": \d+/"_version": $body._version/ s/"_seq_no": \d+/"_seq_no": $body._seq_no/ s/"_primary_term": 1/"_primary_term": $body._primary_term/] This additional information will then enable, for example, creating a visualization of the H3 cell, its children and its intersecting non-children cells. diff --git a/docs/reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md b/docs/reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md index 9153cb77056bc..9e66aca453231 100644 --- a/docs/reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md @@ -39,4 +39,5 @@ The following example adds all user details for the current authenticated user t ] } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/ip-location-processor.md b/docs/reference/ingestion-tools/enrich-processor/ip-location-processor.md index cf9d202917c79..c317e7508ec0f 100644 --- a/docs/reference/ingestion-tools/enrich-processor/ip-location-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/ip-location-processor.md @@ -93,6 +93,7 @@ Which returns: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term":1/"_primary_term" : $body._primary_term/] Here is an example that uses the default country database and adds the geographical information to the `geo` field based on the `ip` field. Note that this database is downloaded automatically. So this: @@ -137,6 +138,7 @@ returns this: } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] Not all IP addresses find geo information from the database, When this occurs, no `target_field` is inserted into the document. @@ -178,5 +180,6 @@ Which returns: } } ``` +% TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] diff --git a/docs/reference/ingestion-tools/enrich-processor/join-processor.md b/docs/reference/ingestion-tools/enrich-processor/join-processor.md index 3a58553c5c554..5cc958cd1d9a7 100644 --- a/docs/reference/ingestion-tools/enrich-processor/join-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/join-processor.md @@ -30,4 +30,5 @@ $$$join-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/json-processor.md b/docs/reference/ingestion-tools/enrich-processor/json-processor.md index f985f65834834..7637259646d7d 100644 --- a/docs/reference/ingestion-tools/enrich-processor/json-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/json-processor.md @@ -37,6 +37,7 @@ Suppose you provide this configuration of the `json` processor: } } ``` +% NOTCONSOLE If the following document is processed: @@ -45,6 +46,7 @@ If the following document is processed: "string_source": "{\"foo\": 2000}" } ``` +% NOTCONSOLE after the `json` processor operates on it, it will look like: @@ -56,6 +58,7 @@ after the `json` processor operates on it, it will look like: } } ``` +% NOTCONSOLE If the following configuration is provided, omitting the optional `target_field` setting: @@ -66,6 +69,7 @@ If the following configuration is provided, omitting the optional `target_field` } } ``` +% NOTCONSOLE then after the `json` processor operates on this document: @@ -74,6 +78,7 @@ then after the `json` processor operates on this document: "source_and_target": "{\"foo\": 2000}" } ``` +% NOTCONSOLE it will look like: @@ -84,6 +89,7 @@ it will look like: } } ``` +% NOTCONSOLE This illustrates that, unless it is explicitly named in the processor configuration, the `target_field` is the same field provided in the required `field` configuration. diff --git a/docs/reference/ingestion-tools/enrich-processor/kv-processor.md b/docs/reference/ingestion-tools/enrich-processor/kv-processor.md index 35950d7108118..c401c3eb4bc8a 100644 --- a/docs/reference/ingestion-tools/enrich-processor/kv-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/kv-processor.md @@ -20,6 +20,7 @@ For example, if you have a log message which contains `ip=1.2.3.4 error=REFUSED` } } ``` +% NOTCONSOLE ::::{tip} Using the KV Processor can result in field names that you cannot control. Consider using the [Flattened](/reference/elasticsearch/mapping-reference/flattened.md) data type instead, which maps an entire object as a single field and allows for simple searches over its contents. diff --git a/docs/reference/ingestion-tools/enrich-processor/lowercase-processor.md b/docs/reference/ingestion-tools/enrich-processor/lowercase-processor.md index e9d36781821bd..bc87fe36aef0d 100644 --- a/docs/reference/ingestion-tools/enrich-processor/lowercase-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/lowercase-processor.md @@ -29,4 +29,5 @@ $$$lowercase-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/network-direction-processor.md b/docs/reference/ingestion-tools/enrich-processor/network-direction-processor.md index 2c90392e439d5..59ac2329973ce 100644 --- a/docs/reference/ingestion-tools/enrich-processor/network-direction-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/network-direction-processor.md @@ -100,4 +100,6 @@ Which produces the following result: ] } ``` +% TESTRESPONSE[s/.../"_index":"_index","_id":"_id","_version":"-3","_ingest":{"timestamp":$body.docs.0.doc._ingest.timestamp},/] +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/pipeline-processor.md b/docs/reference/ingestion-tools/enrich-processor/pipeline-processor.md index c4da79bfce30b..3f5545d67dabb 100644 --- a/docs/reference/ingestion-tools/enrich-processor/pipeline-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/pipeline-processor.md @@ -28,6 +28,7 @@ $$$pipeline-options$$$ } } ``` +% NOTCONSOLE The name of the current pipeline can be accessed from the `_ingest.pipeline` ingest metadata key. @@ -71,6 +72,7 @@ PUT _ingest/pipeline/pipelineB ] } ``` +% TEST[continued] Now indexing a document while applying the outer pipeline will see the inner pipeline executed from the outer pipeline: @@ -80,6 +82,7 @@ PUT /my-index-000001/_doc/1?pipeline=pipelineB "field": "value" } ``` +% TEST[continued] Response from the index request: @@ -98,6 +101,7 @@ Response from the index request: "_primary_term": 1 } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/] Indexed document: @@ -108,4 +112,5 @@ Indexed document: "outer_pipeline_set": "outer" } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/redact-processor.md b/docs/reference/ingestion-tools/enrich-processor/redact-processor.md index 92cb5fd044b6a..8e362f5b08f36 100644 --- a/docs/reference/ingestion-tools/enrich-processor/redact-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/redact-processor.md @@ -80,6 +80,7 @@ The document in the response still contains the `message` field but now the IP a ] } ``` +% TESTRESPONSE[s/2023-02-01T16:08:39.419056008Z/$body.docs.0.doc._ingest.timestamp/] The IP address is replaced with the word `client` because that is what is specified in the Grok pattern `%{IP:client}`. The `<` and `>` tokens which surround the pattern name are configurable using the `prefix` and `suffix` options. @@ -135,6 +136,7 @@ In the response both the IP `55.3.244.1` and email address `test@elastic.co` hav ] } ``` +% TESTRESPONSE[s/2023-02-01T16:53:14.560005377Z/$body.docs.0.doc._ingest.timestamp/] ## Custom patterns [redact-custom-patterns] @@ -197,6 +199,7 @@ The username is redacted in the response. ] } ``` +% TESTRESPONSE[s/2023-02-01T16:53:14.560005377Z/$body.docs.0.doc._ingest.timestamp/] ## Grok watchdog [grok-watchdog-redact] diff --git a/docs/reference/ingestion-tools/enrich-processor/registered-domain-processor.md b/docs/reference/ingestion-tools/enrich-processor/registered-domain-processor.md index e11ae91b7b4a0..4c18f74fc7ef0 100644 --- a/docs/reference/ingestion-tools/enrich-processor/registered-domain-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/registered-domain-processor.md @@ -72,4 +72,5 @@ Which produces the following result: ] } ``` +% TESTRESPONSE[s/.../"_index":"_index","_id":"_id","_version":"-3","_ingest":{"timestamp":$body.docs.0.doc._ingest.timestamp},/] diff --git a/docs/reference/ingestion-tools/enrich-processor/remove-processor.md b/docs/reference/ingestion-tools/enrich-processor/remove-processor.md index 7201391ffb750..ceba23d69a150 100644 --- a/docs/reference/ingestion-tools/enrich-processor/remove-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/remove-processor.md @@ -31,6 +31,7 @@ Here is an example to remove a single field: } } ``` +% NOTCONSOLE To remove multiple fields, you can use the following query: @@ -41,6 +42,7 @@ To remove multiple fields, you can use the following query: } } ``` +% NOTCONSOLE You can also choose to remove all fields other than a specified list: @@ -51,4 +53,5 @@ You can also choose to remove all fields other than a specified list: } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/rename-processor.md b/docs/reference/ingestion-tools/enrich-processor/rename-processor.md index 35c93d9094889..f1384787914b5 100644 --- a/docs/reference/ingestion-tools/enrich-processor/rename-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/rename-processor.md @@ -31,4 +31,5 @@ $$$rename-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/reroute-processor.md b/docs/reference/ingestion-tools/enrich-processor/reroute-processor.md index 7464f351b2d73..4d6a225de321f 100644 --- a/docs/reference/ingestion-tools/enrich-processor/reroute-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/reroute-processor.md @@ -31,6 +31,7 @@ Note that the client needs to have permissions to the final target. Otherwise, t ```js {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [8-dt9H8BqGblnY2uSI--] of user [elastic/fleet-server] on indices [logs-foo-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]"} ``` +% NOTCONSOLE $$$reroute-options$$$ @@ -56,6 +57,7 @@ The `if` option can be used to define the condition in which the document should } } ``` +% NOTCONSOLE The dataset and namespace options can contain either a single value or a list of values that are used as a fallback. If a field reference evaluates to `null`, is not present in the document, the next value or field reference is used. If a field reference evaluates to a non-`String` value, the processor fails. @@ -72,4 +74,5 @@ In the following example, the processor would first try to resolve the value for } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/script-processor.md b/docs/reference/ingestion-tools/enrich-processor/script-processor.md index bf30543f3dfdf..4235db6590cc8 100644 --- a/docs/reference/ingestion-tools/enrich-processor/script-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/script-processor.md @@ -89,6 +89,7 @@ The processor produces: ] } ``` +% TESTRESPONSE[s/.../"_index":"_index","_id":"_id","_version":"-3","_ingest":{"timestamp":$body.docs.0.doc._ingest.timestamp},/] ## Access metadata fields [script-processor-access-metadata-fields] @@ -142,4 +143,5 @@ The processor changes the document’s `_index` to `fr-catalog` from `generic-in ] } ``` +% TESTRESPONSE[s/.../"_id":"_id","_version":"-3","_ingest":{"timestamp":$body.docs.0.doc._ingest.timestamp},/] diff --git a/docs/reference/ingestion-tools/enrich-processor/set-processor.md b/docs/reference/ingestion-tools/enrich-processor/set-processor.md index 48052f02de8c8..92d0e40c54523 100644 --- a/docs/reference/ingestion-tools/enrich-processor/set-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/set-processor.md @@ -34,6 +34,7 @@ $$$set-options$$$ } } ``` +% NOTCONSOLE This processor can also be used to copy data from one field to another. For example: @@ -89,6 +90,7 @@ Result: ] } ``` +% TESTRESPONSE[s/2019-03-11T21:54:37.909224Z/$body.docs.0.doc._ingest.timestamp/] This processor can also access array fields using dot notation: @@ -147,6 +149,7 @@ Result: ] } ``` +% TESTRESPONSE[s/2023-05-05T16:04:16.456475214Z/$body.docs.0.doc._ingest.timestamp/] The contents of a field including complex values such as arrays and objects can be copied to another field using `copy_from`: @@ -198,4 +201,5 @@ Result: ] } ``` +% TESTRESPONSE[s/2020-09-30T12:55:17.742795Z/$body.docs.0.doc._ingest.timestamp/] diff --git a/docs/reference/ingestion-tools/enrich-processor/sort-processor.md b/docs/reference/ingestion-tools/enrich-processor/sort-processor.md index b5e9cbf0a5df2..78f93d263fdd3 100644 --- a/docs/reference/ingestion-tools/enrich-processor/sort-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/sort-processor.md @@ -30,4 +30,5 @@ $$$sort-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/split-processor.md b/docs/reference/ingestion-tools/enrich-processor/split-processor.md index a063cc3431f58..058cc95aca75d 100644 --- a/docs/reference/ingestion-tools/enrich-processor/split-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/split-processor.md @@ -32,6 +32,7 @@ $$$split-options$$$ } } ``` +% NOTCONSOLE 1. Treat all consecutive whitespace characters as a single separator @@ -47,4 +48,5 @@ If the `preserve_trailing` option is enabled, any trailing empty fields in the i } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/terminate-processor.md b/docs/reference/ingestion-tools/enrich-processor/terminate-processor.md index 066ee1ae440c2..dede34db50b91 100644 --- a/docs/reference/ingestion-tools/enrich-processor/terminate-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/terminate-processor.md @@ -29,4 +29,5 @@ $$$terminate-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/trim-processor.md b/docs/reference/ingestion-tools/enrich-processor/trim-processor.md index 316d1c696b518..e53fb046ff3fd 100644 --- a/docs/reference/ingestion-tools/enrich-processor/trim-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/trim-processor.md @@ -34,4 +34,5 @@ $$$trim-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/uppercase-processor.md b/docs/reference/ingestion-tools/enrich-processor/uppercase-processor.md index 382574820e872..3a098a658c777 100644 --- a/docs/reference/ingestion-tools/enrich-processor/uppercase-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/uppercase-processor.md @@ -29,4 +29,5 @@ $$$uppercase-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/uri-parts-processor.md b/docs/reference/ingestion-tools/enrich-processor/uri-parts-processor.md index ec274fcf26b85..0b974e7d23e8d 100644 --- a/docs/reference/ingestion-tools/enrich-processor/uri-parts-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/uri-parts-processor.md @@ -41,6 +41,7 @@ Here is an example definition of the URI parts processor: ] } ``` +% NOTCONSOLE When the above processor executes on the following document: @@ -51,6 +52,7 @@ When the above processor executes on the following document: } } ``` +% NOTCONSOLE It produces this result: @@ -72,4 +74,5 @@ It produces this result: } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/urldecode-processor.md b/docs/reference/ingestion-tools/enrich-processor/urldecode-processor.md index cf74d0266a1bb..6667c00cd109a 100644 --- a/docs/reference/ingestion-tools/enrich-processor/urldecode-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/urldecode-processor.md @@ -29,4 +29,5 @@ $$$urldecode-options$$$ } } ``` +% NOTCONSOLE diff --git a/docs/reference/ingestion-tools/enrich-processor/user-agent-processor.md b/docs/reference/ingestion-tools/enrich-processor/user-agent-processor.md index e4a88eca835a9..fc731ba52a9fa 100644 --- a/docs/reference/ingestion-tools/enrich-processor/user-agent-processor.md +++ b/docs/reference/ingestion-tools/enrich-processor/user-agent-processor.md @@ -73,6 +73,7 @@ Which returns } } ``` +% TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/] ### Using a custom regex file [_using_a_custom_regex_file] diff --git a/docs/reference/ingestion-tools/search-connectors/api-tutorial.md b/docs/reference/ingestion-tools/search-connectors/api-tutorial.md index 2130a16404d5b..91eb27f2c9bab 100644 --- a/docs/reference/ingestion-tools/search-connectors/api-tutorial.md +++ b/docs/reference/ingestion-tools/search-connectors/api-tutorial.md @@ -2,7 +2,7 @@ navigation_title: "API tutorial" applies_to: stack: ga - serverless: + serverless: elasticsearch: ga mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/es-connectors-tutorial-api.html @@ -66,6 +66,7 @@ Let’s test that we can access {{es}}: ```sh curl -s -X GET -u elastic:$ELASTIC_PASSWORD http://localhost:9200 ``` +% NOTCONSOLE Note: With {{es}} running locally, you will need to pass the username and password to authenticate against {{es}} in the configuration file for the connector service. @@ -87,6 +88,7 @@ curl -s -X PUT http://localhost:9200/_connector/my-connector-id \ "service_type": "postgresql" }' ``` +% NOTCONSOLE Refer to connectors-tutorial-api-create-api-key for instructions on creating an API key. @@ -120,6 +122,7 @@ Run the following command to download the file to the `~/data` directory: ```sh curl -L https://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_PostgreSql.sql -o ~/data/Chinook_PostgreSql.sql ``` +% NOTCONSOLE Now we need to import the example data into the PostgreSQL container and create the tables. @@ -160,6 +163,7 @@ PUT _connector/my-connector-id "service_type": "postgresql" } ``` +% TEST[skip:TODO] ::::{tip} `service_type` refers to the third-party data source you’re connecting to. @@ -217,6 +221,7 @@ POST /_security/api_key } } ``` +% TEST[skip:TODO] You’ll need to use the `encoded` value from the response as the `elasticsearch.api_key` in your configuration file. @@ -272,6 +277,7 @@ Verify your connector is connected by getting the connector status (should be `n ```console GET _connector/my-connector-id ``` +% TEST[skip:TODO] ### Configure connector [es-connectors-tutorial-api-update-connector-configuration] @@ -304,6 +310,7 @@ PUT _connector/my-connector-id/_configuration } } ``` +% TEST[skip:TODO] ::::{note} Configuration details are specific to the connector type. The keys and values will differ depending on which third-party data source you’re connecting to. Refer to the individual connectors-references,connector references for these configuration details. @@ -323,6 +330,7 @@ POST _connector/_sync_job "job_type": "full" } ``` +% TEST[skip:TODO] To store data in {{es}}, the connector needs to create an index. When we created the connector, we specified the `music` index. The connector will create and configure this {{es}} index before launching the sync job. @@ -340,6 +348,7 @@ Use the [Get sync job API](https://www.elastic.co/docs/api/doc/elasticsearch/ope ```console GET _connector/_sync_job?connector_id=my-connector-id&size=1 ``` +% TEST[skip:TODO] The job document will be updated as the sync progresses, you can check it as often as you’d like to poll for updates. @@ -350,12 +359,14 @@ Verify that data is present in the `music` index with the following API call: ```console GET music/_count ``` +% TEST[skip:TODO] {{es}} stores data in documents, which are JSON objects. List the individual documents with the following API call: ```console GET music/_search ``` +% TEST[skip:TODO] ## Troubleshooting [es-connectors-tutorial-api-troubleshooting] @@ -365,6 +376,7 @@ Use the following command to inspect the latest sync job’s status: ```console GET _connector/_sync_job?connector_id=my-connector-id&size=1 ``` +% TEST[skip:TODO] If the connector encountered any errors during the sync, you’ll find these in the `error` field. @@ -376,12 +388,14 @@ To delete the connector and its associated sync jobs run this command: ```console DELETE _connector/my-connector-id&delete_sync_jobs=true ``` +% TEST[skip:TODO] This won’t delete the Elasticsearch index that was created by the connector to store the data. Delete the `music` index by running the following command: ```console DELETE music ``` +% TEST[skip:TODO] To remove the PostgreSQL container, run the following commands: diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-azure-blob.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-azure-blob.md index ddcb94a8e08a9..92388b4f43775 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-azure-blob.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-azure-blob.md @@ -54,6 +54,7 @@ PUT _connector/my-azure_blob_storage-connector "service_type": "azure_blob_storage" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -146,6 +147,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-box.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-box.md index bd27e45192c02..f30ae72626e25 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-box.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-box.md @@ -53,6 +53,7 @@ PUT _connector/my-box-connector "service_type": "box" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -194,6 +195,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-confluence.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-confluence.md index 517e2b3fc1869..1a49c24927288 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-confluence.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-confluence.md @@ -55,6 +55,7 @@ PUT _connector/my-confluence-connector "service_type": "confluence" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -204,6 +205,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -308,6 +310,7 @@ This connector supports [advanced sync rules](/reference/ingestion-tools/search- } ] ``` +% NOTCONSOLE **Example 2**: Queries for indexing data based on `created` and `lastmodified` time. @@ -321,6 +324,7 @@ This connector supports [advanced sync rules](/reference/ingestion-tools/search- } ] ``` +% NOTCONSOLE **Example 3**: Query for indexing only given types in a **Space** with key *SD*. @@ -331,6 +335,7 @@ This connector supports [advanced sync rules](/reference/ingestion-tools/search- } ] ``` +% NOTCONSOLE ::::{note} Syncing recently created/updated items in Confluence may be delayed when using advanced sync rules, because the search endpoint used for CQL queries returns stale results in the response. For more details refer to the following issue in the [Confluence documentation](https://jira.atlassian.com/browse/CONFCLOUD-73997). diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-dropbox.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-dropbox.md index eff4299810dda..4ce1c4095c31a 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-dropbox.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-dropbox.md @@ -51,6 +51,7 @@ PUT _connector/my-dropbox-connector "service_type": "dropbox" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -214,6 +215,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -338,6 +340,7 @@ $$$es-connectors-dropbox-client-sync-rules-advanced-example-1$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-dropbox-client-sync-rules-advanced-example-2$$$ **Example: Query with file extension filter** @@ -355,6 +358,7 @@ $$$es-connectors-dropbox-client-sync-rules-advanced-example-2$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-dropbox-client-sync-rules-advanced-example-3$$$ **Example: Query with file category filter** @@ -376,6 +380,7 @@ $$$es-connectors-dropbox-client-sync-rules-advanced-example-3$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-dropbox-client-sync-rules-advanced-limitations$$$ **Limitations** diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-github.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-github.md index 664d61b127a5d..eedec8de91281 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-github.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-github.md @@ -52,6 +52,7 @@ PUT _connector/my-github-connector "service_type": "github" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -239,6 +240,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -361,6 +363,7 @@ $$$es-connectors-github-client-sync-rules-advanced-branch$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-github-client-sync-rules-advanced-issue-key$$$ **Indexing document based on issue query related to bugs via issue key** @@ -375,6 +378,7 @@ $$$es-connectors-github-client-sync-rules-advanced-issue-key$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-github-client-sync-rules-advanced-pr-key$$$ **Indexing document based on PR query related to open PR’s via PR key** @@ -389,6 +393,7 @@ $$$es-connectors-github-client-sync-rules-advanced-pr-key$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-github-client-sync-rules-advanced-issue-query-branch-name$$$ **Indexing document and files based on queries and branch name** @@ -405,6 +410,7 @@ $$$es-connectors-github-client-sync-rules-advanced-issue-query-branch-name$$$ } ] ``` +% NOTCONSOLE ::::{note} All documents pulled by a given rule are indexed regardless of whether the document has already been indexed by a previous rule. This can lead to document duplication, but the indexed documents count will differ in the logs. Check the Elasticsearch index for the actual document count. @@ -431,6 +437,7 @@ $$$es-connectors-github-client-sync-rules-advanced-overlapping$$$ } ] ``` +% NOTCONSOLE ::::{note} If `GitHub App` is selected as the authentication method, the "OWNER/" portion of the "OWNER/REPO" repository argument must be provided. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-gmail.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-gmail.md index 54ae9acf152fb..a18cb9af96339 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-gmail.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-gmail.md @@ -49,6 +49,7 @@ PUT _connector/my-gmail-connector "service_type": "gmail" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -174,6 +175,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -264,7 +266,7 @@ For example: ] } ``` - +% NOTCONSOLE ### Document level security [es-connectors-gmail-client-document-level-security] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-google-cloud.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-google-cloud.md index 2ab55b6fba7aa..58e968d22b7cf 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-google-cloud.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-google-cloud.md @@ -60,6 +60,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-google-drive.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-google-drive.md index 374fd9bfe96cc..10f6ef6cc4ee2 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-google-drive.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-google-drive.md @@ -127,6 +127,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-graphql.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-graphql.md index 02565fb5f2ee1..b564ab24bb664 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-graphql.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-graphql.md @@ -45,6 +45,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-jira.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-jira.md index 8da14162ddb85..77ee80802eccf 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-jira.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-jira.md @@ -55,6 +55,7 @@ PUT _connector/my-jira-connector "service_type": "jira" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -204,6 +205,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -319,6 +321,7 @@ This connector supports [advanced sync rules](/reference/ingestion-tools/search- } ] ``` +% NOTCONSOLE **Example 2**: Query to index data based on priority of issues for given projects. @@ -329,6 +332,7 @@ This connector supports [advanced sync rules](/reference/ingestion-tools/search- } ] ``` +% NOTCONSOLE **Example 3**: Query to index data based on assignee and created time. @@ -339,6 +343,7 @@ This connector supports [advanced sync rules](/reference/ingestion-tools/search- } ] ``` +% NOTCONSOLE ### Document level security [es-connectors-jira-client-document-level-security] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-mongodb.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-mongodb.md index 68bd416e00ed0..6104ffaa9cba8 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-mongodb.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-mongodb.md @@ -114,6 +114,7 @@ PUT _connector/my-mongodb-connector "service_type": "mongodb" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -196,6 +197,7 @@ Incorrect (`new Date()` will be interpreted as string): } } ``` +% NOTCONSOLE Correct (usage of [$$NOW](https://www.mongodb.com/docs/manual/reference/aggregation-variables/#mongodb-variable-variable.NOW)): @@ -206,7 +208,7 @@ Correct (usage of [$$NOW](https://www.mongodb.com/docs/manual/reference/aggregat { "$addFields": { "current_date": { - "$toDate": "$$NOW" + "$toDate": "$NOW" } } }, @@ -224,6 +226,7 @@ Correct (usage of [$$NOW](https://www.mongodb.com/docs/manual/reference/aggregat } } ``` +% NOTCONSOLE #### Connecting with self-signed or custom CA TLS Cert [es-connectors-mongodb-client-known-issues-tls-with-invalid-cert] @@ -272,6 +275,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -413,6 +417,7 @@ For example: } } ``` +% NOTCONSOLE `find` queries also support additional options, for example the `projection` object: @@ -433,6 +438,7 @@ For example: } } ``` +% NOTCONSOLE Where the available options are: diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-ms-sql.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-ms-sql.md index 75ce8073139cb..07e3063bdc772 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-ms-sql.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-ms-sql.md @@ -47,6 +47,7 @@ PUT _connector/my-mssql-connector "service_type": "mssql" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -200,6 +201,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -333,6 +335,7 @@ These rules fetch all records from both the `employee` and `customer` tables. Th } ] ``` +% NOTCONSOLE $$$es-connectors-ms-sql-client-sync-rules-example-one-where$$$ **Example: One WHERE query** @@ -347,6 +350,7 @@ This rule fetches only the records from the `employee` table where the `emp_id` } ] ``` +% NOTCONSOLE $$$es-connectors-ms-sql-client-sync-rules-example-one-join$$$ **Example: One JOIN query** @@ -361,6 +365,7 @@ This rule fetches records by performing an INNER JOIN between the `employee` and } ] ``` +% NOTCONSOLE ::::{warning} When using advanced rules, a query can bypass the configuration field `tables`. This will happen if the query specifies a table that doesn’t appear in the configuration. This can also happen if the configuration specifies `*` to fetch all tables while the advanced sync rule requests for only a subset of tables. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-mysql.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-mysql.md index ef1f3f5d4b5cc..6fbe3acdf69a2 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-mysql.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-mysql.md @@ -48,6 +48,7 @@ PUT _connector/my-mysql-connector "service_type": "mysql" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -274,6 +275,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -360,6 +362,7 @@ For example: } ] ``` +% NOTCONSOLE ::::{warning} When using advanced rules, a query can bypass the configuration field `tables`. This will happen if the query specifies a table that doesn’t appear in the configuration. This can also happen if the configuration specifies `*` to fetch all tables while the advanced sync rule requests for only a subset of tables. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-network-drive.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-network-drive.md index 11630d0dc49d6..a5f33531fd891 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-network-drive.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-network-drive.md @@ -107,6 +107,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -246,6 +247,7 @@ $$$es-connectors-network-drive-client-indexing-files-and-folders-recursively-wit } ] ``` +% NOTCONSOLE $$$es-connectors-network-drive-client-indexing-files-and-folders-directly-inside-folder$$$ **Indexing files and folders directly inside folder** @@ -257,6 +259,7 @@ $$$es-connectors-network-drive-client-indexing-files-and-folders-directly-inside } ] ``` +% NOTCONSOLE $$$es-connectors-network-drive-client-indexing-files-and-folders-directly-inside-a-set-of-folders$$$ **Indexing files and folders directly inside a set of folders** @@ -268,6 +271,7 @@ $$$es-connectors-network-drive-client-indexing-files-and-folders-directly-inside } ] ``` +% NOTCONSOLE $$$es-connectors-network-drive-client-excluding-files-and-folders-that-match-a-pattern$$$ **Excluding files and folders that match a pattern** @@ -279,6 +283,7 @@ $$$es-connectors-network-drive-client-excluding-files-and-folders-that-match-a-p } ] ``` +% NOTCONSOLE ### Content extraction [es-connectors-network-drive-client-content-extraction] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-notion.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-notion.md index 114611022170c..a25cb3506903e 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-notion.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-notion.md @@ -63,6 +63,7 @@ PUT _connector/my-notion-connector "service_type": "notion" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -129,6 +130,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -303,6 +305,7 @@ Indexing every page where the title contains `Demo Page`: ] } ``` +% NOTCONSOLE **Example 2** @@ -320,6 +323,7 @@ Indexing every database where the title contains `Demo Database`: ] } ``` +% NOTCONSOLE **Example 3** @@ -343,6 +347,7 @@ Indexing every database where the title contains `Demo Database` and every page ] } ``` +% NOTCONSOLE **Example 4** @@ -360,6 +365,7 @@ Indexing all pages in the workspace: ] } ``` +% NOTCONSOLE **Example 5** @@ -374,6 +380,7 @@ Indexing all the pages and databases connected to the workspace: ] } ``` +% NOTCONSOLE **Example 6** @@ -394,6 +401,7 @@ Indexing all the rows of a database where the record is `true` for the column `T ] } ``` +% NOTCONSOLE **Example 7** @@ -408,6 +416,7 @@ Indexing all rows of a specific database: ] } ``` +% NOTCONSOLE **Example 8** @@ -442,6 +451,7 @@ Indexing all blocks defined in `searches` and `database_query_filters`: ] } ``` +% NOTCONSOLE ::::{note} In this example the `filter` object syntax for `database_query_filters` is defined per the [Notion documentation](https://developers.notion.com/reference/post-database-query-filter). diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-onedrive.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-onedrive.md index 726bb40c65900..f003733074c04 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-onedrive.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-onedrive.md @@ -51,6 +51,7 @@ PUT _connector/my-onedrive-connector "service_type": "onedrive" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -136,6 +137,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -311,6 +313,7 @@ This rule skips indexing for files with `.xlsx` and `.docx` extensions. All othe } ] ``` +% NOTCONSOLE $$$es-connectors-onedrive-client-sync-rules-advanced-examples-2$$$ **Example 2** @@ -325,6 +328,7 @@ This rule focuses on indexing files and folders owned by `user1-domain@onmicroso } ] ``` +% NOTCONSOLE $$$es-connectors-onedrive-client-sync-rules-advanced-examples-3$$$ **Example 3** @@ -339,6 +343,7 @@ This rule indexes only the files and folders directly inside the root folder, ex } ] ``` +% NOTCONSOLE $$$es-connectors-onedrive-client-sync-rules-advanced-examples-4$$$ **Example 4** @@ -354,6 +359,7 @@ This rule indexes files and folders owned by `user1-domain@onmicrosoft.com` and } ] ``` +% NOTCONSOLE $$$es-connectors-onedrive-client-sync-rules-advanced-examples-5$$$ **Example 5** @@ -370,6 +376,7 @@ This example contains two rules. The first rule indexes all files and folders ow } ] ``` +% NOTCONSOLE $$$es-connectors-onedrive-client-sync-rules-advanced-examples-6$$$ **Example 6** @@ -387,6 +394,7 @@ This example contains two rules. The first rule indexes all files owned by `user } ] ``` +% NOTCONSOLE ### Content Extraction [es-connectors-onedrive-client-content-extraction] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-oracle.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-oracle.md index 05a76194612f8..c11a4e00734bc 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-oracle.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-oracle.md @@ -38,6 +38,7 @@ PUT _connector/my-oracle-connector "service_type": "oracle" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -195,6 +196,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-outlook.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-outlook.md index 97f1b51f60a0e..118cd89af53a2 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-outlook.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-outlook.md @@ -41,6 +41,7 @@ PUT _connector/my-outlook-connector "service_type": "outlook" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -192,6 +193,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-postgresql.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-postgresql.md index 9c81d134f6e96..afd4eb537c8be 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-postgresql.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-postgresql.md @@ -47,6 +47,7 @@ PUT _connector/my-postgresql-connector "service_type": "postgresql" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -213,6 +214,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -347,6 +349,7 @@ $$$es-connectors-postgresql-client-sync-rules-advanced-examples-1$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-postgresql-client-sync-rules-advanced-examples-1-id-columns$$$ **Multiple table queries with `id_columns`** @@ -371,6 +374,7 @@ In 8.15.0, we added a new optional `id_columns` field in our advanced sync rules } ] ``` +% NOTCONSOLE This example uses the `id_columns` field to specify the unique fields `emp_id` and `c_id` for the `employee` and `customer` tables, respectively. @@ -385,6 +389,7 @@ $$$es-connectors-postgresql-client-sync-rules-advanced-examples-2$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-postgresql-client-sync-rules-advanced-examples-3$$$ **`JOIN` operations** @@ -397,6 +402,7 @@ $$$es-connectors-postgresql-client-sync-rules-advanced-examples-3$$$ } ] ``` +% NOTCONSOLE ::::{warning} When using advanced rules, a query can bypass the configuration field `tables`. This will happen if the query specifies a table that doesn’t appear in the configuration. This can also happen if the configuration specifies `*` to fetch all tables while the advanced sync rule requests for only a subset of tables. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-redis.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-redis.md index 6173e0a1b0a7e..1b6cce16d729f 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-redis.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-redis.md @@ -43,6 +43,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -219,6 +220,7 @@ Provide at least one of the following: `key_pattern` or `type`, or both. } ] ``` +% NOTCONSOLE #### Example 2 [es-connectors-redis-connector-advanced-sync-rules-example-2] @@ -233,6 +235,7 @@ Provide at least one of the following: `key_pattern` or `type`, or both. } ] ``` +% NOTCONSOLE #### Example 3 [es-connectors-redis-connector-advanced-sync-rules-example-3] @@ -246,6 +249,7 @@ Provide at least one of the following: `key_pattern` or `type`, or both. "key_pattern": "test[123]" } ``` +% NOTCONSOLE #### Example 4 [es-connectors-redis-connector-advanced-sync-rules-example-4] @@ -260,6 +264,7 @@ Provide at least one of the following: `key_pattern` or `type`, or both. } ] ``` +% NOTCONSOLE #### Example 5 [es-connectors-redis-connector-advanced-sync-rules-example-5] @@ -274,6 +279,7 @@ Provide at least one of the following: `key_pattern` or `type`, or both. } ] ``` +% NOTCONSOLE #### Example 6 [es-connectors-redis-connector-advanced-sync-rules-example-6] @@ -289,6 +295,7 @@ Provide at least one of the following: `key_pattern` or `type`, or both. } ] ``` +% NOTCONSOLE #### Example 7 [es-connectors-redis-connector-advanced-sync-rules-example-7] @@ -303,6 +310,7 @@ Provide at least one of the following: `key_pattern` or `type`, or both. } ] ``` +% NOTCONSOLE ## Connector Client operations [es-connectors-redis-connector-connector-client-operations] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-run-from-docker.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-run-from-docker.md index d70e2bd663b9d..f9dfeaf9688f5 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-run-from-docker.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-run-from-docker.md @@ -22,6 +22,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output /connectors-config/config.yml ``` +% NOTCONSOLE Don’t forget to change the `--output` argument value to the path where you want to save the `config.yml` file on your local system. But keep note of where you wrote this file, as it is required in the `docker run` step below. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-s3.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-s3.md index 4bc26ac94905e..cca4f2244faa2 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-s3.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-s3.md @@ -44,6 +44,7 @@ PUT _connector/my-s3-connector "service_type": "s3" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -168,6 +169,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -277,6 +279,7 @@ $$$es-connectors-s3-client-sync-rules-advanced-examples$$$ ] ``` +% NOTCONSOLE **Example**: Fetch files/folder starting with `folder1`. @@ -288,6 +291,7 @@ $$$es-connectors-s3-client-sync-rules-advanced-examples$$$ } ] ``` +% NOTCONSOLE **Fetching files and folders by specifying extensions** @@ -302,6 +306,7 @@ $$$es-connectors-s3-client-sync-rules-advanced-examples$$$ } ] ``` +% NOTCONSOLE ### Content extraction [es-connectors-s3-client-content-extraction] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-salesforce.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-salesforce.md index dcbdc1621b98a..40a53910d9a9d 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-salesforce.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-salesforce.md @@ -53,6 +53,7 @@ PUT _connector/my-salesforce-connector "service_type": "salesforce" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -192,6 +193,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -363,6 +365,7 @@ $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-query-language$$$ } ] ``` +% NOTCONSOLE **Example**: Fetch documents using SOSL query. @@ -374,6 +377,7 @@ $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-query-language$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-objects$$$ **Fetch standard and custom objects using SOQL and SOSL queries** @@ -392,6 +396,7 @@ $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-objects$$$ } ] ``` +% NOTCONSOLE **Example**: Fetch documents for custom objects via SOQL and SOSL query. @@ -407,6 +412,7 @@ $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-objects$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-standard-custom-fields$$$ **Fetch documents with standard and custom fields** @@ -421,6 +427,7 @@ $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-standard-custom-fie } ] ``` +% NOTCONSOLE **Example**: Fetch documents with all custom fields for Connector object. @@ -432,6 +439,7 @@ $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-standard-custom-fie } ] ``` +% NOTCONSOLE **Example**: Fetch documents with all standard fields for Account object. @@ -443,6 +451,7 @@ $$$es-connectors-salesforce-client-sync-rules-advanced-fetch-standard-custom-fie } ] ``` +% NOTCONSOLE ### Documents and syncs [es-connectors-salesforce-client-documents-syncs] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-servicenow.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-servicenow.md index 3977cc1d983d1..7c7d7e528e538 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-servicenow.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-servicenow.md @@ -49,6 +49,7 @@ PUT _connector/my-servicenow-connector "service_type": "servicenow" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -225,6 +226,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -315,6 +317,7 @@ $$$es-connectors-servicenow-client-sync-rules-number-incident-service$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-servicenow-client-sync-rules-active-false-user-service$$$ **Indexing document based on user activity state for User service** @@ -327,6 +330,7 @@ $$$es-connectors-servicenow-client-sync-rules-active-false-user-service$$$ } ] ``` +% NOTCONSOLE $$$es-connectors-servicenow-client-sync-rules-author-administrator-knowledge-service$$$ **Indexing document based on author name for Knowledge service** @@ -339,6 +343,7 @@ $$$es-connectors-servicenow-client-sync-rules-author-administrator-knowledge-ser } ] ``` +% NOTCONSOLE ### End-to-end Testing [es-connectors-servicenow-client-connector-client-operations-testing] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint-online.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint-online.md index 04af7d5e75a84..2486bd21e4d46 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint-online.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint-online.md @@ -283,6 +283,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. @@ -401,6 +402,7 @@ Example: "skipExtractingDriveItemsOlderThan": 60 } ``` +% NOTCONSOLE This rule will not extract content of any drive items (files in document libraries) that haven’t been modified for 60 days or more. @@ -514,7 +516,7 @@ POST INDEX_NAME/_update_by_query?conflicts=proceed } } ``` - +% TEST[skip:TODO] ### Document-level security [es-connectors-sharepoint-online-client-dls] diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint.md index 15b349ba4b049..0b077e185f8e8 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-sharepoint.md @@ -58,6 +58,7 @@ PUT _connector/my-sharepoint_server-connector "service_type": "sharepoint_server" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -194,6 +195,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-slack.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-slack.md index 75b6cbb5b217a..07b09d26bac3d 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-slack.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-slack.md @@ -56,6 +56,7 @@ PUT _connector/my-slack-connector "service_type": "slack" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -155,6 +156,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-teams.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-teams.md index df0fe4daf0bf3..32486190c99c9 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-teams.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-teams.md @@ -53,6 +53,7 @@ PUT _connector/my-microsoft_teams-connector "service_type": "microsoft_teams" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -185,6 +186,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-connectors-zoom.md b/docs/reference/ingestion-tools/search-connectors/es-connectors-zoom.md index f740658b8f88c..610ce3849478e 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-connectors-zoom.md +++ b/docs/reference/ingestion-tools/search-connectors/es-connectors-zoom.md @@ -53,6 +53,7 @@ PUT _connector/my-zoom-connector "service_type": "zoom" } ``` +% TEST[skip:can’t test in isolation] :::::{dropdown} You’ll also need to create an API key for the connector to use. ::::{note} @@ -184,6 +185,7 @@ Download the sample configuration file. You can either download it manually or r ```sh curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml ``` +% NOTCONSOLE Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name. diff --git a/docs/reference/ingestion-tools/search-connectors/es-dls-e2e-guide.md b/docs/reference/ingestion-tools/search-connectors/es-dls-e2e-guide.md index 6d368643c8112..fe25b5ba29e0a 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-dls-e2e-guide.md +++ b/docs/reference/ingestion-tools/search-connectors/es-dls-e2e-guide.md @@ -116,6 +116,7 @@ The access control index will contain documents similar to this example: } } ``` +% NOTCONSOLE This document contains the Elasticsearch query that describes which documents the user `john@example.com` has access to. The access control information is stored in the `access_control` field. In this case the user has access only to documents that contain `"john@example.co"` or `"Engineering Members"` in the `_allow_access_control` field. @@ -181,6 +182,7 @@ POST /_security/api_key } } ``` +% TEST[skip:TODO] The response will look like this: @@ -193,6 +195,7 @@ The response will look like this: "encoded": "Qk05dy1JZ0JhRDNyNGpLQ3MwUmk6elRzdGU5QjZUY21SSWdkMldnQ1RMZw==" } ``` +% NOTCONSOLE The `api_key` field contains the API key that can be used to query the Search Application with the appropriate DLS restrictions. @@ -225,6 +228,7 @@ GET .search-acl-filter-source1 } } ``` +% NOTCONSOLE ```js GET .search-acl-filter-source2 @@ -246,6 +250,7 @@ GET .search-acl-filter-source2 } } ``` +% NOTCONSOLE `.search-acl-filter-source1` and `.search-acl-filter-source2` define the access control identities for `source1` and `source2`. diff --git a/docs/reference/ingestion-tools/search-connectors/es-dls-overview.md b/docs/reference/ingestion-tools/search-connectors/es-dls-overview.md index bb71c462dc0e5..92f9a4bd8fb03 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-dls-overview.md +++ b/docs/reference/ingestion-tools/search-connectors/es-dls-overview.md @@ -76,6 +76,7 @@ These documents define the access control policy for the data indexed into Elast } } ``` +% NOTCONSOLE In this example, the identity object specifies the identity of the user that this document pertains to. The `query` object then uses a template to list the parameters that form the access control policy for this identity. It also contains the query `source`, which will specify a query to fetch all content documents the identity has access to. The `_id` could be, for example, the email address or the username of a user. The exact content and structure of `identity` depends on the corresponding implementation. @@ -97,6 +98,7 @@ Content documents contain the actual data from your 3rd party source. A specific ] } ``` +% NOTCONSOLE ### Access control sync vs content sync [es-dls-overview-sync-type-comparison] @@ -145,6 +147,7 @@ One access control document: } } ``` +% NOTCONSOLE Let’s see which of the following example documents these permissions can access, and why. @@ -158,6 +161,7 @@ Let’s see which of the following example documents these permissions can acces ] } ``` +% NOTCONSOLE The user `example username` will have access to this document as he’s part of the corresponding group and his username and email address are also explicitly part of `_allow_access_control`. @@ -169,6 +173,7 @@ The user `example username` will have access to this document as he’s part of ] } ``` +% NOTCONSOLE The user `example username` will also have access to this document as they are part of the `example group`. @@ -180,6 +185,7 @@ The user `example username` will also have access to this document as they are p ] } ``` +% NOTCONSOLE The user `example username` won’t have access to this document because their email does not match `another.user@example.com`. @@ -189,6 +195,7 @@ The user `example username` won’t have access to this document because their e "_allow_access_control": [] } ``` +% NOTCONSOLE No one will have access to this document as the `_allow_access_control` field is empty. @@ -224,6 +231,7 @@ GET .search-acl-filter-source1 } } ``` +% NOTCONSOLE ```js GET .search-acl-filter-source2 @@ -245,6 +253,7 @@ GET .search-acl-filter-source2 } } ``` +% NOTCONSOLE `.search-acl-filter-source1` and `.search-acl-filter-source2` define the access control identities for `source1` and `source2`. @@ -294,6 +303,7 @@ POST /_security/api_key } } ``` +% TEST[skip:TODO] #### Workflow guidance [es-dls-overview-multiple-connectors-workflow-guidance] diff --git a/docs/reference/ingestion-tools/search-connectors/es-sync-rules.md b/docs/reference/ingestion-tools/search-connectors/es-sync-rules.md index 04742067e280f..5a563b2d4c218 100644 --- a/docs/reference/ingestion-tools/search-connectors/es-sync-rules.md +++ b/docs/reference/ingestion-tools/search-connectors/es-sync-rules.md @@ -234,6 +234,7 @@ A sample apartment looks like this in the `.json` format: } } ``` +% NOTCONSOLE The target data set should fulfill the following conditions: @@ -303,6 +304,7 @@ Let’s assume that the apartment data is stored inside a MongoDB instance. For } ] ``` +% NOTCONSOLE To create these advanced sync rules navigate to the sync rules creation dialog and select the *Advanced rules* tab. You can now paste your aggregation pipeline into the input field under `aggregate.pipeline`: diff --git a/docs/reference/query-languages/eql-ex-threat-detection.md b/docs/reference/query-languages/eql-ex-threat-detection.md index c28a50df4bdfa..ebb1237dd5b8a 100644 --- a/docs/reference/query-languages/eql-ex-threat-detection.md +++ b/docs/reference/query-languages/eql-ex-threat-detection.md @@ -68,6 +68,7 @@ GET /my-data-stream/_eql/search?filter_path=-hits.events <1> "size": 200 <3> } ``` +% TEST[setup:atomic_red_regsvr32] 1. `?filter_path=-hits.events` excludes the `hits.events` property from the response. This search is only intended to get an event count, not a list of matching events. 2. Matches any event with a `process.name` of `regsvr32.exe`. @@ -90,6 +91,7 @@ The response returns 143 related events. } } ``` +% TESTRESPONSE[s/"took": 60/"took": $body.took/] ## Check for command line artifacts [eql-ex-check-for-command-line-artifacts] @@ -104,6 +106,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:atomic_red_regsvr32] The query matches one event with an `event.type` of `creation`, indicating the start of a `regsvr32.exe` process. Based on the event’s `process.command_line` value, `regsvr32.exe` used `scrobj.dll` to register a script, `RegSvr32.sct`. This fits the behavior of a Squiblydoo attack. @@ -150,6 +153,9 @@ The query matches one event with an `event.type` of `creation`, indicating the s } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/] +% TESTRESPONSE[s/"_id": "gl5MJXMBMk1dGnErnBW8"/"_id": $body.hits.events.0._id/] ## Check for malicious script loads [eql-ex-check-for-malicious-script-loads] @@ -164,6 +170,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:atomic_red_regsvr32] The query matches an event, confirming `scrobj.dll` was loaded. @@ -200,6 +207,9 @@ The query matches an event, confirming `scrobj.dll` was loaded. } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/] +% TESTRESPONSE[s/"_id": "ol5MJXMBMk1dGnErnBW8"/"_id": $body.hits.events.0._id/] ## Determine the likelihood of success [eql-ex-detemine-likelihood-of-success] @@ -223,6 +233,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:atomic_red_regsvr32] The query matches a sequence, indicating the attack likely succeeded. @@ -329,4 +340,9 @@ The query matches a sequence, indicating the attack likely succeeded. } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.sequences.0.events.0._index/] +% TESTRESPONSE[s/"_id": "gl5MJXMBMk1dGnErnBW8"/"_id": $body.hits.sequences.0.events.0._id/] +% TESTRESPONSE[s/"_id": "ol5MJXMBMk1dGnErnBW8"/"_id": $body.hits.sequences.0.events.1._id/] +% TESTRESPONSE[s/"_id": "EF5MJXMBMk1dGnErnBa9"/"_id": $body.hits.sequences.0.events.2._id/] diff --git a/docs/reference/query-languages/eql-syntax.md b/docs/reference/query-languages/eql-syntax.md index 87fd794a2c010..bc9c2b10351c2 100644 --- a/docs/reference/query-languages/eql-syntax.md +++ b/docs/reference/query-languages/eql-syntax.md @@ -815,6 +815,7 @@ A data set contains the following `process` events in ascending chronological or { "index" : { "_id": "11" } } { "user": { "name": "root" }, "process": { "name": "cat" }, ...} ``` +% NOTCONSOLE An EQL sequence query searches the data set: diff --git a/docs/reference/query-languages/eql.md b/docs/reference/query-languages/eql.md index 1e80e0ad48145..b3dd1c0cc57eb 100644 --- a/docs/reference/query-languages/eql.md +++ b/docs/reference/query-languages/eql.md @@ -49,6 +49,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] By default, basic EQL queries return the 10 most recent matching events in the `hits.events` property. These hits are sorted by timestamp, converted to milliseconds since the [Unix epoch](https://en.wikipedia.org/wiki/Unix_time), in ascending order. @@ -104,6 +105,10 @@ By default, basic EQL queries return the 10 most recent matching events in the ` } } ``` +% TESTRESPONSE[s/"took": 60/"took": $body.took/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/] +% TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/] +% TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/] Use the `size` parameter to get a smaller or larger set of hits: @@ -116,6 +121,7 @@ GET /my-data-stream/_eql/search "size": 50 } ``` +% TEST[setup:sec_logs] ## Search for a sequence of events [eql-search-sequence] @@ -132,6 +138,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] The response’s `hits.sequences` property contains the 10 most recent matching sequences. @@ -188,6 +195,11 @@ The response’s `hits.sequences` property contains the 10 most recent matching } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"total": ...,/"total": { "value": 1, "relation": "eq" },/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.sequences.0.events.0._index/] +% TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/] +% TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/] Use [`with maxspan`](/reference/query-languages/eql-syntax.md#eql-with-maxspan-keywords) to constrain matching sequences to a timespan: @@ -201,6 +213,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] Use `!` to match [missing events](/reference/query-languages/eql-syntax.md#eql-missing-events): events in a sequence that do not meet a condition within a given timespan: @@ -215,6 +228,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] Missing events are indicated in the response as `missing": true`: @@ -276,6 +290,11 @@ Missing events are indicated in the response as `missing": true`: } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"total": ...,/"total": { "value": 1, "relation": "eq" },/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2023.07.04-000001"/"_index": $body.hits.sequences.0.events.0._index/] +% TESTRESPONSE[s/"_id": "AnpTIYkBrVQ2QEgsWg94"/"_id": $body.hits.sequences.0.events.0._id/] +% TESTRESPONSE[s/"_id": "BHpTIYkBrVQ2QEgsWg94"/"_id": $body.hits.sequences.0.events.2._id/] Use the [`by` keyword](/reference/query-languages/eql-syntax.md#eql-by-keyword) to match events that share the same field values: @@ -289,6 +308,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] If a field value should be shared across all events, use the `sequence by` keyword. The following query is equivalent to the previous one. @@ -302,6 +322,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] The `hits.sequences.join_keys` property contains the shared field values. @@ -320,6 +341,9 @@ The `hits.sequences.join_keys` property contains the shared field values. } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"hits": ...,/"hits": { "total": { "value": 1, "relation": "eq" },/] +% TESTRESPONSE[s/"events": .../"events": $body.hits.sequences.0.events/] Use the [`until` keyword](/reference/query-languages/eql-syntax.md#eql-until-keyword) to specify an expiration event for sequences. Matching sequences must end before this event. @@ -334,6 +358,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] ## Sample chronologically unordered events [eql-search-sample] @@ -567,6 +592,7 @@ GET /my-index*/_eql/search """ } ``` +% TEST[continued] By default, the response’s `hits.sequences` property contains up to 10 samples. Each sample has a set of `join_keys` and an array with one matching event for each of the filters. Events are returned in the order of the filters they match: @@ -693,6 +719,7 @@ By default, the response’s `hits.sequences` property contains up to 10 samples } } ``` +% TESTRESPONSE[skip:Response is illustrative only] 1. The events in the first sample have a value of `doom` for `host`. 2. This event matches the first filter. @@ -714,6 +741,7 @@ GET /my-index*/_eql/search """ } ``` +% TEST[continued] This query will return samples where each of the events shares the same value for `os` or `op_sys`, as well as for `host`. For example: @@ -843,6 +871,7 @@ This query will return samples where each of the events shares the same value fo } } ``` +% TESTRESPONSE[skip:Response is illustrative only] 1. The events in this sample have a value of `doom` for `host` and a value of `redhat` for `os` or `op_sys`. @@ -862,6 +891,7 @@ GET /my-index*/_eql/search """ } ``` +% TEST[continued] 1. Retrieve up to 2 samples per set of join keys. 2. Retrieve up to 20 samples in total. @@ -882,6 +912,7 @@ GET /my-data-stream/_eql/search?filter_path=hits.events._source.@timestamp,hits. """ } ``` +% TEST[setup:sec_logs] The API returns the following response. @@ -939,6 +970,7 @@ GET /my-data-stream/_eql/search?filter_path=-hits.events._source ] } ``` +% TEST[setup:sec_logs] 1. Both full field names and wildcard patterns are accepted. 2. Use the `format` parameter to apply a custom format for the field’s values. @@ -987,6 +1019,11 @@ The response includes values as a flat list in the `fields` section for each hit } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"total": ...,/"total": { "value": 2, "relation": "eq" },/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/] +% TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/] +% TESTRESPONSE[s/ ....\n/$body.hits.events.1/] ## Use runtime fields [eql-use-runtime-fields] @@ -1013,6 +1050,7 @@ GET /my-data-stream/_eql/search?filter_path=-hits.events._source ] } ``` +% TEST[setup:sec_logs] The API returns: @@ -1039,6 +1077,11 @@ The API returns: } } ``` +% TESTRESPONSE[s/ ...\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/] +% TESTRESPONSE[s/"total": ...,/"total": { "value": 2, "relation": "eq" },/] +% TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/] +% TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/] +% TESTRESPONSE[s/ ....\n/$body.hits.events.1/] ## Specify a timestamp or event category field [specify-a-timestamp-or-event-category-field] @@ -1055,6 +1098,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] The event category field must be mapped as a [`keyword`](/reference/elasticsearch/mapping-reference/keyword.md) family field type. The timestamp field should be mapped as a [`date`](/reference/elasticsearch/mapping-reference/date.md) field type. [`date_nanos`](/reference/elasticsearch/mapping-reference/date_nanos.md) timestamp fields are not supported. You cannot use a [`nested`](/reference/elasticsearch/mapping-reference/nested.md) field or the sub-fields of a `nested` field as the timestamp or event category field. @@ -1076,6 +1120,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] ## Filter using Query DSL [eql-search-filter-query-dsl] @@ -1098,6 +1143,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] ## Run an async EQL search [eql-search-async] @@ -1115,6 +1161,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] If the request doesn’t finish within the timeout period, the search becomes async and returns a response that includes: @@ -1134,12 +1181,18 @@ The async search continues to run in the background without blocking other reque "hits": ... } ``` +% TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/] +% TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/] +% TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/] +% TESTRESPONSE[s/"took": 2000/"took": $body.took/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] To check the progress of an async search, use the [get async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) with the search ID. Specify how long you’d like for complete results in the `wait_for_completion_timeout` parameter. ```console GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s ``` +% TEST[skip: no access to search ID] If the response’s `is_running` value is `false`, the async search has finished. If the `is_partial` value is `false`, the returned search results are complete. @@ -1153,12 +1206,16 @@ If the response’s `is_running` value is `false`, the async search has finished "hits": ... } ``` +% TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/] +% TESTRESPONSE[s/"took": 2000/"took": $body.took/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] Another more lightweight way to check the progress of an async search is to use the [get async EQL status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-get-status) with the search ID. ```console GET /_eql/search/status/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE= ``` +% TEST[skip: no access to search ID] ```console-result { @@ -1169,6 +1226,8 @@ GET /_eql/search/status/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FS "completion_status": 200 } ``` +% TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/] +% TESTRESPONSE[s/"expiration_time_in_millis": 1611690295000/"expiration_time_in_millis": $body.expiration_time_in_millis/] ## Change the search retention period [eql-search-store-async-eql-search] @@ -1185,18 +1244,21 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] You can use the [get async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search)'s `keep_alive` parameter to later change the retention period. The new retention period starts after the get request runs. ```console GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d ``` +% TEST[skip: no access to search ID] Use the [delete async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) to manually delete an async EQL search before the `keep_alive` period ends. If the search is still ongoing, {{es}} cancels the search request. ```console DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE= ``` +% TEST[skip: no access to search ID] ## Store synchronous EQL searches [eql-search-store-sync-eql-search] @@ -1213,6 +1275,7 @@ GET /my-data-stream/_eql/search """ } ``` +% TEST[setup:sec_logs] The response includes a search ID. `is_partial` and `is_running` are `false`, indicating the EQL search was synchronous and returned complete results. @@ -1226,12 +1289,16 @@ The response includes a search ID. `is_partial` and `is_running` are `false`, in "hits": ... } ``` +% TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/] +% TESTRESPONSE[s/"took": 52/"took": $body.took/] +% TESTRESPONSE[s/"hits": .../"hits": $body.hits/] Use the [get async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) to get the same results later: ```console GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY= ``` +% TEST[skip: no access to search ID] Saved synchronous searches are still subject to the `keep_alive` parameter’s retention period. When this period ends, the search and its results are deleted. @@ -1272,6 +1339,8 @@ PUT /_cluster/settings } } ``` +% TEST[setup:host] +% TEST[s/127.0.0.1:930\d+/${transport_host}/] To target a data stream or index on a remote cluster, use the `:` syntax. @@ -1283,6 +1352,9 @@ GET /cluster_one:my-data-stream,cluster_two:my-data-stream/_eql/search """ } ``` +% TEST[continued] +% TEST[setup:sec_logs] +% TEST[teardown:data_stream_cleanup] ## EQL circuit breaker settings [eql-circuit-breaker] diff --git a/docs/reference/query-languages/esql/esql-multivalued-fields.md b/docs/reference/query-languages/esql/esql-multivalued-fields.md index b5807b466c417..95e768707f9b0 100644 --- a/docs/reference/query-languages/esql/esql-multivalued-fields.md +++ b/docs/reference/query-languages/esql/esql-multivalued-fields.md @@ -40,6 +40,7 @@ Multivalued fields come back as a JSON array: ] } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] The relative order of values in a multivalued field is undefined. They’ll frequently be in ascending order but don’t rely on that. @@ -88,6 +89,7 @@ And {{esql}} sees that removal: ] } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] But other types, like `long` don’t remove duplicates. @@ -131,6 +133,7 @@ And {{esql}} also sees that: ] } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] This is all at the storage layer. If you store duplicate `long`s and then convert them to strings the duplicates will stay: @@ -157,6 +160,7 @@ POST /_query "query": "FROM mv | EVAL b=TO_STRING(b) | LIMIT 2" } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] ```console-result { @@ -172,6 +176,7 @@ POST /_query ] } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] ## `null` in a list [esql-multivalued-nulls] @@ -189,6 +194,7 @@ POST /_query "query": "FROM mv | LIMIT 1" } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] ```console-result { @@ -202,6 +208,7 @@ POST /_query ] } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] ## Functions [esql-multivalued-fields-functions] @@ -217,6 +224,11 @@ POST /mv/_bulk?refresh { "index" : {} } { "a": 2, "b": 3 } ``` +% TEST[continued] +% TEST[warning:Line 1:16: evaluation of [b + 2] failed, treating result as null. Only first 20 failures recorded.] +% TEST[warning:Line 1:16: java.lang.IllegalArgumentException: single-value function encountered multi-value] +% TEST[warning:Line 1:23: evaluation of [a + b] failed, treating result as null. Only first 20 failures recorded.] +% TEST[warning:Line 1:23: java.lang.IllegalArgumentException: single-value function encountered multi-value] ```console POST /_query @@ -224,6 +236,11 @@ POST /_query "query": "FROM mv | EVAL b + 2, a + b | LIMIT 4" } ``` +% TEST[continued] +% TEST[warning:Line 1:16: evaluation of [b + 2] failed, treating result as null. Only first 20 failures recorded.] +% TEST[warning:Line 1:16: java.lang.IllegalArgumentException: single-value function encountered multi-value] +% TEST[warning:Line 1:23: evaluation of [a + b] failed, treating result as null. Only first 20 failures recorded.] +% TEST[warning:Line 1:23: java.lang.IllegalArgumentException: single-value function encountered multi-value] ```console-result { @@ -241,6 +258,7 @@ POST /_query ] } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] Work around this limitation by converting the field to single value with one of: @@ -258,6 +276,7 @@ POST /_query "query": "FROM mv | EVAL b=MV_MIN(b) | EVAL b + 2, a + b | LIMIT 4" } ``` +% TEST[continued] ```console-result { @@ -275,4 +294,5 @@ POST /_query ] } ``` +% TESTRESPONSE[s/"took": 28/"took": "$body.took"/] diff --git a/docs/reference/query-languages/esql/esql-syntax.md b/docs/reference/query-languages/esql/esql-syntax.md index e7d6427392703..7a5dee950897b 100644 --- a/docs/reference/query-languages/esql/esql-syntax.md +++ b/docs/reference/query-languages/esql/esql-syntax.md @@ -160,6 +160,7 @@ FROM library """ } ``` +% TEST[setup:library] You can also use [query parameters](docs-content://explore-analyze/query-filter/languages/esql-rest.md#esql-rest-params) in function named parameters: @@ -175,4 +176,5 @@ FROM library "params": [300, "Frank Herbert", 2] } ``` +% TEST[setup:library] diff --git a/docs/reference/query-languages/query-dsl-function-score-query.md b/docs/reference/query-languages/query-dsl-function-score-query.md index 2829bd4adac5b..91a8dda6f37c3 100644 --- a/docs/reference/query-languages/query-dsl-function-score-query.md +++ b/docs/reference/query-languages/query-dsl-function-score-query.md @@ -26,6 +26,7 @@ GET /_search } } ``` +% TEST[setup:my_index] 1. See [Function score](#score-functions) for a list of supported functions. @@ -58,6 +59,7 @@ GET /_search } } ``` +% TEST[setup:my_index] 1. Boost for the whole query. 2. See [Function score](#score-functions) for a list of supported functions. @@ -153,6 +155,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ::::{important} In {{es}}, all document scores are positive 32-bit floating point numbers. @@ -189,6 +192,7 @@ GET /_search } } ``` +% TEST[setup:my_index] Note that unlike the `custom_score` query, the score of the query is multiplied with the result of the script scoring. If you wish to inhibit this, set `"boost_mode": "replace"` @@ -200,6 +204,7 @@ The `weight` score allows you to multiply the score by the provided `weight`. Th ```js "weight" : number ``` +% NOTCONSOLE ## Random [function-random] @@ -226,6 +231,7 @@ GET /_search } } ``` +% TEST[setup:my_index] ## Field Value factor [function-field-value-factor] @@ -249,6 +255,7 @@ GET /_search } } ``` +% TEST[setup:my_index] Which will translate into the following formula for scoring: @@ -308,6 +315,7 @@ To use distance scoring on a query that has numerical fields, the user has to de } } ``` +% NOTCONSOLE 1. The `DECAY_FUNCTION` should be one of `linear`, `exp`, or `gauss`. 2. The specified field must be a numeric, date, or geopoint field. @@ -332,6 +340,7 @@ GET /_search } } ``` +% TEST[setup:my_index] 1. The date format of the origin depends on the [`format`](/reference/elasticsearch/mapping-reference/mapping-date-format.md) defined in your mapping. If you do not define the origin, the current time is used. 2. The `offset` and `decay` parameters are optional. @@ -425,6 +434,7 @@ Example: "multi_value_mode": "avg" } ``` +% NOTCONSOLE @@ -450,6 +460,7 @@ The function for `price` in this case would be } } ``` +% NOTCONSOLE 1. This decay function could also be `linear` or `exp`. @@ -464,6 +475,7 @@ and for `location`: } } ``` +% NOTCONSOLE 1. This decay function could also be `linear` or `exp`. diff --git a/docs/reference/query-languages/query-dsl-geo-bounding-box-query.md b/docs/reference/query-languages/query-dsl-geo-bounding-box-query.md index 99fd7732fbb1a..31d69830b26e7 100644 --- a/docs/reference/query-languages/query-dsl-geo-bounding-box-query.md +++ b/docs/reference/query-languages/query-dsl-geo-bounding-box-query.md @@ -65,6 +65,7 @@ PUT /my_geoshapes/_doc/1 } } ``` +% TESTSETUP Use a `geo_bounding_box` filter to match `geo_point` values that intersect a bounding box. To define the box, provide geopoint values for two opposite corners. diff --git a/docs/reference/query-languages/query-dsl-geo-distance-query.md b/docs/reference/query-languages/query-dsl-geo-distance-query.md index 267a8e0ef1c52..f1335b13e8ba7 100644 --- a/docs/reference/query-languages/query-dsl-geo-distance-query.md +++ b/docs/reference/query-languages/query-dsl-geo-distance-query.md @@ -65,6 +65,7 @@ PUT /my_geoshapes/_doc/1 } } ``` +% TESTSETUP Use a `geo_distance` filter to match `geo_point` values within a specified distance of another geopoint: diff --git a/docs/reference/query-languages/query-dsl-geo-grid-query.md b/docs/reference/query-languages/query-dsl-geo-grid-query.md index 9eb21e652348d..fc598dbab45bc 100644 --- a/docs/reference/query-languages/query-dsl-geo-grid-query.md +++ b/docs/reference/query-languages/query-dsl-geo-grid-query.md @@ -49,6 +49,7 @@ PUT /my_locations/_doc/3?refresh "name": "Musée du Louvre" } ``` +% TESTSETUP ## geohash grid [query-dsl-geo-grid-query-geohash] @@ -68,6 +69,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 10/"took" : $body.took/] ```console-result { @@ -103,6 +105,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 10/"took" : $body.took/] We can extract the documents on one of those buckets by executing a geo_grid query using the bucket key with the following syntax: @@ -118,6 +121,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 1/"took" : $body.took/] ```console-result { @@ -150,6 +154,8 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 1/"took" : $body.took/] +% TESTRESPONSE[s/"took" : 1/"took" : $body.took/] ## geotile grid [query-dsl-geo-grid-query-geotile] @@ -170,6 +176,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 1/"took" : $body.took/] ```console-result { @@ -205,6 +212,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 1/"took" : $body.took/] We can extract the documents on one of those buckets by executing a geo_grid query using the bucket key with the following syntax: @@ -220,6 +228,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 1/"took" : $body.took/] ```console-result { @@ -272,6 +281,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 2/"took" : $body.took/] ```console-result { @@ -307,6 +317,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 2/"took" : $body.took/] We can extract the documents on one of those buckets by executing a geo_grid query using the bucket key with the following syntax: @@ -322,6 +333,7 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 26/"took" : $body.took/] ```console-result { @@ -354,5 +366,6 @@ GET /my_locations/_search } } ``` +% TESTRESPONSE[s/"took" : 26/"took" : $body.took/] diff --git a/docs/reference/query-languages/query-dsl-geo-polygon-query.md b/docs/reference/query-languages/query-dsl-geo-polygon-query.md index 95081e413407e..35a3d3457f63d 100644 --- a/docs/reference/query-languages/query-dsl-geo-polygon-query.md +++ b/docs/reference/query-languages/query-dsl-geo-polygon-query.md @@ -39,6 +39,7 @@ GET /_search } } ``` +% TEST[warning:Deprecated field [geo_polygon] used, replaced by [[geo_shape] query where polygons are defined in geojson or wkt]] ## Query options [_query_options_2] @@ -81,6 +82,7 @@ GET /_search } } ``` +% TEST[warning:Deprecated field [geo_polygon] used, replaced by [[geo_shape] query where polygons are defined in geojson or wkt]] ### Lat lon as string [_lat_lon_as_string_2] @@ -110,6 +112,7 @@ GET /_search } } ``` +% TEST[warning:Deprecated field [geo_polygon] used, replaced by [[geo_shape] query where polygons are defined in geojson or wkt]] ### Geohash [_geohash_4] @@ -137,6 +140,7 @@ GET /_search } } ``` +% TEST[warning:Deprecated field [geo_polygon] used, replaced by [[geo_shape] query where polygons are defined in geojson or wkt]] ## `geo_point` type [_geo_point_type] diff --git a/docs/reference/query-languages/query-dsl-geo-shape-query.md b/docs/reference/query-languages/query-dsl-geo-shape-query.md index 0a9c1d5bf87e6..ae3622c9d77c6 100644 --- a/docs/reference/query-languages/query-dsl-geo-shape-query.md +++ b/docs/reference/query-languages/query-dsl-geo-shape-query.md @@ -40,6 +40,7 @@ POST /example/_doc?refresh } } ``` +% TESTSETUP The following query will find the point using {{es}}'s `envelope` GeoJSON extension: @@ -87,6 +88,7 @@ PUT /example_points/_doc/1?refresh "location": [13.400544, 52.530286] } ``` +% TEST[continued] Using the same query, the documents with matching `geo_point` fields are returned. @@ -113,6 +115,7 @@ GET /example_points/_search } } ``` +% TEST[continued] ```console-result { @@ -144,6 +147,7 @@ GET /example_points/_search } } ``` +% TESTRESPONSE[s/"took" : 17/"took" : $body.took/] ## Pre-indexed shape [_pre_indexed_shape] diff --git a/docs/reference/query-languages/query-dsl-has-child-query.md b/docs/reference/query-languages/query-dsl-has-child-query.md index cb37096b5b040..3c9a79d06ac1e 100644 --- a/docs/reference/query-languages/query-dsl-has-child-query.md +++ b/docs/reference/query-languages/query-dsl-has-child-query.md @@ -38,6 +38,7 @@ PUT /my-index-000001 } } ``` +% TESTSETUP ### Example query [has-child-query-ex-query] diff --git a/docs/reference/query-languages/query-dsl-has-parent-query.md b/docs/reference/query-languages/query-dsl-has-parent-query.md index 74fa3b4cfdeb4..b3c822bb0f0bf 100644 --- a/docs/reference/query-languages/query-dsl-has-parent-query.md +++ b/docs/reference/query-languages/query-dsl-has-parent-query.md @@ -39,6 +39,7 @@ PUT /my-index-000001 } } ``` +% TESTSETUP ### Example query [has-parent-query-ex-query] diff --git a/docs/reference/query-languages/query-dsl-knn-query.md b/docs/reference/query-languages/query-dsl-knn-query.md index 7c7c96245060e..55369c287e9ae 100644 --- a/docs/reference/query-languages/query-dsl-knn-query.md +++ b/docs/reference/query-languages/query-dsl-knn-query.md @@ -32,6 +32,7 @@ PUT my-image-index } } ``` +% TEST[continued] 1. Index your data. @@ -158,6 +159,7 @@ POST my-image-index/_search } } ``` +% TEST[continued] ## Hybrid search with knn query [knn-query-in-hybrid-search] @@ -194,6 +196,7 @@ POST my-image-index/_search } } ``` +% TEST[continued] ## Knn query inside a nested query [knn-query-with-nested-query] @@ -225,6 +228,7 @@ A sample query can look like below: } } ``` +% NOTCONSOLE ## Knn query with aggregations [knn-query-aggregations] diff --git a/docs/reference/query-languages/query-dsl-nested-query.md b/docs/reference/query-languages/query-dsl-nested-query.md index 16f431caf0230..e113461b09766 100644 --- a/docs/reference/query-languages/query-dsl-nested-query.md +++ b/docs/reference/query-languages/query-dsl-nested-query.md @@ -52,6 +52,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] @@ -175,6 +176,7 @@ PUT /drivers/_doc/2?refresh } } ``` +% TEST[continued] You can now use a multi-level nested query to match documents based on the `make` and `model` fields. @@ -201,6 +203,7 @@ GET /drivers/_search } } ``` +% TEST[continued] The search request returns the following response: @@ -245,6 +248,7 @@ The search request returns the following response: } } ``` +% TESTRESPONSE[s/"took" : 5/"took": $body.took/] ### `must_not` clauses and `nested` queries [must-not-clauses-and-nested-queries] @@ -320,6 +324,7 @@ POST my-index/_search } } ``` +% TEST[s/_search/_search?filter_path=hits.hits/] The search returns: @@ -360,6 +365,7 @@ The search returns: } } ``` +% TESTRESPONSE[s/...//] 1. This nested object matches the query. As a result, the search returns the object’s parent document as a hit. 2. This nested object doesn’t match the query. Since another nested object in the same document does match the query, the search still returns the parent document as a hit. @@ -388,6 +394,8 @@ POST my-index/_search } } ``` +% TEST[continued] +% TEST[s/_search/_search?filter_path=hits.hits/] The search returns: @@ -413,6 +421,7 @@ The search returns: } } ``` +% TESTRESPONSE[s/...//] diff --git a/docs/reference/query-languages/query-dsl-percolate-query.md b/docs/reference/query-languages/query-dsl-percolate-query.md index 7dc42febff15f..60cdfa20cbc5a 100644 --- a/docs/reference/query-languages/query-dsl-percolate-query.md +++ b/docs/reference/query-languages/query-dsl-percolate-query.md @@ -50,6 +50,7 @@ PUT /my-index-000001/_doc/1?refresh } } ``` +% TEST[continued] Match a document to the registered percolator queries: @@ -66,6 +67,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] The above request will yield the following response: @@ -105,6 +107,7 @@ The above request will yield the following response: } } ``` +% TESTRESPONSE[s/"took": 13,/"took": "$body.took",/] 1. The query with id `1` matches our document. 2. The `_percolator_document_slot` field indicates which document has matched with this query. Useful when percolating multiple document simultaneously. @@ -174,6 +177,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] At index time terms are extracted from the percolator query and the percolator can often determine whether a query matches just by looking at those extracted terms. However, computing scores requires to deserialize each matching query and run it against the percolated document, which is a much more expensive operation. Hence if computing scores is not required the `percolate` query should be wrapped in a `constant_score` query or a `bool` query’s filter clause. @@ -210,6 +214,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] 1. The documents array contains 4 documents that are going to be percolated at the same time. @@ -250,6 +255,7 @@ GET /my-index-000001/_search } } ``` +% TESTRESPONSE[s/"took": 13,/"took": "$body.took",/] 1. The `_percolator_document_slot` indicates that the first, second and last documents specified in the `percolate` query are matching with this query. @@ -271,6 +277,7 @@ PUT /my-index-000001/_doc/2 "message" : "A new bonsai tree in the office" } ``` +% TEST[continued] Index response: @@ -305,6 +312,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] 1. The version is optional, but useful in certain cases. We can ensure that we are trying to percolate the document we just have indexed. A change may be made after we have indexed, and if that is the case the search request would fail with a version conflict error. @@ -333,6 +341,7 @@ PUT /my-index-000001/_doc/3?refresh } } ``` +% TEST[continued] Save another query: @@ -346,6 +355,7 @@ PUT /my-index-000001/_doc/4?refresh } } ``` +% TEST[continued] Execute a search request with the `percolate` query and highlighting enabled: @@ -367,6 +377,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] This will yield the following response. @@ -431,6 +442,7 @@ This will yield the following response. } } ``` +% TESTRESPONSE[s/"took": 7,/"took": "$body.took",/] 1. The terms from each query have been highlighted in the document. @@ -468,6 +480,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] The slightly different response: @@ -518,6 +531,7 @@ The slightly different response: } } ``` +% TESTRESPONSE[s/"took": 13,/"took": "$body.took",/] 1. The highlight fields have been prefixed with the document slot they belong to, in order to know which highlight field belongs to what document. @@ -557,6 +571,7 @@ PUT /my-index-000001/_doc/5?refresh } } ``` +% TEST[continued] ```console GET /my-index-000001/_search @@ -582,6 +597,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] ```console-result { @@ -639,6 +655,7 @@ GET /my-index-000001/_search } } ``` +% TESTRESPONSE[s/"took": 55,/"took": "$body.took",/] 1. The first document matched only the first sub-query. 2. The second document matched only the second sub-query. @@ -679,6 +696,7 @@ GET /my-index-000001/_search } } ``` +% TEST[continued] 1. The `name` parameter will be used to identify which percolator document slots belong to what `percolate` query. @@ -723,6 +741,7 @@ The above search request returns a response similar to this: } } ``` +% TESTRESPONSE[s/"took": 13,/"took": "$body.took",/] 1. The `_percolator_document_slot_query1` percolator slot field indicates that these matched slots are from the `percolate` query with `_name` parameter set to `query1`. diff --git a/docs/reference/query-languages/query-dsl-query-string-query.md b/docs/reference/query-languages/query-dsl-query-string-query.md index 76899ea99c31c..d34989825e3b3 100644 --- a/docs/reference/query-languages/query-dsl-query-string-query.md +++ b/docs/reference/query-languages/query-dsl-query-string-query.md @@ -452,6 +452,7 @@ GET /my-index-000001/_search } } ``` +% TEST[setup:my_index] The reserved characters are: `+ - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /` diff --git a/docs/reference/query-languages/query-dsl-range-query.md b/docs/reference/query-languages/query-dsl-range-query.md index e541a656de617..a48917599bae9 100644 --- a/docs/reference/query-languages/query-dsl-range-query.md +++ b/docs/reference/query-languages/query-dsl-range-query.md @@ -216,6 +216,7 @@ GET /_search } } ``` +% TEST[continued] 1. Indicates that `date` values use a UTC offset of `+01:00`. 2. With a UTC offset of `+01:00`, {{es}} converts this date to `2019-12-31T23:00:00 UTC`. diff --git a/docs/reference/query-languages/query-dsl-rank-feature-query.md b/docs/reference/query-languages/query-dsl-rank-feature-query.md index 247e981851932..1b9a0c5ac72e9 100644 --- a/docs/reference/query-languages/query-dsl-rank-feature-query.md +++ b/docs/reference/query-languages/query-dsl-rank-feature-query.md @@ -58,6 +58,7 @@ PUT /test } } ``` +% TESTSETUP Index several documents to the `test` index. diff --git a/docs/reference/query-languages/query-dsl-script-query.md b/docs/reference/query-languages/query-dsl-script-query.md index 3cb070210b733..7033d9fb9e925 100644 --- a/docs/reference/query-languages/query-dsl-script-query.md +++ b/docs/reference/query-languages/query-dsl-script-query.md @@ -41,6 +41,8 @@ GET /_search } } ``` +% TEST[setup:ledger] +% TEST[s/_search/_search?filter_path=hits.hits&sort=amount/] You can achieve the same results in a search query by using runtime fields. Use the [`fields`](/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API to fetch values as part of the same query: @@ -71,6 +73,8 @@ GET /_search "fields": [{"field": "amount.signed"}] } ``` +% TEST[setup:ledger] +% TEST[s/_search/_search?filter_path=hits.hits.fields&sort=amount.signed:desc/] ## Top-level parameters for `script` [script-top-level-params] diff --git a/docs/reference/query-languages/query-dsl-script-score-query.md b/docs/reference/query-languages/query-dsl-script-score-query.md index 9cc61cfb45c5f..04299ece540e0 100644 --- a/docs/reference/query-languages/query-dsl-script-score-query.md +++ b/docs/reference/query-languages/query-dsl-script-score-query.md @@ -88,6 +88,7 @@ We suggest using these predefined functions instead of writing your own. These f "source" : "saturation(doc['my-int'].value, 1)" } ``` +% NOTCONSOLE #### Sigmoid [script-score-sigmoid] @@ -99,6 +100,7 @@ We suggest using these predefined functions instead of writing your own. These f "source" : "sigmoid(doc['my-int'].value, 2, 1)" } ``` +% NOTCONSOLE #### Random score function [random-score-function] @@ -112,6 +114,7 @@ We suggest using these predefined functions instead of writing your own. These f "source" : "randomScore(100, '_seq_no')" } ``` +% NOTCONSOLE If the `fieldName` parameter is omitted, the internal Lucene document ids will be used as a source of randomness. This is very efficient, but unfortunately not reproducible since documents might be renumbered by merges. @@ -120,6 +123,7 @@ If the `fieldName` parameter is omitted, the internal Lucene document ids will b "source" : "randomScore(100)" } ``` +% NOTCONSOLE Note that documents that are within the same shard and have the same value for field will get the same score, so it is usually desirable to use a field that has unique values for all documents across a shard. A good default choice might be to use the `_seq_no` field, whose only drawback is that scores will change if the document is updated since update operations also update the value of the `_seq_no` field. @@ -143,6 +147,7 @@ You can read more about decay functions [here](/reference/query-languages/query- } } ``` +% NOTCONSOLE 1. Using `params` allows to compile the script only once, even if params change. @@ -165,6 +170,7 @@ You can read more about decay functions [here](/reference/query-languages/query- } } ``` +% NOTCONSOLE #### Decay functions for date fields [decay-functions-date-fields] @@ -184,6 +190,7 @@ You can read more about decay functions [here](/reference/query-languages/query- } } ``` +% NOTCONSOLE ::::{note} Decay functions on dates are limited to dates in the default format and default time zone. Also calculations with `now` are not supported. @@ -239,6 +246,7 @@ What you used in `script_score` of the Function Score query, you can copy into t } } ``` +% NOTCONSOLE #### `random_score` [random-score] @@ -258,6 +266,7 @@ Use `randomScore` function as described in [random score function](#random-score } } ``` +% NOTCONSOLE For checking if a document has a missing value, you can use `doc['field'].size() == 0`. For example, this script will use a value `1` if a document doesn’t have a field `field`: @@ -269,6 +278,7 @@ For checking if a document has a missing value, you can use `doc['field'].size() } } ``` +% NOTCONSOLE This table lists how `field_value_factor` modifiers can be implemented through a script: @@ -542,6 +552,7 @@ You can check if a document has a value for the field `my_vector` with `doc['my_ ```js "source": "doc['my_vector'].size() == 0 ? 0 : cosineSimilarity(params.queryVector, 'my_vector')" ``` +% NOTCONSOLE #### Accessing vectors directly [vector-functions-accessing-vectors] @@ -647,6 +658,7 @@ PUT my-index-bit-vectors/_doc/3 POST my-index-bit-vectors/_refresh ``` +% TEST[continued] 1. The number of dimensions or bits for the `bit` vector. 2. This vector represents 5 bytes, or `5 * 8 = 40` bits, which equals the configured dimensions @@ -670,6 +682,7 @@ GET my-index-bit-vectors/_search } } ``` +% TEST[continued] 1. This vector is 40 bits, and thus will compute a bitwise `&` operation with the stored vectors. @@ -692,6 +705,7 @@ GET my-index-bit-vectors/_search } } ``` +% TEST[continued] 1. This vector is 40 individual dimensions, and thus will sum the floating point values using the stored `bit` vector as a mask. @@ -726,6 +740,7 @@ GET /my-index-000001/_explain/0 } } ``` +% TEST[setup:my_index] Note that the `explanation` will be null when using in a normal `_search` request, so having a conditional guard is best practice. diff --git a/docs/reference/query-languages/query-dsl-semantic-query.md b/docs/reference/query-languages/query-dsl-semantic-query.md index c142648503460..9ce4b95e17b23 100644 --- a/docs/reference/query-languages/query-dsl-semantic-query.md +++ b/docs/reference/query-languages/query-dsl-semantic-query.md @@ -28,6 +28,7 @@ GET my-index-000001/_search } } ``` +% TEST[skip: Requires inference endpoints] ## Top-level parameters for `semantic` [semantic-query-params] @@ -72,6 +73,7 @@ POST my-index/_search } } ``` +% TEST[skip: Requires inference endpoints] You can also use semantic_text as part of [Reciprocal Rank Fusion](/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md) to make ranking relevant results easier: @@ -107,4 +109,5 @@ GET my-index/_search } } ``` +% TEST[skip: Requires inference endpoints] diff --git a/docs/reference/query-languages/query-dsl-shape-query.md b/docs/reference/query-languages/query-dsl-shape-query.md index d94c012981c4c..948eaac87f00c 100644 --- a/docs/reference/query-languages/query-dsl-shape-query.md +++ b/docs/reference/query-languages/query-dsl-shape-query.md @@ -40,6 +40,7 @@ PUT /example/_doc/1?refresh=wait_for } } ``` +% TESTSETUP The following query will find the point using the Elasticsearch’s `envelope` GeoJSON extension: diff --git a/docs/reference/query-languages/query-dsl-sparse-vector-query.md b/docs/reference/query-languages/query-dsl-sparse-vector-query.md index b13105f4a90d5..f013105572231 100644 --- a/docs/reference/query-languages/query-dsl-sparse-vector-query.md +++ b/docs/reference/query-languages/query-dsl-sparse-vector-query.md @@ -31,6 +31,7 @@ GET _search } } ``` +% TEST[skip: Requires inference] ## Example request using precomputed vectors [_example_request_using_precomputed_vectors] @@ -46,6 +47,7 @@ GET _search } } ``` +% TEST[skip: TBD] ## Top level parameters for `sparse_vector` [sparse-vector-field-params] @@ -101,6 +103,7 @@ GET my-index/_search } } ``` +% TEST[skip: Requires inference] Multiple `sparse_vector` queries can be combined with each other or other query types. This can be achieved by wrapping them in [boolean query clauses](/reference/query-languages/query-dsl-bool-query.md) and using linear boosting: @@ -141,6 +144,7 @@ GET my-index/_search } } ``` +% TEST[skip: Requires inference] This can also be achieved using [reciprocal rank fusion (RRF)](/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md), through an [`rrf` retriever](/reference/elasticsearch/rest-apis/retrievers.md#rrf-retriever) with multiple [`standard` retrievers](/reference/elasticsearch/rest-apis/retrievers.md#standard-retriever). @@ -194,6 +198,7 @@ GET my-index/_search } } ``` +% TEST[skip: Requires inference] ## Example ELSER query with pruning configuration and rescore [sparse-vector-query-with-pruning-config-and-rescore-example] @@ -238,6 +243,7 @@ GET my-index/_search } } ``` +% TEST[skip: Requires inference] ::::{note} When performing [cross-cluster search](docs-content://solutions/search/cross-cluster-search.md), inference is performed on the local cluster. diff --git a/docs/reference/query-languages/query-dsl-text-expansion-query.md b/docs/reference/query-languages/query-dsl-text-expansion-query.md index e5839efd881c0..3f003a40d709a 100644 --- a/docs/reference/query-languages/query-dsl-text-expansion-query.md +++ b/docs/reference/query-languages/query-dsl-text-expansion-query.md @@ -38,6 +38,7 @@ GET _search } } ``` +% TEST[skip: TBD] ## Top level parameters for `text_expansion` [text-expansion-query-params] @@ -91,6 +92,7 @@ GET my-index/_search } } ``` +% TEST[skip: TBD] Multiple `text_expansion` queries can be combined with each other or other query types. This can be achieved by wrapping them in [boolean query clauses](/reference/query-languages/query-dsl-bool-query.md) and using linear boosting: @@ -133,6 +135,7 @@ GET my-index/_search } } ``` +% TEST[skip: TBD] This can also be achieved using [reciprocal rank fusion (RRF)](/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md), through an [`rrf` retriever](/reference/elasticsearch/rest-apis/retrievers.md#rrf-retriever) with multiple [`standard` retrievers](/reference/elasticsearch/rest-apis/retrievers.md#standard-retriever). @@ -186,6 +189,7 @@ GET my-index/_search } } ``` +% TEST[skip: TBD] ## Example ELSER query with pruning configuration and rescore [text-expansion-query-with-pruning-config-and-rescore-example] @@ -230,6 +234,7 @@ GET my-index/_search } } ``` +% TEST[skip: TBD] ::::{note} Depending on your data, the text expansion query may be faster with `track_total_hits: false`. diff --git a/docs/reference/query-languages/query-dsl-weighted-tokens-query.md b/docs/reference/query-languages/query-dsl-weighted-tokens-query.md index 06be623bee1f2..a14aef5d0a06a 100644 --- a/docs/reference/query-languages/query-dsl-weighted-tokens-query.md +++ b/docs/reference/query-languages/query-dsl-weighted-tokens-query.md @@ -43,6 +43,7 @@ POST _search } } ``` +% TEST[skip: TBD] ## Top level parameters for `weighted_token` [weighted-token-query-params] @@ -110,4 +111,5 @@ GET my-index/_search } } ``` +% TEST[skip: TBD] diff --git a/docs/reference/query-languages/regexp-syntax.md b/docs/reference/query-languages/regexp-syntax.md index 20d469f576290..db106701989af 100644 --- a/docs/reference/query-languages/regexp-syntax.md +++ b/docs/reference/query-languages/regexp-syntax.md @@ -56,6 +56,7 @@ GET my-index-000001/_search } } ``` +% TEST[continued] :::: diff --git a/docs/reference/query-languages/sql-data-types.md b/docs/reference/query-languages/sql-data-types.md index f7bee95267e91..7ed10380ccc66 100644 --- a/docs/reference/query-languages/sql-data-types.md +++ b/docs/reference/query-languages/sql-data-types.md @@ -89,6 +89,7 @@ Consider the following `string` mapping: } } ``` +% NOTCONSOLE The following SQL query: