ES|QL change_point docs and tech preview

docs/reference/query-languages/esql/README.md

@@ -1,12 +1,25 @@
 The ES|QL documentation is composed of static content and generated content.
 The static content exists in this directory and can be edited by hand.
-However, the sub-directories `_snippets`, `images` and `kibana` contain mostly
-generated content.
+However, the subdirectories `_snippets`, `images` and `kibana` contain a mix
+of static and generated content, and so updating them is a bit more involved.
+
+## Static content
+
+The root `esql` directory and the following two subdirectories contain static content:
+* `commands` - contains the static content for the ES|QL commands.
+  This content will typically contain mostly `include` directives for content in the `_snippets` or `images` directories.
+* `functions-operators` - contains the static content for the ES|QL functions and operators.
+  Again this will contain mostly `include` directives for content in the `_snippets` or `images` directories.
+
+## Mixed static and generated content
+
+Generated content is created by running the ESQL tests in the `x-pack/plugin/esql` module.
+It will be written into three subdirectories of the `esql` directory:
 
 ### _snippets
 
-In `_snippets` there are files that can be included within other files
-using the [File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
+In `_snippets` there are files that can be included within other files using the
+[File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
 feature of the Elastic Docs V3 system.
 Most, but not all, files in this directory are generated.
 In particular the directories `_snippets/functions/*` and `_snippets/operators/*`
@@ -21,7 +34,7 @@ contain subdirectories that are mostly generated:
 
 Most functions can use the generated docs generated in the `layout` directory.
 If we need something more custom for the function we can make a file in this
-directory that can `include::` any parts of the files above.
+directory that can `include` any parts of the files above.
 
 To regenerate the files for a function run its tests using gradle.
 For example to generate docs for the `CASE` function:
@@ -34,17 +47,140 @@ To regenerate the files for all functions run all of ESQL's tests using gradle:
 ./gradlew :x-pack:plugin:esql:test
 ```
 
+#### Lists
+
+The `_snippets/lists` directory contains re-usable content for lists of commands, functions or operators.
+Whenever adding a command, function or operator, you usually need to add it to one of these lists.
+The lists should also match natural groupings of the commands, functions or operators.
+For example, when adding an aggregation function, add to the `aggregation-functions.md` file.
+
+#### Commands
+
+The `_snippets/commands` directory contains the content for the ES|QL commands.
+There are two subdirectories, one static and one generated:
+* `layout` - contains the static content for the ES|QL commands.
+  The files in this directory are the main content for the documentation for the commands.
+  They are not generated, and so this is the primary place to edit the content, or add new commands.
+* `examples` - contains the generated content for the ES|QL commands.
+  The files in this directory are generated from the test `CommandDocsTests` in the `x-pack/plugin/esql` module.
+  The structure of the subdirectories mimics the csv-spec files and test tags used in the tests.
+
+Including generated examples in the command documentation is done by using the include directive.
+
 ### images
 
-The `images` directory contains `functions` and `operators` sub-directories with
+The `images` directory contains `functions` and `operators` subdirectories with
 the `*.svg` files used to describe the syntax of each function or operator.
 These are all generated by the same tests that generate the functions and operators docs above.
 
 ### kibana
 
-The `kibana` directory contains `definition` and `docs` sub-directories that are generated:
+The `kibana` directory contains `definition` and `docs` subdirectories that are generated:
 
 * `kibana/definition` - function definitions for kibana's ESQL editor
 * `kibana/docs` - the inline docs for kibana
 
 These are also generated as part of the unit tests described above.
+
+## Under the hood
+
+There are three overlapping mechanisms for generating the content:
+* The `AbstractFunctionTestCase` class generates the content for all the functions and most operators.
+  This class makes use of the `DocsV3Support` class to generate the content.
+  It uses the `@FunctionInfo` and `@Param` annotations on function and operator classes to know what content should be generated.
+  All tests that extend this class will automatically generate the content for the functions they test.
+* Some operators do not have a clear class or test class, and so the content is generated by custom
+  tests that do not extend the `AbstractOperatorTestCase` class. See, for example, operators such as `Cast ::`,
+  which uses `CastOperatorTests` to call directly into the `DocsV3Support` class to generate the content.
+* Commands do not have dedicated classes or test classes with annotation that can be used.
+  For this reason, the command documentation is generated by the `CommandDocsTests` class.
+  Currently, this only covers tested examples used in the documentation, and all other commands
+  content is static.
+  Since there are no annotations to mark which examples to use, the command documentation
+  relies on the docs author providing the knowledge of which examples to use by creating subdirectories
+  and examples files that match the csv-spec files and tags to include.
+
+To help differentiate between the static and generated content, the generated content is prefixed with a comment:
+```
+% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
+```
+
+## Tutorials
+
+### Adding a new command
+
+When adding a new command, for example adding the `CHANGE_POINT` command, do the following:
+1. Create a new file in the `_snippets/commands/layout` directory with the name of the command, for example `change_point.md`.
+2. Add the content for the command to the file. See other files in this directory for examples.
+3. Add the command to the list in `_snippets/lists/processing-commands.md`.
+4. Add an include directive to the `commands/processing-commands.md` file to include the new command.
+5. Add tested examples to the `_snippets/commands/examples` directory. See below for details.
+
+### Adding examples to commands
+
+When adding tested examples to a command, for example adding an example to the `CHANGE_POINT` command, do the following:
+* Make sure you have an example in an appropriate csv-spec file in the `x-pack/plugin/esql/qa/testFixtures/src/main/resources/` directory.
+* Make sure the example has a tag that is unique in that file, and matches the intent of the test, or the docs reason for including that test.
+* If you only want to show the query, and no results, then do not tag the results table,
+  otherwise tag the results table with a tag that has the same name as the query tag, but with the suffix `-result`.
+* Create a file with the name of the tag in a subdirectory with the name of the csv-spec file
+  in the `_snippets/commands/examples` directory. While you could add the content to that file, it is not necessary, merely that the file exists
+* Run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
+
+For example, we tag the following test in change_point.csv-spec:
+
+```
+example for docs
+required_capability: change_point
+
+// tag::changePointForDocs[]
+ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
+| MV_EXPAND key
+| EVAL value = CASE(key<13, 0, 42)
+| CHANGE_POINT value ON key
+| WHERE type IS NOT NULL
+// end::changePointForDocs[]
+;
+
+// tag::changePointForDocs-result[]
+key:integer | value:integer | type:keyword | pvalue:double
+13          | 42            | step_change  | 0.0
+// end::changePointForDocs-result[]
+;
+```
+
+Then we create the file `_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md` with the content:
+```
+This should be overwritten
+```
+
+Then we run the test `CommandDocsTests` in the `x-pack/plugin/esql` module to generate the content.
+
+Now the content of the changePointForDocs.md file should have been updated:
+
+```
+% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
+
+\```esql
+ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
+| MV_EXPAND key
+| EVAL value = CASE(key<13, 0, 42)
+| CHANGE_POINT value ON key
+| WHERE type IS NOT NULL
+\```
+
+| key:integer | value:integer | type:keyword | pvalue:double |
+| --- | --- | --- | --- |
+| 13 | 42 | step_change | 0.0 |
+```
+
+Finally include this file in the `CHANGE_POINT` command file `_snippets/commands/layout/change_point.md`:
+
+```
+**Examples**
+
+The following example shows the detection of a step change:
+
+:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
+:::
+```

docs/reference/query-languages/esql/_snippets/commands/examples/change_point.csv-spec/changePointForDocs.md

@@ -0,0 +1,13 @@
+% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
+
+```esql
+ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
+| MV_EXPAND key
+| EVAL value = CASE(key<13, 0, 42)
+| CHANGE_POINT value ON key
+| WHERE type IS NOT NULL
+```
+
+| key:integer | value:integer | type:keyword | pvalue:double |
+| --- | --- | --- | --- |
+| 13 | 42 | step_change | 0.0 |

docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md

@@ -0,0 +1,58 @@
+## `CHANGE_POINT` [esql-change_point]
+
+:::{note}
+The `CHANGE_POINT` command requires a [platinum license](https://www.elastic.co/subscriptions).
+:::
+
+::::{warning}
+This functionality is in technical preview and may be
+changed or removed in a future release. Elastic will work to fix any
+issues, but features in technical preview are not subject to the support
+SLA of official GA features.
+::::
+
+`CHANGE_POINT` detects spikes, dips, and change points in a metric.
+
+**Syntax**
+
+```esql
+CHANGE_POINT value [ON key] [AS type_name, pvalue_name]
+```
+
+**Parameters**
+
+`value`
+:   The column with the metric in which you want to detect a change point.
+
+`key`
+:   The column with the key to order the values by. If not specified, `@timestamp` is used.
+
+`type_name`
+:   The name of the output column with the change point type. If not specified, `type` is used.
+
+`pvalue_name`
+:   The name of the output column with the p-value that indicates how extreme the change point is. If not specified, `pvalue` is used.
+
+**Description**
+
+`CHANGE_POINT` detects spikes, dips, and change points in a metric. The command adds columns to
+the table with the change point type and p-value, that indicates how extreme the change point is
+(lower values indicate greater changes).
+
+The possible change point types are:
+* `dip`: a significant dip occurs at this change point
+* `distribution_change`: the overall distribution of the values has changed significantly
+* `spike`: a significant spike occurs at this point
+* `step_change`: the change indicates a statistically significant step up or down in value distribution
+* `trend_change`: there is an overall trend change occurring at this point
+
+::::{note}
+There must be at least 22 values for change point detection. Fewer than 1,000 is preferred.
+::::
+
+**Examples**
+
+The following example shows the detection of a step change:
+
+:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
+:::

docs/reference/query-languages/esql/_snippets/lists/processing-commands.md

@@ -1,3 +1,4 @@
+* [`CHANGE_POINT`](../../commands/processing-commands.md#esql-change_point)
 * [`DISSECT`](../../commands/processing-commands.md#esql-dissect)
 * [`DROP`](../../commands/processing-commands.md#esql-drop)
 * [`ENRICH`](../../commands/processing-commands.md#esql-enrich)

docs/reference/query-languages/esql/commands/processing-commands.md

@@ -17,6 +17,9 @@ mapped_pages:
 :::{include} ../_snippets/lists/processing-commands.md
 :::
 
+:::{include} ../_snippets/commands/layout/change_point.md
+:::
+
 :::{include} ../_snippets/commands/layout/dissect.md
 :::
 

x-pack/plugin/esql/qa/testFixtures/src/main/resources/change_point.csv-spec

@@ -1198,3 +1198,22 @@ true        | 1             | null         | null
 true        | 1             | null         | null         
 true        | 1             | null         | null         
 ;
+
+
+example for docs
+required_capability: change_point
+
+// tag::changePointForDocs[]
+ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
+| MV_EXPAND key
+| EVAL value = CASE(key<13, 0, 42)
+| CHANGE_POINT value ON key
+| WHERE type IS NOT NULL
+// end::changePointForDocs[]
+;
+
+// tag::changePointForDocs-result[]
+key:integer | value:integer | type:keyword | pvalue:double
+13          | 42            | step_change  | 0.0          
+// end::changePointForDocs-result[]
+;

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4

@@ -55,10 +55,10 @@ processingCommand
     | enrichCommand
     | mvExpandCommand
     | joinCommand
+    | changePointCommand
     // in development
     | {this.isDevVersion()}? inlinestatsCommand
     | {this.isDevVersion()}? lookupCommand
-    | {this.isDevVersion()}? changePointCommand
     | {this.isDevVersion()}? completionCommand
     | {this.isDevVersion()}? insistCommand
     | {this.isDevVersion()}? forkCommand
@@ -257,7 +257,7 @@ inlinestatsCommand
     ;
 
 changePointCommand
-    : DEV_CHANGE_POINT value=qualifiedName (ON key=qualifiedName)? (AS targetType=qualifiedName COMMA targetPvalue=qualifiedName)?
+    : CHANGE_POINT value=qualifiedName (ON key=qualifiedName)? (AS targetType=qualifiedName COMMA targetPvalue=qualifiedName)?
     ;
 
 insistCommand

x-pack/plugin/esql/src/main/antlr/lexer/ChangePoint.g4

@@ -9,7 +9,7 @@ lexer grammar ChangePoint;
 //
 // | CHANGE_POINT command
 //
-DEV_CHANGE_POINT : {this.isDevVersion()}? 'change_point'  -> pushMode(CHANGE_POINT_MODE);
+CHANGE_POINT : 'change_point' -> pushMode(CHANGE_POINT_MODE);
 
 mode CHANGE_POINT_MODE;
 CHANGE_POINT_PIPE : PIPE -> type(PIPE), popMode;

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

@@ -843,7 +843,7 @@ public enum Cap {
         /**
          * Support change point detection "CHANGE_POINT".
          */
-        CHANGE_POINT(Build.current().isSnapshot()),
+        CHANGE_POINT,
 
         /**
          * Fix for https://github.com/elastic/elasticsearch/issues/120817