Re-enable use_doc_values_skipper and add specialized lucene query for @timestamp field filtering.

[martijnvg]

Relates to #127263

[martijnvg]

Elastic/logs run comparing main (baseline) to this branch (contender):

|                                                        Metric |                                                              Task |         Baseline |        Contender |            Diff |   Unit |    Diff % |
|--------------------------------------------------------------:|------------------------------------------------------------------:|-----------------:|-----------------:|----------------:|-------:|----------:|
|                    Cumulative indexing time of primary shards |                                                                   |   1378.05        |   1433.49        |     55.4404     |    min |    +4.02% |
|             Min cumulative indexing time across primary shard |                                                                   |     13.0375      |     12.9382      |     -0.09927    |    min |    -0.76% |
|          Median cumulative indexing time across primary shard |                                                                   |     48.7185      |     49.362       |      0.6435     |    min |    +1.32% |
|             Max cumulative indexing time across primary shard |                                                                   |    528.288       |    549.375       |     21.0876     |    min |    +3.99% |
|           Cumulative indexing throttle time of primary shards |                                                                   |      0           |      0           |      0          |    min |     0.00% |
|    Min cumulative indexing throttle time across primary shard |                                                                   |      0           |      0           |      0          |    min |     0.00% |
| Median cumulative indexing throttle time across primary shard |                                                                   |      0           |      0           |      0          |    min |     0.00% |
|    Max cumulative indexing throttle time across primary shard |                                                                   |      0           |      0           |      0          |    min |     0.00% |
|                       Cumulative merge time of primary shards |                                                                   |    417.51        |    445.672       |     28.1619     |    min |    +6.75% |
|                      Cumulative merge count of primary shards |                                                                   |    685           |    625           |    -60          |        |    -8.76% |
|                Min cumulative merge time across primary shard |                                                                   |      2.49278     |      1.83402     |     -0.65877    |    min |   -26.43% |
|             Median cumulative merge time across primary shard |                                                                   |      8.05628     |      8.50478     |      0.4485     |    min |    +5.57% |
|                Max cumulative merge time across primary shard |                                                                   |    205.172       |    222.632       |     17.4602     |    min |    +8.51% |
|              Cumulative merge throttle time of primary shards |                                                                   |    112.126       |    106.872       |     -5.25342    |    min |    -4.69% |
|       Min cumulative merge throttle time across primary shard |                                                                   |      0.386633    |      0.296383    |     -0.09025    |    min |   -23.34% |
|    Median cumulative merge throttle time across primary shard |                                                                   |      1.54876     |      1.39224     |     -0.15652    |    min |   -10.11% |
|       Max cumulative merge throttle time across primary shard |                                                                   |     52.067       |     45.6801      |     -6.38687    |    min |   -12.27% |
|                     Cumulative refresh time of primary shards |                                                                   |     33.1347      |     40.2614      |      7.12675    |    min |   +21.51% |
|                    Cumulative refresh count of primary shards |                                                                   |   3209           |   3118           |    -91          |        |    -2.84% |
|              Min cumulative refresh time across primary shard |                                                                   |      0.16525     |      0.1076      |     -0.05765    |    min |   -34.89% |
|           Median cumulative refresh time across primary shard |                                                                   |      0.352325    |      0.489192    |      0.13687    |    min |   +38.85% |
|              Max cumulative refresh time across primary shard |                                                                   |     20.1349      |     23.7248      |      3.58995    |    min |   +17.83% |
|                       Cumulative flush time of primary shards |                                                                   |     62.4967      |     78.0751      |     15.5783     |    min |   +24.93% |
|                      Cumulative flush count of primary shards |                                                                   |   1733           |   1724           |     -9          |        |    -0.52% |
|                Min cumulative flush time across primary shard |                                                                   |      1.26627     |      1.75812     |      0.49185    |    min |   +38.84% |
|             Median cumulative flush time across primary shard |                                                                   |      3.76469     |      4.74942     |      0.98472    |    min |   +26.16% |
|                Max cumulative flush time across primary shard |                                                                   |     10.5378      |     12.9788      |      2.44095    |    min |   +23.16% |
|                                       Total Young Gen GC time |                                                                   |     85.745       |    120.555       |     34.81       |      s |   +40.60% |
|                                      Total Young Gen GC count |                                                                   |   6663           |   6692           |     29          |        |    +0.44% |
|                                         Total Old Gen GC time |                                                                   |      0           |      0           |      0          |      s |     0.00% |
|                                        Total Old Gen GC count |                                                                   |      0           |      0           |      0          |        |     0.00% |
|                                                    Store size |                                                                   |     41.3076      |     39.4524      |     -1.85513    |     GB |    -4.49% |
|                                                 Translog size |                                                                   |      7.17118e-07 |      7.17118e-07 |      0          |     GB |     0.00% |
|                                        Heap used for segments |                                                                   |      0           |      0           |      0          |     MB |     0.00% |
|                                      Heap used for doc values |                                                                   |      0           |      0           |      0          |     MB |     0.00% |
|                                           Heap used for terms |                                                                   |      0           |      0           |      0          |     MB |     0.00% |
|                                           Heap used for norms |                                                                   |      0           |      0           |      0          |     MB |     0.00% |
|                                          Heap used for points |                                                                   |      0           |      0           |      0          |     MB |     0.00% |
|                                   Heap used for stored fields |                                                                   |      0           |      0           |      0          |     MB |     0.00% |
|                                                 Segment count |                                                                   |     14           |     14           |      0          |        |     0.00% |
|                                   Total Ingest Pipeline count |                                                                   |      4.88617e+08 |      4.88641e+08 |  24000          |        |     0.00% |
|                                    Total Ingest Pipeline time |                                                                   |      4.06342e+07 |      4.17218e+07 |      1.0876e+06 |     ms |    +2.68% |
|                                  Total Ingest Pipeline failed |                                                                   |      0           |      0           |      0          |        |     0.00% |
|                                                Min Throughput |                                                  insert-pipelines |     12.6071      |     13.315       |      0.70789    |  ops/s |    +5.61% |
|                                               Mean Throughput |                                                  insert-pipelines |     12.6071      |     13.315       |      0.70789    |  ops/s |    +5.61% |
|                                             Median Throughput |                                                  insert-pipelines |     12.6071      |     13.315       |      0.70789    |  ops/s |    +5.61% |
|                                                Max Throughput |                                                  insert-pipelines |     12.6071      |     13.315       |      0.70789    |  ops/s |    +5.61% |
|                                      100th percentile latency |                                                  insert-pipelines |   1135.89        |   1070.32        |    -65.5646     |     ms |    -5.77% |
|                                 100th percentile service time |                                                  insert-pipelines |   1135.89        |   1070.32        |    -65.5646     |     ms |    -5.77% |
|                                                    error rate |                                                  insert-pipelines |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                        insert-ilm |     24.717       |     25.8917      |      1.17471    |  ops/s |    +4.75% |
|                                               Mean Throughput |                                                        insert-ilm |     24.717       |     25.8917      |      1.17471    |  ops/s |    +4.75% |
|                                             Median Throughput |                                                        insert-ilm |     24.717       |     25.8917      |      1.17471    |  ops/s |    +4.75% |
|                                                Max Throughput |                                                        insert-ilm |     24.717       |     25.8917      |      1.17471    |  ops/s |    +4.75% |
|                                      100th percentile latency |                                                        insert-ilm |     39.273       |     37.593       |     -1.68002    |     ms |    -4.28% |
|                                 100th percentile service time |                                                        insert-ilm |     39.273       |     37.593       |     -1.68002    |     ms |    -4.28% |
|                                                    error rate |                                                        insert-ilm |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                            validate-package-template-installation |     46.5037      |     48.3365      |      1.83286    |  ops/s |    +3.94% |
|                                               Mean Throughput |                            validate-package-template-installation |     46.5037      |     48.3365      |      1.83286    |  ops/s |    +3.94% |
|                                             Median Throughput |                            validate-package-template-installation |     46.5037      |     48.3365      |      1.83286    |  ops/s |    +3.94% |
|                                                Max Throughput |                            validate-package-template-installation |     46.5037      |     48.3365      |      1.83286    |  ops/s |    +3.94% |
|                                      100th percentile latency |                            validate-package-template-installation |     21.1994      |     20.4         |     -0.79937    |     ms |    -3.77% |
|                                 100th percentile service time |                            validate-package-template-installation |     21.1994      |     20.4         |     -0.79937    |     ms |    -3.77% |
|                                                    error rate |                            validate-package-template-installation |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                   update-custom-package-templates |     30.5803      |     28.3372      |     -2.24308    |  ops/s |    -7.34% |
|                                               Mean Throughput |                                   update-custom-package-templates |     30.5803      |     28.3372      |     -2.24308    |  ops/s |    -7.34% |
|                                             Median Throughput |                                   update-custom-package-templates |     30.5803      |     28.3372      |     -2.24308    |  ops/s |    -7.34% |
|                                                Max Throughput |                                   update-custom-package-templates |     30.5803      |     28.3372      |     -2.24308    |  ops/s |    -7.34% |
|                                      100th percentile latency |                                   update-custom-package-templates |    392.086       |    423.177       |     31.0905     |     ms |    +7.93% |
|                                 100th percentile service time |                                   update-custom-package-templates |    392.086       |    423.177       |     31.0905     |     ms |    +7.93% |
|                                                    error rate |                                   update-custom-package-templates |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                        bulk-index |    577.882       |   1015.65        |    437.768      | docs/s |   +75.75% |
|                                               Mean Throughput |                                                        bulk-index |  58746.6         |  58311           |   -435.619      | docs/s |    -0.74% |
|                                             Median Throughput |                                                        bulk-index |  58823.2         |  58411.2         |   -412.056      | docs/s |    -0.70% |
|                                                Max Throughput |                                                        bulk-index |  60417.2         |  59687           |   -730.211      | docs/s |    -1.21% |
|                                       50th percentile latency |                                                        bulk-index |    307.914       |   1661.4         |   1353.49       |     ms |  +439.57% |
|                                       90th percentile latency |                                                        bulk-index |    624.675       |   2926.45        |   2301.77       |     ms |  +368.48% |
|                                       99th percentile latency |                                                        bulk-index |   1290.95        |   4160.24        |   2869.29       |     ms |  +222.26% |
|                                     99.9th percentile latency |                                                        bulk-index |   2664.75        |   6932.6         |   4267.84       |     ms |  +160.16% |
|                                    99.99th percentile latency |                                                        bulk-index |   4671.87        |   9422.26        |   4750.39       |     ms |  +101.68% |
|                                      100th percentile latency |                                                        bulk-index |   7806.17        |  11269.2         |   3463.06       |     ms |   +44.36% |
|                                  50th percentile service time |                                                        bulk-index |    309.077       |   1666.69        |   1357.62       |     ms |  +439.25% |
|                                  90th percentile service time |                                                        bulk-index |    624.214       |   2931.09        |   2306.87       |     ms |  +369.56% |
|                                  99th percentile service time |                                                        bulk-index |   1289.21        |   4152.87        |   2863.67       |     ms |  +222.13% |
|                                99.9th percentile service time |                                                        bulk-index |   2669.62        |   6934.12        |   4264.5        |     ms |  +159.74% |
|                               99.99th percentile service time |                                                        bulk-index |   4687.56        |   9362.18        |   4674.62       |     ms |   +99.72% |
|                                 100th percentile service time |                                                        bulk-index |   7806.17        |  11269.2         |   3463.06       |     ms |   +44.36% |
|                                                    error rate |                                                        bulk-index |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                 discovery-search-request-size-100 |      5.98603     |      0.439556    |     -5.54647    |  ops/s |   -92.66% |
|                                               Mean Throughput |                                 discovery-search-request-size-100 |     11.1157      |      0.527343    |    -10.5883     |  ops/s |   -95.26% |
|                                             Median Throughput |                                 discovery-search-request-size-100 |     11.1157      |      0.543067    |    -10.5726     |  ops/s |   -95.11% |
|                                                Max Throughput |                                 discovery-search-request-size-100 |     16.2453      |      0.57074     |    -15.6746     |  ops/s |   -96.49% |
|                                       50th percentile latency |                                 discovery-search-request-size-100 |     35.5222      |   1667.32        |   1631.8        |     ms | +4593.75% |
|                                       90th percentile latency |                                 discovery-search-request-size-100 |     49.3952      |   2059.83        |   2010.43       |     ms | +4070.10% |
|                                      100th percentile latency |                                 discovery-search-request-size-100 |    342.628       |   2273.43        |   1930.8        |     ms |  +563.53% |
|                                  50th percentile service time |                                 discovery-search-request-size-100 |     35.5222      |   1667.32        |   1631.8        |     ms | +4593.75% |
|                                  90th percentile service time |                                 discovery-search-request-size-100 |     49.3952      |   2059.83        |   2010.43       |     ms | +4070.10% |
|                                 100th percentile service time |                                 discovery-search-request-size-100 |    342.628       |   2273.43        |   1930.8        |     ms |  +563.53% |
|                                                    error rate |                                 discovery-search-request-size-100 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                 discovery-search-request-size-500 |     10.3302      |      0.580801    |     -9.74943    |  ops/s |   -94.38% |
|                                               Mean Throughput |                                 discovery-search-request-size-500 |     10.9704      |      0.584172    |    -10.3862     |  ops/s |   -94.67% |
|                                             Median Throughput |                                 discovery-search-request-size-500 |     11.1098      |      0.584131    |    -10.5257     |  ops/s |   -94.74% |
|                                                Max Throughput |                                 discovery-search-request-size-500 |     11.3315      |      0.588834    |    -10.7427     |  ops/s |   -94.80% |
|                                       50th percentile latency |                                 discovery-search-request-size-500 |     85.2927      |   1705.86        |   1620.57       |     ms | +1900.01% |
|                                       90th percentile latency |                                 discovery-search-request-size-500 |     89.0998      |   1736.8         |   1647.7        |     ms | +1849.28% |
|                                      100th percentile latency |                                 discovery-search-request-size-500 |    129.419       |   1792.47        |   1663.05       |     ms | +1285.01% |
|                                  50th percentile service time |                                 discovery-search-request-size-500 |     85.2927      |   1705.86        |   1620.57       |     ms | +1900.01% |
|                                  90th percentile service time |                                 discovery-search-request-size-500 |     89.0998      |   1736.8         |   1647.7        |     ms | +1849.28% |
|                                 100th percentile service time |                                 discovery-search-request-size-500 |    129.419       |   1792.47        |   1663.05       |     ms | +1285.01% |
|                                                    error rate |                                 discovery-search-request-size-500 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                discovery-search-request-size-1000 |      6.31458     |      0.547997    |     -5.76658    |  ops/s |   -91.32% |
|                                               Mean Throughput |                                discovery-search-request-size-1000 |      6.41427     |      0.558716    |     -5.85555    |  ops/s |   -91.29% |
|                                             Median Throughput |                                discovery-search-request-size-1000 |      6.43682     |      0.560067    |     -5.87675    |  ops/s |   -91.30% |
|                                                Max Throughput |                                discovery-search-request-size-1000 |      6.47289     |      0.56214     |     -5.91075    |  ops/s |   -91.32% |
|                                       50th percentile latency |                                discovery-search-request-size-1000 |    152.803       |   1777.15        |   1624.35       |     ms | +1063.03% |
|                                       90th percentile latency |                                discovery-search-request-size-1000 |    157.539       |   1816.32        |   1658.78       |     ms | +1052.93% |
|                                      100th percentile latency |                                discovery-search-request-size-1000 |    165.043       |   1836.37        |   1671.33       |     ms | +1012.66% |
|                                  50th percentile service time |                                discovery-search-request-size-1000 |    152.803       |   1777.15        |   1624.35       |     ms | +1063.03% |
|                                  90th percentile service time |                                discovery-search-request-size-1000 |    157.539       |   1816.32        |   1658.78       |     ms | +1052.93% |
|                                 100th percentile service time |                                discovery-search-request-size-1000 |    165.043       |   1836.37        |   1671.33       |     ms | +1012.66% |
|                                                    error rate |                                discovery-search-request-size-1000 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                   discover/search |      0.00114545  |      0.00113118  |     -1e-05      |  ops/s |    -1.25% |
|                                               Mean Throughput |                                                   discover/search |      0.0027079   |      0.00264327  |     -6e-05      |  ops/s |    -2.39% |
|                                             Median Throughput |                                                   discover/search |      0.00291096  |      0.00289877  |     -1e-05      |  ops/s |    -0.42% |
|                                                Max Throughput |                                                   discover/search |      0.0038472   |      0.003833    |     -1e-05      |  ops/s |    -0.37% |
|                                       50th percentile latency |                                                   discover/search |    598.01        |    695.12        |     97.1104     |     ms |   +16.24% |
|                                       90th percentile latency |                                                   discover/search |   1080.71        |  11378.9         |  10298.2        |     ms |  +952.91% |
|                                      100th percentile latency |                                                   discover/search |   1345.69        |  12008.5         |  10662.8        |     ms |  +792.37% |
|                                  50th percentile service time |                                                   discover/search |    596.236       |    583.238       |    -12.9984     |     ms |    -2.18% |
|                                  90th percentile service time |                                                   discover/search |    754.509       |   1032.15        |    277.645      |     ms |   +36.80% |
|                                 100th percentile service time |                                                   discover/search |    824.955       |   5038.66        |   4213.71       |     ms |  +510.78% |
|                                                    error rate |                                                   discover/search |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                discover/visualize |      0.00115745  |      0.0011576   |      0          |  ops/s |    +0.01% |
|                                               Mean Throughput |                                                discover/visualize |      0.00424318  |      0.00428944  |      5e-05      |  ops/s |    +1.09% |
|                                             Median Throughput |                                                discover/visualize |      0.00415848  |      0.00437057  |      0.00021    |  ops/s |    +5.10% |
|                                                Max Throughput |                                                discover/visualize |      0.00703696  |      0.00703759  |      0          |  ops/s |     0.01% |
|                                       50th percentile latency |                                                discover/visualize |    554.132       |    420.239       |   -133.892      |     ms |   -24.16% |
|                                       90th percentile latency |                                                discover/visualize |   8744.4         |  16562.4         |   7818.05       |     ms |   +89.41% |
|                                      100th percentile latency |                                                discover/visualize |  11854.1         |  21312.9         |   9458.79       |     ms |   +79.79% |
|                                  50th percentile service time |                                                discover/visualize |    552.516       |    392.981       |   -159.535      |     ms |   -28.87% |
|                                  90th percentile service time |                                                discover/visualize |   3418.92        |    676.909       |  -2742.01       |     ms |   -80.20% |
|                                 100th percentile service time |                                                discover/visualize |  11851.1         |  21310.6         |   9459.54       |     ms |   +79.82% |
|                                                    error rate |                                                discover/visualize |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                             kafka |      0.00250886  |      0.00250886  |     -0          |  ops/s |    -0.00% |
|                                               Mean Throughput |                                                             kafka |      0.00523737  |      0.00523737  |     -0          |  ops/s |    -0.00% |
|                                             Median Throughput |                                                             kafka |      0.00510488  |      0.00510488  |     -0          |  ops/s |    -0.00% |
|                                                Max Throughput |                                                             kafka |      0.00919642  |      0.0091964   |     -0          |  ops/s |    -0.00% |
|                                       50th percentile latency |                                                             kafka |    462.89        |    497.409       |     34.519      |     ms |    +7.46% |
|                                       90th percentile latency |                                                             kafka |    672.164       |    912.82        |    240.656      |     ms |   +35.80% |
|                                      100th percentile latency |                                                             kafka |    928.598       |   1052.7         |    124.103      |     ms |   +13.36% |
|                                  50th percentile service time |                                                             kafka |    461.501       |    495.357       |     33.8554     |     ms |    +7.34% |
|                                  90th percentile service time |                                                             kafka |    670.721       |    911.082       |    240.36       |     ms |   +35.84% |
|                                 100th percentile service time |                                                             kafka |    927.607       |   1051.5         |    123.889      |     ms |   +13.36% |
|                                                    error rate |                                                             kafka |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                             nginx |      0.00322662  |      0.0032268   |      0          |  ops/s |     0.01% |
|                                               Mean Throughput |                                                             nginx |      0.0113402   |      0.0113412   |      0          |  ops/s |     0.01% |
|                                             Median Throughput |                                                             nginx |      0.0113291   |      0.0113293   |      0          |  ops/s |     0.00% |
|                                                Max Throughput |                                                             nginx |      0.0169387   |      0.0169607   |      2e-05      |  ops/s |    +0.13% |
|                                       50th percentile latency |                                                             nginx |   1755.14        |   1711.15        |    -43.9925     |     ms |    -2.51% |
|                                       90th percentile latency |                                                             nginx |   4056.26        |   2632.06        |  -1424.19       |     ms |   -35.11% |
|                                      100th percentile latency |                                                             nginx |   9099.34        |   7972.91        |  -1126.44       |     ms |   -12.38% |
|                                  50th percentile service time |                                                             nginx |   1740.53        |   1704.37        |    -36.1508     |     ms |    -2.08% |
|                                  90th percentile service time |                                                             nginx |   3747.25        |   1864.75        |  -1882.5        |     ms |   -50.24% |
|                                 100th percentile service time |                                                             nginx |   9097.34        |   7972.44        |  -1124.89       |     ms |   -12.37% |
|                                                    error rate |                                                             nginx |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                            apache |      0.0113955   |      0.0113956   |      0          |  ops/s |     0.00% |
|                                               Mean Throughput |                                                            apache |      0.0355235   |      0.0355244   |      0          |  ops/s |     0.00% |
|                                             Median Throughput |                                                            apache |      0.0392197   |      0.0392207   |      0          |  ops/s |     0.00% |
|                                                Max Throughput |                                                            apache |      0.0681175   |      0.0681224   |      0          |  ops/s |     0.01% |
|                                       50th percentile latency |                                                            apache |    383.251       |    344.776       |    -38.4749     |     ms |   -10.04% |
|                                       90th percentile latency |                                                            apache |    808.973       |    493.265       |   -315.708      |     ms |   -39.03% |
|                                      100th percentile latency |                                                            apache |   1629.82        |    630.919       |   -998.903      |     ms |   -61.29% |
|                                  50th percentile service time |                                                            apache |    380.744       |    343.27        |    -37.4738     |     ms |    -9.84% |
|                                  90th percentile service time |                                                            apache |    806.69        |    491.632       |   -315.058      |     ms |   -39.06% |
|                                 100th percentile service time |                                                            apache |   1626.81        |    628.149       |   -998.661      |     ms |   -61.39% |
|                                                    error rate |                                                            apache |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                       system/auth |      0.027945    |      0.027945    |     -0          |  ops/s |    -0.00% |
|                                               Mean Throughput |                                                       system/auth |      0.04706     |      0.0470583   |     -0          |  ops/s |    -0.00% |
|                                             Median Throughput |                                                       system/auth |      0.049052    |      0.0490519   |     -0          |  ops/s |    -0.00% |
|                                                Max Throughput |                                                       system/auth |      0.0668287   |      0.0668287   |      0          |  ops/s |     0.00% |
|                                       50th percentile latency |                                                       system/auth |    325.755       |    324.961       |     -0.79324    |     ms |    -0.24% |
|                                       90th percentile latency |                                                       system/auth |    416.861       |    416.884       |      0.02313    |     ms |     0.01% |
|                                       99th percentile latency |                                                       system/auth |   3328.97        |   4962.56        |   1633.59       |     ms |   +49.07% |
|                                      100th percentile latency |                                                       system/auth |   3360.94        |  12039.4         |   8678.44       |     ms |  +258.21% |
|                                  50th percentile service time |                                                       system/auth |    323.782       |    323.039       |     -0.74255    |     ms |    -0.23% |
|                                  90th percentile service time |                                                       system/auth |    415.677       |    414.356       |     -1.32138    |     ms |    -0.32% |
|                                  99th percentile service time |                                                       system/auth |   3307.44        |   4962.05        |   1654.61       |     ms |   +50.03% |
|                                 100th percentile service time |                                                       system/auth |   3357.34        |  12038.5         |   8681.2        |     ms |  +258.57% |
|                                                    error rate |                                                       system/auth |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                           system/syslog/dashboard |      0.0120584   |      0.0120591   |      0          |  ops/s |     0.01% |
|                                               Mean Throughput |                                           system/syslog/dashboard |      0.0257428   |      0.0256663   |     -8e-05      |  ops/s |    -0.30% |
|                                             Median Throughput |                                           system/syslog/dashboard |      0.0252098   |      0.0251866   |     -2e-05      |  ops/s |    -0.09% |
|                                                Max Throughput |                                           system/syslog/dashboard |      0.0333828   |      0.0333898   |      1e-05      |  ops/s |    +0.02% |
|                                       50th percentile latency |                                           system/syslog/dashboard |    515.775       |    324.785       |   -190.989      |     ms |   -37.03% |
|                                       90th percentile latency |                                           system/syslog/dashboard |   1082.6         |    392.479       |   -690.122      |     ms |   -63.75% |
|                                      100th percentile latency |                                           system/syslog/dashboard |   2813.66        |    537.464       |  -2276.2        |     ms |   -80.90% |
|                                  50th percentile service time |                                           system/syslog/dashboard |    500.896       |    322.151       |   -178.745      |     ms |   -35.69% |
|                                  90th percentile service time |                                           system/syslog/dashboard |   1008.65        |    371.648       |   -637.001      |     ms |   -63.15% |
|                                 100th percentile service time |                                           system/syslog/dashboard |   2812.14        |    461.675       |  -2350.47       |     ms |   -83.58% |
|                                                    error rate |                                           system/syslog/dashboard |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                system/syslog/lens |      0.0062273   |      0.00622729  |     -0          |  ops/s |    -0.00% |
|                                               Mean Throughput |                                                system/syslog/lens |      0.0107395   |      0.0107395   |     -0          |  ops/s |    -0.00% |
|                                             Median Throughput |                                                system/syslog/lens |      0.0108108   |      0.0108108   |     -0          |  ops/s |    -0.00% |
|                                                Max Throughput |                                                system/syslog/lens |      0.0144265   |      0.0144265   |     -0          |  ops/s |    -0.00% |
|                                       50th percentile latency |                                                system/syslog/lens |    852.272       |    382.708       |   -469.564      |     ms |   -55.10% |
|                                       90th percentile latency |                                                system/syslog/lens |   1552.95        |    859.707       |   -693.247      |     ms |   -44.64% |
|                                      100th percentile latency |                                                system/syslog/lens |   3048.98        |   3525.64        |    476.666      |     ms |   +15.63% |
|                                  50th percentile service time |                                                system/syslog/lens |    851.517       |    363.81        |   -487.707      |     ms |   -57.28% |
|                                  90th percentile service time |                                                system/syslog/lens |   1551.28        |    858.219       |   -693.058      |     ms |   -44.68% |
|                                 100th percentile service time |                                                system/syslog/lens |   3048.48        |   3524.45        |    475.971      |     ms |   +15.61% |
|                                                    error rate |                                                system/syslog/lens |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                   mysql/dashboard |      0.00112499  |      0.00112503  |      0          |  ops/s |     0.00% |
|                                               Mean Throughput |                                                   mysql/dashboard |      0.00505695  |      0.00505707  |      0          |  ops/s |     0.00% |
|                                             Median Throughput |                                                   mysql/dashboard |      0.00525088  |      0.00525098  |      0          |  ops/s |     0.00% |
|                                                Max Throughput |                                                   mysql/dashboard |      0.00741959  |      0.00741986  |      0          |  ops/s |     0.00% |
|                                       50th percentile latency |                                                   mysql/dashboard |    317.451       |    317.334       |     -0.11732    |     ms |    -0.04% |
|                                       90th percentile latency |                                                   mysql/dashboard |    703.364       |    787.754       |     84.3898     |     ms |   +12.00% |
|                                      100th percentile latency |                                                   mysql/dashboard |   1471.68        |   1208.73        |   -262.955      |     ms |   -17.87% |
|                                  50th percentile service time |                                                   mysql/dashboard |    316.045       |    316.134       |      0.08806    |     ms |    +0.03% |
|                                  90th percentile service time |                                                   mysql/dashboard |    605.959       |    745.507       |    139.548      |     ms |   +23.03% |
|                                 100th percentile service time |                                                   mysql/dashboard |   1469.8         |   1207.49        |   -262.306      |     ms |   -17.85% |
|                                                    error rate |                                                   mysql/dashboard |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                             redis |      0.00477853  |      0.00477851  |     -0          |  ops/s |    -0.00% |
|                                               Mean Throughput |                                                             redis |      0.014449    |      0.014574    |      0.00013    |  ops/s |    +0.87% |
|                                             Median Throughput |                                                             redis |      0.0103474   |      0.0104704   |      0.00012    |  ops/s |    +1.19% |
|                                                Max Throughput |                                                             redis |      0.0338698   |      0.0338697   |     -0          |  ops/s |    -0.00% |
|                                       50th percentile latency |                                                             redis |    525.193       |    491.823       |    -33.3691     |     ms |    -6.35% |
|                                       90th percentile latency |                                                             redis |   1635.08        |   4986.75        |   3351.67       |     ms |  +204.98% |
|                                      100th percentile latency |                                                             redis |   2708.29        |   8572           |   5863.71       |     ms |  +216.51% |
|                                  50th percentile service time |                                                             redis |    454.065       |    486.198       |     32.1332     |     ms |    +7.08% |
|                                  90th percentile service time |                                                             redis |   1627.22        |   4443.41        |   2816.19       |     ms |  +173.07% |
|                                 100th percentile service time |                                                             redis |   2706.87        |   5853.72        |   3146.85       |     ms |  +116.25% |
|                                                    error rate |                                                             redis |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                                        mysql/lens |      0.00337623  |      0.00337623  |      0          |  ops/s |     0.00% |
|                                               Mean Throughput |                                                        mysql/lens |      0.0136495   |      0.0136511   |      0          |  ops/s |    +0.01% |
|                                             Median Throughput |                                                        mysql/lens |      0.0140878   |      0.0140878   |      0          |  ops/s |     0.00% |
|                                                Max Throughput |                                                        mysql/lens |      0.0223948   |      0.0223952   |      0          |  ops/s |     0.00% |
|                                       50th percentile latency |                                                        mysql/lens |   1003.93        |    575.047       |   -428.887      |     ms |   -42.72% |
|                                       90th percentile latency |                                                        mysql/lens |   1692.7         |    960.801       |   -731.895      |     ms |   -43.24% |
|                                      100th percentile latency |                                                        mysql/lens |   1788.94        |   1004.66        |   -784.279      |     ms |   -43.84% |
|                                  50th percentile service time |                                                        mysql/lens |   1002.05        |    543.162       |   -458.893      |     ms |   -45.80% |
|                                  90th percentile service time |                                                        mysql/lens |   1504.45        |    958.735       |   -545.717      |     ms |   -36.27% |
|                                 100th percentile service time |                                                        mysql/lens |   1746.83        |   1003.09        |   -743.74       |     ms |   -42.58% |
|                                                    error rate |                                                        mysql/lens |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                               postgresql/overview |      0.00256514  |      0.00256516  |      0          |  ops/s |     0.00% |
|                                               Mean Throughput |                                               postgresql/overview |      0.0277643   |      0.0277644   |      0          |  ops/s |     0.00% |
|                                             Median Throughput |                                               postgresql/overview |      0.0294596   |      0.0294597   |      0          |  ops/s |     0.00% |
|                                                Max Throughput |                                               postgresql/overview |      0.0405392   |      0.0405394   |      0          |  ops/s |     0.00% |
|                                       50th percentile latency |                                               postgresql/overview |   1022.86        |   1018.35        |     -4.51199    |     ms |    -0.44% |
|                                       90th percentile latency |                                               postgresql/overview |   1110.55        |   1133.39        |     22.837      |     ms |    +2.06% |
|                                       99th percentile latency |                                               postgresql/overview |   1674.77        |   1673.81        |     -0.96101    |     ms |    -0.06% |
|                                      100th percentile latency |                                               postgresql/overview |   1675.02        |   1674.59        |     -0.42603    |     ms |    -0.03% |
|                                  50th percentile service time |                                               postgresql/overview |   1019.28        |   1015.45        |     -3.82996    |     ms |    -0.38% |
|                                  90th percentile service time |                                               postgresql/overview |   1051.8         |   1053.17        |      1.37008    |     ms |    +0.13% |
|                                  99th percentile service time |                                               postgresql/overview |   1673.5         |   1672.52        |     -0.97971    |     ms |    -0.06% |
|                                 100th percentile service time |                                               postgresql/overview |   1673.97        |   1672.78        |     -1.19055    |     ms |    -0.07% |
|                                                    error rate |                                               postgresql/overview |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                               postgresql/duration |      0.0110375   |      0.0110375   |      0          |  ops/s |     0.00% |
|                                               Mean Throughput |                                               postgresql/duration |      0.0198035   |      0.0198005   |     -0          |  ops/s |    -0.02% |
|                                             Median Throughput |                                               postgresql/duration |      0.0153796   |      0.0153796   |      0          |  ops/s |     0.00% |
|                                                Max Throughput |                                               postgresql/duration |      0.0377085   |      0.0377086   |      0          |  ops/s |     0.00% |
|                                       50th percentile latency |                                               postgresql/duration |    320.227       |    319.863       |     -0.36441    |     ms |    -0.11% |
|                                       90th percentile latency |                                               postgresql/duration |    566.337       |    599.18        |     32.8427     |     ms |    +5.80% |
|                                      100th percentile latency |                                               postgresql/duration |   3003.57        |   6270.41        |   3266.83       |     ms |  +108.77% |
|                                  50th percentile service time |                                               postgresql/duration |    317.929       |    317.158       |     -0.77045    |     ms |    -0.24% |
|                                  90th percentile service time |                                               postgresql/duration |    476.321       |    486.608       |     10.2866     |     ms |    +2.16% |
|                                 100th percentile service time |                                               postgresql/duration |   3001.7         |   6267.39        |   3265.68       |     ms |  +108.79% |
|                                                    error rate |                                               postgresql/duration |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                  esql_basic_count_group_0_limit_0 |     57.7457      |     56.772       |     -0.9737     |  ops/s |    -1.69% |
|                                               Mean Throughput |                                  esql_basic_count_group_0_limit_0 |     57.7457      |     56.772       |     -0.9737     |  ops/s |    -1.69% |
|                                             Median Throughput |                                  esql_basic_count_group_0_limit_0 |     57.7457      |     56.772       |     -0.9737     |  ops/s |    -1.69% |
|                                                Max Throughput |                                  esql_basic_count_group_0_limit_0 |     57.7457      |     56.772       |     -0.9737     |  ops/s |    -1.69% |
|                                       50th percentile latency |                                  esql_basic_count_group_0_limit_0 |      4.07398     |      3.71753     |     -0.35645    |     ms |    -8.75% |
|                                       90th percentile latency |                                  esql_basic_count_group_0_limit_0 |      5.02665     |      5.0243      |     -0.00235    |     ms |    -0.05% |
|                                      100th percentile latency |                                  esql_basic_count_group_0_limit_0 |      6.58989     |      6.38152     |     -0.20837    |     ms |    -3.16% |
|                                  50th percentile service time |                                  esql_basic_count_group_0_limit_0 |      4.07398     |      3.71753     |     -0.35645    |     ms |    -8.75% |
|                                  90th percentile service time |                                  esql_basic_count_group_0_limit_0 |      5.02665     |      5.0243      |     -0.00235    |     ms |    -0.05% |
|                                 100th percentile service time |                                  esql_basic_count_group_0_limit_0 |      6.58989     |      6.38152     |     -0.20837    |     ms |    -3.16% |
|                                                    error rate |                                  esql_basic_count_group_0_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                  esql_basic_count_group_1_limit_0 |    228.274       |    250.066       |     21.7918     |  ops/s |    +9.55% |
|                                               Mean Throughput |                                  esql_basic_count_group_1_limit_0 |    228.274       |    250.066       |     21.7918     |  ops/s |    +9.55% |
|                                             Median Throughput |                                  esql_basic_count_group_1_limit_0 |    228.274       |    250.066       |     21.7918     |  ops/s |    +9.55% |
|                                                Max Throughput |                                  esql_basic_count_group_1_limit_0 |    228.274       |    250.066       |     21.7918     |  ops/s |    +9.55% |
|                                       50th percentile latency |                                  esql_basic_count_group_1_limit_0 |      3.29422     |      2.98325     |     -0.31097    |     ms |    -9.44% |
|                                       90th percentile latency |                                  esql_basic_count_group_1_limit_0 |      4.02622     |      3.81812     |     -0.2081     |     ms |    -5.17% |
|                                      100th percentile latency |                                  esql_basic_count_group_1_limit_0 |      5.65914     |      4.42683     |     -1.23232    |     ms |   -21.78% |
|                                  50th percentile service time |                                  esql_basic_count_group_1_limit_0 |      3.29422     |      2.98325     |     -0.31097    |     ms |    -9.44% |
|                                  90th percentile service time |                                  esql_basic_count_group_1_limit_0 |      4.02622     |      3.81812     |     -0.2081     |     ms |    -5.17% |
|                                 100th percentile service time |                                  esql_basic_count_group_1_limit_0 |      5.65914     |      4.42683     |     -1.23232    |     ms |   -21.78% |
|                                                    error rate |                                  esql_basic_count_group_1_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                  esql_basic_count_group_2_limit_0 |    241.49        |    299.985       |     58.4943     |  ops/s |   +24.22% |
|                                               Mean Throughput |                                  esql_basic_count_group_2_limit_0 |    241.49        |    299.985       |     58.4943     |  ops/s |   +24.22% |
|                                             Median Throughput |                                  esql_basic_count_group_2_limit_0 |    241.49        |    299.985       |     58.4943     |  ops/s |   +24.22% |
|                                                Max Throughput |                                  esql_basic_count_group_2_limit_0 |    241.49        |    299.985       |     58.4943     |  ops/s |   +24.22% |
|                                       50th percentile latency |                                  esql_basic_count_group_2_limit_0 |      3.19438     |      2.61153     |     -0.58285    |     ms |   -18.25% |
|                                       90th percentile latency |                                  esql_basic_count_group_2_limit_0 |      3.57928     |      2.92365     |     -0.65563    |     ms |   -18.32% |
|                                      100th percentile latency |                                  esql_basic_count_group_2_limit_0 |      3.86213     |      3.16144     |     -0.70069    |     ms |   -18.14% |
|                                  50th percentile service time |                                  esql_basic_count_group_2_limit_0 |      3.19438     |      2.61153     |     -0.58285    |     ms |   -18.25% |
|                                  90th percentile service time |                                  esql_basic_count_group_2_limit_0 |      3.57928     |      2.92365     |     -0.65563    |     ms |   -18.32% |
|                                 100th percentile service time |                                  esql_basic_count_group_2_limit_0 |      3.86213     |      3.16144     |     -0.70069    |     ms |   -18.14% |
|                                                    error rate |                                  esql_basic_count_group_2_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                  esql_basic_count_group_3_limit_0 |    276.493       |    316.637       |     40.1438     |  ops/s |   +14.52% |
|                                               Mean Throughput |                                  esql_basic_count_group_3_limit_0 |    276.493       |    316.637       |     40.1438     |  ops/s |   +14.52% |
|                                             Median Throughput |                                  esql_basic_count_group_3_limit_0 |    276.493       |    316.637       |     40.1438     |  ops/s |   +14.52% |
|                                                Max Throughput |                                  esql_basic_count_group_3_limit_0 |    276.493       |    316.637       |     40.1438     |  ops/s |   +14.52% |
|                                       50th percentile latency |                                  esql_basic_count_group_3_limit_0 |      2.81726     |      2.53922     |     -0.27804    |     ms |    -9.87% |
|                                       90th percentile latency |                                  esql_basic_count_group_3_limit_0 |      3.06565     |      2.71586     |     -0.34979    |     ms |   -11.41% |
|                                      100th percentile latency |                                  esql_basic_count_group_3_limit_0 |      3.5246      |      3.27674     |     -0.24786    |     ms |    -7.03% |
|                                  50th percentile service time |                                  esql_basic_count_group_3_limit_0 |      2.81726     |      2.53922     |     -0.27804    |     ms |    -9.87% |
|                                  90th percentile service time |                                  esql_basic_count_group_3_limit_0 |      3.06565     |      2.71586     |     -0.34979    |     ms |   -11.41% |
|                                 100th percentile service time |                                  esql_basic_count_group_3_limit_0 |      3.5246      |      3.27674     |     -0.24786    |     ms |    -7.03% |
|                                                    error rate |                                  esql_basic_count_group_3_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                  esql_basic_count_group_4_limit_0 |    265.674       |    320.49        |     54.8161     |  ops/s |   +20.63% |
|                                               Mean Throughput |                                  esql_basic_count_group_4_limit_0 |    265.674       |    320.49        |     54.8161     |  ops/s |   +20.63% |
|                                             Median Throughput |                                  esql_basic_count_group_4_limit_0 |    265.674       |    320.49        |     54.8161     |  ops/s |   +20.63% |
|                                                Max Throughput |                                  esql_basic_count_group_4_limit_0 |    265.674       |    320.49        |     54.8161     |  ops/s |   +20.63% |
|                                       50th percentile latency |                                  esql_basic_count_group_4_limit_0 |      2.86714     |      2.4741      |     -0.39304    |     ms |   -13.71% |
|                                       90th percentile latency |                                  esql_basic_count_group_4_limit_0 |      3.18404     |      2.65641     |     -0.52764    |     ms |   -16.57% |
|                                      100th percentile latency |                                  esql_basic_count_group_4_limit_0 |      3.87663     |      3.32318     |     -0.55345    |     ms |   -14.28% |
|                                  50th percentile service time |                                  esql_basic_count_group_4_limit_0 |      2.86714     |      2.4741      |     -0.39304    |     ms |   -13.71% |
|                                  90th percentile service time |                                  esql_basic_count_group_4_limit_0 |      3.18404     |      2.65641     |     -0.52764    |     ms |   -16.57% |
|                                 100th percentile service time |                                  esql_basic_count_group_4_limit_0 |      3.87663     |      3.32318     |     -0.55345    |     ms |   -14.28% |
|                                                    error rate |                                  esql_basic_count_group_4_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |    109.373       |    128.671       |     19.2987     |  ops/s |   +17.64% |
|                                               Mean Throughput |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |    109.373       |    128.671       |     19.2987     |  ops/s |   +17.64% |
|                                             Median Throughput |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |    109.373       |    128.671       |     19.2987     |  ops/s |   +17.64% |
|                                                Max Throughput |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |    109.373       |    128.671       |     19.2987     |  ops/s |   +17.64% |
|                                       50th percentile latency |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |      5.1861      |      4.58104     |     -0.60506    |     ms |   -11.67% |
|                                       90th percentile latency |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |      6.84594     |      5.7897      |     -1.05625    |     ms |   -15.43% |
|                                      100th percentile latency |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |      8.20654     |      7.31492     |     -0.89162    |     ms |   -10.86% |
|                                  50th percentile service time |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |      5.1861      |      4.58104     |     -0.60506    |     ms |   -11.67% |
|                                  90th percentile service time |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |      6.84594     |      5.7897      |     -1.05625    |     ms |   -15.43% |
|                                 100th percentile service time |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |      8.20654     |      7.31492     |     -0.89162    |     ms |   -10.86% |
|                                                    error rate |  esql_time_range_and_date_histogram_two_groups_pre_filter_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |    204.072       |    236.165       |     32.0931     |  ops/s |   +15.73% |
|                                               Mean Throughput | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |    204.072       |    236.165       |     32.0931     |  ops/s |   +15.73% |
|                                             Median Throughput | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |    204.072       |    236.165       |     32.0931     |  ops/s |   +15.73% |
|                                                Max Throughput | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |    204.072       |    236.165       |     32.0931     |  ops/s |   +15.73% |
|                                       50th percentile latency | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |      3.92781     |      3.48068     |     -0.44712    |     ms |   -11.38% |
|                                       90th percentile latency | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |      4.56293     |      3.75421     |     -0.80872    |     ms |   -17.72% |
|                                      100th percentile latency | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |      5.28305     |      4.71267     |     -0.57039    |     ms |   -10.80% |
|                                  50th percentile service time | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |      3.92781     |      3.48068     |     -0.44712    |     ms |   -11.38% |
|                                  90th percentile service time | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |      4.56293     |      3.75421     |     -0.80872    |     ms |   -17.72% |
|                                 100th percentile service time | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |      5.28305     |      4.71267     |     -0.57039    |     ms |   -10.80% |
|                                                    error rate | esql_time_range_and_date_histogram_two_groups_post_filter_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                           esql_dissect_duration_and_stats_limit_0 |    267.087       |    300.725       |     33.638      |  ops/s |   +12.59% |
|                                               Mean Throughput |                           esql_dissect_duration_and_stats_limit_0 |    267.087       |    300.725       |     33.638      |  ops/s |   +12.59% |
|                                             Median Throughput |                           esql_dissect_duration_and_stats_limit_0 |    267.087       |    300.725       |     33.638      |  ops/s |   +12.59% |
|                                                Max Throughput |                           esql_dissect_duration_and_stats_limit_0 |    267.087       |    300.725       |     33.638      |  ops/s |   +12.59% |
|                                       50th percentile latency |                           esql_dissect_duration_and_stats_limit_0 |      2.65533     |      2.32669     |     -0.32864    |     ms |   -12.38% |
|                                       90th percentile latency |                           esql_dissect_duration_and_stats_limit_0 |      2.91161     |      2.5332      |     -0.37841    |     ms |   -13.00% |
|                                      100th percentile latency |                           esql_dissect_duration_and_stats_limit_0 |      3.18015     |      2.71578     |     -0.46437    |     ms |   -14.60% |
|                                  50th percentile service time |                           esql_dissect_duration_and_stats_limit_0 |      2.65533     |      2.32669     |     -0.32864    |     ms |   -12.38% |
|                                  90th percentile service time |                           esql_dissect_duration_and_stats_limit_0 |      2.91161     |      2.5332      |     -0.37841    |     ms |   -13.00% |
|                                 100th percentile service time |                           esql_dissect_duration_and_stats_limit_0 |      3.18015     |      2.71578     |     -0.46437    |     ms |   -14.60% |
|                                                    error rate |                           esql_dissect_duration_and_stats_limit_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                          esql_basic_count_group_0 |    120.11        |    122.9         |      2.78931    |  ops/s |    +2.32% |
|                                               Mean Throughput |                                          esql_basic_count_group_0 |    120.11        |    122.9         |      2.78931    |  ops/s |    +2.32% |
|                                             Median Throughput |                                          esql_basic_count_group_0 |    120.11        |    122.9         |      2.78931    |  ops/s |    +2.32% |
|                                                Max Throughput |                                          esql_basic_count_group_0 |    120.11        |    122.9         |      2.78931    |  ops/s |    +2.32% |
|                                       50th percentile latency |                                          esql_basic_count_group_0 |      4.86711     |      4.72233     |     -0.14478    |     ms |    -2.97% |
|                                       90th percentile latency |                                          esql_basic_count_group_0 |      5.63474     |      5.2778      |     -0.35694    |     ms |    -6.33% |
|                                      100th percentile latency |                                          esql_basic_count_group_0 |      8.61383     |      8.31659     |     -0.29724    |     ms |    -3.45% |
|                                  50th percentile service time |                                          esql_basic_count_group_0 |      4.86711     |      4.72233     |     -0.14478    |     ms |    -2.97% |
|                                  90th percentile service time |                                          esql_basic_count_group_0 |      5.63474     |      5.2778      |     -0.35694    |     ms |    -6.33% |
|                                 100th percentile service time |                                          esql_basic_count_group_0 |      8.61383     |      8.31659     |     -0.29724    |     ms |    -3.45% |
|                                                    error rate |                                          esql_basic_count_group_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                          esql_basic_count_group_1 |      0.616242    |      0.484668    |     -0.13157    |  ops/s |   -21.35% |
|                                               Mean Throughput |                                          esql_basic_count_group_1 |      0.635561    |      0.491151    |     -0.14441    |  ops/s |   -22.72% |
|                                             Median Throughput |                                          esql_basic_count_group_1 |      0.637783    |      0.491719    |     -0.14606    |  ops/s |   -22.90% |
|                                                Max Throughput |                                          esql_basic_count_group_1 |      0.642323    |      0.493928    |     -0.14839    |  ops/s |   -23.10% |
|                                       50th percentile latency |                                          esql_basic_count_group_1 |   1532.84        |   2008.35        |    475.512      |     ms |   +31.02% |
|                                       90th percentile latency |                                          esql_basic_count_group_1 |   1577.69        |   2053.33        |    475.639      |     ms |   +30.15% |
|                                      100th percentile latency |                                          esql_basic_count_group_1 |   1604.62        |   2081.52        |    476.899      |     ms |   +29.72% |
|                                  50th percentile service time |                                          esql_basic_count_group_1 |   1532.84        |   2008.35        |    475.512      |     ms |   +31.02% |
|                                  90th percentile service time |                                          esql_basic_count_group_1 |   1577.69        |   2053.33        |    475.639      |     ms |   +30.15% |
|                                 100th percentile service time |                                          esql_basic_count_group_1 |   1604.62        |   2081.52        |    476.899      |     ms |   +29.72% |
|                                                    error rate |                                          esql_basic_count_group_1 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                          esql_basic_count_group_2 |      0.249418    |      0.240325    |     -0.00909    |  ops/s |    -3.65% |
|                                               Mean Throughput |                                          esql_basic_count_group_2 |      0.272367    |      0.254957    |     -0.01741    |  ops/s |    -6.39% |
|                                             Median Throughput |                                          esql_basic_count_group_2 |      0.275376    |      0.256901    |     -0.01847    |  ops/s |    -6.71% |
|                                                Max Throughput |                                          esql_basic_count_group_2 |      0.282667    |      0.261414    |     -0.02125    |  ops/s |    -7.52% |
|                                       50th percentile latency |                                          esql_basic_count_group_2 |   3387.98        |   3718.74        |    330.768      |     ms |    +9.76% |
|                                       90th percentile latency |                                          esql_basic_count_group_2 |   3418.53        |   3746.32        |    327.791      |     ms |    +9.59% |
|                                      100th percentile latency |                                          esql_basic_count_group_2 |   3436.42        |   3779.97        |    343.545      |     ms |   +10.00% |
|                                  50th percentile service time |                                          esql_basic_count_group_2 |   3387.98        |   3718.74        |    330.768      |     ms |    +9.76% |
|                                  90th percentile service time |                                          esql_basic_count_group_2 |   3418.53        |   3746.32        |    327.791      |     ms |    +9.59% |
|                                 100th percentile service time |                                          esql_basic_count_group_2 |   3436.42        |   3779.97        |    343.545      |     ms |   +10.00% |
|                                                    error rate |                                          esql_basic_count_group_2 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                          esql_basic_count_group_3 |      0.152054    |      0.150647    |     -0.00141    |  ops/s |    -0.93% |
|                                               Mean Throughput |                                          esql_basic_count_group_3 |      0.154463    |      0.153306    |     -0.00116    |  ops/s |    -0.75% |
|                                             Median Throughput |                                          esql_basic_count_group_3 |      0.154644    |      0.153659    |     -0.00098    |  ops/s |    -0.64% |
|                                                Max Throughput |                                          esql_basic_count_group_3 |      0.155785    |      0.15499     |     -0.00079    |  ops/s |    -0.51% |
|                                       50th percentile latency |                                          esql_basic_count_group_3 |   6315.27        |   6313.34        |     -1.93066    |     ms |    -0.03% |
|                                       90th percentile latency |                                          esql_basic_count_group_3 |   6416.54        |   6454.54        |     37.9946     |     ms |    +0.59% |
|                                      100th percentile latency |                                          esql_basic_count_group_3 |   6467.79        |   6538.33        |     70.5317     |     ms |    +1.09% |
|                                  50th percentile service time |                                          esql_basic_count_group_3 |   6315.27        |   6313.34        |     -1.93066    |     ms |    -0.03% |
|                                  90th percentile service time |                                          esql_basic_count_group_3 |   6416.54        |   6454.54        |     37.9946     |     ms |    +0.59% |
|                                 100th percentile service time |                                          esql_basic_count_group_3 |   6467.79        |   6538.33        |     70.5317     |     ms |    +1.09% |
|                                                    error rate |                                          esql_basic_count_group_3 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                          esql_basic_count_group_4 |      0.05524     |      0.0564357   |      0.0012     |  ops/s |    +2.16% |
|                                               Mean Throughput |                                          esql_basic_count_group_4 |      0.0555135   |      0.0569332   |      0.00142    |  ops/s |    +2.56% |
|                                             Median Throughput |                                          esql_basic_count_group_4 |      0.0555238   |      0.056965    |      0.00144    |  ops/s |    +2.60% |
|                                                Max Throughput |                                          esql_basic_count_group_4 |      0.0557211   |      0.0572603   |      0.00154    |  ops/s |    +2.76% |
|                                       50th percentile latency |                                          esql_basic_count_group_4 |  17509           |  17255.9         |   -253.063      |     ms |    -1.45% |
|                                       90th percentile latency |                                          esql_basic_count_group_4 |  18587.6         |  18299.4         |   -288.203      |     ms |    -1.55% |
|                                      100th percentile latency |                                          esql_basic_count_group_4 |  19635.3         |  18363.6         |  -1271.7        |     ms |    -6.48% |
|                                  50th percentile service time |                                          esql_basic_count_group_4 |  17509           |  17255.9         |   -253.063      |     ms |    -1.45% |
|                                  90th percentile service time |                                          esql_basic_count_group_4 |  18587.6         |  18299.4         |   -288.203      |     ms |    -1.55% |
|                                 100th percentile service time |                                          esql_basic_count_group_4 |  19635.3         |  18363.6         |  -1271.7        |     ms |    -6.48% |
|                                                    error rate |                                          esql_basic_count_group_4 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |          esql_time_range_and_date_histogram_two_groups_pre_filter |      0.156772    |      0.141694    |     -0.01508    |  ops/s |    -9.62% |
|                                               Mean Throughput |          esql_time_range_and_date_histogram_two_groups_pre_filter |      0.159359    |      0.142801    |     -0.01656    |  ops/s |   -10.39% |
|                                             Median Throughput |          esql_time_range_and_date_histogram_two_groups_pre_filter |      0.159841    |      0.142994    |     -0.01685    |  ops/s |   -10.54% |
|                                                Max Throughput |          esql_time_range_and_date_histogram_two_groups_pre_filter |      0.160607    |      0.143328    |     -0.01728    |  ops/s |   -10.76% |
|                                       50th percentile latency |          esql_time_range_and_date_histogram_two_groups_pre_filter |   6177.89        |   6949.91        |    772.021      |     ms |   +12.50% |
|                                       90th percentile latency |          esql_time_range_and_date_histogram_two_groups_pre_filter |   6240.33        |   6977.6         |    737.271      |     ms |   +11.81% |
|                                      100th percentile latency |          esql_time_range_and_date_histogram_two_groups_pre_filter |   6470.08        |   7010.4         |    540.322      |     ms |    +8.35% |
|                                  50th percentile service time |          esql_time_range_and_date_histogram_two_groups_pre_filter |   6177.89        |   6949.91        |    772.021      |     ms |   +12.50% |
|                                  90th percentile service time |          esql_time_range_and_date_histogram_two_groups_pre_filter |   6240.33        |   6977.6         |    737.271      |     ms |   +11.81% |
|                                 100th percentile service time |          esql_time_range_and_date_histogram_two_groups_pre_filter |   6470.08        |   7010.4         |    540.322      |     ms |    +8.35% |
|                                                    error rate |          esql_time_range_and_date_histogram_two_groups_pre_filter |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |         esql_time_range_and_date_histogram_two_groups_post_filter |      0.0392349   |      0.0288656   |     -0.01037    |  ops/s |   -26.43% |
|                                               Mean Throughput |         esql_time_range_and_date_histogram_two_groups_post_filter |      0.0393953   |      0.0289043   |     -0.01049    |  ops/s |   -26.63% |
|                                             Median Throughput |         esql_time_range_and_date_histogram_two_groups_post_filter |      0.0393691   |      0.0289012   |     -0.01047    |  ops/s |   -26.59% |
|                                                Max Throughput |         esql_time_range_and_date_histogram_two_groups_post_filter |      0.0396092   |      0.0289885   |     -0.01062    |  ops/s |   -26.81% |
|                                       50th percentile latency |         esql_time_range_and_date_histogram_two_groups_post_filter |  25143.6         |  34410.5         |   9266.91       |     ms |   +36.86% |
|                                       90th percentile latency |         esql_time_range_and_date_histogram_two_groups_post_filter |  26667.1         |  35028.4         |   8361.26       |     ms |   +31.35% |
|                                      100th percentile latency |         esql_time_range_and_date_histogram_two_groups_post_filter |  26869.3         |  35523.2         |   8653.86       |     ms |   +32.21% |
|                                  50th percentile service time |         esql_time_range_and_date_histogram_two_groups_post_filter |  25143.6         |  34410.5         |   9266.91       |     ms |   +36.86% |
|                                  90th percentile service time |         esql_time_range_and_date_histogram_two_groups_post_filter |  26667.1         |  35028.4         |   8361.26       |     ms |   +31.35% |
|                                 100th percentile service time |         esql_time_range_and_date_histogram_two_groups_post_filter |  26869.3         |  35523.2         |   8653.86       |     ms |   +32.21% |
|                                                    error rate |         esql_time_range_and_date_histogram_two_groups_post_filter |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                   esql_dissect_duration_and_stats |      1.4577      |      1.44353     |     -0.01417    |  ops/s |    -0.97% |
|                                               Mean Throughput |                                   esql_dissect_duration_and_stats |      2.00616     |      1.94249     |     -0.06367    |  ops/s |    -3.17% |
|                                             Median Throughput |                                   esql_dissect_duration_and_stats |      2.08823     |      1.98806     |     -0.10017    |  ops/s |    -4.80% |
|                                                Max Throughput |                                   esql_dissect_duration_and_stats |      2.33653     |      2.24711     |     -0.08942    |  ops/s |    -3.83% |
|                                       50th percentile latency |                                   esql_dissect_duration_and_stats |    338.363       |    350.026       |     11.6632     |     ms |    +3.45% |
|                                       90th percentile latency |                                   esql_dissect_duration_and_stats |    359.042       |    371.203       |     12.1609     |     ms |    +3.39% |
|                                      100th percentile latency |                                   esql_dissect_duration_and_stats |    363.492       |    388.316       |     24.8232     |     ms |    +6.83% |
|                                  50th percentile service time |                                   esql_dissect_duration_and_stats |    338.363       |    350.026       |     11.6632     |     ms |    +3.45% |
|                                  90th percentile service time |                                   esql_dissect_duration_and_stats |    359.042       |    371.203       |     12.1609     |     ms |    +3.39% |
|                                 100th percentile service time |                                   esql_dissect_duration_and_stats |    363.492       |    388.316       |     24.8232     |     ms |    +6.83% |
|                                                    error rate |                                   esql_dissect_duration_and_stats |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                        search_basic_count_group_0 |      0.284219    |      0.29206     |      0.00784    |  ops/s |    +2.76% |
|                                               Mean Throughput |                                        search_basic_count_group_0 |      0.285653    |      0.294263    |      0.00861    |  ops/s |    +3.01% |
|                                             Median Throughput |                                        search_basic_count_group_0 |      0.285703    |      0.294342    |      0.00864    |  ops/s |    +3.02% |
|                                                Max Throughput |                                        search_basic_count_group_0 |      0.286198    |      0.295328    |      0.00913    |  ops/s |    +3.19% |
|                                       50th percentile latency |                                        search_basic_count_group_0 |   3482.44        |   3371.2         |   -111.241      |     ms |    -3.19% |
|                                       90th percentile latency |                                        search_basic_count_group_0 |   3509.18        |   3447.35        |    -61.8346     |     ms |    -1.76% |
|                                      100th percentile latency |                                        search_basic_count_group_0 |   3771.13        |   3779.1         |      7.9707     |     ms |    +0.21% |
|                                  50th percentile service time |                                        search_basic_count_group_0 |   3482.44        |   3371.2         |   -111.241      |     ms |    -3.19% |
|                                  90th percentile service time |                                        search_basic_count_group_0 |   3509.18        |   3447.35        |    -61.8346     |     ms |    -1.76% |
|                                 100th percentile service time |                                        search_basic_count_group_0 |   3771.13        |   3779.1         |      7.9707     |     ms |    +0.21% |
|                                                    error rate |                                        search_basic_count_group_0 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                        search_basic_count_group_1 |    292.621       |    294.274       |      1.65305    |  ops/s |    +0.56% |
|                                               Mean Throughput |                                        search_basic_count_group_1 |    292.621       |    294.274       |      1.65305    |  ops/s |    +0.56% |
|                                             Median Throughput |                                        search_basic_count_group_1 |    292.621       |    294.274       |      1.65305    |  ops/s |    +0.56% |
|                                                Max Throughput |                                        search_basic_count_group_1 |    292.621       |    294.274       |      1.65305    |  ops/s |    +0.56% |
|                                       50th percentile latency |                                        search_basic_count_group_1 |      2.43412     |      2.45976     |      0.02563    |     ms |    +1.05% |
|                                       90th percentile latency |                                        search_basic_count_group_1 |      2.78906     |      2.80493     |      0.01587    |     ms |    +0.57% |
|                                      100th percentile latency |                                        search_basic_count_group_1 |      3.01203     |      3.11268     |      0.10065    |     ms |    +3.34% |
|                                  50th percentile service time |                                        search_basic_count_group_1 |      2.43412     |      2.45976     |      0.02563    |     ms |    +1.05% |
|                                  90th percentile service time |                                        search_basic_count_group_1 |      2.78906     |      2.80493     |      0.01587    |     ms |    +0.57% |
|                                 100th percentile service time |                                        search_basic_count_group_1 |      3.01203     |      3.11268     |      0.10065    |     ms |    +3.34% |
|                                                    error rate |                                        search_basic_count_group_1 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                        search_basic_count_group_2 |      0.01594     |      0.0166983   |      0.00076    |  ops/s |    +4.76% |
|                                               Mean Throughput |                                        search_basic_count_group_2 |      0.0160182   |      0.0169526   |      0.00093    |  ops/s |    +5.83% |
|                                             Median Throughput |                                        search_basic_count_group_2 |      0.0160175   |      0.0169314   |      0.00091    |  ops/s |    +5.71% |
|                                                Max Throughput |                                        search_basic_count_group_2 |      0.0160846   |      0.0171817   |      0.0011     |  ops/s |    +6.82% |
|                                       50th percentile latency |                                        search_basic_count_group_2 |  61998.1         |  55763.5         |  -6234.65       |     ms |   -10.06% |
|                                      100th percentile latency |                                        search_basic_count_group_2 |  64251.1         |  61606.4         |  -2644.7        |     ms |    -4.12% |
|                                  50th percentile service time |                                        search_basic_count_group_2 |  61998.1         |  55763.5         |  -6234.65       |     ms |   -10.06% |
|                                 100th percentile service time |                                        search_basic_count_group_2 |  64251.1         |  61606.4         |  -2644.7        |     ms |    -4.12% |
|                                                    error rate |                                        search_basic_count_group_2 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                        search_basic_count_group_3 |      0.0103196   |      0.0113059   |      0.00099    |  ops/s |    +9.56% |
|                                               Mean Throughput |                                        search_basic_count_group_3 |      0.0104489   |      0.0115889   |      0.00114    |  ops/s |   +10.91% |
|                                             Median Throughput |                                        search_basic_count_group_3 |      0.0104853   |      0.0116529   |      0.00117    |  ops/s |   +11.14% |
|                                                Max Throughput |                                        search_basic_count_group_3 |      0.0105418   |      0.0118077   |      0.00127    |  ops/s |   +12.01% |
|                                       50th percentile latency |                                        search_basic_count_group_3 |  93825.9         |  81314           | -12511.9        |     ms |   -13.34% |
|                                      100th percentile latency |                                        search_basic_count_group_3 | 100987           |  85904.7         | -15082.2        |     ms |   -14.93% |
|                                  50th percentile service time |                                        search_basic_count_group_3 |  93825.9         |  81314           | -12511.9        |     ms |   -13.34% |
|                                 100th percentile service time |                                        search_basic_count_group_3 | 100987           |  85904.7         | -15082.2        |     ms |   -14.93% |
|                                                    error rate |                                        search_basic_count_group_3 |      0           |      0           |      0          |      % |     0.00% |
|                                                Min Throughput |                                        search_basic_count_group_4 |      0.00874649  |      0.00991878  |      0.00117    |  ops/s |   +13.40% |
|                                               Mean Throughput |                                        search_basic_count_group_4 |      0.00878338  |      0.00993607  |      0.00115    |  ops/s |   +13.12% |
|                                             Median Throughput |                                        search_basic_count_group_4 |      0.00877701  |      0.00993554  |      0.00116    |  ops/s |   +13.20% |
|                                                Max Throughput |                                        search_basic_count_group_4 |      0.00882665  |      0.0099539   |      0.00113    |  ops/s |   +12.77% |
|                                       50th percentile latency |                                        search_basic_count_group_4 | 111964           | 100138           | -11825.7        |     ms |   -10.56% |
|                                      100th percentile latency |                                        search_basic_count_group_4 | 113138           | 101530           | -11608.6        |     ms |   -10.26% |
|                                  50th percentile service time |                                        search_basic_count_group_4 | 111964           | 100138           | -11825.7        |     ms |   -10.56% |
|                                 100th percentile service time |                                        search_basic_count_group_4 | 113138           | 101530           | -11608.6        |     ms |   -10.26% |
|                                                    error rate |                                        search_basic_count_group_4 |      0           |      0           |      0          |      % |     0.00% |

The discovery* operations have regressed the most. Some search / esql operations have also improved.

[martijnvg]

Looks the large regression are not caused by range queries on timestamp field (TimestampQuery), but by sorting (and not using doc value skippers during sorting).

A lot of time is spent at org.apache.lucene.search.TopFieldCollector.updateMinCompetitiveScore(Scorable). Looks like we need to build a specialized field comparator that makes use of doc values. As such one doesn't exist yet. It looks like numeric comparators only utilize bkd tree if available as comparative iterator.

I suspect some specialization needs to be build for date_histogram aggregation as well, since that aggregation is also often used in the elastic/logs benchmark.