[CI] APMYamlTestSuiteIT test {yaml=/20_metrics_ingest/Test metrics-apm.app-* setting event.ingested via ingest pipeline} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #8014 / graalvm-ce17_checkpart2_java-matrix
• elasticsearch-periodic-platform-support #8011 / rhel-8_platform-support-unix
• elasticsearch-periodic-platform-support #7915 / ubuntu-2204_platform-support-unix

Reproduction Line:

./gradlew ":x-pack:plugin:apm-data:yamlRestTest" --tests "org.elasticsearch.xpack.apmdata.APMYamlTestSuiteIT" -Dtests.method="test {yaml=/20_metrics_ingest/Test metrics-apm.app-* setting event.ingested via ingest pipeline}" -Dtests.seed=BD8F3887A7A3E5C5 -Dtests.locale=ce-Cyrl-RU -Dtests.timezone=ROK -Druntime.java=17

Applicable branches:
8.17

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [/20_metrics_ingest:32]: field [hits.hits.1.fields.event\.ingested.0] is null

Issue Reasons:

• [8.17] 3 failures in test test {yaml=/20_metrics_ingest/Test metrics-apm.app-* setting event.ingested via ingest pipeline} (0.6% fail rate in 501 executions)
• [8.17] 2 failures in pipeline elasticsearch-periodic-platform-support (11.8% fail rate in 17 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 2 failures in test test {yaml=/20_metrics_ingest/Test metrics-apm.app-* setting event.ingested via ingest pipeline} (0.6% fail rate in 356 executions)

Build Scans:

• elasticsearch-intake #14623 / part2
• elasticsearch-periodic-platform-support #5374 / oraclelinux-7_platform-support-unix

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

@inge4pres I believe this is related to the recent APM work. Can you take a look at it?

[elasticsearchmachine]

This has been muted on branch 8.16

Mute Reasons:

• [8.16] 4 failures in test test {yaml=/20_metrics_ingest/Test metrics-apm.app-* setting event.ingested via ingest pipeline} (0.6% fail rate in 665 executions)
• [8.16] 2 failures in pipeline elasticsearch-periodic (9.5% fail rate in 21 executions)

Build Scans:

• elasticsearch-periodic #6086 / release-tests
• elasticsearch-pull-request #53497 / part-2
• elasticsearch-periodic #6039 / openjdk17_checkpart2_java-fips-matrix
• elasticsearch-periodic-platform-support #5973 / almalinux-9_platform-support-unix

[inge4pres]

@inge4pres I believe this is related to the recent APM work. Can you take a look at it?

Any hints on why ingest pipeline would behave differently than before?

We introduced a change in how we populate APM data in 8.16 and 8.17 with #119209 and
#119208 and we haven't seen such failures happen.

The same change is also in main, so should we expect failures also for 8.18?

The failure message is suspicious

java.lang.AssertionError: Failure at [/20_metrics_ingest:32]: field [hits.hits.1.fields.event\.ingested.0] is null

I don't see how it could be null, since we populate it with the ingest pipeline

[simitt]

Nothing in https://github.com/elastic/elasticsearch/blob/8.17/x-pack/plugin/apm-data/src/main/resources/ingest-pipelines/apm%40pipeline.yaml#L27:L36 looks wrong to me and the pipeline is called (e.g. for traces in https://github.com/elastic/elasticsearch/blob/8.17/x-pack/plugin/apm-data/src/main/resources/ingest-pipelines/traces-apm%40pipeline.yaml#L42).
@dakrone I don't think we do anything really special here, do you spot anything obvious? Or could there be cases where the pipeline is not properly loaded?

[dakrone]

It's possible, we've opened #101768 as this is something we have seen rarely before.