[envoyproxy][mongodb_atlas] Update failing mappings when validation based on mappings enabled

[mrodm]

Proposed commit message

Updated failing mappings found for envoyproxy and mongodb_atlas packages when elastic-package uses validation based on mappings.

envoyproxy

Added missing mappings for envoy.thread_local_cluster_manager metrics.
Added the same mappings as the other groups (mean, min, max, count and so on).

Example of error found (buildkite link):

test case failed: one or more errors found in mappings in metrics-envoyproxy.stats index template:
[0] field "envoy.thread_local_cluster_manager.worker_0_clusters_inflated.value" is undefined: field definition not found

mongodb_atlas

Update flattened field definitions to not create dynamic templates.

As they were previously, this dynamic template is created:

      "dynamic_templates": [
        {
          "mongodb_atlas.project.additional_info.*": {
            "path_match": "mongodb_atlas.project.additional_info.*",
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
      ]

With the change of this PR, this mapping is created as:

        "mongodb_atlas": {
          "properties": {
            "project": {
              "properties": {
                "additional_info": {
                  "type": "flattened"
                },
              },
            },
          },
        },

Example of errors found (buildkite link):

test case failed: one or more errors found in mappings in logs-mongodb_atlas.project index template:
[0] field "mongodb_atlas.project.additional_info._t" is undefined: field definition not found
[1] field "mongodb_atlas.project.additional_info.cid" is undefined: field definition not found
[2] field "mongodb_atlas.project.additional_info.cre" is undefined: field definition not found
[3] field "mongodb_atlas.project.additional_info.description" is undefined: field definition not found
...

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Test with elastic-package enabled mappings: https://buildkite.com/elastic/integrations/builds/21928
• Remove debug changes .buildkite folder and go.mod/go.sum files.

How to test this PR locally

Run elastic-package with mappings validation enabled and run system tests:

cd packages/<package>
elastic-package stack up -v -d --version <kibana_constraint>
export ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings
# Add defer-cleanup to check the data ingested in the corresponding indices/data streams
elastic-package test system -v --data-streams memory --defer-cleanup 900s

elastic-package stack down -v

Related issues

• Relates [system tests] Validate fields are documented with dynamic templates if needed elastic-package#2207
• Relates [system tests] Validate fields in transforms are documented based on mappings elastic-package#2341

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package envoyproxy 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
log	1290.32	1028.81	-261.51 (-20.27%)	💔

Package mongodb_atlas 👍(3) 💚(3) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
alert	3115.26	2433.09	-682.17 (-21.9%)	💔
hardware	17857.14	6666.67	-11190.47 (-62.67%)	💔

To see the full report comment with /test benchmark fullreport

[mrodm]

According to the pipeline, thread_local_cluster_manager is a special case:

integrations/packages/envoyproxy/data_stream/stats/elasticsearch/ingest_pipeline/default.yml

Lines 19 to 21 in 9576ef5

	- thread_local_cluster_manager
	- cluster_manager
	- tls_inspector

The other two special cases had definitions, but thread_local_cluster_manager does not.

integrations/packages/envoyproxy/data_stream/stats/fields/fields.yml

Lines 1023 to 1112 in 9576ef5

	- name: cluster_manager
	type: group
	fields:
	- name: '*.count'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	metric_type: counter
	description: Envoyproxy counters
	- name: '*.max'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy max timers metric
	- name: '*.mean_rate'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy mean rate timers metric
	- name: '*.mean'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy mean timers metric
	- name: '*.median'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy median timers metric
	- name: '*.min'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy min timers metric
	- name: '*.stddev'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy standard deviation timers metric
	- name: '*.value'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	metric_type: gauge
	description: Envoyproxy gauges
	- name: tls_inspector
	type: group
	fields:
	- name: '*.count'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	metric_type: counter
	description: Envoyproxy counters
	- name: '*.max'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy max timers metric
	- name: '*.mean_rate'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy mean rate timers metric
	- name: '*.mean'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy mean timers metric
	- name: '*.median'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy median timers metric
	- name: '*.min'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy min timers metric
	- name: '*.stddev'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	description: Envoyproxy standard deviation timers metric
	- name: '*.value'
	type: object
	object_type: double
	object_type_mapping_type: "*"
	metric_type: gauge
	description: Envoyproxy gauges

In this PR, I've added the definition for that group following the same approach of the other two fields. Is that ok?

Depending on the execution, there are documents ingested with these fields.

[mrodm]

Previous definition with .* was not matching the fields under additional_info. And therefore all the fields were undefined:

[0] field "mongodb_atlas.project.additional_info._t" is undefined: field definition not found
[1] field "mongodb_atlas.project.additional_info.cid" is undefined: field definition not found
...

In order to be defined as flattened, and therefore all elements under additional_info be mapped as a whole JSON object, it needs to be defined without * to avoid creating a dynamic template (see PR description).

Example of contents of mongodb_atlas.project.additional_info:

{
  "severity": "INFO",
  "remote_addr": "0.0.0.0",
  "is_mms_admin": false,
  "hidden": false,
  "description": "User was invited to project",
  "source": "USER",
  "et": "INVITED_TO_GROUP",
  "user_id": "sample_user_id",
  "_t": "USER_AUDIT",
  "cre": "2024-02-21T10:00:29Z",
  "un": "[email protected]",
  "id": "65d5c9bd2c86e3377aa5e5e4",
  "target_username": "[email protected]",
  "ut": "LOCAL",
  "cid": "646f4379c47da356740d14ad"
}

[mrodm]

Should this be a bugfix instead?

[mrodm]

To be removed the changes in .buildkite/ folder.

[harnish-crest-data]

Changes for mongodb_atlas package LGTM! 🚀

[kush-elastic]

Reviewed envoyproxy.
LGTM

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 660f4aa

History

• 💚 Build #22192 succeeded 081c29a
• 💚 Build #22186 succeeded ddcb4f4
• 💚 Build #21928 succeeded 0e8447f
• 💚 Build #21872 succeeded af611fc
• 💚 Build #21864 succeeded 8771d6c
• 💔 Build #21799 failed a6cf52953429f3145ea95227fe8fec9cce56d274

cc @mrodm

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package envoyproxy - 0.2.0 containing this change is available at https://epr.elastic.co/package/envoyproxy/0.2.0/

[elastic-vault-github-plugin-prod]

Package mongodb_atlas - 0.2.1 containing this change is available at https://epr.elastic.co/package/mongodb_atlas/0.2.1/