IngestDocument cannot find fields that contain "." in the field name

[desean1625]

Elasticsearch Version

8.15.2

Installed Plugins

No response

Java Version

bundled

OS Version

lunix

Problem Description

IngestDocument doesn't resolve field paths the same way as the rest of the system causing ingest pipelines to fail.

Steps to Reproduce

PUT /users/_doc/1?refresh=wait_for
{
  "email": "[email protected]",
  "first_name": "Mardy",
  "last_name": "Brown",
  "city": "New Orleans",
  "county": "Orleans",
  "state": "LA",
  "zip": 70116,
  "web": "mardy.asciidocsmith.com"
}
PUT /_enrich/policy/users-policy
{
  "match": {
    "indices": "users",
    "match_field": "email",
    "enrich_fields": ["first_name", "last_name", "city", "zip", "state"]
  }
}

POST /_enrich/policy/users-policy/_execute

PUT /_ingest/pipeline/user_lookup
{
  "description" : "Enriching user details to messages",
  "processors" : [
    {
      "enrich" : {
        "policy_name": "users-policy",
        "field" : "system1.email",
        "target_field": "user",
        "max_matches": "1"
      }
    }
  ]
}
PUT /my-index-00001/_doc/my_id?pipeline=user_lookup
{
  "system1.email": "[email protected]"
}

Logs (if relevant)

No response

[gbanasiak]

Thank you for your interest in Elasticsearch. Please try using dot expander processor. We mention the correct way to access flattened objects in here. I'll close this issue. If you run into problems please ask for help in a Discuss post.

[desean1625]

@gbanasiak This doesn't seem to be the best solution, because doing this causes visualizations, anomaly detection, es|ql and many other features to not work. Previous guidance from Elastic has been to flatten all fields.

elastic/kibana#1084