Closed
Description
Describe the feature:
Using the Dot Expander
processor followed by field renames can easily produce empty temporary leftover fields which are difficult to spot.
For example, the Elasticsearch audit pipeline in Filebeat 7.2.0 produces the following document, where event
, origin
, and user
are leftovers from renamed fields:
"elasticsearch": {
"audit": {
"origin": {},
"origin.type": "rest",
"user.realm": "__anonymous",
"layer": "transport",
"request.id": "uGcw5v94RGKJVaT1dmjB1Q",
"indices": [],
"request.name": "RecoveryRequest",
"action": "indices:monitor/recovery",
"event": {},
"user": {},
"user.roles": [
"anonymous_user"
]
}
}
For example, this is the event
part of the pipeline:
{
"dot_expander": {
"field": "event.action",
"path": "elasticsearch.audit"
}
},
{
"rename": {
"field": "elasticsearch.audit.event.action",
"target_field": "event.action"
}
},
{
"dot_expander": {
"field": "event.type",
"path": "elasticsearch.audit"
}
},
{
"rename": {
"field": "elasticsearch.audit.event.type",
"target_field": "elasticsearch.audit.layer"
}
}
Instead of manually cleaning up, I'd suggest the implementation of a new ingest processor to clean up all empty fields at once.
The processor should support an optional path
argument to limit its scope where needed.