-
Notifications
You must be signed in to change notification settings - Fork 25.4k
Closed
Labels
:Data Management/Ingest NodeExecution or management of Ingest Pipelines including GeoIPExecution or management of Ingest Pipelines including GeoIP>enhancementTeam:Data ManagementMeta label for data/management teamMeta label for data/management team
Description
Describe the feature:
Using the Dot Expander
processor followed by field renames can easily produce empty temporary leftover fields which are difficult to spot.
For example, the Elasticsearch audit pipeline in Filebeat 7.2.0 produces the following document, where event
, origin
, and user
are leftovers from renamed fields:
"elasticsearch": {
"audit": {
"origin": {},
"origin.type": "rest",
"user.realm": "__anonymous",
"layer": "transport",
"request.id": "uGcw5v94RGKJVaT1dmjB1Q",
"indices": [],
"request.name": "RecoveryRequest",
"action": "indices:monitor/recovery",
"event": {},
"user": {},
"user.roles": [
"anonymous_user"
]
}
}
For example, this is the event
part of the pipeline:
{
"dot_expander": {
"field": "event.action",
"path": "elasticsearch.audit"
}
},
{
"rename": {
"field": "elasticsearch.audit.event.action",
"target_field": "event.action"
}
},
{
"dot_expander": {
"field": "event.type",
"path": "elasticsearch.audit"
}
},
{
"rename": {
"field": "elasticsearch.audit.event.type",
"target_field": "elasticsearch.audit.layer"
}
}
Instead of manually cleaning up, I'd suggest the implementation of a new ingest processor to clean up all empty fields at once.
The processor should support an optional path
argument to limit its scope where needed.
Metadata
Metadata
Assignees
Labels
:Data Management/Ingest NodeExecution or management of Ingest Pipelines including GeoIPExecution or management of Ingest Pipelines including GeoIP>enhancementTeam:Data ManagementMeta label for data/management teamMeta label for data/management team