Skip to content

Add empty fields cleanup ingest processor #44956

Closed
@peterpramb

Description

@peterpramb

Describe the feature:

Using the Dot Expander processor followed by field renames can easily produce empty temporary leftover fields which are difficult to spot.

For example, the Elasticsearch audit pipeline in Filebeat 7.2.0 produces the following document, where event, origin, and user are leftovers from renamed fields:

"elasticsearch": {
    "audit": {
        "origin": {},
        "origin.type": "rest",
        "user.realm": "__anonymous",
        "layer": "transport",
        "request.id": "uGcw5v94RGKJVaT1dmjB1Q",
        "indices": [],
        "request.name": "RecoveryRequest",
        "action": "indices:monitor/recovery",
        "event": {},
        "user": {},
        "user.roles": [
            "anonymous_user"
        ]
    }
}

For example, this is the event part of the pipeline:

{
    "dot_expander": {
        "field": "event.action",
        "path": "elasticsearch.audit"
    }
},
{
    "rename": {
        "field": "elasticsearch.audit.event.action",
        "target_field": "event.action"
    }
},
{
    "dot_expander": {
        "field": "event.type",
        "path": "elasticsearch.audit"
    }
},
{
    "rename": {
        "field": "elasticsearch.audit.event.type",
        "target_field": "elasticsearch.audit.layer"
    }
}

Instead of manually cleaning up, I'd suggest the implementation of a new ingest processor to clean up all empty fields at once.

The processor should support an optional path argument to limit its scope where needed.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions