Skip to content

Add empty fields cleanup ingest processor #44956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
peterpramb opened this issue Jul 29, 2019 · 3 comments
Closed

Add empty fields cleanup ingest processor #44956

peterpramb opened this issue Jul 29, 2019 · 3 comments
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team

Comments

@peterpramb
Copy link

peterpramb commented Jul 29, 2019

Describe the feature:

Using the Dot Expander processor followed by field renames can easily produce empty temporary leftover fields which are difficult to spot.

For example, the Elasticsearch audit pipeline in Filebeat 7.2.0 produces the following document, where event, origin, and user are leftovers from renamed fields:

"elasticsearch": {
    "audit": {
        "origin": {},
        "origin.type": "rest",
        "user.realm": "__anonymous",
        "layer": "transport",
        "request.id": "uGcw5v94RGKJVaT1dmjB1Q",
        "indices": [],
        "request.name": "RecoveryRequest",
        "action": "indices:monitor/recovery",
        "event": {},
        "user": {},
        "user.roles": [
            "anonymous_user"
        ]
    }
}

For example, this is the event part of the pipeline:

{
    "dot_expander": {
        "field": "event.action",
        "path": "elasticsearch.audit"
    }
},
{
    "rename": {
        "field": "elasticsearch.audit.event.action",
        "target_field": "event.action"
    }
},
{
    "dot_expander": {
        "field": "event.type",
        "path": "elasticsearch.audit"
    }
},
{
    "rename": {
        "field": "elasticsearch.audit.event.type",
        "target_field": "elasticsearch.audit.layer"
    }
}

Instead of manually cleaning up, I'd suggest the implementation of a new ingest processor to clean up all empty fields at once.

The processor should support an optional path argument to limit its scope where needed.

@cbuescher cbuescher added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement labels Jul 29, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@rjernst rjernst added the Team:Data Management Meta label for data/management team label May 4, 2020
@abraxxa
Copy link

abraxxa commented Nov 14, 2023

We would also be very interessted in having such a feature out of the box!

@joegallo
Copy link
Contributor

This is a very good suggestion, and it's one we've gotten a few times -- there's a bit more traction on it under #86157, so I'm going to close this issue in favor of that one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

6 participants