Skip to content

Downsampling fails if source index contains fields of type passthrough #125156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
martijnvg opened this issue Mar 18, 2025 · 1 comment · May be fixed by #127752
Open

Downsampling fails if source index contains fields of type passthrough #125156

martijnvg opened this issue Mar 18, 2025 · 1 comment · May be fixed by #127752
Assignees
Labels
>bug :StorageEngine/Downsampling Downsampling (replacement for rollups) - Turn fine-grained time-based data into coarser-grained data Team:StorageEngine

Comments

@martijnvg
Copy link
Member

martijnvg commented Mar 18, 2025

Downsampling fails if source index contains fields of type passthrough and these fields are configured with time_series_dimension to true. There are two problems that occur in this case:

  • The target index that the downsample api tries to creates is with a flattened like mapping (fields with dots) and so the type of the object field is never specified (1).
  • The downsample api detects all fields with time_series_dimension set to true as dimension fields. This isn't true as passthrough fields are just container fields.

1: This is a field mapping that the downsample tries to create:

        "metrics.process.memory.usage": {
            "type": "aggregate_metric_double",
            "metrics": [
                "max",
                "min",
                "value_count",
                "sum"
            ],
            "default_metric": "max",
            "time_series_metric": "gauge"
        },

The metrics. is an object field here and but in the mapping that this fields get merged with it is a passthrough object field.

@martijnvg martijnvg added :StorageEngine/Downsampling Downsampling (replacement for rollups) - Turn fine-grained time-based data into coarser-grained data >bug labels Mar 18, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@martijnvg martijnvg changed the title Downsampling fails if source contains fields of type passthrough Downsampling fails if source index contains fields of type passthrough Mar 18, 2025
@gmarouli gmarouli linked a pull request May 6, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :StorageEngine/Downsampling Downsampling (replacement for rollups) - Turn fine-grained time-based data into coarser-grained data Team:StorageEngine
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants