Skip to content

Initial skeleton #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 17 commits into
base: main
Choose a base branch
from
Draft

Initial skeleton #1

wants to merge 17 commits into from

Conversation

salvacorts
Copy link
Collaborator

@salvacorts salvacorts commented Dec 2, 2024

What this PR does / why we need it:

This PR adds structured metadata (SM) stats to the TSDB index. For each stream we save a list of available structured metadata field names. We then use these stats to:

  • Return a list of available SM fields for the /api/labels endpoint.
  • On the query path, skip all series not having a given SM field we are filtering by.

On ingesters, for each stream we keep:

  • A closed set of stats: has the stats up to the most recent closed chunk.
  • An open set of stats: these are updated as we receive push requests (closed set + newer data).
    On the flush loop, the Ingester updates the TSDB with the closed set of stats.

We store these stats next to the label set of each stream on TSDB. As for regular labels, SM names are referenced using the symbols table.
image

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@salvacorts
Copy link
Collaborator Author

Reopening. I tried to close it to make the PR against the main Loki repo but couldn't make it work.

@salvacorts salvacorts reopened this Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants