Skip to content

Fix wrong topics in metadata requests #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

lp247
Copy link

@lp247 lp247 commented Jul 8, 2025

consume is failing in the following workflow:

  1. Create topic A
  2. Consume on topic A
  3. Delete topic A
  4. Create topic B
  5. Consume on topic B

The issue is that the consumer still holds a reference to the deleted topic, and metadata requests in the #fetch method fail as metadata for this topic is not available.

This PR solves this issue by untracking all topics before retracking the target topics inside the function #performConsume in the Consumer class.

Edit: This change also disentangles metadata requests with different topics. Previously, metadata requests were deduplicated independently on the topics that were requested, potentially leading to wrong results for subsequent metadata requests. Deduplication is now done based on requested topics.

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good spot! Some changes

// For some unknown reason this is necessary, otherwise the test hangs
// I guess the previous test had the same issue
// TODO: Investigate why this is necessary
await new Promise(resolve => setTimeout(resolve, 1000))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you investigate this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already had a short look at it, but it does not seem easy to find out. I can investigate, but for now, I would keep the investigation independent of this PR if possible.

@lp247
Copy link
Author

lp247 commented Jul 9, 2025

There is one more fix for metadata requests. More on that in the description.

@lp247 lp247 changed the title Untrack deleted topics Fix wrong topics in metadata requests Jul 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants