KAFKA-19070:: Adding task number to user provided client id to ensure each consumer has a unique client ID to avoid metric registration conflicts. #19341
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…each consumer has a unique client ID to avoid metric registration conflicts.
|
@C0urante can you please review this. |
|
@dajac can you please review this. |
|
Hi @mimaison can you please review this. Thanks. |
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
This PR updates the behavior of client.id assignment when a user provides a custom value via override configs in Kafka Connect.
Why
Currently, if a user overrides the client.id, all tasks or consumers inherit the same client ID. While this doesn't cause an immediate failure in Kafka, it leads to the following issues:
Metrics and logs are merged or overwritten, making observability inaccurate.
Quotas and throttling may be applied incorrectly.
Debugging becomes harder due to lack of per-task identity.
According to Kafka core behavior, client.id should be unique per client instance for proper tracking and diagnostics.
How
This PR appends the task number to the user-provided client.id to ensure uniqueness across tasks.
For example:
User provides: client.id=my-custom-client
Final client.id used by task 2: my-custom-client-2
This approach:
Respects the user’s original intent in naming
Guarantees unique client.id values per task
Improves metrics, logging, and debugging consistency