Skip to content

INK-214: log.inkless.enable should only apply to non-internal topics. #261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

gharris1727
Copy link
Contributor

This is a bug introduced in #251 that causes internal topics (such as __consumer_offsets) to be created as inkless topics when log.inkless.enable is specified. Some integration tests explicitly create these internal topics, and trigger this behavior.

@jeqo
Copy link
Contributor

jeqo commented May 1, 2025

I was troubleshooting this as well while trying to deploy this change to Aiven core: as there are many brokers there and consumer offsets topic has 3 replicas the topic never creates as Inkless topics cannot have more than 1 replica.
This is not enough to disable inkless on internal topics: it's only useful for the topic provisioning but the response and configuration stored in KRaft is still with inkless enabled because the broker default is true.

I found these two additional changes to be needed:

For the response building:

            for (String configName : configNames) {
                ConfigEntry entry = effectiveConfig.get(configName);
                String value = entry.isSensitive() ? null : entry.value();
                // Set response with inkless disabled on all internal topics
                if (Topic.isInternal(topic.name()) && configName.equals(INKLESS_ENABLE_CONFIG)) {
                    value = String.valueOf(false);
                }
                result.configs().add(new CreateTopicsResponseData.CreatableTopicConfigs().
                    setName(entry.name()).
                    setValue(value).
                    setReadOnly(entry.isReadOnly()).
                    setConfigSource(KafkaConfigSchema.translateConfigSource(entry.source()).id()).
                    setIsSensitive(entry.isSensitive()));
            }

And for the records to be stored

        records.add(new ApiMessageAndVersion(new TopicRecord().
            setName(topic.name()).
            setTopicId(topicId), (short) 0));
        // ConfigRecords go after TopicRecord but before PartitionRecord(s).
        // Disable inkless on all internal topics on the configs to be stored
        if (Topic.isInternal(topic.name())) {
            configRecords.add(new ApiMessageAndVersion(new ConfigRecord()
                .setName(INKLESS_ENABLE_CONFIG)
                .setValue("false")
                .setResourceName(topic.name())
                .setResourceType(ResourceType.TOPIC.code()), (short) 0));
        }
        records.addAll(configRecords);

Would be great to extend tests to cover these scenarios as well.

@jeqo
Copy link
Contributor

jeqo commented May 5, 2025

I created #268 to unblock aiven-core.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants