fix: topic config should be a map

[kccox]

Description

Change the config property of topics from object({}) to map(string). This is needed to correctly copy the config properties into the topic declaration.

Release required?

• No release
• Patch release (x.x.X)
• Minor release (x.X.x)
• Major release (X.x.x)

Release notes content

The config property for a topic is a set of key-value pairs. Keys and values are both strings. The keys are not legal terraform attributes, e.g. cleanup.policy and retention.bytes (the keys are defined by Kafka and we can't change them). This means the config should be declared as a map(string) so the values are passed to the ibm_event_streams_topic resource.

Run the pipeline

If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.

Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:

/run pipeline

Checklist for reviewers

• If relevant, a test for the change is included or updated with this PR.
• If relevant, documentation for the change is included or updated with this PR.

For mergers

• Use a conventional commit message to set the release level. Follow the guidelines.
• Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
• Use the Squash and merge option.

[kccox]

Additional notes:

• The missing config properties were reported by an internal user.
• I ran terraform apply to check that the topics are created with the right properties, see below.

[kccox]

This worked as expected. I used the main module with tfvars

plan = "enterprise-3nodes-2tb"

topics = [
    {
      name       = "topic-1"
      partitions = 1
      config = {
        "cleanup.policy"  = "delete"
        "retention.ms"    = "86400000"
        "retention.bytes" = "10485760"
        "segment.bytes"   = "10485760"
      }
    },
    {
      name       = "topic-2"
      partitions = 3
      config = {
        "cleanup.policy"  = "compact,delete"
        "retention.ms"    = "43200000"
        "retention.bytes" = "20971520"
        "segment.bytes"   = "20971520"
      }
    }
  ]

terraform plan output:

  # ibm_event_streams_topic.es_topic[0] will be created
  + resource "ibm_event_streams_topic" "es_topic" {
      + config               = {
          + "cleanup.policy"  = "delete"
          + "retention.bytes" = "10485760"
          + "retention.ms"    = "86400000"
          + "segment.bytes"   = "10485760"
        }
      + id                   = (known after apply)
      + kafka_brokers_sasl   = (known after apply)
      + kafka_http_url       = (known after apply)
      + name                 = "topic-1"
      + partitions           = 1
      + resource_instance_id = (known after apply)
    }

  # ibm_event_streams_topic.es_topic[1] will be created
  + resource "ibm_event_streams_topic" "es_topic" {
      + config               = {
          + "cleanup.policy"  = "compact,delete"
          + "retention.bytes" = "20971520"
          + "retention.ms"    = "43200000"
          + "segment.bytes"   = "20971520"
        }
      + id                   = (known after apply)
      + kafka_brokers_sasl   = (known after apply)
      + kafka_http_url       = (known after apply)
      + name                 = "topic-2"
      + partitions           = 3
      + resource_instance_id = (known after apply)
    }

terraform apply output:

ibm_resource_instance.es_instance: Creation complete after 1h48m28s [id=crn:v1:staging:public:messagehub:us-south:a/6db1b0d0b5c54ee5c201552547febcd8:13925a5b-966b-4d97-9739-6a2f41545eec::]
ibm_event_streams_topic.es_topic[0]: Creating...
ibm_event_streams_topic.es_topic[1]: Creating...
ibm_event_streams_topic.es_topic[0]: Creation complete after 4s [id=crn:v1:staging:public:messagehub:us-south:a/6db1b0d0b5c54ee5c201552547febcd8:13925a5b-966b-4d97-9739-6a2f41545eec:topic:topic-1]
ibm_event_streams_topic.es_topic[1]: Creation complete after 4s [id=crn:v1:staging:public:messagehub:us-south:a/6db1b0d0b5c54ee5c201552547febcd8:13925a5b-966b-4d97-9739-6a2f41545eec:topic:topic-2]

Then using the ibmcloud CLI:

% ibmcloud es topic topic-1
Details for topic topic-1
Topic name   Internal?   Partition count   Replication factor
topic-1      false       1                 3

Partition details for topic topic-1
Partition ID   Leader   Replicas   In-sync
0              1        [1 2 0]    [1 2 0]

Configuration parameters for topic topic-1
Name                   Value
cleanup.policy         delete
min.insync.replicas    2
segment.bytes          10485760
retention.ms           86400000
segment.ms             604800000
retention.bytes        10485760
message.audit.enable   false
segment.index.bytes    10485760
OK


% ibmcloud es topic topic-2
Details for topic topic-2
Topic name   Internal?   Partition count   Replication factor
topic-2      false       3                 3

Partition details for topic topic-2
Partition ID   Leader   Replicas   In-sync
0              2        [2 0 1]    [2 0 1]
1              0        [0 1 2]    [0 1 2]
2              1        [1 2 0]    [1 2 0]

Configuration parameters for topic topic-2
Name                   Value
cleanup.policy         compact,delete
min.insync.replicas    2
segment.bytes          20971520
retention.ms           43200000
segment.ms             604800000
retention.bytes        20971520
message.audit.enable   false
segment.index.bytes    10485760
OK

The topics exist and the partitions and config parameters match those provided, or are defaults for parameters not in the config.

[kccox]

/run pipeline

[jor2]

upgrade test failure

[ocofaigh]

It would seem that by converting from object({}) to map(string) it is forcing an update in place on the config. @kccox I assume this is non disruptive, and we should be safe to skip the upgrade test?

[kccox]

Yes, I think it's safe to skip the upgrade test. What appears to be happening is:

PREVIOUS:

• Topic is created without the values for the config, so the topic has the default config values
• Topic is upgraded with the config values, but because this was object({}) none of those were used, and the topic config is unchanged

WITH FIX:

• Topic is created without the values for the config and gets default values
• Topic is upgraded, and now because it's map(string) the properties are used, and the topic config changes

[ocofaigh]

/run pipeline

[terraform-ibm-modules-ops]

🎉 This PR is included in version 2.10.1 🎉

The release is available on:

• GitHub release
• v2.10.1

Your semantic-release bot 📦🚀