Skip to content

liis-dev-team/terraform-yc-kafka

BranchesTags
 
 

Repository files navigation

Yandex Cloud Managed Kafka Cluster

Features

  • Create a Managed Kafka cluster with predefined number of DB hosts
  • Create a list of users and topics with permissions
  • Create a Connectors
  • Easy to use in other resources via outputs

Kafka cluster definition

At first you need to create VPC network with three subnets!

Kafka module requires a following input variables:

  • VPC network id
  • VPC network subnets ids
  • Zones and brokers count

Notes:

  1. users variable defines a list of separate users with a permissions list, which indicates to a list of topics and grants for each of them.

Example

See examples section

Configure Terraform for Yandex Cloud

  • Install YC CLI
  • Add environment variables for terraform auth in Yandex.Cloud
export YC_TOKEN=$(yc iam create-token)
export YC_CLOUD_ID=$(yc config get cloud-id)
export YC_FOLDER_ID=$(yc config get folder-id)

Requirements

Name Version
terraform >= 1.0.0
yandex >= 0.127.0

Providers

Name Version
random 3.6.2
yandex 0.127.0

Modules

No modules.

Resources

Name Type
random_password.password resource
yandex_mdb_kafka_cluster.this resource
yandex_mdb_kafka_connector.this resource
yandex_mdb_kafka_topic.this resource
yandex_mdb_kafka_user.this resource
yandex_client_config.client data source

Inputs

Name Description Type Default Required
access_policy Access policy from other services to the MongoDB cluster. 
object({
    data_transfer = optional(bool, null)
  })
 {} no
assign_public_ip Whether to assign public IP addresses to the instances. bool true no
brokers_count The number of brokers. number 1 no
connectors A list of Kafka connectors to create. 
list(object({
    name       = string
    tasks_max  = optional(number)
    properties = optional(map(string))
    connector_config_mirrormaker = optional(object({
      topics             = optional(string)
      replication_factor = optional(number)
      source_cluster = optional(object({
        alias = optional(string)
        external_cluster = optional(object({
          bootstrap_servers = optional(string)
          sasl_username     = optional(string)
          sasl_password     = optional(string)
          sasl_mechanism    = optional(string)
          security_protocol = optional(string)
        }))
      }))
      target_cluster = optional(object({
        alias        = optional(string)
        this_cluster = optional(object({}))
        external_cluster = optional(object({
          bootstrap_servers = optional(string)
          sasl_username     = optional(string)
          sasl_password     = optional(string)
          sasl_mechanism    = optional(string)
          security_protocol = optional(string)
        }))
      }))
    }))
    connector_config_s3_sink = optional(object({
      topics                = optional(string)
      file_compression_type = optional(string)
      file_max_records      = optional(number)
      s3_connection = optional(object({
        bucket_name = optional(string)
        external_s3 = optional(object({
          endpoint          = optional(string)
          access_key_id     = optional(string)
          secret_access_key = optional(string)
        }))
      }))
    }))
  }))
 [] no
deletion_protection Inhibits deletion of the cluster. bool false no
description Kafka cluster description string "Managed Kafka cluster created by terraform module" no
disk_size The size of the disk in GB. number 32 no
disk_type_id The type of the disk. string "network-ssd" no
environment The environment for the Kafka cluster (e.g. PRESTABLE, PRODUCTION). string "PRODUCTION" no
folder_id Folder id that contains the MongoDB cluster string null no
kafka_config The configuration for the Kafka broker. 
object({
    compression_type                = optional(string)
    auto_create_topics_enable       = optional(bool)
    log_flush_interval_messages     = optional(number)
    log_flush_interval_ms           = optional(number)
    log_flush_scheduler_interval_ms = optional(number)
    log_retention_bytes             = optional(number)
    log_retention_hours             = optional(number)
    log_retention_minutes           = optional(number)
    log_retention_ms                = optional(number)
    log_segment_bytes               = optional(number)
    log_preallocate                 = optional(bool)
    num_partitions                  = optional(number)
    default_replication_factor      = optional(number)
    message_max_bytes               = optional(number)
    replica_fetch_max_bytes         = optional(number)
    ssl_cipher_suites               = optional(list(string))
    offsets_retention_minutes       = optional(number)
    socket_send_buffer_bytes        = optional(number)
    socket_receive_buffer_bytes     = optional(number)
    sasl_enabled_mechanisms         = optional(list(string))
  })
 {} no
kafka_version The Kafka version to use. string "3.5" no
labels A set of label pairs to assing to the Kafka cluster. map(any) {} no
maintenance_window (Optional) Maintenance policy of the MongoDB cluster.
- type - (Required) Type of maintenance window. Can be either ANYTIME or WEEKLY. A day and hour of window need to be specified with weekly window.
- day - (Optional) Day of the week (in DDD format). Allowed values: "MON", "TUE", "WED", "THU", "FRI", "SAT", "SUN"
- hour - (Optional) Hour of the day in UTC (in HH format). Allowed value is between 0 and 23.		 
object({
    type = string
    day  = optional(string, null)
    hour = optional(string, null)
  })
 
{
  "type": "ANYTIME"
}
 no
name The name of the Kafka cluster. string "kafka-cluster" no
network_id The ID of the VPC network where the cluster will be deployed. string n/a yes
resource_preset_id The resource preset ID. string "s3-c2-m8" no
schema_registry Whether to enable the schema registry. bool false no
security_groups_ids_list A list of security group IDs to which the MongoDB cluster belongs list(string) [] no
subnet_ids A list of subnet IDs to deploy the cluster in. list(string) n/a yes
topics A list of Kafka topics to create. 
list(object({
    name               = string
    partitions         = optional(number)
    replication_factor = optional(number)
    topic_config = optional(object({
      cleanup_policy        = optional(string)
      compression_type      = optional(string)
      delete_retention_ms   = optional(number)
      file_delete_delay_ms  = optional(number)
      flush_messages        = optional(number)
      flush_ms              = optional(number)
      min_compaction_lag_ms = optional(number)
      retention_bytes       = optional(number)
      retention_ms          = optional(number)
      max_message_bytes     = optional(number)
      min_insync_replicas   = optional(number)
      segment_bytes         = optional(number)
      preallocate           = optional(bool)
    }), {})
  }))
 [] no
users A list of Kafka users to create. 
list(object({
    name     = string
    password = optional(string)
    permissions = optional(list(object({
      topic_name  = string
      role        = string
      allow_hosts = optional(list(string), [])
    })), [])
  }))
 [] no
zones A list of availability zones. list(string) n/a yes
zookeeper_config The configuration for ZooKeeper nodes. 
object({
    resources = object({
      resource_preset_id = optional(string, "s3-c2-m8")
      disk_type_id       = optional(string, "network-ssd")
      disk_size          = optional(number, 32)
    })
  })
 
{
  "resources": {
    "disk_size": 30,
    "disk_type_id": "network-ssd",
    "resource_preset_id": "s3-c2-m8"
  }
}
 no

Outputs

Name Description
cluster_host_names_list Kafka cluster host name
cluster_id Kafka cluster ID
cluster_name Kafka cluster name
connection_step_1 1 step - Install certificate
connection_step_2 How connect to Kafka cluster?

1. Run connection string from the output value, for example

kafkacat -C <br> -b <FQDN_брокера>:9091 <br> -t <имя_топика> <br> -X security.protocol=SASL_SSL <br> -X sasl.mechanism=SCRAM-SHA-512 <br> -X sasl.username="<логин_потребителя>" <br> -X sasl.password="<пароль_потребителя>" <br> -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z -K:
topics A list of topics names.
users_data A list of users with passwords.

About

Terraform module for Yandex Cloud Managed service for Kafka

Resources

Readme

License

MIT license

Contributing

Contributing
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

5 tags

Packages

No packages published

Languages

  • HCL 100.0%