Terraform module for deploying Materialize on Azure with all required infrastructure components.

This module sets up:

• AKS cluster for Materialize workloads
• Azure Database for PostgreSQL Flexible Server for metadata storage
• Azure Blob Storage for persistence
• Required networking and security configurations
• Managed identities with proper RBAC permissions

The module has been tested with:

• AKS version 1.28
• PostgreSQL 15
• terraform-helm-materialize v0.1.12 (Materialize Operator v25.1.7)

This module requires active Azure credentials in your environment, either set up through environment variables containing the required keys or by logging in with the Azure CLI using:

az login

You also need to set an Azure subscription ID in the subscription_id variable or set the ARM_SUBSCRIPTION_ID environment variable, eg:

export ARM_SUBSCRIPTION_ID="your-subscription-id"

Additionally, this module runs a Python script to generate Azure SAS tokens for the storage account. This requires Python 3.12 or greater.

Before running the module, ensure you have the necessary Python dependencies installed:

1. Install Python 3.12+ if you haven't already.

2. Install the required dependencies using pip:

   pip install -r requirements.txt

   Or alternatively, you can install the dependencies manually:

   pip install azure-identity azure-storage-blob azure-keyvault-secrets azure-mgmt-storage

If you are using a virtual environment, you can set it up as follows:

python -m venv venv
source venv/bin/activate  # On macOS/Linux
venv\Scripts\activate  # On Windows
pip install -r requirements.txt

This will install the required Python packages in a virtual environment.

This module requires an existing Azure Resource Group. You can either:

1. Create one with Terraform before running this module:

   resource "azurerm_resource_group" "materialize" {
     name     = var.resource_group_name
     location = var.location
   }

   Then set the resource_group_name variable in your terraform.tfvars file:

   resource_group_name = "your-desired-rg-name"

2. Use an existing one by just setting the name in your terraform.tfvars file:

   resource_group_name = "your-existing-rg"

The materialize_instances variable is a list of objects that define the configuration for each Materialize instance.

Optional list of additional command-line arguments to pass to the environmentd container. This can be used to override default system parameters or enable specific features.

environmentd_extra_args = [
  "--system-parameter-default=max_clusters=1000",
  "--system-parameter-default=max_connections=1000",
  "--system-parameter-default=max_tables=1000",
]

These flags configure default limits for clusters, connections, and tables. You can provide any supported arguments here.

No providers.

No resources.

object({
    sku_name         = optional(string, "GP_Standard_D2s_v3")
    postgres_version = optional(string, "15")
    password         = string
    username         = optional(string, "materialize")
    db_name          = optional(string, "materialize")
  })

list(object({
    name                              = string
    namespace                         = optional(string)
    database_name                     = string
    environmentd_version              = optional(string)
    cpu_request                       = optional(string, "1")
    memory_request                    = optional(string, "1Gi")
    memory_limit                      = optional(string, "1Gi")
    create_database                   = optional(bool, true)
    create_load_balancer              = optional(bool, true)
    internal_load_balancer            = optional(bool, true)
    in_place_rollout                  = optional(bool, false)
    request_rollout                   = optional(string)
    force_rollout                     = optional(string)
    balancer_memory_request           = optional(string, "256Mi")
    balancer_memory_limit             = optional(string, "256Mi")
    balancer_cpu_request              = optional(string, "100m")
    license_key                       = optional(string)
    authenticator_kind                = optional(string, "None")
    external_login_password_mz_system = optional(string, null)
    environmentd_extra_args           = optional(list(string), [])
  }))

object({
    vnet_address_space   = string
    subnet_cidr          = string
    postgres_subnet_cidr = string
    service_cidr         = string
    docker_bridge_cidr   = string
  })

The AKS cluster can be accessed using the kubectl command-line tool. To authenticate with the cluster, run the following command:

az aks get-credentials --resource-group $(terraform output -raw resource_group_name) --name $(terraform output -json aks_cluster | jq -r '.name')

This command retrieves the AKS cluster credentials and merges them into the ~/.kube/config file. You can now interact with the AKS cluster using kubectl.

By default, two LoadBalancer Services are created for each Materialize instance:

1. One for balancerd, listening on:
   1. Port 6875 for SQL connections to the database.
   2. Port 6876 for HTTP(S) connections to the database.
2. One for the web console, listening on:
   1. Port 8080 for HTTP(S) connections.

The IP addresses of these load balancers will be in the terraform output as load_balancer_details.

TLS support is provided by using cert-manager and a self-signed ClusterIssuer.

More advanced TLS support using user-provided CAs or per-Materialize Issuers are out of scope for this Terraform module. Please refer to the cert-manager documentation for detailed guidance on more advanced usage.

You must upgrade to at least v0.7.x before upgrading to v0.8.x of this terraform code.

Breaking changes:

• The system node pool is renamed and significantly modified, forcing a recreation.
• Openebs is removed, and with it all support for lgalloc, our legacy spill to disk mechanism.

This is an intermediate version to handle some changes that must be applied in stages. It is recommended to upgrade to v0.8.x after upgrading to this version.

Breaking changes:

• Swap is enabled by default.
• Support for lgalloc, our legacy spill to disk mechanism, is deprecated, and will be removed in the next version.
• We now always use two node pools, one for system workloads and one for Materialize workloads.
  • Variables for configuring these node pools have been renamed, so they may be configured separately.

To avoid downtime when upgrading to future versions, you must perform a rollout at this version.

1. Ensure your environmentd_version is at least v26.0.0.
2. Update your request_rollout (and force_rollout if already at the correct environmentd_version).
3. Run terraform apply.

You must upgrade to at least v0.6.x before upgrading to v0.7.0 of this terraform code.

It is strongly recommended to have enabled swap on v0.6.x before upgrading to v0.7.0 or higher.

We recommend upgrading to at least v0.5.10 before upgrading to v0.6.x of this terraform code.

To use swap:

1. Set swap_enabled to true.
2. Ensure your environmentd_version is at least v26.0.0.
3. Update your request_rollout (and force_rollout if already at the correct environmentd_version).
4. Run terraform apply.

This will create a new node group configured for swap, and migrate your clusterd pods there.

This version is missing the updated helm chart. Skip this version, go to v0.6.1.

We now install cert-manager and configure a self-signed ClusterIssuer by default.

Due to limitations in Terraform, it cannot plan Kubernetes resources using CRDs that do not exist yet. We have worked around this for new users by only generating the certificate resources when creating Materialize instances that use them, which also cannot be created on the first run.

For existing users upgrading Materialize instances not previously configured for TLS:

1. Leave install_cert_manager at its default of true.
2. Set use_self_signed_cluster_issuer to false.
3. Run terraform apply. This will install cert-manager and its CRDs.
4. Set use_self_signed_cluster_issuer back to true (the default).
5. Update the request_rollout field of the Materialize instance.
6. Run terraform apply. This will generate the certificates and configure your Materialize instance to use them.