Skip to content

Commit e53a949

Browse files
JamesTimmsbharathkkb
JamesTimms
and
bharathkkb
authored
feat!: add gpu node autoscaling support (terraform-google-modules#807) (terraform-google-modules#944)
* feature: add gpu node autoscaling support (terraform-google-modules#807) * add gpu node autoscaling support for top level module * add gpu node autoscaling support for beta-private-cluster module * feature: add gpu node autoscaling support for all modules (terraform-google-modules#807) * add gpu node autoscaling support for all modules * feature: add gpu node autoscaling support for all modules (terraform-google-modules#807) * updater example/node_pool cluster_autoscaling var to work with gpu_resources * feature: add gpu node autoscaling support for all modules (terraform-google-modules#807) * fix example/node_pool formatting error * feature: add gpu node autoscaling support for all modules (terraform-google-modules#807) * Format gpu_resource to meet linter requirements * Update examples/node_pool/ * feature: add gpu node autoscaling support for all modules (terraform-google-modules#807) * Add v16.0 upgrade guide * Update node_pool test to specify `gpu_resources` * feature: add gpu node autoscaling support for all modules (terraform-google-modules#807) * updates upgrade guide Co-authored-by: Bharath KKB <[email protected]> Co-authored-by: Bharath KKB <[email protected]>
1 parent 433ab2f commit e53a949

File tree

27 files changed

+71
-24
lines changed

27 files changed

+71
-24
lines changed

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ Then perform the following commands on the root folder:
127127
| add\_shadow\_firewall\_rules | Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled). | `bool` | `false` | no |
128128
| basic\_auth\_password | The password to be used with Basic Authentication. | `string` | `""` | no |
129129
| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
130-
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> })</pre> | <pre>{<br> "enabled": false,<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
130+
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))<br> })</pre> | <pre>{<br> "enabled": false,<br> "gpu_resources": [],<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
131131
| cluster\_ipv4\_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | `any` | `null` | no |
132132
| cluster\_resource\_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | `map(string)` | `{}` | no |
133133
| configure\_ip\_masq | Enables the installation of ip masquerading, which is usually no longer required when using aliasied IP addresses. IP masquerading uses a kubectl call, so when you have a private cluster, you will need access to the API server. | `bool` | `false` | no |

‎autogen/main/main.tf.tmpl

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,15 +55,15 @@ locals {
5555

5656
release_channel = var.release_channel != null ? [{ channel : var.release_channel }] : []
5757

58-
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? [{
58+
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? concat([{
5959
resource_type = "cpu"
6060
minimum = var.cluster_autoscaling.min_cpu_cores
6161
maximum = var.cluster_autoscaling.max_cpu_cores
6262
}, {
6363
resource_type = "memory"
6464
minimum = var.cluster_autoscaling.min_memory_gb
6565
maximum = var.cluster_autoscaling.max_memory_gb
66-
}] : []
66+
}], var.cluster_autoscaling.gpu_resources) : []
6767

6868

6969
custom_kube_dns_config = length(keys(var.stub_domains)) > 0

‎autogen/main/variables.tf.tmpl

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,7 @@ variable "cluster_autoscaling" {
251251
max_cpu_cores = number
252252
min_memory_gb = number
253253
max_memory_gb = number
254+
gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))
254255
})
255256
default = {
256257
enabled = false
@@ -261,6 +262,7 @@ variable "cluster_autoscaling" {
261262
min_cpu_cores = 0
262263
max_memory_gb = 0
263264
min_memory_gb = 0
265+
gpu_resources = []
264266
}
265267
description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
266268
}

‎docs/upgrading_to_v16.0.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Upgrading to v16.0
2+
3+
The v16.0 release of *kubernetes-engine* is a backwards incompatible release.
4+
5+
### cluster_autoscaling modified
6+
The `cluster_autoscaling` variable has been modified to require a `gpu_resources` value. If you have enabled `cluster_autoscaling` and do not require `gpu_resources`, you can set it to an empty list as shown below.
7+
8+
```diff
9+
module "gke" {
10+
source = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
11+
- version = "~> 15.0"
12+
+ version = "~> 16.0"
13+
14+
cluster_autoscaling = {
15+
enabled = true
16+
autoscaling_profile = "BALANCED"
17+
min_cpu_cores = 1
18+
max_cpu_cores = 100
19+
min_memory_gb = 1
20+
max_memory_gb = 1000
21+
+ gpu_resources = []
22+
}
23+
}
24+
```

‎examples/node_pool/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ This example illustrates how to create a cluster with multiple custom node-pool
77

88
| Name | Description | Type | Default | Required |
99
|------|-------------|------|---------|:--------:|
10-
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
10+
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> gpu_resources = list(object({<br> resource_type = string<br> minimum = number<br> maximum = number<br> }))<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "gpu_resources": [],<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
1111
| cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no |
1212
| compute\_engine\_service\_account | Service account to associate to the nodes in the cluster | `any` | n/a | yes |
1313
| ip\_range\_pods | The secondary ip range to use for pods | `any` | n/a | yes |

‎examples/node_pool/variables.tf

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,11 @@ variable "cluster_autoscaling" {
6060
max_cpu_cores = number
6161
min_memory_gb = number
6262
max_memory_gb = number
63+
gpu_resources = list(object({
64+
resource_type = string
65+
minimum = number
66+
maximum = number
67+
}))
6368
})
6469
default = {
6570
enabled = false
@@ -68,6 +73,7 @@ variable "cluster_autoscaling" {
6873
min_cpu_cores = 0
6974
max_memory_gb = 0
7075
min_memory_gb = 0
76+
gpu_resources = []
7177
}
7278
description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
7379
}

‎main.tf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,15 +51,15 @@ locals {
5151

5252
release_channel = var.release_channel != null ? [{ channel : var.release_channel }] : []
5353

54-
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? [{
54+
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? concat([{
5555
resource_type = "cpu"
5656
minimum = var.cluster_autoscaling.min_cpu_cores
5757
maximum = var.cluster_autoscaling.max_cpu_cores
5858
}, {
5959
resource_type = "memory"
6060
minimum = var.cluster_autoscaling.min_memory_gb
6161
maximum = var.cluster_autoscaling.max_memory_gb
62-
}] : []
62+
}], var.cluster_autoscaling.gpu_resources) : []
6363

6464

6565
custom_kube_dns_config = length(keys(var.stub_domains)) > 0

‎modules/beta-private-cluster-update-variant/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ Then perform the following commands on the root folder:
162162
| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
163163
| cloudrun | (Beta) Enable CloudRun addon | `bool` | `false` | no |
164164
| cloudrun\_load\_balancer\_type | (Beta) Configure the Cloud Run load balancer type. External by default. Set to `LOAD_BALANCER_TYPE_INTERNAL` to configure as an internal load balancer. | `string` | `""` | no |
165-
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
165+
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "gpu_resources": [],<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
166166
| cluster\_ipv4\_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | `any` | `null` | no |
167167
| cluster\_resource\_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | `map(string)` | `{}` | no |
168168
| cluster\_telemetry\_type | Available options include ENABLED, DISABLED, and SYSTEM\_ONLY | `string` | `null` | no |

‎modules/beta-private-cluster-update-variant/main.tf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,15 +51,15 @@ locals {
5151

5252
release_channel = var.release_channel != null ? [{ channel : var.release_channel }] : []
5353

54-
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? [{
54+
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? concat([{
5555
resource_type = "cpu"
5656
minimum = var.cluster_autoscaling.min_cpu_cores
5757
maximum = var.cluster_autoscaling.max_cpu_cores
5858
}, {
5959
resource_type = "memory"
6060
minimum = var.cluster_autoscaling.min_memory_gb
6161
maximum = var.cluster_autoscaling.max_memory_gb
62-
}] : []
62+
}], var.cluster_autoscaling.gpu_resources) : []
6363

6464

6565
custom_kube_dns_config = length(keys(var.stub_domains)) > 0

‎modules/beta-private-cluster-update-variant/variables.tf

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@ variable "cluster_autoscaling" {
241241
max_cpu_cores = number
242242
min_memory_gb = number
243243
max_memory_gb = number
244+
gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))
244245
})
245246
default = {
246247
enabled = false
@@ -249,6 +250,7 @@ variable "cluster_autoscaling" {
249250
min_cpu_cores = 0
250251
max_memory_gb = 0
251252
min_memory_gb = 0
253+
gpu_resources = []
252254
}
253255
description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
254256
}

‎modules/beta-private-cluster/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,7 @@ Then perform the following commands on the root folder:
140140
| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
141141
| cloudrun | (Beta) Enable CloudRun addon | `bool` | `false` | no |
142142
| cloudrun\_load\_balancer\_type | (Beta) Configure the Cloud Run load balancer type. External by default. Set to `LOAD_BALANCER_TYPE_INTERNAL` to configure as an internal load balancer. | `string` | `""` | no |
143-
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
143+
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "gpu_resources": [],<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
144144
| cluster\_ipv4\_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | `any` | `null` | no |
145145
| cluster\_resource\_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | `map(string)` | `{}` | no |
146146
| cluster\_telemetry\_type | Available options include ENABLED, DISABLED, and SYSTEM\_ONLY | `string` | `null` | no |

‎modules/beta-private-cluster/main.tf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,15 +51,15 @@ locals {
5151

5252
release_channel = var.release_channel != null ? [{ channel : var.release_channel }] : []
5353

54-
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? [{
54+
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? concat([{
5555
resource_type = "cpu"
5656
minimum = var.cluster_autoscaling.min_cpu_cores
5757
maximum = var.cluster_autoscaling.max_cpu_cores
5858
}, {
5959
resource_type = "memory"
6060
minimum = var.cluster_autoscaling.min_memory_gb
6161
maximum = var.cluster_autoscaling.max_memory_gb
62-
}] : []
62+
}], var.cluster_autoscaling.gpu_resources) : []
6363

6464

6565
custom_kube_dns_config = length(keys(var.stub_domains)) > 0

‎modules/beta-private-cluster/variables.tf

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@ variable "cluster_autoscaling" {
241241
max_cpu_cores = number
242242
min_memory_gb = number
243243
max_memory_gb = number
244+
gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))
244245
})
245246
default = {
246247
enabled = false
@@ -249,6 +250,7 @@ variable "cluster_autoscaling" {
249250
min_cpu_cores = 0
250251
max_memory_gb = 0
251252
min_memory_gb = 0
253+
gpu_resources = []
252254
}
253255
description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
254256
}

‎modules/beta-public-cluster-update-variant/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ Then perform the following commands on the root folder:
156156
| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
157157
| cloudrun | (Beta) Enable CloudRun addon | `bool` | `false` | no |
158158
| cloudrun\_load\_balancer\_type | (Beta) Configure the Cloud Run load balancer type. External by default. Set to `LOAD_BALANCER_TYPE_INTERNAL` to configure as an internal load balancer. | `string` | `""` | no |
159-
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
159+
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))<br> })</pre> | <pre>{<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "gpu_resources": [],<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
160160
| cluster\_ipv4\_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | `any` | `null` | no |
161161
| cluster\_resource\_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | `map(string)` | `{}` | no |
162162
| cluster\_telemetry\_type | Available options include ENABLED, DISABLED, and SYSTEM\_ONLY | `string` | `null` | no |

‎modules/beta-public-cluster-update-variant/main.tf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,15 +51,15 @@ locals {
5151

5252
release_channel = var.release_channel != null ? [{ channel : var.release_channel }] : []
5353

54-
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? [{
54+
autoscaling_resource_limits = var.cluster_autoscaling.enabled ? concat([{
5555
resource_type = "cpu"
5656
minimum = var.cluster_autoscaling.min_cpu_cores
5757
maximum = var.cluster_autoscaling.max_cpu_cores
5858
}, {
5959
resource_type = "memory"
6060
minimum = var.cluster_autoscaling.min_memory_gb
6161
maximum = var.cluster_autoscaling.max_memory_gb
62-
}] : []
62+
}], var.cluster_autoscaling.gpu_resources) : []
6363

6464

6565
custom_kube_dns_config = length(keys(var.stub_domains)) > 0

‎modules/beta-public-cluster-update-variant/variables.tf

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@ variable "cluster_autoscaling" {
241241
max_cpu_cores = number
242242
min_memory_gb = number
243243
max_memory_gb = number
244+
gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))
244245
})
245246
default = {
246247
enabled = false
@@ -249,6 +250,7 @@ variable "cluster_autoscaling" {
249250
min_cpu_cores = 0
250251
max_memory_gb = 0
251252
min_memory_gb = 0
253+
gpu_resources = []
252254
}
253255
description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
254256
}

0 commit comments

Comments
 (0)