Full AWS gp3 support for iops and througput config. (zalando#1261)

Jan-M · FxKu · web-flow · commit 4a88f00a3f8e · 2021-01-25T10:07:18.000+01:00
Support new AWS EBS volume type `gp3` with `iops` and `throughput` in the manifest.

Co-authored-by: Felix Kunde &lt;felix-kunde@gmx.de&gt;
diff --git a/charts/postgres-operator/crds/postgresqls.yaml b/charts/postgres-operator/crds/postgresqls.yaml
@@ -557,6 +557,8 @@ spec:
                 required:
                   - size
                 properties:
+                  iops:
+                    type: integer
                   size:
                     type: string
                     pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
@@ -565,6 +567,8 @@ spec:
                     type: string
                   subPath:
                     type: string
+                  throughput:
+                    type: integer
           status:
             type: object
             additionalProperties:
diff --git a/docs/developer.md b/docs/developer.md
@@ -235,6 +235,24 @@ Then you can for example check the Patroni logs:
 kubectl logs acid-minimal-cluster-0
 ```
 
+## Unit tests with Mocks and K8s Fake API
+
+Whenever possible you should rely on leveraging proper mocks and K8s fake client that allows full fledged testing of K8s objects in your unit tests.
+
+To enable mocks, a code annotation is needed:
+[Mock code gen annotation](https://github.com/zalando/postgres-operator/blob/master/pkg/util/volumes/volumes.go#L3)
+
+To generate mocks run:
+```bash
+make mocks
+```
+
+Examples for mocks can be found in:
+[Example mock usage](https://github.com/zalando/postgres-operator/blob/master/pkg/cluster/volumes_test.go#L248)
+
+Examples for fake K8s objects can be found in:
+[Example fake K8s client usage](https://github.com/zalando/postgres-operator/blob/master/pkg/cluster/volumes_test.go#L166)
+
 ## End-to-end tests
 
 The operator provides reference end-to-end (e2e) tests to
diff --git a/docs/reference/cluster_manifest.md b/docs/reference/cluster_manifest.md
@@ -338,13 +338,13 @@ archive is supported.
   the url to S3 bucket containing the WAL archive of the remote primary.
   Required when the `standby` section is present.
 
-## EBS volume resizing
+## Volume properties
 
 Those parameters are grouped under the `volume` top-level key and define the
 properties of the persistent storage that stores Postgres data.
 
 * **size**
-  the size of the target EBS volume. Usual Kubernetes size modifiers, i.e. `Gi`
+  the size of the target volume. Usual Kubernetes size modifiers, i.e. `Gi`
   or `Mi`, apply. Required.
 
 * **storageClass**
@@ -356,6 +356,14 @@ properties of the persistent storage that stores Postgres data.
 * **subPath**
   Subpath to use when mounting volume into Spilo container. Optional.
 
+* **iops**
+  When running the operator on AWS the latest generation of EBS volumes (`gp3`)
+  allows for configuring the number of IOPS. Maximum is 16000. Optional.
+
+* **throughput**
+  When running the operator on AWS the latest generation of EBS volumes (`gp3`)
+  allows for configuring the throughput in MB/s. Maximum is 1000. Optional.
+
 ## Sidecar definitions
 
 Those parameters are defined under the `sidecars` key. They consist of a list
diff --git a/docs/reference/operator_parameters.md b/docs/reference/operator_parameters.md
@@ -373,10 +373,13 @@ configuration they are grouped under the `kubernetes` key.
   possible value is `parallel`.
 
 * **storage_resize_mode**
-  defines how operator handels the difference between requested volume size and
-  actual size. Available options are: ebs - tries to resize EBS volume, pvc -
-  changes PVC definition, off - disables resize of the volumes. Default is "pvc".
-  When using OpenShift please use one of the other available options.
+  defines how operator handles the difference between the requested volume size and
+    the actual size. Available options are:
+    1. `ebs` : operator resizes EBS volumes directly and executes `resizefs` within a pod
+    2. `pvc` : operator only changes PVC definition
+    3. `off` : disables resize of the volumes.
+    4. `mixed` :operator  uses AWS API to adjust size, throughput, and IOPS, and calls pvc change for file system resize
+    Default is "pvc".
 
 ## Kubernetes resource requests
 
diff --git a/go.mod b/go.mod
@@ -3,7 +3,7 @@ module github.com/zalando/postgres-operator
 go 1.15
 
 require (
-	github.com/aws/aws-sdk-go v1.36.3
+	github.com/aws/aws-sdk-go v1.36.29
 	github.com/golang/mock v1.4.4
 	github.com/lib/pq v1.9.0
 	github.com/motomux/pretty v0.0.0-20161209205251-b2aad2c9a95d
diff --git a/go.sum b/go.sum
@@ -35,6 +35,8 @@ github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbt
 github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
 github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
 github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
+github.com/Storytel/gomock-matchers v1.2.0 h1:VPsbL6c/9/eCa4rH13LOEXPsIsnA1z+INamGIx1lWQo=
+github.com/Storytel/gomock-matchers v1.2.0/go.mod h1:7HEuwyU/eq/W3mrSqPSYETGXiTyU2um0Rrb+dh5KmKM=
 github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
@@ -45,8 +47,8 @@ github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5
 github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
-github.com/aws/aws-sdk-go v1.36.3 h1:KYpG5OegwW3xgOsMxy01nj/Td281yxi1Ha2lJQJs4tI=
-github.com/aws/aws-sdk-go v1.36.3/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
+github.com/aws/aws-sdk-go v1.36.29 h1:lM1G3AF1+7vzFm0n7hfH8r2+750BTo+6Lo6FtPB7kzk=
+github.com/aws/aws-sdk-go v1.36.29/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
 github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -182,6 +184,7 @@ github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/mock v1.2.1-0.20190311213431-837231f7bb37/go.mod h1:L3bP22mxdfCUHSUVMs+SPJMx55FrxQew7MSXT11Q86g=
 github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
 github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=
 github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
@@ -525,6 +528,7 @@ golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGm
 golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190125232054-d66bd3c5d5a6/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190221204921-83362c3779f5/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
 golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
diff --git a/manifests/complete-postgres-manifest.yaml b/manifests/complete-postgres-manifest.yaml
@@ -44,6 +44,8 @@ spec:
   volume:
     size: 1Gi
 #    storageClass: my-sc
+#    iops: 1000  # for EBS gp3
+ #   throughput: 250  # in MB/s for EBS gp3
   additionalVolumes:
     - name: empty
       mountPath: /opt/empty
diff --git a/manifests/postgresql.crd.yaml b/manifests/postgresql.crd.yaml
@@ -553,6 +553,8 @@ spec:
                 required:
                   - size
                 properties:
+                  iops:
+                    type: integer
                   size:
                     type: string
                     pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
@@ -561,6 +563,8 @@ spec:
                     type: string
                   subPath:
                     type: string
+                  throughput:
+                    type: integer
           status:
             type: object
             additionalProperties:
diff --git a/pkg/apis/acid.zalan.do/v1/crds.go b/pkg/apis/acid.zalan.do/v1/crds.go
@@ -835,6 +835,9 @@ var PostgresCRDResourceValidation = apiextv1.CustomResourceValidation{
 						Type:     "object",
 						Required: []string{"size"},
 						Properties: map[string]apiextv1.JSONSchemaProps{
+							"iops": {
+								Type: "integer",
+							},
 							"size": {
 								Type:        "string",
 								Description: "Value must not be zero",
@@ -846,6 +849,9 @@ var PostgresCRDResourceValidation = apiextv1.CustomResourceValidation{
 							"subPath": {
 								Type: "string",
 							},
+							"throughput": {
+								Type: "integer",
+							},
 						},
 					},
 				},
diff --git a/pkg/apis/acid.zalan.do/v1/postgresql_type.go b/pkg/apis/acid.zalan.do/v1/postgresql_type.go
@@ -118,6 +118,7 @@ type Volume struct {
 	SubPath      string `json:"subPath,omitempty"`
 	Iops         *int64 `json:"iops,omitempty"`
 	Throughput   *int64 `json:"throughput,omitempty"`
+	VolumeType   string `json:"type,omitempty"`
 }
 
 // AdditionalVolume specs additional optional volumes for statefulset
diff --git a/pkg/cluster/sync.go b/pkg/cluster/sync.go
@@ -53,41 +53,15 @@ func (c *Cluster) Sync(newSpec *acidv1.Postgresql) error {
 		return err
 	}
 
-	c.logger.Debugf("syncing volumes using %q storage resize mode", c.OpConfig.StorageResizeMode)
-
 	if c.OpConfig.EnableEBSGp3Migration {
 		err = c.executeEBSMigration()
 		if nil != err {
 			return err
 		}
 	}
 
-	if c.OpConfig.StorageResizeMode == "mixed" {
-		// mixed op uses AWS API to adjust size,throughput,iops and calls pvc chance for file system resize
-
-		// resize pvc to adjust filesystem size until better K8s support
-		if err = c.syncVolumeClaims(); err != nil {
-			err = fmt.Errorf("could not sync persistent volume claims: %v", err)
-			return err
-		}
-	} else if c.OpConfig.StorageResizeMode == "pvc" {
-		if err = c.syncVolumeClaims(); err != nil {
-			err = fmt.Errorf("could not sync persistent volume claims: %v", err)
-			return err
-		}
-	} else if c.OpConfig.StorageResizeMode == "ebs" {
-		// potentially enlarge volumes before changing the statefulset. By doing that
-		// in this order we make sure the operator is not stuck waiting for a pod that
-		// cannot start because it ran out of disk space.
-		// TODO: handle the case of the cluster that is downsized and enlarged again
-		// (there will be a volume from the old pod for which we can't act before the
-		//  the statefulset modification is concluded)
-		if err = c.syncVolumes(); err != nil {
-			err = fmt.Errorf("could not sync persistent volumes: %v", err)
-			return err
-		}
-	} else {
-		c.logger.Infof("Storage resize is disabled (storage_resize_mode is off). Skipping volume sync.")
+	if err = c.syncVolumes(); err != nil {
+		return err
 	}
 
 	if err = c.enforceMinResourceLimits(&c.Spec); err != nil {
@@ -590,48 +564,6 @@ func (c *Cluster) syncRoles() (err error) {
 	return nil
 }
 
-// syncVolumeClaims reads all persistent volume claims and checks that their size matches the one declared in the statefulset.
-func (c *Cluster) syncVolumeClaims() error {
-	c.setProcessName("syncing volume claims")
-
-	act, err := c.volumeClaimsNeedResizing(c.Spec.Volume)
-	if err != nil {
-		return fmt.Errorf("could not compare size of the volume claims: %v", err)
-	}
-	if !act {
-		c.logger.Infof("volume claims do not require changes")
-		return nil
-	}
-	if err := c.resizeVolumeClaims(c.Spec.Volume); err != nil {
-		return fmt.Errorf("could not sync volume claims: %v", err)
-	}
-
-	c.logger.Infof("volume claims have been synced successfully")
-
-	return nil
-}
-
-// syncVolumes reads all persistent volumes and checks that their size matches the one declared in the statefulset.
-func (c *Cluster) syncVolumes() error {
-	c.setProcessName("syncing volumes")
-
-	act, err := c.volumesNeedResizing(c.Spec.Volume)
-	if err != nil {
-		return fmt.Errorf("could not compare size of the volumes: %v", err)
-	}
-	if !act {
-		return nil
-	}
-
-	if err := c.resizeVolumes(); err != nil {
-		return fmt.Errorf("could not sync volumes: %v", err)
-	}
-
-	c.logger.Infof("volumes have been synced successfully")
-
-	return nil
-}
-
 func (c *Cluster) syncDatabases() error {
 	c.setProcessName("syncing databases")
 
diff --git a/pkg/cluster/volumes.go b/pkg/cluster/volumes.go
diff --git a/pkg/cluster/volumes_test.go b/pkg/cluster/volumes_test.go
diff --git a/pkg/util/volumes/ebs.go b/pkg/util/volumes/ebs.go
diff --git a/pkg/util/volumes/volumes.go b/pkg/util/volumes/volumes.go

Original file line number	Diff line number	Diff line change
`@@ -118,6 +118,7 @@ type Volume struct {`
`118`	`118`	SubPath string `json:"subPath,omitempty"`
`119`	`119`	Iops *int64 `json:"iops,omitempty"`
`120`	`120`	Throughput *int64 `json:"throughput,omitempty"`
	`121`	+ VolumeType string `json:"type,omitempty"`
`121`	`122`	`}`
`122`	`123`
`123`	`124`	`// AdditionalVolume specs additional optional volumes for statefulset`