Allow configuring pod priority globally and per cluster. (zalando#353)

* Allow configuring pod priority globally and per cluster.

Allow to specify pod priority class for all pods managed by the operator,
as well as for those belonging to individual clusters.

Controlled by the pod_priority_class_name operator configuration
parameter and the podPriorityClassName manifest option.

See https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
for the explanation on how to define priority classes since Kubernetes 1.8.

Some import order changes are due to go fmt.
Removal of OrphanDependents deprecated field.

Code review by @zerg-junior

Author: alexeyklyukin

docs/reference/cluster_manifest.md

@@ -89,7 +89,14 @@ Those are parameters grouped directly under  the `spec` key in the manifest.
   examples](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
   for details on tolerations and possible values of those keys. When set, this
   value overrides the `pod_toleration` setting from the operator. Optional.
-
+  
+* **podPriorityClassName**
+   a name of the [priority
+   class](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass)
+   that should be assigned to the cluster pods. When not specified, the value
+   is taken from the `pod_priority_class_name` operator parameter, if not set
+   then the default priority class is taken. The priority class itself must be defined in advance.
+   
 ## Postgres parameters
 
 Those parameters are grouped under the `postgresql` top-level key.

docs/reference/operator_parameters.md

@@ -116,10 +116,15 @@ configuration they are grouped under the `kubernetes` key.
   option. If not defined, a simple definition that contains only the name will be used. The default is empty.
 
 * **pod_service_account_role_binding_definition**
-  This definition must bind pod service account to a role with permission sufficient for the pods to start and for Patroni to access k8s endpoints; service account on its own lacks any such rights starting with k8s v1.8. If not excplicitly defined by the user, a simple definition that binds the account to the operator's own 'zalando-postgres-operator' cluster role will be used. The default is empty.
+  This definition must bind pod service account to a role with permission
+  sufficient for the pods to start and for Patroni to access k8s endpoints;
+  service account on its own lacks any such rights starting with k8s v1.8. If
+  not excplicitly defined by the user, a simple definition that binds the
+  account to the operator's own 'zalando-postgres-operator' cluster role will
+  be used. The default is empty.
 
 * **pod_terminate_grace_period**
-  Patroni pods are [terminated
+  Postgres pods are [terminated
   forcefully](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods)
   after this timeout. The default is `5m`.
 
@@ -151,7 +156,7 @@ configuration they are grouped under the `kubernetes` key.
   name of the secret containing infrastructure roles names and passwords.
 
 * **pod_role_label**
-  name of the label assigned to the postgres pods (and services/endpoints) by
+  name of the label assigned to the Postgres pods (and services/endpoints) by
   the operator. The default is `spilo-role`.
 
 * **cluster_labels**
@@ -168,7 +173,7 @@ configuration they are grouped under the `kubernetes` key.
   considered `ready`. The operator uses values of those labels to detect the
   start of the Kubernetes cluster upgrade procedure and move master pods off
   the nodes to be decommissioned. When the set is not empty, the operator also
-  assigns the `Affinity` clause to the postgres pods to be scheduled only on
+  assigns the `Affinity` clause to the Postgres pods to be scheduled only on
   `ready` nodes. The default is empty.
 
 * **toleration**
@@ -184,6 +189,13 @@ configuration they are grouped under the `kubernetes` key.
   All variables from that ConfigMap are injected to the pod's environment, on
   conflicts they are overridden by the environment variables generated by the
   operator. The default is empty.
+  
+* **pod_priority_class_name**
+  a name of the [priority
+  class](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass)
+  that should be assigned to the Postgres pods. The priority class itself must be defined in advance.
+  Default is empty (use the default priority class).
+  
 
 ## Kubernetes resource requests
 
@@ -231,8 +243,8 @@ CRD-based configuration.
   possible issues faster. The default is `10m`.
 
 * **pod_deletion_wait_timeout**
-  timeout when waiting for the pods to be deleted when removing the cluster or
-  recreating pods. The default is `10m`.
+  timeout when waiting for the Postgres pods to be deleted when removing the
+  cluster or recreating pods. The default is `10m`.
 
 * **ready_wait_interval**
   the interval between consecutive attempts waiting for the postgres CRD to be
@@ -285,18 +297,19 @@ either. In the CRD-based configuration those options are grouped under the
 
 * **wal_s3_bucket**
   S3 bucket to use for shipping WAL segments with WAL-E. A bucket has to be
-  present and accessible by Patroni managed pods. At the moment, supported
-  services by Spilo are S3 and GCS. The default is empty.
+  present and accessible by Postgres pods. At the moment, supported services by
+  Spilo are S3 and GCS. The default is empty.
 
 * **log_s3_bucket**
   S3 bucket to use for shipping postgres daily logs. Works only with S3 on AWS.
-  The bucket has to be present and accessible by Patroni managed pods. At the
-  moment Spilo does not yet support this. The default is empty.
+  The bucket has to be present and accessible by Postgres pods. At the moment
+  Spilo does not yet support this. The default is empty.
 
 * **kube_iam_role**
-  AWS IAM role to supply in the `iam.amazonaws.com/role` annotation of Patroni
+  AWS IAM role to supply in the `iam.amazonaws.com/role` annotation of Postgres
   pods. Only used when combined with
-  [kube2iam](https://github.com/jtblin/kube2iam) project on AWS. The default is empty.
+  [kube2iam](https://github.com/jtblin/kube2iam) project on AWS. The default is
+  empty.
 
 * **aws_region**
   AWS region used to store ESB volumes. The default is `eu-central-1`.

pkg/cluster/cluster.go

@@ -12,11 +12,11 @@ import (
 	"time"
 
 	"github.com/Sirupsen/logrus"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
-	"k8s.io/api/core/v1"
 	"k8s.io/api/apps/v1beta1"
+	"k8s.io/api/core/v1"
 	policybeta1 "k8s.io/api/policy/v1beta1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/rest"
 	"k8s.io/client-go/tools/cache"
 
@@ -91,7 +91,7 @@ type compareStatefulsetResult struct {
 
 // New creates a new cluster. This function should be called from a controller.
 func New(cfg Config, kubeClient k8sutil.KubernetesClient, pgSpec spec.Postgresql, logger *logrus.Entry) *Cluster {
-	orphanDependents := true
+	deletePropagationPolicy := metav1.DeletePropagationOrphan
 
 	podEventsQueue := cache.NewFIFO(func(obj interface{}) (string, error) {
 		e, ok := obj.(spec.PodEvent)
@@ -113,7 +113,7 @@ func New(cfg Config, kubeClient k8sutil.KubernetesClient, pgSpec spec.Postgresql
 			Services:  make(map[PostgresRole]*v1.Service),
 			Endpoints: make(map[PostgresRole]*v1.Endpoints)},
 		userSyncStrategy: users.DefaultUserSyncStrategy{},
-		deleteOptions:    &metav1.DeleteOptions{OrphanDependents: &orphanDependents},
+		deleteOptions:    &metav1.DeleteOptions{PropagationPolicy: &deletePropagationPolicy},
 		podEventsQueue:   podEventsQueue,
 		KubeClient:       kubeClient,
 	}
@@ -601,7 +601,7 @@ func (c *Cluster) Delete() {
 	}
 
 	for _, obj := range c.Secrets {
-		if delete, user := c.shouldDeleteSecret(obj); !delete {
+		if doDelete, user := c.shouldDeleteSecret(obj); !doDelete {
 			c.logger.Warningf("not removing secret %q for the system user %q", obj.GetName(), user)
 			continue
 		}
@@ -951,11 +951,11 @@ func (c *Cluster) deletePatroniClusterEndpoints() error {
 		return util.NameFromMeta(ep.ObjectMeta), err
 	}
 
-	delete := func(name string) error {
+	deleteEndpointFn := func(name string) error {
 		return c.KubeClient.Endpoints(c.Namespace).Delete(name, c.deleteOptions)
 	}
 
-	return c.deleteClusterObject(get, delete, "endpoint")
+	return c.deleteClusterObject(get, deleteEndpointFn, "endpoint")
 }
 
 func (c *Cluster) deletePatroniClusterConfigMaps() error {
@@ -964,9 +964,9 @@ func (c *Cluster) deletePatroniClusterConfigMaps() error {
 		return util.NameFromMeta(cm.ObjectMeta), err
 	}
 
-	delete := func(name string) error {
+	deleteConfigMapFn := func(name string) error {
 		return c.KubeClient.ConfigMaps(c.Namespace).Delete(name, c.deleteOptions)
 	}
 
-	return c.deleteClusterObject(get, delete, "configmap")
+	return c.deleteClusterObject(get, deleteConfigMapFn, "configmap")
 }

pkg/cluster/exec.go

@@ -5,9 +5,9 @@ import (
 	"fmt"
 	"strings"
 
+	"k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/client-go/kubernetes/scheme"
-	"k8s.io/api/core/v1"
 	"k8s.io/client-go/tools/remotecommand"
 
 	"github.com/zalando-incubator/postgres-operator/pkg/spec"
@@ -59,9 +59,9 @@ func (c *Cluster) ExecCommand(podName *spec.NamespacedName, command ...string) (
 	}
 
 	err = exec.Stream(remotecommand.StreamOptions{
-		Stdout:             &execOut,
-		Stderr:             &execErr,
-		Tty:                false,
+		Stdout: &execOut,
+		Stderr: &execErr,
+		Tty:    false,
 	})
 
 	if err != nil {

pkg/cluster/k8sres.go

@@ -6,6 +6,7 @@ import (
 	"sort"
 
 	"github.com/Sirupsen/logrus"
+
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
@@ -15,6 +16,7 @@ import (
 	policybeta1 "k8s.io/api/policy/v1beta1"
 
 	"github.com/zalando-incubator/postgres-operator/pkg/spec"
+	"github.com/zalando-incubator/postgres-operator/pkg/util"
 	"github.com/zalando-incubator/postgres-operator/pkg/util/constants"
 	"k8s.io/apimachinery/pkg/labels"
 )
@@ -399,6 +401,7 @@ func generatePodTemplate(
 	terminateGracePeriod int64,
 	podServiceAccountName string,
 	kubeIAMRole string,
+	priorityClassName string,
 ) (*v1.PodTemplateSpec, error) {
 
 	terminateGracePeriodSeconds := terminateGracePeriod
@@ -416,6 +419,10 @@ func generatePodTemplate(
 		podSpec.Affinity = nodeAffinity
 	}
 
+	if priorityClassName != "" {
+		podSpec.PriorityClassName = priorityClassName
+	}
+
 	template := v1.PodTemplateSpec{
 		ObjectMeta: metav1.ObjectMeta{
 			Labels:    labels,
@@ -662,7 +669,7 @@ func (c *Cluster) generateStatefulSet(spec *spec.PostgresSpec) (*v1beta1.Statefu
 		c.containerName(), c.logger)
 
 	// pickup the docker image for the spilo container
-	effectiveDockerImage := getEffectiveDockerImage(c.OpConfig.DockerImage, spec.DockerImage)
+	effectiveDockerImage := util.Coalesce(spec.DockerImage, c.OpConfig.DockerImage)
 
 	volumeMounts := generateVolumeMounts()
 
@@ -696,6 +703,7 @@ func (c *Cluster) generateStatefulSet(spec *spec.PostgresSpec) (*v1beta1.Statefu
 	}
 
 	tolerationSpec := tolerations(&spec.Tolerations, c.OpConfig.PodToleration)
+	effectivePodPriorityClassName := util.Coalesce(spec.PodPriorityClassName, c.OpConfig.PodPriorityClassName)
 
 	// generate pod template for the statefulset, based on the spilo container and sidecards
 	if podTemplate, err = generatePodTemplate(
@@ -707,8 +715,13 @@ func (c *Cluster) generateStatefulSet(spec *spec.PostgresSpec) (*v1beta1.Statefu
 		nodeAffinity(c.OpConfig.NodeReadinessLabel),
 		int64(c.OpConfig.PodTerminateGracePeriod.Seconds()),
 		c.OpConfig.PodServiceAccountName,
-		c.OpConfig.KubeIAMRole); err != nil {
-			return nil, fmt.Errorf("could not generate pod template: %v", err)
+		c.OpConfig.KubeIAMRole,
+		effectivePodPriorityClassName); err != nil{
+		return nil, fmt.Errorf("could not generate pod template: %v", err)
+	}
+
+	if err != nil {
+		return nil, fmt.Errorf("could not generate pod template: %v", err)
 	}
 
 	if volumeClaimTemplate, err = generatePersistentVolumeClaimTemplate(spec.Volume.Size,
@@ -737,13 +750,6 @@ func (c *Cluster) generateStatefulSet(spec *spec.PostgresSpec) (*v1beta1.Statefu
 	return statefulSet, nil
 }
 
-func getEffectiveDockerImage(globalDockerImage, clusterDockerImage string) string {
-	if clusterDockerImage == "" {
-		return globalDockerImage
-	}
-	return clusterDockerImage
-}
-
 func generateScalyrSidecarSpec(clusterName, APIKey, serverURL, dockerImage string,
 	containerResources *spec.Resources, logger *logrus.Entry) *spec.Sidecar {
 	if APIKey == "" || dockerImage == "" {

pkg/cluster/pod.go

@@ -4,8 +4,8 @@ import (
 	"fmt"
 	"math/rand"
 
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	"github.com/zalando-incubator/postgres-operator/pkg/spec"
 	"github.com/zalando-incubator/postgres-operator/pkg/util"

pkg/cluster/resources.go

@@ -5,11 +5,11 @@ import (
 	"strconv"
 	"strings"
 
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
-	"k8s.io/api/core/v1"
 	"k8s.io/api/apps/v1beta1"
+	"k8s.io/api/core/v1"
 	policybeta1 "k8s.io/api/policy/v1beta1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
 
 	"github.com/zalando-incubator/postgres-operator/pkg/util"
 	"github.com/zalando-incubator/postgres-operator/pkg/util/constants"
@@ -272,10 +272,10 @@ func (c *Cluster) replaceStatefulSet(newStatefulSet *v1beta1.StatefulSet) error
 	c.logger.Debugf("replacing statefulset")
 
 	// Delete the current statefulset without deleting the pods
-	orphanDepencies := true
+	deletePropagationPolicy := metav1.DeletePropagationOrphan
 	oldStatefulset := c.Statefulset
 
-	options := metav1.DeleteOptions{OrphanDependents: &orphanDepencies}
+	options := metav1.DeleteOptions{PropagationPolicy: &deletePropagationPolicy}
 	if err := c.KubeClient.StatefulSets(oldStatefulset.Namespace).Delete(oldStatefulset.Name, &options); err != nil {
 		return fmt.Errorf("could not delete statefulset %q: %v", statefulSetName, err)
 	}

pkg/cluster/sync.go

@@ -6,7 +6,6 @@ import (
 
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	policybeta1 "k8s.io/api/policy/v1beta1"
-	"k8s.io/api/policy/v1beta1"
 	"k8s.io/api/core/v1"
 
 	"github.com/zalando-incubator/postgres-operator/pkg/spec"
@@ -188,7 +187,7 @@ func (c *Cluster) syncEndpoint(role PostgresRole) error {
 
 func (c *Cluster) syncPodDisruptionBudget(isUpdate bool) error {
 	var (
-		pdb   *v1beta1.PodDisruptionBudget
+		pdb   *policybeta1.PodDisruptionBudget
 		err   error
 	)
 	if pdb, err = c.KubeClient.PodDisruptionBudgets(c.Namespace).Get(c.podDisruptionBudgetName(), metav1.GetOptions{}); err == nil {

pkg/cluster/util.go

@@ -11,11 +11,11 @@ import (
 	"strings"
 	"time"
 
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/labels"
-	"k8s.io/api/core/v1"
 	"k8s.io/api/apps/v1beta1"
+	"k8s.io/api/core/v1"
 	policybeta1 "k8s.io/api/policy/v1beta1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
 
 	"github.com/zalando-incubator/postgres-operator/pkg/spec"
 	"github.com/zalando-incubator/postgres-operator/pkg/util"

pkg/cluster/volumes.go

@@ -5,9 +5,9 @@ import (
 	"strconv"
 	"strings"
 
+	"k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/api/core/v1"
 
 	"github.com/zalando-incubator/postgres-operator/pkg/spec"
 	"github.com/zalando-incubator/postgres-operator/pkg/util"