SA ownreference isn't updated during the operator upgrade

[horis233]

Bug Report

What did you do?
A clear and concise description of the steps you took (or insert a code snippet).

We upgrade operators by switching the catalog source image.

What did you expect to see?
A clear and concise description of what you expected to happen (or insert a code snippet).

We expect all the operators are upgraded successfully to the succeeded status.

What did you see instead? Under which circumstances?
A clear and concise description of what you expected to happen (or insert a code snippet).

Several new operators are in pending status. These operators can't be upgraded to the later version.

In the CSV status, it shows service account is owned by another ClusterServiceVersion

      name: ibmlicensings.operator.ibm.com
      status: Present
      uuid: 5db2e0ca-29fb-49df-977a-6343129a271b
      version: v1
    - group: ''
      kind: ServiceAccount
      message: Service account is owned by another ClusterServiceVersion
      name: ibm-licensing-operator
      status: PresentNotSatisfied
      version: v1
    - group: ''
      kind: ServiceAccount
      message: Service account is owned by another ClusterServiceVersion
      name: ibm-license-service
      status: PresentNotSatisfied
      version: v1

The install plan of the operator has been completed, but the service account still references the old CSV.

Environment

• operator-lifecycle-manager version:

0.17.0

• Kubernetes version information:

  name: kube-apiserver
  version: 1.20.0-beta.2
  

• Kubernetes cluster kind:

OCP 4.7.13

Possible Solution

Delete and reinstall these operators

Additional context
Add any other context about the problem here.

I am wondering if it is because the owner reference of the service account isn't updated during the update in OLM 0.17.0
https://github.com/operator-framework/operator-lifecycle-manager/blob/v0.17.0/pkg/controller/operators/catalog/step_ensurer.go#L138

I believe this PR is a fix to solve the issue: #2076

@benluddy Could you help to confirm that?

Could we get which OCP versions have this fix?

cc @pgodowski

[njhale]

The referenced patch has been included in OCP version 4.7.11, according to the errata.

Since you're experiencing this on 4.7.13, I think we may still have an issue. Will attempt to replicate on master (under the operator-framework org).

[benluddy]

The v0.17.0 you're seeing is the upstream OLM version at the time OCP v4.7 diverged from upstream -- you can determine the commit digest using oc adm release info --commits 4.7.13, which includes the change from the linked PR (

operator-lifecycle-manager/pkg/controller/operators/catalog/step_ensurer.go

Line 159 in df08e59

sa.OwnerReferences = mergedOwnerReferences(preSa.OwnerReferences, sa.OwnerReferences)

).

Since you observed this on OpenShift, I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1974438 to track this downstream as well. Can you provide precise steps to reproduce there, please?

[horis233]

Since we don't know exactly about the root cause and this issue can't be reproduced on all the operators, we don't know how to reproduce the issue stably yet. However, we can see this issue happens very often. I am looking for some hits from code to see if I know the cause.

Also, I can provide some logs and YAML files I observed in the cluster to help us troubleshoot this issue.

I found a common symptom in the operators which hit this issue. There is a serviceaccount that appears twice in the installplan.

    - resolving: operand-deployment-lifecycle-manager.v1.6.0
      resource:
        group: ''
        kind: ServiceAccount
        manifest: >-
          {"kind":"ConfigMap","name":"efed617aec34db68cb4c70a7315f92d51827a3d77f8ffc53b1e751f002b2f7c","namespace":"openshift-marketplace","catalogSourceName":"opencloud-operators","catalogSourceNamespace":"openshift-marketplace","replaces":"operand-deployment-lifecycle-manager.v1.5.1","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandBindInfo\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandConfig\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandRegistry\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandRequest\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-odlm\",\"version\":\"1.6.0\"}}]}"}
        name: operand-deployment-lifecycle-manager
        sourceName: opencloud-operators
        sourceNamespace: openshift-marketplace
        version: v1
....
    - resolving: operand-deployment-lifecycle-manager.v1.6.0
      resource:
        group: ''
        kind: ServiceAccount
        manifest: >-
          {"kind":"ConfigMap","name":"efed617aec34db68cb4c70a7315f92d51827a3d77f8ffc53b1e751f002b2f7c","namespace":"openshift-marketplace","catalogSourceName":"opencloud-operators","catalogSourceNamespace":"openshift-marketplace","replaces":"operand-deployment-lifecycle-manager.v1.5.1","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandBindInfo\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandConfig\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandRegistry\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"OperandRequest\",\"version\":\"v1alpha1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-odlm\",\"version\":\"1.6.0\"}}]}"}
        name: operand-deployment-lifecycle-manager
        sourceName: opencloud-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Present

Operator looks like this in the openshift console:

This happens when we upgrade an operator called operand-deployment-lifecycle-manager

I added these yaml files in it: previous CSV (1.5.1), upcoming CSV (1.6.0), installplan and subscription.
operand-deployment-lifecycle-manager.zip

[benluddy]

Thanks for sharing that additional information. It looks like the bundle itself contains a ServiceAccount manifest with the same name as the ServiceAccount that is generated automatically as part of the ClusterServiceVersion spec:

operator-lifecycle-manager/pkg/controller/registry/resolver/rbac.go

Line 65 in b262c0b

func RBACForClusterServiceVersion(csv *v1alpha1.ClusterServiceVersion) (map[string]*OperatorPermissions, error) {

The generated ServiceAccount manifest includes an ownerReference to its ClusterServiceVersion, but the one in the bundle does not. The catalog operator uses the first manifest for an InstallPlan step with a given GVK and Name, which might explain why this doesn't seem to reproduce 100% of the time:

operator-lifecycle-manager/pkg/controller/operators/catalog/manifests.go

Lines 51 to 67 in b262c0b

	usteps, err := r.unpackedStepsForBundle(step.Resolving, ref)
	if err != nil {
	return "", err
	}
	log.Debugf("checking cache for unpacked step")
	// need to find the real manifest from the unpacked steps
	for _, u := range usteps {
	if u.Name == step.Resource.Name &&
	u.Kind == step.Resource.Kind &&
	u.Version == step.Resource.Version &&
	u.Group == step.Resource.Group {
	manifest = u.Manifest
	log.WithField("manifest", manifest).Debug("step replaced with unpacked value")
	break
	}
	}

[horis233]

@benluddy

Thanks for the information. Yes indeed, these operators have a service account file in their operator bundles. I believe the service account file was generated by a specific version of operator-sdk (like 1.3?).

I will remove these service account files to work around this issue.

[kevinrizza]

This pr introduces validation to ensure that bundles can't be built this way:

operator-framework/api#144

[dinhxuanvu]

This issue has been resolved so we will close this. Thanks.