Go operator with sdk fails to start with set in config.json failed: permission denied #4813

planetf1 · 2021-04-20T15:13:17Z

Bug Report

What did you do?

Created a clean operator project using the 1.6.1 operator sdk (macOS)

operator-sdk init --domain egeria-project.org --license apache2 --owner 'Contributors to the Egeria project' --project-name 'egeria' --repo 'github.com/odpi/egeria-k8s-operator'
operator-sdk create api --group egeria --version v1alpha1 --kind EgeriaPlatform

then built...

checked the operator

$ kubectl get deployment -n egeria-system                                             [16:11:17]
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
egeria-controller-manager   0/1     1            0           2m18s
jonesn:test3/ $ kubectl get pods -n egeria-system                                                   [16:12:08]
NAME                                         READY   STATUS                 RESTARTS   AGE
egeria-controller-manager-55bb64c6c9-pxc4w   1/2     CreateContainerError   0          2m32s
jonesn:test3/ $ kubectl describe pod egeria-controller-manager-55bb64c6c9-pxc4w -n egeria-system    [16:12:23]
Name:         egeria-controller-manager-55bb64c6c9-pxc4w
Namespace:    egeria-system
Priority:     0
Node:         10.242.128.21/10.242.128.21
Start Time:   Tue, 20 Apr 2021 16:09:50 +0100
Labels:       control-plane=controller-manager
              pod-template-hash=55bb64c6c9
Annotations:  cni.projectcalico.org/podIP: 172.17.120.212/32
              cni.projectcalico.org/podIPs: 172.17.120.212/32
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "ips": [
                        "172.17.120.212"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "",
                    "ips": [
                        "172.17.120.212"
                    ],
                    "default": true,
                    "dns": {}
                }]
              openshift.io/scc: restricted
Status:       Pending
IP:           172.17.120.212
IPs:
  IP:           172.17.120.212
Controlled By:  ReplicaSet/egeria-controller-manager-55bb64c6c9
Containers:
  kube-rbac-proxy:
    Container ID:
    Image:         gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
    Image ID:
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from egeria-controller-manager-token-wnzkb (ro)
  manager:
    Container ID:  cri-o://2559d37e22c8cc9e80362e0462c259327e3986ef50ccd5ba15c83d16a9b13778
    Image:         odpi/egeria-k8s-operator:0.1.7
    Image ID:      docker.io/odpi/egeria-k8s-operator@sha256:48da92aacf0ed0be8e5ad64843f1d0c722587d59b1ac76821971d3f41be8ca8a
    Port:          <none>
    Host Port:     <none>
    Command:
      /manager
    Args:
      --health-probe-bind-address=:8081
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
    State:          Running
      Started:      Tue, 20 Apr 2021 16:09:57 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Liveness:     http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:    http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from egeria-controller-manager-token-wnzkb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  egeria-controller-manager-token-wnzkb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  egeria-controller-manager-token-wnzkb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       2m38s                 default-scheduler  Successfully assigned egeria-system/egeria-controller-manager-55bb64c6c9-pxc4w to 10.242.128.21
  Normal   AddedInterface  2m36s                 multus             Add eth0 [172.17.120.212/32]
  Warning  Failed          2m36s                 kubelet            Error: container create failed: time="2021-04-20T10:09:52-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Normal   Pulling         2m36s                 kubelet            Pulling image "odpi/egeria-k8s-operator:0.1.7"
  Normal   Started         2m31s                 kubelet            Started container manager
  Normal   Pulled          2m31s                 kubelet            Successfully pulled image "odpi/egeria-k8s-operator:0.1.7" in 4.935205588s
  Normal   Created         2m31s                 kubelet            Created container manager
  Warning  Failed          2m30s                 kubelet            Error: container create failed: time="2021-04-20T10:09:58-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          2m29s                 kubelet            Error: container create failed: time="2021-04-20T10:09:59-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          2m16s                 kubelet            Error: container create failed: time="2021-04-20T10:10:12-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          2m2s                  kubelet            Error: container create failed: time="2021-04-20T10:10:26-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          109s                  kubelet            Error: container create failed: time="2021-04-20T10:10:39-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          94s                   kubelet            Error: container create failed: time="2021-04-20T10:10:54-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          80s                   kubelet            Error: container create failed: time="2021-04-20T10:11:08-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          68s                   kubelet            Error: container create failed: time="2021-04-20T10:11:20-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          55s                   kubelet            (combined from similar events): Error: container create failed: time="2021-04-20T10:11:33-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Normal   Pulled          40s (x11 over 2m36s)  kubelet            Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0" already present on machine
jonesn:test3/ $

What did you expect to see?

Operator starts up correctly

What did you see instead? Under which circumstances?

The operator pod fails to run, reporting:

  Normal   Scheduled       4m17s                 default-scheduler  Successfully assigned egeria-system/egeria-controller-manager-5676cd584d-zqfw2 to 10.242.128.21
  Normal   AddedInterface  4m15s                 multus             Add eth0 [172.17.120.211/32]
  Normal   Pulling         4m15s                 kubelet            Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"
  Warning  Failed          4m10s                 kubelet            Error: container create failed: time="2021-04-20T09:41:02-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Normal   Pulling         4m10s                 kubelet            Pulling image "odpi/egeria-k8s-operator:0.1.6"
  Normal   Pulled          4m10s                 kubelet            Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0" in 4.28458144s
  Normal   Pulled          4m5s                  kubelet            Successfully pulled image "odpi/egeria-k8s-operator:0.1.6" in 5.052850106s
  Normal   Created         4m5s                  kubelet            Created container manager
  Normal   Started         4m5s                  kubelet            Started container manager
  Warning  Failed          4m4s                  kubelet            Error: container create failed: time="2021-04-20T09:41:08-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
  Warning  Failed          4m3s                  kubelet            Error: container create failed: time="2021-04-20T09:41:09-05:00" level=error msg="container_linux.go:366: starting container process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"

Environment

Operator type:

< /language go >

Kubernetes cluster type:

OpenShift
ROKS/IBM Cloud
4.6.22

operator-sdk version: "v1.6.1", commit: "e6981d812a759442a583d8ee2fae269507c408d4", kubernetes version: "1.19.4", go version: "go1.15.5", GOOS: "darwin", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.15.11 darwin/amd64

$ kubectl version

$ kubectl version [15:57:55]
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0+d46d32f", GitCommit:"d46d32fbc3f0c35d2855a4589062aaf6335acd48", GitTreeState:"clean", BuildDate:"2021-03-11T01:16:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.21) and server (1.19) exceeds the supported minor version skew of +/-1

(aware of skew)

Possible Solution

Some additional security context or kubeproxy changes needed?

Additional context

This is very similar to #4684 - though that related primarily to an ansible based operator (didn't want to confuse the discussion there)

It would appear the operator sdk is not setting up sufficient permissions to work in current openshift versions?

The text was updated successfully, but these errors were encountered:

planetf1 · 2021-04-20T15:27:48Z

Checking what has been built

manager.yaml :

      securityContext:
       runAsNonRoot: true

        securityContext:
          allowPrivilegeEscalation: false

planetf1 · 2021-04-20T16:08:44Z

This might be related -- it's likely the version of OpenShift being used in roks hasn't yet been fixed? https://bugzilla.redhat.com/show_bug.cgi?id=1874057

estroz · 2021-04-20T23:40:34Z

This is the exact same root cause as #4684. The current recommendation is to use the kube-rbac-proxy image built for OpenShift.

/triage duplicate

planetf1 · 2021-04-21T08:59:18Z

Thanks. I wasn't entirely clear, but having updated the image for kube-proxy-rbac I can configure the operator now works. thanks

estroz · 2021-05-10T18:05:12Z

Closing due to duplicate

planetf1 changed the title ~~Go operator with sdk~~ Go operator with sdk fails to start with set in config.json failed: permission denied Apr 20, 2021

openshift-ci-robot added the triage/duplicate Indicates an issue is a duplicate of other open issue. label Apr 20, 2021

estroz closed this as completed May 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Go operator with sdk fails to start with set in config.json failed: permission denied #4813

Go operator with sdk fails to start with set in config.json failed: permission denied #4813

planetf1 commented Apr 20, 2021

planetf1 commented Apr 20, 2021

Uh oh!

planetf1 commented Apr 20, 2021

Uh oh!

estroz commented Apr 20, 2021

Uh oh!

planetf1 commented Apr 21, 2021

Uh oh!

estroz commented May 10, 2021

Uh oh!

Go operator with sdk fails to start with set in config.json failed: permission denied #4813

Go operator with sdk fails to start with set in config.json failed: permission denied #4813

Comments

planetf1 commented Apr 20, 2021

Bug Report

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

Possible Solution

Additional context

planetf1 commented Apr 20, 2021

Uh oh!

planetf1 commented Apr 20, 2021

Uh oh!

estroz commented Apr 20, 2021

Uh oh!

planetf1 commented Apr 21, 2021

Uh oh!

estroz commented May 10, 2021

Uh oh!