Upgrading K8s Cluster Failed

[ahmadamirahmadi1401]

CloudStack 4.19.1.3
My upgrade from K8S Version 1.31.8 into 1.32.4 was unsuccessful.

2025-06-03 07:42:57,510 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done

[weizhouapache]

are there more logs ?

[ahmadamirahmadi1401]

@weizhouapache
These have been repeated over and over again.

2025-06-03 07:42:57,510 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
unpacking k8s.gcr.io/sig-storage/csi-attacher:v3.0.2 (sha256:6f80b12657a7e0a5c683b24e806c4bbbe33a43e39b041fe9b7514d665d478ea4)...done
unpacking registry.k8s.io/sig-storage/csi-attacher:v4.6.1 (sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94)...done
unpacking k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1 (sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108)...done
unpacking registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1 (sha256:f25af73ee708ff9c82595ae99493cdef9295bd96953366cddf36305f82555dac)...done
unpacking k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4 (sha256:bec571992d40203edcd056ac0b0d97003887ee5e4be144c41932d18639673b03)...done
unpacking registry.k8s.io/sig-storage/csi-provisioner:v5.0.1 (sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385)...done
unpacking registry.k8s.io/sig-storage/csi-resizer:v1.11.1 (sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0 (sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.16-0 (sha256:c6a9d11cc5c04b114ccdef39a9265eeef818e3d02f5359be035ae784097fdec5)...done
unpacking registry.k8s.io/kube-apiserver:v1.32.4 (sha256:631c6cc78b2862be4fed7df3384a643ef7297eebadae22e8ef9cbe2e19b6386f)...done
unpacking registry.k8s.io/kube-controller-manager:v1.32.4 (sha256:25e29187ea66f0ff9b9a00114849c3a30b649005c900a8b2a69e3f3fa56448fb)...done
unpacking registry.k8s.io/kube-proxy:v1.32.4 (sha256:152638222ecf265eb8e5352e3c50e8fc520994e8ffcff1ee1490c975f7fc2b36)...done
unpacking registry.k8s.io/kube-scheduler:v1.32.4 (sha256:09c55f8dac59a4b8e5e354140f5a4bdd6fa9bd95c42d6bcba6782ed37c31b5a2)...done
unpacking registry.k8s.io/sig-storage/livenessprobe:v2.12.0 (sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8 (sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.10 (sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a)...done
unpacking registry.mainco.com/weaveworks/weave-kube:latest (sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking registry.mainco.com/weaveworks/weave-npc:latest (sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.10
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks

========

2025-06-03 07:42:57,511 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) Failed to upgrade Kubernetes cluster : 82c623a8ac92d1ffee84962c976e6800-01jwtbe6dpqkmw633gwkaagpba, unable to upgrade Kubernetes node on VM : k8s-82c623a8ac92d1ffee84962c976e6800-01j-control-19734b7e1d6, retries left: 3

2025-06-03 07:43:55,561 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done

=====
2025-06-03 07:43:56,433 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
mount: /mnt/k8sdisk: /dev/sr0 already mounted on /mnt/k8sdisk.
cp: cannot create regular file '/opt/bin/kubeadm': Text file busy

[weizhouapache]

it looks like the issue is caused by

cp: cannot create regular file '/opt/bin/kubeadm': Text file busy

[ahmadamirahmadi1401]

Yes, but I did not make any changes to the file and this is the default behavior when upgrading.
This happens for other versions as well. For example, the following error was obtained for upgrading a Kubernetes cluster from version 1.30.5 to version 1.31.8. And in the end, it failed to unmount the ISO and got an alert.

2025-06-03 18:40:17,001 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-34:ctx-a226f7b8 job-32868 ctx-5a7f130c) (logid:f244ee01) SSH execution of command sudo ./upgrade-kubernetes.sh13249113815295454536.sh 1.31.8 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
unpacking k8s.gcr.io/sig-storage/csi-attacher:v3.0.2 (sha256:6f80b12657a7e0a5c683b24e806c4bbbe33a43e39b041fe9b7514d665d478ea4)...done
unpacking registry.k8s.io/sig-storage/csi-attacher:v4.6.1 (sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94)...done
unpacking k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1 (sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108)...done
unpacking registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1 (sha256:f25af73ee708ff9c82595ae99493cdef9295bd96953366cddf36305f82555dac)...done
unpacking k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4 (sha256:bec571992d40203edcd056ac0b0d97003887ee5e4be144c41932d18639673b03)...done
unpacking registry.k8s.io/sig-storage/csi-provisioner:v5.0.1 (sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385)...done
unpacking registry.k8s.io/sig-storage/csi-resizer:v1.11.1 (sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0 (sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.15-0 (sha256:a6dc63e6e8cfa0307d7851762fa6b629afb18f28d8aa3fab5a6e91b4af60026a)...done
unpacking registry.k8s.io/kube-apiserver:v1.31.8 (sha256:30090db6a7d53799163ce82dae9e8ddb645fd47db93f2ec9da0cc787fd825625)...done
unpacking registry.k8s.io/kube-controller-manager:v1.31.8 (sha256:29eaddc64792a689df48506e78bbc641d063ac8bb92d2e66ae2ad05977420747)...done
unpacking registry.k8s.io/kube-proxy:v1.31.8 (sha256:dd0c9a37670f209947b1ed880f06a2e93e1d41da78c037f52f94b13858769838)...done
unpacking registry.k8s.io/kube-scheduler:v1.31.8 (sha256:22994a2632e81059720480b9f6bdeb133b08d58492d0b36dfd6e9768b159b22a)...done
unpacking registry.k8s.io/sig-storage/livenessprobe:v2.12.0 (sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8 (sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.10 (sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a)...done
unpacking registry.mainx.com/weaveworks/weave-kube:latest (sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking registry.mainx.com/weaveworks/weave-npc:latest (sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.9
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks

====================
2025-06-03 18:43:04,106 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-34:ctx-a226f7b8 job-32868 ctx-5a7f130c) (logid:f244ee01) Failed to upgrade Kubernetes cluster : e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade Kubernetes node on VM : e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089
com.cloud.utils.exception.CloudRuntimeException: Failed to upgrade Kubernetes cluster : e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade Kubernetes node on VM : e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.logTransitStateDetachIsoAndThrow(KubernetesClusterActionWorker.java:266)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeKubernetesClusterNodes(KubernetesClusterUpgradeWorker.java:127)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeCluster(KubernetesClusterUpgradeWorker.java:177)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.upgradeKubernetesCluster(KubernetesClusterManagerImpl.java:1622)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy466.upgradeKubernetesCluster(Unknown Source)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd.execute(UpgradeKubernetesClusterCmd.java:114)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

2025-06-03 18:43:04,685 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:ctx-a226f7b8 job-32868) (logid:f244ee01) Complete async job-32868, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to upgrade Kubernetes cluster : e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade Kubernetes node on VM : e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089"}

@weizhouapache

[ahmadamirahmadi1401]

its another failed in upgrafing from 1.30.5 into 1.30.13
2025-06-07 14:03:11,554 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-88:ctx-2e915bda job-33084 ctx-196a511a) (logid:16a30b3c) Failed to upgrade Kubernetes cluster : 22af645d1859cb5ca6da0c484f1f37ea-01jx57z4hpk29trtvk84bk664k, unable to upgrade Kubernetes node on VM : k8s-22af645d1859cb5ca6da0c484f1f37ea-01j-control-1974a805516
com.cloud.utils.exception.CloudRuntimeException: Failed to upgrade Kubernetes cluster : 22af645d1859cb5ca6da0c484f1f37ea-01jx57z4hpk29trtvk84bk664k, unable to upgrade Kubernetes node on VM : k8s-22af645d1859cb5ca6da0c484f1f37ea-01j-control-1974a805516
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.logTransitStateDetachIsoAndThrow(KubernetesClusterActionWorker.java:266)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeKubernetesClusterNodes(KubernetesClusterUpgradeWorker.java:127)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeCluster(KubernetesClusterUpgradeWorker.java:177)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.upgradeKubernetesCluster(KubernetesClusterManagerImpl.java:1622)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy466.upgradeKubernetesCluster(Unknown Source)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd.execute(UpgradeKubernetesClusterCmd.java:114)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

[ahmadamirahmadi1401]

Why does the middle log say the upgrade was successful, but the next log says the upgrade failed?

What do you think is the problem? Do you have a solution to suggest?

2025-06-08 07:14:30,886 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-6:ctx-559bc6d5 job-33225 ctx-3b4cbb1d) (logid:33c83a6c) SSH execution of command sudo ./upgrade-kubernetes.sh4071027720196768380.sh 1.30.13 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
unpacking k8s.gcr.io/sig-storage/csi-attacher:v3.0.2 (sha256:6f80b12657a7e0a5c683b24e806c4bbbe33a43e39b041fe9b7514d665d478ea4)...done
unpacking registry.k8s.io/sig-storage/csi-attacher:v4.6.1 (sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94)...done
unpacking k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1 (sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108)...done
unpacking registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1 (sha256:f25af73ee708ff9c82595ae99493cdef9295bd96953366cddf36305f82555dac)...done
unpacking k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4 (sha256:bec571992d40203edcd056ac0b0d97003887ee5e4be144c41932d18639673b03)...done
unpacking registry.k8s.io/sig-storage/csi-provisioner:v5.0.1 (sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385)...done
unpacking registry.k8s.io/sig-storage/csi-resizer:v1.11.1 (sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0 (sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.15-0 (sha256:a6dc63e6e8cfa0307d7851762fa6b629afb18f28d8aa3fab5a6e91b4af60026a)...done
unpacking registry.k8s.io/kube-apiserver:v1.30.13 (sha256:bd68d81c20ad5781adec9f6eae24c83d6f66c3afc0d4baa32d3f2d865c82d436)...done
unpacking registry.k8s.io/kube-controller-manager:v1.30.13 (sha256:b85a5d785cc00b03613f67a7afca1e91b2a0ccf4ced8e69d38bda5f686980f3f)...done
unpacking registry.k8s.io/kube-proxy:v1.30.13 (sha256:f68590d1921db1f9a9b56898a66cf3c67fcdcb5be132f7640af8ac1297371e4a)...done
unpacking registry.k8s.io/kube-scheduler:v1.30.13 (sha256:bfa5f10dc3e14316785f23f8e1f4e4dbc556b033270d91185fca0577353cad33)...done
unpacking registry.k8s.io/sig-storage/livenessprobe:v2.12.0 (sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8 (sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.9 (sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097)...done
unpacking registry.mainxcloud.com/weaveworks/weave-kube:latest (sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking registry.mainxcloud.com/weaveworks/weave-npc:latest (sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.9
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.30.13"
[upgrade/versions] Cluster version: v1.30.13
[upgrade/versions] kubeadm version: v1.30.13
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.30.13" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4014411918"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Current and new manifests of kube-apiserver are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Current and new manifests of kube-controller-manager are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Current and new manifests of kube-scheduler are equal, skipping upgrade
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1007267359/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.30.13". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
serviceaccount/weave-net unchanged
clusterrole.rbac.authorization.k8s.io/weave-net unchanged
clusterrolebinding.rbac.authorization.k8s.io/weave-net unchanged
role.rbac.authorization.k8s.io/weave-net unchanged
rolebinding.rbac.authorization.k8s.io/weave-net unchanged
daemonset.apps/weave-net configured
serviceaccount/cloudstack-csi-controller unchanged
clusterrole.rbac.authorization.k8s.io/cloudstack-csi-controller-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/cloudstack-csi-controller-binding unchanged
csidriver.storage.k8s.io/csi.cloudstack.apache.org unchanged
deployment.apps/cloudstack-csi-controller unchanged
daemonset.apps/cloudstack-csi-node unchanged
mount: /mnt/k8sdisk: /dev/sr0 already mounted on /mnt/k8sdisk.

2025-06-08 07:13:42,139 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-6:ctx-559bc6d5 job-33225 ctx-3b4cbb1d) (logid:33c83a6c) Failed to upgrade Kubernetes cluster : c, unable to upgrade Kubernetes node on VM : c-control-1974e576a15, retries left: 2
@weizhouapache @DaanHoogland

[weizhouapache]

Why does the middle log say the upgrade was successful, but the next log says the upgrade failed?

What do you think is the problem? Do you have a solution to suggest?

the process has some steps, including the upgrade of kuberbetes clusters.
it looks like kubernetes cluster has been upgraded successfully, there are an issue in the post-upgrade step.