upgrading to runc 1.1.6 / 1.1.7 breaks

[BenTheElder]

See: #3220, https://kubernetes.slack.com/archives/CEKK1KTN2/p1683851267796889

#3221 and #3222 have test results.

The failure mode is like:

{ failed [FAILED] failed to run command '/agnhost dns-suffix' on pod, stdout: , stderr: , err: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "69cc58960cac2f68567eec2a84011360392a649b12c91f955d6ddfeb6e34b80a": OCI runtime exec failed: exec failed: unable to start container process: error adding pid 78936 to cgroups: failed to write 78936: openat2 /sys/fs/cgroup/unified/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod60faed4d_72a9_453e_be90_24e96a46d7e5.slice/cri-containerd-d9b097ed59d27ead25448f023222ac459cc26d37682af62785f486008271b361.scope/cgroup.procs: no such file or directory: unknown: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "69cc58960cac2f68567eec2a84011360392a649b12c91f955d6ddfeb6e34b80a": OCI runtime exec failed: exec failed: unable to start container process: error adding pid 78936 to cgroups: failed to write 78936: openat2 /sys/fs/cgroup/unified/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod60faed4d_72a9_453e_be90_24e96a46d7e5.slice/cri-containerd-d9b097ed59d27ead25448f023222ac459cc26d37682af62785f486008271b361.scope/cgroup.procs: no such file or directory: unknown

This is limited to cgroups v1, this happens in our CI environment which is further awful by way of host node containerd => dockerd (in CI cluster pod) => kind (running against that nested dockerd)

[BenTheElder]

xref: kubernetes/k8s.io#5276 (we'll need to make sure we continue to cover v1 elsewhere for some time)

[BenTheElder]

I ran an image built for k8s 1.27.1 with #3221 but on a GKE 1.26 / cgroupv2 node pool and no issues so far. While checking into kubernetes/k8s.io#5276

Eventually a lot of these headaches will go away with v1 goes away, but not just yet, probably another 1-2 years.

[BenTheElder]

In the CI nested environment we have:

runc version 1.1.5
commit: v1.1.5-0-gf19387a
spec: 1.0.2-dev
go: go1.19.7
libseccomp: 2.5.1

with docker 23.0.4

However there is also the host CI node level containerd/runc. I don't think I have direct access to the k8s infra CI nodes so it's not quite as easy to confirm the versions there.

[BenTheElder]

With #3221 rebased atop #3241 / #3240 runc 1.1.6 is working in Prow CI.

That leaves k8s < 1.24 to consider. Looking into options.

[BenTheElder]

I think we can just push older k8s versions with a base image from to make -C images/base push EXTRA_BUILD_OPT=--build-arg=RUNC_VERSION=v1.1.5 TAG_SUFFIX=_runc-v1.1.5 until we phase them out and add a release note about this.

[BenTheElder]

There's still some less frequent issues with misc controller in kubernetes CI. ref: #3250 (comment)

The host runtime is not aware of misc, and probably won't be for a while.