No VMs start after Renew Host Security Keys on newer Ubuntu and newer libvirt

[StepBee]

problem

Dear CloudStack community,

we discovered in bug which prevents VM start on (at least) newer Ubuntu Versions or newer Libvirt version after provisioning host security keys.

Looks like @rohityadavcloud stumbled across the same bug in #9984 but did not follow up on the root cause

The issue is in

cloudstack/scripts/util/keystore-cert-import

Line 125 in 3e3a0c0

QEMU_GROUP=$(sed -n 's/^group=//p' /etc/libvirt/qemu.conf | awk -F'"' '{print $2}' | tail -n1)

In newer versions of Libvirt, the default user and group changed from user:root and group:root to user: libvirt-qemu and group: kvm

The mentioned line in the script is checking the config file /etc/libvirt/qemu.conf for the pattern "group=" and will change the owner group of the new host security key certificate files to the group name found in the config file.
If "group=" is not set the chgrp command is not executed and the certificate files will stay with owner root:root
Since the new default user and group of libvirt is libvirt-qemu:kvm the libvirt process will not be able to access the certificate files in standard configuration ("user=" and "group=" are not specifically set and are commented in qemu.conf) and will raise the error

Cannot load CA certificate '/etc/pki/libvirt-vnc/ca-cert.pem': Error while reading file.

The second issue is the pattern the script is using to search for the group.
The search does not include spaces like "group = kvm", only "group=kvm" will work - but spaces are the default in the commented lines in qemu.conf.

versions

Ubuntu 24.04.2 LTS
libvirt version: 10.0.0
Apache Cloudstack 20.1.0

The steps to reproduce the bug

1. Install CloudStack on one of the newer Ubuntu Releases
2. Execute "Provision Host Security Keys"
3. Try to start a VM on the host

What to do about it?

To fix the issue two steps are required from my perspective:

1. replacing the matching pattern in the script

   cloudstack/scripts/util/keystore-cert-import

   Line 125 in 3e3a0c0

   QEMU_GROUP=$(sed -n 's/^group=//p' /etc/libvirt/qemu.conf | awk -F'"' '{print $2}' | tail -n1)

   
   with an expression allowing spaces as well like for example
   $(sed -n 's/^group\s*=//p' /etc/libvirt/qemu.conf | awk -F'"' '{print $2}' | tail -n1)

2. Adjust the installation documentation for (at least) Ubuntu adding the requirement to specifically define "group = kvm" in /etc/libvirt/qemu.conf

[weizhouapache]

thanks @StepBee

it looks like the expression does not work with group = kvm (no ")

[StepBee]

thanks @weizhouapache indeed.

An idea for a working replacement could be
QEMU_GROUP=$(sed -n 's/^group\s*=//p' /tmp/qemu.conf | awk -F'=' '{print $1}' | tr -d '"' |tr -d ' ' | tail -n1)
which takes care of spaces and quotation marks.