RPM sandbox directory empty since rpmbuild 4.20.0

[teodor-pripoae]

After updating system rpmbuild to 4.20.0 building RPM packages fail.

OS: OpenSuse Tumbleweed.

Building rules_pkg project fails with:

Starting local Bazel server (8.1.0) and connecting to it...
WARNING: WORKSPACE support will be removed in Bazel 9 (late 2025), please migrate to Bzlmod, see https://bazel.build/external/migration.
DEBUG: /home/toni/code/gh/rules_pkg/pkg/mappings.bzl:307:34: same source mapped to different locations <generated file tests/an_executable> an_executable an_executable.runfiles/_main/tests/an_executable
INFO: From MakeRpm tests/rpm/test_rpm_bzip2-1.1.1-2222.noarch.rpm:
Error calling rpmbuild:
Building target platforms: noarch
Building for target noarch
Executing(%mkbuilddir): /usr/bin/bash -e /tmp/tmpgpatiq6v/TMP/rpm-tmp.UTgJ0s
+ umask 022
+ cd /tmp/tmpgpatiq6v/BUILD/test_rpm_bzip2-1.1.1-build
+ test -d /tmp/tmpgpatiq6v/BUILD/test_rpm_bzip2-1.1.1-build
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w /tmp/tmpgpatiq6v/BUILD/test_rpm_bzip2-1.1.1-build
+ /usr/bin/rm -rf /tmp/tmpgpatiq6v/BUILD/test_rpm_bzip2-1.1.1-build
+ /usr/bin/mkdir -p /tmp/tmpgpatiq6v/BUILD/test_rpm_bzip2-1.1.1-build
+ /usr/bin/mkdir -p /tmp/tmpgpatiq6v/BUILD/test_rpm_bzip2-1.1.1-build/SPECPARTS
+ RPM_EC=0
++ jobs -p
+ exit 0
Executing(%install): /usr/bin/bash -e /tmp/tmpgpatiq6v/TMP/rpm-tmp.xnb04w
+ umask 022
+ cd /tmp/tmpgpatiq6v/BUILD/test_rpm_bzip2-1.1.1-build
+ /usr/bin/rm -rf /tmp/tmpgpatiq6v/BUILDROOT
+ /usr/bin/mkdir -p /tmp/tmpgpatiq6v
+ /usr/bin/mkdir /tmp/tmpgpatiq6v/BUILDROOT
++ dirname /test/a.ar
+ install -d /tmp/tmpgpatiq6v/BUILDROOT//test
+ cp tests/testdata/a.ar /tmp/tmpgpatiq6v/BUILDROOT//test/a.ar
cp: cannot stat 'tests/testdata/a.ar': No such file or directory
error: Bad exit status from /tmp/tmpgpatiq6v/TMP/rpm-tmp.xnb04w (%install)

RPM build errors:
    Bad exit status from /tmp/tmpgpatiq6v/TMP/rpm-tmp.xnb04w (%install)

No RPM file created.

[kellyma2]

What do your rules and overall configuration look like?
Does this work with previous versions of rpmbuild on tumbleweed?

How rpmbuild functions varies between OSes (and potentially OS versions) and there's some logic in https://github.com/bazelbuild/rules_pkg/blob/main/toolchains/rpm/rpmbuild_configure.bzl to figure out which family it belongs to. If you're able to patch that with the appropriate values for OpenSuSE tumbleweed and test it out (looks superficially like a Fedora variant) than that would be very helpful.

[teodor-pripoae]

Yeah, it worked perfectly on 4.19.x. Since tumbleweed is a rolling release like Arch, we get latest versions when they are released. So it may be something related to 4.20.x.

I first discovered the bug while building RPMs on my project which used to work under 4.19 but I've also tested building rules_pkg and it fails with the same errors so it's not specific to my configuration, it seems to be a rpm build issue.

Maybe it's related to how Opensuse packages rpmbuild if on Fedora 41 is working. I will try on a Fedora 41 VM to see if I can reproduce.

[kellyma2]

Thanks. More details definitely appreciated. It's definitely going to be related to underlying rpmbuild behaviour - this specific part varies between fedora(40?) and some other RPM-based distros. rpmbuild is actually rpmbuild+shell scripts+macros (the latter especially being distro specific) so OpenSuSE may have changed something with the upgrade.

[teodor-pripoae]

Ok, so I've tested on a Fedora 41 vagrant box and it fails with the same errors.

I was not able to build main branch due to some stardoc error [1] but I've tested on 8c23560 and I get about the same rpmbuild error, probably from another failing test.

++ dirname /test_debuginfo
+ install -d /tmp/tmpzy_ydcbc/BUILDROOT//
+ cp bazel-out/k8-fastbuild/bin/tests/rpm/test_debuginfo /tmp/tmpzy_ydcbc/BUILDROOT//test_debuginfo
cp: cannot stat 'bazel-out/k8-fastbuild/bin/tests/rpm/test_debuginfo': No such file or directory
error: Bad exit status from /tmp/tmpzy_ydcbc/TMP/rpm-tmp.xLFWw5 (%install)

RPM build warnings:
    %source_date_epoch_from_changelog is set, but %changelog has no entries to take a date from

RPM build errors:
    Bad exit status from /tmp/tmpzy_ydcbc/TMP/rpm-tmp.xLFWw5 (%install)

No RPM file created.
ERROR: /home/vagrant/code/gh/rules_pkg/tests/rpm/BUILD:608:8: output 'tests/rpm/test_debuginfo_rpm-1-0..rpm' was not created
ERROR: /home/vagrant/code/gh/rules_pkg/tests/rpm/BUILD:608:8: MakeRpm tests/rpm/test_debuginfo_rpm-1-0..rpm failed: not all outputs were created or valid

I have created a vagrant file to reproduce on Fedora 41:

Vagrant.configure("2") do |config|
  # Use the Fedora Cloud Base image
  config.vm.box = "fedora/41-cloud-base"

  # Use the libvirt provider
  config.vm.provider "libvirt" do |libvirt|
    libvirt.qemu_use_session = false  # Use system libvirt, requires root privileges
    libvirt.cpus = 8                  # Number of CPU cores
    libvirt.memory = 16384            # Memory allocation in MB
    libvirt.disk_bus = "virtio"       # Use virtio for disk I/O
  end

  # Configure networking
  config.vm.network "public_network", dev: "eno1"

  # Set up SSH access
  config.ssh.username = "vagrant"
  #config.ssh.private_key_path = "~/.ssh/id_rsa"
  config.ssh.forward_agent = true

  # Optional: provision with cloud-init or shell script
  config.vm.provision "shell", inline: <<-SHELL
    echo "Updating system packages..."
    dnf update -y
    dnf install -y git gcc g++ make rpmbuild wget
    wget -O /usr/bin/bazel https://github.com/bazelbuild/bazelisk/releases/download/v1.25.0/bazelisk-linux-amd64 && sudo chmod +x /usr/bin/bazel
    sudo -u vagrant bash -c 'mkdir /home/vagrant/code && cd /home/vagrant/code && git clone https://github.com/bazelbuild/rules_pkg.git && cd rules_pkg && git checkout 1772ebd'
  SHELL
end

Then just run vagrant ssh and try to build the repository.

[1] Error when building on main, which also fails on CI:

ERROR: /home/vagrant/code/gh/rules_pkg/doc_build/BUILD:65:12: in deps attribute of starlark_doc_extract rule //doc_build:pkg_rpm_gen.extract: missing bzl_library targets for Starlark module(s) //toolchains/rpm:rpmbuild_configure.bzl. Since this rule was created by the macro 'stardoc', the error might have been caused by the macro implementation
ERROR: /home/vagrant/code/gh/rules_pkg/doc_build/BUILD:65:12: Analysis of target '//doc_build:pkg_rpm_gen.extract' failed

[teodor-pripoae]

On Fedora 40 with rpm 4.19.1.1 it builds fine:

INFO: Found 412 targets...
INFO: Elapsed time: 109.797s, Critical Path: 35.20s
INFO: 1578 processes: 1030 internal, 537 linux-sandbox, 2 local, 9 worker.
INFO: Build completed successfully, 1578 total actions

[teodor-pripoae]

Update:

It seems that starting from RPM 4.20, it is creating a new directory inside the BUILD directory. The bazel-out files are not in the current directory but one directory above.

The build script has this additional content:

  cd "/tmp/tmp0og0g8op/BUILD/testrpm-0.0.1-build"
  /usr/bin/rm -rf "$RPM_BUILD_ROOT"
  /usr/bin/mkdir -p `dirname "$RPM_BUILD_ROOT"`
  /usr/bin/mkdir "$RPM_BUILD_ROOT"

The directory structure for 4.19:

PWD: /tmp/tmp2sb0xt_f/BUILD

/SRPMS
/RPMS
/TMP
/TMP/rpm-tmp.1jeWa0
/BUILDROOT
/test-rpm.spec
/BUILD
/BUILD/bazel-out
/BUILD/bazel-out/k8-fastbuild
/BUILD/bazel-out/k8-fastbuild/bin
/BUILD/bazel-out/k8-fastbuild/bin/archive.tar.gz
/SOURCES

The directory structure for 4.20

PWD: /tmp/tmpepxelsjt/BUILD/test-rpm-0-build

/SRPMS
/RPMS
/TMP
/TMP/rpm-tmp.zceJrK
/test-rpm.spec
/BUILD
/BUILD/bazel-out
/BUILD/bazel-out/k8-fastbuild
/BUILD/bazel-out/k8-fastbuild/bin
/BUILD/bazel-out/k8-fastbuild/bin/archive.tar.gz
/BUILD/test-rpm-0-build
/BUILD/test-rpm-0-build/SPECPARTS
/SOURCES

[kellyma2]

Interesting. Can you distill your setup into something that I can use to reproduce this?

[teodor-pripoae]

Hi,

Try this Dockerfile.

FROM registry.fedoraproject.org/fedora:41

RUN dnf install -y git gcc g++ rpm-build wget curl && \
    wget -O /usr/bin/bazel https://github.com/bazelbuild/bazelisk/releases/download/v1.25.0/bazelisk-linux-amd64 && \
    chmod +x /usr/bin/bazel && \
    dnf clean all && \
    mkdir /app && \
    useradd app && \
    chown app:app /app

USER app

RUN cd /app && \
    git clone https://github.com/bazelbuild/rules_pkg.git

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

WORKDIR /app/rules_pkg

CMD ["bazel", "build", "//..."]

$ podman build -t rules_pkg:41 -f Dockerfile.fc41 .
$ podman run -it localhost/rules_pkg:41

If you replace the base image with registry.fedoraproject.org/fedora:40 and run the following it will pass:

$ podman build -t rules_pkg:40 -f Dockerfile.fc40 .
$ podman run -it localhost/rules_pkg:40