use of low level runtime in nvidia-container-runtime

[jgforbes]

I am having an issue with CDI and rootless podman on Ubuntu 24.04.2 LTS when using nvidia-container-runtime.

$ nvidia-container-runtime --version
NVIDIA Container Runtime version 1.17.8
commit: f202b80a9b9d0db00d9b1d73c0128c8962c55f4d
spec: 1.2.1

crun version 1.14.1
commit: de537a7965bfbe9992e2cfae0baeb56a08128171
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL

When using a podman command form like so:
podman run --device /dev/dri --device nvidia.com/gpu=all --runtime /usr/bin/nvidia-container-runtime $CONTAINER

and the config file /etc/nvidia-container-runtime/config.toml contains:

[nvidia-container-runtime]
debug = "/var/log/nvidia-container-runtime.log"
log-level = "debug"
mode = "auto"
runtimes = ["crun"]

I get this error:

Error: /usr/bin/nvidia-container-runtime: time="2025-06-16T16:57:14-04:00" level=error 
msg="runc create failed: unable to start container process: error during container init: 
error creating device nodes: open /home/jeffforbes/.local/share/containers/storage/overlay/41bdd82a8b4480b8cc2dbdb124ed6df12cb700cb1c84d7717606a5a43a7c7121/merged/dev/dri/card1: 
permission denied": OCI permission denied

and there is no log file.
when using "--runtime /usr/bin/crun" the container runs correctly.

Is the nvidia-container-runtime configuration to use crun being ignored? Why no logging?

[elezar]

@jgforbes when using Podman's native CDI support (i.e. the --device nvidia.com/gpu=all flag), the nvidia-container-runtime should not be required. What I expect is happening here is that because you're running rootless, it is not the /etc/nvidia-container-runtime/config.toml that is being loaded, but that the config file at ${XDG_CONFIG_HOME}/nvidia-container-runtime/config.toml is being loaded (or the default config being used).