Seeking ContainerOS-friendly Driver Deployment with CDI Path Consistency

[wqlparallel]

Hi,

We're implementing GPU support on ContainerOS-based Kubernetes nodes with two critical requirements:

1. Security-driven path isolation: All driver files (.so, bins, configs) must reside under a single host path (e.g., /usr/local/nvidia/driver), avoiding writes to /usr/lib//etc per ContainerOS hardening policies

2. CDI path preservation: nvidia-ctk cdi generate must produce container paths identical to standard gpu-operator output for compatibility

Current workaround (inefficient):

1. Build driver in privileged container

2. Copy entire container rootfs (→ ~1GB bloat) to isolated host dir

3. Manual kernel module loading

Key questions:

1. Path mapping: Is there supported configuration (e.g., env vars, config files) to:

• Specify custom host driver path while
• Maintaining default container paths in CDI specs (like /usr/lib/x86_64-linux-gnu/libcuda.so)

2. Minimal file set: Are there tools/patterns to identify/copy only driver-essential files (avoiding 1GB+ OS cruft) while keeping CDI generation intact?

3. Production-grade alternatives: Any recommendations for:

• Single-directory driver containment
• CDI path consistency
• Driver lifecycle decoupled from containers?

Would appreciate insights on achieving secure path isolation without sacrificing CDI compatibility or disk efficiency!

[elezar]

@wqlparallel this sounds like something akin to what we already do for our driver container. It may be more useful to start discussion in the gpu-operator repo.

@cdesiniotis @tariq1890 do you have any thoughts?

[wqlparallel]

@wqlparallel this sounds like something akin to what we already do for our driver container. It may be more useful to start discussion in the gpu-operator repo.

@cdesiniotis @tariq1890 do you have any thoughts?

Appreciate your suggestions, but we’re currently not looking to use the Driver Container for a specific reason.

The main issue is that this approach ties the driver’s lifecycle (loading and unloading) to the start and stop of the Driver Container. If the Driver Container crashes or needs a restart, we would have to evict all GPU application containers that are using that driver. This presents an unacceptable risk of disruption and interruption to our ongoing GPU workloads.

Given the importance of keeping our services up and running smoothly, we prefer to avoid this potential downtime.