refactor(container): reduce layers and image size. Add `.devcontainer.json` sample. #192

jorge-ortega · 2025-04-07T01:39:38Z

Combines the RUN commands to reduce image layers where possible.
Clean apt/dnf package cache, remove downloaded package files,
Temporally mount rust-toolchain.toml when installing rust.
Add a sample .devcontainer.json, configured with our ubuntu24 image, with NVidia gpus and capabilities set, and includes git. Users can copy this to .devcontainer/devcontainer.json to make additional changes.

adamcavendish · 2025-04-07T01:57:31Z

This is not worthwhile since most of the image size comes from NVidia container. Using more layers improves caching to test locally and smaller layers help on bad networking conditions.

container/rockylinux9-cuda12/Dockerfile

container/ubuntu22-cuda11/Dockerfile

adamcavendish · 2025-04-07T02:12:16Z

container/rockylinux9-cuda12/Dockerfile

-RUN curl -sSf -L -O https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/llvm7.0-static-7.0.1-7.el8.x86_64.rpm
-RUN dnf -y install ./*.rpm
-RUN ln -s /usr/bin/llvm-config-7-64 /usr/bin/llvm-config
+RUN curl -sSf -L -O https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/l/libffi3.1-3.1-36.el9.x86_64.rpm && \


Merging curl RUNs are fine since I think no many people would like to modify it.

jorge-ortega · 2025-04-07T02:44:53Z

I'm not able to build the ubuntu24 in main unless unless I add apt-get update to the second RUN, or combine the first two:

| main > v1.87.0-nightly > docker build -f .\container\ubuntu24-cuda12\Dockerfile .
[+] Building 5.1s (7/21)                                                                                                    docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                        0.0s
 => => transferring dockerfile: 1.68kB                                                                                                      0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04                                                       0.0s
 => [internal] load .dockerignore                                                                                                           0.0s
 => => transferring context: 2B                                                                                                             0.0s
 => [ 1/17] FROM docker.io/nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04                                                                       0.0s
 => [internal] load build context                                                                                                           0.0s
 => => transferring context: 40B                                                                                                            0.0s
 => CACHED [ 2/17] RUN apt-get update                                                                                                       0.0s
 => ERROR [ 3/17] RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y install     build-essential     clang     curl     libssl-dev     libt  5.0s
------
 > [ 3/17] RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y install     build-essential     clang     curl     libssl-dev     libtinfo-dev     pkg-config     xz-utils     zlib1g-dev:
5.004 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata_2025a-0ubuntu0.24.04_all.deb  404  Not Found [IP: 91.189.91.81 80]
5.004 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I'm not sure if its docker desktop thing, but I'd figure if I needed to make that change, might as well combine a few others for good measure. Image sizes goes down from 15gb to 13.2, about 12%. Not a lot, but not a little.

I was mostly thinking of these for CI then as a full dev setup. I have a devcontainer.json in my workspace that uses the image with a few other changes to include git, rust-analyzer component and vscode extension, and run args to enable the GPU. I can include that here if that would balance out your DX aspect.

adamcavendish · 2025-04-07T02:57:46Z

I'm not able to build the ubuntu24 in main unless unless I add apt-get update to the second RUN, or combine the first two:

| main > v1.87.0-nightly > docker build -f .\container\ubuntu24-cuda12\Dockerfile .
[+] Building 5.1s (7/21)                                                                                                    docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                        0.0s
 => => transferring dockerfile: 1.68kB                                                                                                      0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04                                                       0.0s
 => [internal] load .dockerignore                                                                                                           0.0s
 => => transferring context: 2B                                                                                                             0.0s
 => [ 1/17] FROM docker.io/nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04                                                                       0.0s
 => [internal] load build context                                                                                                           0.0s
 => => transferring context: 40B                                                                                                            0.0s
 => CACHED [ 2/17] RUN apt-get update                                                                                                       0.0s
 => ERROR [ 3/17] RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y install     build-essential     clang     curl     libssl-dev     libt  5.0s
------
 > [ 3/17] RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y install     build-essential     clang     curl     libssl-dev     libtinfo-dev     pkg-config     xz-utils     zlib1g-dev:
5.004 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata_2025a-0ubuntu0.24.04_all.deb  404  Not Found [IP: 91.189.91.81 80]
5.004 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I'm not sure if its docker desktop thing, but I'd figure if I needed to make that change, might as well combine a few others for good measure. Image sizes goes down from 15gb to 13.2, about 12%. Not a lot, but not a little.

I was mostly thinking of these for CI then as a full dev setup. I have a devcontainer.json in my workspace that uses the image with a few other changes to include git, rust-analyzer component and vscode extension, and run args to enable the GPU. I can include that here if that would balance out your DX aspect.

Emm ... yeah, seems that we have to do apt update maybe because of the tzdata has some security updates. For 12% size change I'm ok with that. The comments would need to live before the RUN commands.

adamcavendish · 2025-04-07T02:59:09Z

I'm not able to build the ubuntu24 in main unless unless I add apt-get update to the second RUN, or combine the first two:

| main > v1.87.0-nightly > docker build -f .\container\ubuntu24-cuda12\Dockerfile .
[+] Building 5.1s (7/21)                                                                                                    docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                        0.0s
 => => transferring dockerfile: 1.68kB                                                                                                      0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04                                                       0.0s
 => [internal] load .dockerignore                                                                                                           0.0s
 => => transferring context: 2B                                                                                                             0.0s
 => [ 1/17] FROM docker.io/nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04                                                                       0.0s
 => [internal] load build context                                                                                                           0.0s
 => => transferring context: 40B                                                                                                            0.0s
 => CACHED [ 2/17] RUN apt-get update                                                                                                       0.0s
 => ERROR [ 3/17] RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y install     build-essential     clang     curl     libssl-dev     libt  5.0s
------
 > [ 3/17] RUN DEBIAN_FRONTEND=noninteractive apt-get -qq -y install     build-essential     clang     curl     libssl-dev     libtinfo-dev     pkg-config     xz-utils     zlib1g-dev:
5.004 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata_2025a-0ubuntu0.24.04_all.deb  404  Not Found [IP: 91.189.91.81 80]
5.004 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I'm not sure if its docker desktop thing, but I'd figure if I needed to make that change, might as well combine a few others for good measure. Image sizes goes down from 15gb to 13.2, about 12%. Not a lot, but not a little.

I was mostly thinking of these for CI then as a full dev setup. I have a devcontainer.json in my workspace that uses the image with a few other changes to include git, rust-analyzer component and vscode extension, and run args to enable the GPU. I can include that here if that would balance out your DX aspect.

For devcontainer, yeah, I think we can add that. Do you think we should just provide a sample .devcontainer.json or ask developers to use ours?

jorge-ortega · 2025-04-07T03:15:52Z

Just a sample they can start with. I'll prefix the file as such and add .devcontainer folder to gitignore. Users can rename that or copy to their .devcontainer.

adamcavendish · 2025-04-07T04:49:58Z

.devcontainer.json

+    "dockerfile": "${localWorkspaceFolder}/container/ubuntu24-cuda12/Dockerfile",
+    "context": "${localWorkspaceFolder}"
+  },
+  "runArgs": ["--runtime=nvidia", "--gpus", "all"],


To enable OptiX on Linux desktop distributions with graphics cards, we would need to enable extra capabilities. Maybe we can just use all?

Reference: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html#driver-capabilities

Based on the reference, we would need to set the environment variable:

NVIDIA_DRIVER_CAPABILITIES=all

Set using containerEnv.

I can't verify the optix examples run atm. A WSL setup is more convoluted, requiring having to copy over driver files.

That's fine. I think we can start with it.

container/rockylinux9-cuda12/Dockerfile

- Combines the RUN commands to reduce image layers where possible. - Clean apt/dnf package cache and remove downloaded package files.

Includes a `.devcontainer.json` in the project root to allow users to start a development container in supported supported tools. Updated README.md to include how to make customizations. In vscode at least, the presesne of both `.devcontainer.json` and `.devcontainer/devcontainer.json` brings up a selection menu to choose which to use.

jorge-ortega · 2025-04-07T18:48:34Z

Now that the images are public, I switched to using the ghcr.io image in .devcontainer.json.

adamcavendish · 2025-04-08T02:57:03Z

LGTM. Would you like to change the title and description before merging?

jorge-ortega force-pushed the reduce-container-layers branch from dc32288 to 6598f9e Compare April 7, 2025 01:57