Skip to content

Update NVIDIA driver symlink script#158

Merged
bedroge merged 6 commits intoEESSI:mainfrom
casparvl:link_nvidia_drivers
Feb 18, 2026
Merged

Update NVIDIA driver symlink script#158
bedroge merged 6 commits intoEESSI:mainfrom
casparvl:link_nvidia_drivers

Conversation

@casparvl
Copy link
Contributor

@casparvl casparvl commented Feb 4, 2026

We'll need the following variant symlinks to be in place before this script can work as intended:

ln -s '$(EESSI_202506_NVIDIA_OVERRIDE:-/cvmfs/software.eessi.io/defaults/nvidia)' /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia
ln -s '$(EESSI_202506_NVIDIA_OVERRIDE:-/cvmfs/software.eessi.io/defaults/nvidia)' /cvmfs/software.eessi.io/versions/2025.06/compat/linux/aarch64/lib/nvidia
ln -s '$(EESSI_202506_NVIDIA_OVERRIDE:-/cvmfs/software.eessi.io/defaults/nvidia)' /cvmfs/software.eessi.io/versions/2025.06/compat/linux/riscv64/lib/nvidia

And then:

ln -s '$(EESSI_NVIDIA_OVERRIDE_DEFAULT:-/dev/null)' /cvmfs/software.eessi.io/defaults/nvidia

This can then be quite easily tested from within the container:

./eessi_container.sh -a rw -r software.eessi.io -b $<host-software-layer-scripts>:/software-layer-scripts --nvidia all
cd /software-layer-scripts/scripts/gpu_support/nvidia
./link_nvidia_host_libraries.sh

This should error out stating that the variant symlink resolves to /dev/null. Then, you can change /etc/cvmfs/default.local to set e.g. EESSI_NVIDIA_OVERRIDE_DEFAULT (e.g. to /opt/eessi/nvidia) and run the linking script again - this should the install the symlinks.

@casparvl
Copy link
Contributor Author

casparvl commented Feb 4, 2026

Although we don't have the symlinks yet, I can actually already test this in the container - it will just create the symlinks in /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia/ in the writeable overlay. That's fine.

What I did:

$ cd /software-layer-scripts/scripts/gpu_support/nvidia/
$ umask 0022
$  source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash
# For some reason this failed to load the module - some module cache issue?
$ module load EESSI/2025.06
$ cat > dummy.c <<'EOF'
int main(void) { return 0; }
EOF
$ gcc -Wall -Wl,--no-as-needed -lcuda dummy.c -o dummy -L /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia/
# singularity has /.singularity.d/libs with the CUDA drivers in the LD_LIBRARY_PATH, but those are not the ones we want to find...
$  unset LD_LIBRARY_PATH
$ ldd dummy
Apptainer> ldd dummy
        linux-vdso.so.1 (0x00007ffc59bb4000)
        libcuda.so.1 => /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia/libcuda.so.1 (0x000014f19b377000)
...

Works as intended. After implementing the variant symlinks, we should retest, try to use the EESSI_NVIDIA_OVERRIDE_DEFAULT symlink, and, once that works, try again using the EESSI_202506_NVIDIA_OVERRIDE variant symlink.

@bedroge
Copy link
Contributor

bedroge commented Feb 17, 2026

Tested in the container using EESSI 2025.06 and without having configured the variant symlinks:

ERROR: /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia is a symlink pointing to /cvmfs/software.eessi.io/defaults/nvidia, which is a symlink pointing to /dev/null
If you want to symlink the drivers in a single location for all EESSI versions, please define the EESSI_NVIDIA_OVERRIDE_DEFAULT variant symlink in your local CVMFS configuration to point to writeable location. This will change the target of symlink /cvmfs/software.eessi.io/defaults/nvidia.
If you want to symlink the drivers only for this version of EESSI (2025.06), please define the EESSI__NVIDIA_OVERRIDE variant symlink in your local CVMFS configuration to point to writeable location. This will change the target of symlink /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia.

With the variant symlink reconfigured as EESSI_NVIDIA_OVERRIDE_DEFAULT=/opt/eessi/nvidia:

Ensure the final target of /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia (/opt/eessi/nvidia) exists
Host NVIDIA GPU drivers linked successfully for EESSI

Wiping that dir and doing it again using EESSI_202506_NVIDIA_OVERRIDE=/opt/eessi/nvidia yields the same result.

Also checked the symlinks, and the pointed to the expected locations.

# Do some checks on existence of links and that we don't end up at /dev/null (the default), so we can print some informative information
# One downside is that we can't explicitely check if something is a variant symlink, so we'll just assume that if it's a link AND it
# lives in our CVMFS repository, it must be a variant symlink
nvidia_trusted_dir="${EESSI_EPREFIX}/lib/nvidia"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that the script will no longer work for 2023.06?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, yeah, that's annoying, this script is in an unversioned prefix. I mean, if we deploy this only for 2025.06, we keep the old version for 2023.06. But then if we want to update that, we have to revert all changes, etc. Maybe we should just duplicate the script? I.e. create something like scripts/gpu_support/nvidia/2023.06/link_nvidia_host_libraries.sh? Or should it be at higher level 2023.06/scripts/gpu_support...?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I did solve it differently in the end. Only the symlink_mode function changed. I now just duplicated it, and call the correct one based on the value of EESSI_VERSION (note that check_eessi_initialized is called before symlink_mode, so we can be sure that EESSI is initialized and that EESSI_VERSION has been set by the time it gets to that part of the script).

The duplication means there is some code duplication - but actually not to much: the function has changed substantially, so I think the small amount of duplication that is still left is acceptable - especially since it's unlikely we'll still change anything on the 2023.06 side.

@casparvl
Copy link
Contributor Author

casparvl commented Feb 18, 2026

Tested using:

./eessi_container.sh --nvidia all --extra-bind-paths $PWD:/software-layer-scripts --fakeroot --pass-through --writable-tmpfs
source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash
cd /software-layer-scripts/scripts/gpu_support/nvidia/
umask 0022
./link_nvidia_host_libraries.sh

Gives:

ERROR: /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia is a symlink pointing to /cvmfs/software.eessi.io/defaults/nvidia, which is a symlink pointing to /dev/null
If you want to symlink the drivers in a single location for all EESSI versions, please define the EESSI_NVIDIA_OVERRIDE_DEFAULT variant symlink in your local CVMFS configuration to point to writeable location. This will change the target of symlink /cvmfs/software.eessi.io/defaults/nvidia.
If you want to symlink the drivers only for this version of EESSI (2025.06), please define the EESSI__NVIDIA_OVERRIDE variant symlink in your local CVMFS configuration to point to writeable location. This will change the target of symlink /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia.

As expected. Then, modify the CVMFS config:

echo EESSI_NVIDIA_OVERRIDE_DEFAULT=/opt/eessi/nvidia >> /etc/cvmfs/default.local
cvmfs_config reload

And run again:

./link_nvidia_host_libraries.sh
...
Ensure the final target of /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia (/opt/eessi/nvidia) exists
Host NVIDIA GPU drivers linked successfully for EESSI

The result looks fine:

ls -al /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia
lrwxrwxrwx 1 root root 40 Feb 17 13:09 /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia -> /cvmfs/software.eessi.io/defaults/nvidia
ls -al /cvmfs/software.eessi.io/defaults/nvidia
lrwxrwxrwx 1 root root 17 Feb 17 13:09 /cvmfs/software.eessi.io/defaults/nvidia -> /opt/eessi/nvidia
ls -al /opt/eessi/nvidia/
total 24
drwxr-xr-x 2 root root 4096 Feb 18 10:41 .
drwxr-x--- 3 root root 4096 Feb 18 10:41 ..
-rw-r--r-- 1 root root    4 Feb 18 10:41 cuda_version.txt
-rw-r--r-- 1 root root   10 Feb 18 10:41 driver_version.txt
lrwxrwxrwx 1 root root   41 Feb 18 10:41 libcudadebugger.so.1 -> /.singularity.d/libs/libcudadebugger.so.1
lrwxrwxrwx 1 root root   31 Feb 18 10:41 libcuda.so -> /.singularity.d/libs/libcuda.so
lrwxrwxrwx 1 root root   33 Feb 18 10:41 libcuda.so.1 -> /.singularity.d/libs/libcuda.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:41 libEGL_nvidia.so.0 -> /.singularity.d/libs/libEGL_nvidia.so.0
lrwxrwxrwx 1 root root   30 Feb 18 10:41 libEGL.so -> /.singularity.d/libs/libEGL.so
lrwxrwxrwx 1 root root   32 Feb 18 10:41 libEGL.so.1 -> /.singularity.d/libs/libEGL.so.1
lrwxrwxrwx 1 root root   37 Feb 18 10:41 libGLdispatch.so -> /.singularity.d/libs/libGLdispatch.so
lrwxrwxrwx 1 root root   39 Feb 18 10:41 libGLdispatch.so.0 -> /.singularity.d/libs/libGLdispatch.so.0
lrwxrwxrwx 1 root root   45 Feb 18 10:41 libGLESv1_CM_nvidia.so.1 -> /.singularity.d/libs/libGLESv1_CM_nvidia.so.1
lrwxrwxrwx 1 root root   36 Feb 18 10:41 libGLESv1_CM.so -> /.singularity.d/libs/libGLESv1_CM.so
lrwxrwxrwx 1 root root   38 Feb 18 10:41 libGLESv1_CM.so.1 -> /.singularity.d/libs/libGLESv1_CM.so.1
lrwxrwxrwx 1 root root   42 Feb 18 10:41 libGLESv2_nvidia.so.2 -> /.singularity.d/libs/libGLESv2_nvidia.so.2
lrwxrwxrwx 1 root root   33 Feb 18 10:41 libGLESv2.so -> /.singularity.d/libs/libGLESv2.so
lrwxrwxrwx 1 root root   35 Feb 18 10:41 libGLESv2.so.2 -> /.singularity.d/libs/libGLESv2.so.2
lrwxrwxrwx 1 root root   29 Feb 18 10:41 libGL.so -> /.singularity.d/libs/libGL.so
lrwxrwxrwx 1 root root   31 Feb 18 10:41 libGL.so.1 -> /.singularity.d/libs/libGL.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:41 libGLX_nvidia.so.0 -> /.singularity.d/libs/libGLX_nvidia.so.0
lrwxrwxrwx 1 root root   30 Feb 18 10:41 libGLX.so -> /.singularity.d/libs/libGLX.so
lrwxrwxrwx 1 root root   32 Feb 18 10:41 libGLX.so.0 -> /.singularity.d/libs/libGLX.so.0
lrwxrwxrwx 1 root root   34 Feb 18 10:41 libnvcuvid.so -> /.singularity.d/libs/libnvcuvid.so
lrwxrwxrwx 1 root root   36 Feb 18 10:41 libnvcuvid.so.1 -> /.singularity.d/libs/libnvcuvid.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:41 libnvidia-cfg.so.1 -> /.singularity.d/libs/libnvidia-cfg.so.1
lrwxrwxrwx 1 root root   51 Feb 18 10:41 libnvidia-eglcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-eglcore.so.590.48.01
lrwxrwxrwx 1 root root   47 Feb 18 10:41 libnvidia-egl-wayland.so.1 -> /.singularity.d/libs/libnvidia-egl-wayland.so.1
lrwxrwxrwx 1 root root   40 Feb 18 10:41 libnvidia-encode.so -> /.singularity.d/libs/libnvidia-encode.so
lrwxrwxrwx 1 root root   42 Feb 18 10:41 libnvidia-encode.so.1 -> /.singularity.d/libs/libnvidia-encode.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:41 libnvidia-fbc.so.1 -> /.singularity.d/libs/libnvidia-fbc.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:41 libnvidia-glcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-glcore.so.590.48.01
lrwxrwxrwx 1 root root   48 Feb 18 10:41 libnvidia-glsi.so.590.48.01 -> /.singularity.d/libs/libnvidia-glsi.so.590.48.01
lrwxrwxrwx 1 root root   53 Feb 18 10:41 libnvidia-glvkspirv.so.590.48.01 -> /.singularity.d/libs/libnvidia-glvkspirv.so.590.48.01
lrwxrwxrwx 1 root root   51 Feb 18 10:41 libnvidia-gpucomp.so.590.48.01 -> /.singularity.d/libs/libnvidia-gpucomp.so.590.48.01
lrwxrwxrwx 1 root root   48 Feb 18 10:41 libnvidia-gtk3.so.590.48.01 -> /.singularity.d/libs/libnvidia-gtk3.so.590.48.01
lrwxrwxrwx 1 root root   38 Feb 18 10:41 libnvidia-ml.so.1 -> /.singularity.d/libs/libnvidia-ml.so.1
lrwxrwxrwx 1 root root   40 Feb 18 10:41 libnvidia-nvvm.so.4 -> /.singularity.d/libs/libnvidia-nvvm.so.4
lrwxrwxrwx 1 root root   42 Feb 18 10:41 libnvidia-opencl.so.1 -> /.singularity.d/libs/libnvidia-opencl.so.1
lrwxrwxrwx 1 root root   47 Feb 18 10:41 libnvidia-opticalflow.so.1 -> /.singularity.d/libs/libnvidia-opticalflow.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:41 libnvidia-ptxjitcompiler.so.1 -> /.singularity.d/libs/libnvidia-ptxjitcompiler.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:41 libnvidia-rtcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-rtcore.so.590.48.01
lrwxrwxrwx 1 root root   47 Feb 18 10:41 libnvidia-tls.so.590.48.01 -> /.singularity.d/libs/libnvidia-tls.so.590.48.01
lrwxrwxrwx 1 root root   36 Feb 18 10:41 libnvoptix.so.1 -> /.singularity.d/libs/libnvoptix.so.1
lrwxrwxrwx 1 root root   35 Feb 18 10:41 libOpenCL.so.1 -> /.singularity.d/libs/libOpenCL.so.1
lrwxrwxrwx 1 root root   33 Feb 18 10:41 libOpenGL.so -> /.singularity.d/libs/libOpenGL.so
lrwxrwxrwx 1 root root   35 Feb 18 10:41 libOpenGL.so.0 -> /.singularity.d/libs/libOpenGL.so.0

(of course, we don't need to symlink this in a container, as singularity sets LD_LIBRARY_PATH, but fine)

Similarly, if I set

echo EESSI_202506_NVIDIA_OVERRIDE=/opt/eessi/nvidia202506 >> /etc/cvmfs/default.local
cvmfs_config reload
./link_nvidia_host_libraries.sh
...
Ensure the final target of /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia (/opt/eessi/nvidia202506) exists
Host NVIDIA GPU drivers linked successfully for EESSI

And the result looks as expected:

{EESSI/2025.06} Apptainer> ls -al /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia
lrwxrwxrwx 1 root root 23 Feb 17 13:09 /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/nvidia -> /opt/eessi/nvidia202506
{EESSI/2025.06} Apptainer> ls -al /opt/eessi/nvidia202506/
total 24
drwxr-xr-x 2 root root 4096 Feb 18 10:47 .
drwxr-x--- 4 root root 4096 Feb 18 10:47 ..
-rw-r--r-- 1 root root    4 Feb 18 10:47 cuda_version.txt
-rw-r--r-- 1 root root   10 Feb 18 10:47 driver_version.txt
lrwxrwxrwx 1 root root   41 Feb 18 10:47 libcudadebugger.so.1 -> /.singularity.d/libs/libcudadebugger.so.1
lrwxrwxrwx 1 root root   31 Feb 18 10:47 libcuda.so -> /.singularity.d/libs/libcuda.so
lrwxrwxrwx 1 root root   33 Feb 18 10:47 libcuda.so.1 -> /.singularity.d/libs/libcuda.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:47 libEGL_nvidia.so.0 -> /.singularity.d/libs/libEGL_nvidia.so.0
lrwxrwxrwx 1 root root   30 Feb 18 10:47 libEGL.so -> /.singularity.d/libs/libEGL.so
lrwxrwxrwx 1 root root   32 Feb 18 10:47 libEGL.so.1 -> /.singularity.d/libs/libEGL.so.1
lrwxrwxrwx 1 root root   37 Feb 18 10:47 libGLdispatch.so -> /.singularity.d/libs/libGLdispatch.so
lrwxrwxrwx 1 root root   39 Feb 18 10:47 libGLdispatch.so.0 -> /.singularity.d/libs/libGLdispatch.so.0
lrwxrwxrwx 1 root root   45 Feb 18 10:47 libGLESv1_CM_nvidia.so.1 -> /.singularity.d/libs/libGLESv1_CM_nvidia.so.1
lrwxrwxrwx 1 root root   36 Feb 18 10:47 libGLESv1_CM.so -> /.singularity.d/libs/libGLESv1_CM.so
lrwxrwxrwx 1 root root   38 Feb 18 10:47 libGLESv1_CM.so.1 -> /.singularity.d/libs/libGLESv1_CM.so.1
lrwxrwxrwx 1 root root   42 Feb 18 10:47 libGLESv2_nvidia.so.2 -> /.singularity.d/libs/libGLESv2_nvidia.so.2
lrwxrwxrwx 1 root root   33 Feb 18 10:47 libGLESv2.so -> /.singularity.d/libs/libGLESv2.so
lrwxrwxrwx 1 root root   35 Feb 18 10:47 libGLESv2.so.2 -> /.singularity.d/libs/libGLESv2.so.2
lrwxrwxrwx 1 root root   29 Feb 18 10:47 libGL.so -> /.singularity.d/libs/libGL.so
lrwxrwxrwx 1 root root   31 Feb 18 10:47 libGL.so.1 -> /.singularity.d/libs/libGL.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:47 libGLX_nvidia.so.0 -> /.singularity.d/libs/libGLX_nvidia.so.0
lrwxrwxrwx 1 root root   30 Feb 18 10:47 libGLX.so -> /.singularity.d/libs/libGLX.so
lrwxrwxrwx 1 root root   32 Feb 18 10:47 libGLX.so.0 -> /.singularity.d/libs/libGLX.so.0
lrwxrwxrwx 1 root root   34 Feb 18 10:47 libnvcuvid.so -> /.singularity.d/libs/libnvcuvid.so
lrwxrwxrwx 1 root root   36 Feb 18 10:47 libnvcuvid.so.1 -> /.singularity.d/libs/libnvcuvid.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:47 libnvidia-cfg.so.1 -> /.singularity.d/libs/libnvidia-cfg.so.1
lrwxrwxrwx 1 root root   51 Feb 18 10:47 libnvidia-eglcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-eglcore.so.590.48.01
lrwxrwxrwx 1 root root   47 Feb 18 10:47 libnvidia-egl-wayland.so.1 -> /.singularity.d/libs/libnvidia-egl-wayland.so.1
lrwxrwxrwx 1 root root   40 Feb 18 10:47 libnvidia-encode.so -> /.singularity.d/libs/libnvidia-encode.so
lrwxrwxrwx 1 root root   42 Feb 18 10:47 libnvidia-encode.so.1 -> /.singularity.d/libs/libnvidia-encode.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:47 libnvidia-fbc.so.1 -> /.singularity.d/libs/libnvidia-fbc.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:47 libnvidia-glcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-glcore.so.590.48.01
lrwxrwxrwx 1 root root   48 Feb 18 10:47 libnvidia-glsi.so.590.48.01 -> /.singularity.d/libs/libnvidia-glsi.so.590.48.01
lrwxrwxrwx 1 root root   53 Feb 18 10:47 libnvidia-glvkspirv.so.590.48.01 -> /.singularity.d/libs/libnvidia-glvkspirv.so.590.48.01
lrwxrwxrwx 1 root root   51 Feb 18 10:47 libnvidia-gpucomp.so.590.48.01 -> /.singularity.d/libs/libnvidia-gpucomp.so.590.48.01
lrwxrwxrwx 1 root root   48 Feb 18 10:47 libnvidia-gtk3.so.590.48.01 -> /.singularity.d/libs/libnvidia-gtk3.so.590.48.01
lrwxrwxrwx 1 root root   38 Feb 18 10:47 libnvidia-ml.so.1 -> /.singularity.d/libs/libnvidia-ml.so.1
lrwxrwxrwx 1 root root   40 Feb 18 10:47 libnvidia-nvvm.so.4 -> /.singularity.d/libs/libnvidia-nvvm.so.4
lrwxrwxrwx 1 root root   42 Feb 18 10:47 libnvidia-opencl.so.1 -> /.singularity.d/libs/libnvidia-opencl.so.1
lrwxrwxrwx 1 root root   47 Feb 18 10:47 libnvidia-opticalflow.so.1 -> /.singularity.d/libs/libnvidia-opticalflow.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:47 libnvidia-ptxjitcompiler.so.1 -> /.singularity.d/libs/libnvidia-ptxjitcompiler.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:47 libnvidia-rtcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-rtcore.so.590.48.01
lrwxrwxrwx 1 root root   47 Feb 18 10:47 libnvidia-tls.so.590.48.01 -> /.singularity.d/libs/libnvidia-tls.so.590.48.01
lrwxrwxrwx 1 root root   36 Feb 18 10:47 libnvoptix.so.1 -> /.singularity.d/libs/libnvoptix.so.1
lrwxrwxrwx 1 root root   35 Feb 18 10:47 libOpenCL.so.1 -> /.singularity.d/libs/libOpenCL.so.1
lrwxrwxrwx 1 root root   33 Feb 18 10:47 libOpenGL.so -> /.singularity.d/libs/libOpenGL.so
lrwxrwxrwx 1 root root   35 Feb 18 10:47 libOpenGL.so.0 -> /.singularity.d/libs/libOpenGL.so.0

@casparvl casparvl marked this pull request as ready for review February 18, 2026 09:48
@casparvl
Copy link
Contributor Author

casparvl commented Feb 18, 2026

Also tested 2023.06 behavior:

source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
cd /software-layer-scripts/scripts/gpu_support/nvidia/
umask 0022
./link_nvidia_host_libraries.sh
...
Linking drivers to the host_injection folder
Successfully created symlink between latest and host in /cvmfs/software.eessi.io/host_injections/nvidia/x86_64
Successfully created symlink between /cvmfs/software.eessi.io/host_injections/nvidia/x86_64/latest and lib in /cvmfs/software.eessi.io/host_injections/2023.06/compat/linux/x86_64
Host NVIDIA GPU drivers linked successfully for EESSI
 ls -al /opt/eessi/2023.06/compat/linux/x86_64/lib/
total 24
drwxr-xr-x 2 root root 4096 Feb 18 10:51 .
drwxr-xr-x 3 root root 4096 Feb 18 10:51 ..
-rw-r--r-- 1 root root    4 Feb 18 10:51 cuda_version.txt
-rw-r--r-- 1 root root   10 Feb 18 10:51 driver_version.txt
lrwxrwxrwx 1 root root   41 Feb 18 10:51 libcudadebugger.so.1 -> /.singularity.d/libs/libcudadebugger.so.1
lrwxrwxrwx 1 root root   31 Feb 18 10:51 libcuda.so -> /.singularity.d/libs/libcuda.so
lrwxrwxrwx 1 root root   33 Feb 18 10:51 libcuda.so.1 -> /.singularity.d/libs/libcuda.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:51 libEGL_nvidia.so.0 -> /.singularity.d/libs/libEGL_nvidia.so.0
lrwxrwxrwx 1 root root   30 Feb 18 10:51 libEGL.so -> /.singularity.d/libs/libEGL.so
lrwxrwxrwx 1 root root   32 Feb 18 10:51 libEGL.so.1 -> /.singularity.d/libs/libEGL.so.1
lrwxrwxrwx 1 root root   37 Feb 18 10:51 libGLdispatch.so -> /.singularity.d/libs/libGLdispatch.so
lrwxrwxrwx 1 root root   39 Feb 18 10:51 libGLdispatch.so.0 -> /.singularity.d/libs/libGLdispatch.so.0
lrwxrwxrwx 1 root root   45 Feb 18 10:51 libGLESv1_CM_nvidia.so.1 -> /.singularity.d/libs/libGLESv1_CM_nvidia.so.1
lrwxrwxrwx 1 root root   36 Feb 18 10:51 libGLESv1_CM.so -> /.singularity.d/libs/libGLESv1_CM.so
lrwxrwxrwx 1 root root   38 Feb 18 10:51 libGLESv1_CM.so.1 -> /.singularity.d/libs/libGLESv1_CM.so.1
lrwxrwxrwx 1 root root   42 Feb 18 10:51 libGLESv2_nvidia.so.2 -> /.singularity.d/libs/libGLESv2_nvidia.so.2
lrwxrwxrwx 1 root root   33 Feb 18 10:51 libGLESv2.so -> /.singularity.d/libs/libGLESv2.so
lrwxrwxrwx 1 root root   35 Feb 18 10:51 libGLESv2.so.2 -> /.singularity.d/libs/libGLESv2.so.2
lrwxrwxrwx 1 root root   29 Feb 18 10:51 libGL.so -> /.singularity.d/libs/libGL.so
lrwxrwxrwx 1 root root   31 Feb 18 10:51 libGL.so.1 -> /.singularity.d/libs/libGL.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:51 libGLX_nvidia.so.0 -> /.singularity.d/libs/libGLX_nvidia.so.0
lrwxrwxrwx 1 root root   30 Feb 18 10:51 libGLX.so -> /.singularity.d/libs/libGLX.so
lrwxrwxrwx 1 root root   32 Feb 18 10:51 libGLX.so.0 -> /.singularity.d/libs/libGLX.so.0
lrwxrwxrwx 1 root root   34 Feb 18 10:51 libnvcuvid.so -> /.singularity.d/libs/libnvcuvid.so
lrwxrwxrwx 1 root root   36 Feb 18 10:51 libnvcuvid.so.1 -> /.singularity.d/libs/libnvcuvid.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:51 libnvidia-cfg.so.1 -> /.singularity.d/libs/libnvidia-cfg.so.1
lrwxrwxrwx 1 root root   51 Feb 18 10:51 libnvidia-eglcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-eglcore.so.590.48.01
lrwxrwxrwx 1 root root   47 Feb 18 10:51 libnvidia-egl-wayland.so.1 -> /.singularity.d/libs/libnvidia-egl-wayland.so.1
lrwxrwxrwx 1 root root   40 Feb 18 10:51 libnvidia-encode.so -> /.singularity.d/libs/libnvidia-encode.so
lrwxrwxrwx 1 root root   42 Feb 18 10:51 libnvidia-encode.so.1 -> /.singularity.d/libs/libnvidia-encode.so.1
lrwxrwxrwx 1 root root   39 Feb 18 10:51 libnvidia-fbc.so.1 -> /.singularity.d/libs/libnvidia-fbc.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:51 libnvidia-glcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-glcore.so.590.48.01
lrwxrwxrwx 1 root root   48 Feb 18 10:51 libnvidia-glsi.so.590.48.01 -> /.singularity.d/libs/libnvidia-glsi.so.590.48.01
lrwxrwxrwx 1 root root   53 Feb 18 10:51 libnvidia-glvkspirv.so.590.48.01 -> /.singularity.d/libs/libnvidia-glvkspirv.so.590.48.01
lrwxrwxrwx 1 root root   51 Feb 18 10:51 libnvidia-gpucomp.so.590.48.01 -> /.singularity.d/libs/libnvidia-gpucomp.so.590.48.01
lrwxrwxrwx 1 root root   48 Feb 18 10:51 libnvidia-gtk3.so.590.48.01 -> /.singularity.d/libs/libnvidia-gtk3.so.590.48.01
lrwxrwxrwx 1 root root   38 Feb 18 10:51 libnvidia-ml.so.1 -> /.singularity.d/libs/libnvidia-ml.so.1
lrwxrwxrwx 1 root root   40 Feb 18 10:51 libnvidia-nvvm.so.4 -> /.singularity.d/libs/libnvidia-nvvm.so.4
lrwxrwxrwx 1 root root   42 Feb 18 10:51 libnvidia-opencl.so.1 -> /.singularity.d/libs/libnvidia-opencl.so.1
lrwxrwxrwx 1 root root   47 Feb 18 10:51 libnvidia-opticalflow.so.1 -> /.singularity.d/libs/libnvidia-opticalflow.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:51 libnvidia-ptxjitcompiler.so.1 -> /.singularity.d/libs/libnvidia-ptxjitcompiler.so.1
lrwxrwxrwx 1 root root   50 Feb 18 10:51 libnvidia-rtcore.so.590.48.01 -> /.singularity.d/libs/libnvidia-rtcore.so.590.48.01
lrwxrwxrwx 1 root root   47 Feb 18 10:51 libnvidia-tls.so.590.48.01 -> /.singularity.d/libs/libnvidia-tls.so.590.48.01
lrwxrwxrwx 1 root root   36 Feb 18 10:51 libnvoptix.so.1 -> /.singularity.d/libs/libnvoptix.so.1
lrwxrwxrwx 1 root root   35 Feb 18 10:51 libOpenCL.so.1 -> /.singularity.d/libs/libOpenCL.so.1
lrwxrwxrwx 1 root root   33 Feb 18 10:51 libOpenGL.so -> /.singularity.d/libs/libOpenGL.so
lrwxrwxrwx 1 root root   35 Feb 18 10:51 libOpenGL.so.0 -> /.singularity.d/libs/libOpenGL.so.0
...
ls -al /opt/eessi/2023.06/compat/linux/x86_64/lib
lrwxrwxrwx 1 root root 61 Feb 18 10:51 /opt/eessi/2023.06/compat/linux/x86_64/lib -> /cvmfs/software.eessi.io/host_injections/nvidia/x86_64/latest
ls -al /cvmfs/software.eessi.io/host_injections/nvidia/x86_64/latest
lrwxrwxrwx 1 root root 4 Feb 18 10:51 /cvmfs/software.eessi.io/host_injections/nvidia/x86_64/latest -> host

Looks as expected!

Copy link
Contributor

@bedroge bedroge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested it for both versions a few more times in the EESSI container, and all my attempts worked fine. Thanks @casparvl !

@bedroge
Copy link
Contributor

bedroge commented Feb 18, 2026

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx

@eessi-bot-deucalion
Copy link

eessi-bot-deucalion bot commented Feb 18, 2026

New job on instance eessi-bot-deucalion for repository eessi.io-2023.06-software
Building on: a64fx
Building for: aarch64/a64fx
Job dir: /home/eessibot/new-bot/jobs/2026.02/pr_158/972316

date job status comment
Feb 18 12:45:39 UTC 2026 submitted job id 972316 awaits release by job manager
Feb 18 12:46:26 UTC 2026 released job awaits launch by Slurm scheduler
Feb 18 12:47:29 UTC 2026 running job 972316 is running
Feb 18 12:56:55 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-972316.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2023.06-software-linux-aarch64-a64fx-17714189750.tar.zstsize: 0 MiB (9673 bytes)
entries: 1
modules under 2023.06/software/linux/aarch64/a64fx/modules/all
no module files in tarball
software under 2023.06/software/linux/aarch64/a64fx/software
no software packages in tarball
reprod directories under 2023.06/software/linux/aarch64/a64fx/reprod
no reprod directories in tarball
other under 2023.06/software/linux/aarch64/a64fx
2023.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh
Feb 18 12:56:55 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] ( 1/10) Skipping test: nodes in this partition only have 30720 MiB memory available (per node) according to the current ReFrame configuration, but 49152 MiB is needed
[ SKIP ] ( 2/10) Skipping test: nodes in this partition only have 30720 MiB memory available (per node) according to the current ReFrame configuration, but 49152 MiB is needed
[ SKIP ] ( 3/10) Skipping test: nodes in this partition only have 30720 MiB memory available (per node) according to the current ReFrame configuration, but 49152 MiB is needed
[ SKIP ] ( 4/10) Skipping test: nodes in this partition only have 30720 MiB memory available (per node) according to the current ReFrame configuration, but 49152 MiB is needed
[ OK ] ( 5/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:a64fx+default
P: perf: 578.984 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 6/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:a64fx+default
P: perf: 576.458 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:a64fx+default
P: latency: 1.65 us (r:0, l:None, u:None)
[ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:a64fx+default
P: latency: 1.69 us (r:0, l:None, u:None)
[ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:a64fx+default
P: bandwidth: 8629.32 MB/s (r:0, l:None, u:None)
[ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:a64fx+default
P: bandwidth: 8728.7 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 6/10 test case(s) from 10 check(s) (0 failure(s), 4 skipped, 0 aborted)
Details
✅ job output file slurm-972316.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Feb 18 13:15:39 UTC 2026 uploaded transfer of eessi-2023.06-software-linux-aarch64-a64fx-17714189750.tar.zst to S3 bucket succeeded

@eessi-bot-deucalion
Copy link

eessi-bot-deucalion bot commented Feb 18, 2026

New job on instance eessi-bot-deucalion for repository eessi.io-2025.06-software
Building on: a64fx
Building for: aarch64/a64fx
Job dir: /home/eessibot/new-bot/jobs/2026.02/pr_158/972317

date job status comment
Feb 18 12:45:45 UTC 2026 submitted job id 972317 awaits release by job manager
Feb 18 12:46:23 UTC 2026 released job awaits launch by Slurm scheduler
Feb 18 12:47:32 UTC 2026 running job 972317 is running
Feb 18 12:52:47 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-972317.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-aarch64-a64fx-17714189120.tar.zstsize: 0 MiB (9673 bytes)
entries: 1
modules under 2025.06/software/linux/aarch64/a64fx/modules/all
no module files in tarball
software under 2025.06/software/linux/aarch64/a64fx/software
no software packages in tarball
reprod directories under 2025.06/software/linux/aarch64/a64fx/reprod
no reprod directories in tarball
other under 2025.06/software/linux/aarch64/a64fx
2025.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh
Feb 18 12:52:47 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:a64fx+default [Skipping test: nodes in this partition only have 30720 MiB memory available (per node) according to the current ReFrame configuration, but 49152 MiB is needed]
[ SKIP ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:a64fx+default [Skipping test: nodes in this partition only have 30720 MiB memory available (per node) according to the current ReFrame configuration, but 49152 MiB is needed]
[ OK ] (3/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:a64fx+default
P: latency: 0.87 us (r:0, l:None, u:None)
[ OK ] (4/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:a64fx+default
P: bandwidth: 7989.26 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 2/4 test case(s) from 4 check(s) (0 failure(s), 2 skipped, 0 aborted)
Details
✅ job output file slurm-972317.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Feb 18 13:15:48 UTC 2026 uploaded transfer of eessi-2025.06-software-linux-aarch64-a64fx-17714189120.tar.zst to S3 bucket succeeded

@casparvl casparvl added the enhancement New feature or request label Feb 18, 2026
@bedroge
Copy link
Contributor

bedroge commented Feb 18, 2026

Staging PR merged.

@bedroge bedroge merged commit 8ad1a7c into EESSI:main Feb 18, 2026
69 checks passed
@boegel
Copy link
Contributor

boegel commented Feb 18, 2026

@casparvl Do we need to update docs accordingly @ https://eessi.io/docs/site_specific_config/gpu/#exposing-nvidia-gpu-drivers ?

@bedroge
Copy link
Contributor

bedroge commented Feb 18, 2026

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:deploy enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments