GPU Status not reporting properly (despite GPU being used : I can see it in jtop)

[Bigsausage64]

LocalAI version:

LocalAI v3.12.1 installed in docker
image used : quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64

Environment, CPU architecture, OS, and Version:

Running on a Jetson Orin Nano 8 gb
uname gives "Linux 5.15.185-tegra #1 SMP PREEMPT Thu Jan 15 19:24:38 PST 2026 aarch64 aarch64 aarch64 GNU/Linux"

Describe the bug

GPU status is showing the name of the GPU "Orin (nvgpu)", but usage stays at 0, RAM usage also stays at 0, and the total amount of memory is also 0

To Reproduce

Install LocalAI in docker and start using models. GPU info stays at 0 despite GPU being used. I can see the GPU and RAM activity in jtop, no problem. I followed the setup procedure found here : https://localai.io/reference/nvidia-l4t/ and I built the container myself

Expected behavior

GPU usage ands RAM usage should report properly

Logs

Additional context

[localai-bot]

Thank you for reporting this GPU status issue on your Jetson Orin Nano setup.

To help debug why the GPU metrics are showing 0 despite the GPU being active, could you please provide:

1. Debug logs: Run LocalAI with flag or set environment variable and share any GPU-related log entries

2. Custom container details: You mentioned building the container yourself - could you share:

   • The Dockerfile or build command used
   • Any modifications made to the official quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64 image

3. GPU detection verification:

4. LocalAI startup command: The exact command or docker-compose configuration you're using

This appears to be a platform-specific configuration issue with NVIDIA L4T on Jetson. The fact that jtop shows activity confirms the GPU is working - the issue is likely in how LocalAI queries the GPU metrics on this specific platform.

Once we have these details, we can determine if this is:

• A custom container build issue (missing GPU monitoring libraries)
• An NVIDIA L4T specific configuration problem
• A bug in LocalAI's GPU detection code for Jetson platforms

[localai-bot]

Thanks for reporting this GPU status issue on Jetson Orin Nano.

To help diagnose this, could you please provide:

1. Debug logs: Run LocalAI with --debug flag or set DEBUG=true environment variable and share the relevant GPU-related log output

2. Docker command: The exact docker run command you're using to start LocalAI

3. Container inspection: Output of nvidia-smi or jtop from inside the container to confirm GPU visibility

4. LocalAI version: You mentioned v3.12.1, but could you confirm with localai --version from inside the container?

This will help us determine if this is a container configuration issue, a GPU detection problem, or a bug in the status reporting code.

[Bigsausage64]

Hi, please find below some details

1. Debug is activated since the beginning of my testing. The log file is massive, I'll try to clean it a bit

2. I didn't change anything to the image and I used the command provided to build the container:

 
git clone https://github.com/mudler/LocalAI

cd LocalAI

docker build --build-arg SKIP_DRIVERS=true --build-arg BUILD_TYPE=cublas --build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r36.4.0 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-core .

3. To run the container, I'm also using the command provided on localai.io website:

docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64

4. I'm using version3.12.1
   root@35a9c3c78a04:/# ./local-ai --version v3.12.1

5. here is the outpu of nvidia-smi:

root@35a9c3c78a04:/# nvidia-smi 
Tue Mar 10 21:01:27 2026       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 540.5.0                Driver Version: 540.5.0      CUDA Version: 12.6     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Orin (nvgpu)                  N/A  | N/A              N/A |                  N/A |
| N/A   N/A  N/A               N/A /  N/A | Not Supported        |     N/A          N/A |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

To be fair, nvidia-smi is doing this even after a fresh installation of Jetson Orin OS.