[BUG] EXL3 reserve allocations ignored for device IDs > 0

[figuernd]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.12

Describe the bug

Autosplit reserve (and the equivalent for tensor parallel) ONLY works for device ID 0.

Reproduction steps

In a system with multiple GPUs try something like autosplit_reserve: [1024,1024]. Device ID 1 will not reserve anything.

Expected behavior

Should reserve for each device as described in the documentation.

Logs

No response

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[frenzybiscuit]

Cannot reproduce

(venv) tabbyapi@llm:~/tabbyapi/tabbyAPI$ ./start.sh
Activating venv
pip 24.0 from /home/tabbyapi/tabbyapi/tabbyAPI/venv/lib/python3.12/site-packages/pip (python 3.12)
Loaded your saved preferences from `start_options.json`
Starting TabbyAPI...
2025-12-26 12:58:43.542 INFO:     Using backend exllamav3
2025-12-26 12:58:43.544 INFO:     exllamav3 version: 0.0.18
2025-12-26 12:58:43.764 WARNING:  The provided model does not have vision capabilities that are supported by ExllamaV3. Vision input is disabled.
2025-12-26 12:58:43.765 WARNING:  Draft model is disabled because a model name wasn't provided. Please check your config.yml!
2025-12-26 12:58:43.766 INFO:     Attempting to load a prompt template if present.
2025-12-26 12:58:43.769 INFO:     Using template "chat_template" for chat completions.
2025-12-26 12:58:43.771 INFO:     Loading model: /home/tabbyapi/tabbyapi/exl3/MS2411-123B-Animus-Redux-V12.0-4.25BPW-H16-EXL3
2025-12-26 12:58:43.771 INFO:     Loading with tensor parallel
 -- NCCL init: world_size 4, rank 3, device 0, init_method tcp://127.0.0.1:33567
 -- NCCL init: world_size 4, rank 0, device 1, init_method tcp://127.0.0.1:33567
 -- NCCL warmup, device 0, please wait...
 -- NCCL init: world_size 4, rank 1, device 2, init_method tcp://127.0.0.1:33567
 -- NCCL warmup, device 2, please wait...
 -- NCCL init: world_size 4, rank 2, device 3, init_method tcp://127.0.0.1:33567
 -- NCCL warmup, device 3, please wait...
 -- NCCL warmup, device 1, please wait...
 -- Finished NCCL warmup, device 2
 -- Finished NCCL warmup, device 0
 -- Finished NCCL warmup, device 1
 -- Finished NCCL warmup, device 3
2025-12-26 12:59:29.034 INFO:     Model successfully loaded.
Loading model modules ���������������������������������������� 100% 91/91 0:00:00
2025-12-26 12:59:29.037 INFO:     Your API key is: removed
2025-12-26 12:59:29.037 INFO:     Your admin key is: removed
2025-12-26 12:59:29.037 INFO:
2025-12-26 12:59:29.037 INFO:     If these keys get compromised, make sure to delete api_tokens.yml and restart the server. Have fun!
2025-12-26 12:59:29.038 INFO:     Generation logging is disabled
2025-12-26 12:59:29.038 INFO:     Applied sampler overrides from file.
2025-12-26 12:59:29.039 INFO:     Developer documentation: http://192.168.11.3:5000/redoc
2025-12-26 12:59:29.042 INFO:     Starting OAI API
2025-12-26 12:59:29.042 INFO:     Completions: http://192.168.11.3:5000/v1/completions
2025-12-26 12:59:29.043 INFO:     Chat completions: http://192.168.11.3:5000/v1/chat/completions
2025-12-26 12:59:29.064 INFO:     Started server process [79315]
2025-12-26 12:59:29.065 INFO:     Waiting for application startup.
2025-12-26 12:59:29.065 INFO:     Application startup complete.
2025-12-26 12:59:29.066 INFO:     Uvicorn running on http://192.168.11.3:5000 (Press CTRL+C to quit)

Using the following:

  tensor_parallel: true

  # Sets a backend type for tensor parallelism. (default: native).
  # Options: native, nccl
  # Native is recommended for PCIe GPUs
  # NCCL is recommended for NVLink.
  tensor_parallel_backend: nccl

  # Automatically allocate resources to GPUs (default: True).
  # Not parsed for single GPU users.
  gpu_split_auto: true

  # Reserve VRAM used for autosplit loading (default: 96 MB on GPU 0).
  # Represented as an array of MB per GPU.
  autosplit_reserve: [256, 256, 256, 256]

  # An integer array of GBs of VRAM to split between GPUs (default: []).
  # Used with tensor parallelism.
  gpu_split: [24, 24, 24, 24]

[figuernd]

Disable tensor parallel and change autosplit reserve to [256, 256, 256, 24576] - assuming you have 24GB in device 3, it will ignore the reservation still use device 3.

[CNmnius]

I'm not sure if it is relevant, but found a weird behavior about autosplit_reserve settings.

GPU0: 5070Ti, GPU1: 4070TiS.
Windows 11, NVIDIA-SMI CUDA 13.1, NVCC CUDA 12.8

• default setting: autosplit_reserve: [96]
  When I load Mistral-Small-3.2-24B-Instruct-2506-exl3_4.0bpw with default settings (TP disabled), the program does not seem equally divide the model on both GPU. Using more VRAM on GPU1.

• alternative setting 1: autosplit_reserve: [96, 10240]
  I intended to utilize more VRAM on GPU0. However the result is same as the default setting.

• alternative setting 2: autosplit_reserve: [10240, 96]
  This setting works as I intended, using up 15 GB of VRAM on GPU0 and 5 GB on GPU1.