You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/rpc.md
+30-12Lines changed: 30 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,13 @@
1
1
# Building and Using the RPC Server with `stable-diffusion.cpp`
2
2
3
-
This guide covers how to build a version of the RPC server from `llama.cpp` that is compatible with your version of `stable-diffusion.cpp` to manage multi-backends setups. RPC allows you to offload specific model components to a remote server.
3
+
This guide covers how to build a version of [the RPC server from `llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md) that is compatible with your version of `stable-diffusion.cpp` to manage multi-backends setups. RPC allows you to offload specific model components to a remote server.
4
4
5
5
> **Note on Model Location:** The model files (e.g., `.safetensors` or `.gguf`) remain on the **Client** machine. The client parses the file and transmits the necessary tensor data and computational graphs to the server. The server does not need to store the model files locally.
6
6
7
7
## 1. Building `stable-diffusion.cpp` with RPC client
8
8
9
9
First, you should build the client application from source. It requires `GGML_RPC=ON` to include the RPC backend to your client.
10
+
10
11
```bash
11
12
mkdir build
12
13
cd build
@@ -16,7 +17,7 @@ cmake .. \
16
17
cmake --build . --config Release -j $(nproc)
17
18
```
18
19
19
-
> **Note:** Ensure you add the other flags you would normally use (e.g., `-DSD_VULKAN=ON`, `-DSD_CUDA=ON`, `-DSD_HIPBLAS=ON`, or `-DGGML_METAL=ON`), for more information about building `stable-diffusion.cpp` from source, please refer to the `build.md` documentation.
20
+
> **Note:** Ensure you add the other flags you would normally use (e.g., `-DSD_VULKAN=ON`, `-DSD_CUDA=ON`, `-DSD_HIPBLAS=ON`, or `-DGGML_METAL=ON`), for more information about building `stable-diffusion.cpp` from source, please refer to the [build.md](build.md) documentation.
To save on download time and storage, you can use a shallow clone to download only the target commit:
44
+
To save on download time and storage, you can use a shallow clone to download only the target commit:
44
45
```bash
45
46
mkdir -p llama.cpp
46
47
cd llama.cpp
@@ -54,15 +55,16 @@ To save on download time and storage, you can use a shallow clone to download on
54
55
55
56
The RPC server acts as the worker. You must explicitly enable the **backend** (the hardware interface, such as CUDA for Nvidia, Metal for Apple Silicon, or Vulkan) when building, otherwise the server will default to using only the CPU.
56
57
57
-
To find the correct flags, refer to the official documentation for the `llama.cpp` repository.
58
+
To find the correct flagsfor your system, refer to the official documentation for the [`llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) repository.
58
59
59
60
>**Crucial:** You must include the compiler flags required to satisfy the API compatibility with `stable-diffusion.cpp` (`-DGGML_MAX_NAME=128`). Without this flag, `GGML_MAX_NAME` will default to `64`for the server, and data transfers between the client and server will fail. Of course, `-DGGML_RPC` must also be enabled.
60
61
>
61
62
> I recommend disabling the `LLAMA_CURL` flag to avoid unnecessary dependencies, and disabling shared library builds to avoid potential conflicts.
62
63
63
-
>**Build Target:** We are specifically building the `rpc-server` target. This prevents the build system from compiling the entire `llama.cpp` suite (like `llama-cli`), making the build significantly faster.
64
+
>**Build Target:** We are specifically building the `rpc-server` target. This prevents the build system from compiling the entire `llama.cpp` suite (like `llama-server`), making the build significantly faster.
@@ -112,10 +116,13 @@ Start the server. It listens for connections on the default address (usually `lo
112
116
113
117
**On the Server :**
114
118
If running on the same machine, you can use the default address:
119
+
115
120
```bash
116
121
./rpc-server
117
122
```
123
+
118
124
If you want to allow connections from other machines on the network:
125
+
119
126
```bash
120
127
./rpc-server --host 0.0.0.0
121
128
```
@@ -129,13 +136,16 @@ If you want to allow connections from other machines on the network:
129
136
We're assuming the server is running on your local machine, and listening on the default port `50052`. If it's running on a different machine, you can replace `localhost` with the IP address of the server.
130
137
131
138
**On the Client:**
139
+
132
140
```bash
133
141
./sd-cli --rpc localhost:50052 --list-devices
134
142
```
143
+
135
144
If the server is running and the client is able to connect, you should see `RPC0 localhost:50052`in the list of devices.
136
145
137
-
Example output:
146
+
Example output:
138
147
(Client built without GPU acceleration, two GPUs available on the server)
148
+
139
149
```
140
150
List of available GGML devices:
141
151
Name Description
@@ -166,23 +176,31 @@ Example: A main machine (192.168.1.10) with 3 GPUs, with one GPU running CUDA an
166
176
**On the first machine (Running two server instances):**
@@ -199,4 +217,4 @@ The client will map these servers to sequential device IDs (e.g., RPC0 from the
199
217
200
218
## 6. Performance Considerations
201
219
202
-
RPC performance is heavily dependent on network bandwidth, as large weights and activations must be transferred back and forth over the network, especially for large models, or when using high resolutions. For best results, ensure your network connection is stable and has sufficient bandwidth (>1Gbps recommended).
220
+
RPC performance is heavily dependent on network bandwidth, as large weights and activations must be transferred back and forth over the network, especially for large models, or when using high resolutions. For best results, ensure your network connection is stable and has sufficient bandwidth (>1Gbps recommended).
0 commit comments