You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -12,60 +12,79 @@ Since 2021 the development of built-in ClickHouse® alternative for Zookeeper is
12
12
13
13
See slides: https://presentations.clickhouse.com/meetup54/keeper.pdf and video https://youtu.be/IfgtdU1Mrm0?t=2682
14
14
15
-
## Current status (last updated: July 2023)
15
+
## Current status (last updated: March 2026)
16
16
17
-
Since version 23.3 we recommend using clickhouse-keeper for new installations.
17
+
ClickHouse Keeper is the recommended choice for new installations. It yields better performance in many cases due to the new features, like async replication or multi read. Some ClickHouse server features cannot be used without Keeper, for example the S3Queue.
18
18
19
-
Even better if you will use the latest version of clickhouse-keeper (currently it's 23.7), and it's not necessary to use the same version of clickhouse-keeper as ClickHouse itself.
19
+
- Use the latest Keeper version available in your supported upgrade path whenever possible.
20
+
- The Keeper version doesn’t need to match the ClickHouse server version
21
+
- Modern Keeper usually performs better than older versions because the codebase has matured significantly, new protocol feature flags have been added, and internal replication has improved.
20
22
21
23
For existing systems that currently use Apache Zookeeper, you can consider upgrading to clickhouse-keeper especially if you will [upgrade ClickHouse](https://altinity.com/clickhouse-upgrade-overview/) also.
22
24
23
-
But please remember that on very loaded systems the change can give no performance benefits or can sometimes lead to a worse performance.
25
+
{{% alert title="Warning" color="warning" %}}
26
+
Before upgrading ClickHouse Keeper from version older than 23.9 please check Upgrade caveat for async_replication [Upgrade caveat for async_replication](https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-zookeeper/clickhouse-keeper#upgrade-caveat-for-async_replication)
27
+
{{% /alert %}}
24
28
25
-
The development pace of keeper code is [still high](https://github.com/ClickHouse/ClickHouse/pulls?q=is%3Apr+keeper)
26
-
so every new version should bring improvements / cover the issues, and stability/maturity grows from version to version, so
27
-
if you want to play with clickhouse-keeper in some environment - please use [the most recent ClickHouse releases](https://altinity.com/altinity-stable/)! And of course: share your feedback :)
29
+
## How does clickhouse-keeper differ from Zookeeper?
28
30
29
-
## How does clickhouse-keeper work?
31
+
Keeper is optimized for ClickHouse workloads and written in C++ (and can be used as single-binary), so it don't need any external dependencies. It uses the same **client** protocol but both are implementing different consensus protocol: Zookeeper is using ZAB, while ClickHouse Keeper implements eBay NuRAFT [GitHub - eBay/NuRaft: C++ implementation of Raft core logic as a replication library](https://github.com/eBay/NuRaft) which improves stability and performance of base RAFT protocol.
30
32
31
-
Official docs: https://clickhouse.com/docs/en/guides/sre/keeper/clickhouse-keeper/
33
+
ClickHouse Keeper can also run in embedded mode, operating as a separate thread within the ClickHouse server process, which may be suitable for testing purposes or smaller instances where some performance can be sacrificed for simplicity
32
34
33
-
ClickHouse-keeper still need to be started additionally on few nodes (similar to 'normal' zookeeper) and speaks normal zookeeper protocol - needed to simplify A/B tests with real zookeeper.
35
+
## Migration and upgrade guide
34
36
35
-
To test that you need to run 3 instances of clickhouse-server (which will mimic zookeeper) with an extra config like that:
37
+
- A mixed ZooKeeper / ClickHouse Keeper quorum is not supported. Those are different consensus protocols.
38
+
- ZooKeeper snapshots and transaction logs are not format-compatible with Keeper. For data migration use `clickhouse-keeper-converter`.
39
+
- If the above is too complex you can switch to new, empty Keeper ensemble and recreate the Keeper metadata using `SYSTEM RESTORE REPLICA` calls. This method takes longer time but it is suitable for smaller clusters. Check [procedure to restore multiple tables in RO mode article](https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-check-replication-ddl-queue/#procedure-to-restore-multiple-tables-in-read-only-mode-per-replica)
40
+
- Keep in mind that some metadata is available in ZooKeeper only and will be lost if you don't migrate with clickhouse-keeper-converter using above guide. For example: Distributed DDL queue, RBAC data (if configured), etc. Check [Keeper depended features](https://kb.altinity.com/altinity-kb-setup-and-maintenance/keeper-dependent-features) for more information.
`async_replication` is an internal Keeper optimization for RAFT replication and it's turned on by default starting from [25.10](https://github.com/ClickHouse/ClickHouse/pull/88515) . It does not change ClickHouse replicated table semantics, but it can improve Keeper performance.
40
45
41
-
or event single instance with config like that: [https://github.com/ClickHouse/ClickHouse/blob/master/tests/config/config.d/keeper_port.xml](https://github.com/ClickHouse/ClickHouse/blob/master/tests/config/config.d/keeper_port.xml)
If you upgrade directly from a version older than `23.9` to `25.10+`:
43
47
44
-
And point all the ClickHouses (zookeeper config section) to those nodes / ports.
48
+
- either upgrade Keeper to `23.9+` first, and then continue to `25.10+`
49
+
- or temporarily set `keeper_server.coordination_settings.async_replication=0` during the upgrade and enable it after the upgrade is finished
45
50
46
-
Latest version is recommended (even testing / master builds). We will be thankful for any feedback.
51
+
### Keeper in kubernetes
52
+
53
+
If you run ClickHouse on Kubernetes with Altinity operator, Keeper can be managed as a dedicated `ClickHouseKeeperInstallation` resource (often abbreviated as CHK). That is usually the cleanest way to run and upgrade a separate Keeper ensemble on Kubernetes. Please check examples [here](https://github.com/Altinity/clickhouse-operator/blob/master/docs/chk-examples/01-chi-simple-with-keeper.yaml).
See https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-zookeeper/clickhouse-keeper-initd/
62
+
63
+
## More than 3 Keeper nodes
64
+
65
+
The main issue with a larger Keeper ensemble is that it takes more time to re-elect a leader, and commits take longer, which can slow down insertions and DDL queries.
66
+
67
+
It should be fine, but we don’t recommend running more than three Keeper nodes (excluding observers).
68
+
69
+
Increasing the number of nodes offers no significant advantages (unless you need to tolerate the simultaneous failure of two Keeper nodes). In terms of performance, it doesn’t perform better—and may even perform worse—and it consumes additional resources (ZooKeeper requires fast, dedicated disks to perform well, as well as some RAM and CPU).
57
70
58
-
## Example of a simple cluster with 2 nodes of ClickHouse using built-in keeper
71
+
## clickhouse-keeper-client
59
72
60
-
For example you can start two ClickHouse nodes (hostname1, hostname2)
73
+
In clickhouse-keeper-client, paths are now parsed more strictly and must be passed as string literals. In practice, this means using single quotes around paths—for example, ls '/' instead of ls /, and get '/clickhouse/path' instead of get /clickhouse/path.
74
+
75
+
## Example of a simple cluster
76
+
77
+
The Keeper ensemble size must be odd because it requires a majority (50% + 1 nodes) to form a quorum. A 2-node Keeper setup will lose quorum after a single node failure, so the recommended number of Keeper replicas is 3.
78
+
79
+
On `hostname1` and `hostname2` below, ClickHouse can use the embedded Keeper cluster from `<keeper_server>`, so a separate client-side `<keeper>` section is not required. If your ClickHouse servers connect to an external Keeper or ZooKeeper ensemble, see [ClickHouse config for Keeper]({{< ref "clickhouse-keeper-clickhouse-config" >}}).
0 commit comments