From 05143cb35e9d128505d84a26a414876d65f43e65 Mon Sep 17 00:00:00 2001 From: Ivan Orekhov Date: Mon, 9 Feb 2026 12:23:02 +0300 Subject: [PATCH 1/8] fix: 1st version of ldaps docs --- .../admin-manual/auth/authentication/ldap.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md index 464e5c9b52fcd..15c83ae59d8de 100644 --- a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md +++ b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md @@ -49,13 +49,22 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ``` ldap_authentication_enabled = true ldap_host = ladp-host - ldap_port = 389 + ldap_port = 389 # change it if ldap_use_ssl specified to true as different port (636) is used for LDAPS ldap_admin_name = uid=admin,o=emr ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr + ldap_use_ssl = true # specify true to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP ``` +> Important for LDAPS: +> When `ldap_use_ssl = true`, ensure your LDAP server certificate is trusted by the Doris FE JVM. +> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): +> ``` +> # Example for JDK 17 +> JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +> ``` + 3. After starting `fe`, log in to Doris with `root` or `admin` account and set the LDAP admin password: ```sql @@ -250,7 +259,9 @@ You can refresh the cache with the `refresh ldap` statement. See [REFRESH-LDAP]( ## Known Limitations -- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, and between `fe` and LDAP service. +- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, but connection between `fe` and LDAP service can be secured optionally. +- **For `ldap_use_ssl = false` (default behavior)**: Passwords are transmitted in plain text between the Doris FE and the LDAP server. +- **To secure the connection**: Set `ldap_use_ssl = true` to encrypt traffic between Doris FE and the LDAP server. Note that SSL/TLS encryption between the **client and Doris FE** must be configured separately. ## FAQ From 4a9b9b75d4441107cc65c7c73ac3bf5c74ec1b1b Mon Sep 17 00:00:00 2001 From: Ivan Orekhov Date: Mon, 9 Feb 2026 12:30:10 +0300 Subject: [PATCH 2/8] fix: improved version of ldaps docs --- .../admin-manual/auth/authentication/ldap.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md index 15c83ae59d8de..06692107ceb1a 100644 --- a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md +++ b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md @@ -49,17 +49,23 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ``` ldap_authentication_enabled = true ldap_host = ladp-host - ldap_port = 389 # change it if ldap_use_ssl specified to true as different port (636) is used for LDAPS + # change ldap_port value if ldap_use_ssl specified to true as different port (636) is used for LDAPS + ldap_port = 389 ldap_admin_name = uid=admin,o=emr ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr - ldap_use_ssl = true # specify true to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP + # specify ldap_use_ssl to true to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP + ldap_use_ssl = true ``` > Important for LDAPS: +> > When `ldap_use_ssl = true`, ensure your LDAP server certificate is trusted by the Doris FE JVM. -> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): +> +> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. +> +> Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): > ``` > # Example for JDK 17 > JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." From e4bf2ace8ace08781b97b75243e74968cadaae7d Mon Sep 17 00:00:00 2001 From: Ivan Orekhov Date: Mon, 9 Feb 2026 12:39:17 +0300 Subject: [PATCH 3/8] fix: final version of DORIS documentation --- docs/admin-manual/auth/authentication/ldap.md | 21 +++++++++++++++++-- .../admin-manual/auth/authentication/ldap.md | 21 +++++++++++++++++-- .../admin-manual/auth/authentication/ldap.md | 2 +- 3 files changed, 39 insertions(+), 5 deletions(-) diff --git a/docs/admin-manual/auth/authentication/ldap.md b/docs/admin-manual/auth/authentication/ldap.md index 464e5c9b52fcd..9a05239d10abd 100644 --- a/docs/admin-manual/auth/authentication/ldap.md +++ b/docs/admin-manual/auth/authentication/ldap.md @@ -49,13 +49,28 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ``` ldap_authentication_enabled = true ldap_host = ladp-host - ldap_port = 389 + # change ldap_port value if ldap_use_ssl specified to true as different port (636) is used for LDAPS + ldap_port = 389 ldap_admin_name = uid=admin,o=emr ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr + # specify ldap_use_ssl to use to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP + ldap_use_ssl = true ``` +> Important for LDAPS: +> +> When `ldap_use_ssl = true`, ensure your LDAP server certificate is trusted by the Doris FE JVM. +> +> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. +> +> Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): +> ``` +> # Example for JDK 17 +> JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +> ``` + 3. After starting `fe`, log in to Doris with `root` or `admin` account and set the LDAP admin password: ```sql @@ -250,7 +265,9 @@ You can refresh the cache with the `refresh ldap` statement. See [REFRESH-LDAP]( ## Known Limitations -- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, and between `fe` and LDAP service. +- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, but connection between `fe` and LDAP service can be secured optionally. +- **For `ldap_use_ssl = false` (default behavior)**: Passwords are transmitted in plain text between the Doris FE and the LDAP server. +- **To secure the connection**: Set `ldap_use_ssl = true` to encrypt traffic between Doris FE and the LDAP server. Note that SSL/TLS encryption between the **client and Doris FE** must be configured separately. ## FAQ diff --git a/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md b/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md index 464e5c9b52fcd..fdcf429b3bf6e 100644 --- a/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md +++ b/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md @@ -49,13 +49,28 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ``` ldap_authentication_enabled = true ldap_host = ladp-host - ldap_port = 389 + # change ldap_port value if ldap_use_ssl specified to true as different port (636) is used for LDAPS + ldap_port = 389 ldap_admin_name = uid=admin,o=emr ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr + # specify ldap_use_ssl to true to use to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP + ldap_use_ssl = true ``` +> Important for LDAPS: +> +> When `ldap_use_ssl = true`, ensure your LDAP server certificate is trusted by the Doris FE JVM. +> +> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. +> +> Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): +> ``` +> # Example for JDK 17 +> JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +> ``` + 3. After starting `fe`, log in to Doris with `root` or `admin` account and set the LDAP admin password: ```sql @@ -250,7 +265,9 @@ You can refresh the cache with the `refresh ldap` statement. See [REFRESH-LDAP]( ## Known Limitations -- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, and between `fe` and LDAP service. +- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, but connection between `fe` and LDAP service can be secured optionally. +- **For `ldap_use_ssl = false` (default behavior)**: Passwords are transmitted in plain text between the Doris FE and the LDAP server. +- **To secure the connection**: Set `ldap_use_ssl = true` to encrypt traffic between Doris FE and the LDAP server. Note that SSL/TLS encryption between the **client and Doris FE** must be configured separately. ## FAQ diff --git a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md index 06692107ceb1a..fdcf429b3bf6e 100644 --- a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md +++ b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md @@ -55,7 +55,7 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr - # specify ldap_use_ssl to true to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP + # specify ldap_use_ssl to true to use to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP ldap_use_ssl = true ``` From 07f2ae658d08ecacf1b4ea9851c89b8c7133bf80 Mon Sep 17 00:00:00 2001 From: Ivan Orekhov Date: Mon, 16 Feb 2026 12:22:22 +0300 Subject: [PATCH 4/8] fix: fixed typo within description of ldap_use_ssl --- docs/admin-manual/auth/authentication/ldap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin-manual/auth/authentication/ldap.md b/docs/admin-manual/auth/authentication/ldap.md index 9a05239d10abd..99067ea5b1a3c 100644 --- a/docs/admin-manual/auth/authentication/ldap.md +++ b/docs/admin-manual/auth/authentication/ldap.md @@ -55,7 +55,7 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr - # specify ldap_use_ssl to use to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP + # specify ldap_use_sslto true to use to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP ldap_use_ssl = true ``` From da2f8fff11acd739f256d4902db9055897d16ca4 Mon Sep 17 00:00:00 2001 From: Ivan Orekhov Date: Mon, 16 Feb 2026 12:28:15 +0300 Subject: [PATCH 5/8] fix: typo on ldap_use_ssl description --- docs/admin-manual/auth/authentication/ldap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin-manual/auth/authentication/ldap.md b/docs/admin-manual/auth/authentication/ldap.md index 99067ea5b1a3c..9486166340b2b 100644 --- a/docs/admin-manual/auth/authentication/ldap.md +++ b/docs/admin-manual/auth/authentication/ldap.md @@ -55,7 +55,7 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr - # specify ldap_use_sslto true to use to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP + # specify ldap_use_ssl to true to use to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP ldap_use_ssl = true ``` From 602a7bad079a9e49c6aa7800f839daf6b0566b2b Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Mon, 30 Mar 2026 12:50:56 -0700 Subject: [PATCH 6/8] opt --- docs/admin-manual/auth/authentication/ldap.md | 79 ++++++++++++++----- .../admin-manual/auth/authentication/ldap.md | 64 ++++++++++++++- .../admin-manual/auth/authentication/ldap.md | 64 ++++++++++++++- .../admin-manual/auth/authentication/ldap.md | 64 ++++++++++++++- .../admin-manual/auth/authentication/ldap.md | 79 ++++++++++++++----- .../admin-manual/auth/authentication/ldap.md | 79 ++++++++++++++----- 6 files changed, 363 insertions(+), 66 deletions(-) diff --git a/docs/admin-manual/auth/authentication/ldap.md b/docs/admin-manual/auth/authentication/ldap.md index 9486166340b2b..00c5c38b23c66 100644 --- a/docs/admin-manual/auth/authentication/ldap.md +++ b/docs/admin-manual/auth/authentication/ldap.md @@ -2,7 +2,7 @@ { "title": "LDAP", "language": "en", - "description": "Detailed guide on Apache Doris federated authentication: unified identity verification and group authorization through LDAP integration, covering configuration steps, login methods, permission mapping rules, and common troubleshooting." + "description": "Detailed guide on Apache Doris federated authentication: unified identity verification and group authorization through LDAP/LDAPS integration, covering configuration steps, login methods, permission mapping rules, and common troubleshooting." } --- @@ -49,27 +49,14 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ``` ldap_authentication_enabled = true ldap_host = ladp-host - # change ldap_port value if ldap_use_ssl specified to true as different port (636) is used for LDAPS - ldap_port = 389 + ldap_port = 389 ldap_admin_name = uid=admin,o=emr ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr - # specify ldap_use_ssl to true to use to switch to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP - ldap_use_ssl = true ``` -> Important for LDAPS: -> -> When `ldap_use_ssl = true`, ensure your LDAP server certificate is trusted by the Doris FE JVM. -> -> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. -> -> Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): -> ``` -> # Example for JDK 17 -> JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." -> ``` + > To enable LDAPS (encrypted connection to the LDAP server), see the [LDAPS (Encrypted Connection)](#ldaps-encrypted-connection) section below. 3. After starting `fe`, log in to Doris with `root` or `admin` account and set the LDAP admin password: @@ -248,6 +235,52 @@ If user jack belongs to LDAP groups `doris_rd`, `doris_qa`, `doris_pm`, and Dori > - Which `group` a `user` belongs to is independent of the LDAP tree's organizational structure. `user2` in the example above doesn't necessarily belong to `group2`. > - To make `user2` belong to `group2`, you need to explicitly add `user2` to the `member` attribute of `group2`. +## LDAPS (Encrypted Connection) + +:::info Supported since version 4.0.5 +::: + +By default, Doris communicates with the LDAP server over plain LDAP (unencrypted). Starting from version 4.0.5, Doris supports LDAPS (LDAP over SSL/TLS) to encrypt the connection between Doris FE and the LDAP server. + +### Enabling LDAPS + +To enable LDAPS, add the following configuration to `fe/conf/ldap.conf`: + +``` +ldap_use_ssl = true +``` + +When `ldap_use_ssl` is set to `true`, Doris uses the `ldaps://` protocol to connect to the LDAP server. You should also update the port to the standard LDAPS port (typically `636`): + +``` +ldap_host = ldap-host +ldap_port = 636 +ldap_use_ssl = true +``` + +### Configuring Certificate Trust + +When using LDAPS, the LDAP server's SSL certificate must be trusted by the Doris FE JVM. If your LDAP server uses a certificate signed by a well-known public CA, no additional configuration is needed. + +If using a custom or self-signed Certificate Authority (CA), you must import the CA certificate into a Java trustStore and configure the JVM to use it. Add the following parameters to `JAVA_OPTS` in `fe/conf/fe.conf`: + +``` +# Example for JDK 17 +JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +``` + +> **Steps to import a self-signed CA certificate:** +> +> 1. Obtain the CA certificate file (e.g., `ca.crt`). +> 2. Import it into a Java trustStore using `keytool`: +> +> ```shell +> keytool -importcert -alias ldap-ca -keystore /path/to/your/cacerts -file /path/to/ca.crt -storepass changeit -noprompt +> ``` +> +> 3. Configure the trustStore path in `JAVA_OPTS` as shown above. +> 4. Restart the Doris FE for the changes to take effect. + ## Cache Management To avoid frequent access to LDAP services, Doris caches LDAP information in memory. @@ -265,9 +298,8 @@ You can refresh the cache with the `refresh ldap` statement. See [REFRESH-LDAP]( ## Known Limitations -- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, but connection between `fe` and LDAP service can be secured optionally. -- **For `ldap_use_ssl = false` (default behavior)**: Passwords are transmitted in plain text between the Doris FE and the LDAP server. -- **To secure the connection**: Set `ldap_use_ssl = true` to encrypt traffic between Doris FE and the LDAP server. Note that SSL/TLS encryption between the **client and Doris FE** must be configured separately. +- Doris's LDAP functionality only supports cleartext password verification for the client-to-FE segment, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`. SSL/TLS encryption between the client and Doris FE must be configured separately (see [Client Connection](#step-2-client-connection)). +- For the FE-to-LDAP segment, the connection is unencrypted by default (`ldap_use_ssl = false`). To encrypt this segment, set `ldap_use_ssl = true` to use LDAPS (see [LDAPS (Encrypted Connection)](#ldaps-encrypted-connection)). ## FAQ @@ -283,3 +315,12 @@ Check the following items step by step: 2. Check whether the expected `group` is located under the organizational structure corresponding to `ldap_group_basedn`. 3. Check whether the expected `group` contains the `member` attribute. 4. Check whether the `member` attribute of the expected `group` contains the current user's `dn`. + +**Q: LDAPS connection fails, how to troubleshoot?** + +Check the following items step by step: + +1. Confirm that `ldap_use_ssl = true` is set in `fe/conf/ldap.conf`. +2. Confirm that `ldap_port` is set to the correct LDAPS port (typically `636`). +3. Verify that the LDAP server's SSL certificate is trusted by the JVM. Check the `fe.log` for SSL handshake errors such as `SSLHandshakeException` or `PKIX path building failed`. +4. If using a self-signed CA, ensure the CA certificate has been imported into the trustStore and the trustStore path is correctly configured in `JAVA_OPTS`. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/authentication/ldap.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/authentication/ldap.md index a1edb207002c9..81f53bfa68463 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/authentication/ldap.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/authentication/ldap.md @@ -2,7 +2,7 @@ { "title": "LDAP", "language": "zh-CN", - "description": "详解 Apache Doris 联邦认证：通过集成 LDAP 实现统一身份验证与组授权，涵盖配置步骤、登录方式、权限映射规则及常见问题排查。" + "description": "详解 Apache Doris 联邦认证：通过集成 LDAP/LDAPS 实现统一身份验证与组授权，涵盖配置步骤、登录方式、权限映射规则及常见问题排查。" } --- @@ -44,7 +44,7 @@ Doris 支持接入第三方 LDAP 服务，提供验证登录和组授权两大 ### 第一步：配置 Doris 1. 在 `fe/conf/fe.conf` 中设置认证方式：`authentication_type=ldap`。 -2. 在 `fe/conf/ldap.conf` 中配置 LDAP 服务的连接信息。 +2. 在 `fe/conf/ldap.conf` 中配置 LDAP 服务的连接信息： ``` ldap_authentication_enabled = true @@ -56,6 +56,8 @@ Doris 支持接入第三方 LDAP 服务，提供验证登录和组授权两大 ldap_group_basedn = ou=group,o=emr ``` + > 如需启用 LDAPS（加密连接至 LDAP 服务器），请参阅下文 [LDAPS（加密连接）](#ldaps加密连接) 章节。 + 3. 启动 `fe` 后，使用 `root` 或 `admin` 账号登录 Doris，设置 LDAP 管理员密码： ```sql @@ -233,6 +235,52 @@ member: uid=jack,ou=aidp,dc=domain,dc=com > - `user` 属于哪个 `group` 与 LDAP 树的组织结构无关。上文示例中的 `user2` 并不一定属于 `group2`。 > - 若想让 `user2` 属于 `group2`，需要在 `group2` 的 `member` 属性中显式添加 `user2`。 +## LDAPS（加密连接） + +:::info 自 4.0.5 版本开始支持 +::: + +默认情况下，Doris 通过明文 LDAP 协议与 LDAP 服务器通信。自 4.0.5 版本起，Doris 支持 LDAPS（LDAP over SSL/TLS），用于加密 Doris FE 与 LDAP 服务器之间的连接。 + +### 启用 LDAPS + +在 `fe/conf/ldap.conf` 中添加以下配置以启用 LDAPS： + +``` +ldap_use_ssl = true +``` + +当 `ldap_use_ssl` 设置为 `true` 时，Doris 使用 `ldaps://` 协议连接 LDAP 服务器。同时应将端口更新为标准 LDAPS 端口（通常为 `636`）： + +``` +ldap_host = ldap-host +ldap_port = 636 +ldap_use_ssl = true +``` + +### 配置证书信任 + +使用 LDAPS 时，LDAP 服务器的 SSL 证书必须被 Doris FE 的 JVM 所信任。如果 LDAP 服务器使用的证书由知名公共 CA 签发，则无需额外配置。 + +如果使用自定义或自签名 CA，需要将 CA 证书导入 Java trustStore，并配置 JVM 使用该 trustStore。在 `fe/conf/fe.conf` 的 `JAVA_OPTS` 中添加以下参数： + +``` +# JDK 17 示例 +JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +``` + +> **导入自签名 CA 证书的步骤：** +> +> 1. 获取 CA 证书文件（例如 `ca.crt`）。 +> 2. 使用 `keytool` 将其导入 Java trustStore： +> +> ```shell +> keytool -importcert -alias ldap-ca -keystore /path/to/your/cacerts -file /path/to/ca.crt -storepass changeit -noprompt +> ``` +> +> 3. 按上述方式在 `JAVA_OPTS` 中配置 trustStore 路径。 +> 4. 重启 Doris FE 使配置生效。 + ## 缓存管理 为了避免频繁访问 LDAP 服务，Doris 会将 LDAP 信息缓存到内存中。 @@ -250,7 +298,8 @@ member: uid=jack,ou=aidp,dc=domain,dc=com ## 已知限制 -- 目前 Doris 的 LDAP 功能仅支持明文密码验证，即用户登录时，密码在 `client` 与 `fe` 之间、`fe` 与 LDAP 服务之间均以明文形式传输。 +- Doris 的 LDAP 功能在客户端到 FE 的链路上仅支持明文密码验证，即用户登录时，密码在 `client` 与 `fe` 之间以明文形式传输。客户端与 Doris FE 之间的 SSL/TLS 加密需要单独配置（详见[客户端连接](#第二步客户端连接)）。 +- FE 到 LDAP 服务器的链路默认使用明文传输（`ldap_use_ssl = false`）。如需加密该链路，请设置 `ldap_use_ssl = true` 以启用 LDAPS（详见 [LDAPS（加密连接）](#ldaps加密连接)）。 ## 常见问题 @@ -266,3 +315,12 @@ member: uid=jack,ou=aidp,dc=domain,dc=com 2. 检查预期的 `group` 是否位于 `ldap_group_basedn` 对应的组织结构下。 3. 检查预期的 `group` 是否包含 `member` 属性。 4. 检查预期 `group` 的 `member` 属性中是否包含当前用户的 `dn`。 + +**Q：LDAPS 连接失败，如何排查？** + +按以下步骤逐项检查： + +1. 确认 `fe/conf/ldap.conf` 中已设置 `ldap_use_ssl = true`。 +2. 确认 `ldap_port` 已设置为正确的 LDAPS 端口（通常为 `636`）。 +3. 检查 LDAP 服务器的 SSL 证书是否被 JVM 信任。查看 `fe.log` 中是否有 `SSLHandshakeException` 或 `PKIX path building failed` 等 SSL 握手错误。 +4. 如果使用自签名 CA，确认 CA 证书已导入 trustStore，且 `JAVA_OPTS` 中的 trustStore 路径配置正确。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/auth/authentication/ldap.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/auth/authentication/ldap.md index a1edb207002c9..81f53bfa68463 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/auth/authentication/ldap.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/auth/authentication/ldap.md @@ -2,7 +2,7 @@ { "title": "LDAP", "language": "zh-CN", - "description": "详解 Apache Doris 联邦认证：通过集成 LDAP 实现统一身份验证与组授权，涵盖配置步骤、登录方式、权限映射规则及常见问题排查。" + "description": "详解 Apache Doris 联邦认证：通过集成 LDAP/LDAPS 实现统一身份验证与组授权，涵盖配置步骤、登录方式、权限映射规则及常见问题排查。" } --- @@ -44,7 +44,7 @@ Doris 支持接入第三方 LDAP 服务，提供验证登录和组授权两大 ### 第一步：配置 Doris 1. 在 `fe/conf/fe.conf` 中设置认证方式：`authentication_type=ldap`。 -2. 在 `fe/conf/ldap.conf` 中配置 LDAP 服务的连接信息。 +2. 在 `fe/conf/ldap.conf` 中配置 LDAP 服务的连接信息： ``` ldap_authentication_enabled = true @@ -56,6 +56,8 @@ Doris 支持接入第三方 LDAP 服务，提供验证登录和组授权两大 ldap_group_basedn = ou=group,o=emr ``` + > 如需启用 LDAPS（加密连接至 LDAP 服务器），请参阅下文 [LDAPS（加密连接）](#ldaps加密连接) 章节。 + 3. 启动 `fe` 后，使用 `root` 或 `admin` 账号登录 Doris，设置 LDAP 管理员密码： ```sql @@ -233,6 +235,52 @@ member: uid=jack,ou=aidp,dc=domain,dc=com > - `user` 属于哪个 `group` 与 LDAP 树的组织结构无关。上文示例中的 `user2` 并不一定属于 `group2`。 > - 若想让 `user2` 属于 `group2`，需要在 `group2` 的 `member` 属性中显式添加 `user2`。 +## LDAPS（加密连接） + +:::info 自 4.0.5 版本开始支持 +::: + +默认情况下，Doris 通过明文 LDAP 协议与 LDAP 服务器通信。自 4.0.5 版本起，Doris 支持 LDAPS（LDAP over SSL/TLS），用于加密 Doris FE 与 LDAP 服务器之间的连接。 + +### 启用 LDAPS + +在 `fe/conf/ldap.conf` 中添加以下配置以启用 LDAPS： + +``` +ldap_use_ssl = true +``` + +当 `ldap_use_ssl` 设置为 `true` 时，Doris 使用 `ldaps://` 协议连接 LDAP 服务器。同时应将端口更新为标准 LDAPS 端口（通常为 `636`）： + +``` +ldap_host = ldap-host +ldap_port = 636 +ldap_use_ssl = true +``` + +### 配置证书信任 + +使用 LDAPS 时，LDAP 服务器的 SSL 证书必须被 Doris FE 的 JVM 所信任。如果 LDAP 服务器使用的证书由知名公共 CA 签发，则无需额外配置。 + +如果使用自定义或自签名 CA，需要将 CA 证书导入 Java trustStore，并配置 JVM 使用该 trustStore。在 `fe/conf/fe.conf` 的 `JAVA_OPTS` 中添加以下参数： + +``` +# JDK 17 示例 +JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +``` + +> **导入自签名 CA 证书的步骤：** +> +> 1. 获取 CA 证书文件（例如 `ca.crt`）。 +> 2. 使用 `keytool` 将其导入 Java trustStore： +> +> ```shell +> keytool -importcert -alias ldap-ca -keystore /path/to/your/cacerts -file /path/to/ca.crt -storepass changeit -noprompt +> ``` +> +> 3. 按上述方式在 `JAVA_OPTS` 中配置 trustStore 路径。 +> 4. 重启 Doris FE 使配置生效。 + ## 缓存管理 为了避免频繁访问 LDAP 服务，Doris 会将 LDAP 信息缓存到内存中。 @@ -250,7 +298,8 @@ member: uid=jack,ou=aidp,dc=domain,dc=com ## 已知限制 -- 目前 Doris 的 LDAP 功能仅支持明文密码验证，即用户登录时，密码在 `client` 与 `fe` 之间、`fe` 与 LDAP 服务之间均以明文形式传输。 +- Doris 的 LDAP 功能在客户端到 FE 的链路上仅支持明文密码验证，即用户登录时，密码在 `client` 与 `fe` 之间以明文形式传输。客户端与 Doris FE 之间的 SSL/TLS 加密需要单独配置（详见[客户端连接](#第二步客户端连接)）。 +- FE 到 LDAP 服务器的链路默认使用明文传输（`ldap_use_ssl = false`）。如需加密该链路，请设置 `ldap_use_ssl = true` 以启用 LDAPS（详见 [LDAPS（加密连接）](#ldaps加密连接)）。 ## 常见问题 @@ -266,3 +315,12 @@ member: uid=jack,ou=aidp,dc=domain,dc=com 2. 检查预期的 `group` 是否位于 `ldap_group_basedn` 对应的组织结构下。 3. 检查预期的 `group` 是否包含 `member` 属性。 4. 检查预期 `group` 的 `member` 属性中是否包含当前用户的 `dn`。 + +**Q：LDAPS 连接失败，如何排查？** + +按以下步骤逐项检查： + +1. 确认 `fe/conf/ldap.conf` 中已设置 `ldap_use_ssl = true`。 +2. 确认 `ldap_port` 已设置为正确的 LDAPS 端口（通常为 `636`）。 +3. 检查 LDAP 服务器的 SSL 证书是否被 JVM 信任。查看 `fe.log` 中是否有 `SSLHandshakeException` 或 `PKIX path building failed` 等 SSL 握手错误。 +4. 如果使用自签名 CA，确认 CA 证书已导入 trustStore，且 `JAVA_OPTS` 中的 trustStore 路径配置正确。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/auth/authentication/ldap.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/auth/authentication/ldap.md index a1edb207002c9..81f53bfa68463 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/auth/authentication/ldap.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/auth/authentication/ldap.md @@ -2,7 +2,7 @@ { "title": "LDAP", "language": "zh-CN", - "description": "详解 Apache Doris 联邦认证：通过集成 LDAP 实现统一身份验证与组授权，涵盖配置步骤、登录方式、权限映射规则及常见问题排查。" + "description": "详解 Apache Doris 联邦认证：通过集成 LDAP/LDAPS 实现统一身份验证与组授权，涵盖配置步骤、登录方式、权限映射规则及常见问题排查。" } --- @@ -44,7 +44,7 @@ Doris 支持接入第三方 LDAP 服务，提供验证登录和组授权两大 ### 第一步：配置 Doris 1. 在 `fe/conf/fe.conf` 中设置认证方式：`authentication_type=ldap`。 -2. 在 `fe/conf/ldap.conf` 中配置 LDAP 服务的连接信息。 +2. 在 `fe/conf/ldap.conf` 中配置 LDAP 服务的连接信息： ``` ldap_authentication_enabled = true @@ -56,6 +56,8 @@ Doris 支持接入第三方 LDAP 服务，提供验证登录和组授权两大 ldap_group_basedn = ou=group,o=emr ``` + > 如需启用 LDAPS（加密连接至 LDAP 服务器），请参阅下文 [LDAPS（加密连接）](#ldaps加密连接) 章节。 + 3. 启动 `fe` 后，使用 `root` 或 `admin` 账号登录 Doris，设置 LDAP 管理员密码： ```sql @@ -233,6 +235,52 @@ member: uid=jack,ou=aidp,dc=domain,dc=com > - `user` 属于哪个 `group` 与 LDAP 树的组织结构无关。上文示例中的 `user2` 并不一定属于 `group2`。 > - 若想让 `user2` 属于 `group2`，需要在 `group2` 的 `member` 属性中显式添加 `user2`。 +## LDAPS（加密连接） + +:::info 自 4.0.5 版本开始支持 +::: + +默认情况下，Doris 通过明文 LDAP 协议与 LDAP 服务器通信。自 4.0.5 版本起，Doris 支持 LDAPS（LDAP over SSL/TLS），用于加密 Doris FE 与 LDAP 服务器之间的连接。 + +### 启用 LDAPS + +在 `fe/conf/ldap.conf` 中添加以下配置以启用 LDAPS： + +``` +ldap_use_ssl = true +``` + +当 `ldap_use_ssl` 设置为 `true` 时，Doris 使用 `ldaps://` 协议连接 LDAP 服务器。同时应将端口更新为标准 LDAPS 端口（通常为 `636`）： + +``` +ldap_host = ldap-host +ldap_port = 636 +ldap_use_ssl = true +``` + +### 配置证书信任 + +使用 LDAPS 时，LDAP 服务器的 SSL 证书必须被 Doris FE 的 JVM 所信任。如果 LDAP 服务器使用的证书由知名公共 CA 签发，则无需额外配置。 + +如果使用自定义或自签名 CA，需要将 CA 证书导入 Java trustStore，并配置 JVM 使用该 trustStore。在 `fe/conf/fe.conf` 的 `JAVA_OPTS` 中添加以下参数： + +``` +# JDK 17 示例 +JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +``` + +> **导入自签名 CA 证书的步骤：** +> +> 1. 获取 CA 证书文件（例如 `ca.crt`）。 +> 2. 使用 `keytool` 将其导入 Java trustStore： +> +> ```shell +> keytool -importcert -alias ldap-ca -keystore /path/to/your/cacerts -file /path/to/ca.crt -storepass changeit -noprompt +> ``` +> +> 3. 按上述方式在 `JAVA_OPTS` 中配置 trustStore 路径。 +> 4. 重启 Doris FE 使配置生效。 + ## 缓存管理 为了避免频繁访问 LDAP 服务，Doris 会将 LDAP 信息缓存到内存中。 @@ -250,7 +298,8 @@ member: uid=jack,ou=aidp,dc=domain,dc=com ## 已知限制 -- 目前 Doris 的 LDAP 功能仅支持明文密码验证，即用户登录时，密码在 `client` 与 `fe` 之间、`fe` 与 LDAP 服务之间均以明文形式传输。 +- Doris 的 LDAP 功能在客户端到 FE 的链路上仅支持明文密码验证，即用户登录时，密码在 `client` 与 `fe` 之间以明文形式传输。客户端与 Doris FE 之间的 SSL/TLS 加密需要单独配置（详见[客户端连接](#第二步客户端连接)）。 +- FE 到 LDAP 服务器的链路默认使用明文传输（`ldap_use_ssl = false`）。如需加密该链路，请设置 `ldap_use_ssl = true` 以启用 LDAPS（详见 [LDAPS（加密连接）](#ldaps加密连接)）。 ## 常见问题 @@ -266,3 +315,12 @@ member: uid=jack,ou=aidp,dc=domain,dc=com 2. 检查预期的 `group` 是否位于 `ldap_group_basedn` 对应的组织结构下。 3. 检查预期的 `group` 是否包含 `member` 属性。 4. 检查预期 `group` 的 `member` 属性中是否包含当前用户的 `dn`。 + +**Q：LDAPS 连接失败，如何排查？** + +按以下步骤逐项检查： + +1. 确认 `fe/conf/ldap.conf` 中已设置 `ldap_use_ssl = true`。 +2. 确认 `ldap_port` 已设置为正确的 LDAPS 端口（通常为 `636`）。 +3. 检查 LDAP 服务器的 SSL 证书是否被 JVM 信任。查看 `fe.log` 中是否有 `SSLHandshakeException` 或 `PKIX path building failed` 等 SSL 握手错误。 +4. 如果使用自签名 CA，确认 CA 证书已导入 trustStore，且 `JAVA_OPTS` 中的 trustStore 路径配置正确。 diff --git a/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md b/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md index fdcf429b3bf6e..00c5c38b23c66 100644 --- a/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md +++ b/versioned_docs/version-3.x/admin-manual/auth/authentication/ldap.md @@ -2,7 +2,7 @@ { "title": "LDAP", "language": "en", - "description": "Detailed guide on Apache Doris federated authentication: unified identity verification and group authorization through LDAP integration, covering configuration steps, login methods, permission mapping rules, and common troubleshooting." + "description": "Detailed guide on Apache Doris federated authentication: unified identity verification and group authorization through LDAP/LDAPS integration, covering configuration steps, login methods, permission mapping rules, and common troubleshooting." } --- @@ -49,27 +49,14 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ``` ldap_authentication_enabled = true ldap_host = ladp-host - # change ldap_port value if ldap_use_ssl specified to true as different port (636) is used for LDAPS - ldap_port = 389 + ldap_port = 389 ldap_admin_name = uid=admin,o=emr ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr - # specify ldap_use_ssl to true to use to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP - ldap_use_ssl = true ``` -> Important for LDAPS: -> -> When `ldap_use_ssl = true`, ensure your LDAP server certificate is trusted by the Doris FE JVM. -> -> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. -> -> Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): -> ``` -> # Example for JDK 17 -> JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." -> ``` + > To enable LDAPS (encrypted connection to the LDAP server), see the [LDAPS (Encrypted Connection)](#ldaps-encrypted-connection) section below. 3. After starting `fe`, log in to Doris with `root` or `admin` account and set the LDAP admin password: @@ -248,6 +235,52 @@ If user jack belongs to LDAP groups `doris_rd`, `doris_qa`, `doris_pm`, and Dori > - Which `group` a `user` belongs to is independent of the LDAP tree's organizational structure. `user2` in the example above doesn't necessarily belong to `group2`. > - To make `user2` belong to `group2`, you need to explicitly add `user2` to the `member` attribute of `group2`. +## LDAPS (Encrypted Connection) + +:::info Supported since version 4.0.5 +::: + +By default, Doris communicates with the LDAP server over plain LDAP (unencrypted). Starting from version 4.0.5, Doris supports LDAPS (LDAP over SSL/TLS) to encrypt the connection between Doris FE and the LDAP server. + +### Enabling LDAPS + +To enable LDAPS, add the following configuration to `fe/conf/ldap.conf`: + +``` +ldap_use_ssl = true +``` + +When `ldap_use_ssl` is set to `true`, Doris uses the `ldaps://` protocol to connect to the LDAP server. You should also update the port to the standard LDAPS port (typically `636`): + +``` +ldap_host = ldap-host +ldap_port = 636 +ldap_use_ssl = true +``` + +### Configuring Certificate Trust + +When using LDAPS, the LDAP server's SSL certificate must be trusted by the Doris FE JVM. If your LDAP server uses a certificate signed by a well-known public CA, no additional configuration is needed. + +If using a custom or self-signed Certificate Authority (CA), you must import the CA certificate into a Java trustStore and configure the JVM to use it. Add the following parameters to `JAVA_OPTS` in `fe/conf/fe.conf`: + +``` +# Example for JDK 17 +JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +``` + +> **Steps to import a self-signed CA certificate:** +> +> 1. Obtain the CA certificate file (e.g., `ca.crt`). +> 2. Import it into a Java trustStore using `keytool`: +> +> ```shell +> keytool -importcert -alias ldap-ca -keystore /path/to/your/cacerts -file /path/to/ca.crt -storepass changeit -noprompt +> ``` +> +> 3. Configure the trustStore path in `JAVA_OPTS` as shown above. +> 4. Restart the Doris FE for the changes to take effect. + ## Cache Management To avoid frequent access to LDAP services, Doris caches LDAP information in memory. @@ -265,9 +298,8 @@ You can refresh the cache with the `refresh ldap` statement. See [REFRESH-LDAP]( ## Known Limitations -- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, but connection between `fe` and LDAP service can be secured optionally. -- **For `ldap_use_ssl = false` (default behavior)**: Passwords are transmitted in plain text between the Doris FE and the LDAP server. -- **To secure the connection**: Set `ldap_use_ssl = true` to encrypt traffic between Doris FE and the LDAP server. Note that SSL/TLS encryption between the **client and Doris FE** must be configured separately. +- Doris's LDAP functionality only supports cleartext password verification for the client-to-FE segment, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`. SSL/TLS encryption between the client and Doris FE must be configured separately (see [Client Connection](#step-2-client-connection)). +- For the FE-to-LDAP segment, the connection is unencrypted by default (`ldap_use_ssl = false`). To encrypt this segment, set `ldap_use_ssl = true` to use LDAPS (see [LDAPS (Encrypted Connection)](#ldaps-encrypted-connection)). ## FAQ @@ -283,3 +315,12 @@ Check the following items step by step: 2. Check whether the expected `group` is located under the organizational structure corresponding to `ldap_group_basedn`. 3. Check whether the expected `group` contains the `member` attribute. 4. Check whether the `member` attribute of the expected `group` contains the current user's `dn`. + +**Q: LDAPS connection fails, how to troubleshoot?** + +Check the following items step by step: + +1. Confirm that `ldap_use_ssl = true` is set in `fe/conf/ldap.conf`. +2. Confirm that `ldap_port` is set to the correct LDAPS port (typically `636`). +3. Verify that the LDAP server's SSL certificate is trusted by the JVM. Check the `fe.log` for SSL handshake errors such as `SSLHandshakeException` or `PKIX path building failed`. +4. If using a self-signed CA, ensure the CA certificate has been imported into the trustStore and the trustStore path is correctly configured in `JAVA_OPTS`. diff --git a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md index fdcf429b3bf6e..00c5c38b23c66 100644 --- a/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md +++ b/versioned_docs/version-4.x/admin-manual/auth/authentication/ldap.md @@ -2,7 +2,7 @@ { "title": "LDAP", "language": "en", - "description": "Detailed guide on Apache Doris federated authentication: unified identity verification and group authorization through LDAP integration, covering configuration steps, login methods, permission mapping rules, and common troubleshooting." + "description": "Detailed guide on Apache Doris federated authentication: unified identity verification and group authorization through LDAP/LDAPS integration, covering configuration steps, login methods, permission mapping rules, and common troubleshooting." } --- @@ -49,27 +49,14 @@ In LDAP, data is organized in a tree structure. Here's an example of a typical L ``` ldap_authentication_enabled = true ldap_host = ladp-host - # change ldap_port value if ldap_use_ssl specified to true as different port (636) is used for LDAPS - ldap_port = 389 + ldap_port = 389 ldap_admin_name = uid=admin,o=emr ldap_user_basedn = ou=people,o=emr ldap_user_filter = (&(uid={login})) ldap_group_basedn = ou=group,o=emr - # specify ldap_use_ssl to true to use to secured LDAPS protocol, specify false or comment property to use default behavior with plain LDAP - ldap_use_ssl = true ``` -> Important for LDAPS: -> -> When `ldap_use_ssl = true`, ensure your LDAP server certificate is trusted by the Doris FE JVM. -> -> If using a custom or self-signed Certificate Authority (CA), you must configure the Java trustStore. -> -> Add the following parameters to JAVA_OPTS in `fe/conf/fe.conf` (adjust the path to your cacerts file): -> ``` -> # Example for JDK 17 -> JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." -> ``` + > To enable LDAPS (encrypted connection to the LDAP server), see the [LDAPS (Encrypted Connection)](#ldaps-encrypted-connection) section below. 3. After starting `fe`, log in to Doris with `root` or `admin` account and set the LDAP admin password: @@ -248,6 +235,52 @@ If user jack belongs to LDAP groups `doris_rd`, `doris_qa`, `doris_pm`, and Dori > - Which `group` a `user` belongs to is independent of the LDAP tree's organizational structure. `user2` in the example above doesn't necessarily belong to `group2`. > - To make `user2` belong to `group2`, you need to explicitly add `user2` to the `member` attribute of `group2`. +## LDAPS (Encrypted Connection) + +:::info Supported since version 4.0.5 +::: + +By default, Doris communicates with the LDAP server over plain LDAP (unencrypted). Starting from version 4.0.5, Doris supports LDAPS (LDAP over SSL/TLS) to encrypt the connection between Doris FE and the LDAP server. + +### Enabling LDAPS + +To enable LDAPS, add the following configuration to `fe/conf/ldap.conf`: + +``` +ldap_use_ssl = true +``` + +When `ldap_use_ssl` is set to `true`, Doris uses the `ldaps://` protocol to connect to the LDAP server. You should also update the port to the standard LDAPS port (typically `636`): + +``` +ldap_host = ldap-host +ldap_port = 636 +ldap_use_ssl = true +``` + +### Configuring Certificate Trust + +When using LDAPS, the LDAP server's SSL certificate must be trusted by the Doris FE JVM. If your LDAP server uses a certificate signed by a well-known public CA, no additional configuration is needed. + +If using a custom or self-signed Certificate Authority (CA), you must import the CA certificate into a Java trustStore and configure the JVM to use it. Add the following parameters to `JAVA_OPTS` in `fe/conf/fe.conf`: + +``` +# Example for JDK 17 +JAVA_OPTS_FOR_JDK_17 = "-Djavax.net.ssl.trustStore=/path/to/your/cacerts -Djavax.net.ssl.trustStorePassword=changeit ..." +``` + +> **Steps to import a self-signed CA certificate:** +> +> 1. Obtain the CA certificate file (e.g., `ca.crt`). +> 2. Import it into a Java trustStore using `keytool`: +> +> ```shell +> keytool -importcert -alias ldap-ca -keystore /path/to/your/cacerts -file /path/to/ca.crt -storepass changeit -noprompt +> ``` +> +> 3. Configure the trustStore path in `JAVA_OPTS` as shown above. +> 4. Restart the Doris FE for the changes to take effect. + ## Cache Management To avoid frequent access to LDAP services, Doris caches LDAP information in memory. @@ -265,9 +298,8 @@ You can refresh the cache with the `refresh ldap` statement. See [REFRESH-LDAP]( ## Known Limitations -- Currently, Doris's LDAP functionality only supports cleartext password verification, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`, but connection between `fe` and LDAP service can be secured optionally. -- **For `ldap_use_ssl = false` (default behavior)**: Passwords are transmitted in plain text between the Doris FE and the LDAP server. -- **To secure the connection**: Set `ldap_use_ssl = true` to encrypt traffic between Doris FE and the LDAP server. Note that SSL/TLS encryption between the **client and Doris FE** must be configured separately. +- Doris's LDAP functionality only supports cleartext password verification for the client-to-FE segment, meaning that when users log in, passwords are transmitted in plaintext between `client` and `fe`. SSL/TLS encryption between the client and Doris FE must be configured separately (see [Client Connection](#step-2-client-connection)). +- For the FE-to-LDAP segment, the connection is unencrypted by default (`ldap_use_ssl = false`). To encrypt this segment, set `ldap_use_ssl = true` to use LDAPS (see [LDAPS (Encrypted Connection)](#ldaps-encrypted-connection)). ## FAQ @@ -283,3 +315,12 @@ Check the following items step by step: 2. Check whether the expected `group` is located under the organizational structure corresponding to `ldap_group_basedn`. 3. Check whether the expected `group` contains the `member` attribute. 4. Check whether the `member` attribute of the expected `group` contains the current user's `dn`. + +**Q: LDAPS connection fails, how to troubleshoot?** + +Check the following items step by step: + +1. Confirm that `ldap_use_ssl = true` is set in `fe/conf/ldap.conf`. +2. Confirm that `ldap_port` is set to the correct LDAPS port (typically `636`). +3. Verify that the LDAP server's SSL certificate is trusted by the JVM. Check the `fe.log` for SSL handshake errors such as `SSLHandshakeException` or `PKIX path building failed`. +4. If using a self-signed CA, ensure the CA certificate has been imported into the trustStore and the trustStore path is correctly configured in `JAVA_OPTS`. From 8eb131389071fab9178a51028899c42c92aa085b Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Mon, 30 Mar 2026 12:55:21 -0700 Subject: [PATCH 7/8] opt --- docs/lakehouse/catalogs/iceberg-catalog.mdx | 54 ++++++------ .../lakehouse/catalogs/iceberg-catalog.mdx | 68 ++++++++------- .../lakehouse/catalogs/iceberg-catalog.mdx | 86 ++++++++++++------- .../lakehouse/catalogs/iceberg-catalog.mdx | 86 ++++++++++++------- .../lakehouse/catalogs/iceberg-catalog.mdx | 71 +++++++++------ .../lakehouse/catalogs/iceberg-catalog.mdx | 71 +++++++++------ 6 files changed, 259 insertions(+), 177 deletions(-) diff --git a/docs/lakehouse/catalogs/iceberg-catalog.mdx b/docs/lakehouse/catalogs/iceberg-catalog.mdx index 99aaccccd9ee5..ce3418e3cc1c3 100644 --- a/docs/lakehouse/catalogs/iceberg-catalog.mdx +++ b/docs/lakehouse/catalogs/iceberg-catalog.mdx @@ -392,7 +392,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 'warehouse' = 'gs://bucket/iceberg_warehouse', 'gs.access_key' = '', 'gs.secret_key' = '', - 'fs.gcs.support'='true' + 'fs.gcs.support' = 'true' ); ``` @@ -406,7 +406,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 'azure.account_name' = '', 'azure.account_key' = '', 'azure.endpoint' = 'https://.blob.core.windows.net', - 'fs.azure.support'='true' + 'fs.azure.support' = 'true' ); ``` @@ -546,7 +546,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 's3.access_key' = '', 's3.secret_key' = '' ); - ``` + ``` @@ -651,7 +651,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please ```sql CREATE CATALOG iceberg_dlf_catalog_catalog PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='dlf', + 'iceberg.catalog.type' = 'dlf', 'warehouse' = 'oss://bucket/iceberg-dlf-oss-warehouse', 'dlf.uid' = '203225413946383283', 'dlf.catalog_id' = 'p2_regression_case', @@ -671,7 +671,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please ```sql CREATE CATALOG iceberg_dlf_catalog_catalog PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='dlf', + 'iceberg.catalog.type' = 'dlf', 'warehouse' = 'oss://bucket/iceberg-dlf-oss-warehouse', 'dlf.uid' = '203225413946383283', 'dlf.catalog.id' = 'catalog_id', @@ -1071,7 +1071,7 @@ This is an experimental feature, supported since version 4.1.0. 'warehouse' = 'hdfs://127.0.0.1:8520/iceberg-fs-hdfs-warehouse', 'fs.defaultFS' = 'hdfs://127.0.0.1:8520', 'hadoop.security.authentication' = 'kerberos', - 'hadoop.kerberos.principal'='hive/presto-master.docker.cluster@LABS.TERADATA.COM', + 'hadoop.kerberos.principal' = 'hive/presto-master.docker.cluster@LABS.TERADATA.COM', 'hadoop.kerberos.keytab' = '/keytabs/hive-presto-master.keytab' ); @@ -1145,7 +1145,7 @@ This is an experimental feature, supported since version 4.1.0. 'type' = 'iceberg', 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 'gs://bucket/iceberg_warehouse', - 'fs.gcs.support'='true', + 'fs.gcs.support' = 'true', 'gs.access_key' = '', 'gs.secret_key' = '' ); @@ -1155,9 +1155,9 @@ This is an experimental feature, supported since version 4.1.0. ```sql CREATE CATALOG test_iceberg_fs_on_minio PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='hadoop', + 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 's3://warehouse/wh', - 'fs.minio.support'='true', + 'fs.minio.support' = 'true', 'minio.endpoint' = 'http://127.0.0.1:19001', 'minio.access_key' = 'admin', 'minio.secret_key' = 'password', @@ -1319,20 +1319,22 @@ This is an experimental feature, supported since version 4.1.0.
3.0.6+ Version - - Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation. - ```sql - CREATE CATALOG test_s3tables_catalog PROPERTIES ( - 'type' = 'iceberg', - 'iceberg.catalog.type' = 's3tables', - 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', - 's3.region' = 'ap-east-1', - 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', - 's3.access_key' = '', - 's3.secret_key' = '' - ); - ``` - + + + Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation. + ```sql + CREATE CATALOG test_s3tables_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 's3tables', + 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', + 's3.region' = 'ap-east-1', + 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', + 's3.access_key' = '', + 's3.secret_key' = '' + ); + ``` + +
## Query Operations @@ -1395,9 +1397,9 @@ SELECT * FROM iceberg_table FOR VERSION AS OF 123456789; ### Branch and Tag -> Since 3.1.0 +> Since 3.1.0. > -> For creating, dropping and managing branch and tag, please refer to [Managing Branch & Tag] +> For creating, dropping and managing branch and tag, please refer to the [Managing Branch & Tag] section. Reading specific branches and tags of Iceberg tables is supported. @@ -2906,7 +2908,7 @@ You can use the following SQL to analyze the data distribution and delete file c CASE WHEN file_size_in_bytes BETWEEN 0 AND 8 * 1024 * 1024 THEN '0-8M' WHEN file_size_in_bytes BETWEEN 8 * 1024 * 1024 + 1 AND 32 * 1024 * 1024 THEN '8-32M' - WHEN file_size_in_bytes BETWEEN 2 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' + WHEN file_size_in_bytes BETWEEN 32 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' WHEN file_size_in_bytes BETWEEN 128 * 1024 * 1024 + 1 AND 512 * 1024 * 1024 THEN '128-512M' WHEN file_size_in_bytes > 512 * 1024 * 1024 THEN '> 512M' ELSE 'Unknown' diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx index 48d5dc912a968..018bc76a690be 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx @@ -55,11 +55,11 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * `dlf`：使用阿里云 DLF 作为元数据服务。 - * `s3tables`: 使用 AWS S3 Tables Catalog 访问 [S3 Table Bucket](https://aws.amazon.com/s3/features/tables/). + * `s3tables`：使用 AWS S3 Tables Catalog 访问 [S3 Table Bucket](https://aws.amazon.com/s3/features/tables/)。 * `` - Iceberg 的仓库路径。当 `` 为 `hadoop` 时，需指定这个参数。 + Iceberg 的仓库路径。当 `` 为 `hadoop` 时，需指定该参数。 `warehouse` 的路径必须指向 `Database` 路径的上一级。如您的表路径是 `s3://bucket/path/to/db1/table1`，那么 `warehouse` 应该是 `s3://bucket/path/to/`。 @@ -118,7 +118,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [FileSystem](../metastores/filesystem.md) -> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参考【基础示例】章节。 +> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参阅【基础示例】章节。 ### Iceberg Catalog 功能支持矩阵 @@ -126,7 +126,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( | -------------------- | ------ | -------- | -------------- | -------------- | ------------------ | | Hive MetaStore | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Glue | ✅ | ✅ | ❌ | ❌ | ✅ | -| Aliyun DLF | ✅ | ✅ | ❌ | ❌ | ❌ | +| Aliyun DLF | ✅ | ✅ | ❌ | ❌ | ❌ | | Iceberg Rest Catalog | ✅ | ✅ | 取决于服务提供方 | 取决于服务提供方 | 取决于服务提供方 | | FileSystem | ✅ | ✅ | ✅ | ✅ | ✅ | @@ -150,7 +150,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [MINIO](../storages/minio.md) -> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参考【基础示例】章节。 +> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参阅【基础示例】章节。 ### 支持的数据格式 @@ -190,7 +190,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( ## Namespace 映射 -Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespace 可以有多级（Nested Namespace）。 +Iceberg 的元数据层级关系是 Catalog -> Namespace -> Table。其中 Namespace 可以有多级（Nested Namespace）。 ``` ┌─────────┐ @@ -984,7 +984,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'org.postgresql.Driver', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1005,7 +1005,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'com.mysql.cj.jdbc.Driver', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1024,7 +1024,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'org.sqlite.JDBC', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1131,7 +1131,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'type' = 'iceberg', 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 'gs://bucket/iceberg_warehouse', - 'fs.gcs.support'='true', + 'fs.gcs.support' = 'true', 'gs.access_key' = '', 'gs.secret_key' = '' ); @@ -1334,20 +1334,22 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac
3.0.6+ 版本 - - 可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。 - ```sql - CREATE CATALOG test_s3tables_catalog PROPERTIES ( - 'type' = 'iceberg', - 'iceberg.catalog.type' = 's3tables', - 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', - 's3.region' = 'ap-east-1', - 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', - 's3.access_key' = '', - 's3.secret_key' = '' - ); - ``` - + + + 可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。 + ```sql + CREATE CATALOG test_s3tables_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 's3tables', + 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', + 's3.region' = 'ap-east-1', + 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', + 's3.access_key' = '', + 's3.secret_key' = '' + ); + ``` + +
## 查询操作 @@ -1408,9 +1410,9 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 868895038966572; ### Branch 和 Tag -> 该功能自 3.1.0 版本支持 +> 该功能自 3.1.0 版本支持。 > -> 关于 Branch、Tag 的创建、删除和维护操作，请参阅【管理 Branch & Tag】 +> 关于 Branch、Tag 的创建、删除和维护操作，请参阅【管理 Branch & Tag】章节。 支持读取指定 Iceberg 表的分支（Branch）和标签（Tag）。 @@ -1434,7 +1436,7 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 'tag1'; > 该功能自 3.1.0 版本支持 -支持查询 Iceberg 视图。视图查询方式和普通表方式一样。有以下几点注意事项： +支持查询 Iceberg 视图。视图查询方式与普通表方式一样。有以下几点注意事项： - 仅支持 `hms` 类型的 Iceberg Catalog。 - 视图的定义 SQL 需要与 Doris SQL 方言兼容，否则会出现解析错误（后续版本会提供方言转换功能）。 @@ -1474,7 +1476,7 @@ DESC iceberg_tbl; > 该功能自 3.1.0 版本支持 -Doris 支持查询 Iceberg 系统表，用来查询表的相关元信息。支持使用系统表查看快照历史、清单文件、数据文件、分区等信息。 +Doris 支持查询 Iceberg 系统表，用于查询表的相关元信息。支持使用系统表查看快照历史、清单文件、数据文件、分区等信息。 要访问 Iceberg 表的元数据，可以在表名后添加 `$` 符号，后跟系统表名称： @@ -1984,7 +1986,7 @@ SWITCH iceberg; CREATE DATABASE [IF NOT EXISTS] iceberg_db; ``` -也可以使用全限定名创建，或指定 location（目前只有 hms 类型的 Catalog 支持指定 location），如： +也可以使用全限定名创建，或指定 location（目前只有 `hms` 类型的 Catalog 支持指定 location），如： ```sql CREATE DATABASE [IF NOT EXISTS] iceberg.iceberg_db; @@ -2011,7 +2013,7 @@ DROP DATABASE [IF EXISTS] iceberg.iceberg_db; ``` :::caution -对于 Iceberg Database，必须先删除这个 Database 下的所有表后，才能删除 Database，否则会报错 +对于 Iceberg Database，必须先删除该 Database 下的所有表后，才能删除 Database，否则会报错。 ::: ### 创建和删除表 @@ -2406,9 +2408,9 @@ DROP DATABASE [IF EXISTS] iceberg.iceberg_db; ## Iceberg 表操作 -> 该功能自 4.0.2 版本开始支持。具体支持版本参见各个操作的说明 +> 该功能自 4.0.2 版本开始支持。具体支持版本参见各个操作的说明。 > -> 该功能是实验性功能。 +> 该功能为实验性功能。 Doris 支持通过 `ALTER TABLE EXECUTE` 语法来执行 Iceberg 表的特定操作（Actions）。该语法提供了统一的接口来执行表优化和快照管理等各种操作。 @@ -2873,7 +2875,7 @@ EXECUTE set_current_snapshot ("ref" = "v1.0"); CASE WHEN file_size_in_bytes BETWEEN 0 AND 8 * 1024 * 1024 THEN '0-8M' WHEN file_size_in_bytes BETWEEN 8 * 1024 * 1024 + 1 AND 32 * 1024 * 1024 THEN '8-32M' - WHEN file_size_in_bytes BETWEEN 2 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' + WHEN file_size_in_bytes BETWEEN 32 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' WHEN file_size_in_bytes BETWEEN 128 * 1024 * 1024 + 1 AND 512 * 1024 * 1024 THEN '128-512M' WHEN file_size_in_bytes > 512 * 1024 * 1024 THEN '> 512M' ELSE 'Unknown' diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx index 9badeb0d509f1..018bc76a690be 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx @@ -55,11 +55,11 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * `dlf`：使用阿里云 DLF 作为元数据服务。 - * `s3tables`: 使用 AWS S3 Tables Catalog 访问 [S3 Table Bucket](https://aws.amazon.com/s3/features/tables/). + * `s3tables`：使用 AWS S3 Tables Catalog 访问 [S3 Table Bucket](https://aws.amazon.com/s3/features/tables/)。 * `` - Iceberg 的仓库路径。当 `` 为 `hadoop` 时，需指定这个参数。 + Iceberg 的仓库路径。当 `` 为 `hadoop` 时，需指定该参数。 `warehouse` 的路径必须指向 `Database` 路径的上一级。如您的表路径是 `s3://bucket/path/to/db1/table1`，那么 `warehouse` 应该是 `s3://bucket/path/to/`。 @@ -118,7 +118,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [FileSystem](../metastores/filesystem.md) -> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参考【基础示例】章节。 +> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参阅【基础示例】章节。 ### Iceberg Catalog 功能支持矩阵 @@ -126,7 +126,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( | -------------------- | ------ | -------- | -------------- | -------------- | ------------------ | | Hive MetaStore | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Glue | ✅ | ✅ | ❌ | ❌ | ✅ | -| Aliyun DLF | ✅ | ✅ | ❌ | ❌ | ❌ | +| Aliyun DLF | ✅ | ✅ | ❌ | ❌ | ❌ | | Iceberg Rest Catalog | ✅ | ✅ | 取决于服务提供方 | 取决于服务提供方 | 取决于服务提供方 | | FileSystem | ✅ | ✅ | ✅ | ✅ | ✅ | @@ -140,6 +140,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [Azure Blob](../storages/azure-blob.md) +* [Apache Ozone](../storages/ozone.md)（自 4.0.4 起支持） + * [阿里云 OSS](../storages/aliyun-oss.md) * [腾讯云 COS](../storages/tencent-cos.md) @@ -148,7 +150,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [MINIO](../storages/minio.md) -> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参考【基础示例】章节。 +> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参阅【基础示例】章节。 ### 支持的数据格式 @@ -188,7 +190,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( ## Namespace 映射 -Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespace 可以有多级（Nested Namespace）。 +Iceberg 的元数据层级关系是 Catalog -> Namespace -> Table。其中 Namespace 可以有多级（Nested Namespace）。 ``` ┌─────────┐ @@ -982,7 +984,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'org.postgresql.Driver', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1003,7 +1005,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'com.mysql.cj.jdbc.Driver', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1022,7 +1024,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'org.sqlite.JDBC', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1129,7 +1131,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'type' = 'iceberg', 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 'gs://bucket/iceberg_warehouse', - 'fs.gcs.support'='true', + 'fs.gcs.support' = 'true', 'gs.access_key' = '', 'gs.secret_key' = '' ); @@ -1149,6 +1151,22 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac ); ``` + + 自 4.0.4 起支持 + ```sql + CREATE CATALOG iceberg_fs_on_ozone_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'hadoop', + 'warehouse' = 's3a://bucket/iceberg_warehouse', + 'fs.ozone.support' = 'true', + 'ozone.endpoint' = 'http://ozone-s3g:9878', + 'ozone.access_key' = '', + 'ozone.secret_key' = '', + 'ozone.region' = 'us-east-1', + 'ozone.use_path_style' = 'true' + ); + ``` + ```sql CREATE CATALOG test_iceberg_fs_on_minio PROPERTIES ( @@ -1316,20 +1334,22 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac
3.0.6+ 版本 - - 可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。 - ```sql - CREATE CATALOG test_s3tables_catalog PROPERTIES ( - 'type' = 'iceberg', - 'iceberg.catalog.type' = 's3tables', - 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', - 's3.region' = 'ap-east-1', - 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', - 's3.access_key' = '', - 's3.secret_key' = '' - ); - ``` - + + + 可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。 + ```sql + CREATE CATALOG test_s3tables_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 's3tables', + 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', + 's3.region' = 'ap-east-1', + 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', + 's3.access_key' = '', + 's3.secret_key' = '' + ); + ``` + +
## 查询操作 @@ -1390,9 +1410,9 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 868895038966572; ### Branch 和 Tag -> 该功能自 3.1.0 版本支持 +> 该功能自 3.1.0 版本支持。 > -> 关于 Branch、Tag 的创建、删除和维护操作，请参阅【管理 Branch & Tag】 +> 关于 Branch、Tag 的创建、删除和维护操作，请参阅【管理 Branch & Tag】章节。 支持读取指定 Iceberg 表的分支（Branch）和标签（Tag）。 @@ -1416,7 +1436,7 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 'tag1'; > 该功能自 3.1.0 版本支持 -支持查询 Iceberg 视图。视图查询方式和普通表方式一样。有以下几点注意事项： +支持查询 Iceberg 视图。视图查询方式与普通表方式一样。有以下几点注意事项： - 仅支持 `hms` 类型的 Iceberg Catalog。 - 视图的定义 SQL 需要与 Doris SQL 方言兼容，否则会出现解析错误（后续版本会提供方言转换功能）。 @@ -1456,7 +1476,7 @@ DESC iceberg_tbl; > 该功能自 3.1.0 版本支持 -Doris 支持查询 Iceberg 系统表，用来查询表的相关元信息。支持使用系统表查看快照历史、清单文件、数据文件、分区等信息。 +Doris 支持查询 Iceberg 系统表，用于查询表的相关元信息。支持使用系统表查看快照历史、清单文件、数据文件、分区等信息。 要访问 Iceberg 表的元数据，可以在表名后添加 `$` 符号，后跟系统表名称： @@ -1966,7 +1986,7 @@ SWITCH iceberg; CREATE DATABASE [IF NOT EXISTS] iceberg_db; ``` -也可以使用全限定名创建，或指定 location（目前只有 hms 类型的 Catalog 支持指定 location），如： +也可以使用全限定名创建，或指定 location（目前只有 `hms` 类型的 Catalog 支持指定 location），如： ```sql CREATE DATABASE [IF NOT EXISTS] iceberg.iceberg_db; @@ -1993,7 +2013,7 @@ DROP DATABASE [IF EXISTS] iceberg.iceberg_db; ``` :::caution -对于 Iceberg Database，必须先删除这个 Database 下的所有表后，才能删除 Database，否则会报错 +对于 Iceberg Database，必须先删除该 Database 下的所有表后，才能删除 Database，否则会报错。 ::: ### 创建和删除表 @@ -2388,9 +2408,9 @@ DROP DATABASE [IF EXISTS] iceberg.iceberg_db; ## Iceberg 表操作 -> 该功能自 4.0.2 版本开始支持。具体支持版本参见各个操作的说明 +> 该功能自 4.0.2 版本开始支持。具体支持版本参见各个操作的说明。 > -> 该功能是实验性功能。 +> 该功能为实验性功能。 Doris 支持通过 `ALTER TABLE EXECUTE` 语法来执行 Iceberg 表的特定操作（Actions）。该语法提供了统一的接口来执行表优化和快照管理等各种操作。 @@ -2855,7 +2875,7 @@ EXECUTE set_current_snapshot ("ref" = "v1.0"); CASE WHEN file_size_in_bytes BETWEEN 0 AND 8 * 1024 * 1024 THEN '0-8M' WHEN file_size_in_bytes BETWEEN 8 * 1024 * 1024 + 1 AND 32 * 1024 * 1024 THEN '8-32M' - WHEN file_size_in_bytes BETWEEN 2 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' + WHEN file_size_in_bytes BETWEEN 32 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' WHEN file_size_in_bytes BETWEEN 128 * 1024 * 1024 + 1 AND 512 * 1024 * 1024 THEN '128-512M' WHEN file_size_in_bytes > 512 * 1024 * 1024 THEN '> 512M' ELSE 'Unknown' diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx index 9badeb0d509f1..018bc76a690be 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx @@ -55,11 +55,11 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * `dlf`：使用阿里云 DLF 作为元数据服务。 - * `s3tables`: 使用 AWS S3 Tables Catalog 访问 [S3 Table Bucket](https://aws.amazon.com/s3/features/tables/). + * `s3tables`：使用 AWS S3 Tables Catalog 访问 [S3 Table Bucket](https://aws.amazon.com/s3/features/tables/)。 * `` - Iceberg 的仓库路径。当 `` 为 `hadoop` 时，需指定这个参数。 + Iceberg 的仓库路径。当 `` 为 `hadoop` 时，需指定该参数。 `warehouse` 的路径必须指向 `Database` 路径的上一级。如您的表路径是 `s3://bucket/path/to/db1/table1`，那么 `warehouse` 应该是 `s3://bucket/path/to/`。 @@ -118,7 +118,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [FileSystem](../metastores/filesystem.md) -> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参考【基础示例】章节。 +> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参阅【基础示例】章节。 ### Iceberg Catalog 功能支持矩阵 @@ -126,7 +126,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( | -------------------- | ------ | -------- | -------------- | -------------- | ------------------ | | Hive MetaStore | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Glue | ✅ | ✅ | ❌ | ❌ | ✅ | -| Aliyun DLF | ✅ | ✅ | ❌ | ❌ | ❌ | +| Aliyun DLF | ✅ | ✅ | ❌ | ❌ | ❌ | | Iceberg Rest Catalog | ✅ | ✅ | 取决于服务提供方 | 取决于服务提供方 | 取决于服务提供方 | | FileSystem | ✅ | ✅ | ✅ | ✅ | ✅ | @@ -140,6 +140,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [Azure Blob](../storages/azure-blob.md) +* [Apache Ozone](../storages/ozone.md)（自 4.0.4 起支持） + * [阿里云 OSS](../storages/aliyun-oss.md) * [腾讯云 COS](../storages/tencent-cos.md) @@ -148,7 +150,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [MINIO](../storages/minio.md) -> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参考【基础示例】章节。 +> 注意：不同 Doris 版本所支持的服务类型和参数略有区别，请参阅【基础示例】章节。 ### 支持的数据格式 @@ -188,7 +190,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( ## Namespace 映射 -Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespace 可以有多级（Nested Namespace）。 +Iceberg 的元数据层级关系是 Catalog -> Namespace -> Table。其中 Namespace 可以有多级（Nested Namespace）。 ``` ┌─────────┐ @@ -982,7 +984,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'org.postgresql.Driver', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1003,7 +1005,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'com.mysql.cj.jdbc.Driver', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1022,7 +1024,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'iceberg.jdbc.init-catalog-tables' = 'true', 'iceberg.jdbc.schema-version' = 'V1', 'iceberg.jdbc.driver_class' = 'org.sqlite.JDBC', - 'iceberg.jdbc.driver_url' = '' + 'iceberg.jdbc.driver_url' = '', 'warehouse' = 's3://bucket/warehouse', 's3.access_key' = '', 's3.secret_key' = '', @@ -1129,7 +1131,7 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac 'type' = 'iceberg', 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 'gs://bucket/iceberg_warehouse', - 'fs.gcs.support'='true', + 'fs.gcs.support' = 'true', 'gs.access_key' = '', 'gs.secret_key' = '' ); @@ -1149,6 +1151,22 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac ); ``` + + 自 4.0.4 起支持 + ```sql + CREATE CATALOG iceberg_fs_on_ozone_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'hadoop', + 'warehouse' = 's3a://bucket/iceberg_warehouse', + 'fs.ozone.support' = 'true', + 'ozone.endpoint' = 'http://ozone-s3g:9878', + 'ozone.access_key' = '', + 'ozone.secret_key' = '', + 'ozone.region' = 'us-east-1', + 'ozone.use_path_style' = 'true' + ); + ``` + ```sql CREATE CATALOG test_iceberg_fs_on_minio PROPERTIES ( @@ -1316,20 +1334,22 @@ Iceberg 的元数层级关系是 Catalog -> Namespace -> Table。其中 Namespac
3.0.6+ 版本 - - 可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。 - ```sql - CREATE CATALOG test_s3tables_catalog PROPERTIES ( - 'type' = 'iceberg', - 'iceberg.catalog.type' = 's3tables', - 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', - 's3.region' = 'ap-east-1', - 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', - 's3.access_key' = '', - 's3.secret_key' = '' - ); - ``` - + + + 可参阅 [集成 S3 Tables](../best-practices/doris-aws-s3tables.md) 文档。 + ```sql + CREATE CATALOG test_s3tables_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 's3tables', + 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', + 's3.region' = 'ap-east-1', + 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', + 's3.access_key' = '', + 's3.secret_key' = '' + ); + ``` + +
## 查询操作 @@ -1390,9 +1410,9 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 868895038966572; ### Branch 和 Tag -> 该功能自 3.1.0 版本支持 +> 该功能自 3.1.0 版本支持。 > -> 关于 Branch、Tag 的创建、删除和维护操作，请参阅【管理 Branch & Tag】 +> 关于 Branch、Tag 的创建、删除和维护操作，请参阅【管理 Branch & Tag】章节。 支持读取指定 Iceberg 表的分支（Branch）和标签（Tag）。 @@ -1416,7 +1436,7 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 'tag1'; > 该功能自 3.1.0 版本支持 -支持查询 Iceberg 视图。视图查询方式和普通表方式一样。有以下几点注意事项： +支持查询 Iceberg 视图。视图查询方式与普通表方式一样。有以下几点注意事项： - 仅支持 `hms` 类型的 Iceberg Catalog。 - 视图的定义 SQL 需要与 Doris SQL 方言兼容，否则会出现解析错误（后续版本会提供方言转换功能）。 @@ -1456,7 +1476,7 @@ DESC iceberg_tbl; > 该功能自 3.1.0 版本支持 -Doris 支持查询 Iceberg 系统表，用来查询表的相关元信息。支持使用系统表查看快照历史、清单文件、数据文件、分区等信息。 +Doris 支持查询 Iceberg 系统表，用于查询表的相关元信息。支持使用系统表查看快照历史、清单文件、数据文件、分区等信息。 要访问 Iceberg 表的元数据，可以在表名后添加 `$` 符号，后跟系统表名称： @@ -1966,7 +1986,7 @@ SWITCH iceberg; CREATE DATABASE [IF NOT EXISTS] iceberg_db; ``` -也可以使用全限定名创建，或指定 location（目前只有 hms 类型的 Catalog 支持指定 location），如： +也可以使用全限定名创建，或指定 location（目前只有 `hms` 类型的 Catalog 支持指定 location），如： ```sql CREATE DATABASE [IF NOT EXISTS] iceberg.iceberg_db; @@ -1993,7 +2013,7 @@ DROP DATABASE [IF EXISTS] iceberg.iceberg_db; ``` :::caution -对于 Iceberg Database，必须先删除这个 Database 下的所有表后，才能删除 Database，否则会报错 +对于 Iceberg Database，必须先删除该 Database 下的所有表后，才能删除 Database，否则会报错。 ::: ### 创建和删除表 @@ -2388,9 +2408,9 @@ DROP DATABASE [IF EXISTS] iceberg.iceberg_db; ## Iceberg 表操作 -> 该功能自 4.0.2 版本开始支持。具体支持版本参见各个操作的说明 +> 该功能自 4.0.2 版本开始支持。具体支持版本参见各个操作的说明。 > -> 该功能是实验性功能。 +> 该功能为实验性功能。 Doris 支持通过 `ALTER TABLE EXECUTE` 语法来执行 Iceberg 表的特定操作（Actions）。该语法提供了统一的接口来执行表优化和快照管理等各种操作。 @@ -2855,7 +2875,7 @@ EXECUTE set_current_snapshot ("ref" = "v1.0"); CASE WHEN file_size_in_bytes BETWEEN 0 AND 8 * 1024 * 1024 THEN '0-8M' WHEN file_size_in_bytes BETWEEN 8 * 1024 * 1024 + 1 AND 32 * 1024 * 1024 THEN '8-32M' - WHEN file_size_in_bytes BETWEEN 2 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' + WHEN file_size_in_bytes BETWEEN 32 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' WHEN file_size_in_bytes BETWEEN 128 * 1024 * 1024 + 1 AND 512 * 1024 * 1024 THEN '128-512M' WHEN file_size_in_bytes > 512 * 1024 * 1024 THEN '> 512M' ELSE 'Unknown' diff --git a/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx b/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx index 6c1f2d11accf7..ce3418e3cc1c3 100644 --- a/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx +++ b/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx @@ -127,6 +127,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [AWS S3](../storages/s3.md) * [Google Cloud Storage](../storages/gcs.md) * [Azure Blob](../storages/azure-blob.md) +* [Apache Ozone](../storages/ozone.md) (supported since 4.0.4) * [Aliyun OSS](../storages/aliyun-oss.md) * [Tencent COS](../storages/tencent-cos.md) * [Huawei OBS](../storages/huawei-obs.md) @@ -391,7 +392,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 'warehouse' = 'gs://bucket/iceberg_warehouse', 'gs.access_key' = '', 'gs.secret_key' = '', - 'fs.gcs.support'='true' + 'fs.gcs.support' = 'true' ); ``` @@ -405,7 +406,23 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 'azure.account_name' = '', 'azure.account_key' = '', 'azure.endpoint' = 'https://.blob.core.windows.net', - 'fs.azure.support'='true' + 'fs.azure.support' = 'true' + ); + ``` + + + Supported since 4.0.4 + ```sql + CREATE CATALOG iceberg_fs_on_ozone_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'hadoop', + 'warehouse' = 's3a://bucket/iceberg_warehouse', + 'fs.ozone.support' = 'true', + 'ozone.endpoint' = 'http://ozone-s3g:9878', + 'ozone.access_key' = '', + 'ozone.secret_key' = '', + 'ozone.region' = 'us-east-1', + 'ozone.use_path_style' = 'true' ); ``` @@ -529,7 +546,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 's3.access_key' = '', 's3.secret_key' = '' ); - ``` + ``` @@ -634,7 +651,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please ```sql CREATE CATALOG iceberg_dlf_catalog_catalog PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='dlf', + 'iceberg.catalog.type' = 'dlf', 'warehouse' = 'oss://bucket/iceberg-dlf-oss-warehouse', 'dlf.uid' = '203225413946383283', 'dlf.catalog_id' = 'p2_regression_case', @@ -654,7 +671,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please ```sql CREATE CATALOG iceberg_dlf_catalog_catalog PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='dlf', + 'iceberg.catalog.type' = 'dlf', 'warehouse' = 'oss://bucket/iceberg-dlf-oss-warehouse', 'dlf.uid' = '203225413946383283', 'dlf.catalog.id' = 'catalog_id', @@ -1054,7 +1071,7 @@ This is an experimental feature, supported since version 4.1.0. 'warehouse' = 'hdfs://127.0.0.1:8520/iceberg-fs-hdfs-warehouse', 'fs.defaultFS' = 'hdfs://127.0.0.1:8520', 'hadoop.security.authentication' = 'kerberos', - 'hadoop.kerberos.principal'='hive/presto-master.docker.cluster@LABS.TERADATA.COM', + 'hadoop.kerberos.principal' = 'hive/presto-master.docker.cluster@LABS.TERADATA.COM', 'hadoop.kerberos.keytab' = '/keytabs/hive-presto-master.keytab' ); @@ -1128,7 +1145,7 @@ This is an experimental feature, supported since version 4.1.0. 'type' = 'iceberg', 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 'gs://bucket/iceberg_warehouse', - 'fs.gcs.support'='true', + 'fs.gcs.support' = 'true', 'gs.access_key' = '', 'gs.secret_key' = '' ); @@ -1138,9 +1155,9 @@ This is an experimental feature, supported since version 4.1.0. ```sql CREATE CATALOG test_iceberg_fs_on_minio PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='hadoop', + 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 's3://warehouse/wh', - 'fs.minio.support'='true', + 'fs.minio.support' = 'true', 'minio.endpoint' = 'http://127.0.0.1:19001', 'minio.access_key' = 'admin', 'minio.secret_key' = 'password', @@ -1302,20 +1319,22 @@ This is an experimental feature, supported since version 4.1.0.
3.0.6+ Version - - Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation. - ```sql - CREATE CATALOG test_s3tables_catalog PROPERTIES ( - 'type' = 'iceberg', - 'iceberg.catalog.type' = 's3tables', - 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', - 's3.region' = 'ap-east-1', - 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', - 's3.access_key' = '', - 's3.secret_key' = '' - ); - ``` - + + + Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation. + ```sql + CREATE CATALOG test_s3tables_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 's3tables', + 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', + 's3.region' = 'ap-east-1', + 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', + 's3.access_key' = '', + 's3.secret_key' = '' + ); + ``` + +
## Query Operations @@ -1378,9 +1397,9 @@ SELECT * FROM iceberg_table FOR VERSION AS OF 123456789; ### Branch and Tag -> Since 3.1.0 +> Since 3.1.0. > -> For creating, dropping and managing branch and tag, please refer to [Managing Branch & Tag] +> For creating, dropping and managing branch and tag, please refer to the [Managing Branch & Tag] section. Reading specific branches and tags of Iceberg tables is supported. @@ -2889,7 +2908,7 @@ You can use the following SQL to analyze the data distribution and delete file c CASE WHEN file_size_in_bytes BETWEEN 0 AND 8 * 1024 * 1024 THEN '0-8M' WHEN file_size_in_bytes BETWEEN 8 * 1024 * 1024 + 1 AND 32 * 1024 * 1024 THEN '8-32M' - WHEN file_size_in_bytes BETWEEN 2 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' + WHEN file_size_in_bytes BETWEEN 32 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' WHEN file_size_in_bytes BETWEEN 128 * 1024 * 1024 + 1 AND 512 * 1024 * 1024 THEN '128-512M' WHEN file_size_in_bytes > 512 * 1024 * 1024 THEN '> 512M' ELSE 'Unknown' diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx b/versioned_docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx index 6c1f2d11accf7..ce3418e3cc1c3 100644 --- a/versioned_docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx +++ b/versioned_docs/version-4.x/lakehouse/catalogs/iceberg-catalog.mdx @@ -127,6 +127,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [AWS S3](../storages/s3.md) * [Google Cloud Storage](../storages/gcs.md) * [Azure Blob](../storages/azure-blob.md) +* [Apache Ozone](../storages/ozone.md) (supported since 4.0.4) * [Aliyun OSS](../storages/aliyun-oss.md) * [Tencent COS](../storages/tencent-cos.md) * [Huawei OBS](../storages/huawei-obs.md) @@ -391,7 +392,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 'warehouse' = 'gs://bucket/iceberg_warehouse', 'gs.access_key' = '', 'gs.secret_key' = '', - 'fs.gcs.support'='true' + 'fs.gcs.support' = 'true' ); ``` @@ -405,7 +406,23 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 'azure.account_name' = '', 'azure.account_key' = '', 'azure.endpoint' = 'https://.blob.core.windows.net', - 'fs.azure.support'='true' + 'fs.azure.support' = 'true' + ); + ``` + + + Supported since 4.0.4 + ```sql + CREATE CATALOG iceberg_fs_on_ozone_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'hadoop', + 'warehouse' = 's3a://bucket/iceberg_warehouse', + 'fs.ozone.support' = 'true', + 'ozone.endpoint' = 'http://ozone-s3g:9878', + 'ozone.access_key' = '', + 'ozone.secret_key' = '', + 'ozone.region' = 'us-east-1', + 'ozone.use_path_style' = 'true' ); ``` @@ -529,7 +546,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please 's3.access_key' = '', 's3.secret_key' = '' ); - ``` + ``` @@ -634,7 +651,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please ```sql CREATE CATALOG iceberg_dlf_catalog_catalog PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='dlf', + 'iceberg.catalog.type' = 'dlf', 'warehouse' = 'oss://bucket/iceberg-dlf-oss-warehouse', 'dlf.uid' = '203225413946383283', 'dlf.catalog_id' = 'p2_regression_case', @@ -654,7 +671,7 @@ Support for Nested Namespace needs to be explicitly enabled. For details, please ```sql CREATE CATALOG iceberg_dlf_catalog_catalog PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='dlf', + 'iceberg.catalog.type' = 'dlf', 'warehouse' = 'oss://bucket/iceberg-dlf-oss-warehouse', 'dlf.uid' = '203225413946383283', 'dlf.catalog.id' = 'catalog_id', @@ -1054,7 +1071,7 @@ This is an experimental feature, supported since version 4.1.0. 'warehouse' = 'hdfs://127.0.0.1:8520/iceberg-fs-hdfs-warehouse', 'fs.defaultFS' = 'hdfs://127.0.0.1:8520', 'hadoop.security.authentication' = 'kerberos', - 'hadoop.kerberos.principal'='hive/presto-master.docker.cluster@LABS.TERADATA.COM', + 'hadoop.kerberos.principal' = 'hive/presto-master.docker.cluster@LABS.TERADATA.COM', 'hadoop.kerberos.keytab' = '/keytabs/hive-presto-master.keytab' ); @@ -1128,7 +1145,7 @@ This is an experimental feature, supported since version 4.1.0. 'type' = 'iceberg', 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 'gs://bucket/iceberg_warehouse', - 'fs.gcs.support'='true', + 'fs.gcs.support' = 'true', 'gs.access_key' = '', 'gs.secret_key' = '' ); @@ -1138,9 +1155,9 @@ This is an experimental feature, supported since version 4.1.0. ```sql CREATE CATALOG test_iceberg_fs_on_minio PROPERTIES ( 'type' = 'iceberg', - 'iceberg.catalog.type'='hadoop', + 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 's3://warehouse/wh', - 'fs.minio.support'='true', + 'fs.minio.support' = 'true', 'minio.endpoint' = 'http://127.0.0.1:19001', 'minio.access_key' = 'admin', 'minio.secret_key' = 'password', @@ -1302,20 +1319,22 @@ This is an experimental feature, supported since version 4.1.0.
3.0.6+ Version - - Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation. - ```sql - CREATE CATALOG test_s3tables_catalog PROPERTIES ( - 'type' = 'iceberg', - 'iceberg.catalog.type' = 's3tables', - 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', - 's3.region' = 'ap-east-1', - 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', - 's3.access_key' = '', - 's3.secret_key' = '' - ); - ``` - + + + Please refer to the [S3 Tables Integration](../best-practices/doris-aws-s3tables.md) documentation. + ```sql + CREATE CATALOG test_s3tables_catalog PROPERTIES ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 's3tables', + 'warehouse' = 'arn:aws:s3tables:ap-east-1:9527:bucket/doris-s3-table-bucket', + 's3.region' = 'ap-east-1', + 's3.endpoint' = 'https://s3.ap-east-1.amazonaws.com', + 's3.access_key' = '', + 's3.secret_key' = '' + ); + ``` + +
## Query Operations @@ -1378,9 +1397,9 @@ SELECT * FROM iceberg_table FOR VERSION AS OF 123456789; ### Branch and Tag -> Since 3.1.0 +> Since 3.1.0. > -> For creating, dropping and managing branch and tag, please refer to [Managing Branch & Tag] +> For creating, dropping and managing branch and tag, please refer to the [Managing Branch & Tag] section. Reading specific branches and tags of Iceberg tables is supported. @@ -2889,7 +2908,7 @@ You can use the following SQL to analyze the data distribution and delete file c CASE WHEN file_size_in_bytes BETWEEN 0 AND 8 * 1024 * 1024 THEN '0-8M' WHEN file_size_in_bytes BETWEEN 8 * 1024 * 1024 + 1 AND 32 * 1024 * 1024 THEN '8-32M' - WHEN file_size_in_bytes BETWEEN 2 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' + WHEN file_size_in_bytes BETWEEN 32 * 1024 * 1024 + 1 AND 128 * 1024 * 1024 THEN '32-128M' WHEN file_size_in_bytes BETWEEN 128 * 1024 * 1024 + 1 AND 512 * 1024 * 1024 THEN '128-512M' WHEN file_size_in_bytes > 512 * 1024 * 1024 THEN '> 512M' ELSE 'Unknown' From 7af76bee91ce240ed7e7c0bfaf1a2f37d6eada51 Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Mon, 30 Mar 2026 13:00:17 -0700 Subject: [PATCH 8/8] ozone --- .../lakehouse/catalogs/iceberg-catalog.mdx | 2 - .../version-4.x/lakehouse/storages/ozone.md | 69 +++++++++++++++++++ sidebars.ts | 2 +- .../lakehouse/catalogs/iceberg-catalog.mdx | 1 - .../version-4.x/lakehouse/storages/ozone.md | 69 +++++++++++++++++++ versioned_sidebars/version-4.x-sidebars.json | 3 +- 6 files changed, 141 insertions(+), 5 deletions(-) create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/storages/ozone.md create mode 100644 versioned_docs/version-4.x/lakehouse/storages/ozone.md diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx index 018bc76a690be..99c370968387b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx @@ -140,8 +140,6 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [Azure Blob](../storages/azure-blob.md) -* [Apache Ozone](../storages/ozone.md)（自 4.0.4 起支持） - * [阿里云 OSS](../storages/aliyun-oss.md) * [腾讯云 COS](../storages/tencent-cos.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/storages/ozone.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/storages/ozone.md new file mode 100644 index 0000000000000..e95455ed7ffc8 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/storages/ozone.md @@ -0,0 +1,69 @@ +--- +{ + "title": "Apache Ozone", + "language": "zh-CN", + "description": "自 4.0.4 版本起，Doris 支持通过 S3 Gateway 访问 Apache Ozone。", + "sidebar_label": "Apache Ozone" +} +--- + +# Apache Ozone + +自 4.0.4 版本起，Doris 支持通过 S3 Gateway 访问 Apache Ozone。 + +本文档介绍访问 Apache Ozone 所需的参数，这些参数适用于以下场景： + +- Catalog 属性 +- Table Valued Function 属性 +- Broker Load 属性 +- Export 属性 +- Outfile 属性 + +**如果要将 Ozone 作为独立存储类型使用，需要显式配置 `"fs.ozone.support" = "true"`。** + +## 参数总览 + +| 属性名称 | 曾用名 | 描述 | 默认值 | 是否必须 | +| --- | --- | --- | --- | --- | +| ozone.endpoint | s3.endpoint | Ozone S3 Gateway 访问端点，例如 `http://ozone-s3g:9878` | 无 | 是 | +| ozone.region | s3.region | Ozone S3 Gateway 区域 | `us-east-1` | 否 | +| ozone.access_key | s3.access_key, s3.access-key-id | 用于认证的 Access Key | 无 | 否* | +| ozone.secret_key | s3.secret_key, s3.secret-access-key | 用于认证的 Secret Key | 无 | 否* | +| ozone.session_token | s3.session_token, s3.session-token | Session Token | 无 | 否 | +| ozone.connection.maximum | s3.connection.maximum | 最大连接数 | `100` | 否 | +| ozone.connection.request.timeout | s3.connection.request.timeout | 请求超时时间（毫秒） | `10000` | 否 | +| ozone.connection.timeout | s3.connection.timeout | 连接超时时间（毫秒） | `10000` | 否 | +| ozone.use_path_style | use_path_style, s3.path-style-access | 是否使用 path-style 访问 | `true` | 否 | +| ozone.force_parsing_by_standard_uri | force_parsing_by_standard_uri | 是否强制使用标准 URI 解析 | `false` | 否 | +| fs.ozone.support | | 是否启用 Ozone 存储类型 | `false` | 是 | + +说明： + +- `ozone.access_key` 和 `ozone.secret_key` 需要成对配置。 +- `fs.s3a.*` 参数不会被 Ozone 属性直接解析，请使用 `ozone.*` 或兼容的 `s3.*` 参数。 +- Ozone 支持 `s3://`、`s3a://`、`s3n://` 三种 URI 协议。 +- 仅配置 `ozone.endpoint`（或 `s3.endpoint`）不会自动启用 Ozone，必须同时设置 `fs.ozone.support=true`。 + +## 配置示例 + +使用 `ozone.*` 参数： + +```properties +"fs.ozone.support" = "true", +"ozone.endpoint" = "http://ozone-s3g:9878", +"ozone.access_key" = "hadoop", +"ozone.secret_key" = "hadoop", +"ozone.region" = "us-east-1", +"ozone.use_path_style" = "true" +``` + +使用兼容的 `s3.*` 参数： + +```properties +"fs.ozone.support" = "true", +"s3.endpoint" = "http://ozone-s3g:9878", +"s3.access_key" = "hadoop", +"s3.secret_key" = "hadoop", +"s3.region" = "us-east-1", +"s3.path-style-access" = "true" +``` diff --git a/sidebars.ts b/sidebars.ts index 0398a2810e3f8..48c1d55f75218 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -433,7 +433,6 @@ const sidebars: SidebarsConfig = { items: [ 'lakehouse/storages/hdfs', 'lakehouse/storages/s3', - 'lakehouse/storages/ozone', 'lakehouse/storages/azure-blob', 'lakehouse/storages/gcs', 'lakehouse/storages/aliyun-oss', @@ -442,6 +441,7 @@ const sidebars: SidebarsConfig = { 'lakehouse/storages/baidu-bos', 'lakehouse/storages/minio', 'lakehouse/storages/juicefs', + 'lakehouse/storages/ozone' ], }, { diff --git a/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx b/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx index ce3418e3cc1c3..4e68627f658ad 100644 --- a/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx +++ b/versioned_docs/version-3.x/lakehouse/catalogs/iceberg-catalog.mdx @@ -127,7 +127,6 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * [AWS S3](../storages/s3.md) * [Google Cloud Storage](../storages/gcs.md) * [Azure Blob](../storages/azure-blob.md) -* [Apache Ozone](../storages/ozone.md) (supported since 4.0.4) * [Aliyun OSS](../storages/aliyun-oss.md) * [Tencent COS](../storages/tencent-cos.md) * [Huawei OBS](../storages/huawei-obs.md) diff --git a/versioned_docs/version-4.x/lakehouse/storages/ozone.md b/versioned_docs/version-4.x/lakehouse/storages/ozone.md new file mode 100644 index 0000000000000..784f3f6fd162d --- /dev/null +++ b/versioned_docs/version-4.x/lakehouse/storages/ozone.md @@ -0,0 +1,69 @@ +--- +{ + "title": "Apache Ozone", + "language": "en", + "description": "Starting from version 4.0.4, Doris supports accessing Apache Ozone through the S3 Gateway.", + "sidebar_label": "Apache Ozone" +} +--- + +# Apache Ozone + +Starting from version 4.0.4, Doris supports accessing Apache Ozone through the S3 Gateway. + +This document describes the parameters required to access Apache Ozone. These parameters apply to: + +- Catalog properties +- Table Valued Function properties +- Broker Load properties +- Export properties +- Outfile properties + +**To use Ozone as a dedicated storage type, configure `"fs.ozone.support" = "true"` explicitly.** + +## Parameter Overview + +| Property Name | Legacy Name | Description | Default Value | Required | +| --- | --- | --- | --- | --- | +| ozone.endpoint | s3.endpoint | Ozone S3 Gateway endpoint, for example `http://ozone-s3g:9878` | None | Yes | +| ozone.region | s3.region | Region of Ozone S3 Gateway | `us-east-1` | No | +| ozone.access_key | s3.access_key, s3.access-key-id | Access key used for authentication | None | No* | +| ozone.secret_key | s3.secret_key, s3.secret-access-key | Secret key used for authentication | None | No* | +| ozone.session_token | s3.session_token, s3.session-token | Session token | None | No | +| ozone.connection.maximum | s3.connection.maximum | Maximum number of connections | `100` | No | +| ozone.connection.request.timeout | s3.connection.request.timeout | Request timeout in milliseconds | `10000` | No | +| ozone.connection.timeout | s3.connection.timeout | Connection timeout in milliseconds | `10000` | No | +| ozone.use_path_style | use_path_style, s3.path-style-access | Whether to use path-style access | `true` | No | +| ozone.force_parsing_by_standard_uri | force_parsing_by_standard_uri | Whether to force standard URI parsing | `false` | No | +| fs.ozone.support | | Whether to enable Ozone as storage type | `false` | Yes | + +Notes: + +- `ozone.access_key` and `ozone.secret_key` must be configured together. +- `fs.s3a.*` keys are not parsed directly by Ozone properties. Use `ozone.*` or compatible `s3.*` keys. +- Ozone supports `s3://`, `s3a://`, and `s3n://` URI schemas. +- Setting `ozone.endpoint` (or `s3.endpoint`) alone does not enable Ozone. You must set `fs.ozone.support=true`. + +## Example Configuration + +Using `ozone.*` keys: + +```properties +"fs.ozone.support" = "true", +"ozone.endpoint" = "http://ozone-s3g:9878", +"ozone.access_key" = "hadoop", +"ozone.secret_key" = "hadoop", +"ozone.region" = "us-east-1", +"ozone.use_path_style" = "true" +``` + +Using compatible `s3.*` aliases: + +```properties +"fs.ozone.support" = "true", +"s3.endpoint" = "http://ozone-s3g:9878", +"s3.access_key" = "hadoop", +"s3.secret_key" = "hadoop", +"s3.region" = "us-east-1", +"s3.path-style-access" = "true" +``` diff --git a/versioned_sidebars/version-4.x-sidebars.json b/versioned_sidebars/version-4.x-sidebars.json index 73619501c953a..4379cd338e234 100644 --- a/versioned_sidebars/version-4.x-sidebars.json +++ b/versioned_sidebars/version-4.x-sidebars.json @@ -441,7 +441,8 @@ "lakehouse/storages/huawei-obs", "lakehouse/storages/baidu-bos", "lakehouse/storages/minio", - "lakehouse/storages/juicefs" + "lakehouse/storages/juicefs", + "lakehouse/storages/ozone" ] }, {