Multiple opencloud replicas not working with keycloak and opencloud.replicas description misleading

[Tim-herbie]

Hello,

I discovered Opencloud a few days ago and have started exploring the application — it looks great so far, well done!

I'm planning to set up a high-availability (HA) Opencloud deployment on Kubernetes and found this Helm chart — which looks very solid overall, great work! However, I’m a bit confused by the description of the opencloud.replicas value. It states that persistence can be disabled when deploying more than one replica.

In my tests, if I disable persistence, I can indeed run multiple replicas — but when I delete a pod, the files are no longer visible in the Opencloud UI, even though they are still present in S3. So at the moment, the only reliable way would be to use RWS volumes for the data and config directory. I tested that and were able to have two replicas, but after that I can´t login to opencloud anymore and get the message Access denied.

It seems that opencloud can´t handle the sessions for already logged in users with keycloak. Do I something wrong or is this still a bug?

HA Opencloud Setup with Keycloak

global:
  domain:
    keycloak: keycloak.mydomain.de
    opencloud: opencloud.mydomain.de
    wopi: wopiserver.mydomain.de
    collabora: collabora.mydomain.de
    onlyoffice: onlyoffice.mydomain.de
    companion: companion.mydomain.de

keycloak:
  enabled: false
  internal:
    enabled: false
  external:
    # Enable external Keycloak
    enabled: true
    # External Keycloak URL
    url: "https://keycloak.mydomain.de"
    # External Keycloak realm
    realm: "master"
    # External Keycloak client ID
    clientId: "web"

postgres:
  enabled: false

opencloud:
  enabled: true
  replicas: 2
  logLevel: "warn"
  persistence:
    enabled: true
    # Size of the persistent volume for data
    size: 5Gi
    # Size of the persistent volume for config
    configSize: 3Gi
    # Access mode (ReadWriteOnce or ReadWriteMany for multiple replicas)
    accessMode: ReadWriteMany
  storage:
    s3:
      internal:
        enabled: false
        persistence:
          enabled: false
      external:
        enabled: true
        endpoint: "https://minio-api.mydomain.de"
        region: "us-east-1"
        accessKey: "opencloud"
        secretKey: "my-super-secret-key"
        bucket: "opencloud"
        createBucket: false
  env:
    - name: OC_OIDC_ISSUER
      value: https://keycloak.mydomain.de/realms/master
    - name: "WEB_OIDC_METADATA_URL"
      value: "https://keycloak.mydomain.de/realms/master/.well-known/openid-configuration"
    - name: "WEB_OIDC_CLIENT_ID"
      value: "web"
    - name: "PROXY_OIDC_REWRITE_WELLKNOWN"
      value: "true"
    - name: "PROXY_USER_OIDC_CLAIM"
      value: "preferred_username"
    - name: "PROXY_USER_CS3_CLAIM"
      value: "username"
    - name: "GRAPH_ASSIGN_DEFAULT_USER_ROLE"
      value: "false"
    - name: "GRAPH_USERNAME_MATCH"
      value: "none"
    - name: "PROXY_AUTOPROVISION_ACCOUNTS"
      value: "true"
    - name: GRAPH_LDAP_SERVER_WRITE_ENABLED
      value: "true"
    - name: "PROXY_ROLE_ASSIGNMENT_DRIVER"
      value: "oidc"
    - name: "PROXY_ROLE_ASSIGNMENT_OIDC_CLAIM"
      value: "roles"
    - name: "PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD"
      value: "none"
    - name: OC_ADMIN_USER_ID
      value: ""

Here are some pod logs:

kubectl logs opencloud-opencloud-55c4cd5bd9-qm7t9 -f                                                                                                                              ─╯
Defaulted container "opencloud" out of: opencloud, init-config (init), init-drawio (init), init-externalsites (init), init-importer (init), init-jsonviewer (init), init-progressbars (init), init-unzip (init), init-web-extensions (init)
2025/05/28 18:58:31 Could not create config: config file already exists, use --force-overwrite to overwrite or --diff to show diff
{"level":"warn","service":"nats","time":"2025-05-28T18:58:31Z","message":"Filestore [KV_eventhistory] Stream state outdated, last block has additional entries, will rebuild"}
{"level":"warn","service":"nats","time":"2025-05-28T18:58:31Z","message":"Filestore [KV_eventhistory] Recovering stream state from index errored: prior state file"}
{"level":"warn","service":"nats","time":"2025-05-28T18:58:31Z","message":"Filestore [main-queue] Stream state outdated, last block has additional entries, will rebuild"}
{"level":"warn","service":"nats","time":"2025-05-28T18:58:31Z","message":"Filestore [main-queue] Recovering stream state from index errored: prior state file"}
{"level":"warn","service":"proxy","time":"2025-05-28T18:58:33Z","message":"basic auth enabled, use only for testing or development"}
{"level":"warn","service":"storage-system","pkg":"rhttp","time":"2025-05-28T18:58:33Z","message":"missing or incomplete nats configuration. Events will not be published."}
{"level":"warn","service":"ocm","pkg":"rhttp","time":"2025-05-28T18:58:33Z","message":"missing or incomplete nats configuration. Events will not be published."}
{"level":"warn","service":"ocm","pkg":"rhttp","traceid":"d444aed34a96e64732666d570f72aa61","time":"2025-05-28T18:58:33Z","message":"core access token not set"}
{"level":"warn","service":"ocm","pkg":"rhttp","traceid":"d444aed34a96e64732666d570f72aa61","host":"127.0.0.1","method":"GET","uri":"/","url":"/","proto":"HTTP/1.1","status":404,"size":19,"start":"28/May/2025:18:58:33 +0000","end":"28/May/2025:18:58:33 +0000","time_ns":113649,"time":"2025-05-28T18:58:33Z","message":"http"}
{"level":"warn","service":"ocm","pkg":"rhttp","traceid":"5cc2efee8d7d11f6f352755ec535fc54","time":"2025-05-28T18:58:33Z","message":"core access token not set"}
{"level":"warn","service":"ocm","pkg":"rhttp","traceid":"5cc2efee8d7d11f6f352755ec535fc54","host":"127.0.0.1","method":"GET","uri":"/","url":"/","proto":"HTTP/1.1","status":404,"size":19,"start":"28/May/2025:18:58:33 +0000","end":"28/May/2025:18:58:33 +0000","time_ns":198562,"time":"2025-05-28T18:58:33Z","message":"http"}
{"level":"warn","service":"frontend","pkg":"rhttp","traceid":"c5a63c2ab2d1e217a5521ef7ff64d9dd","time":"2025-05-28T18:58:34Z","message":"core access token not set"}
{"level":"warn","service":"frontend","pkg":"rhttp","traceid":"c5a63c2ab2d1e217a5521ef7ff64d9dd","host":"127.0.0.1","method":"GET","uri":"/","url":"/","proto":"HTTP/1.1","status":401,"size":0,"start":"28/May/2025:18:58:34 +0000","end":"28/May/2025:18:58:34 +0000","time_ns":609080,"time":"2025-05-28T18:58:34Z","message":"http"}
{"level":"warn","service":"frontend","pkg":"rhttp","traceid":"7f083b1d82e226405dd3706fb1c146ab","time":"2025-05-28T18:58:37Z","message":"core access token not set"}
{"level":"error","service":"graph","error":"LDAP Result Code 200 \"Network Error\": dial tcp [::1]:9235: connect: connection refused","time":"2025-05-28T18:58:49Z","message":"could not get ldap Connection"}
{"level":"error","service":"graph","request-id":"opencloud-opencloud-55c4cd5bd9-qm7t9/Uu2RydGzJC-000035","error":"LDAP Result Code 200 \"Network Error\": dial tcp [::1]:9235: connect: connection refused","time":"2025-05-28T18:58:49Z","message":"failed to add user"}
{"level":"error","service":"graph","request-id":"opencloud-opencloud-55c4cd5bd9-qm7t9/Uu2RydGzJC-000035","error":"generalException: failed to add user","time":"2025-05-28T18:58:49Z","message":"could not create user: backend error"}
{"level":"warn","service":"proxy","OData Error":"failed to add user","time":"2025-05-28T18:58:49Z","message":"Error Response"}
{"level":"error","service":"proxy","error":"500 Internal Server Error","time":"2025-05-28T18:58:49Z","message":"Error creating user"}
{"level":"error","service":"proxy","error":"500 Internal Server Error","time":"2025-05-28T18:58:49Z","message":"Autoprovisioning user failed"}
{"level":"error","service":"graph","error":"LDAP Result Code 200 \"Network Error\": dial tcp [::1]:9235: connect: connection refused","time":"2025-05-28T18:59:38Z","message":"could not get ldap Connection"}
{"level":"error","service":"graph","request-id":"opencloud-opencloud-55c4cd5bd9-qm7t9/Uu2RydGzJC-000066","error":"LDAP Result Code 200 \"Network Error\": dial tcp [::1]:9235: connect: connection refused","time":"2025-05-28T18:59:38Z","message":"failed to add user"}
{"level":"error","service":"graph","request-id":"opencloud-opencloud-55c4cd5bd9-qm7t9/Uu2RydGzJC-000066","error":"generalException: failed to add user","time":"2025-05-28T18:59:38Z","message":"could not create user: backend error"}
{"level":"warn","service":"proxy","OData Error":"failed to add user","time":"2025-05-28T18:59:38Z","message":"Error Response"}
{"level":"error","service":"proxy","error":"500 Internal Server Error","time":"2025-05-28T18:59:38Z","message":"Error creating user"}
{"level":"error","service":"proxy","error":"500 Internal Server Error","time":"2025-05-28T18:59:38Z","message":"Autoprovisioning user failed"}

And the error from the GUI:

Nicht angemeldet
Das könnte aufgrund einer routinemäßigen Abmeldung aus Sicherheitsgründen geschehen sein oder Ihr Benutzerkonto ist inaktiv oder noch nicht freigeschaltet. Bitte versuchen Sie es nach einiger Zeit erneut oder wenden Sie sich an Ihre Administration.

Doc

P.S. I would change the description for the value opencloud.replicas, because it´s misleading IMHO:

Current:

Number of replicas (Note: When using multiple replicas, persistence should be disabled or use a storage class that supports ReadWriteMany access mode)

New:

Number of replicas (Note: When using multiple replicas, use a storage class that supports ReadWriteMany access mode)

[michaelstingl]

@Tim-herbie Thanks for the detailed report! This is indeed a significant issue with running multiple OpenCloud replicas.

Root Cause Analysis

The issue you're experiencing is due to OpenCloud's embedded IDM (LDAP) service architecture:

1. Embedded LDAP: OpenCloud runs an embedded LDAP service (IDM) on localhost:9235
2. Local State: Each replica has its own local LDAP database (BoltDB at /var/lib/opencloud/idm/idm.boltdb)
3. No Replication: The embedded IDM doesn't support replication across instances
4. Autoprovisioning Conflicts: When users are autoprovisioned, they're created locally in each instance's LDAP, leading to inconsistencies

The error in your logs confirms this:

"error":"LDAP Result Code 200 \"Network Error\": dial tcp [::1]:9235: connect: connection refused"

OpenCloud's Architecture Limitations

According to the IDM service documentation:

"It is mainly targeted at small OpenCloud installations. For larger setups it is recommended to replace IDM with a "real" LDAP server or to switch to an external identity management solution."

Multi-replica deployments definitely fall into the "larger setups" category.

Solution

For HA deployments, you should:

1. Use external LDAP/Keycloak (which you're already doing ✅)
2. Disable the embedded IDM service completely
3. Ensure all OpenCloud instances point to the same external identity provider

Helm Chart Documentation

You're absolutely right about the misleading opencloud.replicas description. The current text suggests disabling persistence would work, but that's incorrect. The documentation should be updated to clarify that multiple replicas require:

• External identity provider (Keycloak/LDAP)
• Shared storage (S3 or ReadWriteMany volumes)
• Distributed cache configuration (Redis/NATS)

Would you like to create a PR to update the documentation?

cc @butonic @micbar @rhafer - This might need deeper discussion about OpenCloud's HA architecture and whether the embedded IDM should be automatically disabled when replicas > 1.

[micbar]

Correct.

Multiple replicas need RWX storage (also for S3)

We suggest either

1. CephFS
2. GPFS
3. NFS (slowest version, only nfsv4 with xattr support)

[Tim-herbie]

@michaelstingl
Is it yet possible to disable the embedded IDM service completely or to integrate a dedicated redis service for the cache?

[micbar]

Use a nats cluster for the cache. There is a helm chart for it.

[Tim-herbie]

Are you sure that it´s possible to use a external nats right now?

The parameter NATS_NATS_HOST seems to bind the port on the opencloud pod and not the server which should be used.

EDIT: I found the doc and will test it: https://github.com/opencloud-eu/opencloud/blob/main/services/nats/README.md

[michaelstingl]

Hi @Tim-herbie,

While PR #84 doesn't directly address the multiple replicas issue, we've added a warning in the README about this limitation:

Note: When using internal Keycloak with multiple OpenCloud replicas (opencloud.replicas > 1), you must use an external shared database or LDAP. The embedded IDM does not support replication. See issue #53 for details.

This is now documented in the Keycloak Settings section to prevent others from running into the same issue.

The proper solution for HA deployments remains:

1. Use external Keycloak (now properly supported with PR feat: implement external Keycloak support (fixes #75, #82) #84)
2. Or use external LDAP/database for the embedded IDM

Thanks for reporting this issue - it's an important limitation that needed to be clearly documented!

[Tim-herbie]

@michaelstingl Thank you for your help.
I haven’t found a way to disable the embedded IDM. Is there a parameter or configuration option to turn it off?

Here’s what I’ve done so far:

Connected an external Keycloak instance

Configured RWX volumes and external S3 (MinIO)

However, I’m still running into the same issue.
I also tried to set up an external NATS instance, but I haven’t been successful yet.

At this point, I’m not sure how to completely disable the embedded IDM and rely solely on Keycloak for identity and access management.

[michaelstingl]

@Tim-herbie I think there's some confusion - I'm just a Helm chart reviewer, not an OpenCloud backend expert. For questions about disabling the embedded IDM, you'll need help from @micbar who knows the OpenCloud architecture.

@micbar Could you clarify if it's possible to disable the embedded IDM service when using external Keycloak? This seems like a backend limitation that prevents true HA deployments.

This might need to be addressed in the OpenCloud backend (https://github.com/opencloud-eu/opencloud) rather than in the Helm charts.

[micbar]

OC_EXCLUDE_RUN_SERVICES=idp,idm