Operator does not pick up ClickHouseInstallation after applying ClickHouseOperatorConfiguration without a restart

[patsevanton]

When a ClickHouseOperatorConfiguration resource is applied to change the watched namespaces after the operator is already running, the operator does not dynamically reload its namespace watch configuration. As a result, any ClickHouseInstallation resources created in the newly watched namespace are silently ignored until the operator pod is manually restarted.

Steps to reproduce

1. Install the Altinity ClickHouse Operator (v0.26.0) via Helm into the clickhouse-operator namespace:

helm upgrade --install clickhouse-operator altinity/altinity-clickhouse-operator \
  --version 0.26.0 \
  --namespace clickhouse-operator \
  --create-namespace \
  --wait

2. Apply a ClickHouseOperatorConfiguration that tells the operator to watch the clickhouse namespace:

kubectl apply -n clickhouse-operator -f clickhouse-operator-config.yaml

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseOperatorConfiguration"
metadata:
  name: "watch-clickhouse-namespace"
spec:
  watch:
    namespaces:
      include:
        - "clickhouse"
      exclude: []

3. Create a ClickHouseInstallation in the clickhouse namespace:

kubectl apply -f clickhouse.yaml

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: sentry-clickhouse
  namespace: clickhouse
spec:
  configuration:
    clusters:
      - name: single-node
        layout:
          shardsCount: 1
          replicasCount: 1
    users:
      default/networks/ip:
        - "0.0.0.0/0"
  defaults:
    templates:
      podTemplate: clickhouse-single-node
      dataVolumeClaimTemplate: data-volume
  templates:
    podTemplates:
      - name: clickhouse-single-node
        spec:
          containers:
            - name: clickhouse
              image: altinity/clickhouse-server:25.3.6.10034.altinitystable
    volumeClaimTemplates:
      - name: data-volume
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

4. Wait 2+ minutes and observe that no pods, services, statefulsets, or events are created in the clickhouse namespace:

$ kubectl get pods -n clickhouse
No resources found in clickhouse namespace.

$ kubectl get clickhouseinstallation -n clickhouse
NAME                STATUS   CLUSTERS   HOSTS   HOSTS-COMPLETED   AGE
sentry-clickhouse                                                  2m14s

$ kubectl get svc -n clickhouse
No resources found in clickhouse namespace.

$ kubectl get statefulsets -n clickhouse
No resources found in clickhouse namespace.

The ClickHouseInstallation STATUS, CLUSTERS, and HOSTS fields remain empty — the operator has not started reconciliation.

5. Restart the operator:

kubectl rollout restart deployment -n clickhouse-operator

6. After the restart, the operator immediately picks up the ClickHouseInstallation and begins provisioning ClickHouse — pods, services, and statefulsets appear within seconds:

$ kubectl get pods -n clickhouse
NAME                                      READY   STATUS    RESTARTS   AGE
chi-sentry-clickhouse-single-node-0-0-0   1/1     Running   0          63s

$ kubectl get clickhouseinstallation -n clickhouse
NAME                STATUS      CLUSTERS   HOSTS   HOSTS-COMPLETED   AGE
sentry-clickhouse   Completed   1          1                         9m15s

$ kubectl get svc -n clickhouse
NAME                                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
chi-sentry-clickhouse-single-node-0-0   ClusterIP   None         <none>        9000/TCP,8123/TCP,9009/TCP   5m59s
clickhouse-sentry-clickhouse            ClusterIP   None         <none>        8123/TCP,9000/TCP            10s

Expected behavior

The operator should dynamically reload its configuration when a ClickHouseOperatorConfiguration resource is created or updated, and begin watching the newly specified namespaces without requiring a manual restart. Any ClickHouseInstallation resources already present in those namespaces should be reconciled automatically.

Actual behavior

The operator ignores the ClickHouseOperatorConfiguration change at runtime. It only picks up the new namespace watch configuration after a pod restart (kubectl rollout restart). Until then, ClickHouseInstallation resources in the new namespace are not reconciled at all.

Workaround

Restart the operator deployment after applying ClickHouseOperatorConfiguration:

kubectl rollout restart deployment -n clickhouse-operator

Environment

• Altinity ClickHouse Operator: v0.26.0 (Helm chart)
• Kubernetes: Yandex Cloud Managed Kubernetes
• ClickHouse image: altinity/clickhouse-server:25.3.6.10034.altinitystable

[alex-zaitsev]

This is by design. Some changes in operator configuration may trigger restart of all managed CHI, so there is two step process. But perhaps we can make it configurable.