Support hot-reloading of Ditto-specific HOCON configuration via Kubernetes ConfigMap file watching

[thjaeckle]

Motivation

When running Ditto on Kubernetes, configuration is typically provided via ConfigMaps mounted as .conf files into pods. Currently, any configuration change — even to Ditto-specific application settings — requires a pod restart to take effect. This is disruptive for operational tuning (e.g., adjusting logging levels, timeouts, rate limits, or feature toggles) and causes unnecessary downtime.

Additionally, the Helm chart currently requires manual updates whenever new HOCON configuration options are introduced in Ditto. This is cumbersome, error-prone, and means not all configurations are exposed via values.yaml.

Kubernetes already updates ConfigMap-mounted files atomically (via symlink swap) without restarting pods. Ditto should be able to leverage this mechanism for its own configuration namespace.

Proposal

Introduce a dynamic configuration reload mechanism for ditto.*-prefixed HOCON configuration, combined with a pass-through HOCON block in the Helm chart that eliminates the need to manually map every HOCON key to a Helm value.

Static configuration (unchanged behavior)

• Scope: Pekko, JVM, and infrastructure-level settings (dispatchers, serialization, cluster config, database connections, etc.)
• Deployment: Mounted via a "static" ConfigMap
• Reload: Requires pod restart (as today)

Dynamic configuration (new capability)

• Scope: ditto.*-prefixed application configuration only
• Deployment: Mounted via a separate "dynamic" ConfigMap, populated from a pass-through HOCON block in values.yaml
• Reload: Automatically detected and applied without pod restart

Design Considerations

Pass-through HOCON in Helm values

Instead of mapping every ditto.* config key to an individual Helm value (which requires manual Helm chart updates for each new config option), the dynamic ConfigMap is populated from a free-form HOCON pass-through block in values.yaml:

# values.yaml
things:
  # Structured Helm values for critical/common settings (image, resources, replicas, etc.)
  replicaCount: 1

  # Free-form HOCON pass-through — rendered directly into the dynamic ConfigMap
  dynamicConfig: |
    ditto.things {
      log-incoming-messages = true
      thing {
        snapshot.interval = 30s
      }
    }

The dynamic ConfigMap template simply renders it:

data:
  ditto-dynamic.conf: {{ .Values.things.dynamicConfig | quote }}

This approach:

• Eliminates the sync burden — users can set any ditto.* key without a Helm chart update
• Is proven at scale — used by Kafka, Elasticsearch, Cassandra, and most mature Helm charts
• Pairs naturally with hot-reload — the pass-through block is the content of the dynamic ConfigMap

Structured Helm values are kept for critical operational settings (image tags, replica counts, resource limits, database URIs, etc.) that benefit from schema validation and Helm upgrade semantics.

File watching mechanism

Kubernetes updates ConfigMap mounts by creating a new timestamped directory and atomically swapping the ..data symlink. Java NIO's WatchService can be unreliable with symlinks in containers. A robust alternative is periodic polling (e.g., every 30 seconds) checking file modification time or content hash via the Pekko Scheduler.

Reload flow

1. Detect change in the mounted dynamic config file (poll-based)
2. Call ConfigFactory.invalidateCaches() to clear Typesafe Config's internal cache
3. Re-parse only the dynamic config file via ConfigFactory.parseFile()
4. Publish a config-changed event on the ActorSystem EventStream
5. Subscribed actors/extensions receive the new config and apply it

Helm chart: preventing restarts for dynamic config changes

The Ditto Helm chart should support separating configuration into:

• A static ConfigMap — triggers pod restart on change (via checksum/config-static pod annotation)
• A dynamic ConfigMap — does not trigger pod restart (no checksum annotation); contents are updated in-place by Kubernetes and hot-reloaded by Ditto

The existing checksum/config annotation on the pod template must reference only the static ConfigMap. The dynamic ConfigMap is mounted as an additional volume but deliberately excluded from checksum annotations, so changes to it do not cause a rolling restart.

Gradual adoption

Not all ditto.* configuration needs to be hot-reloadable from the start. Individual config consumers (actors, extensions) opt in by subscribing to config-change events. Non-subscribed components continue to use their startup config. This allows incremental migration of settings to dynamic reload.

Requirements

Must Have

1. Polling-based file change detection for a configurable config file path, using the Pekko Scheduler
2. EventStream-based notification when the dynamic config changes
3. Separation of static and dynamic config — dynamic reload only applies to ditto.*-prefixed configuration
4. Pass-through HOCON block in Helm values.yaml (dynamicConfig) rendered directly into the dynamic ConfigMap, eliminating the need for manual Helm chart updates per config key
5. Helm chart support for mounting a separate dynamic ConfigMap with no checksum annotation (no pod restart on change)
6. Logging of config reload events (success, failure, what changed)

Should Have

1. Configurable poll interval (default: 30s)
2. Validation of new config before applying (parse errors should be logged and ignored, keeping the previous valid config)
3. Initial set of hot-reloadable settings (e.g., logging levels, feature toggles) as proof of concept

Non-Functional Requirements

• No impact on startup behavior when no dynamic config file is present
• Reload must be non-blocking and safe for the actor system
• Must work across all five Ditto services