FE and BE store persisted mutable settings in fe_custom.conf and be_custom.conf. Cloud BE also saves configuration received in FE heartbeats. MS normally persists mutable settings into its startup configuration file, which its container entrypoint can replace from a ConfigMap on restart.
The operator can preserve these settings separately from user ConfigMaps for DorisCluster FE/BE and DorisDisaggregatedCluster FE/BE/MS. It does not change user ConfigMaps, mount an empty volume over the image's conf directory, create new PVCs, or change StatefulSet PVC templates.
On its first reconciliation, the operator records a cluster-level default in doris.apache.org/runtime-config-policy:
A new-cluster component without a suitable PVC keeps its existing behavior and emits RuntimeConfigStorageUnavailable. Its StatefulSet is marked unavailable for automatic persistence; adding storage later does not silently migrate it. Stateless clusters are not made durable by this feature.
An explicit request is strict:
spec: runtimeConfigPersistence: enabled: true
Missing safe storage or an incomplete existing-cluster migration blocks this request. Set enabled: false before initial deployment to opt out. Turning off an already activated policy is rejected: doing so could discard configuration that overrides the ConfigMap. No automatic reverse migration is provided.
The operator selects a writable, existing per-Pod PVC mount in this order:
meta_dir / doris-meta).storage_root_path / storage).A file_cache_path root or overlapping directory is excluded because cache maintenance can delete the entire root. EmptyDir, read-only mounts and mounts using an existing subPathExpr are excluded. A cache-only cloud BE needs a safe persistent log/configuration volume before explicit enablement. Directory selection is recorded on the StatefulSet and is not silently switched later.
A directory .doris-runtime-config/<pod-name> inside the selected mount is exposed at /var/lib/doris/runtime-config. The Pod name separates overrides even when an existing PVC is shared. Existing subPath mounts retain their prefix.
For DDC compute-group scale-down, the operator retains the selected runtime configuration PVC so scaling back to the same ordinal can recover its overrides and migration marker. Other PVCs, including file cache PVCs, follow the existing cleanup policy. Retained configuration claims continue to consume storage while the ordinal is absent. This does not change deletion of the cluster or manual PVC deletion, and does not initialize cloud identity for entirely new nodes.
An init container uses the same image and security context as the main container to create the directory and prepare a writable copy of the base ConfigMap configuration. When no base ConfigMap exists, it uses the image's base configuration. It injects:
| Component | Injected setting | Persisted file |
|---|---|---|
| FE | custom_config_dir=/var/lib/doris/runtime-config | fe_custom.conf |
| BE | custom_config_dir=/var/lib/doris/runtime-config | be_custom.conf |
| MS | custom_conf_path=/var/lib/doris/runtime-config/ms_custom.conf | ms_custom.conf |
The main container retains its normal entrypoint. Runtime files are created only when absent; image/configuration updates never truncate them. Directory mounts allow the kernel to create temporary files and replace files with rename.
Explicit custom paths already backed by writable PVCs remain user-managed. An explicit nondefault path without durable backing is rejected. ConfigMap-backed initial *_custom.conf files seed the persistent file once and are excluded from the prepared ConfigMap directory to prevent read-only symlinks.
Images must contain Bash, cp, awk and the existing entrypoint, and support the corresponding kernel configuration setting. Nonroot deployments must supply working PVC/emptyDir permissions (for example the appropriate Pod fsGroup). This feature does not change container users or elevate privileges.
Upgrade CRDs and the operator first. Leave the persistence spec unset.
Stop dynamic configuration updates, scaling, image updates, and concurrent automation for the migration window. All existing Pods must be Ready.
Back up the live overrides before replacing any Pod:
python3 tools/runtime-config-migrate.py \ --kind ddc --namespace example --name doris \ --backup-dir ./runtime-backup # Use --kind dcr for DorisCluster; --context selects a kubectl context.
Review the private backup directory. Files are mode 0600 and the directory is mode 0700; configuration values are not printed or stored in annotations. In particular, MS candidates contain effective settings that differ from the currently mounted ConfigMap. The tool cannot reconstruct which historical values were deliberately changed: review MS candidates and remove obsolete overrides. FDB endpoints and configuration-path settings are excluded.
Apply the reviewed backup:
python3 tools/runtime-config-migrate.py \ --kind ddc --namespace example --name doris \ --backup-dir ./runtime-backup --apply
The tool checks cluster, StatefulSet and Pod identities, replicas, generations, and unchanged live source configuration. It copies each backup through the old Pod‘s existing PVC mount, verifies the copy, and only then patches the migration annotation and enabled: true. It never silently replaces a different existing backup. The operator validates the approval before changing workloads. Migrated ordinals require their on-disk migration marker; newly added ordinals initialize normally. Follow the operator’s normal rolling update and verify loaded settings and a subsequent Pod replacement before reopening configuration updates.
If there is no suitable PVC, an explicit custom path already exists, an MS base ConfigMap is unavailable, or membership changes, the tool stops. Such clusters need an individually planned storage/configuration migration. The tool does not recreate StatefulSets or recover overrides after an old Pod is already lost. Keep local backups until validation is complete.
Persistence closes the loss of mutable configuration during Pod replacement. A new cloud BE still needs its first FE heartbeat. Kernel query admission and FE scheduling must independently prevent queries before required initialization.