VMware CBT migration architecture

This document describes the implemented Apache CloudStack architecture for warm VMware-to-KVM migration based on VMware Changed Block Tracking (CBT). Detailed API, database, configuration, and operational information is in README.md. KVM host preparation is documented in ../vmware-cbt-migration.md.

Design invariant

The KVM-side replica remains byte-compatible with the VMware source disk until the final CBT cycle has completed. Raw VMware CBT ranges are never applied to a disk after guest conversion has changed its layout.

The migration therefore has two distinct phases:

1. Replication: copy the initial source-equivalent baseline and apply CBT delta ranges to it.
2. Finalization: after the source VM is powered off, apply one final delta and run virt-v2v locally against the current KVM-side replica.

This avoids a second full VMware read during cutover. CloudStack does not power off the source VM; shutdown remains an explicit operator action.

Components

Management server

The API commands are:

• checkVmwareCbtMigrationPrerequisites
• startVmwareCbtMigration
• listVmwareCbtMigrations
• syncVmwareCbtMigration
• cutoverVmwareCbtMigration
• cancelVmwareCbtMigration
• deleteVmwareCbtMigration

VmwareCbtMigrationManagerImpl owns migration orchestration, host and storage validation, state transitions, cutover policy, agent command dispatch, import, and cleanup. VmwareCbtMigrationServiceImpl performs VMware inventory, snapshot, power-state, CBT enablement, and changed-area operations.

VmwareCbtStorageTarget classifies the selected primary storage and determines the target representation and finalization constraints.

Persistence

The feature persists its state in three CloudStack tables:

• cloud.vmware_cbt_migration: one row per migration session.
• cloud.vmware_cbt_migration_disk: source and target metadata for each disk, including the current per-disk VMware changeId.
• cloud.vmware_cbt_migration_cycle: metrics and status for each delta cycle.

The schema and configuration entries are delivered through the standard 4.23.0.0-to-24.0.0 database upgrade path:

• engine/schema/src/main/resources/META-INF/db/schema-42300to2400.sql
• engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade42300to2400.java

KVM agent

The management server sends these commands to the selected conversion host:

• VmwareCbtPrepareCommand: create or resolve target disks and perform the initial VDDK full copy.
• VmwareCbtSyncCommand: read and apply VMware CBT ranges.
• VmwareCbtCutoverCommand: finalize the current replica with virt-v2v.
• VmwareCbtCleanupCommand: terminate migration-owned processes and remove migration-owned temporary artifacts.
• VmwareCbtRbdProbeCommand: actively validate RBD create/write/read/delete access for the selected pool.

The corresponding KVM wrappers are LibvirtVmwareCbtPrepareCommandWrapper, LibvirtVmwareCbtSyncCommandWrapper, LibvirtVmwareCbtCutoverCommandWrapper, LibvirtVmwareCbtCleanupCommandWrapper, and LibvirtVmwareCbtRbdProbeCommandWrapper.

User interface

The workflow is integrated into Import/Export Instances. The UI starts async operations through the API and uses listVmwareCbtMigrations as the detailed source for migration, disk, and cycle state.

End-to-end flow

VMware source VM
    |
    | preflight and source discovery
    v
baseline VMware snapshot
    |
    | nbdkit-vddk full logical disk read
    v
source-equivalent replica on KVM primary storage
    |
    | repeated VMware snapshots + QueryChangedDiskAreas
    | changed ranges copied through VDDK and patched in place
    v
ReadyForCutover
    |
    | operator powers off the VMware VM
    | final CBT delta
    | local virt-v2v finalization
    v
CloudStack KVM import

Preflight

Preflight resolves the source, destination cluster, conversion host, and primary storage. It validates source CBT state, source disks, snapshots, storage support, host block-copy and in-place-finalization capabilities, RBD or LINSTOR access where applicable, Windows conversion dependencies, and optional compute-offering sizing.

The selected conversion host reports capability through host details. The principal gates are:

• host.vddk.blockcopy.support
• host.vddk.blockcopy.inplace.finalization.support
• host.vddk.blockcopy.rbd.support
• vddk.lib.dir
• host.vddk.version
• qemu and virt-v2v version details

Initial synchronization

Start creates the migration and disk records, enables CBT when required, creates a baseline VMware snapshot, prepares destination targets, and dispatches VmwareCbtPrepareCommand.

The KVM agent exposes each snapshot disk with nbdkit-vddk. It uses qemu-img convert for the general copy path and can use nbdcopy as an optional accelerator for supported pre-created or NBD-bridged targets. The result remains source-equivalent; virt-v2v is not run during initial sync.

After a successful copy, CloudStack records the baseline per-disk changeId and removes the baseline snapshot when possible.

Delta synchronization

Each synchronization cycle:

1. Creates a VMware snapshot.
2. Calls VMware QueryChangedDiskAreas from the previous per-disk changeId.
3. Persists the changed ranges and cycle state.
4. Opens the snapshot disk through nbdkit-vddk on the KVM host.
5. Copies only the changed ranges and writes them at their original byte offsets with qemu-io.
6. Records changed bytes, duration, dirty rate, and the new changeId.
7. Removes the cycle snapshot when possible.

VmwareCbtMigrationCutoverPolicy evaluates the configured minimum, maximum, and quiet-cycle thresholds and moves the migration to ReadyForCutover when the policy boundary is reached.

Cutover

Cutover is accepted only when the source VM is powered off. CloudStack runs one final CBT cycle, then finalizes the current KVM-side replica with one of these paths:

• virt-v2v-in-place;
• virt-v2v --in-place; or
• regular virt-v2v -o local for QCOW2 file targets only, when the explicit non-in-place fallback configuration is enabled.

After finalization, CloudStack imports the disks through the shared external KVM import path. A successful import sets the migration to Completed. If finalization succeeds but import fails, the migration remains ReadyForImport and a retry repeats only the import stage.

Storage targets

Primary storage	Replica	Initial copy	Delta writes	Finalization
NFS, filesystem, shared mount	QCOW2 file	qemu-img	qemu-io -f qcow2	In-place preferred; explicitly gated local fallback supported
Ceph/RBD	Raw RBD image	Direct qemu RBD path or nbdcopy through a local NBD bridge	qemu-io -f raw	In-place only
LINSTOR	Pre-created raw DRBD block device	nbdcopy when available, otherwise qemu-img	qemu-io -f raw	In-place only

For filesystem targets, temporary replicas are stored under a migration-owned cloudstack-cbt/<migration-uuid>/ directory. RBD and LINSTOR resources use CloudStack-owned migration markers. Cleanup is restricted to these owned paths and resource names.

State model

Migration states:

Created
  -> InitialSync
  -> Replicating
  -> ReadyForCutover
  -> CuttingOver
  -> ReadyForImport
  -> Completed

Failed and Cancelled are terminal states. ReadyForImport is deliberately retryable.

Disk states are Created, Prepared, Syncing, Ready, and Failed. Cycle states are Created, QueryingChangedAreas, CopyingChangedBlocks, Completed, and Failed.

Concurrency and cleanup

Sync and cutover commands are serialized by migration ID. Cancellation sends a best-effort agent cleanup command before marking a non-terminal migration as cancelled. Failed and cancelled migrations can delete their migration-owned temporary replicas when cleanup is requested.

Deleting a completed migration is record-only. It removes CBT bookkeeping but does not delete the imported VM, its CloudStack volumes, or finalized primary storage artifacts.

VMware snapshots are removed after initial and delta attempts when possible. Snapshot identifiers and cleanup errors remain visible in migration state so operators can identify VMware-side cleanup work.

Credential handling

Existing registered vCenter credentials are resolved from CloudStack‘s VMware datacenter record. External-source credentials are accepted by sensitive API commands and the password is stored through CloudStack’s encrypted DAO field handling until the migration no longer needs it.

On the KVM host, the password is written to a temporary owner-only file and passed to nbdkit with password=+<file>. It is not placed directly in the process command line. Returned errors are sanitized before persistence.

Operational boundaries

• The operator is responsible for shutting down the source VM before cutover.
• Initial baseline transfer size depends on the extents exposed by VMware VDDK and the source datastore; CBT optimization applies to subsequent cycles.
• Invalidated VMware CBT state requires a new baseline migration.
• RBD and LINSTOR require in-place finalization and active backend access from the selected conversion host.
• Windows guests require a usable virtio-win driver bundle on that host.
• The source VMware VM remains authoritative until cutover succeeds.