This document describes the implemented VMware Changed Block Tracking (CBT) migration flow for Apache CloudStack and the host-level prerequisites required on KVM migration hosts.
The goal of this flow is to reduce the downtime and bandwidth needed when migrating VMware guests to KVM. Instead of repeatedly copying the full VMware disk, CloudStack performs one initial full synchronization and then uses VMware CBT to copy only changed disk ranges until the VM is quiet enough for final cutover.
No additional CloudStack software is installed on the VMware ESXi hosts or vCenter for this flow. The extra migration tooling is installed on the selected CloudStack KVM host that performs the VDDK read, virt-v2v conversion, and CBT delta application work.
The implementation contains the CloudStack management model, API surface, state tracking, cutover policy, VMware changed-block query orchestration, KVM agent initial sync, KVM agent delta sync, cutover finalization, import retry handling, delete/cancel cleanup, and UI status tracking.
Start a VMware CBT migration session. CloudStack records the destination zone, KVM cluster, optional conversion host, optional storage pool, source vCenter/ESXi details, and discovered VMware disks.
Run the initial full sync. The KVM agent exposes each VMware snapshot disk through nbdkit-vddk and copies it to a source-equivalent target on KVM primary storage. It uses qemu-img convert for the general path and can use nbdcopy as an optional accelerator for supported targets. virt-v2v does not run in this phase.
Register target disks. The management server records the target disk path, format, initial VMware CBT change ID, and snapshot reference for each migrated disk.
Run CBT delta synchronization cycles. For each cycle CloudStack creates a VMware snapshot, calls QueryChangedDiskAreas() for the source disks, dispatches changed byte ranges to the selected KVM host, records copied bytes and dirty rate, and removes the VMware snapshot.
Decide when to cut over. The decision is controlled by global settings:
vmware.cbt.migration.min.cyclesvmware.cbt.migration.max.cyclesvmware.cbt.migration.quiet.cyclesvmware.cbt.migration.quiet.bytesvmware.cbt.migration.quiet.dirty.rateLong-running KVM agent commands are controlled separately by:
vmware.cbt.migration.agent.command.timeoutFinal cutover. After the operator powers down the VMware VM, CloudStack performs the final delta sync, runs virt-v2v locally against the current KVM-side replica, and imports/registers the VM on KVM.
CBT initial full sync, regular delta sync, final delta sync, and cutover finalization run as KVM agent commands. The command wait value and the agent-side child process timeout are both controlled by:
vmware.cbt.migration.agent.command.timeout=86400
The value is in seconds. The default is 24 hours. It is intentionally separate from convert.vmware.instance.to.kvm.timeout, which applies to the older OVF/VDDK ConvertInstanceCommand import path, and from remote.kvm.instance.disks.copy.timeout, which applies to remote KVM unmanaged disk copy imports.
For very large source disks, set vmware.cbt.migration.agent.command.timeout above the worst-case expected duration. A 1 TiB initial sync can easily require multiple hours depending on VDDK transport, ESXi storage, destination storage, and sparse extent behavior. The setting is finite because the shared CloudStack script runner treats zero as its default one-hour timeout, not as “no timeout”.
CBT migration runs on a KVM host in the destination cluster. If the API call does not specify a conversion host, CloudStack selects an enabled KVM routing host in the destination cluster that reports:
host.vddk.blockcopy.support=true
The KVM agent reports this capability during host startup. It is true only when the host satisfies the required VDDK, virt-v2v, qemu-img, qemu-nbd, and qemu-io checks.
Related host detail keys:
host.vddk.support vddk.lib.dir host.vddk.version host.vddk.blockcopy.support host.vddk.blockcopy.inplace.finalization.support host.vddk.blockcopy.rbd.support host.qemu.img.version host.qemu.nbd.version host.qemu.io.version host.virtv2v.version host.virtv2v.inplace.version
Restart cloudstack-agent after installing packages or changing agent.properties so these details are refreshed on the management server.
Each KVM host that can be selected for VMware CBT migration needs:
virt-v2v for final cutover conversion.nbdkit and an nbdkit VDDK plugin.lib64/libvixDiskLib.so.qemu-img for target image inspection, creation, and manipulation.qemu-nbd for block device exposure during incremental copy workflows.qemu-io for applying and verifying changed block ranges.nbdcopy from libnbd to accelerate supported initial full-copy paths. CloudStack falls back to qemu-img when it is unavailable.virtio-win drivers when migrating Windows guests.Destination-specific requirements:
virt-v2v support (the virt-v2v-in-place binary or the virt-v2v --in-place option).virt-v2v support, and the conversion host must be a LINSTOR satellite connected to the destination pool (the CloudStack primary storage must be attached to the host, and the host's hostname must match its LINSTOR node name). Target volumes are pre-created through the Linstor storage adaptor and written as raw data to the local DRBD device path, so no extra qemu block driver is needed. DRBD permits only one writer: make sure no other node holds the target resource Primary while a migration writes to it.You do not have to install these packages on every KVM host in the zone unless you want every KVM host to be eligible for VMware CBT migration. A common deployment model is to prepare a smaller set of conversion-capable KVM hosts and explicitly select one of them for migration work.
The KVM agent checks these commands:
virt-v2v --version nbdkit vddk --dump-plugin libdir=<vddk-lib-dir> qemu-img --version qemu-nbd --version qemu-io --version
On Ubuntu and Debian hosts the conversion support check also verifies that the nbdkit package is installed with dpkg -l nbdkit.
Install VMware VDDK on every KVM host that may run a VMware-to-KVM conversion or CBT replication session. A common layout is:
mkdir -p /opt/vmware tar -xzf VMware-vix-disklib-*.tar.gz -C /opt/vmware
The extracted directory is usually named:
/opt/vmware/vmware-vix-disklib-distrib
CloudStack auto-detects the first directory named vmware-vix-disklib-distrib. If auto-detection is not desirable, configure the path explicitly in the KVM agent properties:
vddk.lib.dir=/opt/vmware/vmware-vix-disklib-distrib
Optional VDDK settings:
vddk.transports=nbdssl:nbd vddk.thumbprint=<vcenter-sha1-thumbprint>
If vddk.thumbprint is not set, CloudStack attempts to compute the vCenter thumbprint on the KVM host.
On an EL8-compatible KVM host, the CloudStack agent package already brings in the normal KVM runtime dependencies such as libvirt, qemu-kvm, qemu-img, and Python libvirt bindings. The VMware CBT path additionally needs virt-v2v, nbdkit, the nbdkit VDDK plugin, qemu-nbd support, and VDDK itself.
Example package installation:
dnf install -y cloudstack-agent dnf install -y virt-v2v nbdkit qemu-img qemu-kvm-core qemu-kvm-block-curl libnbd dnf install -y nbdkit-plugin-vddk || dnf install -y nbdkit-vddk-plugin
Package names can differ between RHEL, Oracle Linux, Rocky, AlmaLinux, and enabled module streams. The important validation is not the exact package name, but that these commands work:
virt-v2v --version qemu-img --version qemu-nbd --version qemu-io --version nbdkit vddk --dump-plugin libdir=/opt/vmware/vmware-vix-disklib-distrib
For Windows guests, install a virtio-win package or otherwise make the VirtIO drivers available to virt-v2v. Some EL8 environments provide this from an additional repository rather than the base distribution repositories.
Distribution virtio-win packages can lag the upstream project and omit drivers for recent Windows releases. virt-v2v treats missing drivers as a non-fatal warning and would deliver a guest that cannot boot from its VirtIO disk, so both the cold import and the CBT cutover conversion fail fast instead. When virt-v2v reports that no VirtIO drivers are available, the operation is aborted and identifies the conversion host's /usr/share/virtio-win/virtio-win.iso as the file to update. Install the upstream latest-virtio ISO on every conversion host:
curl -L -o /usr/share/virtio-win/virtio-win.iso \ https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso
Windows Server 2025 additionally rejects the optional smbus driver's signature at first boot, which loops the virt-v2v firstboot driver installation forever; rebuild the ISO without smbus.inf/smbus.cat before converting Windows Server 2025 guests (the admin guide page “Importing VMware VMs into KVM” carries the exact commands). Older Windows releases work with the unmodified ISO.
Those releases require CPU features that the KVM agent's default guest CPU model (qemu64) does not expose. The conversion itself is unaffected — the disks convert, the drivers install, the imported instance starts and reaches its UEFI firmware — but the guest then drops into Preparing Automatic Repair instead of the login screen, on every boot. Nothing in the migration output indicates a problem, so it reads like a corrupt image.
Set a modern named model on every KVM host that will run such guests, in /etc/cloudstack/agent/agent.properties, and restart the agent:
guest.cpu.mode=custom guest.cpu.model=<a model your hosts support>
The model has to be one the host can actually provide, and that differs by processor vendor and generation — Broadwell-noTSX-IBRS or newer on Intel, EPYC-IBPB or newer on AMD. Check before setting it:
virsh domcapabilities | grep "usable='yes'"
A model reported as usable='no' is refused at domain start with “the CPU is incompatible with host CPU: Host CPU does not provide required features”, and the instance never launches — including system VMs, if the host is the one running them. In a cluster, pick a model every host reports as usable so live migration stays possible.
guest.cpu.mode=host-model and host-passthrough also work and need no such check, but they tie the instance to the host's CPU and prevent live migration between hosts with different processors.
This applies equally to instances deployed natively on KVM — it is a host prerequisite for the guest OS, not a property of the migration.
If the nbdkit VDDK plugin package conflicts with the installed nbdkit module stream, align the enabled AppStream or vendor repository so nbdkit and the VDDK plugin come from compatible builds.
On Ubuntu 24.04, the base KVM and conversion packages are available from the standard repositories, but the nbdkit VDDK plugin is not necessarily provided by the default Ubuntu archive. Plan for an internal package, a vendor-provided package, or a locally built nbdkit VDDK plugin.
Example base package installation:
apt-get update apt-get install -y cloudstack-agent apt-get install -y qemu-utils qemu-system-x86 libvirt-daemon-system libvirt-clients apt-get install -y virt-v2v nbdkit libguestfs-tools libnbd-bin
Then install or build the nbdkit VDDK plugin and verify:
nbdkit vddk --dump-plugin libdir=/opt/vmware/vmware-vix-disklib-distrib
qemu-img and qemu-nbd are provided by qemu-utils on Ubuntu 24.04.
For Windows guests, ensure a virtio-win package or equivalent VirtIO driver bundle is available. The KVM agent check looks for a package named virtio-win on Debian-based hosts, so an internal package with that name is the easiest way to satisfy the current check. The upstream-ISO, smbus and guest CPU model notes above apply here as well.
The selected KVM migration host must be able to reach:
The VMware account used for migration must be able to discover the VM, create and remove snapshots, read virtual disk data, and query changed disk areas.
VMware CBT must be enabled and valid for the source VM disks before incremental cycles can be trusted. A full resync is required when VMware invalidates CBT state, for example after CBT reset, certain disk changes, failed snapshot consolidation, or storage operations that change disk backing identity.
Avoid or explicitly handle VMware disk types that do not participate cleanly in CBT, such as independent disks and unsupported raw device mappings.
Both the cold import and the CBT warm migration derive sensible defaults from the source VM instead of falling back to generic import defaults. Everything below applies only when the caller did not specify the value explicitly - operator-provided parameters always win.
config.guestFullName / config.guestId, available regardless of power state). Windows releases newer than the guest OS catalog map to the nearest older entry of the same family (for example Windows Server 2025 maps to the Windows Server 2022 type until the catalog catches up); the mapped type keeps Windows clock and Hyper-V enlightenment semantics correct on KVM.UEFI setting (SECURE when the source has secure boot enabled) and the q35 machine type. Previously every import defaulted to BIOS, which leaves a UEFI source's GPT/EFI disk unbootable.vga; the libvirt default (legacy cirrus) renders a blank console on Windows Server 2025 Server Core.nicipaddresslist entry keeps its guest-configured address when that address fits the target network’s CIDR, is not the gateway and is free; otherwise the NIC falls back to normal allocation. DHCP NICs are untouched and pick up the virtual router's lease after migration. Capture requires VMware Tools running in the source, which the warm flow guarantees (the source is live when the migration starts).Run this checklist on every candidate KVM migration host:
virt-v2v --version qemu-img --version qemu-nbd --version qemu-io --version nbdkit vddk --dump-plugin libdir=/opt/vmware/vmware-vix-disklib-distrib test -f /opt/vmware/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so systemctl restart cloudstack-agent
Then confirm the host details from the CloudStack API or UI show:
host.vddk.support=true host.vddk.blockcopy.support=true host.vddk.blockcopy.inplace.finalization.support=<true-or-false> host.vddk.blockcopy.rbd.support=<true-or-false> host.qemu.img.version=<detected> host.qemu.nbd.version=<detected> host.qemu.io.version=<detected> host.virtv2v.version=<detected>
If host.vddk.blockcopy.support is false, check the agent log on the KVM host and validate each command above manually. In-place and RBD capability can be false on hosts intended only for QCOW2 fallback, but those hosts are not eligible for RBD or LINSTOR cutover.
nbdssl or another encrypted VDDK transport where supported.convert.instance.env.tmpdir and convert.instance.env.virtv2v.tmpdir when the default temporary filesystem is too small.vmware.cbt.migration.agent.command.timeout before migrating very large VMs if the baseline copy or cutover finalization can exceed 24 hours.cs- plus the recorded volume path, e.g. cs-cbt-<marker>-<disk> for warm migrations). This is expected and does not affect volume lifecycle operations, which address volumes by their recorded path.