docs/vmware-cbt/README.md

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# VMware CBT Migration Implementation Notes

## Introduction

This document describes the VMware Changed Block Tracking (CBT) migration design
and implementation for Apache CloudStack 24.0.0. It is written as source material
for an Apache CloudStack CWIKI architecture page and should be treated as an
implementation guide, not only as a design sketch.

The feature provides a warm VMware-to-KVM migration flow. CloudStack creates a
KVM-side source-equivalent disk replica from a VMware VDDK snapshot, keeps that
replica current with one or more VMware CBT delta cycles, requires the operator
to power off the VMware source VM, runs one final CBT delta cycle, finalizes the
replica with `virt-v2v`, and imports the finalized disk(s) as a regular
CloudStack KVM VM.

The essential architectural point is that VMware CBT migration is a two-phase
disk strategy:

- Phase 1 keeps a KVM-side source-equivalent replica current using VDDK and
  VMware CBT.
- Phase 2 finalizes the fully synchronized replica with `virt-v2v` only after
  the source VM is powered off.

This gives CloudStack warm migration behavior without trying to apply VMware
changed blocks to a disk that has already been transformed for KVM, and without
requiring a second full VMware read at cutover time.

## Problem Statement

The existing VMware import paths are useful for one-time imports, but they do
not provide a warm migration loop where the target disk is kept current while
the source VM continues to run on VMware. Large VMs therefore either require a
longer downtime window or repeated full-copy style attempts.

VMware CBT provides the changed-block metadata needed for a replication-style
workflow, but the design must avoid a common trap: once `virt-v2v` modifies a
disk for KVM guest bootability, VMware CBT byte ranges can no longer be safely
applied to that transformed disk. The implementation must therefore keep the
replication target source-equivalent until final cutover, then run guest
conversion only after the final delta is applied.

## Goals

- Provide an operator-driven warm migration flow from VMware to CloudStack KVM.
- Support both existing registered vCenter records and external vCenter details
  supplied during import.
- Run the initial full copy once through VDDK and `qemu-img`.
- Apply subsequent VMware CBT delta cycles to the KVM-side replica.
- Make long-running start, sync, and cutover operations asynchronous.
- Keep migration progress visible through list APIs and the UI migration table.
- Validate destination storage, host capabilities, and compute offering sizing
  before expensive work starts.
- Persist external vCenter credentials only through CloudStack encrypted DAO
  fields and never expose passwords in API responses.
- Reuse the existing CloudStack converted-VM import path after cutover
  finalization.

## Scope

Current implementation scope:

- Target hypervisor: KVM.
- Source hypervisor: VMware through VDDK and VMware CBT.
- Admin API surface for preflight, start, list, sync, cutover, cancel, and
  delete.
- Async API execution for long-running start, sync, and cutover operations.
- Management-server state model for migrations, disks, and delta cycles.
- External vCenter credential persistence for long-lived UI sessions, encrypted
  with CloudStack DAO encryption.
- KVM agent wrappers for initial VDDK full sync, delta block copy, cutover
  finalization, and cleanup.
- Filesystem-like primary storage target support using QCOW2 files.
- Ceph/RBD primary storage target support using raw RBD images.
- Non-in-place `virt-v2v` fallback for QCOW2 file targets, gated by global
  configuration.
- Early compute-offering validation against discovered VMware VM CPU and memory.
- Dynamic/custom service offering support through the same `details` keys used
  by the existing OVF/VDDK import flow.
- UI integration in the Import/Export Instances area, including job polling and
  table auto-refresh while CBT migrations are active.

## Non-Goals

Not implemented or intentionally blocked:

- Non-in-place `virt-v2v` fallback for Ceph/RBD targets. RBD targets require
  in-place finalization because fallback produces local QCOW2 files.
- Automatic source VM shutdown during cutover.
- Continuous byte-level progress from `qemu-img convert`. The UI currently
  receives state transitions, disk target paths, delta cycle byte counts, dirty
  rates, and cycle descriptions through the list API.
- Applying CBT deltas to a guest-converted disk.
- Replacing the existing OVF/VDDK one-time import flows.

## Key Principles

- CloudStack does not power off the VMware source VM automatically. The
  operator must shut it down before final cutover.
- CloudStack does not store vCenter passwords in clear text.
- Initial sync and delta sync keep the target disk source-equivalent.
- `virt-v2v` runs only at finalization, after the final CBT delta cycle.
- The final `virt-v2v` work is local to the KVM-side replica and does not
  perform a second full VMware read.
- The detailed status source is `listVmwareCbtMigrations`, not only the async
  job result.
- Failed final import after successful finalization is retryable through the
  `ReadyForImport` state.

## High-Level Architecture

The implementation has four main layers:

- CloudStack management-server APIs and orchestration.
- CloudStack database state for migration sessions, source disks, and delta
  cycles.
- KVM agent commands that perform VDDK reads, QCOW2 or raw RBD writes,
  `virt-v2v` finalization, and cleanup.
- UI workflow in the Import/Export Instances area.

Management-server API and orchestration:

- `StartVmwareCbtMigrationCmd`
- `CheckVmwareCbtMigrationPrerequisitesCmd`
- `ListVmwareCbtMigrationsCmd`
- `SyncVmwareCbtMigrationCmd`
- `CutoverVmwareCbtMigrationCmd`
- `CancelVmwareCbtMigrationCmd`
- `DeleteVmwareCbtMigrationCmd`
- `VmwareCbtMigrationManagerImpl`
- `VmwareCbtStorageTarget`
- `VmwareCbtMigrationCutoverPolicy`

Persistence:

- `cloud.vmware_cbt_migration`
- `cloud.vmware_cbt_migration_disk`
- `cloud.vmware_cbt_migration_cycle`

KVM agent wrappers:

- `LibvirtVmwareCbtPrepareCommandWrapper`
- `LibvirtVmwareCbtSyncCommandWrapper`
- `LibvirtVmwareCbtCutoverCommandWrapper`
- `LibvirtVmwareCbtCleanupCommandWrapper`

UI:

- `ui/src/views/tools/ImportUnmanagedInstance.vue`
- `ui/src/views/tools/ManageInstances.vue`
- `ui/src/views/tools/VmwareCbtMigrations.vue`

## Architecture Diagrams

The diagrams below are intentionally plain text so they can be pasted directly
into Apache CWIKI source and remain editable.

### End-To-End CBT Flow

```text
VMware source VM
     |
     | VMware snapshot + CBT metadata
     v
VDDK / nbdkit-vddk on KVM conversion host
     |
     | Initial baseline: full logical disk read
     v
CloudStack primary storage
     |
     | Source-equivalent replica
     |   - qcow2 file on filesystem-like storage
     |   - raw RBD image on Ceph/RBD storage
     v
Delta sync cycles
     |
     | CBT changed ranges only
     v
Operator powers off source VM
     |
     | Cutover: final delta + virt-v2v finalization
     v
CloudStack KVM VM
```

### Initial Sync Transfer Behavior

VMware CBT initial sync is not guest-filesystem-used-space-oriented. CloudStack
does not inspect the guest filesystem to decide which files are in use; it reads
the disk image exposed by VDDK/NBD and writes the target replica.

The amount of data transferred depends on what VDDK exposes for the source disk
and datastore. Some source paths expose the baseline as one allocated logical
range, while others expose sparse `hole,zero` extents. The destination can remain
thin or sparse when the source and copy path expose zero ranges correctly, but
operators must size the initial-copy window for the full logical disk capacity.

The KVM agent uses `qemu-img convert` for the general full-copy path. When
`nbdcopy` is installed, it can accelerate supported pre-created raw targets and
RBD targets served through a local `qemu-nbd` bridge. `nbdcopy` is optional;
capability and correctness do not depend on it.

```text
Example VMware disk capacity: 5 GiB
Guest-used data:              near 0 GiB
Datastore/path behavior:      datastore-dependent

During initial sync:
  VMware/VDDK exposes allocation/extents to NBD.
  CloudStack reads the ranges that qemu sees through that NBD export.
  qemu-img writes a qcow2 target and can preserve thin/sparse output.

Possible outcome:             full logical transfer or sparse-aware transfer
Target allocation:            backend- and extent-dependent
```

### Linked Clone And Full Clone Baselines

The initial baseline reads the logical disk presented by VDDK. Linked clone vs
full clone is not enough by itself to predict transfer volume. The decisive
question is whether VDDK/NBD exposes trustworthy allocation and zero extents for
the source datastore and disk chain.

```text
Linked clone source
  child delta VMDK
       |
       v
  parent/base VMDK chain
       |
       v
  VDDK presents one logical disk

Full clone source
  single/full VMDK
       |
       v
  VDDK presents one logical disk

CBT initial sync:
  both are read as the full logical disk capacity.
  The target qcow2 can still be sparse/thin after the write.
```

### Delta Sync And Cutover Eligibility

```text
Initial sync completed
        |
        v
Delta cycle 1 -> changed bytes / dirty rate measured
        |
        v
Delta cycle 2 -> quiet-cycle counters updated
        |
        v
Repeat until one boundary is met:

  min cycles satisfied
  AND quiet bytes threshold satisfied
  AND quiet dirty-rate threshold satisfied
  AND quiet cycle count satisfied

OR

  max cycle count reached

        |
        v
Migration becomes ReadyForCutover
```

### Cutover Finalization Paths

```text
Final CBT delta sync
      |
      v
Check selected conversion host capability
      |
      +--> virt-v2v-in-place available
      |        |
      |        v
      |   run virt-v2v-in-place finalization
      |
      +--> virt-v2v supports --in-place
      |        |
      |        v
      |   run virt-v2v --in-place finalization
      |
      +--> non-in-place fallback explicitly allowed
               |
               v
          run regular virt-v2v -o local fallback finalization

Result:
  finalized qcow2 moved to primary storage root
  VM imported with VirtIO root disk controller
```

### Final Disk Placement

```text
During sync:
  /mnt/<pool-uuid>/cloudstack-cbt/<migration-uuid>/...

After finalization/import:
  /mnt/<pool-uuid>/<generated-volume-uuid>.qcow2

CloudStack DB target path:
  updated to the final primary-storage-root location
```

## End-To-End Flow

### 1. Preflight

`checkVmwareCbtMigrationPrerequisites` validates the source and destination
before a long-running job is started. The command is synchronous and returns a
structured preflight response with pass, warn, and fail findings.

The preflight checks include:

- Destination zone and KVM cluster lookup.
- Destination primary storage lookup or implicit pool selection.
- Storage target classification.
- KVM conversion host selection.
- Host VMware CBT capability checks.
- Host in-place finalization capability checks.
- Host qemu RBD capability checks and an active temporary RBD create/write/read/delete
  probe when the selected primary storage is Ceph/RBD.
- vCenter source resolution, either by `existingvcenterid` or external
  vCenter fields.
- Source VM discovery.
- Source VM CBT support.
- Source VM CBT enabled state.
- Source VM consolidation-needed state.
- Existing VMware snapshot count.
- Source disk discovery.
- Windows guest conversion dependency check on the selected KVM conversion host
  when the source VM is a Windows guest.
- Optional compute offering sizing validation.

The preflight response includes, among other fields:

- `ready`
- `storagewritertype`
- `storagewritersupported`
- `storagerequiresinplacefinalization`
- `convertinstancehostinplacefinalizationsupported`
- `noninplacefinalizationfallbackallowed`
- `noninplacefinalizationfallbacksupported`
- `sourcecpunumber`
- `sourcecpuspeed`
- `sourcememory`
- `sourceguestosid`
- `sourceguestos`
- `disk`
- `finding`

If CBT is supported but not enabled, preflight reports a warning. The start
path can enable CBT before creating the baseline snapshot.

The source guest OS fields are populated from the VMware inventory and are used
to decide whether Windows-specific conversion prerequisites apply. They do not
change the migration mode or disk-copy behavior by themselves.

### Guest OS Mapping And Source OS Labels

VMware exposes more than one guest OS signal. The VM's configured guest OS
option can differ from the runtime guest OS label reported by VMware Tools. For
example, a powered-on VM can show `Ubuntu Linux (64-bit)` in the vSphere summary
while VM Options still show `Other (64-bit)`. CloudStack discovery and preflight
therefore treat source guest OS values as inventory facts rather than as a safe
automatic target OS selection.

The target CloudStack guest OS is the optional `osid` supplied to
`startVmwareCbtMigration` and used later by the shared KVM import path. Operators
should verify this value before starting migration, especially for generic VMware
guest IDs such as `ubuntu64Guest`, which can represent multiple Ubuntu releases.
If the desired target guest OS is not selectable, the corresponding
`guest_os_hypervisor` mapping must exist for the source hypervisor version.

### 2. Start And Initial Full Sync

`startVmwareCbtMigration` is async. It creates the migration record, persists
source disk rows, creates a baseline VMware snapshot, prepares target paths, and
dispatches the initial full sync to the selected KVM host.

For Windows source VMs, the start path repeats the same conversion dependency
check reported by preflight before the baseline snapshot and initial sync are
started. This prevents a long CBT copy from producing a disk that cannot be made
bootable because the selected KVM conversion host is missing `virtio-win`.

State transitions:

- `Created`
- `InitialSync`
- `Replicating` when the initial VDDK full sync completes successfully
- `Failed` if any initial copy or metadata step fails

Important implementation details:

- For filesystem-like storage, target paths are prepared before dispatch and
  are shown in API/UI while the sync is running.
- Default target path pattern with explicit primary storage:
  `/mnt/<storage-pool-uuid>/cloudstack-cbt/<migration-uuid>/<disk>.qcow2`
- Default target path pattern without explicit pool:
  `/var/lib/libvirt/images/cloudstack-cbt/<migration-uuid>/<disk>.qcow2`
- The agent writes the vCenter password to a temporary password file and passes
  it to `nbdkit` as `password=+<file>`.
- After successful copy, management refreshes and records the baseline disk
  `changeId` values from the VMware snapshot.
- The baseline snapshot is removed after the initial sync attempt when possible.

### 3. Delta Synchronization

`syncVmwareCbtMigration` is async and serialized by migration ID through
`BaseAsyncCmd.migrationSyncObject`.

Allowed migration states:

- `Replicating`
- `ReadyForCutover`

Delta cycle sequence:

1. Create a new `vmware_cbt_migration_cycle` row.
2. Create a VMware snapshot for the cycle.
3. Query VMware CBT changed disk areas from the previous per-disk `changeId`.
4. Persist cycle state as `QueryingChangedAreas`, then `CopyingChangedBlocks`.
5. Send changed block ranges to the KVM agent.
6. Agent copies only those ranges from the VDDK NBD source into the target
   replica.
7. Management records changed bytes, duration, dirty rate, and the next
   per-disk `changeId`.
8. The cycle snapshot is removed when possible.
9. The cutover policy decides whether to remain in `Replicating` or move to
   `ReadyForCutover`.

The cycle response exposes:

- `cyclenumber`
- `snapshotmor`
- `changedbytes`
- `dirtyrate`
- `duration`
- `state`
- `description`
- `created`
- `lastupdated`

Delta sync progress is represented as state and cycle metadata rather than a
streamed progress bar. While a cycle is running, the row description can show
what the manager knows, for example that changed areas are being queried or
that a number of ranges has been dispatched. After the agent returns, the cycle
shows copied bytes, dirty rate, duration, and a textual summary.

### 4. Cutover

`cutoverVmwareCbtMigration` is async and serialized by migration ID.

Allowed migration states:

- `ReadyForCutover`
- `ReadyForImport`

Cutover requires the source VM to be powered off. CloudStack checks the VMware
power state and rejects cutover if the source VM is still powered on:

```text
Cannot cut over VMware CBT migration <uuid> while source VM <name> is in power state PowerOn.
Gracefully shut down the source VM, then retry cutover.
```

CloudStack does not attempt a graceful shutdown or power-off. The operator must
shut down the VMware source VM before final cutover. This avoids surprising the
operator and avoids needing to encode site-specific guest shutdown policy in the
migration path.

Cutover sequence:

1. Verify source VM is powered off.
2. Set migration state to `CuttingOver`.
3. Run a final CBT delta sync while the VM is off.
4. Run `virt-v2v` finalization on the KVM host.
5. Move finalized disk files to the primary storage root with generated UUID
   names, then update disk target paths from the agent result.
6. Set state to `ReadyForImport`.
7. Call the shared external KVM VM import path.
8. On success, set state to `Completed`, store the new CloudStack VM ID, and
   clear stored external vCenter credentials.
9. On import failure, keep state as `ReadyForImport` so the operator can retry
   cutover to repeat only the import step against already finalized disks.

### 5. Cancel And Delete

`cancelVmwareCbtMigration` is synchronous. If the migration is not terminal, it
sends a best-effort cleanup command, marks the migration `Cancelled`, and clears
stored external vCenter credentials.

`deleteVmwareCbtMigration` is synchronous. It is allowed for `Failed`,
`Cancelled`, and `Completed` migrations. With `cleanup=true` (the default), it
sends a cleanup command before deleting failed or cancelled child rows and the
parent row. `Completed` migration deletion is always record-only: CloudStack
removes the CBT bookkeeping rows but never deletes the imported VM, CloudStack
volumes, finalized target disks, or primary-storage files.

Deletion behavior:

- Deletes `vmware_cbt_migration_cycle` rows.
- Deletes `vmware_cbt_migration_disk` rows.
- Deletes the `vmware_cbt_migration` row.
- Clears stored source credentials before row removal.
- For `Failed` or `Cancelled` rows, cleans target files only if the paths are
  under the migration-specific `/cloudstack-cbt/<migration-uuid>/` directory
  and cleanup is enabled. For RBD targets, cleanup removes only temporary RBD
  image names containing the CloudStack-owned `cloudstack-cbt-<migration-uuid>-`
  marker.
- For `Completed` rows, ignores cleanup and removes only CBT migration records.
- Skips immediate cleanup if another active migration is running on the same
  conversion host, to avoid disturbing active agent-side work.

The KVM cleanup wrapper resolves migration directories from disk target paths
and removes only the migration-specific directory tree for file targets. For RBD
targets it uses the destination storage pool and deletes only marked temporary
images, never arbitrary RBD images.

## Conversion Model: One VMware Baseline Read

The feature deliberately separates replication from final conversion.

Initial full sync does not run `virt-v2v`. The KVM agent opens the VMware source
snapshot through `nbdkit` with the VDDK plugin. Filesystem-like primary storage
writes QCOW2 with `qemu-img convert`:

```text
nbdkit -r -U - vddk ... --run \
  'qemu-img convert -f raw -O qcow2 "$uri" "<targetPath>"'
```

Ceph/RBD primary storage is a raw RBD image. When `nbdcopy` is available, the
agent pre-creates the image, exposes it through a temporary local `qemu-nbd`
bridge, and copies NBD to NBD. Otherwise it uses `qemu-img convert` directly
against the qemu RBD URL. LINSTOR similarly uses `nbdcopy` when available for
its pre-created raw DRBD device and falls back to `qemu-img convert`.

Example RBD fallback path:

```text
nbdkit -r -U - vddk ... --run \
  'qemu-img convert -f raw -O raw "$uri" "rbd:<pool>/<image>:..."'
```

This produces a source-equivalent replica on the selected primary storage. It
is not yet a CloudStack-imported VM disk. It is a replication target that can
safely receive raw changed blocks from later CBT cycles.

Delta sync also does not run `virt-v2v`. Management creates a VMware snapshot,
queries changed disk areas using VMware CBT metadata, and dispatches changed
byte ranges to the KVM agent. The agent opens the snapshot through `nbdkit`,
parses the temporary nbdkit Unix socket from the NBD URI, uses `qemu-img
convert --image-opts` to materialize bounded raw chunks for only the changed
ranges, and patches the replica with `qemu-io`. File targets use `-f qcow2`;
RBD targets use `-f raw` against the qemu RBD URL for the target image.

Final cutover is where `virt-v2v` runs. After the source VM is powered off, the
manager runs one final CBT delta cycle so the replica is current. Then the agent
runs one of the finalization paths:

- `virt-v2v-in-place`, when available.
- `virt-v2v --in-place`, when the installed `virt-v2v` supports it.
- Regular `virt-v2v -o local`, only when the non-in-place fallback is explicitly
  enabled and the target is a QCOW2 file target.

Ceph/RBD targets require one of the in-place finalization paths. They do not
allow non-in-place fallback because fallback creates local QCOW2 output rather
than modifying the raw RBD image that CloudStack will import.

So there is not a second full VMware read during finalization. The final
`virt-v2v` step is local work against the up-to-date KVM-side replica. It may
still require local storage reads and writes, especially in fallback mode, but
it is not another VDDK full copy from vCenter.

The initial baseline may read up to the full logical VMware disk range,
regardless of guest filesystem usage. This does not imply thick allocation on
the destination: QCOW2, RBD, and LINSTOR can remain sparse or thin when the
source and copy path expose zero ranges correctly. Delta cycles are CBT-range
based and copy only changed ranges.

This design avoids applying future CBT raw block deltas to a disk that has
already been modified by `virt-v2v`. Applying VMware CBT ranges to a converted
guest disk would be unsafe because the layout no longer represents the source
VMware disk byte-for-byte.

## API Changes

All commands are admin-only.

### `checkVmwareCbtMigrationPrerequisites`

Synchronous preflight command.

Required:

- `zoneid`
- `clusterid`
- `sourcevmname`

Source vCenter selection:

- Existing registered vCenter: `existingvcenterid`
- External vCenter: `vcenter`, `datacentername`, `username`, `password`

Optional:

- `hostip`
- `clustername`
- `convertinstancehostid`
- `convertinstancestoragepoolid`
- `serviceofferingid`
- `details`

The command has `requestHasSensitiveInfo=true` because external vCenter
credentials can be provided.

### `startVmwareCbtMigration`

Async command. Starts the migration and initial full sync.

Required:

- `zoneid`
- `clusterid`
- `serviceofferingid`
- `sourcevmname`

Source vCenter selection:

- Existing registered vCenter: `existingvcenterid`
- External vCenter: `vcenter`, `datacentername`, `username`, `password`

Optional import inputs:

- `displayname`
- `hostname`
- `account`
- `domainid`
- `projectid`
- `templateid`
- `osid`
- `datadiskofferinglist`
- `nicnetworklist`
- `nicipaddresslist`
- `forced`

Optional conversion inputs:

- `convertinstancehostid`
- `convertinstancestoragepoolid`
- `details`

If `convertinstancestoragepoolid` is omitted, CloudStack auto-selects the only
CBT-compatible primary storage pool in the destination cluster/zone. Supported
target pool types are `NetworkFilesystem`, `Filesystem`, `SharedMountPoint`,
and `RBD`. If more than one compatible pool is available, the operator must
provide the pool explicitly.

Useful `details` keys:

- `vddk.lib.dir`
- `vddk.transports`
- `vddk.thumbprint`
- `cpuNumber`
- `cpuSpeed`
- `memory`

The command has `requestHasSensitiveInfo=true` and
`responseHasSensitiveInfo=false`.

### `listVmwareCbtMigrations`

Synchronous status command and the main progress source for UI and automation.

Filters:

- `id`
- `zoneid`
- `accountid`
- `vcenter`
- `sourcevmname`
- `state`

The response includes migration-level state, current step, current step
duration, last error, cycle counters, changed byte totals, disks, and cycles.
`currentstepduration` is populated for non-terminal migrations and is intended
for the UI table marker similar to Import VM Tasks.

### `syncVmwareCbtMigration`

Async command. Runs one delta sync cycle.

Required:

- `id`

Optional:

- `username`
- `password`

For an external vCenter migration, stored credentials are used when available.
If override credentials are supplied, both username and password must be
provided. Overrides are stored back to the migration so a UI operator can return
later and continue syncing.

### `cutoverVmwareCbtMigration`

Async command. Runs final powered-off sync, finalizes disks, and imports the VM.

Required:

- `id`

Optional:

- `username`
- `password`

The command fails early if the source VMware VM is not powered off.

### `cancelVmwareCbtMigration`

Synchronous command.

Required:

- `id`

Cancels a non-terminal migration, attempts cleanup, and clears stored external
source credentials.

### `deleteVmwareCbtMigration`

Synchronous command.

Required:

- `id`

Optional:

- `cleanup` (default `true`)

Allowed for `Failed`, `Cancelled`, and `Completed` migrations. Completed
migration deletes are record-only and never remove the imported CloudStack VM,
volumes, finalized target disks, or primary-storage files.

## Configuration And Cutover Policy

The cutover recommendation is based on global settings:

| Global setting | Default | Meaning |
| --- | ---: | --- |
| `vmware.cbt.migration.min.cycles` | `1` | Minimum number of manual/API delta cycles that must complete before quiet-cycle readiness can make the migration eligible for cutover. |
| `vmware.cbt.migration.max.cycles` | `5` | Maximum number of manual/API delta cycles before CloudStack marks the migration ready for cutover even if the quiet-cycle thresholds were not met. |
| `vmware.cbt.migration.quiet.cycles` | `2` | Number of consecutive quiet cycles required before CloudStack marks the migration ready for cutover. |
| `vmware.cbt.migration.quiet.bytes` | `1073741824` | Maximum changed bytes in one cycle for that cycle to be considered quiet. The default is 1 GiB. |
| `vmware.cbt.migration.quiet.dirty.rate` | `16777216` | Maximum changed bytes per second for that cycle to be considered quiet. The default is 16 MiB/s. |
| `vmware.cbt.migration.agent.command.timeout` | `86400` | Timeout in seconds for long-running VMware CBT data-plane commands on the KVM agent. This covers initial full sync, regular delta sync, final delta sync, and cutover finalization. The default is 24 hours. |

The manager tracks:

- Number of completed delta cycles.
- Number of consecutive quiet cycles.
- Last changed bytes.
- Last dirty rate.
- Total changed bytes.

A cycle is quiet when its changed bytes and dirty rate are below the configured
thresholds. Both thresholds must pass. For example, with defaults, a cycle that
copies 1.2 GiB at 10 MiB/s is not quiet because the changed-byte threshold is
exceeded even though the dirty-rate threshold passed.

Once enough quiet cycles have accumulated, or once the maximum manual/API
delta-cycle count is reached, the migration moves to `ReadyForCutover`.

The operator can still manually run sync cycles while the migration is in
`Replicating` or `ReadyForCutover`.

The UI intentionally exposes the Cutover button only when the server-side
migration state is exactly `ReadyForCutover`. Before that point the UI exposes
Sync delta, because the server has not yet recommended final cutover. If a
cutover is started, CloudStack creates one additional final CBT delta cycle
inside the cutover operation. Therefore the final displayed cycle count can be
one higher than `vmware.cbt.migration.max.cycles`: for example, five ordinary
delta cycles can make the migration ready, and cutover can then record cycle 6
as the final synchronization cycle.

### Timeouts And Large VMs

The long-running CBT work is done by KVM agent commands. CloudStack sets the
agent command `wait` value from
`vmware.cbt.migration.agent.command.timeout`, and the KVM wrappers use the same
value as the timeout for child processes such as `nbdkit`, `qemu-img`, and
`virt-v2v`.

The timeout is finite by design. The shared CloudStack `Script` helper treats a
zero timeout as its default one-hour timeout, so `0` is not a safe way to mean
"run forever". For large source disks, tune the timeout above the expected
worst-case duration. A rough estimate is:

```text
timeout_seconds >= source_bytes / expected_bytes_per_second * safety_factor
```

For example, a 1 TiB initial sync at 100 MiB/s is roughly 3 hours before
overhead; at 25 MiB/s it is roughly 12 hours. The default 24-hour timeout is
intended to be safe for large migrations while still
protecting the agent from permanently stuck child processes.

Related import settings that operators often notice are:

| Global setting | Applies to CBT? | Default | Meaning |
| --- | --- | ---: | --- |
| `convert.vmware.instance.to.kvm.timeout` | No | `3` | OVF/VDDK import timeout, in hours, for the `ConvertInstanceCommand` virt-v2v path. CBT cutover has its own timeout because it uses CBT-specific agent commands. |
| `remote.kvm.instance.disks.copy.timeout` | No | `30` | Remote KVM unmanaged import disk-copy timeout, in minutes. It is unrelated to VMware CBT. |

If a CBT command exceeds `vmware.cbt.migration.agent.command.timeout`, the
migration is marked failed with the agent error details in `last_error`.
Failed or cancelled migration records can be deleted with cleanup enabled to
remove CloudStack-owned `cloudstack-cbt/<migration-uuid>` working files.

## Database Changes

The feature targets the Apache CloudStack 24.0.0 release, so the schema changes
are delivered through the canonical 4.23.0.0 to 24.0.0 upgrade path:

- `engine/schema/src/main/resources/META-INF/db/schema-42300to2400.sql`
- `engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade42300to2400.java`

### `cloud.vmware_cbt_migration`

One row per migration session.

Important columns:

- `uuid`
- `zone_id`
- `account_id`
- `user_id`
- `vm_id`
- `existing_vcenter_id`
- `source_username`
- `source_password`
- `destination_cluster_id`
- `convert_host_id`
- `storage_pool_id`
- `display_name`
- `host_name`
- `template_id`
- `service_offering_id`
- `guest_os_id`
- `data_disk_offering_map`
- `nic_network_map`
- `nic_ip_address_map`
- `import_details`
- `forced`
- `vcenter`
- `datacenter`
- `source_host`
- `source_cluster`
- `source_vm_name`
- `vddk_lib_dir`
- `vddk_transports`
- `vddk_thumbprint`
- `state`
- `current_step`
- `last_error`
- `completed_cycles`
- `quiet_cycles`
- `total_changed_bytes`
- `last_changed_bytes`
- `last_dirty_rate`
- `created`
- `updated`
- `removed`

`source_password` is annotated with `@Encrypt` in `VmwareCbtMigrationVO`, using
the same CloudStack encrypted DAO field mechanism used by VMware datacenter
credentials and VM VNC passwords.

### `cloud.vmware_cbt_migration_disk`

One row per source VMware disk.

Important columns:

- `migration_id`
- `source_disk_id`
- `source_disk_device_key`
- `source_disk_path`
- `datastore_name`
- `capacity_bytes`
- `target_path`
- `target_format`
- `change_id`
- `snapshot_moref`
- `state`

Disk states:

- `Created`
- `Prepared`
- `Syncing`
- `Ready`
- `Failed`

### `cloud.vmware_cbt_migration_cycle`

One row per delta cycle.

Important columns:

- `migration_id`
- `cycle_number`
- `snapshot_moref`
- `changed_bytes`
- `dirty_rate`
- `duration`
- `state`
- `description`

Cycle states:

- `Created`
- `QueryingChangedAreas`
- `CopyingChangedBlocks`
- `Completed`
- `Failed`

## Migration State Machine

Migration states:

- `Created`
- `InitialSync`
- `Replicating`
- `ReadyForCutover`
- `CuttingOver`
- `ReadyForImport`
- `Completed`
- `Failed`
- `Cancelled`

Terminal states:

- `Completed`
- `Failed`
- `Cancelled`

Normal successful path:

```text
Created
  -> InitialSync
  -> Replicating
  -> ReadyForCutover
  -> CuttingOver
  -> ReadyForImport
  -> Completed
```

`ReadyForImport` is intentionally retryable. If the final `virt-v2v`
finalization succeeded but CloudStack VM import failed, the next cutover command
does not rerun the final CBT copy and finalization. It retries the VM import
against the already finalized disk paths.

## Storage Target And Finalization

`VmwareCbtStorageTarget` maps primary storage to a target writer:

- `NetworkFilesystem`, `Filesystem`, and `SharedMountPoint` use `QCOW2_FILE`.
- No explicit pool also uses a filesystem QCOW2 target under the default local
  CBT path.
- `RBD` maps to `RBD_RAW`. Initial sync writes a raw RBD image with `nbdcopy`
  through a local NBD bridge when available, otherwise with
  `qemu-img convert -O raw`. Delta sync writes changed ranges with `qemu-io -f
  raw`, and cutover finalization is in-place only.
- `Linstor` maps to `RAW_BLOCK_DEVICE`. The agent pre-creates each target
  volume at source capacity through the Linstor storage adaptor (qemu cannot
  create DRBD devices the way it creates RBD images), then the initial sync
  writes the local DRBD device with `nbdcopy` when available or
  `qemu-img convert -n -O raw` otherwise. Delta sync patches changed ranges
  with `qemu-io -f raw` against the device path, and cutover finalization is
  in-place only, using a plain `<disk type='block'>` libvirt XML (no qemu-nbd
  bridges are needed for local block devices).
  Target names are `cbt-<mig8>-<disk-id>` where `mig8` is the first eight
  characters of the migration UUID: LINSTOR resource names ("cs-" + name) are
  capped at 48 characters, so the RBD-style naming with the full migration
  UUID does not fit; the short marker still guards cleanup on both the
  management server and the agent. The conversion host must be a LINSTOR
  satellite connected to the pool; host selection and the preflight storage
  probe (create a 4 MiB volume, qemu-io write/read on the device, delete)
  enforce this.
- Other primary storage types are unsupported.

For QCOW2 file targets, the initial replica is stored under the selected
primary storage:

```text
/mnt/<pool-uuid>/cloudstack-cbt/<migration-uuid>/<disk>.qcow2
```

If no storage pool is selected, the fallback path is:

```text
/var/lib/libvirt/images/cloudstack-cbt/<migration-uuid>/<disk>.qcow2
```

For Ceph/RBD targets, the initial replica is a raw RBD image in the selected
pool:

```text
<ceph-pool>/cloudstack-cbt-<migration-uuid>-<source-disk-id>-<source-name>
```

Only the image name is persisted as the migration disk target path. The agent
reconstructs the qemu RBD URL from the selected storage pool when writing
initial and delta data. During in-place `virt-v2v` finalization, the agent
starts a temporary localhost `qemu-nbd` bridge for each RBD image and gives
`virt-v2v` a libvirt XML disk that points to that local NBD endpoint. This keeps
RBD credentials and monitor options in the qemu layer and presents a local NBD
endpoint to the conversion appliance.

The preferred cutover finalization is in-place:

- `virt-v2v-in-place`, if present.
- `virt-v2v --in-place`, if the installed `virt-v2v` supports it.

If neither in-place method is available, QCOW2 file targets can use regular
`virt-v2v -o local` only when this global configuration is enabled:

```text
vmware.cbt.allow.non.inplace.finalization=true
```

Default:

```text
false
```

When fallback is enabled:

- The fallback is only allowed for supported QCOW2 file targets.
- The agent stages both `TMPDIR` and `virt-v2v` output under the current
  migration directory on the selected primary storage.
- The agent validates free space before running fallback finalization.
- Required free space is conservatively estimated as two times the summed disk
  capacity.
- After successful fallback finalization, source replica disks are deleted.

Fallback is never allowed for RBD targets. RBD migrations require
`virt-v2v-in-place` or `virt-v2v --in-place` support on the selected conversion
host.

Example fallback output path:

```text
/mnt/<pool-uuid>/cloudstack-cbt/<migration-uuid>/virt-v2v-output-<random>/<migration-uuid>-sda
```

After successful finalization, both in-place and fallback paths are normalized
to the same final layout used by the OVF/VDDK import flow: the KVM agent moves
each finalized QCOW2 file to the selected primary storage root and gives it a
generated UUID filename:

```text
/mnt/<pool-uuid>/<generated-volume-uuid>
```

The agent returns the relocated target paths in the cutover answer. The
management server persists them in `vmware_cbt_migration_disk.target_path`
through the normal disk-result update path, so no manual database update or
schema change is required for this relocation. CloudStack then imports the VM
using the flat root-level file name, matching the final placement used by
regular OVF/VDDK migrations.

For RBD targets, successful in-place finalization keeps the raw RBD image in
place and CloudStack imports it by image name from the selected RBD storage
pool. No post-finalization file move is performed, and completed migration
deletion remains record-only.

RBD finalization creates only temporary localhost `qemu-nbd` processes and
temporary XML/script files on the conversion host. The wrapper installs a shell
trap to stop those bridge processes and remove the PID files after `virt-v2v`
finishes or fails. The persistent storage artifact remains the raw RBD image
whose name contains the `cloudstack-cbt-<migration-uuid>-` marker until
CloudStack imports it or cleanup removes it for a failed/cancelled migration.

CloudStack imports finalized CBT disks with the KVM `virtio` root disk
controller. The conversion host must have `virtio-win` available for Windows
guests so that `virt-v2v` can enable the matching boot driver before the VM is
started under KVM. This keeps the persisted `rootDiskController` detail aligned
with the controller model used by the converted guest.

## Ceph/RBD Conversion Host Requirements

Ceph/RBD support relies on the same KVM host plumbing that normal CloudStack RBD
primary storage uses, plus the CBT-specific VMware conversion tools. The
selected conversion host must be able to access the destination RBD pool from
both Java/libvirt storage code and the qemu command-line tools used by the CBT
wrappers.

Required conversion-host capabilities:

- CloudStack KVM agent with the normal RBD primary-storage dependencies.
- The selected KVM/conversion host must already be able to use the destination
  CloudStack RBD primary storage pool.
- `librados` and `librbd` client libraries compatible with the Ceph cluster.
- Java RADOS/RBD bindings used by CloudStack's RBD storage adaptor.
- qemu RBD block driver support so `qemu-img` and `qemu-io` can open
  `rbd:<pool>/<image>:...` URLs.
- VDDK and `nbdkit-vddk` for VMware source reads.
- `virt-v2v-in-place` or `virt-v2v --in-place` for RBD cutover finalization.
- `virtio-win` or equivalent VirtIO driver media/packages for Windows guests,
  same as the existing VMware import requirements.

CloudStack RBD primary-storage configuration must be valid on the selected
KVM/conversion host. For CBT-to-RBD, CloudStack uses the existing KVM
storage-pool metadata to build qemu RBD URLs, including monitor addresses,
authentication mode, Ceph user, and secret. The CBT qemu path therefore does not
rely on `/etc/ceph/ceph.conf` or a local keyring as its primary credential source
when CloudStack supplies explicit RBD URL options.

A local Ceph configuration and keyring can still be useful for operator
diagnostics and site-local tooling, for example running `ceph -s`,
`rbd ls <pool>`, or similar commands without manually passing monitor and
authentication options each time. Sites may choose to install those files for
operational convenience, but they should not be documented as the required
credential source for CBT RBD writes.

The Ceph client packages on every KVM/conversion host should match the Ceph
cluster major release, or at least be explicitly supported by that cluster
release. Avoid relying on the distribution's stock Ceph packages if they are
older than the deployed Ceph cluster. Ancient `librbd`, `librados`, qemu RBD
block-driver, or Java binding packages can fail in ways that look like CBT
errors even though the actual problem is a client/cluster compatibility mismatch.

Practical guidance:

- If the Ceph cluster is deployed from upstream Ceph packages, install the
  corresponding upstream Ceph client repository on all KVM/conversion hosts.
- If the Ceph cluster is deployed from a vendor repository, use that vendor's
  matching client packages on the KVM/conversion hosts.
- Keep `ceph-common`, `librados`, `librbd`, qemu RBD block-driver packages, and
  Java RADOS/RBD bindings aligned as a set. Do not mix a modern Ceph cluster
  with old OS-default client libraries unless the Ceph vendor explicitly
  supports that combination.
- Verify the KVM host can already use the selected RBD primary storage for
  ordinary CloudStack VM volumes before enabling CBT migrations to the same
  pool.
- Verify that qemu tools can see RBD support. Package names vary by OS, but on
  many EL-like distributions this is provided by a qemu RBD block-driver package
  such as `qemu-kvm-block-rbd`; on Debian/Ubuntu-like systems it is commonly
  part of qemu block extras.

CBT preflight and `startVmwareCbtMigration` actively probe RBD access when the
selected destination pool is RBD. The management server sends the selected
conversion host a small probe command using a temporary image named
`cloudstack-cbt-probe-<uuid>`. The KVM agent builds the qemu RBD URL from the
existing CloudStack storage-pool metadata, runs `qemu-img create`, writes and
reads a 4 KiB pattern with `qemu-io`, and removes the temporary image through
CloudStack's existing RBD storage adaptor. This catches qemu RBD driver issues,
Ceph monitor/authentication problems, and Java RADOS/RBD cleanup failures before
the long-running VMware copy starts.

Useful host checks:

```bash
ceph -s
ceph osd lspools
rbd ls <pool>
qemu-img --help | grep -i rbd
qemu-io --help | grep -i rbd
```

Useful package/version checks:

```bash
ceph --version
rbd --version
ldconfig -p | egrep 'librbd|librados'
java -version
```

Useful CloudStack database check after the host reconnects:

```sql
SELECT h.id, h.name, hd.name, hd.value
FROM cloud.host h
JOIN cloud.host_details hd ON hd.host_id = h.id
WHERE h.name = '<kvm-host-name>'
  AND hd.name IN (
    'host.vddk.blockcopy.support',
    'host.vddk.blockcopy.inplace.finalization.support',
    'host.vddk.blockcopy.rbd.support',
    'host.vddk.support',
    'host.vddk.version',
    'host.virtv2v.inplace.version',
    'vddk.lib.dir'
  )
ORDER BY hd.name;
```

An RBD-capable conversion host should already report VMware CBT capability as
`true` before a migration starts, and the selected destination storage pool must
be the RBD pool that the host can access. If these host details are stale after
installing packages, restart `cloudstack-agent`, ensure the management server is
running, and wait for host details to be refreshed.

## Host Capability Reporting

The agent checks these on startup/ready and sends them as host details. Management then persists them into `cloud.host_details`.

| Host detail | How agent checks it |
| --- | --- |
| `host.virtv2v.version` | Runs `virt-v2v --version`. On Ubuntu/Debian it also checks `dpkg -l nbdkit`. Version is parsed from `virt-v2v --version`. |
| `vddk.lib.dir` | Reads `vddk.lib.dir` from `agent.properties`; if blank/invalid, auto-detects with `find / -type d -name 'vmware-vix-disklib-distrib' 2>/dev/null \| head -n 1`. Valid means `<dir>/lib64/libvixDiskLib.so*` exists. |
| `host.vddk.version` | Runs `nbdkit vddk --dump-plugin libdir=<vddk-lib-dir>` and parses `vddk_library_version=<n>`. |
| `host.vddk.support` | True only if `virt-v2v`/`nbdkit` support is present, VDDK dir is valid, and `nbdkit-vddk-plugin` can load VDDK via `nbdkit vddk --dump-plugin libdir=<dir>`. |
| `host.qemu.img.version` | Runs `qemu-img --version`, parses first non-empty line. |
| `host.qemu.nbd.version` | Runs `qemu-nbd --version`, parses first non-empty line. |
| `host.qemu.io.version` | Runs `qemu-io --version`, parses first non-empty line. |
| `host.vddk.blockcopy.support` | True only if `host.vddk.support` is true and `qemu-img --version`, `qemu-nbd --version`, and `qemu-io --version` all exit `0`. |
| `host.vddk.blockcopy.inplace.finalization.support` | True only if CBT support is true and either `virt-v2v-in-place --version` works, or `virt-v2v --help 2>&1 \| grep -q -- '--in-place'` succeeds. |
| `host.virtv2v.inplace.version` | If `virt-v2v-in-place` exists, parses `virt-v2v-in-place --version`. If only `virt-v2v --in-place` exists, reports the normal `virt-v2v` version. |
| `host.vddk.blockcopy.rbd.support` | True only if in-place finalization support is true and `qemu-img --help` advertises the `rbd` block driver. |

Important nuance: `host.vddk.blockcopy.rbd.support` is only the coarse host capability. For an actual selected RBD pool, preflight/start also runs an active probe: create a temporary `cloudstack-cbt-probe-<uuid>` RBD image, write/read 4 KiB using `qemu-io`, then delete it through CloudStack's RBD storage adapter. That catches Ceph auth, monitor, qemu RBD, librados/librbd, and Java binding problems.

Windows source VMs also require `virtio-win` on the selected KVM conversion
host. CBT preflight and `startVmwareCbtMigration` use the same
`CheckConvertInstanceCommand` path as OVF/VDDK import with Windows guest
conversion enabled, so a missing `virtio-win` package is reported before the
initial sync starts.

VDDK detection uses either agent configuration or auto-detection. The agent
validates the VDDK library by checking that `lib64/libvixDiskLib.so*` exists
and that `nbdkit vddk --dump-plugin libdir=<path>` reports a library version.

## VDDK Details

VDDK options can be provided as API `details` at migration start:

- `vddk.lib.dir`
- `vddk.transports`
- `vddk.thumbprint`

If a value is not provided by the API, the agent uses its configured or detected
value. If `vddk.thumbprint` is not provided, the agent attempts to fetch the
vCenter SHA1 certificate thumbprint with `openssl`.

The implementation does not hard-code one VDDK version. The installed VDDK
build must be compatible with the host libraries and pass the agent capability
check.

## Credential Handling

There are two source credential modes:

- Existing registered vCenter: credentials are read from the registered
  `vmware_data_center` record.
- External vCenter: credentials are supplied to start/preflight and stored on
  the migration record so later UI sync/cutover actions can run without asking
  again.

External credential behavior:

- `source_password` is stored through the CloudStack encrypted DAO field
  mechanism via `@Encrypt`.
- Start, preflight, sync, and cutover commands that can carry credentials are
  marked `requestHasSensitiveInfo=true`.
- API responses are marked `responseHasSensitiveInfo=false`.
- Agent-side VDDK commands pass the password through a temporary password file
  and `nbdkit` `password=+<file>`, not by embedding the clear-text password in
  the command line.
- Management sanitizes returned error messages by replacing the resolved source
  password with `******` before storing `last_error` or cycle descriptions.
- Stored external credentials are cleared when the migration completes,
  when it is cancelled, and before deletion.

The implementation avoids logging clear-text passwords. As with other
CloudStack secret handling, this also depends on callers not placing secrets in
unrelated free-form log messages.

## Compute Offering And Preflight Validation

CBT migration uses the same eventual import path as the existing OVF/VDDK VM
import code. The final CloudStack VM sizing still comes from the selected
service offering and any dynamic offering details.

The CBT-specific difference is timing: CBT validates obvious sizing problems
early, before the long initial full sync begins. This is meant to avoid spending
hours copying disks only to fail final VM import because the selected offering
is too small.

The source VM sizing captured during preflight/start includes:

- `sourcecpunumber`
- `sourcecpuspeed`
- `sourcememory`

Validation rule:

- If the source value and requested value are both known and greater than zero,
  the requested value must be greater than or equal to the source value.
- Unknown or zero source values are not used to reject the offering.

Validated fields:

- CPU number
- CPU speed
- Memory

Fixed offerings:

- CPU number comes from `service_offering.cpu`.
- CPU speed comes from `service_offering.speed`.
- Memory comes from `service_offering.ram_size`.

Dynamic/custom offerings:

- Caller details can provide `cpuNumber`, `cpuSpeed`, and `memory`.
- If caller details are missing, offering minimums such as `mincpunumber` and
  `minmemory` can be used where applicable.
- This mirrors the shared import behavior in
  `UnmanagedVMsManagerImpl.addServiceOfferingDetailsToParams`.

Example custom offering details:

```text
details[0].key=cpuNumber
details[0].value=4
details[1].key=cpuSpeed
details[1].value=2100
details[2].key=memory
details[2].value=4096
```

This early validation is not intended to be stricter than OVF/VDDK import. It
is intended to fail earlier for the same class of sizing mismatch.

## UI Changes

The CBT UI is integrated into the Import/Export Instances area.

Implemented UI behavior:

- Start migration uses the async `startVmwareCbtMigration` job.
- Sync delta uses the async `syncVmwareCbtMigration` job.
- Cutover uses the async `cutoverVmwareCbtMigration` job.
- Each async action uses CloudStack job polling.
- The CBT migration table also calls `listVmwareCbtMigrations` as the detailed
  status source.
- The table auto-refreshes while the CBT tab is active and any migration is in
  an active state or a CBT action job is still running.

Active states for UI refresh:

- `Created`
- `InitialSync`
- `Replicating`
- `CuttingOver`

The table exposes:

- Migration state.
- Current step.
- Completed cycles.
- Quiet cycles.
- Last changed bytes.
- Last dirty rate.
- Total changed bytes.
- Last error.
- Per-disk target path, target format, change ID, snapshot MOR, and state.
- Per-cycle state, changed bytes, dirty rate, duration, description, created,
  and updated timestamps.

Initial sync progress is currently coarse. While `qemu-img convert` runs on the
agent, management can show the migration state, current step, target path, and
disk state, but not a continuously updated percentage. Delta syncs have better
post-cycle metrics because changed bytes and dirty rate are returned by the
cycle result.

## Failure And Retry Semantics

Initial full sync failure:

- Migration moves to `Failed`.
- Created or syncing disks are marked `Failed`.
- Last error is stored on the migration.
- Delete with cleanup can remove the migration directory.

Delta sync failure:

- The cycle is marked `Failed`.
- The migration moves to `Failed`.
- Last error and cycle description are stored.
- VMware cycle snapshot removal is still attempted.

Cutover finalization failure:

- Migration moves to `Failed`.
- The source VM remains under operator control in vCenter.
- Delete with cleanup can remove replicated/finalized target directories if the
  paths are under the migration directory.

Import failure after successful finalization:

- Migration remains `ReadyForImport`.
- Target disk paths point at the finalized disk files.
- Retrying cutover retries the import step.

Cancel:

- Best-effort cleanup.
- State becomes `Cancelled`.
- Stored external credentials are cleared.

Delete:

- `Failed`, `Cancelled`, or `Completed`.
- Optional cleanup, default `true`, applies only to `Failed` and `Cancelled`.
- `Completed` deletion is record-only and intentionally preserves the imported
  VM and all destination storage artifacts.
- Child rows and parent row are removed.

## Security And Logging Considerations

Important security choices:

- Credential-bearing commands are marked sensitive.
- External vCenter password storage uses encrypted DAO fields.
- Passwords are passed to VDDK through temp files instead of clear-text command
  arguments.
- Password temp files are owner-readable and owner-writable where POSIX
  permissions are supported.
- Returned failure messages are sanitized against the active source password
  before persistence.
- CBT delta and cutover agent wrappers include the last useful command output
  line in returned failure details when available. This keeps management server
  and UI errors actionable for `qemu-nbd`, `qemu-io`, `nbdkit`, and `virt-v2v`
  failures without logging or returning clear-text passwords.

Operational caution:

- `source_username` is not encrypted.
- vCenter host, datacenter, VM name, datastore names, disk paths, change IDs,
  and snapshot MORs are not treated as secrets and can appear in API responses
  and logs.
- Operators should still avoid manually pasting credentials into log-visible
  free-form fields.

## Testing Strategy

Relevant unit coverage includes:

- `VmwareCbtMigrationOfferingValidationTest`
- `VmwareCbtMigrationDeletePolicyTest`
- `VmwareCbtStorageTargetTest`
- `VmwareCbtMigrationCutoverPolicyTest`
- `LibvirtVmwareCbtSyncCommandWrapperTest`
- `VmwareCbtSyncPlanTest`
- `LibvirtVmwareCbtCutoverCommandWrapperTest`
- `LibvirtVmwareCbtCleanupCommandWrapperTest`
- `LibvirtVmwareCbtRbdProbeCommandWrapperTest`
- `LibvirtStoragePoolTest`

The tests cover:

- Fixed offering validation.
- Dynamic/custom offering detail handling.
- Storage target classification.
- Non-in-place finalization gating.
- Cutover policy decisions.
- Delta copy script planning.
- Chunked changed-range reads from the nbdkit Unix socket.
- RBD probe create/write/read/delete command planning.
- RBD finalization through temporary localhost `qemu-nbd` bridges.
- Fallback `virt-v2v` output path handling.
- Cleanup scoping to migration directories.
- Cleanup scoping to migration-owned RBD image names.
- Nested relative KVM volume path resolution under filesystem pools.

## Operational Verification

Useful host capability check:

```bash
cmk list hosts name=<kvm-host-name> details=all | egrep \
'host.vddk.blockcopy.support|host.vddk.blockcopy.inplace.finalization.support|host.vddk.blockcopy.rbd.support|host.vddk.support|host.vddk.version|host.virtv2v.version|host.virtv2v.inplace.version|host.qemu.img.version|host.qemu.nbd.version|host.qemu.io.version|vddk.lib.dir'
```

Useful VDDK plugin check on the KVM host:

```bash
nbdkit vddk --dump-plugin libdir=/opt/vmware-vix-disklib-distrib | \
  egrep 'vddk_library_version|vddk_dll|vddk_transport_modes'
```

Useful process check during initial sync:

```bash
pgrep -af 'nbdkit|qemu-img|qemu-nbd|nbdcopy'
```

Useful target disk inspection:

```bash
qemu-img info /mnt/<pool-uuid>/cloudstack-cbt/<migration-uuid>/<disk>.qcow2
```

## References

- [Apache CloudStack: Importing VMware VMs into KVM](https://docs.cloudstack.apache.org/en/latest/adminguide/virtual_machines/importing_vmware_vms_into_kvm.html)
- [Broadcom vSphere `QueryChangedDiskAreas` API](https://developer.broadcom.com/xapis/virtual-infrastructure-json-api/latest/sdk/vim25/release/VirtualMachine/moId/QueryChangedDiskAreas/post/)
- [Broadcom Virtual Disk API disk operations](https://developer.broadcom.com/xapis/virtual-disk-api/latest/vddkFunctions.6.4.html)
- [Broadcom VDDK CBT notes and limitations](https://developer.broadcom.com/xapis/virtual-disk-api/latest/vddkBkupVadp.9.5.html)
- [nbdkit VDDK plugin](https://libguestfs.org/nbdkit-vddk-plugin.1.html)