docs/development.md

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Development

This document describes how to run a fully simulated development and test
environment for the CloudStack Kubernetes Provider: a real Kubernetes API
server, a real CloudStack management server and the CCM itself, all in
containers on your workstation. Nothing is mocked — the CCM makes genuine
CloudStack API calls and the resulting load balancer rules, public IPs and
firewall/ACL rules are real database objects you can inspect.

The pieces are:

| Component | What provides it |
| --- | --- |
| Kubernetes API server + kubelets | a [kind](https://kind.sigs.k8s.io/) cluster |
| CloudStack management server | the [`apache/cloudstack-simulator`](https://hub.docker.com/r/apache/cloudstack-simulator) container |
| Cloud controller manager | this repository, either in-cluster or as a host process |

## Prerequisites

* Docker
* [kind](https://kind.sigs.k8s.io/) v0.30 or later
* `kubectl`
* Go 1.23 or later
* [cmk](https://github.com/apache/cloudstack-cloudmonkey) (CloudMonkey), the
  CloudStack CLI — the harness drives the CloudStack API through it
* `jq` and `curl`
* About 12 GB of free disk and 8 GB of RAM

> **linux/amd64 only.** `apache/cloudstack-simulator` is not published for
> arm64. On Apple Silicon you can run it under emulation with
> `--platform linux/amd64` (expect the simulator to take three to five times
> longer to start), point the environment at a simulator running on an x86
> host, or build an arm64 image yourself from `tools/docker/` in the
> [apache/cloudstack](https://github.com/apache/cloudstack) repository.

## Quickstart

```bash
make e2e-up          # simulator + zone + kind cluster + CloudStack VMs + CCM
make test-e2e        # phase 1: load balancer, nodes, annotations
make e2e-vpc         # switch the environment to a VPC in a project
make test-e2e-vpc    # phase 2: VPC / network ACL
make e2e-down        # tear everything down
```

Phase 2 builds on phase 1, so run them in that order; both test targets stop
with a clear message if the environment they need is not up.

`make e2e-up` takes about seven minutes once the simulator image is pulled —
roughly 90 seconds for the simulator to start, two and a half minutes to
deploy the zone, and the rest for the kind cluster, the VMs and the CCM. The
first run also has to pull a ~2 GB image.

Once it is up, try the thing the CCM exists for:

```bash
export KUBECONFIG=hack/e2e/_out/kubeconfig
kubectl create deployment web --image=nginx
kubectl expose deployment web --port=80 --type=LoadBalancer
kubectl get svc web -w
```

The service gets an `EXTERNAL-IP` from the simulator's public IP range
(`192.168.2.0/24`), and the corresponding rule shows up in CloudStack:

```bash
cmk -c hack/e2e/_out/cmk.ini listLoadBalancerRules listall=true
```

The CloudStack UI is also available: run the simulator with `-p 8081:5050`
and open <http://localhost:8081/>, logging in as `admin` / `password`.

## How the harness talks to CloudStack

The scripts call `cmk` directly, always in the form

```bash
cmk -c hack/e2e/_out/cmk.ini <command> [key=value ...]
```

so every CloudStack command in the harness is one you can paste into a shell.
The config file is generated by the harness rather than read from
`~/.cmk/config`, so your own cmk profiles are left alone. (cmk takes its
config path only from `-c` or `$HOME`, with no environment variable for it,
which is why the flag is repeated rather than hidden behind a wrapper.)

The generated profile authenticates with `admin`/`password` rather than API
keys, because the harness has to talk to CloudStack *before* any keys exist —
it is what mints them. Two cmk defaults do real work here:

* `asyncblock = true` — cmk waits for async jobs such as
  `deployVirtualMachine` and returns the finished result, so nothing in the
  harness polls `queryAsyncJobResult`.
* `output = json` — responses come back *without* the `<command>response`
  envelope, so a zone list is `.zone[0].id`, not
  `.listzonesresponse.zone[0].id`. Worth knowing if you compare the scripts
  against raw API output.

## What the scripts do

`hack/e2e/up.sh` chains four numbered scripts. Each is independently runnable
and safe to re-run. All tunables live in `hack/e2e/env.sh` and can be
overridden from the environment.

### 1. `10-simulator-up.sh` — simulator and zone

Creates a docker bridge network (`cs-ccm-e2e`, `172.30.0.0/24`) that both the
simulator and the kind nodes will join, then starts the simulator on it:

```bash
docker network create --subnet 172.30.0.0/24 cs-ccm-e2e
docker run -d --name cloudstack-simulator --network cs-ccm-e2e \
    -p 127.0.0.1:8080:8080 apache/cloudstack-simulator:4.22.1.0
```

The image exposes three ports and it matters which one you use:

| Port | What it is |
| --- | --- |
| **8080** | the management server API (`/client/api`) — **use this one** |
| 8096 | the unauthenticated integration API, used by marvin |
| 5050 | the Vue UI development server, which proxies to 8080 |

The upstream simulator README suggests `-p 8080:5050`, which publishes the
*UI*. For API access, publish container port 8080 directly.

Readiness is checked in three stages rather than with a fixed sleep: jetty
answering at all, then the API accepting admin credentials, then
`listManagementServersMetrics` returning a server. The last one matters
because it proves a management server is registered and running, not just that
the API answers.

The zone is then deployed with marvin, which is preinstalled in the image:

```bash
docker exec cloudstack-simulator python3 \
    /root/tools/marvin/marvin/deployDataCenter.py -i /root/setup/dev/advanced.cfg
```

This creates the `Sandbox-simulator` advanced zone with a public IP range of
`192.168.2.2`–`192.168.2.200`.

Finally the script mints admin API keys for the CCM:

```bash
cmk -c hack/e2e/_out/cmk.ini listUsers username=admin      # -> the user id
cmk -c hack/e2e/_out/cmk.ini getUserKeys id=<user id>      # -> apikey, secretkey
```

`listUsers` is not a substitute for `getUserKeys` — it returns the API key but
never the secret. `registerUserKeys` is used only when no key pair exists yet,
because it *rotates* the keys, which would break a simulator you are reusing.

Keys land in `hack/e2e/_out/keys.env`.

### 2. `20-kind-up.sh` — the Kubernetes cluster

```bash
KIND_EXPERIMENTAL_DOCKER_NETWORK=cs-ccm-e2e kind create cluster \
    --name cs-ccm-e2e --config hack/e2e/kind-config.yaml
```

The cluster config does two important things:

* `cloud-provider: external` in every node's `kubeletExtraArgs`, so nodes
  register with the `node.cloudprovider.kubernetes.io/uninitialized` taint.
  Removing that taint is the CCM's job, and is how you know it works.
* `kubelet-preferred-address-types: InternalIP` on the API server. Once the
  CCM initializes a node it sets the node's Hostname address to the CloudStack
  instance's hostname, which for the simulator is the simulated hypervisor
  agent and is not resolvable. Without this setting, `kubectl logs` and
  `kubectl exec` stop working after node initialization.

The cluster is named so that node names are deterministic:
`cs-ccm-e2e-control-plane`, `cs-ccm-e2e-worker`, `cs-ccm-e2e-worker2`. Two
workers exist so tests can check that the control plane node — which kubeadm
labels `node.kubernetes.io/exclude-from-external-load-balancers` — is left out
of load balancer membership.

The script records each node's IP on the shared docker network into
`hack/e2e/_out/node-ips`. The next step depends on it.

> CoreDNS stays `Pending` until the CCM removes the uninitialized taint. That
> is expected; don't wait for it.

### 3. `30-topology-isolated.sh` — matching CloudStack VMs

**This is the part that makes or breaks the environment.** The CCM looks up
each Kubernetes node by name in CloudStack, so a VM must exist whose name
exactly matches the node name. On top of that, kind starts kubelet with
`--node-ip=<the node's docker IP>`, and the CCM's node controller refuses to
initialize a node whose kubelet-reported IP is not among the addresses the
cloud provider reports for it. So the VMs must also carry the *same IP
addresses* as the kind node containers.

The script therefore aligns the zone's guest CIDR with the docker subnet,
creates an isolated network on it, and deploys one VM per node pinned to that
node's IP:

```bash
cmk updateZone id=$ZONE guestcidraddress=172.30.0.0/24
cmk createNetwork name=ccm-e2e-iso networkofferingid=$OFFERING \
    gateway=172.30.0.1 netmask=255.255.255.0 zoneid=$ZONE
cmk deployVirtualMachine name=cs-ccm-e2e-worker displayname=cs-ccm-e2e-worker \
    ipaddress=172.30.0.4 networkids=$NET ...
```

The offering used is `DefaultIsolatedNetworkOfferingWithSourceNatService`,
which provides the **Firewall** service — so on this network the CCM manages
firewall rules. (The VPC scenario below uses an offering that provides
**NetworkACL** instead, exercising the other branch.)

All offerings and templates are looked up by name, because their UUIDs differ
between simulator deployments.

### 4. `40-ccm-deploy.sh` — the controller

Generates two `cloud-config` files that differ **only in `api-url`**:

* `hack/e2e/_out/cloud-config` — used by the in-cluster deployment, pointing
  at the simulator's IP on the `cs-ccm-e2e` docker network.
* `hack/e2e/_out/cloud-config-host` — used when you run the CCM as a host
  process, pointing at `http://localhost:8080/client/api`.

The in-cluster config must use the simulator's **IP address**, not its
container name or network alias: pods have their own network namespace and
cannot reach Docker's embedded DNS resolver, and `host.docker.internal` does
not exist on Linux Docker Engine.

Both configs set `zone` explicitly. If `zone` is empty the CCM tries to
detect it by looking up its own pod, which cannot work when running as a host
process.

The script then loads the image into kind and applies the repository's
[`deployment.yaml`](../deployment.yaml), changed for testing before it is
applied: the local image with `imagePullPolicy: Never`, `--leader-elect=false`
(single replica, faster startup), `--v=4` for useful logs, and higher CPU
limits — the stock manifest's `limits.cpu: 50m` throttles informer startup
badly on shared CI runners. Applying the stock manifest first would start the
released image, which can keep running for its 30s grace period and reconcile
the test Services next to the build under test.

Finally it waits for every node to lose the uninitialized taint.

## Running the CCM as a host process

For interactive development and debugging, skip step 4 and run the binary
directly against the same environment:

```bash
make
./cloudstack-ccm \
    --cloud-provider=external-cloudstack \
    --cloud-config=hack/e2e/_out/cloud-config-host \
    --kubeconfig=hack/e2e/_out/kubeconfig \
    --leader-elect=false \
    --v=4
```

If the in-cluster CCM is already running, scale it down first so the two do
not fight over the same services:

```bash
kubectl -n kube-system scale deployment/cloud-controller-manager --replicas=0
```

### Debugging

You can use the VS Code extension
[Go](https://marketplace.visualstudio.com/items?itemName=golang.go) to debug
the CCM. Add the following to `.vscode/launch.json`:

```json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch CloudStack CCM",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/cmd/cloudstack-ccm",
            "env": {},
            "args": [
                "--cloud-provider=external-cloudstack",
                "--cloud-config=${workspaceFolder}/hack/e2e/_out/cloud-config-host",
                "--kubeconfig=${workspaceFolder}/hack/e2e/_out/kubeconfig",
                "--leader-elect=false",
                "--v=4"
            ],
            "showLog": true,
            "trace": "verbose"
        },
        {
            "name": "Attach to Process",
            "type": "go",
            "request": "attach",
            "mode": "local",
            "processId": 0
        }
    ]
}
```

To debug against a real CloudStack installation instead of the simulator,
point `--cloud-config` at your own `cloud-config` and `--kubeconfig` at your
cluster's kubeconfig.

## The VPC scenario

`make e2e-vpc` (that is, `hack/e2e/50-topology-vpc.sh`) switches the
environment to a VPC network so the
Network ACL code path can be exercised. It creates a VPC, a **custom** ACL list,
a tier network and per-node VMs, then re-points the CCM at them and restarts it.

Two details are worth knowing:

* The ACL list must be a custom one. The CCM deliberately refuses to add rules
  to the built-in `default_allow` and `default_deny` lists.
* Everything is created inside a **CloudStack project**. The CCM matches VM
  names across the whole account, and fails with `found hosts that belong to
  different networks` if the matched VMs are spread over several networks.
  Because CloudStack hides project resources from non-project queries and vice
  versa, putting the VPC VMs in a project makes the two scenarios mutually
  invisible without needing a second cluster or a second account.

The tier reuses the same subnet as the isolated network, so the VMs keep the
same IP addresses and node initialization continues to work after the switch.

## Running the tests

Unit tests need nothing but Go:

```bash
make test
```

The end-to-end suite needs the environment above. It is behind the `e2e` build
tag, so it never runs as part of `make test` or `go build ./...`:

```bash
make test-e2e        # phase 1
make test-e2e-vpc    # phase 2, after `make e2e-vpc`
```

Configuration comes from the environment, using the same variable names as the
existing opt-in acceptance tests in `cloudstack_test.go`. The make targets set
these for you from `hack/e2e/_out/`; the table matters if you invoke `go test`
directly:

| Variable | Meaning |
| --- | --- |
| `KUBECONFIG` | cluster under test |
| `CS_API_URL` | CloudStack API endpoint as reachable from the test process |
| `CS_API_KEY`, `CS_SECRET_KEY` | CloudStack credentials |
| `CS_PROJECT_ID` | optional; set during the VPC phase |

When any of them is missing the tests skip rather than fail. The same suite
runs against a real CloudStack installation — just point the variables at it.

Each test creates its own namespace and cleans up after itself. Because load
balancer provisioning is asynchronous, all assertions poll rather than
assuming immediate consistency.

The VPC tests count every rule on their port in the tier's ACL list, and some
create rules of their own on 8080 and 8085 the way an operator would, deleting
them on cleanup. They stop at once if a rule for their port is left over from
an earlier run.

### Known limitation: provider IDs

kind starts kubelet with `--provider-id=kind://docker/<cluster>/<node>`, and
Kubernetes only allows a node's provider ID to be set once. In this
environment the CCM therefore never assigns the
`external-cloudstack://<instance UUID>` provider ID it would set on a real
cluster. `TestNode_ProviderID` detects this, logs the value it *would* have
assigned, and reports itself as skipped, so the gap stays visible instead of
quietly passing. Everything else about node initialization — taint removal,
labels, addresses — is exercised normally.

## Continuous integration

[`.github/workflows/e2e-simulator.yml`](../.github/workflows/e2e-simulator.yml)
runs this environment on every pull request and every push to `main`, as a
matrix of the latest two Kubernetes versions against the latest two CloudStack
releases. The CloudStack axis is not only version coverage: CloudStack 4.22
and later update a load balancer rule's CIDR list in place, while earlier
versions delete and recreate the rule, so both branches get tested.

All matrix cells run in parallel and a shared build job compiles the CCM image
once, so the whole workflow takes about as long as a single run — roughly
fifteen minutes, most of it the simulator image pull and zone deployment.

To change the versions under test, edit the `k8s` and `acs` lists in the
matrix. Both use explicit patch-level tags
([`kindest/node`](https://hub.docker.com/r/kindest/node/tags) and
[`apache/cloudstack-simulator`](https://hub.docker.com/r/apache/cloudstack-simulator/tags)),
so a run is reproducible; avoid floating tags like `latest`.

## Troubleshooting

| Symptom | Cause |
| --- | --- |
| `LB service provider cannot support this rule` on a VPC | The VPC virtual router accepts only a restricted set of public load balancer ports. 80 and 8080 work; an arbitrary high port such as 8081 is rejected. Pick a port the router supports when adding a VPC test. |
| CCM exits with `no CloudStack version returned by the management server` | `listCapabilities` answered without a `cloudstackversion`. Set `version` in the `cloud-config` to pin it and skip the lookup. |
| Nodes keep the uninitialized taint; CCM logs `provided node ip for node "..." is not valid` | The CloudStack VM's NIC IP does not match the IP kubelet registered with. Recreate the VM with `ipaddress=` set to the kind node's docker IP. |
| Services stay `<pending>`; CCM logs `none of the hosts matched the list of VMs retrieved from CS API` | No CloudStack VM has a name matching a Kubernetes node name. |
| CCM logs `found hosts that belong to different networks` | VMs matching the node names exist on more than one network — typically leftovers from a previous scenario. |
| No ACL rules are created on a VPC network | The tier uses `default_allow` or `default_deny`; the CCM only manages rules on custom ACL lists. Or the list already has an ingress rule for that port that the CCM did not create; the CCM then relies on that rule and adds none of its own. |
| A VPC test fails at once with `leftover ACL rule` | A rule for its port survived an earlier run. Delete it from the tier's ACL list and rerun. |
| CoreDNS stuck `Pending` | Expected until the CCM removes the uninitialized taint. If it persists, the CCM is not working — check its logs. |
| `kubectl logs`/`exec` fail after nodes initialize | The API server is preferring the Hostname address, which the CCM set to the CloudStack instance hostname. Use `kubelet-preferred-address-types: InternalIP` as the provided kind config does. |
| Simulator never becomes ready | It runs `mvn jetty:run` and fetches from Maven Central at startup. Check `docker logs cloudstack-simulator`. |