Network Namespace: support custom actions for Policy-Based Routing
diff --git a/Network-Namespace/README.md b/Network-Namespace/README.md index b57982b..3889941 100644 --- a/Network-Namespace/README.md +++ b/Network-Namespace/README.md
@@ -743,6 +743,56 @@ > are NOT removed on destroy — they may still be used by other networks or for > VM connectivity. +### VPC lifecycle commands: `implement-vpc`, `shutdown-vpc`, `destroy-vpc` + +These commands manage VPC-level state. Called by `NetworkExtensionElement` when +implementing, shutting down, or destroying a VPC (before or after per-tier +network operations). + +#### `implement-vpc` + +``` +network-namespace-wrapper.sh implement-vpc \ + --vpc-id <vpc-id> \ + --cidr <vpc-cidr> +``` + +Actions: +1. Create the shared VPC namespace `cs-vpc-<vpc-id>`. +2. Enable IP forwarding inside the namespace. +3. Create iptables chains for NAT and filter rules. +4. Save VPC metadata (CIDR, gateway) to state files under `/var/lib/cloudstack/<ext-name>/vpc-<vpc-id>/`. + +> This command runs **before** any tier networks are implemented. Tier networks +> inherit the same namespace and host assignment. + +#### `shutdown-vpc` + +``` +network-namespace-wrapper.sh shutdown-vpc \ + --vpc-id <vpc-id> +``` + +Actions: +1. Flush all iptables rules (ingress, egress, NAT chains inside the namespace). +2. Stop all services (dnsmasq, haproxy, apache2, password-server) for all tiers. +3. Keep the namespace and tier veths intact (tiers may restart). + +> Called when the VPC is shut down; tier networks may be restarted later. + +#### `destroy-vpc` + +``` +network-namespace-wrapper.sh destroy-vpc \ + --vpc-id <vpc-id> +``` + +Actions: +1. Remove the entire namespace `cs-vpc-<vpc-id>` (deletes all interfaces inside). +2. Remove VPC-wide state directory `/var/lib/cloudstack/<ext-name>/vpc-<vpc-id>/`. + +> This is the final cleanup step; after this, the VPC namespace is gone. + ### `assign-ip` Called when a public IP is associated with the network (including source NAT). @@ -929,6 +979,55 @@ `default_egress_allow` policy (allow-by-default or deny-by-default) to VM outbound traffic on `-i vn-<vlan>-<id>`. +### `apply-network-acl` + +Apply Network ACL (Access Control List) rules for VPC networks. + +``` +network-namespace-wrapper.sh apply-network-acl \ + --network-id <id> \ + --vlan <vlan-id> \ + --acl-rules <base64-json> \ + [--vpc-id <vpc-id>] +``` + +The `--acl-rules` value is a Base64-encoded JSON array of ACL rule objects: +```json +[ + { + "id": 1, + "number": 100, + "trafficType": "Ingress", + "action": "Allow", + "protocol": "tcp", + "portStart": 80, + "portEnd": 80, + "sourceCidrs": ["0.0.0.0/0"] + }, + { + "id": 2, + "number": 200, + "trafficType": "Egress", + "action": "Allow", + "protocol": "all", + "destCidrs": ["0.0.0.0/0"] + } +] +``` + +iptables design: + +* **Ingress rules** (filter FORWARD, chain `CS_EXTNET_ACL_IN_<networkId>`): + Matches `-i vn-<vlan>-<id>` (traffic entering the VM namespace), + ordered by rule number. Actions: ACCEPT or DROP. + +* **Egress rules** (filter FORWARD, chain `CS_EXTNET_ACL_OUT_<networkId>`): + Matches `-o vn-<vlan>-<id>` (traffic leaving the VM namespace), + ordered by rule number. Actions: ACCEPT or DROP. + +Both chains are inserted at position 1 of `CS_EXTNET_FWD_<networkId>` so ACL rules +take precedence over the catch-all ACCEPT rules. + ### `config-dhcp-subnet` / `remove-dhcp-subnet` Configure or tear down dnsmasq DHCP service for the network inside the namespace. @@ -1107,6 +1206,37 @@ |--------|-------------| | `reboot-device` | Bounces the guest veth pair (`vh-<vlan>-<id>` down → up) | | `dump-config` | Prints namespace IP addresses, iptables rules, and per-network state to stdout | +| `pbr-create-table` | Create or update a routing-table entry in `/etc/iproute2/rt_tables` | +| `pbr-delete-table` | Remove a routing-table entry from `/etc/iproute2/rt_tables` | +| `pbr-list-tables` | List non-comment routing-table entries from `/etc/iproute2/rt_tables` | +| `pbr-add-route` | Add/replace an `ip route` entry in a specific routing table inside the namespace | +| `pbr-delete-route` | Delete an `ip route` entry from a specific routing table inside the namespace | +| `pbr-list-routes` | List routes from one table (or all tables) inside the namespace | +| `pbr-add-rule` | Add an `ip rule` policy rule mapped to a specific routing table inside the namespace | +| `pbr-delete-rule` | Delete an `ip rule` policy rule mapped to a specific routing table inside the namespace | +| `pbr-list-rules` | List policy rules (or only rules for one table) inside the namespace | + +PBR action parameter keys (`--action-params` JSON): + +| Action | Required keys | Optional keys | +|--------|---------------|---------------| +| `pbr-create-table` | `table-id` (or `id`), `table-name` (or `table`) | — | +| `pbr-delete-table` | `table-id` or `table-name` | — | +| `pbr-list-tables` | — | — | +| `pbr-add-route` | `table`, `route` | — | +| `pbr-delete-route` | `table`, `route` | — | +| `pbr-list-routes` | — | `table` | +| `pbr-add-rule` | `table`, `rule` | — | +| `pbr-delete-rule` | `table`, `rule` | — | +| `pbr-list-rules` | — | `table` | + +Examples (equivalent to direct Linux commands): + +* `{"table-id":"100","table-name":"isp1"}` → `100 isp1` +* `{"table":"isp1","route":"default via 192.168.1.1 dev eth0"}` +* `{"table":"vpn1","route":"default dev wg0"}` +* `{"table":"isp1","rule":"from 10.10.1.0/24"}` +* `{"table":"vpn1","rule":"to 10.10.2.0/24"}` To add custom actions, place an executable script at `${STATE_DIR}/hooks/custom-action-<name>.sh` @@ -1207,6 +1337,34 @@ "parameters[0].key=threshold" "parameters[0].value=90" ``` +### PBR custom-action examples + +```bash +# 1) Create action definitions (once per extension) +cmk addCustomAction extensionid=<ext-uuid> name=pbr-create-table resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-add-route resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-add-rule resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-list-tables resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-list-routes resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-list-rules resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-delete-rule resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-delete-route resourcetype=Network +cmk addCustomAction extensionid=<ext-uuid> name=pbr-delete-table resourcetype=Network + +# 2) Execute against a network +cmk runNetworkCustomAction networkid=<network-uuid> actionid=<pbr-create-table-id> \ + "parameters[0].key=table-id" "parameters[0].value=100" \ + "parameters[1].key=table-name" "parameters[1].value=isp1" + +cmk runNetworkCustomAction networkid=<network-uuid> actionid=<pbr-add-route-id> \ + "parameters[0].key=table" "parameters[0].value=isp1" \ + "parameters[1].key=route" "parameters[1].value=default via 192.168.1.1 dev eth0" + +cmk runNetworkCustomAction networkid=<network-uuid> actionid=<pbr-add-rule-id> \ + "parameters[0].key=table" "parameters[0].value=isp1" \ + "parameters[1].key=rule" "parameters[1].value=from 10.10.1.0/24" +``` + CloudStack calls `NetworkExtensionElement.runCustomAction()`, which issues: ```bash network-namespace.sh custom-action \ @@ -1226,6 +1384,23 @@ ## Developer / testing notes +### VPC Support + +The extension now supports **VPC (Virtual Private Cloud)** networks in addition to +isolated networks. Key differences from isolated networks: + +* **Namespace sharing**: All tiers of a VPC share a single namespace (`cs-vpc-<vpcId>`) + instead of each network getting its own (`cs-net-<networkId>`). +* **Host affinity**: All tiers of a VPC land on the same KVM host via stable hash-based + selection using the VPC ID as the routing key. +* **VPC-level operations**: `implement-vpc`, `shutdown-vpc`, `destroy-vpc` commands + manage VPC-wide state (namespace creation/teardown). +* **VPC tier operations**: `implement-network`, `shutdown-network`, `destroy-network` + commands manage per-tier bridges and routes; the namespace is preserved across + tier lifecycle operations. + +### Integration tests + The integration smoke test at `test/integration/smoke/test_network_extension_namespace.py` exercises the full lifecycle against real KVM hosts in the zone. @@ -1248,10 +1423,12 @@ * Create / list / update / delete external network device. * Full network lifecycle: implement → assign-ip (source NAT) → static NAT → port forwarding → firewall rules → DHCP/DNS → shutdown / destroy. +* VPC multi-tier networks with shared namespace and automatic host affinity. * NSP state transitions: Disabled → Enabled → Disabled → Deleted. * Tests `test_04`, `test_05`, `test_06` (DHCP, DNS, LB) require `arping`, `dnsmasq`, and `haproxy` on the KVM hosts; the test skips them automatically if these tools are not installed. +* Script cleanup on both management server and KVM hosts after each test. Run the test: ```bash
diff --git a/Network-Namespace/network-namespace-wrapper.sh b/Network-Namespace/network-namespace-wrapper.sh index c8af377..d237de8 100755 --- a/Network-Namespace/network-namespace-wrapper.sh +++ b/Network-Namespace/network-namespace-wrapper.sh
@@ -1397,7 +1397,7 @@ dhcp-hostsfile=${dhcp_hosts} addn-hosts=${hosts} dhcp-optsfile=${dhcp_opts} -log-facility=/var/log/cloudstack/network-namespace-dnsmasq-${NETWORK_ID}.log +log-facility=/var/log/cloudstack/extensions/${_WRAPPER_EXT_DIR}/dnsmasq-${NETWORK_ID}.log EOF # Add DHCP option 15 (domain-search) when provided by the caller if [ -n "${DOMAIN}" ]; then @@ -1649,7 +1649,7 @@ ${authz_line} DocumentRoot ${www} -ErrorLog /var/log/cloudstack/network-namespace-apache2-${NETWORK_ID}.log +ErrorLog /var/log/cloudstack/extensions/${_WRAPPER_EXT_DIR}/apache2-${NETWORK_ID}.log <VirtualHost ${listen_ip}:80> ServerName metadata @@ -1718,7 +1718,7 @@ local script_f; script_f=$(_passwd_server_script) local pid_f; pid_f=$(_passwd_server_pid) local passwd_f; passwd_f=$(_passwd_file) - local log_f; log_f="/var/log/cloudstack/network-namespace-passwd-${NETWORK_ID}.log" + local log_f; log_f="/var/log/cloudstack/extensions/${_WRAPPER_EXT_DIR}/passwd-${NETWORK_ID}.log" mkdir -p "$(dirname "${script_f}")" touch "${passwd_f}" @@ -2721,6 +2721,159 @@ ############################################################################## # Command: custom-action +_pbr_param() { + # Return the first non-empty key from ACTION_PARAMS_JSON. + local _k _v + for _k in "$@"; do + _v=$(_json_get "${ACTION_PARAMS_JSON}" "${_k}") + if [ -n "${_v}" ]; then + echo "${_v}" + return 0 + fi + done + echo "" +} + +_pbr_table_file() { echo "/etc/iproute2/rt_tables"; } + +_pbr_create_table() { + local tid tname tf tmp + tid="$(_pbr_param table-id table_id id tableid)" + tname="$(_pbr_param table-name table_name name tablename table)" + [ -z "${tid}" ] && die "pbr-create-table: missing table id" + [ -z "${tname}" ] && die "pbr-create-table: missing table name" + + tf="$(_pbr_table_file)" + grep -Eq "^[[:space:]]*${tid}[[:space:]]+${tname}([[:space:]]|$)" "${tf}" 2>/dev/null && { + echo "pbr-create-table: exists ${tid} ${tname}" + return 0 + } + + tmp=$(mktemp /tmp/cs-extnet-rt-tables-XXXXXX) + awk -v tid="${tid}" -v tname="${tname}" ' + BEGIN { done = 0 } + { + if ($0 ~ "^[[:space:]]*#" || $0 ~ "^[[:space:]]*$") { print; next } + if ($1 == tid || $2 == tname) { + if (!done) { + print tid " " tname + done = 1 + } + next + } + print + } + END { + if (!done) print tid " " tname + } + ' "${tf}" > "${tmp}" + cat "${tmp}" > "${tf}" + rm -f "${tmp}" 2>/dev/null || true + echo "pbr-create-table: OK ${tid} ${tname}" +} + +_pbr_delete_table() { + local tid tname tf tmp + tid="$(_pbr_param table-id table_id id tableid)" + tname="$(_pbr_param table-name table_name name tablename table)" + [ -z "${tid}" ] && [ -z "${tname}" ] && die "pbr-delete-table: missing table id/name" + + tf="$(_pbr_table_file)" + tmp=$(mktemp /tmp/cs-extnet-rt-tables-XXXXXX) + awk -v tid="${tid}" -v tname="${tname}" ' + { + if ($0 ~ "^[[:space:]]*#" || $0 ~ "^[[:space:]]*$") { print; next } + if ((tid != "" && $1 == tid) || (tname != "" && $2 == tname)) { + next + } + print + } + ' "${tf}" > "${tmp}" + cat "${tmp}" > "${tf}" + rm -f "${tmp}" 2>/dev/null || true + echo "pbr-delete-table: OK id=${tid:-n/a} name=${tname:-n/a}" +} + +_pbr_list_tables() { + awk ' + { + if ($0 ~ "^[[:space:]]*#" || $0 ~ "^[[:space:]]*$") next + print + } + ' "$(_pbr_table_file)" +} + +_pbr_add_route() { + local table route + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + route="$(_pbr_param route route-spec route_spec)" + [ -z "${table}" ] && die "pbr-add-route: missing table" + [ -z "${route}" ] && die "pbr-add-route: missing route spec" + [ -z "${NAMESPACE}" ] && die "pbr-add-route: namespace not resolved" + + # replace is idempotent and avoids duplicate route errors. + ip netns exec "${NAMESPACE}" sh -c "ip route replace ${route} table ${table}" + echo "pbr-add-route: OK table=${table} route=${route}" +} + +_pbr_delete_route() { + local table route + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + route="$(_pbr_param route route-spec route_spec)" + [ -z "${table}" ] && die "pbr-delete-route: missing table" + [ -z "${route}" ] && die "pbr-delete-route: missing route spec" + [ -z "${NAMESPACE}" ] && die "pbr-delete-route: namespace not resolved" + + ip netns exec "${NAMESPACE}" sh -c "ip route del ${route} table ${table}" 2>/dev/null || true + echo "pbr-delete-route: OK table=${table} route=${route}" +} + +_pbr_list_routes() { + local table + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + [ -z "${NAMESPACE}" ] && die "pbr-list-routes: namespace not resolved" + if [ -n "${table}" ]; then + ip netns exec "${NAMESPACE}" ip route show table "${table}" + else + ip netns exec "${NAMESPACE}" ip route show table all + fi +} + +_pbr_add_rule() { + local table rule + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + rule="$(_pbr_param rule rule-spec rule_spec)" + [ -z "${table}" ] && die "pbr-add-rule: missing table" + [ -z "${rule}" ] && die "pbr-add-rule: missing rule spec" + [ -z "${NAMESPACE}" ] && die "pbr-add-rule: namespace not resolved" + + ip netns exec "${NAMESPACE}" sh -c "ip rule add ${rule} table ${table}" 2>/dev/null || true + echo "pbr-add-rule: OK table=${table} rule=${rule}" +} + +_pbr_delete_rule() { + local table rule + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + rule="$(_pbr_param rule rule-spec rule_spec)" + [ -z "${table}" ] && die "pbr-delete-rule: missing table" + [ -z "${rule}" ] && die "pbr-delete-rule: missing rule spec" + [ -z "${NAMESPACE}" ] && die "pbr-delete-rule: namespace not resolved" + + ip netns exec "${NAMESPACE}" sh -c "ip rule del ${rule} table ${table}" 2>/dev/null || true + echo "pbr-delete-rule: OK table=${table} rule=${rule}" +} + +_pbr_list_rules() { + local table + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + [ -z "${NAMESPACE}" ] && die "pbr-list-rules: namespace not resolved" + if [ -n "${table}" ]; then + ip netns exec "${NAMESPACE}" ip rule show | grep -E "[[:space:]]lookup[[:space:]]+${table}([[:space:]]|$)" || true + else + ip netns exec "${NAMESPACE}" ip rule show + fi +} + cmd_custom_action() { NETWORK_ID="" VPC_ID="" @@ -2753,6 +2906,7 @@ CHOSEN_ID="${VPC_ID:-${NETWORK_ID}}" _load_state + acquire_lock "${NETWORK_ID}" log "custom-action: network=${NETWORK_ID} ns=${NAMESPACE} action=${ACTION_NAME} params=${ACTION_PARAMS_JSON}" @@ -2782,16 +2936,45 @@ echo "=== VPC/shared state ($(_vpc_state_dir)) ===" ls -la "$(_vpc_state_dir)/" 2>/dev/null || echo "(no vpc state)" ;; + pbr-create-table) + _pbr_create_table + ;; + pbr-delete-table) + _pbr_delete_table + ;; + pbr-list-tables) + _pbr_list_tables + ;; + pbr-add-route) + _pbr_add_route + ;; + pbr-delete-route) + _pbr_delete_route + ;; + pbr-list-routes) + _pbr_list_routes + ;; + pbr-add-rule) + _pbr_add_rule + ;; + pbr-delete-rule) + _pbr_delete_rule + ;; + pbr-list-rules) + _pbr_list_rules + ;; *) local hook="${STATE_DIR}/hooks/custom-action-${ACTION_NAME}.sh" if [ -x "${hook}" ]; then exec "${hook}" --network-id "${NETWORK_ID}" --action "${ACTION_NAME}" \ --action-params "${ACTION_PARAMS_JSON}" else - die "Unknown action '${ACTION_NAME}'. Built-ins: reboot-device, dump-config" + die "Unknown action '${ACTION_NAME}'. Built-ins: reboot-device, dump-config, pbr-*" fi ;; esac + + release_lock } ##############################################################################