Network Namespace: support VPC
diff --git a/Network-Namespace/README.md b/Network-Namespace/README.md index 3bfe1bf..b57982b 100644 --- a/Network-Namespace/README.md +++ b/Network-Namespace/README.md
@@ -67,8 +67,8 @@ │ NetworkExtensionElement.java │ │ │ executes (path resolved from Extension record) │ │ ▼ │ -│ /etc/cloudstack/extensions/<ext-name>/ │ -│ network-namespace.sh │ +│ /usr/share/cloudstack-management/extensions/<ext-name>/ │ +│ <ext-name>.sh (network-namespace.sh) │ └──────────────────────┬───────────────────────────────────┘ │ SSH (host : port from extension details) │ credentials from extension_resource_map_details @@ -82,23 +82,24 @@ │ │ │ HOST side │ │ ───────────────────────────────────────────────── │ -│ eth0.1910 ─────────────────────────────────┐ │ +│ eth1.1910 ─────────────────────────────────┐ │ │ (VLAN sub-iface) │ │ -│ breth0-1910 (bridge) │ -│ veth-host-1910 ────────────────────────────┘ │ +│ breth1-1910 (bridge) │ +│ vh-1910-d1 ─────────────────────────────────┘ │ │ │ │ -│ NAMESPACE cs-net-209 (or cs-net-<vpcId>) │ +│ NAMESPACE cs-net-209 (isolated) │ +│ cs-vpc-5 (VPC, vpc-id=5) │ │ ───────────────────────────────────────────────── │ -│ veth-ns-1910 ← gateway IP 10.1.1.1/24 │ +│ vn-1910-d1 ← gateway IP 10.1.1.1/24 │ │ │ │ PUBLIC side (source-NAT IP 10.0.56.4 on VLAN 101): │ │ │ │ HOST side │ -│ eth0.101 ─────────────────────────────────┐ │ -│ breth0-101 (bridge) │ +│ eth1.101 ─────────────────────────────────┐ │ +│ breth1-101 (bridge) │ │ vph-101-209 ────────────────────────────────┘ │ │ │ │ -│ NAMESPACE cs-net-209 (or cs-net-<vpcId>) │ +│ NAMESPACE cs-net-209 (or cs-vpc-<vpcId>) │ │ vpn-101-209 ← source-NAT IP 10.0.56.4/32 │ │ default route → 10.0.56.1 (upstream gateway) │ └──────────────────────────────────────────────────────────┘ @@ -108,25 +109,23 @@ | Object | Name pattern | Example (VLAN 1910, net 209, pub-VLAN 101) | |--------|--------------|-------------------------------------------| -| Namespace (standalone network) | `cs-net-<networkId>` | `cs-net-209` | -| Namespace (VPC network) | `cs-net-<vpcId>` | `cs-net-5` | -| Guest host bridge | `br<ethX>-<vlan>` | `breth0-1910` | +| Namespace (isolated network) | `cs-net-<networkId>` | `cs-net-209` | +| Namespace (VPC network) | `cs-vpc-<vpcId>` | `cs-vpc-5` | +| Guest host bridge | `br<ethX>-<vlan>` | `breth1-1910` | | Guest veth – host side | `vh-<vlan>-<id>` | `vh-1910-d1` | | Guest veth – namespace side | `vn-<vlan>-<id>` | `vn-1910-d1` | -| Public host bridge | `br<pub_ethX>-<pvlan>` | `breth0-101` | +| Public host bridge | `br<pub_ethX>-<pvlan>` | `breth1-101` | | Public veth – host side | `vph-<pvlan>-<id>` | `vph-101-209` | | Public veth – namespace side | `vpn-<pvlan>-<id>` | `vpn-101-209` | -`ethX` (and `pub_ethX`) is the **physical NIC** resolved from the -`kvmnetworklabel` (`public_kvmnetworklabel`) stored in the physical-network -extension details: +`ethX` (and `pub_ethX`) is the NIC specified in the `guest.network.device` +(and `public.network.device`) key when registering the extension on the +physical network. Both default to `eth1` when not explicitly set. -* `eth1` → `eth1` (not in `/sys/devices/virtual/net/` → already physical) -* `cloudbr1` → `eth1` (virtual bridge → first non-virtual - `/sys/class/net/cloudbr1/brif/` member) - -Both labels are automatically included in `--physical-network-extension-details` -by `NetworkExtensionElement` — no extra registration step is needed. +> **Note:** when `<vlan>` or `<id>` would make the interface name exceed the +> Linux 15-character limit, the `<id>` portion is shortened to its hex +> representation (for numeric IDs) or a 6-character MD5 prefix (for +> non-numeric IDs). **Key design principles:** @@ -139,9 +138,10 @@ VLAN sub-interfaces live on the **host** (not inside the namespace) so that guest VMs whose NICs are connected to `brethX-<vlan>` reach the namespace gateway without any additional configuration. -* **VPC networks** share a single namespace per VPC (`cs-net-<vpcId>`). Multiple - guest VLANs are each connected via their own veth pair (`veth-host-<vlan>` / - `veth-ns-<vlan>`). +* **VPC networks** share a single namespace per VPC (`cs-vpc-<vpcId>`). Multiple + guest VLANs are each connected via their own veth pair (`vh-<vlan>-<id>` / + `vn-<vlan>-<id>`). +* **Isolated networks** each get their own namespace (`cs-net-<networkId>`). * The two scripts are intentionally decoupled: you can replace either script with a custom implementation (Python, Go, etc.) as long as the interface contract (arguments and exit codes) is maintained. @@ -208,36 +208,56 @@ During package installation the `network-namespace.sh` script is deployed to: ``` -/etc/cloudstack/extensions/<extension-name>/network-namespace.sh +/usr/share/cloudstack-management/extensions/<extension-name>/<extension-name>.sh ``` +The extension path is set to `network-namespace` at creation time; +`NetworkExtensionElement` looks for `<extensionName>.sh` inside the directory. In **developer mode** the extensions directory defaults to `extensions/` relative -to the repo root working directory, so `extensions/network-namespace/network-namespace.sh` -is found automatically when `path=network-namespace` is set at extension creation -time (CloudStack looks for `<extensionName>.sh` inside the directory). +to the repo root, so `extensions/network-namespace/network-namespace.sh` is +found automatically. ### Remote network device -Copy `network-namespace-wrapper.sh` to each remote device that will act as the -network gateway: +Copy `network-namespace-wrapper.sh` to **each** remote device that will act as the +network gateway, inside a subdirectory named after the extension: ```bash # From the CloudStack source tree: +DEVICE=root@<kvm-host> +EXT_NAME=network-namespace # must match the extension name in CloudStack + +ssh ${DEVICE} "mkdir -p /etc/cloudstack/extensions/${EXT_NAME}" scp extensions/network-namespace/network-namespace-wrapper.sh \ - root@<device>:/etc/cloudstack/extensions/network-namespace-wrapper.sh -chmod +x /etc/cloudstack/extensions/network-namespace-wrapper.sh + ${DEVICE}:/etc/cloudstack/extensions/${EXT_NAME}/${EXT_NAME}-wrapper.sh +ssh ${DEVICE} "chmod +x /etc/cloudstack/extensions/${EXT_NAME}/${EXT_NAME}-wrapper.sh" ``` -The default path expected by `network-namespace.sh` is -`/etc/cloudstack/extensions/network-namespace/network-namespace-wrapper.sh`. -You can override this per-physical-network by passing a `script_path` detail -when calling `registerExtension` (see below). +The wrapper derives its state directory and log path from the directory it is +installed in: + +* **State:** `/var/lib/cloudstack/<ext-name>/` + (e.g. `/var/lib/cloudstack/network-namespace/`) +* **Log (wrapper):** `/var/log/cloudstack/extensions/<ext-name>/<ext-name>.log` + (e.g. `/var/log/cloudstack/extensions/network-namespace/network-namespace.log`) +* **Log (proxy, on management server):** `/var/log/cloudstack/extensions/<ext-name>.log` **Prerequisites on the remote device:** -* `iproute2` (`ip link`, `ip addr`, `ip netns`) -* `iptables` -* `sshd` running and reachable from the management server -* The SSH user must have permission to run `ip` and `iptables` (root or `sudo`) + +| Package / tool | Purpose | +|----------------|---------| +| `iproute2` (`ip`, `ip netns`) | Namespace, bridge, veth, route management | +| `iptables` + `iptables-save` | NAT and filter rules inside namespace | +| `arping` | Gratuitous ARP after public IP assignment | +| `dnsmasq` | DHCP and DNS service inside namespace | +| `haproxy` | Load balancing inside namespace | +| `apache2` (Debian/Ubuntu) or `httpd` (RHEL/CentOS) | Metadata / user-data HTTP service (port 80) | +| `python3` | DHCP options parsing, haproxy config generation, vm-data processing | +| `util-linux` (`flock`) | Serialise concurrent operations per network | +| `sshd` | Reachable from the management server on the configured port (default 22) | + +The SSH user must have permission to run `ip`, `iptables`, `iptables-save`, +and `ip netns exec` (root or passwordless `sudo` for those commands). --- @@ -321,6 +341,24 @@ physical network and enables it automatically. The NSP name is the **extension name** — not the generic string `NetworkExtension`. +After registering, set the connection details for the remote KVM device(s): + +```bash +cmk updateRegisteredExtension \ + extensionid=<extension-uuid> \ + resourcetype=PhysicalNetwork \ + resourceid=<phys-net-uuid> \ + "details[0].key=hosts" "details[0].value=192.168.10.1,192.168.10.2" \ + "details[1].key=username" "details[1].value=root" \ + "details[2].key=sshkey" "details[2].value=<pem-key-contents>" \ + "details[3].key=guest.network.device" "details[3].value=eth1" \ + "details[4].key=public.network.device" "details[4].value=eth1" +``` + +The `hosts` value is a comma-separated list of KVM host IPs; `ensure-network-device` +picks one per network and stores it in `--network-extension-details`. Use `sshkey` +(PEM private key) for passwordless authentication, or `password` + `sshpass`. + Verify: ```bash cmk listNetworkServiceProviders physicalnetworkid=<phys-net-uuid> @@ -369,8 +407,8 @@ ``` > The `serviceCapabilityList` entries must match the values declared in the -> extension's `network.capabilities` detail. If the extension's JSON does not -> declare a capability value for a service, CloudStack accepts any value (or no +> extension's `network.service.capabilities` detail. If the extension's JSON does +> not declare a capability value for a service, CloudStack accepts any value (or no > value) without error. ### 4. Create an isolated network @@ -402,9 +440,11 @@ --cidr 10.0.1.0/24 ``` -The wrapper creates a VLAN sub-interface and Linux bridge, assigns the gateway -IP to the bridge, enables IP forwarding, and creates dedicated per-network -iptables chains (`CS_EXTNET_42` in `nat` and `CS_EXTNET_FWD_42` in `filter`). +The wrapper creates a VLAN sub-interface and Linux bridge, a guest veth pair +(`vh-100-2a`/`vn-100-2a`), assigns the gateway IP to the namespace veth, +enables IP forwarding inside the namespace, and creates per-network iptables +chains: `CS_EXTNET_42_PR` (nat PREROUTING), `CS_EXTNET_42_POST` (nat +POSTROUTING), and `CS_EXTNET_FWD_42` (filter FORWARD). ### 5. Acquire a public IP and enable Source NAT @@ -426,10 +466,15 @@ ``` The wrapper: -1. Adds `203.0.113.10/32` as a secondary address on the physical interface. -2. Adds an iptables SNAT rule: traffic from `10.0.1.0/24` outbound → source `203.0.113.10`. -3. Adds an iptables FORWARD rule allowing traffic from the guest CIDR to the - physical interface. +1. Creates public VLAN sub-interface `eth1.<pvlan>` and bridge `breth1-<pvlan>` on the host. +2. Creates veth pair `vph-<pvlan>-42` (host, in bridge) / `vpn-<pvlan>-42` (namespace). +3. Assigns `203.0.113.10/32` to `vpn-<pvlan>-42` **inside the namespace**. +4. Adds host route `203.0.113.10/32 dev vph-<pvlan>-42` so the host can reach it. +5. Adds an iptables SNAT rule in `CS_EXTNET_42_POST`: traffic from `10.0.1.0/24` + out `vpn-<pvlan>-42` → source `203.0.113.10`. +6. Adds an iptables FORWARD ACCEPT rule in `CS_EXTNET_FWD_42` for the guest CIDR. +7. If `--public-gateway` is set, adds/replaces the namespace default route via + `vpn-<pvlan>-42`. When the IP is released (via `disassociateIpAddress`), `release-ip` is called, which removes all associated rules and the IP address. @@ -454,15 +499,15 @@ --private-ip 10.0.1.5 ``` -iptables rules added: +iptables rules added (all run inside the namespace via `ip netns exec`): ```bash -# DNAT inbound -iptables -t nat -A CS_EXTNET_42 -d 203.0.113.20 -j DNAT --to-destination 10.0.1.5 -# SNAT outbound -iptables -t nat -A CS_EXTNET_42 -s 10.0.1.5 -o eth0 -j SNAT --to-source 203.0.113.20 -# FORWARD inbound + outbound -iptables -t filter -A CS_EXTNET_FWD_42 -d 10.0.1.5 -o cs-br-42 -j ACCEPT -iptables -t filter -A CS_EXTNET_FWD_42 -s 10.0.1.5 -i cs-br-42 -j ACCEPT +# DNAT inbound (CS_EXTNET_42_PR = nat PREROUTING chain) +iptables -t nat -A CS_EXTNET_42_PR -d 203.0.113.20 -j DNAT --to-destination 10.0.1.5 +# SNAT outbound (CS_EXTNET_42_POST = nat POSTROUTING chain) +iptables -t nat -A CS_EXTNET_42_POST -s 10.0.1.5 -o vpn-<pvlan>-42 -j SNAT --to-source 203.0.113.20 +# FORWARD inbound + outbound (CS_EXTNET_FWD_42 = filter FORWARD chain) +iptables -t filter -A CS_EXTNET_FWD_42 -d 10.0.1.5 -o vn-100-2a -j ACCEPT +iptables -t filter -A CS_EXTNET_FWD_42 -s 10.0.1.5 -i vn-100-2a -j ACCEPT ``` ```bash @@ -498,14 +543,14 @@ --protocol TCP ``` -iptables rules added: +iptables rules added (inside the namespace): ```bash -# DNAT inbound -iptables -t nat -A CS_EXTNET_42 -p tcp -d 203.0.113.20 --dport 2222 \ +# DNAT inbound (CS_EXTNET_42_PR = nat PREROUTING chain) +iptables -t nat -A CS_EXTNET_42_PR -p tcp -d 203.0.113.20 --dport 2222 \ -j DNAT --to-destination 10.0.1.5:22 -# FORWARD +# FORWARD (CS_EXTNET_FWD_42 = filter FORWARD chain) iptables -t filter -A CS_EXTNET_FWD_42 -p tcp -d 10.0.1.5 --dport 22 \ - -o cs-br-42 -j ACCEPT + -o vn-100-2a -j ACCEPT ``` Port ranges (e.g. `80:90`) are supported and passed verbatim to iptables `--dport`. @@ -533,11 +578,16 @@ The wrapper: 1. Removes jump rules from PREROUTING, POSTROUTING, and FORWARD. -2. Flushes and deletes iptables chains `CS_EXTNET_42` and `CS_EXTNET_FWD_42`. -3. Brings down and deletes bridge `cs-br-42`. -4. Brings down and deletes VLAN interface `eth0.100` (VLAN ID read from state if - not passed in arguments). -5. Removes all state under `/var/lib/cloudstack/network-namespace/42/`. +2. Flushes and deletes iptables chains `CS_EXTNET_42_PR`, `CS_EXTNET_42_POST`, + `CS_EXTNET_FWD_42`, and any `CS_EXTNET_FWRULES_42` / `CS_EXTNET_FWI_*` chains. +3. Deletes public veth pairs (`vph-<pvlan>-42` / `vpn-<pvlan>-42`) that were + created during `assign-ip` (read from state files). +4. On `destroy`: also deletes the guest veth host-side (`vh-100-2a`) and removes + the namespace `cs-net-42` entirely. +5. Removes all state under `/var/lib/cloudstack/<ext-name>/network-42/`. + +> The host bridge `breth1-100` and VLAN sub-interface `eth1.100` are **not** +> removed — they may still be used by other networks or for VM connectivity. ### 9. Unregister and delete the extension @@ -547,10 +597,15 @@ cmk deleteNetworkServiceProvider id=<nsp-uuid> # Remove external network device credentials (if any) -# Device credentials are stored as `extension_resource_map_details` for the -# extension registration. Remove or update them via `updateExtension` or -# by unregistering the extension from the physical network (unregisterExtension) -# and then updating the Extension record if necessary. +# Device credentials are stored as extension_resource_map_details for the +# extension registration. Remove or update them via `updateRegisteredExtension` +# (set cleanupdetails=true to wipe all details) or by supplying new details. +# Example: clear all registration details for a physical network: +cmk updateRegisteredExtension \ + extensionid=<extension-uuid> \ + resourcetype=PhysicalNetwork \ + resourceid=<phys-net-uuid> \ + cleanupdetails=true # Unregister the extension from the physical network cmk unregisterExtension \ @@ -575,14 +630,18 @@ # Register two extensions, each backed by a different device cmk registerExtension id=<ext-a-uuid> resourcetype=PhysicalNetwork resourceid=<pn-uuid> cmk registerExtension id=<ext-b-uuid> resourcetype=PhysicalNetwork resourceid=<pn-uuid> -``` -# Store device connection details and script_path as registration details -# (use updateNetworkServiceProvider or updateExtension details in the API / CMK) -# Example: set hosts, sshkey, script_path for the registered extension on the physical network -# Note: details are stored in extension_resource_map_details for the registration -cmk updateExtension id=<ext-uuid> "details[0].key=hosts" "details[0].value=10.0.0.1,10.0.0.2" \ - "details[1].key=script_path" "details[1].value=/etc/cloudstack/extensions/network-namespace/network-namespace-wrapper.sh" +# Store device connection details as registration details for each extension. +# Details are stored in extension_resource_map_details for the registration. +# Example: set hosts and guest/public network devices for ext-a on the physical network: +cmk updateRegisteredExtension \ + extensionid=<ext-a-uuid> \ + resourcetype=PhysicalNetwork \ + resourceid=<pn-uuid> \ + "details[0].key=hosts" "details[0].value=10.0.0.1,10.0.0.2" \ + "details[1].key=guest.network.device" "details[1].value=eth1" \ + "details[2].key=public.network.device" "details[2].value=eth1" +``` When creating network offerings, reference the specific extension name: @@ -613,10 +672,11 @@ `--option value` pairs. All commands: -* Write timestamped entries to `/var/log/cloudstack/network-namespace.log`. -* Use a per-network flock file (`/var/lib/cloudstack/network-namespace/lock-<id>`) - to serialise concurrent operations. -* Persist state under `/var/lib/cloudstack/network-namespace/<network-id>/`. +* Write timestamped entries to `/var/log/cloudstack/extensions/<ext-name>/<ext-name>.log`. +* Use a per-network flock file (`${STATE_DIR}/lock-network-<id>`) — or + `lock-vpc-<id>` for VPC networks — to serialise concurrent operations. +* Persist state under `/var/lib/cloudstack/<ext-name>/network-<network-id>/` + (or `vpc-<vpc-id>/` for VPC-wide shared state such as public IPs). ### `implement` @@ -632,17 +692,17 @@ ``` Actions: -1. Create namespace `cs-net-<vpc-id>` (VPC) or `cs-net-<network-id>` (standalone). -2. Resolve `ethX` from `kvmnetworklabel` in `--physical-network-extension-details`. -3. Create VLAN sub-interface `ethX.<vlan>` on the host. -4. Create host bridge `brethX-<vlan>` and attach `ethX.<vlan>` to it. -5. Create veth pair `veth-host-<vlan>` (host) / `veth-ns-<vlan>` (namespace). - Attach host end to `brethX-<vlan>`. -6. Assign `<gateway>/<prefix>` to `veth-ns-<vlan>` inside the namespace. +1. Create namespace `cs-vpc-<vpc-id>` (VPC) or `cs-net-<network-id>` (isolated). +2. Resolve `GUEST_ETH` from `guest.network.device` in `--physical-network-extension-details` + (defaults to `eth1` when absent). +3. Create VLAN sub-interface `GUEST_ETH.<vlan>` on the host. +4. Create host bridge `br<GUEST_ETH>-<vlan>` and attach `GUEST_ETH.<vlan>` to it. +5. Create veth pair `vh-<vlan>-<id>` (host, in bridge) / `vn-<vlan>-<id>` (namespace). +6. Assign `<gateway>/<prefix>` to `vn-<vlan>-<id>` inside the namespace. 7. Enable IP forwarding inside the namespace. -8. Create iptables chains `CS_EXTNET_<id>_PR` (PREROUTING DNAT), - `CS_EXTNET_<id>_POST` (POSTROUTING SNAT), and `CS_EXTNET_FWD_<id>` (FORWARD). -9. Save VLAN, gateway, CIDR, namespace, network-id or vpc-id to state files. +8. Create iptables chains `CS_EXTNET_<id>_PR` (nat PREROUTING DNAT), + `CS_EXTNET_<id>_POST` (nat POSTROUTING SNAT), and `CS_EXTNET_FWD_<id>` (filter FORWARD). +9. Save VLAN, gateway, CIDR, namespace, and network-id / vpc-id to state files. ### `shutdown` @@ -654,12 +714,15 @@ ``` Actions: -1. Flush and remove iptables chains (PREROUTING, POSTROUTING, FORWARD jumps + - chain contents). -2. Delete public veth pairs (`vph-<pvlan>-<id>`) that were created during - `assign-ip` (read from state). -3. Keep namespace and guest veth (`veth-host-<vlan>` / `veth-ns-<vlan>`) intact — - guest VMs can still connect to `brethX-<vlan>`. +1. Stop dnsmasq, haproxy, apache2, and password-server processes running inside + the namespace (if any). +2. Flush and remove iptables chains (PREROUTING, POSTROUTING, FORWARD jumps + + chain contents), including `CS_EXTNET_FWRULES_<id>` and all `CS_EXTNET_FWI_*` + ingress chains. +3. Delete public veth pairs (`vph-<pvlan>-<id>` / `vpn-<pvlan>-<id>`) that were + created during `assign-ip` (read from state). +4. Keep namespace and guest veth (`vh-<vlan>-<id>` / `vn-<vlan>-<id>`) intact — + guest VMs can still connect to `br<GUEST_ETH>-<vlan>`. ### `destroy` @@ -671,14 +734,14 @@ ``` Actions (superset of shutdown): -1. Delete guest veth host-side (`veth-host-<vlan>`). -2. Delete public veth pairs. +1. Delete guest veth host-side (`vh-<vlan>-<id>`). +2. Delete public veth pairs (`vph-<pvlan>-<id>` / `vpn-<pvlan>-<id>`). 3. Delete the namespace (removes all interfaces inside it). -4. Remove state directory. +4. Remove per-network state directory `network-<id>/`. -> The host bridge `brethX-<vlan>` and VLAN sub-interface `ethX.<vlan>` are NOT -> removed on destroy — they may still be used by other networks or for VM -> connectivity. +> The host bridge `br<GUEST_ETH>-<vlan>` and VLAN sub-interface `GUEST_ETH.<vlan>` +> are NOT removed on destroy — they may still be used by other networks or for +> VM connectivity. ### `assign-ip` @@ -699,10 +762,11 @@ ``` Actions: -1. Resolve `pub_ethX` from `public_kvmnetworklabel` (falls back to `kvmnetworklabel`). -2. Create VLAN sub-interface `pub_ethX.<pvlan>` and bridge `brpub_ethX-<pvlan>` on the host. +1. Resolve `PUB_ETH` from `public.network.device` in `--physical-network-extension-details` + (defaults to `eth1` when absent). +2. Create VLAN sub-interface `PUB_ETH.<pvlan>` and bridge `br<PUB_ETH>-<pvlan>` on the host. 3. Create veth pair `vph-<pvlan>-<id>` (host) / `vpn-<pvlan>-<id>` (namespace). - Attach host end to `brpub_ethX-<pvlan>`. + Attach host end to `br<PUB_ETH>-<pvlan>`. 4. Assign `<public-ip>/32` (or `/<prefix>` if `--public-cidr` given) to `vpn-<pvlan>-<id>` inside the namespace. 5. Add host route `<public-ip>/32 dev vph-<pvlan>-<id>` so the host can reach it. @@ -760,10 +824,10 @@ |-------|-------|------| | `nat` | `CS_EXTNET_<id>_PR` | `-d <public-ip> -j DNAT --to-destination <private-ip>` | | `nat` | `CS_EXTNET_<id>_POST` | `-s <private-ip> -o vpn-<pvlan>-<id> -j SNAT --to-source <public-ip>` | -| `filter` | `CS_EXTNET_FWD_<id>` | `-d <private-ip> -o veth-ns-<vlan> -j ACCEPT` | -| `filter` | `CS_EXTNET_FWD_<id>` | `-s <private-ip> -i veth-ns-<vlan> -j ACCEPT` | +| `filter` | `CS_EXTNET_FWD_<id>` | `-d <private-ip> -o vn-<vlan>-<id> -j ACCEPT` | +| `filter` | `CS_EXTNET_FWD_<id>` | `-s <private-ip> -i vn-<vlan>-<id> -j ACCEPT` | -State saved to `/var/lib/cloudstack/network-namespace/<id>/static-nat/<public-ip>`. +State saved to `${STATE_DIR}/network-<id>/static-nat/<public-ip>`. ### `delete-static-nat` @@ -792,17 +856,17 @@ --protocol tcp|udp ``` -iptables rules added: +iptables rules added (inside the namespace): | Table | Chain | Rule | |-------|-------|------| -| `nat` | `CS_EXTNET_<id>` | `-p <proto> -d <public-ip> --dport <public-port> -j DNAT --to-destination <private-ip>:<private-port>` | -| `filter` | `CS_EXTNET_FWD_<id>` | `-p <proto> -d <private-ip> --dport <private-port> -o cs-br-<id> -j ACCEPT` | +| `nat` | `CS_EXTNET_<id>_PR` | `-p <proto> -d <public-ip> --dport <public-port> -j DNAT --to-destination <private-ip>:<private-port>` | +| `filter` | `CS_EXTNET_FWD_<id>` | `-p <proto> -d <private-ip> --dport <private-port> -o vn-<vlan>-<id> -j ACCEPT` | Port ranges (`80:90`) are passed verbatim to iptables `--dport`. State saved to -`/var/lib/cloudstack/network-namespace/<id>/port-forward/<proto>_<public-ip>_<public-port>`. +`${STATE_DIR}/network-<id>/port-forward/<proto>_<public-ip>_<public-port>`. ### `delete-port-forward` @@ -818,6 +882,217 @@ Removes the DNAT and FORWARD rules added by `add-port-forward`. +### `apply-fw-rules` + +Called when CloudStack applies or removes firewall rules for the network. + +``` +network-namespace-wrapper.sh apply-fw-rules \ + --network-id <id> \ + --vlan <vlan-id> \ + --fw-rules <base64-json> \ + [--vpc-id <vpc-id>] +``` + +The `--fw-rules` value is a Base64-encoded JSON object: +```json +{ + "default_egress_allow": true, + "cidr": "10.0.1.0/24", + "rules": [ + { + "type": "ingress", + "protocol": "tcp", + "portStart": 22, + "portEnd": 22, + "publicIp": "203.0.113.10", + "sourceCidrs": ["0.0.0.0/0"] + }, + { + "type": "egress", + "protocol": "all", + "sourceCidrs": ["0.0.0.0/0"] + } + ] +} +``` + +iptables design (two independent parts, both inside the namespace): + +* **Ingress** (mangle PREROUTING, per public IP): + Per-public-IP chains `CS_EXTNET_FWI_<pubIp>` check traffic *before* DNAT so + the match is against the real public destination IP. Traffic not matched by + explicit ALLOW rules is dropped. + +* **Egress** (filter FORWARD, chain `CS_EXTNET_FWRULES_<networkId>`): + Inserted at position 1 of `CS_EXTNET_FWD_<networkId>`. Applies the + `default_egress_allow` policy (allow-by-default or deny-by-default) to VM + outbound traffic on `-i vn-<vlan>-<id>`. + +### `config-dhcp-subnet` / `remove-dhcp-subnet` + +Configure or tear down dnsmasq DHCP service for the network inside the namespace. + +**`config-dhcp-subnet` arguments:** +``` +network-namespace-wrapper.sh config-dhcp-subnet \ + --network-id <id> \ + --gateway <gw> \ + --cidr <cidr> \ + [--dns <dns-server>] \ + [--domain <domain>] \ + [--vpc-id <vpc-id>] +``` + +Actions: writes a dnsmasq configuration file under +`${STATE_DIR}/network-<id>/dnsmasq/` and starts or reloads the dnsmasq process +inside the namespace. DNS on port 53 is **disabled** by `config-dhcp-subnet` +(use `config-dns-subnet` to enable it). + +**`remove-dhcp-subnet` arguments:** +``` +network-namespace-wrapper.sh remove-dhcp-subnet --network-id <id> +``` + +Actions: stops dnsmasq and removes the dnsmasq configuration directory. + +### `add-dhcp-entry` / `remove-dhcp-entry` + +Add or remove a static DHCP host reservation (MAC → IP mapping) from dnsmasq. + +``` +network-namespace-wrapper.sh add-dhcp-entry \ + --network-id <id> \ + --mac <mac> \ + --ip <vm-ip> \ + [--hostname <name>] \ + [--default-nic true|false] +``` + +When `--default-nic false`, the DHCP option 3 (default gateway) is suppressed +for that MAC so the VM does not get a competing default route via a secondary NIC. + +``` +network-namespace-wrapper.sh remove-dhcp-entry \ + --network-id <id> \ + --mac <mac> +``` + +### `set-dhcp-options` + +Set extra DHCP options for a specific NIC (identified by `--nic-id`) using a +JSON map of option-code → value pairs. + +``` +network-namespace-wrapper.sh set-dhcp-options \ + --network-id <id> \ + --nic-id <nic-id> \ + --options '{"119":"example.com"}' +``` + +### `config-dns-subnet` / `remove-dns-subnet` + +Enable or disable DNS (port 53) in the dnsmasq instance. + +``` +network-namespace-wrapper.sh config-dns-subnet \ + --network-id <id> \ + --gateway <gw> \ + --cidr <cidr> \ + [--extension-ip <ip>] \ + [--domain <domain>] \ + [--vpc-id <vpc-id>] +``` + +Actions: like `config-dhcp-subnet` but enables DNS on port 53. Also registers a +`data-server` hostname entry (using `--extension-ip` if provided, otherwise +`--gateway`) for metadata service discovery. + +``` +network-namespace-wrapper.sh remove-dns-subnet --network-id <id> +``` + +Actions: disables DNS (rewrites config to disable port 53) but keeps DHCP running. + +### `add-dns-entry` / `remove-dns-entry` + +Add or remove a hostname → IP mapping in the dnsmasq hosts file. + +``` +network-namespace-wrapper.sh add-dns-entry \ + --network-id <id> \ + --ip <vm-ip> \ + --hostname <name> + +network-namespace-wrapper.sh remove-dns-entry \ + --network-id <id> \ + --ip <vm-ip> +``` + +### `save-vm-data` + +Write the full VM metadata/userdata/password set for a VM in a single call. +Called on network restart and VM deploy. + +``` +network-namespace-wrapper.sh save-vm-data \ + --network-id <id> \ + --ip <vm-ip> \ + --vm-data <base64-json> +``` + +The `--vm-data` value is a Base64-encoded JSON array of `{dir, file, content}` +entries (same format as `generateVmData()` in the Java layer). Writes files +under `${STATE_DIR}/network-<id>/metadata/<vm-ip>/latest/`. After writing, +starts or reloads both the **apache2 metadata HTTP service** (port 80) and the +**VR-compatible password server** (port 8080) inside the namespace. + +### `save-userdata` / `save-password` / `save-sshkey` / `save-hypervisor-hostname` + +Granular variants that write individual VM metadata fields: + +``` +network-namespace-wrapper.sh save-userdata --network-id <id> --ip <vm-ip> --userdata <base64> +network-namespace-wrapper.sh save-password --network-id <id> --ip <vm-ip> --password <plain> +network-namespace-wrapper.sh save-sshkey --network-id <id> --ip <vm-ip> --sshkey <base64> +network-namespace-wrapper.sh save-hypervisor-hostname \ + --network-id <id> --ip <vm-ip> --hypervisor-hostname <name> +``` + +Each command writes the relevant file and restarts/reloads apache2 (and +the password server, for `save-password`). + +### `apply-lb-rules` + +Apply or revoke load-balancing rules via haproxy inside the namespace. + +``` +network-namespace-wrapper.sh apply-lb-rules \ + --network-id <id> \ + --lb-rules <json-array> \ + [--vpc-id <vpc-id>] +``` + +`--lb-rules` is a JSON array of LB rule objects. Set `"revoke": true` on a +rule to remove it. The wrapper regenerates the haproxy configuration from the +persistent per-rule JSON files under `${STATE_DIR}/network-<id>/haproxy/` and +reloads haproxy inside the namespace. haproxy is stopped when no active rules +remain. + +### `restore-network` + +Batch-restore DHCP/DNS/metadata/services for all VMs on a network in a single +call. Invoked on network restart to rebuild all state at once instead of N +per-VM calls. + +``` +network-namespace-wrapper.sh restore-network \ + --network-id <id> \ + --restore-data <base64-json> \ + [--gateway <gw>] [--cidr <cidr>] [--dns <dns>] \ + [--domain <dom>] [--extension-ip <ip>] [--vpc-id <vpc-id>] +``` + ### `custom-action` ``` @@ -830,11 +1105,12 @@ | Action | Description | |--------|-------------| -| `reboot-device` | Bounces the bridge: `ip link set cs-br-<id> down && up` | -| `dump-config` | Prints iptables rules and bridge/interface state to stdout | +| `reboot-device` | Bounces the guest veth pair (`vh-<vlan>-<id>` down → up) | +| `dump-config` | Prints namespace IP addresses, iptables rules, and per-network state to stdout | To add custom actions, place an executable script at -`/var/lib/cloudstack/network-namespace/hooks/custom-action-<name>.sh`. +`${STATE_DIR}/hooks/custom-action-<name>.sh` +(e.g. `/var/lib/cloudstack/network-namespace/hooks/custom-action-<name>.sh`). Unknown action names are delegated to the hook if present; otherwise the command fails with a descriptive error. @@ -861,37 +1137,34 @@ | `username` | SSH user — default: `root` | | `password` | SSH password via `sshpass` — sensitive, not logged | | `sshkey` | PEM-encoded SSH private key — sensitive, not logged; preferred over password | +| `guest.network.device` | Host NIC for guest (internal) traffic, e.g. `eth1` — defaults to `eth1` when absent | +| `public.network.device` | Host NIC for public (NAT/external) traffic, e.g. `eth1` — defaults to `eth1` when absent | -These keys are **automatically injected** by `NetworkExtensionElement` from the -physical network record — no manual registration needed: +This key is **automatically injected** by `NetworkExtensionElement` from the +physical network record: | JSON key | Description | |----------|-------------| | `physicalnetworkname` | Physical network name from CloudStack DB | -| `kvmnetworklabel` | KVM guest traffic label (e.g. `eth0`, `cloudbr0`) | -| `vmwarenetworklabel` | VMware guest traffic label | -| `xennetworklabel` | XenServer guest traffic label | -| `public_kvmnetworklabel` | KVM public traffic label (used for public bridges) | -| `public_vmwarenetworklabel` | VMware public traffic label | -The wrapper script uses `kvmnetworklabel` (and `public_kvmnetworklabel`) to -derive the physical NIC `ethX` via `/sys/devices/virtual/net/` inspection, then -names bridges as `brethX-<vlan>`. +The wrapper script uses `guest.network.device` (and `public.network.device`) to +name bridges as `br<eth>-<vlan>` and veth pairs as `vh-<vlan>-<id>` / +`vn-<vlan>-<id>` (guest) and `vph-<pvlan>-<id>` / `vpn-<pvlan>-<id>` (public). ### Per-network details (keys in `--network-extension-details`) | JSON key | Description | |----------|-------------| | `host` | Previously selected host IP (set by `ensure-network-device`) | -| `namespace` | Linux network namespace name (e.g. `cs-net-<networkId>`) | +| `namespace` | Linux network namespace name (e.g. `cs-net-<networkId>` or `cs-vpc-<vpcId>`) | ### Additional per-command arguments | CLI Argument | Commands | Description | |--------------|----------|-------------| -| `--vpc-id <id>` | all | Inject when network belongs to a VPC; namespace becomes `cs-net-<vpcId>` | +| `--vpc-id <id>` | all | Present when the network belongs to a VPC; namespace becomes `cs-vpc-<vpcId>` | | `--public-vlan <pvlan>` | `assign-ip`, `release-ip` | Public IP's VLAN tag (e.g. `101`) | -| `--network-id <id>` | `assign-ip`, `release-ip`, `add-static-nat`, `delete-static-nat`, `add-port-forward`, `delete-port-forward` | VPC ID if VPC network, else network ID — used in public veth names (`vph-<pvlan>-<id>`, `vpn-<pvlan>-<id>`) | +| `--network-id <id>` | most | Network ID — CHOSEN_ID for veth names is `<vpc-id>` when VPC, else `<network-id>` | ### Action parameters (custom-action only) @@ -955,44 +1228,51 @@ The integration smoke test at `test/integration/smoke/test_network_extension_namespace.py` -exercises the full lifecycle using a **Linux network namespace** on the Marvin -node as the simulated remote device: +exercises the full lifecycle against real KVM hosts in the zone. ``` -Marvin node (this machine — also acts as the remote network device) - ├── ip netns add cs-extnet-<id> ← isolated namespace - ├── ~/.ssh/authorized_keys ← test RSA key ← management server connects here - └── /etc/cloudstack/extensions/ - └── network-namespace-wrapper.sh ← copied from repo by test +Management server + └── /usr/share/cloudstack-management/extensions/<ext-name>/ + └── network-namespace.sh ← deployed / referenced by test + SSHes to KVM host + runs network-namespace-wrapper.sh <cmd> <args> -KVM hosts in the zone (best-effort, skipped if none found) - └── /etc/cloudstack/extensions/ - └── network-namespace-wrapper.sh ← copied by KvmHostDeployer - -Management server (may be the same machine) +KVM host(s) in the zone └── /etc/cloudstack/extensions/<ext-name>/ - └── network-namespace.sh ← deployed by test (static copy) - reads CS_PHYSICAL_NETWORK_EXTENSION_DETAILS (JSON) - CS_NETWORK_EXTENSION_DETAILS (JSON) - SSHes back to Marvin node :22 - runs ip netns exec cs-extnet-<id> <script> <args> + └── network-namespace-wrapper.sh ← copied to KVM hosts by test setup + creates cs-net-<id> or cs-vpc-<id> namespaces + manages bridges, veth pairs, iptables, dnsmasq, haproxy, apache2 ``` The test covers: * Create / list / update / delete external network device. * Full network lifecycle: implement → assign-ip (source NAT) → static NAT → - port forwarding → shutdown / destroy. + port forwarding → firewall rules → DHCP/DNS → shutdown / destroy. * NSP state transitions: Disabled → Enabled → Disabled → Deleted. +* Tests `test_04`, `test_05`, `test_06` (DHCP, DNS, LB) require `arping`, + `dnsmasq`, and `haproxy` on the KVM hosts; the test skips them automatically + if these tools are not installed. Run the test: ```bash cd test/integration/smoke -nosetests test_network_extension_provider.py \ +python -m pytest test_network_extension_namespace.py \ --with-marvin --marvin-config=<config.cfg> \ -s -a 'tags=advanced,smoke' 2>&1 | tee /tmp/extnet-test.log ``` -**Prerequisites:** -* `iproute2` on the Marvin node (`ip netns list` must succeed). -* The Marvin node must be reachable by SSH from the management server on port 22. -* Set `MARVIN_NODE_IP=<ip>` if auto-detection of the Marvin node IP fails. +**Prerequisites on KVM hosts:** +* `iproute2` (`ip`, `ip netns`) +* `iptables` + `iptables-save` +* `arping` (for GARP on IP assignment) +* `dnsmasq` (DHCP + DNS — required for `test_04` / DNS tests) +* `haproxy` (LB — required for `test_05` / LB tests) +* `apache2` / `httpd` (metadata HTTP service — required for UserData tests) +* `python3` (vm-data processing, haproxy config generation) +* `util-linux` (`flock`) (lock serialization) +* SSH access from management server (root or sudo-capable user) + +**Prerequisites on the Marvin / test runner node:** +* Python Marvin library installed (`pip install -r requirements.txt`) +* A valid Marvin config file pointing to the CloudStack environment +* The test runner must be able to SSH to the management server and to KVM hosts
diff --git a/Network-Namespace/network-namespace-wrapper.sh b/Network-Namespace/network-namespace-wrapper.sh index 8051c01..c8af377 100755 --- a/Network-Namespace/network-namespace-wrapper.sh +++ b/Network-Namespace/network-namespace-wrapper.sh
@@ -349,6 +349,7 @@ nat_chain() { echo "${CHAIN_PREFIX}_${1}"; } filter_chain() { echo "${CHAIN_PREFIX}_FWD_${1}"; } firewall_chain() { echo "${CHAIN_PREFIX}_FWRULES_${1}"; } +acl_chain() { echo "${CHAIN_PREFIX}_ACL_${1}"; } ensure_public_ip_on_namespace() { local public_ip="$1" public_cidr="$2" pveth_n="$3" pveth_h="$4" addr_spec prefix @@ -460,10 +461,15 @@ LB_RULES_JSON="[]" DEFAULT_NIC="true" VM_DATA="" + VM_DATA_FILE="" DOMAIN="" EXTENSION_IP="" RESTORE_DATA="" + RESTORE_DATA_FILE="" FW_RULES_JSON="" + FW_RULES_FILE="" + ACL_RULES_JSON="" + ACL_RULES_FILE="" while [ $# -gt 0 ]; do case "$1" in @@ -495,10 +501,15 @@ --lb-rules) LB_RULES_JSON="$2"; shift 2 ;; --default-nic) DEFAULT_NIC="$2"; shift 2 ;; --vm-data) VM_DATA="$2"; shift 2 ;; + --vm-data-file) VM_DATA_FILE="$2"; shift 2 ;; --domain) DOMAIN="$2"; shift 2 ;; --extension-ip) EXTENSION_IP="$2"; shift 2 ;; --restore-data) RESTORE_DATA="$2"; shift 2 ;; + --restore-data-file) RESTORE_DATA_FILE="$2"; shift 2 ;; --fw-rules) FW_RULES_JSON="$2"; shift 2 ;; + --fw-rules-file) FW_RULES_FILE="$2"; shift 2 ;; + --acl-rules) ACL_RULES_JSON="$2"; shift 2 ;; + --acl-rules-file) ACL_RULES_FILE="$2"; shift 2 ;; # consumed by _pre_scan_args — skip silently --physical-network-extension-details|--network-extension-details) shift 2 ;; @@ -577,7 +588,7 @@ } ############################################################################## -# Command: implement +# Command: implement-network # # 1. Create namespace cs-net-<id> # 2. Create host bridge br<ethX>-<vlan> with ethX.<vlan> sub-interface @@ -586,11 +597,11 @@ # 5. Set up iptables chains inside namespace ############################################################################## -cmd_implement() { +cmd_implement_network() { parse_args "$@" acquire_lock "${NETWORK_ID}" - log "implement: network=${NETWORK_ID} ns=${NAMESPACE} vlan=${VLAN} gw=${GATEWAY} cidr=${CIDR}" + log "implement-network: network=${NETWORK_ID} ns=${NAMESPACE} vlan=${VLAN} gw=${GATEWAY} cidr=${CIDR}" local veth_h veth_n nchain_pr nchain_post fchain veth_h=$(veth_host_name "${VLAN}" "${CHOSEN_ID}") @@ -706,22 +717,22 @@ _dump_iptables "${NAMESPACE}" release_lock - log "implement: done network=${NETWORK_ID} namespace=${NAMESPACE}" + log "implement-network: done network=${NETWORK_ID} namespace=${NAMESPACE}" } ############################################################################## -# Command: shutdown +# Command: shutdown-network # Flush iptables chains and remove this network's veth pairs. # For VPC networks the shared namespace is preserved (other tiers still use it). # For isolated networks the namespace is also removed. ############################################################################## -cmd_shutdown() { +cmd_shutdown_network() { parse_args "$@" _load_state acquire_lock "${NETWORK_ID}" - log "shutdown: network=${NETWORK_ID} ns=${NAMESPACE} vpc=${VPC_ID}" + log "shutdown-network: network=${NETWORK_ID} ns=${NAMESPACE} vpc=${VPC_ID}" local nchain_pr nchain_post fchain nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" @@ -752,7 +763,7 @@ if [ -f "${tier_f}" ]; then local owner_tier; owner_tier=$(cat "${tier_f}" 2>/dev/null || true) if [ -n "${owner_tier}" ] && [ "${owner_tier}" != "${NETWORK_ID}" ]; then - log "shutdown: skipping veth for $(basename "${f%.pvlan}") (owned by tier ${owner_tier})" + log "shutdown-network: skipping veth for $(basename "${f%.pvlan}") (owned by tier ${owner_tier})" continue fi fi @@ -760,7 +771,7 @@ pvlan=$(cat "${f}") pveth_h=$(pub_veth_host_name "${pvlan}" "${CHOSEN_ID}") ip link del "${pveth_h}" 2>/dev/null || true - log "shutdown: removed public veth ${pveth_h}" + log "shutdown-network: removed public veth ${pveth_h}" done fi @@ -768,7 +779,7 @@ local veth_h veth_h=$(veth_host_name "${VLAN}" "${CHOSEN_ID}") ip link del "${veth_h}" 2>/dev/null || true - log "shutdown: removed guest veth ${veth_h}" + log "shutdown-network: removed guest veth ${veth_h}" # Clean transient public IP state. # For isolated networks the state dir is per-network, so wipe it entirely. @@ -791,28 +802,28 @@ # across tiers and must only be deleted when the last tier is destroyed. if [ -z "${VPC_ID}" ]; then ip netns del "${NAMESPACE}" 2>/dev/null || true - log "shutdown: deleted namespace ${NAMESPACE}" + log "shutdown-network: deleted namespace ${NAMESPACE}" else - log "shutdown: preserved shared namespace ${NAMESPACE} (VPC tier)" + log "shutdown-network: preserved shared namespace ${NAMESPACE} (VPC tier)" fi release_lock - log "shutdown: done network=${NETWORK_ID}" + log "shutdown-network: done network=${NETWORK_ID}" } ############################################################################## -# Command: destroy +# Command: destroy-network # Delete this network's state entirely. -# For VPC networks the shared namespace is only deleted when this is the last -# remaining tier (tracked via vpc-<vpcId>/tiers/<networkId> marker files). +# For VPC tier networks the shared namespace is preserved — the namespace is +# removed only when shutdownVpc() calls shutdown-vpc or destroy-vpc. ############################################################################## -cmd_destroy() { +cmd_destroy_network() { parse_args "$@" _load_state acquire_lock "${NETWORK_ID}" - log "destroy: network=${NETWORK_ID} ns=${NAMESPACE} vpc=${VPC_ID}" + log "destroy-network: network=${NETWORK_ID} ns=${NAMESPACE} vpc=${VPC_ID}" # Remove this tier's guest veth host-side local veth_h @@ -831,7 +842,7 @@ if [ -f "${tier_f}" ]; then local owner_tier; owner_tier=$(cat "${tier_f}" 2>/dev/null || true) if [ -n "${owner_tier}" ] && [ "${owner_tier}" != "${NETWORK_ID}" ]; then - log "destroy: skipping veth for $(basename "${f%.pvlan}") (owned by tier ${owner_tier})" + log "destroy-network: skipping veth for $(basename "${f%.pvlan}") (owned by tier ${owner_tier})" continue fi fi @@ -855,29 +866,18 @@ # Deregister this tier from VPC tracking if [ -n "${VPC_ID}" ]; then rm -f "${vsd}/tiers/${NETWORK_ID}" 2>/dev/null || true - # Only delete the VPC-wide namespace and state when no more tiers remain - local remaining_tiers - remaining_tiers=$(find "${vsd}/tiers/" -type f 2>/dev/null | wc -l) - if [ "${remaining_tiers}" -le 0 ]; then - if ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then - ip netns del "${NAMESPACE}" - log "destroy: deleted shared namespace ${NAMESPACE} (last VPC tier)" - fi - rm -rf "${vsd}" - log "destroy: removed VPC state dir ${vsd}" - else - log "destroy: preserved namespace ${NAMESPACE} (${remaining_tiers} tier(s) remaining)" - fi + # The VPC namespace is managed by shutdown-vpc / destroy-vpc — do NOT delete it here. + log "destroy-network: deregistered tier ${NETWORK_ID} from VPC ${VPC_ID} (namespace preserved)" else # Isolated network: delete the namespace directly if ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then ip netns del "${NAMESPACE}" - log "destroy: deleted namespace ${NAMESPACE}" + log "destroy-network: deleted namespace ${NAMESPACE}" fi fi release_lock - log "destroy: done network=${NETWORK_ID}" + log "destroy-network: done network=${NETWORK_ID}" } ############################################################################## @@ -961,7 +961,9 @@ fi # ---- Source NAT ---- - if [ "${SOURCE_NAT}" = "true" ] && [ -n "${CIDR}" ]; then + # For VPC tiers the SNAT rule covers the entire VPC CIDR and is set up by + # implement-vpc (using --vpc-cidr). Duplicate SNAT rules here would conflict. + if [ "${SOURCE_NAT}" = "true" ] && [ -n "${CIDR}" ] && [ -z "${VPC_ID}" ]; then ip netns exec "${NAMESPACE}" iptables -t nat \ -C "${nchain_post}" -s "${CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" 2>/dev/null || \ ip netns exec "${NAMESPACE}" iptables -t nat \ @@ -971,6 +973,8 @@ ip netns exec "${NAMESPACE}" iptables -t filter \ -A "${fchain}" -o "${pveth_n}" -s "${CIDR}" -j ACCEPT log "Source NAT: ${CIDR} -> ${PUBLIC_IP} via ${pveth_n}" + elif [ "${SOURCE_NAT}" = "true" ] && [ -n "${VPC_ID}" ]; then + log "assign-ip: skipping SNAT rules for VPC tier (managed by implement-vpc)" fi # ---- Persist state ---- @@ -2231,20 +2235,36 @@ acquire_lock "${NETWORK_ID}" log "save-vm-data: network=${NETWORK_ID} ip=${VM_IP}" [ -z "${VM_IP}" ] && die "save-vm-data: missing --ip" - [ -z "${VM_DATA}" ] && die "save-vm-data: missing --vm-data" + + local vm_data_file="${VM_DATA_FILE}" + local cleanup_vm_data_file="false" + if [ -z "${vm_data_file}" ]; then + [ -z "${VM_DATA}" ] && die "save-vm-data: missing --vm-data or --vm-data-file" + vm_data_file=$(mktemp /tmp/cs-extnet-vm-data-XXXXXX) + cleanup_vm_data_file="true" + printf '%s' "${VM_DATA}" > "${vm_data_file}" + fi + [ -f "${vm_data_file}" ] || die "save-vm-data: payload file not found: ${vm_data_file}" local meta_dir; meta_dir=$(_metadata_dir) local passwd_f; passwd_f=$(_passwd_file) mkdir -p "${meta_dir}" "$(dirname "${passwd_f}")" touch "${passwd_f}" - python3 - "${VM_IP}" "${meta_dir}" "${passwd_f}" "${VM_DATA}" << 'PYEOF' + python3 - "${VM_IP}" "${meta_dir}" "${passwd_f}" "${vm_data_file}" << 'PYEOF' import base64, json, os, sys vm_ip = sys.argv[1] meta_dir = sys.argv[2] passwd_f = sys.argv[3] -data_b64 = sys.argv[4] +data_file = sys.argv[4] + +try: + with open(data_file, 'r', encoding='utf-8') as f: + data_b64 = f.read().strip() +except Exception as e: + print(f"save-vm-data: failed to read vm-data file: {e}", file=sys.stderr) + sys.exit(1) # Decode the outer base64 wrapper, then parse the JSON array try: @@ -2321,6 +2341,10 @@ print(f"save-vm-data: wrote {len(entries)} entries for {vm_ip}") PYEOF + if [ "${cleanup_vm_data_file}" = "true" ]; then + rm -f "${vm_data_file}" 2>/dev/null || true + fi + _write_apache2_conf _svc_start_or_reload_apache2 _svc_start_or_reload_passwd_server @@ -2389,6 +2413,15 @@ acquire_lock "${NETWORK_ID}" log "apply-fw-rules: network=${NETWORK_ID} ns=${NAMESPACE}" + local fw_rules_file="${FW_RULES_FILE}" + local cleanup_fw_rules_file="false" + if [ -z "${fw_rules_file}" ]; then + fw_rules_file=$(mktemp /tmp/cs-extnet-fw-rules-XXXXXX) + cleanup_fw_rules_file="true" + printf '%s' "${FW_RULES_JSON:-}" > "${fw_rules_file}" + fi + [ -f "${fw_rules_file}" ] || die "apply-fw-rules: payload file not found: ${fw_rules_file}" + local veth_n fchain fw_chain veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") fchain=$(filter_chain "${NETWORK_ID}") @@ -2406,15 +2439,22 @@ ip netns exec "${NAMESPACE}" iptables -t filter -N "${fw_chain}" # ---- 4. Build iptables rules via Python ---- - python3 - "${NAMESPACE}" "${FW_RULES_JSON:-}" "${veth_n}" \ + python3 - "${NAMESPACE}" "${fw_rules_file}" "${veth_n}" \ "${fw_chain}" << 'PYEOF' import base64, json, re, subprocess, sys namespace = sys.argv[1] -rules_b64 = sys.argv[2] +rules_file = sys.argv[2] veth_n = sys.argv[3] fw_chain = sys.argv[4] # filter table egress chain (CS_EXTNET_FWRULES_<N>) +try: + with open(rules_file, 'r', encoding='utf-8') as f: + rules_b64 = f.read().strip() +except Exception as e: + print(f"apply-fw-rules: failed to read rules file: {e}", file=sys.stderr) + sys.exit(1) + # Prefix for per-public-IP ingress chains in the mangle table. # e.g. CS_EXTNET_FWI_10.0.56.20 FW_INGRESS_PREFIX = 'CS_EXTNET_FWI_' @@ -2592,6 +2632,11 @@ PYEOF local py_exit=$? + + if [ "${cleanup_fw_rules_file}" = "true" ]; then + rm -f "${fw_rules_file}" 2>/dev/null || true + fi + if [ ${py_exit} -ne 0 ]; then # Python script failed — leave chain empty but continue so that the # fchain catch-all rules remain effective (fail-open for existing traffic). @@ -2768,7 +2813,16 @@ _load_state acquire_lock "${NETWORK_ID}" log "restore-network: network=${NETWORK_ID} ns=${NAMESPACE}" - [ -z "${RESTORE_DATA}" ] && die "restore-network: missing --restore-data" + + local restore_data_file="${RESTORE_DATA_FILE}" + local cleanup_restore_data_file="false" + if [ -z "${restore_data_file}" ]; then + [ -z "${RESTORE_DATA}" ] && die "restore-network: missing --restore-data or --restore-data-file" + restore_data_file=$(mktemp /tmp/cs-extnet-restore-data-XXXXXX) + cleanup_restore_data_file="true" + printf '%s' "${RESTORE_DATA}" > "${restore_data_file}" + fi + [ -f "${restore_data_file}" ] || die "restore-network: payload file not found: ${restore_data_file}" local dhcp_hosts; dhcp_hosts=$(_dnsmasq_dhcp_hosts) local dhcp_opts; dhcp_opts=$(_dnsmasq_dhcp_opts) @@ -2780,19 +2834,26 @@ touch "${dhcp_hosts}" "${dhcp_opts}" "${dns_hosts}" "${passwd_f}" python3 - \ - "${RESTORE_DATA}" \ + "${restore_data_file}" \ "${dhcp_hosts}" "${dhcp_opts}" "${dns_hosts}" \ "${meta_dir}" "${passwd_f}" \ << 'PYEOF' import base64, json, os, sys -restore_b64 = sys.argv[1] +restore_file = sys.argv[1] dhcp_hosts_f = sys.argv[2] dhcp_opts_f = sys.argv[3] dns_hosts_f = sys.argv[4] meta_dir = sys.argv[5] passwd_f = sys.argv[6] +try: + with open(restore_file, 'r', encoding='utf-8') as f: + restore_b64 = f.read().strip() +except Exception as e: + print(f"restore-network: failed to read restore-data file: {e}", file=sys.stderr) + sys.exit(1) + # ---- Decode the outer base64, then parse JSON ---- try: data = json.loads(base64.b64decode(restore_b64).decode('utf-8')) @@ -2938,16 +2999,16 @@ PYEOF # ------------------------------------------------------------------ - # Decode dhcp_enabled / dns_enabled from RESTORE_DATA so we can + # Decode dhcp_enabled / dns_enabled from restore_data_file so we can # reconfigure dnsmasq correctly even when the namespace (and therefore - # dnsmasq.conf) was deleted and recreated. RESTORE_DATA is passed as - # a positional argument to avoid shell-quoting issues with base64 data. + # dnsmasq.conf) was deleted and recreated. # ------------------------------------------------------------------ local _r_flags - _r_flags=$(python3 - "${RESTORE_DATA}" 2>/dev/null << 'PYFLAGSEOF' -import base64, json, sys + _r_flags=$(python3 - "${restore_data_file}" 2>/dev/null << 'PYFLAGSEOF' +import base64, json, sys, pathlib try: - d = json.loads(base64.b64decode(sys.argv[1]).decode('utf-8')) + b64_data = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8').strip() + d = json.loads(base64.b64decode(b64_data).decode('utf-8')) dhcp = 'true' if d.get('dhcp_enabled', False) else 'false' dns = 'true' if d.get('dns_enabled', False) else 'false' print(dhcp + ' ' + dns) @@ -2955,6 +3016,10 @@ print('false false') PYFLAGSEOF ) + + if [ "${cleanup_restore_data_file}" = "true" ]; then + rm -f "${restore_data_file}" 2>/dev/null || true + fi local _r_dhcp _r_dns _r_dhcp="${_r_flags%% *}"; _r_dhcp="${_r_dhcp:-false}" _r_dns="${_r_flags##* }"; _r_dns="${_r_dns:-false}" @@ -2990,6 +3055,344 @@ } ############################################################################## +# Helpers: parse VPC-level args (no --network-id required) +############################################################################## + +parse_vpc_args() { + VPC_ID="" + NAMESPACE="" + VPC_CIDR="" + PUBLIC_IP="" + PUBLIC_VLAN="" + PUBLIC_GATEWAY="" + PUBLIC_CIDR="" + SOURCE_NAT="false" + + while [ $# -gt 0 ]; do + case "$1" in + --vpc-id) VPC_ID="$2"; shift 2 ;; + --namespace) NAMESPACE="$2"; shift 2 ;; + --cidr) VPC_CIDR="$2"; shift 2 ;; + --public-ip) PUBLIC_IP="$2"; shift 2 ;; + --public-vlan) PUBLIC_VLAN="$2"; shift 2 ;; + --public-gateway) PUBLIC_GATEWAY="$2"; shift 2 ;; + --public-cidr) PUBLIC_CIDR="$2"; shift 2 ;; + --source-nat) SOURCE_NAT="$2"; shift 2 ;; + # consumed by _pre_scan_args — skip silently + --physical-network-extension-details|--network-extension-details) + shift 2 ;; + *) shift ;; + esac + done + + [ -z "${VPC_ID}" ] && die "Missing --vpc-id" + + if [ -z "${NAMESPACE}" ]; then + local NS_FROM_DETAILS + NS_FROM_DETAILS=$(_json_get "${EXTENSION_DETAILS}" "namespace") + NAMESPACE="${NS_FROM_DETAILS:-cs-vpc-${VPC_ID}}" + fi + + # Normalise VLANs + if [ -n "${PUBLIC_VLAN}" ]; then + PUBLIC_VLAN=$(normalize_vlan "${PUBLIC_VLAN}") + fi +} + +############################################################################## +# Command: implement-vpc +# Creates the VPC namespace, enables IP forwarding, and optionally sets up +# VPC-level source NAT (--public-ip / --public-vlan / --source-nat true). +# State is persisted to ${STATE_DIR}/vpc-<vpcId>/ +############################################################################## + +cmd_implement_vpc() { + parse_vpc_args "$@" + log "implement-vpc: vpc=${VPC_ID} ns=${NAMESPACE} cidr=${VPC_CIDR}" + + # ---- 1. Create (or ensure) VPC namespace ---- + if ! ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns add "${NAMESPACE}" + log "implement-vpc: created namespace ${NAMESPACE}" + fi + ip netns exec "${NAMESPACE}" ip link set lo up 2>/dev/null || true + + # Disable IPv6 inside the namespace + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.all.disable_ipv6=1 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.default.disable_ipv6=1 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.lo.disable_ipv6=1 >/dev/null 2>&1 || true + + # ---- 2. IP forwarding ---- + ip netns exec "${NAMESPACE}" sysctl -w net.ipv4.ip_forward=1 >/dev/null 2>&1 || true + + # ---- 3. VPC-level source NAT (if source NAT IP provided) ---- + # Creates the public veth pair and SNAT rule for the entire VPC CIDR. + if [ -n "${PUBLIC_IP}" ] && [ -n "${PUBLIC_VLAN}" ] && [ "${SOURCE_NAT}" = "true" ] && [ -n "${VPC_CIDR}" ]; then + local pveth_h pveth_n pub_br + pveth_h=$(pub_veth_host_name "${PUBLIC_VLAN}" "${VPC_ID}") + pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${VPC_ID}") + ensure_host_bridge "${PUB_ETH}" "${PUBLIC_VLAN}" + pub_br=$(host_bridge_name "${PUB_ETH}" "${PUBLIC_VLAN}") + + if ! ip link show "${pveth_h}" >/dev/null 2>&1; then + ip link add "${pveth_h}" type veth peer name "${pveth_n}" + ip link set "${pveth_n}" netns "${NAMESPACE}" + ip link set "${pveth_h}" master "${pub_br}" + ip link set "${pveth_h}" up + ip netns exec "${NAMESPACE}" ip link set "${pveth_n}" up + log "implement-vpc: created public veth ${pveth_h} <-> ${pveth_n}" + else + ip link set "${pveth_h}" up 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set "${pveth_n}" up 2>/dev/null || true + fi + + # Assign public IP + local ADDR_SPEC + if [ -n "${PUBLIC_CIDR}" ] && echo "${PUBLIC_CIDR}" | grep -q '/'; then + local PREFIX + PREFIX=$(echo "${PUBLIC_CIDR}" | cut -d'/' -f2) + ADDR_SPEC="${PUBLIC_IP}/${PREFIX}" + else + ADDR_SPEC="${PUBLIC_IP}/32" + fi + ip netns exec "${NAMESPACE}" ip addr show "${pveth_n}" 2>/dev/null | \ + grep -q "${PUBLIC_IP}/" || \ + ip netns exec "${NAMESPACE}" ip addr add "${ADDR_SPEC}" dev "${pveth_n}" + + # Host route + ip route show | grep -q "^${PUBLIC_IP}" || \ + ip route add "${PUBLIC_IP}/32" dev "${pveth_h}" 2>/dev/null || true + + # Default route inside namespace toward upstream gateway + if [ -n "${PUBLIC_GATEWAY}" ]; then + ip netns exec "${NAMESPACE}" ip route replace default \ + via "${PUBLIC_GATEWAY}" dev "${pveth_n}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" ip route add default \ + via "${PUBLIC_GATEWAY}" dev "${pveth_n}" 2>/dev/null || true + log "implement-vpc: default route via ${PUBLIC_GATEWAY} dev ${pveth_n}" + fi + + # VPC SNAT rule — covers the entire VPC CIDR (all tiers) + # Use a VPC-level POSTROUTING chain: CS_EXTNET_<vpcId>_VPC_POST + local vpc_post_chain="${CHAIN_PREFIX}_${VPC_ID}_VPC_POST" + ensure_chain nat "${vpc_post_chain}" + ensure_jump nat POSTROUTING "${vpc_post_chain}" + + ip netns exec "${NAMESPACE}" iptables -t nat \ + -C "${vpc_post_chain}" -s "${VPC_CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t nat \ + -A "${vpc_post_chain}" -s "${VPC_CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" + log "implement-vpc: VPC SNAT ${VPC_CIDR} -> ${PUBLIC_IP} via ${pveth_n}" + + # Persist public IP state + local vsd="${STATE_DIR}/vpc-${VPC_ID}" + mkdir -p "${vsd}/ips" + echo "true" > "${vsd}/ips/${PUBLIC_IP}" + echo "${PUBLIC_VLAN}" > "${vsd}/ips/${PUBLIC_IP}.pvlan" + fi + + # ---- 4. Persist VPC state ---- + local vsd="${STATE_DIR}/vpc-${VPC_ID}" + mkdir -p "${vsd}" + echo "${NAMESPACE}" > "${vsd}/namespace" + [ -n "${VPC_CIDR}" ] && echo "${VPC_CIDR}" > "${vsd}/cidr" + + log "implement-vpc: done vpc=${VPC_ID} namespace=${NAMESPACE}" +} + +############################################################################## +# Command: shutdown-vpc +# Removes the VPC namespace after all tiers have been shut down. +# Called by shutdownVpc() in NetworkExtensionElement after all tiers are gone. +############################################################################## + +cmd_shutdown_vpc() { + parse_vpc_args "$@" + log "shutdown-vpc: vpc=${VPC_ID} ns=${NAMESPACE}" + + if ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns del "${NAMESPACE}" + log "shutdown-vpc: deleted namespace ${NAMESPACE}" + else + log "shutdown-vpc: namespace ${NAMESPACE} not found (already removed?)" + fi + + log "shutdown-vpc: done vpc=${VPC_ID}" +} + +############################################################################## +# Command: destroy-vpc +# Destroys the VPC namespace and removes all VPC state. +############################################################################## + +cmd_destroy_vpc() { + parse_vpc_args "$@" + log "destroy-vpc: vpc=${VPC_ID} ns=${NAMESPACE}" + + if ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns del "${NAMESPACE}" + log "destroy-vpc: deleted namespace ${NAMESPACE}" + fi + + local vsd="${STATE_DIR}/vpc-${VPC_ID}" + rm -rf "${vsd}" + log "destroy-vpc: removed VPC state dir ${vsd}" + + log "destroy-vpc: done vpc=${VPC_ID}" +} + +############################################################################## +# Command: apply-network-acl +# Applies VPC network ACL rules to the FORWARD chain inside the namespace. +# Rules are passed as a Base64-encoded JSON array via --acl-rules-file or +# --acl-rules. Each rule has: +# number, action (allow|deny), trafficType (ingress|egress), +# protocol, portStart, portEnd, icmpType, icmpCode, sourceCidrs[] +############################################################################## + +cmd_apply_network_acl() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + log "apply-network-acl: network=${NETWORK_ID} ns=${NAMESPACE} cidr=${CIDR}" + + local acl_rules_file="${ACL_RULES_FILE}" + local cleanup_acl_file="false" + if [ -z "${acl_rules_file}" ]; then + acl_rules_file=$(mktemp /tmp/cs-extnet-acl-rules-XXXXXX) + cleanup_acl_file="true" + printf '%s' "${ACL_RULES_JSON:-}" > "${acl_rules_file}" + fi + [ -f "${acl_rules_file}" ] || die "apply-network-acl: payload file not found: ${acl_rules_file}" + + local veth_n acl_chain_name fchain + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + acl_chain_name=$(acl_chain "${NETWORK_ID}") + fchain=$(filter_chain "${NETWORK_ID}") + + # ---- 1. Remove existing jump from fchain to acl chain (idempotent) ---- + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -j "${acl_chain_name}" 2>/dev/null || true + + # ---- 2. Flush and delete old ACL chain ---- + ip netns exec "${NAMESPACE}" iptables -t filter -F "${acl_chain_name}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter -X "${acl_chain_name}" 2>/dev/null || true + + # ---- 3. Create fresh ACL chain ---- + ip netns exec "${NAMESPACE}" iptables -t filter -N "${acl_chain_name}" + + # ---- 4. Build iptables ACL rules via Python ---- + python3 - "${NAMESPACE}" "${acl_rules_file}" "${veth_n}" \ + "${acl_chain_name}" "${CIDR:-}" << 'PYEOF' +import base64, json, subprocess, sys + +namespace = sys.argv[1] +rules_file = sys.argv[2] +veth_n = sys.argv[3] +acl_chain = sys.argv[4] +net_cidr = sys.argv[5] # tier CIDR (may be empty) + +try: + with open(rules_file, 'r', encoding='utf-8') as f: + rules_b64 = f.read().strip() +except Exception as e: + print(f"apply-network-acl: failed to read rules file: {e}", file=sys.stderr) + sys.exit(1) + +def _run(table, *args): + cmd = ['ip', 'netns', 'exec', namespace, 'iptables', '-t', table] + list(args) + r = subprocess.run(cmd, capture_output=True) + if r.returncode != 0: + print(f"iptables ({table}): {r.stderr.decode().strip()}", file=sys.stderr) + return r + +def iptf(*args): + _run('filter', *args) + +if rules_b64: + try: + rules = json.loads(base64.b64decode(rules_b64).decode('utf-8')) + except Exception as e: + print(f"apply-network-acl: failed to decode rules: {e}", file=sys.stderr) + sys.exit(1) +else: + rules = [] + +# Always allow RELATED,ESTABLISHED so active sessions are not dropped. +iptf('-A', acl_chain, '-m', 'state', '--state', 'RELATED,ESTABLISHED', '-j', 'ACCEPT') + +for rule in sorted(rules, key=lambda r: r.get('number', 999)): + direction = rule.get('trafficType', 'ingress').lower() + protocol = (rule.get('protocol') or 'all').lower() + port_start = rule.get('portStart') + port_end = rule.get('portEnd') + icmp_type = rule.get('icmpType') + icmp_code = rule.get('icmpCode') + src_cidrs = rule.get('sourceCidrs') or ['0.0.0.0/0'] + action = 'ACCEPT' if rule.get('action', 'deny').lower() == 'allow' else 'DROP' + + for src_cidr in src_cidrs: + a = [] + if direction == 'ingress': + # Traffic toward VMs (going OUT on guest veth_n into the tier subnet) + a = ['-o', veth_n] + if net_cidr: + a += ['-d', net_cidr] + if src_cidr and src_cidr not in ('0.0.0.0/0', '::/0', ''): + a += ['-s', src_cidr] + else: + # Traffic FROM VMs (coming IN on guest veth_n from the tier subnet) + a = ['-i', veth_n] + if net_cidr: + a += ['-s', net_cidr] + # For egress rules sourceCidrs is used as destination filter + if src_cidr and src_cidr not in ('0.0.0.0/0', '::/0', ''): + a += ['-d', src_cidr] + + if protocol not in ('all', ''): + a += ['-p', protocol] + if protocol in ('tcp', 'udp') and port_start is not None: + port_spec = str(port_start) + if port_end is not None and port_end != port_start: + port_spec = f"{port_start}:{port_end}" + a += ['--dport', port_spec] + elif protocol == 'icmp' and icmp_type is not None and icmp_type != -1: + icmp_spec = str(icmp_type) + if icmp_code is not None and icmp_code != -1: + icmp_spec += f"/{icmp_code}" + a += ['--icmp-type', icmp_spec] + + iptf('-A', acl_chain, *a, '-j', action) + +# Default: DROP all unmatched traffic (implicit deny at end of ACL) +iptf('-A', acl_chain, '-j', 'DROP') + +print(f"apply-network-acl: applied {len(rules)} ACL rule(s) to chain {acl_chain}") +PYEOF + + local py_exit=$? + + if [ "${cleanup_acl_file}" = "true" ]; then + rm -f "${acl_rules_file}" 2>/dev/null || true + fi + + if [ ${py_exit} -ne 0 ]; then + log "apply-network-acl: Python rule builder exited ${py_exit}; ACL chain may be incomplete" + fi + + # ---- 5. Insert jump from fchain to acl chain at position 1 ---- + # ACL rules take precedence over the catch-all ACCEPT rules in fchain. + if ip netns exec "${NAMESPACE}" iptables -t filter -n -L "${fchain}" >/dev/null 2>&1; then + ip netns exec "${NAMESPACE}" iptables -t filter \ + -I "${fchain}" 1 -j "${acl_chain_name}" 2>/dev/null || true + log "apply-network-acl: inserted ACL jump in ${fchain}" + fi + + release_lock + log "apply-network-acl: done network=${NETWORK_ID}" +} + +############################################################################## # Main dispatcher ############################################################################## @@ -2999,9 +3402,13 @@ shift || true case "${COMMAND}" in - implement) cmd_implement "$@" ;; - shutdown) cmd_shutdown "$@" ;; - destroy) cmd_destroy "$@" ;; + implement-network) cmd_implement_network "$@" ;; + shutdown-network) cmd_shutdown_network "$@" ;; + destroy-network) cmd_destroy_network "$@" ;; + # VPC lifecycle + implement-vpc) cmd_implement_vpc "$@" ;; + shutdown-vpc) cmd_shutdown_vpc "$@" ;; + destroy-vpc) cmd_destroy_vpc "$@" ;; assign-ip) cmd_assign_ip "$@" ;; release-ip) cmd_release_ip "$@" ;; add-static-nat) cmd_add_static_nat "$@" ;; @@ -3028,15 +3435,19 @@ # Load balancing (haproxy) apply-fw-rules) cmd_apply_fw_rules "$@" ;; apply-lb-rules) cmd_apply_lb_rules "$@" ;; + # ACL rules (VPC network ACLs) + apply-network-acl) cmd_apply_network_acl "$@" ;; # Custom actions custom-action) cmd_custom_action "$@" ;; "") - echo "Usage: $0 {implement|shutdown|destroy|assign-ip|release-ip|" \ + echo "Usage: $0 {implement-network|shutdown-network|destroy-network|" \ + "implement-vpc|shutdown-vpc|destroy-vpc|" \ + "assign-ip|release-ip|" \ "add-static-nat|delete-static-nat|add-port-forward|delete-port-forward|" \ "config-dhcp-subnet|remove-dhcp-subnet|add-dhcp-entry|remove-dhcp-entry|set-dhcp-options|" \ "config-dns-subnet|remove-dns-subnet|add-dns-entry|remove-dns-entry|" \ "save-userdata|save-password|save-sshkey|save-hypervisor-hostname|save-vm-data|restore-network|" \ - "apply-fw-rules|apply-lb-rules|custom-action} [options]" >&2 + "apply-fw-rules|apply-lb-rules|apply-network-acl|custom-action} [options]" >&2 exit 1 ;; *) echo "Unknown command: ${COMMAND}" >&2
diff --git a/Network-Namespace/network-namespace.sh b/Network-Namespace/network-namespace.sh index 4155f01..090b46f 100755 --- a/Network-Namespace/network-namespace.sh +++ b/Network-Namespace/network-namespace.sh
@@ -152,6 +152,10 @@ NETWORK_ID="" CURRENT_DETAILS="{}" VPC_ID="" +VM_DATA_FILE="" +FW_RULES_FILE="" +RESTORE_DATA_FILE="" +ACL_RULES_FILE="" FORWARD_ARGS=() while [ $# -gt 0 ]; do @@ -173,6 +177,18 @@ --current-details) CURRENT_DETAILS="${2:-{}}" shift 2 ;; + --vm-data-file) + VM_DATA_FILE="${2:-}" + shift 2 ;; + --fw-rules-file) + FW_RULES_FILE="${2:-}" + shift 2 ;; + --acl-rules-file) + ACL_RULES_FILE="${2:-}" + shift 2 ;; + --restore-data-file) + RESTORE_DATA_FILE="${2:-}" + shift 2 ;; *) FORWARD_ARGS+=("$1") shift ;; @@ -280,12 +296,28 @@ fi } +upload_file_to_remote() { + local host="$1" local_file="$2" tag="$3" + [ -f "${local_file}" ] || die "Missing local payload file: ${local_file}" 1 + + local remote_tmp + remote_tmp=$(ssh_exec "${host}" "mktemp /tmp/cs-extnet-${tag}-XXXXXX") || \ + die "Failed to create remote temp file for ${tag}" 2 + remote_tmp=$(printf '%s' "${remote_tmp}" | tr -d '\r\n') + [ -n "${remote_tmp}" ] || die "Failed to resolve remote temp file for ${tag}" 2 + + cat "${local_file}" | ssh_exec "${host}" "cat > '${remote_tmp}' && chmod 600 '${remote_tmp}'" || \ + die "Failed to upload payload file for ${tag}" 2 + + printf '%s' "${remote_tmp}" +} + # --------------------------------------------------------------------------- # ensure-network-device # --------------------------------------------------------------------------- if [ "${COMMAND}" = "ensure-network-device" ]; then - [ -z "${NETWORK_ID}" ] && die "ensure-network-device: missing --network-id" 1 + [ -z "${NETWORK_ID}" ] && [ -z "${VPC_ID}" ] && die "ensure-network-device: missing --network-id or --vpc-id" 1 if [ ${#HOST_LIST[@]} -eq 0 ]; then die "ensure-network-device: no hosts configured. Set 'hosts' in registerExtension details." 1 @@ -311,7 +343,7 @@ h="${h// /}" if [ "${h}" = "${CURRENT_HOST}" ]; then if host_reachable "${CURRENT_HOST}"; then - log "ensure-network-device: network=${NETWORK_ID} keeping current host=${CURRENT_HOST}" + log "ensure-network-device: ${NETWORK_ID:+network=${NETWORK_ID} }${VPC_ID:+vpc=${VPC_ID} }keeping current host=${CURRENT_HOST}" if [ -n "${VPC_ID}" ]; then printf '{"host":"%s","namespace":"%s","vpc_id":"%s"}\n' \ "${CURRENT_HOST}" "${NAMESPACE}" "${VPC_ID}" @@ -350,7 +382,7 @@ _H="${HOST_LIST[$_IDX]// /}" if host_reachable "${_H}"; then _SELECTED_HOST="${_H}" - log "ensure-network-device: network=${NETWORK_ID} hash-selected host=${_SELECTED_HOST} (key=${_ROUTE_KEY}, idx=${_IDX})" + log "ensure-network-device: ${NETWORK_ID:+network=${NETWORK_ID} }${VPC_ID:+vpc=${VPC_ID} }hash-selected host=${_SELECTED_HOST} (key=${_ROUTE_KEY}, idx=${_IDX})" break fi log "ensure-network-device: host ${_H} not reachable, trying next" @@ -386,6 +418,28 @@ remote_args+=("'${arg//"'"/"'\\''"}'" ) done +REMOTE_PAYLOAD_FILES=() +if [ -n "${VM_DATA_FILE}" ]; then + REMOTE_VM_DATA_FILE=$(upload_file_to_remote "${REMOTE_HOST}" "${VM_DATA_FILE}" "vm-data") + REMOTE_PAYLOAD_FILES+=("${REMOTE_VM_DATA_FILE}") + remote_args+=("'--vm-data-file'" "'${REMOTE_VM_DATA_FILE//"'"/"'\\''"}'") +fi +if [ -n "${FW_RULES_FILE}" ]; then + REMOTE_FW_RULES_FILE=$(upload_file_to_remote "${REMOTE_HOST}" "${FW_RULES_FILE}" "fw-rules") + REMOTE_PAYLOAD_FILES+=("${REMOTE_FW_RULES_FILE}") + remote_args+=("'--fw-rules-file'" "'${REMOTE_FW_RULES_FILE//"'"/"'\\''"}'") +fi +if [ -n "${ACL_RULES_FILE}" ]; then + REMOTE_ACL_RULES_FILE=$(upload_file_to_remote "${REMOTE_HOST}" "${ACL_RULES_FILE}" "acl-rules") + REMOTE_PAYLOAD_FILES+=("${REMOTE_ACL_RULES_FILE}") + remote_args+=("'--acl-rules-file'" "'${REMOTE_ACL_RULES_FILE//"'"/"'\\''"}'") +fi +if [ -n "${RESTORE_DATA_FILE}" ]; then + REMOTE_RESTORE_DATA_FILE=$(upload_file_to_remote "${REMOTE_HOST}" "${RESTORE_DATA_FILE}" "restore-data") + REMOTE_PAYLOAD_FILES+=("${REMOTE_RESTORE_DATA_FILE}") + remote_args+=("'--restore-data-file'" "'${REMOTE_RESTORE_DATA_FILE//"'"/"'\\''"}'") +fi + PHYS_ESCAPED="${PHYS_DETAILS//\'/\'\\\'\'}" EXT_ESCAPED="${EXTENSION_DETAILS//\'/\'\\\'\'}" REMOTE_CMD="'${REMOTE_SCRIPT}' '${COMMAND}' ${remote_args[*]} --physical-network-extension-details '${PHYS_ESCAPED}' --network-extension-details '${EXT_ESCAPED}'" @@ -395,6 +449,12 @@ RC=0 ssh_exec "${REMOTE_HOST}" "${REMOTE_CMD}" || RC=$? +if [ ${#REMOTE_PAYLOAD_FILES[@]} -gt 0 ]; then + for _rf in "${REMOTE_PAYLOAD_FILES[@]}"; do + ssh_exec "${REMOTE_HOST}" "rm -f '${_rf}'" >/dev/null 2>&1 || true + done +fi + if [ ${RC} -ne 0 ]; then if [ ${RC} -eq 255 ]; then log "SSH connection failed (rc=255): host=${REMOTE_HOST}:${REMOTE_PORT} user=${REMOTE_USER}"