Proxmox SDN Network Extension for Apache CloudStack

This directory contains the Proxmox SDN NetworkOrchestrator extension — a CloudStack plugin that delegates network lifecycle operations to the Proxmox VE SDN API.

The extension maps CloudStack network concepts (isolated networks, VPCs, VLANs, source NAT, static NAT, port forwarding, firewall rules) onto Proxmox SDN primitives (zones, VNets, subnets) and manages NAT/firewall rules on the Proxmox gateway node via iptables.

This extension works alongside the Proxmox.sh VM orchestrator extension: proxmox-sdn.sh creates VLAN networks (VNets) on Proxmox SDN, while Proxmox.sh creates VMs and attaches NICs to those VNet bridges. VM metadata (userdata, passwords, SSH keys, network config) is delivered via a cloud-init ISO attached to the VM.

The extension is implemented as a shell script executed by NetworkExtensionElement — no separate plugin JAR is required.

Table of Contents

1. Architecture
2. Proxmox SDN Concepts
3. CloudStack ↔ Proxmox SDN Mapping
4. Directory Contents
5. Prerequisites
6. Installation
7. Step-by-step API Setup
   • 1. Create the Extension
   • 2. Register with Physical Network
   • 3. Create Network Offering
   • 4. Create an Isolated Network
   • 5. VPC Setup
   • 6. Cleanup
8. Extension Details Reference
9. Supported Commands
10. Custom Actions
11. Cloud-Init Integration
12. Integration with Proxmox VM Extension
13. Proxmox SDN Zone Types
14. Limitations and Future Work

Architecture

┌──────────────────────────────────────────────────────────────┐
│  CloudStack Management Server                                │
│                                                              │
│  NetworkExtensionElement.java                                │
│      │ executes (path resolved from Extension record)        │
│      ▼                                                       │
│  /usr/share/cloudstack-management/extensions/<ext-name>/     │
│      proxmox-sdn.sh                                          │
│          │                                                   │
│          │ 1. Proxmox REST API (HTTPS :8006)                 │
│          │    → /api2/json/cluster/sdn/zones                 │
│          │    → /api2/json/cluster/sdn/vnets                 │
│          │    → /api2/json/cluster/sdn/vnets/<vnet>/subnets  │
│          │    → PUT /api2/json/cluster/sdn  (apply changes)  │
│          │                                                   │
│          │ 2. SSH to gateway node (for iptables NAT/FW)      │
│          ▼                                                   │
└──────────┬───────────────────────────────────────────────────┘
           │
           ▼
┌──────────────────────────────────────────────────────────────┐
│  Proxmox VE Cluster                                          │
│                                                              │
│  SDN Layer (managed via API)                                 │
│  ┌────────────────────────────────────────────────────┐      │
│  │  Zone: cszone (type: vlan, bridge: vmbr0)          │      │
│  │                                                    │      │
│  │  ┌──────────────────────────────────────────┐      │      │
│  │  │  VNet: cs42 (tag: 100)                   │      │      │
│  │  │    Subnet: 10.0.1.0/24                   │      │      │
│  │  │      Gateway: 10.0.1.1                   │      │      │
│  │  │      SNAT: enabled                       │      │      │
│  │  └──────────────────────────────────────────┘      │      │
│  │                                                    │      │
│  │  ┌──────────────────────────────────────────┐      │      │
│  │  │  VNet: cs55 (tag: 200)                   │      │      │
│  │  │    Subnet: 10.0.2.0/24                   │      │      │
│  │  │      Gateway: 10.0.2.1                   │      │      │
│  │  └──────────────────────────────────────────┘      │      │
│  └────────────────────────────────────────────────────┘      │
│                                                              │
│  NAT/Firewall (iptables on gateway node)                     │
│  ┌────────────────────────────────────────────────────┐      │
│  │  Chain CS_PVE_SDN_42_PR (nat PREROUTING)           │      │
│  │    DNAT rules for static NAT / port forwarding     │      │
│  │                                                    │      │
│  │  Chain CS_PVE_SDN_42_POST (nat POSTROUTING)        │      │
│  │    SNAT rules for source NAT                       │      │
│  │                                                    │      │
│  │  Chain CS_PVE_SDN_FWD_42 (filter FORWARD)          │      │
│  │    Forwarding rules for guest traffic              │      │
│  └────────────────────────────────────────────────────┘      │
└──────────────────────────────────────────────────────────────┘

Key design principles:

• The proxmox-sdn.sh script runs entirely on the management server. Network topology (zones, VNets, subnets) is managed via the Proxmox REST API — no SSH is needed for SDN operations.

• NAT and firewall rules are managed via iptables on a designated gateway node in the Proxmox cluster, accessed via SSH or the Proxmox API execute endpoint.

• Proxmox SDN handles VLAN trunking and DHCP (via dnsmasq on nodes) natively.

• VMs connect to CloudStack networks via Proxmox VNet bridges that are automatically created on all cluster nodes when SDN changes are applied.

Proxmox SDN Concepts

Refer to the Proxmox SDN documentation for full details.

Zones

A Zone defines a transport domain — the method by which VNets are isolated from each other:

VNets

A VNet is a virtual network within a zone. Each VNet:

• Has a unique name (max 8 characters in Proxmox)
• Has a tag (VLAN ID for VLAN zones)
• Creates a Linux bridge on each cluster node (e.g., vnet0)
• VMs connect to VNets by assigning their NICs to the VNet bridge

Subnets

A Subnet within a VNet defines:

• IP address range (CIDR)
• Gateway IP
• SNAT flag (for source NAT on the nodes)
• DHCP range and DNS server
• IPAM integration (PVE, NetBox, phpIPAM)

CloudStack ↔ Proxmox SDN Mapping

Directory Contents

Prerequisites

Proxmox VE Cluster

• Proxmox VE 7.x or 8.x with SDN enabled

  • SDN is available in Proxmox VE 7.0+ (Technology Preview) and fully supported in Proxmox VE 8.1+
  • Ensure libpve-network-perl is installed:

    apt install libpve-network-perl
    

• API Token with SDN permissions:

  # On a Proxmox node, create an API token:
  pveum user token add root@pam cloudstack --privsep 0
  # Or with specific permissions:
  pveum role add CloudStackSDN -privs "SDN.Allocate SDN.Audit SDN.Use Sys.Modify Sys.Audit"
  pveum acl modify / --roles CloudStackSDN --users root@pam --tokens root@pam!cloudstack
  

• SDN enabled in the datacenter configuration:

  # Verify SDN is configured:
  pvesh get /cluster/sdn
  

• Physical bridge configured (e.g., vmbr0) that carries VLAN trunks (for VLAN zones) or has IP connectivity between nodes (for VXLAN/EVPN zones).

CloudStack Management Server

• curl — for Proxmox API calls
• jq — for JSON parsing (recommended; fallback to grep-based parsing)
• bash 4+ — for associative arrays and advanced string handling
• SSH access to the gateway node (for iptables management)

Installation

Management Server

During package installation, proxmox-sdn.sh is deployed to:

/usr/share/cloudstack-management/extensions/<extension-name>/proxmox-sdn.sh

In developer mode the extensions directory defaults to extensions/ relative to the repo root, so extensions/proxmox-sdn/proxmox-sdn.sh is found automatically.

Step-by-step API Setup

1. Create the Extension

cmk createExtension \
    name=proxmox-sdn \
    type=NetworkOrchestrator \
    path=proxmox-sdn \
    "details[0].key=network.services" \
    "details[0].value=SourceNat,StaticNat,PortForwarding,Firewall,Gateway" \
    "details[1].key=network.service.capabilities" \
    "details[1].value={\"SourceNat\":{\"SupportedSourceNatTypes\":\"peraccount\",\"RedundantRouter\":\"false\"},\"Firewall\":{\"TrafficStatistics\":\"per public ip\"}}"

For a full-featured extension including DHCP:

cmk createExtension \
    name=proxmox-sdn-full \
    type=NetworkOrchestrator \
    path=proxmox-sdn \
    "details[0].key=network.services" \
    "details[0].value=SourceNat,StaticNat,PortForwarding,Firewall,Gateway,Dhcp,Dns" \
    "details[1].key=network.service.capabilities" \
    "details[1].value={\"SourceNat\":{\"SupportedSourceNatTypes\":\"peraccount\",\"RedundantRouter\":\"false\"},\"Firewall\":{\"TrafficStatistics\":\"per public ip\"}}"

Verify:

cmk listExtensions name=proxmox-sdn

2. Register with Physical Network

cmk registerExtension \
    id=<extension-uuid> \
    resourcetype=PhysicalNetwork \
    resourceid=<phys-net-uuid>

Set Proxmox connection details:

cmk updateRegisteredExtension \
    extensionid=<extension-uuid> \
    resourcetype=PhysicalNetwork \
    resourceid=<phys-net-uuid> \
    "details[0].key=url"                      "details[0].value=proxmox1.example.com" \
    "details[1].key=user"                     "details[1].value=root@pam" \
    "details[2].key=token"                    "details[2].value=cloudstack" \
    "details[3].key=secret"                   "details[3].value=<api-token-secret>" \
    "details[4].key=node"                     "details[4].value=pve1" \
    "details[5].key=verify_tls_certificate"   "details[5].value=false" \
    "details[6].key=sdn.zone.type"            "details[6].value=vlan" \
    "details[7].key=sdn.zone.bridge"          "details[7].value=vmbr0" \
    "details[8].key=public.bridge"            "details[8].value=vmbr0" \
    "details[9].key=gateway.node"             "details[9].value=pve1"

For SSH-based iptables management on the gateway node, add SSH credentials:

cmk updateRegisteredExtension \
    extensionid=<extension-uuid> \
    resourcetype=PhysicalNetwork \
    resourceid=<phys-net-uuid> \
    "details[0].key=ssh.key"   "details[0].value=$(cat ~/.ssh/id_rsa)" \
    "details[1].key=ssh.user"  "details[1].value=root"

Verify the NSP was created:

cmk listNetworkServiceProviders physicalnetworkid=<phys-net-uuid>
# → "proxmox-sdn" should appear as Enabled

3. Create Network Offering

cmk createNetworkOffering \
    name="Proxmox SDN Isolated" \
    displaytext="Isolated network via Proxmox SDN" \
    guestiptype=Isolated \
    traffictype=GUEST \
    supportedservices="SourceNat,StaticNat,PortForwarding,Firewall,Gateway" \
    "serviceProviderList[0].service=SourceNat"      "serviceProviderList[0].provider=proxmox-sdn" \
    "serviceProviderList[1].service=StaticNat"      "serviceProviderList[1].provider=proxmox-sdn" \
    "serviceProviderList[2].service=PortForwarding" "serviceProviderList[2].provider=proxmox-sdn" \
    "serviceProviderList[3].service=Firewall"       "serviceProviderList[3].provider=proxmox-sdn" \
    "serviceProviderList[4].service=Gateway"        "serviceProviderList[4].provider=proxmox-sdn" \
    "serviceCapabilityList[0].service=SourceNat" \
    "serviceCapabilityList[0].capabilitytype=SupportedSourceNatTypes" \
    "serviceCapabilityList[0].capabilityvalue=peraccount"

cmk updateNetworkOffering id=<offering-uuid> state=Enabled

4. Create an Isolated Network

cmk createNetwork \
    name=my-proxmox-network \
    displaytext="My Proxmox SDN network" \
    networkofferingid=<offering-uuid> \
    zoneid=<zone-uuid>

When a VM is deployed, CloudStack triggers implement-network, which:

1. Creates a VLAN zone cszone (if not already present) in Proxmox SDN
2. Creates a VNet cs<networkId> with the assigned VLAN tag
3. Creates a subnet with gateway and CIDR
4. Applies SDN changes (PUT /cluster/sdn)
5. Sets up iptables NAT chains on the gateway node

5. VPC Setup

For VPC networks, create a VPC offering and VPC:

# Create VPC offering
cmk createVPCOffering \
    name="Proxmox SDN VPC" \
    displaytext="VPC via Proxmox SDN" \
    supportedservices="SourceNat,StaticNat,PortForwarding,Firewall,NetworkACL" \
    "serviceProviderList[0].service=SourceNat"      "serviceProviderList[0].provider=proxmox-sdn" \
    "serviceProviderList[1].service=StaticNat"      "serviceProviderList[1].provider=proxmox-sdn" \
    "serviceProviderList[2].service=PortForwarding" "serviceProviderList[2].provider=proxmox-sdn" \
    "serviceProviderList[3].service=Firewall"       "serviceProviderList[3].provider=proxmox-sdn" \
    "serviceProviderList[4].service=NetworkACL"     "serviceProviderList[4].provider=proxmox-sdn"

# Create VPC
cmk createVPC \
    name=my-vpc \
    displaytext="My Proxmox SDN VPC" \
    vpcofferingid=<vpc-offering-uuid> \
    zoneid=<zone-uuid> \
    cidr=10.0.0.0/16

# Create VPC tier network offering and tiers...

Each VPC gets its own Proxmox SDN VLAN zone. VPC tier networks are VNets within the VPC zone.

6. Cleanup

# Delete network
cmk deleteNetwork id=<network-uuid>

# Unregister and delete extension
cmk unregisterExtension id=<ext-uuid> resourcetype=PhysicalNetwork resourceid=<phys-net-uuid>
cmk deleteExtension id=<ext-uuid>

Extension Details Reference

Physical Network Extension Details

Set via updateRegisteredExtension:

Per-Network Extension Details

Stored in network_details by ensure-network-device, forwarded as --network-extension-details:

Supported Commands

Network Lifecycle

VPC Lifecycle

IP Management

Firewall

DHCP/DNS/Metadata

These commands are handled by Proxmox SDN natively or delegated to cloud-init:

Custom Actions

Example:

cmk addCustomAction extensionid=<ext-uuid> name=dump-config resourcetype=Network
cmk runNetworkCustomAction networkid=<network-uuid> actionid=<action-uuid>

Cloud-Init Integration

VM metadata (userdata, passwords, SSH keys, network configuration) is delivered via a cloud-init ISO (with the cidata volume label) created by the extension and attached to the VM via the Proxmox API.

When CloudStack calls save-vm-data, save-userdata, save-password, or save-sshkey, the extension:

1. Creates a temporary directory with three files:

   • meta-data — instance ID and local hostname
   • network-config — NoCloud v2 format with static IP, gateway, and DNS from the network details
   • user-data — CloudStack userdata, password, and/or SSH keys merged into a #cloud-config document

2. Generates an ISO image using genisoimage (or mkisofs/xorrisofs) with the cidata volume label

3. Uploads the ISO to Proxmox storage (configurable via cloudinit.storage, default: local)

4. Attaches the ISO to the VM as ide2 (CD-ROM drive) via the Proxmox API

The VM's Proxmox VMID is resolved from the --vmid argument, the VM_DATA_FILE JSON, or a local IP→VMID mapping maintained by the extension.

Integration with Proxmox VM Extension

This SDN extension is designed to work alongside the Proxmox.sh VM orchestrator extension (extensions/Proxmox/proxmox.sh).

Workflow:

1. Network creation — CloudStack calls proxmox-sdn.sh implement-network, which creates a VNet (e.g., cs42) in Proxmox SDN with the assigned VLAN tag. The VNet name becomes a Linux bridge on every cluster node.

2. VM creation — CloudStack calls proxmox.sh create, which creates a QEMU VM and attaches NICs to the VNet bridge using the VLAN tag and MAC address from CloudStack.

3. Metadata delivery — CloudStack calls proxmox-sdn.sh save-vm-data (and related commands), which creates a cloud-init ISO with network config, userdata, password, and SSH keys, and attaches it to the VM.

The VNet name returned by ensure-network-device in the extension details JSON ("vnet": "cs42") is the bridge name that the VM orchestrator uses as network_bridge.

Proxmox SDN Zone Types

VLAN Zone (Recommended for isolated networks)

Best for traditional L2 isolation. Requires 802.1Q VLAN trunk on the physical bridge.

details: sdn.zone.type=vlan, sdn.zone.bridge=vmbr0

Each CloudStack network gets a VNet with the VLAN tag assigned by CloudStack. The Proxmox VNet creates a VLAN-aware bridge on each cluster node.

Simple Zone (For testing)

No isolation — all VNets share the same bridge. Useful for testing.

details: sdn.zone.type=simple

QinQ Zone (For service providers)

Stacked VLANs (802.1ad). Each VNet uses a customer VLAN inside a service VLAN.

details: sdn.zone.type=qinq, sdn.zone.bridge=vmbr0, sdn.service.vlan=100

Limitations and Future Work

Current Limitations

1. DHCP: Proxmox SDN provides DHCP via dnsmasq on nodes, but per-MAC static reservations must be configured through Proxmox IPAM (PVE built-in, NetBox, or phpIPAM). The extension currently delegates DHCP to Proxmox SDN's native mechanisms.

2. Load Balancing: Not natively supported by Proxmox SDN. A future version could deploy HAProxy on the gateway node.

3. VNet Name Length: Proxmox limits VNet names to 8 characters. The extension generates short names (e.g., cs42, vt5n42) but this limits the maximum network/VPC IDs that can be represented without collision.

4. Multi-node Gateway: Currently, a single gateway node handles all NAT/firewall rules. A future version could distribute rules across multiple nodes for HA.

5. Cloud-init ISO tools: The management server must have genisoimage, mkisofs, or xorrisofs installed to create cloud-init ISOs.

Future Enhancements

• Proxmox IPAM Integration: Register VM MAC→IP mappings in the Proxmox IPAM database for static DHCP reservations.
• Proxmox Firewall API: Use the Proxmox cluster/node firewall API instead of raw iptables for better integration.
• HA Gateway: Distribute NAT/firewall rules across multiple Proxmox nodes with VRRP or similar.
• IPv6 Support: Extend subnet configuration with IPv6 CIDR and dual-stack support.