FR01: Netris Integration (#1)

* Add Netris module and Add netris provider

* Fix

* Add Netris Provider to the zone creation wizard

* add steps to zone wizard for adding netris controller and public traffic

* cleanup

* Add missing config key

* Add routed mode offerings for Netris VPC (#3)

* Add routed mode offerings for Netris VPC

* update vpc offering name

* generalize the offering creation method for network providers

* log

* remove debug log

* fix failing build

* Add dependency and Netris API client (#4)

* Add dependency and first approach to Netris API client

* Fix authentication and create Netris API client, in progress sites listing

* Fix get sites

* Support for listing VPCs (#5)

* List tenants

* Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack (#7)

* Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack

* Pass default auth scheme for now

* Support adding netris provider to CloudStack and Netris VPC Creation (#6)

* Support adding netris provider to CloudStack

* revert marvin change

* add license and perform session check when provider is added

* add license and remove unused import

* fix build failure - uunused imports

* address comments

* fix provider name

* add Netris network element

* add license

* Add netris management APIs and netris service provider

* add license

* revert change

* remove other network elements from Netris element

* fix api name in doc generator

* remove logs

* move session alive check to CheckHealthCommand exec

* Fix zone creation wizard to configure netris provider

* Upgrade GSON version - from PR 8756

* Add additional parametes to the add Netris provider API

* add netris as a host

* add additional params to the resoponse and update UI

* Rename site to site_name

* Create Netris VPC (#8)

* Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack (#7)

* Delegate API classes creation to the SDK and simply invoke the desired API class through CloudStack

* Pass default auth scheme for now

* Drop for_nsx and for_tungten columns in favour of checking the provider on the ntwserviceprovider map table

* Remove missing setForTungsten occurrence

* Remove forNsx from VPC offerings

* Create Netris VPC

* Fix VPC offerings listing and remove unused dao

* Create VPC fixes

* Upgrade GSON version - from PR 8756

* Fix VPC creation response by using the latest SDK code

* Fix unit test

* Remove unused import

* Fix NSX unit tests after refactoring

* Add Netris key to the VLAN Details table (#10)

* Add Netris key to the VLAN Details table

* update for_<provider> column to be generic

* Fix VPC and add IPAM allocation for the VPC CIDR (#9)

* Fix VPC and add IPAM allocation for the VPC CIDR

* Remove VPC logic

* Use zoneId accountId and domainId on resources creation

* Fix naming

* Fix VR public nic issue

* Fix Netris Public IP for VPC source NAT allocation

* Add Netris VPC Subnets and vNets (#11)

* Add Netris VPC Subnets and vNets

* fix compilation errors

* Add netris subnet

* refactor naming convention to differentiate between VPC tiers and Isolated networks

* revert marvin change

* fix constructor - build failure

* Add support to filter netris offerings, delete netris provider when zone is being deleted

* Fix build

* Fix VPC creation

* Fix vnet creation

* unnecesary log

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix unit tests

* Add support to delete VNets and Subnets (#13)

* Add support to delete VNets and Subnets

* Add support to delete vnet resources

* Add support to delete vnet resources

* extract code to method

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Add missing suffix return (#14)

* Set up Netris Public range on new zone addition (#15)

* Set up Netris Public range on new zone addition

* Add dependency to calculate subnet containing a start and end IP

* Remove unused import

* Move dependency to the netris module

* Rename Netris IP range

* Refactor logic

* Revert "Refactor logic"

This reverts commit 7ec36a81320444c37e7bb914dd895060b663411b.

* Fix setup range after adding Netris Provider

* Fix VXLAN range adding on zone creation

* Pass VXLAN ID during creation of Netris vNets (#16)

* add zone params to accepts management vnet

* Release vxlan associated to the netris broadcast domain type

* handle update network broadcast uri

* Update Subnet purpose for Netris Public Traffic (#17)

* Update Subnet purpose for Netris Public Traffic

* search for existing subnet of common purpose type

* Fix VR Public IP address (#20)

* Fix VR Public IP address

* Do not set the Public IP range on Netris side that is not part of the Netris IP Public Pool

* Leave only systemvms tag for the first element

* Fix NSX compatibility

* Pass network gateway instead of network CIDR for Netris vNet creation (#21)

* Run moodifyvxlan script if broadcast domain type is Netris (#18)

* Add support to create Netris VPC / Network offerings (#22)

* Add support to create Netris VPC / Network offerings

* fix support services for netris provider type

* Phase4 - Add support for Source NAT, Static NAT and Port Forwarding (#19)

* Run moodifyvxlan script if broadcast domain type is Netris

* Add Netris NAT offerings

* Add support to add Source nat rules for Natted offering

* fix api params while creating Netris source NAT rule

* Add support to add and delete source nat rule on netris

* Add support to create /32 NAT subnet

* Add support to add and delete Static NAT rules in Netris (#23)

* Add support to add and delete Static NAT rules in Netris

* fix static nat creation on netris & removal of subnet on deletion of static nat rule

* remove nat subnet after deltion of the static nat rule

* add check to see if subnet already exists and add license header

* Add port forwarding rules as DNAT rules in Netris (#24)

* Add port forwarding rules as DNAT rules in Netris

* Fixes

* Allow removing DNAT rules

* Fixes

* Fix subnet search

* Fix update SNAT only for SNAT rules

* Address comments

* Fix

* Fix netris pom xml

* Fix SNAT rule creation

* Fix IP and port placements (#27)

* Fix IP and port placements

* fix dnat to IP for PF rules

* change dnatport

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>

* List only Netris Public IPs for NAT operations (#26)

* List only Netris Public IPs for NAT operations

* rename getter and change type

* fix failing unit tests

* list all IPs if forProvider is not passed

* fix list public IPs for external providers with additional IP range

* filter provider Ips in a zone with external provider setup

* Prevent acquiring IP that is not from the external provider range

* formating

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Support to pass provider when creating public ip range and create IPAM on Netris (#28)

* UI: support to pass provider when creating public ip range

* prevent adding public ip range for a provider that isnt supported in zone

* Create public range on Netris when created on CloudStack

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Revert UI filtration for public IPs (#29)

* Fix issue with pagination of public addresses listed after filtering for external providers

* Revert UI filteration for public IPs for external network provider enabled zones

* Fix unit tests (#30)

* Add Netris Tag parameter to the Network provider and fix zone creation wizard (#33)

* Add Netris Tag parameter to the Network provider

* remove unused import

* Fix public IP ranges creation on zone creation (#34)

* use single quotes

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>

* Fix SourceAddress for SNAt to VPC cidr (#35)

* Fix VPC/network offering service list for external network providers in Routed mode (#32)

* Fix network offering service list for external network providers in Routed mode

* filter out unsupported services based on network mode

* fix supported services list for vpc offering for external providers in Routed mode

* Add support to add and delete and update static routes on Netris (#37)

* Add support to add static routes in Netris

* support to delete static routes on netris

* add defensive check for nextHop

* Add support to update static routes

* add state

* pass empty list for switched to avoid timeout

* Netris: search static route by name and next hop if exists

---------

Co-authored-by: Wei Zhou <weizhou@apache.org>

* Netris FR1b: Support Remote Access VPN and Site-to-Site VPN in VPC VR (#41)

* Static Routes: support nexthop

* Update api/src/main/java/org/apache/cloudstack/api/command/user/vpc/CreateStaticRouteCmd.java

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* PR#10064 VR: apply iptables rules when add/remove static routes

* PR#10065 UI: fix cannot open 'Edit tags' modal for static routes

* PR#10066 Static Routes: fix check on wrong global configuration

* PR#10067 VR: fix site-2-site VPN if split connections is enabled

* PR#10081 server: do not allocate nic on public network for NSX VPC VR

* PR#10082 UI: create VPC network offering with conserve mode

* PR#10083 VR: allow outgoing traffic from RAS/VPN clients

* PR#10086 server: fix typo removeaccessvpn in VirtualRouterElement

* server: Add check on Public IP for remote access VPN

* Revert "PR#10083 VR: allow outgoing traffic from RAS/VPN clients"

This reverts commit 2f9b9f428947cac91de322fbdf4a980902a1c0a0.

* VPC: fetch same used IP for domain router if VR is not Source NAT

* VR: pass has_public_network to VR and configure RA/S2S VPN left peers

* Revert "PR#10081 server: do not allocate nic on public network for NSX VPC VR"

This reverts commit 809e269ed6b361d9df1fcef6537762c5612863e0.

* VPC: fetch same used IP for domain router if VR is not Source NAT (v2)

* VR: fix /etc/hosts and nameservers in dnsmasq.conf if VPC VR is not guest gateway

prior to this PR
```
root@r-1167-VM:~# cat /etc/hosts
127.0.0.1	localhost
127.0.1.1	r-1167-VM
::1	localhost ip6-localhost ip6-loopback
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.21.1.33	dummy-vpc-vpn-001
172.21.1.1	r-1167-VM data-server

root@r-1167-VM:~# cat /etc/dnsmasq.d/cloud.conf
dhcp-hostsfile=/etc/dhcphosts.txt
listen-address=127.0.0.1,172.21.1.234
dhcp-range=set:interface-eth1-0,172.21.1.234,static
dhcp-option=tag:interface-eth1-0,15,cs2cloud.internal
dhcp-option=tag:interface-eth1-0,6,172.21.1.1,10.0.32.1,8.8.8.8
dhcp-option=tag:interface-eth1-0,3,172.21.1.1
dhcp-option=eth1,26,1500
dhcp-option=tag:interface-eth1-0,1,255.255.255.0
```

the lines should be
```
172.21.1.234  r-1167-VM data-server

dhcp-option=tag:interface-eth1-0,6,10.0.32.1,8.8.8.8
```

* server: Enable static NAT for Domain router if it is not Source NAT

* server: Enable static NAT for Domain router on UI

* server: assign Public IP to VPC VR and enable static nat if VR is not Source NAT

* server: configure dns1 if VR is not Source NAT

* server: remove check on Firewall service when list network service providers

* UI: remove dot from message.enabled.vpn

* systemvm: add default route via first guest gateway if VR does not have public IP/interface

* VR: add fw_dhcpserver for shared network

* VR: pass has_public_network to VR and configure RA/S2S VPN left peers (v2)

* UI: fix request error when create a VPC tier in a non-Netris/NSX env

* systemvm: add default route via first guest gateway (v2)

* VR: configure iptables rules for S2S vpn on first guest interface

* VR: allow FORWARD to guest interfaces if VR is not Public

* VR: configure remote access vpn on first guest interface if not public

* VR: fix error 789 in RA VPN client when both RA and S2S are configured

* server: Apply Static Route for RA/S2S VPN in VPC VR

* VR: do not set mark for Public interface when VR is not really public

* VPN: do not disable static nat if it is used by a RA/S2S VPN

* server: skip check on network conserve mode if disable/enable RA VPN on Router IP

* server: set forRouter to false when release a IP

* VR: diable IP spoofing protection on default guest network

* VR: fix iptables rules only when only S2S vpn is enabled

* UI: show 'VPN Connections' section

* VPC: new methods to configure/reconfigure Static NAT for VPC VR

* API: set Type in ip address response to DomainRouter if it is used by VR

* server: do not allow IP release if it is used by RA or S2S VPN gateway

* VR: check if interface is added

* VR: add default route only when ip is associated to first guest interface

* VR: fix ipsec conf for l2tp and s2s vpn

* server: save placeholder IP for VPC VR to fix the new VR IP when vpc tier is auto-shutdown

* server: get non-placeholder NIC for VPC VR

* VR: wait 15 seconds after starting password server

* server: fix unable to configure static nat due to 'invalid virtual machine id'

* UI: fix link of router in info card

* VPC: apply static route for VPC VPN if needed (refactoring)

* server: fix VR IP of first VPC tier is the VM gateway

* server: update or remove all existing static routes when shutdown a network

* server: update ipaddress after disabling static nat to fix vpc deletion issue

* servr: disable remote access VPN as part of VPC dstroy

* server: apply static routes when implement a vpc tier

* server: apply static routes even if next hop is null

* server: fix Cannot invoke "com.cloud.vm.NicProfile.getRequestedIPv4()" because "requested" is null

* Netris: Update Vpn provider to VpcVirtualRouter

* Netris: Add Vpn service to network offerings and networks

* server: fix CIDR of VPN ip range

* server: set isVrGuestGateway by SoureNat/Gateway service with Provider.VPCVirtualRouter

* VR: password server takes 10-15 seconds to start if VR IP is not configured in /etc/hosts

* Netris: add back routesPutBody.setStateStatus

* engine/schema: remove SQL changes in schema-41910to42000.sql

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* Add support for Gateway service for Netris VPC and network offerings (#39)

* Add support for Gateway service for Netris VPC and network offerings

* Restore UserData service

* add gateway only to vpc service

* Add support for gateway service for external network providers for networks in routed mode

* add support for gateway svc

* Revert "add support for gateway svc"

This reverts commit 06645cd1c6d08a81ede5d1431497ea3f2efdc5dc.

* Fix VPC offering creation

* Fix VR public NIC after Gateway service is set to Netris

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Netris VPN: Fix s2s vpn status update and isolated network implementation (#42)

* server: fix NPE when deploy vm on isolated network

* vpn: fix s2s vpn status is not updated

Prior to this fix
```
java.lang.IllegalArgumentException: Class com.cloud.agent.api.CheckS2SVpnConnectionsAnswer declares multiple JSON fields named 'details'; conflict is caused by fields com.cloud.agent.api.CheckS2SVpnConnectionsAnswer#details and com.cloud.agent.api.Answer#details
	at com.cloud.agent.transport.ResponseTest.testCheckS2SVpnConnectionsAnswer(ResponseTest.java:42)
```

* test: fix test_01_vpn_usage as now it is only possible to create VPN on Source NAT if it uses VR

* VR: fix unable to create remote access VPN on regular isolated network

the error is
```
  File "/opt/cloud/bin/configure.py", line 1242, in process
    self.remoteaccessvpn_iptables(self.dbag['public_interface'], public_ip, self.dbag[public_ip])
                                  ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
KeyError: 'public_interface'
```

* Release NAT IP subnet when VPC is removed or IP is released (#44)

* Release NAT IP subnet when VPC is removed or IP is released

* add license

* Add support to add IPv6 Public IP range as IPAM Allocation / Subnet on Netris (#36)

* Add support to add IPv6 Public IP range as IPAM Allocation / Subnet on Netris

* Add ipam alloc and subnet for the ipv6 subnet associated to the vpc tier network

* remove commented code

* Phase5 - Support for ACLs in Netris (#31)

* Add support for Netris ACLs

* acl support

* Make acl api call to netris to create the rule

* refactor add acl rule to populate the right fields

* support icmp type acl rule

* acl rule creation - move netrisnetworkRule

* Update ACL naming on Netris

* Add support for Deletion of netris acls

* Add support to delete and re-order ACL rules

* support creation of default acl rules and replacing acl rules

* fix NSXNetworkRule

* Add global routing flag on subnet creation (#45)

* Support change snat ip (#46)

* Support updating VPC Source NAT IP

* Optimize code

* Update source NAT IP

* Fix naming convention for NAT subnets to follow other resources (#47)

* Fix naming convention for NAT subnets to follow other resources

* Use vpc ID for nat subnets

* Use new nat subnet name for deletion of static nat rule

* fix naming convevntion for nat subnet

* Keep Vpn service to default VPC offering with Natted mode only (#50)

* Add Vpn service to default VPC offering with Routed mode

* Revert change on VPC offering and fix VPN service only for Netris NAT mode

* Validate if given CIDR belongs to a bigger allocation in Netris before creating the zone-level allocation (#48)

* Validate if given CIDR belongs to a bigger allocation in Netris before creating

* rename method

* Phase5 - Support for LB - create, delete and Update operations (#49)

* Add support for Netris ACLs

* acl support

* Make acl api call to netris to create the rule

* refactor add acl rule to populate the right fields

* support icmp type acl rule

* acl rule creation - move netrisnetworkRule

* Update ACL naming on Netris

* Add support for Deletion of netris acls

* Add support to delete and re-order ACL rules

* support creation of default acl rules and replacing acl rules

* fix NSXNetworkRule

* Fix naming convention for NAT subnets to follow other resources

* Use vpc ID for nat subnets

* Phase5 - Support for LB - create, delete and Update operations

* Use new nat subnet name for deletion of static nat rule

* add support to add netris lb rule

* support deletion of LB rule on Netris

* add checks when editing unsupported fields of LB rule for Netris and hide columns on the UI

* fix test failure

* fix imports

* add license

* address comments

* Enable Autoscaling on Netris for CPU and memory (#51)

* Enable Autoscaling on Netris for CPU and memory

* Fix monitor autoscale group and cleanup

* Rename autoscaling group method

* Integrate Autoscaling by allowing to update LB rules

* Refactor according to the SDK changes

* Fix the test failures noticed on #44 (#52)

* Increase code coverage (#54)

* Increase code coverage

* More unit tests

* Remove credentials and mock api client

* NetrisResource tests

* Fix unit test

* Add support to add and remove ACL rules when CIDR list is passed when creating LB rules (#53)

* Add support to add and remove ACL rules when CIDR list is passed when creating LB rules

* add deny all rule

* delete the deny rule as well

* Fix build (#57)

* Prevent Index Out of Bounds exception when naming IPAM subnets (#58)

* Prevent Index Out of Bounds exception when naming IPAM subnets

* fix linter

* Delete netris IPv6 subnet (#59)

* Netris VPN: add static route when update a non-existent static route (#60)

* Fix VPC tier creation failure - prevent creating IPv6 IPAM allocation if it already exists (#61)

* Update netris VPC and tier name (#56)

* Update netris VPC and tier name

* add support to update vpc tier name

* add license

* support editing names of dual stack VPCs

* VR/server: configure default gateway and RA/S2S VPN on the IP/interface with minimum network_id (#43)

* server: fix NPE when deploy vm on isolated network

* vpn: fix s2s vpn status is not updated

Prior to this fix
```
java.lang.IllegalArgumentException: Class com.cloud.agent.api.CheckS2SVpnConnectionsAnswer declares multiple JSON fields named 'details'; conflict is caused by fields com.cloud.agent.api.CheckS2SVpnConnectionsAnswer#details and com.cloud.agent.api.Answer#details
	at com.cloud.agent.transport.ResponseTest.testCheckS2SVpnConnectionsAnswer(ResponseTest.java:42)
```

* test: fix test_01_vpn_usage as now it is only possible to create VPN on Source NAT if it uses VR

* VR: fix unable to create remote access VPN on regular isolated network

the error is
```
  File "/opt/cloud/bin/configure.py", line 1242, in process
    self.remoteaccessvpn_iptables(self.dbag['public_interface'], public_ip, self.dbag[public_ip])
                                  ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
KeyError: 'public_interface'
```

* VR/server: configure default gateway and RA/S2S VPN on the IP/interface with minimum network_id

* Don't add deny rule if no CIDR list is passed (#62)

* Hide the Stickiness Configure button for Netris Load Balancers (#72)

* Update IPAM subnet purpose to nat before NAT operations if its different (#71)

* Netris VPN: create vpc gateway with specified IP (#63)

* Netris: fix UnsupportedOperationException when create VPC offering with NATTED mode (#75)

fixes
```
2025-03-21T10:42:55,039 ERROR [c.c.a.ApiServer] (qtp1513608173-21:[ctx-f9c7f002, ctx-bcfe846d]) (logid:e12e798f) unhandled exception executing api command: [Ljava.lang.String;@3a1416cd java.lang.UnsupportedOperationException
        at java.base/java.util.AbstractList.add(AbstractList.java:153)
        at java.base/java.util.AbstractList.add(AbstractList.java:111)
        at org.apache.cloudstack.api.command.admin.vpc.CreateVPCOfferingCmd.getServiceProviderMapForExternalProvider(CreateVPCOfferingCmd.java:248)
```

* [UI] Zone wizard creation improvements - rename hostname to url and remove port for Netris Provider (#77)

* [UI] Zone wizard creation improvements - rename hostname to url and remove port for Netris Provider

* Fix schema column for url instead of hostname

* Fix Static NAT rules naming (#83)

* Netris: create VPN gateway with specified public IP on UI (#82)

* Netris vpn: apply static routes when start or delete a VPN connection (#85)

* Netris VPN: apply static routes when start S2S VPN

* Netris: list static routes and revoke the routes which are not needed

* Netris: use route name (x.x.x.0/x) instead of prefix (x.x.x.0) and get clean cidr list

* Netris VPN: fix NPE when list static routes

* Update plugins/network-elements/netris/src/main/java/org/apache/cloudstack/service/NetrisApiClientImpl.java

* Delete IPv6 allocation after tier removal on VPC with dual stack offering (#86)

* Netris pass v6 gateway (#87)

* pass v6 gateway to netris

* pass v6 gateway to netris

* refactor to address comments

* remove imports

* [VR] Fix IPv6 NIC IP on the VR (#89)

* Add support to edit ACL rules (#74)

* Add support to edit ACL rules

* add support to update acl rules

* remove test file

* VR: advertise SLAAC prefix only if VR is gateway (#91)

* Make reorder ACL items invoke Netris controller (#90)

* VR: fix radvd misconfiguration for non-netris env (#92)

* [VR] Fix object comparisson to string comparisson on python (#93)

* Fix unit tests for ACL (#94)

* Use the previously assigned vNet for Netris Network when it transitions to Implemented state after gc (#88)

* Use the previously assigned vNet for Netris Network when it transitions to Implemented state after gc

* Fix unit tests

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>