Table of Contents
Hairpinning of floating IPs in flat/VLAN provider networks¶
https://git.opendaylight.org/gerrit/#/q/topic:hairpinning
This feature enables VM instances connected to the same router to communicate with each other using their floating ip addresses directly without traversing via the external gateway.
Problem description¶
Local and East/West communication between VMs using floating ips for flat/VLAN provider types is not handled internally by the pipeline currently. As a result, this type of traffic is mistakenly classified as North/South and routed to the external network gateway.
Today, SNATted traffic to flat/VLAN network is routed directly to the external gateway after traversing the SNAT/outbound NAPT pipeline using OF group per external network subnet. The group itself sets the destination mac as the mac address of the external gw associated with the floating ip/ router gw and output to the provider network port via the egress table. This workflow would be changed to align with the VxLAN provider type and direct SNATted traffic back to the FIB where the destination can then resolved to be floating ip on local or remote compute node.
Proposed change¶
The vpn-id used for classification of floating ips and router gateway external addresses in flat/VLAN provider networks is based on the external network id. It will be changed to reflect the subnet id associated with the floating ip/router gateway. This will allow traffic from the SNAT/outbound NAPT table to be resubmitted back to the FIB while preserving the subnet id.
Each floating ip already has VRF entry in the fib table. The vpn-id of this entry will also be based on the subnet id of the floating ip instead of the external network id. If the VM associated with the floating ip is located on remote compute node, the traffic will be routed to the remote compute based on the provider network of the subnet from which the floating ip was allocated e.g. if the private network is VxLAN and the external network is VLAN provider, traffic to floating ip on remote compute node will be routed to the provider port associated with the VLAN provider and not the tunnel associated with the VxLAN provider.
In the FIB table of the egress node, the destination mac will be replaced with the mac address of the floating ip in case of routing to remote compute node. This will allow traffic from flat/VLAN provider enter the L3 pipeline for DNAT of the floating ip.
Default flow will be added to the FIB table for each external subnet-id. If no floating ip match was found in the FIB table for the subnet id, the traffic will be sent to the group of the external subnet. Each group entry will perform the following: (a) replace the destination mac address to the external gateway mac address (b) send the traffic to the provider network via the egress table.
Ingress traffic from flat/VLAN provider network is bounded to L3VPN service using vpn-id of the external network id. To allow traffic classification based on subnet id for floating ips and router gateway ips, the GW MAC table will replace the vpn-id of the external network with the vpn-id of the subnet id of the floating ip. For ingress traffic to router gateway mac, the vpn-id of the correct subnet will be deterined at the FIB table based on the router gateway fixed ip.
A new model will be introduced to contain the new vpn/subnet associations -
odl-nat:subnets-networks
. This model will be filled only for external flat/VLAN provider networks and will take precedence overodl-nat:external-networks
model for selection of vpn-id. BGPVPN use cases won’t be affected by these changes as this model will not be applicable for these scenarios.
Pipeline changes¶
Egress traffic from VM with floating IP to the internet¶
For Pre SNAT, SNAT, FIB tables the vpn-id will be based on the subnet-id of the floating ip
Packets from SNAT table resubmitted back to the FIB rather than straight to the external network subnet-id group. In the FIB table it should be matched against a new flow with lower priority than any other flow containing dst-ip match. Traffic will be redirected based on the vpn-id of the floating ip subnet to the external network subnet-id group.
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=router-id
=>GW Mac table (19)match: vpn-id=router-id,dst-mac=router-interface-mac
=>FIB table (21)match: vpn-id=router-id
=>Pre SNAT table (26)match: vpn-id=router-id,src-ip=vm-ip set vpn-id=fip-subnet-id,src-ip=fip
=>SNAT table (28)match: vpn-id=fip-subnet-id,src-ip=fip set src-mac=fip-mac
=>FIB table (21)match: vpn-id=fip-subnet-id
=>Subnet-id group:set dst-mac=ext-subnet-gw-mac, reg6=provider-lport-tag
=>Egress table (220) output to provider network
Ingress traffic from the internet to VM with floating IP¶
For GW MAC, FIB table the vpn-id will be based on the subnet-id of the floating ip
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=ext-net-id
=>GW Mac table (19)match: vpn-id=ext-net-id,dst-mac=floating-ip-mac set vpn-id=fip-subnet-id
=>FIB table (21)match: vpn-id=fip-subnet-id,dst-ip=fip
=>Pre DNAT table (25)match: dst-ip=fip set vpn-id=router-id,dst-ip=vm-ip
=>DNAT table (27)match: vpn-id=router-id,dst-ip=vm-ip
=>FIB table (21)match: vpn-id=router-id,dst-ip=vm-ip
=>Local Next-Hop group:set dst-mac=vm-mac, reg6=vm-lport-tag
=>Egress table (220) output to VM port
Egress traffic from VM with no associated floating IP to the internet - NAPT switch¶
For Outbound NAPT, NAPT PFIB and FIB tables the vpn-id will be based on the subnet-id of the router gateway
Packets from NAPT PFIB table resubmitted back to the FIB rather than straight to the external network subnet-id group. In the FIB table it should be matched against a new flow with lower priority than any other flow containing dst-ip match. Traffic will be redirected based on the vpn-id of the router gateway subnet to the external network subnet-id group.
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=router-id
=>GW Mac table (19)match: vpn-id=router-id,dst-mac=router-interface-mac
=>FIB table (21)match: vpn-id=router-id
=>Pre SNAT table (26)match: vpn-id=router-id
=>Outbound NAPT table (46)match: src-ip=vm-ip,port=int-port set src-ip=router-gw-ip,vpn-id=router-gw-subnet-id,port=ext-port
=>NAPT PFIB table (47)match: vpn-id=router-gw-subnet-id
=>FIB table (21)match: vpn-id=router-gw-subnet-id
=>Subnet-id group:set dst-mac=ext-subnet-gw-mac, reg6=provider-lport-tag
=>Egress table (220) output to provider network
Ingress traffic from the internet to VM with no associated floating IP - NAPT switch¶
For FIB table the vpn-id will be based on the subnet-id of the router gateway
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=ext-net-id
=>GW Mac table (19)match vpn-id=ext-net-id,dst-mac=router-gw mac
=>FIB table (21)match: vpn-id=ext-net-id,dst-ip=router-gw set vpn-id=router-gw-subnet-id
=>Inbound NAPT table (44)match: dst-ip=router-gw,port=ext-port set dst-ip=vm-ip,vpn-id=router-id,port=int-port
=>PFIB table (47)match: vpn-id=router-id
=>FIB table (21)match: vpn-id=router-id,dst-ip=vm-ip
=>Local Next-Hop group:set dst-mac=vm-mac,reg6=vm-lport-tag
=>Egress table (220) output to VM port
Hairpinning - VM traffic to floating ip on the same compute node¶
For Pre SNAT, SNAT, FIB tables the vpn-id will be based on the subnet-id of the floating ips
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=router-id
=>GW Mac table (19)match: vpn-id=router-id,dst-mac=router-interface-mac
=>FIB table (21)match: vpn-id=router-id
=>Pre SNAT table (26)match: vpn-id=router-id,src-ip=src-vm-ip set vpn-id=fip-subnet-id,src-ip=src-fip
=>SNAT table (28)match: vpn-id=fip-subnet-id,src-ip=src-fip set src-mac=src-fip-mac
=>FIB table (21)match: vpn-id=fip-subnet-id,dst-ip=dst-fip
=>Pre DNAT table (25)match: dst-ip=dst-fip set vpn-id=router-id,dst-ip=dst-vm-ip
=>DNAT table (27)match: vpn-id=router-id,dst-ip=dst-vm-ip
=>FIB table (21)match: vpn-id=router-id,dst-ip=dst-vm-ip
=>Local Next-Hop group:set dst-mac=dst-vm-mac,reg6=dst-vm-lport-tag
=>Egress table (220) output to VM port
Hairpinning - VM traffic to floating ip on remote compute node¶
VM originating the traffic (Ingress DPN):¶
For Pre SNAT, SNAT, FIB tables the vpn-id will be based on the subnet-id of the floating ip
The destination mac is updated by the FIB table to be the floating ip mac. Traffic is sent to the egress DPN over the port of the flat/VLAN provider network.
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=router-id
=>GW Mac table (19)match: vpn-id=router-id,dst-mac=router-interface-mac
=>FIB table (21)match: vpn-id=router-id
=>Pre SNAT table (26)match: vpn-id=router-id,src-ip=src-vm-ip set vpn-id=fip-subnet-id,src-ip=src-fip
=>SNAT table (28)match: vpn-id=fip-subnet-id,src-ip=src-fip set src-mac=src-fip-mac
=>FIB table (21)match: vpn-id=fip-subnet-id,dst-ip=dst-fip set dst-mac=dst-fip-mac, reg6=provider-lport-tag
=>Egress table (220) output to provider network
VM receiving the traffic (Egress DPN):¶
For GW MAC, FIB table the vpn-id will be based on the subnet-id of the floating ip
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=ext-net-id
=>GW Mac table (19)match: vpn-id=ext-net-id,dst-mac=dst-fip-mac set vpn-id=fip-subnet-id
=>FIB table (21)match: vpn-id=fip-subnet-id,dst-ip=dst-fip
=>Pre DNAT table (25)match: dst-ip=dst-fip set vpn-id=router-id,dst-ip=dst-vm-ip
=>DNAT table (27)match: vpn-id=router-id,dst-ip=dst-vm-ip
=>FIB table (21)match: vpn-id=router-id,dst-ip=dst-vm-ip
=>Local Next-Hop group:set dst-mac=dst-vm-mac,lport-tag=dst-vm-lport-tag
=>Egress table (220) output to VM port
Hairpinning - traffic from VM with no associated floating IP to floating ip on remote compute node¶
VM originating the traffic (Ingress DPN) is non-NAPT switch:¶
No flow changes required. Traffic will be directed to NAPT switch and directed to the outbound NAPT table straight from the internal tunnel table
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=router-id
=>GW Mac table (19)match: vpn-id=router-id,dst-mac=router-interface-mac
=>FIB table (21)match: vpn-id=router-id
=>Pre SNAT table (26)match: vpn-id=router-id
=>NAPT Groupoutput to tunnel port of NAPT switch
=>
VM originating the traffic (Ingress DPN) is the NAPT switch:¶
For Outbound NAPT, NAPT PFIB, Pre DNAT, DNAT and FIB tables the vpn-id will be based on the common subnet-id of the router gateway and the floating-ip.
Packets from NAPT PFIB table resubmitted back to the FIB where they will be matched against the destnation floating ip.
The destination mac is updated by the FIB table to be the floating ip mac. Traffic is sent to the egress DPN over the port of the flat/VLAN provider network.
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=router-id
=>GW Mac table (19)match: vpn-id=router-id,dst-mac=router-interface-mac
=>FIB table (21)match: vpn-id=router-id
=>Pre SNAT table (26)match: vpn-id=router-id
=>Outbound NAPT table (46)match: src-ip=vm-ip,port=int-port set src-ip=router-gw-ip,vpn-id=router-gw-subnet-id,port=ext-port
=>NAPT PFIB table (47)match: vpn-id=router-gw-subnet-id
=>FIB table (21)match: vpn-id=router-gw-subnet-id dst-ip=dst-fip set dst-mac=dst-fip-mac, reg6=provider-lport-tag
=>Egress table (220) output to provider network
VM receiving the traffic (Egress DPN):¶
For GW MAC, FIB table the vpn-id will be based on the subnet-id of the floating ip
Classifier table (0) =>Dispatcher table (17)l3vpn service: set vpn-id=ext-net-id
=>GW Mac table (19)match: vpn-id=ext-net-id,dst-mac=dst-fip-mac set vpn-id=fip-subnet-id
=>FIB table (21)match: vpn-id=fip-subnet-id,dst-ip=dst-fip
=>Pre DNAT table (25)match: dst-ip=dst-fip set vpn-id=router-id,dst-ip=dst-vm-ip
=>DNAT table (27)match: vpn-id=router-id,dst-ip=dst-vm-ip
=>FIB table (21)match: vpn-id=router-id,dst-ip=dst-vm-ip
=>Local Next-Hop group:set dst-mac=dst-vm-mac,lport-tag=dst-vm-lport-tag
=>Egress table (220) output to VM port
Yang changes¶
odl-nat module will be enhanced with the following container
container external-subnets {
list subnets {
key id;
leaf id {
type yang:uuid;
}
leaf vpnid {
type yang:uuid;
}
leaf-list router-ids {
type yang:uuid;
}
leaf external-network-id {
type yang:uuid;
}
}
}
This model will be filled out only for flat/VLAN external network provider types.
If this model is missing, vpn-id will be taken from odl-nat:external-networks
model
to maintain compatibility with BGPVPN models.
odl-nat:ext-routers
container will be enhanced with the list of the external subnet-ids
associated with the router.
container ext-routers {
list routers {
key router-name;
leaf router-name {
type string;
}
...
leaf-list external-subnet-id {
type yang:uuid; }
}
}
}
Targeted Release¶
Carbon
Alternatives¶
None
Usage¶
Create external network with two subnets¶
neutron net-create public-net -- --router:external --is-default --provider:network_type=flat
--provider:physical_network=physnet1
neutron subnet-create --ip_version 4 --gateway 10.64.0.1 --name public-subnet1 <public-net-uuid> 10.64.0.0/16
-- --enable_dhcp=False
neutron subnet-create --ip_version 4 --gateway 10.65.0.1 --name public-subnet2 <public-net-uuid> 10.65.0.0/16
-- --enable_dhcp=False
Create internal networks with subnets¶
neutron net-create private-net1
neutron subnet-create --ip_version 4 --gateway 10.0.123.1 --name private-subnet1 <private-net1-uuid>
10.0.123.0/24
neutron net-create private-net2
neutron subnet-create --ip_version 4 --gateway 10.0.124.1 --name private-subnet2 <private-net2-uuid>
10.0.124.0/24
neutron net-create private-net3
neutron subnet-create --ip_version 4 --gateway 10.0.125.1 --name private-subnet3 <private-net3-uuid>
10.0.125.0/24
neutron net-create private-net4
neutron subnet-create --ip_version 4 --gateway 10.0.126.1 --name private-subnet4 <private-net4-uuid>
10.0.126.0/24
Create two router instances and connect each router to one internal subnet and one external subnet¶
neutron router-create router1
neutron router-interface-add <router1-uuid> <private-subnet1-uuid>
neutron router-gateway-set --fixed-ip subnet_id=<public-subnet1-uuid> <router1-uuid> <public-net-uuid>
neutron router-create router2
neutron router-interface-add <router2-uuid> <private-subnet2-uuid>
neutron router-gateway-set --fixed-ip subnet_id=<public-subnet2-uuid> <router2-uuid> <public-net-uuid>
Create router instance connected to both external subnets and the remaining internal subnets¶
neutron router-create router3
neutron router-interface-add <router3-uuid> <private-subnet3-uuid>
neutron router-interface-add <router3-uuid> <private-subnet4-uuid>
neutron router-gateway-set --fixed-ip subnet_id=<public-subnet1-uuid> --fixed-ip subnet_id=<public-subnet2-uuid>
<router3-uuid> <public-net-uuid>
Create floating ips from both subnets¶
neutron floatingip-create --subnet <public-subnet1-uuid> public-net
neutron floatingip-create --subnet <public-subnet1-uuid> public-net
neutron floatingip-create --subnet <public-subnet2-uuid> public-net
Create 2 VM instance in each subnet and associate with floating ips¶
nova boot --image <image-id> --flavor <flavor-id> --nic net-id=<private-net1-uuid> VM1
nova floating-ip-associate VM1 <fip1-public-subnet1>
nova boot --image <image-id> --flavor <flavor-id> --nic net-id=<private-net1-uuid> VM2
nova floating-ip-associate VM2 <fip2-public-subnet1>
nova boot --image <image-id> --flavor <flavor-id> --nic net-id=<private-net2-uuid> VM3
nova floating-ip-associate VM3 <fip1-public-subnet2>
nova boot --image <image-id> --flavor <flavor-id> --nic net-id=<private-net2-uuid> VM4
nova boot --image <image-id> --flavor <flavor-id> --nic net-id=<private-net3-uuid> VM5
nova boot --image <image-id> --flavor <flavor-id> --nic net-id=<private-net4-uuid> VM6
Connectivity tests¶
Connect to the internet from all VMs.
VM1
andVM2
will route traffic through external gateway 10.64.0.1VM3
andVM4
route traffic through external gateway 10.65.0.1.Connect to the internet from
VM5
andVM6
. Each connection will be routed to different external gateway with the corresponding subnet router-gateway ip.Hairpinning when source VM is associated with floating ip - ping between
VM1
andVM2
using their floating ips.Hairpinning when source VM is not associated with floating ip - ping from
VM4
toVM3
using floating ip. SinceVM4
has no associated floating ip a NAPT entry will be allocated using the router-gateway ip.
Features to Install¶
odl-netvirt-openstack
Implementation¶
Assignee(s)¶
- Primary assignee:
Yair Zinger <yair.zinger@hpe.com>
- Other contributors:
Tali Ben-Meir <tali@hpe.com>
Work Items¶
https://trello.com/c/uDcQw95v/104-pipeline-changes-fip-w-multiple-subnets-in-ext-net-hairpinning
Add external-subnets model
Add vpn-instances for external flat/VLAN sunbets
Change pipeline to prefer vpn-id from external-subnets over vpn-id from external-networks
Add write metadata to GW MAC table for floating ip/router gw mac addresses
Add default subnet-id match in FIB table to external subnet group entry
- Changes in remote next-hop flow for floating ip in FIB table
Set destination mac to floating ip mac
Set egress actions to provider port of the network attached to the floating ip subnet
Resubmit SNAT + Outbound NAPT flows to FIB table
Dependencies¶
None