.. contents:: Table of Contents :depth: 3 =================================================================== Discovery of directly connected PNFs in Flat/VLAN provider networks =================================================================== https://git.opendaylight.org/gerrit/#/q/topic:directly_connected_pnf_discovery This features enables discovering and directing traffic to Physical Network Functions (PNFs) in Flat/VLAN provider and tenant networks, by leveraging Subnet-Route feature. Problem description =================== PNF is a device which has not been created by Openstack but connected to the hypervisors L2 broadcast domain and configured with ip from one of the neutron subnets. Ideally, L2/L3 communication between VM instances and PNFs on flat/VLAN networks would be routed similarly to inter-VM communication. However, there are two main issues preventing direct communication to PNFs. * L3 connectivity of tenant network and VLAN provider network, between VMs and PNFs. A VM is located in a tenant network, A PNF is located in a provider network (external network). Both networks are connected via a router. The only way for VMs to communicate with a PNF is via additional hop which is the external gateway, instead of directly. * L3 connectivity between VMs and PNFs in a two diffrent tenant networks, connected by a router, which is not supported and have two problems. First, traffic initiated from a VMs towards a PNF is dropped because there isn't an appropriate rule in FIB table (table 21) to route that traffic. Second, in the other direction, PNFs are not able to resolve their default gateway. We want to leverage the Subnet-Route and Aliveness-Monitor features in order to address the above issues. Subnet-Route ------------ Today, Subnet-Route feature enables ODL to route traffic to a destination IP address, even for ip addresses that have not been statically configured by OpenStack, in the FIB table. To achieve that, the FIB table contains a flow that match all IP packets in a given subnet range. How that works? * A flow is installed in the FIB table, matching on subnet prefix and vpn-id of the network, with a goto-instruction directing packets to table 22. There, packets are punted to the controller. * ODL hold the packets, and initiate an ARP request towards the destination IP. * Upon receiving ARP reply, ODL installs exact IP match flow in FIB table to direct all further traffic towards the newly learnt MAC of the destination IP Current limitations of Subnet-Route feature: * Works for BGPVPN only * May cause traffic lost due to "swallowing" the packets punted from table 22. * Uses the source MAC and source IP from the punted packet. Aliveness monitor ----------------- After ODL learns a mac that is associated with an ip address, ODL schedule an arp monitor task, with the purpose of verifying that the device is still alive and responding. This is done by periodically sending arp requests to the device. Current limitation: Aliveness monitor was not designed for monitoring devices behind flat/VLAN provider network ports. Use Cases --------- * L3 connectivity of tenant network and VLAN provider network, between VMs and PNFs. * VMs in a private network, PNFs in external network * L3 connectivity between VMs and PNFs in a two diffrent tenant networks. Proposed change =============== Subnet-route ------------ * Upon OpenStack configuration of a Subnet in a provider network, a new vrf entry with subnet-route augmentation will be created. * Upon associataion of neutron router with a subnet in a tenant network, a new vrf entry with subnet-route augmentation will be created. * Upon receiving ARP reply, install exact IP match flow in FIB table to direct all further traffic towards the newly resolved PNF, on all relevant computes nodes, which will be discussed later * Packets that had been punted to controller will be resubmitted to the openflow pipeline after installation of exact match flow. Communication between VMs in tenant networks and PNFs in provider networks. --------------------------------------------------------------------------- In this scenario a VM in a private tenant network wants to communicate with a PNF in the (external) provider network * The controller will hold the packets, and initiate an ARP request towards the PNF IP. an ARP request will have source MAC and IP the router gateway and will be sent from the NAPT switch. * ARP packets will be punted from the NAPT switch only. * Upon receiving ARP reply, install exact IP match flow in FIB table to direct all further traffic towards the newly resolved PNF, on all compute nodes that are associated with the external network. * leveraging Aliveness monitor feature to monitor PNFs. The controller will send ARP requests from the NAPT switch. Communication between VMs and PNFs in different tenant networks. ---------------------------------------------------------------- In this scenario a VM and a PNF, in different private networks of the same tenant, wants to communicate. For each subnet prefix, a designated switch will be chosen to communicate directly with the PNFs in that subnet prefix. That means sending ARP requests to the PNFs and receiving their traffic. **Note: IP traffic from VM instances will retain the src MAC of the VM instance, instead of replacing it with the router-interface-mac, in order to prevent MAC momvements in the underlay switches. This is a limitation until NetVirt supports a MAC per hypervisor implementation.** * A subnet flow will be installed in the FIB table, matching the subnet prefix and vpn-id of the router. * ARP request will have a source MAC and IP of the router interface, and will be sent via the provider port in the designated switch. * ARP packets will be punted from the designated switch only. * Upon receiving an ARP reply, install exact IP match flow in FIB table to direct all further traffic towards the newly resolved PNF, on all computes related to the router * ARP responder flow: a new ARP responder flow will be installed in the designated switch This flow will response for ARP requests from a PNF and the response MAC will be the router interface MAC. This flow will use the LPort-tag of the provider port. * Split Horizon protection disabling: traffic from PNFs, arrives to the primary switch(via a provider port) due to the ARP responder rule described above, and will need to be directed to the proper compute of the designated VM (via a provider port). This require disabling the split horizon protection. In order to protects against infinite loops, the packet TTL will be decreased. * leveraging Aliveness monitor, the controller will send ARP requests from the designated switch. ARP messages -------------- ARP messages in the Flat/Vlan provider and tenant networks will be punted from a designated switch, in order to avoid a performance issue in the controller, of dealing with broadcast packets that may be received in multiple provider ports. In external networks this switch is the NAPT switch. Pipeline changes ---------------- First use-case depends on hairpinning spec [2], the flows presented here reflects that dependency. Egress traffic from VM with floating IP to an unresolved PNF in external network ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Packets in FIB table after translation to FIP, will match on subnet flow and will be punted to controller from Subnet Route table. Then, ARP request will be generated and be sent to the PNF. No flow changes are required in this part. | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: vpn-id=router-id,dst-mac=router-interface-mac`` => | FIB table (21) ``match: vpn-id=router-id`` => | Pre SNAT table (26) ``match: vpn-id=router-id,src-ip=vm-ip set vpn-id=ext-subnet-id,src-ip=fip`` => | SNAT table (28) ``match: vpn-id=ext-subnet-id,src-ip=fip set src-mac=fip-mac`` => | FIB table (21) ``match: vpn-id=ext-subnet-id, dst-ip=ext-subnet-ip`` => | Subnet Route table (22): => Output to Controller | - After receiving ARP response from the PNF a new exact IP flow will be installed in table 21. No other flow changes are required. | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: vpn-id=router-id,dst-mac=router-interface-mac`` => | FIB table (21) ``match: vpn-id=router-id`` => | Pre SNAT table (26) ``match: vpn-id=router-id,src-ip=vm-ip set vpn-id=ext-subnet-id,src-ip=fip`` => | SNAT table (28) ``match: vpn-id=ext-subnet-id,src-ip=fip set src-mac=fip-mac`` => | FIB table (21) ``match: vpn-id=ext-subnet-id, dst-ip=pnf-ip, set dst-mac=pnf-mac, reg6=provider-lport-tag`` => | Egress table (220) output to provider port | Egress traffic from VM using NAPT to an unresolved PNF in external network ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Ingress-DPN is not the NAPT switch, no changes required. Traffic will be directed to NAPT switch and directed to the outbound NAPT table straight from the internal tunnel table | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: vpn-id=router-id,dst-mac=router-interface-mac`` => | FIB table (21) ``match: vpn-id=router-id`` => | Pre SNAT table (26) ``match: vpn-id=router-id`` => | NAPT Group ``output to tunnel port of NAPT switch`` | - Ingress-DPN is the NAPT switch. Packets in FIB table after translation to NAPT, will match on subnet flow and will be punted to controller from Subnet Route table. Then, ARP request will be generated and be sent to the PNF. No flow changes are required. | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: vpn-id=router-id,dst-mac=router-interface-mac`` => | FIB table (21) ``match: vpn-id=router-id`` => | Pre SNAT table (26) ``match: vpn-id=router-id`` => | Outbound NAPT table (46) ``match: src-ip=vm-ip,port=int-port set src-ip=router-gw-ip,vpn-id=router-gw-subnet-id,port=ext-port`` => | NAPT PFIB tabl (47) ``match: vpn-id=router-gw-subnet-id`` => | FIB table (21) ``match: vpn-id=ext-subnet-id, dst-ip=ext-subnet-ip`` => | Subnet Route table (22) => Output to Controller | - After receiving ARP response from the PNF a new exact IP flow will be installed in table 21. No other changes required. | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: vpn-id=router-id,dst-mac=router-interface-mac`` => | FIB table (21) ``match: vpn-id=router-id`` => | Pre SNAT table (26) ``match: vpn-id=router-id`` => | Outbound NAPT table (46) ``match: vpn-id=router-id TBD set vpn-id=external-net-id`` => | NAPT PFIB table (47) ``match: vpn-id=external-net-id`` => | FIB table (21) ``match: vpn-id=ext-network-id, dst-ip=pnf-ip set dst-mac=pnf-mac, reg6=provider-lport-tag`` => | Egress table (220) output to provider port | Egress traffic from VM in private network to an unresolved PNF in another private network ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Packet from a VM is punted to the controller, no flow changes are required. | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: vpn-id=router-id,dst-mac=router-interface-mac`` => | FIB table (21) ``match: vpn-id=router-id dst-ip=subnet-ip`` => | Subnet Route table (22): => Output to Controller | - After receiving ARP response from the PNF a new exact IP flow will be installed in table 21. | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: vpn-id=router-id,dst-mac=router-interface-mac`` => | FIB table (21) ``match: vpn-id=router-id dst-ip=pnf-ip set dst-mac=pnf-mac, reg6=provider-lport-tag`` => | Egress table (220) output to provider port | Ingress traffic to VM in private network from a PNF in another private network ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - New flow in table 19, to distinguish our new use-case, in which we want to decrease the TTL of the packet | Classifier table (0) => | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` => | GW Mac table (19) ``match: lport-tag=provider-port, vpn-id=router-id, dst-mac=router-interface-mac, set split-horizon-bit = 0, decrease-ttl`` => | FIB table (21) ``match: vpn-id=router-id dst-ip=vm-ip set dst-mac=vm-mac reg6=provider-lport-tag`` => | Egress table (220) output to provider port | Yang changes ------------ In odl-l3vpn module, adjacency-list grouping will be enhanced with the following field :: grouping adjacency-list { list adjacency { key "ip_address"; ... leaf phys-network-func { type boolean; default false; description "Value of True indicates this is an adjacency of a device in a provider network"; } } } An adjacency that is added as a result of a PNF discovery, is a primary adjacency with an empty next-hop-ip list. This is not enough to distinguish PNF at all times. This new field will help us identify this use-case in a more robust way. Configuration impact --------------------- A configuration mode will be available to turn this feature ON/OFF. Clustering considerations ------------------------- None Other Infra considerations -------------------------- None Security considerations ------------------------------ None Scale and Performance Impact ---------------------------- All traffic of PNFs in each subnet-prefix sends their traffic to a designated switch. Targeted Release ----------------- Carbon Alternatives ------------ None Usage ===== Create external network with a subnet ------------------------------------- :: neutron net-create public-net -- --router:external --is-default --provider:network_type=flat --provider:physical_network=physnet1 neutron subnet-create --ip_version 4 --gateway 10.64.0.1 --name public-subnet1 10.64.0.0/16 -- --enable_dhcp=False Create internal networks with subnets ------------------------------------- :: neutron net-create private-net1 neutron subnet-create --ip_version 4 --gateway 10.0.123.1 --name private-subnet1 10.0.123.0/24 neutron net-create private-net2 neutron subnet-create --ip_version 4 --gateway 10.0.124.1 --name private-subnet2 10.0.124.0/24 Create a router instance and connect it to an internal subnet and an external subnet ------------------------------------------------------------------------------------ This will allow communication with PNFs in provider network :: neutron router-create router1 neutron router-interface-add neutron router-gateway-set --fixed-ip subnet_id= Create a router instance and connect to it to two internal subnets ------------------------------------------------------------------ This will allow East/West communication between VMs and PNFs :: neutron router-create router1 neutron router-interface-add neutron router-interface-add Features to Install ------------------- odl-netvirt-openstack REST API -------- CLI --- Implementation ============== Assignee(s) ----------- Primary assignee: Tomer Pearl Other contributors: Yakir Dorani Work Items ---------- * Configure subnet-route flows upon ext-net configuration / router association * Solve traffic lost issues of punted packets from table 22 * Enable aliveness monitoring on external interfaces. * Add ARP responder flow for L3-PNF * Add ARP packet-in from primary switch only * Disable split-horizon and enable TTL decrease for L3-PNF Dependencies ============ This feature depends on hairpinning feature [2] Testing ======= Unit Tests ---------- Unit tests will be added for the new functionality Integration Tests ----------------- CSIT ---- Will need to see if a PNF could be simulated in CSIT Documentation Impact ==================== References ========== [1] https://docs.google.com/presentation/d/1ByvEQXUtIyH-H7Bin6OBJNrHjOv-3hpHYzU6Sf6hDbA/edit#slide=id.g11657174d1_0_31 [2] http://docs.opendaylight.org/en/latest/submodules/netvirt/docs/specs/hairpinning-flat-vlan.html