Netvirt Documentation¶

Use Cases¶

Use cases addressed by this feature.

Pipeline changes¶

Any changes to pipeline must be captured explicitly in this section.

Yang changes¶

This should detail any changes to yang models.

 module example {
     namespace "urn:opendaylight:netvirt:example";
     prefix "example";

     import ietf-yang-types {prefix yang; revision-date "2013-07-15";}

     description "An example YANG model.";

     revision 2017-02-14 { description "Initial revision"; }
 }

Configuration impact¶

Any configuration parameters being added/deprecated for this feature? What will be defaults for these? How will it impact existing deployments?

Note that outright deletion/modification of existing configuration is not allowed due to backward compatibility. They can only be deprecated and deleted in later release(s).

Clustering considerations¶

This should capture how clustering will be supported. This can include but not limited to use of CDTCL, EOS, Cluster Singleton etc.

Other Infra considerations¶

This should capture impact from/to different infra components like MDSAL Datastore, karaf, AAA etc.

Security considerations¶

Document any security related issues impacted by this feature.

Scale and Performance Impact¶

What are the potential scale and performance impacts of this change? Does it help improve scale and performance or make it worse?

Targeted Release¶

What release is this feature targeted for?

Alternatives¶

Alternatives considered and why they were not selected.

Features to Install¶

odl-netvirt-openstack

Identify existing karaf feature to which this change applies and/or new karaf features being introduced. These can be user facing features which are added to integration/distribution or internal features to be used by other projects.

REST API¶

Sample JSONS/URIs. These will be an offshoot of yang changes. Capture these for User Guide, CSIT, etc.

CLI¶

Any CLI if being added.

Assignee(s)¶

Who is implementing this feature? In case of multiple authors, designate a primary assignee and other contributors.

Primary assignee:

<developer-a>, <irc nick>, <email>

Other contributors:

<developer-b>, <irc nick>, <email> <developer-c>, <irc nick>, <email>

Work Items¶

Break up work into individual items. This should be a checklist on a Trello card for this feature. Provide the link to the trello card or duplicate it.

Unit Tests¶

Integration Tests¶

CSIT¶

Use Cases¶

Collection and retrieval of information on the number of packets dropped by the ACL rules

• Done for all (VM) ports in which ACL is configured
• The information in these counters are made available to the operator, on demand, through an API
• Service Orchestrator/operator can also specify ports selectively where ACL rules are configured

ACL Changes¶

Current Stateful ACL implementation has drop flows for all ports combined for a device. This needs to be modified to have drop flows for each of the OF ports connected to VMs (Neutron Ports).

With the current implementation, drop flows are as below:

cookie=0x6900000, duration=938.964s, table=252, n_packets=0, n_bytes=0, priority=62020,
        ct_state=+inv+trk actions=drop

cookie=0x6900000, duration=938.969s, table=252, n_packets=0, n_bytes=0, priority=50,
        ct_state=+new+trk actions=drop

Now, for supporting Drop packets statistics per port, ACL will be updated to replace above flows with new DROP flows with lport tag as metadata for each of the VM (Neutron port) being added to OVS as specified below:

cookie=0x6900001, duration=938.964s, table=252, n_packets=0, n_bytes=0, priority=62015,
        metadata=0x10000000000/0xffffff0000000000, ct_state=+inv+trk actions=drop

cookie=0x6900001, duration=938.969s, table=252, n_packets=0, n_bytes=0, priority=50,
        metadata=0x10000000000/0xffffff0000000000, ct_state=+new+trk actions=drop

Drop flows details explained above are for pipeline egress direction. For ingress side, similar drop flows would be added with table=41.

Also, new cookie value 0x6900001 would be added with drop flows to identify it uniquely and priority 62015 would be used with +inv+trk flows to give higher priority for +est and +rel flows.

Drop packets statistics support¶

ODL Controller will be updated to provide a new RPC/NB REST API <get-acl-port-statistics> in ACL module with ACL Flow Stats Request and ACL Flow Stats Response messages. This RPC/API will retrieve details of dropped packets by Security Group rules for all the neutron ports specified as part of ACL Flow Stats Request. The retrieved information (instantaneous) received in the OF reply message is formatted as ACL Flow Stats Response message before sending it as a response towards the NB.

<get-acl-port-statistics> RPC/API implementation would be triggering opendaylight-direct-statistics:get-flow-statistics request of OFPlugin towards OVS to get the flow statistics of ACL tables (ingress / egress) for the required ports.

ACL Flow Stats Request/Response messages are explained in subsequent sections.

Pipeline changes¶

No changes needed in OF pipeline. But, new flows as specified in above section would be added for each of the Neutron ports being added.

Yang changes¶

New yang file will be created with RPC as specified below:

 module acl-live-statistics {
     namespace "urn:opendaylight:netvirt:acl:live:statistics";

     prefix "acl-stats";

     import ietf-interfaces {prefix if;}
     import aclservice {prefix aclservice; revision-date "2016-06-08";}

     description "YANG model describes RPC to retrieve ACL live statistics.";

     revision "2016-11-29" {
         description "Initial revision of ACL live statistics";
     }

     typedef direction {
         type enumeration {
             enum ingress;
             enum egress;
             enum both;
         }
     }

     grouping acl-drop-counts {
         leaf drop-count {
             description "Packets/Bytes dropped by ACL rules";
             type uint64;
         }
         leaf invalid-drop-count {
             description "Packets/Bytes identified as invalid";
             type uint64;
         }
     }

     grouping acl-stats-output {
         description "Output for ACL port statistics";
         list acl-interface-stats {
             key "interface-name";
             leaf interface-name {
                 type leafref {
                     path "/if:interfaces/if:interface/if:name";
                 }
             }
             list acl-drop-stats {
                 max-elements "2";
                 min-elements "0";
                 leaf direction {
                     type identityref {
                         base "aclservice:direction-base";
                     }
                 }
                 container packets {
                     uses acl-drop-counts;
                 }
                 container bytes {
                     uses acl-drop-counts;
                 }
             }
             container error {
                 leaf error-message {
                     type string;
                 }
             }
         }
     }

     grouping acl-stats-input {
         description "Input parameters for ACL port statistics";

         leaf direction {
             type identityref {
                 base "aclservice:direction-base";
             }
             mandatory "true";
         }
         leaf-list interface-names {
             type leafref {
                 path "/if:interfaces/if:interface/if:name";
             }
             max-elements "unbounded";
             min-elements "1";
         }
     }

     rpc get-acl-port-statistics {
         description "Get ACL statistics for given list of ports";

         input {
             uses acl-stats-input;
         }
         output {
             uses acl-stats-output;
         }
     }
 }

Configuration impact¶

No configuration parameters being added/deprecated for this feature

Clustering considerations¶

No additional changes required to be done as only one RPC is being supported as part of this feature.

Other Infra considerations¶

N.A.

Security considerations¶

N.A.

Scale and Performance Impact¶

N.A.

Targeted Release¶

Carbon

Alternatives¶

Dispatcher table (table 17 and table 220) based approach of querying drop packets count was considered. ie., arriving drop packets count by below rule:

<total packets entered ACL tables> - <total packets entered subsequent service>

This approach was not selected as this only provides total packets dropped count per port by ACL services and does not provide details of whether it’s dropped by ACL rules or for some other reasons.

Features to Install¶

odl-netvirt-openstack

REST API¶

{
    "input":
    {
         "direction": "both",
         "interface-names": [
             "4ae8cd92-48ca-49b5-94e1-b2921a2661c5",
             "6c53df3a-3456-11e5-a151-feff819cdc9f"
         ]
    }
}

{
    "output": {
    "acl-port-stats": [
    {
        "interface-name": "4ae8cd92-48ca-49b5-94e1-b2921a2661c5",
        "acl-drop-stats": [
        {
            "direction": "ingress",
            "bytes": {
                "invalid-drop-count": "0",
                "drop-count": "300"
            },
            "packets": {
                "invalid-drop-count": "0",
                "drop-count": "4"
            }
        },
        {
            "direction": "egress",
            "bytes": {
                "invalid-drop-count": "168",
                "drop-count": "378"
            },
            "packets": {
                "invalid-drop-count": "2",
                "drop-count": "9"
            }
        }]
    },
    {
        "interface-name": "6c53df3a-3456-11e5-a151-feff819cdc9f",
        "acl-drop-stats": [
        {
            "direction": "ingress",
            "bytes": {
                "invalid-drop-count": "1064",
                "drop-count": "1992"
            },
            "packets": {
                "invalid-drop-count": "18",
                "drop-count": "23"
             }
        },
        {
            "direction": "egress",
            "bytes": {
                "invalid-drop-count": "462",
                "drop-count": "476"
             },
            "packets": {
                "invalid-drop-count": "11",
                "drop-count": "6"
            }
        }]
    }]
}

{
    "output":
    {
        "acl-port-stats": [
        {
            "interface-name": "4ae8cd92-48ca-49b5-94e1-b2921a2661c5",
            "acl-drop-stats": [
            {
                "direction": "ingress",
                "bytes": {
                    "invalid-drop-count": "0",
                    "drop-count": "300"
                },
                "packets": {
                    "invalid-drop-count": "0",
                    "drop-count": "4"
                }
            },
            {
                "direction": "egress",
                "bytes": {
                    "invalid-drop-count": "168",
                    "drop-count": "378"
                },
                "packets": {
                    "invalid-drop-count": "2",
                    "drop-count": "9"
                }
            },
            {
                "interface-name": "6c53df3a-3456-11e5-a151-feff819cdc9f",
                "error": {
                    "error-message": "Interface not found in datastore."
                }
            }]
        }]
    }
}

CLI¶

No CLI being added for this feature

Assignee(s)¶

Primary assignee:

<Somashekar Byrappa>

Other contributors:

<Shashidhar R>

Work Items¶

1. Adding new drop rules per port (in table 41 and 252)
2. Yang changes
3. Supporting new RPC

Unit Tests¶

Following test cases will need to be added/expanded

1. Verify ACL STAT RPC with single Neutron port
2. Verify ACL STAT RPC with multiple Neutron ports
3. Verify ACL STAT RPC with invalid Neutron port
4. Verify ACL STAT RPC with mode set to “transparent/learn/stateless”

Also, existing unit tests will be updated to include new drop flows.

Integration Tests¶

Integration tests will be added, once IT framework is ready

CSIT¶

Following test cases will need to be added/expanded

1. Verify ACL STAT RPC with single Neutron port with different directions (ingress, egress, both)
2. Verify ACL STAT RPC with multiple Neutron ports with different directions (ingress, egress, both)
3. Verify ACL STAT RPC with invalid Neutron port
4. Verify ACL STAT RPC with combination of valid and invalid Neutron ports
5. Verify ACL STAT RPC with combination of Neutron ports with few having port-security-enabled as true and others having false

Use Cases¶

Neutron configuration of security rules, configured with remote SGs. This optimization will be relevant only when there is a single security group that is associated with the port. In case more than one security group is associated with the port - we will fallback to the current implementation which allows full functionality but with possible flow scaling issues.

Rules with a remote ACL are used to allow certain types of packets only between VMs in certain security groups. For example, configuring rules with the parent security group also configured as a remote security group, allows to configure rules applied only for traffic between VMs in the same security group.

This will be done in the ACL implementation, so any ACL configured with a remote ACL via a different northbound or REST would also be handled.

Pipeline changes¶

ACL3 will be added, and the flows in ACL2/ACL3 will be modified as noted above in the proposed change:

Table Numbering:

Currently the Ingress ACLs use tables 40,41,42 and the Egress ACLs use tables 251,252,253.

Table 43 is already proposed to be taken by SNAT, and table 254 is considered invalid by OVS. To overcome this and align Ingress/Egress with symmetric numbering, I propose the following change:

• Ingress ACLs: 211, 212, 213, 214
• Egress ACLs: 241, 242, 243, 244

ACL1: INGRESS/EGRESS_ACL_TABLE ACL2: INGRESS/EGRESS_ACL_REMOTE_ACL_TABLE ACL3: INGRESS/EGRESS_ACL_FILTER_TABLE

ACL4 is used only for Learn implementation for which an extra table is required.

Yang changes¶

None.

Configuration impact¶

None.

Clustering considerations¶

None.

Other Infra considerations¶

None.

Security considerations¶

None.

Scale and Performance Impact¶

See example in description. The scale of the flows will be drastically reduced when using remote ACLs.

Targeted Release¶

Carbon

Alternatives¶

For fully optimized support in all scenarios for remote SGs, meaning including support for ports with multiple ACLs on them, we did consider implementing a similar optimization.

However, for this to happen due to OpenFlow limitations we would need to introduce an internal dispatcher inside the ACL services, meaning we loop the ACL service multiple times, each time setting a different metadata SG value for the port.

For another approach we could use a bitmask, but this would limit the number of possible SGs to be the number of bits in the mask, which is much too low for any reasonable use case.

Features to Install¶

Install the ODL Karaf feature for NetVirt (no change):

• odl-netvirt-openstack

REST API¶

None.

CLI¶

Refer to the Neutron CLI Reference [1] for the Neutron CLI command syntax for managing Security Rules with Remote Security Groups.

Assignee(s)¶

Who is implementing this feature? In case of multiple authors, designate a primary assigne and other contributors.

Primary assignee:

• Alon Kochba <alonko@hpe.com>
• Aswin Suryanarayanan <asuryana@redhat.com>

Other contributors:

• ?

Work Items¶

Task list in Carbon Trello

Unit Tests¶

Integration Tests¶

CSIT¶

We should add tests verifying remote SG configuration functionality. There should be at least:

• One security rule allowing ICMP traffic between VMs in the same SG.
• One positive test, checking ICMP connectivity works between two VMs using the same SG.
• One negative test, checking ICMP connectivity does not work between two VMs, one using the SG configured with the rule above, and the other using a separate security group with all directions allowed.

Use Cases¶

The new ACL implementation will affect the below use cases,

• VM Creation/Deletion with SG
• SG Rule addition and removal to/from existing SG associated to ports

Pipeline changes¶

The propose changes includes:

• New tables 210 and 240
• Re-purposed tables 211, 212, 241, 242

The propose will re-purpose the table 211 and 212 of egress, table 241 and 242 of ingress.

Currently, for egress, we are using the table 211 for source match and 212 for destination match. In new propose, we will use the new table 210 for source match, table 211 for destination match and table 212 for new flow installation when we delete the SG flow.

For Egress, the traffic will use the tables in following order,

17 >> 210 >> 211 >> 212 >> 213.

Similarly, for ingress, currently we are using the table 241 for destination match and 242 for source match. In new propose, we will use the new table 240 for destination match, table 241 for source match and table 242 for new flow installation when we delete the SG flow.

For Ingress, the traffic will use the tables in following order,

220 >> 240 >> 241 >> 242 >> 243

flow will be added in table 212/242, and the match condition of ACL4 flows will be modified as noted above in the proposed change:

Yang changes¶

The nicira-action.yang and the openflowplugin-extension-nicira-action.yang needs to be updated with ct_mark action. The action structure shall be

grouping ofj-nx-action-conntrack-grouping {
    container nx-action-conntrack {
        leaf flags {
            type uint16;
        }
        leaf zone-src {
            type uint32;
        }
        leaf conntrack-zone {
            type uint16;
        }
        leaf recirc-table {
            type uint128;
        }
        leaf experimenter-id {
            type oft:experimenter-id;
        }
        list ct-actions{
            uses ofpact-actions;
        }
    }
 }

The nicira-match.yang and the openflowplugin-extension-nicira-match.yang needs to be updated with the ct_mark match.

grouping ofj-nxm-nx-match-ct-mark-grouping{
       container ct-mark-values {
          leaf ct-mark {
             type uint32;
          }
           leaf mask {
             type uint32;
          }
      }
  }

Configuration impact¶

None.

Clustering considerations¶

None.

Other Infra considerations¶

None.

Security considerations¶

None.

Scale and Performance Impact¶

When we delete the SG rule from the VM, A new flow will be added in the flow table 212 to flip the value of ct_mark of ongoing traffics. This flow will have a time out based on the protocol as mentioned in the proposed changes section. The packets of ongoing traffic will be recommitted and will do the set filed of ct_mark until the flow reaches the time out.

Targeted Release¶

Carbon

Alternatives¶

While deleting a SG flow from the flow table, we will add a DROP flow with the highest priority in the ACL4 table. This DROP flow will drop the packets and it will stop the existing traffic. Similarly, when we restore the same rule again, we will delete the DROP flow from the ACL4 table which will enable the existing traffic.

But this approach will be effective only if the VM do not have any duplicate flows. With the current ACL implementation, if we associate two SGs which having similar set of SG rule, netvirt will install the two set of flows with different priority for the same VM.

As per above approach, if we dissociate any one of SG from the VM, It will add the DROP flow in ACL4 table which will stops the existing traffic irrespective of there is still another flow available in ACL4, to make the traffic possible.

Features to Install¶

Install the ODL Karaf feature for NetVirt (no change):

• odl-netvirt-openstack

REST API¶

None.

CLI¶

Refer to the Neutron CLI Reference [1] for the Neutron CLI command syntax for managing Security Rules.

Assignee(s)¶

Who is implementing this feature? In case of multiple authors, designate a primary assignee and other contributors.

Primary assignee:

• VinothB <vinothb@hcl.com>
• Balakrishnan Karuppasamy <balakrishnan.ka@hcl.com>

Other contributors:

• ?

Work Items¶

None

Unit Tests¶

Integration Tests¶

CSIT¶

We should add tests verifying ACL change reflection on existing traffic. There should be at least:

• One security rule allowing ICMP traffic between VMs in the same SG.
• One positive test, checking ICMP connectivity works between two VMs using the same SG. Delete all the rules from the SG without disturbing the already established traffic. It should stop the traffic.
• One positive test, checking ICMP connectivity works between two VMs,one using the SG, configured with the ICMP rule, Delete and restore the ICMP rule immediately. This should stop and resume the ICMP traffic after restoring the ICMP rule.
• One positive test, checking ICMP connectivity between VMs, using the SG, configured with ICMP ALL and Other protocol ANY rule. Delete the ICMP rule from the SG, It should not stop the ICMP traffic.
• One negative test, checking ICMP connectivity between two VMs, one using the SG, configured with the ICMP and TCP rules above, and delete the TCP rule. This should not affect the ICMP traffic.

Use Cases¶

The following use case will be realized by the implementation

External Network Access The SNAT enables the VM in a tenant network access the external network without using a floating ip. It uses NAPT for sharing the external ip address across multiple VMs that share the same router gateway.

Pipeline changes¶

The ovs based NAPT flows will replace the controller based NAPT flows. The changes are limited to the designated switch for the router. Below is the illustration for flat external network.

Outbound NAPT

Table 26 (PSNAT Table) => submits the packet to netfilter to check whether it is an existing connection. Resubmits the packet back to 46.

Table 46 (NAPT OUTBOUND TABLE) => if it is an established connection, it indicates the translation is done and the packet is forwarded to table 47 after writing the external network metadata.

If it is a new connection the connection will be committed to netfilter and this entry will be used for NAPT. The translated packet will be resubmitted to table 47. The external network metadata will be written before sending the packet to netfilter.

Table 47 (NAPT FIB TABLE) => The translated packet will be sent to the egress group.

Sample Flows

table=26, priority=5,ip,metadata=0x222e2/0xfffffffe actions=ct(table=46,zone=5003,nat)
table=46, priority=6,ct_state=+snat,ip,metadata=0x222e2/0xfffffffe actions=set_field:0x222e0->metadata,resubmit(,47)
table=46, priority=5,ct_state=+new+trk,ip,metadata=0x222e2/0xfffffffe actions=set_field:0x222e0->metadata,ct(commit,table=47,zone=5003,nat(src=192.168.111.21))
table=47, n_packets=0, n_bytes=0, priority=6,ct_state=+snat,ip,nw_src=192.168.111.21 actions=group:200000

Inbound NAPT

Table 44 (NAPT INBOUND Table)=> submits the packet to netfilter to check for an existing connection after changing the metadata to that of the internal network. The packet will be submitted back to table 47.

Table 47 (NAPT FIB TABLE) => The translated packet will be submitted back to table 21.

Sample Flows

table=21, priority=42,ip,metadata=0x222e0/0xfffffffe,nw_dst=192.168.111.21 actions=resubmit(,44)
table=44, priority=10,ip,metadata=0x222e0/0xfffffffe,nw_dst=192.168.111.21 actions=set_field:0x222e2->metadata,ct(table=47,zone=5003,nat)
table=47, priority=5,ct_state=+dnat,ip actions=resubmit(,21)

Yang changes¶

The nicira-action.yang and the openflowplugin-extension-nicira-action.yang needs to be updated with nat action. The action structure shall be

typedef nx-action-nat-range-present {
    type enumeration {
        enum NX_NAT_RANGE_IPV4_MIN {
            value 1;
            description "IPV4 minimum value is present";
        }
        enum NX_NAT_RANGE_IPV4_MAX {
            value 2;
            description "IPV4 maximum value is present";
        }
        enum NX_NAT_RANGE_IPV6_MIN {
            value 4;
            description "IPV6 minimum value is present in range";
        }
        enum NX_NAT_RANGE_IPV6_MAX {
            value 8;
            description "IPV6 maximum value is present in range";
        }
        enum NX_NAT_RANGE_PROTO_MIN {
            value 16;
            description "Port minimum value is present in range";
        }
        enum NX_NAT_RANGE_PROTO_MAX {
            value 32;
            description "Port maximum value is present in range";
        }
    }
 }

typedef nx-action-nat-flags {
    type enumeration {
        enum NX_NAT_F_SRC {
            value 1;
            description "Source nat is selected ,Mutually exclusive with NX_NAT_F_DST";
        }
        enum NX_NAT_F_DST {
            value 2;
            description "Destination nat is selected";
        }
        enum NX_NAT_F_PERSISTENT {
            value 4;
            description "Persistent flag is selected";
        }
        enum NX_NAT_F_PROTO_HASH {
            value 8;
            description "Hash mode is selected for port mapping, Mutually exclusive with
            NX_NAT_F_PROTO_RANDOM ";
        }
        enum NX_NAT_F_PROTO_RANDOM {
            value 16;
            description "Port mapping will be randomized";
        }
    }
 }

grouping ofj-nx-action-conntrack-grouping {
    container nx-action-conntrack {
        leaf flags {
            type uint16;
        }
        leaf zone-src {
            type uint32;
        }
        leaf conntrack-zone {
            type uint16;
        }
        leaf recirc-table {
            type uint8;
        }
        leaf experimenter-id {
            type oft:experimenter-id;
        }
        list ct-actions{
            uses ofpact-actions;
        }
    }
 }

grouping ofpact-actions {
    description
       "Actions to be performed with conntrack.";
    choice ofpact-actions {
         case nx-action-nat-case {
            container nx-action-nat {
                leaf flags {
                    type uint16;
                }
                leaf range_present {
                    type uint16;
                }
                leaf ip-address-min {
                    type inet:ip-address;
                }
                leaf ip-address-max {
                    type inet:ip-address;
                }
                leaf port-min {
                    type uint16;
                }
                leaf port-max {
                    type uint16;
                }
            }
        }
    }
}

For the new configuration knob a new yang natservice-config shall be added in NAT service, with the container for holding the NAT mode configured. It will have two options controller and conntrack, with controller being the default.

container natservice-config {
  config true;
  leaf nat-mode {
      type enumeration {
          enum "controller";
          enum "conntrack";
      }
      default "controller";
  }
}

Configuration impact¶

The proposed change requires the NAT service to provide a configuration knob to switch between the controller based/conntrack based implementation. A new configuration file netvirt-natservice-config.xml shall be added with default value controller.

<natservice-config xmlns="urn:opendaylight:netvirt:natservice-config">
  <nat-mode>controller</nat-mode>
</natservice-config>

The dynamic update of nat-mode will not be supported. To change the nat-mode the controller cluster needs to be restarted after changing the nat-mode. On restart the NAT translation lifecycle will be reset and after the controller comes up in the updated nat-mode, a new set of switches will be elected as designated NAPT switches and it can be different from the ones that were forwarding traffic earlier.

Clustering considerations¶

NA

Other Infra considerations¶

The implementation requires ovs2.6 with the kernel module installed. OVS currently does not support SNAT connection tracking for dpdk datapath. It would be supported in some future release.

Security considerations¶

NA

Scale and Performance Impact¶

The new SNAT implementation is expected to improve the performance when compared to the existing one and will reduce the flows in ovs pipeline.

Targeted Release¶

Carbon

Alternatives¶

An alternative implementation of X NAPT switches was discussed, which will not be a part of this document but will be considered as a further enhancement.

Create External Network¶

Create an external flat network and subnet

neutron net-create ext1 --router:external  --provider:physical_network public --provider:network_type flat
neutron subnet-create --allocation-pool start=<start-ip>,end=<end-ip> --gateway=<gw-ip> --disable-dhcp --name subext1 ext1 <subnet-cidr>

Create Internal Network¶

Create an internal n/w and subnet

neutron net-create vx-net1 --provider:network_type vxlan
neutron subnet-create vx-net1 <subnet-cidr> --name vx-subnet1

Create Router¶

Create a router and add an interface to internal n/w. Set the external n/w as the router gateway.

neutron router-create router1
neutron router-interface-add  router1 vx-subnet1
neutron router-gateway-set router1 ext1
nova boot --poll --flavor m1.tiny --image $(nova image-list | grep 'uec\s' | awk '{print $2}' | tail -1) --nic net-id=$(neutron net-list | grep -w vx-net1 | awk '{print $2}') vmvx2

Features to Install¶

odl-netvirt-openstack

REST API¶

NA

CLI¶

A new command line, display-napt-switch, will be added to display the current designated NAPT switch selected for each router. It shall show the below info.

router id | Host Name of designated NAPT switch | Management Ip of the designated NAPT switch

Assignee(s)¶

Aswin Suryanarayanan <asuryana@redhat.com>

Work Items¶

https://trello.com/c/DMLsrLfq/9-snat-decentralized-ovs-nat-based

• Write a framework which can support multiple modes of NAT implementation.
• Add support in openflow plugin for conntrack nat actions.
• Add support in genius for conntrack nat actions.
• Add a config parameter to select between controller based and conntrack based.
• Add the flow programming for SNAT in netvirt.
• Add the new HA framework.
• Add the command to display the designated NAPT switch.
• Write Unit tests for conntrack based snat.

Unit Tests¶

Unit test needs to be added for the new snat mode. It shall use the component tests framework

Integration Tests¶

Integration tests needs to be added for the conntrack snat flows.

CSIT¶

Run the CSIT with conntrack based SNAT configured.

Use Cases¶

This feature will cover the following use cases:

Pipeline changes¶

No new pipeline changes are introduced as part of this feature. The pipeline flow between VM instances in remote sites is similar to the current implementation of cross compute intra-DC traffic since the realization of remote compute nodes is similar to local ones.

Yang changes¶

The following new yang models will be introduced as part of the federation plugin API bundle:

module federation-plugin {
   yang-version 1;
   namespace "urn:opendaylight:netvirt:federation:plugin";
   prefix "federation-plugin";

   import yang-ext {
        prefix ext;
        revision-date "2013-07-09";
   }

   import ietf-yang-types {
        prefix yang;
   }

   import network-topology {
        prefix topo;
   }

   import opendaylight-inventory {
        prefix inv;
   }

   import ietf-interfaces {
        prefix if;
   }

   import elan {
        prefix elan;
   }

   import l3vpn {
        prefix l3vpn;
   }

   import neutronvpn {
        prefix nvpn;
   }

   revision "2017-02-19" {
       description "Federation plugin model";
   }

   grouping shadow-properties {
       leaf shadow {
           type boolean;
           description "Represents whether this is a federated entity";
       }
       leaf generation-number {
           type int32;
           description "The current generation number of the federated entity";
       }
       leaf remote-ip {
           type string;
           description "The IP address of the original site of the federated entity";
       }
   }

   augment "/topo:network-topology/topo:topology/topo:node" {
       ext:augment-identifier "topology-node-shadow-properties";
       uses shadow-properties;
   }

   augment "/inv:nodes/inv:node" {
       ext:augment-identifier "inventory-node-shadow-properties";
       uses shadow-properties;
   }

   augment "/if:interfaces/if:interface" {
       ext:augment-identifier "if-shadow-properties";
       uses shadow-properties;
   }

   augment "/elan:elan-interfaces/elan:elan-interface" {
       ext:augment-identifier "elan-shadow-properties";
       uses shadow-properties;
   }

   augment "/l3vpn:vpn-interfaces/l3vpn:vpn-interface" {
       ext:augment-identifier "vpn-shadow-properties";
       uses shadow-properties;
   }
}

module federation-plugin-manager {
   yang-version 1;
   namespace "urn:opendaylight:netvirt:federation:plugin:manager";
   prefix "federation-plugin-manager";

   import yang-ext {
       prefix ext;
       revision-date "2013-07-09";
  }

  import ietf-yang-types {
       prefix yang;
  }

  revision "2017-02-19" {
      description "Federation plugin model";
  }

   identity mgr-context {
       description "Identity for a routed RPC";
   }

   container routed-container {
       list route-key-item {
           key "id";
           leaf id {
               type string;
           }

           ext:context-instance "mgr-context";
       }
   }

   container federated-networks {
       list federated-network {
           key self-net-id;
           uses federated-nets;
       }
   }

   container federation-generations {
       description
               "Federation generation information for a remote site.";
       list remote-site-generation-info {
           max-elements "unbounded";
           min-elements "0";
           key "remote-ip";
           leaf remote-ip {
               mandatory true;
               type string;
               description "Remote site IP address.";
           }
           leaf generation-number {
               type int32;
               description "The current generation number used for the remote site.";
           }
       }
   }

   grouping federated-nets {
       leaf self-net-id {
           type string;
           description "UUID representing the self net";
       }
       leaf self-subnet-id {
           type yang:uuid;
           description "UUID representing the self subnet";
       }
       leaf self-tenant-id {
           type yang:uuid;
           description "UUID representing the self tenant";
       }
       leaf subnet-ip {
           type string;
           description "Specifies the subnet IP in CIDR format";
       }

       list site-network {
           key id;
           leaf id {
               type string;
               description "UUID representing the site ID (from xsite manager)";
           }
           leaf site-ip {
               type string;
               description "Specifies the site IP";
           }
           leaf site-net-id {
               type string;
               description "UUID of the network in the site";
           }
           leaf site-subnet-id {
               type yang:uuid;
               description "UUID of the subnet in the site";
           }
           leaf site-tenant-id {
               type yang:uuid;
               description "UUID of the tenant holding this network in the site";
           }
       }
   }
}

module federation-plugin-rpc {
   yang-version 1;
   namespace "urn:opendaylight:netvirt:federation:plugin:rpc";
   prefix "federation-plugin-rpc";

   import yang-ext {
       prefix ext;
       revision-date "2013-07-09";
  }

  import ietf-yang-types {
       prefix yang;
  }

  import federation-plugin-manager {
       prefix federation-plugin-manager;
  }

  revision "2017-02-19" {
      description "Federation plugin model";
  }

   rpc update-federated-networks {
       input {
           list federated-networks-in {
               key self-net-id;
               uses federation-plugin-manager:federated-nets;
               description "Contain all federated networks in this site that are still
                            connected, a federated network that does not appear will be considered
                            disconnected";
           }
       }
   }
}

module federation-plugin-routed-rpc {
   yang-version 1;
   namespace "urn:opendaylight:netvirt:federation:plugin:routed:rpc";
   prefix "federation-plugin-routed-rpc";

   import yang-ext {
       prefix ext;
       revision-date "2013-07-09";
   }

   import ietf-yang-types {
       prefix yang;
   }

   import federation-plugin-manager {
       prefix federation-plugin-manager;
   }

   revision "2017-02-19" {
       description "Federation plugin model";
   }

   rpc update-federated-networks {
       input {
           leaf route-key-item {
               type instance-identifier;
               ext:context-reference federation-plugin-manager:mgr-context;
           }

           list federated-networks-in {
               key self-net-id;
               uses federation-plugin-manager:federated-nets;
           }
       }
   }
}

Configuration impact¶

None.

Clustering considerations¶

The federation plugin will be active only on one of the ODL instances in the cluster. The cluster singleton service infrastructure will be used in order to register the federation plugin routed RPCs only on the selected ODL instance.

Other Infra considerations¶

None

Security considerations¶

None

Scale and Performance Impact¶

None

Targeted Release¶

Carbon

Alternatives¶

None

Features to Install¶

odl-netvirt-federation

This is a new feature that will load odl-netvirt-openstack and the federation service features. It will not be installed by default and requires manual startup using karaf feature:install command.

REST API¶

Connecting neutron networks from remote sites

URL: restconf/operations/federation-plugin-manager:update-federated-networks

Sample JSON data

{
   "input": {
       "federated-networks-in": [
           {
               "self-net-id": "c4976ee7-c5cd-4a5e-9cf9-261f28ba7920",
               "self-subnet-id": "93dee7cb-ba25-4318-b60c-19a15f2c079a",
               "subnet-ip": "10.0.123.0/24",
               "site-network": [
                   {
                       "id": "c4976ee7-c5cd-4a5e-9cf9-261f28ba7922",
                       "site-ip": "10.0.43.146",
                       "site-net-id": "c4976ee7-c5cd-4a5e-9cf9-261f28ba7921",
                       "site-subnet-id": "93dee7cb-ba25-4318-b60c-19a15f2c079b",
                   }
               ]
           }
       ]
   }
}

CLI¶

None.

Assignee(s)¶

Primary assignee:

Tali Ben-Meir <tali@hpe.com>

Other contributors:

Guy Sela <guy.sela@hpe.com>

Shlomi Alfasi <shlomi.alfasi@hpe.com>

Yair Zinger <yair.zinger@hpe.com>

Work Items¶

Trello card https://trello.com/c/mgdUO6xx/154-federation-plugin-for-netvirt

Since the code was already implemented in downstream no work items will be defined

Unit Tests¶

End-to-end component service will test the federation plugin on top of the federation service.

Integration Tests¶

None

CSIT¶

The CSIT infrastructure will be enhanced to support connect/disconnect operations between sites using update-federated-networks RPC call.

A new federation suite will test L2 and L3 connectivity between remote sites and will be based on the existing L2/L3 connectivity suites. CSIT will load sites A,B and C in 1-node/3-node deployment options to run the following tests:

Use Cases¶

a DHCP packet that is received by the odl, from a port that is managed by Netvirt and was configured using the netvirt API, rather then the neutron API, in a way that there is no pre-allocated IP for network interfaces behind that port - will be handled by the DHCP dynamic allocation pool that is configured on the network associated with the receiving OVS port.

Pipeline changes¶

This new rule in table 60 will be responsible for forwarding dhcp packets to the controller:

cookie=0x6800000, duration=121472.576s, table=60, n_packets=1, n_bytes=342, priority=49,udp,tp_src=68,tp_dst=67 actions=CONTROLLER:65535

Yang changes¶

New YANG model to support the configuration of the DHCP allocation pools and allocations, per network and subnet.

• Allocation-Pool: configuration of allocation pool parameters like range, gateway and dns servers.
• Allocation-Instance: configuration of static IP address allocation and Neutron pre-allocated addresses, per MAC address.

 container dhcp_allocation_pool {
     config true;
     description "contains DHCP Server dynamic allocations";

     list network {
         key "network-id";
         leaf network-id {
             description "network (elan-instance) id";
             type string;
         }
         list allocation {
             key "subnet";
             leaf subnet {
                 description "subnet for the dhcp to allocate ip addresses";
                 type inet:ip-prefix;
             }

             list allocation-instance {
                 key "mac";
                 leaf mac {
                     description "requesting mac";
                     type yang:phys-address;
                 }
                 leaf allocated-ip {
                     description "allocated ip address";
                     type inet:ip-address;
                 }
             }
         }
         list allocation-pool {
             key "subnet";
             leaf subnet {
                 description "subnet for the dhcp to allocate ip addresses";
                 type inet:ip-prefix;
             }
             leaf allocate-from {
                 description "low allocation limit";
                 type inet:ip-address;
             }
             leaf allocate-to {
                 description "high allocation limit";
                 type inet:ip-address;
             }
             leaf gateway {
                 description "default gateway for dhcp allocation";
                 type inet:ip-address;
             }
             leaf-list dns-servers {
                 description "dns server list";
                 type inet:ip-address;
             }
             list static-routes {
                 description "static routes list for dhcp allocation";
                 key "destination";
                 leaf destination {
                     description "destination in CIDR format";
                     type inet:ip-prefix;
                 }
                 leaf nexthop {
                     description "router ip address";
                     type inet:ip-address;
                 }
             }
         }
     }
 }

Configuration impact¶

The feature is activated in the configuration (disabled by default).

adding dhcp-dynamic-allocation-pool-enabled leaf to dhcpservice-config:

 container dhcpservice-config {
     leaf controller-dhcp-enabled {
         description "Enable the dhcpservice on the controller";
         type boolean;
         default false;
     }

     leaf dhcp-dynamic-allocation-pool-enabled {
         description "Enable dynamic allocation pool on controller dhcpservice";
         type boolean;
         default false;
     }
 }

and netvirt-dhcpservice-config.xml:

<dhcpservice-config xmlns="urn:opendaylight:params:xml:ns:yang:dhcpservice:config">
  <controller-dhcp-enabled>false</controller-dhcp-enabled>
  <dhcp-dynamic-allocation-pool-enabled>false</dhcp-dynamic-allocation-pool-enabled>
</dhcpservice-config>

Clustering considerations¶

Support clustering.

Other Infra considerations¶

None.

Security considerations¶

None.

Scale and Performance Impact¶

None.

Targeted Release¶

Carbon.

Alternatives¶

Implement and maintain an external DHCP server.

Features to Install¶

This feature can be used by installing odl-netvirt-openstack. This feature doesn’t add any new karaf feature.

REST API¶

Introducing a new REST API for the feature

{"dhcp_allocation_pool": {
  "network": [
    {
      "network-id": "d211a14b-e5e9-33af-89f3-9e43a270e0c8",
      "allocation-pool": [
        {
          "subnet": "10.1.1.0/24",
          "dns-servers": [
            "8.8.8.8"
          ],
          "gateway": "10.1.1.1",
          "allocate-from": "10.1.1.2",
          "allocate-to": "10.1.1.200"
          "static-routes": [
            {
              "destination": "5.8.19.24/16",
              "nexthop": "10.1.1.254"
            }
          ]
]}]}}

{"dhcp_allocation_pool": {
  "network": [
    {
      "network-id": "d211a14b-e5e9-33af-89f3-9e43a270e0c8",
      "allocation": [
        {
          "subnet": "10.1.1.0/24",
          "allocation-instance": [
            {
              "mac": "fa:16:3e:9d:c6:f5",
              "allocated-ip": "10.1.1.2"
            }
]}]}]}}

CLI¶

None.

Assignee(s)¶

Primary assignee:

Shai Haim (shai.haim@hpe.com)

Other contributors:

Alex Feigin (alex.feigin@hpe.com)

Work Items¶

Here is the link for the Trello Card: https://trello.com/c/0mgGyJuV/153-dhcp-server-dynamic-allocation-pool

Unit Tests¶

N.A.

Integration Tests¶

N.A.

CSIT¶

N.A.

Subnet-Route¶

Today, Subnet-Route feature enables ODL to route traffic to a destination IP address, even for ip addresses that have not been statically configured by OpenStack, in the FIB table. To achieve that, the FIB table contains a flow that match all IP packets in a given subnet range. How that works?

• A flow is installed in the FIB table, matching on subnet prefix and vpn-id of the network, with a goto-instruction directing packets to table 22. There, packets are punted to the controller.
• ODL hold the packets, and initiate an ARP request towards the destination IP.
• Upon receiving ARP reply, ODL installs exact IP match flow in FIB table to direct all further traffic towards the newly learnt MAC of the destination IP

Current limitations of Subnet-Route feature:

• Works for BGPVPN only
• May cause traffic lost due to “swallowing” the packets punted from table 22.
• Uses the source MAC and source IP from the punted packet.

Aliveness monitor¶

After ODL learns a mac that is associated with an ip address, ODL schedule an arp monitor task, with the purpose of verifying that the device is still alive and responding. This is done by periodically sending arp requests to the device.

Current limitation: Aliveness monitor was not designed for monitoring devices behind flat/VLAN provider network ports.

Use Cases¶

• L3 connectivity of tenant network and VLAN provider network, between VMs and PNFs.

  • VMs in a private network, PNFs in external network

• L3 connectivity between VMs and PNFs in a two diffrent tenant networks.

Subnet-route¶

• Upon OpenStack configuration of a Subnet in a provider network, a new vrf entry with subnet-route augmentation will be created.
• Upon associataion of neutron router with a subnet in a tenant network, a new vrf entry with subnet-route augmentation will be created.
• Upon receiving ARP reply, install exact IP match flow in FIB table to direct all further traffic towards the newly resolved PNF, on all relevant computes nodes, which will be discussed later
• Packets that had been punted to controller will be resubmitted to the openflow pipeline after installation of exact match flow.

Communication between VMs in tenant networks and PNFs in provider networks.¶

In this scenario a VM in a private tenant network wants to communicate with a PNF in the (external) provider network

• The controller will hold the packets, and initiate an ARP request towards the PNF IP. an ARP request will have source MAC and IP the router gateway and will be sent from the NAPT switch.
• ARP packets will be punted from the NAPT switch only.
• Upon receiving ARP reply, install exact IP match flow in FIB table to direct all further traffic towards the newly resolved PNF, on all compute nodes that are associated with the external network.
• leveraging Aliveness monitor feature to monitor PNFs. The controller will send ARP requests from the NAPT switch.

Communication between VMs and PNFs in different tenant networks.¶

In this scenario a VM and a PNF, in different private networks of the same tenant, wants to communicate. For each subnet prefix, a designated switch will be chosen to communicate directly with the PNFs in that subnet prefix. That means sending ARP requests to the PNFs and receiving their traffic.

Note: IP traffic from VM instances will retain the src MAC of the VM instance, instead of replacing it with the router-interface-mac, in order to prevent MAC momvements in the underlay switches. This is a limitation until NetVirt supports a MAC per hypervisor implementation.

• A subnet flow will be installed in the FIB table, matching the subnet prefix and vpn-id of the router.
• ARP request will have a source MAC and IP of the router interface, and will be sent via the provider port in the designated switch.
• ARP packets will be punted from the designated switch only.
• Upon receiving an ARP reply, install exact IP match flow in FIB table to direct all further traffic towards the newly resolved PNF, on all computes related to the router
• ARP responder flow: a new ARP responder flow will be installed in the designated switch This flow will response for ARP requests from a PNF and the response MAC will be the router interface MAC. This flow will use the LPort-tag of the provider port.
• Split Horizon protection disabling: traffic from PNFs, arrives to the primary switch(via a provider port) due to the ARP responder rule described above, and will need to be directed to the proper compute of the designated VM (via a provider port). This require disabling the split horizon protection. In order to protects against infinite loops, the packet TTL will be decreased.
• leveraging Aliveness monitor, the controller will send ARP requests from the designated switch.

ARP messages¶

ARP messages in the Flat/Vlan provider and tenant networks will be punted from a designated switch, in order to avoid a performance issue in the controller, of dealing with broadcast packets that may be received in multiple provider ports. In external networks this switch is the NAPT switch.

Pipeline changes¶

First use-case depends on hairpinning spec [2], the flows presented here reflects that dependency.

Yang changes¶

In odl-l3vpn module, adjacency-list grouping will be enhanced with the following field

 grouping adjacency-list {
  list adjacency {
    key "ip_address";
    ...
    leaf phys-network-func {
         type boolean;
         default false;
         description "Value of True indicates this is an adjacency of a device in a provider network";
    }
  }
}

An adjacency that is added as a result of a PNF discovery, is a primary adjacency with an empty next-hop-ip list. This is not enough to distinguish PNF at all times. This new field will help us identify this use-case in a more robust way.

Configuration impact¶

A configuration mode will be available to turn this feature ON/OFF.

Clustering considerations¶

None

Other Infra considerations¶

None

Security considerations¶

None

Scale and Performance Impact¶

All traffic of PNFs in each subnet-prefix sends their traffic to a designated switch.

Targeted Release¶

Carbon

Alternatives¶

None

Create external network with a subnet¶

neutron net-create public-net -- --router:external --is-default --provider:network_type=flat
--provider:physical_network=physnet1
neutron subnet-create --ip_version 4 --gateway 10.64.0.1 --name public-subnet1 <public-net-uuid> 10.64.0.0/16
-- --enable_dhcp=False

Create internal networks with subnets¶

neutron net-create private-net1
neutron subnet-create --ip_version 4 --gateway 10.0.123.1 --name private-subnet1 <private-net1-uuid>
10.0.123.0/24
neutron net-create private-net2
neutron subnet-create --ip_version 4 --gateway 10.0.124.1 --name private-subnet2 <private-net2-uuid>
10.0.124.0/24

Create a router instance and connect it to an internal subnet and an external subnet¶

This will allow communication with PNFs in provider network

neutron router-create router1
neutron router-interface-add <router1-uuid> <private-subnet1-uuid>
neutron router-gateway-set --fixed-ip subnet_id=<public-subnet1-uuid> <router1-uuid> <public-net-uuid>

Create a router instance and connect to it to two internal subnets¶

This will allow East/West communication between VMs and PNFs

neutron router-create router1
neutron router-interface-add <router1-uuid> <private-subnet1-uuid>
neutron router-interface-add <router1-uuid> <private-subnet2-uuid>

Features to Install¶

odl-netvirt-openstack

REST API¶

CLI¶

Assignee(s)¶

Primary assignee:

Tomer Pearl <tomer.pearl@hpe.com>

Other contributors:

Yakir Dorani <yakir.dorani@hpe.com>

Work Items¶

• Configure subnet-route flows upon ext-net configuration / router association
• Solve traffic lost issues of punted packets from table 22
• Enable aliveness monitoring on external interfaces.
• Add ARP responder flow for L3-PNF
• Add ARP packet-in from primary switch only
• Disable split-horizon and enable TTL decrease for L3-PNF

Unit Tests¶

Unit tests will be added for the new functionality

Integration Tests¶

CSIT¶

Will need to see if a PNF could be simulated in CSIT

Use Cases¶

• ECMP forwarding of traffic entering a DC (i.e. Spraying of DC-GW -> OVS traffic across multiple Compute Nodes & VMs). In this case, DC-GW can load balance the traffic if a static route can be reachable through multiple NOVA VMs (say VM1 and VM2 connected on different compute nodes) running some networking application (example: vRouter).
• ECMP forwarding of traffic exiting a DC (i.e. Spraying of OVS -> DC-GW traffic across multiple DC Gateways). In this case, a Compute Node can LB the traffic if external route can be reachable from multiple DC-GWs.
• ECMP forwarding of intra-DC traffic (i.e. Spraying of traffic within DC across multiple Compute Nodes & VMs) This is similar to UC1, but load balancing behavior is applied on remote Compute Node for intra-DC communication.
• OVS -> DC-GW tunnel status based ECMP for inter and intra-DC traffic. Tunnel status based on monitoring (BFD) is considered in ECMP path set determination.

Pipeline changes¶

cookie=0x8000003, duration=46.020s, table=21, n_packets=0, n_bytes=0, priority=34,ip,metadata=0x222e4/0xfffffffe, nw_dst=100.0.0.0/24 actions=write_actions(group:150002)
group_id=150002,type=select,bucket=weight:50,actions=group:150001,bucket=weight:50,actions=group:150000
group_id=150001,type=all,bucket=actions=set_field:fa:16:3e:34:ff:58->eth_dst,load:0x200->NXM_NX_REG6[],resubmit(,220)
group_id=150000,type=all,bucket=actions=set_field:fa:16:3e:eb:61:39->eth_dst,load:0x100->NXM_NX_REG6[],resubmit(,220)

YANG changes¶

Changes will be needed in l3vpn.yang , odl-l3vpn.yang and odl-fib.yang to support ECMP functionality.

 grouping vpn-af-config {
     description "A set of configuration parameters that is applicable to both IPv4 and
     IPv6 address family for a VPN instance .";

     leaf-list route-distinguisher {
         description "The route-distinguisher command configures a route distinguisher (RD)
         for the IPv4 or IPv6 address family of a VPN instance.
         Format is ASN:nn or IP-address:nn.";
         config "true";
         type string{
             length "3..21";
         }
     }
 }

ECMP forwarding through multiple Compute Node and VMs¶

In some cases, extra route can be added which can have reachability through multiple Nova VMs. These VMs can be either connected on same compute node (or) different Compute Nodes. When VMs are in different compute nodes, DC-GW should learn all the route paths such that ECMP behavior can be applied for these multi path routes. When VMs are co-located in same compute node, DC-GW will not perform ECMP and compute node performs traffic splitting instead.

ECMP forwarding for dispersed VMs¶

When configured extra route are reached through nova VMs which are connected with different compute node, then it is ODL responsibility to advertise these multiple route paths (but with same MPLS label) to Quagga BGP which in turn sends these routes into DC-GW. But DC-GW replaces the existing route with a new route received from the peer if the NLRI (prefix) is same in the two routes.

This is true even when multipath is enabled on the DC-GW and it is as per standard BGP RFC 4271, Section 9 UPDATE Message Handling. Hence the route is lost in DC-GW even before path computation for multipath is applied.This scenario is solved by adding multiple route distinguisher (RDs) for the vpn instance and let ODL uses the list of RDs to advertise the same prefix with different BGP NHs. Multiple RDs will be supported only for BGP VPNs.

ECMP forwarding for co-located VMs¶

When extra routes on VM interfaces are connected with same compute node, LFIB/FIB and Terminating service table flow entries should be programmed so that traffic can be load balanced between local VMs. This can be done by creating load balancing next hop group for each vpn-to-extraroute (if nexthop-ip-list size is greater than 1) with buckets pointing to the actual VMs next hop group on source Compute Node. Even for the co-located VMs, VPN interface manager should assign separate RDs for each adjacency of same dest IP prefix and let route can be advertised again to Quagga BGP with same next hop (TEP IP address). This will enable DC-Gateway to realize ECMP behavior when an IP prefix can be reachable through multiple co located VMs on one Compute Node and an another VM connected on different Compute Node.

To create load balancing next hop group, the dest IP prefix is used as the key to generate group id. When any of next hop is removed, then adjust load balancing nexthop group so that traffic can be sent through active next hops.

ECMP forwarding through two DC-Gateways¶

The current ITM implementation provides support for creating multiple GRE tunnels for the provided list of DC-GW IP addresses from compute node. This should help in creating corresponding load balancing group whenever Quagga BGP is advertising two routes on same IP prefix pointing to multiple DC GWs. The group id of this load balancing group can be derived from sorted order of DC GW TEP IP addresses with the following format dc_gw_tep_ip _address_1: dc_gw_tep_ip_address_2. This will be useful when multiple external IP prefixes share the same next hops. The load balancing next hop group buckets is programmed according to sorted remote end point DC-Gateway IP address. The support of action move:NXM_NX_REG0(1) -> MPLS label is not supported in ODL openflowplugin. It has to be implemented. Since there are two DC gateways present for the data center, it is possible that multiple equal cost routes are supplied to ODL by Quagga BGP like Fig 2. The current Quagga BGP doesn’t have multipath support and it will be done. When Quagga BGP announces route with multiple paths, then it is ODL responsibility to program Fib entries in all compute nodes where VPN instance blueprint is present, so that traffic can be load balanced between these two DC gateways. It requires changes in existing odl-fib.yang model (like below) to support multiple routes for same destination IP prefix.

BGPManager should be able to create vrf entry for the advertised IP prefix with multiple route paths. VrfEntryListener listens to DCN on these vrf entries and program Fib entries (21) based on number route paths available for given IP prefix. For the given (external) destination IP prefix, if there is only one route path exists, use the existing approach to program FIB table flow entry matches on (vpnid, ipv4_dst) and actions with push MPLS label and output to gre tunnel port. For the given (external) destination IP prefix, if there are two route paths exist, then retrieve next hop ip address from routes list in the same sorted order (i.e. using same logic which is used to create buckets for load balancing next hop group for DC- Gateway IP addresses), then program FIB table flow entry with an instruction like Fig 3. It should have two set field actions where first action sets MPLS label to NX_REG0 for first sorted DC-GW IP address and second action sets MPLS label to NX_REG1 for the second sorted DC-GW IP address. When more than two DC Gateways are used, then more number of NXM Registries have to be used to push appropriate MPLS label before sending it to next hop group. It needs operational DS container to have mapping between DC Gateway IP address and NXM_REG. When one of the route is withdrawn for the IP prefix, then modify the FIB table flow entry with with push MPLS label and output to the available gre tunnel port.

ECMP for Intra-DC L3VPN communication¶

ECMP within data center is required to load balance the data traffic when extra route can be reached through multiple next hops (i.e. Nova VMs) when these are connected with different compute nodes. It mainly deals with how Compute Nodes can spray the traffic when dest IP prefix can be reached through two or more VMs (next hops) which are connected with multiple compute nodes.

When there are multiple RDs (if VPN is of type BGP VPN) assigned to VPN instance so that VPN engine can be advertise IP route with different RDs to achieve ECMP behavior in DC-GW as mentioned before. But for intra-DC, this doesn’t make any more sense since it’s all about programming remote FIB entries on computes nodes to achieve data traffic spray behavior.

Irrespective of RDs, when multiple next hops (which are from different Compute Nodes) are present for the extra-route adjacency, then FIB Manager has to create load balancing next hop group in remote compute node with buckets pointing with targeted Compute Node VxLAN tunnel ports.

To allocate group id for this load balancing next hop, the same destination IP prefix is used as the group key. The remote FIB table flow should point to this next hop group after writing prefix label into tunnel_id. The bucket weight of remote next hop is adjusted according to number of VMs associated to given extra route and on which compute node the VMs are connected. For example, two compute node having one VM each, then bucket weight is 50 each. One compute node having two VMs and another compute node having one VM, then bucket weight is 66 and 34 each. The hop-count property in vrfEntry data store helps to decide what is the bucket weight for each bucket.

ECMP Path decision based on Internal/External Tunnel Monitoring¶

ODL will use GRE-KA or BFD protocol to implement monitoring of GRE external tunnels. This implementation detail is out of scope in this document. Based on the tunnel state, GRE Load Balancing Group is adjusted accordingly as mentioned like below.

GRE tunnel state handling¶

As soon as GRE tunnel interface is created in ODL, interface manager uses alivenessmonitor to monitor the GRE tunnels for its liveness using GRE Keep-alive protocol. When tunnel state changes, it has to handled accordingly to adjust above load balancing group so that data traffic is sent to only active DC-GW tunnel. This can be done with listening to update StateTunnelList DCN.

When one GRE tunnel is operationally going down, then retrieve the corresponding bucket from the load balancing group and delete it. When GRE tunnel comes up again, then add bucket back into load balancing group and reprogram it.

When both GRE tunnels are going down, then just recreate load balancing group with empty. Withdraw the routes from that particular DC-GW. With the above implementation, there is no need of modifying Fib entries for GRE tunnel state changes.

But when BGP Quagga withdrawing one of the route for external IP prefix, then reprogram FIB flow entry (21) by directly pointing to output=<gre_port> after pushing MPLS label.

VxLAN tunnel state handling¶

Similarly, when VxLAN tunnel state changes, the Load Balancing Groups in Compute Nodes have to be updated accordingly so that traffic can flow through active VxLAN tunnels. It can be done by having config mapping between target data-path-id to next hop group Ids and vice versa.

For both GRE and VxLAN tunnel monitoring, L3VPN has to implement the following YANG model to update load balancing next hop group buckets according to tunnel status.

When one of the VxLAN/GRE tunnel is going down, then retrieve nexthop-key from dpid-l3vpn-lb-nexthops by providing tep-device-ids from src-info and dst-info of StateTunnelList while handling its update DCN.

After retrieving next hop key, fetch target-device-id list from l3vpn-lb-nexthops and reprogram VxLAN/GRE load balancing group in each remote Compute Node based on tunnel state between source and destination Compute Node. Similarly, when tunnel comes up, then logic have to be rerun to add its bucket back into Load balancing group.

Assumptions¶

The support for action move:NXM_NX_REG0(1) -> MPLS label is already available in Compute Node.

Reboot Scenarios¶

This feature support all the following Reboot Scenarios for EVPN:

• Entire Cluster Reboot
• Leader PL reboot
• Candidate PL reboot
• OVS Datapath reboots
• Multiple PL reboots
• Multiple Cluster reboots
• Multiple reboots of the same OVS Datapath.
• Openstack Controller reboots

Clustering considerations¶

The feature should operate in ODL Clustered environment reliably.

Other Infra considerations¶

N.A.

Security considerations¶

N.A.

Scale and Performance Impact¶

Not covered by this Design Document.

Targeted Release¶

Carbon.

Alternatives¶

Alternatives considered and why they were not selected.

Features to Install¶

This feature doesn’t add any new karaf feature.

REST API¶

Assignee(s)¶

Primary assignee(s):

• Manu B <manu.b@ericsson.com>
• Kency Kurian <kency.kurian@ericsson.com>
• Gobinath <gobinath@ericsson.com>
• P Govinda Rajulu <p.govinda.rajulu@ericsson.com>

Other contributors:

• Periyasamy Palanisamy <periyasamy.palanisamy@ericsson.com>

Work Items¶

The Trello cards have already been raised for this feature under l3vpn_ecmp.

Link for the Trello Card: https://trello.com/c/8E3LWIkq/121-ecmp-support-for-bgp-based-l3vpn-l3vpn-ecmp

Unit Tests¶

Appropriate UTs will be added for the new code coming in once framework is in place.

Integration Tests¶

There won’t be any Integration tests provided for this feature.

CSIT¶

CSIT will be enhanced to cover this feature by providing new CSIT tests.

Use Cases¶

• Tracking East/West communication between local VMs.
• Tracking East/West communication between VMs that are located in different compute nodes.
• Tracking communication between a local VM and an IP located in an external network.
• Tracking TCP/UDP traffic sent from/to a VM.
• Tracking dropped packets between 2 VMs.

Pipeline changes¶

Two new tables will be used: table 219 for outgoing traffic from the VM, and table 249 for incoming traffic from the VM. In both ingress and egress pipelines, the counter service will be just after the appropriate ACL service. The default rule will resubmit traffic to the appropriate dispatcher table.

Assuming we want statistics on VM A traffic, received or sent from VM B.

Yang changes¶

Netvirt Statistics module will be enhanced with the following RPC:

grouping result {
    list counterResult {
        key id;
        leaf id {
            type string;
        }
        list groups {
            key name;
            leaf name {
                type string;
            }
            list counters {
                key name;
                leaf name {
                    type string;
                }
                leaf value {
                    type uint64;
                }
            }
        }
    }
}

grouping filters {
    leaf-list groupFilters {
        type string;
    }
    leaf-list counterFilter {
        type string;
    }
}

grouping elementRequestData {
    container filters {
            container tcpFilter {
            leaf on {
                type boolean;
            }
            leaf srcPort {
                type int32;
                default -1;
            }
            leaf dstPort {
                type int32;
                default -1;
            }
        }

        container udpFilter {
            leaf on {
                type boolean;
            }
            leaf dstPort {
                type int32;
                default -1;
            }
            leaf srcPort {
                type int32;
                default -1;
            }
        }

        container ipFilter {
            leaf ip {
                type string;
                default "";
            }
        }
    }
}

container elementCountersRequestConfig {
    list counterRequests {
        key "requestId";
        leaf requestId {
            type string;
        }
        leaf lportTag {
            type int32;
        }
        leaf dpn {
            type uint64;
        }
        leaf portId {
            type string;
        }
        leaf trafficDirection {
            type string;
        }
        uses elementRequestData;
    }
}

rpc acquireElementCountersRequestHandler {
    input {
        leaf portId {
            type string;
        }
        container incomingTraffic {
            uses elementRequestData;
        }
        container outgoingTraffic {
            uses elementRequestData;
        }
        uses filters;
    }
    output {
        leaf incomingTrafficHandler {
            type string;
        }
        leaf outcoingTrafficHandler {
            type string;
        }
    }
}

rpc releaseElementCountersRequestHandler {
    input {
        leaf handler {
            type string;
        }
    }
    output {
    }
}

rpc getElementCountersByHandler {
    input {
        leaf handler {
            type string;
        }
    }
    output {
        uses result;
    }
}

Configuration impact¶

The described above YANG model will be saved in the data store.

Clustering considerations¶

None

Other Infra considerations¶

None

Security considerations¶

None

Scale and Performance Impact¶

Since adding the new service is done by a request (as well as removing it), not all packets will be sent to the new tables described above.

Targeted Release¶

Carbon

Alternatives¶

None

Features to Install¶

odl-netvirt-openstack

REST API¶

CLI¶

Assignee(s)¶

Primary assignee:

Guy Regev <guy.regev@hpe.com>

Other contributors:

TBD

Work Items¶

https://trello.com/c/88MnwGwb/129-element-to-element-counters

• Add new service in Genius.
• Implement new rules installation.
• Update Netvirt Statistics module to support the new counters request.

Unit Tests¶

Integration Tests¶

CSIT¶

Use Cases¶

• Local and East/West communication between VMs co-located on the same compute node using associated floating ip.
• Local and East/West communication between VMs located on different compute nodes using associated floating ip.

Pipeline changes¶

Yang changes¶

odl-nat module will be enhanced with the following container

container external-subnets {
  list subnets  {
    key id;
    leaf id {
       type yang:uuid;
    }
    leaf vpnid {
       type yang:uuid;
    }
    leaf-list router-ids {
       type yang:uuid;
    }
    leaf external-network-id {
       type yang:uuid;
    }
  }
}

This model will be filled out only for flat/VLAN external network provider types. If this model is missing, vpn-id will be taken from odl-nat:external-networks model to maintain compatibility with BGPVPN models.

odl-nat:ext-routers container will be enhanced with the list of the external subnet-ids associated with the router.

container ext-routers {
  list routers {
    key router-name;
    leaf router-name {
      type string;
    }
    ...

    leaf-list external-subnet-id {
      type yang:uuid; }
    }
  }
}

Configuration impact¶

None

Clustering considerations¶

None

Other Infra considerations¶

None

Security considerations¶

None

Scale and Performance Impact¶

None

Targeted Release¶

Carbon

Alternatives¶

None

Create external network with two subnets¶

neutron net-create public-net -- --router:external --is-default --provider:network_type=flat
--provider:physical_network=physnet1
neutron subnet-create --ip_version 4 --gateway 10.64.0.1 --name public-subnet1 <public-net-uuid> 10.64.0.0/16
-- --enable_dhcp=False
neutron subnet-create --ip_version 4 --gateway 10.65.0.1 --name public-subnet2 <public-net-uuid> 10.65.0.0/16
-- --enable_dhcp=False

Create internal networks with subnets¶

neutron net-create private-net1
neutron subnet-create --ip_version 4 --gateway 10.0.123.1 --name private-subnet1 <private-net1-uuid>
10.0.123.0/24
neutron net-create private-net2
neutron subnet-create --ip_version 4 --gateway 10.0.124.1 --name private-subnet2 <private-net2-uuid>
10.0.124.0/24
neutron net-create private-net3
neutron subnet-create --ip_version 4 --gateway 10.0.125.1 --name private-subnet3 <private-net3-uuid>
10.0.125.0/24
neutron net-create private-net4
neutron subnet-create --ip_version 4 --gateway 10.0.126.1 --name private-subnet4 <private-net4-uuid>
10.0.126.0/24

Create two router instances and connect each router to one internal subnet and one external subnet¶

neutron router-create router1
neutron router-interface-add <router1-uuid> <private-subnet1-uuid>
neutron router-gateway-set --fixed-ip subnet_id=<public-subnet1-uuid> <router1-uuid> <public-net-uuid>
neutron router-create router2
neutron router-interface-add <router2-uuid> <private-subnet2-uuid>
neutron router-gateway-set --fixed-ip subnet_id=<public-subnet2-uuid> <router2-uuid> <public-net-uuid>

Create router instance connected to both external subnets and the remaining internal subnets¶

neutron router-create router3
neutron router-interface-add <router3-uuid> <private-subnet3-uuid>
neutron router-interface-add <router3-uuid> <private-subnet4-uuid>
neutron router-gateway-set --fixed-ip subnet_id=<public-subnet1-uuid> --fixed-ip subnet_id=<public-subnet2-uuid>
<router3-uuid> <public-net-uuid>