OpFlex agent-ovs User Guide

Introduction

agent-ovs is a policy agent that works with OVS to enforce a group-based policy networking model with locally attached virtual machines or containers. The policy agent is designed to work well with orchestration tools like OpenStack.

Agent Configuration

The agent configuration is handled using its config file which is by default found at “/etc/opflex-agent-ovs/opflex-agent-ovs.conf”

Here is an example configuration file that documents the available options:

{
    // Logging configuration
    // "log": {
    //     // Set the log level.
    //     // Possible values in descending order of verbosity:
    //     // "debug5"-"debug0", "debug" (synonym for "debug0"),
    //     // "info", "warning", "error", "fatal"
    //     // Default: "info"
    //     "level": "info"
    // },

    // Configuration related to the OpFlex protocol
    "opflex": {
        // The policy domain for this agent.
        "domain": "openstack",

        // The unique name in the policy domain for this agent.
        "name": "example-agent",

        // a list of peers to connect to, by hostname and port.  One
        // peer, or an anycast pseudo-peer, is sufficient to bootstrap
        // the connection without needing an exhaustive list of all
        // peers.
        "peers": [
            // EXAMPLE:
            // {"hostname": "10.0.0.30", "port": 8009}
        ],

        "ssl": {
            // SSL mode.  Possible values:
            // disabled: communicate without encryption (default)
            // encrypted: encrypt but do not verify peers
            // secure: encrypt and verify peer certificates
            "mode": "encrypted",

            // The path to a directory containing trusted certificate
            // authority public certificates, or a file containing a
            // specific CA certificate.
            // Default: "/etc/ssl/certs"
            "ca-store": "/etc/ssl/certs"
        },

        "inspector": {
            // Enable the MODB inspector service, which allows
            // inspecting the state of the managed object database.
            // Default: true
            "enabled": true,

            // Listen on the specified socket for the inspector
            // Default: "/var/run/opflex-agent-ovs-inspect.sock"
            "socket-name": "/var/run/opflex-agent-ovs-inspect.sock"
        },

        "notif": {
            // Enable the agent notification service, which sends
            // notifications to interested listeners over a UNIX
            // socket.
            // Default: true
            "enabled": true,

            // Listen on the specified socket for the inspector
            // Default: "/var/run/opflex-agent-ovs-notif.sock"
            "socket-name": "/var/run/opflex-agent-ovs-notif.sock",

            // Set the socket owner user after binding if the user
            // exists
            // Default: do not set the owner
            // "socket-owner": "root",

            // Set the socket group after binding if the group name
            // exists
            // Default: do not set the group
            "socket-group": "opflexep",

            // Set the socket permissions after binding to the
            // specified octal permissions mask
            // Default: do not set the permissions
            "socket-permissions": "770"
        }
    },

    // Endpoint sources provide metadata about local endpoints
    "endpoint-sources": {
        // Filesystem path to monitor for endpoint information
        // Default: no endpoint sources
        "filesystem": ["/var/lib/opflex-agent-ovs/endpoints"]
    },

    // Service sources provide metadata about services that can
    // provide functionality for local endpoints
    "service-sources": {
        // Filesystem path to monitor for service information
        // Default: no service sources
        "filesystem": ["/var/lib/opflex-agent-ovs/services"]
    },

    // Renderers enforce policy obtained via OpFlex.
    // Default: no renderers
    "renderers": {
        // Stitched-mode renderer for interoperating with a
        // hardware fabric such as ACI
        // EXAMPLE:
        "stitched-mode": {
            // "Integration" bridge used to enforce contracts and forward
            // packets
            "int-bridge-name": "br-int",

            // "Access" bridge used to enforce access control and enforce
            // security groups.
            "access-bridge-name": "br-access",

            // Set encapsulation type.  Must set either vxlan or vlan.
            "encap": {
                // Encapsulate traffic with VXLAN.
                "vxlan" : {
                    // The name of the tunnel interface in OVS
                    "encap-iface": "br0_vxlan0",

                    // The name of the interface whose IP should be used
                    // as the source IP in encapsulated traffic.
                    "uplink-iface": "team0.4093",

                    // The vlan tag, if any, used on the uplink interface.
                    // Set to zero or omit if the uplink is untagged.
                    "uplink-vlan": 4093,

                    // The IP address used for the destination IP in
                    // the encapsulated traffic.  This should be an
                    // anycast IP address understood by the upstream
                    // stiched-mode fabric.
                    "remote-ip": "10.0.0.32",

                    // UDP port number of the encapsulated traffic.
                    "remote-port": 8472
                }

                // Encapsulate traffic with a locally-significant VLAN
                // tag
                // EXAMPLE:
                // "vlan" : {
                //     // The name of the uplink interface in OVS
                //     "encap-iface": "team0"
                // }
            },

            // Configure forwarding policy
            "forwarding": {
                // Configure the virtual distributed router
                "virtual-router": {
                    // Enable virtual distributed router.  Set to true
                    // to enable or false to disable.
                    // Default: true.
                    "enabled": true,

                    // Override MAC address for virtual router.
                    // Default: "00:22:bd:f8:19:ff"
                    "mac": "00:22:bd:f8:19:ff",

                    // Configure IPv6-related settings for the virtual
                    // router
                    "ipv6" : {
                        // Send router advertisement messages in
                        // response to router solicitation requests as
                        // well as unsolicited advertisements.  This
                        // is not required in stitched mode since the
                        // hardware router will send them.
                        "router-advertisement": false
                    }
                },

                // Configure virtual distributed DHCP server
                "virtual-dhcp": {
                    // Enable virtual distributed DHCP server.  Set to
                    // true to enable or false to disable.
                    // Default: true
                    "enabled": true,

                    // Override MAC address for virtual dhcp server.
                    // Default: "00:22:bd:f8:19:ff"
                    "mac": "00:22:bd:f8:19:ff"
                },

                "endpoint-advertisements": {
                    // Set mode for generation of periodic ARP/NDP
                    // advertisements for endpoints.  Possible values:
                    // disabled: Do not send advertisements
                    // gratuitous-unicast: Send gratuitous endpoint
                    //   advertisements as unicast packets to the router
                    //   mac.
                    // gratuitous-broadcast: Send gratuitous endpoint
                    //   advertisements as broadcast packets.
                    // router-request: Unicast a spoofed request/solicitation
                    //   for the subnet's gateway router.
                    // Default: router-request.
                    "mode": "gratuitous-broadcast"
                }
            },

            // Location to store cached IDs for managing flow state
            // Default: "/var/lib/opflex-agent-ovs/ids"
            "flowid-cache-dir": "/var/lib/opflex-agent-ovs/ids",

            // Location to write multicast groups for the mcast-daemon
            // Default: "/var/lib/opflex-agent-ovs/mcast/opflex-groups.json"
            "mcast-group-file": "/var/lib/opflex-agent-ovs/mcast/opflex-groups.json"
        }
    }
}

Endpoint Registration

The agent learns about endpoints using endpoint metadata files located by default in “/var/lib/opflex-agent-ovs/endpoints”.

These are JSON-format files such as the (unusually complex) example below:

{
    "uuid": "83f18f0b-80f7-46e2-b06c-4d9487b0c754",
    "policy-space-name": "test",
    "endpoint-group-name": "group1",
    "interface-name": "veth0",
    "ip": [
        "10.0.0.1", "fd8f:69d8:c12c:ca62::1"
    ],
    "dhcp4": {
        "ip": "10.200.44.2",
        "prefix-len": 24,
        "routers": ["10.200.44.1"],
        "dns-servers": ["8.8.8.8", "8.8.4.4"],
        "domain": "example.com",
        "static-routes": [
            {
                "dest": "169.254.169.0",
                "dest-prefix": 24,
                "next-hop": "10.0.0.1"
            }
        ]
    },
    "dhcp6": {
        "dns-servers": ["2001:4860:4860::8888", "2001:4860:4860::8844"],
        "search-list": ["test1.example.com", "example.com"]
    },
    "ip-address-mapping": [
        {
           "uuid": "91c5b217-d244-432c-922d-533c6c036ab4",
           "floating-ip": "5.5.5.1",
           "mapped-ip": "10.0.0.1",
           "policy-space-name": "common",
           "endpoint-group-name": "nat-epg"
        },
        {
           "uuid": "22bfdc01-a390-4b6f-9b10-624d4ccb957b",
           "floating-ip": "fdf1:9f86:d1af:6cc9::1",
           "mapped-ip": "fd8f:69d8:c12c:ca62::1",
           "policy-space-name": "common",
           "endpoint-group-name": "nat-epg"
        }
    ],
    "mac": "00:00:00:00:00:01",
    "promiscuous-mode": false
}

The possible parameters for these files are:

uuid

A globally unique ID for the endpoint

endpoint-group-name

The name of the endpoint group for the endpoint

policy-space-name

The name of the policy space for the endpoint group.

interface-name

The name of the OVS interface to which the endpoint is attached

ip

A list of strings contains either IPv4 or IPv6 addresses that the endpoint is allowed to use

mac

The MAC address for the endpoint’s interface.

promiscuous-mode

Allow traffic from this VM to bypass default port security

dhcp4

A distributed DHCPv4 configuration block (see below)

dhcp6

A distributed DHCPv6 configuration block (see below)

ip-address-mapping

A list of IP address mapping configuration blocks (see below)

DHCPv4 configuration blocks can contain the following parameters:

ip

the IP address to return with DHCP. Must be one of the configured IPv4 addresses.

prefix

the subnet prefix length

routers

a list of default gateways for the endpoint

dns

a list of DNS server addresses

domain

The domain name parameter to send in the DHCP reply

static-routes

A list of static route configuration blocks, which contains a “dest”, “dest-prefix”, and “next-hop” parameters to send as static routes to the end host

DHCPv6 configuration blocks can contain the following parameters:

dns

A list of DNS servers for the endpoint

search-patch

The DNS search path for the endpoint

IP address mapping configuration blocks can contain the following parameters:

uuid

a globally unique ID for the virtual endpoint created by the mapping.

floating-ip

Map using DNAT to this floating IPv4 or IPv6 address

mapped-ip

the source IPv4 or IPv6 address; must be one of the IPs assigned to the endpoint.

endpoint-group-name

The name of the endpoint group for the NATed IP

policy-space-name

The name of the policy space for the NATed IP

Inspector

The Opflex inspector is a useful command-line tool that will allow you to inspect the state of the managed object database for the agent for debugging and diagnosis purposes.

The command is called “gbp_inspect” and takes the following arguments:

# gbp_inspect -h
Usage: gbp_inspect [options]
Allowed options:
  -h [ --help ]                         Print this help message
  --log arg                             Log to the specified file (default
                                        standard out)
  --level arg (=warning)                Use the specified log level (default
                                        warning)
  --syslog                              Log to syslog instead of file or
                                        standard out
  --socket arg (=/usr/var/run/opflex-agent-ovs-inspect.sock)
                                        Connect to the specified UNIX domain
                                        socket (default /usr/var/run/opfl
                                        ex-agent-ovs-inspect.sock)
  -q [ --query ] arg                    Query for a specific object with
                                        subjectname,uri or all objects of a
                                        specific type with subjectname
  -r [ --recursive ]                    Retrieve the whole subtree for each
                                        returned object
  -f [ --follow-refs ]                  Follow references in returned objects
  --load arg                            Load managed objects from the specified
                                        file into the MODB view
  -o [ --output ] arg                   Output the results to the specified
                                        file (default standard out)
  -t [ --type ] arg (=tree)             Specify the output format: tree,
                                        asciitree, list, or dump (default tree)
  -p [ --props ]                        Include object properties in output

Here are some examples of the ways to use this tool.

You can get information about the running system using one or more queries, which consist of an object model class name and optionally the URI of a specific object. The simplest query is to get a single object, nonrecursively:

# gbp_inspect -q DmtreeRoot
───⦁ DmtreeRoot,/
# gbp_inspect -q GbpEpGroup
───⦁ GbpEpGroup,/PolicyUniverse/PolicySpace/test/GbpEpGroup/group1/
───⦁ GbpEpGroup,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/
# gbp_inspect -q GbpEpGroup,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/
───⦁ GbpEpGroup,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/

You can also display all the properties for each object:

# gbp_inspect -p -q GbpeL24Classifier
───⦁ GbpeL24Classifier,/PolicyUniverse/PolicySpace/test/GbpeL24Classifier/classifier4/
      {
        connectionTracking : 1 (reflexive)
        dFromPort          : 80
        dToPort            : 80
        etherT             : 2048 (ipv4)
        name               : classifier4
        prot               : 6
      }
───⦁ GbpeL24Classifier,/PolicyUniverse/PolicySpace/test/GbpeL24Classifier/classifier3/
      {
        etherT : 34525 (ipv6)
        name   : classifier3
        order  : 100
        prot   : 58
      }
───⦁ GbpeL24Classifier,/PolicyUniverse/PolicySpace/test/GbpeL24Classifier/classifier1/
      {
        etherT : 2054 (arp)
        name   : classifier1
        order  : 102
      }
───⦁ GbpeL24Classifier,/PolicyUniverse/PolicySpace/test/GbpeL24Classifier/classifier2/
      {
        etherT : 2048 (ipv4)
        name   : classifier2
        order  : 101
        prot   : 1
      }

You can also request to get the all the children of an object you query for:

# gbp_inspect -r -q GbpEpGroup,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/
──┬⦁ GbpEpGroup,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/
  ├──⦁ GbpeInstContext,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/GbpeInstContext/
  ╰──⦁ GbpEpGroupToNetworkRSrc,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/GbpEpGroupToNetworkRSrc/

You can also follow references found in any object downloads:

# gbp_inspect -fr -q GbpEpGroup,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/
──┬⦁ GbpEpGroup,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/
  ├──⦁ GbpeInstContext,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/GbpeInstContext/
  ╰──⦁ GbpEpGroupToNetworkRSrc,/PolicyUniverse/PolicySpace/common/GbpEpGroup/nat-epg/GbpEpGroupToNetworkRSrc/
──┬⦁ GbpBridgeDomain,/PolicyUniverse/PolicySpace/common/GbpBridgeDomain/bd_ext/
  ╰──⦁ GbpBridgeDomainToNetworkRSrc,/PolicyUniverse/PolicySpace/common/GbpBridgeDomain/bd_ext/GbpBridgeDomainToNetworkRSrc/
──┬⦁ GbpFloodDomain,/PolicyUniverse/PolicySpace/common/GbpFloodDomain/fd_ext/
  ╰──⦁ GbpFloodDomainToNetworkRSrc,/PolicyUniverse/PolicySpace/common/GbpFloodDomain/fd_ext/GbpFloodDomainToNetworkRSrc/
──┬⦁ GbpRoutingDomain,/PolicyUniverse/PolicySpace/common/GbpRoutingDomain/rd_ext/
  ├──⦁ GbpRoutingDomainToIntSubnetsRSrc,/PolicyUniverse/PolicySpace/common/GbpRoutingDomain/rd_ext/GbpRoutingDomainToIntSubnetsRSrc/152/%2fPolicyUniverse%2fPolicySpace%2fcommon%2fGbpSubnets%2fsubnets_ext%2f/
  ╰──⦁ GbpForwardingBehavioralGroupToSubnetsRSrc,/PolicyUniverse/PolicySpace/common/GbpRoutingDomain/rd_ext/GbpForwardingBehavioralGroupToSubnetsRSrc/
──┬⦁ GbpSubnets,/PolicyUniverse/PolicySpace/common/GbpSubnets/subnets_ext/
  ├──⦁ GbpSubnet,/PolicyUniverse/PolicySpace/common/GbpSubnets/subnets_ext/GbpSubnet/subnet_ext4/
  ╰──⦁ GbpSubnet,/PolicyUniverse/PolicySpace/common/GbpSubnets/subnets_ext/GbpSubnet/subnet_ext6/