WO2016123314A1

WO2016123314A1 - Data loop determination in a software-defined network

Info

Publication number: WO2016123314A1
Application number: PCT/US2016/015322
Authority: WO
Inventors: Celestian K. SEBASTIAN; Prasanth GOPINATHAN NAIR SARASWATHYAMMA; Vijeesh ERANKOTTE PANAYAMTHATTA
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2015-01-28
Filing date: 2016-01-28
Publication date: 2016-08-04

Abstract

In some examples, a method includes installing, with a software-defined network (SDN) controller in an SDN containing a plurality of controlled network nodes, a flow rule at a controlled network node to forward a time-to-live (TTL) expired packet to the SDN controller; determining which of the plurality of controlled network nodes are involved with a loop in the SDN by injecting a controlled network node with a series of marker packets having equivalent packet headers as the TTL expired packet; determining that a flow rule used for a marker packet by a given controlled network node does not correspond to a flow rule for the marker packet in an SDN controller database; and adjusting a flow rule on the given controlled network node to remedy the loop.

Description

DATA LOOP DETERMINATION IN A SOFTWARE-DEFINED NETWORK BACKGROUN D

[0001] Computer networks can be used to allow networked devices, such as personal computers, servers, and data storage devices to exchange data. Computer networks often include intermediary datapath devices such as network switches, gateways, and routers, to flow traffic along selected data routing paths between networked devices. A data routing path can, for example, be selected by a network controller, administrator, or another entity, and can, for example, be based on network conditions, network equipment capabilities, or other factors.

BRIEF DESCRIPTION OF DRAWINGS

[0002] FIG. 1 is a diagram of an example software-defined network with traffic along first and second example datapaths.

[0003] FIG. 2 is a flowchart outlining an example method performed by an example

software-defined network controller.

[0004] FIG. 3 is a diagram of an example software-defined network controller.

[0005] FIG. 4 is a diagram of an example software-defined network controller connected to an example network switch.

[0006] FIG. 5 is a diagram of an example machine-readable storage medium.

[0007] FIG. 6 is a diagram of an example computing system including the example machine- readable storage medium of FIG. 5.

DETAILED DESCRIPTION

[0008] As described above, computer networks can be used to allow networked devices to exchange data over selected data routing paths. In a software-defined network (SDN), control decisions for routing traffic through the network can be decoupled from the network's physical infrastructure. For example, an SDN controller can be programmed to decide how to route packets over the network and to inform controlled intermediary datapath devices, such as network switches, about these decisions. One way in which an SDN controller can inform such devices about its decisions is to install flow rules onto hardware of the devices (e.g., within a Ternary Content Addressable Memory (TCAM) of a controlled network switch). Such flow rules can, for example, be carefully determined by the SDN controller so as to prevent data loops or other unintended network routing behavior.

[0009] In some SDN networks, other entities, such as a network administrator or another network device can install their own flow rules onto devices which may conflict with flow rules installed by the SDN controller. For example, in some network devices, if there are two matching flow rules for a given packet header, the flow rule with a higher priority will be executed in lieu of a flow rule with a lower priority. In some situations, flow rules installed by another entity can conflict with flow rules installed by the SDN controller and can result in data loops or other unintended network routing behavior.

[0010] Certain implementations of the present disclosure seek to address the above issues by leveraging an SDN controller's ability to inject packets and install flow rules onto controlled network nodes. As but one example, and as described in further detail below, an SDN controller can be programmed to install a flow rule at a controlled network node to forward a time-to-live (TTL) expired packet to the SDN controller. The SDN controller can then determine which of the plurality of controlled network nodes are involved with a loop in the SDN by injecting a controlled network node with a series of marker packets having equivalent packet headers as the TTL expired packet and increasing TTL values. In this example implementation, the SDN controller can then determine that a flow rule used for a marker packet by a given controlled network node does not correspond to a flow rule for the marker packet in an SDN controller database and can adjust a flow rule on the given controlled network node to remedy the loop.

[0011] FIG. 1 is a diagram of an example software-defined network (SDN) 100 including an SDN controller 102 having various modules 122, 124, 126, 128, and 130. The structure and functionality of the various modules of SDN controller 102 are described in detail below with respect to FIG. 3. FIG. 1 depicts traffic along a first example datapath between a source node 132 and a destination node 134, the first datapath being defined by network nodes 104, 108, 110, 116, 118, and 120. FIG. 1 also depicts traffic along a second example datapath between source node 132 towards destination node 134. The second datapath is defined by network nodes 104, 108, 110, 116, and 112. As depicted in FIG. 1, the second datapath results in an infinite data loop including nodes 110, 116, and 112, with data packets in the loop eventually expiring in accordance with their respective time-to-live (TTL) expiration value.

[0012] As described above, in an SDN, control decisions for routing traffic through the network can be decoupled from the network's physical infrastructure. For example, SDN controller 102 can be used to instruct network nodes to flow traffic along a selected routing path defined by the nodes. In some implementations, these nodes can, for example, be in the form of network switches or other intermediary network devices. The use of such software-defined networking can provide other functionality. For example, one or more SDN applications can be installed on or interface with SDN controller 102 to meet customer use cases, such as to achieve a desired throughput (or another quality of service) over SDN 100, enforce security provisions for SDN 100, or provide another suitable service or functionality.

[0013] The functionality of SDN controller 102 can, for example, be implemented in part via a software program on a standalone machine, such as a standalone server. In some implementations, SDN controller 102 can be implemented on multi-purpose machines, such as a suitable desktop computer, laptop, tablet, or the like. In some implementations, SDN controller 102 can be implemented on a suitable non-host network node, such as certain types of network switches. It is appreciated that the functionality of SDN controller 102 may be split among multiple controllers or other devices. For example, SDN 100 is described and illustrated as including only one SDN controller 102. However, it is appreciated that the disclosure herein can be implemented in SDNs with multiple controllers. For example, in some SDNs, network devices are in communication with multiple controllers such that control of the network can be smoothly handed over from a first controller to a second controller if a first controller fails or is otherwise out of operation. As another example, multiple controllers can work together to concurrently control an SDN. In such SDNs, a first controller can, for example, control certain network devices while a second controller can control other network devices. In view of the above, reference in this application to a single SDN controller 102 that controls the operation of SDN 100 is intended to include such multiple controller configurations (and other suitable multiple controller configurations).

[0014] Source node 132 and destination node 134 can, for example, be in the form of

network hosts or other types of network nodes. For example, one or both of source node 132 and destination node 134 can be in the form of suitable servers, desktop computers, laptops, printers, etc. As but one example, source node 132 can be in the form of a desktop computer including a monitor for presenting information to an operator and a keyboard and mouse for receiving input from an operator, and destination node 134 can be in the form of a standalone storage server appliance. It is appreciated that source node 132 and destination node 134 can be endpoint nodes on SDN 100, intermediate nodes between endpoint nodes, or other types of network nodes.

[0015] Nodes 104, 106, 108, 110, 112, 114, 116, 118, and 120 can, for example, be in the form of switches or other multi-port network bridges that process and forward data at the data link layer. In some implementations, one or more of the nodes can be in the form of multilayer switches that operate at multiple layers of the OSI model (e.g., the data link and network layers). Although the term "switch" is used throughout this description, it is appreciated that this term can refer broadly to other suitable network data forwarding devices. For example, a general purpose computer can include suitable hardware and machine-readable instructions that allow the computer to function as a network switch. It is appreciated that the term "switch" can include other network data path elements in the form of suitable routers, gateways and other devices that provide switch-like functionality for SDN 100.

[0016] Nodes within SDN 100 can forward traffic along a datapath based on metadata

within the traffic. For example, traffic received at the node can be in the form of a packet. For illustration, the term "packet" is used herein, however, it is appreciated that "packet" can refer to any suitable protocol data unit (PDU). The packet can, for example, include payload data as well as metadata in the form of control data.

Control data can, for example, provide data to assist the node with reliably delivering the payload data. For example, control data can include network addresses for source node 132 and destination node 134, error detection codes, sequencing information, packet size of the packet, a time-to-live (TTL) value, etc. In contrast, payload data can include data carried on behalf of an application for use by source node 132 or destination node 134. Each node within SDN 100, for example, help manage the flow of data across a network by only transmitting a received message to a destination device for which the message was intended (or to an intermediary device en route to the destination device). In some implementations, these nodes can rely on flow entries in flow tables stored on a machine-readable medium within each switch (or otherwise accessible by each switch). Each flow rule in a flow table can, for example, contain information such as: (1) match fields to match against packets (e.g., an ingress port and specific packet header fields), (2) a priority value for the flow rule to allow prioritization over other flow entries, (3) counters that are updated when packets are matched, (4) instructions to modify the action set or pipeline processing, (5) timeouts indicating a maximum amount of time or idle time before a flow is expired by the switch, and (6) a cookie value which can be used by the SDN controller to filter flow statistics, flow modification, and flow deletion.

[0017] The various nodes within SDN 100 are connected via one or more data channels, which can, for example be in the form of data cables or wireless data channels. Although a single link between each network node is illustrated, it is appreciated that each single link may include multiple wires or data channels. For illustration, FIG. 1 depicts SDN controller 102 as being connected to each network nodes via broken lines to illustrate control channels between SDN controller 102 and respective nodes, however it is appreciated that SDN controller 102 may be directly connected to only one or a few network nodes, while being indirectly connected to other nodes of SDN 100. As but one example, SDN controller 102 can be directly connected to node 106 via an Ethernet cable, while being indirectly connected to node 110 (e.g., by relying on node 106 as an intermediary for communication with node 110).

[0018] SDN 100 can, for example, be implemented through the use of an SDN controller 102 that interfaces with various SDN-compatible devices via a suitable Application Program Interface ("API"), or another suitable protocol (e.g., OpenFlow and/or simple network management protocol (SNMP)). In some implementations, SDN controller 102 may interface with controlled network devices via an interface channel that connects each controlled device to SDN controller 102 to allow SDN controller 102 to configure and manage each device, receive events from each device, and send packets using each device.

[0019] As used herein, the term "controlled" and similar terminology in the context of SDN- compatible network nodes, such as "controlled switches," is intended to include devices within the control domain of SDN controller 102 or otherwise controllable by SDN controller 102. Such a controlled node can, for example, communicate with SDN controller 102 and SDN controller 102 is able to manage the node in accordance with an SDN protocol, such as the OpenFlow protocol. For example, an OpenFlow- compatible switch controlled by SDN controller 102 can permit SDN controller 102 to add, update, and delete flow entries in flow tables of the switch using suitable SDN commands.

[0020] In the example SDN 100 depicted in FIG. 1, the various network nodes are in the form of intermediary nodes (controlled network switches 104, 106, 108, 110, 112, 114, 116, 118, and 120) and host devices (source node 132 and destination node 134). It is appreciated however, that the implementations described herein can be used or adapted for networks including more or fewer devices, different types of devices, and different network arrangements. It is further appreciated that the disclosure herein can apply to suitable SDNs (e.g., certain hybrid or heterogeneous SDNs) in which some devices are controlled by an SDN controller and some devices are not controlled by the SDN controller. For example, in some implementations, at least one node along a given datapath is controlled by SDN controller 102 and at least one node along the given datapath is not controlled by SDN controller 102.

[0021] FIG. 2 illustrates a flowchart for a method 136 relating to the use of an SDN

controller to determine and remedy a loop in an SDN. For illustration, the description of method 136 and its component steps make reference to example SDN 100 and elements thereof, such as for example SDN controller 102, node 110, source node 132, destination node 134, etc., however, it is appreciated that method 136 or aspects thereof can be used or otherwise applicable for any suitable network or network element described herein or otherwise. For example, method 136 can be applied to computer networks with different network topologies than those illustrated in FIG. 1.

[0022] Method 136 includes a step 138 of installing a flow rule at one or more controlled network nodes to forward TTL expired packets to SDN controller 102. For purposes of illustration, an example controlled network node referred to in step 138 can be node 110. In some implementations, SDN controller 102 installs the flow rules referred to in step 138 on all controlled network nodes within SDN 100, whereas in some implementations, the flow rules referred to in step 138 are only installed on a subset of controlled network nodes within SDN 100. It is appreciated that the selection of one or more controlled network nodes upon which to install the flow rule can be automatically decided by SDN controller 102 based on one or more network factors and/or can be manually selected by a network administrator. In some implementations, SDN controller 102 may be aware of a loop along a datapath between source node 132 and destination node 134 and may select the controlled network node to be any controlled network node between source node 132 and destination node 134. In some implementations, SDN controller 102 can select the controlled network node to be a first network node along a datapath between source node 132 and destination node 134 (e.g., node 104 in FIG. 1).

[0023] The flow rule installed by SDN controller 102 on network node 110 can, for example, provide an instruction to controlled network node 110 to forward to SDN controller 102 any packet having a TTL value of 1 or 0 (or another suitable TTL value indicating that the packet is expired). As used herein, the term "expired" can refer to packets that are either already expired (e.g., a TTL value of 0) or are near expiration (e.g., a TTL value of 1, 2, etc.). It is appreciated that the flow rule can provide for certain filters such that only some expired packets are forwarded to SDN controller 102. For example, in some implementations, SDN controller 102 can only be concerned with expired packets en route to a specific destination node or originating from a specific source node. As such, the flow rule can provide that controlled network node 110 only forward expired packets if they are en route to the specific destination node or originate from the specific source node. It is appreciated that other suitable forwarding rules can be put in place. As described above, SDN controller 102 may be in communication with controlled network node 110 via an intermediate network node (e.g., network node 112). In such a situation, the flow rule may instruct network node 110 to forward the expired packet to network node 112 en route to SDN controller 102.

[0024] Method 136 includes a step 140 of determining which of a plurality of controlled network nodes (e.g., nodes 110, 116, and 112 along second datapath) are involved with a loop in SDN 100 by injecting a controlled network node with a series of marker packets having equivalent packet headers as the TTL expired packet and increasing TTL values. In some implementations, the controlled network node (e.g., node 112) injected with the series of marker packets can, for example, be a different node on which SDN controller 102 installed instructions to forward TTL expired packets (i.e., node 110 in this example). However, for purposes of this example, the controlled network node injected with the series of marker packets is the same controlled network node on which SDN controller 102 installed instructions to forward TTL expired packets (i.e., node 110 in this example). It is appreciated that the selection of a controlled network node to inject with the series of marker packets can be made based on one or more network factors or other factors by SDN controller 102, a network administrator, and/or another entity.

[0025] As provided above, the series of marker packets have equivalent packet headers as the TTL expired packet. As used herein, the term "equivalent" can refer to packet headers having the same destination node or other relevant routing metadata. In some implementations, an equivalent packet header can include the same source node and the same destination node or other packet metadata. It is appreciated that in some situations, the marker packets may have identical packet headers as the TTL expired packets, whereas in other situations, the packet headers can merely be equivalent for purposes of analyzing nodes within SDN 100.

[0026] As provided above, the series of marker packets have increasing TTL values. In some implementations, step 140 includes injecting a first marker packet having a TTL value of 1, a second marker packet having a TTL value of 2, a third marker packet having a TTL value of 3, and so on. In some implementations, SDN controller 102 stops injecting packets when the given controlled network node reports expiration of a marker packet in the set of marker packets. It is appreciated that the TTL values can be integer values incremented by one unit for each TTL. In some implementations, the TTL values can be incremented by more than one unit (e.g., a first marker packet having a TTL value of 2, a second marker packet having a TTL value of 4, a third marker packet having a TTL value of 6, and so on). It is appreciated that the TTL increment value can be a fixed increment or a variable increment.

[0027] Method 136 includes a step 142 of determining that a flow rule used for a marker packet by a given controlled network node does not correspond to a flow rule for the marker packet in an SDN controller database. The term "SDN controller database" as used herein can, for example, refer to a flow rule database created by another entity and referenced by SDN controller 102, as well as a flow rule database created by SDN controller 102 and referenced by SDN controller 102. For example, an SDN controller database can include a flow rule for a packet having a specific source node 132 and a specific destination node 134. In accordance with step 142, SDN controller 102 can check whether the actual flow rule used for the marker packet by the given controlled node corresponds to the flow rule within the SDN controller database. This can, in some implementations, be accomplished by querying the given controlled network node with SDN controller 102 using a statistics collection feature and asking for the flow rule by the given controlled network node that was used to actually forward the marker packet.

[0028] With reference to SDN 100 depicted in FIG. 1, take for example a flow rule in the

SDN controller database that indicates that a packet en route to a given destination node (e.g., destination node 134) should be forwarded from the given controlled node (e.g., node 110) to a specific downstream node (e.g., node 116). In this example, suppose that the flow rule by the given controlled network node that was used to forward the marker packet indicates that the packet en route to the given destination node (e.g., destination node 134) is to be forwarded from the given controlled node (e.g., node 110) to a different downstream node (e.g., node 114). In such a situation, SDN controller 102 will then determine that the flow rule used for a marker packet by the given controlled network node does not correspond to the flow rule for the marker packet in its SDN controller database.

[0029] In some implementations, the given controlled network node (e.g., node 114)

involved in step 142 is a different node on which SDN controller 102 installed instructions to forward TTL expired packets (i.e., node 110 in this example) and/or which is injected with the series of marker packets (i.e., node 110 in this example). However, for purposes of this example, the given controlled network node involved in step 142 is the same controlled network node on which SDN controller 102 installed instructions to forward TTL expired packets (i.e., node 110 in this example). For example, in some implementations, the given controlled node is the first controlled network node along the routing path of the TTL expired packet (i.e., node 104 in this example). That is, the given controlled node does not necessarily need to be within a loop along the datapath. It is appreciated that the selection of the given controlled network node used in step 142 can be made based on one or more network factors or other factors by SDN controller 102, a network administrator, and/or another entity.

[0030] Method 136 includes a step 144 of adjusting a flow rule on the given controlled network node (node 110 in this example) to remedy the loop in response to determining that the flow rule used for the marker packet by the given controlled network node (node 110 in this example) does not correspond to a flow rule for the marker packet in an SDN controller database. It is appreciated that step 144 can include: (1) adjusting the flow rule used for the marker packet by the given controlled network node, (2) adjusting a flow rule for the marker packet not used by the given controlled node, or adjusting another flow rule stored on any controlled network node within SDN 100.

[0031] For example, in some implementations, step 144 includes increasing a priority of a flow rule for the marker packet installed by SDN controller 102 on the given controlled network node (node 110 in this example) to be greater than the flow rule used for the marker packet by the given controlled network node. That is, suppose the flow rule used for the marker packet by the given controlled network node was installed by a command line interface (CLI) by a network administrator and has a priority value of 1000. In such a situation, SDN 100 can include increasing the priority of a flow rule for the marker packet installed by SDN controller 102 to a priority value of 2000. As a result, the given controlled network node 110 will use the higher priority rule (i.e., the rule installed by SDN controller 102). As described above, flow rules installed by SDN controller 102 can be carefully selected so as to prevent data loops or other unintended network routing behavior. By using such flow rules installed by SDN controller 102, the loop identified by SDN controller 102 can be effectively remedied.

[0032] As described above, SDN controller 102 can adjust a priority of an existing flow rule.

It is appreciated that in some implementations, SDN controller 102 can install another flow rule in the given controlled network node having a higher priority instead of or in addition to adjust a priority of an existing flow rule. In some implementations, SDN controller 102 can delete flow rules on the given controlled network node 110, add flow rules, or otherwise modify flow rules in order to remedy the loop within SDN 100.

[0033] In some implementations, method 136 includes additional steps of (1) injecting, with SDN controller 102, the marker packet at the controlled network node (e.g., node 110), (2) receiving, with SDN controller 102, the marker packet from one of the plurality of controlled network nodes, and (3) determining, with SDN controller 102, that a loop is present in SDN 100 in response to receiving the marker packet. For example, it is appreciated that SDN controller 102 can receive the marker packet from a controlled network node (e.g., node 110) that is different than the controlled network node injected with the marker packet (e.g., node 112).

[0034] Although the flowchart of FIG. 2 shows a specific order of performance, it is

appreciated that this order may be rearranged into another suitable order, may be executed concurrently or with partial concurrence, or a combination thereof.

Likewise, suitable additional and/or comparable steps may be added to method 136 or other methods described herein in order to achieve the same or comparable functionality. In some implementations, one or more steps are omitted. For example, in some implementations, step 144 of adjusting the flow rule on the given controlled network node is omitted from method 136. It is appreciated that steps corresponding to additional or alternative functionality of other implementations described herein can be incorporated in method 136. For example, steps corresponding to the functionality of various modules of SDN controller 102 can be incorporated in method 136 even if such functionality is not explicitly characterized herein as a step in a method.

[0035] FIG. 3 illustrates SDN controller 102 in the form of functional modules that can, for example, be operative to execute one or more steps of method 136 or other methods described herein. As used herein, the term "module" refers to a combination of hardware (e.g., a processor such as an integrated circuit or other circuitry) and software (e.g., machine- or processor-executable instructions, commands, or code such as firmware, programming, or object code). A combination of hardware and software can include hardware only (i.e., a hardware element with no software elements), software hosted at hardware (e.g., software that is stored at a memory and executed or interpreted at a processor), or at hardware and software hosted at hardware. Additionally, as used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, the term "module" is intended to mean one or more modules or a combination of modules. Each module of SDN controller 102 can include one or more machine-readable storage mediums and one or more computer processors. For example, software that provides the functionality of modules on SDN controller 102 can be stored on a memory of a computer to be executed by a processor of the computer.

[0036] The implementation of SDN controller 102 of FIG. 3 includes flow rule installation module 122, packet header determination module 124, marker packet injection module 126, loop determination module 128, and flow rule comparison module 130, which were referenced above in FIG. 1 and are described in further detail below. It is appreciated that other modules can be added to SDN controller 102 for additional or alternative functionality. For example, another implementation of SDN controller 102 (described with respect to FIG. 4) includes additional modules, such as a communication module with a data port. [0037] Flow rule installation module 122 of SDN controller 102 includes a combination of hardware and software to allow SDN controller 102 to install flow entries at network nodes controlled by the SDN controller to forward time-to-live (TTL) expired packets to the SDN controller. Flow rule installation module 122 can, for example, include one or more machine-readable storage mediums and one or more computer processors to implement one or more aspects of step 1 (or other steps) of method 136 described above with respect to FIG. 2. In some implementations, flow rule installation module 122 is to install flow entries at network nodes controlled by the SDN controller to remedy a loop within SDN 100.

[0038] Packet header determination module 124 of SDN controller 102 includes a

combination of hardware and software to allow SDN controller 102 to determine a level 3 ("L3") packet header for a received TTL expired packet. As used herein, the term "level 3 packet header" can refer to the level 3 or network layer of the Open Systems Interconnection model (OSI) which partitions the internal functions of a communication system into various abstraction layers. This level can, for example, refer to the structuring and managing of a multi-node network, including addressing, routing and traffic control. Examples of L3 packets headers can include IPv4, I Pv6, IPsec, packet headers. Packet header determination module 124 can, for example, include one or more machine-readable storage mediums and one or more computer processors to implement one or more aspects of step 140 (and/or other steps) of method 136 described above with respect to FIG. 2.

[0039] With reference to a L3 packet header of an IPv4 packet as an example, such a packet can include header information such as: (1) a 4-bit "Version" field which can, for example, identify that the packet is an IPv4 version packet, (2) a 4-bit "Internet Header Length (IHL)" field, which can identify the number of 32-bit words in the header, (3) a 6-bit "Differentiated Services Code Point (DSCP)" field which can, for example, identify whether the packet is used for real-time data streaming (e.g., VoIP) or another differentiated service, (4) a 2-bit "Explicit Congestion Notification (ECN)" field, which can, for example, enable end-to-end notification of network congestion without dropping packets, (5) a 16-bit "Total Length" field, which can identify the entire packet size, including header and data, in bytes, (6) a 16-bit "Identification" field to uniquely identify a group of fragments of a single IP datagram, (7) a 3-bit "Flags" field to control and allow fragmentation, (8) a 13-bit "Fragment Offset" field to specify an offset of a particular fragment relative to the beginning of an original unfragmented IP datagram, (8) an 8-bit "Time To Live (TTL)" field to help prevent datagrams from persisting on a network by limiting a datagram's lifetime, (9) an 8-bit "Protocol" field to define a protocol used in the data portion of the IP datagram, (10) a 16-bit "Header Checksum" value used for error-checking of the header, (11) a 32- bit "Source IP address", (12) a 32-bit "Destination IP address, and (12) a 32-bit "Options" field used to identify various options for the IP datagram. It is appreciated that the above example L3 packet header and the description of its fields are used solely for illustration and that implementations of the disclosure provided herein can include additional, alternative, or fewer L3 packet header fields.

[0040] Marker packet injection module 126 of SDN controller 102 includes a combination of hardware and software to allow SDN controller 102 to inject a network node controlled by SDN controller 102 with a series of marker packets, wherein each marker packet of the series of marker packets has an equivalent L3 packet header as the received TTL expired packet and wherein the marker packets of the series of marker packets have increasing TTL values. Marker packet injection module 126 can, for example, include one or more machine-readable storage mediums and one or more computer processors to implement one or more aspects of step 140 (or other steps) of method 136 described above with respect to FIG. 2.

[0041] Loop determination module 128 of SDN controller 102 includes a combination of hardware and software to allow SDN controller 102 to determine controlled network nodes involved in a loop based on received TTL expired packets having equivalent L3 packet headers. Loop determination module 128 can, for example, include one or more machine-readable storage mediums and one or more computer processors to implement one or more aspects of step 140 (and/or other steps) of method 136 described above with respect to FIG. 2.

[0042] Flow rule comparison module 130 of SDN controller 102 includes a combination of hardware and software to allow SDN controller 102 to determine whether a flow rule used by a given controlled network node involved in the loop for a TTL expired packet corresponds to a flow rule for the marker packet in an SDN controller database for the given controlled network node or a flow rule otherwise expected by SDN controller 102 for the marker packet. Flow rule comparison module 130 can, for example, include one or more machine-readable storage mediums and one or more computer processors to implement one or more aspects of step 142 (and/or other steps) of method 136 described above with respect to FIG. 2.

[0043] FIG. 4 illustrates another example of an SDN controller 102 connected to an example network switch (e.g., network node 110) in the form of functional modules that can, for example, be operative to execute one or more steps of method 136 or other methods described herein. Network node 110 is referred to as a network switch for illustration, but it is appreciated that aspects of network node/switch 110 described herein can be incorporated in another suitable network data forwarding device. SDN controller 102 as depicted in FIG. 4 includes flow rule installation module 122, packet header determination module 124, marker packet injection module 126, loop determination module 128, and flow rule comparison module 130 as described above with respect to FIG. 3. SDN controller 102 of FIG. 4 further includes a communication module 146, Input/Output (I/O) module 148, and flow rule adjustment module 150 as described in further detail below. It is appreciated that various modules of SDN controller 102 of FIG. 3 are depicted in the implementation of SDN controller 102 of FIG. 4 for illustration. However, it is appreciated that SDN controller 102 of FIG. 4 can include alternative, additional, or fewer modules than SDN controller 102 described in FIG. 3. For example, in some implementations, SDN controller 102 of FIG. 4 does not include flow rule installation module 122. Example network switch 110 includes forwarding module 152 and communication module 154, as described in further detail below.

[0044] Communication modules 146 and 154 of SDN controller 102 and network switch 110 are a combination of hardware and software to allow networked communication between SDN controller 102, network switch 110, and other elements of SDN 100. Communication modules 146 and 154 can include matching or non-matching physical data ports 156, 158 to connect to elements of SDN 100. For example, in some implementations, communication modules 146, 154 can each include a network interface controller having an Ethernet port, whereas in some

implementations, communication module 146 can include an Ethernet port and communication module 154 can include a Fibre Channel port. In some

implementations, communication modules 146, 154 can include wired or wireless communication interface. Communication modules 146, 154 can, in some implementations, provide for virtual network ports. In some implementations, communication modules 146, 154 includes hardware in the form of a hard drive, related firmware, and other software for allowing the hard drive to operatively communicate with other hardware of SDN controller 102 or network switch 110. Communication modules 146, 154 can include information for use with

communication modules 146, 154, such as firmware for implementing physical or virtual network ports.

[0045] I/O module 148 of SDN controller 102 is a combination of hardware and software to allow an operator or other entity to view and/or interact with SDN controller 102. Example of suitable I/O modules can include modules for monitors, printers, keyboards, mouses, styluses, touchscreens, speakers, etc. I/O devices for such modules can be connected to elements of SDN controller 102 via wired or wireless links. It is appreciated that network switch 110 can also include an appropriate I/O module of its own.

[0046] Flow rule adjustment module 150 is a combination of hardware and software to adjust a flow rule on a controlled network node (e.g., network switch 110) to remedy a loop within SDN 100 in response to determining that a flow rule used for a marker packet by a controlled network node (e.g., network switch 110 or another node) does not correspond to a flow rule for the marker packet in an SDN controller database or otherwise expected by SDN controller 102.

[0047] Forwarding module 152 of network switch 110 includes a combination of hardware and software to allow network switch 110 to extract a set of fields from a received data packets to determine flow routing instructions and to forward the packet according to the flow routing instructions. Network switch 110 can, for example, include one or more machine-readable storage mediums and one or more computer processors, such as the mediums and processors described herein to implement one or more aspects of method 136 or other methods described herein.

[0048] It is appreciated that certain modules described herein or otherwise can, in some implementations, share hardware, software, or data with other modules. For example, in some implementations, communication module 146 of SDN controller 102 can share a computer-readable medium with loop determination module 128, whereas in some implementations, communication module 146 and loop determination module 128 use separate mediums. It is appreciated that any modules can share hardware, software, or data with any other module in order to achieve their respective objectives.

[0049] FIGs. 5 and 6 respectively illustrate example machine storage medium 164 (which stores expired packet forwarding instructions 166, loop determination instructions 168, and flow rule adjustment instructions 170) and an example computing system 160 that includes medium 164. Computing system 160 can, for example, be in the form of an SDN controller or can otherwise provide SDN controller functionality for a network by executing one or more steps of method 136 and/or other methods described herein. The description of computing system 160 refers to elements of SDN 100 for illustration, however, it is appreciated that computing system 160 can be used with any suitable network. Computing system 160 includes a processor 162 and machine-readable storage medium 164 as described further below. It is appreciated that computing system 160 can include additional elements, such as input/output (I/O) devices, a communication interface, etc. It is appreciated that one or hardware or software elements for SDN controller 102 described herein may be implemented in computing system 160. In some implementations, software that provides the functionality of SDN controller 102 can be stored on machine-readable storage medium 164 of computing system 160 to be executed by processor 162 of computing system 160.

[0050] Processor 162 of computing system 160 can, for example, be in the form of a central processing unit (CPU), a semiconductor-based microprocessor, a digital signal processor (DSP) such as a digital image processing unit, other hardware devices or processing elements suitable to retrieve and execute instructions stored in medium 164, or suitable combinations thereof. Processor 162 can, for example, include single or multiple cores on a chip, multiple cores across multiple chips, multiple cores across multiple devices, or suitable combinations thereof. Processor 162 can be functional to fetch, decode, and execute instructions as described herein. As an alternative or in addition to retrieving and executing instructions, processor 162 can, for example, include at least one integrated circuit (IC), other control logic, other electronic circuits, or suitable combination thereof that include a number of electronic components for performing the functionality of instructions stored on medium 164. Processor 162 can, for example, be implemented across multiple processing units and instructions may be implemented by different processing units in different areas of computing system 160.

[0051] Medium 164 of computing system 160 can, for example, be in the form of a non- transitory machine-readable storage medium, such as a suitable electronic, magnetic, optical, or other physical storage apparatus to contain or store information such as machine-readable instructions 166, 168, and 170. Such instructions can be operative to perform one or more functions described herein, such as those described above with respect to method 136 or other methods described herein. Medium 164 can include additional network relation information, such as for example, data, such as network performance data relating to SDN 100.

[0052] Medium 164 can, for example, be housed within the same housing as processor 162 for computing system 160, such as within a computing tower case for computing system 160. In some implementations, medium 164 and processor 162 are housed in different housings. As used herein, the term "machine-readable storage medium" can, for example, include Random Access Memory (RAM), flash memory, a storage drive (e.g., a hard disk), any type of storage disc (e.g., a Compact Disc Read Only Memory (CD-ROM), any other type of compact disc, a DVD, etc.), and the like, or a combination thereof. In some implementations, medium 164 can correspond to a memory including a main memory, such as a Random Access Memory (RAM), where software may reside during runtime, and a secondary memory. The secondary memory can, for example, include a nonvolatile memory where a copy of machine- readable instructions are stored. It is appreciated that instructions and data can be stored on separate machine-readable storage mediums and multiple mediums can be treated as a single medium 164 for purposes of description.

[0053] Expired packet forwarding instructions 166 can, for example, run on computing system 160 to install, at a plurality of network nodes controlled by SDN controller 102, a flow rule to forward TTL expired packets to SDN controller 102. Instructions 166 can incorporate one or more aspects of step 138 (and/or other steps) of method 136 described above and/or one or more aspects of flow rule installation module 122 (and/or other modules) of SDN controller 102 or vice versa.

[0054] Loop determination instructions 168 can, for example, run on computing system 160 to determine which of the plurality of controlled network nodes are involved with a loop in the SDN by injecting a controlled network node with a series of marker packets having equivalent routing headers (e.g., L3 routing headers) as the TTL expired packet, wherein the series of marker packets have increasing TTL values. Instructions 168 can incorporate one or more aspects of step 140 (and/or other steps) of method 136 described above and/or one or more aspects of packet header determination module 124, marker packet injection module 126, and loop determination module 128 (and/or other modules) of SDN controller 102 or vice versa.

[0055] Flow rule adjustment instructions 170 can, for example, run on computing system 160 to adjust a flow rule on a given controlled network node to remedy a loop in response to determining that a flow rule used for an injected marker packet does not correspond to a flow rule for the marker packet provided by SDN controller 102. Instructions 170 can incorporate one or more aspects of step 142 (and/or other steps) of method 144 described above and/or one or more aspects of flow rule adjustment module 150 (and/or other modules) of SDN controller 102 or vice versa.

[0056] While certain implementations have been shown and described above, various changes in form and details may be made. For example, some features that have been described in relation to one implementation and/or process can be related to other implementations. In other words, processes, features, components, and/or properties described in relation to one implementation can be useful in other implementations. As another example, functionalities discussed above in relation to specific modules or elements can be included at different modules, engines, or elements in other implementations.

[0057] As used herein, the term "provide" includes push mechanisms (e.g., sending data independent of a request for that data), pull mechanisms (e.g., delivering data in response to a request for that data), and store mechanisms (e.g., storing data at an intermediary at which the data can be accessed). Furthermore, as used herein, the term "based on" means "based at least in part on." Thus, a feature that is described based on some cause, can be based only on the cause, or based on that cause and on one or more other causes.

[0058] Furthermore, it should be understood that the systems, networks, and methods described herein can include various combinations and/or sub-combinations of the components and/or features of the different implementations described. Thus, features described with reference to one or more implementations can be combined with other implementations described herein.

Claims

CLAIMS What is claimed is:

1. A method comprising: installing, with a software-defined network (SDN) controller in an SDN containing a plurality of controlled network nodes, a flow rule at a controlled network node to forward a time-to-live (TTL) expired packet to the SDN controller; determining which of the plurality of controlled network nodes are involved with a data loop in the SDN by injecting a controlled network node with a series of marker packets having equivalent packet headers as the TTL expired packet, wherein the series of marker packets have increasing TTL values; determining that a flow rule used for a marker packet by a given controlled network node does not correspond to a flow rule for the marker packet in an SDN controller database; and adjusting a flow rule on the given controlled network node to remedy the data loop in response to determining that the flow rule used for the marker packet by the given controlled network node does not correspond to a flow rule for the marker packet in an SDN controller database.

2. The method of claim 1, wherein the given controlled network node is the same node injected with the series of marker packets having increasing TTL values.

3. The method of claim 1, wherein adjusting a flow rule on the given controlled network node to remedy the data loop includes adjusting the flow rule used for the marker packet by the given controlled network node.

4. The method of claim 1, wherein adjusting a flow rule on the given controlled network node to remedy the data loop includes adjusting a flow rule for the marker packet not used by the given controlled node.

5. The method of claim 1, wherein adjusting a flow rule on the given controlled network node to remedy the data loop includes increasing a priority of a flow rule for the marker packet installed by the SDN controller on the given controlled network node to be greater than the flow rule used for the marker packet by the given controlled network node.

6. The method of claim 1, further comprising:

injecting, with the SDN controller, the marker packet at a controlled network node; receiving, with the SDN controller, the marker packet from one of the plurality of controlled network nodes; and

determining, with the SDN controller, that a data loop is present in the SDN in response to receiving the marker packet.

7. The method of claim 1, wherein the given controlled network node is the network node that forwarded the TTL expired packet to the SDN controller.

8. The method of claim 1, wherein the given controlled node is the first controlled network node along the routing path of the TTL expired packet.

9. The method of claim 1, wherein determining which of the plurality of controlled network nodes are involved with a data loop in the SDN includes injecting, at the given controlled network node with the SDN controller, a first marker packet having a TTL value of 1, and

wherein the SDN controller stops injecting packets when the given controlled network node reports expiration of a marker packet in the set of marker packets.

10. A software-defined network (SDN) controller comprising:

a flow rule installation module to install flow entries at network nodes controlled by the SDN controller to forward time-to-live (TTL) expired packets to the SDN controller; a packet header determination module to determine a level 3 (L3) packet header for a received TTL expired packet;

a marker packet injection module to inject a network node controlled by the SDN controller with a series of marker packets, wherein each marker packet of the series of marker packets has an equivalent L3 packet header as the received TTL expired packet and wherein the marker packets of the series of marker packets have increasing TTL values; a loop determination module to determine controlled network nodes involved in a data loop based on received TTL expired packets having equivalent L3 packet headers; and a flow rule comparison module to determine whether a flow rule used by a given controlled network node involved in the data loop for a TTL expired packet corresponds to a flow rule for the marker packet in an SDN controller database for the given controlled network node.

11. The SDN controller of claim 10₇ wherein the flow rule installation module is to install flow entries at network nodes controlled by the SDN controller to remedy the data loop.

12. The SDN controller of claim 10, wherein a marker packet in the series of marker packets is to include a TTL value of 1.

13. A non-transitory machine-readable storage medium encoded with instructions executable by a processor of a software-defined network (SDN) controller, the instructions comprising:

expired packet forwarding instructions to install, at a plurality of network nodes controlled by the SDN controller, a flow rule to forward time-to-live (TTL) expired packets to the SDN controller;

loop determination instructions to determine which of the plurality of controlled network nodes are involved with a data loop in the SDN by injecting a controlled network node with a series of marker packets having equivalent routing headers as the TTL expired packet, wherein the series of marker packets have increasing TTL values;

flow rule adjustment instructions to adjust a flow rule on a given controlled network node to remedy the data loop in response to determining that a flow rule used for an injected marker packet does not correspond to a flow rule for the marker packet provided by the SDN controller.

14. The medium of claim 13, wherein the flow rule for the marker packet provided by the SDN controller is to be determined by the SDN controller.

15. The medium of claim 13, wherein the flow rule for the marker packet provided by the SDN controller is to be determined by the SDN controller by reference to an SDN controller database.