WO2022139808A1 - Architecture de réseau étendu défini par logiciel à faible latence - Google Patents

Architecture de réseau étendu défini par logiciel à faible latence Download PDF

Info

Publication number
WO2022139808A1
WO2022139808A1 PCT/US2020/066589 US2020066589W WO2022139808A1 WO 2022139808 A1 WO2022139808 A1 WO 2022139808A1 US 2020066589 W US2020066589 W US 2020066589W WO 2022139808 A1 WO2022139808 A1 WO 2022139808A1
Authority
WO
WIPO (PCT)
Prior art keywords
core
network node
qos
virtual queue
rate
Prior art date
Application number
PCT/US2020/066589
Other languages
English (en)
Inventor
Yan Sun
Xilei WANG
Original Assignee
Futurewei Technologies, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Futurewei Technologies, Inc. filed Critical Futurewei Technologies, Inc.
Priority to CN202080106997.2A priority Critical patent/CN116438787A/zh
Priority to PCT/US2020/066589 priority patent/WO2022139808A1/fr
Publication of WO2022139808A1 publication Critical patent/WO2022139808A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • H04L47/62—Queue scheduling characterised by scheduling criteria
    • H04L47/6215—Individual queue per QOS, rate or priority
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • H04L47/52—Queue scheduling by attributing bandwidth to queues
    • H04L47/527—Quantum based scheduling, e.g. credit or deficit based scheduling or token bank
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
    • H04L47/2441—Traffic characterised by specific attributes, e.g. priority or QoS relying on flow classification, e.g. using integrated services [IntServ]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00—Packet switching elements
    • H04L49/30—Peripheral units, e.g. input or output ports
    • H04L49/3045—Virtual queuing

Definitions

  • This disclosure is related to the technical field of software defined wide area networks, in particular low latency software defined wide area network architecture.
  • WAN wide area network
  • MPLS multi-protocol label switching
  • Some data may be transmitted via the Internet.
  • Data transmitted via the Internet does not typically follow a defined route.
  • Transmitting data via the Internet is typically cheaper than transmitting data via an MPLS circuit.
  • transmitting data via the Internet is typically slower and less reliable than transmitting data via an MPLS circuit.
  • SD-WAN software defined WAN
  • a first aspect relates to network node comprising a memory; and a processor coupled to the memory, the processor comprising a first core and a second core, the processor configured to receive instructions from the memory which, when executed by the processor, cause the network node to create a first virtual queue associated with the first core and a first quality of service (QoS); create a second virtual queue associated with the second core and the first QoS; write, at a first time, a first packet associated with the first QoS to the first virtual queue by the first core; and write, at substantially the first time, a second packet associated with the first QoS to the second virtual queue by the second core.
  • QoS quality of service
  • the instructions further cause the network node to determine supply rates for the first virtual queue and the second virtual queue based on the first QoS and one or more of a number of packets associated with the first QoS or a volume of data associated with the first QoS.
  • the instructions further cause the network node to transmit data associated with the first QoS according to the supply rates
  • the instructions further cause the network node to determine, by the first core, a first demand rate of the first virtual queue; and determine, by the second core, a second demand rate of the second virtual queue.
  • the first demand rate is based upon a first quantity of packets of the first QoS at the first core
  • the second demand rate is based upon a second quantity of packets of the first QoS at the second core
  • the processor further comprises a scheduler, and wherein the instructions further cause the network node to determine, by the scheduler, a total demand rate comprising the first demand rate and the second demand rate; determine, by the scheduler, a product of the first demand rate and a rate limit assigned to the first QoS; determine, by the scheduler, a supply rate for the first virtual queue as a quotient of the product and the total demand rate; and transmit the supply rate to the first core.
  • the first demand rate comprises a write permission for the first core
  • the supply rate comprises a read permission for the first core
  • the instructions further cause the network node to determine, by the first core, a third quantity of tokens based upon the supply rate; and send, by the first core, the first packet to a destination when the third quantity is above a threshold.
  • a second aspect relates to a method in a network node comprising a processor, the method comprising creating a first virtual queue associated with a first core of the processor and a first quality of service (QoS); creating a second virtual queue associated with a second core of the processor and the first QoS; writing, at a first time, a first packet associated with the first QoS to the first virtual queue by the first core; and writing, at substantially the first time, a second packet associated with the first QoS to the second virtual queue by the second core.
  • QoS quality of service
  • the method provides techniques that improve throughput of a queue by writing to virtual queues of the same QoS at substantially the same time, delay associated with locking a queue is avoided.
  • the method further comprises determining supply rates for the first virtual queue and the second virtual queue based on the first QoS and one or more of a number of packets associated with the first QoS or a volume of data associated with the first QoS.
  • the method further comprises transmitting data associated with the first QoS according to the supply rates.
  • the method further comprises determining, by the first core, a first demand rate of the first virtual queue; and determining, by the second core, a second demand rate of the second virtual queue.
  • the first demand rate is based upon a first quantity of packets of the first QoS at the first core
  • the second demand rate is based upon a second quantity of packets of the first QoS at the second core.
  • the method further comprises determining, by a scheduler, a total demand rate comprising the first demand rate and the second demand rate; determining, by the scheduler, a product of the first demand rate and a rate limit assigned to the first QoS; determining, by the scheduler, a supply rate for the first virtual queue as a quotient of the product and the total demand rate; and transmitting the supply rate to the first core.
  • the first demand rate comprises a write permission for the first core
  • the supply rate comprises a read permission for the first core.
  • the method further comprises determining, by the first core, a third quantity of tokens based upon the supply rate; and sending, by the first core, the first packet to a destination when the third quantity is above a threshold.
  • a third aspect relates to a computer program product comprising instructions embodied on a computer readable medium which, when executed by a network node comprising a processor, cause the network node to create a first virtual queue associated with a first core of the processor and a first quality of service (QoS); create a second virtual queue associated with a second core of the processor and the first QoS; write, at a first time, a first packet associated with the first QoS to the first virtual queue by the first core; and write, at substantially the first time, a second packet associated with the first QoS to the second virtual queue by the second core.
  • QoS quality of service
  • the computer program product includes computer instructions to reduce latency in a queue by writing to virtual queues of the same QoS at substantially the same time, delay associated with locking a queue is avoided.
  • the instructions further cause the network node to determine supply rates for the first virtual queue and the second virtual queue based on the first QoS and one or more of a number of packets associated with the first QoS or a volume of data associated with the first QoS.
  • the instructions further cause the network node to transmit data associated with the first QoS according to the supply rates.
  • the instructions further cause the network node to determine, by the first core, a first demand rate of the first virtual queue; and determine, by the second core, a second demand rate of the second virtual queue.
  • the first demand rate is based upon a first quantity of packets of the first QoS at the first core
  • the second demand rate is based upon a second quantity of packets of the first QoS at the second core.
  • the instructions further cause the network node to determine, by a scheduler, a total demand rate comprising the first demand rate and the second demand rate; determine, by the scheduler, a product of the first demand rate and a rate limit assigned to the first QoS; determine, by the scheduler, a supply rate for the first virtual queue as a quotient of the product and the total demand rate; and transmit the supply rate to the first core.
  • the first demand rate comprises a write permission for the first core
  • the supply rate comprises a read permission for the first core
  • the instructions further cause the network node to determine, by the first core, a third quantity of tokens based upon the supply rate; and send, by the first core, the first packet to a destination when the third quantity is above a threshold.
  • the instructions further cause the network node to write the first packet without locking the first virtual queue.
  • a fourth aspect relates to a software-defined wide area network (SD-WAN) comprising a first network node configured to receive, from a plurality of network nodes, network status reports and encryption information; generate a first tunnel through the SD-WAN based on the network status reports; and transmit, to the plurality of network nodes, configuration files comprising information of the first tunnel, encryption information of the plurality of network nodes, and network status information of the plurality of nodes; and a second network node of the plurality of network nodes, the second network node configured to set a service level obligation (SLO) for an application; set a priority for the SLO; transmit a packet of the application using the first tunnel when the SLO of the application is met by the tunnel; and identify a second tunnel for the packet when the SLO of the application is not met by the tunnel.
  • the first tunnel comprises an internet-based tunnel
  • the second tunnel comprises a multiprotocol label
  • each of the network status reports comprises bandwidth available at one of the plurality of network nodes, latency of the one of the plurality of network nodes, and packet drop rate of the one of the plurality of network nodes.
  • the second network node is further configured to identify a third network node in a down state; and report the down state of the third network node to the first network node.
  • the first network node is further configured to transmit updated configuration files to the plurality of network nodes in response to the down state of the third network node.
  • the second network node is further configured to create a first virtual queue associated with a first quality of service (QoS) and a first core of a processor of the second network node; create a second virtual queue associated with a second core of the processor and the first QoS; write, at a first time, a first packet associated with the first QoS to the first virtual queue by the first core; and write, at substantially the first time, a second packet associated with the first QoS to the second virtual queue by the second core.
  • QoS quality of service
  • a sixth aspect relates to a method in a software-defined wide area network (SD-WAN), the method comprising receiving, by a first network node from a plurality of network nodes, network status reports and encryption information; generating, by the first network node, a first tunnel through the SD-WAN based on the network status reports; transmitting, by the first network node to the plurality of network nodes, configuration files comprising information of the first tunnel, encryption information of the plurality of network nodes, and network status information of the plurality of nodes; setting, by a second network node of the plurality of network nodes, a service level obligation (SLO) for an application; setting, by the second network node, a priority for the SLO; transmitting, by the second network node, a packet of the application using the first tunnel when the SLO of the application is met by the tunnel; and identifying, by the second network node, a second tunnel for the packet when the SLO of the application is not met by the tunnel.
  • SLO service level obligation
  • each of the network status reports comprises bandwidth available at one of the plurality of network nodes, latency of the one of the plurality of network nodes, and packet drop rate of the one of the plurality of network nodes.
  • the method further comprises identifying, by the second network node, a third network node in a down state; and reporting, by the second network node, the down state of the third network node to the first network node.
  • the method further comprises transmitting, by the first network node, updated configuration files to the plurality of network nodes in response to the down state of the third network node.
  • the method further comprises creating, by the second network node, a first virtual queue associated with a first quality of service (QoS) and a first core of a processor of the second network node; creating, by the second network node, a second virtual queue associated with a second core of the processor and the first QoS; writing, by the second network node at a first time, a first packet associated with the first QoS to the first virtual queue by the first core; and writing, by the second network node at substantially the first time, a second packet associated with the first QoS to the second virtual queue by the second core.
  • QoS quality of service
  • FIG. 1 is a diagram of an embodiment of a low-latency SD-WAN architecture.
  • FIG. 2 is a diagram of an embodiment of a controller and agent for a low-latency SD-
  • FIG. 3 is a diagram of an embodiment of a traffic flow control technique.
  • FIG. 4 is a diagram of an embodiment of a single-core rate limiting engine.
  • FIG. 5 is a diagram of an embodiment of a multi-core rate limiting engine.
  • FIG. 6 is a diagram of an embodiment of a multi-core lock free rate limiting engine.
  • FIG. 7 is a diagram of an embodiment of a method for lock-free rate limiting.
  • FIG. 8 is a schematic diagram of an electronic device according to an embodiment of the disclosure.
  • FIG. 9 is a diagram of a means for lock-free rate limiting.
  • An SD-WAN provides certain benefits relative to traditional hardware based WANs.
  • An SD-WAN may be configured to provide network redundancy, using software based routes, when one route fails, another route may be determined and used for data transmission.
  • a failure in a hardware based WAN results in communication loss requiring hardware reconfiguration or repair.
  • An SD-WAN may be used to offload traffic from more expensive MPLS circuits to less expensive internet based circuits. Further, higher priority traffic may be prioritized for MPLS circuits while lower priority traffic may be transmitted using internet based circuits. Traffic priority may be determined based upon the type of data carried by the traffic.
  • SD-WAN provides more secure solutions leveraging various encryption standards to secure data transmissions.
  • SD-WANs may have unpredictable performance, especially for latency sensitive applications.
  • the low-latency SD-WAN architecture described herein provides a high-performance software-defined client-server architecture for routing packets to encrypted endpoints via an overlay network. Further, a lock-free rate-limiting approach is described to provide latency sensitive packets with higher bandwidth and shorter latency.
  • a controller and agents may be deployed in the network. Agents may reside at network nodes of the SD-WAN and one or more controllers may be deployed to control the agents and SD-WAN behavior at various locations in the network. Network nodes may include any device on the network that is capable of creating, receiving, or transmitting information through the SD-WAN. Each agent will collect network information such as latency and bandwidth between network equipment and send the network information to the controller.
  • the controller will generate configuration files based on the network information and distribute the configuration files to the agents.
  • the agent can also be used as an endpoint or a router. If an agent is used as a router, the agent will decrypt the outer layer encrypted header of a packet and forward to the next agent in the route.
  • the controller has an overall view of all agents in the network and gathers network status such as bandwidth and latency and other communication characteristics from the agents. While the example provides for a single controller, multiple controllers may be used in a network, for example for redundancy.
  • the controller coordinates creation of encrypted tunnels for communication between the controller and agents.
  • a tunnel is a communication channel between network nodes. Tunnels may be encrypted.
  • a tunnel is created by a controller by identifying ingress and egress points of the tunnel and in some cases intermediate network nodes used to transmit packets through a network. Packets traversing the tunnel may be encapsulated with identifiers to allow transmission through the network using he tunnel. The identifiers may identify to network nodes a tunnel to transmit the packet over.
  • the controller will generate configuration files for all agents based on network status, and distribute configuration files to the agents. Each configuration file includes peer agents’ encryption information such as public key and peer agents' network status such as latency, bandwidth and other communication characteristics.
  • the agent can select from multiple routes for transmitting packets to meet service-level objectives (SLOs).
  • SLO may be achieved by using routes that provide a quality of service (QoS) meeting the needs of the SLO.
  • Agents may set SLOs for different applications before encryption, and set different priority for different SLOs.
  • the agent selects a route based on the network status information received from the controller and the SLO of the to-be transmitted packet.
  • Agents process packets based on their priorities to achieve a particular QoS. Packet priorities may be set based at least on their latency sensitivity levels. In a hybrid SD-WAN/MPLS network, the agent may transmit only high priority packets via the MPLS.
  • Agents can also be used as a router and forward packets to a new destination.
  • the sender When an agent is used as a router, the sender will add another outer packet header for this router, and the agent will process and strip off the outer header and forward the packet to the destination or to next agent which also works as a router. Routing decisions are made locally by the agent to reduce latency (i.e., there is no delay waiting for the controller to provide routing decisions). If an agent goes down or otherwise loses connectivity, other agents will report to the controller and the controller will reconfigure affected agents.
  • QoS includes quantifying the performance of a communication route or other service.
  • QoS may be used to prioritize certain types of traffic over others in order to ensure higher priority traffic is communicated to meet a SLO.
  • Rate limiting is used to control the rate of traffic in and out of a communication device. Rate limiting is used in the low-latency SD- WAN architecture to ensure latency sensitive traffic is transmitted to meet SLOs.
  • lock-free rate limiting uses virtual queues to isolate simultaneous access to the same queue by different central processing unit (CPU) cores. Two parameters are proposed to synchronize the QoS constraints among multiple cores and multiple virtual queues; the parameters are a demand rate determined by the core and a supply rate determined by a scheduler.
  • the lock-free rate limiting may experience a 50% higher limiting rate than locking rate limiting approaches.
  • the lock-free rate limiting is scalable over any number of cores in a multi-core system to support multiple virtual queues each associated with one of a group of QoS supported by the multi-core system.
  • Virtual queues associated with a QoS class are used to isolate simultaneous access to the same queue by different CPU cores.
  • a particular QoS has a virtual queue associated with multiple cores in a multi-core system.
  • Two parameters, demand rate and supply rate, are associated with each virtual queue to synchronize the QoS constraints among multi cores.
  • Supply rate is one or a set of parameters which determines the actual packet transmission rate of a virtual queue. Under given supply rate parameters, the virtual queue is expected to dequeue at a specific average rate.
  • Demand rate is one or a set of parameters determined by a core based upon usage of a virtual queue.
  • Each CPU core needs write permission to the demand rate parameters and read permission to the supply rate parameters of its own virtual queues.
  • a scheduler which periodically fetches the demand rate of each virtual queue and recalculates the supply rate parameters correspondingly, takes charge of demand rate collection and supply rate update.
  • the lock-free rate limiting can be used in a variety of multi-core apparatuses or systems including, but not limited to, switches, routers, firewalls, intrusion detection systems, carrier networks, virtualized networks and other cloud based environments.
  • FIG. 1 is a diagram of an embodiment of a low-latency SD-WAN architecture 100.
  • the architecture 100 includes location A 110, location B 120, location C 130, and location D 140.
  • the embodiment of FIG. 1 includes 4 locations, other embodiments may include any number of locations.
  • the locations may be geographically separate locations that have a need to share data or other electronic communications, the locations may be co-located in a same geographic location but on separate communication networks, or a combination of geographically separate and co-located locations.
  • the locations communicate via one or more communication channels.
  • the communication channels may include various forms of electronic communication between the locations.
  • Channel 112 is used for communication between location A 110 and location B 120.
  • Channel 122 is used for communication between location B 120 and location C 130.
  • Channel 116 is used for communication between location A 110 and location D 140.
  • Channel 114 is used for communication between location A 110 and location C 130.
  • Channel 142 is used for communication between location D 140 and location C 130. In other embodiments, more or less communication channels may be used for communication between locations.
  • the SD-WAN architecture 100 also includes a controller 150 to control communications in the SD-WAN.
  • Controller 150 may be in communication with agents 115, 125, 135, and 145 at each of the locations 110, 120, 130, 140.
  • the agents 115, 125, 135, and 145 are configured to communicate with the controller 150 to control communications at their respective location.
  • the controller 150 communicates with agents 115, 125, 135, and 145 via secured communication channels 152, 154, 156, and 158.
  • the agents 115, 125, 135, and 145 may communicate various statistics related to the communications channels connected to their respective location. For example, available bandwidth and latency of the communications channel may be transmitted to the controller 150.
  • FIG. 2 is a diagram of an embodiment of a controller 210 and agent 220 for an SD-WAN.
  • controller 210 may function as controller 150 and agent 220 may function as agents 115, 125, 135, and 145.
  • Controller 210 includes tunnel setup engine 212 in communication with tunnel setup engine 225 at agent 220, a configuration engine 214, and a global tunnel status monitor 216 in communication with a tunnel status monitor 226 at agent 220.
  • Agent 220 includes a servicelevel objective (SLO) setting engine 221, a route selection component 222, an encryption/ decryption engine 223, a rate limiting engine 224, the tunnel setup engine 225, and the tunnel status monitor 226.
  • SLO servicelevel objective
  • Agent 220 is configured to process packets for transmission over an SD-WAN.
  • a packet may be received for transmission from a data source in communication with the agent 220.
  • the SLO setting engine 221 determines an SLO for the packet.
  • An SLO may include various desired values or ranges of values for transmission characteristics used for transmitting the packet.
  • SLO may include latency, error rate, throughput, bandwidth, path availability, and/or other transmission characteristics.
  • the SLO may be determined based upon the type of packet to be transmitted. Certain applications may require less latency than others, for example video conferencing in real-time may require less latency than a large file transfer. Thus, the application that created that packet may be taken into consideration when determining an SLO. Other factors may be considered, for example, the sender of the packet, the receiver of the packet, or other characteristics.
  • the route selection module 222 selects a route for the packet through the SD-WAN.
  • the route may be selected to meet the desired SLO. Additional factors may be considered in combination with meeting the desired SLO. For example, the cost of the transmission may be taken into consideration. If two paths can meet the SLO, the cheaper alternative may be selected for transmission. Other factors may be considered during route selection.
  • the route selection module 222 identifies a tunnel to route the packet through to the destination.
  • Tunnel setup engine 225 receives instructions from tunnel setup engine 212 at the controller 210 for setting up tunnels through the SD-WAN.
  • the tunnels may be created to meet certain SLO criteria, to provide direct routes between locations, or for redundancy through the SD-WAN. If a tunnel fails, the tunnel setup engine 212 may create or identify other tunnels to replace the failed tunnel.
  • Tunnel status monitor 226 receives status updates from the global tunnel status monitor 216 of tunnels available for routing packets. These status updates may be used by the route selection module 222 in identifying a tunnel for routing the packet.
  • Tunnel status monitor 226 also provides status updates to the global tunnel status monitor 216 of tunnels in communication with agent 220. These updates may be provided to other agents in the SD-WAN for use in route selection.
  • One or more tunnels may be used to route a packet from source to destination in the SD-WAN.
  • the encryption/decryption engine 223 may encrypt the packet for transmission through a tunnel. Similarly, received packets may be decrypted by the encryption/decryption engine 223. Finally, the rate limiting engine 224 controls the rate at which packets are transited over various tunnels to meet SLO requirements and/or other transmission requirements of the SD-WAN. The rate limiting engine 224 may also control packet transmission rate to prevent overloading of tunnels in the SD-WAN.
  • FIG. 3 is a diagram of an embodiment of a traffic flow control 300.
  • Traffic flow control 300 may be implemented in apparatuses, e.g. controller 150 and/or agents 115, 125, 135, and 145, for controlling the flow of packet traffic from the apparatus. Packet traffic may be controlled using a token technique.
  • a traffic type or transmission interface may have an associated bucket 320 for storing tokens 310.
  • the bucket 320 and tokens 310 may be implemented in software or hardware as a count-up or count-down register.
  • the bucket 320 may be assigned a quantity of tokens 310 used for transmitting packets 340.
  • Tokens 310 may be assigned by a controller (not pictured) or some other device to control data rates in a network where the traffic flow control 300 is implemented.
  • the quantity of tokens 310 may be periodically refreshed and or adjusted based upon factors associated with an associated transmission interface, packet processing capabilities, or other characteristics that may affect packet transmission. Periodically refreshed tokens 310 may result in a desired transmission rate per time period. For example, fifty tokens per second added to the bucket may allow for fifty bytes per second to be transmitted.
  • Transmission queue 330 stores packets 340 that are awaiting transmission. The packets 340 may be broken up into smaller sizes for transmission by a transmitting apparatus, e.g. packets may be broken up into one byte segments. Each byte 360 must use a token 310 in order to be transmitted. When no more tokens 310 are in bucket 320, transmission is stopped until additional tokens 310 are received.
  • packets 340 may be dropped and not transmitted even when additional tokens 310 become available. Packets 340 that are dropped may be selected on various criteria individually or combined, for example, oldest packet in the queue, newest packet in the queue, traffic type of packet, etc. The bytes 360 are allowed to dequeue 350 to the transmission interface when paired with a token 310. While only a single bucket 320 and transmission queue 330 are shown, a system may include multiple transmission queues 330 of varying sizes for processing of various different types of traffic, where each type of traffic may be associated with a single transmission queue 330 and bucket 320. Higher priority transmission queues 330 may receive more tokens 310 than lower priority transmission queues 330.
  • Management of the traffic flow control 300 may be performed by a controller, e.g., controller 210 or an agent under control of a controller.
  • FIG. 4 is a diagram of an embodiment of a single-core rate limiting engine 400.
  • the single-core rate limiting engine 400 controls transmission rate of packets using a single-core 420 of a CPU.
  • a packet may be received by a network interface 410.
  • the single-core 420 packet reception module 422 may communicate with the network interface 410 to receive the packet.
  • Packet classification module 424 may identify a QoS of the packet and write the packet to a corresponding QoS queue 430.
  • Each QoS queue 430 corresponds to a single QoS.
  • a QoS may be associated with an SLO.
  • the single-core rate limiting engine 400 is limited to processing a single packet at a time for transmission and thus the processing capabilities of single-core 420 limit the throughput of the single-core rate limiting engine 400.
  • Packets are pushed from the QoS queues 430 to packet transmission module 440.
  • the packets may be pushed based on a traffic flow control, for example traffic flow control 300.
  • the packets are transmitted using network interface 450.
  • Network interface 450 may be the same as network interface 410 or different.
  • FIG. 5 is a diagram of an embodiment of a multi-core rate limiting engine 500.
  • the multi-core rate limiting engine 500 uses four cores 520 of a CPU.
  • Each core 520 includes a packet reception module 522 and a packet classification module 524.
  • a packet may be received by a network interface 510.
  • the packet may be received by a packet reception module 522 in communication with the network interface 510.
  • Packets may be assigned to the cores 520 in a sequential manner, e.g., first packet is assigned to core 0, second packet assigned to core 1, and so forth.
  • Packets may be assigned to the cores 520 based on a current workload of the cores 520, e.g., a core 520 with the lowest workload may be assigned a packet for processing.
  • Packet classification module 524 may identify a QoS of the packet and write the packet to a corresponding QoS queue 530.
  • Each QoS queue 530 corresponds to a single QoS. When a packet is written to a QoS queue 530, the QoS queue 530 is locked and only one core 520 is allowed to write to the QoS queue 530.
  • a bottleneck may occur if multiple cores 520 have packets to be written to the same QoS queue 530 because other cores 520 have to wait until a currently writing core 520 releases the lock on the particular QoS queue 530.
  • Packets are pushed from the QoS queues 530 to packet transmission module 540.
  • the packets may be pushed based on a traffic flow control, for example traffic flow control 300.
  • the packets are transmitted using network interface 550.
  • Network interface 550 may be the same as network interface 510 or different.
  • a lock-free rate limiting engine may be used to overcome the bottle neck that occurs in a multi-core system when queues are locked for writing.
  • the lock-free rate limiting engine may include a plurality of virtual queues stored in a memory of the device where the lock-free rate limiting engine is implemented.
  • a queue (which may be referred to in this example as a traditional queue) will be locked each time a core writes to the traditional queue.
  • a queue which may be referred to in this example as a traditional queue
  • Each core sequentially writes to the traditional queue in a serial fashion.
  • One or several virtual queues of the lock-free rate limiting engine may be associated with a single core. Each virtual queue may have a relationship with a single core.
  • Associating a virtual queue with only one core permits the core to write to the virtual queue without contending with other cores.
  • Several virtual queues, each associated with different cores, may function to store data that would have been written to a single traditional queue. Thus, more than one core may write data in parallel to the virtual queues that would have been written serially using a traditional queue.
  • each traditional queue is associated with a particular type of data, e.g., a QoS class, an application, a specific format, etc.
  • Several virtual queues are associated with the same type of data in the lock-free rate limiting engine. Thus, in a traditional queue system, if several cores have data for a particular application, the cores would write the data serially to the traditional queue associated with the particular application.
  • Each core may have a virtual queue for one or more types of data.
  • the cores may be associated with different numbers of virtual queues.
  • Transmission rates may vary for different types of data.
  • a type of data may have an associated committed rate.
  • the committed rate is a transmission rate for the type of data.
  • Several virtual queues may be associated with the same type of data. Thus, the virtual queues associated with the same type of data share the committed rate for the type of data.
  • a controller may be configured to control the transmission rate from each of the virtual queues to ensure the combined traffic from the virtual queues does not exceed the committed rate for the type of data.
  • Each core may provide a demand rate for the type of data to the controller.
  • a demand rate for each type of data handled by the core may be provided to the controller. If several cores are all queuing the same type of data for transmission, the controller will read the demand rate from each of the cores and determine a supply rate for each core.
  • the supply rate is the rate at which a particular core may dequeue data packets of a particular data type from the corresponding virtual queue.
  • Each data type may be associated with a QoS.
  • each core will have a demand rate for each of the virtual queues associated with the core and an assigned supply rate for each of the virtual queues associated with the core.
  • Supply rates and demand rates may be provided as tokens for transmission of packets, as data volumes, e.g., bits or bytes, or as some other unit to measure the flow of data to and from the virtual queues.
  • Supply rates may be periodically updated by the controller. For example, a core is assigned a certain supply rate for a given period, e.g., 500 bits during a one millisecond period. The core may transmit 500 bits of data from the corresponding virtual queue during one millisecond. The controller may read demand rates and determine a new supply rate for the virtual queue. The new supply rate is assigned to the virtual queue, and the core transmits data from the virtual queue accordingly.
  • Each virtual queue has a defined storage size.
  • the core may not be able to write additional data to the virtual queue.
  • data in the virtual queue may be dropped or new data may be dropped and not be added to the virtual queue. Whether a packet is dropped or not may be determined on a number of criteria, for example, priority of the data, source of the data, destination of the data, size of the data, etc.
  • FIG. 6 is a diagram of an embodiment of a multi-core lock free rate limiting engine 600.
  • the multi-core lock free rate limiting engine 600 uses four cores 620 of a CPU for processing packets.
  • a fifth core 630 is used for scheduling packet transmissions.
  • Cores 620 each include a packet reception module 622 and a packet classification module 624.
  • a packet may be received by a network interface 610.
  • the packet may be received by a packet reception module 622 in communication with the network interface 610.
  • Packets may be assigned to the cores 620 in a sequential manner, e.g., first packet is assigned to core 0, second packet assigned to core 1, and so forth.
  • Packets may be assigned to the cores 620 based on a current workload of the cores 620, e.g., a core 620 with the lowest workload may be assigned a packet for processing. Other methods may be used for assigning packets to the cores 620.
  • Each core 620 is associated with several virtual queues 640.
  • Virtual queues 640 may occupy a memory or some other data storage area of a network node comprising the multi-core lock free rate limiting engine 600.
  • a complete queue comprises one virtual queue 640 associated with each core 620.
  • a complete queue associated with QoS Class 0 includes virtual queues 640 identified as QoS class 00, 10, 20, and 30.
  • Each core 620 writes to several dedicated virtual queues 640 associated with different QoS values.
  • multi-core lock free rate limiting engine 600 provides an advantage over multi-core rate limiting engine 500 in that a queue does not need to be locked for multiple cores 620 to substantially simultaneously write to the queue.
  • Writing substantially simultaneously includes writing to two virtual queues of the same QoS classes without locking a queue. Thus, the period of time it would take to lock and unlock a physical queue for writing may be considered as substantially simultaneously.
  • a different core 630 is used to schedule transmissions from the virtual queues 640.
  • Core 630 includes a scheduler 632.
  • Scheduler 632 monitors a demand rate 642 and determines a supply rate 644, based on the demand rate 642, of each core's virtual queues 640.
  • Scheduler 632 may use the demand rate 642 and supply rate 644 associated with each virtual class queue to synchronize the QoS constraints among cores 620.
  • Supply rate 644 determines the packet transmission rate of one or more virtual queues 640. Based on supply rate 644 parameters, the virtual queue 640 is expected to dequeue at a specific average rate.
  • Scheduler 632 assigns a supply rate 644 to a virtual queue 640, and the virtual queue 640 dequeues a number of bytes or packets based on the assigned supply rate 644.
  • Each core 620 has write permission to the demand rate 642 parameters and read permission to the supply rate 644 parameters of its own virtual queues 640.
  • Scheduler 632 periodically fetches the demand rate 642 of each virtual queue 640 and recalculates the supply rate 644 parameters correspondingly. If the demand rate 642 is high, the scheduler 632 may assign additional supply rate 644. Other factors may be considered in assigning supply rate 644, e.g., QoS of the virtual queues 640, priority of traffic in the virtual queue 640, etc.
  • Packets are pushed from the virtual queues 640 to packet transmission module 650.
  • the packets are transmitted using network interface 660.
  • Network interface 660 may be the same as network interface 610 or different.
  • the multi-core lock free rate limiting engine 600 may be implemented with any number of cores and any number of QoS classes/virtual queues.
  • the scheduler 632 may be executed by one of cores 620.
  • the multi-core lock free rate limiting engine 600 may be implemented in an SD-WAN, an MPLS system, or other systems with multi-core processors and multiple QoS classes for transmission.
  • Table 1 is an algorithm 1 used for processing packets in a virtual queue 640 by a core 620.
  • Inputs to algorithm 1 include a supply rate sr(ij) received from scheduler 632 for a particular core i and data type j on said core z, a packet pkt with length len, and a current time t.
  • Data type j is associated with one of the virtual queues of core i.
  • Data type j may also be associated with a QoS class.
  • Time difference t diff determined by subtracting a time of last processing tiast from time t.
  • num tokens is the number of available tokens (e.g., supply tokens 644) for the data type j.
  • num tokens is determined by summing a previous num tokens with the product of t diff and sr(ij).
  • the current packet is processed by the droppacket(pkt) function.
  • the droppacket(pkt) function may include dropping the packet, queueing the packet for later future transmission, fragmenting the packet, or other functions for handling a packet when the supply tokens for the virtual queue are exhausted in a particular transmission period. Otherwise, the current packet is transmitted, e.g., dequeued, from the virtual queue by the sendpacket(pkt) function.
  • num tokens is adjusted by subtracting len.
  • a packet is transmitted from a virtual queue when a sufficient number of tokens are available.
  • the number of tokens is adjusted downward based on the length of the packet.
  • the supply rate 644 assigned by scheduler 632 to a virtual queue 640 is used to determine the number of tokens the virtual queue 640 may use for transmitting packets from the virtual queue 640.
  • Supply rate 644 is calculated based on demand rate 642 using algorithm 2 in table 2.
  • Inputs to algorithm 2 include a number of CPU cores m, a number of QoS classes n , a demand rate for a particular core and virtual queue dr(ij), and a committed rate limit for a QoS class cr(j).
  • the committed rate cr(j) for a QoS class may be received from a controller 210 in an SD- WAN.
  • Committed rate cr(j) is a transmission rate associated with the QoS class.
  • the committed rate cr(j) is a target transmission rate used to comply with the transmission rate associated with a QoS class.
  • Committed rate cr(f) may be assigned by a controller internal to the multi-core lock free rate limiting engine 600, e.g, scheduler 632, or external to the multi-core lock free rate limiting engine 600, e.g. an internet service provider (ISP).
  • Committed rate cr(j) may be assigned based on the QoS classes used in transmitting packets. Each QoS class is associated with one virtual queue at each core.
  • the demand rate dr(i,j) is determined by each core based upon the number of packets associated with the particular QoS. The core provides the demand rate dr(i,j) to the scheduler 632 for computing the source rate sr.
  • the demand rate dr of each core is summed to determine a total demand rate sum for the QoS class j.
  • the source rate sr for each core and virtual queue combination is determined by multiplying the demand rate for that core and virtual queue combination by the committed rate cr(j) for that QoS associated with the virtual queue and then dividing the product by the total demand rate sum.
  • a core and virtual queue combination is one virtual queue at one of the cores.
  • FIG. 7 is a flow diagram of an embodiment of a method 700 for lock-free rate limiting.
  • Method 700 begins at block 710 when a network device creates a first queue associated with a first core of a CPU and a first QoS class of traffic handled by the network device.
  • the method 700 continues at block 720 when the network device creates a second queue associated with a second core of the CPU and the first QoS class of traffic.
  • the first CPU writes a packet to the first virtual queue.
  • the second CPU writes a packet to the second virtual queue.
  • multiple cores may write packets associated with the same QoS class to virtual queues of the QoS at the same time.
  • the lock-free rate limiting achieves a higher packet throughput resulting in support of a higher maximum supported limiting rate.
  • FIG. 8 is a schematic diagram of an electronic device 800 according to an embodiment of the disclosure.
  • the electronic device 800 is suitable for implementing the disclosed embodiments as described herein.
  • the electronic device 800 comprises ingress ports 810 and receiver units (Rx) 820 for receiving data; a processor, logic unit, or central processing unit (CPU) 830 to process the data; transmitter units (Tx) 840 and egress ports 850 for transmitting the data; and a memory 860 for storing the data.
  • the electronic device 800 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports 810, the receiver units 820, the transmitter units 840, and the egress ports 850 for egress or ingress of optical or electrical signals.
  • OE optical-to-electrical
  • EO electrical-to-optical
  • the processor 830 is implemented by hardware and software.
  • the processor 830 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs).
  • the processor 830 is in communication with the ingress ports 810, receiver units 820, transmitter units 840, egress ports 850, and memory 860.
  • the memory 860 comprises one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution.
  • the memory 860 may be volatile and/or non-volatile and may be read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), and/or static random-access memory (SRAM).
  • FIG. 9 is a diagram of a means for lock-free rate limiting 900, for example a network node in a SD-WAN.
  • the means for lock-free rate limiting 900 includes memory means 920, for example memory 860; and processing means 910, for example processor 930.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

La présente invention concerne un nœud de réseau comprenant une mémoire ; et un processeur couplé à la mémoire. Le processeur comprend un premier cœur et un second cœur, le processeur étant configuré pour recevoir des instructions de la mémoire qui, lorsqu'elles sont exécutées par le processeur, amènent le nœud de réseau à créer une première file d'attente virtuelle associée au premier cœur et une première qualité de service (QoS) ; créer une seconde file d'attente virtuelle associée au second cœur et à la première QoS ; écrire, à un premier instant, un premier paquet associé à la première QoS dans la première file d'attente virtuelle par le premier cœur ; et écrire, sensiblement au premier instant, un second paquet associé à la première QoS dans la seconde file d'attente virtuelle par le second cœur.
PCT/US2020/066589 2020-12-22 2020-12-22 Architecture de réseau étendu défini par logiciel à faible latence WO2022139808A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080106997.2A CN116438787A (zh) 2020-12-22 2020-12-22 低延迟软件定义广域网架构
PCT/US2020/066589 WO2022139808A1 (fr) 2020-12-22 2020-12-22 Architecture de réseau étendu défini par logiciel à faible latence

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2020/066589 WO2022139808A1 (fr) 2020-12-22 2020-12-22 Architecture de réseau étendu défini par logiciel à faible latence

Publications (1)

Publication Number Publication Date
WO2022139808A1 true WO2022139808A1 (fr) 2022-06-30

Family

ID=74181379

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2020/066589 WO2022139808A1 (fr) 2020-12-22 2020-12-22 Architecture de réseau étendu défini par logiciel à faible latence

Country Status (2)

Country Link
CN (1) CN116438787A (fr)
WO (1) WO2022139808A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3076621A1 (fr) * 2015-03-31 2016-10-05 Cavium, Inc. Procédé et appareil permettant d'utiliser des listes de mémoire multiples reliées
US20180123715A1 (en) * 2015-10-30 2018-05-03 Citrix Systems, Inc. Method for packet scheduling using multiple packet schedulers
US20180212889A1 (en) * 2017-01-25 2018-07-26 Futurewei Technologies, Inc. Multi-core lock-free rate limiting apparatus and method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3076621A1 (fr) * 2015-03-31 2016-10-05 Cavium, Inc. Procédé et appareil permettant d'utiliser des listes de mémoire multiples reliées
US20180123715A1 (en) * 2015-10-30 2018-05-03 Citrix Systems, Inc. Method for packet scheduling using multiple packet schedulers
US20180212889A1 (en) * 2017-01-25 2018-07-26 Futurewei Technologies, Inc. Multi-core lock-free rate limiting apparatus and method

Also Published As

Publication number Publication date
CN116438787A (zh) 2023-07-14

Similar Documents

Publication Publication Date Title
US11985060B2 (en) Dragonfly routing with incomplete group connectivity
US11412076B2 (en) Network access node virtual fabrics configured dynamically over an underlay network
US10965586B2 (en) Resilient network communication using selective multipath packet flow spraying
US20200169513A1 (en) Fabric control protocol for data center networks with packet spraying over multiple alternate data paths
US7397809B2 (en) Scheduling methods for combined unicast and multicast queuing
US11595315B2 (en) Quality of service in virtual service networks
US8660001B2 (en) Method and apparatus for providing per-subscriber-aware-flow QoS
WO2022139808A1 (fr) Architecture de réseau étendu défini par logiciel à faible latence
US20050083960A1 (en) Method and apparatus for transporting parcels of data using network elements with network element storage

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20839523

Country of ref document: EP

Kind code of ref document: A1