EP3758313B1 - Überlastungssteuerung für rechenzentren-netzwerke mit niedriger latenz - Google Patents

Überlastungssteuerung für rechenzentren-netzwerke mit niedriger latenz Download PDF

Info

Publication number
EP3758313B1
EP3758313B1 EP19212316.4A EP19212316A EP3758313B1 EP 3758313 B1 EP3758313 B1 EP 3758313B1 EP 19212316 A EP19212316 A EP 19212316A EP 3758313 B1 EP3758313 B1 EP 3758313B1
Authority
EP
European Patent Office
Prior art keywords
rtt
network device
topology
network
packet
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP19212316.4A
Other languages
English (en)
French (fr)
Other versions
EP3758313A1 (de
Inventor
Keon Jang
Inho Cho
Yi Cui
Nandita Dukkipati
David Wetherall
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google LLC filed Critical Google LLC
Publication of EP3758313A1 publication Critical patent/EP3758313A1/de
Application granted granted Critical
Publication of EP3758313B1 publication Critical patent/EP3758313B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/27—Evaluation or update of window size, e.g. using information derived from acknowledged [ACK] packets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0852—Delays
    • H04L43/0864—Round trip delays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/12—Avoiding congestion; Recovering from congestion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/18—End to end
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/25—Flow control; Congestion control with rate being modified by the source upon detecting a change of network conditions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/28—Flow control; Congestion control in relation to timing considerations
    • H04L47/283—Flow control; Congestion control in relation to timing considerations in response to processing delays, e.g. caused by jitter or round trip time [RTT]

Definitions

  • Datacenter networks may include network devices that direct the transmission of data in the form of packets from a source (e.g., a computer, a server, a datacenter, a router, etc.) to a destination (e.g., another computer, another server, another datacenter, another router, etc.).
  • the network devices determine how the data is transmitted from the source to the destination.
  • the management of packet transmission is carried out on a per flow basis.
  • a network flow refers to a unique stream of data packets that make up a distinct communication stream between two end points.
  • a flow is typically identified by a 5-tuple of information.
  • the 5-tuple of information for a flow typically includes a source internet protocol (IP) address, source port number, destination IP address, destination port number, and a protocol identifier.
  • IP internet protocol
  • other header data fields can be used to define the 5-tuple.
  • the 5-tuple may include source and destination MAC addresses.
  • the congestion window size governs how many data packets can be transmitted for a given flow before some of the packets are acknowledged as having been received by the destination network device.
  • the network devices tend to reduce the size of the congestion windows of flows they are handling as congestion increases, and to increase the size of the congestion windows as congestion decreases.
  • Typical congestion window management techniques fail to adequately distinguish between different sources of congestion, particularly in the context of cloud computing.
  • JIANXIN ZHOU ET AL "Improve tcp performance in ad hoc network by TCP-RC" (7 September 2003 ), discloses TCP-RC (TCP- Recomputation of cwnd and ssthresh) re- computes cwnd (congestion window) and ssthresh (slow start thresh) after route reconstruction, thus adjusting TCP transmission rate adaptively according to current available network capacity of the TCP connection, and keeps a stable load factor of TCP connection as route is changing.
  • PATIL MINAKSHEE ET AL "Enhancing TCP performance in multihop ad hoc networks",2013 FOURTH INTERNATIONAL CONFERENCE ON COMPUTING, COMMUNICATIONS AND NETWORKING TECHNOLOGIES (ICCCNT), IEEE, 4 July 2013 (2013-07-04) discloses an adaptive congestion window mechanism for TCP round- trip time estimation using the information from the routing layer, wherein the round-trip time (RTT) is split into two parts: 1) congestion RTT and 2) contention RTT. Congestion RTT is used to determine bandwidth-delay product (BDP). Contention RTT is used to determine the contention situation.
  • RTT round-trip time
  • Systems and methods of this disclosure relate generally to transmitting data packets over a data network.
  • a data packet is transmitted from a source to a destination in a data network.
  • the transmission of the data packet is managed based on the characteristics of a flow over the data network.
  • the characteristics may include, for example, a target round-trip time (RTT), an actual round trip time, and a congestion window size associated with the flow.
  • RTT target round-trip time
  • actual round trip time an actual round trip time
  • congestion window size associated with the flow.
  • Congestion when a data packet takes a longer period of time to be transmitted from a source to a destination, there are multiple factors that may individually or collectively cause the delay in transmission.
  • Congestion can take different forms, particularly in cloud computing contexts in which network devices handle traffic associated with multiple virtualized computing devices, either in the form of virtual machines or containerized environments.
  • Congestion broadly speaking, can be divided into at least two types of congestion, fabric congestion and engine congestion.
  • Fabric congestion refers to congestion across the network infrastructure between the source and destination of the traffic. Fabric congestion can be caused by limitations on the physical bandwidth of links between nodes in the network and/or between bandwidth limitations of switching and/or routing devices located between the source and destination of the network traffic.
  • Engine congestion relates to delays introduced by bandwidth constraints within the source and destination network devices, themselves. For example, delays resulting from a packet remaining in a transmission queue at the source node, or from delays an ingress engine at the packet destination being able to process a received packet. Such processing may include guest to physical address translation, decryption, or the execution of other network functions that are carried out on the packet before it is transferred to the actual destination computing environment, e.g., a specific virtual machine or container executing on the destination network device. Particularly in the cloud computing environment, where a given network device may host multiple computing environments, and therefore may receive more network traffic, a great percentage of congestion experienced in data center environments results from engine congestion in comparison to computing environments in which a networking device only hosts a single computing environment.
  • RTT round trip time
  • the congestion window size is the number of packets associated with a flow that can be transmitted without having one of such packets acknowledged as having been received. For example for with a congestion window of six for a given flow, if the source network device of the has transmitted six packets associated with the flow, it cannot send additional packets associated with the flow until it has received an acknowledgement message from the destination network device that one or more of such packets has been received. For each received packet, the source network device can transmit one additional packet associated with the flow. Decreasing the size of the congestion window for a flow generally decreases the number of packets associated with the packet on the network at a given time, reducing congestion.
  • congestion window size can be increased to allow for greater throughput. If implemented properly, this traffic management technique can be effective for both fabric congestion and for engine congestion. Systems and methods disclosed herein set forth congestion window management techniques that, in some implementations, are effective for distinguishing between and separately managing for both fabric and engine congestion.
  • FIG. 1A is an example of data network 100 according to one or more embodiments.
  • Data network 100 includes network devices 110-1-110-3 (hereinafter, collectively referred to as “network devices 110") connected by data links 120-1-120-12 (herein after, collectively referred to as “data links 120”) and switches 115-1-115-6 (herein after, collectively referred to as "switches 115").
  • network devices 110 may be one or more electronic devices configured to transmit data within data network 100.
  • each of network devices 110 may represent a server, host computer, or other end node in a computer network.
  • Data links 120 allow data to be transmitted between network devices 110.
  • Data links 120 may be any type of hardwired and/or wireless data connection.
  • a data link 120 may be a fiber optic link, an electrically conductive link (e.g., copper wire), or wireless link.
  • Data links 120 may be bidirectional or unidirectional.
  • a path may be an ordered set of data links and/or network devices that are connected by data links.
  • data packet may be sent from network device 110-1 (e.g., a source) to network device 110-2 (e.g., a destination) via the data network 100.
  • the data packet may be routed over a number of different paths.
  • the data packet sent by network device 110-1 to network device 110-2 may traverse data links 120-1, 120-4, and 120-12.
  • the data packet may traverse data links 120-2, 120-3, and 120-12.
  • packets traverse the shortest paths possible between source and destination, though a packet may traverse a longer path if a data link 120 or switch 115 along the shortest path is congested or otherwise impaired in a manner that makes traversing a longer path more efficient.
  • acknowledgement messages usually traverse the same path from the source to the destination, but in the reverse order, so that the acknowledgement message traverses the same data links 120 and same switches 115.
  • the switches 115 may be switches, hubs, bridges, routers, or other network device that forwards a packet along a path towards its intended destination.
  • Such switches may be packet switches, which read the headers of the packets traversing the switch to identify which of several network interfaces to forward the packet, or a circuit switch (e.g., a crossbar switch or passive optical switch) which passively forwards a packet out an appropriate network interface without evaluating the header of the packet.
  • a circuit switch e.g., a crossbar switch or passive optical switch
  • Such forwarding can be based on the time of receipt of the packet (e.g., for scheduled transmissions) or based on the frequency of an optical carrier wave used to transmit the packet.
  • FIG. 1B is another block diagram representing the network 100, emphasizing features of an illustrative network device 110 suitable for inclusion in the network 100. As shown in Figure 1B , each network device 110 is communicatively coupled to other network devices 110 via switches 115 and data links 120.
  • the network 100 is a network facilitating interactions between participant devices.
  • An illustrative example network 100 is the Internet; however, in other implementations, the network 100 may be another network, such as a local network within a data center, a network fabric, or any other local area or wide area network.
  • the network 100 may be composed of multiple connected sub-networks or autonomous networks.
  • the network 100 can be a local-area network (LAN), such as a company intranet, a metropolitan area network (MAN), a wide area network (WAN), an inter-network such as the Internet, or a peer-to-peer network; for example, an ad hoc Wi-Fi peer-to-peer network.
  • LAN local-area network
  • MAN metropolitan area network
  • WAN wide area network
  • An inter-network such as the Internet
  • peer-to-peer network for example, an ad hoc Wi-Fi peer-to-peer network.
  • Any type and/or form of data network and/or communication network can be used for the
  • the network device 110 is a host device or server executing one or more virtual machines 150a-150c (generally virtual machines 150) on one or more CPUs of the network device 110.
  • virtual machines 150 instead of virtual machines 150, the network device 110 hosts a plurality of containerized computing environments. It would be understood by a person of ordinary skill in the art that discussion herein regarding virtual machine environments can also apply to containerized computing environments.
  • the network device 110 has configuration similar to that of a computing system 910 as shown in Figure 7 .
  • the memory 118 can have a configuration similar to that of a memory 970 as shown in Figure 7
  • the network interface card 140 can have configuration similar to that of a network interface card 922 and/or a network interface controller 920 as shown in Figure 7 .
  • the computing system 910 is described in more detail below, in reference to Figure 7 .
  • the elements shown in the computing system 910 illustrated in Figure 7 do not all need to be present in some implementations of the network device 110 illustrated in Figure 1B .
  • the network device 110 includes a network interface driver 122, a memory 118, a network interface card 140, a real OS 220, a hypervisor 130, and the virtual machines 150.
  • the hypervisor 130 manages operation of the virtual machines 150, including spinning up and terminating the virtual machines 150, managing allocation of memory 118 to the virtual machines 150, and migrating the virtual machines 150 to other network devices.
  • the network device 110 includes the memory 118.
  • the memory 118 may store computer executable instructions for a real operating system (OS) 220 and a transport protocol module 145 (such as a TCP protocol module or the TCP layer of the network stack) to be executed on a processor of the network device 110.
  • the memory 118 may store computer executable instructions related to the operation and use of a network interface driver 122.
  • the memory 118 may store computer executable instructions for the hypervisor 130 to be executed on a processor of the network device 110.
  • the memory 118 may store computer executable instructions for the virtual machines 150 executing on the network device 110.
  • the memory 118 may store data for forwarding to the virtual machines 150, processing by the virtual machines 150, and/or transmission from the virtual machines 150.
  • the memory 118 may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), a ferroelectric random access memory (FRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), and/or a flash memory.
  • RAM random access memory
  • DRAM dynamic random access memory
  • SRAM static random access memory
  • SDRAM synchronous dynamic random access memory
  • FRAM ferroelectric random access memory
  • ROM read only memory
  • PROM programmable read only memory
  • EPROM erasable programmable read only memory
  • EEPROM electrically erasable programmable read only memory
  • Functionality implemented within the TCP layer of a network device 110 can additionally or alternatively execute in another network protocol module within the transport layer, the network layer or a combined transport/network layer of a network protocol stack.
  • the functionality can be implemented in a user datagram protocol (UDP) module, reliable datagram protocol (RDP) module, reliable user datagram protocol (RUDP) module, or a datagram congestion control protocol (DCCP) module.
  • UDP user datagram protocol
  • RDP reliable datagram protocol
  • RUDP reliable user datagram protocol
  • DCCP datagram congestion control protocol
  • a network layer, a transport layer, or a combined transport/network layer will generally be referred to as a packet layer of the network protocol stack.
  • the network interface driver 122 can include a network interface driver software module running on a real OS.
  • a network interface driver such as the network interface driver 122, can be a collection of computer executable instructions stored in the memory 118 that when executed by a processor help facilitate network communications.
  • the network interface driver 122 may be implemented as logic implemented in a hardware processor or other integrated circuit, or as a combination of hardware and software logic.
  • the network interface driver 122 can communicate with one of the software virtual machines 150 (or applications executing on the virtual machines 150), via a guest OS of a virtual machine (or in some implementations, through a hypervisor and the guest OS if operating in a virtual machine environment).
  • the network interface driver 122 is included within a first layer of a transmission control protocol (TCP) stack of the real OS 220 of the network device 110 and communicates with a software module or application that is included in an upper layer of the TCP stack.
  • TCP transmission control protocol
  • the network interface driver 122 is included within a transport layer of a TCP stack and communicates with a software module or application that is included in an application layer of the TCP stack.
  • the network interface driver 122 is included within a link layer of a TCP stack and communicates with a TCP/IP module that is included in an internet/transport layer of the TCP stack.
  • the functionality is additionally or alternatively configured to receive packets from another network or transport layer protocol module, such as a user datagram protocol (UDP) module, reliable datagram protocol (RDP) module, reliable user datagram protocol (RUDP) module, or a datagram congestion control protocol (DCCP) module.
  • UDP user datagram protocol
  • RDP reliable datagram protocol
  • RUDP reliable user datagram protocol
  • DCCP datagram congestion control protocol
  • the network interface driver 122 can be included as a portion of the network interface card 140.
  • the network interface card 140 includes an ingress engine 141, an egress engine 144, and a memory 146.
  • the ingress engine 141 and egress engine 144 can be functional blocks of one or more processors of the network interface card 140, such as an application-specific integrated circuit (ASIC) having discrete functional blocks.
  • ASIC application-specific integrated circuit
  • the ingress engine 141 and egress engine 144 can be functional blocks of software executing on one or more general-purpose processors of the network interface card 140.
  • the functions of the ingress engine 141 and egress engine 144 can be combined or further subdivided into different functional blocks.
  • the memory 146 can store computer executable instructions for execution by network interface card 140, such as instructions relating to the operation of the ingress engine 141 and the egress engine 144.
  • the memory 146 can provide for caches, buffers, translation tables, and/or time-indexed data structures used by the network interface card 140, virtual machines 150, and/or hypervisor 130 for the transmission and receipt of data packets over the network 100.
  • the memory 146 can store, for each virtual machine 150, one or more receive queues and transmit queues.
  • the memory 146 can store tables containing translations of virtual memory addresses associated with virtual machines 150 to physical memory addresses of regions of the network device 110 in memory 118 respectively allocated to the virtual machines 150.
  • the memory 146 may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), a ferroelectric random access memory (FRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), and/or a flash memory.
  • RAM random access memory
  • DRAM dynamic random access memory
  • SRAM static random access memory
  • SDRAM synchronous dynamic random access memory
  • FRAM ferroelectric random access memory
  • ROM read only memory
  • PROM programmable read only memory
  • EPROM erasable programmable read only memory
  • EEPROM electrically erasable programmable read only memory
  • the ingress engine 141 of the network interface card 140 is responsible for handling received packets.
  • the ingress engine 141 may manage multiple receipt queues maintained in the memory 146 for storing received packets until such packets are processed by the network interface driver 122 and are passed on to a destination virtual machine 150.
  • the ingress engine 141 maintains a separate queue for each virtual machine 150.
  • the ingress engine 141 maintains separate queues for packets marked with having different quality of service flags.
  • the ingress engine 141 executes a received packet scheduler for controlling the timing of processing of packets in the respective queues by the network interface driver 122. In some implementations, such scheduling functionality is integrated instead into the network interface driver 122.
  • the egress engine 144 manages the transmission of packets received by the network interface card 140 from the network interface driver 122. Such packets include both traditional data packets transmitted by applications executing the virtual machines 150, as well as control packets, such as acknowledgement packets sent to acknowledge receipt of packets, session initiation packets and session termination packets used to set up and tear down communication sessions between network devices. In some implementations, the egress engine 144 maintains multiple transmit queues within the memory 146. The queues may be associated with individual virtual machines 150 or quality of service identifiers, or shared across multiple virtual machines 150 and/or quality of service indicators.
  • the egress engine 144 may also include a packet builder that takes data to be transmitted by the network interface card 140, breaks the data down into multiple packets, and adds any necessary header information to the packets prior to transmission.
  • packet building functionality may be implemented by the network interface driver 122 or the functionality may be split between the network interface driver 122 and the network interface card 140.
  • Figure 1C is an example block diagram 180 depicting example processing steps performed by an example network device 110 with respect to received data packets 182.
  • Figure 1C depicts certain processing performed directly by a network interface card 140 of a network device 110, as well as processing performed by the processor (and software executing thereon) of the network device 110 itself.
  • data packets 182 are received as a stream of bits 183 (encoded using an optical or electrical encoding scheme), which are converted into received data packets 182.
  • the data packets are processed by the ingress engine 141 and are stored in various queues 183a-183c (herein after referred to as "queues 183") in the memory 146 of the network interface card 140 associated with each of the virtual machines 150 executing on the network device 110.
  • queues 183 may receive packets 182 at different rates, and therefore the queues 183 associated with the respective virtual machines 150 may have different numbers of packets stored therein.
  • packets may be removed from the respective queues 183 at different rates, which may result in further variation in the number of packets 182 stored in the queues 183.
  • the rate at which packets 182 are processed from the respective queues 183 is controlled by a scheduler 184.
  • the scheduler 184 is shown in Figure 1C as a functional block (implemented e.g., as part of the network interface driver 122 executing on a processor 185 of the network device 110). However, in other implementations, the scheduler 184 may be implemented on the network interface card 140 as part of the ingress engine 141.
  • the packet processing rates may be set equally for each virtual machine.
  • certain virtual machines 150 may be given preferential packet processing rates.
  • the packet processing rates are set based on processor cycle allocations to each virtual machine 150, as different packets require different numbers of processor cycles to be processed, or the rates may be set based on a relative absolute packet processing allocation, e.g., using a round robin or other scheduling algorithm.
  • the processor 185 is configured to execute additional network functions, including a decryption function 186, an address translation function 187, and a firewall function 188. Different packets may require different ones of these network functions to be applied, depending on the type of communication involved.
  • the network interface driver 122 Upon completion of processing a received packet 182, the network interface driver 122 generates an acknowledgement packet 189 to be transmitted back to the source of the received packet 183.
  • each individual packet is not acknowledged, but groups of received packets 182 are acknowledged in a collective acknowledgement packet 189. Some network protocols do not require receipt acknowledgement at all.
  • the acknowledgement packet 189 is then transmitted by the network interface card 140 over a physical interface on to a data link 120. While not explicitly shown in Figure 1C , before the data packet including the acknowledgement message is finally transmitted, it may be temporarily stored in a transmission queue. Similarly, new data packets (i.e., not acknowledgement messages) may also be stored in transmission queues on the network interface card 140 prior to transmission, depending on the level congestion at the network interface card 140.
  • the engine RTT refers to the time consumed in the processing of a received packet and the corresponding acknowledgement message packet by the network interface cards 140 and network interface drivers 122 of the source and destination network devices of the packet. Such delays reflect congestion when any part of that processing is delayed due to the processing of other packets being transmitted and/or received by the respective network interface drivers 122 and network interface cards 140 of the source and destination network devices 110.
  • the remaining portion of the RTT is referred to as the fabric RTT, or the portion of the RTT during which the packet (and its corresponding acknowledgement) traverse the data links 120 and switches 115 between the source and destination network devices 110.
  • the ingress engine 141, egress engine 144, and/or network interface driver 122 may provide time stamps in acknowledgement packets indicating when packets are received and transmitted by the network interface cards 140 of the source and destination network devices 110.
  • the source network interface card 140 may store for each yet-to-be-acknowledged packet a transmit time indicating the time the packet was actually transmitted by the network interface card 140.
  • a destination network interface card 140 may store for each received packet a time stamp indicating when the packet was received.
  • a difference between a current time and the stored receipt time can be added to the acknowledgement packet to indicate a measure of engine RTT.
  • the egress engine 144 of the network interface card 140 can include a calculated engine RTT value for the most recently received packet as the most current indication of any congestion at the destination network device 110.
  • the network device 110 Upon the network interface card 140 of the original source network device 110 receiving an acknowledgement message, the network device 110 (either at the ingress engine 141 or the network interface driver 122) can calculate the total RTT for the acknowledged packet(s), as well as subtract the engine RTT included in the acknowledgement packet to break the total RTT into its engine RTT and fabric RTT components.
  • FIG 2 is a flow chart illustrating an example process 200 for managing network traffic in a data network, such as the data network 100 shown in Figure 1A .
  • the blocks of the example process 200 are described herein with reference to the components and/or processes described herein.
  • the blocks of the process 200 may be implemented, for example, by one or more components or processors of network devices 110 of Figure 1A or 1C .
  • the process 200 may be carried out by some combination of the network interface driver 122, ingress engine 141, and egress engine 144 of a network device 110.
  • one or more of the blocks may be implemented apart from other blocks, and by one or more different processors or controllers.
  • the blocks of the example process 200 are described as occurring serially, or linearly. However, multiple blocks of the example process 200 may occur in parallel. In addition, the blocks of the example process 200 need not be performed in the order shown and/or one or more of the blocks of the example process 200 need not be performed.
  • a current round-trip time is measured.
  • a RTT is the combination of an engine RTT and a fabric RTT.
  • the total RTT for a packet can be measured by a source network device 110 by subtracting an acknowledgement message receipt time stamp recorded by the source network device network interface card 140 from a stored packet transmission time stamp.
  • a total RTT time can be based on software time stamps stored when a packet is transmitted from the network interface driver 122 of the source network device to the network interface card 140 of the source network device 110 and the software time stamps stored when a packet acknowledgement message is passed from the source network device network interface card 140 to the source network device network interface driver 122.
  • a target RTT may be set by a network administrator or a traffic management application as a desired amount of time it should take for a packet to be successfully transmitted and acknowledged.
  • a single target total RTT can be set for all packets transmitted over a network.
  • different target total RTT values can be set for different traffic flows. For example, a different target total RTT values may be set for flows having different quality of service indicators. Alternatively, or in addition, target total RTT values may be set based on service level obligations (SLOs) agreed upon between a cloud service provider and a cloud tenant.
  • SLOs service level obligations
  • a base target total RTT may be set for one or more flows, which may then be increased based on other factors, including network topology and current network congestion levels.
  • a total target RTT in a data center environment may range from about 50 microseconds to about 200 microseconds. In some implementations, a total target RTT is set between about 100 microseconds and 200 microseconds.
  • the size of a congestion window for the network flow i.e., the set of packets associated with the same tuple of header information, e.g., a source internet protocol (IP) address, source port number, destination IP address, destination port number, and a protocol identifier
  • IP internet protocol
  • Decreasing the congestion window for the network flow limits the number of data packets being transmitted across the switches 115 and data links 120 carrying packets associated with the flow, and also limits the number of packets being processed by the destination network device.
  • the congestion window can be decreased by a fixed increment, for example by subtracting a fixed number.
  • the congestion window can be reduced by multiplying the current congestion window by a factor that is less than 1.
  • the size of a congestion window for data network 100 is increased to allow more packets to be transmitted over data network 100.
  • the congestion window can be increased by a fixed increment, for example by adding a fixed number.
  • the congestion window can be increased by multiplying the current congestion window by a factor that is greater than 1.
  • the congestion window size is only increased or decreased if the current RTT differs from the target RTT by more than a threshold percent (e.g., more than about 3% to more than about 10%, e.g., more than about 5%).
  • the data packets are transmitted over data network 100 based on the adjusted congestion window size (e.g., decreased or increased window congestion size). Then, the process 200 is repeated from block 202 when a next packet acknowledgement message is received.
  • the adjusted congestion window size e.g., decreased or increased window congestion size
  • Figure 3 is a flow chart illustrating another example process 300 for managing network traffic.
  • the process 300 is a more refined process for managing network traffic based on round trip time, separately evaluating engine round trip time and fabric round trip time.
  • the process 300 builds on the recognition that delays introduced by fabric congestion are different than delays introduced by engine congestion, and thus such delays should be evaluated and managed separately.
  • the one or more of the blocks of the process 300 may be implemented, for example, by one or more components or processors of network devices 110 of Figure 1 .
  • the process 300 may be carried out by some combination of the network interface driver 122, ingress engine 141, and egress engine 144 of a network device 110.
  • one or more of the blocks may be implemented apart from other blocks, and by one or more different processors or controllers.
  • the blocks of the example process 300 are described as occurring serially, or linearly. However, multiple blocks of the example process 300 may occur in parallel. In addition, the blocks of the example process 300 need not be performed in the order shown and/or one or more of the blocks of the example process 300 need not be performed.
  • an engine RTT and a fabric RTT are determined.
  • the engine RTT is the length of time it takes from when a destination network device receives a data packet to when the destination generates and transmits an acknowledgement of receipt of the data packet.
  • the fabric RTT is the amount of time the original data packet and the acknowledgment packet spend traversing the network 100 between the source and destination network devices 110.
  • the engine RTT can be calculated by the destination network interface card by the egress engine of the network interface card taking a difference between when a packet is received and a corresponding acknowledgement packet is transmitted. The RTT can then be added to the acknowledgement message upon transmission.
  • the engine RTT can be determined at block 301 by extracting the engine RTT from the acknowledgement message.
  • the fabric RTT can be calculated at block 302 by subtracting the engine RTT from the total RTT.
  • the total RTT can be determined by a source network interface card calculating a difference between a stored transmission time stamp for a packet and a time stamp associated with the receipt of a corresponding acknowledgement packet.
  • the total RTT can be determined based on software time stamps stored when a packet is forwarded to a network interface card and when an acknowledgement message is forwarded from the network interface card to the network interface driver of the source network device.
  • RTT analysis was carried out against a total target RTT.
  • two RTT analyses are conducted. One is based on a target engine RTT. The other is based on a target fabric RTT. As with the total target RTT, the same target engine RTT value and target fabric RTT value may be used for all flows. In some implementations, different target engine RTT values and/or fabric RTT values may be used based on a variety of factors, including tenant SLOs, quality of service indicators associated with the flow, or other factors. As discussed further below, in some implementations, target engine RTT values for a given flow may be adjusted based on current network load metrics (e.g., the current congestion window).
  • current network load metrics e.g., the current congestion window
  • the target fabric RTT may be adjusted based on network topology information (e.g., the number of data links or number of packet switches packets of the flow traverse). Suitable values for a target fabric RTT range from about 30 microseconds to about 180 microseconds. Suitable values for target engine RTT range from about 20 microseconds to about 100 microseconds.
  • a congestion window when decreased, can be decreased by a fixed increment, for example by subtracting a fixed number. In some implications, a congestion window can be reduced by multiplying the current congestion window by a factor that is less than 1. Similarly, when increased, in some implementations, the congestion window can be increased by a fixed increment, for example by adding a fixed number. In some implications, the congestion window can be increased by multiplying the current congestion window by a factor that is greater than 1.
  • either the engine-based congestion window or the fabric-based congestion window, whichever is smaller, is selected for use as the current congestion window.
  • the selected engine-based congestion window or fabric-based congestion window is then used to transmit packets at block 310.
  • Figure 4 is a flow chart illustrating and example process 400 for scaling a target RTT based on a network topology, according to one or more embodiments.
  • the process 400 can be used to scale a target total RTT suitable for use in the process 200 shown in Figure 2 .
  • the process 400 can be used to scale a target fabric RTT suitable for use in the process 300 shown in Figure 3 .
  • the various blocks of the example process 400 are described herein with reference to the components and/or processes described herein.
  • One or more of the blocks of the process 400 may be implemented, for example, by one or more components or processors of network devices 110 shown in Figure 1 .
  • the process 400 may be carried out by some combination of the network interface driver 122, ingress engine 141, and egress engine 144 of a network device 110.
  • one or more of the blocks may be implemented apart from other blocks, and by one or more different processors or controllers.
  • the blocks of the example process 400 are described as occurring serially, or linearly. However, multiple blocks of the example process 400 may occur in parallel.
  • the blocks of the example process 400 need not be performed in the order shown and/or one or more of the blocks of the example process 400 need not be performed.
  • the target RTT for a flow is obtained.
  • the target RTT may be a total target RTT or a target fabric RTT.
  • the number of hops packets associated with the flow traverse between the source and the destination is determined. The number of hops may correspond to the number of data links 120, the total number of switches 115, or the number of packet switches, data packets of the flow traverse from its source network device to its destination network device.
  • the number of hops may be determined by evaluating a TTL field included in an packet acknowledgement message or any other packet received from the destination network device in association with the flow (e.g., as part of a handshaking protocol used in setting up the flow), assuming that such packets traverse the same path the corresponding data packet originally traversed, but in reverse order.
  • the number of hops is determined by the source network device transmitting a traceroute or similar control packet to the destination network device.
  • a new target RTT is determined based on the obtained target RTT and the determined number of hops.
  • a configurable amount of time is added to the target RTT for each hop in the path traversed by packets in the flow. Suitable amounts of time range from between 3 microseconds per hop to 10 microseconds per hop.
  • the per-hop time value may be set by a traffic management application or a system administrator. The per-hop time value may be applied equally to all flows or different per-hop time values may be added to different flows based on, e.g., operative SLOs or the type of communication medium the packets traverse.
  • different per-hop time increments may be added based on whether the hops are optical or electrical. For example, for a flow traversing hops, and the per-hop time increment is five microseconds, the target RTT would be increased by 15 microseconds.
  • the process moves on to block 408 where the data packet transmission for the flow is managed based on the determined new target RTT.
  • the process 400 is repeated upon a determination that the number of hops associated with a flow has changed. For example, TTL values in packet acknowledgement messages may be regularly monitored for a change from prior TTL values to determine if a hop count has changed.
  • FIG. 5 is a flow chart illustrating another example process 500 for scaling a target RTT.
  • the process 500 scales a target RTT based on a current network load, rather than based on network topology.
  • the process 500 can be used, in various implementations, for scaling total target RTT values, fabric RTT values, and/or engine RTT values.
  • the various blocks of the example process 500 are described herein with reference to the components and/or processes described herein.
  • the process 500 may be carried out by some combination of the network interface driver 122, ingress engine 141, and egress engine 144 of a network device 110.
  • One or more of the blocks of process 500 may be implemented, for example, by one or more components or processors of network devices 110 of Figure 1A or 1C .
  • one or more of the blocks may be implemented apart from other blocks, and by one or more different processors or controllers.
  • the blocks of the example process 500 are described as occurring serially, or linearly. However, multiple blocks of the example process 500 may occur in parallel.
  • the blocks of the example process 500 need not be performed in the order shown and/or one or more of the blocks of the example process 500 need not be performed.
  • the base target RTT for a flow is obtained.
  • the target RTT for the flow can be a target total RTT, a target engine RTT, and/or a target fabric RTT, and can be set by a network administrator or traffic management application.
  • the size of the currently applied congestion window for the flow is obtained.
  • a scaled target RTT is determined based on the base target RTT and the currently applied congestion window. In this process, the current congestion window size is used as a proxy for the overall load on the portion of the network traversed by the flow. In other implementations, other metrics of network load may be employed instead of the congestion window.
  • the addition value or multiplication factor is inversely proportional to the size the of current congestion window.
  • the addition value or multiplication factor may be in the form of x ⁇ MaxCongestionWindowSize CurrentCongestionWindowSize , where x is a positive number.
  • the x is a positive number greater than or equal to 1.0.
  • x is a positive number less than or equal to 1.0.
  • the addition value or multiplication factor can be of the form of A CurrentCongestionWindowSize , here A is a positive number.
  • a ceiling and/or a floor may be included in the calculation of the new target RTT.
  • the new target RTT may have a ceiling of 300 microseconds a floor of about 50 microseconds.
  • network traffic is managed based on the new target RTT, e.g., as described in relation to Figures 2 and 3 .
  • the process may be carried out in response to the congestion window for the flow changing by more than a threshold percentage (e.g., 20-30%) from the congestion window used to calculate the new target RTT.
  • FIG 6 is a flow chart illustrating another example process 600 for scaling target RTT values.
  • the process 600 scales a target engine RTT based on a current network load and a target fabric RTT based on network topology.
  • the process 600 can be used, e.g., to determine appropriate target engine and fabric RTT values to use in the process 300 shown in Figure 3 .
  • the various blocks of the example process 600 are described herein with reference to the components and/or processes described herein.
  • One or more of the blocks of process 600 may be implemented, for example, by one or more components or processors of network devices 110 of Figure 1A or 1C .
  • the process 600 may be carried out by some combination of the network interface driver 122, ingress engine 141, and egress engine 144 of a network device 110.
  • one or more of the blocks may be implemented apart from other blocks, and by one or more different processors or controllers.
  • the blocks of the example process 600 are described as occurring serially, or linearly. However, multiple blocks of the example process 600 may occur in parallel.
  • the blocks of the example process 600 need not be performed in the order shown and/or one or more of the blocks of the example process 600 need not be performed.
  • the process 600 includes obtaining base target RTT(s) for a flow at block 602.
  • a base total target RTT is obtained, as described in relation to block 204 for Figure 2 .
  • a base target engine RTT and a base target fabric RTT are obtained at block 602. Obtaining a base target fabric RTT is discussed in relation to block 402 of Figure 4 . Obtaining a base target engine RTT is discussed in relation to block 502 of Figure 5 .
  • the process also includes obtaining the hop count for the flow and the currently applied congestion window size for the flow at block 604. These process steps can be carried out as described above in relation to blocks 404 and 504 of Figures 4 and 5 , respectively.
  • new target RTT(s) are determined based on the obtained base target RTT(s), the hop count, and the currently applied congestion window size for the flow.
  • a scaled target total RTT can be calculated by increasing the base target total RTT based on both the current congestion window size and the hop count associated with the flow. For example, the base target total RTT can be incremented based on the hop count as described in relation to block 406 of Figure 4 , and the resultant value can be further adjusted as described in relation to block 506 of Figure 5 based on the congestion window size.
  • the base target total RTT can first be adjusted based on congestion window size in accordance with block 506 of Figure 5 and then incremented based on hop count in accordance with block 406 of Figure 4 .
  • the target fabric RTT can be increased based on the obtained hop count for the flow as described in relation to block 406 of Figure 4 and the target fabric RTT can be increased based on the congestion window size as described in relation to block 506 of Figure 5 .
  • network traffic is managed based on the new target RTT(s).
  • the new target RTT(s) can be used for example, in the processes 200 and 300 shown in Figures 2 and 3 , respectively.
  • the process 600 may be carried out any time the congestion window for a flow changes.
  • the process 600 may be repeated in response to changes in topology (i.e., flow hop count) or flow congestion window size, as described above.
  • FIG 7 is a block diagram illustrating a general architecture for a computer system 900 that may be employed to implement elements of the systems and methods described and illustrated herein, according to an illustrative implementation.
  • the computing system 900 may be utilized in implementing the methods of operating, for example, network devices 110 shown in Figures 1-5 .
  • the computing system 910 includes at least one processor 950 for performing actions in accordance with instructions and one or more memory devices 970 or 975 for storing instructions and data.
  • the illustrated example computing system 910 includes one or more processors 950 in communication, via a bus 915, with at least one network interface controller 920 with one or more network interface ports 922 connecting to one or more network devices 924, memory 970, and any other devices 980, e.g., an I/O interface.
  • a processor 950 will execute instructions received from memory.
  • the processor 950 illustrated incorporates, or is directly connected to, cache memory 975.
  • the processor 950 may be any logic circuitry that processes instructions, e.g., instructions fetched from the memory 970 or cache 975.
  • the processor 950 is a microprocessor unit or special purpose processor.
  • the computing device 900 may be based on any processor, or set of processors, capable of operating as described herein.
  • the processor 950 can be capable of executing the methods operating network devices 110 shown in Figures 2-6 .
  • the processor 950 may be a single core or multi-core processor.
  • the processor 950 may be multiple processors.
  • the memory 970 may be any device suitable for storing computer readable data.
  • the memory 970 may be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media and memory devices, semiconductor memory devices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magnetic disks, magneto optical disks, and optical discs (e.g., CD ROM, DVD-ROM, and Blu-ray ® discs).
  • a computing system 900 may have any number of memory devices 970.
  • the memory 970 can include instructions corresponding to the method of methods operating network devices 110 shown in Figures 2-6 .
  • the memory 970 supports virtualized or containerized memory accessible by virtual machine or container execution environments provided by the computing system 910.
  • the cache memory 975 is generally a form of computer memory placed in close proximity to the processor 950 for fast read times. In some implementations, the cache memory 975 is part of, or on the same chip as, the processor 950. In some implementations, there are multiple levels of cache 975, e.g., L2 and L3 cache layers.
  • the network interface controller 920 manages data exchanges via the network interfaces 922 (also referred to as network interface ports).
  • the network interface controller 920 handles the physical and data link layers of the OSI model for network communication. In some implementations, some of the network interface controller's tasks are handled by the processor 950. In some implementations, the network interface controller 920 is part of the processor 950.
  • a computing system 910 has multiple network interface controllers 920.
  • the network interfaces 922 are connection points for physical network links.
  • the network interface controller 920 supports wireless network connections and an interface port 922 is a wireless receiver/transmitter.
  • a computing device 910 exchanges data with other network devices 924 via physical or wireless links to a network interfaces 922.
  • the network interface controller 920 implements a network protocol such as Ethernet.
  • the other network devices 924 are connected to the computing device 910 via a network interface port 922.
  • the other network devices 924 may be peer computing devices, network devices, or any other computing device with network functionality.
  • a first network device 924 may be a network device such as a hub, a bridge, a switch, or a router, connecting the computing device 910 to a data network such as the Internet.
  • the other devices 980 may include an I/O interface, external serial device ports, and any additional co-processors.
  • a computing system 910 may include an interface (e.g., a universal serial bus (USB) interface) for connecting input devices (e.g., a keyboard, microphone, mouse, or other pointing device), output devices (e.g., video display, speaker, or printer), or additional memory devices (e.g., portable flash drive or external media drive).
  • a computing device 900 includes an additional device 980 such as a coprocessor, e.g., a math co-processor can assist the processor 950 with high precision or complex calculations.
  • Implementations of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software embodied on a tangible medium, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs embodied on a tangible medium, i.e., one or more modules of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, a data processing apparatus.
  • a computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them.
  • the computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).
  • the computer storage medium may be tangible and non-transitory.
  • the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
  • the operations may be executed within the native environment of the data processing apparatus or within one or more virtual machines or containers hosted by the data processing apparatus.
  • a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment.
  • a computer program may, but need not, correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).
  • a computer program can be deployed to be executed on one computer or on multiple computers or one or more virtual machines or containers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • Examples of communication networks include a local area network ("LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
  • the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • references to "or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms.
  • the labels “first,” “second,” “third,” and so forth are not necessarily meant to indicate an ordering and are generally used merely to distinguish between like or similar items or elements.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Environmental & Geological Engineering (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Claims (10)

  1. Verfahren (400) zum Steuern einer Überlastung in einer Cloud-Computing-Umgebung, wobei Netzwerkgeräte mit mehreren virtualisierten Computing-Geräten innerhalb der Cloud-Computing-Umgebung assoziierten Verkehr handhaben, wobei das Verfahren folgendes umfasst:
    Erhalten (402) einer Basisziel-Paketumlaufzeit, RTT (= Round-Trip Time), für Pakete eines Netzwerkflusses, der von einem Quellnetzwerkgerät (110-1) zu einem Zielnetzwerkgerät (110-2) übertragene Pakete enthält, wobei die Basisziel-RTT eine erwünschte Menge an Zeit ist, die es für ein Paket dauern sollte, erfolgreich übertragen und bestätigt zu werden;
    Bestimmen (404) einer Anzahl von Hops, die mit dem Netzwerkfluss assoziierte Pakete zwischen dem Quellnetzwerkgerät (110-1) und dem Zielnetzwerkgerät (110-2) durchlaufen, wobei die Anzahl von Hops durch Auswerten eines TTL-Felds bestimmt wird, das in einem vom Zielnetzwerkgerät (110-2) in Assoziation mit dem Netzwerkfluss empfangenen Paket enthalten ist, unter Annahme, dass solche Paket denselben Pfad durchlaufen, den das Paket ursprünglich durchlief;
    Bestimmen (406) einer durch Topologie skalierten Ziel-RTT für den Netzwerkfluss basierend auf der Basisziel-RTT und der bestimmten Anzahl von Hops, wobei ein Bestimmen der durch Topologie skalierten Ziel-RTT ein Addieren eines Werts gleich einem Zeitinkrement pro Hop, multipliziert mit der bestimmten Anzahl von Hops, zur Basisziel-RTT umfasst; und
    Managen (408) einer Überlastungsfenstergröße für den Netzwerkfluss basierend auf der durch Topologie skalierten Ziel-RTT durch:
    Senden, durch eine Netzwerkschnittstellenkarte (140) des Quellnetzwerkgeräts (110-1), eines mit dem Netzwerkfluss assoziierten ersten Pakets zum Zielnetzwerkgerät (110-2) basierend auf einer ersten Überlastungsfenstergröße;
    Empfangen, durch die Netzwerkschnittstellenkarte (140) des Quellnetzwerkgeräts (110-1), eines zweiten Pakets, das eine Bestätigungsnachricht enthält, und eine Differenz zwischen einer Zeit, die anzeigt, wann das Zielnetzwerkgerät (110-2) das zweite Paket sendete, und einer beim Zielnetzwerkgerät (110-2) gespeicherten Zeit, die anzeigt, wann die Zielnetzwerkgerät (110-2) das erste Paket empfing, wobei die Differenz eine Engine-RTT anzeigt; und
    Einstellen der Überlastungsfenstergröße basierend auf einem Vergleich (204) der Engine-RTT mit der durch Topologie skalierten Ziel-RTT.
  2. Verfahren nach Anspruch 1, wobei in Reaktion darauf, dass die Engine-RTT größer ist (206) als die durch Topologie skalierte RTT, das Verfahren ein Reduzieren der Überlastungsfenstergröße umfasst; und/oder
    wobei in Reaktion darauf, dass die Engine-RTT nicht größer ist (208) als die durch Topologie skalierte RTT, das Verfahren ein Erhöhen der Überlastungsfenstergröße umfasst.
  3. Verfahren nach Anspruch 1, wobei ein Einstellen der Überlastungsfenstergröße basierend auf einem Vergleich der Engine-RTT mit der durch Topologie skalierten Ziel-RTT ein Einstellen der Überlastungsfenstergröße nur dann umfasst, wenn sich die Engine-RTT von der durch Topologie skalierten Ziel-RTT um mehr als ein Schwellenmaß unterscheidet.
  4. Verfahren nach Anspruch 3, wobei das Schwellenmaß zwischen etwa 3% und etwa 10% der durch Topologie skalierten Ziel-RTT liegt.
  5. Verfahren nach einem der Ansprüche 1 bis 4,
    das weiterhin ein Bestimmen umfasst, dass sich die Anzahl von Hops, die von mit dem Netzwerkfluss assoziierten Paketen vom Quellnetzwerkgerät zum Zielnetzwerkgerät durchlaufen sind, geändert hat, und in Reaktion darauf ein Berechnen einer zweiten durch Topologie skalierten Ziel-RTT für den Netzwerkfluss; und/oder
    das weiterhin ein Bestimmen einer zweiten durch Topologie skalierten Ziel-RTT für einen zweiten Netzwerkfluss umfasst, der vom Quellnetzwerkgerät (110-1) zu einem zweiten Zielnetzwerkgerät übertragene Pakete enthält, wobei das zweite Zielnetzwerkgerät unterschiedlich vom Zielnetzwerkgerät (110-2) ist und die zweite durch Topologie skalierte Ziel-RTT eine andere als die durch Topologie skalierte Ziel-RTT ist.
  6. System (910), umfassend:
    einen oder mehrere Prozessoren (950); und
    ein nichtflüchtiges maschinenlesbares Medium (970), das darin gespeicherte Anweisungen umfasst, die dann, wenn sie durch den einen oder die mehreren Prozessoren ausgeführt werden, veranlassen, dass der eine oder die mehreren Prozessoren Operationen durchführt oder durchführen, um eine Überlastung in einer Cloud-Computing-Umgebung zu steuern, wobei Netzwerkgeräte mit mehreren virtualisierten Computing-Geräten innerhalb der Cloud-Computing-Umgebung assoziierten Verkehr handhaben, wobei die Operationen folgendes umfassen:
    Erhalten (402) einer Basisziel-Paketumlaufzeit, RTT (= Round-Trip Time), für Pakete eines Netzwerkflusses, der von einem Quellnetzwerkgerät (110-1) zu einem Zielnetzwerkgerät (110-2) übertragene Pakete enthält, wobei die Basisziel-RTT eine erwünschte Menge an Zeit ist, die es für ein Paket dauern sollte, erfolgreich übertragen und bestätigt zu werden;
    Bestimmen (404) einer Anzahl von Hops, die mit dem Netzwerkfluss assoziierte Pakete zwischen dem Quellnetzwerkgerät (110-1) und dem Zielnetzwerkgerät (110-2) durchlaufen, wobei die Anzahl von Hops durch Auswerten eines TTL-Felds bestimmt wird, das in einem vom Zielnetzwerkgerät (110-2) in Assoziation mit dem Netzwerkfluss empfangenen Paket enthalten ist, unter Annahme, dass solche Paket denselben Pfad durchlaufen, den das Paket ursprünglich durchlief;
    Bestimmen (406) einer durch Topologie skalierten Ziel-RTT für den Netzwerkfluss basierend auf der Basisziel-RTT und der bestimmten Anzahl von Hops, wobei ein Bestimmen der durch Topologie skalierten Ziel-RTT ein Addieren eines Werts gleich einem Zeitinkrement pro Hop, multipliziert mit der bestimmten Anzahl von Hops, zur Basisziel-RTT umfasst; und
    Managen (408) einer Überlastungsfenstergröße für den Netzwerkfluss basierend auf der durch Topologie skalierten Ziel-RTT durch:
    Senden, durch eine Netzwerkschnittstellenkarte (140) des Quellnetzwerkgeräts (110-1), eines mit dem Netzwerkfluss assoziierten ersten Pakets zum Zielnetzwerkgerät (110-2) basierend auf einer ersten Überlastungsfenstergröße;
    Empfangen, durch die Netzwerkschnittstellenkarte (140) des Quellnetzwerkgeräts (110-1), eines zweiten Pakets, das eine Bestätigungsnachricht enthält, und eine Differenz zwischen einer Zeit, die anzeigt, wann das Zielnetzwerkgerät (110-2) das zweite Paket sendete, und einer beim Zielnetzwerkgerät (110-2) gespeicherten Zeit, die anzeigt, wann die Zielnetzwerkgerät (110-2) das erste Paket empfing, wobei die Differenz eine Engine-RTT anzeigt; und
    Einstellen der Überlastungsfenstergröße basierend auf einem Vergleich (204) der Engine-RTT mit der durch Topologie skalierten Ziel-RTT.
  7. System nach Anspruch 6, wobei in Reaktion darauf, dass die Engine-RTT größer ist (206) als die durch Topologie skalierte RTT, das Verfahren ein Reduzieren der Überlastungsfenstergröße umfasst; und/oder
    wobei in Reaktion darauf, dass die Engine-RTT nicht größer ist (208) als die durch Topologie skalierte RTT, das Verfahren ein Erhöhen der Überlastungsfenstergröße umfasst.
  8. System nach Anspruch 6, wobei ein Einstellen der Überlastungsfenstergröße basierend auf einem Vergleich der Engine-RTT mit der durch Topologie skalierten Ziel-RTT ein Einstellen der Überlastungsfenstergröße nur dann umfasst, wenn sich die Engine-RTT von der durch Topologie skalierten Ziel-RTT um mehr als ein Schwellenmaß unterscheidet.
  9. System nach Anspruch 8, wobei das Schwellenmaß zwischen etwa 3% und etwa 10% der durch Topologie skalierten Ziel-RTT liegt.
  10. System nach einem der Ansprüche 6 bis 9, wobei die Operationen weiterhin ein Bestimmen umfassen, dass sich die Anzahl von Hops, die von mit dem Netzwerkfluss assoziierten Paketen vom Quellnetzwerkgerät zum Zielnetzwerkgerät durchlaufen sind, geändert hat, und in Reaktion darauf ein Berechnen einer zweiten durch Topologie skalierten Ziel-RTT für den Netzwerkfluss; und/oder
    die Operationen weiterhin ein Bestimmen einer zweiten durch Topologie skalierten Ziel-RTT für einen zweiten Netzwerkfluss umfassen, der vom Quellnetzwerkgerät (110-1) zu einem zweiten Zielnetzwerkgerät übertragene Pakete enthält, wobei das zweite Zielnetzwerkgerät unterschiedlich vom Zielnetzwerkgerät (110-2) ist und die zweite durch Topologie skalierte Ziel-RTT eine andere als die durch Topologie skalierte Ziel-RTT ist.
EP19212316.4A 2019-06-27 2019-11-28 Überlastungssteuerung für rechenzentren-netzwerke mit niedriger latenz Active EP3758313B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US16/455,115 US11496404B2 (en) 2019-06-27 2019-06-27 Congestion control for low latency datacenter networks

Publications (2)

Publication Number Publication Date
EP3758313A1 EP3758313A1 (de) 2020-12-30
EP3758313B1 true EP3758313B1 (de) 2024-05-08

Family

ID=68732775

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19212316.4A Active EP3758313B1 (de) 2019-06-27 2019-11-28 Überlastungssteuerung für rechenzentren-netzwerke mit niedriger latenz

Country Status (4)

Country Link
US (2) US11496404B2 (de)
EP (1) EP3758313B1 (de)
DK (1) DK3758313T3 (de)
FI (1) FI3758313T3 (de)

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022054283A1 (ja) 2020-09-14 2022-03-17 日本電信電話株式会社 情報処理システム、情報処理方法およびプログラム
JP7549254B2 (ja) * 2020-09-14 2024-09-11 日本電信電話株式会社 情報処理システム、接続制御方法およびプログラム
WO2022064640A1 (ja) 2020-09-25 2022-03-31 日本電信電話株式会社 通信システム、振分装置、データ振分方法及びプログラム
WO2022070247A1 (ja) 2020-09-29 2022-04-07 日本電信電話株式会社 情報処理システム、情報処理方法およびプログラム
JP7620233B2 (ja) 2020-09-29 2025-01-23 日本電信電話株式会社 情報処理システムおよび情報処理方法
US12550037B2 (en) 2020-10-22 2026-02-10 Ntt, Inc. Optical path design apparatus, optical path design method and program
US12494859B2 (en) 2020-10-22 2025-12-09 Ntt, Inc. Optical path design apparatus, optical path design method and program
JP7485990B2 (ja) 2020-11-16 2024-05-17 日本電信電話株式会社 通信経路探索装置、通信経路探索方法、及びプログラム
CN114679413B (zh) * 2022-05-27 2022-08-09 鹏城实验室 异构网络的拥塞控制方法、装置、设备及存储介质
US20230123387A1 (en) * 2022-12-16 2023-04-20 Intel Corporation Window-based congestion control

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2001249893A1 (en) * 2000-04-06 2001-10-23 Rensselaer Polytechnic Institute System and method of source based multicast congestion control
JP5867160B2 (ja) * 2012-02-28 2016-02-24 富士通株式会社 通信制御装置、通信制御方法および通信制御プログラム
US20150319063A1 (en) * 2014-04-30 2015-11-05 Jive Communications, Inc. Dynamically associating a datacenter with a network device

Also Published As

Publication number Publication date
EP3758313A1 (de) 2020-12-30
DK3758313T3 (da) 2024-07-29
FI3758313T3 (fi) 2024-07-23
US11496404B2 (en) 2022-11-08
US11750525B2 (en) 2023-09-05
US20230033871A1 (en) 2023-02-02
US20200412657A1 (en) 2020-12-31

Similar Documents

Publication Publication Date Title
US11750525B2 (en) Congestion control for low latency datacenter networks
US10771389B2 (en) Virtual tunnel endpoints for congestion-aware load balancing
US10193810B2 (en) Congestion-aware load balancing
He et al. AC/DC TCP: Virtual congestion control enforcement for datacenter networks
JP6670109B2 (ja) ネットワーク内のスケーラブルなフロー及び輻輳制御
US11509593B2 (en) Congestion control for low latency datacenter networks
US11805071B2 (en) Congestion control processing method, packet forwarding apparatus, and packet receiving apparatus
WO2016162833A1 (en) Method and system for traffic pattern generation in a software-defined networking (sdn) system
US11533265B2 (en) Alleviating flow congestion at forwarding elements
US9832072B1 (en) Self-configuring computer network router
JP7657264B2 (ja) トランスポート制御プロトコルのトリップタイムの推定
WO2017213745A1 (en) Self-protecting computer network router with queue resource manager
Sreekumari et al. Transport protocols for data center networks: a survey of issues, solutions and challenges
Turull et al. Performance evaluation of OpenFlow controllers for network virtualization
JP2015533045A (ja) 通信システム、通信方法、情報処理装置、通信制御方法及びプログラム
M. Abdelmoniem et al. Reducing latency in multi-tenant data centers via cautious congestion watch
Tan et al. DCQUIC: flexible and reliable software-defined data center transport
US20160149817A1 (en) Analysis device
De Almeida et al. iRED: A disaggregated P4-AQM fully implemented in programmable data plane hardware
Ciko et al. LGC-ShQ: Datacenter congestion control with queueless load-based ECN marking
Bakhshi Evaluating Modern TCP Congestion Control Algorithms
US12526234B1 (en) Load balancing using base transport header
Wang et al. Application of recent reliable transport layer advancements to DoD SATCOM network challenges
HK40043996A (en) Predictable virtualized nic
Yu Analysis of High-speed Data Transfer Protocols

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20210629

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20220411

REG Reference to a national code

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: H04L0012801000

Ref country code: DE

Ref document number: 602019051857

Country of ref document: DE

Ipc: H04L0047120000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: H04L 47/27 20220101ALI20230628BHEP

Ipc: H04L 47/10 20220101ALI20230628BHEP

Ipc: H04L 47/25 20220101ALI20230628BHEP

Ipc: H04L 47/12 20220101AFI20230628BHEP

INTG Intention to grant announced

Effective date: 20230712

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20231023

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTC Intention to grant announced (deleted)
INTG Intention to grant announced

Effective date: 20231205

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602019051857

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: FI

Ref legal event code: FGE

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

Effective date: 20240726

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240809

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240909

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1685916

Country of ref document: AT

Kind code of ref document: T

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240909

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240808

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240908

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240809

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240808

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602019051857

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20250211

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20241128

REG Reference to a national code

Ref country code: CH

Ref legal event code: U11

Free format text: ST27 STATUS EVENT CODE: U-0-0-U10-U11 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20251201

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20251126

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20251128

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20251127

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FI

Payment date: 20251125

Year of fee payment: 7

Ref country code: DK

Payment date: 20251125

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20251125

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: BE

Payment date: 20251127

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: CH

Payment date: 20251201

Year of fee payment: 7

Ref country code: SE

Payment date: 20251127

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IE

Payment date: 20251127

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20191128

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20191128