US20190140962A1 - Receiver-directed computer network congestion control system - Google Patents
Receiver-directed computer network congestion control system Download PDFInfo
- Publication number
- US20190140962A1 US20190140962A1 US16/092,890 US201716092890A US2019140962A1 US 20190140962 A1 US20190140962 A1 US 20190140962A1 US 201716092890 A US201716092890 A US 201716092890A US 2019140962 A1 US2019140962 A1 US 2019140962A1
- Authority
- US
- United States
- Prior art keywords
- sender
- computer
- receiver
- message
- senders
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
- G06F15/17306—Intercommunication techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
- H04L47/2475—Traffic characterised by specific attributes, e.g. priority or QoS for supporting traffic characterised by the type of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/16—Arrangements for providing special services to substations
- H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
- H04L12/185—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast with management of multicast group membership
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/16—Arrangements for providing special services to substations
- H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
- H04L12/1886—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast with traffic restrictions for efficiency improvement, e.g. involving subnets or subdomains
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/28—Flow control; Congestion control in relation to timing considerations
- H04L47/283—Flow control; Congestion control in relation to timing considerations in response to processing delays, e.g. caused by jitter or round trip time [RTT]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Definitions
- the present application relates to computer network systems, and more specifically, to a computer network congestion control system.
- Data center network performance is critical for most modern web/cloud services whose backends run within data centers. Many interactive datacenter applications have tight latency requirements due to stringent service-level agreements which directly impact revenue (e.g., under 200 ms for Web Search). Datacenter network latency is a major component of the application latency. While the networks have employed TCP, Remote Direct Memory Access (RDMA) is known to reduce latencies significantly compared to TCP by bypassing the operating system via hardware support at the network interface (e.g., RDMA over Infiniband and RDMA over Converged Ethernet (RoCE) can cut TCP's latency by 10 ⁇ ). As such, RDMA has been proposed as a faster replacement for TCP in datacenter applications.
- RDMA Remote Direct Memory Access
- RDMA provides hop-by-hop flow control but not end-to-end congestion control. While such flow control results in a lossless network, any congestion causes the well-known tree saturation problem where congestion at a switch spreads upstream in a tree rooted at the congested switch. This upstream spreading occurs due to the backpressure exerted by flow control where a sender is blocked because of a blocked receiver and so on. Tree saturation blocks even unrelated flows and results in a precipitous drop in the network throughput.
- the second issue is the incast problem, where multiple flows collide at a switch causing queuing delays and long latency tails. While good network design and load balancing avoid persistent congestion, sporadic congestion is inevitable in real deployments. While such occasional congestion affects only a small fraction of the flows (e.g., 0.1%), datacenter applications' unique characteristics imply that this infrequent problem affecting only the tail of the latency distribution worsens the average case. Because the applications often aggregate replies from many nodes for each query, waiting for the slowest reply worsens overall response latency and, alternately, dropping the reply worsens response quality (e.g., because WebSearch aggregates replies from 1000 nodes, the 99.9th percentile reply latency affects the overall average response time).
- DCQCN Datacenter QCN
- ECN Early Congestion Notification
- TIMELY targets latency tail in RDMA and uses round-trip times (RTT) measurements, instead of ECN marks, as an indication of imminent congestion.
- RTT round-trip times
- bandwidth share indicator value comprises the number of senders seen by the receiver.
- the bandwidth share indicator value may comprise a percentage bandwidth share allocated to the sender computer to allow for varying priorities between senders.
- each sender may also include the incast degree, which is programmed in the application, to the receiver. This strategy enables the receiver to send back the sender count to all the senders as soon the first sender's packets arrive, even before the rest of the senders' packets arrive.
- the sender count and the incast degree look-ahead enable the receiver-directed system to achieve accurate and faster convergence of sending rates, without any repeated adjustments.
- a method of managing congestion in a computer network comprising maintaining a set of active senders in a network interface of a receiver computer in the computer network, receiving a message from a sender computer in the network, and providing an acknowledgement message to the sender computer, the acknowledgement message including a bandwidth share indicator value for the sender computer, the bandwidth share indicator based on the set of active senders.
- a computer server comprising a network interface card having a network interface memory.
- the network interface card is configured to maintain a set of active senders in the network interface card, receive a message from a sender computer in the network, and provide an acknowledgement message to the sender computer, the acknowledgement message including a bandwidth share indicator value for the sender computer, the bandwidth share indicator based on the set of active senders.
- a computer server comprising a network interface card having a network interface memory, the network interface card configured to send a message to a receiver computer over a computer network, receive an acknowledgement message from the receiver computer, the acknowledgement message including a bandwidth share indicator value for the computer server, the bandwidth share indicator based on the set of active senders, and set a message transmission rate from the computer server to the receiver computer based on the bandwidth share indicator value.
- network interface card comprising a computer processor and a network interface memory communicatively connected to the computer processor, wherein the computer processor is configured to maintain a set of active senders in the network interface card, receive a message from a sender computer in the network and provide an acknowledgement message to the sender computer, the acknowledgement message including a bandwidth share indicator value for the sender computer, the bandwidth share indicator based on the set of active senders.
- a network interface card comprising a computer processor an a network interface memory communicatively connected to the computer processor, wherein the computer processor is configured to send a message to a receiver computer over a computer network, receive an acknowledgement message from the receiver computer, the acknowledgement message including a bandwidth share indicator value for the computer server, the bandwidth share indicator based on the set of active senders, and set a message transmission rate from the computer server to the receiver computer based on the bandwidth share indicator value.
- FIG. 1 is a diagram illustrating a plurality of computer servers connected by a network in a data center according to various aspects.
- FIG. 2 a is a timing diagram illustrating a single receiver (D) receiving messages from two senders (S 1 and S 2 ) and the corresponding bandwidth share indicator messages according to various aspects.
- FIG. 2 a is a routing diagram illustrating the single receiver (D) receiving messages from two senders (S 1 and S 2 ) and the corresponding bandwidth share indicator messages according to various aspects.
- FIG. 3 a is a graph showing the interaction of long and short flows according to various aspects.
- FIG. 4 a is a flowchart illustrating a process for congestion control according to one embodiment.
- FIG. 4 b is a flowchart illustrating a process for congestion control according to one embodiment.
- FIG. 1 shows a data center 100 having a plurality of computer servers 120 connected by a network 122 according to one embodiment. Although only two servers 120 are shown in FIG. 1 , the data center 100 may have many more servers 120 .
- the network 122 may be configured to support Infiniband or RoCE (RDMA over converged Ethernet) for sending and receiving messages between the servers 120 .
- the computer servers may include a network interface card (NIC) 107 , a system central processing unit (CPU) 113 , and a system memory 109 .
- the NIC 107 may also comprise a network interface memory 115 and a processing unit 117 .
- the servers 120 are configured such that when a receiver server 120 receives a message from a sender server 120 over the network 122 , the receiver replies to the sender with an acknowledgement message (ACK).
- the receiver includes a value representing a bandwidth share indicator value for the sender server 120 .
- the bandwidth share indicator value comprises a value n representing the number of current senders.
- the bandwidth share indicator value may comprise other values, such as a percentage bandwidth share that is being allocated to the sender server 120 .
- a percentage bandwidth share may be allocated to the sender server 120 .
- certain senders may be allocated a higher or lower bandwidth share depending on assigned priorities.
- the term “line rate” shall be interpreted to mean the maximum rate at which a network interface card 107 in a receiver computer is able to process incoming messages.
- the bandwidth share indicator value may be implemented as a hardware circuit (e.g., a latch) in the NIC 107 separate from the network interface memory 115 . In other embodiments, the bandwidth share indicator value is maintained within the network interface memory 115 of the NIC 107 . Also, the congestion control methodology described herein may be implemented by software running on the NIC 107 processor 117 and memory 115 or may be implemented by a dedicated hardware circuit within the NIC 107 . Further, the congestion control methodology described herein may be implemented by software running on the server 120 system processor 113 and system memory 109 .
- FIGS. 2 a and 2 b illustrate one example where the computer servers 120 are configured such that a single receiver (D in FIGS. 2 a and 2 b , which may be one of servers 120 ) is receiving a steady long flow from one sender (S 1 , which may be another one of servers 120 ) at a line rate.
- the sender S 1 continues to transmit at the line rate without throttling as it sees the n value (which represents the number of current senders) remain at 1 in the ACKs from the receiver D.
- the ACKs cause the sender NICs to throttle their transmission rate to 1/n (in the case of 2 senders, the rate would be reduced to 1 ⁇ 2 of the line rate), which can be sustained in steady state.
- the system 100 thereby provides two key benefits.
- the continuous feedback mechanism means that congestion control feed-back to senders is fast.
- the senders are given an accurate and precise rate not to exceed.
- the disclosed process seamlessly handles flow “churn” by constantly or periodically sending updated bandwidth share indicator values.
- FIGS. 2 a and 2 b are for one simplified example in which the senders S 1 and S 2 have equal priority which results in an equal bandwidth share.
- the NIC 107 sends a value that indicates the share of bandwidth that each sender may use. For example, if sender S 1 was to be given twice as much bandwidth as sender S 2 because S 1 has higher priority than S 2 , then the receiver D would do a weighted count and would communicate the number 3 instead of 2. The receivers would then divide their local priority (2 for S 1 and 1 for S 2 ) to arrive at their bandwidth fractions of 2 ⁇ 3 and 1 ⁇ 3 respectively.
- the receiver D can compute a function of recent sender history and send that resulting value to the senders in the acknowledgement messages. Senders, in turn may use another function to deduce bandwidth share from that value.
- the illustrated examples assume that the receiver-side function is the sender-count and the sender side function is the line rate divided by the sender count.
- the functions in [0027] above ensure that the sum of bandwidth sent to all senders matches the receiver's line rate. (If the sum exceeds line rate, there may be network congestion. Similarly, if the sum falls short of the line rate, network bandwidth goes underutilized.)
- FIG. 3( a ) illustrates the interaction of long and short flows according to one embodiment.
- the presence of short flows can be treated as a case of flow-churn; the long flows throttle their rates according to the number of short flows, but only for the duration of the short flows, as shown in FIG. 3( b ) .
- the disclosed system 100 addresses one other challenge of short flows: accurate counting of flows.
- two incasts to the same destination say D
- senders of the two incasts ender S is part of both incast groups.
- S's two incast flows would be serialized at S's NIC
- D's NIC should count S exactly once when determining n. If the disclosed system counts each flow as a sender, the two flows from S would artificially bloat n resulting in lost throughput.
- this single connection would be assumed to be active for the entire duration of the two real short flows (including any timegap between messages).
- the disclosed system 100 maintains a list of the senders having at least one active message flow, referred to herein as the active flow set (AFS).
- AFS active flow set
- the flowchart in FIG. 4 a describes a process 400 for AFS tracking operation for messages from sender S (which may be one of the servers 120 ) according to one embodiment.
- the NIC 107 determines if the sender of the message is already present in the AFS (stage 404 , FIG. 4 a ). If not, the sender of the message is added to the AFS and a message count associated with that sender is initialized to 1 (stage 406 ). If the sender associated with the received message is already in the AFS, the message count associated with that sender is incremented (stage 408 ), and the system is ready to process the next message (stage 410 ).
- 4 b shows a similar process 420 for processing messages when the message is completed (e.g., a message end marker is received).
- the message count associated with the sender is decremented (stage 424 ).
- the system determines whether the message count associated with the sender is equal to 0 (stage 426 ). If not, the system is ready to process the next message (stage 430 ).
- the system decrements the sender count for the AFS only when all the messages from that sender terminate (stage 428 ), as shown in FIG. 4( b ) .
- the system 100 may provide a look-ahead notification of an upcoming set of incast flows that are part of an incast group. For example, if each incast message carries (1) information that it is part of a 20-flow incast and (2) the list of the 20 senders, the receiver NIC can advertise rate limits that incorporate 20 senders even before all 20 senders' messages arrive at the receiver. In other words, the AFS can be populated with the set of senders in advance of actual message arrival. This results in faster backoff on the part of the long flows as shown by the dotted line in FIG. 3( b ) . For accurate counting, the system may treat any flow as if it begins when the lookahead notification first arrives. The ending of flows may be handled similarly as in the case of FIG. 4 without the lookahead.
- the disclosed system may utilize DCQCN which uses ECN marks to achieve congestion control for in-network congestion.
- DCQCN uses ECN marks to achieve congestion control for in-network congestion.
- the NIC cannot distinguish between the ECN marks due to in-network congestion and the ECN marks at the last-hop switch due to end-point contention (even end-point contention results in ECN marks)
- additional handling is implemented by the disclosed system to account for this ambiguity.
- the ECN-based congestion control mechanisms may overthrottle the sending rates even if the congestion is limited to the end-points.
- the disclosed system may include a test to distinguish between in-network congestion and end-point congestion.
- the sytsem infers that the ECN marks are for in-network contention and includes them in the ACKs. However, if the incoming traffic is at the line rate, but the packets are ECN-marked, the system infers that end-point contention is responsible, in which case the system will include the n values without any ECN mark on the returning ACKs to the sender.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
Description
- The present application is related to and claims the priority benefit of U.S. Provisional Patent Application Ser. No. 62/312,097, filed Mar. 23, 2016, the contents of which is hereby incorporated by reference in its entirety into the present disclosure.
- The present application relates to computer network systems, and more specifically, to a computer network congestion control system.
- Data center network performance is critical for most modern web/cloud services whose backends run within data centers. Many interactive datacenter applications have tight latency requirements due to stringent service-level agreements which directly impact revenue (e.g., under 200 ms for Web Search). Datacenter network latency is a major component of the application latency. While the networks have employed TCP, Remote Direct Memory Access (RDMA) is known to reduce latencies significantly compared to TCP by bypassing the operating system via hardware support at the network interface (e.g., RDMA over Infiniband and RDMA over Converged Ethernet (RoCE) can cut TCP's latency by 10×). As such, RDMA has been proposed as a faster replacement for TCP in datacenter applications.
- Employing RDMA in datacenters, however, poses two challenges. First, RDMA provides hop-by-hop flow control but not end-to-end congestion control. While such flow control results in a lossless network, any congestion causes the well-known tree saturation problem where congestion at a switch spreads upstream in a tree rooted at the congested switch. This upstream spreading occurs due to the backpressure exerted by flow control where a sender is blocked because of a blocked receiver and so on. Tree saturation blocks even unrelated flows and results in a precipitous drop in the network throughput. While this problem is well-known in multiprocessor interconnection networks, those solutions are not applicable to data centers because they (a) do not offer end-to-end congestion control, (b) rely on observing the state of the entire network whose scale is much smaller than that of datacenter networks, and (c) employ common, network-wide throttling as response whereas datacenters need per-flow response to maintain high throughput.
- The second issue is the incast problem, where multiple flows collide at a switch causing queuing delays and long latency tails. While good network design and load balancing avoid persistent congestion, sporadic congestion is inevitable in real deployments. While such occasional congestion affects only a small fraction of the flows (e.g., 0.1%), datacenter applications' unique characteristics imply that this infrequent problem affecting only the tail of the latency distribution worsens the average case. Because the applications often aggregate replies from many nodes for each query, waiting for the slowest reply worsens overall response latency and, alternately, dropping the reply worsens response quality (e.g., because WebSearch aggregates replies from 1000 nodes, the 99.9th percentile reply latency affects the overall average response time). This problem exists in TCP where incast causes packet drops and retransmissions which result in delays. While RDMA is lossless and hence does not incut packet drops and retransmissions, incast-induced queuing delays still affect RDMA's latency tail.
- Recent work in RDMA-based datacenter networks target these two problems. For example, Datacenter QCN (DCQCN) proposes congestion control for RDMA. DCQCN relies on Early Congestion Notification (ECN) marks to detect the onset of congestion. Another work, called TIMELY, targets latency tail in RDMA and uses round-trip times (RTT) measurements, instead of ECN marks, as an indication of imminent congestion. Both DCQCN and TIMELY retain TCP's approach of constantly adjusting the sending rates and cut back the rates upon imminent congestion to reduce the impact on latency. Unfortunately, because ECN marks and RTT measurements need a few round-trips to take effect, the schemes are slow to improve latency of the applications predominantly short flows (e.g., each flow lasts only a handful of round-trips). At the same time, the schemes also incur throughput loss due to overshooting and undershooting of the sending rates during the adjustment. Therefore, improvements are needed in the field.
- The present disclosure provides receiver-directed apportioning by adding a bandwidth share indicator value to the acknowledgement messages sent by the receiver to the senders. In certain embodiments, bandwidth share indicator value comprises the number of senders seen by the receiver. In other embodiments, the bandwidth share indicator value may comprise a percentage bandwidth share allocated to the sender computer to allow for varying priorities between senders. In the acknowledgement message, each sender may also include the incast degree, which is programmed in the application, to the receiver. This strategy enables the receiver to send back the sender count to all the senders as soon the first sender's packets arrive, even before the rest of the senders' packets arrive. Thus, the sender count and the incast degree look-ahead enable the receiver-directed system to achieve accurate and faster convergence of sending rates, without any repeated adjustments.
- According to one aspect, a method of managing congestion in a computer network is provided, comprising maintaining a set of active senders in a network interface of a receiver computer in the computer network, receiving a message from a sender computer in the network, and providing an acknowledgement message to the sender computer, the acknowledgement message including a bandwidth share indicator value for the sender computer, the bandwidth share indicator based on the set of active senders.
- According to another aspect, a computer server is provided, comprising a network interface card having a network interface memory. The network interface card is configured to maintain a set of active senders in the network interface card, receive a message from a sender computer in the network, and provide an acknowledgement message to the sender computer, the acknowledgement message including a bandwidth share indicator value for the sender computer, the bandwidth share indicator based on the set of active senders.
- According to another aspect, a computer server is provided, comprising a network interface card having a network interface memory, the network interface card configured to send a message to a receiver computer over a computer network, receive an acknowledgement message from the receiver computer, the acknowledgement message including a bandwidth share indicator value for the computer server, the bandwidth share indicator based on the set of active senders, and set a message transmission rate from the computer server to the receiver computer based on the bandwidth share indicator value.
- According to another aspect, network interface card is provided, comprising a computer processor and a network interface memory communicatively connected to the computer processor, wherein the computer processor is configured to maintain a set of active senders in the network interface card, receive a message from a sender computer in the network and provide an acknowledgement message to the sender computer, the acknowledgement message including a bandwidth share indicator value for the sender computer, the bandwidth share indicator based on the set of active senders.
- According to another aspect, a network interface card is provided, comprising a computer processor an a network interface memory communicatively connected to the computer processor, wherein the computer processor is configured to send a message to a receiver computer over a computer network, receive an acknowledgement message from the receiver computer, the acknowledgement message including a bandwidth share indicator value for the computer server, the bandwidth share indicator based on the set of active senders, and set a message transmission rate from the computer server to the receiver computer based on the bandwidth share indicator value.
- In the following description and drawings, identical reference numerals have been used, where possible, to designate identical features that are common to the drawings.
-
FIG. 1 is a diagram illustrating a plurality of computer servers connected by a network in a data center according to various aspects. -
FIG. 2a is a timing diagram illustrating a single receiver (D) receiving messages from two senders (S1 and S2) and the corresponding bandwidth share indicator messages according to various aspects. -
FIG. 2a is a routing diagram illustrating the single receiver (D) receiving messages from two senders (S1 and S2) and the corresponding bandwidth share indicator messages according to various aspects. -
FIG. 3a is a graph showing the interaction of long and short flows according to various aspects. -
FIG. 4a is a flowchart illustrating a process for congestion control according to one embodiment. -
FIG. 4b is a flowchart illustrating a process for congestion control according to one embodiment. - The attached drawings are for purposes of illustration and are not necessarily to scale.
- In the following description, some aspects will be described in terms that would ordinarily be implemented as software programs. Those skilled in the art will readily recognize that the equivalent of such software can also be constructed in hardware, firmware, or micro-code. Because data-manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, systems and methods described herein. Other aspects of such algorithms and systems, and hardware or software for producing and otherwise processing the signals involved therewith, not specifically shown or described herein, are selected from such systems, algorithms, components, and elements known in the art. Given the systems and methods as described herein, software not specifically shown, suggested, or described herein that is useful for implementation of any aspect is conventional and within the ordinary skill in such arts.
-
FIG. 1 shows adata center 100 having a plurality ofcomputer servers 120 connected by anetwork 122 according to one embodiment. Although only twoservers 120 are shown inFIG. 1 , thedata center 100 may have manymore servers 120. Thenetwork 122 may be configured to support Infiniband or RoCE (RDMA over converged Ethernet) for sending and receiving messages between theservers 120. As shown, the computer servers may include a network interface card (NIC) 107, a system central processing unit (CPU) 113, and asystem memory 109. TheNIC 107 may also comprise anetwork interface memory 115 and aprocessing unit 117. - The
servers 120 are configured such that when areceiver server 120 receives a message from asender server 120 over thenetwork 122, the receiver replies to the sender with an acknowledgement message (ACK). As part of the acknowledgement message, the receiver includes a value representing a bandwidth share indicator value for thesender server 120. In certain embodiments, the bandwidth share indicator value comprises a value n representing the number of current senders. Once it receives the ACK message with the n value, the sender sets its send rate to be 1/n. In this way, the rate of messages received by the receiver from all senders will be reduced to the line rate, thereby alleviating congestion at that receiver. In other embodiments, the bandwidth share indicator value may comprise other values, such as a percentage bandwidth share that is being allocated to thesender server 120. For example, certain senders may be allocated a higher or lower bandwidth share depending on assigned priorities. As used herein, the term “line rate” shall be interpreted to mean the maximum rate at which anetwork interface card 107 in a receiver computer is able to process incoming messages. - In certain embodiments, the bandwidth share indicator value may be implemented as a hardware circuit (e.g., a latch) in the
NIC 107 separate from thenetwork interface memory 115. In other embodiments, the bandwidth share indicator value is maintained within thenetwork interface memory 115 of theNIC 107. Also, the congestion control methodology described herein may be implemented by software running on theNIC 107processor 117 andmemory 115 or may be implemented by a dedicated hardware circuit within theNIC 107. Further, the congestion control methodology described herein may be implemented by software running on theserver 120system processor 113 andsystem memory 109. -
FIGS. 2a and 2b illustrate one example where thecomputer servers 120 are configured such that a single receiver (D inFIGS. 2a and 2b , which may be one of servers 120) is receiving a steady long flow from one sender (S1, which may be another one of servers 120) at a line rate. The sender S1 continues to transmit at the line rate without throttling as it sees the n value (which represents the number of current senders) remain at 1 in the ACKs from the receiver D. When a second sender (S2, which may also be one of the servers 120) initiates another long flow to the same receiver D, there is contention at the leaf-level switch 202 (which is one of many network switches 201 in the network 122), as shown inFIG. 2(b) , where the solid and broken lines show the two flows. As messages from each of the two flows arrive at the destination receiver server D in an interleaved fashion, theNIC 107 in the receiver D includes the updated n=2 value with the ACKs to each sender (S1 and S2). The ACKs cause the sender NICs to throttle their transmission rate to 1/n (in the case of 2 senders, the rate would be reduced to ½ of the line rate), which can be sustained in steady state. Thesystem 100 thereby provides two key benefits. First, the continuous feedback mechanism means that congestion control feed-back to senders is fast. Second, the senders are given an accurate and precise rate not to exceed. The disclosed process seamlessly handles flow “churn” by constantly or periodically sending updated bandwidth share indicator values. - The example in
FIGS. 2a and 2b are for one simplified example in which the senders S1 and S2 have equal priority which results in an equal bandwidth share. More generally, theNIC 107 sends a value that indicates the share of bandwidth that each sender may use. For example, if sender S1 was to be given twice as much bandwidth as sender S2 because S1 has higher priority than S2, then the receiver D would do a weighted count and would communicate thenumber 3 instead of 2. The receivers would then divide their local priority (2 for S1 and 1 for S2) to arrive at their bandwidth fractions of ⅔ and ⅓ respectively. - In general, the receiver D can compute a function of recent sender history and send that resulting value to the senders in the acknowledgement messages. Senders, in turn may use another function to deduce bandwidth share from that value. For simplicity, the illustrated examples assume that the receiver-side function is the sender-count and the sender side function is the line rate divided by the sender count.
- In the preferred method, the functions in [0027] above ensure that the sum of bandwidth sent to all senders matches the receiver's line rate. (If the sum exceeds line rate, there may be network congestion. Similarly, if the sum falls short of the line rate, network bandwidth goes underutilized.)
- For short flows interacting with long flows, the
system 100 may implement the same mechanism to ensure that the latency of short-flows is not hurt.FIG. 3(a) illustrates the interaction of long and short flows according to one embodiment. A long flow that contends with k other short flows is directed to reduce its sending rate to 1/(k+1) because the total number of senders is n=k+1. While this throttling helps the short flows' latency, such throttling is short-lived and does not hurt the long flow's throughput. The presence of short flows can be treated as a case of flow-churn; the long flows throttle their rates according to the number of short flows, but only for the duration of the short flows, as shown inFIG. 3(b) . - The disclosed
system 100 addresses one other challenge of short flows: accurate counting of flows. Consider a case where two incasts to the same destination (say D) begin close in time and there is an overlap in the senders of the two incasts (sender S is part of both incast groups). Because S's two incast flows would be serialized at S's NIC, D's NIC should count S exactly once when determining n. If the disclosed system counts each flow as a sender, the two flows from S would artificially bloat n resulting in lost throughput. Consider another case where two short flows use one persistent connection but there are no flow begin/end markers. Then, this single connection would be assumed to be active for the entire duration of the two real short flows (including any timegap between messages). Such an assumption would lead to overcounting of flows in the idle period between flows, resulting in throughput loss. To address the first case, in certain embodiments the disclosedsystem 100 maintains a list of the senders having at least one active message flow, referred to herein as the active flow set (AFS). - The flowchart in
FIG. 4a describes aprocess 400 for AFS tracking operation for messages from sender S (which may be one of the servers 120) according to one embodiment. Upon receiving a new message (e.g., begin of a new message), theNIC 107 determines if the sender of the message is already present in the AFS (stage 404,FIG. 4a ). If not, the sender of the message is added to the AFS and a message count associated with that sender is initialized to 1 (stage 406). If the sender associated with the received message is already in the AFS, the message count associated with that sender is incremented (stage 408), and the system is ready to process the next message (stage 410). The flowchart ofFIG. 4b shows asimilar process 420 for processing messages when the message is completed (e.g., a message end marker is received). After a message end marker has been received (stage 422), the message count associated with the sender is decremented (stage 424). The system then determines whether the message count associated with the sender is equal to 0 (stage 426). If not, the system is ready to process the next message (stage 430). When multiple short messages from the same sender are concurrently active, the system decrements the sender count for the AFS only when all the messages from that sender terminate (stage 428), as shown inFIG. 4(b) . With the above tracking in place, the disclosed receiver-directedrate control system 100 can use the number of elements in the AFS as the n value (i.e., n=|AFS|), which is transmitted by the receiver to the sender by theNIC 107 as part of the ACK message. - In certain embodiments, the
system 100 may provide a look-ahead notification of an upcoming set of incast flows that are part of an incast group. For example, if each incast message carries (1) information that it is part of a 20-flow incast and (2) the list of the 20 senders, the receiver NIC can advertise rate limits that incorporate 20 senders even before all 20 senders' messages arrive at the receiver. In other words, the AFS can be populated with the set of senders in advance of actual message arrival. This results in faster backoff on the part of the long flows as shown by the dotted line inFIG. 3(b) . For accurate counting, the system may treat any flow as if it begins when the lookahead notification first arrives. The ending of flows may be handled similarly as in the case ofFIG. 4 without the lookahead. - Since in-network congestion (as opposed to end-point congestion) is not visible at the NIC, the disclosed system may utilize DCQCN which uses ECN marks to achieve congestion control for in-network congestion. However, because the NIC cannot distinguish between the ECN marks due to in-network congestion and the ECN marks at the last-hop switch due to end-point contention (even end-point contention results in ECN marks), additional handling is implemented by the disclosed system to account for this ambiguity. Without any additional safeguards, the ECN-based congestion control mechanisms may overthrottle the sending rates even if the congestion is limited to the end-points. To avoid such overthrottling, the disclosed system may include a test to distinguish between in-network congestion and end-point congestion. If the incoming datarate at the receiver's NIC is below the line rate, the sytsem infers that the ECN marks are for in-network contention and includes them in the ACKs. However, if the incoming traffic is at the line rate, but the packets are ECN-marked, the system infers that end-point contention is responsible, in which case the system will include the n values without any ECN mark on the returning ACKs to the sender.
- The invention is inclusive of combinations of the aspects described herein. References to “a particular aspect” and the like refer to features that are present in at least one aspect of the invention. Separate references to “an aspect” (or “embodiment”) or “particular aspects” or the like do not necessarily refer to the same aspect or aspects; however, such aspects are not mutually exclusive, unless so indicated or as are readily apparent to one of skill in the art. The use of singular or plural in referring to “method” or “methods” and the like is not limiting. The word “or” is used in this disclosure in a non-exclusive sense, unless otherwise explicitly noted.
- The invention has been described in detail with particular reference to certain preferred aspects thereof, but it will be understood that variations, combinations, and modifications can be effected by a person of ordinary skill in the art within the spirit and scope of the invention.
Claims (26)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/092,890 US11190454B2 (en) | 2016-03-23 | 2017-03-23 | Receiver-directed computer network congestion control system |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662312097P | 2016-03-23 | 2016-03-23 | |
PCT/US2017/023910 WO2017165711A1 (en) | 2016-03-23 | 2017-03-23 | Receiver-directed computer network congestion control system |
US16/092,890 US11190454B2 (en) | 2016-03-23 | 2017-03-23 | Receiver-directed computer network congestion control system |
Publications (2)
Publication Number | Publication Date |
---|---|
US20190140962A1 true US20190140962A1 (en) | 2019-05-09 |
US11190454B2 US11190454B2 (en) | 2021-11-30 |
Family
ID=59900789
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/092,890 Active US11190454B2 (en) | 2016-03-23 | 2017-03-23 | Receiver-directed computer network congestion control system |
Country Status (2)
Country | Link |
---|---|
US (1) | US11190454B2 (en) |
WO (1) | WO2017165711A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11102129B2 (en) * | 2018-09-09 | 2021-08-24 | Mellanox Technologies, Ltd. | Adjusting rate of outgoing data requests for avoiding incast congestion |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022228641A1 (en) * | 2021-04-26 | 2022-11-03 | Huawei Technologies Co., Ltd. | Distribution of computation tasks over a plurality of nodes in a computer network |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8024417B2 (en) * | 2008-06-04 | 2011-09-20 | Microsoft Corporation | Simple flow control protocol over RDMA |
US20100183024A1 (en) * | 2009-01-21 | 2010-07-22 | Brocade Communications Systems, Inc | Simplified rdma over ethernet and fibre channel |
US20120054362A1 (en) * | 2010-08-31 | 2012-03-01 | Canon Kabushiki Kaisha | Mechanism for autotuning mass data transfer from a sender to a receiver over parallel connections |
US8873385B2 (en) | 2010-12-07 | 2014-10-28 | Microsoft Corporation | Incast congestion control in a network |
US9331955B2 (en) * | 2011-06-29 | 2016-05-03 | Microsoft Technology Licensing, Llc | Transporting operations of arbitrary size over remote direct memory access |
US9455927B1 (en) * | 2012-10-25 | 2016-09-27 | Sonus Networks, Inc. | Methods and apparatus for bandwidth management in a telecommunications system |
-
2017
- 2017-03-23 WO PCT/US2017/023910 patent/WO2017165711A1/en active Application Filing
- 2017-03-23 US US16/092,890 patent/US11190454B2/en active Active
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11102129B2 (en) * | 2018-09-09 | 2021-08-24 | Mellanox Technologies, Ltd. | Adjusting rate of outgoing data requests for avoiding incast congestion |
Also Published As
Publication number | Publication date |
---|---|
WO2017165711A1 (en) | 2017-09-28 |
US11190454B2 (en) | 2021-11-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11362960B2 (en) | Layer 3 fair rate congestion control notification | |
Munir et al. | Minimizing flow completion times in data centers | |
US9497125B2 (en) | Congestion control enforcement in a virtualized environment | |
EP2537301B1 (en) | Control of packet transfer through a multipath session comprising a single congestion window | |
EP3120253B1 (en) | Flow aware buffer management for data center switches | |
CN109039936B (en) | Transmission rate control method, device, sending equipment and receiving equipment | |
US20120314578A1 (en) | Flow-Based Adaptive Private Network With Multiple WAN-Paths | |
US20060203730A1 (en) | Method and system for reducing end station latency in response to network congestion | |
CN111835649B (en) | Data transmission method, device and related components | |
KR20210028722A (en) | Congestion control method and network device | |
CN112019395B (en) | Method, network device and system for measurement of network | |
Xue et al. | Dart: Divide and specialize for fast response to congestion in RDMA-based datacenter networks | |
WO2017114231A1 (en) | Packet sending method, tcp proxy, and tcp client | |
Sreekumari et al. | Transport protocols for data center networks: a survey of issues, solutions and challenges | |
Arslan et al. | Bolt:{Sub-RTT} Congestion Control for {Ultra-Low} Latency | |
CN111224888A (en) | Method for sending message and message forwarding equipment | |
US11190454B2 (en) | Receiver-directed computer network congestion control system | |
Irteza et al. | On the coexistence of transport protocols in data centers | |
Halepoto et al. | Scheduling over dissimilar paths using CMT-SCTP | |
EP3108631B1 (en) | Buffer bloat control | |
Arianfar et al. | Deadline-based resource management for information-centric networks | |
Xue et al. | Fast congestion control in RDMA-based datacenter networks | |
Lu et al. | EQF: An explicit queue-length feedback for TCP congestion control in datacenter networks | |
Zhang et al. | Efficient data center flow scheduling without starvation using expansion ratio | |
Lautenschlaeger | A deterministic tcp bandwidth sharing model |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
AS | Assignment |
Owner name: PURDUE RESEARCH FOUNDATION, INDIANA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:VIJAYKUMAR, TERANI N;VAMANAN, BALAJEE;THOTTETHODI, MITHUNA SHAMABHAT;SIGNING DATES FROM 20210304 TO 20210305;REEL/FRAME:055916/0737 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
AS | Assignment |
Owner name: PURDUE RESEARCH FOUNDATION, INDIANA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:XUE, JIACHEN;REEL/FRAME:057788/0927 Effective date: 20210507 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |