JP2024508220A - System and method for push data communication - Google Patents

Abstract

A data packet communication system is described that can be implemented as a physical push data packet communication device, such as a router, a gateway, or a controller circuit coupled to a router or gateway adapted to control data packet communication. The data packet communication system is adapted to assess network capacity of each of the plurality of networks at the time of a monitored communication event and allocate data packets accordingly. [Selection diagram] Figure 1F

Description

CROSS REFERENCES This application is a reference to U.S. Patent Application No. 63/139,286 and No. No. 63/183,952 and claims all benefits, including priority, thereof, which are incorporated herein by reference in their entirety.

This application incorporates by reference the following applications in their entirety:
PCT Application No. PCT/CA2017/051584, filed on December 21, 2017, entitled “PACKET TRANSMISSION SYSTEM AND METHOD”.

PCT Application No. PCT/CA2018/051012, filed on August 22, 2018, entitled “SYSTEM AND METHOD FOR ASSESESSING COMMUNICATION RESOURCES”.

PCT Application No. PCT/CA2020/051090, filed on August 7, 2020, entitled “SYSTEMS AND METHODS FOR MANAGING DATA PACKET COMMUNICATIONS”.

Embodiments of the present disclosure relate to the field of electronic data packet communications, and more specifically, embodiments relate to devices, systems, and methods for push data packet communications where multiple network connections are available.

Introduction Existing data packet communication solutions for the transmission of data packets over multiple available network connections rely on static routing logic to determine how to convey data packets across the connections. There is. For example, the system may employ a timer for each available connection and use each timer to determine when to remove packets from the input queue for transmission on the connection associated with the timer. Decide whether to pull. These solutions require that these connections trade performance for simplicity of implementation, have similar operational characteristics and availability, and use the next available transmission over a connection with the most recently expired timer. Assume that you want to schedule data packets. Although the timer approach is simple, it can compromise computational efficiency in certain situations.

Communication systems that treat multiple network connections as multiple instances of connections with similar operational characteristics (e.g., LACP, ML-PPP) may result in certain networks being underutilized or overutilized. As such, if multiple networks have different operating characteristics, it may lead to less effective use of multiple connections. This inaccuracy of use can result in greater latency of data packet delivery, less reliable data packet delivery, more expensive data packet delivery, reduced throughput, and so on.

Connection management solutions also encounter difficulties in handling applications where the available connections or the operational characteristics of these connections change over time. For example, timer-based systems can be insensitive to changes in network performance because the timer for each connection fires independently, so it does not provide an accurate view of network performance for all other connections. There is a possibility that there is no.

A method of data communication that more efficiently utilizes multiple connections to transmit data packets with lower latency, increased throughput, and enables more reliable or less expensive data packet delivery. Systems and methods for this are desirable.

The systems and methods described herein include push data packet communication systems, such as a router, a gateway, or a controller circuit coupled to a router or gateway adapted to control data packet communications. may be implemented as a physical push data packet communication device. In another embodiment, the system may be implemented as software for use with electronic hardware, such as a communications controller board with a processor, or as software resident in memory as a set of instructions executed by the electronic hardware. . The push data packet communication system operates to improve data communication across multiple connections (e.g., latency, reliability) based on monitored data communication characteristics (e.g., network traffic) associated with one of the connections. Various embodiments are described herein to improve performance (performance, throughput, error correction). A corresponding computer-readable medium (e.g., a non-transitory computer-readable medium storing a set of machine-interpretable instructions for execution on the processor) that operates in conjunction with the data storage, computer processor, and computer memory also includes: provided.

Embodiments described herein provide technical improvements over alternative approaches such as the use of static routing tables, pull scheduling (eg, use of connection-specific timers that are not necessarily synchronized). For example, the Linux kernel TCP stack uses a "pull" model to reduce recovery time for retransmitted packets. See: https://groups. google. com/g/bbr-dev/c/d14XAcuTBxo/m/Pwui1o3MBAAJ.

The push data packet communication system described herein provides connection (link ) adapted to provide routers that can be adapted for aggregation.

As described in various embodiments herein, a multipath connection aggregation system (alternatively referred to as a mixed connection system) includes a first input queue and scheduling stage, a second connection a characteristic monitoring stage (e.g., providing congestion control and packet buffering, and encapsulating pacing instructions), and for delivering packets to the network interface (e.g., according to the pacing instructions provided in the second stage); It can be modeled as a three-stage data pipeline, with a third transmission stage.

Push data packet communication systems include a scheduler configured to queue data packets for transmission in response to monitored communication events (also referred to as trigger events). Example triggering events (other than timers) include ACK/NACK of previously transmitted WAN packets, arrival of new data to the input queue, lack of feedback for a given amount of time, etc. The scheduler allocates data packets to various networks based on monitored characteristics at the time of the trigger event.

The monitored characteristics of an individual connection are a subset of the connection's operational characteristics, including estimated throughput, measured and/or expected latency, measured packet loss, and others that may be derived from the measured or expected values. may include, but are not limited to, the characteristics of In a clear example, the operational characteristics may include estimated throughput, measured and/or expected latency, measured packet loss, and other characteristics that may be derived from the measured or expected values and are monitored. The characteristics may be limited to estimated throughput.

In contrast to pull schedulers, push schedulers evaluate monitored characteristics of each of multiple networks at the time of a monitored communication event and allocate data packets to connections that have available transmission capacity. In an exemplary embodiment, the push scheduler determines, in addition to available transmission capacity, data packet (application) requirements, network ordering preferences configured by an administrator, and monitored characteristics (other than transmission capacity) of the connection. Take into consideration. The push scheduler determines whether there is available transmission capacity associated with the network (available transmission capacity is continuously updated to reflect the allocated data packets) and whether data packets are available for transmission. Continue to allocate data packets to available networks if available. In this way, push schedulers may utilize the available transmission capacity of multiple networks to a greater extent compared to pull schedulers.

One aspect of the push scheduler is a method for configuring the push scheduler to computationally determine or establish a sort order for controlling the routing of data packets to various network connections. Factors that influence sort order decisions include keeping the sort order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting is subject to self-induced out-of-order events and jitter). events) and considering both measured characteristics of the connection and unmeasured external directives.

The measured characteristics are: packet loss during three separate sliding windows (500ms, 3s, 30s) (further summarized as the worst value between the three windows), current network round trip time (RTT), It may include bandwidth and round-trip propagation time (BBRv1) parameters, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives can include priority order (PR) rules, among others, but can also include flow-specific rules/requirements.

In some embodiments, the measured characteristics of BBRv1 are different from the design of these characteristics in the original BBRv1 IETF draft. For example, given embodiments that may operate on systems where the system execution times are less consistent, the size of the sliding window used to calculate BtlBw may vary either statically or dynamically. . Also, the input parameters used to calculate RtProp may be slightly modified for similar execution consistency reasons.

A specific improvement based on improving the networking routing order to control the usage of network interfaces takes advantage of the combined characteristics of round trip time (RTT) and packet loss to give RTT, packet loss, and MSS. Some embodiments are described that predict throughput based on a Mathis Equation to establish an upper bound on TCP throughput. Mathis factors can be utilized to modify the sort order for routing networked packets.

In some embodiments, push scheduling may also be configured to implicitly trigger a sort order along one or more implicit sorting dimensions over a period of time. The implicit sorting dimension forces connections that encounter periodic bad events (eg, latency spikes due to congestion) to bubble to the bottom of the list. Connections with consistent behavior naturally bubble to the top and stay near the top.

Pull scheduling does not allow for increased utilization of network capacity because pulls occur based on connection-specific timers that are triggered independently of each other. The pull scheduler system includes a shared pull scheduler and timer associated with each network of a plurality of available networks, each timer having a wake-up value. When the timer wakes up and pulls a packet from the pull scheduler, the decision about which packets are scheduled is made by monitoring the current connection, since the potential wake-up times of other connections are unknown. It is based solely on characteristics. The resulting packet scheduling decision is locally optimal for each individual connection at the time of the pull event, but packets scheduled at an earlier pull event on one connection may could result in globally suboptimal decisions that would have been more optimally scheduled. The pull scheduler is configured to pull an amount of packets from the input queue based on the estimated bit rate of the corresponding network and the duration between each timer wakeup (e.g., 1/50 Hz, or 20 ms). .

A technical challenge associated with push schedulers is that data packets belonging to a particular data flow (e.g., packets containing audio files, video, etc.) can only be monitored by a single network, or similar monitoring system, so that they can meet the requirements of the flow. The aim is to ensure that the network is transmitted over a relatively stable set of networks that share the same characteristics. For example, transmitting audio packets belonging to a single data flow across two networks with different latencies means that the sequencer (i.e., the hardware or software responsible for reassembling the packets after transmission) buffers the audio packets. reordering and, in some cases, adding latency or failing to deliver data packets in their original order, resulting in high-latency, inconsistent and Perceiving poor networking encounters. A push scheduler that groups connections with similar monitored characteristics provides more direct control to allocate data packets to connections that best meet flow requirements.

The proposed push scheduler, in some embodiments, relies on the comparison of monitored characteristics (e.g., round-trip time, bottleneck bandwidth, etc.) with reference ranges of network operating characteristics (e.g., reference ranges for transmission). The range is configured to order the plurality of networks based on the range defined to be insensitive to small variations in the measured operational characteristics. The use of ranges allows the push scheduler to remain responsive to changes in the monitored characteristics of the network while building a degree of insensitivity to smaller variations in the monitored characteristics.

The proposed push scheduler allocates data packets to multiple networks based on data packet ordering and assigns data packets belonging to a particular flow to a single or relatively stable network based on monitored characteristics of the networks. may be configured to facilitate assignment to a given set. In an exemplary embodiment, the push scheduler remembers the ordering and continues to send packets according to the ordering until the ordering is updated as a result of new or updated monitored characteristics.

The monitored characteristics may include a capacity factor, which is an aggregate value defined at least in part by the bottleneck bandwidth divided by the round-trip propagation time. The capacity factor may indicate connections that achieve higher bandwidth per millisecond of round-trip propagation time, indicative of an efficient network. Additionally, the capacity factor scales the bottleneck bandwidth to the nearest digit (e.g., 1, 10, 100, 1000, 10 , 000, etc.) or an integer multiple thereof (eg, 2, 3...20, 30, etc.). Similarly, the capacity factor uses a bounded round-trip propagation time that is related to the measured round-trip propagation time to reduce the sensitivity of the capacity factor to variations in the measured round-trip propagation time. can be determined by

In an exemplary embodiment, the proposed push scheduler may further order multiple networks based on user input criteria. For example, the ordering may be based, at least in part, on the user assigning a priority to each of the available networks, which allows the user to assign priorities to each of the available networks, such as costs, future reliability or It becomes possible to provide preferences as to the order in which networks should be used based on external factors such as information about availability.

In response to the technical challenge of utilizing greater amounts of existing network transmission capacity, push schedulers improve transmission requirements associated with data packets queued for transmission by using a flow classification engine. (e.g., the system can be configured to identify and/or utilize predefined transmission requirements associated with the identified data flow type and identify a particular data flow type. standards that can be used). By identifying or exploiting the transmission requirements of data packets and comparing these transmission requirements to the monitored characteristics of multiple networks, the push scheduler determines whether the data packets are transmitted over networks with preferred monitored characteristics. In order to maximize the amount of data packets, it may be possible to allocate the data packets to networks that match the required transmission requirements. Although it may still be possible to transmit packets over a network whose monitored characteristics do not match the transmission requirements, interpreting the "late" or out-of-order delivery of packets as network degradation and thereby preventing the attempted This can result in applications generating packets that degrade the quality of service provided. Exemplary transmission requirements may include required latency throughout, and the like.

In an example embodiment, the push scheduler may use flow classification engine packets within flows, each flow having an assigned set of transmission requirements. In another embodiment, each flow inherently has transmission requirements that form the characteristics of the flow. This flow mapping is based on one or more elements within or associated with the data packet's header. These flow transmission requirements may then be used to assign the packet to one of multiple connections with comparable monitored characteristics. In the example where data packets are classified as VPN data flows, they have an inherent requirement that the data be delivered in order, and these requirements dictate that a flow be sent over a connection with a certain minimum reliability. May include transmission requirements.

For example, TCP/IP headers are identified as TCP/IP 5-tuples (e.g., source and destination IP addresses, source and destination port numbers, and protocol) in a data flow that is more sensitive to ordering. It may be determined that it is a virtual private network (VPN) data flow. As a result, the scheduler requires that all of the data packets of a VPN data flow be delivered in order, regardless of the presence of other indicators such as DSCP tags on individual packets that may conflict with this requirement. The transmission requirements may be received from a flow classification engine that provides a flow classification engine. For example, some of the packets may have a DSCP tag that requires low latency, while other packets may require high throughput. The scheduler may map (assign) packets to connections with monitored characteristics that meet the specific transmission requirements of the DSCP tag, as long as they do not violate the ordering requirements of the encapsulated VPN data flow.

According to another example, a push scheduler may use a flow classification engine to map packets to flows whose transmission requirements prioritize low latency over reliability or throughput (e.g., sources may may indicate the type of data such as a live video stream that may be sensitive to).

According to further exemplary embodiments, the push scheduler may assign priority transmission requirements to individual data packets independently of transmission requirements derived from flow classification. For example, the push scheduler may be configured to assign the highest level of priority to packets that are queued for retransmission (ie, the packets were previously transmitted and failed).

The technical challenges associated with push schedulers implementing data flow or data packet priority transmission requirements are that non-urgent data packets are continually moved from the transmission queue, resulting in "starvation" of non-urgent packets in the system. That's true.

The push scheduler may be configured to pace the rate of packet transmission by allocating a nominal wall clock time for transmission to each data packet in response to a starvation of non-urgent packets. In an exemplary embodiment, packets are assigned wall clock time based on the estimated bottleneck bandwidth and the size of the data packet to prevent congestion.

The network output queue to which a data packet is allocated in a pipeline stage is configured to transmit the data packet according to the data packet's assigned nominal wall clock time, giving priority to priority packets and without moving non-urgent packets. can be done. For example, at time zero, a non-emergency data packet may be assigned a nominal wall clock time for transmission at time three. The urgent data packet may be scheduled for transmission at time 3 based on the amount of available transmission capacity in the network, which available transmission capacity is reduced to incorporate the non-urgent data packet.

Continuing with the example, the push scheduler may be configured to queue non-urgent data packets for later transmission after a nominal wall clock time. For example, at time 0, if there is network capacity available at time 2, the push scheduler may send a non-urgent data packet with a timestamp of time 3 for transmission at time 2. Subsequently, the non-urgent data packet may be rescheduled for transmission at time 3 in response to receiving a packet with a more urgent nominal wall clock time.

A technical challenge associated with push schedulers is that pushing data packets based on estimated available network transmission capacity results in the system becoming less robust in the event of network failures or an unacceptable amount of network congestion. It's about getting.

The push scheduler may be configured to monitor network operating characteristics associated with each of the available networks to determine whether the monitored characteristics meet a congestion threshold. In an exemplary embodiment, the congestion threshold is established from a baseline profile associated with each network. For example, the lowest latency measured for packets transmitted over a particular network can form a baseline value for the network, and subsequent transmissions along that network can be compared to the baseline value. can. In another example embodiment, the threshold may be based, at least in part, on a baseline of packet round-trip times. In other embodiments, the baseline may be calculated using more complex statistical techniques than taking minimum latency.

To increase the robustness of the push scheduler, the scheduler may reduce the amount of transmission capacity defined as available for transmission in response to detecting a network failure or an unacceptable amount of network congestion. , or may be configured to update available transmission capacity other than in accordance with each monitored communication event. In an exemplary embodiment, the transmission capacity of the network is updated only in response to receiving an acknowledgment that a previously transmitted data packet has been received (e.g., in response to receiving an ACK). can be done. According to some embodiments, for example, the defined transmission capacity is reduced to a theoretical transmission capacity that does not include a static queue of packets in buffers associated with the network. In a further exemplary embodiment, the defined transmission capacity is reduced to be equal to the amount of data packets in-flight across a particular network.

In some exemplary embodiments, one of the networks of the plurality of networks operates according to Bandwidth on Demand (BOD) principles, and the network administrator and/or automated network monitoring agent: Allocate available bandwidth for transmissions to transmitters based on previous transmissions or previous requests for bandwidth. This allocation may increase or decrease the allocated bandwidth depending on the administrator's or agent's prediction of future bandwidth usage. During this bandwidth allocation process, packet loss and/or latency may temporarily increase, requiring the scheduler to make more intelligent decisions to allocate packets to connections. For example, probing packets used to request more bandwidth may be marked with a DSCP tag indicating that the probing packet is of lower priority and is therefore a candidate to be dropped first if necessary. It is.

The technical challenges associated with a push scheduler pushing data packets onto a Bandwidth-on-Demand network is that the Bandwidth-on-Demand network updates the available bandwidth for transmission infrequently and takes advantage of the available capacity. This can be difficult. For example, a bandwidth on demand network may have available capacity, but as a result of allocation delays, this capacity may go unused. Alternatively, if the push scheduler does not consistently use sufficient capacity on the network, the administrator/agent may elect to reduce the allocated capacity, potentially reducing the network's monitored capacity. can cause significant changes in the characteristics of

The push scheduler defines the transmission capacity of the bandwidth-on-demand network as greater than the received allocated transmission capacity, and stores the excess transmission capacity in a buffer associated with the bandwidth-on-demand network; Bandwidth on Demand The network administrator may be configured to over-advertise the required amount of bandwidth. As a result of over-advertising the required bandwidth, the bandwidth-on-demand network may begin to allocate greater amounts of unused bandwidth to the push scheduler. Furthermore, as a result of defining the transmission capacity as being larger than the received allocated transmission capacity, the push scheduler does not limit the transmission capacity to take advantage of any increase in the received allocated transmission capacity that may occur subsequently. can ensure that data packets are available.

A further technical challenge associated with pull scheduler systems is that data packets with specified requirements are difficult to be fairly allocated for transmission. For example, if a certain type of data packet requires a connection with a certain latency (e.g., a video feed), the pull scheduler system may bypass that packet until an available network that meets the latency requirement wakes up. or may be delayed. In this manner, depending on when the data packet enters the input queue, latency sensitive data packets may be delayed, further exacerbating the latency problem.

Rate-based congestion control typically adjusts rates based on delay and loss. Congestion detection needs to be performed with a certain time lag to avoid overreacting to spurious congestion events such as delay spikes. Despite the fact that there are more than one congestion indicators, the result is that there is still only one mechanism for adjusting the transmission rate. This makes it difficult to simultaneously achieve the goals of high throughput and rapid response to congestion.

In the figures, embodiments are shown by way of example. It is to be expressly understood that the description and drawings are for purposes of illustration only and as an aid to understanding only.

Embodiments will now be described, by way of example only, with reference to the accompanying drawings.
1 is a schematic diagram of a pull communication system; FIG. 1 is a schematic diagram of a push communication system, according to some example embodiments; FIG. 2 is a flowchart of a method for transmitting data packets to push packets to multiple networks based on availability, according to some example embodiments. 2 is a flowchart of another method for transmitting data packets for ordering connections based on connection metadata, according to some example embodiments. 3 is a flowchart of a further method for transmitting data packets based on a selected order of networks for assigning packets to networks, according to some example embodiments. 2 is a process flow illustrating dimensions used to sort network connections to control the routing of data packets, according to some example embodiments. 2 is a schematic diagram of scheduling data packets of different priority levels according to some example embodiments; FIG. 1B is a series of schematic diagrams of the push communication system of FIG. 1B scheduling data packets of different priority levels, according to some example embodiments; FIG. 1B is a schematic diagram of connection selection with the push communication system of FIG. 1B, according to some example embodiments; FIG. 1B is a schematic diagram of data packet selection using the push communication system of FIG. 1B, according to some example embodiments; FIG. 1B is a schematic diagram of data packet allocation using the push communication system of FIG. 1B, according to some example embodiments; FIG. 1B is a schematic diagram of another data packet allocation using the push communication system of FIG. 1B, according to some example embodiments; FIG. 1B is a schematic diagram of the push communication system of FIG. 1B scheduling two independent data flows for transmission, according to some example embodiments; FIG. 1B is a schematic diagram of the push communication system of FIG. 1B scheduling two dependent data flows for transmission, according to some example embodiments; FIG. 5 shows a corresponding screenshot of an interface for managing data packet transmission, according to an example embodiment; FIG. 5 shows a corresponding screenshot of an interface for managing data packet transmission, according to an example embodiment; FIG. 5 shows a corresponding screenshot of an interface for managing data packet transmission, according to an example embodiment; FIG. 5 shows a corresponding screenshot of an interface for managing data packet transmission, according to an example embodiment; FIG. 5 shows a corresponding screenshot of an interface for managing data packet transmission, according to an example embodiment; FIG. FIG. 2 is a diagram of data packet transmission, according to some example embodiments. FIG. 3 is two diagrams of unidirectional data packet transmission, according to some example embodiments. FIG. 3 is two diagrams of data packet transmissions that occur in response to acknowledgments, according to some example embodiments. FIG. 7 is two further diagrams of data packet transmissions that occur without requiring acknowledgment to send further data packets, according to some example embodiments; FIG. 3 is an illustration of in-flight goodput data, according to some example embodiments. FIG. 2 is another illustration of in-flight goodput data, according to some example embodiments. FIG. 6 is a further illustration of in-flight goodput data, according to some example embodiments. 2 is a series of diagrams illustrating data packet transmission where a connection is experiencing congestion, according to some example embodiments; FIG. 2 is a series of diagrams illustrating data packet transmission after transmission capacity adjustment in response to a connection encountering congestion, according to some example embodiments; FIG. 1 is an example schematic diagram of a computing device, according to some embodiments. FIG.

Systems, methods, and devices are described for transmitting data packets in which multiple connections for transmission are available, alternatively referred to as mixed routers or mixed systems.

Existing solutions for transmitting data packets across multiple available connections for transmission are typically implemented as "pull" architectures, where data packets are routed across multiple available connections. Transmitted in response to an event generated or defined by a device or system, such as an associated internal timer.

The internal timer may be responsive to various connection parameters (e.g., a timer duration configured to reduce recovery time for retransmitted packets), but not to connection parameters of other available connections.

As a result of the independence of timer events, pull-based systems can suffer from underutilization (e.g., if the timer value is too long, resulting in idle connections) or overutilization (e.g., poor performance but low reliability). data packets may be allocated to connections), which may result in greater latency in data packet delivery, reduced throughput, less reliable data packet delivery, more expensive data packet delivery, etc. can be brought about.

Methods, systems, and devices for transmitting data packets according to a push architecture for transmitting data packets over multiple available connections are proposed herein. The disclosed methods, systems, and devices obtain a subset of operational characteristics (monitored characteristics) of each of a plurality of connections in response to a monitored communication event; and a scheduler configured to iteratively allocate available data packets for transmission to any of the determined connections based on the determined transmission capacity. In exemplary embodiments, other monitored characteristics of the network may be used in conjunction with the transmission requirements associated with the data packet to assign the data packet to a connection that matches the data packet's transmission requirements.

Scheduling data packets in push architectures where multiple connections are available has not been used. Typical multipath systems (alternatively referred to as mixed connection systems) allocate packets and flows to connections on a static basis because they do not consider the flow requirements or operational characteristics of the connections. For example, a multihomed Internet Protocol (IP) router assigns packets to connections purely based on the destination IP address in the packet header, which matches against the router's destination routing table. If connection allocations do not change frequently, it is easier to implement a pull architecture. For example, there is no need to handle the case where a connection is assigned a large queue of packets by a push scheduler, but then has to be recalled or reallocated because the connection goes offline.

The proposed method, system, and device receive a data flow comprising one or more data packets from an input network through an input connection.

The proposed method, system, and device include a three-stage data pipeline that transmits data packets over a mixed set of connections to remote peers. New data destined for multiple connections is passed to a first pipeline stage where the new data is queued and then passed to a second pipeline stage associated with each connection in the mixed connection. It will be done. The second pipeline stage compiles statistics about the connections underlying the second pipeline stage and passes the packet to the third pipeline stage. The second stage may provide congestion control functionality and packet buffering to ensure that packets are available to the next pipeline stage. The third stage then writes the packet to the network interface, possibly according to pacing or timing hints provided by the second stage.

Another embodiment may have a single second stage instance that then provides packets to the third stage instances, each of the third stages having a network interface that shares a physical interface card. Manage packet transmission in sets. Other embodiments may exist with different numbers of instances of each of the pipeline stages.

FIG. 1A is a schematic diagram of a pull communication system 100A, according to some example embodiments.

When each connection timer (e.g., timers 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) wakes up, the connection timer invokes pull scheduler 130 to transfer data packets from input queue 102. request data packets for. Data packets may be requested in bursts, with the burst size nominally reflecting 20 ms (1/50 Hz) worth of bytes at the estimated bit rate of each timer's corresponding connection. A limitation associated with such pull timers is that the number of bytes requested upon wake-up of each timer does not reflect the complete congestion window (e.g., representing "capacity") of the connection.

Because each timer fires independently, pull scheduler 130 does not have visibility into the overall transmission status in order to make globally optimal scheduling decisions. For example, connection 110A may be a terrestrial broadband connection and connection 110B may be a bandwidth-on-demand (BoD) satellite connection. Connection 110A may be preferable to connection 110B as a result of expense, latency, etc., but when connection 110B timer 108B wakes up and calls pull scheduler 130, pull scheduler 130 uses the next It has no visibility into wake-up time (since timer 108A fires independently). Pull scheduler 130 determines whether connection 110A can process all of the data packets in input queue 102 or whether additional data packets should be assigned to connection 110B to complete the transmission. does not have the basis of

If connection 110B is useful, pull scheduler 130 determines which connection should be used to transmit which of the data packets available for transmission, since only pull scheduler 130 includes visibility into the timer. It is not necessary to have a means for determining.

Data packets are transmitted across the connection and then received by receiver 116 via connection 114. Receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via local area network (LAN) 120.

In contrast to pull-based architectures, the proposed system may be able to schedule data packets on specific connections to increase efficient connection utilization.

FIG. 1B is a schematic diagram of a push communication system 100B, according to some example embodiments. In the exemplary embodiment, system 100B is designed to aggregate connection types that provide multiple access (MA) on a shared communication channel using multiplexing techniques. For example, orthogonal codes assigned to each node in a code division multiple access (CDMA) network enable the nodes to share a set of frequencies and transmit simultaneously without interfering with each other.

A characteristic of connection types that use these multiplexing techniques is that the reallocation of available capacity to nodes in the network is rapid, typically on the order of tens to hundreds of milliseconds. . In the exemplary embodiment, system 100B is designed with this characteristic in mind and reallocates traffic among available WAN connections on this time scale.

In contrast, connections using demand allocation multiple access (DAMA) techniques, also known as Bandwidth on Demand (BoD), share a communication channel by having nodes negotiate access to the channel for a period of time. Once allocated, the node uses the negotiated portion exclusively. Therefore, sharing of the available capacity is done sequentially through the nodes using negotiated portions of the channel.

With BoD connections, this negotiation and subsequent reallocation of capacity between nodes typically takes on the order of seconds (mid-single digits to low double digits) to complete. In an exemplary embodiment, system 100B will require modifications (described herein) to effectively use these types of connections.

Long reallocation times (with associated increased packet loss and/or latency) are interpreted as congestion in current implementations, and typical congestion control methods reduce usage on connections where congestion is detected. , to avoid overloading these connections. However, BoD connections actually require the opposite behavior (at least temporarily) in order to request and possibly obtain increased allocation of capacity.

System 100B includes an input queue 102 that receives a data flow containing one or more data packets from an input network for transmission. Examples of data flows may be file transfers, video file transfers, audio file transfers, voice over internet protocol (VoIP) calls, or data packets associated with a virtual private network (VPN).

Input queue 102 also has a drop policy on packets if a scheduler (as described herein) is unable to service input queue 102 at the same rate as new packets arrive from LAN-side clients. Responsible for applying. This typically occurs when LAN clients are transmitting at a higher rate than the aggregate WAN transmission capacity.

System 100B includes three stages for transmitting data packets to a remote peer via a set of intermixed connections according to a push architecture, the stages alternatively referred to as pipeline or pipeline stages. Ru.

The first stage 104 includes a flow classification engine 104A, a scheduler 104B (hereinafter alternatively referred to as push scheduler 104), and receives data packets from the input queue 102 and forwards them to the second stage 106. Queue data packets for passing. In the exemplary embodiment, first stage 104 includes all of input queue 102, flow classification engine 104A, and scheduler 104B.

Input queue 102 buffers packets received from clients on the sender's LAN side and, in some cases, utilizes or cooperates with flow classification engine 104A to route data packets to flows. (alternatively referred to as mapping data packets to flows). The input queue 102 ensures that data flows are serviced fairly by the first stage 104 scheduler.

In an exemplary embodiment, the first stage 104, or the auxiliary flow classification engine 104A of the first stage 104, monitors changes in user-configured settings associated with the assignment of data packets to connections; Be notified. User-configured settings provide information about how connections should be assigned to priority levels, when those priority levels should be activated, as well as specific monitored packets belonging to a flow. Contains user-configurable flow classification rules that may favor connections with characteristics (e.g., packets belonging to the video stream class may prefer connections with low latency, whereas packets belonging to the file download class may prefer connections with lower latency). may prefer connections that have lower monetary costs).

A first stage 104 periodically updates current metadata (e.g., operational characteristics) for each of a plurality of connection monitoring instances (as described herein) corresponding to a WAN connection that includes mixed links. will be notified. This metadata is collected by monitoring instance 106 and includes the connection's estimated bottleneck bandwidth, estimated minimum round-trip time, recent actual round-trip time, current congestion window for the connection, and previously may include reception/loss status for packets sent to the stage. Using this metadata, the first stage 104 also extends the metadata to track an estimate of the bytes that the first stage 104 has scheduled in-flight on this connection, and adds that total to new bytes. , retransmission bytes, or dummy/probing bytes. All of the metadata collected and tracked by the first stage 104 is referred to as a monitored characteristic.

Using all available monitored characteristics, the first stage 104 then determines whether the packet should be sent to the next stage and, if so, which connection should receive the packet. The decision will be made as to whether or not. The set of packets provided to a connection monitoring instance may include duplicates of previously sent packets (either to the same connection monitoring instance or to a different connection monitoring instance). Alternatively, the scheduler may determine that a given connection monitoring instance is not currently eligible to receive packets even if that instance's connections currently have capacity. Assignment of packets to specific Stage 3 instances is discussed elsewhere in this document.

The second stage 106 includes a connection monitoring instance for each of the plurality of available connections (eg, connection monitoring instances 106A, 106B, and 106N associated with connections 110A, 110B, and 110N, respectively). The connection monitoring instance of the second stage 106 monitors the associated connection and determines or obtains statistics associated with the connection's performance (eg, latency), referred to as behavioral characteristics. The second stage 106 may provide congestion control functionality and packet buffering to ensure that packets are available to the next stage. Second stage 106 passes the packet to third stage 108, which includes an output queue for each connection (eg, output queues 108A, 108B, and 108N). Third stage 108 then writes the packet to the network interface, possibly according to pacing or timing hints provided by second stage 106.

In some embodiments, the second stage 106 uses one or more R2RI protocols (e.g., PPPoE (RFC5578), R2CP, DLEP (RFC8175) to better support the bandwidth allocation process for BoD connections. )) can be implemented.

The second stage 106 is responsible for receiving packets from the first stage 104, queuing these packets locally, and providing groups of packets as bursts to the next pipeline stage. The second stage 106 also tracks metadata about the second stage's 106 associated connections based on acknowledgments received from peers or recipients.

The purpose of the queue in the second stage 106 is to ensure that there is always a packet available to the next pipeline stage to write to the network device for transmission. Some embodiments may include priority metadata as part of each packet information so that new packets arriving from the first stage 104 provide preference to higher priority packets. may be reordered within the queue.

In order to ensure that the second stage 106 buffers enough packets so that the packets can be provided to the third stage 108, the second stage 106 transfers the modified metadata to the first may be provided to stage 104. For example, according to some example embodiments, the second stage 106 writes the second stage for when the first stage 104 completes writing the actual congestion window packet. may choose to report (i.e., "over-advertise") a larger congestion window to provide additional packets that may be buffered at stage 106 of.

The second stage 106 may also provide modified metadata to the first stage 104 to facilitate changing the allocation of capacity in the BoD connection (i.e., to allocate higher capacity). To trigger a BoD connection, have the first stage 104 schedule packets on this connection as if the first stage 104 had a higher capacity). This modified metadata may include information that breaks down the higher capacity into different requested types/priorities. For example, higher incremental capacity used to probe a BoD connection is typically comprised of redundant/dummy packets since probe packets are more likely to be dropped by the BoD connection.

Advertising or over-advertising capacity from the second stage 106 to the first stage 104 need not necessarily result in immediate probing for higher incremental capacity. The first stage 104 ultimately still makes decisions about how packets are assigned to connections. One embodiment may wait until the CWND on all non-BoD connections is fully consumed (or close to fully consumed) before instructing the second stage 106 to begin the DAMA reallocation process. .

The queue in the second stage 106 also allows for fast notification of failures to previous stages if something happens to the underlying network connection. Packets provided to the third stage 108 and written to the network must be assumed to be in transit, even if the network interface subsequently reports down, and those packets are marked lost. This means that a timeout must occur before you can get it. However, packets that are still queued in the second stage 106 may be marked as lost as soon as the second stage 106 is notified that the connection is no longer available, thereby These packets are immediately eligible for prioritized retransmission.

Before being passed to the third stage 108, some embodiments provide a time stamp to each packet that indicates when the next stage should attempt to send that packet over the connection. One may try to explicitly pace the rate of packet transmission. Such timestamps are used to determine the estimated bottleneck bandwidth of a connection (e.g., the connection to which the packet is being allocated) to prevent network congestion at or before the network bottleneck link. , and the packet size.

FIG. 2 illustrates, at 200, an example embodiment in which the second stage 106 supports prioritized packets. In FIG. 2, the head of the queue is displayed on the right side, and the tail of the queue is displayed on the left side. At the top of FIG. 2 (eg, first step), the first stage 104 provides a new list of prioritized packets to the connection monitoring instance of the second stage 106. The bottom part of Figure 2 (e.g., the second step) shows that these new packets have been added to the priority queue in priority order, so that all priority 1 packets are replaced by any priority 2 packets. packets are dequeued before they are dequeued. Packets of the same priority are kept in sequential order.

In an exemplary embodiment not shown, the second stage 106 may have a single connection monitoring instance that then provides packets to the third stage instances, each of which has a physical interface. Manages the transmission of packets across a set of network interfaces that share a card. Other embodiments may exist with different numbers of instances of each of the pipeline stages.

Feedback in the form of metadata or callbacks may be sent from a stage to a previous stage, or a stage may receive notifications from different parts of the system. Any of these actions may trigger (ie, push) a stage to send a packet to the next stage.

Third stage 108 transmits packets from second stage 106 via one or more connection transmission instances (e.g., connection transmission instances 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively). and writes these packets to network 112. In some embodiments, all second stage 106 connection monitoring instances may provide packets to a single connection transmission instance, or each connection monitoring instance may be associated with a connection transmission instance.

Some embodiments of the third stage 108 may use a queuing mechanism to allow previous stages to provide hints as to when to send a packet. For example, a hint may occur in the form of a "nominal transmission time" associated with the packet as metadata. If the next packet to be transmitted has a later nominal transmission time, the third stage 108 will wait until that time before transmitting the packet. Urgent packets may be flagged with an immediate timestamp that may cause the urgent packet to be sent before other queued packets. To prevent starvation of non-urgent packets, some embodiments may queue urgent packets after non-urgent packets that have a past nominal transmission time.

Some embodiments may include a callback function as part of the packet metadata that may be used to trigger a notification to the second stage 106 to indicate that the packet has been sent. Such notifications may be used to trigger a new batch of packets to be pushed from the second stage 106 to the third stage 108.

FIG. 3 illustrates, at 300, a time course of an embodiment that includes an "on transmit" hint (nominal transmit time) requesting that packets provided by the second stage 106 not be transmitted before a specified time. Show progress. Packets with a "when sent" time of 0 should be sent as soon as possible. FIG. 3 illustrates examples for all of these cases (eg, the use of past nominal transmission times for urgent packets and how the system avoids starvation of non-urgent packets).

In the state depicted at t=1, the packet queue of the third stage 108 instance contains four packets, three of which are currently eligible for transmission, and the second stage 106 An instance of is about to provide four new packets, the first of which has sequence 5 (sequence order 5) and should be sent immediately.

In the state depicted at t=2, a new packet has been added to the third stage 108 priority queue. This embodiment chose to queue the packet with sequence 5 for transmission after the packet with sequence 2 because sequence 2 was already eligible for transmission at the time sequence 5 was queued. Please note that In this way, freezing the order of packets at the head of the packet queue means that a constant stream of immediately sent new urgent packets is eligible to be sent long before the most recent immediately sent urgent packet arrives. Prevent starvation if it is possible to prevent non-immediate packets from being sent even though they might have been.

In the situation depicted at t=3, the third stage 108 dequeued the first four packets from the output queue and provided these packets to the network interface. Since there are currently no packets eligible to be sent until time t=4, the third stage 108 instance will not write until time t=4, even if there is current write capacity on the network interface. , or it would be necessary to wait to send more packets until a new packet arrives that should be sent immediately or at time t=3.

Packets may be passed from one stage to the next in turn at the time each stage determines to be optimal.

Similar to FIG. 1A, data packets are transmitted across the connection and then received by receiver 116 via connection 114. Receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via local area network (LAN) 120.

In contrast to system 100A, push communication system 100B does not include an independent timer associated with each connection; instead, the scheduling of packets onto connections is done at a central location, and each connection is monitored. The push scheduler 104 includes a push scheduler 104 that accesses the properties of the push scheduler 104 that is activated in response to one or more monitored events that occur other than based on a timer. For example, one or more monitored communication events may include an acknowledgment/negative acknowledgment (ACK/NACK) from a recipient of a previously transmitted data packet, or the arrival of a new data packet to input queue 102. may be included.

In an exemplary embodiment, the entire connection (congestion window) CWND may potentially be consumed or allocated by the push scheduler 104, rather than just the 20ms of the packet being transmitted.

With access to the monitored characteristics of each connection, push scheduler 104 may be configured to optimize packet scheduling decisions. For example, data flows with specific transmission requirements may be preferentially assigned to connections that can meet the requirements associated with the data flow, rather than being assigned based largely on timer wake order. In one illustrative example, the first data flow requires transmission with a latency of less than 50 ms.

Connections 110A, 110B, and 110N have latencies of 5ms, 45ms, and 80ms, respectively. Using pull scheduler 130, if the 50Hz timer for connection 110B (45ms latency) wakes up first, then that timer schedules packets for this first data flow onto connection 110B. would be possible. Using the push scheduler 104, data flows can be preferentially scheduled on connection 110A (5ms latency) until the CWND of connection 110A is completely consumed before overflowing onto connection 110B (45ms latency). .

To enable bidirectional communication, a single instance of a multipath endpoint can include all components described in FIG. 1B. Specifically, one or more instances of a sender 100 to support upstream communications to other multipath endpoints and a receiver to support downstream communications from other multipath endpoints. 116.

According to some embodiments, for example, one or more of the available connections is a BoD connection, and the push scheduler 104 requests more capacity from the network based on criteria such as: To do this, more data can be allocated to BoD connections.

While the CWND of connection 110A is fully consumed (or approaching fully consumed), more packets are eligible to be transmitted on connection 110B (e.g., based on per-flow rules). is available in the input queue 102.

In an exemplary embodiment, push communication system 100B may enable more accurate/fair quality of service (QoS) and active queue management (AQM) packet dropping rules applied to input queues 102. Referring again to the previous example, a flow with a 50ms latency requirement will never be able to utilize an 80ms connection. Since the push scheduler 104 has a consistent snapshot of the entire system, it can calculate the partial usage of flows excluding 80ms connections and their fair share of the total capacity.

A potential advantage of the push scheduling approach is that a complete set of metadata about multiple connections, including intermixed links, is available to the push scheduler 104 to optimally allocate packets to target connections. be.

FIG. 1C is a flowchart of a method 110C for transmitting data packets using system 100B or other push communication systems, according to some example embodiments.

At block 1130, method 100C detects when the input queue contains unallocated data packets available for transmission.

At block 1132, method 100C includes assigning one or more of the unallocated data packets to one or more corresponding output queues of a plurality of networks having suitable monitored characteristics.

At block 1134, method 100C includes updating the monitored characteristics of the corresponding plurality of networks to reflect the assigned one or more data packets.

At block 1136, method 100C communicates the assigned data packet from the output queue corresponding to each network of the plurality of networks to the destination device.

In an exemplary embodiment, the method for transmitting data packets in system 100B includes a loop of the following steps each time push scheduler 104 is triggered by an external event.
1. requesting a data packet from input queue 102;
2. Assigns received packets to connections for transmission,
3. a) until there are no packets available in the input queue 102, or b) until there is no available transmission capacity (e.g., as defined by a congestion window (CWND) metric) for any of the active connections. Repeat steps 1 and 2.

The described method may be implemented such that the push scheduler 104 does not consider any data flow-specific requirements. In such embodiments, system 100B assigns received packets based on connection ordering.

The ordering may be based on monitored characteristics of the connections, and may include, for example, a subset of operational characteristics estimated by a congestion control algorithm. In one embodiment, the BBRv1 algorithm is used, but other embodiments may include examples such as BBRv2, TCP Reno, TCP LEDBAT, proprietary implementations, etc. to handle specific connection types such as BoD. . In the subsequent discussion of operational characteristics specific to BBRv1, for illustrative purposes only, the two endpoints are modeled as a series of interconnected pipes, where each pipe segment can have a different diameter and length. , referring to the pipe analogy.

The operational characteristic is the minimum time required for a transmitted packet to reach the destination endpoint (e.g., receiver) and be acknowledged (e.g., via an ACK message from the receiver to the transmitter). It can be a measure of This duration is named "Round Trip Propagation Delay" (abbreviated as "RtProp") and is measured in units of time (eg, milliseconds). In terms of pipe analogy, this value represents the total length of all pipe segments from the transmitter to the receiver.

The operating characteristic may be the maximum rate at which data can move between endpoints along a pipe segment with a minimum diameter. The smallest pipe diameter, referred to as the "bottleneck" segment, determines the maximum rate at which data can move between transmitter and receiver, and this is referred to as the "bottleneck bandwidth" ("BtlBw"). (abbreviated)). For example, the method used to estimate BtlBw is described in the IETF draft: https://tools. ietf. org/html/draft-cheng-iccrg-delivery-rate-estimation-00.

Table 1 shows an exemplary set of WAN connections and characteristics of the WAN connections, an exemplary subset of operational characteristics selected by scheduler 104 to be the monitored characteristics, and the resulting sort order.

Table 1: Exemplary WAN Connections and WAN Connection Sort Order

This example method sorts connections across the following dimensions and moves to the next dimension only if the current dimension compares as equal across connections.
1. Increase preference priority level 2. Increase RtProp 3. Decrease BtlBw 4. Increment interface name.

Note how the result of the sort operation keeps the order of the connections relatively stable, since the sort dimension is a parameter that is not expected to change frequently.

This stable sort order is intentional, so packets that arrive at the input queue tend to be "sticky" for the same set of connections (those sorted at the top of the list) and are A request "overflows" up to a lower sorted connection only if it exceeds the total CWND of the higher sorted connection.

Note how the sorting criteria in this example is not flow specific. For example, an administrator or end user may only want high-priority application flows to be transmitted over metered Cell 1 and Cell 0 connections (an example of a transmission requirement); , meaning that we would want the resulting sort order to place cell 1 and cell 0 lower in the list. Similarly, there may be some flows that have a maximum latency limit, for example in VoIP calls where the end user considers the maximum latency limit unusable for interactive conversations if the latency exceeds 150ms. be. For these flows, it may be desirable to have a resulting sort order that uses RtProp as the primary sorting dimension. Embodiments that extend the sorting criteria to be flow specific are discussed in subsequent paragraphs.

The scheduler is also responsible for determining the packet retransmission policy. In an exemplary embodiment, system 100B is configured such that the scheduler always marks a packet for retransmission if the packet is reported as missing by the receiver. While this is desirable for many flows, other flows may not want this at all (due to additional latency or packets potentially arriving out of order making the packets unusable by the application). Or, you may only want retransmission attempts within a certain period of time.

FIG. 1D is a flowchart of another method for transmitting data packets, according to some example embodiments.

One embodiment of a simple scheduling loop is as follows.
1. Perform any accounting required to update connection metadata and determine the set of active connections.
2. Determining a preferred ordering of connections based on connection metadata (monitored characteristics).
3. While there are packets to send and there is network capacity available in the set of active connections:
a. Get the next packet to be sent.
b. Queue packets to be transmitted on the first connection in the ordering that has available capacity.
c. Update connection metadata (monitored characteristics) to reflect scheduled packets.
d. repeat.
4. For each connection that has queued packets, send those packets to the next pipeline stage for that connection.

Such embodiments treat all packets as equivalent for purposes of connection preferred ordering. In this case, the packets can be considered to belong to the same class of flows (even though separate packets may belong to logically separate flows, from the scheduler's perspective there are no connections between these flows for connection allocation purposes). There is no distinction).

FIG. 1E is a flowchart of a further method of transmitting data packets, according to some example embodiments.

between flows of different classes (i.e., ICMP echo requests/responses vs. video streams vs. file transfers), and even between individual flows within the same class (video streams between hosts A and B vs. hosts A and C). An embodiment of a scheduler that can distinguish between video streams) may be provided as follows.
1. At block 1102, any necessary accounting of connection metadata updates (monitored characteristics) is performed to determine the set of active connections.
2. While the active connection set has capacity (block 1114);
a. At block 1103, a selector is constructed based on the currently active connection set.
b. At block 1105, a selector is used to select a packet to transmit over one of the available connections. If no packet is selected, exit the loop.
c. At block 1107, an ordering of active connections is determined based on the flow of packets and flow classification in combination with connection metadata (monitored characteristics).
d. At block 1110, packets are queued to be transmitted on the first connection in the ordering that has available capacity.
e. At block 1112, connection metadata is updated to reflect the scheduled packet.
f. repeat.
3. At block 1116, for each connection that has queued packets, those packets are sent to the next pipeline stage for that connection.

These two sample methods (the methods shown in Figures 1D and 1E) are similar, with the method shown in Figure 1D being a simplified and optimized version of the more general method in Figure 1E. be.

The first method can be converted to the second method by defining a single flow class and defining a selector that matches all packets. With this in mind, the second method (FIG. 1E) will be discussed in more detail, and the same discussion will be applicable to the first method (FIG. 1D), subject to the above limitations.

The first step is to update any pending metadata accounting (monitored characteristics) to determine the set of active connections. In one embodiment, the set of active connections includes connections that are not subject to unreliable connection backoff (i.e., connections whose use has been restricted due to recent failures to ensure packet delivery). It is.

Other embodiments may further limit the set of active connections, e.g., by eliminating connections whose metadata (monitored characteristics) do not match any of the currently defined flow class requirements (i.e. , assuming that all packets must match one defined flow class, a connection can be considered inactive if it is never selected by any flow class, e.g. , if all flow classes specify a maximum RTT of 500ms, connections with RTT > 500ms may be considered inactive).

A set of active connections has the capacity to send packets if at least one connection in the set has an available CWND (congestion window) and in-flight bytes currently tracked by the scheduler for this connection. means that the number does not exceed the CWND determined for this connection. In one embodiment, the CWND for the connection may be the CWND reported by the next pipeline stage, and if throttling or priority routing is enabled, the scheduler reduces the reported CWND. You may choose to do so.

In one embodiment, a connection that is throttled to a particular rate has its CWND reduced by the same factor as the throttle rate over the bottleneck bandwidth (or, if the throttle rate is higher than the bandwidth estimate, the CWND is , unchanged). Similarly, the CWND of a connection may be reduced if this CWND is subject to priority routing (where this embodiment reduces the total goodput (throughput excluding overhead) of all connections of higher priority level ) supports connection grouping into priority levels that are activated when the level is below a configured threshold for the activated level). Various methods for determining CWND reduction in these scenarios are discussed below in connection with FIGS. 8 and 9.

Connections assigned to priority level that are inactive (total goodput + unused bitrate for all connections at higher priority level meets or exceeds activation bitrate for priority level) has an effective CWND of zero and is a non-primary priority level connection whose estimated aggregate goodput bitrate of all connections in the currently active priority level set is the lowest active If the priority activation threshold is met or exceeded, set the CWND of these connections to zero.

In other embodiments using priority routing, non-primary level connections are configured such that these connections only contribute enough to close the gap to the configured activation threshold. may be capped at a rate lower than the bottleneck bandwidth of This can be done to reduce costs, for example, if the non-primary connection is a more expensive connection intended to be used only in emergencies. Similar to throttling, CWND is reduced to reflect the capped rate at which these connections should be sent.

The loop in step 2 of the method can run as long as the connections in the active set have capacity.

Step 2a (block 1103) constructs a selector that is used to find packets that are eligible to be transmitted on one of the active connections with active capacity. In some embodiments, the selector consists of a set of flow classes that are equivalent to a set of currently active connections with capacity.

For example, if a flow class requires connections with a maximum latency of 500ms, and all active connections with available capacity have latencies greater than 500ms, that flow class may be excluded from the selector. Dew. Some embodiments determine that an active You may choose to include flow classes in the selector that would not normally match the connection's criteria. In that case, embodiments may choose to violate the match criteria in favor of transmitting the packet rather than holding the packet until it times out and is dropped. Other embodiments may choose to enforce a strict match policy in which packets matching flow classes that do not have eligible connections are not sent.

Another embodiment, illustrated in the exemplary sort order in the process flow of FIG. 1F, utilizes alternative techniques for computationally determining or establishing a sort order for controlling the routing of data packets to various network connections. can do.

Some or all of the input parameters may be rounded/bucketized before being used in sort order determination. The technical improvement from rounding and bucketing is to keep the sort order as "stable" as possible so that changes in sort order do not occur all the time (e.g., without rounding and bucketing, two Close values can cause "jitter" by continuously having sort changes as their values vary over time.

Factors that influence sort order decisions include keeping the sort order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting is subject to self-induced out-of-order events and jitter). events) and considering both measured characteristics of the connection and unmeasured external directives.

The following values may be mentioned for rounding:.
- Latency related values (RtProp, RTT) are floored 1ms after rounding and rounded to the nearest multiple of 50ms.
●Throughput-related values (BtlBw) are rounded to the "nearest digit".
● Packet loss (worst value between three sliding windows) is rounded to the nearest digit. The raw values here are restricted to the range [0,100], so the output values are set (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20 , 30, 40, 50, 60, 70, 80, 90, 100).

An example of rounding to the nearest digit is provided in the table below. The raw value, the calculated number of digits of the raw value, and then the resulting rounded value are shown.

A specific improvement based on improving the networking routing order to control the usage of network interfaces takes advantage of the combined characteristics of round trip time (RTT) and packet loss to give RTT, packet loss, and MSS. Some embodiments are described that predict throughput based on the Mathis equation to establish an upper bound on TCP throughput. Mathis coefficients can be utilized to modify the sort order for routing networked packets.

The measured characteristics are: packet loss during three separate sliding windows (500ms, 3s, 30s) (further summarized as the worst value between the three windows), current network round trip time (RTT), It may include Bandwidth Round Trip Propagation Time (BBRv1) parameters, RtProp (Network Propagation Delay), and BtlBw (Throughput Estimate). External directives can include priority order (PR) rules, among others, but can also include flow-specific rules/requirements.

In some embodiments, the measured characteristics of BBRv1 are different from the design of these characteristics in the original BBRv1 IETF draft. For example, to account for embodiments that may be executed on systems where the execution times of embodiments are inconsistent, the size of the sliding window used to calculate BtlBw is increased to cover longer durations. (hiding variability due to execution inconsistencies), but if network loss or latency increase events occur, those events may cause BtlBw to actually change and hide any long history in the sliding window. The window duration can also be shortened immediately, indicating that it must be forgotten.

For similar execution inconsistency reasons, the input parameters used to measure RtProp may be changed to include more than the network propagation delay, adding a fraction of the execution/processing time, which In some embodiments, this may be due to the time being potentially non-trivial and contributing consistently to the actual delay encountered in transmitting a packet and waiting for a response, from network propagation delays alone. ``System RtProp'' as opposed to ``Network RtProp''. In these embodiments, "System RtProp" and "Network RtProp" may be used for various calculations depending on the intended purpose. For example, "System RtProp" is appropriate so that the pipeline includes enough data in flight to account for processing delays when calculating CWND. However, when sorting network connections to determine packet routing, "Network RtProp" is more appropriate because the preferred routing decision is independent of system execution delay.

Packet loss, latency, and throughput are not completely independent variables. Those three combinations influence each other depending on the congestion control behavior of the application. This is because, for example, with TCP, higher packet loss results in longer periods of radio "silence" (underutilized channels) as the sender waits for a DUP ACK or RTO to occur. be.

Therefore, several composite attributes are determined and used in the final sorting method. in particular,
Mathis Factor: This composite attribute is based on the Mathis Relation that predicts throughput considering RTT and packet loss and provides an upper bound on TCP throughput considering RTT, packet loss, and MSS. When comparing the attributes of any two network connections to determine the sort order of these network connections, this relationship indicates that the expected throughput is (RTT*sqrt(packet loss )) is proportional to the reciprocal of This relationship is utilized in this example as a practical implementation for establishing coefficients that are utilized in directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis coefficient as follows: MAX (Rounded Rtt, 1 ms) * sqrt (Rounded Summated Packet Loss) Connections with smaller Mathis coefficient values are considered better. Because the calculation is a function of two independently measured characteristics, the relationship between these characteristics determines how the connections compare to each other. Mathis coefficients are maintained and stored in data objects, such as corresponding Mathis coefficient data values associated with each connection, and may be stored, for example, in an array of data values along with other monitored characteristics.

For example, at one extreme, two connections with 0% packet loss will always have the same Mathis coefficient, and the actual RTT values of these connections are irrelevant. In this scenario, the Mathis coefficient does not affect the sort order and is determined by subsequent comparisons of the measured attribute and the composite attribute.

At the other extreme, if two connections have 100% packet loss (rounded aggregated packet loss = 1.0), then only the rounded Rtt values of these connections have a way of comparing with each other. It is supposed to be decided. In some embodiments, this is undesirable, so connections with 100% packet loss are filtered from the available selections earlier.

Between the two extremes, connections with both low RTT and low packet loss are to be compared better than connections with higher values. However, connections with a high percentage of losses can still compensate by having a low RTT and can have the same Mathis coefficient as connections with a low percentage of losses. for example,
Connection A: 100ms RTT, 0.01 (1%) loss => Mathis coefficient = 10
Connection B: 19ms RTT, 0.25 (25%) loss => Mathis coefficient = 9.5

In this example, even though connection B has significantly higher loss (25% versus 1%), connection B's lower RTT compensates for it and makes it match a better Mathis factor than connection A. . The explanation for this is that connection B's lower RTT exchanges packet acknowledgments (positive and negative) for lost packets, and even though connection A has a smaller percentage loss, the higher Allows any necessary packet retransmissions to be completed faster than connection A with RTT.

Capacity Factor: This composite attribute considers BtlBw and RtProp and is determined as follows: Rounded BtlBw/MAX (Rounded RtProp, 1ms). The unit is "bytes per millisecond/second", which has no physical meaning, but provides a way to normalize the bandwidth that a connection can achieve per unit of RtProp. In other words, a connection that can achieve more bandwidth per RtProp millisecond of the connection is considered more efficient than a connection that achieves less bandwidth. Efficient transmission in this context means connections that achieve more bandwidth per millisecond of propagation delay (RtProp).

Once all of the raw input has been processed and the composite properties have been computed as described above, the actual sorting mechanism can operate sequentially as follows.

The transition from one comparison to the next occurs only if all previous rows compare equal. If the comparison is not equal, the remainder of the comparison is irrelevant and is omitted. In some embodiments, the selection of comparisons is made on a flow classification basis (e.g., VPN vs. VoiP vs. ping packets vs. FEC packets), and for each different flow, the comparison criteria list may be different (e.g., each may have different predefined most important characteristics, second most important characteristics, third most important characteristics, etc.). Each of these characteristics can be iterated one after the other, most important first, etc., until the connections themselves can be sorted into an ordered sequence. In some embodiments, depending on the flow classification, the first preliminary step is to select a subset of connections that are suitable for transmitting the types of data packets in the flow classification, and then the packet allocation is An order is given to them before they are executed.
1. A connection that is completely down (100% aggregated loss) is considered a worse connection than a connection that has less than 100% aggregated loss.
2. PR priority (the higher the priority, the better)
3. Mathis coefficient (lower is better)
4. Rounded RTT (lower is better)
5. Capacity coefficient (the higher the better)
6. Rounded RtProp (lower is better)
7. Rounded BtlBw (higher is better)
8. Connection ID (lower is better)

A rendering of the sorting mechanism is shown in Figure 1F. In an exemplary sorting mechanism, the real-time flows first include a comparison of RTT/RtProp, then an equal comparison, then the real-time flows compare the reliability, and then, if also equal, the transmission rate Compare etc. Depending on the flow type, there can be different sort orders. For example, a flow desiring throughput may first compare transmission rates. If they are equal, compare based on reliability. If still equal, the real-time flow may decide to evaluate nothing else and simply choose at will.

In some embodiments, flow requirements (eg, real-time vs. throughput) may determine the order and number of comparisons. For example, a real-time flow may prioritize comparing only rounded RTT and rounded RtProp, ignoring all other attributes. The throughput flow may be determined to compare only the Mathis coefficient and the capacity coefficient.

The techniques described above relate to explicit sorting dimensions. There are also implicit sorting dimensions that tend to arise as a by-product of typical application traffic patterns.

Most application traffic tends to be bursty or adaptive and therefore does not require the use of all WAN connections. This means that connections currently at the bottom of the sort order may not carry traffic on these connections. If these connections are not, then their explicit sorting dimensions are not updated, so these connections tend to maintain their position near the bottom of the sort order.

In some embodiments, push scheduling may also be configured to implicitly trigger a sort order along one or more implicit sorting dimensions over a period of time. The implicit sorting dimension forces connections that encounter periodic bad events (eg, latency spikes due to congestion) to bubble to the bottom of the list. Connections with consistent behavior naturally bubble to the top and stay near the top.

This is generally considered a good attribute of the algorithm, but it can be modified in the future to be influenced by external factors. For example, if a flow has specific requirements that cannot currently be met by connections near the top of the sort order, generate probing traffic on connections near the bottom to refresh the explicit sorting dimensions of those connections. Sometimes that makes sense.

FIG. 4A depicts an embodiment in which, at 400A, a selector is generated using a predefined flow class and current connection state.

In this example, there are three connections, each with a set of monitored characteristics, and three predefined flow classes, each defining match criteria and preferences for sorting connections that meet the match criteria. .

In this embodiment, the selector consists of a set of flow classes whose match criteria match at least one connection with available capacity. In this example, flow class 1 and flow class 2 both have match criteria that match connection 1 and connection 2, while flow class 3 does not match any connections and is not included in the selector.

Even if both connection 1 and connection 2 match, connection 2 is excluded from the selector because it does not have an available congestion window and cannot currently send packets. When constructing the selector, one embodiment examines only connection 1 to ensure that connection 2 matches, since the flow class can be included as soon as a matching connection with an available congestion window is found. It may be determined that flow class 1 and flow class 2 can be included in the selector without the need for confirmation.

Step 2b (block 1105) uses the selector from step 2a to find packets on the retransmission or transmission queues that are eligible to be (re)transmitted on active connections that have capacity. If no such packet is found, the loop ends. This can occur even if some active connections have capacity and there are packets on the input or retransmission queues.

FIG. 4B illustrates an embodiment with a priority input queue that is an ordered set of flows at 400B, where flows on the right side of the queue should send packets before flows on the left side. It is. Each flow has a flow ID and each flow belongs to one of predefined flow classes. Each flow also includes an ordered queue of packets, so if a flow is selected, the packets at the head of the queue are removed.

In this example, the selector from FIG. 4A is used to evaluate the flows in the input queue from right to left, and the selector selects the first flow that has the flow class listed in the selector. In this case, flows with ID5 and ID2 are evaluated and rejected because they both have flow class 3, which is not one of the flow classes from the selector. Flow 7 is then evaluated and matches because it belongs to flow class 1, which is one of the flow classes in the selector. A packet is then extracted from the head of the packet queue for flow 7 and stamped with flow metadata.

Once a packet is selected, step 2c (block 1105) builds an ordering of active connections. The ordering chosen is such that it is based on the rules provided in the flow class definition, if the embodiment supports it, or otherwise generally acceptable ordering. Some embodiments may use the same ordering for all flows that match a flow class, and some embodiments may make the ordering flow dependent (e.g., all flows are sticky to the same connection). (makes all packets in a flow sticky to a given connection without the need for The ordering may be built as needed at each iteration of the loop in step 2, and some embodiments may ), you may choose to cache the ordering and clear the cache.

One embodiment may reduce capacity to zero by scheduling packets on a connection, or scheduling packets on a different connection may activate a set of connections at a previously unused priority level. The ordering is constructed ignoring whether the connection currently has a capacity, since if we happen to change the metadata mix enough to cause the Building the ordering in this way allows the ordering to be cached and reused if the only items that change are the effective CWND and in-flight byte count for the connection.

Flows that traverse system 100B tend to behave better if the flow's packets traverse the same WAN connection whenever possible. Therefore, one embodiment uses a stable sorting criterion so that orderings created for the same flow are consistent unless the monitored characteristics of one or more of the connections change significantly. It's good to build an ordering. Therefore, operating characteristics such as RTT and RtProp, as well as bandwidth, are poorly chosen because they can vary over short periods of time, making the sort order unstable.

The conversion from "noisy" raw operating characteristics to monitored characteristics may be performed using statistical filtering methods. Statistical filtering methods can range from traditional measures such as mean, variance, ANOVA, etc., to the field of (quantized) rank-order statistics (rank-order filtering). For example, the monitored characteristic may be a range of raw operating characteristics, and the range may be defined by a particular value (eg, 1-100 ms), an order of magnitude, or the like. Specific examples follow in subsequent paragraphs, including the examples shown in Table 2.

Some embodiments may use machine learning techniques to convert operational characteristics to monitored characteristics. For example, a trained model can be developed to predict impending failure of a connection, so that an ordering can be constructed that sorts the matching connections to the end of the list. Another exemplary model is used to detect the type of connection (e.g., wired, cellular, WiFi, satellite, BoD, etc.) and use it to detect new monitored connections that cannot be easily measured in real time. statistics (e.g., susceptibility to buffer bloat).

One embodiment uses the following order for all packets (later criteria are only used if all earlier criteria are equal).

"Bucketed" drop probability. Connections may be grouped into buckets defined by packet count-based or size-based (eg, bytes) drop probability ranges. Connections in buckets with smaller drop probability ranges appear first in the ordering, and connections in buckets with larger drop probability ranges appear later in the ordering.

Connection priority group (provided by administrator or end user). Connections with higher priority are ranked faster (primary > secondary > tertiary, etc.)

"Bucketed" RTT. Latency is rounded to the nearest multiple of 50ms. Bucketing latency values allows normal variations in latency to be hidden from the ordering, while allowing large changes in latency to be reflected in the ordering.

"Capacity factor". This is an aggregate value measured in "bytes per second per millisecond," rounding the bottleneck bandwidth estimate to the nearest digit or an integer multiple of the nearest digit (e.g., from 1 to 1, 10 and 11 to 10, 15 to 20, 86000 to 90000) and dividing it by RtProp bucketed to the nearest 50ms. A connection that achieves a higher bandwidth per ms of RtProp is considered better than a connection that has a lower bandwidth per ms of RtProp.

Determining the capacity factor for the raw BtlBW and RtProp values is highly variable, but the capacity factor can be determined by rounding BtlBW to the nearest digit or an integer multiple of the nearest digit and bucketing RtProp. Stabilizing stabilizes the value over time unless large changes are observed. Table 2 shows some examples of bucketing, rounding, and capacity factor calculations. Note that in this embodiment, the capacity factor is rounded to the nearest integer, and a bucketed RtProp value of 0 is treated as 1 for purposes of capacity factor calculation.

"Bucketed" RtProp. Lower values of RtProp bucketed to the nearest 50 ms are preferred over higher values.

"Rolled" BtlBW. Higher values of BtlBW rounded to the nearest digit or an integer multiple of the nearest digit are preferred over lower values.

If all of the above criteria compare equal, the connection ID (assigned when the connection was enumerated) is used to break the balance deterministically. Other embodiments may use more complex tie-breaking mechanisms, such as hashing the attributes of a flow into an integer modulo of the number of connections to balance.
Table 2: Examples of bucketing, rounding, and capacity factors

One embodiment that differentiates between flow types may modify the above order to favor connections with higher capacity factors over lower latency connections for file transfer flows, e.g., depending on the flow type's transmission requirements. and may prioritize low-latency connections over high-bandwidth connections for video streams.

One embodiment that performs strict flow pinning to a connection (or set of connections) may choose to only allow exact matches of connection IDs (possibly excluding other connections from ordering altogether). ). Such pinning rules are typically provided by administrators or end users and are intended to override any ordering decisions determined by the monitored characteristics. Therefore, even if the monitored characteristics indicate that the connection is unavailable or has failed, flows will continue to have the same ordering and therefore connectivity through multipath system 100B. will no longer have.

Some embodiments may create monitored characteristics that are composites of other monitored characteristics or operational characteristics and use these composites to determine the ordering. For example, a composite reliability metric that combines bucketed drop probability, (running) latency (RtProp) variance, and (running) throughput into a weighted metric that represents the overall reliability for a connection. can be created. Connections that rank high in this metric will be reserved for flows that require high reliability (high reliability may not mean high speed).

In general, if a selector is generated that allows a packet to be selected, then that packet has an ordering that allows the packet to be scheduled on some connection that was used to generate the selector. should be generated.

FIG. 4C shows an example of how packet flow classes can be used to construct connection ordering at 400C. In the illustrated embodiment, the flow class in the packet's metadata is used to reference the flow class definition. Flow class match criteria are used to select which connections should be included in the constructed ordering (in other words, a flow class defines a set of transmission requirements that can be associated with a particular flow class). ), in this case connection 1 and connection 2 both match, while connection 3 does not match. Note that connection 2 is included even though it does not have a congestion window available (ie, is currently not eligible to send packets). This makes it easy to cache orderings and allow them to be reused even if the state of the available congestion window should change (e.g., when the scheduler (if you initially prevent sending on a connection because of a problem, but then decide to re-enable the connection because the current data mix on the enabled connection does not meet the minimum system throughput).

Once the set of connections is selected, the ordering criteria from the flow class (eg, a subset of the transmission requirements associated with the flow class) is used to sort the matching connections into an ordered list. In this example, the first criterion is the capacity factor. Both connections have the same capacity factor. The second criterion is RTT, both connections have the same RTT. The third criterion is BtlBW (bottleneck bandwidth), where connection 2 appears before connection 1 in the ordering because it has a higher value.

A set of ordering criteria (eg, transmission requirements) in a flow class allows packets to be scheduled on the available connections that are most optimal for the type of flow to which they belong. For example, packets belonging to the "Video Encoder" flow class may preferentially use connections with the lowest latency, while packets belonging to the "File Transfer" flow may have higher or highly variable latencies. In some cases, high throughput connections may be prioritized.

Once the ordering is determined for the connections, step 2d (block 1107) uses the ordering to allocate packets to connections that have available bandwidth. The packet is placed on the transmit queue for the connection.

FIG. 4D shows, at 400D, using the ordering from FIG. 4C to select a connection for a given packet. In this example, the first connection in the ordering is connection 2, but connection 2 is not currently eligible to send packets because it does not have an available congestion window. Connection 1 is then checked and selected as it can send the packet, and the packet is queued to be sent to the stage 3 pipeline for connection 1.

The in-flight byte's connection metadata (monitored characteristics) is then adjusted as part of step 2e. In some embodiments, this includes distinguishing between good put bytes and retransmission/dummy/probing bytes when tracking CWND consumption for a connection. Such adjustments may affect the goodput and/or potential goodput for the priority group that may activate the next priority level connection in the next iteration of the loop in step 2. Various methods for converting units of volume (CWND bytes) to rates (goodput) are discussed following the discussion of FIGS. 8 and 9.

When step 3 is reached, either the available CWND for each connection is full or there are no packets eligible for transmission. The packets queued in the per-connection output queue in step 2 are then sent to the appropriate instance of the next stage of the pipeline.

In some embodiments, an explicit objective function can be used to determine the desired tradeoff when there are multiple flows belonging to flow classes with the same priority but with competing requirements. can.

The embodiments described in FIGS. 4A-4D illustrate objective functions that implement fairness between flows with equal priority. The selector services each flow that matches the flow class in a round-robin manner.

A fair implementation in this embodiment uses the scarcity round robin scheduling algorithm described in RFC 8290. A brief overview of this algorithm is that it utilizes a "byte credit" scheme, allocating a quantum of byte credits to each queue in each round of round iterations across queues.

The number of bytes that can be obtained for transmission from each queue in each round is nominally limited to the number of available credits the queue has. Fairness naturally arises if the byte credit quantum for each queue is equal.

In some embodiments, the objective function may allow intentional unfairness to be configured by an administrator or end user. For example, weights may be configured on the matching flows, which may ultimately result in unequal byte credit quantum given to the queue at each iteration.

For example, there can be five different weight levels:
●Best (5)
●High (4)
●Standard (3)
●Low (2)
●Minimum (1)

The value assigned to each weight represents the ratio between the byte credit quantum of these queues. For example, if the quantum corresponding to the standard is 30KB, a flow marked low will receive (2/3)*30KB=20KB quantum in each round. The highest marked flow would receive (5/3)*30KB=50KB.

Note that the absolute numbers in this example are not important. If the standard quantum were 5KB instead of 30KB, the weights would still scale the relative quantum appropriately for other priority levels, and the end result would be the same in terms of overall fairness. Let's become.

Other embodiments may allow the weights to be arbitrary integers rather than the five fixed values in the example above. This would allow administrators or end users to configure even greater inequity between flows in the objective function if desired.

Also note that, like all quality of service (QoS), these weight settings are only important when there is contention for available WAN capacity. Since the absence of contention means that the data in all queues fits perfectly within the available capacity, the ratio between queue usage is a natural deviation from applications generating different volumes of data. It is a ratio.

Other embodiments may have completely different objective functions that do not consider fairness at all. for example,
● Guarantee that the maximum number of flows is satisfied by the available WAN capacity, or
● Maximize the total throughput achieved by the served flows, or ● Minimize changes to the current bandwidth allocation on BoD connections.

In some embodiments, the method in which the WAN connection is used can modify the attributes of the WAN connection, making it ineligible to service a particular flow class. For example, some types of connections can achieve high throughput, but at the cost of incurring a higher RTT.

An embodiment with an objective function that seeks to maximize throughput would choose to service flows with high throughput requirements, which would increase the WAN connection RTT and reduce the WAN connection to This would make it unsuitable to service flow classes with low latency requirements.

Conversely, embodiments with a goal function that seeks to serve the maximum number of flows may choose the opposite trade-off so that low-latency flow classes can continue to be served by this WAN connection.

In some embodiments, the connection sort order of individual schedulers is maintained per flow rather than globally for all flows.

This allows flow-specific requirements to influence the sorting order so that the flow's packets are assigned to connections in a way that meets the needs of the connections.

For example, a VoIP flow with a preferred maximum latency tolerance tolerance of 150ms would want a sorting criterion that places connections with RTTs greater than 150ms at the bottom of the list. Conversely, TCP flows that do not have specific latency requirements may instead either ignore RTT completely or use RTT only as one of many secondary criteria. One would want to prioritize connection throughput in this TCP flow sorting criterion.

In some embodiments, these flow requirements and preferences are configured in the form of rules consisting of match criteria and resulting behaviors.

For example, the match criteria can take the form of an IP3-tuple (protocol, source IP, and destination IP) or an IP5-tuple (protocol, source IP, source port, destination IP, and destination port). Behavior takes the form of explicit preferences for latency or throughput, with target preferred values for either.

Examples of other heuristic-based match criteria may include:
● DSCP tag ● Any other header field available in the IP (v4 or v6) header ● Any other header field in the TCP header or UDP header ● The cumulative volume of data transmitted, e.g. a volume threshold TCP flows that exceed the cumulative duration of the flow may match as "bulk data transfers" while those that do not may not match as interactive SSH/telnet sessions. Regular expression match Clear text parameters extracted from TLS handshake (e.g. SNI, CN, SubjectAltName)

Examples of other operations that can affect scheduler sort order include:
● Specific preferences for connection order (e.g., the user may prefer sorting expensive connections to the bottom of the list for low priority flows)
● Jitter tolerance ● Maximum latency tolerance ● Desired amount of redundancy (e.g. FEC)
● Whether retransmission of lost packets is desired, and if so, whether there is a maximum time limit at which retransmission attempts should stop ● Whether explicit packet pacing is desired ● For example, 30 per second A real-time video application that transmits video in frames may transmit video frames of video exactly 33.3 milliseconds apart and would not want multipath system 100B to change the pacing.
- In contrast, bursty applications may benefit from having the scheduler explicitly pace bursts of video at the aggregate WAN rate.
● Desired throughput (e.g. minimum, target, maximum)

Rules (both match criteria and actions) can be automatically generated using machine learning techniques that analyze traffic patterns.

Inputs (features) to the ML method may include the following:
i. Cumulative volume of data transmitted ii. Packet size distribution (histogram)
iii. Inter-packet spacing (both pacing and jitter)
iv. Packet frequency and grouping v. Packet header fields (Ethernet, IP, TCP, UDP, etc.)
vi. Packet content (payload)
vii. IP addressing as well as the intended destination of the packet (e.g. SNI, CN, and SubjectAltName fields in the TLS handshake and exchanged certificates)
viii. Cumulative duration of flow ix. Time x. Number of simultaneous flows to the same destination or from the same source xi. Current or historical predictions (labels) of any simultaneous flows to the same destination or from the same source (e.g., some applications open multiple flows, one as a control plane and one as a control plane) as a data plane and knowing that there is an existing control plane flow can help predict the data plane flow)

Predictions (labels) from the ML method may include any of the aforementioned behaviors, determined based on the behavior of the training corpus.

In some embodiments, a rule may have the concept of a subordinate rule that applies further matching criteria and more specific behavior while still being managed by a parent rule. for example,
- Any VPN flow identified by an IP5 tuple may have rules defined with match criteria and actions. This would be considered the "parent" rule.
- A VPN will carry many internal flows within the VPN from applications protected by the VPN. Normally these internal flows would be completely hidden from the match criteria (encrypted by the VPN).
However, some VPNs choose to expose a limited set of information about internal flows by setting a DSCP tag on encrypted packets that corresponds to a preference for internal (clear text) packets. It is possible.

Although dependent rules may be created that match DSCP tags, the resulting available behaviors will still be governed by the parent rule. for example,
a. Generally, VPNs require packets to be delivered in order (VPNs can tolerate out-of-order, but only up to a small window).
b. The behavior configured by subordinate rules that match DSCP tags must not result in parent rules delivering packets significantly out of order.

For example, a dependent rule requiring a maximum latency limit of 150 ms and a second dependent rule requiring maximum throughput may allow interleaved packets from these two dependent rules to be assigned to connections with significantly different latencies. may violate the ordering requirements of the parent rule.

FIG. 5A illustrates two independent flows whose rules and behavior are completely independent. In this example, since SIP flows require low latency, the behavior causes scheduler 104B to transmit only packets for SIP flows on connection 1. Conversely, FTP flows require high throughput, so scheduler 104B transmits only packets for FTP flows on connection 2.

When packets are transmitted over the air for these two flows and reach sequencer 118, they are put back in order and transmitted independently to their final destination. In this example, sequencer 118 is transmitting packets belonging to a SIP flow that have already arrived and are in the correct order, regardless of the state of the FTP flow in which the packets are still being transmitted over the air.

FIG. 5B illustrates the concept of dependent flows. In this example, the same SIP and FTP flows exist, but before scheduler 104B sees these flows, they pass through the VPN client for encryption and encapsulation.

When coming out of the VPN client, the packets from both flows have the same IP5 tuple, so the scheduler 104B and sequencer 118 treat it as the parent flow and are constrained by the requirements of the parent flow.

The presence of dependent flows (SIP and FTP) is communicated using DSCP tags, which control how scheduler 104B assigns packets to connections 1 and 2 for transmission. In this example, the assignments are the same as in FIG. 5A.

However, parent flow constraints require that packets be transmitted from sequencer 118 in the same order as they arrive at scheduler 104B. Therefore, in this example, even if the packets belonging to the subordinate SIP flow have already reached the sequencer 118, the packets belonging to the subordinate SIP flow are transmitted until the packets belonging to the subordinate FTP flow arrive and are transmitted first. I can't.

FIG. 6A, at 600A, is a sample embodiment of a rules management page. The configured rules are displayed in a table, with each row summarizing match criteria and behavior. The order in which the rules are listed is the order in which packets match the rules. FIG. 6B shows a workflow for changing the order of rules at 600B.

FIG. 6C shows how each row in the table can be expanded at 600C to show details of the match criteria and behavior, as well as the currently active flows through the system that match this rule.

Each rule in the table includes edit and delete icons.

Figure 6D shows the dialog that appears when the edit button is clicked at 600D. Match criteria and behavior for the rule can be changed in this dialog.

There is a separate Add Rule link on the main rules management screen, and FIG. 6E shows the mode dialog that appears when clicked at 600E.

In some embodiments, rules are configured at the sender and automatically pushed to the receiver. By default, rules are symmetric and bidirectional. This means that once a rule is added and the rule behavior is configured, matching pair rules are now added transparently, but the source and destination match criteria are now swapped. There is. It would be possible to add asymmetric rules (different behavior for either direction of the same flow) via the advanced configuration screen.

In one embodiment, the scheduler probes various network characteristics of the plurality of connections (to determine the measured characteristics for each connection) and uses the measured characteristics in combination with transmission requirements to Make transmission decisions.

For example, one such decision is determining how many packets to transmit on a particular connection and when to transmit these packets.

Some embodiments are volume-based. Such embodiments operate by determining a limit on the volume of data (eg, in bytes or packets) that can be transmitted over a particular connection. Once enough packets have been transmitted for that amount of volume, transmission will stop until some feedback regarding these packets is received. If feedback is received before the full volume limit has been transmitted, the transmission may continue without stopping as long as the volume of data transmitted for which no feedback has been received exceeds the determined volume limit. can.

In one embodiment, the volume of data is chosen to be the congestion window (CWND) of the connection. Simply put, CWND is the maximum volume of data that can be sent on a connection without causing significant congestion on the connection before any feedback is received. There are many methods for estimating CWND. It is assumed that the scheduler has access to one such method and can receive an estimate of CWND via that method.

In some embodiments, a scheduler is required to determine the rate at which data is being transmitted (eg, in bytes per second). This rate can then be used to make other decisions regarding packet transmission.

For example, in one embodiment, if the rate of data being transmitted on the currently active network connection(s) is less than the rate of data being received from the source application for transmission, the scheduler may A decision may be made to activate more connections to increase the aggregate transmission rate to accommodate the application rate.

In another example, an embodiment may have to meet quality of service (QoS) requirements, which can be expressed in terms of guaranteed transmission rates. If the aggregate rate falls below the required level, the scheduler may decide to activate more connections to increase the aggregate transmission rate to meet the QoS requirements.

In yet another example, an embodiment may measure goodput (i.e., the rate at which application data is successfully passed through a network) for purposes such as ensuring application performance levels, optimizing performance, reporting, etc. may be required.

In one embodiment, TCP performance may be optimized by pacing transmitted packets at an aggregate rate achieved by multiple connections. See, for example, PCT Application No. PCT/CA2020/051090.

In embodiments where rates are required for some functionality, a volume-based scheduler requires a way to convert the volume of data being transmitted into a rate.

The trivial technique of converting volume to rate by dividing the volume by the transmitted time duration generally yields inaccurate results. Such techniques measure the transmission rate at the transmitter moving through the network, rather than the actual rate data. The actual rate at which data moves through a network is truly determined by the rate at the slowest link in a multi-link network, commonly referred to as the network bottleneck.

The characteristics of bottleneck links in a network are particularly important to transmitters or receivers, who are aware that multi-link networks can change the combination of links used at any time, transparently to the transmitter and receiver. Not available. However, congestion control methods can generate estimates based on observed performance at the transmitter and receiver.

In some embodiments, the congestion control method may provide an estimate of the transmission rate of a bottleneck link. This is sometimes referred to as the network bottleneck bandwidth BtlBw.

In some embodiments, the network round-trip propagation time RtProp may be estimated by sampling the time between when a packet is sent and when feedback regarding the packet is received.

The combination of BtlBw and RtProp can be used to determine a network attribute called bandwidth-delay product BDP as follows.
BDP=BtlBw×RtProp

The most common unit for BDP is the byte. The following units are assumed for these quantities: (i) bytes per second for BtlBw, (ii) seconds for RtProp, and thus (iii) bytes for BDP.

BDP refers to the amount of data that would be present in a network (in-flight) when operating under ideal conditions, assuming that data is always available for transmission. This network will transfer data at the network's nominal rate BtlBw and will provide feedback on every packet just after the RtProp time to send that packet. FIG. 7 shows two examples at 700. The top half of Figure 7 illustrates this concept using an example in which the BDP consists of 16 equally sized packets. In reality, the amount and size of the packets can be different as long as the total size is equal to the value calculated using BDP.

In some volume-based embodiments, BDP can be used to estimate various data transfer rates, some examples of which are described above.

For the in-flight volume under consideration, assuming ideal conditions, BtlBw, which is only a fraction of the network's nominal rate, can be achieved if the volume is smaller than the BDP. This fractional value can be calculated as the ratio of volume to BDP. The resulting achieved rate is determined as follows:
Achieved rate corresponding to in-flight volume = Volume/BDP x BtlBw

In reality, networks rarely operate under ideal conditions. For example, packets may be temporarily buffered before being delivered to the receiving end, and feedback may be delayed at the receiver or by the network. In some cases, packets may be lost and never reach the receiver, thereby not triggering feedback. In other cases, the feedback itself may be lost. Loss may be estimated based on, for example, when new packet feedback is received or on timer expiration.

Waiting for delayed feedback (which may or may not arrive) to trigger the next transmission event may artificially limit the achievable rate in the network.

FIG. 8A shows an example of a receiver in FIG. 800A that aggregates feedback and sends an acknowledgment back to the sender once for every four packets received.

FIG. 8B continues to illustrate in diagram 800B how, for example, aggregation of acknowledgments results in the overall throughput being less than the capacity of the network. The sender waits longer before starting to send the next group of packets. Overall, the transmitter ends up transmitting the same amount of packets as in the ideal case of FIG. 7, but over a longer period of time, resulting in reduced throughput.

In one embodiment, this artificial limitation is overcome by specifying an additional allowed amount of data to transmit even if feedback of previously transmitted packets has not yet been received. This means that the volume of in-flight data can exceed the BDP. However, in such a case, the network is not actually transferring data at the higher rate, as application of the achieved rate formula above may suggest. The additional allowance simply allows the transmitter to maintain a constant rate equal to BtlBw for a longer period of time. Therefore, the estimated transfer rate of the volume at hand is capped at BtlBw.

FIG. 8C continues to illustrate in diagram 800C how a transmitter may use such additional transmission allowance to continue transmitting packets, e.g., before an acknowledgment is received. There is. The volume of in-flight data, 19 packets, exceeds the BDP of 16 packets. However, the actual rate matches the network capacity, and the calculated rate capping achieves the same result by using the in-flight volume minimum and BDP.

In some embodiments, similar techniques can be used to estimate the goodput that may be achieved on the network, hereinafter referred to as potential goodput.

A particular volume of data can be divided into a goodput portion and other portions. The goodput portion contains newly transmitted application data. The other part includes retransmissions of data that cannot be counted as newly transmitted application data, such as previously lost data, control data, probing data, etc.

If the total volume of data is larger than BDP, packet and/or feedback delays are assumed as above. Assume there is no extra space available for additional goodput. The resulting (potential) good put rate is estimated as follows:
Good put rate = Good put portion/total volume x BtlBw

For example, FIG. 9A shows a network with a 16-packet BDP at 900A. To achieve a throughput equal to the network capacity, the volume taken in-flight is 19 packets. However, only 14 packets are goodput, resulting in a goodput rate of 7/8 of the network's capacity. If the volume of data is less than the BDP, the difference is considered to be extra space that is assumed to be available for additional goodput.

FIG. 9B shows an example in 900B where only 3 packets are in-flight, resulting in an extra space of 13 packets that can be assumed to be good put when calculating the potential good put rate. This extra space must be limited by the CWND determined by the congestion control method. At any given time, the additional volume that congestion control allows is the difference between CWND and total volume. If the extra space in the goodput exceeds this congestion control allowance, the space is simply limited to the congestion control allowance.

FIG. 9C shows an example where CWND is reduced to 11 packets in 900C, which limits the extra space assumed available for additional goodput to 8 packets.

The resulting potential goodput rate is estimated as follows:
Potential goodput rate = (goodput portion + limited goodput extra space) / BDP x BtlBw

Continuing with the example above, Figures 9B and 9C show how the potential goodput equations above result in a lower potential goodput rate for the scenario in Figure 9C, which is as expected considering the CWND reduction enforced by congestion control methods.

Although the above technique is described for goodput and potential goodput rates, the technique can be extended to calculate the rate or potential rate of any type of data that partially constitutes a total volume. Can be easily modified.

Below is a numerical example applying the above equation to a network with attributes typical of those encountered over LTE cellular networks.
BtlBw=10Mbps
RtProp=60ms
BDP=10Mbps*60ms=75KB
CWND=60KB
Goodput part = 25KB
Other parts = 10KB
Total volume = 25KB + 10KB = 35KB
Goodput extra space = BDP - Total volume = 75KB - 35KB = 40KB
Congestion control allowance = CWND - total volume = 60KB - 35KB = 25KB
Limited Goodput Extra Space =
Min (goodput extra space, congestion control allowance) =
Min (40KB, 25KB) = 25KB
Potential good put rate =
(Goodput portion + limited goodput extra space)/BDPxBtlBw=
(25KB+25KB)/75KBx10Mbps=50/75x10Mbps=6.67Mbps

Systems that communicate over IP networks often implement congestion control mechanisms that seek to achieve fair network utilization by all network nodes employing similar congestion control mechanisms.

Congestion control mechanisms often operate by monitoring communications occurring on a network, possibly probing the network as necessary, and deriving network attributes based on the results. A model representing the network is created, and subsequent communications are guided based on this model. For example, the model can track network attributes such as current maximum throughput achieved, current and minimum round trip times for packets, packet loss events, etc.

Based on the created model and tracked information, the congestion control mechanism can determine when the network is underperforming, indicating that the network may be experiencing congestion. The congestion control mechanism may then take corrective actions aimed at reducing or eliminating network congestion and restoring desired network performance.

In traditional congestion control implementations (eg, TCP), packet loss is used as an indicator of network congestion, and corrective action (eg, reducing CWND) is taken in response to the loss.

In one embodiment, packet latency, ie, the time required for a packet to traverse a network from a transmitter to a receiver, is an indicator of network performance. High latency packets may, in some cases, be of no use to the receiver and may be discarded, wasting the transmission of these packets. For example, real-time VoIP applications that require low conversational delays between participants are designed to avoid using packets carrying voice data that are older than the maximum allowed conversational delay.

In such embodiments, the congestion control mechanism may monitor packet latency and attempt to control the latency so that the packet latency does not exceed an acceptable level.

In one embodiment, the lowest latency measured for a packet may form a baseline value to which subsequent measurements may be compared. If subsequent measurements exceed a threshold that is a function of the baseline value, the congestion control mechanism considers the network to be experiencing congestion.

In another embodiment, a similar technique uses a packet's round-trip time, which is the time that elapses between sending a packet and receiving an acknowledgment for it, instead of latency. .

In one embodiment, when a congestion control mechanism determines that a network is experiencing congestion based on increased packet delay, a congestion window (CWND) may be transmitted on this network without receiving any feedback. Reduced to limit the amount of data. The objective is to reduce the volume of data that will be or is currently buffered by the network, which is a common cause of increased packet delay.

In one embodiment, the congestion window is reduced to the bandwidth-delay product (BDP) of the network. In theory, BDP assumes that packets are transmitted by the transmitter at a constant pace that reflects the network throughput (BtlBw), without forming a static queue of packets in its buffer. Represents the volume of data that a network should be able to deliver. This is therefore expected to allow the network to at least partially recover and reduce packet delays back to acceptable levels.

For example, assume a network with a BDP of 16 packets and a congestion control mechanism that has determined a CWND of 24 packets to account for acknowledgment aggregation. FIG. 10A illustrates how congestion from other network nodes can cause a transmitter to dominate the network at 1000A. In this case, the transmitter does not reduce the transmitter's congestion window and takes the entire amount in-flight. This causes the network buffer to fill up and drop extra packets that cannot be buffered. Even after congestion has cleared and normal draining of network buffers continues, Figure 10A shows how to increase the round-trip time of all subsequent packets and keep the network buffers full. It illustrates how a static queue could be formed inside the network buffer to make it easier.

Continuing with this example, FIG. 10B illustrates how a transmitter reducing the transmitter's congestion window to BDP at 1000B avoids both packet drops and the formation of a static queue. ing. The transmitter may determine that congestion is occurring, for example, by detecting that no feedback has been received for a period that exceeds a baseline duration for feedback that the transmitter has measured in past transmissions. . In another example, a transmitter detects congestion by receiving feedback indicating that the amount of previously transmitted packets that exceed the baseline loss encountered in past transmissions is not being delivered to the receiver. It can be determined that In response to this congestion determination, reducing the congestion window matching the BDP reduces the amount of packets that will be taken in-flight. Consequently, this reduces the possibility of filling up network buffers, which previously led to packet drops. Additionally, bringing only BDP packets in-flight allows the network to completely drain the network buffer before a new packet from the transmitter reaches the network buffer again (if the network attribute is (assuming no changes).

In another embodiment, the congestion window is reduced to be equal to the volume of packets currently in flight.

Once the congestion control mechanism determines that an acceptable level of performance has been restored, the network is considered to no longer experience congestion.

In one embodiment, a reduction in packet delay below a threshold is an indication that network performance has been restored.

In one embodiment, the threshold is determined as a function of the baseline delay value.

In another embodiment, transmitting a particular volume of data without incurring any loss is an indication that network performance has been restored.

In one embodiment, the volume of data is equal to the BDP.

In another embodiment, the volume of data is equal to the original value of the congestion window before it was reduced by a congestion control mechanism in response to congestion.

Push architecture improves system performance with non-BoD connections for the most common application types. Table 3 summarizes the improvements observed experimentally.
Table 3: Application performance improvement due to push scheduler

The push scheduling method also makes the priority routing behavior more intuitive. For example, in one embodiment, the decision whether to engage lower priority connections depends solely on how the BtlBw of those connections compares to a configured threshold.

For example, consider the following scenario:
wan0
●Priority: Primary ●Target threshold: Not applicable ●BtlBw: 10Mbps
wan1
●Priority: Secondary ●Target threshold: 15Mbps
●BtlBw: 30Mbps

With a pull scheduler, each time WAN1's 50Hz timer wakes up and requests a packet, the scheduler will determine that the desired target threshold (15Mbps) is greater than the threshold that WAN0 alone can provide (10Mbps). Since the scheduler knows this, it provides packets for transmission to WAN1.

However, the scheduler does this regardless of the rate at which packets arrive at the input. For example, if the input rate does not exceed 10 Mbps, it would be more optimal if WAN1 was not involved, since the traffic can be completely contained by WAN0 alone.

With a push scheduler, WAN0 would be sorted to the top of the list, and packets would be sent to WAN0 only after the WAN0 CWND is completely consumed, even though the threshold indicates that WAN1 should be active. scheduled on. As a result, there is less unintended use of lower priority connections, improving performance.

In summary, a multipath system scheduler can be implemented as either a pull type (100A) or a push type (100B). Push is better because the scheduler can make packet scheduling decisions in a self-determined rhythm, rather than being driven by the rate at which the target connection is ready to transmit. This self-determining rhythm allows consideration of all factors that are important to the application that generated the packet and/or the user of the system.

For example, packets generated by a TCP-based application that requires maximum throughput will prefer that the packet be scheduled on connections that are ordered from best Mathis coefficient to worst Mathis coefficient.

Consider also a more complex example using the same TCP-based application, but where the user has configured multipath system 100B with priority routing rules and the achievable goodput at 110B does not drop below 5 Mbps. connection 110B is given priority over all other connections 110N. In a hypothetical scenario where 110B has high throughput, low latency, but high packet loss, 110B would have a worse Mathis factor than all other connections 110N.

Push scheduler 100B makes complex scheduling decisions that initially respect user preferences for using connection 110B, and also converts the in-flight CWND of connection 110B to a bit rate and uses it to By estimating the available goodput of 110B, one could calculate the achievable goodput of 110B. If this availability drops below the configured 5 Mbps threshold, 110B can schedule packets on the remaining connections 110N, ordered from best Mathis coefficient to worst Mathis coefficient.

FIG. 11 is a schematic diagram of a computing device 1100 that may be used to implement system 100B, according to one embodiment.

As depicted, computing device 1100 includes at least one processor 1102, memory 1104, at least one I/O interface 1106, and at least one network interface 1108.

Each processor 1102 may include, for example, a microprocessor or microcontroller (e.g., a dedicated microprocessor or microcontroller), a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable readout It may be a special purpose memory (PROM) or various combinations thereof.

Memory 1104 may include, for example, random access memory (RAM), read only memory (ROM), compact disk read only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EPROM). It may include a combination of various types of computer memory, located either internally or externally, such as Enabled Programmable Read Only Memory (EEPROM), Ferroelectric RAM (FRAM), etc.

Each I/O interface 1106 interconnects the computing device 1100 with one or more input devices, such as a keyboard, mouse, camera, touch screen, and microphone, or with one or more output devices, such as a display screen and speakers. make it possible to

Each network interface 1108 can be used to connect the computing device 1100 to the Internet, Ethernet, Plain Old Telephone Service (POTS) lines, public switched telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial Data including cable, fiber optic, satellite, mobile, wireless (e.g., Wi-Fi, WiMAX), SS7 signaling networks, fixed line, local area networks, wide area networks, and others, including various combinations thereof. Communicate with other components, exchange data with other components, access and connect to network resources, provide applications, and connect other components by connecting to a network (or networks) that may carry Enables you to run computing applications.

Although one computing device 1100 is shown for simplicity only, system 100B may include multiple computing devices 1100. Computing devices 1100 may be the same or different types of devices. Computing devices 1100 may be directly coupled, indirectly coupled via a network, distributed over a wide geographic area, or connected via a network (which may be referred to as "cloud computing"). may be connected in a variety of ways, including.

For example, and without limitation, computing device 1100 may include a server, network appliance, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, mobile phone, smartphone device, UMPC tablet, video display terminal. , a game console, or various other computing devices that can be configured to perform the methods described herein.

Applicants emphasize that the described embodiments and examples are illustrative and non-limiting. Practical implementations of the features may incorporate combinations of some or all of the features, and the features described herein should not be taken as an indicator of future or existing product plans.

The terms "connected" or "coupled" refer to both direct couplings (two elements joined to each other are in contact with each other) and indirect couplings (at least one additional element is located between the two elements). may include both.

Although embodiments have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope. Furthermore, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.

Throughout the foregoing description, numerous references may be made to packet controllers, servers, instances, interfaces, portals, platforms, or other systems formed from computing devices. It is understood that use of such term is deemed to refer to one or more computing devices having at least one processor configured to execute software instructions stored on computer-readable tangible non-transitory media. I want to be For example, a packet controller may implement the technical solution of the described embodiments in the form of a software product. The software product may be stored on a non-volatile or non-transitory storage medium, which may be a CD-ROM, a USB flash disk, or a hard disk (removable or otherwise). The software product includes a number of instructions that enable a computing device (eg, a packet controller or a network device) to perform the methods provided by the embodiments.

As can be understood, the embodiments described and illustrated above are intended to be exemplary only.

Claims (42)

a packet communications controller coupled to a plurality of network connections, each having a corresponding output queue from a plurality of output queues, the packet communications controller being a processor;
monitoring and maintaining, in a data object, network characteristics associated with each network of the plurality of networks such that each network of the plurality of networks can be ordered in an ordered sequence based on the network characteristics;
When it is determined that a trigger event has occurred,
establishing the ordered sequence for network connections of the selected set of network connections based on the network characteristics;
one or more of the unallocated data packets of the input queue by assigning the one or more data packets to consecutive available network connections based on availability and the ordering sequence; assigned to one or more corresponding output queues of the network;
updating the monitored characteristics of the corresponding plurality of networks to reflect the assigned one or more data packets, and communicating the assigned data packets from the plurality of output queues to a destination device. A packet communications controller, including a processor configured to. the packet communication controller is a push scheduler, and the processor is configured to: determine whether monitored characteristics of the plurality of networks are favorable;
further configured to measure transmission capacity of each of the plurality of network connections;
The packet communication of claim 1, wherein the controller is configured to allocate the one or more data packets while the transmission capacity of one or more of the plurality of network connections is not exhausted. controller. The processor transmits the one or more data packets upon detecting a new data packet in the input queue or upon receiving an acknowledgment that one or more previous data packets were received at the destination device. The packet communications controller of claim 1, further configured to allocate. The processor,
For each of the plurality of network connections, the monitored characteristic:
estimating at least one of a transmission rate, a round trip time, and a round trip propagation time of the bottleneck link;
The at least one of the estimated round-trip propagation time, the estimated round-trip time, and the estimated transmission rate is defined as a round-trip propagation time, a reference range of round-trip times, and a reference range of transmission. for each of the at least one of the estimated round-trip propagation time, the estimated round-trip time, and the estimated transmission rate under consideration, as compared to a respective reference range of at least one of the establishing a score value for;
The packet communications controller of claim 1, further configured to measure by ordering the plurality of networks in the ordered sequence based on the score value for each of the plurality of networks. The estimate of the at least one of the transmission rate, round-trip time, and round-trip propagation time of the bottleneck link is one of the transmission rate, the round-trip time, and the round-trip propagation time of the bottleneck link. estimating a plurality of score values for at least two, said ordered sequence is based on a comparison of said score values corresponding to a predefined most important characteristic, and said ordered sequence is based on a comparison of said score values corresponding to said predefined most important characteristic; 5. The packet communication controller of claim 4, wherein if the corresponding score values are equal, proceed to the next predefined most important characteristic for comparison. The processor is configured to perform a flow classification of the unallocated data packets, and based on the flow classification, the predefined most important characteristic and the predefined next most important characteristic are determined. 6. The packet communication controller of claim 5, wherein the packet communication controller is identified. The processor,
measuring modified monitored characteristics, including system overhead processing time, for each of the plurality of network connections, and utilizing either the original characteristics or the modified characteristics;
determining at least one of an estimate of the transmission rate, the round trip time, the round trip propagation time, a bandwidth delay product (BDP), and a congestion window of a bottleneck link;
The ordering of the plurality of networks is further based on the at least one of the estimates of the transmission rate of a bottleneck link, the round trip time, the round trip propagation time, the BDP, or the congestion window. The packet communication controller according to claim 4. The processor,
one or more corresponding sliding window histories for each of the plurality of network connections based on other measured characteristics including at least one of packet loss in the round trip time or change in the round trip time; 8. The packet communications controller of claim 7, further configured to modify the measured characteristic by dynamically adjusting the duration of the packet communication controller. The processor,
estimating a Mathis coefficient data value indicating a relative upper limit of achievable throughput for the network connection of the plurality of network connections, taking into account the round-trip time and packet loss of the network connection;
further configured to compare the estimated Mathis coefficient data value to a reference range of Mathis coefficient data values;
5. The packet communications controller of claim 4, wherein the ordered sequence of the plurality of networks is further based on the comparison of the estimated Mathis coefficient data values and the reference range. The processor is configured to perform a flow classification of the unallocated data packets, and based on the flow classification, the predefined most important characteristic and the predefined next most important characteristic are determined. A packet communication controller according to any one of claims 7 to 9, wherein the packet communication controller is identified. In order to order the plurality of networks, the processor:
estimating a capacity factor indicative of efficient transmission defined at least in part by dividing the reference range of transmission rates of the bottleneck link by the reference range of round-trip propagation times;
5. The packet communication of claim 4, further configured to compare the estimated capacity factor to a reference range of capacity factors, and order the plurality of networks based on the comparison of the capacity factors. controller. In order to define the transmission capacity, the processor:
determining whether one or more of the plurality of networks allocates transmission capacity using a demand allocation multiple access (DAMA) method;
configured to define the transmission capacity of the one or more DAMA networks to be greater than an allocated transmission capacity received from the DAMA network;
The processor,
In response to determining that the received allocated transmission capacity of one or more DAMA networks is currently exhausted but has more potential capacity that can be allocated through demonstration of a request, one The packet of claim 2, further configured to allocate the probe data packets or more to the one or more output queues corresponding to the one or more DAMA networks having potential transmission capacity. communication controller. The processor,
maintaining a nominal time for each of the plurality of networks to have available preferred monitored characteristics;
while assigning the one or more data packets to the one or more output queues, appending the maintained nominal time to each of the one or more data packets; and The packet communications controller of claim 1, configured to schedule data packets for pushing from the respective output queues based on a nominal time. The processor,
13. The packet communications controller of claim 12, configured to add a prioritized nominal time to the one or more data packets identified as packets for retransmission. each of the one or more data packets is associated with one or more data flows, and the processor:
the one or more data packets;
estimating a data packet classification associated with each of the data packets based on a header associated with the one or more data packets;
determining one or more transmission requirements associated with the estimated data packet classification;
2. The network according to claim 1, configured to allocate the one or more data packets by allocating the one or more data packets to the plurality of networks that satisfy the corresponding one or more transmission requirements of the one or more data packets. packet communication controller. The processor,
maximizing one of several data flows serviced, maximizing transmission power, or minimizing changes in current bandwidth allocation for one or more DAMA networks of said plurality of networks; 13. The packet communications controller of claim 12, configured to allocate data packets to the plurality of networks based on an objective function including one of: The processor,
determining whether the one or more data packets are associated with one or more data flows that include one or more previously allocated data packets, and the one or more data packets are associated with one or more previously allocated data packets; associating the one or more data packets with the previously allocated data packet in response to determining that the one or more data packets are associated with one or more data flows including the previously allocated data packet; 13. The packet communication controller of claim 12, wherein the packet communication controller is configured to allocate to the plurality of networks. 3. The packet communications controller of claim 2, wherein the transmission capacity is at least partially defined by BDP, CWND, or in-flight data packets. The processor,
for each network, including a conversion between a rate measurement measured by at least one of goodput and throughput, and a volume measurement including at least one of BDP, CWND, and in-flight data. maintaining baseline monitored characteristics;
determining whether the updated monitored characteristic varies by a threshold amount from the baseline operating characteristic;
for each network in response to the updated monitored characteristic varying by the threshold amount from the baseline operation being monitored;
16. The packet communications controller of claim 15, further configured to: assign the one or more data packets to one or more output queues of other available networks. The processor,
maintain baseline monitored characteristics for each network;
determining whether the updated monitored characteristic varies by a threshold amount from the baseline operating characteristic, and the updated monitored characteristic varies by the threshold amount from the baseline operating characteristic. In response, for each of said networks:
Define the transmission capacity of each of the networks as loss recovery capacity,
The packet communications controller of claim 1, configured to assign the one or more data packets to the corresponding output queue of the network based on the loss recovery capability. A method for packet communication using a packet communication controller coupled to a plurality of network connections, each having a corresponding output queue from a plurality of output queues, the method comprising:
monitoring and maintaining, in a data object, network characteristics associated with each network of the plurality of networks such that each network of the plurality of networks can be ordered in an ordered sequence based on the network characteristics;
When it is determined that a trigger event has occurred,
establishing the ordered sequence for network connections of the selected set of network connections based on the network characteristics;
one or more of the unallocated data packets of the input queue by assigning the one or more data packets to consecutive available network connections based on availability and the ordering sequence; assigning to one or more corresponding output queues of the network;
updating the monitored characteristic of the corresponding plurality of networks to reflect the assigned one or more data packets;
communicating the assigned data packets from the plurality of output queues to a destination device. The packet communication controller is a push scheduler, and the method includes:
measuring the transmission capacity of each of the plurality of network connections;
The allocation of the one or more data packets may include assigning the one or more data packets to the plurality of network connections only if the transmission capacity of the corresponding one of the plurality of network connections is not exhausted. 22. The method of claim 21, comprising assigning to the corresponding connection of: The allocation of the one or more data packets is performed upon detecting a new data packet in the input queue or upon receiving an acknowledgment that one or more previous data packets have been received at the destination device. The method according to item 21. For each of the plurality of network connections, the monitored characteristic:
estimating at least one of a transmission rate, a round trip time, and a round trip propagation time of the bottleneck link;
The at least one of the estimated round-trip propagation time, the estimated round-trip time, and the estimated transmission rate is defined as a round-trip propagation time, a reference range of round-trip times, and a reference range of transmission. for each of the at least one of the estimated round-trip propagation time, the estimated round-trip time, and the estimated transmission rate under consideration, as compared to a respective reference range of at least one of the establishing a score value for;
22. The method of claim 21, further comprising: ordering the plurality of networks in the ordered sequence based on the score value for each of the plurality of networks. The estimate of the at least one of the transmission rate, round-trip time, and round-trip propagation time of the bottleneck link is one of the transmission rate, the round-trip time, and the round-trip propagation time of the bottleneck link. estimating a plurality of score values for at least two, said ordered sequence is based on a comparison of said score values corresponding to a predefined most important characteristic, and said ordered sequence is based on a comparison of said score values corresponding to said predefined most important characteristic; 25. The method of claim 24, wherein if the corresponding score values are equal, proceed to the next predefined most important characteristic for comparison. 4. The method of claim 1, further comprising performing a flow classification of the unallocated data packets, and based on the flow classification, the predefined most important characteristic and the predefined next most important characteristic are identified. 25. The method described in 25. measuring modified monitored characteristics, including system overhead processing time, for each of the plurality of network connections, utilizing either the original characteristics or the modified characteristics;
determining at least one of an estimate of the transmission rate, the round trip time, the round trip propagation time, a bandwidth delay product (BDP), and a congestion window of a bottleneck link;
The ordered sequence of the plurality of networks further comprises: the at least one of the estimates of the transmission rate of a bottleneck link, the round-trip time, the round-trip propagation time, the BDP, and the congestion window; 25. The method according to claim 24, based on. one or more corresponding sliding window histories for each of the plurality of network connections based on other measured characteristics including at least one of packet loss in the round trip time or change in the round trip time; 28. The method of claim 27, further comprising modifying the measured characteristic by dynamically adjusting the duration of. estimating a Mathis coefficient data value indicating a relative upper limit of achievable throughput for the network connection of the plurality of network connections, taking into account the round trip time and packet loss of the network connection;
further comprising: comparing the estimated Mathis coefficient data value to a reference range of Mathis coefficient data values to determine a Mathis coefficient score;
25. The method of claim 24, wherein the ordered sequence of the plurality of networks is further based on the comparison of the estimated Mathis coefficient data values and the reference range. In order to order the plurality of networks, the method includes:
estimating a capacity factor indicative of efficient transmission defined at least in part by dividing the reference range of transmission rates of the bottleneck link by the reference range of round-trip propagation times;
comparing the estimated capacity factor to a reference range of capacity factors to establish an estimated capacity factor score;
25. The method of claim 24, further comprising: ordering the plurality of networks based on the comparison of the capacity factors. The processor is configured to perform a flow classification of the unallocated data packets, and based on the flow classification, the predefined most important characteristic and the predefined next most important characteristic are determined. 31. A method according to any one of claims 27 to 30, wherein said method is identified. In order to define the transmission capacity, the method comprises:
determining whether one or more of the plurality of networks allocates transmission capacity using a demand allocation multiple access (DAMA) method;
defining the transmission capacity of the one or more DAMA networks to be greater than the allocated transmission capacity received from the DAMA networks;
In response to determining that the received allocated transmission capacity of one or more DAMA networks is currently exhausted but has more potential capacity that can be allocated through demonstration of a request, one 23. The method of claim 22, further comprising assigning the one or more probe data packets to the one or more output queues corresponding to the one or more DAMA networks having potential transmission capacity. maintaining a nominal time for each of the plurality of networks to have available preferred monitored characteristics;
adding the maintained nominal time to each of the one or more data packets while assigning the one or more data packets to the one or more output queues;
22. The method of claim 21, further comprising: scheduling data packets for pushing from the respective output queues based on the nominal time associated with the respective data packets. 34. The method of claim 33, further comprising adding a prioritized nominal time to the one or more data packets identified as packets for retransmission. each of the one or more data packets is associated with one or more data flows;
The allocation of the one or more data packets comprises:
estimating a data packet classification associated with each of the data packets based on a header associated with the data packets;
determining one or more transmission requirements associated with the estimated data packet classification;
22. The method of claim 21, comprising: assigning the one or more data packets to the plurality of networks that satisfy the corresponding one or more transmission requirements of the one or more data packets. The assignment of the one or more data packets to the plurality of networks maximizes one of several data flows served, maximizes transmission power, or maximizes the transmission power of the plurality of networks. 32. The method of claim 31, based on an objective function comprising one of: minimizing a change in current bandwidth allocation for the one or more DAMA networks. determining whether the one or more data packets are associated with one or more data flows that include one or more previously allocated data packets;
assigning the one or more data packets to the plurality of networks associated with the previously assigned data packets, wherein the one or more data packets are assigned to the one or more previously assigned data packets; 34. The method of claim 33, wherein the method is performed in response to the determining that the data flow is associated with one or more data flows including: 23. The method of claim 22, wherein the transmission capacity is at least partially defined by BDP, CWND, or a number of in-flight data packets. The method includes:
for each network, including a conversion between a rate measurement measured by at least one of goodput and throughput, and a volume measurement including at least one of BDP, CWND, and in-flight data. maintaining baseline monitored characteristics;
further comprising determining whether the updated monitored characteristic varies by a threshold amount from the baseline operating characteristic;
for each of the networks, in response to the updated monitored characteristic varying by the threshold amount from the baseline behavior monitored to one or more output queues of other available networks; 22. The method of claim 21, wherein the allocation of the one or more data packets is performed. The method includes:
maintaining baseline monitored characteristics for each network;
monitoring whether the updated monitored characteristic varies by a threshold amount from the baseline operating characteristic;
for each network in response to the updated monitored characteristic varying by the threshold amount from the baseline operation being monitored;
further comprising: defining the transmission capacity of each of the networks as a loss recovery capability;
22. The method of claim 21, wherein the assignment of the one or more data packets to the corresponding output queue of the network is based on the loss recovery capability. A non-transitory computer-readable medium having computer-interpretable instructions stored thereon, the computer-interpretable instructions, when executed by a processor, causing the processor to perform the method of claim 21 or claim 22. , a non-transitory computer-readable medium. A system for packet communication using multiple network connections,
in response to detecting one or more monitored packet communication events;
defining a transmission capacity of each of the plurality of network connections;
In response to determining that the input queue contains unallocated data packets available for transmission or that the updated transmission capacity of one or more of the plurality of network connections is not exhausted;
requesting one or more data packets of the unallocated data packets from the input queue;
assigning the one or more data packets to one or more corresponding output queues of the plurality of networks having available transmission capacity;
updating the transmission capacity to reflect the assigned one or more data packets, and communicating the assigned data packets from the output queue corresponding to each network of the plurality of networks to a destination device. a receiver configured to:
a receiver configured to receive the communicated assigned data packets and reconstruct the received data packets.

Images