CN113454958A

CN113454958A - Method and apparatus for determining TCP congestion window

Info

Publication number: CN113454958A
Application number: CN201980092354.4A
Authority: CN
Inventors: 杨彪
Original assignee: Nokia Shanghai Bell Co Ltd
Current assignee: Nokia Shanghai Bell Co Ltd
Priority date: 2019-02-18
Filing date: 2019-02-18
Publication date: 2021-09-28
Also published as: WO2020168443A1

Abstract

The present invention will provide an appropriate scheme for determining the TCP congestion window. According to an exemplary embodiment, an apparatus for determining a TCP congestion window is provided, wherein the apparatus comprises means for: obtaining measurement related data for a UE in a TCP connection; a recommended congestion window is determined for the UE using reinforcement learning based on the measurement related data. The invention has the following advantages: fewer RTTs are used on the heuristic algorithm to find the optimal congestion window and the accuracy of the recommended congestion window is improved.

Description

Method and apparatus for determining TCP congestion window

Technical Field

Various exemplary embodiments relate generally to wireless communication technology and, more particularly, to determining a TCP congestion window

Background

The Transmission Control Protocol (TCP) is one of the main internet protocols. Many mobile-based internet applications rely on TCP, such as WWW, email, video, and FTP. One of the main features of TCP is congestion control that controls the rate of data entering the network. In congestion control, the congestion window is one of the factors that defines the number of bytes that can be in operation in the network. TCP uses some heuristic algorithms to find the optimal value of the congestion window, such as slow start or congestion avoidance. Proper setting of the congestion window can greatly improve TCP-based application latency and goodput.

In mobile cellular networks, the available bandwidth for each UE is dynamically changing, since the communication channel quality is changing rapidly. The available bandwidth for each UE is affected by the UE location, UE movement speed or cell load, and even has a relation to vendor specific RAN implementations, such as MAC schedulers. Therefore, it is difficult to quickly adapt the optimal congestion window for a cellular network between TCP connections. The present invention proposes a RAN-dominated congestion window control based on reinforcement learning to solve this problem.

In current TCP implementations, TCP uses a heuristic algorithm to find the optimal congestion window for the TCP connection. Depending on the TCP protocol stack configuration and implementation, TCP starts with a conservative initial congestion window set to 3/10. After TCP connection setup, the slow start algorithm of TCP congestion control will take longer to find the appropriate congestion window.

One problem is that many TCP-based applications over mobile networks require only a very short TCP connection to complete data transfer, such as short video, still images, gif images, or still html in social media. It is unreasonable to spend so many RTTs on the heuristic to find the optimal congestion window, thus increasing the latency of the end-user application. Another problem is that the radio signal quality can change rapidly. For cellular networks between TCP connections, the RAN network is typically a bandwidth bottleneck, and the RAN network is unstable and thus has a large negative impact on TCP performance.

Disclosure of Invention

The present invention will provide an appropriate scheme for determining the TCP congestion window, overcoming the drawbacks of the prior art.

According to an exemplary embodiment, an apparatus for determining a TCP congestion window is provided, wherein the apparatus comprises means for:

obtaining measurement related data for a UE in a TCP connection;

a recommended congestion window is determined for the UE using reinforcement learning based on the measurement related data.

According to one exemplary embodiment, a method for determining a TCP congestion window is provided, wherein the method comprises:

measurement related data is obtained for a UE in a TCP connection.

According to an example embodiment, a computer program product is provided, wherein the computer program comprises a non-transitory computer-readable medium storing computer program code, which when executed by an apparatus, causes the apparatus to perform at least the following: obtaining measurement related data for a UE in a TCP connection; a recommended congestion window is determined for the UE using reinforcement learning based on the measurement related data.

The invention has the following advantages: by using reinforcement learning to determine a recommended congestion window for the UE based on measurement related data, fewer RTTs are used on heuristic algorithms to find an optimal congestion window, thereby reducing the latency of end-user applications and increasing the throughput for end-user applications. Furthermore, the accuracy of the recommended congestion window is improved by using reinforcement learning to determine the recommended congestion window, especially for cellular networks between TCP connections.

Drawings

Other features, objects and advantages of the present invention will become more apparent from the detailed description of the non-limiting embodiments set forth below when taken in conjunction with the drawings.

Fig. 1 illustrates an exemplary flow diagram of a method for determining a TCP congestion window according to one embodiment;

FIG. 2 illustrates a schematic diagram of an exemplary network environment, according to one embodiment;

fig. 3 shows a schematic diagram of an apparatus for determining a TCP congestion window according to one embodiment;

fig. 4 depicts a high-level block diagram of an apparatus for determining a TCP congestion window in accordance with one embodiment.

The same or similar reference numerals in the drawings denote the same or similar components.

Detailed Description

Further description of the present invention will be given below with reference to the accompanying drawings.

Technical term definition:

The word "exemplary" is used herein to mean "serving as an example, instance, or illustration.

Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this detailed description section are provided to enable persons skilled in the art to make or use the exemplary embodiments of the disclosure and are not provided to limit the scope of the disclosure.

In some exemplary embodiments, the method according to the present disclosure is implemented by the apparatus.

In this context, the apparatus may be a base station or may be a component or device capable of implementing all the steps of the respective method, which may be comprised in a base station or other device with equivalent or similar functionality.

Preferably, the device is in a Radio Intelligent Controller (RIC).

Fig. 1 illustrates an exemplary flow diagram of a method for sharing control plane capacity according to one embodiment.

The method according to one embodiment comprises steps S101 and S102.

As shown in fig. 1, in step 101, the apparatus obtains measurement related data for a UE in a TCP connection.

Measurement related data in this context comprises cellular measurement data and/or UE measurement data.

The cellular measurement data comprises all kinds of data related to cellular measurements, such as the number of UEs in the cell, cellular radio resource utilization, etc. The UE measurement data includes all kinds of data related to UE measurement, such as UE CQI, UE throughput, and the like.

In particular, the apparatus obtains measurement-related data from the RAN.

In step 102, the apparatus determines a recommended congestion window for the UE using a reinforcement learning model based on the measurement related data.

Wherein a reinforcement learning model is used to estimate the optimal TCP congestion window for each UE. The input into the reinforcement learning model is the measurement related data and the output is the optimal congestion window for the UE.

There are two examples to show how the device determines the recommended congestion window.

Example 1:

the cellular measurement data comprises UE measurement data. In step S102, the apparatus determines a recommended congestion window for the UE using a reinforcement learning model based on the UE measurement data.

Example 2:

the cellular measurement data includes UE measurement data and cellular measurement data. In step S102, the apparatus determines a recommended congestion window for the UE using a reinforcement learning model based on the cellular measurement data and the UE measurement data.

In an exemplary embodiment, the method according to this exemplary embodiment further comprises step S103.

In step 103, during a TCP setup period, the apparatus determines an initial recommended congestion window for the UE using a reinforcement learning model based on measurement related data.

The manner in which the initial recommended congestion window is determined is similar to that of step 102 above and therefore will not be described in detail.

In an exemplary embodiment, the method according to this exemplary embodiment further comprises step S104.

In step 104, the apparatus builds and trains a reinforcement learning model for the UE based on the measurement-related data.

The reinforcement learning model employs a reward function to minimize the total transmission time corresponding to the session and/or the total transmission data corresponding to the session.

In one example, the apparatus builds and trains a reinforcement learning model for the UE. The reinforcement learning model uses a deep neural network as a policy network. The measurement related data further comprises a total transmission time corresponding to the session and a total transmission data corresponding to the session. Inputs into the reinforcement learning model include cell load, number of UEs, other cell measurement data, CQI, SINR, RTT, UE throughput, other UE measurement data. The output of the reinforcement learning model is the optimal congestion window for the UE.

The reward function is defined in the reinforcement learning model as follows:

reward f (T _ session, D _ session)

Where T _ session is the total transmission time corresponding to the session and D _ session is the total transmission data corresponding to the session. The goal of the bonus function is to minimize T _ session and D _ session.

Those skilled in the art will recognize that there are many types of reinforcement learning models that are suitable for use in the present invention, and those skilled in the art can select an appropriate reinforcement learning model according to actual needs.

Alternatively, the apparatus builds and trains a reinforcement learning model for each UE.

Alternatively, the apparatus builds and trains a reinforcement learning model for each UE. Then, UEs from one cell share the same reinforcement learning model.

In an exemplary embodiment, the method according to this exemplary embodiment further comprises step S105.

In step 105, the apparatus sends information recommending a congestion window to one or more other network entities.

Preferably, the apparatus sends information of the recommended congestion window to an MEC (moving edge computing).

Or the device generates and updates the recommended congestion window table according to the information of the recommended congestion window.

According to embodiments of the present disclosure, by using a reinforcement learning model to determine a recommended congestion window for a UE based on measurement related data, fewer RTTs are used on heuristic algorithms to find an optimal congestion window, thereby reducing the latency of end-user applications and increasing the throughput for end-user applications. Furthermore, the accuracy of the recommended congestion window is improved by using reinforcement learning to determine the recommended congestion window, especially for cellular networks between TCP connections.

FIG. 2 illustrates a schematic diagram of an exemplary network environment, according to one embodiment. As shown in fig. 2, the network environment includes an MEC 201, an RIC 202, a RAN 203, a TCP server 204, and a UE 205.

The device is in RIC 202. The device obtains measurement related data from the RAN 203. In training the reinforcement learning model for the UE 205, the apparatus obtains from the MEC 201 a total transmission time corresponding to the session and total transmission data corresponding to the session.

MEC 201 collects measurement related data, such as total transmission time and/or total transmission data, and provides the measurement related data to RIC 202. MEC 201 receives information recommending a congestion window from RIC 202 and then sends it to TCP server 204 via an ACK packet.

During the TCP setup period, MEC 201 performs the following actions:

capturing and decoding a first ACK packet from UP upstream during a three-way handshake of a TCP connection setup phase;

constructing the TCP options as ACK headers;

obtain a recommended initial congestion window for the session from RIC 202;

inserting the recommended congestion window into the TCP option;

sending the modified ACK packet to the TCP server 204;

the timestamp of the ACK packet is stored for the session.

During TCP connection activity, MEC 201 performs the following actions:

decoding the URI information from the http header;

collecting a total transmission time corresponding to the session and total transmission data corresponding to the session;

obtain a recommended congestion window for the session from RIC 202;

inserting the recommended congestion window into the TCP option;

the modified ACK packet is sent to the TCP server 204.

Fig. 3 shows a schematic diagram of an apparatus for determining a TCP congestion window according to one embodiment.

As shown in fig. 3, an apparatus 301 (referred to as "obtaining apparatus") and an apparatus 302 (referred to as "determining apparatus") are included in the apparatus.

The obtaining means obtains measurement-related data for the UE in the TCP connection.

In particular, the obtaining means obtains measurement related data from the RAN.

The determining means determines a recommended congestion window for the UE using a reinforcement learning model based on the measurement related data.

There are two examples to show how the determining means determines the recommended congestion window.

Example 1:

the cellular measurement data includes UE measurement data, and the determining means determines a recommended congestion window for the UE using a reinforcement learning model based on the UE measurement data.

Example 2:

the cellular measurement data includes UE measurement data and cellular measurement data, and the determining means determines a recommended congestion window for the UE using a reinforcement learning model based on the cellular measurement data and the UE measurement data.

In an exemplary embodiment, the apparatus according to this exemplary embodiment further comprises means 303 (referred to as "initial determining means").

During the TCP setup period, the initial determination means determines an initial recommended congestion window for the UE using a reinforcement learning model based on the measurement related data.

The manner in which the initial recommended congestion window is determined is similar to the previous determination means and will therefore not be described in detail.

In an exemplary embodiment, the apparatus according to this exemplary embodiment further comprises an apparatus 304 (referred to as "training apparatus").

The training apparatus builds and trains a reinforcement learning model for the UE based on the measurement-related data.

Alternatively, the training apparatus establishes and trains a reinforcement learning model for each UE.

Alternatively, the training apparatus establishes and trains a reinforcement learning model for each UE. Then, UEs from one cell share the same reinforcement learning model.

In an exemplary embodiment, the apparatus according to this exemplary embodiment further comprises an apparatus 305 (referred to as "transmitting apparatus").

The transmitting means transmits information recommending a congestion window to one or more other network entities.

Preferably, the transmitting means transmits the information of the recommended congestion window to an MEC (moving edge computing).

Wherein the first device comprises: at least one processor 401; and at least one memory 402 comprising computer program code. The at least one memory 402 and the computer program code are configured to, with the at least one processor 401, cause the apparatus to perform at least the following: obtaining measurement related data for a UE in a TCP connection; a recommended congestion window is determined for the UE using reinforcement learning based on the measurement related data.

The operation of the device is similar to the steps described above and will not be repeated here.

A computer program product is also disclosed. The computer program product includes a non-transitory computer-readable medium storing computer program code, which, when executed by an apparatus, causes the apparatus to perform at least the following: selecting a plurality of data nodes to receive control plane signaling messages for an upcoming UE; a control plane signaling message is sent from the control node to the selected data node.

It should be noted that the present invention may be implemented in software and/or in a combination of software and hardware, for example, the present invention may be implemented using Application Specific Integrated Circuits (ASICs), general purpose computers or any other similar hardware devices. In one embodiment, the software program of the present invention may be executed by a processor to perform the aforementioned steps or functions. Also, the software programs (including associated data structures) of the present invention can be stored in a computer-readable recording medium, such as a RAM memory, a magneto-optical drive or floppy disk, and the like. Further, some of the steps or functions of the present invention may be implemented by using hardware such as a circuit which implements various steps or functions in cooperation with a processor.

Furthermore, portions of the present invention may be applied as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or technical solutions according to the present invention through the operation of the computer. Program instructions for invoking the inventive methods may be stored in a fixed or removable recording medium and/or transmitted via a broadcast or other signal bearing medium and/or stored in an operating memory of a computer device executing according to the program instructions. An embodiment according to the invention herein comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, which apparatus is triggered to operate a method and/or technical solution according to the aforementioned embodiments of the invention when the computer program instructions are executed by the processor.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. It will furthermore be obvious that the word "comprising" does not exclude other elements or steps, that the singular does not exclude the plural, and that a plurality of elements or means recited in a device claim may also be implemented by one element or means in software or hardware, and that the words such as "first" and "second" are used merely to indicate a title and not any particular order.

Claims

1. A method for determining a TCP congestion window, wherein the method comprises:

obtaining measurement related data for a UE in a TCP connection;

2. The method of claim 1, wherein the method further comprises:

a reinforcement learning model is built and trained for the UE based on the measurement-related data.

3. The method of claim 1 or 2, wherein the method further comprises:

during the TCP setup period, an initial recommended congestion window is determined for the UE using reinforcement learning based on measurement related data.

4. The method according to any of the preceding claims, wherein measurement related data comprises cellular measurement data and UE measurement data.

5. The method of any preceding claim, wherein the method further comprises:

information recommending a congestion window is sent to one or more other network entities.

6. An apparatus for determining a TCP congestion window, wherein the apparatus comprises means for:

obtaining measurement related data for a UE in a TCP connection;

7. The apparatus of claim 6, wherein the apparatus is further configured to:

8. The apparatus of any preceding claim, wherein the apparatus is further configured to:

9. The apparatus of any preceding claim, wherein measurement related data comprises cellular measurement data and UE measurement data.

10. The apparatus of any preceding claim, wherein the apparatus is further configured to:

11. The apparatus of any preceding claim, wherein the apparatus is in a Radio Intelligent Controller (RIC).

12. The apparatus of any preceding claim, wherein the apparatus comprises:

at least one processor; and

at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform operations.

13. A computer program product comprising a non-transitory computer readable medium having computer program code stored therein, the computer program code, when executed by an apparatus, causing the apparatus to perform at least the following:

obtaining measurement related data for a UE in a TCP connection;