WO2020168443A1

WO2020168443A1 - Method and apparatus for determining tcp congestion window

Info

Publication number: WO2020168443A1
Application number: PCT/CN2019/075341
Authority: WO
Inventors: Biao Yang
Original assignee: Nokia Shanghai Bell Co., Ltd.
Priority date: 2019-02-18
Filing date: 2019-02-18
Publication date: 2020-08-27
Also published as: CN113454958A

Abstract

The present invention is to provide a suitable scheme for determining TCP congestion window. According to one example embodiment, an apparatus for determining TCP congestion window is provided, wherein the apparatus comprising means for performing: obtaining measurement related data for a UE in TCP connection; determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning. With the present invention, the following advantages are achieved: less RTT on heuristic algorithm are used to find the optimal congestion window and the accuracy of recommended congestion windows is increased.

Description

METHOD AND APPARATUS FOR DETERMINING TCP CONGESTION WINDOW

FIELD OF THE INVENTION

Various example embodiments relate generally to the technology of wireless communication and, more specifically, relates to determining TCP congestion window.

DESCRIPTION OF THE RELATED ART

Transmission Control Protocol (TCP) is one of the main protocols of internet. Many mobile based internet applications such as WWW, Email, Video and FTP rely on TCP. One main feature of TCP is Congestion Control which controls the rate of data entering the network. In Congestion Control, the Congestion Window is one of the factors which defines the number of bytes can be on-the-fly in the network. TCP uses some heuristic algorithms like Slow Start or Congestion Avoidance to find an optimal value of Congestion Window. Proper setting of Congestion Window can greatly improve TCP based applications latency and goodput.

In mobile cellular network, the available bandwidth for each UE is dynamic changing as the communication channel quality is changed quickly. The available bandwidth for each UE is influenced by UE location, UE moving speed, or cell load, even more it has relationship with vendor specific RAN’s implementation like MAC scheduler. Hence, it’s hard to quickly adapt an optimal congestion windows for cellular network in-between TCP connection. This invention proposes one RAN dominated Congestion Window Control based on Reinforcement Learning to solve this issue.

In current TCP implementation, TCP uses heuristic algorithms to find the optimal Congestion Window for TCP connection. TCP starts with a conservative initial congestion window which is set to 3/10 depending on the TCP stack configuration and implementation. After TCP connection setup, Slow Start algorithm of TCP congestion control will spend long time to find the proper Congestion Window.

One problem is quite many of TCP based application through mobile network only need short TCP connection to finish the data transfer, like short video in social media, static images, gif image, or static html. It’s not reasonable to spend so many RTT on heuristic algorithm to find the optimal Congestion Window, then increases the latency of end user application. Another problem is radio channel quality varying quickly. For cellular network in between TCP connection, RAN network is usually the bottleneck for the bandwidth, which is unstable and then has big negative impact to TCP performance.

SUMMARY OF THE INVENTION

The present invention is to provide a suitable scheme for determining TCP congestion window so as to overcome the defect in the prior art.

According to one example embodiment, an apparatus for determining TCP congestion window is provided, wherein the apparatus comprising means for performing:

obtaining measurement related data for a UE in TCP connection;

determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.

According to one example embodiment, a method for determining TCP congestion window is provided, wherein said method comprising:

obtaining measurement related data for a UE in TCP connection;

According to one example embodiment, a computer program product is provided, where the computer program comprising a non-transitory computer-readable medium storing computer program code thereon which when executed by a device causes the device to perform at least: obtaining measurement related data for a UE in TCP connection; determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.

With the present invention, the following advantages are achieved: By determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning, less RTT on heuristic algorithm are used to find the optimal congestion window, thereby decreases the latency of end user application and increases the throughput for end user application. Also, the accuracy of recommended congestion windows is increased by determining the recommended congestion windows using reinforcement learning, especially for cellular network in-between TCP connection.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, purposes and advantages of the invention will become more explicit by means of reading the detailed statement of the non-restrictive embodiments made with reference to the accompanying drawings.

FIG. 1 shows an exemplary flow chart of a method for determining TCP congestion window according to an embodiment;

FIG. 2 shows the schematic diagram of an exemplary network environment according to an embodiment.

FIG. 3 shows the schematic diagram of apparatus for determining TCP congestion window according to an embodiment;

FIG. 4 shows a high-level block diagram of the apparatus for determining TCP congestion window according to an embodiment.

The same or similar reference signs in the drawings represent the same or similar component parts.

.

DETAILED DESCRIPTION OF THE INVENTION

Further description of this invention would be given as follow by reference of the drawings.

Definition of Technical Terms:

The word "exemplary" is used herein to mean "serving as an example, instance, or illustration. "

Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the disclosure and not to limit the scope of the disclosure.

In some example embodiment, the method according to this disclosure is implemented by the apparatus.

Herein, the apparatus could be a base station, or could be a component or device that is able to implement all the steps of the according method, which could be included in a base station or other equipment with equivalent or similar functions.

Preferably, the apparatus is in a Radio Intelligence Controller (RIC) .

FIG. 1 shows an exemplary flow chart of a method for sharing control plane capacity according to an embodiment.

The method according to an exemplary including steps S101 and S102.

As shown in FIG. 1, in step 101, the apparatus obtains measurement related data for a UE in TCP connection.

The measurement related data herein includes cell measurement data and/or UE measurement data.

The cell measurement data includes all kinds of data related to cell measurement, such as the number of UEs in the cell, cell radio resource utilization etc. The UE measurement data includes all kinds of data related to UE measurement, such as UE CQI, UE throughput etc.

Specifically, the apparatus obtains the measurement related data from RAN.

In step 102, the apparatus determines the recommended congestion windows for the UE based on the measurement related data, using a reinforcement learning model.

Wherein the reinforcement learning model is used to estimate optimal TCP congestion windows for each UE. The input into the reinforcement learning model is the measurement related data, and the output is the optimal congestion windows for the UE.

There are two examples to show how apparatus determines the recommended congestion windows.

Example 1:

The cell measurement data includes UE measurement data. In step S102, the apparatus determines the recommended congestion windows for the UE based on the UE measurement data, using a reinforcement learning model.

Example 2:

The cell measurement data includes UE measurement data and cell measurement data. In step S102, the apparatus determines the recommended congestion windows for the UE based on the cell measurement data and UE measurement data, using a reinforcement learning model.

In one exemplary embodiment, the method according to the exemplary embodiment further includes step S103.

In step 103, during TCP setup period, the apparatus determines the initial recommended congestion windows for the UE based on the measurement related data, using a reinforcement learning model.

The way of determining the initial recommended congestion windows are similar as step102 above, which will not go into any more details.

In one exemplary embodiment, the method according to the exemplary embodiment further includes step S104.

In step 104, the apparatus establishes and trains a reinforcement learning model for the UE based on measurement related data.

The reinforcement learning model adopts reward function to minimize total transmission time for the session and/or total transmitted data for the session.

In one example, the apparatus establishes and trains a reinforcement learning model for the UE. The reinforcement learning model uses deep neural network as the policy network. The measurement related data further includes total transmission time for the session and total transmitted data for the session. The input into the reinforcement learning model includes Cell load, UE numbers, other cell measurement data, CQI, SINR, RTT, UE throughput, other UE measurement data. The output of the reinforcement learning model is the optimal congestion windows for the UE.

Reward Function is defined in the reinforcement learning model, as below:

Reward = f (T_session, D_session)

where T_session is the total transmission time for the session, and D_session is the total transmitted data for the session. The target of the Reward Function is to minimize T_session and D_session.

As those skilled in the art would realize, various types of reinforcement learning model are applicable to the present invention, those skilled in the art may choose the appropriate reinforcement learning model according practical needs.

Alternatively, the apparatus establishes and trains a reinforcement learning model for each UE.

Alternatively, the apparatus establishes and trains a reinforcement learning model for each cell. Then the UEs from one cell share the same reinforcement learning model.

In one exemplary embodiment, the method according to the exemplary embodiment further includes step S105.

In step 105, the apparatus sends the information of the recommended congestion windows to one or more other network entities.

Preferably, the apparatus sends the information of the recommended congestion windows to MEC (Mobile Edge Computing) .

Alternatively, the apparatus generates and updates a recommended congestion windows table according to the information of the recommended congestion windows.

According to the embodiments of the present disclosure, by determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning, less RTT on heuristic algorithm are used to find the optimal congestion window, thereby decreases the latency of end user application and increases the throughput for end user application. Also, the accuracy of recommended congestion windows is increased by determining the recommended congestion windows using reinforcement learning, especially for cellular network in-between TCP connection.

FIG. 2 shows the schematic diagram of an exemplary network environment according to an embodiment. As shown by FIG. 2, the network environment includes MEC 201, RIC 202, RAN 203, TCP Server 204 and UE 205.

The apparatus is in RIC 202. The apparatus obtains the measurement related data from RAN 203. When training the reinforcement learning model for the UE 205, the apparatus obtains total transmission time for the session and total transmitted data for the session from MEC 201 .

MEC 201 collects measurement related data, such as total transmission time and/or total transmitted data and provides the measurement related data to RIC 202. And MEC 201 receives the information of the recommended congestion windows from RIC202, and then sends it to TCP server 204 via ACK packet.

During TCP setup period, MEC 201 performs actions as below:

Catching and decoding the first ACK packet from UE upstreaming during three handshake of TCP connection setup phase;

Constructing TCP options into ACK header;

Getting the recommended initial congestion windows for the session from RIC 202;

Inserting the recommended congestion windows into the TCP options;

Sending the modified ACK packet to TCP server 204;

Storing timestamp of this ACK packet for this session.

During TCP connection is alive, MEC 201 performs actions as below:

Decoding the URI information from http header;

Collecting total transmission time for the session and total transmitted data for the session;

Getting the recommended congestion windows for the session from RIC 202;

Inserting the recommended congestion windows into the TCP options;

Sending the modified ACK packet to TCP server 204;

FIG. 3 shows the schematic diagram of apparatus for determining TCP congestion window according to an embodiment.

As shown in FIG. 3, means 301 (called “obtaining means” ) and means 302 (called “determining means” ) are included in the apparatus.

The obtaining means obtains measurement related data for a UE in TCP connection.

Specifically, the obtaining means obtains the measurement related data from RAN.

The determining means determines the recommended congestion windows for the UE based on the measurement related data, using a reinforcement learning model.

There are two examples to show how determining means determines the recommended congestion windows.

Example 1:

The cell measurement data includes UE measurement data., the determining means determines the recommended congestion windows for the UE based on the UE measurement data, using a reinforcement learning model.

Example 2:

The cell measurement data includes UE measurement data and cell measurement data, the determining means determines the recommended congestion windows for the UE based on the cell measurement data and UE measurement data, using a reinforcement learning model.

In one exemplary embodiment, the apparatus according to the exemplary further includes means 303 (called “initial determining means” ) .

During TCP setup period, the initial determining means determines the initial recommended congestion windows for the UE based on the measurement related data, using a reinforcement learning model.

The way of determining the initial recommended congestion windows are similar as determining means above, which will not go into any more details.

In one exemplary embodiment, the apparatus according to the exemplary further includes means 304 (called “training means” ) .

The training means establishes and trains a reinforcement learning model for the UE based on measurement related data.

Alternatively, the training means establishes and trains a reinforcement learning model for each UE.

Alternatively, the training means establishes and trains a reinforcement learning model for each cell. Then the UEs from one cell share the same reinforcement learning model.

In one exemplary embodiment, the apparatus according to the exemplary further includes means 305 (called “sending means” ) .

The sending means sends the information of the recommended congestion windows to one or more other network entities.

Preferably, the sending means sends the information of the recommended congestion windows to MEC (Mobile Edge Computing) .

Wherein the first apparatus comprises at least one processor 401; and at least one memory 402 including computer program code. The at least one memory 402 and the computer program code configured to, with the at least one processor 401, cause the apparatus to perform operation of at least the following: obtaining measurement related data for a UE in TCP connection; determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.

The operations of the apparatus are similar with the steps that have been described above and will not be repeated herein.

Also, a computer program product is disclosed. The computer program product comprising a non-transitory computer-readable medium storing computer program code thereon which when executed by a device causes the device to perform at least: selecting multiple data nodes to receive the control plane signaling message of incoming UEs; sending the control plane signaling message from control node to the selected data nodes.

It needs to note that the present invention can be implemented in software and/or a combination of software and hardware, for example, the invention can be implemented by using an Application Specific Integrated Circuit (ASIC) , a general purpose computer or any other similar hardware equipment. In one embodiment, the software program of this invention can be executed by a processor to accomplish the aforesaid steps or functions. Likewise, the software program (including the relevant data structure) of the invention can be stored in a computer readable recording medium, for example, RAM memory, magneto-optical drive or floppy disk and similar devices. In addition, some steps or functions of the invention can be realized by using hardware, for example, a circuit that cooperates with the processor to perform various kind of steps or functions.

In addition, part of the invention can be applied as a computer program product, such as a computer program instruction, when the instruction is executed by the computer, the method and/or technical solution according to this invention may be called or provided through an operation of the computer. However, the program instruction for calling the method of the invention may possibly be stored in a fixed or movable recording medium, and/or be transmitted via broadcasting or other signal carrier mediums, and/or be stored in the operation memory of a computer device that is running according to said program instruction. Here, there is one device included according to an embodiment of the invention, said device comprises a memory for storing computer program instructions and a processor for executing program instructions, this device is triggered to operate the methods and/or technical solutions based on the aforesaid embodiments of the invention when the computer program instructions are executed by said processor.

To those skilled in the art, apparently the present invention is not limited to the details of the aforementioned exemplary embodiments, moreover, under the premise of not deviating from the spirit or fundamental characteristics of the invention, this invention can be accomplished in other specific forms. Therefore, the embodiments should be considered exemplary and non-restrictive no matter from which point, the scope of the invention is defined by the appended claims instead of the above description, and aims at covering the meanings of the equivalent components falling into the claims and all changes within the scope in this invention. Any reference sign in the claims shall not be deemed as limiting the concerned claims. Besides, apparently the word “comprise/include” does not exclude other components or steps, singular numbers does not exclude complex numbers, the plurality of components or means mentioned in device claims may also be accomplished by one component or means through software or hardware, the wording like first and second are only used to represent names rather than any specific order.

Claims

A method for determining TCP congestion window, wherein said method comprising:

obtaining measurement related data for a UE in TCP connection;

determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.
The method of claim 1 wherein the method further comprising:

establishing and training a reinforcement learning model for the UE based on measurement related data.
The method of claim 1 or 2 wherein the method further comprising:

during TCP setup period, determining the initial recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.
The method of any proceeding claim wherein the measurement related data includes cell measurement data and UE measurement data.
The method of any proceeding claim wherein the method further comprising:

sending the information of the recommended congestion windows to one or more other network entities.
An apparatus for determining TCP congestion window, wherein the apparatus comprising means for performing:

obtaining measurement related data for a UE in TCP connection;

determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.
The apparatus of claim 6 wherein the means are further configured to perform:

establishing and training a reinforcement learning model for the UE based on measurement related data.
The apparatus of any proceeding wherein the means are further configured to perform:

during TCP setup period, determining the initial recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.
The apparatus of any proceeding wherein the measurement related data includes cell measurement data and UE measurement data.
The apparatus of any proceeding wherein the means are further configured to perform:

sending the information of the recommended congestion windows to one or more other network entities.
The apparatus of any proceeding claim wherein the apparatus is in a radio intelligence controller (RIC) .
The apparatus of any proceeding claim wherein the means comprises:

at least one processor; and

at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the performance of the apparatus.
A computer program product comprising a non-transitory computer-readable medium storing computer program code thereon which when executed by a device causes the device to perform at least:

obtaining measurement related data for a UE in TCP connection;

determining the recommended congestion windows for the UE based on the measurement related data, using reinforcement learning.