WO2021189957A1 - Data transmission method, data transmission system, device and storage medium - Google Patents

Abstract

Provided are a data transmission method, a data transmission system, a device and a storage medium. The method comprises: querying a historical transmission record, and acquiring a historical transmission parameter; determining a current transmission parameter according to the historical transmission parameter; performing, according to the current transmission parameter, initial fragmentation on data to be transmitted and then transmitting same; determining whether the current transmission state is normal; and if the current transmission state is normal, adjusting the current transmission parameter according to a first preset rule, and performing, according to the adjusted current transmission parameter, secondary fragmentation on the remaining data to be transmitted and then transmitting same. A current transmission parameter is determined according to a historical transmission parameter, and data to be transmitted is fragmented according to the current transmission parameter and is then transmitted, such that too many fragmentation and transmission request attempts caused by a solidified fragmentation manner can be prevented from occurring, thereby reducing time loss. By determining the current transmission state, the current transmission parameter is dynamically adjusted, such that the network bandwidth can be utilized to the maximum extent, the transmission time is reduced, the transmission efficiency is improved, the bandwidth utilization rate is improved, and the transmission delay is reduced.

Description

A data transmission method, data transmission system, equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on October 14, 2020, the application number is 202011099350.X, and the invention title is "a data transmission method, data transmission system, equipment, and storage medium". The entire content is incorporated into this application by reference.

Technical field

This application belongs to the field of data processing, and specifically relates to a data transmission method, data transmission system, equipment, and storage medium.

Background technique

With the advent of the era of big data and 5G communications, the amount of data generated in various fields of production and life has increased exponentially. With the continuous support of large-capacity storage devices and technologies, the file capacity of these data is also increasing day by day, and the transmission of large-capacity data files has gradually become a rigid demand in various fields. In some specific fields and special scenarios, large-capacity data files still have to be transmitted over the network. Network data transmission is an extremely important part of the field of information technology. However, most of the current big data processing technologies tend to be mobile computing rather than data transmission. For the unavoidable large-capacity data transmission, there is no high-availability and stable transmission. Strategy.

The inventor realizes that the transmission speed and stability of large-capacity data transmission in high-speed and stable network scenarios, such as the same local area network or the same data center, are acceptable. In addition to taking a long time, such scenarios , The possibility of serious problems such as transmission failure is small. However, in some scenarios where the transmission bandwidth is relatively narrow or the bandwidth is unstable, such as large-capacity data transmission across LANs or different data centers, if special technical solutions and transmission strategies are not used, there is a high probability that transmission will be interrupted or time-consuming. The timeout is too long and the transfer cannot be completed eventually.

In view of this situation, most of the existing technical solutions currently adopt a fragmented transmission strategy based on resumable transmission, that is, large-capacity data files are fragmented, and when a fragmented transmission fails, restart from the breakpoint. The transmission of the fragments, thereby avoiding retransmission of the entire file. This method of purely using fragmented transmission, although it can complete the transmission, cannot effectively use the network bandwidth, especially in some cases where the bandwidth is unstable, the existing fragmented transmission method cannot make the most of the network bandwidth, resulting in Excessive time loss during transmission and low bandwidth utilization.

Summary of the invention

The embodiment of the first aspect of the present application proposes a data transmission method, including:

Query historical transmission records and obtain historical transmission parameters;

Determine current transmission parameters according to the historical transmission parameters;

According to the current transmission parameters, the data to be transmitted is transmitted after initial fragmentation;

Determine whether the current transmission status is normal;

If it is normal, the current transmission parameters are adjusted according to the first preset rule, and the remaining data to be transmitted is transmitted after being fragmented twice according to the adjusted current transmission parameters.

The embodiment of the second aspect of the present application provides a data transmission system, including:

The query module is used to query historical transmission records and obtain historical transmission parameters;

The judgment calculation module is configured to determine the current transmission parameters according to the historical transmission parameters; determine whether the current transmission status is normal, and if it is normal, adjust the current transmission parameters according to the first preset rule;

The transmission module is configured to perform initial fragmentation and transmission of the data to be transmitted according to the current transmission parameters; and perform second fragmentation and transmission of the remaining data to be transmitted according to the adjusted current transmission parameters.

The embodiment of the third aspect of the present application provides a computer device, including a memory and a processor, and computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processing The device executes the following steps of the data transmission method described in the first aspect:

Query historical transmission records and obtain historical transmission parameters;

Determine current transmission parameters according to the historical transmission parameters;

According to the current transmission parameters, the data to be transmitted is transmitted after initial fragmentation;

Determine whether the current transmission status is normal;

If it is normal, the current transmission parameters are adjusted according to the first preset rule, and the remaining data to be transmitted is transmitted after being fragmented twice according to the adjusted current transmission parameters.

The embodiment of the fourth aspect of the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the following first The steps of the data transmission method described in the aspect:

Query historical transmission records and obtain historical transmission parameters;

Determine current transmission parameters according to the historical transmission parameters;

According to the current transmission parameters, the data to be transmitted is transmitted after initial fragmentation;

Determine whether the current transmission status is normal;

If it is normal, the current transmission parameters are adjusted according to the first preset rule, and the remaining data to be transmitted is transmitted after being fragmented twice according to the adjusted current transmission parameters.

Description of the drawings

By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only used for the purpose of illustrating the preferred embodiments, and are not considered as a limitation to the application. Also, throughout the drawings, the same reference symbols are used to denote the same components. In the attached picture:

FIG. 1 shows a schematic diagram of the steps of a data transmission method provided by an embodiment of the present application;

FIG. 2 shows a schematic flowchart of a data transmission method provided by an embodiment of the present application;

FIG. 3 shows a schematic structural diagram of a data transmission system provided by an embodiment of the present application;

FIG. 4 shows a schematic structural diagram of a computer device provided by an embodiment of the present application;

FIG. 5 shows a schematic diagram of a storage medium provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions, and advantages of this application clearer and clearer, the following further describes the application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

It can be understood that the terms "first", "second", etc. used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

In one embodiment, a data transmission method is proposed, which can determine the current transmission parameters based on historical transmission parameters, and then perform fragmentation and transmission of the data to be transmitted according to the current transmission parameters, thereby avoiding the traditional simple fragmentation method. Direct fragmentation regardless of the current bandwidth, resulting in excessive fragmentation transmission request attempts and increased time loss; by judging the current transmission status in real time, dynamically adjusting the current transmission parameters and fragment size, so as to maximize the use of network bandwidth , Reduce transmission time, improve transmission efficiency, increase bandwidth utilization, and reduce transmission delay.

Data transmission through the operation process shown in Figure 1 specifically includes the following steps:

Step 101: Query historical transmission records and obtain historical transmission parameters.

As shown in Figure 2, when the client is preparing to transfer large-capacity data files across a local area network or across data centers, the client needs to initiate a large-capacity data file transfer request first, query the database in the client, and perform data from the previous The historical transmission parameters corresponding to the file receiving end (the server receiving the file) are obtained from the historical transmission record saved during transmission. The historical transmission parameters are the historical transmission parameters of the large-capacity data file transmission between the client that is going to send the large-capacity data file transmission and the server that will receive the file before this, including the fragment size, the number of transmission threads, Parameters related to data transmission and bandwidth, such as response time and number of retries.

Step 102: Determine current transmission parameters according to historical transmission parameters.

From the historical transmission parameters, one or more sets of historical parameter vectors are obtained, where the historical parameter vectors are vectors composed of related historical parameters in different time periods in the database. The parameter vector includes parameters related to data transmission and bandwidth, such as the size of the fragment, the number of transmission threads, the response time, and the number of retries. According to the obtained one or more sets of historical parameter vectors, the weighted average smoothing formula is used to calculate the current transmission parameters. The weighted average smoothing formula is:

in,

Is the current transmission parameter, i is the group number of the historical parameter vector,

Is the historical parameter vector, a _i is the weighted component, usually, the closer the time is, the higher the weight is, n is a positive integer, t is the time, and t ₀ is the starting point of the query period corresponding to the current transmission time, namely It is a transmission time period corresponding to the current transmission time. The transmission time period can be 30 minutes, 1 hour, or 2 hours. T ₀ is the interval period of the query time period, usually 24 hours as an interval period. Assuming that the time when the client initiates a large-capacity data file transfer request is at 13:35 on August 18, 2020, with 1 hour as the transmission time period, it corresponds to the current transmission time (13:35 on August 18, 2020) The start point of the query time period is 13:00, and the end point of the query time period is 14:0. That is, query today (18th) 13:00 to 14:00, yesterday (17th) 13:00 to 14:00, the day before yesterday (16th) 13:00 to 14:00 and before From 13:0 to 14:0 of the date of the historical transmission parameters, one or more sets of historical parameter vectors including the fragment size, the number of transmission threads, the response time, and the number of retries are obtained from these historical transmission parameters. Suppose that only the historical parameter vectors of the 17th and 16th are obtained, that is, the number of groups of the historical parameter vector is 2, because the historical parameter vector

Including the shard size Mt, the number of threads Pt, the response time Rt and the number of retries Ct, so the weighted average smoothing formula is used and the shard size Mt, the number of threads Pt, and the response time Rt in the historical parameter vector on the 17th and 16th And the number of retries Ct, determine the fragment size m, the number of threads p, the response time r, and the number of retries c in the previous transmission parameters.

Take the fragment size m as an example,

in,

Is the 17-day slice size in the historical parameter vector,

Is the 16-day slice size in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Take the number of threads p as an example,

in,

Is the number of threads on the 17th in the historical parameter vector,

Is the number of threads on the 16th in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Take the response time r as an example,

in,

Is the response time of 17 days in the historical parameter vector,

Is the 16-day response time in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Take the number of retries c as an example,

in,

Is the number of retries on the 17th day in the historical parameter vector,

Is the number of retries for 16 days in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Other parameters related to data transmission and bandwidth can also be calculated by the above formula.

Step 103: According to the current transmission parameters, the data to be transmitted is transmitted after initial fragmentation.

According to the fragment size m previously calculated by the client, the large-capacity data file to be transmitted is fragmented to obtain fragmented data, and the transmission parameters are set according to the number of threads p, response time r, and number of retries c calculated before , To transmit the fragmented data of large-capacity data files to the server. Among them, before judging the current transmission status, the fragmentation of the data to be transmitted is the initial fragmentation.

If the historical transmission parameter corresponding to the current transmission time is not found in the historical transmission record, the preset transmission parameter is used directly as the current transmission parameter. That is, the client cannot obtain the historical transmission parameters corresponding to the server receiving the file from the historical transmission records, and uses the transmission parameters with the preset values to fragment the large-capacity data file to be transmitted, set the transmission parameters, and transmit Fragmentation of large-capacity data files to the server.

Step 104: Determine whether the current transmission status is normal.

Determine whether there is an abnormal situation of timeout error during the transmission of the fragmented data of the data to be transmitted. Specifically, it is determined whether the fragmented transmission of the transmitted data is normal, and whether an abnormal situation of timeout occurs. Among them, the abnormal conditions of timeout include: whether a timeout error occurs, whether the number of retries has been exceeded, whether the response time has been exceeded, etc.

Step 105: If it is normal, adjust the current transmission parameters according to the first preset rule, and perform the second fragmentation and transmission of the remaining data to be transmitted according to the current transmission parameters adjusted according to the first preset rule. Increase the fragment size and the number of transmission threads in the current transmission parameters according to the first preset rule, and use the increased fragment size and the number of transmission threads as the current transmission parameters, and perform second fragmentation and transmission of the remaining data to be transmitted . Among them, the remaining data to be transmitted is fragmented according to the adjusted current transmission parameters, which is the second fragmentation.

Wherein, after the remaining data to be transmitted is fragmented and transmitted for the second time according to the adjusted current transmission parameters, the judgment of step 104 is continued again, and step 105 is selected again according to whether the transmission status is normal, and so on. .

Specifically, increase the fragment size and the number of transmission threads in the current transmission parameters according to the first preset rule, including: if the transmission status is normal, no timeout abnormality occurs, that is, no timeout error occurs, and the number of retries is not exceeded. If the response time and other parameters are not exceeded, the current transmission parameters are adjusted according to the increment ratio and increment value preset in the first preset rule. It is also possible to use only one of no timeout error, exceeding the number of retries, exceeding the response time, or other conditions as the criterion for judging that the state is normal.

In addition to increasing the fragment size and the number of transmission threads in the current transmission parameters, it also includes adjusting (such as increasing or decreasing) the response time and the number of retries. Whether the response time and the number of retries are adjusted, and whether to increase or decrease, can be specifically set as required.

Take, for example, the adjustment of increasing fragment size, response time, and number of retries.

The increment ratios of the fragment size, response time, and the number of retries can be 5%, 10%, 15%, 20%, etc., and corresponding increment ratios can also be set for the fragment size, response time, and number of retries. Taking the increment ratio of the fragment size, response time and the number of retries as an example, the fragment size m is adjusted to (1+0.1)m, and the response time r is adjusted to (1+ 0.1)r, the number of retries c is adjusted to (1+0.1)c. The incremental value of the number of threads, response time, and number of retries can be increased by a fixed value. Assuming that the increment value of the number of threads is 1, the increment value of the response time is 5 milliseconds (ms), and the increment value of the number of retries is 5, the adjusted number of threads is p+1, and the response time is r+5 Milliseconds, the number of retries is c+5.

According to the current transmission parameters after incremental adjustment, the remaining data to be transmitted that has not yet been transmitted are fragmented and sent correspondingly. When the segment is sent, its current transmission parameter is the current transmission parameter after adjustment. That is, use the adjusted number of threads to be p+1, the response time to be r+5 milliseconds, and the number of retries to be c+5 to transmit a fragment with a fragment size of (1+0.1)m.

After incrementing (increasing the fragment size and the number of transmission threads in the current transmission parameters), it continues to determine whether the current transmission status is normal. If the transmission status is normal, continue to increase the segment size, number of transmission threads, response time, and number of retries in the current transmission parameters according to the first preset rule, and adjust the segment size, number of transmission threads, and response according to the increment. Time and the number of retries, and perform corresponding secondary fragmentation and transmission of the remaining data to be transmitted that has not yet been transmitted. If there is no abnormal transmission state until the data transmission is completed, the current transmission parameters when the data transmission is completed are saved. If there is an abnormal transmission status, save the current transmission parameters to the historical transmission record, use the current transmission parameters to continue to transmit the remaining data to be transmitted, or after decrementing the current transmission parameters, save the current decrement adjustment Transmission parameters, and use the current transmission parameters after decrement adjustment to perform secondary segmentation and transmission of the remaining data to be transmitted.

In step 104, after judging whether the current transmission status is normal, the method further includes:

If it is abnormal, save the current transmission parameters to the historical transmission record, and use the current transmission parameters to continue to transmit the remaining data to be transmitted.

The above steps are to determine whether the transmission status is normal according to whether the transmission of the transmitted data fragments is normal, whether there is a timeout error, whether the number of retries is exceeded, whether the response time is exceeded, etc., to determine whether the transmission status is normal. If the parameters such as the number of times and response time appear and exceed, or one of them appears or exceeds (that is, to determine whether the fragmented transmission of the transmitted data is normal, it can also only be caused by a timeout error, exceeding the number of retries, exceeding the response time, or other conditions If one of the items is used as the criterion for abnormal status), then continue to use the current transmission parameters to transmit the fragmented data, and no further adjustments are made until the transmission is completed, and the current transmission parameters are saved.

If an abnormal situation occurs, you can also choose one of the following two decrement methods for subsequent transmission and save the adjusted parameters. Two reduction methods include: using the current transmission parameters before the last current parameter adjustment as the adjusted current transmission parameters, or adjusting the current transmission parameters, reducing the fragment size and the number of transmission threads in the current transmission parameters as the adjusted current transmission parameters Current transmission parameters. Whether the response time and the number of retries are adjusted, and whether to increase or decrease, can be specifically set as required.

Among them, use the current transmission parameters before the last adjustment of the current parameters as the adjusted current transmission parameters, which means that if the current transmission parameters are adjusted transmission parameters, use the transmission parameters before the current transmission parameter adjustments as the adjusted current transmission parameters. Current transmission parameters, data transmission. That is, assuming that the current transmission parameters are: the fragment size is (1+0.1)m, the response time is (1+0.1)r, the number of retries c is (1+0.1)c, and the number of threads is p+1; The transmission parameters before the adjustment of the current transmission parameters are: the fragment size is m, the response time is r, the number of retries is c, and the number of threads is p; use these parameters as the adjusted transmission parameters, transmit data, and save these parameters, No further adjustments will be made afterwards until the data transfer is complete.

Adjust the current transmission parameters, reduce the fragment size and the number of transmission threads in the current transmission parameters as the adjusted current transmission parameters, which means that according to the preset decrement ratio and decrement value, adjust the current transmission parameters to reduce the score Chip size, reduced response time, number of retries, and number of threads. The reduction ratio can be 5%, 10%, 15%, 20%, and so on. The decrement value can be a fixed value corresponding to the reduction in response time, the number of retries, and the number of threads. After decrementing the current transmission parameters, use and save the decremented current transmission parameters to transmit data, and then do not adjust until the data transmission is completed.

After the adjustments are made according to the above two reduction methods, the historical transmission records of the adjusted current transmission parameters are saved to the client for subsequent data transmission, and the current transmission parameters are no longer adjusted. The saved adjusted current transmission parameters and the current transmission parameters that have not been adjusted all correspond to the transmission time period.

That is, when the transmission status is abnormal, there are three alternative methods.

According to the current transmission parameters after the decrement adjustment, the remaining to-be-transmitted data that has not yet been transmitted is subjected to corresponding secondary fragmentation and transmission. According to one of the above three methods selected, when the segment is sent, its current transmission parameter is the current transmission parameter after adjustment or the current transmission parameter without adjustment.

The embodiment of the application utilizes the time period characteristics of the transmission network, determines the current transmission parameters through historical transmission parameters according to the universal characteristics of the transmission network, and then transmits the data to be transmitted in fragments according to the current transmission parameters, thereby avoiding the traditional The naive fragmentation method directly fragments without considering the current bandwidth, resulting in excessive fragmentation transmission request attempts and increased time loss, which makes up for the low transmission efficiency and bandwidth utilization of the existing fragmentation transmission scheme; By judging the current transmission status, quantifying and analyzing the real-time parameters of the network, optimizing the fragmentation transmission strategy, dynamically adjusting the current transmission parameters and fragment size, and making maximum use in the large-capacity data transmission scenario across the local area network or across the data center Network bandwidth, thereby reducing transmission time, improving transmission efficiency, increasing bandwidth utilization, reducing transmission delay, and optimizing fragmentation transmission strategy, network transmission efficiency and bandwidth utilization, thereby improving the robustness of the transmission scheme and ensuring high connection availability , And has the universal characteristics of transmission network in the scene of large-capacity data transmission across local area networks or across data centers.

Most transmission networks have tidal characteristics, that is, have similar periodic transmission performance in a fixed time period. The reason is that the network load has a great correlation with the user's work and rest and usage habits. For example, transoceanic networks usually reach the peak of transmission around 12 noon. Aiming at this network feature, using the historical transmission parameters of the same period of time to optimize the strategy in the fragmented transmission process can improve the transmission success rate on the one hand, and improve the efficiency of network transmission on the other hand.

Based on the historical transmission parameters in the historical transmission records, when large-capacity data files are transmitted, the periodical characteristics of network transmission are used to perform dynamic and real-time intelligent fragmentation transmission of large-capacity data files, effectively using network bandwidth. By recording and reading the historical transmission records of the current time period, the fragmented transmission strategy of the current time period is optimized. At the same time, the current transmission parameters are adjusted in a small range until the error retry rate increases, thereby further optimizing the fragmentation strategy on the basis of the optimal transmission strategy (parameter). By optimizing the fragmented transmission strategy, not only the transmission success rate of large-capacity data files is improved, but also the transmission efficiency of the network is improved.

As shown in FIG. 3, in one embodiment, a data transmission system is provided, including:

The query module 301 is used to query historical transmission records and obtain historical transmission parameters;

The judgment calculation module 302 is configured to determine the current transmission parameters based on the historical transmission parameters; determine whether the current transmission status is normal, and if it is normal, adjust the current transmission parameters according to the first preset rule;

The transmission module 303 is configured to perform initial fragmentation and transmission of the data to be transmitted according to the current transmission parameters; and perform second fragmentation and transmission of the remaining data to be transmitted according to the adjusted current transmission parameters.

The query module 301 includes:

Data query unit for querying historical transmission records;

The data extraction unit is configured to obtain historical transmission parameters from the historical transmission records, and send the historical transmission parameters to the judgment calculation module 302.

When the data transmission system (client) is preparing to transmit large-capacity data files across the local area network or across the data center, the data transmission system needs to first query the database through the data query unit, and then obtain the data from the historical transmission record through the data extraction unit. The historical transmission parameters corresponding to the file receiving end (the server receiving the file). The historical transmission parameters are the historical transmission parameters for the large-capacity data file transmission between the data transmission system that is going to send the large-capacity data file transmission and the server that will receive the file before this.

The judgment calculation module 302 includes:

The parameter determination unit is configured to determine the current transmission parameters using the weighted average smoothing formula according to the historical transmission parameters, and send the current transmission parameters to the transmission module 303;

The state judgment unit is used to judge whether the current transmission state is normal, and send the judgment result of the transmission state to the parameter adjustment unit;

The parameter adjustment unit is configured to adjust the current transmission parameters according to the first preset rule according to the judgment result of the transmission state, and send the adjusted current transmission parameters to the transmission module 303.

The above parameter determination unit specifically obtains one or more sets of historical parameter vectors from historical transmission parameters, where the parameter vectors include: fragment size, number of transmission threads, response time and number of retries, etc., as well as data transmission and bandwidth Related parameters. According to the obtained one or more sets of historical parameter vectors, the weighted average smoothing formula is used to calculate the current transmission parameters. The weighted average smoothing formula is:

in,

Is the current transmission parameter, i is the group number of the historical parameter vector,

Is the historical parameter vector, a _i is the weighted component, usually, the closer the time is, the higher the weight is, n is a positive integer, t is the time, and t ₀ is the starting point of the query period corresponding to the current transmission time, namely It is a transmission time period corresponding to the current transmission time. The transmission time period can be 30 minutes, 1 hour, or 2 hours. T ₀ is the interval period of the query time period, usually 24 hours as an interval period. Assuming that the time when the client initiates a large-capacity data file transfer request is at 13:35 on August 18, 2020, with 1 hour as the transmission time period, it corresponds to the current transmission time (13:35 on August 18, 2020) The start point of the query time period is 13:0, and the end point of the query time period is 14:0. That is, query today (18th) 13:00 to 14:00, yesterday (17th) 13:00 to 14:00, the day before yesterday (16th) 13:00 to 14:00 and before From 13:0 to 14:0 of the date of the historical transmission parameters, one or more sets of historical parameter vectors including the fragment size, the number of transmission threads, the response time, and the number of retries are obtained from these historical transmission parameters. Suppose that only the historical parameter vectors of the 17th and 16th are obtained, that is, the number of groups of the historical parameter vector is 2, because the historical parameter vector

Including the shard size Mt, the number of threads Pt, the response time Rt and the number of retries Ct, so the weighted average smoothing formula is used and the shard size Mt, the number of threads Pt, and the response time Rt in the historical parameter vector on the 17th and 16th And the number of retries Ct, determine the fragment size m, the number of threads p, the response time r, and the number of retries c in the previous transmission parameters.

Take the fragment size m as an example,

in,

Is the 17-day slice size in the historical parameter vector,

Is the 16-day slice size in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Take the number of threads p as an example,

in,

Is the number of threads on the 17th in the historical parameter vector,

Is the number of threads on the 16th in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Take the response time r as an example,

in,

Is the response time of 17 days in the historical parameter vector,

Is the 16-day response time in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Take the number of retries c as an example,

in,

Is the number of retries on the 17th day in the historical parameter vector,

Is the number of retries for 16 days in the historical parameter vector, and _{the value of a 1} is greater than the value of a ₂ .

Other parameters related to data transmission and bandwidth can also be calculated by the above formula.

The above-mentioned state judgment unit specifically judges whether the current transmission state is normal and whether an abnormal situation of timeout occurs. Among them, the abnormal conditions of timeout include: whether a timeout error occurs, whether the number of retries has been exceeded, whether the response time has been exceeded, etc., to determine whether the current transmission status is normal. If there is no timeout error, the number of retries and the response time are not exceeded, the current transmission status is normal, and the current transmission status is sent to the parameter adjustment unit; One of the number of attempts, exceeding the response time, or other conditions is the criterion for judging that the state is normal. If a timeout error occurs, the number of retries is exceeded, and the response time is exceeded, the transmission status is abnormal, and the transmission status is sent to the parameter adjustment unit; the transmission status can also be judged only by the occurrence of a timeout error, exceeding the number of retries, and exceeding the response One item of time or other conditions is used as a criterion for judging abnormal conditions.

The above-mentioned parameter adjustment unit specifically adjusts the current transmission parameters according to the transmission status, and sends the adjusted current transmission parameters to the transmission module 303. Determine whether the current transmission status is normal, and if it is normal, adjust the current transmission parameters according to the first preset rule.

Determine whether there is an abnormal situation of timeout error during the transmission of the fragmented data of the data to be transmitted. If not (the transmission status is normal), adjust the current transmission parameters according to the first preset rule (incrementally adjust the current transmission parameters) , Increase the fragment size and the number of transmission threads in the current transmission parameters. Specifically, it is judged whether the fragmented transmission of the transmitted data is normal, whether a timeout error occurs, whether the number of retries is exceeded, whether the response time is exceeded, etc., if the transmission status is normal, no timeout abnormality occurs, that is, no timeout error occurs, no If the number of retries is exceeded and the response time and other parameters are not exceeded, the current transmission parameters are adjusted according to the increment ratio and increment value preset according to the first preset rule. It is also possible to use only one of no timeout error, exceeding the number of retries, exceeding the response time, or other conditions as the criterion for judging that the state is normal. The increment ratios of the fragment size, response time, and the number of retries can be 5%, 10%, 15%, 20%, etc., and corresponding increment ratios can also be set for the fragment size, response time, and number of retries. Taking the increment ratio of the fragment size, response time and the number of retries as an example, the fragment size m is adjusted to (1+0.1)m, and the response time r is adjusted to (1+ 0.1)r, the number of retries c is adjusted to (1+0.1)c. The incremental value of the number of threads, response time, and number of retries can be increased by a fixed value. Assuming that the increment value of the number of threads is 1, the increment value of the response time is 5 milliseconds (ms), and the increment value of the number of retries is 5, the adjusted number of threads is p+1, and the response time is r+5 Milliseconds, the number of retries is c+5.

In addition to increasing the fragment size and the number of transmission threads in the current transmission parameters, it also includes adjusting (such as increasing or decreasing) the response time and the number of retries. Whether the response time and the number of retries are adjusted, and whether to increase or decrease, can be specifically set as required.

If it is abnormal (the transmission status is abnormal), continue to use the current transmission parameters. That is to determine whether the fragmented transmission of the transmitted data is normal, whether there is a timeout error, whether it exceeds the number of retries, whether it exceeds the response time, etc., if the transmission status is abnormal, that is, if the parameters such as timeout error, number of retries, and response time all appear and Exceeded, or one of them appeared or exceeded (that is, to determine whether the fragmented transmission of the transmitted data is normal, or only one of the occurrence of timeout errors, exceeding the number of retries, exceeding the response time, or other conditions is the judgment of abnormal status Standard), then continue to use the current transmission parameters and no further adjustments will be made later, send the unadjusted (unchanged) current transmission parameters to the transmission module 303, and save the current transmission parameters.

You can also choose one of the following two decrement methods for subsequent transmission. Two reduction methods include: using the current transmission parameters before the last current parameter adjustment as the adjusted current transmission parameters, or adjusting the current transmission parameters, reducing the fragment size and the number of transmission threads in the current transmission parameters as the adjusted current transmission parameters Current transmission parameters.

Among them, use the current transmission parameters before the last adjustment of the current parameters as the adjusted current transmission parameters, which means that if the current transmission parameters are adjusted transmission parameters, use the transmission parameters before the current transmission parameter adjustments as the adjusted current transmission parameters. Current transmission parameters, data transmission. That is, assuming that the current transmission parameters are: the fragment size is (1+0.1)m, the response time is (1+0.1)r, the number of retries c is (1+0.1)c, and the number of threads is p+1; The transmission parameters before the current transmission parameter adjustment are: the fragment size is m, the response time is r, the number of retries is c, and the number of threads is p; these parameters are used as the adjusted transmission parameters and sent to the transmission module 303.

Adjust the current transmission parameters, reduce the fragment size and the number of transmission threads in the current transmission parameters as the adjusted current transmission parameters, which means that according to the preset decrement ratio and decrement value, adjust the current transmission parameters to reduce the score Chip size, reduced response time, number of retries, and number of threads. The reduction ratio can be 5%, 10%, 15%, 20%, and so on. The decrement value can be a fixed value corresponding to the reduction in response time, the number of retries, and the number of threads. After the current transmission parameters are decremented and adjusted, these parameters are used as the adjusted transmission parameters and sent to the transmission module 303.

The parameter adjustment unit is also used to send the unadjusted current transmission parameter or the current transmission parameter after decrement adjustment to the historical transmission record of the storage module for storage. After adjusting according to the above two reduction methods, or after continuing to use the unadjusted current transmission parameters, save the current transmission parameters (that is, reduction or unadjusted current transmission parameters) to the client's historical transmission records for later use Data transmission, and no longer any adjustments to the current transmission parameters. The saved current transmission parameters all correspond to the transmission time period.

The transmission module 303 includes:

The transmission adjustment unit is configured to perform initial fragmentation of the data to be transmitted according to the current transmission parameters; perform secondary fragmentation of the remaining data to be transmitted according to the current transmission parameters adjusted by the first preset rule;

The data transmission unit is used to transmit the data to be transmitted after the fragmentation (after the initial fragmentation or the second fragmentation).

The above-mentioned transmission adjustment unit is specifically configured to perform secondary fragmentation of the data to be transmitted according to the current transmission parameters. The client fragments the large-capacity data file to be transmitted according to the previously calculated fragment size m, and sets the transmission parameters according to the previously calculated number of threads p, response time r, and number of retries c to transmit large-capacity data Fragmentation of the file to the server.

If the historical transmission parameter corresponding to the current transmission time is not found in the historical transmission record, the preset transmission parameter is used directly as the current transmission parameter. That is, the client cannot obtain the historical transmission parameters corresponding to the server receiving the file from the historical transmission records, and uses the transmission parameters with the preset values to fragment the large-capacity data file to be transmitted, set the transmission parameters, and transmit Fragmentation of large-capacity data files to the server.

The remaining data to be transmitted that has not yet been transmitted is divided into corresponding fragments according to the current transmission parameter, the current transmission parameter after the incremental adjustment or the decrement adjustment is performed, and then the data is sent through the data transmission unit.

The state judging unit continues to judge whether the current transmission state is normal after incrementing (increasing the fragment size and the number of transmission threads in the current transmission parameter). If the current transmission status is normal, the parameter adjustment unit will continue to increase the fragment size, number of transmission threads, response time, and retry times in the current transmission parameters, and adjust the fragment size, number of transmission threads, response time, and retry according to the incremental adjustment. Number of attempts. The transmission adjustment unit performs corresponding fragmentation on the remaining data to be transmitted that has not yet been transmitted, and then sends it. If there is no abnormal transmission state until the data transmission is completed, the parameter adjustment unit saves the current transmission parameters when the data transmission is completed. If an abnormal transmission state occurs, the transmission adjustment unit adjusts the current transmission parameters according to the above two methods, or does not adjust the current transmission parameters, and the parameter adjustment unit saves the current transmission parameters after decrement adjustment or the current transmission parameters that have not been adjusted. Transmission parameters.

In the data transmission system provided by the embodiment of the application, the current transmission parameters are determined through historical transmission parameters, and then the data to be transmitted is fragmented and transmitted according to the current transmission parameters, thereby avoiding the situation that the traditional simple fragmentation method does not consider the current bandwidth. Direct fragmentation, resulting in excessive fragmentation transmission request attempts and increased time loss; by judging the current transmission status, dynamically adjusting the current transmission parameters and fragment size, the network bandwidth can be maximized, thereby reducing the transmission time and improving Transmission efficiency improves bandwidth utilization and reduces transmission delay.

In one embodiment, a computer device is proposed. As shown in FIG. 4, the computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. Wherein, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When the computer-readable instructions are executed by the processor, the processor can implement a Kind of data transmission method. The processor of the computer equipment is used to provide calculation and control capabilities, and supports the operation of the entire computer equipment. A computer readable instruction may be stored in the memory of the computer device, and when the computer readable instruction is executed by the processor, the processor may execute a data transmission method. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand that the structure shown in FIG. 4 is only a block diagram of part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.

The computer equipment includes a memory, a processor, and a computer program that is stored on the memory and can run on the processor. When the processor executes the computer program, the following steps are implemented: query historical transmission records, obtain historical transmission parameters; determine according to historical transmission parameters Current transmission parameters; according to the current transmission parameters, the data to be transmitted is divided into pieces and transmitted; it is judged whether the current transmission status is normal, and if it is normal, the current transmission parameters are adjusted according to the first preset rule; the data to be transmitted is performed according to the adjusted current transmission parameters Transfer after fragmentation.

In one embodiment, a storage medium storing computer-readable instructions is provided. The storage medium may be volatile or non-volatile. As shown in Figure 5, when the computer-readable instruction is executed by one or more processors, one or more processors perform the following steps: query historical transmission records, obtain historical transmission parameters; determine current transmission according to historical transmission parameters Parameters; according to the current transmission parameters, the data to be transmitted is initially fragmented and then transmitted; it is judged whether the current transmission status is normal, and if it is normal, the current transmission parameters are adjusted according to the first preset rule; the remaining transmission parameters are adjusted according to the adjusted current transmission parameters The data is transmitted after the second fragmentation.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a computer readable storage medium. When executed, it may include the procedures of the above-mentioned method embodiments. Among them, the aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the various technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, All should be considered as the scope of this specification.

The above-mentioned embodiments only express several implementation manners of the present application, and their description is relatively specific and detailed, but they should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims (20)

1. A data transmission method, which includes:

   Query historical transmission records and obtain historical transmission parameters;

   Determine current transmission parameters according to the historical transmission parameters;

   According to the current transmission parameters, the data to be transmitted is transmitted after initial fragmentation;

   Determine whether the current transmission status is normal;

   If it is normal, the current transmission parameters are adjusted according to the first preset rule, and the remaining data to be transmitted is transmitted after being fragmented twice according to the adjusted current transmission parameters.
2. The data transmission method according to claim 1, wherein the determining current transmission parameters according to the historical transmission parameters comprises:

   Obtain one or more sets of historical parameter vectors from the historical transmission parameters,

   According to the obtained one or more sets of historical parameter vectors, a weighted average smoothing formula is used to calculate the current transmission parameters.
3. The data transmission method according to claim 2, wherein the historical parameter vector and the current transmission parameter both include: fragment size, number of transmission threads, response time and/or number of retries.
4. The data transmission method according to claim 1, wherein said determining whether the current transmission status is normal comprises:

   Determine whether there is an abnormal situation of timeout during the transmission of the fragmented data of the data to be transmitted.
5. The data transmission method according to claim 1, wherein if it is normal, the current transmission parameters are adjusted according to a first preset rule, and the remaining data to be transmitted is divided into two parts according to the adjusted current transmission parameters. Post-slice transmission, including:

   Increase the fragment size and the number of transmission threads in the current transmission parameters according to the first preset rule;

   Using the increased fragment size and the number of transmission threads as the current transmission parameters, the remaining transmission data is fragmented for a second time and then transmitted.
6. The data transmission method according to claim 1, wherein after the current transmission state is normal, the method further comprises:

   If it is abnormal, save the current transmission parameters to the historical transmission record;

   Use the current transmission parameters to continue to transmit the remaining data to be transmitted.
7. The data transmission method according to claim 1, wherein before said determining the current transmission parameters according to the historical transmission parameters, the method further comprises:

   If the historical transmission parameter corresponding to the current transmission time is not found in the historical transmission record, the preset transmission parameter is used as the current transmission parameter.
8. A data transmission system, which includes:

   The query module is used to query historical transmission records and obtain historical transmission parameters;

   The judgment calculation module is configured to determine the current transmission parameters according to the historical transmission parameters; determine whether the current transmission status is normal, and if it is normal, adjust the current transmission parameters according to the first preset rule;

   The transmission module is configured to perform initial fragmentation and transmission of the data to be transmitted according to the current transmission parameters; and perform second fragmentation and transmission of the remaining data to be transmitted according to the adjusted current transmission parameters.
9. A computer device includes a memory and a processor, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor is caused to execute the data transmission method described below. step:

   Query historical transmission records and obtain historical transmission parameters;

   Determine current transmission parameters according to the historical transmission parameters;

   According to the current transmission parameters, the data to be transmitted is transmitted after initial fragmentation;

   Determine whether the current transmission status is normal;

   If it is normal, the current transmission parameters are adjusted according to the first preset rule, and the remaining data to be transmitted is transmitted after being fragmented twice according to the adjusted current transmission parameters.
10. The computer device according to claim 9, wherein the determining the current transmission parameter according to the historical transmission parameter comprises:

    Obtain one or more sets of historical parameter vectors from the historical transmission parameters,

    According to the obtained one or more sets of historical parameter vectors, a weighted average smoothing formula is used to calculate the current transmission parameters.
11. The computer device according to claim 9, wherein said determining whether the current transmission status is normal comprises:

    Determine whether there is an abnormal situation of timeout during the transmission of the fragmented data of the data to be transmitted.
12. 9. The computer device according to claim 9, wherein, if it is normal, the current transmission parameter is adjusted according to a first preset rule, and the remaining data to be transmitted is fragmented twice according to the adjusted current transmission parameter Later transmission, including:

    Increase the fragment size and the number of transmission threads in the current transmission parameters according to the first preset rule;

    Using the increased fragment size and the number of transmission threads as the current transmission parameters, the remaining transmission data is fragmented for a second time and then transmitted.
13. The computer device according to claim 9, wherein after the current transmission status is normal, it further comprises:

    If it is abnormal, save the current transmission parameters to the historical transmission record;

    Use the current transmission parameters to continue to transmit the remaining data to be transmitted.
14. The computer device according to claim 9, wherein before the determining the current transmission parameter according to the historical transmission parameter, the method further comprises:

    If the historical transmission parameter corresponding to the current transmission time is not found in the historical transmission record, the preset transmission parameter is used as the current transmission parameter.
15. A storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data transmission method as described below:

    Query historical transmission records and obtain historical transmission parameters;

    Determine current transmission parameters according to the historical transmission parameters;

    According to the current transmission parameters, the data to be transmitted is transmitted after initial fragmentation;

    Determine whether the current transmission status is normal;

    If it is normal, the current transmission parameters are adjusted according to the first preset rule, and the remaining data to be transmitted is transmitted after being fragmented twice according to the adjusted current transmission parameters.
16. The storage medium according to claim 15, wherein the determining the current transmission parameter according to the historical transmission parameter comprises:

    Obtain one or more sets of historical parameter vectors from the historical transmission parameters,

    According to the obtained one or more sets of historical parameter vectors, a weighted average smoothing formula is used to calculate the current transmission parameters.
17. The storage medium according to claim 15, wherein said determining whether the current transmission status is normal comprises:

    Determine whether there is an abnormal situation of timeout during the transmission of the fragmented data of the data to be transmitted.
18. 15. The storage medium according to claim 15, wherein if it is normal, the current transmission parameters are adjusted according to a first preset rule, and the remaining data to be transmitted is fragmented twice according to the adjusted current transmission parameters Later transmission, including:

    Increase the fragment size and the number of transmission threads in the current transmission parameters according to the first preset rule;

    Using the increased fragment size and the number of transmission threads as the current transmission parameters, the remaining transmission data is fragmented for a second time and then transmitted.
19. The storage medium according to claim 15, wherein after the current transmission status is normal, it further comprises:

    If it is abnormal, save the current transmission parameters to the historical transmission record;

    Use the current transmission parameters to continue to transmit the remaining data to be transmitted.
20. The storage medium according to claim 15, wherein before the determining the current transmission parameter according to the historical transmission parameter, the method further comprises:

    If the historical transmission parameter corresponding to the current transmission time is not found in the historical transmission record, the preset transmission parameter is used as the current transmission parameter.

Images