TW202129516A - A method and a system for a data transmission method based on asynchronous response - Google Patents

Abstract

The present invention discloses a data transmission method based on asynchronous response, and the method includes: an extraction step that extracts data from a first data end in a cycle and stores the data; a flow speed adjustment step that confirms a second flow speed of the data transmitted in the cycle based on a load result feedback from a second data end in the previous cycle and a first flow speed of the data transmitted in the previous cycle; and a loading step that loads the data to the second data end in the cycle according to the second flow speed.

Description

Method and system for transmitting data based on asynchronous response

The present invention relates to the field of data transmission, in particular to a method and system for transmitting data based on an asynchronous response.

In the process of transferring data from the data source to the data destination, an ETL (Extract-Transform-Load) system that can extract data, convert the format of the data, and load the data to the data destination is usually used. Since the ETL system needs to run frequently occurring tasks (for example, processing batch data) in a fixed cycle and the ETL system does not require high real-time data, it is difficult to use the traditional ETL system in scenarios that require high real-time data. In the process of data transmission, there is a need to adjust the flow rate of data transmission. A commonly used method of adjusting the flow rate is to use the system load of the data transmission system as a hard indicator for adjustment. For example, when the system load is higher than a certain threshold, the flow rate is prohibited or reduced, and when the system load starts to decrease, the flow rate is restored. Another commonly used method of adjusting the flow rate is the adaptive current limiting protection method, which uses the system load of the data transmission system as the starting value of the controlled flow rate, which is determined by the current request response time of the data transmission system and the processing request rate The current flow rate allowed, so as to increase the flow rate without the data transmission system being worn down, rather than having to fall below a certain threshold.

In order to increase the flow rate as much as possible, the asynchronous response mode can also be used, that is, the module responsible for data loading in the data transmission system only sends the data to the downstream data destination without waiting for the specific loading result. The data transmission system can The execution result of data loading is processed by another module. In other words, the data loaded in a cycle may not be able to know whether the data is successfully loaded in the current cycle.

However, in the case of using the asynchronous response mode, the first method mentioned above takes the system load of the data transmission system as a hard indicator, and data transmission is not allowed if the system load exceeds the system load, thus limiting the processing capacity of the data transmission system and the data destination. , Which reduces the data transmission volume of the data transmission system; in addition, when the downstream data destination fails or expands and other events that affect processing capabilities, this method cannot detect and adjust the flow rate in time, which may result in idle or idle data transmission systems. Abnormal situations such as loss of large quantities of data. In addition, for the second method mentioned above, the method adjusts the flow rate according to the processing capacity of the data transmission system, and cannot dynamically control the flow rate according to the processing capacity of the downstream data destination. It is not applicable to the data destination. The processing capacity is weaker than that of the data transmission system. Ability situation.

An aspect of the present invention provides a method for transmitting data based on an asynchronous response, which includes: an extraction step: extracting and storing the data from a first data terminal in a period; and a flow rate adjustment step: based on the previous period of the period The loading result fed back by the second data terminal in the cycle and the first flow rate at which the data was transmitted in the previous cycle determine the second flow rate at which the data is transmitted in the cycle; and the loading step: in the cycle Load the data to the second data terminal according to the second flow rate.

Another aspect of the present invention provides a method for transmitting data based on an asynchronous response, including: an extraction step: extracting first data from a first data terminal in a cycle, and storing the first data in a first queue; Flow rate adjustment step: determining to transmit the data in the cycle based on the first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate at which the data was transmitted in the previous cycle And the loading step: load the first data in the first queue and the second data in the second queue to the second data terminal in the cycle according to the second flow rate .

Another aspect of the present invention provides a system for transmitting data based on an asynchronous response, including: a data extraction module for extracting and storing the data from a first data terminal in a cycle; and a flow rate adjustment module, which It is used to determine the second data transmission in the cycle based on the loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate at which the data was transmitted in the previous cycle Flow rate; and a data loading module for loading the data to the second data terminal according to the second flow rate in the cycle.

Another aspect of the present invention provides a system for transmitting data based on an asynchronous response, including: a data extraction module for extracting first data from a first data terminal in a cycle, and storing the first data in In the first queue; a flow rate adjustment module, which is used to determine based on the first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate at which the data was transmitted in the previous cycle The second flow rate for transmitting the data in the period; and a data loading module for storing the first data and the second data in the first queue according to the second flow rate in the period The second data in the queue is loaded to the second data terminal.

Another aspect of the present invention provides a computer-readable medium having computer-readable instructions stored thereon, and the computer-readable instructions can execute the method according to the embodiments of the present invention when the computer-readable instructions are executed by a computer.

The embodiment of the present invention can automatically adjust the flow rate of the data transmission system according to the feedback loading result in the case of asynchronous response, thereby increasing the data as much as possible under the precondition of ensuring that the data destination is not overwhelmed and reducing data loss. The throughput of the transmission system.

The above description is only an overview of the technical solution of the present invention. In order to understand the technical means of the present invention more clearly, it can be implemented in accordance with the content of the specification, and in order to make the above and other objectives, features and advantages of the present invention more obvious and understandable. In the following, specific embodiments of the present invention are specifically cited.

The principle and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and then implement the present invention, but not to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Fig. 1 shows a schematic diagram of a data transmission system according to an embodiment of the present invention.

As shown in Figure 1, the system includes five modules: a data extraction module, a data loading module, an asynchronous response processing module, a timing scan module, and a flow rate adjustment module. The system also includes a data queue to be loaded, a data queue to be reloaded, and a data map to be confirmed.

The data extraction module can extract data from the data source, convert the format of the data and store it in the data queue to be loaded, and wait if the queue is full. In this article, one piece of data can represent one piece of business, and the size of each piece of data can be different.

The data loading module can take out the data from the data queue to be loaded, and judge whether the data can be loaded to the data destination through the flow rate adjustment module. The data loading module may also include a normal data loading module and a failed data loading module, which are respectively used to load the data in the data queue to be loaded and the data queue to be reloaded. Each time data is loaded, the data loading module can simultaneously store the data in the to-be-confirmed data map. In some embodiments, the data map to be confirmed is a data storage structure that can be used to temporarily store data dumped from the data loading module. The Key field in the data map to be confirmed stores the identity of the data, and the Value field stores the value of the data.

The asynchronous response processing module can process the processing result (that is, the loading result of the loaded data, which is success or failure) fed back by the data destination, and count the amount of successful or failed data. For the data whose processing result is successful, the asynchronous response processing module can directly delete the data from the to-be-confirmed data Map. For the data whose processing result is failure, the asynchronous response processing module can send the failed data from the data map to be confirmed to the data queue to be reloaded to be loaded by the data loading module. If the number of failures to load certain data exceeds the predetermined number (ie, the upper limit of the number of reloads), the data is discarded.

The timing scan module can count the data that has not received the processing result of the data destination after a certain period of time by scanning the to-be-confirmed data map (that is, the timeout data that has not received the response), and send the data from the to-be-confirmed data map To the data queue to be reloaded. The timing scanning module can also calculate the statistical data loading success rate in the current cycle by scanning the processing results from the data destination stored in the asynchronous response processing module, and send the success rate to the flow rate adjustment module.

The flow rate adjustment module can achieve flow rate control by adjusting the TPS (Transaction Per Second, transaction volume per second, such as the number of transactions processed per second) of the data transmission system. First, you can set an ideal maximum TPS value as the upper limit of the flow rate, and increase or decrease the flow rate (TPS value) of the data transmission system according to the success rate of data loading in the current cycle, so as to realize the flow rate adjustment of the system.

Figure 2 shows the data extraction method that can be used by the data extraction module. As shown in Figure 2, the method includes the following steps:

(1) The data extraction module extracts data from the data source and converts the data in a unified format;

(2) If the queue to be loaded is not full at this time, store the converted format data directly into it, and then continue to perform step (1), otherwise perform step (3);

(3) If the queue to be loaded is full at this time, accumulate the number of transmission failures for the current data. If the number of transmission failures does not exceed the upper limit of the number of transmission failures, the data extraction module sleeps for a predetermined time and re-executes step (2), otherwise The data is discarded, and step (1) is executed again.

Figure 3 shows a timing diagram of the method of loading data that the data loading module can use. As shown in Figure 3, the data loading module can obtain the converted format data from the data queue to be loaded. After obtaining the data to be loaded, in order to ensure that the current flow rate of the data transmission system does not exceed the allowable flow rate upper limit, the data loading module needs to apply to the flow rate adjustment module for a token that allows data to be loaded, only after the token is obtained , Can send the data to be loaded to the data destination.

When the flow rate adjustment module receives an application for a token, it checks whether there are still available tokens, and if there is no available token temporarily, the application is suspended and waits for a new token to be generated. The flow rate adjustment module can periodically generate or clear tokens, for example, empty the remaining tokens at the end of each cycle, and generate a number of tokens corresponding to the upper limit of the current system allowable flow rate (for example, the upper limit of the flow rate) It is 1000, which means, for example, that it can process up to 1000 data per second (for example, 1000 transactions), then the flow rate adjustment module can generate 1000 tokens to allow processing of 1000 data). The upper limit of the flow rate will be updated according to the data loading status of the data destination. By modifying the number of tokens allowed in a cycle, dynamic adjustment of the system flow rate can be achieved.

Fig. 4 shows a timing diagram of a method for adjusting a flow rate according to an embodiment of the present invention. The method involves an asynchronous response processing module, a timing scanning module, and a flow rate adjustment module. The operations that can be performed by the method include counting the processing results of the data destination, counting the data that has not responded to timeout, calculating the loading success rate in the current cycle, and updating the upper limit of the flow rate allowed by the system.

The asynchronous response processing module is used to receive the loading result fed back by the data destination, and process the corresponding data (that is, delete the successfully loaded data from the data map to be confirmed, or delete the failed data or the unreceived response The timeout data is sent to the data queue to be reloaded), and the value of the loading success counter or loading failure counter is updated according to the feedback loading result.

The timing scan module can scan in each cycle to calculate the success rate of data loading at the data destination in the current cycle (this data is the data that should be loaded by the data destination in the previous cycle, which can also be called historical data). This triggers the flow rate adjustment module to adjust the flow rate. For calculating the success rate, the timing scan module can first count the amount of timeout data that has not received a response in the current period (for example, stored in the timeout data counter), and then obtain the current time data load success counter and load failure counter values , Calculate the success rate according to the following formula:

Among them, S is the success rate of data loading in the current cycle, Cs and Cf are the values of the current load success counter and load failure counter, respectively, and Ct is the value of the timeout data counter for which no response is currently received. The value of each counter is cleared immediately after the calculation of S is completed to ensure that it will not interfere with the calculation of the next cycle.

The method of adjusting the flow rate can be summarized as follows:

(1) Configure the upper limit Tmax of acceptable flow rate according to the processing capacity of the data destination;

(2) If the success rate of data loading fed back from the data destination in the current cycle is lower than the lower limit of acceptable success rate, the flow rate of the data transmission system is lowered according to a fixed step. If the flow rate is lower than the lower limit of the flow rate, The lower limit of the flow rate is taken as the actual flow rate (the number of tokens corresponding to the determined flow rate value is generated). If the flow rate before the reduction has reached the acceptable lower limit of the flow rate, the system flow rate is no longer lowered, and manual intervention is required;

(3) If the success rate of data loading in the current cycle is greater than the success rate adjustment threshold (which is greater than the lower limit of the success rate) and the current system flow rate is less than Tmax, then the flow rate can be increased according to a fixed step, if the flow rate is higher than the upper limit of the flow rate , The upper limit of the flow rate is taken as the actual flow rate (the number of tokens corresponding to it is generated according to the determined flow rate value). If the flow rate before the increase has reached Tmax, the system flow rate value will not be increased;

(4) If the success rate is greater than the lower limit of the success rate and less than the success rate adjustment threshold, and the flow rate is less than Tmax, the data transmission system and the downstream system (that is, the data destination) are considered to be in a stable state, and there is no need to adjust the flow rate of the data transmission system ；

(5) If the flow rate of the data transmission system is maintained at Tmax, and the calculated success rate is always greater than or equal to the success rate adjustment threshold, try to increase Tmax; if the success rate drops immediately after increasing Tmax, the previous Tmax value will be restored. No more attempts to increase Tmax within (for example, the same day);

(6) If the data destination is expanded, Tmax can be manually modified to the upper limit of the flow rate that is compatible with the expanded processing capacity;

(7) If the calculated success rate is greater than or equal to the success rate adjustment threshold and the data transmission system increases the flow rate, the calculated success rate is lower than the lower limit of the success rate, causing the data transmission system to lower the flow rate, and first adjust the flow rate and then lower it The flow rate process continues many times, so this may mean that the calculated Tmax does not meet the current status quo, and manual intervention is required. For example, the value of Tmax can be manually reduced.

An embodiment of the method for realizing the above-mentioned adjusting flow rate will be described below. First, define each parameter, and its meaning is as follows:

Tmax: In an ideal state, the maximum TPS that the data destination can carry (that is, the upper limit of the flow rate), which can be manually configured and modified by the system operation and maintenance personnel;

Tmin: The lowest TPS acceptable to the data destination, if it is lower than this value, manual intervention is required for investigation;

Smin: The minimum success rate of data loading allowed by the data destination;

Sadp: The lowest acceptable success rate when TPS is increased (that is, the success rate adjustment threshold). Only when the current success rate is greater than or equal to this value, you can try to increase the value of TPS;

st: TPS adjustment step length, that is, the value of TPS is adjusted up or down in a single time;

Tcur: The currently allowed TPS value of the data destination. When the data transmission system is started, Tcur can be arbitrarily configured so that Tcur is not greater than Tmax.

In addition, this article defines a continuous TPS up-regulation and down-regulation or a continuous TPS down-regulation and up-regulation as an oscillation process. If oscillations occur continuously, it means that the value of Tmax does not match the current state of the data destination, and the operation and maintenance personnel need to be reminded to do so. Check and confirm, if necessary, modify the value of Tmax.

Based on this, for example, the flag bit UD can be used to record each adjustment operation:

UD=00: no adjustment; UD=10: increase TPS; UD=01: decrease TPS

In addition, use UDp to record the last flow rate adjustment operation. For example, you can calculate UD|UDp (or operation) to indicate whether oscillations have occurred (for example, UD|UDp=11, indicating that an oscillation has occurred), use Ccon to record the number of continuous oscillations, and use Mcon to indicate the maximum acceptable number of continuous oscillations .

Based on the above definition, the specific implementation of the method of adjusting the flow rate is as follows:

● When S≥Sadp and Tcur ＜Tmax, UD=10 at this time, then update Tcur+=st and calculate UD|UDp:

If UD|UDp=11, modify Ccon+=1, UDp=10;

If UD|UDp!=11, modify Ccon=0, UDp=10.

●When S≥Sadp and Tcur=Tmax, Tcur remains unchanged, modify Ccon=0, UDp=00

●When S≤Smin and Tcur ＞Tmin, UD=01 at this time, update Tcur-=st, calculate UD|UDp:

If UD|UDp=11, modify Ccon+=1, UDp=01;

If UD|UDp!=11, modify Ccon=0, UDp=01.

●When S≤Smin and Tcur=Tmin, Tcur remains unchanged, modify Ccon=0, UDp=00, and an alarm is required to prompt manual intervention;

●When Smin＜S＜Sadp, Tcur remains unchanged, modify Ccon=0, UDp=00.

Each time the algorithm for adjusting the flow rate is executed, the corresponding token number is first generated according to the updated TPS value allowed by the data destination, and Ccon is checked at the same time. If Ccon≥Mcon, it means that the number of consecutive oscillations of the data transmission system exceeds the upper limit, that is, the value of Tmax is not suitable for the current data destination. Therefore, the data transmission system will issue an alarm to remind operation and maintenance personnel to intervene in the investigation.

In order to reduce abnormal data (including data that failed to load and timeout data that did not receive a response), the asynchronous response processing module or the timed scanning module will send the abnormal data to the data queue to be reloaded, and load it through the data Failed data in the module. Reload the module to reload the abnormal data.

Fig. 5 shows a flowchart of a method for processing abnormal data according to an embodiment of the present invention. The methods used to handle abnormal data are as follows:

(1) The asynchronous response processing module judges after receiving the processing result. When the result is successful, it directly removes the data from the data map to be confirmed, and the process ends; otherwise, it proceeds to step (2);

(2) If the result is not successful, update the number of failures. If the number of failures exceeds the upper limit, directly remove the data from the to-be-confirmed data Map and report an exception (such as an alarm), otherwise go to step (3);

(3) If the data queue to be reloaded is not full at this time, transfer the data directly to the data queue to be reloaded, otherwise it will be recorded as a transfer failure, directly update the number of failures, and continue to step (4);

(4) If the number of data failures exceeds the upper limit of failure, the data is directly discarded and an exception is reported.

Otherwise, it will continue to be stored in the to-be-confirmed data Map, and then go to step (3) after waiting for a period of time.

It can be seen from Figure 1 and Figure 5 that the method of handling abnormal data can also reduce the impact of abnormal data on the system by loading isolation, that is, using the normal data loading module to be responsible for processing the data in the data queue to be loaded, and the use fails. The data loading module is responsible for processing the data in the data queue to be reloaded, and the two modules are executed in parallel. Based on this, the TPS value Tcur allowed by the current system or data destination calculated by the flow rate adjustment module can be subdivided into normal data loading TPS (denoted as Tnor) and abnormal data loading TPS (denoted as Tuno). Different loading modules Request the respective load tokens to realize the load isolation of normal data and abnormal data. The specific method is as follows:

(1) If the success rate is not 100%, multiply the success rate by the current flow rate by the flow rate loaded as normal data, use the remaining part of the current flow rate as the flow rate loaded with abnormal data, and update the flow rate of the abnormal data at the current flow rate The proportion in the data transmission system will be used in subsequent cycles. The specific implementation method can be, for example:

When S≠100%, Tnor= Tcur*S, Tuno=Tcur*(1-S); update R _P = Tuno/ Tcur.

(2) If the success rate is 100%, the flow rate of abnormal data loading is the minimum value of the number of data in the data queue to be reloaded, and the product of the above proportion and the current flow rate. Therefore, the flow rate of the normal data load is the difference between the current flow rate and the flow rate of the abnormal data load. The percentage of the flow rate of abnormal data in the current flow rate can be updated for the data transmission system to use in subsequent cycles. The specific implementation method can be, for example:

When S=100%, Tuno=min(L _w R _P * Tcur), Tnor= Tcur-Tuno, update R _P = Tuno/ Tcur.

Among them, S is the success rate of data loading in the current cycle, Rp is the proportion of abnormal data loading TPS in the total TPS of the system during the previous week, and Lu is the number of data in the data queue to be reloaded (for example, the number of transactions) ).

The program code used to perform the operations of the present invention can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural programming. Language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computing device, partly executed on the user's equipment, partly executed on the user's computing device, or entirely executed on the remote computing device or server. In the case of remote computing devices, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (for example, using Internet services) Provider to connect via the Internet).

In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that these operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution.

It should be noted that although several software devices/modules and sub-devices/modules that implement the above methods are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more devices described above can be embodied in one device/module. Conversely, the features and functions of one device/module described above can be further divided into multiple devices/modules to be embodied.

Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined for performance. Benefit, this division is only for ease of presentation. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

without.

[Fig. 1] A schematic diagram showing a data transmission system according to an embodiment of the present invention. [Fig. 2] A flowchart showing a method for extracting data according to an embodiment of the present invention. [Fig. 3] A timing chart showing a method for loading data according to an embodiment of the present invention. [Fig. 4] A timing chart showing a method for adjusting a flow rate according to an embodiment of the present invention. [Fig. 5] A flowchart showing a method for processing abnormal data according to an embodiment of the present invention.

Claims (39)

A method for transmitting data based on an asynchronous response, including: Extraction step: extract and store the data from the first data terminal in one cycle; Flow rate adjustment step: determining the data transmission rate in the cycle based on the loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate at which the data was transmitted in the previous cycle Second flow rate; and Loading step: loading the data to the second data terminal according to the second flow rate in the cycle. According to the method described in claim 1, the extraction step includes: (A) Extract the data from the first data terminal, and convert the data according to a predetermined format; (B) Determine whether the queue is full, if not, store the converted data in the queue and go to step (a), otherwise go to step (c); (C) Accumulate the number of failed transmissions once for the converted data. If the number of failed transmissions does not exceed the upper limit of the number of failed transmissions, wait a predetermined time before re-entering step (b), otherwise discard the converted data And go to step (a). The method according to claim 1, wherein the loading result is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, The quantity of historical data of loading failure fed back by the second data terminal, wherein the flow rate adjustment step further includes: counting timeouts that have not received the feedback from the second data terminal after a predetermined time has passed in the previous cycle The number of historical data, and the success rate of loading historical data fed back by the second data terminal in the last cycle, the success rate is used to determine the second flow rate, wherein the success rate is passed Calculated by the following formula:

Where, S is the success rate, Cs is the number of historical data successfully loaded by the second data terminal in the previous cycle, and Cf is the number of historical data from the second data in the previous cycle. The number of historical data of loading failure fed back by the terminal, and Ct is the number of time-out historical data that has not received the feedback from the second data terminal after a predetermined time has been exceeded in the previous cycle. The method according to claim 3, wherein the flow rate adjustment step includes: (A) If the success rate is less than or equal to the lower limit of the success rate and the first flow rate is higher than the lower limit of the flow rate, the first flow rate is reduced according to a first predetermined value to obtain the second flow rate. If the flow rate is less than the lower limit of the flow rate, the lower limit of the flow rate is taken as the second flow rate; (B) If the success rate is less than or equal to the lower limit of the success rate and the first flow rate is equal to the lower limit of the flow rate, then the lower limit of the flow rate is used as the second flow rate; (C) If the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the upper limit of the flow rate, increase the first flow rate according to a second predetermined value to obtain the second flow rate, if the increased first flow rate Is greater than the upper limit of the flow rate, then the upper limit of the flow rate is taken as the second flow rate; (D) If the success rate is higher than the success rate adjustment threshold and the first flow rate reaches the upper flow rate limit, then the upper flow rate limit is used as the second flow rate; or (E) If the success rate is between the success rate lower limit and the success rate adjustment threshold, the first flow rate is used as the second flow rate. The method according to claim 4, wherein the flow rate adjustment step further includes: when the second flow rate is maintained at the upper limit of the flow rate for a first predetermined number of cycles, and the success rate is greater than the success rate When the threshold is adjusted, the upper limit of the flow rate and the second flow rate are increased according to a second predetermined value. The method according to claim 5, wherein after the step of increasing the upper limit of the flow rate and the second flow rate according to a second predetermined value, if the success rate drops in the next period of the period, Then, the increased upper flow rate limit and the increased second flow rate are restored to the upper flow rate limit and the second flow rate, and the upper flow rate limit is not increased for a second predetermined number of cycles. The method according to claim 4, wherein the method further includes: If the following steps are repeated in multiple cycles and the number of repeated operations reaches the preset number of times, it is determined that the upper limit of the flow rate is set incorrectly and an alarm message is issued: Perform the step (a) directly after performing the step (c), or Perform the step (c) directly after performing the step (a). The method according to claim 3, wherein the loading step includes: Generating a token for loading the data according to the second flow rate and the success rate in the cycle; Extract the data in the queue according to the token; and Loading the extracted data in the queue to the second data terminal. A method for transmitting data based on an asynchronous response, including: Extraction step: extract first data from the first data terminal in one cycle, and store the first data in the first queue; Flow rate adjustment step: determining to transmit the data in the cycle based on the first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate at which the data was transmitted in the previous cycle The second flow rate; and Loading step: loading the first data in the first queue and the second data in the second queue to the second data terminal according to the second flow rate in the cycle. According to the method according to claim 9, the method further includes: storing the first data and the second data in a data structure as candidate data. According to the method of claim 10, the method further includes: determining a candidate in the data structure based on a second loading result fed back by the second data terminal in the period or at least one period after the period Whether the data is loaded successfully, if successful, delete the candidate data in the data structure, if unsuccessful, judge whether the number of times the candidate data has been loaded exceeds a predetermined number of times, if it exceeds the predetermined number of times, then The candidate data is deleted from the data structure, and if the predetermined number of times is not exceeded, the candidate data is sent to the second queue as the second data. According to the method described in claim 9, the extraction step includes: (A) Extract the first data from the first data terminal, and convert the first data in a predetermined format; (B) Determine whether the first queue is full, if not, store the converted first data in the first queue and go to step (a), otherwise go to step (c); (C) Accumulate the number of transmission failures once for the converted first data. If the number of transmission failures does not exceed the upper limit of the number of transmission failures, then wait for a predetermined time and then re-enter the step (B), otherwise discard the converted first data and go to step (a). The method according to claim 9, wherein the first loading result is the number of successfully loaded historical data fed back by the second data terminal in the last cycle, The number of historical data of loading failures fed back by the second data terminal, wherein the flow rate adjustment step further includes: counting data that have not received the feedback from the second data terminal after a predetermined time has passed in the previous cycle The number of time-out historical data, and the success rate of loading historical data fed back by the second data terminal in the last cycle, and the success rate is used to determine the second flow rate, wherein the success The rate is calculated by the following formula:

Where, S is the success rate, Cs is the number of historical data successfully loaded by the second data terminal in the previous cycle, and Cf is the number of historical data from the second data in the previous cycle. The number of historical data of loading failure fed back by the terminal, and Ct is the number of time-out historical data that has not received the feedback from the second data terminal after a predetermined time has been exceeded in the previous cycle. The method according to claim 13, wherein the flow rate adjustment step includes: (A) If the success rate is less than or equal to the lower limit of the success rate and the first flow rate is higher than the lower limit of the flow rate, the first flow rate is reduced according to a first predetermined value to obtain the second flow rate. If the flow rate is less than the lower limit of the flow rate, the lower limit of the flow rate is taken as the second flow rate; (B) If the success rate is less than or equal to the lower limit of the success rate and the first flow rate is equal to the lower limit of the flow rate, then the lower limit of the flow rate is used as the second flow rate; (C) If the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the upper limit of the flow rate, increase the first flow rate according to a second predetermined value to obtain the second flow rate, if the increased first flow rate Is greater than the upper limit of the flow rate, then the upper limit of the flow rate is taken as the second flow rate; (D) If the success rate is higher than the success rate adjustment threshold and the first flow rate reaches the upper flow rate limit, then the upper flow rate limit is used as the second flow rate; or (E) If the success rate is between the success rate lower limit and the success rate adjustment threshold, the first flow rate is used as the second flow rate. The method according to claim 14, wherein the step of adjusting the flow rate further comprises: when the second flow rate remains at the upper limit of the flow rate for a first predetermined number of cycles, and the success rate is greater than the success rate When the threshold is adjusted, the upper limit of the flow rate and the second flow rate are increased according to a second predetermined value. The method according to claim 15, wherein, after the step of increasing the upper limit of the flow rate and the second flow rate according to a second predetermined value, if the success rate drops in the next period of the period, Then, the increased upper flow rate limit and the increased second flow rate are restored to the upper flow rate limit and the second flow rate, and the upper flow rate limit is not increased for a second predetermined number of cycles. The method according to claim 14, wherein the method further includes: if the following steps are repeatedly performed in a plurality of cycles and the number of repeated operations reaches a preset number of times, determining that the upper limit of flow rate is set incorrectly and Send out warning message: Perform the step (a) directly after performing the step (c), or Perform the step (c) directly after performing the step (a). The method according to claim 13, wherein the loading step includes: (A) Generate a first token for loading the first data and a second token for loading the second data in the cycle according to the second flow rate and the success rate; (B) Extracting the first data in the first queue according to the first token and extracting the second data in the second queue according to the second token; and (C) Loading the extracted first data and the extracted second data to the second data terminal. The method according to claim 18, wherein the step (a) comprises: when the success rate is less than 100%, multiplying the success rate by the second flow rate as loading the first data The second flow rate is subtracted from the difference between the flow rate loaded with the first data to obtain the flow rate loaded with the second data, and the flow rate loaded with the second data is updated with the The ratio of the second flow rate, and the first token is generated according to the flow rate at which the first data is loaded and the second token is generated according to the flow rate at which the second data is loaded; or When the success rate is equal to 100%, the minimum value among the number of data in the second queue and the product of the ratio and the second flow rate is used as the flow rate for loading the second data , Subtract the second flow rate from the difference between the flow rate loaded with the second data to obtain the flow rate loaded with the first data, update the ratio, and load the first data according to the The first token is generated at a flow rate of and the second token is generated according to the flow rate at which the second data is loaded. A computer-readable medium having computer-readable instructions stored thereon, and when the computer-readable instructions are executed by a computer, the method according to any one of claims 1-19 can be executed. A data transmission system based on asynchronous response, including: A data extraction module for extracting and storing the data from the first data terminal in one cycle; The flow rate adjustment module is used to determine the transmission in the cycle based on the loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate at which the data was transmitted in the previous cycle The second flow rate of the data; and The data loading module is used for loading the data to the second data terminal according to the second flow rate in the cycle. For the system described in claim 21, the data extraction module can also perform the following steps: (A) Extract the data from the first data terminal, and convert the data according to a predetermined format; (B) Determine whether the queue is full, if not, store the converted data in the queue and go to step (a), otherwise go to step (c); (C) Accumulate the number of failed transmissions once for the converted data. If the number of failed transmissions does not exceed the upper limit of the number of failed transmissions, wait for a predetermined time before re-entering step (b), otherwise discard the converted data And go to step (a). The system according to claim 21, wherein the loading result is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, The number of historical data of loading failure fed back by the second data terminal, wherein the system further includes a timing scanning module for counting that the second data has not been received for more than a predetermined time in the previous cycle The number of timeout historical data fed back by the terminal, calculate the success rate of loading historical data fed back by the second data terminal in the previous cycle, and send the success rate to the flow rate adjustment module, wherein , The success rate is calculated by the following formula:

Wherein, S is the success rate, Cs is the number of historical data successfully loaded by the second data terminal in the previous cycle, and Cf 5 is the success rate in the previous cycle. The number of historical data of loading failure fed back by the data terminal, and Ct is the number of time-out historical data fed back by the second data terminal that has not received the feedback from the second data terminal after a predetermined time has been exceeded in the previous cycle. The system according to claim 23, wherein the flow rate adjustment module further includes: The first module is used to reduce the first flow rate according to a first predetermined value to obtain the second flow rate when the success rate is less than or equal to the lower limit of the success rate and the first flow rate is higher than the lower limit of the flow rate , In the case that the reduced first flow rate is less than the lower limit of the flow rate, the lower limit of the flow rate is used as the second flow rate; The second module is configured to use the lower limit of the flow rate as the second flow rate when the success rate is less than or equal to the lower limit of the success rate and the first flow rate is equal to the lower limit of the flow rate; The third module is used to increase the first flow rate according to a second predetermined value to obtain the second flow rate when the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the upper flow rate limit, In the case where the increased first flow rate is greater than the upper flow rate limit, use the upper flow rate limit as the second flow rate; The fourth module is configured to use the upper flow rate as the second flow rate when the success rate is higher than the success rate adjustment threshold and the first flow rate reaches the upper flow rate limit; or The fifth module is configured to use the first flow rate as the second flow rate when the success rate is between the success rate lower limit and the success rate adjustment threshold. The system according to claim 24, wherein the flow rate adjustment module further includes: a sixth module, which is used to maintain the second flow rate at the upper limit of the flow rate for a first predetermined number of cycles, and When the success rate is greater than the success rate adjustment threshold, the upper limit of the flow rate and the second flow rate are increased according to a second predetermined value. The system according to claim 25, wherein, after increasing the upper limit of the flow rate and the second flow rate according to a second predetermined value, the flow rate adjustment module can also be able to report successfully in the next cycle of the cycle In the case of a decrease in the flow rate, the increased flow rate upper limit and the increased second flow rate are restored to the flow rate upper limit and the second flow rate, and the flow rate upper limit is not increased for a second predetermined number of cycles. The system according to claim 24, wherein the flow rate adjustment module is further configured to repeatedly perform the following steps in a plurality of cycles and the number of repeated operations reaches a preset number of times to determine the flow rate The upper limit is set incorrectly and a warning message is issued: Use the first module directly after using the third module; or Use the third module directly after using the first module. The system according to claim 23, wherein the flow rate adjustment module further includes: A module for generating a token for loading the data according to the second flow rate and the success rate in the cycle; The data loading module further includes: A module for extracting data in the queue according to the token; and A module for loading the extracted data in the queue to the second data terminal. A data transmission system based on asynchronous response, including: A data extraction module, which is used to extract first data from the first data terminal in a cycle, and store the first data in a first queue; A flow rate adjustment module, which is used to determine that in the cycle based on the first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate that transmits the data in the previous cycle The second flow rate at which the data is transmitted; and The data loading module is configured to load the first data in the first queue and the second data in the second queue to the second data terminal according to the second flow rate in the cycle. The system according to claim 29, wherein the data loading module is further configured to store the first data and the second data in a data structure as candidate data. For example, in the system according to claim 30, the system further includes: an asynchronous response processing module, which is configured to respond based on the second data feedback from the second data terminal in the period or at least one period after the period. The loading result determines whether the candidate data in the data structure is loaded successfully, if successful, delete the candidate data in the data structure, if unsuccessful, judge whether the candidate data has been loaded more than a predetermined number of times, If the predetermined number of times is exceeded, the candidate data is deleted in the data structure, and if the predetermined number of times is not exceeded, the candidate data is sent to the second queue as the second data. For the system described in claim 29, the data extraction module can also perform the following steps: (A) Extract the first data from the first data terminal, and convert the first data in a predetermined format; (B) Determine whether the first queue is full, if not, store the converted first data in the first queue and go to step (a), otherwise go to step (c); (C) Accumulate the number of transmission failures once for the converted first data. If the number of transmission failures does not exceed the upper limit of the number of transmission failures, wait for a predetermined time and re-enter step (b), otherwise discard the converted The first data and go to step (a). The system according to claim 29, wherein the first loading result is the number of historical data of successful loading fed back by the second data terminal in the last period of the 15 The number of historical data of loading failures fed back by the second data terminal, wherein the system further includes a timing scanning module, which is used to count the number of the failure to receive the data after a predetermined time has passed in the previous cycle. The number of timeout historical data fed back by the second data terminal, and the success rate of loading historical data fed back by the second data terminal in the previous cycle is calculated, and the success rate is sent to the flow rate adjustment Module, wherein the success rate is calculated by the following formula:

Where, S is the success rate, Cs is the number of historical data successfully loaded by the second data terminal in the previous cycle, and Cf is the number of historical data from the second data in the previous cycle. The number of historical data of loading failure fed back by the terminal, Ct 25 is the number of time-out historical data fed back by the second data terminal that has not received the feedback of the second data terminal after a predetermined time has been exceeded in the previous cycle. The system according to claim 33, wherein the flow rate adjustment module further includes: The first module is used to reduce the first flow rate according to a first predetermined value to obtain the second flow rate when the success rate is less than or equal to the lower limit of the success rate and the first flow rate is higher than the lower limit of the flow rate , In the case that the reduced first flow rate is less than the lower limit of the flow rate, the lower limit of the flow rate is used as the second flow rate; The second module is configured to use the lower limit of the flow rate as the second flow rate when the success rate is less than or equal to the lower limit of the success rate and the first flow rate is equal to the lower limit of the flow rate; The third module is used to increase the first flow rate according to a second predetermined value to obtain the second flow rate when the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the upper flow rate limit, In the case where the increased first flow rate is greater than the upper flow rate limit, use the upper flow rate limit as the second flow rate; The fourth module is configured to use the upper flow rate as the second flow rate when the success rate is higher than the success rate adjustment threshold and the first flow rate reaches the upper flow rate limit; or The fifth module is configured to use the first flow rate as the second flow rate when the success rate is between the success rate lower limit and the success rate adjustment threshold. The system according to claim 34, wherein the flow rate adjustment module further includes: a sixth module, which is used to maintain the second flow rate at the upper flow rate limit for a first predetermined number of cycles, and When the success rate is greater than the success rate adjustment threshold, the upper limit of the flow rate and the second flow rate are increased according to a second predetermined value. The system according to claim 35, wherein, after the upper limit of the flow rate and the second flow rate are increased according to a second predetermined value, the flow rate adjustment module is also able to adjust the flow rate in the next period of the period. In the case where the success rate drops, the increased flow rate upper limit and the increased second flow rate are restored to the flow rate upper limit and the second flow rate, and the flow rate upper limit is not increased for a second predetermined number of cycles. The system according to claim 34, wherein the flow rate adjustment module is further configured to repeatedly perform the following steps in a plurality of cycles and the number of repeated operations reaches a preset number of times to determine the flow rate The upper limit is set incorrectly and a warning message is issued: Use the first module directly after using the third module; or Use the third module directly after using the first module. The system according to claim 33, wherein the flow rate adjustment module further includes: A module for generating a first token for loading the first data and a second token for loading the second data according to the second flow rate and the success rate in the cycle; The data loading module further includes: A module for extracting the first data in the first queue according to the first token and extracting the second data in the second queue according to the second token; and A module for loading the extracted first data and the extracted second data to the second data terminal. The system according to claim 38, wherein the device generates a first token for loading the first data and a first token for loading the first data according to the second flow rate and the success rate in the cycle. The second token module of the second data further includes: When the success rate is less than 100%, the success rate is multiplied by the second flow rate as the flow rate for loading the first data, and the second flow rate is subtracted from the loading of the first data. The difference between the flow rate of one data is used to obtain the flow rate at which the second data is loaded, the ratio of the flow rate at which the second data is loaded to the second flow rate is updated, and the first data is loaded according to the ratio A module for generating the first token according to the flow rate of loading the second data and generating the second token according to the flow rate of loading the second data; and When the success rate is equal to 100%, use the smallest value among the number of data in the second queue and the product of the ratio and the second flow rate as the loading of the second data The second flow rate is subtracted from the difference between the flow rate loaded with the second data to obtain the flow rate loaded with the first data, the ratio is updated, and the second flow rate is loaded according to the A flow rate of one data generates the first token and a module that generates the second token according to the flow rate of the loaded second data.

Images