TWI818188B - A method and a system for a data transmission method based on asynchronous response - Google Patents

Abstract

The present invention discloses a data transmission method based on asynchronous response, and the method includes: an extraction step that extracts data from a first data end in a cycle and stores the data; a flow speed adjustment step that confirms a second flow speed of the data transmitted in the cycle based on a load result feedback from a second data end in the previous cycle and a first flow speed of the data transmitted in the previous cycle; and a loading step that loads the data to the second data end in the cycle according to the second flow speed.

Description

Method and system for transmitting data based on asynchronous response

The present invention relates to the field of data transmission, and in particular to a method and system for transmitting data based on asynchronous response.

In the process of transmitting data from the data source to the data destination, an ETL (Extract-Transform-Load) system that can extract data, convert the format of the data, and load the data to the data destination is usually used. Since the ETL system requires recurring tasks that run on a fixed cycle (for example, processing batch data) and the ETL system does not have high requirements for real-time data, it is difficult to use traditional ETL systems in scenarios that require high real-time data. In the process of transmitting data, there is a need to adjust the flow rate of data transmission. A common method to adjust the flow rate is to use the system load of the data transmission system as a hard indicator for adjustment. For example, when the system load is higher than a certain threshold, the flow rate is disabled or reduced, and when the system load begins to decrease, the flow rate is restored. Another commonly used method to adjust the flow rate is the adaptive current limiting protection method. This method uses the system load of the data transmission system as the starting value of the controlled flow rate, which is determined by the current request response time of the data transmission system and the processed request rate. The currently allowed flow rate, allowing the data transfer system to be increased without being overwhelmed, rather than necessarily falling below a certain threshold.

In order to increase the flow rate as much as possible, asynchronous response mode can also be used, that is, the module responsible for data loading in the data transmission system only sends the data to the downstream data destination without waiting for the specific loading result. The data transmission system can The execution results of data loading are processed through another module. In other words, data loaded in one cycle may not necessarily be known in the current cycle whether the data is loaded successfully.

However, in the case of using asynchronous response mode, the above-mentioned first method uses the system load of the data transmission system as a hard indicator. If the system load exceeds the system load, data is not allowed to be transmitted, thus limiting the processing capabilities of the data transmission system and the data destination. , reducing the data transmission volume of the data transmission system; in addition, when the downstream data destination fails or expands or other events that affect processing capabilities, this method cannot sense and adjust the flow rate in a timely manner, which may cause the data transmission system to become idle or Abnormal situations such as massive data loss. In addition, for the second method mentioned above, this method adjusts the flow rate according to the processing capacity of the data transmission system. It cannot dynamically control the flow rate according to the processing capacity of the downstream data destination. It is not applicable to the data destination whose processing capacity is weaker than that of the data transmission system. ability situation.

One aspect of the present invention provides a method for transmitting data based on asynchronous response, including: an extraction step: extracting and storing the data from the first data terminal in a cycle; a flow rate adjustment step: based on the previous data in the cycle. The loading result fed back by the second data terminal in the cycle and the first flow rate of transmitting the data in the previous cycle determine the second flow rate of transmitting the data in the cycle; and the loading step: in the cycle Loading the data to the second data port according to the second flow rate.

Another aspect of the present invention provides a method for transmitting data based on asynchronous responses, including: an extraction step: extracting first data from a first data terminal in one cycle, and storing the first data in a first queue; Flow rate adjustment step: determining to transmit the data in the cycle based on the first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate for transmitting the data in the previous cycle. a second flow rate; and a loading step: loading the first data in the first queue and the second data in the second queue to the second data terminal according to the second flow rate in the cycle .

Another aspect of the present invention provides a system for transmitting data based on asynchronous response, including: a data extraction module, which is used to extract and store the data from the first data terminal in one cycle; a flow rate adjustment module, which For determining a second flow rate for transmitting the data in the cycle based on the loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate for transmitting the data in the previous cycle. flow rate; and a data loading module configured to load the data to the second data port according to the second flow rate during the cycle.

Another aspect of the present invention provides a system for transmitting data based on asynchronous response, including: a data extraction module, which is used to extract the first data from the first data terminal in one cycle, and store the first data in In the first queue; a flow rate adjustment module configured to determine the first flow rate based on the first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate for transmitting the data in the previous cycle. a second flow rate for transmitting the data in the period; and a data loading module configured to load the first data and the second data in the first queue according to the second flow rate in the period. The second data in the queue is loaded into the second data terminal.

Yet another aspect of the present invention provides a computer-readable medium having computer-readable instructions stored thereon. The computer-readable instructions, when executed by a computer, can perform the method according to the embodiment of the present invention.

Embodiments of the present invention can automatically adjust the flow rate of the data transmission system according to the feedback loading results in the case of asynchronous responses, thereby improving data as much as possible on the premise of ensuring that the data destination is not overwhelmed and reduces data loss. The throughput of the transmission system.

The above description is only an overview of the technical solution of the present invention. In order to have a clearer understanding of the technical means of the present invention, it can be implemented according to the content of the description, and in order to make the above and other objects, features and advantages of the present invention more obvious and understandable. , the specific embodiments of the present invention are listed below.

The principles and spirit of the invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are only provided to enable those skilled in the art to better understand and implement the present invention, but are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Figure 1 shows a schematic diagram of a data transmission system according to an embodiment of the present invention.

As shown in Figure 1, the system includes five modules: data extraction module, data loading module, asynchronous response processing module, timing scanning module and flow rate adjustment module. The system also includes a data queue to be loaded, a data queue to be reloaded, and a data map to be confirmed.

The data extraction module can extract data from the data source, convert the data into a format and store it in the data queue to be loaded. If the queue is full, it will wait. In this article, one data can represent a business, and the size of each data can be different.

The data loading module can retrieve data from the data queue to be loaded, and determine whether the data can currently be loaded to the data destination through the flow rate adjustment module. The data loading module may also include a normal data loading module and a failed data loading module, which are respectively used to load data in the data queue to be loaded and the data queue to be reloaded. Each time data is loaded, the data loading module can simultaneously store the data in the data map to be confirmed. In some embodiments, the data to be confirmed Map is a data storage structure that can be used to temporarily store data transferred from the data loading module. The Key field in the data Map to be confirmed stores the identity of the data, and the Value field stores the value of the data.

The asynchronous response processing module can process the processing result fed back by the data destination (i.e., the loading result of the loaded data, which is success or failure), and count the amount of successful or failed data. For data whose processing result is successful, the asynchronous response processing module can directly delete the data from the data to be confirmed Map. For data whose processing result is failed, the asynchronous response processing module can send the data with failed loading from the data map to be confirmed to the data queue to be reloaded to be loaded by the data loading module. If the number of times a certain data load fails exceeds a predetermined number of times (that is, the upper limit of reload times), the data is discarded.

The scheduled scanning module can scan the data map to be confirmed to count the data that has not received the processing result of the data destination for more than a certain period of time (that is, the timeout data for which no response has been received), and send the data from the data map to be confirmed. to the data queue to be reloaded. The timing scanning module can also calculate the statistical data loading success rate in the current cycle by scanning the processing results from the data destination stored in the asynchronous response processing module, and send the success rate to the flow rate adjustment module.

The flow rate adjustment module can achieve flow rate control by adjusting the TPS (Transaction Per Second, business processing volume per second, such as the number of transactions processed per second) of the data transmission system. First, you can set a maximum TPS value under the most ideal situation as the upper limit of the flow rate, and increase or decrease the flow rate (TPS value) of the data transmission system according to the success rate of data loading in the current cycle, thereby realizing the flow rate adjustment of the system.

Figure 2 shows the methods that the data extraction module can use to extract data. As shown in Figure 2, the method includes the following steps:

(1) The data extraction module extracts data from the data source and converts the data into a unified format;

(2) If the queue to be loaded is not full at this time, the converted format data will be directly stored in it, and then step (1) will continue, otherwise step (3) will be executed;

(3) If the queue to be loaded is full at this time, the number of transmission failures will be accumulated once for the current data. If the number of transmission failures does not exceed the upper limit of the number of transmission failures, the data extraction module will sleep for a predetermined time and re-execute step (2), otherwise Discard the data and start step (1) again.

Figure 3 shows a sequence diagram of a method for loading data that can be used by the data loading module. As shown in Figure 3, the data loading module can obtain the converted format data from the data queue to be loaded. After obtaining the data to be loaded, in order to ensure that the flow rate of the current data transmission system does not exceed the allowed flow rate upper limit, the data loading module needs to apply to the flow rate adjustment module for a token that allows loading of data. Only after obtaining the token , the data to be loaded can be sent to the data destination.

When the flow rate adjustment module receives an application for a token, it checks whether there are still available tokens. If there are no available tokens, it suspends the application and waits for new tokens to be generated. The flow rate adjustment module can generate or clear tokens periodically, for example, clear the remaining tokens at the end of each cycle, and generate a number of tokens corresponding to the flow rate upper limit allowed by the current system (for example, the flow rate upper limit is 1000, which means for example that up to 1000 data can be processed per second (for example, 1000 transactions), then the flow rate adjustment module can generate 1000 tokens to allow 1000 data to be processed). The upper limit of the flow rate is updated based on the data loading status of the data destination. By modifying the number of tokens allowed in a cycle, the system flow rate can be dynamically adjusted.

Figure 4 shows a timing diagram of a method for adjusting flow rate according to an embodiment of the invention. The method involves an asynchronous response processing module, a timing scanning module and a flow rate adjustment module. The operations that this method can perform include statistics on the processing results of the data destination, statistics on timeout unanswered data, calculation of the loading success rate in the current cycle, and updating the upper limit of the flow rate allowed by the system.

The asynchronous response processing module is used to receive the loading results fed back by the data destination and process the corresponding data (that is, delete the successfully loaded data from the data map to be confirmed, or delete the loaded data that failed or no response was received) The timeout data is sent to the data queue to be reloaded), and at the same time, the value of the load success counter or load failure counter is updated according to the feedback load result.

The scheduled scanning module can scan in each cycle and calculate the success rate of loading data by the data destination in the current cycle (this data is the data that should be loaded by the data destination in the previous cycle, which can also be called historical data). This triggers the flow rate adjustment module to adjust the flow rate. For calculating the success rate, the scheduled scan module can first count the amount of all timeout data that has not received a response in the current cycle (for example, stored in the timeout data counter), and then obtain the values of the data loading success counter and loading failure counter at the current moment. , calculate the success rate according to the following formula:

Among them, S is the success rate of data loading in the current cycle, Cs and Cf are the values of the current loading success counter and loading failure counter respectively, and Ct is the value of the timeout data counter for which no response has been received. Clear the value of each counter immediately after the S calculation is completed to ensure that there will be no interference with the calculation of the next cycle.

This method of adjusting flow rate can be summarized as follows:

(1) Configure the acceptable flow rate upper limit Tmax according to the processing capability of the data destination;

(2) If the success rate of data loading fed back from the data destination in the current cycle is lower than the acceptable lower limit of success rate, the flow rate of the data transmission system is reduced according to a fixed step size. If the flow rate is lower than the lower limit of the flow rate, Then the lower limit of the flow rate is used as the actual flow rate (the number of corresponding tokens is generated based on the determined flow rate value). If the flow rate before the reduction has reached the acceptable lower limit of the flow rate, the system flow rate will no longer be reduced, and manual intervention is required for investigation;

(3) If the success rate of data loading in the current cycle is greater than the success rate adjustment threshold (which is greater than the lower limit of the success rate) and the current system flow rate is less than Tmax, then the flow rate can be increased according to a fixed step size. If the flow rate is higher than the upper limit of the flow rate , then the upper limit of the flow rate is regarded as the actual flow rate (the number of corresponding tokens is generated based on the determined flow rate value). If the flow rate before the increase has reached Tmax, the system flow rate value will no longer be increased;

(4) If the success rate is greater than the lower limit of the success rate and less than the success rate adjustment threshold, and the flow rate is less than Tmax, it is considered that both the data transmission system and the downstream system (i.e., the data destination) are in a stable state, and there is no need to adjust the flow rate of the data transmission system. ;

(5) If the flow rate of the data transmission system is maintained at Tmax, and the calculated success rate is always greater than or equal to the success rate adjustment threshold, try to increase Tmax; if the success rate drops immediately after increasing Tmax, restore the previous Tmax value, and in a certain period No more attempts to increase Tmax will be made during the day (for example, on the same day);

(6) If the data destination is expanded, Tmax can be manually modified to the upper limit of the flow rate that is compatible with the expanded processing capacity;

(7) If the calculated success rate is greater than or equal to the success rate adjustment threshold and the data transmission system increases the flow rate, the calculated success rate is lower than the lower limit of the success rate, causing the data transmission system to decrease the flow rate, and first increase the flow rate and then decrease it. If the flow rate process continues multiple times, this may mean that the calculated Tmax does not meet the current status and requires manual intervention. For example, the Tmax value can be manually reduced.

An embodiment of the method for realizing the above-mentioned adjustment of flow rate will be described below. First, define each parameter, its meaning is as follows:

Tmax: Under ideal conditions, the maximum TPS (i.e., the upper limit of flow rate) that the data destination can carry, which can be manually configured and modified by system operation and maintenance personnel;

Tmin: The lowest TPS acceptable by the data destination. If it is lower than this value, manual intervention is required;

Smin: The minimum success rate of data loading allowed by the data destination;

Sadp: The lowest acceptable success rate when TPS is increased (i.e., the success rate adjustment threshold). Only when the current success rate is greater than or equal to this value can you try to increase the value of TPS;

st: TPS adjustment step size, that is, the value of a single upward or downward adjustment of TPS;

Tcur: The TPS value currently allowed by the data destination. Tcur can be configured arbitrarily when the data transmission system is started, so that Tcur is not greater than Tmax.

In addition, this article defines a continuous TPS increase and decrease or a continuous TPS decrease and increase as a oscillation process. If the oscillation continues, it means that the Tmax value does not match the current status of the data destination, and the operation and maintenance personnel need to be reminded. Check and confirm, and modify the value of Tmax if necessary.

Based on this, the individual adjustment operations can be recorded, for example, using the flag UD:

UD=00: Unadjusted; UD=10: Increase TPS; UD=01: Decrease TPS

In addition, use UDp to record the last flow rate adjustment operation. For example, you can calculate UD|UDp (or operation) to indicate whether oscillation occurs (for example, UD|UDp=11, indicating that a oscillation occurred), use Ccon to record the number of continuous oscillations, and use Mcon to indicate the maximum acceptable number of continuous oscillations. .

Based on the above definition, the specific implementation method of adjusting the flow rate is as follows:

● When S≥Sadp and Tcur <Tmax, then UD=10, then update Tcur+=st and calculate UD|UDp:

If UD|UDp=11, modify Ccon+=1, UDp=10;

If UD|UDp!=11, modify Ccon=0 and UDp=10.

●When S≥Sadp and Tcur =Tmax, Tcur remains unchanged, modify Ccon=0, UDp=00

●When S≤Smin and Tcur>Tmin, then UD=01, update Tcur-=st, and calculate UD|UDp:

If UD|UDp=11, modify Ccon+=1, UDp=01;

If UD|UDp!=11, modify Ccon=0 and UDp=01.

●When S≤Smin and Tcur =Tmin, Tcur remains unchanged, modify Ccon=0, UDp=00, and an alarm needs to be issued to prompt manual intervention;

●When Smin＜S＜Sadp, Tcur remains unchanged, modify Ccon=0, UDp=00.

Each time the flow rate adjustment algorithm is executed, the corresponding number of tokens is first generated based on the updated TPS value allowed by the data destination, and Ccon is checked at the same time. If Ccon≥Mcon, it means that the number of continuous oscillations of the data transmission system exceeds the upper limit, that is, the value of Tmax is not suitable for the current data destination. Therefore, the data transmission system will issue an alarm to remind operation and maintenance personnel to intervene.

In order to reduce abnormal data (including data that failed to load and timeout data that did not receive a response), the asynchronous response processing module or the scheduled scanning module will send the abnormal data to the data queue to be reloaded, and load it through the data Failed data reloading in the module The module reloads the abnormal data.

Figure 5 shows a flowchart of a method for processing abnormal data according to an embodiment of the present invention. The methods used to handle abnormal data are as follows:

(1) The asynchronous response processing module makes a judgment after receiving the processing result. When the result is successful, the data is directly removed from the data map to be confirmed, and the process ends; otherwise, proceed to step (2);

(2) When the result is not successful, update the number of failures. If the number of failures exceeds the upper limit, directly remove the data from the data map to be confirmed and report an exception (such as issuing an alarm), otherwise proceed to step (3);

(3) If the data queue to be reloaded is not full at this time, the data will be directly transferred to the data queue to be reloaded. Otherwise, it will be recorded as a transfer failure, and the number of failures will be directly updated, and step (4) will be continued;

(4) If the number of data failures exceeds the upper limit of failures, the data will be discarded directly and an exception will be reported.

Otherwise, it will continue to be stored in the data map to be confirmed, wait for a period of time and then enter step (3).

As can be seen from Figure 1 and Figure 5, the method of processing abnormal data can also reduce the impact of abnormal data on the system through load isolation, that is, the normal data loading module is used to process the data in the data queue to be loaded, and the use fails The data loading module is responsible for processing the data in the data queue to be reloaded, and the two modules execute in parallel. Based on this, the TPS value Tcur allowed by the current system or data destination calculated by the flow rate adjustment module can be subdivided into normal data loading TPS (denoted as Tnor) and abnormal data loading TPS (denoted as Tuno). Different loading modules Request respective load tokens to achieve load isolation of normal data and abnormal data. The specific method is as follows:

(1) If the success rate is not 100%, multiply the success rate and the current flow rate as the flow rate for normal data loading, use the remaining part of the current flow rate as the flow rate for abnormal data loading, and update the flow rate for abnormal data at the current flow rate proportion for the data transmission system to use in subsequent cycles. The specific implementation method can be, for example:

When S≠100%, Tnor= Tcur*S, Tuno=Tcur*(1-S); update R _P = Tuno/ Tcur.

(2) If the success rate is 100%, the flow rate of abnormal data loading is the minimum value of the number of data in the data queue to be reloaded and the product of the above proportion and the current flow rate. Therefore, the flow rate for normal data loading is the difference between the current flow rate and the flow rate for abnormal data loading. The proportion of the flow rate of abnormal data in the current flow rate can be updated for use by the data transmission system in subsequent cycles. The specific implementation method can be, for example:

When S=100%, Tuno=min(L _w R _P * Tcur), Tnor= Tcur-Tuno, update R _P = Tuno/ Tcur.

Among them, S is the data loading success rate in the current cycle, Rp is the proportion of abnormal data loading TPS in the total system TPS in the previous cycle, and Lu is the number of data in the data queue to be reloaded (for example, the number of transactions ).

Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., as well as conventional procedural programming. Language—such as "C" or a similar programming language. The program code may execute entirely on the user's computing device, partly on the user's computing device, partly on the user's computing device, or entirely on a remote computing device or server. In situations involving remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., utilizing an Internet service provider to connect via the Internet).

Furthermore, although the operations of the methods of the present invention are depicted in a particular order in the drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be broken down into multiple steps for execution.

It should be noted that although several software devices/modules and sub-devices/modules for implementing the above method are mentioned in the above detailed description, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more devices described above may be embodied in one device/module. Conversely, the features and functions of one device/module described above may be further divided into being embodied by multiple devices/modules.

Although the spirit and principles of the invention have been described with reference to a number of specific embodiments, it should be understood that the invention is not limited to the specific embodiments disclosed, nor does the division into aspects mean that features in these aspects cannot be combined. Benefit, this division is only for convenience of expression. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

without.

[Fig. 1] shows a schematic diagram of a data transmission system according to an embodiment of the present invention. [Fig. 2] A flowchart showing a method for extracting data according to an embodiment of the present invention. [Fig. 3] A sequence diagram showing a method for loading data according to an embodiment of the present invention. [Fig. 4] A timing chart showing a method for adjusting a flow rate according to an embodiment of the present invention. [Fig. 5] A flowchart showing a method for processing abnormal data according to an embodiment of the present invention.

Claims (31)

A method of transmitting data based on asynchronous response, including: an extraction step: extracting and storing the data from a first data terminal in one cycle; a flow rate adjustment step: based on the second data in the previous cycle of the cycle The loading result fed back by the terminal and the first flow rate for transmitting the data in the previous cycle determine the second flow rate for transmitting the data in the cycle; and the loading step: in the cycle according to the second flow rate The data is loaded to the second data terminal at a flow rate, where the loading result is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, The number of historical data of loading failures fed back by the second data terminal in the cycle, wherein the flow rate adjustment step further includes: counting whether the second data has not been received after a predetermined time in the previous cycle The number of timeout historical data fed back by the end, and the success rate is calculated through the following formula:

Where, S is the success rate, C _s is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, and C _f is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle. The number of historical data of loading failure fed back by the second data terminal, C _t is the number of timeout historical data fed back by the second data terminal that has not been received after the predetermined time in the previous cycle, wherein, the The loading step includes: generating a token for loading the data according to the second flow rate and the success rate in the cycle; extracting the data in the queue according to the token; and transferring the extracted queue The data in is loaded into the second data port. As in the method described in claim 1, the extracting step includes: (a) extracting the data from the first data terminal and converting the data according to a predetermined format; (b) judging whether the queue is full, and if not is full, then store the converted data in the queue and enter step (a), otherwise enter step (c); (c) accumulate one number of transmission failures for the converted data, if the number of transmission failures has not been If the upper limit of the number of failed transmissions is exceeded, step (b) is entered again after waiting for a predetermined time; otherwise, the converted data is discarded and step (a) is entered. The method according to claim 1, wherein the flow rate adjustment step further includes: (a) if the success rate is less than or equal to a success rate lower limit and the first flow rate is higher than the flow rate lower limit, reducing the flow rate by a first predetermined value The first flow rate is used to obtain the second flow rate. If the reduced first flow rate is less than the flow rate lower limit, the flow rate lower limit is used as the second flow rate; (b) If the success rate is less than or equal to the If the success rate is lower limit and the first flow rate is equal to the flow rate lower limit, then the flow rate lower limit is used as the second flow rate; (c) If the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the flow rate If the upper limit is reached, increase the first flow rate according to a second predetermined value to obtain the second flow rate. If the increased first flow rate is greater than the upper limit of the flow rate, then use the upper limit of the flow rate as the second flow rate; (d ) If the success rate is higher than the success rate adjustment threshold and the first flow rate reaches the flow rate upper limit, then use the flow rate upper limit as the second flow rate; or (e) if the success rate is within the success rate If the rate is between the lower limit and the success rate adjustment threshold, the first flow rate is used as the second flow. The method of claim 3, wherein the flow rate adjustment step further includes: when the second flow rate remains at the flow rate upper limit within a first predetermined number of periods, and the success rate is greater than the success rate When adjusting the threshold, the flow rate upper limit and the second flow rate are increased according to a second predetermined value. The method of claim 4, wherein after the step of increasing the flow rate upper limit and the second flow rate according to a second predetermined value, if the success rate decreases in the next cycle of the cycle, Then, the increased flow rate upper limit and the increased second flow rate are restored to the flow rate upper limit and the second flow rate, and the flow rate upper limit is no longer increased within a second predetermined number of periods. The method of claim 3, wherein the method further includes: if the following steps are repeatedly performed in multiple cycles and the number of times the repeated operations are performed reaches a preset number of times, determining that the flow rate upper limit is set incorrectly and Issue an alarm message: perform step (a) directly after performing step (c), or perform step (c) directly after performing step (a). A method of transmitting data based on asynchronous response, including: an extraction step: extracting first data from a first data terminal in a cycle, and storing the first data in a first queue; a flow rate adjustment step: based on the The first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate of transmitting the data in the previous cycle determine the second flow rate of transmitting the data in the cycle; and loading Step: Load the first data in the first queue and the second data in the second queue to the second data terminal according to the second flow rate in the cycle, wherein the first The loading result is the number of historical data with successful loading fed back by the second data terminal in the previous cycle, and the number of historical data with failed loading fed back by the second data terminal in the previous cycle. , wherein the flow rate adjustment step further includes: counting the number of timeout historical data that has not received feedback from the second data terminal for more than a predetermined time in the previous cycle, and calculating the number of timeout historical data in the previous cycle. The success rate of loading historical data fed back by the second data terminal is used to determine the second flow rate, where the success rate is calculated by the following formula:

Where, S is the success rate, C _s is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, and C _f is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle. The number of historical data of loading failure fed back by the second data terminal, C _t is the number of timeout historical data fed back by the second data terminal that has not been received after the predetermined time in the previous cycle, wherein, the The loading step includes: (a) generating a first token for loading the first data and a second token for loading the second data according to the second flow rate and the success rate in the cycle. card; (b) extracting the first data in the first queue according to the first token and extracting the second data in the second queue according to the second token; and (c) extracting the extracted first data and the extracted second data is loaded into the second data port. The method according to claim 7, further comprising: storing the first data and the second data in a data structure as candidate data. The method of claim 8, further comprising: determining a candidate in the data structure based on a second loading result fed back by the second data terminal in the cycle or at least one cycle after the cycle. Whether the data is loaded successfully, if successful, delete the candidate data in the data structure, if not, determine whether the number of times the candidate data has been loaded exceeds a predetermined number of times, if it exceeds the predetermined number of times, then delete the candidate data in the data structure. Delete the candidate data from the above data structure, if it does not exceed the the predetermined times, then the candidate data is sent to the second queue as the second data. The method of claim 7, the extracting step includes: (a) extracting the first data from the first data terminal and converting the first data in a predetermined format; (b) determining the first queue Whether it is full, if not, store the converted first data in the first queue and enter step (a), otherwise enter step (c); (c) for the converted first data The number of transmission failures is accumulated once. If the number of transmission failures does not exceed the upper limit of the number of transmission failures, step (b) is entered again after waiting for a predetermined time. Otherwise, the converted first data is discarded and step (a) is entered. The method of claim 7, wherein the flow rate adjustment step further includes: (a) if the success rate is less than or equal to a success rate lower limit and the first flow rate is higher than the flow rate lower limit, reducing the flow rate by a first predetermined value The first flow rate is used to obtain the second flow rate. If the reduced first flow rate is less than the flow rate lower limit, the flow rate lower limit is used as the second flow rate; (b) If the success rate is less than or equal to the If the success rate is lower limit and the first flow rate is equal to the flow rate lower limit, then the flow rate lower limit is used as the second flow rate; (c) If the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the flow rate If the upper limit is reached, increase the first flow rate according to a second predetermined value to obtain the second flow rate. If the increased first flow rate is greater than the upper limit of the flow rate, then use the upper limit of the flow rate as the second flow rate; (d ) If the success rate is higher than the success rate adjustment threshold and the first flow rate reaches the flow rate upper limit, then use the flow rate upper limit as the second flow rate; or (e) if the success rate is within the success rate If the flow rate is between the lower limit of the flow rate and the success rate adjustment threshold, the first flow rate is used as the second flow rate. The method according to claim 11, wherein the flow rate adjustment step further includes: when the When the second flow rate remains at the upper flow rate limit within a first predetermined number of periods, and the success rate is greater than the success rate adjustment threshold, the upper flow rate limit and the second flow rate are increased according to a second predetermined value. The method of claim 12, wherein after the step of increasing the flow rate upper limit and the second flow rate according to a second predetermined value, if the success rate decreases in the next cycle of the cycle, Then, the increased flow rate upper limit and the increased second flow rate are restored to the flow rate upper limit and the second flow rate, and the flow rate upper limit is no longer increased within a second predetermined number of periods. The method of claim 11, wherein the method further includes: if the following steps are repeatedly performed in multiple cycles and the number of times the repeated operations are performed reaches a preset number of times, determining that the flow rate upper limit is set incorrectly and Issue an alarm message: perform step (a) directly after performing step (c), or perform step (c) directly after performing step (a). The method of claim 7, wherein the step (a) includes: when the success rate is less than 100%, multiplying the success rate and the second flow rate to load the first data flow rate, subtract the difference between the flow rate for loading the first data from the second flow rate to obtain the flow rate for loading the second data, update the flow rate for loading the second data and the flow rate for loading the second data. The ratio of the second flow rate, and the first token is generated according to the flow rate of loading the first data and the second token is generated according to the flow rate of loading the second data; or when the When the success rate is equal to 100%, the minimum value of the number of data in the second queue, the product of the ratio and the second flow rate is used as the flow rate for loading the second data, and Subtract the difference between the flow rate loading the second data from the second flow rate to obtain the flow rate loading the first data, update the ratio, and generate a flow rate based on the flow rate loading the first data The first token and the second token are generated according to the flow rate of loading the second data. A computer-readable medium having computer-readable instructions stored thereon, the computer The readable instructions, when executed by a computer, can perform the method as claimed in any one of claims 1-15. A system for transmitting data based on asynchronous response, including: a data extraction module, which is used to extract and store the data from the first data terminal in a cycle; a flow rate adjustment module, which is used based on the cycle The loading result fed back by the second data terminal in the previous cycle and the first flow rate of transmitting the data in the previous cycle determine the second flow rate of transmitting the data in the cycle; and the data loading module , which is used to load the data to the second data terminal according to the second flow rate in the cycle, wherein the loading result is fed back by the second data terminal in the previous cycle The number of historical data with successful loading, and the number of historical data with failed loading fed back by the second data terminal in the previous cycle, wherein the system also includes a timing scanning module for counting The number of timeout historical data fed back by the second data terminal that has not been received after a predetermined time in the previous cycle is calculated. The number of loading historical data fed back by the second data terminal in the previous cycle is calculated. The success rate is sent to the flow rate adjustment module, where the success rate is calculated by the following formula:

Wherein, S is the success rate, C _s is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, and C _f5 is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle. The number of historical data of loading failure fed back by the second data terminal, C _t is the number of timeout historical data fed back by the second data terminal that has not been received after the predetermined time in the previous cycle, wherein, the The flow rate adjustment module includes: a module for generating a token for loading the data according to the second flow rate and the success rate in the cycle; the data loading module includes: a module for generating the token according to the second flow rate and the success rate. a module for extracting data from the token queue; and a module for loading the extracted data from the queue into the second data terminal. As in the system described in claim 17_, the data extraction module can also perform the following steps: (a) extract the data from the first data terminal and convert the data according to a predetermined format; (b) determine Whether the queue is full, if not, store the converted data in the queue and enter step (a), otherwise enter step (c); (c) accumulate the number of transmission failures for the converted data, If the number of failed transmissions does not exceed the upper limit of the number of failed transmissions, step (b) will be entered again after waiting for a predetermined time; otherwise, the converted data will be discarded and step (a) will be entered. The system according to claim 17, wherein the flow rate adjustment module further includes: a first module, which is used when the success rate is less than or equal to the lower limit of the success rate and the first flow rate is higher than the lower limit of the flow rate. , reducing the first flow rate according to a first predetermined value to obtain the second flow rate, and when the reduced first flow rate is less than the lower limit of the flow rate, the lower limit of the flow rate is used as the second flow rate; A second module, which is used to use the lower limit of flow rate as the second flow rate when the success rate is less than or equal to the lower limit of success rate and the first flow rate is equal to the lower limit of flow rate; a third module, which For when the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the flow rate upper limit, increase the first flow rate according to a second predetermined value to obtain the second flow rate. After the increase, the first flow rate is When the flow rate is greater than the upper limit of the flow rate, the upper limit of the flow rate is used as the second flow rate; a fourth module is used when the success rate is higher than the success rate adjustment threshold and the first flow rate reaches In the case of an upper limit of the flow rate, the upper limit of the flow rate is used as the second flow rate; or a fifth module is used when the success rate is between the lower limit of the success rate and the adjustment threshold of the success rate. , taking the first flow rate as the second flow rate. The system of claim 19, wherein the flow rate adjustment module further includes: a sixth module configured to maintain the upper flow rate limit when the second flow rate remains at the upper limit of the flow rate within a first predetermined number of periods, and When the success rate is greater than the success rate adjustment threshold, the flow rate upper limit and the second flow rate are increased according to a second predetermined value. The system of claim 20, wherein after increasing the flow rate upper limit and the second flow rate according to a second predetermined value, the flow rate adjustment module can also succeed in the next cycle of the cycle. If the flow rate decreases, the increased flow rate upper limit and the increased second flow rate are restored to the flow rate upper limit and the second flow rate, and the flow rate upper limit is no longer increased within a second predetermined number of periods. The system according to claim 19, wherein the flow rate adjustment module is also used to repeatedly perform the following steps in multiple cycles and determine the flow rate when the number of times of the repeated operations reaches a preset number. The upper limit is set incorrectly and an alarm message is issued: use the first module directly after using the third module; or use the third module directly after using the first module. A system for transmitting data based on asynchronous response, including: a data extraction module, which is used to extract first data from a first data terminal in a cycle, and store the first data in a first queue; flow rate adjustment A module configured to determine, based on the first loading result fed back by the second data terminal in the previous cycle of the cycle and the first flow rate of the data transmitted in the previous cycle, the data transmitted in the cycle. a second flow rate of the data; and a data loading module configured to load the first data in the first queue and the second data in the second queue according to the second flow rate in the cycle. to the second data terminal, wherein the first loading result is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, The number of historical data of loading failures fed back by the second data terminal, wherein the system also includes a timing scanning module for counting whether the second data has not been received after a predetermined time in the previous cycle. The number of timeout historical data fed back by the data terminal, and the success rate of loading historical data fed back by the second data terminal in the previous cycle is calculated, and the success rate is sent to the flow rate adjustment module. , where the success rate is calculated by the following formula:

Where, S is the success rate, C _s is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle, and C _f is the number of successfully loaded historical data fed back by the second data terminal in the previous cycle. The number of historical data of loading failure fed back by the second data terminal, C _t25 is the number of timeout historical data that has not been received from the second data terminal after exceeding the predetermined time in the previous cycle, wherein, the The flow rate adjustment module includes: a first token for loading the first data and a second token for loading the second data according to the second flow rate and the success rate in the cycle. Token module; the data loading module includes: a module for extracting the first data in the first queue according to the first token and extracting the second data in the second queue according to the second token. a module; and a module for loading the extracted first data and the extracted second data into the second data port. The system according to claim 23, wherein the data loading module is further configured to store the first data and the second data in a data structure as candidate data. The system according to claim 24, the system further includes: an asynchronous response processing module, which is used to feedback the second data signal based on the second data terminal in the period or at least one period after the period. The loading result determines whether the candidate data in the data structure is loaded successfully. If successful, the candidate data is deleted from the data structure. If unsuccessful, it is determined whether the number of times the candidate data has been loaded exceeds a predetermined number of times. If the predetermined number of times is exceeded, the candidate data is deleted from the data structure; if the predetermined number of times is not exceeded, the candidate data is sent to the second queue as the second data. As in the system described in claim 23, the data extraction module can also perform the following steps: (a) extract the first data from the first data terminal and convert the first data according to a predetermined format; ( b) Determine whether the first queue is full. If not, store the converted first data in the first queue and enter step (a). Otherwise, enter step (c); (c) For the The converted first data accumulates a number of transmission failures. If the number of transmission failures does not exceed the upper limit of the number of transmission failures, then wait for a predetermined time and then re-enter step (b), otherwise discard the converted first data and enter Step (a). The system according to claim 23, wherein the flow rate adjustment module further includes: a first module, which is used when the success rate is less than or equal to the lower limit of the success rate and the first flow rate is higher than the lower limit of the flow rate. , reducing the first flow rate according to a first predetermined value to obtain the second flow rate, and when the reduced first flow rate is less than the lower limit of the flow rate, the lower limit of the flow rate is used as the second flow rate; Two modules, which are used to use the lower limit of the flow rate as the second flow rate when the success rate is less than or equal to the lower limit of the success rate and the first flow rate is equal to the lower limit of the flow rate; A third module configured to increase the first flow rate by a second predetermined value to obtain the second flow rate when the success rate is higher than the success rate adjustment threshold and the flow rate is lower than the flow rate upper limit, When the increased first flow rate is greater than the upper limit of the flow rate, the upper limit of the flow rate is used as the second flow rate; a fourth module is used when the success rate is higher than the success rate adjustment threshold and the When the first flow rate reaches the upper limit of the flow rate, the upper limit of the flow rate is used as the second flow rate; or a fifth module is used when the success rate is between the lower limit of the success rate and the adjustment threshold of the success rate. In the case of time, the first flow rate is regarded as the second flow rate. The system of claim 27, wherein the flow rate adjustment module further includes: a sixth module configured to maintain the upper flow rate limit when the second flow rate is maintained at the upper limit of the flow rate within a first predetermined number of periods, and When the success rate is greater than the success rate adjustment threshold, the flow rate upper limit and the second flow rate are increased according to a second predetermined value. The system of claim 28, wherein after increasing the flow rate upper limit and the second flow rate according to the second predetermined value, the flow rate adjustment module can also adjust the flow rate in the next cycle of the cycle. In the case where the success rate decreases, the increased flow rate upper limit and the increased second flow rate are restored to the flow rate upper limit and the second flow rate, and the flow rate upper limit is no longer increased within a second predetermined number of periods. The system according to claim 27, wherein the flow rate adjustment module is also used to determine the flow rate when the following steps are repeatedly performed in multiple cycles and the number of times the repeated operations are performed reaches a preset number of times. The upper limit is set incorrectly and an alarm message is issued: use the first module directly after using the third module; or use the third module directly after using the first module. The system of claim 23, wherein the step of generating a first token for loading the first data and a first token for loading the first data according to the second flow rate and the success rate in the cycle. describe The module for the second token of the second data further includes: when the success rate is less than 100%, multiplying the success rate and the second flow rate to serve as the flow rate for loading the first data, Subtract the difference between the flow rate loading the first data from the second flow rate to obtain the flow rate loading the second data, and update the flow rate loading the second data and the second flow rate. The ratio of the flow rate, and a module for generating the first token according to the flow rate for loading the first data and generating the second token according to the flow rate for loading the second data; and When the success rate is equal to 100%, the minimum value of the number of data in the second queue and the product of the ratio and the second flow rate is used as the flow rate for loading the second data, Subtract the difference between the flow rates loading the second data from the second flow rate to obtain the flow rate loading the first data, update the ratio, and calculate the flow rate based on the flow rate loading the first data. A module that generates the first token based on the flow rate and generates the second token based on the flow rate loading the second data.