TW201828059A - Message flow rate control method, apparatus and related system capable of reducing the amount of message discarding and increasing stability and processing efficiency of the overall data processing system - Google Patents

Abstract

The present invention discloses a message flow rate control method, an apparatus and a related system. The present invention performs sampling according to message receiving amount of each processing node. Once the actual message receiving amount of a processing node is not within a range of the corresponding predetermined flow rate threshold value, a corresponding flow rate control request is generated, and the message transmitting speed of an upstream data node corresponding to the processing node is adjusted according to the flow rate control request, so as to realize dynamic control of message flow rate between any two adjacent upstream and downstream data nodes. Compared with the prior art, the information transmitting rate of the data source is adjusted in particular according to the number of unprocessed messages of the last processing node, the present invention has higher control accuracy, and no message backlog or message idle occurs in any processing node, thereby reducing the amount of message discarding and improving the stability and processing efficiency of the overall data processing system.

Description

 Message flow control method, device and related system  

The invention relates to the field of distributed real-time computing technology, in particular to a message flow control method, device and related system.

Most of the existing data processing systems adopt a pipelined processing method. As shown in the topology diagram shown in Figure 1, after the message is sent from the data source S, it flows gradually downstream according to the logical relationship between the processing nodes; the total number of processing nodes and each The number of nodes on the pipeline depends on the actual data processing logic. Figure 1 only takes the three processing nodes A, B, and C as an example. The arrows indicate the direction of the message flow.

Under normal circumstances, the message sending rate of the upstream source or processing node should match the message processing rate of the adjacent downstream processing node; if the upstream node has a message traffic peak and/or the downstream processing node has insufficient processing capacity, As a result, the upstream message transmission rate is greater than the downstream message processing rate, which causes a large amount of message backlog, processor pressure increase, and even some messages are discarded due to processing timeout, and the actual effective processing capability of the system is degraded. In particular, real-time computing systems such as Storm, Jstorm, and Spark streaming, the message traffic is constantly changing. When the traffic peaks, each node is required to process the received message as quickly as possible. Otherwise, it will affect other normal running nodes or Calculation task.

In the related art, the following flow control strategy is adopted: a flow control value R0 is configured in the data source to represent the number of outstanding messages allowed by the system, and during the operation of the system, the actual uncompleted message is determined according to the message processing condition of the last processing node in the pipeline. The quantity R, if R>R0, indicates that the message traffic is too large, then the control data source reduces the number of message transmissions; conversely, if R<R0, indicating that the message traffic is too small, the control data source increases the number of message transmissions. The above traffic control policy has serious drawbacks in practical applications; for example, if the flow control value is configured too high and the message discarding amount is too high, the system instability will still occur, but when the flow control value is configured too low, It also causes the computing resources to be idle.

In order to solve the above technical problem, the present invention discloses a message flow control method, apparatus and related system.

A first aspect of the present invention provides a message flow control method, the method being applied to a data processing system, the data processing system including a data source and a plurality of processing nodes; the method comprising: separately processing a message for each processing node Receiving, sampling, and determining, according to the sampling result, whether the actual message receiving amount of the corresponding processing node is within a preset traffic threshold interval; determining the actual message receiving amount of the first processing node according to the sampling result corresponding to the first processing node When not in the preset traffic threshold interval, the corresponding flow control request is generated; and the message sending rate of the upstream data node adjacent to the first processing node is adjusted according to the flow control request; wherein the upstream data node may For the data source or processing node.

In combination with the first aspect, in a first feasible implementation manner of the first aspect, when determining, according to the sampling result corresponding to the first processing node, that the actual message receiving amount of the first processing node is not within a preset traffic threshold interval And generating a corresponding flow control request, including: generating a traffic reduction request when determining that the actual message receiving amount of the first processing node is greater than a preset maximum traffic threshold according to the sampling result corresponding to the first processing node; When the sampling result corresponding to the node determines that the actual message receiving amount of the first processing node is less than the preset minimum traffic threshold, a traffic increase request is generated.

In conjunction with the first possible implementation of the first aspect, in a second possible implementation manner of the first aspect, the sending of the information of the upstream data node adjacent to the first processing node is adjusted according to the flow control request The rate includes: when the flow control request is the traffic reduction request, determining a message sending delay time according to a message processing rate of the first processing node and an original message sending rate of the upstream data node; The sending delay time is superimposed with the original message sending rate to obtain a target message sending rate of the upstream data node.

In combination with the first possible implementation manner of the first aspect, in a third feasible implementation manner of the first aspect, the sending of the information of the upstream data node adjacent to the first processing node is adjusted according to the flow control request The rate includes: when the flow control request is the traffic increase request, gradually increasing the message sending rate of the upstream data node to a preset normal rate according to the preset rate adjustment step.

Combining the first aspect, or the first feasible embodiment of the first aspect, or the second feasible implementation of the first aspect, or the third feasible implementation of the first aspect, in the first aspect In a fourth feasible implementation manner, before adjusting a message sending rate of an upstream data node adjacent to the first processing node according to the flow control request, the method further includes: if the And determining, by the plurality of downstream processing nodes corresponding to the upstream data node, a corresponding flow control request, determining a change rate of the message sending rate caused by the flow control request corresponding to each downstream processing node to the upstream data node; determining a change rate of the message sending rate The target flow control request corresponding to the maximum value is used to adjust the message sending rate of the upstream data node according to the target flow control request.

Combining the first aspect, or the first feasible embodiment of the first aspect, or the second feasible implementation of the first aspect, or the third feasible implementation of the first aspect, in the first aspect In a fifth possible implementation, the method further includes: storing a flow control state of each processing node to a preset storage module; and storing the flow control state stored by the preset storage module according to a preset period Synchronize to the corresponding processing node.

Combining the first aspect, or the first feasible embodiment of the first aspect, or the second feasible implementation of the first aspect, or the third feasible implementation of the first aspect, in the first aspect In the sixth feasible implementation manner, determining, according to the sampling result, whether the actual message receiving amount of the corresponding processing node is within the preset traffic threshold interval, including: calculating, for any processing node, the absent collected in the preset sampling period. The ratio between the number of sampled values in the preset flow threshold interval and the total number of sampled values in the preset sampling period; if the ratio is greater than the preset threshold, determining the actual message receiving amount of the corresponding processing node Not within its preset traffic threshold interval.

A second aspect of the present invention provides a message flow control apparatus, the apparatus being applied to a data processing system, the data processing system including a data source and a plurality of processing nodes; the apparatus comprising: a trigger and a controller; The trigger is set in each processing node, and is used for sampling the message receiving amount of the processing node where it is located, and determining whether the actual message receiving amount of the processing node where the processing node is located is within the preset traffic threshold interval according to the sampling result, and When the actual message receiving amount of the processing node is not within the preset traffic threshold interval, a corresponding flow control request is generated; the controller is disposed in each data node having a downstream processing node, and is used according to the data node where the processing node is located The flow control request of the corresponding downstream processing node adjusts the message sending rate of the data node where the data is located; wherein the data node may be the data source or the processing node.

In combination with the second aspect, in the first feasible implementation manner of the second aspect, the corresponding flow control request is generated when the actual message receiving amount of the processing node at the processing node is not within the preset traffic threshold interval. The trigger is configured to generate a traffic reduction request when the actual message receiving amount of the processing node at the processing node is greater than the preset maximum traffic threshold; and generate traffic when the actual message receiving amount of the processing node at the processing node is greater than the preset minimum traffic threshold. Raise the request.

In combination with the first feasible implementation manner of the second aspect, in the second feasible implementation manner of the second aspect, in order to implement the flow control request according to the downstream processing node corresponding to the data node where the data node is located, the data node where the data node is located is adjusted. a message sending rate, the controller is configured to: when the flow control request is the traffic reduction request, determine a message according to a message processing rate of the downstream processing node, and an original message sending rate of the data node where the data node is located Delaying the time, and superimposing the message sending delay time and the original message sending rate to obtain a target message sending rate of the data node where the data node is located.

In combination with the first feasible implementation manner of the second aspect, in the third feasible implementation manner of the second aspect, in order to implement the flow control request according to the downstream processing node corresponding to the data node where the data node is located, the data node of the data node is adjusted. The message sending rate, the controller is configured to: when the flow control request is the traffic increase request, gradually increase the message sending rate of the data node where the data node is located to a preset normal rate according to the preset rate adjustment step.

In combination with the second aspect, or the second aspect of the first feasible implementation, or the second aspect of the second possible implementation, or the second aspect of the third possible implementation, in the second aspect In a fourth possible implementation, the device further includes: a first coordinator; the first coordinator is disposed in parallel with each processing node in the data processing system, and configured to acquire the traffic sent by the trigger Controlling the request, and forwarding the flow control request to a controller corresponding to the upstream data node; wherein, if the first coordinator acquires multiple flow control sent to a controller of the same upstream data node within a preset time period And the first coordinator determines, by each flow control request, a message transmission rate change caused by the same upstream data node, and sends a flow control request corresponding to the maximum value of the message transmission rate change to the same upstream The controller of the data node.

In combination with the second aspect, or the second aspect of the first feasible implementation, or the second aspect of the second possible implementation, or the second aspect of the third possible implementation, in the second aspect In a fifth possible implementation, the apparatus further includes: a second coordinator; the second coordinator is disposed in parallel with each processing node in the data processing system, and is configured to control a flow control state of each processing node. And storing the flow control state stored by the preset storage module to a corresponding processing node according to a preset period.

In combination with the second aspect, or the second aspect of the first feasible implementation, or the second aspect of the second possible implementation, or the second aspect of the third possible implementation, in the second aspect In the sixth feasible implementation manner, in order to determine whether the actual message receiving amount of the processing node where the processing node is located is within the preset traffic threshold interval according to the sampling result, the trigger is configured to: calculate the preset sampling period and collect The ratio of the number of sampled values that are not within the preset flow threshold range to the total number of sampled values in the preset sampling period, and determines that the processing node is located when the ratio is greater than a preset threshold. The actual message received is not within its preset traffic threshold interval.

The third party of the present invention provides a data processing system including a data source, a plurality of processing nodes, and any of the above-described message flow control devices.

According to the above technical solution, the embodiment of the present invention samples the received message amount of each processing node, and generates a corresponding flow control once the actual message receiving amount of a processing node is not within the corresponding preset traffic threshold interval. And requesting, according to the flow control request, adjusting a message sending rate of the upstream data node corresponding to the processing node, so that dynamic control of message traffic between any two adjacent upstream and downstream data nodes can be implemented, which is based on the prior art only The last method of processing the number of unprocessed messages of the node to adjust the message sending rate of the data source, the control precision of the present invention is higher, and any processing node does not have a message backlog or idle phenomenon, thereby reducing the amount of message discarding and improving The stability and processing efficiency of the entire data processing system.

510‧‧‧ upstream data node

512‧‧‧ controller

520‧‧‧ downstream data node

521‧‧‧ Trigger

A‧‧‧ Processing node

B‧‧‧ Processing node

C‧‧‧ Processing node

C_A‧‧‧ controller

C_S‧‧‧ controller

K‧‧‧ Coordinator

S‧‧‧Source

T_A‧‧‧ trigger

T_B‧‧‧ trigger

T_C‧‧‧ Coordination Service Module

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, for those skilled in the art, Other drawings may also be obtained from these drawings without paying for inventive labor.

1 is a topological structural diagram of a prior art data processing system; FIG. 2 is a flowchart of a message flow control method according to an embodiment of the present invention; FIG. 3 is a topological structural diagram of a data processing system according to an embodiment of the present invention; 4 is a signal flow diagram of a message flow control method based on the topology shown in FIG. 3 according to an embodiment of the present invention; FIG. 5 is a schematic diagram of a working principle of a message flow control apparatus according to an embodiment of the present invention; A topology diagram of another data processing system provided.

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. The following description of the drawings refers to the same or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of devices and methods consistent with aspects of the invention as detailed in the appended claims.

First, an embodiment of the message flow control method provided by the present invention will be described. The method can be applied to any pipelined data processing system including a data source and a plurality of processing nodes, such as an instant computing system such as Storm, Jstorm, Spark streaming, etc., to adjust the message flow between nodes in the pipeline to realize the data source and The message transmission rate of any processing node matches the message processing rate of the corresponding downstream processing node.

2 is a flowchart of a message flow and quantity control method according to an embodiment of the present invention; referring to FIG. 2, the method includes the following steps.

S11. Sampling the message receiving amount of each processing node separately, and determining, according to the sampling result, whether the actual message receiving amount of the corresponding processing node is within a preset traffic threshold threshold interval.

Because the processing capability of each processing node in the data processing pipeline and the number of messages to be processed (that is, the load of the processing node) are different, the embodiment sets a corresponding preset traffic threshold interval for each processing node in advance; The topology diagram shown in 1 is taken as an example. The node to be sampled includes all three processing nodes. According to the processing capability of the node and its load, the preset traffic threshold interval corresponding to the processing node A is set to [A1, B1], and the processing node B is processed. The corresponding preset traffic threshold interval is [A2, B2], and the preset traffic threshold interval corresponding to the processing node C is [A3, B3].

During the running of the system, the message receiving amount of each processing node is respectively sampled, that is, the load of the processing node is corresponding, and the actual message receiving amount of the processing node is determined according to the sampling result of each processing node according to the preset traffic threshold interval. Inside. Still taking FIG. 1 as an example, according to the sampling result of the processing node A, it is determined whether the actual message receiving amount of the processing node A is within the corresponding interval [A1, B1] of the processing node A, correspondingly, according to the sampling result of the processing node B. Whether the actual message reception amount of the processing node B is within the interval [A2, B2], and so on. That is, it is detected in step S11 whether the load of each processing node and its processing capability match.

S12. Generate a corresponding flow control request when determining that the actual message receiving amount of the first processing node is not within the preset traffic threshold interval according to the sampling result corresponding to the first processing node.

S13. Adjust a message sending rate of the upstream data node adjacent to the first processing node according to the flow control request.

In this embodiment, when it is detected that the actual message receiving amount of any processing node is not within its corresponding preset traffic threshold interval, that is, when the load of the processing node does not match its processing capability, a corresponding flow control request is generated, and according to the The flow control request adjusts the message sending rate of the upstream data node corresponding to the processing node.

The upstream data node may be a data source or a processing node according to different locations of the processing nodes in the pipeline. Taking FIG. 1 as an example, for two nodes of the data source and the processing node A, the processing node A is equivalent to the first processing node, the data source S is equivalent to the upstream data node, and the processing node A and the processing node B are two nodes. The processing node B corresponds to the first processing node, and the processing node A corresponds to the upstream data node.

According to the above technical solution, the embodiment of the present invention samples the received message amount of each processing node, and generates a corresponding flow control once the actual message receiving amount of a processing node is not within the corresponding preset traffic threshold interval. And requesting, according to the flow control request, adjusting a message sending rate of the upstream data node corresponding to the processing node, so that dynamic control of message traffic between any two adjacent upstream and downstream data nodes can be implemented, which is based on the prior art only The last method of processing the number of unprocessed messages of the node to adjust the message sending rate of the data source, the control precision of the present invention is higher, and any processing node does not have a message backlog or idle phenomenon, reducing the amount of message discarding and improving the entire data. Handling system stability and processing efficiency.

In a feasible embodiment of the present invention, a trigger may be separately disposed at each processing node of the data processing system, and the steps S11 and S12 are performed through the trigger, and at the same time, the controller is set in the data source and each processing node. The controller performs the above step S13. Still taking FIG. 1 as an example, in the embodiment of the present invention, the controller C_S is set in the data source S, the trigger T_A and the controller C_A are set in the processing node A, and the trigger T_B is set in the processing node B. The trigger T_C is set in node C, as shown in FIG. The solid arrows in Fig. 3 indicate the direction of message transmission between the nodes, and the dashed arrows indicate the direction of flow control request transmission between the nodes. It should be noted that, in the system shown in FIG. 3, since there are no other processing nodes downstream of the processing nodes B and C, there is no need to send a message, so the corresponding controllers may not be set in the processing nodes B and C, in other embodiments. If there are other processing nodes downstream of the processing nodes B and C, a corresponding controller may be added to the processing nodes B and C. Based on the system shown in Figure 3, the message flow control process between nodes is shown in Figure 4.

Referring to FIG. 3 and FIG. 4, for the two nodes of the data source and the processing node A, the processing node A is equivalent to the first processing node, and the data source S is equivalent to the upstream data node, then: in step S111, the trigger T_A is processed. The message receiving amount of the node A is sampled, and it is determined according to the sampling result whether the actual message receiving amount is within the corresponding preset traffic threshold threshold interval [A1, B1]; in step S121, the trigger T_A is determined to be the actual processing node A. When the message reception amount is not in [A1, B1] (greater than B1, or less than A1), that is, the load and processing capability of the processing node A do not match, a flow control request is generated and sent to the data source S; In S131, the controller C_S adjusts the message sending rate of the data source S according to the flow control request sent by the trigger T_A. Through the above steps S111, S121 and S131, the number of messages sent to the processing node A per unit time, that is, the actual message receiving amount of the processing node A per unit time, can be matched with the number of message processing of the processing node A, which can avoid When the message sending rate of the source S is too large, the message is accumulated in the processing node A, and the message sending rate of the data source S is too small, and the processing node A is idle.

Correspondingly, for the two nodes of the processing node A and the processing node B, the processing node B is equivalent to the first processing node, and the processing node A is equivalent to the upstream data node, then: in step S112, the trigger T_B is paired with the processing node B. The received message amount is sampled, and it is judged according to the sampling result whether the actual message receiving amount is within the corresponding preset traffic threshold threshold interval [A2, B2]; in step S122, the trigger T_B is determined to process the actual message receiving of the processing node B. When the quantity is not in [A2, B2], a flow control request is generated and sent to the processing node A; in step S132, the controller C_A adjusts the processing node A to the processing node B according to the flow control request sent by the trigger T_B. The rate at which the message is sent when the message is sent. That is, the actual message reception amount of the processing node B per unit time is matched with the number of message processing through the above-described steps S112, S122, and S132.

For the processing node A and the processing node C, the message flow control process is similar to the above steps S112, S122 and S132, and will not be described again herein.

In a possible embodiment of the present invention, the determining, according to the sampling result, whether the actual message receiving amount of the corresponding processing node is within the preset traffic threshold interval according to the sampling result, specifically includes the following steps: for any processing node, Calculating a ratio between the number of sampled values that are not in the preset flow threshold range and the total number of sampled values in the preset sampling period that are collected in the preset sampling period; if the ratio is greater than the preset threshold, Then, it is determined that the actual message receiving amount of the corresponding processing node is not within the preset traffic threshold interval.

A flow control request is generated for each sampling value that is not within the preset traffic threshold interval. In the above embodiment, when the proportion of the sample value exceeding the preset traffic threshold interval in a preset sampling period is greater than a preset threshold The flow control request is generated, and the adjustment of the message sending rate of the upstream data node is started, which can avoid problems such as frequent system jitter and processing capability degradation.

In a feasible embodiment of the present invention, when the actual message receiving amount of the first processing node is not within the preset traffic threshold interval, the corresponding traffic is generated according to the sampling result corresponding to the first processing node. The control request may include the following two situations: when it is determined that the actual message receiving amount of the first processing node is greater than a preset maximum traffic threshold according to the sampling result corresponding to the first processing node, generating a traffic reduction request; When the sampling result corresponding to the processing node determines that the actual message receiving amount of the first processing node is less than the preset minimum traffic threshold, a traffic increase request is generated.

That is, during the peak traffic period of the system, when the actual message receiving amount of a processing node is greater than the preset maximum traffic threshold, a traffic reduction request is generated to control the upstream data node to reduce the message sending rate; and the message is sent along with the upstream data node. When the rate is decreased, the actual message receiving amount of the processing node is also decreased. When it is lower than the preset minimum traffic threshold, a traffic increase request is generated to stop the lowering of the message sending rate of the upstream data node, so that the message is changed. The send rate is gradually increased until it returns to normal.

In a feasible embodiment of the present invention, the foregoing flow control request includes not only information for indicating that the upstream data node raises or lowers the message sending rate, but also a message processing rate of the processing node that generates the flow control request; When the flow control request is the traffic reduction request, the method for adjusting the message sending rate of the upstream data node in the foregoing step S13 is: when the flow control request is the traffic reduction request, according to the first processing The message processing rate of the node and the original message sending rate of the upstream data node determine a message sending delay time; superimposing the message sending delay time and the original message sending rate to obtain a target message sending rate of the upstream data node .

Specifically, it is assumed that the message processing rate of the first processing node is TPS_DownStream, and the original message sending rate of the upstream data node is TPS_UpStream. The traffic control request is a traffic reduction request, and the UpStream_TPS>TPS_DownStream can be inferred. The upstream node sends a message with a delay of T_delay (that is, the above message transmission delay time) to reduce UpStream_TPS; the message transmission delay time T_delay can be calculated by the following formula: T_delay=1/TPS_DownStream-1/UpStream_TPS.

The above T_delay and UpStream_TPS are superimposed to obtain an adjusted message sending rate, that is, the target message sending rate UpStream_TPS'; the superposition calculation formula may be: UpStream_TPS'=1/[(1/UpStream_TPS)+T_delay].

In addition, in the actual application, because the network or other intermediate links have a certain influence on the message sending rate or the message processing rate, the upstream data node still receives the traffic of the downstream processing node after applying the adjusted message sending rate. The request is controlled, so in other embodiments of the present invention, the fine adjustment of the message transmission delay time may be performed, that is, the fine adjustment of the minimum delay unit T_min_adjust (which may be 1 millisecond) is added on the basis of T_delay, that is, T_delay=1/TPS_DownStream -1/UpStream_TPS+T_min_adjust.

In another possible embodiment of the present invention, when the flow control request is the traffic increase request, the method for adjusting the message sending rate of the upstream data node in step S13 is: adjusting the step according to the preset rate. The message rate of the upstream data node is gradually increased to a preset normal rate.

That is, when the traffic increase request is received, the message sending rate of the upstream data node is gradually increased, instead of being restored to the preset normal rate at one time; compared to the above-mentioned adjustment method based on the message sending delay time T_delay, the received traffic is increased. When requesting, gradually reduce the value of T_delay until 0, and UpStream_TPS' is gradually increased to UpStream_TPS. By adopting this method of gradually increasing the message transmission rate, the message backlog can be avoided and the system can be stabilized.

In addition, when the message sending rate of the upstream data node is restored to the preset normal rate according to the traffic increase request, the response information may be returned to the corresponding downstream processing node to notify the downstream processing node that the control ends; correspondingly, downstream After receiving the response information, the processing node does not send the request, but still checks whether the actual message receiving amount of itself is within the preset traffic threshold interval.

In a feasible embodiment of the present invention, the flow control request generated in step S12 may be directly sent by the first processing node to the corresponding upstream data node; in another feasible embodiment of the present invention, the flow control request may also be The flow control request is forwarded through an independent node, and the independent node receives the flow control request of each processing node, determines its corresponding upstream data node, and forwards the flow control request to the upstream data node. Optionally, the trigger and the controller are set in the node according to the foregoing embodiment, and the independent node in this embodiment may be a coordinator disposed in the system.

In actual application, if there is a plurality of downstream processing nodes in an upstream data node, in the process of receiving and forwarding the flow control request through the independent node, in the same detection period, multiple downstream processing nodes may detect their actual conditions. The message receiving amount is not within the preset traffic threshold interval, so that the flow control request is generated correspondingly. If the message sending rate of the upstream data node is adjusted according to the flow control request of each downstream processing node, it needs to be adjusted multiple times, not only The workload is large, and the system may be unstable due to frequent adjustments.

In view of the above, in a feasible embodiment of the present invention, a plurality of flow control requests sent to the same upstream data node may be filtered, that is, adjusted to be adjacent to the first processing node according to the flow control request. Before the message sending rate of the upstream data node (step S13), the message flow control method may further include the following steps: S14: if a plurality of downstream processing nodes corresponding to the upstream data node generate corresponding traffic within a preset time period Controlling the request, determining a message transmission rate change amount caused by the flow control request corresponding to each downstream processing node to the upstream data node; S15, determining a target flow control request corresponding to a maximum value of the message transmission rate change, according to the The target flow control request adjusts the message sending rate of the upstream data node.

Referring to the system shown in FIG. 1 or FIG. 3, the processing node A has two downstream processing nodes, that is, the processing node B and the processing node C, which may occur in the same detection period, and both the processing node B and the processing node C detect themselves. The actual message receiving amount is not within the preset traffic threshold interval, and the flow control request is generated. For the two flow control requests, a flow control request that maximizes the change rate of the message transmission rate caused by the processing node A is selected through calculation and comparison. And adjusting the message sending rate of the processing node A according to the selected flow control request.

Preferably, the message transmission delay rate T_delay described above can be used to measure the amount of change of the message transmission rate caused by the flow control request to the corresponding processing node.

In the above embodiment, the same upstream data node receives multiple flow control requests, and selects a flow control request for adjusting the amount of change caused by the message sending rate of the upstream data node according to each flow control request, and is used to adjust the upstream data node. The message transmission rate can reduce the number of adjustments, which can reduce the workload of flow control, avoid system jitter caused by frequent adjustments in a short period of time, and improve system stability.

Preferably, the foregoing steps S14 and S15 can be performed through the coordinator described in the foregoing embodiment, and the target flow control request determined in step S15 is sent to the controller corresponding to the upstream data node, so as to avoid the screening process on the upstream data node. The occupation of resources such as memory.

In a feasible embodiment of the present invention, the message flow control method may further include the following steps: S16: storing a flow control state of each processing node to a preset storage module; and S17, setting the preset according to a preset period. The flow control state stored by the storage module is synchronized to the corresponding processing node.

Taking an instant computing system such as Storm and Jstorm as an example, the preset storage module may be a coordination server external to the instant computing system, such as ZooKeeper. Corresponding to the two flow control requests described above, that is, the traffic reduction request and the traffic increase request, the flow control state includes at least two types: for any processing node, if it receives the traffic reduction request, it corresponds to The flow control state is the downflow control state (ie, the processing node sends a message to the downstream processing node at a message sending rate lower than a preset value); if it receives the traffic increase request or does not receive any flow control request, then The corresponding flow control state is an uncontrolled state (ie, the processing node sends a message to the downstream processing node at a preset message sending rate).

In the above technical solution, the instant storage and periodic synchronization of the flow control state of each processing node can ensure that the processing node can still be in the correct flow control state after the failure restart, thereby ensuring the stability of the system.

Preferably, the above steps S16 and S17 can also be performed by the coordinator described above.

Corresponding to the foregoing embodiment of the message flow control method, the embodiment of the present invention further provides a message flow control device applied to a pipelined data processing system, and the pipelined data processing system includes a data source and a plurality of processing nodes; For convenience of description, the data source and each processing node will be collectively referred to as a data node hereinafter. The message flow control device includes: triggers respectively disposed in each of the processing nodes, and controllers respectively disposed in each of the data nodes having the downstream processing nodes. The topology of the data processing system provided with the above message flow control device can be referred to FIG.

The trigger is configured to sample the received message amount of the processing node where the processing node is located, and determine whether the actual message receiving amount of the processing node where the processing node is located is within the preset traffic threshold interval according to the sampling result, and is at the processing node where the processing node is located. When the actual message receiving amount is not within the preset traffic threshold interval, a corresponding flow control request is generated.

The controller is configured to adjust a message sending rate of the data node where the data node is located according to the flow control request of the downstream processing node corresponding to the data node where the data node is located.

Specifically, as shown in FIG. 5, a block diagram of any one of the upstream data node 510 and a downstream data node 520 adjacent to the data processing system provided with the message flow control device; wherein, in a specific case, the upstream data Both the node 510 and the downstream data node 520 are processing nodes. In another specific case, the upstream data node 510 is a data source, and the downstream data node 520 is a processing node. The dashed and dashed arrows in Fig. 5 indicate the structure and information flow that exist in one case and not exist in the other.

Based on the upstream data node 510 and the downstream data node 520, the operation process of the message flow control device is as follows: the trigger 521 of the downstream data node 520 samples the message receiving amount of the downstream data node 520, and determines the downstream data according to the sampling result. Whether the actual message reception amount of the node 520 is within the preset traffic threshold interval of the downstream data node 520, if not, generating a corresponding flow control request, and transmitting the flow control request to the controller 512 of the upstream data node 510; After receiving the flow control request sent by the trigger 521, the device 512 adjusts the message sending rate of the upstream data node 510 according to the flow control request, so that the actual message receiving amount of the downstream data node 520 can be controlled by the downstream data node 520. Set the flow threshold within the interval.

According to the above technical solution, the embodiment of the present invention samples the received message amount of each processing node, and generates a corresponding flow control once the actual message receiving amount of a processing node is not within the corresponding preset traffic threshold interval. And requesting, according to the flow control request, adjusting a message sending rate of the upstream data node corresponding to the processing node, so that dynamic control of message traffic between any two adjacent upstream and downstream data nodes can be implemented, which is based on the prior art only The last method of processing the number of unprocessed messages of the node to adjust the message sending rate of the data source, the control precision of the present invention is higher, and any processing node does not have a message backlog or idle phenomenon, reducing the amount of message discarding and improving the entire data. Handling system stability and processing efficiency.

In a feasible embodiment of the present invention, in order to determine whether the actual message receiving amount of the processing node where the processing node is located is within the preset traffic threshold interval according to the sampling result, the trigger may be configured to: calculate the preset sampling period. The ratio between the number of sampled values that are not within the preset flow threshold range and the total number of sampled values in the preset sampling period, and determines that the processing is performed when the ratio is greater than a preset threshold. The actual message reception of the node is not within its preset traffic threshold interval.

Based on the foregoing technical solution, a proportion of the sample value exceeding the preset traffic threshold interval in a preset sampling period is generated with respect to generating a flow control request for each sample value that is not within the preset traffic threshold interval. When the threshold is greater than the preset threshold, the traffic control request is generated, and the message sending rate of the upstream data node is adjusted to avoid frequent adjustments, such as system jitter and processing capability.

In a feasible embodiment of the present invention, in order to implement a corresponding flow control request when the actual message receiving amount of the processing node at the processing node is not within the preset traffic threshold interval, the trigger may be configured to: When the actual message receiving amount of the processing node is greater than the preset maximum traffic threshold, a traffic reduction request is generated; when the actual message receiving amount of the processing node at the processing node is greater than the preset minimum traffic threshold, a traffic increase request is generated.

In a feasible embodiment of the present invention, in order to implement a flow control request of a downstream processing node corresponding to the data node where the data node is located, and adjust a message sending rate of the data node where the data node is located, the controller may be configured to: When the request is the traffic reduction request, determining a message sending delay time according to the message processing rate of the downstream processing node and the original message sending rate of the data node where the data node is located, and sending the message sending delay time and the original message The rate is superimposed to obtain the target message transmission rate of the data node where it is located.

In another possible embodiment of the present invention, in order to implement a flow control request of a downstream processing node corresponding to the data node where the data node is located, and adjust a message sending rate of the data node where the data node is located, the controller may be configured to: When the control request is the traffic increase request, the step size is adjusted according to the preset rate to gradually increase the message sending rate of the data node where the data node is located to a preset normal rate.

In addition to the trigger and the controller, the message flow control apparatus provided by the present invention may further include a coordinator, which is disposed in parallel with each processing node in the data processing system, and the coordinator is configured to acquire the generated by each trigger. A flow control request forwards the flow control request to a corresponding upstream data node.

FIG. 6 is a topological structural diagram of the system topology structure provided with the message flow control device shown in FIG. 3 after adding a coordinator. Referring to FIG. 6, the coordinator K acquires the flow control request of the trigger T_A, the trigger T_B, and the trigger T_C, respectively, and forwards the flow control request of the trigger T_A to the controller C_S of the data source S, and triggers the trigger T_B and the trigger. The flow control request of the T_C is forwarded to the controller C_A of the processing node A, respectively.

In a possible embodiment of the present invention, the coordinator may be configured to: obtain a plurality of flow control requests sent to a controller of the same upstream data node within a preset time period, where the first coordinator determines Each flow control request sends a rate change of the message to the same upstream data node, and sends a flow control request corresponding to the maximum value of the message transmission rate change to the controller of the same upstream data node.

Still referring to FIG. 6, in the preset time period, the coordinator K obtains both the flow control request Q_B of the trigger T_B and the flow control request Q_C of the trigger T_C, in order to reduce the message of the common upstream processing node A. The number of transmission rate adjustments ensures the stability of the system. The coordinator K determines the amount of change in the message transmission rate caused by the flow control request Q_B and Q_C to the processing node A, and selects a flow control request with a large change in the message transmission rate, which is assumed to be traffic. The control request Q_B, the coordinator K only forwards the flow control request Q_B to the controller C_A, and discards the flow control request Q_C; correspondingly, the controller C_A only needs to adjust the message sending rate of the processing node A according to the flow control request Q_B.

In another possible embodiment of the present invention, the coordinator is further configured to: store the flow control state of each processing node to the preset storage module, and store the preset storage module according to a preset period. The flow control state is synchronized to the corresponding processing node; thus, the processing node can still be in the correct flow control state after the failure restart, thereby ensuring the stability of the system.

In addition, an embodiment of the present invention further provides a data processing system, including a data source, a plurality of processing nodes, and the message flow control device described in any of the above embodiments. The structure of the system can be referred to FIG. 3 and FIG. 6. For the flow control process, reference may be made to the above method and device embodiment, and details are not described herein.

The various embodiments in the present specification are described in a gradual manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the device and system embodiments, the description is relatively simple, since it is substantially similar to the method embodiment, and the relevant portions of the method embodiments can be referred to.

The embodiments of the invention described above are not intended to limit the scope of the invention. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims (15)

 A message flow control method is characterized in that it is applied to a data processing system, the data processing system includes a data source and a plurality of processing nodes; the method comprises: separately sampling the message receiving amount of each processing node, and according to the sampling result Determining whether the actual message receiving amount of the corresponding processing node is within the preset traffic threshold interval; when determining that the actual message receiving amount of the first processing node is not within the preset traffic threshold interval according to the sampling result corresponding to the first processing node, Generating a corresponding flow control request; adjusting a message sending rate of the upstream data node adjacent to the first processing node according to the flow control request; wherein the upstream data node may be the data source or the processing node.    The method of claim 1, wherein the corresponding traffic is generated when the actual message receiving amount of the first processing node is not within the preset traffic threshold interval according to the sampling result corresponding to the first processing node. The control request includes: when determining that the actual message receiving amount of the first processing node is greater than a preset maximum traffic threshold according to the sampling result corresponding to the first processing node, generating a traffic reduction request; determining, according to the sampling result corresponding to the first processing node When the actual message receiving amount of the first processing node is less than the preset minimum traffic threshold, a traffic increase request is generated.    The method of claim 2, wherein adjusting the message transmission rate of the upstream data node adjacent to the first processing node according to the flow control request comprises: the flow control request is the traffic reduction request Determining a message transmission delay time according to the message processing rate of the first processing node and the original message sending rate of the upstream data node; superimposing the message sending delay time and the original message sending rate to obtain the target of the upstream data node The rate at which the message is sent.    The method of claim 2, wherein the message transmission rate of the upstream data node adjacent to the first processing node is adjusted according to the flow control request, including: the flow control request is the traffic increase request The step of adjusting the step size according to the preset rate gradually increases the message sending rate of the upstream data node to a preset normal rate.    The method of any one of claims 1 to 4, wherein before the adjusting the message transmission rate of the upstream data node adjacent to the first processing node according to the flow control request, the method further comprises: If a plurality of downstream processing nodes corresponding to the upstream data node generate corresponding flow control requests within a preset time period, determining a message transmission rate change amount caused by the flow control request corresponding to each downstream processing node to the upstream data node; Determining a target flow control request corresponding to a maximum value of the message transmission rate change, to adjust a message sending rate of the upstream data node according to the target flow control request.    The method of any one of claims 1 to 4, further comprising: storing a flow control state of each processing node to a preset storage module; and the preset storage module according to a preset period The stored flow control state is synchronized to the corresponding processing node.    The method according to any one of claims 1 to 4, wherein, according to the sampling result, it is determined whether the actual message receiving amount of the corresponding processing node is within the preset traffic threshold interval, including: for any processing node And calculating a ratio between the number of sample values that are not in the preset flow threshold range and the total number of sample values in the actual preset sampling period, and if the actual ratio is greater than the preset threshold, It is determined that the actual message receiving amount of the corresponding processing node is not within the preset traffic threshold interval.    A message flow control device is characterized in that it is applied to a data processing system, the data processing system includes a data source and a plurality of processing nodes; the device includes: a trigger and a controller; the trigger is disposed in each processing node, It is used to sample the received message amount of the processing node where it is located, and judge whether the actual message receiving amount of the processing node where it is located is within the preset traffic threshold interval according to the sampling result, and the actual message receiving amount at the processing node where it is located is not When the preset traffic threshold is within the preset interval, a corresponding flow control request is generated; the controller is disposed in each data node having a downstream processing node, and is configured to adjust according to the flow control request of the downstream processing node corresponding to the data node where the data node is located The message sending rate of the data node where it is located; wherein the data node can be the data source or the processing node.    The device according to claim 8, wherein the trigger is configured to: when the actual message receiving amount of the processing node at the processing node is not within the preset traffic threshold interval, the trigger is configured to: When the actual message receiving amount of the processing node is greater than the preset maximum traffic threshold, a traffic reduction request is generated; when the actual message receiving amount of the processing node at the processing node is greater than the preset minimum traffic threshold, a traffic increase request is generated.    The device according to claim 9, wherein the controller is configured to: adjust the message sending rate of the data node where the data node is located according to the flow control request of the downstream processing node corresponding to the data node where the data node is located, and the controller is configured to: When the flow control request is the traffic reduction request, determining a message sending delay time according to the message processing rate of the downstream processing node and the original message sending rate of the data node where the data node is located, and performing the message sending delay time and the original message sending rate. Superimpose to get the target message sending rate of the data node where it is located.    The system of claim 9, wherein the controller is configured to: adjust the message transmission rate of the data node according to the data processing node corresponding to the data node to which the data node is located, and the controller is configured to: When the flow control request is the traffic increase request, the step size is adjusted according to the preset rate to gradually increase the message sending rate of the data node where the data node is located to a preset normal rate.    The system of any one of clauses 8 to 11, wherein the method further includes: a first coordinator; the first coordinator is disposed in parallel with each processing node in the data processing system for acquiring the a flow control request sent by the trigger, and forwarding the flow control request to a controller corresponding to the upstream data node; wherein, if the first coordinator acquires multiple controllers that send to the same upstream data node within a preset time period The flow control request, the first coordinator determines a message transmission rate change caused by each flow control request to the same upstream data node, and sends a flow control request corresponding to the maximum value of the message transmission rate change to the same upstream The controller of the data node.    The system according to any one of claims 8 to 11, further comprising: a second coordinator; the second coordinator is disposed in parallel with each processing node in the data processing system for The flow control state of the processing node is stored in the preset storage module, and the flow control state stored by the preset storage module is synchronized to the corresponding processing node according to a preset period.    The system according to any one of claims 8 to 11, wherein the trigger is determined to determine whether the actual message reception amount of the processing node in which the processing node is located is within the preset traffic threshold interval according to the sampling result. The configuration is: calculating a ratio between the number of sampled values that are not within the preset flow threshold range and the total number of sampled values in the preset sampling period, and the ratio is greater than the preset. At the threshold, it is determined that the actual message received by the processing node at which it is located is not within its preset traffic threshold interval.    A data processing system comprising a data source and a plurality of processing nodes, wherein the system further comprises the message flow control device according to claim 8 of the patent application.