CN111796950B

CN111796950B - Data processing method and system

Info

Publication number: CN111796950B
Application number: CN202010685054.1A
Authority: CN
Inventors: 朱文亮; 叶均杰; 温中凯
Original assignee: Netease Hangzhou Network Co Ltd
Current assignee: Netease Hangzhou Network Co Ltd
Priority date: 2020-07-16
Filing date: 2020-07-16
Publication date: 2023-06-30
Anticipated expiration: 2040-07-16
Also published as: CN111796950A

Abstract

The embodiment of the application provides a data processing method and a data processing system, wherein the method comprises the following steps: responding to the received data to be processed, the first intermediate node obtains a first processing capability, the first intermediate node sends a first message to the second intermediate node, the second intermediate node obtains a second processing capability, and when the first processing capability is higher than the second processing capability, the first message is sent to the third intermediate node, the steps are repeated until a plurality of intermediate nodes are traversed, and the first message is returned to the first intermediate node, and the first intermediate node determines a first target consumption node from the corresponding consumption nodes so that the first target consumption node processes the data to be processed. According to the scheme, the data to be processed are circulated among the intermediate nodes, so that the first target consumption node corresponding to the data to be processed can be rapidly determined, and the data throughput is improved.

Description

Data processing method and system

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method and system.

Background

At present, a typical asynchronous task is usually realized by using a production node and consumption node model, and asynchronous communication is completed between the production node and the consumption node by using a message queue. A message queue refers to a container that holds data during data transmission, i.e., a production node generates data, buffers the data in the message queue, and a consuming node retrieves the data from the message queue for processing.

In the prior art, a production node sends generated data to a central control node, the central control node maintains at least one message queue, when the central control node receives the data sent by the production node, the data is cached in one of the message queues, and the condition of each consumption node is monitored to determine which consumption node the data is sent to.

However, the complexity of deciding which data of which message queue should be consumed by which consuming node is also growing in the face of a large number of consuming nodes, resulting in lower data throughput.

Disclosure of Invention

The data processing method and system are used for improving data throughput through data streaming, and aims at solving the problem that the data throughput is low due to higher complexity when a consumption node for consuming data to be processed is determined in the prior art.

In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

in a first aspect, an embodiment of the present application provides a data processing method, including:

responding to the received data to be processed, the first intermediate node obtains a first processing capability, wherein the first processing capability is used for indicating the processing capability of a corresponding consumption node;

The first intermediate node sends a first message to a second intermediate node, wherein the first message comprises indication information of the first processing capability and an identifier of the first intermediate node;

the second intermediate node obtains a second processing capability, and when the first processing capability is higher than the second processing capability, the second intermediate node sends the first message to a third intermediate node, and the steps are repeated until the plurality of intermediate nodes are traversed, and the first message is returned to the first intermediate node;

and the first intermediate node determines a first target consumption node from the corresponding consumption nodes so that the first target consumption node processes the data to be processed.

In one possible implementation, the method further comprises:

when the first processing capacity is lower than the second processing capacity, the second intermediate node sends a second message to a third intermediate node, and the steps are repeated until the plurality of intermediate nodes are traversed, and the traversed target message is returned to the first intermediate node; the second message includes the indication information of the second processing capability and the identifier of the second intermediate node, and the target message includes: indication information of the highest processing capacity in the traversal process and identification of a target intermediate node corresponding to the highest processing capacity;

When the identification of the target intermediate node is different from the identification of the first intermediate node, the first intermediate node sends the target message to the second intermediate node;

and when the second processing capacity is equal to the highest processing capacity, the second intermediate node determines a second target consumption node from the corresponding consumption nodes so that the second target consumption node processes the data to be processed.

In one possible implementation, the method further comprises:

when the second processing capacity is lower than the highest processing capacity, the second intermediate node sends the target message to the third intermediate node, and the steps are repeated until the target intermediate node corresponding to the highest processing capacity is reached;

and the target intermediate node determines a third target consumption node from the corresponding consumption nodes so that the third target consumption node processes the data to be processed.

In one possible implementation, the method further comprises:

the first target consumption node stores the data to be processed into a corresponding processing sequence, wherein the processing sequence comprises at least one data to be processed;

and if the first target consumption node does not process the data to be processed under the preset condition, the first intermediate node sets the first processing capacity as the preset lowest processing capacity and sends the data to be processed to the second intermediate node.

In one possible implementation, the preset condition includes: the polling times of the first target consumption node are greater than or equal to preset times, wherein the polling times are as follows: and polling the first target consumption node for the number of times of processing the data to be processed.

In one possible implementation, the preset condition further includes: the waiting time length of the data to be processed is longer than or equal to the preset time length.

In one possible implementation, the first intermediate node obtains a first processing capability, including:

and determining the first processing capacity according to the attribute parameters of the consumption nodes corresponding to the first intermediate.

In one possible implementation, the attribute parameters include at least one of the following:

processing speed, remaining memory, remaining hard disk space, number of input and output interfaces.

In one possible implementation, the plurality of intermediate nodes includes a plurality of intermediate parent nodes;

the first intermediate node sends a first message to a second intermediate node, including:

and the first intermediate father node sends the first message to the second intermediate father node.

In one possible implementation, when the first processing capability is higher than the second processing capability, the second intermediate node sends the first message to a third intermediate node, and repeats this step until the plurality of intermediate nodes are traversed, and returns the first message to the first intermediate node, including:

And when the first processing capacity is higher than the second processing capacity, the second intermediate father node sends the first message to a third intermediate father node, and the steps are repeated until the plurality of intermediate father nodes are traversed, and the first message is returned to the first intermediate father node.

In one possible implementation, the intermediate nodes further include an intermediate child node corresponding to each intermediate parent node;

the first intermediate node determines a first target consuming node from the corresponding consuming nodes, including:

the first intermediate father node determines a target intermediate child node from the corresponding intermediate child nodes;

and the target intermediate child node determines the first target consumption node from consumption nodes corresponding to the target intermediate child node.

In one possible implementation, the first intermediate parent node determines a target intermediate child node from corresponding intermediate child nodes, including:

the first intermediate father node sends the data to be processed to a first intermediate child node;

the first intermediate child node obtains a third processing capability in response to the received data to be processed, wherein the third processing capability is used for indicating the processing capability of a consumption node corresponding to the first intermediate child node;

The first intermediate child node sends the third message to a second intermediate child node, wherein the third message comprises indication information of the third processing capability and an identifier of the first intermediate child node;

the second intermediate child node obtains fourth processing capability, and when the third processing capability is higher than the fourth processing capability, the second intermediate child node sends the third message to a third intermediate child node, and the steps are repeated until the intermediate child node corresponding to the first intermediate father node is traversed, and the third message is returned to the first intermediate child node;

the first intermediate child node determines the first intermediate child node as the target intermediate child node.

In a second aspect, another embodiment of the present application provides a data processing system, the system comprising:

a plurality of intermediate node devices and a consuming node device connected to each intermediate node device;

the plurality of intermediate node devices are respectively configured to perform steps performed by any one of the first intermediate node, the second intermediate node, and the third intermediate node that may implement the first aspect;

for each intermediate node device, there is a target consuming node device in the consuming node devices connected to the intermediate node device, where the multiple target consuming node devices are respectively configured to execute steps executed by any one of the first target consuming node, the second target consuming node, and the third target consuming node that may be implemented in the first aspect.

In one possible implementation, the plurality of intermediate node devices and the consuming node device are each separate physical devices.

In one possible implementation, the plurality of intermediate node devices and the consuming node device are hardware nodes of a same physical device, respectively.

The embodiment of the application provides a data processing method and a data processing system, wherein the method comprises the following steps: responding to the received data to be processed, the first intermediate node obtains a first processing capability, the first processing capability is used for indicating the processing capability of the corresponding consumption node, the first intermediate node sends a first message to the second intermediate node, the first message comprises indication information of the first processing capability and identification of the first intermediate node, the second intermediate node obtains the second processing capability, and when the first processing capability is higher than the second processing capability, the second intermediate node sends the first message to the third intermediate node, the steps are repeated until the plurality of intermediate nodes are traversed, the first message is returned to the first intermediate node, and the first intermediate node determines a first target consumption node from the corresponding consumption nodes so that the first target consumption node processes the data to be processed. According to the scheme, the data to be processed are circulated among the intermediate nodes, when the processing capacity of the first intermediate node is higher than that of other intermediate nodes, the first intermediate node determines the first target consumption node, the first target consumption node processes the data to be processed, and the first target consumption node corresponding to the data to be processed can be rapidly determined by adopting the mode, so that the data throughput is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered limiting the scope, and that other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 depicts a schematic diagram of the architecture of an existing data processing system provided by an embodiment of the present application;

FIG. 2 depicts a schematic diagram of a data processing system according to an embodiment of the present application;

fig. 3 shows a flowchart of a data processing method according to an embodiment of the present application;

fig. 4 shows a second flowchart of a data processing method according to an embodiment of the present application;

fig. 5 shows a third flowchart of a data processing method according to an embodiment of the present application;

fig. 6 shows a flowchart of a data processing method according to an embodiment of the present application;

fig. 7 shows a fifth flowchart of a data processing method according to an embodiment of the present application;

FIG. 8 depicts a second schematic architecture of a data processing system provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of a data processing system according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it should be understood that the accompanying drawings in the present application are only for the purpose of illustration and description, and are not intended to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. A flowchart, as used in this application, illustrates operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow diagrams may be implemented out of order and that steps without logical context may be performed in reverse order or concurrently. Moreover, one or more other operations may be added to the flow diagrams and one or more operations may be removed from the flow diagrams as directed by those skilled in the art.

In addition, the described embodiments are only some, but not all, of the embodiments of the present application. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the present application.

It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features stated hereinafter, but not to exclude the addition of other features. Furthermore, the terms first, second, third and fourth in the description and in the claims and drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.

It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

FIG. 1 illustrates a schematic architecture of an existing data processing system provided by an embodiment of the present application, as shown in FIG. 1, an existing data processing system 10 includes: the production node 11, the central control node 12 and the consumption node 13, wherein the production node 11 comprises a first production node and a second production node, the consumption node 13 comprises a first consumption node, a second consumption node and a third consumption node, the central control node 12 maintains at least one message queue, and fig. 1 shows one message queue.

When the production node 11 sends the generated data to the central control node 12, the central control node 12 caches the data in the corresponding message queue, and monitors the condition of each consumption node 13 to determine which consumption node 13 to send the data to; or the central control node 12 only maintains a path with the consuming node 13, and when data is cached in the corresponding message queue, the consuming node 13 decides by itself who to process the data in the message queue.

However, the two approaches described above have the following drawbacks:

first, if the central control node decides which consuming node to send a message to for processing, the consuming node passively receives the data, and although the problem of queuing by the consuming node is avoided, the decision by the central control node may result in hard coupling of communication logic (i.e. the communication process of data transfer) and consuming logic (i.e. deciding which consuming node processes the data), which increases the processing complexity, causes lower data throughput, and is unfavorable for long-term iteration and maintenance.

Second, if the consuming node decides by itself who is processing the data in the message queue, the consuming node actively fetches the data from the message queue, which may cause queuing problem due to the many-to-one competition relationship, wasting time and resulting in lower data throughput.

Based on the above, the embodiment of the application provides a data processing method and system, through circulating data to be processed among intermediate nodes, a first target consumption node corresponding to the data to be processed can be rapidly determined, queuing is avoided, and meanwhile, data throughput is improved.

The technical solutions of the embodiments of the present application are explained below in conjunction with several specific embodiments.

Fig. 2 shows a schematic architecture diagram of a data processing system provided in an embodiment of the present application, and fig. 3 shows a schematic flow diagram of a data processing method provided in an embodiment of the present application.

The data processing method shown in fig. 3 is applied to the data processing system shown in fig. 2, and referring to fig. 3, the data processing system 20 includes a plurality of intermediate nodes 21, and consumption nodes 22 corresponding to each intermediate node, where the intermediate nodes 21 are configured to receive data to be processed, and send the data to be processed to the consumption nodes 22, and the consumption nodes 22 are configured to process the data to be processed received from the intermediate nodes. Wherein the intermediate node 21 may also be referred to as a station for capturing process messages or release messages.

It should be noted that the data processing system may further include a production node 23, where the production node 23 is configured to produce the data to be processed and send the data to be processed to the intermediate node 21.

In a practical application scenario, in a distributed image processing scenario, the production node 23 sends an image to be processed to the intermediate node 21, and the intermediate node 21 sends the image to be processed to the consumption node 22. So that the consumption node 22 processes the received image to be processed, the specific processing procedure is set according to the actual requirement, which is not limited in this embodiment.

As shown in fig. 2, the production nodes 23 are respectively denoted as: the first production node and the second production node; the plurality of intermediate nodes 21 are respectively denoted as: first intermediate node, second intermediate node. First consuming node, second consuming node … … nth consuming node.

It should be noted that, fig. 2 shows that each intermediate node 21 corresponds to one consuming node 22, and in practical application, the number of consuming nodes 22 corresponding to each intermediate node 21 includes, but is not limited to, one, that is, one intermediate node may also correspond to a plurality of consuming nodes.

The data processing method provided in the embodiment of the present application is described in detail below with reference to the description of the data processing system shown in fig. 2.

As shown in fig. 3, the data processing method provided in this embodiment includes the following steps:

s101, responding to received data to be processed, and acquiring a first processing capability by a first intermediate node.

In this embodiment, the first intermediate node is any one of a plurality of intermediate nodes. The production node may generate data to be processed and send the data to be processed to the first intermediate node, and the first intermediate node receives the data to be processed and then obtains a first processing capability in response to the received data to be processed, where the first processing capability is used to indicate a processing capability of a corresponding consumption node. Specifically, the consuming node is a consuming node corresponding to the first intermediate node, and the number of consuming nodes includes, but is not limited to, one.

Optionally, the first intermediate node obtains a first processing capability, including:

and determining the first processing capacity according to the attribute parameters of the consumption nodes corresponding to the first intermediate nodes.

Illustratively, the attribute parameters include at least one of the following:

The processing speed represents the speed of processing data by the consuming node, the remaining space represents the remaining memory space of the consuming node, the remaining hard disk space represents the remaining hard disk space of the consuming node, and the number of input and output interfaces is used for representing the number of input and output interfaces which are still remained currently, that is, when the consuming node can process the data to be processed, the input and output interfaces are needed to exist, so that the data to be processed can be sent to the consuming node through the input interfaces, and the processed data can be sent to other nodes through the output interfaces.

In one possible implementation, determining the first processing capability according to the attribute parameter of the consumption node corresponding to the first intermediate node includes: and carrying out weighted operation on the attribute parameters of the consumption nodes corresponding to the first intermediate nodes to obtain scores, wherein the scores are used for indicating the first processing capacity, the higher the scores are, the stronger the processing capacity is, the lower the scores are, and the lower the processing capacity is.

Of course, the manner of determining the first processing capability includes, but is not limited to, the above example, and the first processing capability may also be scored according to a sum of attribute parameters of the consumption node corresponding to the first intermediate node, which may be specifically determined according to the actual situation, which is not limited in this embodiment.

S102, the first intermediate node sends a first message to the second intermediate node.

The message refers to a data unit exchanged and transmitted in the network, namely, a data block to be sent by a station at one time. The second intermediate node is any one of a plurality of intermediate nodes other than the first intermediate node.

Specifically, the first intermediate node obtains the first processing capability, and may obtain a first packet, and send the first packet to the second intermediate node, where the first packet includes indication information of the first processing capability and an identifier of the first intermediate node.

The indication information of the first processing capability may be the score in step S101, which is not limited in this embodiment, and may be specifically determined according to the actual situation.

S103, the second intermediate node obtains the second processing capacity, and when the first processing capacity is higher than the second processing capacity, the second intermediate node sends a first message to the third intermediate node, and the steps are repeated until the plurality of intermediate nodes are traversed, and the first message is returned to the first intermediate node.

Wherein the third intermediate node is any one of the plurality of intermediate nodes other than the first intermediate node and the second intermediate node. The second processing capability is used for indicating the processing capability of the consuming node corresponding to the second intermediate node, and the implementation manner of the second processing capability is similar to that of the first processing capability, and is not described herein again.

The second intermediate node responds to the received first message, acquires the second processing capability, and when the first processing capability is higher than the second processing capability, sends the first message to the third intermediate node, repeats the steps until a plurality of intermediate nodes are traversed, returns the first message to the first intermediate node, that is, the third intermediate node acquires the processing capability of the corresponding consuming node, when the first processing capability is higher than the processing capability, continues to send the first message to the fourth intermediate node, similarly, sends the first message to the nth intermediate node, and when the first processing capability is higher than the processing capability of the consuming node corresponding to the nth intermediate node, returns the first message to the first intermediate node.

It should be noted that, the first packet may further include a hop count, the first intermediate node may set the hop count to 0 when receiving the data to be processed, and the second intermediate node may further add one to the hop count in the first packet, that is, set to 1 when sending the first packet to the second intermediate node, and so on, where the hop count in the first packet returned to the first intermediate node is equal to the total number n of the plurality of intermediate nodes.

As can be seen from the above, each intermediate node forms a loop, the message information flow is that the messages are forwarded between the loops, in addition, when a certain intermediate node fails, a new loop can be converged, the whole message flow is not affected, the self-recovery capability of the redundant intermediate node loop line is ensured, for example, when a second intermediate node fails, the messages can be transmitted according to the new loop, such as: the original loop is the first intermediate node-the second intermediate node … … nth intermediate node, the current loop is the first intermediate node-the third intermediate node … … nth intermediate node, and of course, the loop may also be the first intermediate node-the fourth intermediate node-the third intermediate node … … nth intermediate node, which is not limited in this embodiment.

S104, the first intermediate node determines a first target consuming node from the corresponding consuming nodes, so that the first target consuming node processes the data to be processed.

The first target consuming node may be any one of consuming nodes corresponding to the first intermediate node, or may have the highest processing capability in the consuming nodes corresponding to the first intermediate node, where the processing capability of the consuming node may be determined according to an attribute parameter of the consuming node, for example, a weighted sum of attribute parameters of the consuming node is adopted, so long as the processing capability of the consuming node can be distinguished, which is not limited in this embodiment.

In this embodiment, when the first intermediate node receives the first message, the first message includes an identifier of the first intermediate node, which indicates that no intermediate node is about to preempt the data to be processed, and one of the consumption nodes corresponding to the first intermediate node processes the data to be processed, and specifically, the first intermediate node determines a first target consumption node from the corresponding consumption nodes, so that the first target consumption node processes the data to be processed.

The data processing method provided in this embodiment includes: responding to the received data to be processed, the first intermediate node obtains a first processing capability, the first intermediate node sends a first message to the second intermediate node, the second intermediate node obtains a second processing capability, and when the first processing capability is higher than the second processing capability, the first message is sent to the third intermediate node, the steps are repeated until a plurality of intermediate nodes are traversed, and the first message is returned to the first intermediate node, and the first intermediate node determines a first target consumption node from the corresponding consumption nodes so that the first target consumption node processes the data to be processed. According to the scheme, the central control node and the centralized message queue are removed, the competition problem caused by one-to-many or less-to-many is solved, the pipeline structure enables the data to be processed to be distributed to all consumption nodes as much as possible, the first target consumption node corresponding to the data to be processed can be rapidly determined by circulating the data to be processed among the intermediate nodes, the data throughput is improved, and in addition, the time of user data circulation replaces the synchronous information time of all consumption nodes.

In an exemplary embodiment, when the first processing capability is lower than the second processing capability, the second intermediate node may further continue to send the second message to the third intermediate node, and repeat this step until a plurality of intermediate nodes are traversed, and return the traversed target message to the first intermediate node. The following describes in detail the embodiment of fig. 4, and fig. 4 shows a second flowchart of the data processing method provided in the embodiment of the present application.

As shown in fig. 4, the data processing method provided in this embodiment may further include the following steps:

s201, when the first processing capacity is lower than the second processing capacity, the second intermediate node sends a second message to the third intermediate node, the steps are repeated until a plurality of intermediate nodes are traversed, and the traversed target message is returned to the first intermediate node.

The target message comprises: indication information of the highest processing capability in the traversal process and identification of the target intermediate node corresponding to the highest processing capability.

On the basis of fig. 2, referring to fig. 6, when the first processing capability is lower than the second processing capability, the second intermediate node sends a second message to the third intermediate node, the second message includes indication information of the second processing capability and identification of the second intermediate node, the steps are repeated until a plurality of intermediate nodes are traversed, the traversed target message is returned to the first intermediate node, that is, the third intermediate node obtains the processing capability of the corresponding consuming node, when the second processing capability is higher than the processing capability, the second message continues to be sent to the fourth intermediate node, and similarly, the nth intermediate node returns the target message to the first intermediate node.

Similarly, when the second processing capability is lower than the processing capability, the third intermediate node sends a third message to the fourth intermediate node, wherein the third message comprises the indication information of the processing capability of the consumption node corresponding to the third intermediate node and the identification of the second intermediate node, and similarly, the nth intermediate node returns the target message to the first intermediate node.

S202, when the identification of the target intermediate node is different from that of the first intermediate node, the first intermediate node sends the target message to the second intermediate node.

And S203, when the second processing capacity is equal to the highest processing capacity, the second intermediate node determines a second target consumption node from the corresponding consumption nodes so that the second target consumption node processes the data to be processed.

The first intermediate node receives a target message, wherein the target message comprises an identifier of a target intermediate node corresponding to the highest processing capacity, and when the identifier of the target intermediate node is different from the identifier of the first intermediate node, the first intermediate node indicates that the intermediate node is to occupy the data to be processed, and then the first intermediate node sends the target message to the second intermediate node.

And then, the second intermediate node receives the target message, and when the second processing capacity is equal to the highest processing capacity, the second intermediate node indicates that the second intermediate node wants to preempt the data to be processed, and the second intermediate node determines a second target consumption node from the corresponding consumption nodes so that the second target consumption node processes the data to be processed.

The second target consuming node may be any one of consuming nodes corresponding to the second intermediate node, or may have the highest processing capability in the consuming nodes corresponding to the second intermediate node, where the processing capability of the consuming node may be determined according to the attribute parameters of the consuming node, for example, a weighted sum of the attribute parameters of the consuming node is adopted, so long as the processing capability of the consuming node can be distinguished.

It should be noted that, the first packet may further include a hop count, the first intermediate node may set the hop count to 0 when receiving the data to be processed, and the second intermediate node may further add one to the hop count in the first packet, that is, set to 1 when sending the second packet to the second intermediate node, and so on, where the hop count in the target packet returned to the first intermediate node is equal to the total number n of the plurality of intermediate nodes.

For example, the scores are used to indicate the processing capability, and the processing capability includes three intermediate nodes, namely, a first intermediate node 1, a second intermediate node 2 and a third intermediate node 3, where the corresponding scores are respectively: 50 minutes, 60 minutes and 55 minutes, the second intermediate node 2 is about to preempt the data to be processed, and the data passes through the third intermediate node 3 (namely, the third intermediate node 3 is about 60 minutes), then returns to the first intermediate node 1 (namely, the first intermediate node is about 60 minutes), and then returns to the second intermediate node 2, and the propagation path of the process is as follows: 1- >2- >3- >1- >2, go through 4 hops altogether, the hop count is 5.

It will be appreciated that in determining that there is preemption of the first intermediate node 1, the reason for the need to transfer again is that: it is necessary for all other intermediate nodes to recognize and know that the data is to be preempted, the above-mentioned captain.

The data processing method provided in this embodiment includes: when the first processing capability is lower than the second processing capability, the second intermediate node sends a second message to the third intermediate node, the steps are repeated until a plurality of intermediate nodes are traversed, the traversed target message is returned to the first intermediate node, the second message comprises indication information of the second processing capability and identification of the second intermediate node, and the target message comprises: and when the second processing capacity is equal to the highest processing capacity, the second intermediate node determines a second target consumption node from the corresponding consumption nodes, so that the second target consumption node processes the data to be processed. According to the scheme, the data to be processed can be transmitted to the consumption node for processing by establishing the circulation, so that the data throughput is improved.

In an exemplary embodiment, when the second processing capability is lower than the highest processing capability, the second intermediate node may further send a target packet to the third intermediate node, and repeat this step until reaching the target intermediate node corresponding to the highest processing capability. In the following, a detailed description will be given with reference to the embodiment of fig. 5, and fig. 5 shows a third flowchart of the data processing method provided in the embodiment of the present application.

As shown in fig. 5, the data processing method provided in this embodiment further includes the following steps:

and S301, when the second processing capacity is lower than the highest processing capacity, the second intermediate node sends a target message to the third intermediate node, and the step is repeated until the target intermediate node corresponding to the highest processing capacity is reached.

S302, the target intermediate node determines a third target consumption node from the corresponding consumption nodes, so that the third target consumption node processes the data to be processed.

And when the first processing capacity is lower than the second processing capacity, the second intermediate node sends a target message to the third intermediate node until reaching the target intermediate node corresponding to the highest processing capacity, namely, when determining that the data to be processed is to be preempted, the second intermediate node sends the target message to the third intermediate node again, if the second processing capacity is lower than the highest processing capacity, the second intermediate node is not the intermediate node for the data to be processed, the second intermediate node sends the target message to the third intermediate node, if the processing capacity of the consumption node corresponding to the third intermediate node is equal to the highest processing capacity, the third intermediate node is indicated to be preempted for the data to be processed, namely, the third intermediate node is the target intermediate node, otherwise, the third intermediate node sends the target message to the fourth intermediate node until reaching the target intermediate node corresponding to the highest processing capacity.

Then, the target intermediate node receives the target message, and determines a third target consuming node from the corresponding consuming nodes, so that the third target consuming node processes the data to be processed.

The third target consuming node may be any one of consuming nodes corresponding to the target intermediate node, or may be a consuming node with the highest processing capability corresponding to the target intermediate node, where the processing capability of the consuming node may be determined according to an attribute parameter of the consuming node, for example, a weighted sum of attribute parameters of the consuming node is adopted, so long as the processing capability of the consuming node can be distinguished, which is not limited in this embodiment.

It should be noted that, when the hop count in the target message is n and the first intermediate node sends the target message to the second intermediate node, the hop count may be set to n+1, and so on, and the hop count is increased by one every time a message is transmitted.

For example, the scores are used to indicate the processing capability, and the processing capability includes three intermediate nodes, namely, a first intermediate node 1, a second intermediate node 2 and a third intermediate node 3, where the corresponding scores are respectively: 50 minutes, 60 minutes and 70 minutes, the second intermediate node 2 and the third intermediate node 3 want to preempt the data to be processed, return to the first intermediate node 1 (namely the third intermediate node is the first 70 minutes), then return to the second intermediate node 2 (namely the second intermediate node is the first 70 minutes), and then return to the C, and the propagation path of the process is as follows: 1- >2- >3- >1- >2- >3, go through 5 hops altogether, the hop count is 6.

It follows that the number of hops in this case is an integer multiple of the intermediate node, which is the worst preemption case, the root cause being: the third intermediate node 3 upstream of the first intermediate node 1 has exactly the highest score, so the number of flows in this case is the largest.

According to the data processing method provided by the embodiment, when the second processing capacity is lower than the highest processing capacity, the second intermediate node sends the target message to the third intermediate node, the steps are repeated until the target intermediate node corresponding to the highest processing capacity is reached, and the target intermediate node determines the third target consumption node from the corresponding consumption nodes, so that the third target consumption node processes the data to be processed. According to the scheme, the circulation flow is established to transfer the data to be processed to the consumption node for processing, the data to be processed can be ensured to be processed through the maximum two circulation time, and the data throughput is effectively improved.

For example, based on the embodiment of fig. 3, when the first intermediate node determines the first target consumption node from the corresponding consumption nodes, the first intermediate node may further send the data to be processed to the first target consumption node, and when the first target consumption node acquires the data to be processed, the data to be processed is stored in the corresponding processing sequence, and when the first target consumption node does not process the data to be processed under the preset condition, the first intermediate node sets the first processing capability to the preset minimum processing capability, and sends the data to be processed to the second intermediate node. In the following, a detailed description will be given with reference to the embodiment of fig. 6, and fig. 6 shows a flowchart of a data processing method according to an embodiment of the present application.

As shown in fig. 6, the data processing method provided in this embodiment further includes the following steps:

s401, the first target consumption node stores the data to be processed into a corresponding processing sequence.

S402, if the first target consumption node does not process the data to be processed under the preset condition, the first intermediate node sets the first processing capacity as the preset lowest processing capacity, and sends the data to be processed to the second intermediate node.

Each consumption node corresponds to at least one processing sequence, and the processing sequence comprises at least one data waiting to be processed, that is, after the first target consumption node receives the data waiting to be processed, the data waiting to be processed is stored in the corresponding processing sequence, and the data waiting to be processed are queued in the processing sequence.

If the first target consumption node does not process the data to be processed under the preset condition, the first intermediate node sets the first processing capacity as the preset lowest processing capacity, that is to say, the first processing capacity of the first intermediate node is the lowest processing capacity, the first intermediate node does not have the opportunity of preempting the data to be processed, and the data to be processed is sent to the second intermediate node.

In one possible implementation, the preset condition includes that the number of polls of the first target consumption node is greater than or equal to a preset number of polls, where the number of polls is: and polling the first target consumption node for the number of times of processing the data to be processed.

In another possible implementation, the preset conditions further include: the waiting time length of the data to be processed is longer than or equal to the preset time length.

That is, when the number of polling times is greater than or equal to a preset number of times, or when the waiting time period of the data to be processed in the processing sequence is greater than or equal to a preset length of time, the first intermediate node sets the first processing capability to a preset minimum processing capability, and transmits the data to be processed to the second intermediate node, or other intermediate nodes.

Then, the second intermediate node transfers the data to be processed according to the steps executed by the first intermediate node in fig. 3, that is, all the intermediate nodes except for the first intermediate node have an opportunity to preempt the data to be processed, and the specific implementation process is similar to that of the embodiment in fig. 3, and will not be repeated here.

It should be noted that, when the data to be processed is sent to the second intermediate node, the second processing capability of the second intermediate node may not be the last processing capability, and may be changed along with the change of the attribute parameter of the consuming node of the second intermediate node object.

It will be appreciated that the reason for the above is that: and (3) asynchronous behavior, namely when the first target consumption node considers that the processing capacity exists, the data to be processed is taken, the processing capacity of the first target consumption node is set to be the lowest when the follow-up finding that the processing capacity does not exist, the data to be processed is sent out again when the preset condition is met, so that the intermediate nodes corresponding to other consumption nodes with the processing capacity are used for preempting, if the processing capacity of the other intermediate nodes is the lowest, the data to be processed is returned to the first intermediate node, and when the preset condition is met, the steps are continuously executed.

In addition, for the second target consuming node in fig. 4, the to-be-processed data may also be stored in the corresponding processing sequence, and if the second target consuming node does not process the to-be-processed data under the preset condition, the second intermediate node sets the second processing capability to the preset minimum processing capability, and sends the to-be-processed data to other intermediate nodes, such as the third intermediate node.

For the third target consuming node in fig. 5, the data to be processed may also be stored in the corresponding processing sequence, and if the third target consuming node does not process the data to be processed under the preset condition, the third intermediate node sets the processing capability of the corresponding consuming node to the preset minimum processing capability, and sends the data to be processed to other intermediate nodes, such as the second intermediate node.

The data processing method provided in this embodiment includes: the first target consumption node stores the data to be processed into a corresponding processing sequence, the processing sequence comprises at least one data to be processed, and if the first target consumption node does not process the data to be processed under the preset condition, the first intermediate node sets the first processing capacity as the preset minimum processing capacity and sends the data to be processed to the second intermediate node. The scheme ensures that the data can be processed robustly, and once a certain consumption node cannot be processed, the opportunity can be given out for other consumption nodes to process, thereby improving the data processing efficiency.

On the basis of the above method embodiment, when the number of consuming nodes is too large, in order to effectively reduce the transmission length of the message, the plurality of intermediate nodes may illustratively include a plurality of intermediate parent nodes, and the plurality of intermediate nodes further include intermediate child nodes corresponding to each intermediate parent node, and first, the intermediate child nodes are transmitted between the intermediate parent nodes, and after a certain intermediate parent node is determined, the intermediate child nodes corresponding to the intermediate parent node are transmitted between the intermediate child nodes.

The first intermediate node sends a first message to the second intermediate node, comprising: the first intermediate father node sends a first message to the second intermediate father node; when the first processing capability is higher than the second processing capability, the second intermediate node sends a first message to the third intermediate node, and repeats the steps until a plurality of intermediate nodes are traversed, and returns the first message to the first intermediate node, including: and when the first processing capacity is higher than the second processing capacity, the second intermediate father node sends a first message to the third intermediate father node, and the steps are repeated until a plurality of intermediate father nodes are traversed, and the first message is returned to the first intermediate father node.

The first intermediate node determines a first target consuming node from the corresponding consuming nodes, comprising: the first intermediate father node determines a target intermediate child node from the corresponding intermediate child nodes; and the target intermediate child node determines a first target consuming node from consuming nodes corresponding to the target intermediate child node.

Fig. 7 shows a fifth flowchart of a data processing method according to an embodiment of the present application, and fig. 8 shows a second flowchart of a data processing system according to an embodiment of the present application.

As shown in fig. 7, the data processing method provided in this embodiment further includes the following steps:

s501, responding to received data to be processed, and acquiring a first processing capability by a first intermediate father node.

Referring to fig. 8, the plurality of intermediate parent nodes include a first intermediate parent node a, a second intermediate parent node B, a third intermediate parent node C, a fourth intermediate parent node D, a fifth intermediate parent node E, and a sixth intermediate parent node F, the first intermediate child node to the fourth intermediate child node corresponding to the first intermediate parent node a are A1, A2, A3, A4, respectively, and so on, and the first intermediate child node to the sixth intermediate child node corresponding to the sixth intermediate parent node F are F1, F2, F3, and F4.

Optionally, the communication duration of the intermediate child node corresponding to each intermediate parent node is within a preset range, that is, the communication durations of A1, A2, A3, A4 are within a certain preset range, and the communication durations of D1, D2, D3, D4 are within another preset range.

In this embodiment, the first intermediate parent node is any one of a plurality of intermediate parent nodes, and in response to data to be processed received from the production node, the first intermediate parent node obtains a first processing capability for indicating a processing capability of a corresponding consumption node, that is, the first processing capability indicates a processing capability of a consumption node corresponding to the first intermediate child node.

The method for acquiring the first processing capability is described with reference to the embodiment of fig. 3, and is not described herein.

S502, the first intermediate father node sends a first message to the second intermediate father node.

S503, when the first processing capacity is higher than the second processing capacity, the second intermediate father node sends a first message to the third intermediate father node, and the steps are repeated until a plurality of intermediate father nodes are traversed, and the first message is returned to the first intermediate father node.

With continued reference to fig. 8, the first intermediate parent node a sends a first message to the second intermediate parent node B, where the first message includes indication information of the first processing capability and an identifier of the first intermediate node, where the identifier of the first intermediate node may be an identifier of the first intermediate parent node a.

The second intermediate parent node is any one of the plurality of intermediate parent nodes other than the first intermediate parent node. The second intermediate father node obtains the second processing capability in response to the received first message, and when the first processing capability is higher than the second processing capability, sends the first message to the third intermediate father node, and repeats this step until a plurality of intermediate father nodes, for example A, B, C, D, E, F shown in fig. 8, are traversed, that is, the third intermediate father node obtains the processing capability of the consumption node corresponding to the corresponding third intermediate child node, and when the first processing capability is higher than the processing capability, continues sending the first message to the fourth intermediate father node, and similarly, sends the first message to the sixth intermediate father node, and when the first processing capability is higher than the processing capability of the consumption node corresponding to the sixth intermediate father node, returns the first message to the first intermediate father node.

It should be noted that, the above description is only given by taking the number of intermediate parent nodes as six as an example, and in practical application, the number of intermediate parent nodes includes but is not limited to six.

S504, the first intermediate father node determines a target intermediate child node from the corresponding intermediate child nodes.

In this embodiment, when a first intermediate parent node receives a first message, where the first message includes an identifier of the first intermediate parent node, which indicates that there is no desire of the intermediate parent node to preempt the data to be processed, the first intermediate parent node obtains a target intermediate child node from a corresponding first intermediate child node, and then the target intermediate child node determines a first target consumption node from consumption nodes corresponding to the target intermediate child node, so that the first target consumption node processes the data to be processed.

With continued reference to fig. 8, the intermediate nodes A1, A2, A3, and A4 corresponding to the first intermediate parent node a may be the target intermediate child node with the highest processing capacity among the intermediate nodes A1, A2, A3, and A4, or any one of the intermediate nodes A1, A2, A3, and A4.

Optionally, the first intermediate parent node determines a target intermediate child node from the corresponding intermediate child nodes, including:

the first intermediate father node sends data to be processed to the first intermediate child node;

Responding to the received data to be processed, the first intermediate child node obtains a third processing capability, and the third processing capability is used for indicating the processing capability of a consumption node corresponding to the first intermediate child node;

the first intermediate child node sends a third message to the second intermediate child node, wherein the third message comprises indication information of third processing capacity and identification of the first intermediate child node;

the second intermediate child node obtains fourth processing capability, and when the third processing capability is higher than the fourth processing capability, the third message is sent to the third intermediate child node, the steps are repeated until the intermediate child node corresponding to the first intermediate father node is traversed, and the third message is returned to the first intermediate child node;

the first intermediate child node determines the first intermediate child node as a target intermediate child node.

Specifically, when the processing capacity of the consumption node corresponding to the first intermediate father node is determined to be highest, the data to be processed is streamed between the intermediate child nodes corresponding to the first intermediate father node, so as to determine a target intermediate child node, and the specific process is as follows: the first intermediate father node a sends the data to be processed to the first intermediate child node, such as A1, then the first intermediate child node obtains a third processing capability, which is used to indicate the processing capability of the consuming node corresponding to the first intermediate child node, then the first intermediate child node A1 sends a third message to the second intermediate child node, such as A2, the third message includes the indication information of the third processing capability and the identifier of the first intermediate child node A1, then the second intermediate child node A2 obtains a fourth processing capability, and when the third processing capability is higher than the fourth processing capability, sends a third message to the third intermediate child node, such as A4, and repeats this step until the intermediate child node corresponding to the first intermediate father node is traversed, such as A1, A2, A3, A4 in fig. 8, and returns a third message to the first intermediate child node A1, and the first intermediate child node determines the first intermediate child node as the target intermediate child node.

The method for obtaining the third processing capability refers to the first processing capability shown in fig. 3, and is not described herein.

In this embodiment, after a certain intermediate parent node is determined, a process of transferring data to be processed between intermediate child nodes corresponding to the intermediate parent node is similar to a process of transferring data to be processed between intermediate nodes in the embodiment of fig. 3, which is both for determining a node with the highest capability.

Of course, the cases of fig. 4, fig. 5, and fig. 6 may also occur during the transfer between intermediate child nodes, and the specific reference may be made to the above embodiments, which are not repeated herein.

S505, the target intermediate child node determines a first target consumption node from consumption nodes corresponding to the target intermediate child node.

After determining a target intermediate child node in the intermediate child nodes corresponding to the first intermediate parent node, the target intermediate child node determines a first target consumption node from consumption nodes corresponding to the target intermediate child node. The first target consuming node may be any consuming node corresponding to the first intermediate sub-node, or may be a consuming node corresponding to the first intermediate sub-node, where the processing capability of the consuming node may be determined according to an attribute parameter of the consuming node, for example, a weighted sum of attribute parameters of the consuming node is adopted, so long as the processing capability of the consuming node can be distinguished, which is not limited in this embodiment.

The data processing method provided in this embodiment includes: responding to received data to be processed, acquiring a first processing capability by a first intermediate father node, sending a first message to a second intermediate father node by the first intermediate father node, sending the first message to a third intermediate father node by the second intermediate father node when the first processing capability is higher than the second processing capability, repeating the steps until a plurality of intermediate father nodes are traversed, returning the first message to the first intermediate father node, determining a target intermediate child node from corresponding intermediate child nodes by the first intermediate father node, and determining a first target consumption node from consumption nodes corresponding to the target intermediate child nodes by the target intermediate child nodes. Therefore, when the number of consumption nodes is particularly huge, the length of the data to be circulated is effectively reduced, so that the data is distributed to the proper consumption nodes, the response speed is improved, and the throughput is further improved.

Based on the same inventive concept, the embodiment of the present application further provides a data processing system corresponding to the data processing method, and since the principle of solving the problem by the system in the embodiment of the present application is similar to that of the data processing method in the embodiment of the present application, the implementation of the system may refer to the implementation of the method, and the repetition is omitted.

FIG. 9 is a schematic diagram of a data processing system according to an embodiment of the present application, and as shown in FIG. 9, the data processing system 30 includes:

a plurality of intermediate node devices 31 and a consuming node device 32 connected to each intermediate node device;

the plurality of intermediate node devices 31 are respectively configured to perform the steps performed by the first intermediate node, the second intermediate node, and the third intermediate node in the above-described method embodiments;

for each intermediate node device, there is a target consuming node device in the consuming node devices connected to the intermediate node device, where the multiple target consuming node devices are respectively configured to execute the steps executed by the first target consuming node, the second target consuming node, and the third target consuming node in the foregoing method embodiment.

Optionally, the plurality of intermediate node devices 31 and the consuming node device 32 are each independent physical devices.

Optionally, the plurality of intermediate node devices 31 and the consuming node device 32 are hardware nodes of the same physical device.

The specific implementation process may refer to the relevant description in the above method embodiment, and will not be described in detail here.

It will be appreciated that the various numerical numbers referred to in the embodiments of the present application are merely for ease of description and are not intended to limit the scope of the embodiments of the present application. The sequence numbers of the above-mentioned processes do not mean the sequence of execution sequence, and the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

The foregoing is merely a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes or substitutions are covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing method, applied to a data processing system, the data processing system including a plurality of intermediate nodes and consumption nodes corresponding to each intermediate node, the method comprising:

The first intermediate node determines a first target consumption node from corresponding consumption nodes so that the first target consumption node processes the data to be processed;

the method further comprises the steps of:

2. The method according to claim 1, wherein the method further comprises:

3. The method according to claim 1, wherein the method further comprises:

4. A method according to claim 3, wherein the preset conditions include: the polling times of the first target consumption node are greater than or equal to preset times, wherein the polling times are as follows: and polling the first target consumption node for the number of times of processing the data to be processed.

5. A method according to claim 3, wherein the preset conditions further comprise: the waiting time length of the data to be processed is longer than or equal to the preset time length.

6. The method of any of claims 1-5, wherein the first intermediate node acquiring a first processing capability comprises:

7. The method of claim 6, wherein the attribute parameters include at least one of the following:

8. The method of claim 1, wherein the plurality of intermediate nodes comprises a plurality of intermediate parent nodes;

9. The method of claim 8, wherein the second intermediate node sending the first message to a third intermediate node when the first processing capability is higher than the second processing capability, repeating the steps until the plurality of intermediate nodes are traversed, and returning the first message to the first intermediate node, comprises:

10. The method of claim 9, wherein the plurality of intermediate nodes further comprises an intermediate child node corresponding to each intermediate parent node;

11. The method of claim 10, wherein the first intermediate parent node determines a target intermediate child node from the corresponding intermediate child nodes, comprising:

The first intermediate child node sends a third message to a second intermediate child node, wherein the third message comprises indication information of the third processing capability and an identifier of the first intermediate child node;

12. A data processing system, comprising:

the plurality of intermediate node devices being configured to perform the steps performed by the first intermediate node, the second intermediate node, and the third intermediate node, respectively, of any of claims 1-11;

for each intermediate node device, there is a target consuming node device in the consuming node devices connected to the intermediate node device, where the plurality of target consuming node devices are configured to execute the steps executed by the first target consuming node, the second target consuming node, and the third target consuming node according to any one of claims 1-11, respectively.

13. The system of claim 12, wherein the plurality of intermediate node devices and the consuming node device are each separate physical devices.

14. The system of claim 12, wherein the plurality of intermediate node devices and the consuming node device are each hardware nodes of a same physical device.