WO2017032212A1

WO2017032212A1 - Data stream processing method and apparatus

Info

Publication number: WO2017032212A1
Application number: PCT/CN2016/093588
Authority: WO
Inventors: 李旭良; 李嘉; 刘杰
Original assignee: 华为技术有限公司
Priority date: 2015-08-27
Filing date: 2016-08-05
Publication date: 2017-03-02
Also published as: CN106487694B; CN106487694A

Abstract

Disclosed are a data stream processing method and apparatus. The method can comprise: a processing node of a real-time calculation system receiving a control stream which is sent by a management node of the real-time calculation system to adjust a topology structure of the real-time calculation system, wherein the control stream is used for describing a current topology structure needing to be updated of the real-time calculation system; the processing node updating a data stream flow table according to the current topology structure needing to be updated described by the control stream, wherein the updated data stream flow table comprises a data stream flow path matching the current topology structure needing to be updated; and when the processing node receives a data stream, the processing node performing service processing on the data stream, and sending the data stream having been subjected to the service processing according to the data stream flow path comprised in the updated data stream flow table. The embodiments of the present invention can dynamically adjust a topology structure of a real-time calculation system.

Description

Data stream processing method and device

Technical field

The present invention relates to the field of communications technologies, and in particular, to a data stream processing method and apparatus.

Background technique

With the advancement of technology and the development of user needs, there is a large amount of real-time data. Among them, real-time data is strong in real-time, and the amount of data is large, and there is no end. At present, real-time data is mainly calculated by flow computing in real-time computing system. Calculation, for example: Storm application. However, in the current real-time computing system, since the code is implemented by relying on the JAVA application programming interface (API), and the package submission task is implemented, the business logic of the real-time computing system that is already running is immutable, that is, real-time. The topology structure of the system is immutable at runtime in the computing system. However, many of the requirements nowadays are to dynamically adjust the processing logic of the real-time computing system, that is, to dynamically adjust the topology of the real-time computing system so that it can adapt to the needs of the business. It can be seen that how to dynamically adjust the topology of the real-time computing system is currently a technical problem that needs to be solved urgently.

Summary of the invention

The embodiment of the invention provides a data stream processing method and device, which can dynamically adjust the topology structure of the real-time computing system.

In a first aspect, an embodiment of the present invention provides a data stream processing method, including:

a processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update structure;

The processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Circulation path

When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service according to the data flow circulation path included in the updated data flow circulation table. The processed data stream.

In a first possible implementation manner of the first aspect, the method further includes:

Transmitting, by the processing node, the updated data flow circulation table to a shared storage node of the real-time computing system;

When the processing node performs failure recovery, the processing node acquires the updated data flow distribution table from the shared storage node, and transmits the data flow according to the updated data flow distribution table.

In conjunction with the first aspect or the first possible implementation of the first aspect, in a second possible implementation of the first aspect, the method further includes:

And the processing node feeds back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the control to the management node flow;

When the update result indicates that the update fails, the processing node receives the control flow sent by the management node, and updates the data flow circulation table according to the topology that is currently updated according to the control flow.

In conjunction with the first aspect, or the first possible implementation of the first aspect, in a third possible implementation manner of the first aspect, the method further includes:

The processing node feeds back an update result of the data flow table update to an output node of the real-time computing system, and the output node summarizes the update result fed back by all processing nodes of the real-time computing system, and Output summary results.

In a second aspect, an embodiment of the present invention provides a data stream processing method, including:

a management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update structure;

Transmitting, by the management node, the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow description The updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs the data flow on the data flow. The service processes, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.

In a first possible implementation manner of the second aspect, the method further includes:

Establishing and maintaining a connection between the management node and each of the processing nodes in the real-time computing system;

The management node sends the control flow to a processing node for performing service processing on the data stream, including:

The management node transmits the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.

In a third aspect, an embodiment of the present invention provides a data stream processing method, including:

A source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update;

And the source node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes a data flow that matches the topology that needs to be updated currently. Circulation path

Sending, by the source node, the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently;

Receiving, by the source node, the data stream, and transmitting the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where the processing node performs a service on the data flow Processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.

In a first possible implementation manner of the third aspect, the method further includes:

Sending, by the source node, the updated data flow circulation table to a shared storage node of the real-time computing system;

When the source node performs fault recovery, the source node acquires the updated data flow flow table from the shared storage node, and performs data flow transmission according to the updated data flow flow table.

With reference to the third aspect, or the first possible implementation manner of the third aspect, in a second possible implementation manner of the third aspect, the method further includes:

Receiving, by the source node, the processing node fed back by the processing node to update an update result of the data flow distribution table;

When the update result indicates that the update fails, the source node sends the control flow to the management node, so that the management node sends the control flow to the processing node, and the processing node follows the The topology that is currently updated as described by the control flow updates the data flow flow table.

In a fourth aspect, an embodiment of the present invention provides a data stream processing apparatus, where the apparatus is applied to a processing node of a real-time computing system, including: a receiving unit, a first updating unit, and a first sending unit, where:

The receiving unit is configured to receive a control flow sent by a management node of a real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update currently. ;

The first update unit is configured to update a data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the topology that needs to be updated currently Structure matching data flow path;

The first sending unit is configured to: when the processing node receives the data stream, perform service processing on the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table. After the data stream.

In a first possible implementation manner of the fourth aspect, the device further includes:

a second sending unit, configured to send the updated data flow distribution table to a shared storage node of the real-time computing system;

And a recovery unit, configured to: when the processing node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.

With reference to the fourth aspect, or the first possible implementation manner of the fourth aspect, in a second possible implementation manner of the fourth aspect, the device further includes:

a first feedback unit, configured to feed back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the update to the management node The control flow;

a second updating unit, configured to: when the update result indicates that the first update unit fails to update, Receiving the control flow sent by the management node, and updating the data flow circulation table according to the topology that is currently updated according to the control flow.

With reference to the fourth aspect, or the first possible implementation manner of the fourth aspect, in a third possible implementation manner of the fourth aspect, the device further includes:

a second feedback unit, configured to feed back an update result of the data flow table update to an output node of the real-time computing system, where the update result is fed back by the output node to all processing nodes of the real-time computing system Summarize and output the summary results.

In a fifth aspect, an embodiment of the present invention provides a data stream processing apparatus, including: a receiving unit and a sending unit, where:

The receiving unit is configured to receive a control flow sent by a source node of a real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update currently. ;

The sending unit is configured to send the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data according to the topology that is currently updated according to the control flow. a flow distribution table, wherein the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node The data stream performs service processing, and transmits the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.

In a first possible implementation manner of the fifth aspect, the device further includes:

a holding unit, configured to establish and maintain a connection with each of the processing nodes in the real-time computing system;

The sending unit is configured to send the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.

In a sixth aspect, an embodiment of the present invention provides a data stream processing apparatus, where the apparatus is applied to a source node of a real-time computing system, including: an acquiring unit, an updating unit, a first sending unit, and a second sending unit, where:

The acquiring unit is configured to acquire a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update at present;

The updating unit is configured to update a data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the current requirement The data flow path that matches the topology to be updated;

The first sending unit is configured to send the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, And causing the processing node to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes a topology that matches the current need to be updated. Data flow path;

The second sending unit is configured to receive a data stream, and send the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where the processing node is The data stream performs service processing, and transmits the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.

In a first possible implementation manner of the sixth aspect, the device further includes:

a third sending unit, configured to send the updated data flow distribution table to a shared storage node of the real-time computing system;

And a recovery unit, configured to: when the source node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.

With reference to the sixth aspect, or the first possible implementation manner of the sixth aspect, in a second possible implementation manner of the sixth aspect, the device further includes:

a receiving unit, configured to receive, by the processing node fed back by the processing node, an update result of updating the data flow distribution table;

a fourth sending unit, configured to: when the update result indicates that the update fails, send the control flow to the management node, so that the management node sends the control flow to the processing node, by the processing node Updating the data flow flow table according to the topology currently required to be updated as described by the control flow.

In the above technical solution, the processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system. The topology structure currently needs to be updated; the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the current needs update Topology structure matching data flow path; when the processing node receives the data stream, the processing node pairs the number Performing service processing according to the flow, and transmitting the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table. In this way, the processing logic of the real-time computing system can be updated by updating the data flow table in real time, that is, the topology of the real-time computing system can be dynamically adjusted.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any creative work.

FIG. 1 is a structural diagram of a real-time computing system to which a data stream processing method according to an embodiment of the present invention is applicable;

2 is a schematic flowchart of a data stream processing method according to an embodiment of the present invention;

3 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention;

4 is a schematic flowchart diagram of another data stream processing method according to an embodiment of the present invention;

FIG. 5 is a schematic flowchart diagram of another data stream processing method according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of control flow transmission using a Storm application as an example according to an embodiment of the present invention; FIG.

FIG. 7 is a schematic diagram of data flow transmission using a Storm application as an example according to an embodiment of the present invention; FIG.

FIG. 8 is a schematic structural diagram of a data stream processing apparatus according to an embodiment of the present invention; FIG.

FIG. 9 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 11 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 12 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 13 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 14 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 15 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 16 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 17 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 18 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention;

FIG. 19 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

Please refer to FIG. 1. FIG. 1 is a structural diagram of a real-time computing system applicable to a data stream processing method according to an embodiment of the present invention. As shown in FIG. 1, the method includes: a supervisory body 11, a source node 12, and one or more The processing node 13 and the management module 14 are respectively connected to the source node 12 and the respective processing nodes 13, and the source node 12 is connected to at least one of the processing nodes 13, and the processing nodes 13 are sequentially connected. In addition, the management module 14 includes a management node 141 and an output node 142, wherein the management node 141 is connected to the source node and the respective processing nodes 13, and the output node 142 is connected to the respective processing nodes 13. The connection relationship between the nodes in the real-time computing system is expressed as a topology of the real-time computing system or as a topology model.

It should be noted that the connection between the nodes described in this embodiment may be understood as a connection on a logical connection, and a data flow or a control flow may be transmitted between the connected nodes.

When the real-time computing system is running, the data stream flows from the source node 12, and then the source node 12 transmits the data stream to the corresponding processing node 13 in accordance with the stored data flow flow table, and the processing node 13 performs the service on the received data stream. Processing, wherein the service processing may include real-time calculation of the data stream or understanding of the stream calculation, the processing node 13 transmitting the service-processed data stream to another processing node 13 according to the stored data flow circulation table, by the processing The node 13 transmits the service-processed data stream to another node or outputs the result according to the stored data flow flow table. In the embodiment of the present invention, the data flow distribution table includes a data flow circulation path that matches the topology structure of the real-time computing system, that is, in the embodiment, the data flow distribution table controls the transmission path or the transmission structure of the data flow, such that Both the source node 12 and the processing node 13 can send the data stream that needs to be sent to the corresponding node through the respective stored data flow distribution table. For example, the topology of the real-time computing system includes that the source node 12 is connected to the processing node A, the processing node A is reconnected to the processing node B, the processing node B is reconnected to the processing node C, and the processing node C is the ending node. Thus, the data flow flow table stored at the source node 12 can include the data flow sent by the source node 12 to the processing section. The mapping of the point A, the data flow distribution table stored by the processing node A may include the mapping of the data stream sent by the processing node A to the processing node B, and the data flow distribution table stored by the processing node B may include the data stream sent by the processing node B. To the mapping of the processing node C, the data flow distribution table stored by the processing node C may include a mapping of the processing node C outputting the business processing result. Of course, in this embodiment, the data flow distribution table stored by each node may include all mappings or only include mappings related to itself.

When the real-time computing system described above is running, the control flow can be transmitted from the source node 12 to the management node 141, wherein the control flow is used to describe the topology that the real-time computing system currently needs to update. The management node 141 then transmits the control flow to each processing node 13, so that the processing nodes 13 can update the stored data flow flow table according to the control flow, thereby realizing the dynamic adjustment of the topology of the real-time computing system.

In addition, in the embodiment of the present invention, the real-time computing system may be a distributed system, that is, the foregoing nodes may be distributed and run in different machines, and of course, some of the nodes may be allowed to run in the same machine, for example, a management node. 121 and output node 122 can run in the same machine or in different machines. In addition, in the embodiment of the present invention, these machines are not limited, for example, these machines may be computers or servers.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of a data stream processing method according to an embodiment of the present invention. As shown in FIG. 2, the method includes the following steps:

201. The processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe that the real-time computing system needs to be updated currently. Topology.

In this embodiment, the topology that needs to be updated in the foregoing control flow may be a complete topology that the real-time computing system needs to update. For example, the foregoing control flow may describe the source node, the processing node, and the management node as shown in FIG. 1 . The topology between the node and the output node. Alternatively, the topology that needs to be updated in the foregoing control flow description may be a topology that needs to be adjusted currently, for example, the processing node A is connected to the processing node B in the original topology, and the processing node B is connected to the processing node C, that is, the data stream. The order of circulation is from processing node A to processing node B to processing node C, and currently needs to be adjusted to process node A to connect processing node C, and processing node C to connect to processing node B. Then, the above control flow may only describe the processing node A connection processing node C, processing section The point C reconnects to the topology of the processing node B, that is, after the update, the flow order of the data stream is from the processing node A to the processing node B to the processing node C.

In addition, in this embodiment, the foregoing control flow can be understood as control flow information, that is, the above control flow can be understood as one piece of information.

202. The processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Distribution path.

In this embodiment, the processing node stores a data flow circulation table that matches the original topology structure. When the control flow is received, that is, when the current topology needs to be updated, the data flow distribution table may be updated according to the topology structure currently required to be updated. . The updated data flow circulation table includes a data flow distribution path that matches the topology that needs to be updated, so that the processing node sends the data flow according to the updated topology when transmitting the data flow.

In addition, the above-mentioned data flow path matching the topology that needs to be updated may be understood as a flow path or a circulation structure of the data flow in the topology that is currently required to be updated. For example, if the topology that needs to be updated is the processing node A connection processing node C, then the updated data flow distribution table includes the data stream transmitted from the processing node A to the processing node C, that is, when the processing node performing step 202 is Processing node A, then the updated data flow flow table may include a data flow path for transmitting the data stream to the processing node C.

203. When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service processed data according to the data flow distribution path included in the updated data flow distribution table. flow.

Through the above steps, the processing node can be sent according to the updated topology when transmitting the data stream, so that the topology of the real-time computing system can be dynamically adjusted. In addition, when the real-time computing system performs a huge amount of business operations, the above-mentioned steps dynamically adjust the topology of the real-time computing system without causing interference to the data stream being processed. In addition, when the real-time computing system is a distributed system, and each node runs on a different machine, dynamically adjusting the topology of the real-time computing system through the above steps can avoid the problem caused by the modification lag of a certain machine.

In this embodiment, the processing node may be any processing node in the real-time computing system.

In this embodiment, the processing node of the real-time computing system receives a control flow sent by the management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control The flow is used to describe a topology that the real-time computing system currently needs to update; the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, where the updated data flow is circulated. The table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs service processing on the data flow, and follows the updated data flow. The data flow distribution path included in the flow table transmits the data stream after the business process. In this way, the processing logic of the real-time computing system can be updated by updating the data flow table in real time, that is, the topology of the real-time computing system can be dynamically adjusted.

Referring to FIG. 3, FIG. 3 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention. As shown in FIG. 3, the method includes the following steps:

301. A processing node of a real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe that the real-time computing system needs to be updated currently. Topology.

302. The processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Distribution path.

In this embodiment, when the processing node updates the data flow distribution table, the processing node may suspend the data flow transmission, but may perform service processing on the data flow. Or the processing node may suspend the service processing and data stream transmission of the data stream when updating the data flow distribution table. When the data flow flow table is updated, the suspended data stream transmission and/or the business processing of the data stream are resumed. In this way, when the topology of the real-time computing system is dynamically adjusted, the error handling of the data stream and the blocking effect of the system may occur.

303. When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service processed data according to the data flow distribution path included in the updated data flow distribution table. flow.

The above method may further comprise the following steps:

304. The processing node sends the updated data flow circulation table to a shared storage node of the real-time computing system.

305. When the processing node performs fault recovery, the processing node acquires the updated data flow flow table from the shared storage node, and performs the number according to the updated data flow circulation table. According to the flow of the transmission.

In this implementation manner, the processing node can obtain the data flow circulation table updated in step 302 directly from the shared storage node when the recovery node recovers, so that the processing node can read from the shared storage node after the failure occurs. The data flow table is initialized and initialized, that is, the high availability (HA) mechanism of dynamically adjusting the topology is completed.

In this embodiment, the foregoing method may further include the following steps:

In this embodiment, the update result may be the update result of step 302. When the step 302 is successfully updated, the source node may be notified of the update of the processing node by the update result. When the update fails, the source node is notified by the above update result that the processing node update fails, so that the source node sends the control flow to the management node again, and the management node sends the control flow to the processing node again to make the processing node again. Update. Certainly, when the source node sends the control flow to the management node again, it may also carry the identification information of the processing node that failed to update, so that the management node may only send the control flow to the processing node that failed the update, without successfully updating. The processing node sends the control flow again to save transmission resources.

This embodiment can implement the correct feedback of the update result, and if the update fails, the update task can be started again.

Processing node feeds back an update result of the data flow table update to an output node of the real-time computing system, and the output node summarizes the update result fed back by all processing nodes of the real-time computing system, and outputs the summary result.

The output node can obtain the update result fed back by each processing node, so that the output node can summarize it, so that the summary result can be output, for example, sending the summary result to the presentation device, or printing the summary result, etc. This allows the user to know the state of the topology adjustment of the real-time computing system.

In this embodiment, a plurality of optional implementation manners are added on the basis of the embodiment shown in FIG. 2, and the topology of the real-time computing system can be dynamically adjusted.

Referring to FIG. 4, FIG. 4 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention. As shown in FIG. 4, the method includes the following steps:

401. The management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe that the real-time computing system needs to be updated currently. Topology.

In this embodiment, the foregoing control flow may include structural information of a topology that needs to be updated, and may also include a control flow decomposition structure identifier of the management node or the management module internal processing logic, that is, the control flow may be decomposed by the structure identifier and The structure information identifies the topology that needs to be updated currently.

402. The management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow. The updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs the data flow on the data flow. The service processes, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.

In this embodiment, the management node may send the control flow to all processing nodes in the real-time computing system, so that all processing nodes update the stored data flow table. For example, the above method may further include the following steps:

The step of the foregoing management node sending the control flow to the processing node for performing service processing on the data stream may include:

This allows broadcast control to be sent to all processing nodes.

Of course, in this embodiment, the management node may only send the control flow to the partial processing node. For example, when the topology that needs to be updated only involves some processing nodes, then the management section The point can only send control flow to these involved processing nodes, so that these are involved in the processing node to update the data flow flow table, and without being involved, the data flow flow table may not be updated.

In this embodiment, the management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system. The topology that currently needs to be updated; the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing node updates according to the current needs described by the control flow The topology update data flow circulation table, wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing is performed by the processing The node performs service processing on the data stream, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table. This allows dynamic adjustment of the topology of the real-time computing system.

Referring to FIG. 5, FIG. 5 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention. As shown in FIG. 5, the method includes the following steps:

501. A source node of a real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update.

The foregoing control flow may be a control flow in which the source node receives user input. The control flow here can refer to the control flow described in the embodiment shown in FIG. 1-4, and will not be repeatedly described herein.

502. The source node updates the data flow distribution table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Distribution path.

After receiving the above control flow, the source node can decompose the control flow, and update the data flow circulation table according to the topology structure that needs to be updated according to the decomposition. In addition, the source node stores or caches the updated data flow table. Of course, the above control flow can also be stored or cached.

503. The source node sends the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node used for performing service processing on the data flow, so that the processing is performed. The node updates the data flow flow table according to the topology that is currently updated as described by the control flow, wherein the updated data flow flow table includes more The new topology matches the data flow path.

Through this step, the control node can send the control flow to the processing node, and the processing node updates the stored data flow table.

504. The source node receives the data stream, and sends the data stream to the processing node according to a data flow path included in the data flow circulation table updated by the source node, where the processing node performs a service on the data stream. Processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.

In this embodiment, the data stream may be used to calculate data currently needed by the system in real time, such as data input by the user or data transmitted by the collection device.

Since the step 502 updates the data flow distribution table, the step 504 sends the data flow according to the updated data flow circulation table, so that the source node can send the data flow according to the updated topology to dynamically adjust the topology of the real-time computing system. .

In this implementation manner, after the source node recovers, the source node can directly obtain the data flow flow table updated in step 502 from the shared storage node, so that the source node can read from the shared storage node after the failure occurs. The data flow table is initialized and initialized, that is, the HA mechanism for dynamically adjusting the topology is completed.

In this implementation manner, the update result of the current update data flow flow table of each processing node can be obtained in time. When one or more processing nodes fail to update, the source node can trigger the management node to The processing node that failed the update sends a control flow so that the processing node that failed the update updates the data flow flow table again.

In this embodiment, the source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update at present; the source And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; Sending, by the source node, the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; Receiving, by the source node, a data stream, and including the data stream according to a data flow flow table updated by the source node Sending to the processing node according to the flow distribution path, the processing node performs service processing on the data flow, and sends the service processed data according to a data flow circulation path included in the data flow distribution table updated by the processing node. flow. The topology of the real-time computing system can be dynamically adjusted.

Referring to FIG. 6, FIG. 6 is a schematic diagram of a control flow transmission using a Storm application as an example, as shown in FIG.

601. The alarm node receives the control flow message, parses the control flow message to obtain the control flow, and updates the data flow flow table according to the control flow, and sends the control flow message to the police station node.

Among them, the Storm application is a real-time computing system. The above alarm node can be understood as the source node in the embodiment shown in Figures 1-5. The alarm node is a Spout function class node of the Storm real-time computing system. The Spout function class refers to a data source class in the Storm application, which is used for receiving and sending an external data stream, or constructing a data stream for transmission by itself.

The police station node is understood to be the management node in the embodiment shown in FIG. 1 to FIG. 5, and the police station node is a Bolt function class node in the Storm application, and depends on the police module body (the management module body in the embodiment shown in FIG. 1) The support is automatically integrated in the Storm application topology, and is connected to all nodes except the alarm node that sends data to the Storm application. Among them, the Bolt function class node is a data processing node in the Storm application, and each Bolt implements different business logics. Multiple Bolt combinations complete complex business logic processing.

602. The police station sends a control flow message to the police node A, the police node B, and the police node C.

The policing node can be understood as a processing node in the embodiment shown in FIG. 1 to FIG. 5, which is a Bolt function class node.

603. The police node A decomposes the control flow message, and updates the data flow flow table according to the control flow obtained by the decomposition.

604. The police node A feeds back the control flow processing message to the alarm node, and feeds back the feedback message of the processing process to the alarm node.

When the update of step 603 fails, the police node A may acquire the control flow message from the alarm node to implement the update again, in which case the control flow processing message fed back to the alarm node may be the updated result of the update again.

605. The police node B decomposes the control flow message, and updates the data flow flow table according to the control flow obtained by the decomposition.

606. The police node B feeds back the control flow processing message to the alarm node, and feeds back the feedback message of the processing process to the alarm node.

607. The police node C decomposes the control flow message, and updates the data flow flow table according to the control flow obtained by the decomposition.

608. The police node C feeds back the control flow processing message to the alarm node, and feeds back the feedback message of the processing process to the alarm node.

609. The alarm node summarizes the updated structure of the feedback, and sends or prints the control flow processing structure according to a predetermined logic.

Referring to FIG. 7, FIG. 7 is a schematic diagram of data flow transmission using a Storm application as an example, as shown in FIG. 7, including:

701. The alarm node receives the data stream, and sends the data stream to the processing node A according to the data flow distribution table.

702. The police node A performs service processing on the received data stream, and sends the service processed data stream to the police node B according to the data flow distribution table.

703. The police node B performs service processing on the received data stream, and follows the data flow distribution table. The service processed data stream is sent to the police node C.

704. The police node C performs service processing on the received data stream, and sends the service processed data stream to the next node or outputs the result according to the data flow distribution table.

Referring to FIG. 8, FIG. 8 is a schematic structural diagram of a data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 8, the method includes: a receiving unit 81, a first updating unit 82, and a first sending unit 83, where:

The receiving unit 81 is configured to receive a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update.

It should be noted that the data stream processing apparatus in this embodiment may be applied to a processing node of a real-time computing system, for example, a processing node as shown in FIG. 1.

In this embodiment, the topology that needs to be updated in the foregoing control flow may be a complete topology that the real-time computing system needs to update. For example, the foregoing control flow may describe the source node, the processing node, and the management node as shown in FIG. 1 . The topology between the node and the output node. Alternatively, the topology that needs to be updated in the foregoing control flow description may be a topology that needs to be adjusted currently, for example, the processing node A is connected to the processing node B in the original topology, and the processing node B is connected to the processing node C, that is, the data stream. The order of circulation is from processing node A to processing node B to processing node C, and currently needs to be adjusted to process node A to connect processing node C, and processing node C to connect to processing node B. Then, the above control flow may only describe the processing node A connection processing node C, and the processing node C reconnects the processing node B topology structure, that is, after the update, the data flow circulation order is the processing node A to the processing node B and then to the processing node. C.

The first update unit 82 is configured to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes the topology that is currently required to be updated. The matching data flow path.

In this embodiment, the processing node stores a data flow circulation table that matches the original topology structure. When the control flow is received, that is, when the current topology needs to be updated, the data flow distribution table may be updated according to the topology structure currently required to be updated. . And the updated data flow flow table includes more needs than current needs. The new topology matches the data flow path, so that the processing node sends the data stream according to the updated topology when sending the data stream.

In addition, the above-mentioned data flow path matching the topology that needs to be updated may be understood as a flow path or a circulation structure of the data flow in the topology that is currently required to be updated. For example, if the topology that needs to be updated is the processing node A connection processing node C, then the updated data flow distribution table includes the data stream transmitted from the processing node A to the processing node C, that is, when the device is applied to the processing node. Processing node A, then the updated data flow flow table may include a data flow path for transmitting the data stream to the processing node C.

The first sending unit 83 is configured to perform service processing on the data stream when the processing node receives the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table. The data stream.

Through the foregoing apparatus, the processing node can be sent according to the updated topology when transmitting the data stream, so that the topology of the real-time computing system can be dynamically adjusted. In addition, when the real-time computing system performs a huge amount of business operations, the above-mentioned steps dynamically adjust the topology of the real-time computing system without causing interference to the data stream being processed. In addition, when the real-time computing system is a distributed system, and each node runs on a different machine, dynamically adjusting the topology of the real-time computing system through the above steps can avoid the problem caused by the modification lag of a certain machine.

In this embodiment, the processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system. The topology structure currently needs to be updated; the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the current needs update a topologically matched data flow path; when the processing node receives the data stream, the processing node performs a service processing on the data stream, and follows a data flow path included in the updated data flow flow table Sending the data stream after the business processing. In this way, the processing logic of the real-time computing system can be updated by updating the data flow table in real time, that is, the topology of the real-time computing system can be dynamically adjusted.

Referring to FIG. 9, FIG. 9 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 9, the method includes: a receiving unit 91, a first updating unit 92, and a first sending list. Yuan 93, where:

The receiving unit 91 is configured to receive a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update.

The first update unit 92 is configured to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the topology that is currently required to be updated. The matching data flow path.

The first sending unit 93 is configured to perform service processing on the data stream when the processing node receives the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table. The data stream.

In this embodiment, the device may further include:

a second sending unit 94, configured to send the updated data flow distribution table to a shared storage node of the real-time computing system;

The recovery unit 95 is configured to: when the processing node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.

In this implementation manner, the processing node can obtain the data flow circulation table updated by the first update unit 92 directly from the shared storage node when the recovery node recovers, so that the processing node can be shared storage after the failure occurs. The data flow table is read and initialized in the node, that is, the HA mechanism for dynamically adjusting the topology is completed.

As shown in FIG. 10, the foregoing apparatus may further include:

a first feedback unit 96, configured to feed back the data stream to a source node of the real-time computing system Updating the result of the update of the table, so that when the update structure indicates that the update fails, the source node sends the control flow to the management node;

a second updating unit 97, configured to: when the update result indicates that the first update unit fails to update, receive the control flow sent by the management node, and update according to the current needs described by the control flow The topology updates the data flow flow table.

In this embodiment, the update result may be an update result of the first update unit 92. When the first update unit 92 is successfully updated, the source node may be notified of the update of the processing node by the update result. When the update fails, the source node is notified by the above update result that the processing node update fails, so that the source node sends the control flow to the management node again, and the management node sends the control flow to the processing node again to make the processing node again. Update. Certainly, when the source node sends the control flow to the management node again, it may also carry the identification information of the processing node that failed to update, so that the management node may only send the control flow to the processing node that failed the update, without successfully updating. The processing node sends the control flow again to save transmission resources.

As shown in FIG. 11, the foregoing apparatus may further include:

a second feedback unit 98, configured to feed back an update result of the data flow table update to an output node of the real-time computing system, and the update result fed back by the output node to all processing nodes of the real-time computing system Summarize and output the summary results.

In this embodiment, a plurality of optional implementation manners are added on the basis of the embodiment shown in FIG. 8, and the topology of the real-time computing system can be dynamically adjusted.

Referring to FIG. 12, FIG. 12 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 12, the method includes: a receiving unit 121 and a sending unit 122, where:

The receiving unit 121 is configured to receive a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing The topology that the system currently needs to update.

It should be noted that the data stream processing apparatus provided in this embodiment may be applied to a management node of a real-time computing system, for example, the management node shown in FIG. 1.

The sending unit 122 is configured to send the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data flow according to the topology that is currently updated according to the control flow. a flow table, wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the data is processed by the processing node The stream performs service processing, and transmits the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.

In this embodiment, the management node may send the control flow to all processing nodes in the real-time computing system, so that all processing nodes update the stored data flow table. For example, as shown in FIG. 13, the foregoing apparatus may further include:

a holding unit 123, configured to establish and maintain a connection with each of the processing nodes in the real-time computing system;

The sending unit 122 is configured to send the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.

This allows broadcast control to be sent to all processing nodes.

Of course, in this embodiment, the management node may only send the control flow to the partial processing node. For example, when the topology that needs to be updated only involves some processing nodes, the management node may only refer to these involved. The processing node sends the control flow so that these are involved in the processing node updating the data flow flow table, and without being involved, the data flow flow table may not be updated.

In this embodiment, the management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system. The topology that currently needs to be updated; the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing node updates according to the current needs described by the control flow Topology update data flow flow table, The updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs a service on the data flow. Processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the updated data flow distribution table. This allows dynamic adjustment of the topology of the real-time computing system.

Referring to FIG. 14, FIG. 14 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 14, the method includes: an obtaining unit 141, an updating unit 142, a first sending unit 143, and a second sending. Unit 144, wherein:

The obtaining unit 141 is configured to acquire a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update;

It should be noted that the data stream processing apparatus provided in this embodiment may be applied to a source node of a real-time computing system, for example, a source node as shown in FIG. 1.

The updating unit 142 is configured to update the data flow distribution table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a topology that matches the current need to be updated. Data flow path.

The first sending unit 143 is configured to send the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node used for performing service processing on the data stream, to And causing the processing node to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes data that matches the topology that needs to be updated currently. Flow path

Through the unit, the control node can send the control flow to the processing node, and the processing node updates the stored data flow table.

a second sending unit 144, configured to receive a data stream, and send the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where The processing node performs service processing on the data stream, and sends the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.

Since the update unit 142 updates the data flow distribution table, so that the second sending unit 144 sends the data flow according to the updated data flow distribution table, the source node can send the data flow according to the updated topology to implement dynamic adjustment of the real-time calculation. The topology of the system.

In this embodiment, as shown in FIG. 15, the foregoing apparatus may further include:

a third sending unit 145, configured to send the updated data flow circulation table to a shared storage node of the real-time computing system;

The recovery unit 146 is configured to: when the source node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.

In this implementation manner, after the source node recovers, the source node may directly obtain the data flow circulation table updated by the update unit 142 from the shared storage node, so that the source node may be in the shared storage node after the failure occurs. Reading and initializing the data flow table, the HA mechanism for dynamically adjusting the topology is completed.

In this embodiment, as shown in FIG. 16, the device may further include:

The receiving unit 147 is configured to receive, by the processing node that is fed back by the processing node, an update result of the data flow distribution table.

The fourth sending unit 148 is configured to: when the update result indicates that the update fails, send the control flow to the management node, so that the management node sends the control flow to the processing node, by the processing The node updates the data flow flow table according to the topology that is currently updated as described by the control flow.

In this implementation manner, the update result of the current update data flow flow table of each processing node can be obtained in time. When the update of one or more processing nodes fails, the source node may trigger the management node to send a control flow to the processing node that failed the update, so that the processing node that failed the update updates the data flow circulation table again.

In this embodiment, the source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a current requirement of the real-time computing system. An updated topology; the source node updates a data flow flow table according to the topology that is currently updated as described by the control flow, wherein the updated data flow flow table includes the topology that needs to be updated currently a data flow path of the structure matching; the source node transmitting the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a service for processing a data stream Processing the node, so that the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the topology that needs to be updated currently a data flow path of the structure matching; the source node receives the data stream, and sends the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, by the processing The node performs service processing on the data stream, and according to the data flow path included in the data flow circulation table updated by the processing node Sending the processed traffic data stream. The topology of the real-time computing system can be dynamically adjusted.

Referring to FIG. 17, FIG. 17 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. The apparatus is applied to a processing node of a real-time computing system, as shown in FIG. 17, and includes: a processor 171, a network interface. 172, a memory 174 for implementing connection communication between the processor 171, the network interface 172, and the memory 173, and a communication bus 174, the processor 171 executing the program stored in the memory 173 Used to implement the following methods:

Receiving, by the management node of the real-time computing system, a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update;

Updating the data flow circulation table according to the topology structure currently required to be updated according to the control flow, wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently;

When the processing node receives the data stream, performing processing on the data stream, and transmitting the data stream after the service processing according to the data stream circulation path included in the updated data stream circulation table.

In this embodiment, the processor 171 can also execute the following procedure:

Transmitting the updated data flow circulation table to a shared storage node of the real-time computing system;

When the processing node performs failure recovery, the updated data flow distribution table is obtained from the shared storage node, and the data flow is transmitted according to the updated data flow distribution table.

In this embodiment, the processor 171 can also execute the following procedure:

And feeding back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the control flow to the management node;

In this embodiment, the processor 171 can also execute the following procedure:

And returning, to the output node of the real-time computing system, an update result of the data flow table update, and the output node summarizes the update result fed back by all processing nodes of the real-time computing system, and outputs a summary result.

Referring to FIG. 18, FIG. 18 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. The apparatus is applied to a management node of a real-time computing system, as shown in FIG. 18, and includes: a processor 181, a network interface. 182. A memory 184 for communicating communications between the processor 181, the network interface 182 and the memory 183, and a communication bus 184, the processor 181 executing the program stored in the memory 183 Used to implement the following methods:

Receiving, by the source node of the real-time computing system, a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update;

Transmitting the control flow to a processing node for performing traffic processing on the data stream to cause the The processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently. When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service processed data according to the data flow distribution path included in the updated data flow circulation table. flow.

In this embodiment, the processor 181 can also execute the following procedure:

Establishing and maintaining a connection with each of the processing nodes in the real-time computing system;

The program executed by the processor 181 to send the control flow to the processing node for performing service processing on the data stream may include:

Referring to FIG. 19, FIG. 19 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. The apparatus is applied to a source node of a real-time computing system, as shown in FIG. 19, and includes: a processor 191, a network interface. 192, a memory 194 for implementing connection communication between the processor 191, the network interface 192, and the memory 193, and a communication bus 194, the processor 191 executing the program stored in the memory 193 Used to implement the following methods:

Obtaining a control flow for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update;

Updating the data stream according to the topology that is currently updated as described by the control flow a table, wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently;

Transmitting the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing node follows the The topology update data flow circulation table currently required to be updated by the control flow, wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently;

Receiving a data stream, and transmitting the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where the processing node performs service processing on the data flow, and according to the The data flow distribution path included in the data flow distribution table updated by the processing node transmits the data flow after the business processing.

In this embodiment, the program executed by the processor 191 may further include:

When the source node performs failure recovery, the updated data flow distribution table is obtained from the shared storage node, and the data flow is transmitted according to the updated data flow distribution table.

Receiving, by the processing node fed back by the processing node, an update result of the data flow distribution table;

And when the update result indicates that the update fails, sending the control flow to the management node, so that the management node sends the control flow to the processing node, which is described by the processing node according to the control flow. The topology that currently needs to be updated updates the data flow flow table.

In this embodiment, the source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update at present; the source And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; Sending, by the source node, the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing The node updates the data flow flow table according to the topology that is currently updated according to the control flow, where the updated data flow flow table includes a data flow path that matches the topology that needs to be updated currently. The source node receives the data stream, and sends the data stream to the processing node according to the data flow circulation path included in the data flow circulation table updated by the source node, and the data stream is processed by the processing node Performing business processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node. The topology of the real-time computing system can be dynamically adjusted.

One of ordinary skill in the art can understand that all or part of the process of implementing the foregoing embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

The above is only the preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, and thus equivalent changes made in the claims of the present invention are still within the scope of the present invention.

Claims

A data stream processing method, comprising:

a processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update structure;

The processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Circulation path

When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.
The method of claim 1 wherein the method further comprises:

Transmitting, by the processing node, the updated data flow circulation table to a shared storage node of the real-time computing system;

When the processing node performs failure recovery, the processing node acquires the updated data flow distribution table from the shared storage node, and transmits the data flow according to the updated data flow distribution table.
The method of claim 1 or 2, wherein the method further comprises:

And the processing node feeds back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the control to the management node flow;

When the update result indicates that the update fails, the processing node receives the control flow sent by the management node, and updates the data flow circulation table according to the topology that is currently updated according to the control flow.
The method of claim 1 or 2, wherein the method further comprises:

The processing node feeds back an update result of the data flow table update to an output node of the real-time computing system, and the output node feeds back all processing nodes of the real-time computing system The updated results are summarized and the summary results are output.
A data stream processing method, comprising:

a management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update structure;

Transmitting, by the management node, the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow description The updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs the data flow on the data flow. The service processes, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.
The method of claim 5, wherein the method further comprises:

Establishing and maintaining a connection between the management node and each of the processing nodes in the real-time computing system;

The management node sends the control flow to a processing node for performing service processing on the data stream, including:

The management node transmits the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.
A data stream processing method, comprising:

A source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update;

And the source node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes a data flow that matches the topology that needs to be updated currently. Circulation path

The source node sends the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, And causing the processing node to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes data that matches the topology that needs to be updated currently. Flow path

Receiving, by the source node, the data stream, and transmitting the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where the processing node performs a service on the data flow Processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.
The method of claim 7 wherein the method further comprises:

Sending, by the source node, the updated data flow circulation table to a shared storage node of the real-time computing system;

When the source node performs fault recovery, the source node acquires the updated data flow flow table from the shared storage node, and performs data flow transmission according to the updated data flow flow table.
The method of claim 7 or 8, wherein the method further comprises:

Receiving, by the source node, the processing node fed back by the processing node to update an update result of the data flow distribution table;

When the update result indicates that the update fails, the source node sends the control flow to the management node, so that the management node sends the control flow to the processing node, and the processing node follows the The topology that is currently updated as described by the control flow updates the data flow flow table.
A data stream processing device, the device being applied to a processing node of a real-time computing system, comprising: a receiving unit, a first updating unit, and a first sending unit, wherein:

The receiving unit is configured to receive a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe that the real-time computing system needs to be updated currently Topology;

The first update unit is configured to update a data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the topology that needs to be updated currently Structure matching data flow path;

The first sending unit is configured to: when the processing node receives the data stream, perform service processing on the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table. After the data stream.
The device of claim 10, wherein the device further comprises:

a second sending unit, configured to send the updated data flow distribution table to a shared storage node of the real-time computing system;

And a recovery unit, configured to: when the processing node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.
The device according to claim 10 or 11, wherein the device further comprises:

a first feedback unit, configured to feed back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the update to the management node The control flow;

a second updating unit, configured to: when the update result indicates that the first update unit fails to update, receive the control flow sent by the management node, and update according to the current flow description described by the control flow The topology updates the data flow flow table.
The device according to claim 10 or 11, wherein the device further comprises:

a second feedback unit, configured to feed back an update result of the data flow table update to an output node of the real-time computing system, where the update result is fed back by the output node to all processing nodes of the real-time computing system Summarize and output the summary results.
A data stream processing device, comprising: a receiving unit and a sending unit, wherein:

The receiving unit is configured to receive a control flow sent by a source node of a real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update currently. ;

The sending unit is configured to send the control flow to a service for processing a data stream. Processing the node, so that the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the topology that needs to be updated currently a data flow path of the structure matching; when the processing node receives the data stream, the processing node performs service processing on the data stream, and sends according to the data flow path included in the updated data flow circulation table The data stream after the business processing.
The device of claim 14 wherein said device further comprises:

a holding unit, configured to establish and maintain a connection with each of the processing nodes in the real-time computing system;

The sending unit is configured to send the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.
A data stream processing device, the device being applied to a source node of a real-time computing system, comprising: an obtaining unit, an updating unit, a first sending unit, and a second sending unit, wherein:

The acquiring unit is configured to acquire a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update at present;

The updating unit is configured to update a data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes a topology matching the current need to be updated. Data flow path;

The first sending unit is configured to send the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, And causing the processing node to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes a topology that matches the current need to be updated. Data flow path;

The second sending unit is configured to receive a data stream, and send the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where the processing node is The data stream performs service processing, and transmits the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.
The device of claim 16 wherein said device further comprises:

a third sending unit, configured to send the updated data flow distribution table to a shared storage node of the real-time computing system;

And a recovery unit, configured to: when the source node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.
The device according to claim 16 or 17, wherein the device further comprises:

a receiving unit, configured to receive, by the processing node fed back by the processing node, an update result of updating the data flow distribution table;

a fourth sending unit, configured to: when the update result indicates that the update fails, send the control flow to the management node, so that the management node sends the control flow to the processing node, by the processing node Updating the data flow flow table according to the topology currently required to be updated as described by the control flow.