WO2017032212A1 - Procédé et appareil de traitement de flux de données - Google Patents

Procédé et appareil de traitement de flux de données Download PDF

Info

Publication number
WO2017032212A1
WO2017032212A1 PCT/CN2016/093588 CN2016093588W WO2017032212A1 WO 2017032212 A1 WO2017032212 A1 WO 2017032212A1 CN 2016093588 W CN2016093588 W CN 2016093588W WO 2017032212 A1 WO2017032212 A1 WO 2017032212A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
data flow
processing
updated
real
Prior art date
Application number
PCT/CN2016/093588
Other languages
English (en)
Chinese (zh)
Inventor
李旭良
李嘉
刘杰
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2017032212A1 publication Critical patent/WO2017032212A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/02—Topology update or discovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Z—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS, NOT OTHERWISE PROVIDED FOR
    • G16Z99/00—Subject matter not provided for in other main groups of this subclass
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass

Definitions

  • the present invention relates to the field of communications technologies, and in particular, to a data stream processing method and apparatus.
  • real-time data is strong in real-time, and the amount of data is large, and there is no end.
  • real-time data is mainly calculated by flow computing in real-time computing system. Calculation, for example: Storm application.
  • the code is implemented by relying on the JAVA application programming interface (API), and the package submission task is implemented, the business logic of the real-time computing system that is already running is immutable, that is, real-time.
  • the topology structure of the system is immutable at runtime in the computing system.
  • the embodiment of the invention provides a data stream processing method and device, which can dynamically adjust the topology structure of the real-time computing system.
  • an embodiment of the present invention provides a data stream processing method, including:
  • a processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update structure;
  • the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Circulation path
  • the processing node When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service according to the data flow circulation path included in the updated data flow circulation table.
  • the processed data stream When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service according to the data flow circulation path included in the updated data flow circulation table.
  • the processed data stream The processed data stream.
  • the method further includes:
  • the processing node When the processing node performs failure recovery, the processing node acquires the updated data flow distribution table from the shared storage node, and transmits the data flow according to the updated data flow distribution table.
  • the method further includes:
  • processing node feeds back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the control to the management node flow;
  • the processing node receives the control flow sent by the management node, and updates the data flow circulation table according to the topology that is currently updated according to the control flow.
  • the method further includes:
  • the processing node feeds back an update result of the data flow table update to an output node of the real-time computing system, and the output node summarizes the update result fed back by all processing nodes of the real-time computing system, and Output summary results.
  • an embodiment of the present invention provides a data stream processing method, including:
  • a management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update structure;
  • the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs the data flow on the data flow.
  • the service processes, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the method further includes:
  • the management node sends the control flow to a processing node for performing service processing on the data stream, including:
  • the management node transmits the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.
  • an embodiment of the present invention provides a data stream processing method, including:
  • a source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update;
  • the source node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes a data flow that matches the topology that needs to be updated currently.
  • the control flow Sending, by the source node, the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently;
  • the method further includes:
  • the source node When the source node performs fault recovery, the source node acquires the updated data flow flow table from the shared storage node, and performs data flow transmission according to the updated data flow flow table.
  • the method further includes:
  • the source node sends the control flow to the management node, so that the management node sends the control flow to the processing node, and the processing node follows the The topology that is currently updated as described by the control flow updates the data flow flow table.
  • an embodiment of the present invention provides a data stream processing apparatus, where the apparatus is applied to a processing node of a real-time computing system, including: a receiving unit, a first updating unit, and a first sending unit, where:
  • the receiving unit is configured to receive a control flow sent by a management node of a real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update currently. ;
  • the first update unit is configured to update a data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the topology that needs to be updated currently Structure matching data flow path;
  • the first sending unit is configured to: when the processing node receives the data stream, perform service processing on the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table. After the data stream.
  • the device further includes:
  • a second sending unit configured to send the updated data flow distribution table to a shared storage node of the real-time computing system
  • a recovery unit configured to: when the processing node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.
  • the device further includes:
  • a first feedback unit configured to feed back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the update to the management node The control flow;
  • a second updating unit configured to: when the update result indicates that the first update unit fails to update, Receiving the control flow sent by the management node, and updating the data flow circulation table according to the topology that is currently updated according to the control flow.
  • the device further includes:
  • a second feedback unit configured to feed back an update result of the data flow table update to an output node of the real-time computing system, where the update result is fed back by the output node to all processing nodes of the real-time computing system Summarize and output the summary results.
  • an embodiment of the present invention provides a data stream processing apparatus, including: a receiving unit and a sending unit, where:
  • the receiving unit is configured to receive a control flow sent by a source node of a real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update currently. ;
  • the sending unit is configured to send the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data according to the topology that is currently updated according to the control flow.
  • a flow distribution table wherein the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node The data stream performs service processing, and transmits the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the device further includes:
  • a holding unit configured to establish and maintain a connection with each of the processing nodes in the real-time computing system
  • the sending unit is configured to send the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.
  • an embodiment of the present invention provides a data stream processing apparatus, where the apparatus is applied to a source node of a real-time computing system, including: an acquiring unit, an updating unit, a first sending unit, and a second sending unit, where:
  • the acquiring unit is configured to acquire a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update at present;
  • the updating unit is configured to update a data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the current requirement The data flow path that matches the topology to be updated;
  • the first sending unit is configured to send the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, And causing the processing node to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes a topology that matches the current need to be updated.
  • Data flow path ;
  • the second sending unit is configured to receive a data stream, and send the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where the processing node is The data stream performs service processing, and transmits the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.
  • the device further includes:
  • a third sending unit configured to send the updated data flow distribution table to a shared storage node of the real-time computing system
  • a recovery unit configured to: when the source node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.
  • the device further includes:
  • a receiving unit configured to receive, by the processing node fed back by the processing node, an update result of updating the data flow distribution table
  • a fourth sending unit configured to: when the update result indicates that the update fails, send the control flow to the management node, so that the management node sends the control flow to the processing node, by the processing node Updating the data flow flow table according to the topology currently required to be updated as described by the control flow.
  • the processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system.
  • the topology structure currently needs to be updated; the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the current needs update Topology structure matching data flow path; when the processing node receives the data stream, the processing node pairs the number Performing service processing according to the flow, and transmitting the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the processing logic of the real-time computing system can be updated by updating the data flow table in real time, that is, the topology of the real-time computing system can be dynamically adjusted.
  • FIG. 1 is a structural diagram of a real-time computing system to which a data stream processing method according to an embodiment of the present invention is applicable;
  • FIG. 2 is a schematic flowchart of a data stream processing method according to an embodiment of the present invention.
  • FIG. 3 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention.
  • FIG. 4 is a schematic flowchart diagram of another data stream processing method according to an embodiment of the present invention.
  • FIG. 5 is a schematic flowchart diagram of another data stream processing method according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic diagram of control flow transmission using a Storm application as an example according to an embodiment of the present invention.
  • FIG. 7 is a schematic diagram of data flow transmission using a Storm application as an example according to an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 10 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 11 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 13 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 14 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 15 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 16 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 17 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 18 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 19 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • FIG. 1 is a structural diagram of a real-time computing system applicable to a data stream processing method according to an embodiment of the present invention.
  • the method includes: a supervisory body 11, a source node 12, and one or more
  • the processing node 13 and the management module 14 are respectively connected to the source node 12 and the respective processing nodes 13, and the source node 12 is connected to at least one of the processing nodes 13, and the processing nodes 13 are sequentially connected.
  • the management module 14 includes a management node 141 and an output node 142, wherein the management node 141 is connected to the source node and the respective processing nodes 13, and the output node 142 is connected to the respective processing nodes 13.
  • the connection relationship between the nodes in the real-time computing system is expressed as a topology of the real-time computing system or as a topology model.
  • connection between the nodes described in this embodiment may be understood as a connection on a logical connection, and a data flow or a control flow may be transmitted between the connected nodes.
  • the data stream flows from the source node 12, and then the source node 12 transmits the data stream to the corresponding processing node 13 in accordance with the stored data flow flow table, and the processing node 13 performs the service on the received data stream.
  • the service processing may include real-time calculation of the data stream or understanding of the stream calculation
  • the processing node 13 transmitting the service-processed data stream to another processing node 13 according to the stored data flow circulation table, by the processing
  • the node 13 transmits the service-processed data stream to another node or outputs the result according to the stored data flow flow table.
  • the data flow distribution table includes a data flow circulation path that matches the topology structure of the real-time computing system, that is, in the embodiment, the data flow distribution table controls the transmission path or the transmission structure of the data flow, such that Both the source node 12 and the processing node 13 can send the data stream that needs to be sent to the corresponding node through the respective stored data flow distribution table.
  • the topology of the real-time computing system includes that the source node 12 is connected to the processing node A, the processing node A is reconnected to the processing node B, the processing node B is reconnected to the processing node C, and the processing node C is the ending node.
  • the data flow flow table stored at the source node 12 can include the data flow sent by the source node 12 to the processing section.
  • the mapping of the point A, the data flow distribution table stored by the processing node A may include the mapping of the data stream sent by the processing node A to the processing node B, and the data flow distribution table stored by the processing node B may include the data stream sent by the processing node B.
  • the data flow distribution table stored by the processing node C may include a mapping of the processing node C outputting the business processing result.
  • the data flow distribution table stored by each node may include all mappings or only include mappings related to itself.
  • control flow can be transmitted from the source node 12 to the management node 141, wherein the control flow is used to describe the topology that the real-time computing system currently needs to update.
  • the management node 141 then transmits the control flow to each processing node 13, so that the processing nodes 13 can update the stored data flow flow table according to the control flow, thereby realizing the dynamic adjustment of the topology of the real-time computing system.
  • the real-time computing system may be a distributed system, that is, the foregoing nodes may be distributed and run in different machines, and of course, some of the nodes may be allowed to run in the same machine, for example, a management node. 121 and output node 122 can run in the same machine or in different machines. In addition, in the embodiment of the present invention, these machines are not limited, for example, these machines may be computers or servers.
  • FIG. 2 is a schematic flowchart of a data stream processing method according to an embodiment of the present invention. As shown in FIG. 2, the method includes the following steps:
  • the processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe that the real-time computing system needs to be updated currently. Topology.
  • the topology that needs to be updated in the foregoing control flow may be a complete topology that the real-time computing system needs to update.
  • the foregoing control flow may describe the source node, the processing node, and the management node as shown in FIG. 1 .
  • the topology that needs to be updated in the foregoing control flow description may be a topology that needs to be adjusted currently, for example, the processing node A is connected to the processing node B in the original topology, and the processing node B is connected to the processing node C, that is, the data stream.
  • the order of circulation is from processing node A to processing node B to processing node C, and currently needs to be adjusted to process node A to connect processing node C, and processing node C to connect to processing node B. Then, the above control flow may only describe the processing node A connection processing node C, processing section The point C reconnects to the topology of the processing node B, that is, after the update, the flow order of the data stream is from the processing node A to the processing node B to the processing node C.
  • control flow information that is, the above control flow can be understood as one piece of information.
  • the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Distribution path.
  • the processing node stores a data flow circulation table that matches the original topology structure.
  • the data flow distribution table may be updated according to the topology structure currently required to be updated.
  • the updated data flow circulation table includes a data flow distribution path that matches the topology that needs to be updated, so that the processing node sends the data flow according to the updated topology when transmitting the data flow.
  • the above-mentioned data flow path matching the topology that needs to be updated may be understood as a flow path or a circulation structure of the data flow in the topology that is currently required to be updated.
  • the topology that needs to be updated is the processing node A connection processing node C
  • the updated data flow distribution table includes the data stream transmitted from the processing node A to the processing node C, that is, when the processing node performing step 202 is Processing node A
  • the updated data flow flow table may include a data flow path for transmitting the data stream to the processing node C.
  • the processing node When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service processed data according to the data flow distribution path included in the updated data flow distribution table. flow.
  • the processing node can be sent according to the updated topology when transmitting the data stream, so that the topology of the real-time computing system can be dynamically adjusted.
  • the above-mentioned steps dynamically adjust the topology of the real-time computing system without causing interference to the data stream being processed.
  • the real-time computing system is a distributed system, and each node runs on a different machine, dynamically adjusting the topology of the real-time computing system through the above steps can avoid the problem caused by the modification lag of a certain machine.
  • the processing node may be any processing node in the real-time computing system.
  • the processing node of the real-time computing system receives a control flow sent by the management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control
  • the flow is used to describe a topology that the real-time computing system currently needs to update;
  • the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, where the updated data flow is circulated.
  • the table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs service processing on the data flow, and follows the updated data flow.
  • the data flow distribution path included in the flow table transmits the data stream after the business process. In this way, the processing logic of the real-time computing system can be updated by updating the data flow table in real time, that is, the topology of the real-time computing system can be dynamically adjusted.
  • FIG. 3 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention. As shown in FIG. 3, the method includes the following steps:
  • a processing node of a real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe that the real-time computing system needs to be updated currently. Topology.
  • the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Distribution path.
  • the processing node when the processing node updates the data flow distribution table, the processing node may suspend the data flow transmission, but may perform service processing on the data flow. Or the processing node may suspend the service processing and data stream transmission of the data stream when updating the data flow distribution table. When the data flow flow table is updated, the suspended data stream transmission and/or the business processing of the data stream are resumed. In this way, when the topology of the real-time computing system is dynamically adjusted, the error handling of the data stream and the blocking effect of the system may occur.
  • the processing node When the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service processed data according to the data flow distribution path included in the updated data flow distribution table. flow.
  • the above method may further comprise the following steps:
  • the processing node sends the updated data flow circulation table to a shared storage node of the real-time computing system.
  • the processing node acquires the updated data flow flow table from the shared storage node, and performs the number according to the updated data flow circulation table. According to the flow of the transmission.
  • the processing node can obtain the data flow circulation table updated in step 302 directly from the shared storage node when the recovery node recovers, so that the processing node can read from the shared storage node after the failure occurs.
  • the data flow table is initialized and initialized, that is, the high availability (HA) mechanism of dynamically adjusting the topology is completed.
  • the foregoing method may further include the following steps:
  • processing node feeds back, to the source node of the real-time computing system, an update result of the data flow table update, so that when the update structure indicates that the update fails, the source node sends the control to the management node flow;
  • the processing node receives the control flow sent by the management node, and updates the data flow circulation table according to the topology that is currently updated according to the control flow.
  • the update result may be the update result of step 302.
  • the source node may be notified of the update of the processing node by the update result.
  • the update fails, the source node is notified by the above update result that the processing node update fails, so that the source node sends the control flow to the management node again, and the management node sends the control flow to the processing node again to make the processing node again.
  • Update Certainly, when the source node sends the control flow to the management node again, it may also carry the identification information of the processing node that failed to update, so that the management node may only send the control flow to the processing node that failed the update, without successfully updating. The processing node sends the control flow again to save transmission resources.
  • This embodiment can implement the correct feedback of the update result, and if the update fails, the update task can be started again.
  • the foregoing method may further include the following steps:
  • Processing node feeds back an update result of the data flow table update to an output node of the real-time computing system, and the output node summarizes the update result fed back by all processing nodes of the real-time computing system, and outputs the summary result.
  • the output node can obtain the update result fed back by each processing node, so that the output node can summarize it, so that the summary result can be output, for example, sending the summary result to the presentation device, or printing the summary result, etc. This allows the user to know the state of the topology adjustment of the real-time computing system.
  • a plurality of optional implementation manners are added on the basis of the embodiment shown in FIG. 2, and the topology of the real-time computing system can be dynamically adjusted.
  • FIG. 4 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention. As shown in FIG. 4, the method includes the following steps:
  • the management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe that the real-time computing system needs to be updated currently. Topology.
  • control flow may include structural information of a topology that needs to be updated, and may also include a control flow decomposition structure identifier of the management node or the management module internal processing logic, that is, the control flow may be decomposed by the structure identifier and The structure information identifies the topology that needs to be updated currently.
  • the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow.
  • the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs the data flow on the data flow.
  • the service processes, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the management node may send the control flow to all processing nodes in the real-time computing system, so that all processing nodes update the stored data flow table.
  • the above method may further include the following steps:
  • the step of the foregoing management node sending the control flow to the processing node for performing service processing on the data stream may include:
  • the management node transmits the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.
  • the management node may only send the control flow to the partial processing node.
  • the management section The point can only send control flow to these involved processing nodes, so that these are involved in the processing node to update the data flow flow table, and without being involved, the data flow flow table may not be updated.
  • the management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system.
  • the topology update data flow circulation table wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing is performed by the processing
  • the node performs service processing on the data stream, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table. This allows dynamic adjustment of the topology of the real-time computing system.
  • FIG. 5 is a schematic flowchart of another data stream processing method according to an embodiment of the present invention. As shown in FIG. 5, the method includes the following steps:
  • a source node of a real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update.
  • control flow may be a control flow in which the source node receives user input.
  • the control flow here can refer to the control flow described in the embodiment shown in FIG. 1-4, and will not be repeatedly described herein.
  • the source node updates the data flow distribution table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow that matches the topology that needs to be updated currently. Distribution path.
  • the source node After receiving the above control flow, the source node can decompose the control flow, and update the data flow circulation table according to the topology structure that needs to be updated according to the decomposition. In addition, the source node stores or caches the updated data flow table. Of course, the above control flow can also be stored or cached.
  • the source node sends the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node used for performing service processing on the data flow, so that the processing is performed.
  • the node updates the data flow flow table according to the topology that is currently updated as described by the control flow, wherein the updated data flow flow table includes more The new topology matches the data flow path.
  • control node can send the control flow to the processing node, and the processing node updates the stored data flow table.
  • the source node receives the data stream, and sends the data stream to the processing node according to a data flow path included in the data flow circulation table updated by the source node, where the processing node performs a service on the data stream. Processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.
  • the data stream may be used to calculate data currently needed by the system in real time, such as data input by the user or data transmitted by the collection device.
  • the step 504 sends the data flow according to the updated data flow circulation table, so that the source node can send the data flow according to the updated topology to dynamically adjust the topology of the real-time computing system. .
  • the foregoing method may further include the following steps:
  • the source node When the source node performs fault recovery, the source node acquires the updated data flow flow table from the shared storage node, and performs data flow transmission according to the updated data flow flow table.
  • the source node can directly obtain the data flow flow table updated in step 502 from the shared storage node, so that the source node can read from the shared storage node after the failure occurs.
  • the data flow table is initialized and initialized, that is, the HA mechanism for dynamically adjusting the topology is completed.
  • the foregoing method may further include the following steps:
  • the source node sends the control flow to the management node, so that the management node sends the control flow to the processing node, and the processing node follows the The topology that is currently updated as described by the control flow updates the data flow flow table.
  • the update result of the current update data flow flow table of each processing node can be obtained in time.
  • the source node can trigger the management node to The processing node that failed the update sends a control flow so that the processing node that failed the update updates the data flow flow table again.
  • the source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update at present; the source And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; Sending, by the source node, the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; Receiving, by the source node, a data stream, and including the data stream according to a data flow flow table updated
  • FIG. 6 is a schematic diagram of a control flow transmission using a Storm application as an example, as shown in FIG.
  • the alarm node receives the control flow message, parses the control flow message to obtain the control flow, and updates the data flow flow table according to the control flow, and sends the control flow message to the police station node.
  • the Storm application is a real-time computing system.
  • the above alarm node can be understood as the source node in the embodiment shown in Figures 1-5.
  • the alarm node is a Spout function class node of the Storm real-time computing system.
  • the Spout function class refers to a data source class in the Storm application, which is used for receiving and sending an external data stream, or constructing a data stream for transmission by itself.
  • the police station node is understood to be the management node in the embodiment shown in FIG. 1 to FIG. 5, and the police station node is a Bolt function class node in the Storm application, and depends on the police module body (the management module body in the embodiment shown in FIG. 1)
  • the support is automatically integrated in the Storm application topology, and is connected to all nodes except the alarm node that sends data to the Storm application.
  • the Bolt function class node is a data processing node in the Storm application, and each Bolt implements different business logics. Multiple Bolt combinations complete complex business logic processing.
  • the police station sends a control flow message to the police node A, the police node B, and the police node C.
  • the policing node can be understood as a processing node in the embodiment shown in FIG. 1 to FIG. 5, which is a Bolt function class node.
  • the police node A decomposes the control flow message, and updates the data flow flow table according to the control flow obtained by the decomposition.
  • the police node A feeds back the control flow processing message to the alarm node, and feeds back the feedback message of the processing process to the alarm node.
  • the police node A may acquire the control flow message from the alarm node to implement the update again, in which case the control flow processing message fed back to the alarm node may be the updated result of the update again.
  • the police node B decomposes the control flow message, and updates the data flow flow table according to the control flow obtained by the decomposition.
  • the police node B feeds back the control flow processing message to the alarm node, and feeds back the feedback message of the processing process to the alarm node.
  • the police node C decomposes the control flow message, and updates the data flow flow table according to the control flow obtained by the decomposition.
  • the police node C feeds back the control flow processing message to the alarm node, and feeds back the feedback message of the processing process to the alarm node.
  • the alarm node summarizes the updated structure of the feedback, and sends or prints the control flow processing structure according to a predetermined logic.
  • FIG. 7 is a schematic diagram of data flow transmission using a Storm application as an example, as shown in FIG. 7, including:
  • the alarm node receives the data stream, and sends the data stream to the processing node A according to the data flow distribution table.
  • the police node A performs service processing on the received data stream, and sends the service processed data stream to the police node B according to the data flow distribution table.
  • the police node B performs service processing on the received data stream, and follows the data flow distribution table.
  • the service processed data stream is sent to the police node C.
  • the police node C performs service processing on the received data stream, and sends the service processed data stream to the next node or outputs the result according to the data flow distribution table.
  • FIG. 8 is a schematic structural diagram of a data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 8, the method includes: a receiving unit 81, a first updating unit 82, and a first sending unit 83, where:
  • the receiving unit 81 is configured to receive a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update.
  • the data stream processing apparatus in this embodiment may be applied to a processing node of a real-time computing system, for example, a processing node as shown in FIG. 1.
  • the topology that needs to be updated in the foregoing control flow may be a complete topology that the real-time computing system needs to update.
  • the foregoing control flow may describe the source node, the processing node, and the management node as shown in FIG. 1 .
  • the topology that needs to be updated in the foregoing control flow description may be a topology that needs to be adjusted currently, for example, the processing node A is connected to the processing node B in the original topology, and the processing node B is connected to the processing node C, that is, the data stream.
  • the order of circulation is from processing node A to processing node B to processing node C, and currently needs to be adjusted to process node A to connect processing node C, and processing node C to connect to processing node B. Then, the above control flow may only describe the processing node A connection processing node C, and the processing node C reconnects the processing node B topology structure, that is, after the update, the data flow circulation order is the processing node A to the processing node B and then to the processing node. C.
  • control flow information that is, the above control flow can be understood as one piece of information.
  • the first update unit 82 is configured to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes the topology that is currently required to be updated.
  • the matching data flow path is configured to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes the topology that is currently required to be updated.
  • the processing node stores a data flow circulation table that matches the original topology structure.
  • the data flow distribution table may be updated according to the topology structure currently required to be updated.
  • the updated data flow flow table includes more needs than current needs.
  • the new topology matches the data flow path, so that the processing node sends the data stream according to the updated topology when sending the data stream.
  • the above-mentioned data flow path matching the topology that needs to be updated may be understood as a flow path or a circulation structure of the data flow in the topology that is currently required to be updated.
  • the topology that needs to be updated is the processing node A connection processing node C
  • the updated data flow distribution table includes the data stream transmitted from the processing node A to the processing node C, that is, when the device is applied to the processing node.
  • Processing node A then the updated data flow flow table may include a data flow path for transmitting the data stream to the processing node C.
  • the first sending unit 83 is configured to perform service processing on the data stream when the processing node receives the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the data stream is configured to perform service processing on the data stream when the processing node receives the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the processing node can be sent according to the updated topology when transmitting the data stream, so that the topology of the real-time computing system can be dynamically adjusted.
  • the above-mentioned steps dynamically adjust the topology of the real-time computing system without causing interference to the data stream being processed.
  • the real-time computing system is a distributed system, and each node runs on a different machine, dynamically adjusting the topology of the real-time computing system through the above steps can avoid the problem caused by the modification lag of a certain machine.
  • the processing node may be any processing node in the real-time computing system.
  • the processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system.
  • the topology structure currently needs to be updated; the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the current needs update a topologically matched data flow path; when the processing node receives the data stream, the processing node performs a service processing on the data stream, and follows a data flow path included in the updated data flow flow table Sending the data stream after the business processing.
  • the processing logic of the real-time computing system can be updated by updating the data flow table in real time, that is, the topology of the real-time computing system can be dynamically adjusted.
  • FIG. 9 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 9, the method includes: a receiving unit 91, a first updating unit 92, and a first sending list. Yuan 93, where:
  • the receiving unit 91 is configured to receive a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update.
  • the data stream processing apparatus in this embodiment may be applied to a processing node of a real-time computing system, for example, a processing node as shown in FIG. 1.
  • the first update unit 92 is configured to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the topology that is currently required to be updated.
  • the matching data flow path is configured to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes the topology that is currently required to be updated.
  • the processing node when the processing node updates the data flow distribution table, the processing node may suspend the data flow transmission, but may perform service processing on the data flow. Or the processing node may suspend the service processing and data stream transmission of the data stream when updating the data flow distribution table. When the data flow flow table is updated, the suspended data stream transmission and/or the business processing of the data stream are resumed. In this way, when the topology of the real-time computing system is dynamically adjusted, the error handling of the data stream and the blocking effect of the system may occur.
  • the first sending unit 93 is configured to perform service processing on the data stream when the processing node receives the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the data stream is configured to perform service processing on the data stream when the processing node receives the data stream, and send the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the device may further include:
  • a second sending unit 94 configured to send the updated data flow distribution table to a shared storage node of the real-time computing system
  • the recovery unit 95 is configured to: when the processing node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.
  • the processing node can obtain the data flow circulation table updated by the first update unit 92 directly from the shared storage node when the recovery node recovers, so that the processing node can be shared storage after the failure occurs.
  • the data flow table is read and initialized in the node, that is, the HA mechanism for dynamically adjusting the topology is completed.
  • the foregoing apparatus may further include:
  • a first feedback unit 96 configured to feed back the data stream to a source node of the real-time computing system Updating the result of the update of the table, so that when the update structure indicates that the update fails, the source node sends the control flow to the management node;
  • a second updating unit 97 configured to: when the update result indicates that the first update unit fails to update, receive the control flow sent by the management node, and update according to the current needs described by the control flow
  • the topology updates the data flow flow table.
  • the update result may be an update result of the first update unit 92.
  • the source node may be notified of the update of the processing node by the update result.
  • the update fails, the source node is notified by the above update result that the processing node update fails, so that the source node sends the control flow to the management node again, and the management node sends the control flow to the processing node again to make the processing node again.
  • Update Certainly, when the source node sends the control flow to the management node again, it may also carry the identification information of the processing node that failed to update, so that the management node may only send the control flow to the processing node that failed the update, without successfully updating. The processing node sends the control flow again to save transmission resources.
  • This embodiment can implement the correct feedback of the update result, and if the update fails, the update task can be started again.
  • the foregoing apparatus may further include:
  • a second feedback unit 98 configured to feed back an update result of the data flow table update to an output node of the real-time computing system, and the update result fed back by the output node to all processing nodes of the real-time computing system Summarize and output the summary results.
  • the output node can obtain the update result fed back by each processing node, so that the output node can summarize it, so that the summary result can be output, for example, sending the summary result to the presentation device, or printing the summary result, etc. This allows the user to know the state of the topology adjustment of the real-time computing system.
  • a plurality of optional implementation manners are added on the basis of the embodiment shown in FIG. 8, and the topology of the real-time computing system can be dynamically adjusted.
  • FIG. 12 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 12, the method includes: a receiving unit 121 and a sending unit 122, where:
  • the receiving unit 121 is configured to receive a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing The topology that the system currently needs to update.
  • the data stream processing apparatus may be applied to a management node of a real-time computing system, for example, the management node shown in FIG. 1.
  • control flow may include structural information of a topology that needs to be updated, and may also include a control flow decomposition structure identifier of the management node or the management module internal processing logic, that is, the control flow may be decomposed by the structure identifier and The structure information identifies the topology that needs to be updated currently.
  • the sending unit 122 is configured to send the control flow to a processing node for performing service processing on the data stream, so that the processing node updates the data flow according to the topology that is currently updated according to the control flow.
  • a flow table wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the data is processed by the processing node
  • the stream performs service processing, and transmits the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table.
  • the management node may send the control flow to all processing nodes in the real-time computing system, so that all processing nodes update the stored data flow table.
  • the foregoing apparatus may further include:
  • a holding unit 123 configured to establish and maintain a connection with each of the processing nodes in the real-time computing system
  • the sending unit 122 is configured to send the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.
  • the management node may only send the control flow to the partial processing node.
  • the management node may only refer to these involved.
  • the processing node sends the control flow so that these are involved in the processing node updating the data flow flow table, and without being involved, the data flow flow table may not be updated.
  • the management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system.
  • the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing node performs a service on the data flow. Processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the updated data flow distribution table. This allows dynamic adjustment of the topology of the real-time computing system.
  • FIG. 14 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention. As shown in FIG. 14, the method includes: an obtaining unit 141, an updating unit 142, a first sending unit 143, and a second sending. Unit 144, wherein:
  • the obtaining unit 141 is configured to acquire a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system currently needs to update;
  • the data stream processing apparatus may be applied to a source node of a real-time computing system, for example, a source node as shown in FIG. 1.
  • control flow may be a control flow in which the source node receives user input.
  • the control flow here can refer to the control flow described in the embodiment shown in FIG. 1-4, and will not be repeatedly described herein.
  • the updating unit 142 is configured to update the data flow distribution table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a topology that matches the current need to be updated. Data flow path.
  • the source node After receiving the above control flow, the source node can decompose the control flow, and update the data flow circulation table according to the topology structure that needs to be updated according to the decomposition. In addition, the source node stores or caches the updated data flow table. Of course, the above control flow can also be stored or cached.
  • the first sending unit 143 is configured to send the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node used for performing service processing on the data stream, to And causing the processing node to update the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow circulation table includes data that matches the topology that needs to be updated currently.
  • control node can send the control flow to the processing node, and the processing node updates the stored data flow table.
  • a second sending unit 144 configured to receive a data stream, and send the data stream to the processing node according to a data flow circulation path included in the data flow circulation table updated by the source node, where The processing node performs service processing on the data stream, and sends the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.
  • the data stream may be used to calculate data currently needed by the system in real time, such as data input by the user or data transmitted by the collection device.
  • the source node can send the data flow according to the updated topology to implement dynamic adjustment of the real-time calculation.
  • the topology of the system Since the update unit 142 updates the data flow distribution table, so that the second sending unit 144 sends the data flow according to the updated data flow distribution table, the source node can send the data flow according to the updated topology to implement dynamic adjustment of the real-time calculation.
  • the topology of the system Since the update unit 142 updates the data flow distribution table, so that the second sending unit 144 sends the data flow according to the updated data flow distribution table, the source node can send the data flow according to the updated topology to implement dynamic adjustment of the real-time calculation.
  • the topology of the system Since the update unit 142 updates the data flow distribution table, so that the second sending unit 144 sends the data flow according to the updated data flow distribution table, the source node can send the data flow according to the updated topology to implement dynamic adjustment of the real-time calculation.
  • the topology of the system Since the update unit 142 updates the data flow distribution table,
  • the foregoing apparatus may further include:
  • a third sending unit 145 configured to send the updated data flow circulation table to a shared storage node of the real-time computing system
  • the recovery unit 146 is configured to: when the source node performs failure recovery, acquire the updated data flow distribution table from the shared storage node, and send the data flow according to the updated data flow distribution table.
  • the source node may directly obtain the data flow circulation table updated by the update unit 142 from the shared storage node, so that the source node may be in the shared storage node after the failure occurs. Reading and initializing the data flow table, the HA mechanism for dynamically adjusting the topology is completed.
  • the device may further include:
  • the receiving unit 147 is configured to receive, by the processing node that is fed back by the processing node, an update result of the data flow distribution table.
  • the fourth sending unit 148 is configured to: when the update result indicates that the update fails, send the control flow to the management node, so that the management node sends the control flow to the processing node, by the processing
  • the node updates the data flow flow table according to the topology that is currently updated as described by the control flow.
  • the update result of the current update data flow flow table of each processing node can be obtained in time.
  • the source node may trigger the management node to send a control flow to the processing node that failed the update, so that the processing node that failed the update updates the data flow circulation table again.
  • the source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a current requirement of the real-time computing system.
  • An updated topology the source node updates a data flow flow table according to the topology that is currently updated as described by the control flow, wherein the updated data flow flow table includes the topology that needs to be updated currently a data flow path of the structure matching;
  • the source node transmitting the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a service for processing a data stream Processing the node, so that the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the topology that needs to be updated currently a data flow path of the structure matching;
  • the source node receives the data stream, and sends the data stream to the processing node according to a data flow circulation path included in the
  • FIG. 17 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • the apparatus is applied to a processing node of a real-time computing system, as shown in FIG. 17, and includes: a processor 171, a network interface. 172, a memory 174 for implementing connection communication between the processor 171, the network interface 172, and the memory 173, and a communication bus 174, the processor 171 executing the program stored in the memory 173 Used to implement the following methods:
  • the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently;
  • the processing node When the processing node receives the data stream, performing processing on the data stream, and transmitting the data stream after the service processing according to the data stream circulation path included in the updated data stream circulation table.
  • the processor 171 can also execute the following procedure:
  • the updated data flow distribution table is obtained from the shared storage node, and the data flow is transmitted according to the updated data flow distribution table.
  • the processor 171 can also execute the following procedure:
  • the processing node receives the control flow sent by the management node, and updates the data flow circulation table according to the topology that is currently updated according to the control flow.
  • the processor 171 can also execute the following procedure:
  • the processing node of the real-time computing system receives a control flow sent by a management node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system.
  • the topology structure currently needs to be updated; the processing node updates the data flow circulation table according to the topology that is currently updated as described by the control flow, wherein the updated data flow circulation table includes the current needs update a topologically matched data flow path; when the processing node receives the data stream, the processing node performs a service processing on the data stream, and follows a data flow path included in the updated data flow flow table Sending the data stream after the business processing.
  • the processing logic of the real-time computing system can be updated by updating the data flow table in real time, that is, the topology of the real-time computing system can be dynamically adjusted.
  • FIG. 18 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • the apparatus is applied to a management node of a real-time computing system, as shown in FIG. 18, and includes: a processor 181, a network interface. 182.
  • a memory 184 for communicating communications between the processor 181, the network interface 182 and the memory 183, and a communication bus 184, the processor 181 executing the program stored in the memory 183 Used to implement the following methods:
  • the processing node Transmitting the control flow to a processing node for performing traffic processing on the data stream to cause the
  • the processing node updates the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently.
  • the processing node receives the data stream, the processing node performs service processing on the data stream, and sends the service processed data according to the data flow distribution path included in the updated data flow circulation table. flow.
  • the processor 181 can also execute the following procedure:
  • the program executed by the processor 181 to send the control flow to the processing node for performing service processing on the data stream may include:
  • the management node transmits the control flow to each of the processing nodes of the real-time computing system in a broadcast manner.
  • the management node of the real-time computing system receives a control flow sent by a source node of the real-time computing system for adjusting a topology of the real-time computing system, where the control flow is used to describe the real-time computing system.
  • the topology update data flow circulation table wherein the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently; when the processing node receives the data flow, the processing is performed by the processing
  • the node performs service processing on the data stream, and sends the data stream after the service processing according to the data flow distribution path included in the updated data flow distribution table. This allows dynamic adjustment of the topology of the real-time computing system.
  • FIG. 19 is a schematic structural diagram of another data stream processing apparatus according to an embodiment of the present invention.
  • the apparatus is applied to a source node of a real-time computing system, as shown in FIG. 19, and includes: a processor 191, a network interface. 192, a memory 194 for implementing connection communication between the processor 191, the network interface 192, and the memory 193, and a communication bus 194, the processor 191 executing the program stored in the memory 193 Used to implement the following methods:
  • control flow for adjusting a topology of the real-time computing system, wherein the control flow is used to describe a topology that the real-time computing system currently needs to update;
  • the updated data flow circulation table includes a data flow circulation path that matches the topology that needs to be updated currently;
  • the program executed by the processor 191 may further include:
  • the updated data flow distribution table is obtained from the shared storage node, and the data flow is transmitted according to the updated data flow distribution table.
  • the program executed by the processor 191 may further include:
  • the topology that currently needs to be updated updates the data flow flow table.
  • the source node of the real-time computing system acquires a control flow for adjusting a topology of the real-time computing system, where the control flow is used to describe a topology that the real-time computing system needs to update at present; the source And updating, by the node, the data flow circulation table according to the topology that is currently updated according to the control flow, where the updated data flow distribution table includes a data flow circulation path that matches the topology that needs to be updated currently; Sending, by the source node, the control flow to a management node of the real-time computing system, so that the management node sends the control flow to a processing node for performing service processing on the data stream, so that the processing The node updates the data flow flow table according to the topology that is currently updated according to the control flow, where the updated data flow flow table includes a data flow path that matches the topology that needs to be updated currently.
  • the source node receives the data stream, and sends the data stream to the processing node according to the data flow circulation path included in the data flow circulation table updated by the source node, and the data stream is processed by the processing node Performing business processing, and transmitting the data stream after the service processing according to a data flow distribution path included in the data flow distribution table updated by the processing node.
  • the topology of the real-time computing system can be dynamically adjusted.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

L'invention concerne un procédé et un appareil de traitement de flux de données. Le procédé peut comprendre : la réception, par un nœud de traitement d'un système de calcul en temps réel, d'un flux de commande envoyé par un nœud de gestion du système de calcul en temps réel afin d'ajuster une structure topologique du système de calcul en temps réel, le flux de commande étant utilisé pour décrire une structure topologique en cours nécessitant une mise à jour du système de calcul en temps réel ; la mise à jour, par le nœud de traitement, d'une table de débit de flux de données en fonction de la structure topologique en cours nécessitant une mise à jour décrite par le flux de commande, la table de débit de flux de données mise à jour comprenant un trajet de débit de flux de données correspondant à la structure topologique en cours nécessitant une mise à jour ; lorsque le nœud de traitement reçoit un flux de données, l'exécution, par le nœud de traitement, d'un traitement de service sur le flux de données et l'envoi du flux de données ayant été soumis au traitement de service conformément au trajet de débit de flux de données compris dans la table de débit de flux de données mise à jour. Les modes de réalisation de la présente invention peuvent ajuster de manière dynamique une structure topologique d'un système de calcul en temps réel.
PCT/CN2016/093588 2015-08-27 2016-08-05 Procédé et appareil de traitement de flux de données WO2017032212A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510534143.5A CN106487694B (zh) 2015-08-27 2015-08-27 一种数据流处理方法和装置
CN201510534143.5 2015-08-27

Publications (1)

Publication Number Publication Date
WO2017032212A1 true WO2017032212A1 (fr) 2017-03-02

Family

ID=58099527

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/093588 WO2017032212A1 (fr) 2015-08-27 2016-08-05 Procédé et appareil de traitement de flux de données

Country Status (2)

Country Link
CN (1) CN106487694B (fr)
WO (1) WO2017032212A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11095522B2 (en) * 2019-08-21 2021-08-17 Microsoft Technology Licensing, Llc Dynamic scaling for data processing streaming system

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110019369B (zh) * 2017-12-31 2022-06-07 中国移动通信集团福建有限公司 共用数据流处理拓扑的方法、装置、设备及介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130290554A1 (en) * 2012-04-26 2013-10-31 Qiming Chen Open station as a stream analysis operator container
CN103716182A (zh) * 2013-12-12 2014-04-09 中国科学院信息工程研究所 一种面向实时云平台的故障检测与容错方法及系统
CN104008007A (zh) * 2014-06-12 2014-08-27 深圳先进技术研究院 基于流式计算和批处理计算的互操作数据处理系统及方法
CN104090886A (zh) * 2013-12-09 2014-10-08 深圳市腾讯计算机系统有限公司 构建用户实时画像的方法及装置
WO2014194251A2 (fr) * 2013-05-30 2014-12-04 Vaibhav Nivargi Appareil et procédé pour analyser de manière collaborative des données provenant de sources de données variées
CN104683445A (zh) * 2015-01-26 2015-06-03 北京邮电大学 分布式实时数据融合系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130290554A1 (en) * 2012-04-26 2013-10-31 Qiming Chen Open station as a stream analysis operator container
WO2014194251A2 (fr) * 2013-05-30 2014-12-04 Vaibhav Nivargi Appareil et procédé pour analyser de manière collaborative des données provenant de sources de données variées
CN104090886A (zh) * 2013-12-09 2014-10-08 深圳市腾讯计算机系统有限公司 构建用户实时画像的方法及装置
CN103716182A (zh) * 2013-12-12 2014-04-09 中国科学院信息工程研究所 一种面向实时云平台的故障检测与容错方法及系统
CN104008007A (zh) * 2014-06-12 2014-08-27 深圳先进技术研究院 基于流式计算和批处理计算的互操作数据处理系统及方法
CN104683445A (zh) * 2015-01-26 2015-06-03 北京邮电大学 分布式实时数据融合系统

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11095522B2 (en) * 2019-08-21 2021-08-17 Microsoft Technology Licensing, Llc Dynamic scaling for data processing streaming system

Also Published As

Publication number Publication date
CN106487694A (zh) 2017-03-08
CN106487694B (zh) 2020-03-27

Similar Documents

Publication Publication Date Title
JP7463544B2 (ja) ブロックチェーンメッセージ処理方法、装置、コンピュータデバイスおよびコンピュータプログラム
US10341792B1 (en) System for distributing audio output using multiple devices
WO2019153488A1 (fr) Procédé de gestion de configuration de service, appareil, support de stockage et serveur
US10069942B2 (en) Method and apparatus for changing configurations
WO2019061720A1 (fr) Système et procédé de synchronisation de données
WO2017092347A1 (fr) Procédé, dispositif et système de mise à jour d'une configuration de client dans un système à mémoire cache
US11252035B2 (en) Data configuration method and apparatus
CN107517227B (zh) 用于分布式一致性系统的会话实现方法以及装置
WO2020211629A1 (fr) Appareil et procédé de surveillance de message de liaison courte basés sur une chaîne de blocs
WO2022120806A1 (fr) Procédé et système de messagerie distribuée multi-nuages pour informatique à haute performance
CN112929225B (zh) 会话异常处理方法、装置、计算机设备和存储介质
WO2017032212A1 (fr) Procédé et appareil de traitement de flux de données
CN112118322A (zh) 一种网络设备的数据同步方法、网络设备及系统
US20210334185A1 (en) Task based service management platform
WO2020010906A1 (fr) Procédé et dispositif d'installation par lots d'un système d'exploitation (os), et dispositif de réseau
CN113342503B (zh) 实时进度反馈方法、装置、设备及存储介质
WO2024002390A1 (fr) Procédé et appareil de réponse de message de dispositif pcb, et dispositif informatique et support de stockage
US10498637B2 (en) Switch processing method, controller, switch, and switch processing system
WO2015117365A1 (fr) Procédé, dispositif et système d'interaction de paquets hello
CN110798513B (zh) 物联网设备互联系统及方法
US20180367448A1 (en) Dataflow consistency verification
WO2022121492A1 (fr) Procédé et appareil de transmission de fichiers, dispositif informatique et support d'enregistrement
WO2021136358A1 (fr) Procédé et système de gestion de dispositif de réseau, et dispositif de réseau
CN114697334A (zh) 一种编排任务的执行方法和装置
US9936055B2 (en) Using multicasting to concurrently image multiple client devices

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16838472

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16838472

Country of ref document: EP

Kind code of ref document: A1