CN106487694A - A kind of data flow processing method and device - Google Patents

A kind of data flow processing method and device Download PDF

Info

Publication number
CN106487694A
CN106487694A CN201510534143.5A CN201510534143A CN106487694A CN 106487694 A CN106487694 A CN 106487694A CN 201510534143 A CN201510534143 A CN 201510534143A CN 106487694 A CN106487694 A CN 106487694A
Authority
CN
China
Prior art keywords
data flow
node
renewal
real time
stream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201510534143.5A
Other languages
Chinese (zh)
Other versions
CN106487694B (en
Inventor
李旭良
李嘉
刘杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201510534143.5A priority Critical patent/CN106487694B/en
Priority to PCT/CN2016/093588 priority patent/WO2017032212A1/en
Publication of CN106487694A publication Critical patent/CN106487694A/en
Application granted granted Critical
Publication of CN106487694B publication Critical patent/CN106487694B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/02—Topology update or discovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Z—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS, NOT OTHERWISE PROVIDED FOR
    • G16Z99/00—Subject matter not provided for in other main groups of this subclass
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass

Abstract

The embodiment of the invention discloses a kind of data flow processing method and device, the method may include：The node that processes of real time computation system receives the controlling stream for adjusting the topological structure of the real time computation system that the management node of the real time computation system sends, wherein, the controlling stream is used for describing the topological structure that the real time computation system is currently needed for updating；According to the topological structure update data stream circulation table for being currently needed for described in described by the controlling stream updating, wherein, the data flow circulation table of the renewal includes the data flow circulation path mated with the topological structure for being currently needed for updating to the node that processes；When the process node receives data flow, the process node carries out Business Processing to the data flow, and the data flow circulation path included according to the data flow circulation table of the renewal sends the data flow after the Business Processing.The embodiment of the present invention can the dynamic topological structure for adjusting real time computation system.

Description

A kind of data flow processing method and device
Technical field
The present invention relates to communication technical field, more particularly to a kind of data flow processing method and device.
Background technology
With the progress of science and technology and the development of user's request, there is now substantial amounts of real time data, wherein, real When real-time property strong, and data volume is big, and endless, at present mainly by real time computation system to flow The mode of calculating is calculated to real time data, for example：Storm is applied.However, current real-time calculating system In system due to be all rely on JAVA application programming interface (Application Programming Interface, API) code is realized, and submission task realization of packing, so in the industry of the real time computation system of operation Business logic is immutable, i.e., in real time computation system operationally, topology (Topology) structure of the system is Immutable.But a lot of demands are desirable to be capable of the process logic of dynamic adjustment real time computation system now, I.e. dynamic adjusts the topological structure of real time computation system, such that it is able to adapt to the demand of business.It can be seen that, at present How the topological structure of dynamic adjustment real time computation system is the current technical problem that is badly in need of and solving.
Content of the invention
A kind of data flow processing method and device is embodiments provided, can dynamic adjustment calculating in real time The topological structure of system.
In a first aspect, the embodiment of the present invention provides a kind of data flow processing method, including：
Real time computation system process that node receives that the management node of the real time computation system sends for adjusting The controlling stream of the topological structure of the whole real time computation system, wherein, the controlling stream is used for describing the reality When computing system be currently needed for update topological structure；
The node that processes is updated according to the topological structure for being currently needed for described in described by the controlling stream updating Data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for opening up for renewal with described Flutter the data flow circulation path of structure matching；
When the process node receives data flow, the process node is carried out at business to the data flow Reason, and the data flow circulation path included according to the data flow circulation table of the renewal sends the Business Processing Data flow afterwards.
In the first possible implementation of first aspect, methods described also includes：
The data flow circulation table of the renewal is sent sharing to the real time computation system by the node that processes Memory node；
When the process node carries out fault recovery, the process node is obtained from the shared memory node The data flow circulation table of the renewal, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
In conjunction with the first possible implementation of first aspect or first aspect, the second of first aspect Plant in possible implementation, methods described also includes：
The node that processes feeds back what the data flow circulation table updated to the source node of the real time computation system Result is updated, during so that the renewal representation updating failure, from the source node to the management node Send the controlling stream；
When the renewal result represents renewal failure, the process node receives what the management node sent The controlling stream, and number is updated according to the topological structure for being currently needed for described in described by the controlling stream updating According to stream circulation table.
In conjunction with the first possible implementation of first aspect or first aspect, the 3rd of first aspect the Plant in possible implementation, methods described also includes：
The node that processes feeds back the data flow circulation table renewal to the output node of the real time computation system Renewal result, by the output node all of to the real time computation system process node feeding back described in Update result to be collected, and export summarized results.
Second aspect, the embodiment of the present invention provide a kind of data flow processing method, including：
The management node of real time computation system receive that the source node of the real time computation system sends for adjusting The controlling stream of the topological structure of the real time computation system, wherein, the controlling stream is described real-time for describing Computing system is currently needed for the topological structure for updating；
The management node sends the controlling stream to the process section for carrying out Business Processing to data flow Point, so that the node that processes is according to the topological structure for being currently needed for described in described by the controlling stream updating Update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for updating with described Topological structure coupling data flow circulation path；When the process node receives data flow, by described Process node carries out Business Processing to the data flow, and include according to the data flow circulation table of the renewal Data flow circulation path sends the data flow after the Business Processing.
In the first possible implementation of second aspect, methods described also includes：
The management node is set up and the company of holding with each the described process node in the real time computation system Connect；
The management node sends the controlling stream to the process section for carrying out Business Processing to data flow Point, including：
The controlling stream is sent to each institute of the real time computation system by the management node with broadcast mode State process node.
The third aspect, the embodiment of the present invention provide a kind of data flow processing method, including：
The source node of real time computation system obtains the control for adjusting the topological structure of the real time computation system Stream, wherein, the controlling stream is used for describing the topological structure that the real time computation system is currently needed for updating；
The source node updates number according to the topological structure for being currently needed for described in described by the controlling stream updating According to stream circulation table, wherein, the data flow circulation table of the renewal includes and the topology for being currently needed for and updating The data flow circulation path of structure matching；
The controlling stream is sent to the source node management node of the real time computation system, so that described Management node sends the controlling stream to the process node for carrying out Business Processing to data flow, so that institute State node is processed according to the topological structure update data stream for being currently needed for described in described by the controlling stream updating Circulation table, wherein, the data flow circulation table of the renewal includes and the topological structure for being currently needed for and updating The data flow circulation path of coupling；
The source node receiving data stream, and the data flow stream that the data flow is updated according to the source node The data flow circulation path that logical table includes is sent to the process node, by the process node to the data Stream carries out Business Processing, and the data flow logical circulation road included according to the data flow circulation table of the process node updates Footpath sends the data flow after the Business Processing.
In the first possible implementation of the third aspect, methods described also includes：
The source node sends the data flow circulation table of the renewal to the shared of the real time computation system and deposits Storage node；
When the source node carries out fault recovery, the source node obtains described from the shared memory node The data flow circulation table of renewal, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
In conjunction with the first possible implementation of the third aspect or the third aspect, the second of the third aspect Plant in possible implementation, methods described also includes：
The source node receives data flow circulation table described in the process node updates for processing node feeding back Renewal result；
When the renewal result represents renewal failure, the source node sends the control to the management node System stream, so that the management node sends the controlling stream to the process node, is pressed by the process node The data flow circulation table is updated according to the topological structure for being currently needed for described in described by the controlling stream updating.
Fourth aspect, the embodiment of the present invention provide a kind of data stream processing device, and described device is applied in real time The process node of computing system, including：Receiving unit, the first updating block and the first transmitting element, wherein：
The receiving unit, for receive real time computation system management node send for adjusting the reality When computing system topological structure controlling stream, wherein, the controlling stream be used for describe described real-time calculate system System is currently needed for the topological structure for updating；
First updating block, for according to opening up for being currently needed for described in described by the controlling stream updating Topology update data flow circulation table is flutterred, wherein, the data flow circulation table of the renewal includes and the current need The data flow circulation path of topological structure coupling to be updated；
First transmitting element, for when the process node receives data flow, to the data flow Business Processing is carried out, and the data flow circulation path included according to the data flow circulation table of the renewal sends institute State the data flow after Business Processing.
In the first possible implementation of fourth aspect, described device also includes：
Second transmitting element, for sending the data flow circulation table of the renewal to the real time computation system Shared memory node；
Recovery unit, for when the process node carries out fault recovery, obtaining from the shared memory node The data flow circulation table of the renewal is taken, and sending out for data flow is carried out according to the data flow circulation table of the renewal Send.
In conjunction with the first possible implementation of fourth aspect or fourth aspect, the second of fourth aspect Plant in possible implementation, described device also includes：
First feedback unit, feeds back the data flow circulation table for the source node to the real time computation system The renewal result of renewal, during so that the renewal representation updating failure, from the source node to the pipe Reason node sends the controlling stream；
Second updating block, for when described renewal result represent first updating block update failure when, The controlling stream of management node transmission is received, and according to the described current need described by the controlling stream Topological structure update data stream circulation table to be updated.
In conjunction with the first possible implementation of fourth aspect or fourth aspect, the 3rd of fourth aspect the Plant in possible implementation, described device also includes：
Second feedback unit, feeds back the data flow circulation for the output node to the real time computation system The renewal result that table updates, by the output node to all of process node feeding back of the real time computation system The renewal result collected, and export summarized results.
5th aspect, the embodiment of the present invention provide a kind of data stream processing device, including：Receiving unit and send out Unit is sent, wherein：
The receiving unit, it is described real-time for adjusting that the source node for receiving real time computation system sends The controlling stream of the topological structure of computing system, wherein, the controlling stream is used for describing the real time computation system It is currently needed for the topological structure for updating；
The transmitting element, for sending the controlling stream to the place for carrying out Business Processing to data flow Reason node, so that the node that processes is according to the topology for being currently needed for described in described by the controlling stream updating Topology update data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for described The data flow circulation path of the topological structure coupling of renewal；When the process node receives data flow, by The process node carries out Business Processing, and the data flow circulation table bag according to the renewal to the data flow The data flow circulation path for including sends the data flow after the Business Processing.
In the first possible implementation of the 5th aspect, described device also includes：
Holding unit, for setting up and the company of holding with each the described process node in the real time computation system Connect；
The transmitting element is used for sending the controlling stream with broadcast mode each to the real time computation system The individual process node.
6th aspect, the embodiment of the present invention provide a kind of data stream processing device, and described device is applied in real time The source node of computing system, including：Acquiring unit, updating block, the first transmitting element and second send single Unit, wherein：
The acquiring unit, for obtaining the controlling stream for adjusting the topological structure of the real time computation system, Wherein, the controlling stream is used for describing the topological structure that the real time computation system is currently needed for updating；
The updating block, for according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；
First transmitting element, for the controlling stream to be sent to the management section of the real time computation system Point, so that the management node sends the controlling stream to the process for carrying out Business Processing to data flow Node, so that the node that processes is according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；
Second transmitting element, for receiving data stream, and by the data flow according to the source node more The data flow circulation path that new data flow circulation table includes is sent to the process node, is saved by described process Put carries out Business Processing to the data flow, and include according to the data flow circulation table of the process node updates Data flow circulation path sends the data flow after the Business Processing.
In the first possible implementation of the 6th aspect, described device also includes：
3rd transmitting element, for sending the data flow circulation table of the renewal to the real time computation system Shared memory node；
Recovery unit, for when the source node carries out fault recovery, obtaining from the shared memory node The data flow circulation table of the renewal, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
In conjunction with the 6th aspect or the first possible implementation of the 6th aspect, the second of the 6th aspect Plant in possible implementation, described device also includes：
Receiving unit, for receiving data flow stream described in the process node updates for processing node feeding back The renewal result of logical table；
4th transmitting element, for when the renewal result represents renewal failure, sending out to the management node The controlling stream is sent, so that the management node sends the controlling stream to the process node, by described Reason node updates the data flow according to the topological structure for being currently needed for described in described by the controlling stream updating Circulation table.
In technique scheme, the management for processing the node reception real time computation system of real time computation system The controlling stream for adjusting the topological structure of the real time computation system that node sends, wherein, the control Flow and be currently needed for, for describing the real time computation system, the topological structure for updating；The process node is according to institute State the topological structure update data stream circulation table for being currently needed for updating described by controlling stream, wherein, institute The data flow circulation table for stating renewal includes the data flow circulation that mates with the topological structure for being currently needed for updating Path；When the process node receives data flow, the process node carries out business to the data flow Process, and the data flow circulation path included according to the data flow circulation table of the renewal is sent at the business Data flow after reason.So can realize by way of real-time update data flow circulation table updating calculating in real time The process logic of system, you can to dynamically adjust the topological structure of real time computation system.
Description of the drawings
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to implementing Example or accompanying drawing to be used needed for description of the prior art are briefly described, it should be apparent that, describe below In accompanying drawing be only some embodiments of the present invention, for those of ordinary skill in the art, do not paying On the premise of going out creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is the framework of the applicable real time computation system of data flow processing method provided in an embodiment of the present invention Figure；
Fig. 2 is a kind of schematic flow sheet of data flow processing method provided in an embodiment of the present invention；
Fig. 3 is the schematic flow sheet of another kind data flow processing method provided in an embodiment of the present invention；
Fig. 4 is the schematic flow sheet of another kind data flow processing method provided in an embodiment of the present invention；
Fig. 5 is the schematic flow sheet of another kind data flow processing method provided in an embodiment of the present invention；
Fig. 6 is the control streaming schematic diagram by taking Storm application as an example provided in an embodiment of the present invention；
Fig. 7 is the data stream transmitting schematic diagram by taking Storm application as an example provided in an embodiment of the present invention；
Fig. 8 is a kind of structural representation of data stream processing device provided in an embodiment of the present invention；
Fig. 9 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 10 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 11 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 12 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 13 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 14 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 15 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 16 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 17 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 18 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention；
Figure 19 is the structural representation of another kind data stream processing device provided in an embodiment of the present invention.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clearly Chu, it is fully described by, it is clear that described embodiment is only a part of embodiment of the present invention, rather than Whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making creation Property work under the premise of the every other embodiment that obtained, belong to the scope of protection of the invention.
It is the applicable real-time calculating of data flow processing method provided in an embodiment of the present invention to refer to Fig. 1, Fig. 1 The Organization Chart of system, as shown in figure 1, including：Subject of supervision 11, source node 12, and also include one Or multiple process nodes 13 and management module 14, wherein, subject of supervision 11 respectively with source node 12 and Each processes node 13 and connects, and source node 12 is at least connected with one of node 13 that processes, and at each Reason node 13 is sequentially connected.In addition, management module 14 includes management node 141 and output node 142, its In, management node 141 and source node and each process node 13 and be connected, output node 142 with each at Reason node 13 connects.Wherein, the annexation performance in above-mentioned real time computation system between each node is the reality When computing system topology (Topology), or be interpreted as topological model.
It should be noted that the connection between node described in the present embodiment is can be understood as in logic connection Connection, the transmission of data flow or controlling stream can be carried out between the node of connection.
In above-mentioned real time computation system operationally, data flow is flowed into from source node 12, then source node 12 Data flow is sent to corresponding process node 13, the process node 13 pairs by the data flow circulation table according to storage The data flow of reception carries out Business Processing, and wherein, the Business Processing can include to count data flow in real time Calculate or stream calculation is interpreted as, the process node 13 circulates table by after Business Processing according to the data flow of storage Data flow is sent to another process node 13, by the process node 13 according to storage data flow circulation table by industry Data flow after business is processed is sent to another node or output result.Wherein, in the embodiment of the present invention, number According to the data flow circulation path that stream circulation table includes to mate with the topological structure of real time computation system, i.e. this enforcement In example, by data flow circulation table come the transmission path of control data stream or transmission structure, so either Source node 12 or process node 13 be able to will be currently needed for sending out by the data flow circulation table of each storage The data flow that send is sent to corresponding node.For example：The topological structure of above-mentioned real time computation system includes that source is saved 12 connection of point processes node A, processes node A reconnection and processes node B, processes node B reconnection and process Node C, it is end node to process node C.So just permissible in the data flow circulation table of the storage of source node 12 Sent the data flow that node A storage is processed to the mapping for processing node A including data flow by source node 12 Circulation table can just include that data flow is sent to the mapping for processing node B by node A is processed, and process node B The data flow circulation table of storage can just include that data flow is sent to the mapping for processing node C by node B is processed, The data flow circulation table for processing node C storage can just include to process reflecting for node C outgoing traffic result Penetrate.Certainly, in the present embodiment, the data flow circulation table of each node storage can include all mappings or Only include the mapping related to oneself.
In above-mentioned real time computation system operationally, controlling stream can be transmitted to management node from source node 12 141, wherein, the controlling stream is used for describing the topological structure that the real time computation system is currently needed for updating.Pipe Control is streaming to each and processes node 13 by reason node 141 again, and so these process nodes 13 just can root The data flow circulation table of storage is updated according to controlling stream, so as to realize the topology knot of dynamic adjustment real time computation system Structure.
In addition, in the embodiment of the present invention, above-mentioned real time computation system can be distributed system, i.e., above-mentioned each Node can be distributed and operate in different machines, it is of course also possible to allow a part of node operation therein In uniform machinery, for example：Management node 121 and output node 122 may operate in uniform machinery or Person is operated in different machines.In addition, in the embodiment of the present invention, these machines are not construed as limiting, such as： These machines can be computer or server etc..
The schematic flow sheet that Fig. 2, Fig. 2 are a kind of data flow processing methods provided in an embodiment of the present invention is referred to, As shown in Fig. 2 comprising the following steps：
201st, the node that processes of real time computation system receives the use that the management node of the real time computation system sends In the controlling stream of the topological structure for adjusting the real time computation system, wherein, the controlling stream is used for describing institute State the topological structure that real time computation system is currently needed for updating.
In the present embodiment, the topological structure for being currently needed for updating of above-mentioned control stream description can be calculated in real time System is currently needed for the complete topology structure for updating, for example：Above-mentioned controlling stream can describe as shown in Figure 1 Topological structure between source node, process node, management node and output node.Or, above-mentioned controlling stream The topological structure for being currently needed for updating of description can be the topological structure for being currently needed for adjusting, for example：Originally Topological structure in process node A connection and process node B, process node B and reconnect and process node C, i.e., The circulation order of data flow is to process node A to arrive process node C again to node B is processed, and is currently needed for adjusting Whole process node C for processing node A connection, process node C and reconnect and process node B.So, above-mentioned Controlling stream can be only that description process node A connection processes node C, process node C and reconnect process node The topological structure of B, that is, after updating, the circulation order of data flow is to process node A to be arrived to node B is processed again Process node C.
In addition, in the present embodiment, above-mentioned controlling stream can be understood as controlling stream information, i.e., above-mentioned controlling stream can To be interpreted as an information.
202nd, process node to update according to the topological structure for being currently needed for described in described by the controlling stream updating Data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for opening up for renewal with described Flutter the data flow circulation path of structure matching.
In the present embodiment, process node and be stored with the data flow circulation table mated with former topological structure, work as reception During to above-mentioned controlling stream, i.e., current topological structure need update when, it is possible to according to be currently needed for update open up Flutter topology update data flow circulation table.And the data flow circulation table after updating includes and is currently needed for opening up for renewal The data flow circulation path of structure matching is flutterred, and the process node can be thus realized when data flow is sent, Be according to renewal after topological structure send data flow.
In addition, the above-mentioned data flow circulation path mated with topological structure that is being currently needed for updating can be understood as The circulation path of data flow or currency structure in the topological structure for being currently needed for updating.For example：Currently need Topological structure to be updated is to process node A connection to process node C, then, the data flow stream of above-mentioned renewal Logical table will include that data flow is transferred to process node C from node A is processed, i.e., when the place of execution step 202 Reason node is for processing node A, then, the data flow circulation table of above-mentioned renewal can just include to spread data Transport to the data flow circulation path for processing node C.
203rd, when the process node receives data flow, the process node enters industry to the data flow Business is processed, and the data flow circulation path included according to the data flow circulation table of the renewal sends the business Data flow after process.
Can realize processing node by above-mentioned steps is entered according to the topological structure for updating when data flow is sent Row sends, such that it is able to realize dynamically adjusting the topological structure of real time computation system.In addition, real-time Computing system is when flood tide service operation is carried out by the topology knot of above-mentioned steps dynamic adjustment real time computation system The interference that structure will not be caused to the data flow that processed.In addition, when real time computation system is distributed System, when each node operates in different machines, by above-mentioned steps dynamic adjustment real time computation system Topological structure can avoid the delayed problem for causing of the modification of certain machine.
In the present embodiment, above-mentioned process node can be any process node in real time computation system.
In the present embodiment, the management node for processing the node reception real time computation system of real time computation system The controlling stream for adjusting the topological structure of the real time computation system for sending, wherein, the controlling stream is used In the topological structure that the description real time computation system is currently needed for updating；The process node is according to the control The described topological structure update data stream circulation table for being currently needed for updating of system stream, wherein, described more New data flow circulation table includes the data flow logical circulation road that mates with the topological structure for being currently needed for updating Footpath；When the process node receives data flow, the process node is carried out at business to the data flow Reason, and the data flow circulation path included according to the data flow circulation table of the renewal sends the Business Processing Data flow afterwards.So can realize by way of real-time update data flow circulation table updating in real time and calculate system The process logic of system, you can to dynamically adjust the topological structure of real time computation system.
Refer to the flow process that Fig. 3, Fig. 3 are another kind data flow processing methods provided in an embodiment of the present invention to illustrate Figure, as shown in figure 3, comprise the following steps：
301st, the node that processes of real time computation system receives the use that the management node of the real time computation system sends In the controlling stream of the topological structure for adjusting the real time computation system, wherein, the controlling stream is used for describing institute State the topological structure that real time computation system is currently needed for updating.
302nd, process node to update according to the topological structure for being currently needed for described in described by the controlling stream updating Data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for opening up for renewal with described Flutter the data flow circulation path of structure matching.
In the present embodiment, node is processed when update data stream circulates table, can suspend data flow transmission, but Business Processing can be carried out to data flow.Or node is processed when update data stream circulates table, can suspend The transmission of Business Processing data stream is carried out to data flow.When data flow circulation table has been updated, recover time-out Data flow sends and/or the Business Processing to data flow.Dynamic adjustment real time computation system so can be implemented in Topological structure when, the error handle of data flow and the occlusive effects performance of system not so occurs.
303rd, when the process node receives data flow, the process node enters industry to the data flow Business is processed, and the data flow circulation path included according to the data flow circulation table of the renewal sends the business Data flow after process.
Said method can also comprise the steps：
304th, process node and the data flow circulation table of the renewal is sent sharing to the real time computation system Memory node.
305th, when the process node carries out fault recovery, the process node is from the shared memory node The data flow circulation table of the renewal is obtained, and data flow is carried out according to the data flow circulation table of the renewal Send.
In the embodiment, it is possible to achieve process node after failure, when being recovered can directly from The data flow circulation table that shared memory node obtaining step 302 updates, is going out such that it is able to realize process node After existing fault, the data flow circulation table can be read and is initialized from shared memory node, that is, is completed dynamic High Availabitity (High Availability, the HA) mechanism of state adjustment topological structure.
In the present embodiment, said method can also comprise the steps：
The node that processes feeds back what the data flow circulation table updated to the source node of the real time computation system Result is updated, during so that the renewal representation updating failure, from the source node to the management node Send the controlling stream；
When the renewal result represents renewal failure, the process node receives what the management node sent The controlling stream, and number is updated according to the topological structure for being currently needed for described in described by the controlling stream updating According to stream circulation table.
In the embodiment, above-mentioned renewal result can be the renewal result of step 302, when step 302 more During new success, it is possible to tell source node above-mentioned process node updates success by above-mentioned renewal result.When more During new failure, tell source node above-mentioned process node updates failure by above-mentioned renewal result, so above-mentioned source Node will send above-mentioned controlling stream to management node again, be sent to the process node from management node again Above-mentioned controlling stream is updated again so as to process node.Certainly, sent to management node in source node again The identification information of the process node for updating failure during above-mentioned controlling stream, can also be carried, can so make pipe Reason node only sends controlling stream again to the process node for updating failure, and not to the process node being updated successfully Controlling stream is sent again, to save transfer resource.
The embodiment can be realized correctly feeding back renewal result, if updating failure, can open again Move more new task.
In the present embodiment, said method can also comprise the steps：
Process node data flow circulation table renewal is fed back more to the output node of the real time computation system New result, by the output node renewal that process node feeding back all of to the real time computation system As a result collected, and exported summarized results.
The output node can obtain the renewal result that each processes node feeding back, and such output node can To collect to which, such that it is able to export summarized results, for example：The summarized results is sent to represent and is set Standby, or the summarized results is printed etc., so as to let the user know that the topological structure of real time computation system The state of adjustment.
In the present embodiment, the embodiment of plurality of optional is increased on the basis of the embodiment shown in Fig. 2, And the topological structure of dynamic adjustment real time computation system can be realized.
Refer to the flow process that Fig. 4, Fig. 4 are another kind data flow processing methods provided in an embodiment of the present invention to illustrate Figure, as shown in figure 4, comprise the following steps：
401st, the management node of real time computation system receives being used for for the source node transmission of the real time computation system The controlling stream of the topological structure of the real time computation system is adjusted, wherein, the controlling stream is described for describing Real time computation system is currently needed for the topological structure for updating.
In the present embodiment, above-mentioned controlling stream can include the structural information of the topological structure for being currently needed for updating, And the controlling stream decomposition texture that can also be appointed including management node or management module internal processing logic Mark, you can be currently needed for the topology for updating to identify by controlling stream decomposition texture mark and structural information Structure.
402nd, management node sends the controlling stream to the process section for carrying out Business Processing to data flow Point, so that the node that processes is according to the topological structure for being currently needed for described in described by the controlling stream updating Update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for updating with described Topological structure coupling data flow circulation path；When the process node receives data flow, by described Process node carries out Business Processing to the data flow, and include according to the data flow circulation table of the renewal Data flow circulation path sends the data flow after the Business Processing.
In the present embodiment, management node can be above-mentioned to all process nodes transmission in real time computation system Controlling stream, the data flow circulation table that so all process nodes all can be stored to which are updated.For example：On The method of stating can also comprise the steps：
The management node is set up and the company of holding with each the described process node in the real time computation system Connect；
Above-mentioned management node sends the controlling stream to the process section for carrying out Business Processing to data flow The step of point, can include：
The controlling stream is sent to each institute of the real time computation system by the management node with broadcast mode State process node.
Can thus realize, by broadcast mode, controlling stream is sent to all process nodes.
Certainly, in the present embodiment, management node can also be to process node to part to send controlling stream, example Such as：When being currently needed for relating only to part process node in the topological structure for updating, then, management node is just Only controlling stream can be sent to these process nodes being involved, so that these are involved process node more New data stream circulation table, then can not update data stream circulation table without being involved.
In the present embodiment, the management node of real time computation system receives the source node of the real time computation system and sends out The controlling stream for adjusting the topological structure of the real time computation system that send, wherein, the controlling stream is used for The topological structure that the real time computation system is currently needed for updating is described；The management node is by the controlling stream Send to the process node for carrying out Business Processing to data flow, so that the process node is according to the control The described topological structure update data stream circulation table for being currently needed for updating of system stream, wherein, described more New data flow circulation table includes the data flow logical circulation road that mates with the topological structure for being currently needed for updating Footpath；When the process node receives data flow, business is carried out to the data flow by the process node Process, and the data flow circulation path included according to the data flow circulation table of the renewal is sent at the business Data flow after reason.The topological structure of dynamic adjustment real time computation system can so be realized.
Refer to the flow process that Fig. 5, Fig. 5 are another kind data flow processing methods provided in an embodiment of the present invention to illustrate Figure, as shown in figure 5, comprise the following steps：
501st, the source node of real time computation system is obtained for the topological structure that adjusts the real time computation system Controlling stream, wherein, the controlling stream is used for describing the topology knot that the real time computation system is currently needed for updating Structure.
Wherein, above-mentioned controlling stream can be the controlling stream of source node receiving user's input.Here controlling stream can With the controlling stream described in the embodiment with reference to shown in Fig. 1-4, explanation is not repeated herein.
502nd, source node updates number according to the topological structure for being currently needed for described in described by the controlling stream updating According to stream circulation table, wherein, the data flow circulation table of the renewal includes and the topology for being currently needed for and updating The data flow circulation path of structure matching.
After source node receives above-mentioned controlling stream, it is possible to which which is decomposed, and according to working as that decomposition is obtained The front topological structure update data stream circulation table for needing to update.In addition, source node can be stored or buffer update Data flow circulation table afterwards.It is, of course, also possible to store or cache above-mentioned controlling stream.
503rd, the controlling stream is sent to source node the management node of the real time computation system, so that described Management node sends the controlling stream to the process node for carrying out Business Processing to data flow, so that institute State node is processed according to the topological structure update data stream for being currently needed for described in described by the controlling stream updating Circulation table, wherein, the data flow circulation table of the renewal includes and the topological structure for being currently needed for and updating The data flow circulation path of coupling.
Can just realize controlling stream being sent to node is processed by management node by the step, process node again The data flow circulation table stored by which is updated.
504th, source node receiving data stream, and the data flow stream that the data flow is updated according to the source node The data flow circulation path that logical table includes is sent to the process node, by the process node to the data Stream carries out Business Processing, and the data flow logical circulation road included according to the data flow circulation table of the process node updates Footpath sends the data flow after the Business Processing.
In the present embodiment, data that above-mentioned data flow can be transmitted with the data of user input or collecting device etc. Real time computation system is currently needed for the data for calculating.
Due to step 502 update data stream circulation table, such step 504 is circulated according to the data flow after renewal Table send data flow, it is possible to realize source node be according to renewal after topological structure send data flow, with reality The topological structure of existing dynamic adjustment real time computation system.
In the present embodiment, said method can also comprise the steps：
The source node sends the data flow circulation table of the renewal to the shared of the real time computation system and deposits Storage node；
When the source node carries out fault recovery, the source node obtains described from the shared memory node The data flow circulation table of renewal, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
In the embodiment, it is possible to achieve source node after failure, can be directly from altogether when being recovered The data flow circulation table of the renewal of memory node obtaining step 502 is enjoyed, event is occurring such that it is able to realize source node After barrier, the data flow circulation table can be read and is initialized from shared memory node, that is, completes dynamic tune The HA mechanism of whole topological structure.
In the present embodiment, said method can also comprise the steps：
The source node receives data flow circulation table described in the process node updates for processing node feeding back Renewal result；
When the renewal result represents renewal failure, the source node sends the control to the management node System stream, so that the management node sends the controlling stream to the process node, is pressed by the process node The data flow circulation table is updated according to the topological structure for being currently needed for described in described by the controlling stream updating.
In the embodiment, each renewal knot for processing the current update data stream circulation table of node can be obtained in time Really.When some or multiple process node updates fail when, source node can trigger management node to renewal The process node of failure sends controlling stream, so as to update the process node of failure, update data stream circulates table again.
In the present embodiment, the source node of real time computation system is obtained for adjusting opening up for the real time computation system The controlling stream of structure is flutterred, wherein, the controlling stream is currently needed for updating for describing the real time computation system Topological structure；The source node is according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；The controlling stream is sent to the reality by the source node When computing system management node so that the management node sends the controlling stream to for data flow The process node of Business Processing is carried out, so that the process node is according to working as described in described by the controlling stream The front topological structure update data stream circulation table for needing to update, wherein, the data flow circulation table bag of the renewal Include the data flow circulation path mated with the topological structure for being currently needed for updating；The source node receives number According to stream, and the data flow circulation that the data flow circulation table that the data flow is updated according to the source node includes Path is sent to the process node, is carried out Business Processing by the process node to the data flow, and is pressed The data flow circulation path included according to the data flow circulation table of the process node updates is sent after the Business Processing Data flow.Can the dynamic topological structure for adjusting real time computation system.
It is that the control streaming by taking Storm application as an example provided in an embodiment of the present invention is shown to refer to Fig. 6, Fig. 6 It is intended to, as shown in fig. 6, including：
601st, warning node receives control flow message, and parsing control flow message obtains controlling stream, and according to the control System stream update data stream circulation table, and control flow message is sent to police station's node.
Wherein, Storm application is real time computation system.Above-mentioned warning node is can be understood as shown in Fig. 1-Fig. 5 Embodiment in source node.The warning node is the Spout function class node of Storm real time computation system, Wherein, Spout function class refers to the data source class in Storm application, concurrent for receiving external data flow Send, or oneself build data flow and send.
Above-mentioned police station's node is interpreted as the management node in the embodiment shown in Fig. 1-Fig. 5, and police station's node is Bolt function class node in Storm application, relies on the police module main body (pipe in embodiment shown in Fig. 1 Reason module bodies) support, inside is automatically integrating in Storm applied topology structure, is applied with Storm In except to oneself send data warning node in addition to all nodes keep connection.Wherein, Bolt function class section Point is the data processing node in Storm application, and each Bolt achieves different service logics, multiple Bolt Combination completes the business logic processing of complexity.
602nd, police station's node will control flow message to be sent to police service node A, police service node B and police service node C.
Wherein, police service node can be understood as the process node in the embodiment shown in Fig. 1-Fig. 5, be Bolt Function class node.
603rd, police service node A decomposes control flow message, and the controlling stream update data stream stream obtained according to decomposition Logical table.
604th, police service node A will control stream process message feedback to alert node, and by the anti-of processing procedure Feedback message feedback is to warning node.
When step 603 updates failure, police service node A can get control flow message from warning node, To realize being updated again, in this case, the control stream process message to alert node feedback can be again The renewal result of secondary renewal.
605th, police service node B decomposes control flow message, and the controlling stream update data stream stream obtained according to decomposition Logical table.
606th, police service node B will control stream process message feedback to alert node, and by the anti-of processing procedure Feedback message feedback is to warning node.
607th, police service node C decomposes control flow message, and the controlling stream update data stream stream obtained according to decomposition Logical table.
608th, police service node C will control stream process message feedback to alert node, and by the anti-of processing procedure Feedback message feedback is to warning node.
609th, alert node collects the more new construction of feedback, and sends according to predetermined logic or print control Stream process structure.
It is that the data stream transmitting by taking Storm application as an example provided in an embodiment of the present invention shows to refer to Fig. 7, Fig. 7 It is intended to, as shown in fig. 7, comprises：
701st, warning node receiving data stream, and data flow is sent to process node according to data flow circulation table A.
702nd, police service node A carries out Business Processing to receiving data flow, and according to data flow circulation table by industry Data flow after business is processed is sent to police service node B.
703rd, police service node B carries out Business Processing to receiving data flow, and according to data flow circulation table by industry Data flow after business is processed is sent to police service node C.
704th, police service node C carries out Business Processing to receiving data flow, and according to data flow circulation table by industry Data flow after business is processed is sent to next node or output result.
The structural representation that Fig. 8, Fig. 8 are a kind of data stream processing devices provided in an embodiment of the present invention is referred to, As shown in figure 8, including：Receiving unit 81, the first updating block 82 and the first transmitting element 83, wherein：
Receiving unit 81, it is described real-time for adjusting that the management node for receiving real time computation system sends The controlling stream of the topological structure of computing system, wherein, the controlling stream is used for describing the real time computation system It is currently needed for the topological structure for updating.
It should be noted that the data stream processing device in the present embodiment can apply to real time computation system Node is processed, for example：Process node as shown in Figure 1.
In the present embodiment, the topological structure for being currently needed for updating of above-mentioned control stream description can be calculated in real time System is currently needed for the complete topology structure for updating, for example：Above-mentioned controlling stream can describe as shown in Figure 1 Topological structure between source node, process node, management node and output node.Or, above-mentioned controlling stream The topological structure for being currently needed for updating of description can be the topological structure for being currently needed for adjusting, for example：Originally Topological structure in process node A connection and process node B, process node B and reconnect and process node C, i.e., The circulation order of data flow is to process node A to arrive process node C again to node B is processed, and is currently needed for adjusting Whole process node C for processing node A connection, process node C and reconnect and process node B.So, above-mentioned Controlling stream can be only that description process node A connection processes node C, process node C and reconnect process node The topological structure of B, that is, after updating, the circulation order of data flow is to process node A to be arrived to node B is processed again Process node C.
In addition, in the present embodiment, above-mentioned controlling stream can be understood as controlling stream information, i.e., above-mentioned controlling stream can To be interpreted as an information.
First updating block 82, for according to being currently needed for the topology for updating described in described by the controlling stream Topology update data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for described The data flow circulation path of the topological structure coupling of renewal.
In the present embodiment, process node and be stored with the data flow circulation table mated with former topological structure, work as reception During to above-mentioned controlling stream, i.e., current topological structure need update when, it is possible to according to be currently needed for update open up Flutter topology update data flow circulation table.And the data flow circulation table after updating includes and is currently needed for opening up for renewal The data flow circulation path of structure matching is flutterred, and the process node can be thus realized when data flow is sent, Be according to renewal after topological structure send data flow.
In addition, the above-mentioned data flow circulation path mated with topological structure that is being currently needed for updating can be understood as The circulation path of data flow or currency structure in the topological structure for being currently needed for updating.For example：Currently need Topological structure to be updated is to process node A connection to process node C, then, the data flow stream of above-mentioned renewal Logical table will include that data flow is transferred to process node C from node A is processed, i.e., when said apparatus are applied to place Reason node is for processing node A, then, the data flow circulation table of above-mentioned renewal can just include to spread data Transport to the data flow circulation path for processing node C.
First transmitting element 83, for when the process node receives data flow, entering to the data flow Row Business Processing, and the data flow circulation path transmission included according to the data flow circulation table of the renewal is described Data flow after Business Processing.
Can realize processing node by said apparatus is entered according to the topological structure for updating when data flow is sent Row sends, such that it is able to realize dynamically adjusting the topological structure of real time computation system.In addition, real-time Computing system is when flood tide service operation is carried out by the topology knot of above-mentioned steps dynamic adjustment real time computation system The interference that structure will not be caused to the data flow that processed.In addition, when real time computation system is distributed System, when each node operates in different machines, by above-mentioned steps dynamic adjustment real time computation system Topological structure can avoid the delayed problem for causing of the modification of certain machine.
In the present embodiment, above-mentioned process node can be any process node in real time computation system.
In the present embodiment, the management node for processing the node reception real time computation system of real time computation system The controlling stream for adjusting the topological structure of the real time computation system for sending, wherein, the controlling stream is used In the topological structure that the description real time computation system is currently needed for updating；The process node is according to the control The described topological structure update data stream circulation table for being currently needed for updating of system stream, wherein, described more New data flow circulation table includes the data flow logical circulation road that mates with the topological structure for being currently needed for updating Footpath；When the process node receives data flow, the process node is carried out at business to the data flow Reason, and the data flow circulation path included according to the data flow circulation table of the renewal sends the Business Processing Data flow afterwards.So can realize by way of real-time update data flow circulation table updating in real time and calculate system The process logic of system, you can to dynamically adjust the topological structure of real time computation system.
Refer to the structural representation that Fig. 9, Fig. 9 are another kind data stream processing devices provided in an embodiment of the present invention Figure, as shown in figure 9, including：Receiving unit 91, the first updating block 92 and the first transmitting element 93, Wherein：
Receiving unit 91, it is described real-time for adjusting that the management node for receiving real time computation system sends The controlling stream of the topological structure of computing system, wherein, the controlling stream is used for describing the real time computation system It is currently needed for the topological structure for updating.
It should be noted that the data stream processing device in the present embodiment can apply to real time computation system Node is processed, for example：Process node as shown in Figure 1.
First updating block 92, for according to being currently needed for the topology for updating described in described by the controlling stream Topology update data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for described The data flow circulation path of the topological structure coupling of renewal.
In the present embodiment, node is processed when update data stream circulates table, can suspend data flow transmission, but Business Processing can be carried out to data flow.Or node is processed when update data stream circulates table, can suspend The transmission of Business Processing data stream is carried out to data flow.When data flow circulation table has been updated, recover time-out Data flow sends and/or the Business Processing to data flow.Dynamic adjustment real time computation system so can be implemented in Topological structure when, the error handle of data flow and the occlusive effects performance of system not so occurs.
First transmitting element 93, for when the process node receives data flow, entering to the data flow Row Business Processing, and the data flow circulation path transmission included according to the data flow circulation table of the renewal is described Data flow after Business Processing.
In the present embodiment, described device can also include：
Second transmitting element 94, for sending the data flow circulation table of the renewal to real-time calculating be The shared memory node of system；
Recovery unit 95, for when the process node carries out fault recovery, from the shared memory node The data flow circulation table of the renewal is obtained, and data flow is carried out according to the data flow circulation table of the renewal Send.
In the embodiment, it is possible to achieve process node after failure, when being recovered can directly from Shared memory node obtains the data flow circulation table that the first updating block 92 updates, such that it is able to realize processing section Point after failure, can read and initialize the data flow circulation table from shared memory node, i.e., complete The HA mechanism of dynamic adjustment topological structure is become.
As shown in Figure 10, said apparatus can also include：
First feedback unit 96, feeds back the data flow circulation for the source node to the real time computation system The renewal result that table updates, during so that the renewal representation updating failure, from the source node to described Management node sends the controlling stream；
Second updating block 97, for when described renewal result represent first updating block update failure when, The controlling stream of management node transmission is received, and according to the described current need described by the controlling stream Topological structure update data stream circulation table to be updated.
In the embodiment, above-mentioned renewal result can be the renewal result of the first updating block 92, when first When updating block 92 is updated successfully, it is possible to tell the above-mentioned process node of source node more by above-mentioned renewal result New success.When failure is updated, tell source node above-mentioned process node updates failure by above-mentioned renewal result, So above-mentioned source node will send above-mentioned controlling stream to management node again, from management node again at this Reason node sends above-mentioned controlling stream and is updated so as to process node again.Certainly, in source node again to pipe When reason node sends above-mentioned controlling stream, the identification information of the process node for updating failure can also be carried, this Sample can make management node only send controlling stream again to the process node for updating failure, and not to being updated successfully Process node send controlling stream again, to save transfer resource.
The embodiment can be realized correctly feeding back renewal result, if updating failure, can open again Move more new task.
As shown in figure 11, said apparatus can also include：
Second feedback unit 98, feeds back the data flow stream for the output node to the real time computation system The renewal result that logical table updates, anti-by output node process node all of to the real time computation system The renewal result of feedback is collected, and exports summarized results.
The output node can obtain the renewal result that each processes node feeding back, and such output node can To collect to which, such that it is able to export summarized results, for example：The summarized results is sent to represent and is set Standby, or the summarized results is printed etc., so as to let the user know that the topological structure of real time computation system The state of adjustment.
In the present embodiment, the embodiment of plurality of optional is increased on the basis of the embodiment shown in Fig. 8, And the topological structure of dynamic adjustment real time computation system can be realized.
Refer to the structure that Figure 12, Figure 12 are another kind data stream processing devices provided in an embodiment of the present invention to show It is intended to, as shown in figure 12, including：Receiving unit 121 and transmitting element 122, wherein：
Receiving unit 121, by receive real time computation system source node send by adjust described real-time based on The controlling stream of the topological structure of calculation system, wherein, the controlling stream is worked as describing the real time computation system The front topological structure for needing to update.
It should be noted that the data stream processing device that the present embodiment is provided can apply to real time computation system Management node, for example：Management node shown in Fig. 1.
In the present embodiment, above-mentioned controlling stream can include the structural information of the topological structure for being currently needed for updating, And the controlling stream decomposition texture that can also be appointed including management node or management module internal processing logic Mark, you can be currently needed for the topology for updating to identify by controlling stream decomposition texture mark and structural information Structure.
Transmitting element 122, for sending the controlling stream to the process for carrying out Business Processing to data flow Node, so that the node that processes is according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；When the process node receives data flow, by institute State process node carries out Business Processing to the data flow, and includes according to the data flow circulation table of the renewal Data flow circulation path send the data flow after the Business Processing.
In the present embodiment, management node can be above-mentioned to all process nodes transmission in real time computation system Controlling stream, the data flow circulation table that so all process nodes all can be stored to which are updated.For example：Such as Shown in Figure 13, said apparatus can also include：
Holding unit 123, for setting up and keeping with each the described process node in the real time computation system Connection；
The transmitting element 122 is used for sending the controlling stream to the real time computation system with broadcast mode Each described process node.
Can thus realize, by broadcast mode, controlling stream is sent to all process nodes.
Certainly, in the present embodiment, management node can also be to process node to part to send controlling stream, example Such as：When being currently needed for relating only to part process node in the topological structure for updating, then, management node is just Only controlling stream can be sent to these process nodes being involved, so that these are involved process node more New data stream circulation table, then can not update data stream circulation table without being involved.
In the present embodiment, the management node of real time computation system receives the source node of the real time computation system and sends out The controlling stream for adjusting the topological structure of the real time computation system that send, wherein, the controlling stream is used for The topological structure that the real time computation system is currently needed for updating is described；The management node is by the controlling stream Send to the process node for carrying out Business Processing to data flow, so that the process node is according to the control The described topological structure update data stream circulation table for being currently needed for updating of system stream, wherein, described more New data flow circulation table includes the data flow logical circulation road that mates with the topological structure for being currently needed for updating Footpath；When the process node receives data flow, business is carried out to the data flow by the process node Process, and the data flow circulation path included according to the data flow circulation table of the renewal is sent at the business Data flow after reason.The topological structure of dynamic adjustment real time computation system can so be realized.
Refer to the structure that Figure 14, Figure 14 are another kind data stream processing devices provided in an embodiment of the present invention to show It is intended to, as shown in figure 14, including：Acquiring unit 141, updating block 142, the first transmitting element 143 With the second transmitting element 144, wherein：
Acquiring unit 141, for obtaining the controlling stream for adjusting the topological structure of the real time computation system, Wherein, the controlling stream is used for describing the topological structure that the real time computation system is currently needed for updating；
It should be noted that the data stream processing device provided in the present embodiment can apply to calculate system in real time The source node of system, for example：Source node as shown in Figure 1.
Wherein, above-mentioned controlling stream can be the controlling stream of source node receiving user's input.Here controlling stream can With the controlling stream described in the embodiment with reference to shown in Fig. 1-4, explanation is not repeated herein.
Updating block 142, for according to being currently needed for the topological structure for updating described in described by the controlling stream Update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for updating with described Topological structure coupling data flow circulation path.
After source node receives above-mentioned controlling stream, it is possible to which which is decomposed, and according to working as that decomposition is obtained The front topological structure update data stream circulation table for needing to update.In addition, source node can be stored or buffer update Data flow circulation table afterwards.It is, of course, also possible to store or cache above-mentioned controlling stream.
First transmitting element 143, for the controlling stream to be sent to the management section of the real time computation system Point, so that the management node sends the controlling stream to the process for carrying out Business Processing to data flow Node, so that the node that processes is according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；
Can just realize controlling stream being sent to node is processed by management node by the unit, process node again The data flow circulation table stored by which is updated.
Second transmitting element 144, for receiving data stream, and the data flow is updated according to the source node The data flow circulation path that includes of data flow circulation table send to the process node, by the process node Business Processing, and the number included according to the data flow circulation table of the process node updates is carried out to the data flow The data flow after the Business Processing is sent according to stream circulation path.
In the present embodiment, data that above-mentioned data flow can be transmitted with the data of user input or collecting device etc. Real time computation system is currently needed for the data for calculating.
As 142 update data stream of updating block circulates table, after such second transmitting element 144 is according to renewal Data flow circulation table send data flow, it is possible to realize source node be according to renewal after topological structure send Data flow, to realize the topological structure of dynamic adjustment real time computation system.
In the present embodiment, as shown in figure 15, said apparatus can also include：
3rd transmitting element 145, for sending the data flow circulation table of the renewal to real-time calculating be The shared memory node of system；
Recovery unit 146, for when the source node carries out fault recovery, obtaining from the shared memory node The data flow circulation table of the renewal is taken, and sending out for data flow is carried out according to the data flow circulation table of the renewal Send.
In the embodiment, it is possible to achieve source node after failure, can be directly from altogether when being recovered Enjoy memory node and the data flow circulation table that updating block 142 updates is obtained, going out such that it is able to realize source node After existing fault, the data flow circulation table can be read and is initialized from shared memory node, that is, is completed dynamic The HA mechanism of state adjustment topological structure.
In the present embodiment, as shown in figure 16, described device can also include：
Receiving unit 147, for receiving data flow described in the process node updates for processing node feeding back The renewal result of circulation table；
4th transmitting element 148, for when the renewal result represents renewal failure, to the management node The controlling stream is sent, so that the management node sends the controlling stream to the process node, by described Process node the data are updated according to the topological structure for being currently needed for described in described by the controlling stream updating Stream circulation table.
In the embodiment, each renewal knot for processing the current update data stream circulation table of node can be obtained in time Really.When some or multiple process node updates fail when, source node can trigger management node to renewal The process node of failure sends controlling stream, so as to update the process node of failure, update data stream circulates table again.
In the present embodiment, the source node of real time computation system is obtained for adjusting opening up for the real time computation system The controlling stream of structure is flutterred, wherein, the controlling stream is currently needed for updating for describing the real time computation system Topological structure；The source node is according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；The controlling stream is sent to the reality by the source node When computing system management node so that the management node sends the controlling stream to for data flow The process node of Business Processing is carried out, so that the process node is according to working as described in described by the controlling stream The front topological structure update data stream circulation table for needing to update, wherein, the data flow circulation table bag of the renewal Include the data flow circulation path mated with the topological structure for being currently needed for updating；The source node receives number According to stream, and the data flow circulation that the data flow circulation table that the data flow is updated according to the source node includes Path is sent to the process node, is carried out Business Processing by the process node to the data flow, and is pressed The data flow circulation path included according to the data flow circulation table of the process node updates is sent after the Business Processing Data flow.Can the dynamic topological structure for adjusting real time computation system.
Refer to the structure that Figure 17, Figure 17 are another kind data stream processing devices provided in an embodiment of the present invention to show It is intended to, the device is applied to the process node of real time computation system, as shown in figure 17, including：Processor 171, Network interface 172, memory 173 and communication bus 174, wherein, the communication bus 174 is used for realizing Connection communication between the processor 171, network interface 172 and memory 173, the processor 171 are held The program stored in the row memory 173 is used for realizing following methods：
Receive the real time computation system management node send for adjusting opening up for the real time computation system The controlling stream of structure is flutterred, wherein, the controlling stream is currently needed for updating for describing the real time computation system Topological structure；
According to the topological structure update data stream circulation for being currently needed for described in described by the controlling stream updating Table, wherein, the data flow circulation table of the renewal includes to mate with the topological structure for being currently needed for updating Data flow circulation path；
When processing node and receiving data flow, carry out Business Processing to the data flow, and according to described more The data flow circulation path that new data flow circulation table includes sends the data flow after the Business Processing.
In the present embodiment, processor 171 can also carry out following program：
The data flow circulation table of the renewal is sent to the shared memory node of the real time computation system；
When the process node carries out fault recovery, the number of the renewal is obtained from the shared memory node According to stream circulation table, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
In the present embodiment, processor 171 can also carry out following program：
Feed back, to the source node of the real time computation system, the renewal result that the data flow circulation table updates, with When making the renewal representation that failure is updated, the control is sent from the source node to the management node Stream；
When the renewal result represents renewal failure, the process node receives what the management node sent The controlling stream, and number is updated according to the topological structure for being currently needed for described in described by the controlling stream updating According to stream circulation table.
In the present embodiment, processor 171 can also carry out following program：
Feed back, to the output node of the real time computation system, the renewal result that the data flow circulation table updates, Carried out by the output node renewal result for processing node feeding back all of to the real time computation system Collect, and export summarized results.
In the present embodiment, the management node for processing the node reception real time computation system of real time computation system The controlling stream for adjusting the topological structure of the real time computation system for sending, wherein, the controlling stream is used In the topological structure that the description real time computation system is currently needed for updating；The process node is according to the control The described topological structure update data stream circulation table for being currently needed for updating of system stream, wherein, described more New data flow circulation table includes the data flow logical circulation road that mates with the topological structure for being currently needed for updating Footpath；When the process node receives data flow, the process node is carried out at business to the data flow Reason, and the data flow circulation path included according to the data flow circulation table of the renewal sends the Business Processing Data flow afterwards.So can realize by way of real-time update data flow circulation table updating in real time and calculate system The process logic of system, you can to dynamically adjust the topological structure of real time computation system.
Refer to the structure that Figure 18, Figure 18 are another kind data stream processing devices provided in an embodiment of the present invention to show It is intended to, the device is applied to the management node of real time computation system, as shown in figure 18, including：Processor 181, Network interface 182, memory 183 and communication bus 184, wherein, the communication bus 184 is used for realizing Connection communication between the processor 181, network interface 182 and memory 183, the processor 181 are held The program stored in the row memory 183 is used for realizing following methods：
Receive the topology for adjusting the real time computation system that the source node of the real time computation system sends The controlling stream of structure, wherein, the controlling stream is currently needed for renewal for describing the real time computation system Topological structure；
The controlling stream is sent to the process node for carrying out Business Processing to data flow, so that the place Reason node is according to the topological structure update data stream circulation for being currently needed for described in described by the controlling stream updating Table, wherein, the data flow circulation table of the renewal includes to mate with the topological structure for being currently needed for updating Data flow circulation path；When the process node receives data flow, by the process node to described Data flow carries out Business Processing, and the data flow circulation path included according to the data flow circulation table of the renewal Send the data flow after the Business Processing.
In the present embodiment, processor 181 can also carry out following program：
Set up and keep being connected with each the described process node in the real time computation system；
What processor 181 was executed sends the controlling stream to the process for carrying out Business Processing to data flow The program of node, can include：
The controlling stream is sent to each institute of the real time computation system by the management node with broadcast mode State process node.
In the present embodiment, the management node of real time computation system receives the source node of the real time computation system and sends out The controlling stream for adjusting the topological structure of the real time computation system that send, wherein, the controlling stream is used for The topological structure that the real time computation system is currently needed for updating is described；The management node is by the controlling stream Send to the process node for carrying out Business Processing to data flow, so that the process node is according to the control The described topological structure update data stream circulation table for being currently needed for updating of system stream, wherein, described more New data flow circulation table includes the data flow logical circulation road that mates with the topological structure for being currently needed for updating Footpath；When the process node receives data flow, business is carried out to the data flow by the process node Process, and the data flow circulation path included according to the data flow circulation table of the renewal is sent at the business Data flow after reason.The topological structure of dynamic adjustment real time computation system can so be realized.
Refer to the structure that Figure 19, Figure 19 are another kind data stream processing devices provided in an embodiment of the present invention to show It is intended to, the device is applied to the source node of real time computation system, as shown in figure 19, including：Processor 191, Network interface 192, memory 193 and communication bus 194, wherein, the communication bus 194 is used for realizing Connection communication between the processor 191, network interface 192 and memory 193, the processor 191 are held The program stored in the row memory 193 is used for realizing following methods：
The controlling stream for adjusting the topological structure of the real time computation system is obtained, wherein, the controlling stream It is currently needed for, for describing the real time computation system, the topological structure for updating；
According to the topological structure update data stream circulation for being currently needed for described in described by the controlling stream updating Table, wherein, the data flow circulation table of the renewal includes to mate with the topological structure for being currently needed for updating Data flow circulation path；
The controlling stream is sent to the management node of the real time computation system, so that the management node will The controlling stream is sent to the process node for carrying out Business Processing to data flow, so that the process node According to the topological structure update data stream circulation table for being currently needed for described in described by the controlling stream updating, its In, the data flow circulation table of the renewal includes the data that mates with the topological structure for being currently needed for updating Stream circulation path；
Receiving data stream, and data flow circulation table that the data flow is updated according to the source node includes Data flow circulation path is sent to the process node, carries out business by the process node to the data flow Process, and the data flow circulation path transmission included according to the data flow circulation table of the process node updates is described Data flow after Business Processing.
In the present embodiment, the program that processor 191 is executed can also include：
The data flow circulation table of the renewal is sent to the shared memory node of the real time computation system；
When the source node carries out fault recovery, the data of the renewal are obtained from the shared memory node Stream circulation table, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
In the present embodiment, the program that processor 191 is executed can also include：
Receive the renewal knot of data flow circulation table described in the process node updates for processing node feeding back Really；
When the renewal result represents renewal failure, the controlling stream is sent to the management node, so that The management node sends the controlling stream to the process node, by the process node according to the control The described topological structure for being currently needed for updating of stream updates the data flow circulation table.
In the present embodiment, the source node of real time computation system is obtained for adjusting opening up for the real time computation system The controlling stream of structure is flutterred, wherein, the controlling stream is currently needed for updating for describing the real time computation system Topological structure；The source node is according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；The controlling stream is sent to the reality by the source node When computing system management node so that the management node sends the controlling stream to for data flow The process node of Business Processing is carried out, so that the process node is according to working as described in described by the controlling stream The front topological structure update data stream circulation table for needing to update, wherein, the data flow circulation table bag of the renewal Include the data flow circulation path mated with the topological structure for being currently needed for updating；The source node receives number According to stream, and the data flow circulation that the data flow circulation table that the data flow is updated according to the source node includes Path is sent to the process node, is carried out Business Processing by the process node to the data flow, and is pressed The data flow circulation path included according to the data flow circulation table of the process node updates is sent after the Business Processing Data flow.Can the dynamic topological structure for adjusting real time computation system.
One of ordinary skill in the art will appreciate that all or part of flow process in above-described embodiment method is realized, Can be by computer program to complete to instruct the hardware of correlation, described program can be stored in a calculating In machine read/write memory medium, the program is upon execution, it may include as the flow process of the embodiment of above-mentioned each method. Wherein, described storage medium can for magnetic disc, CD, read-only memory (Read-Only Memory, ) or random access memory (Random Access Memory, abbreviation RAM) etc. ROM.
Above disclosed be only present pre-ferred embodiments, can not be limited with this certainly the present invention it Interest field, the equivalent variations that is therefore made according to the claims in the present invention, still belong to the scope covered by the present invention.

Claims (18)

1. a kind of data flow processing method, it is characterised in that include：
Real time computation system process that node receives that the management node of the real time computation system sends for adjusting The controlling stream of the topological structure of the whole real time computation system, wherein, the controlling stream is used for describing the reality When computing system be currently needed for update topological structure；
The node that processes is updated according to the topological structure for being currently needed for described in described by the controlling stream updating Data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for opening up for renewal with described Flutter the data flow circulation path of structure matching；
When the process node receives data flow, the process node is carried out at business to the data flow Reason, and the data flow circulation path included according to the data flow circulation table of the renewal sends the Business Processing Data flow afterwards.
2. the method for claim 1, it is characterised in that methods described also includes：
The data flow circulation table of the renewal is sent sharing to the real time computation system by the node that processes Memory node；
When the process node carries out fault recovery, the process node is obtained from the shared memory node The data flow circulation table of the renewal, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
3. method as claimed in claim 1 or 2, it is characterised in that methods described also includes：
The node that processes feeds back what the data flow circulation table updated to the source node of the real time computation system Result is updated, during so that the renewal representation updating failure, from the source node to the management node Send the controlling stream；
When the renewal result represents renewal failure, the process node receives what the management node sent The controlling stream, and number is updated according to the topological structure for being currently needed for described in described by the controlling stream updating According to stream circulation table.
4. method as claimed in claim 1 or 2, it is characterised in that methods described also includes：
The node that processes feeds back the data flow circulation table renewal to the output node of the real time computation system Renewal result, by the output node all of to the real time computation system process node feeding back described in Update result to be collected, and export summarized results.
5. a kind of data flow processing method, it is characterised in that include：
The management node of real time computation system receive that the source node of the real time computation system sends for adjusting The controlling stream of the topological structure of the real time computation system, wherein, the controlling stream is described real-time for describing Computing system is currently needed for the topological structure for updating；
The management node sends the controlling stream to the process section for carrying out Business Processing to data flow Point, so that the node that processes is according to the topological structure for being currently needed for described in described by the controlling stream updating Update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for updating with described Topological structure coupling data flow circulation path；When the process node receives data flow, by described Process node carries out Business Processing to the data flow, and include according to the data flow circulation table of the renewal Data flow circulation path sends the data flow after the Business Processing.
6. method as claimed in claim 5, it is characterised in that methods described also includes：
The management node is set up and the company of holding with each the described process node in the real time computation system Connect；
The management node sends the controlling stream to the process section for carrying out Business Processing to data flow Point, including：
The controlling stream is sent to each institute of the real time computation system by the management node with broadcast mode State process node.
7. a kind of data flow processing method, it is characterised in that include：
The source node of real time computation system obtains the control for adjusting the topological structure of the real time computation system Stream, wherein, the controlling stream is used for describing the topological structure that the real time computation system is currently needed for updating；
The source node updates number according to the topological structure for being currently needed for described in described by the controlling stream updating According to stream circulation table, wherein, the data flow circulation table of the renewal includes and the topology for being currently needed for and updating The data flow circulation path of structure matching；
The controlling stream is sent to the source node management node of the real time computation system, so that described Management node sends the controlling stream to the process node for carrying out Business Processing to data flow, so that institute State node is processed according to the topological structure update data stream for being currently needed for described in described by the controlling stream updating Circulation table, wherein, the data flow circulation table of the renewal includes and the topological structure for being currently needed for and updating The data flow circulation path of coupling；
The source node receiving data stream, and the data flow stream that the data flow is updated according to the source node The data flow circulation path that logical table includes is sent to the process node, by the process node to the data Stream carries out Business Processing, and the data flow logical circulation road included according to the data flow circulation table of the process node updates Footpath sends the data flow after the Business Processing.
8. method as claimed in claim 7, it is characterised in that methods described also includes：
The source node sends the data flow circulation table of the renewal to the shared of the real time computation system and deposits Storage node；
When the source node carries out fault recovery, the source node obtains described from the shared memory node The data flow circulation table of renewal, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
9. method as claimed in claim 7 or 8, it is characterised in that methods described also includes：
The source node receives data flow circulation table described in the process node updates for processing node feeding back Renewal result；
When the renewal result represents renewal failure, the source node sends the control to the management node System stream, so that the management node sends the controlling stream to the process node, is pressed by the process node The data flow circulation table is updated according to the topological structure for being currently needed for described in described by the controlling stream updating.
10. a kind of data stream processing device, described device are applied to the process node of real time computation system, its It is characterised by, including：Receiving unit, the first updating block and the first transmitting element, wherein：
The receiving unit, for receive the real time computation system management node send for adjusting State the controlling stream of the topological structure of real time computation system, wherein, the controlling stream by describe described real-time based on Calculation system is currently needed for the topological structure for updating；
First updating block, for according to opening up for being currently needed for described in described by the controlling stream updating Topology update data flow circulation table is flutterred, wherein, the data flow circulation table of the renewal includes and the current need The data flow circulation path of topological structure coupling to be updated；
First transmitting element, for when the process node receives data flow, to the data flow Business Processing is carried out, and the data flow circulation path included according to the data flow circulation table of the renewal sends institute State the data flow after Business Processing.
11. devices as claimed in claim 10, it is characterised in that described device also includes：
Second transmitting element, for sending the data flow circulation table of the renewal to the real time computation system Shared memory node；
Recovery unit, for when the process node carries out fault recovery, obtaining from the shared memory node The data flow circulation table of the renewal is taken, and sending out for data flow is carried out according to the data flow circulation table of the renewal Send.
12. devices as described in claim 10 or 11, it is characterised in that described device also includes：
First feedback unit, feeds back the data flow circulation table for the source node to the real time computation system The renewal result of renewal, during so that the renewal representation updating failure, from the source node to the pipe Reason node sends the controlling stream；
Second updating block, for when described renewal result represent first updating block update failure when, The controlling stream of management node transmission is received, and according to the described current need described by the controlling stream Topological structure update data stream circulation table to be updated.
13. devices as described in claim 10 or 11, it is characterised in that described device also includes：
Second feedback unit, feeds back the data flow circulation for the output node to the real time computation system The renewal result that table updates, by the output node to all of process node feeding back of the real time computation system The renewal result collected, and export summarized results.
14. a kind of data stream processing devices, it is characterised in that include：Receiving unit and transmitting element, its In：
The receiving unit, it is described real-time for adjusting that the source node for receiving real time computation system sends The controlling stream of the topological structure of computing system, wherein, the controlling stream is used for describing the real time computation system It is currently needed for the topological structure for updating；
The transmitting element, for sending the controlling stream to the place for carrying out Business Processing to data flow Reason node, so that the node that processes is according to the topology for being currently needed for described in described by the controlling stream updating Topology update data flow circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for described The data flow circulation path of the topological structure coupling of renewal；When the process node receives data flow, by The process node carries out Business Processing, and the data flow circulation table bag according to the renewal to the data flow The data flow circulation path for including sends the data flow after the Business Processing.
15. devices as claimed in claim 14, it is characterised in that described device also includes：
Holding unit, for setting up and the company of holding with each the described process node in the real time computation system Connect；
The transmitting element is used for sending the controlling stream with broadcast mode each to the real time computation system The individual process node.
A kind of 16. data stream processing devices, described device are applied to the source node of real time computation system, and which is special Levy and be, including：Acquiring unit, updating block, the first transmitting element and the second transmitting element, wherein：
The acquiring unit, for obtaining the controlling stream for adjusting the topological structure of the real time computation system, Wherein, the controlling stream is used for describing the topological structure that the real time computation system is currently needed for updating；
The updating block, for according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；
First transmitting element, for the controlling stream to be sent to the management section of the real time computation system Point, so that the management node sends the controlling stream to the process for carrying out Business Processing to data flow Node, so that the node that processes is according to the topology knot for being currently needed for described in described by the controlling stream updating Structure update data stream circulation table, wherein, the data flow circulation table of the renewal includes to be currently needed for more with described The data flow circulation path of new topological structure coupling；
Second transmitting element, for receiving data stream, and by the data flow according to the source node more The data flow circulation path that new data flow circulation table includes is sent to the process node, is saved by described process Put carries out Business Processing to the data flow, and include according to the data flow circulation table of the process node updates Data flow circulation path sends the data flow after the Business Processing.
17. devices as claimed in claim 16, it is characterised in that described device also includes：
3rd transmitting element, for sending the data flow circulation table of the renewal to the real time computation system Shared memory node；
Recovery unit, for when the source node carries out fault recovery, obtaining from the shared memory node The data flow circulation table of the renewal, and the transmission of data flow is carried out according to the data flow circulation table of the renewal.
18. devices as described in claim 16 or 17, it is characterised in that described device also includes：
Receiving unit, for receiving data flow stream described in the process node updates for processing node feeding back The renewal result of logical table；
4th transmitting element, for when the renewal result represents renewal failure, sending out to the management node The controlling stream is sent, so that the management node sends the controlling stream to the process node, by described Reason node updates the data flow according to the topological structure for being currently needed for described in described by the controlling stream updating Circulation table.
CN201510534143.5A 2015-08-27 2015-08-27 Data stream processing method and device Active CN106487694B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201510534143.5A CN106487694B (en) 2015-08-27 2015-08-27 Data stream processing method and device
PCT/CN2016/093588 WO2017032212A1 (en) 2015-08-27 2016-08-05 Data stream processing method and apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510534143.5A CN106487694B (en) 2015-08-27 2015-08-27 Data stream processing method and device

Publications (2)

Publication Number Publication Date
CN106487694A true CN106487694A (en) 2017-03-08
CN106487694B CN106487694B (en) 2020-03-27

Family

ID=58099527

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510534143.5A Active CN106487694B (en) 2015-08-27 2015-08-27 Data stream processing method and device

Country Status (2)

Country Link
CN (1) CN106487694B (en)
WO (1) WO2017032212A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110019369A (en) * 2017-12-31 2019-07-16 中国移动通信集团福建有限公司 Method, apparatus, equipment and the medium of shared data stream process topology

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11095522B2 (en) * 2019-08-21 2021-08-17 Microsoft Technology Licensing, Llc Dynamic scaling for data processing streaming system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130290554A1 (en) * 2012-04-26 2013-10-31 Qiming Chen Open station as a stream analysis operator container
CN103716182A (en) * 2013-12-12 2014-04-09 中国科学院信息工程研究所 Failure detection and fault tolerance method and failure detection and fault tolerance system for real-time cloud platform
CN104008007A (en) * 2014-06-12 2014-08-27 深圳先进技术研究院 Interoperability data processing system and method based on streaming calculation and batch processing calculation
CN104090886A (en) * 2013-12-09 2014-10-08 深圳市腾讯计算机系统有限公司 Method and device for constructing real-time portrayal of user
WO2014194251A2 (en) * 2013-05-30 2014-12-04 Vaibhav Nivargi Apparatus and method for collaboratively analyzing data from disparate data sources

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104683445A (en) * 2015-01-26 2015-06-03 北京邮电大学 Distributed real-time data fusion system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130290554A1 (en) * 2012-04-26 2013-10-31 Qiming Chen Open station as a stream analysis operator container
WO2014194251A2 (en) * 2013-05-30 2014-12-04 Vaibhav Nivargi Apparatus and method for collaboratively analyzing data from disparate data sources
CN104090886A (en) * 2013-12-09 2014-10-08 深圳市腾讯计算机系统有限公司 Method and device for constructing real-time portrayal of user
CN103716182A (en) * 2013-12-12 2014-04-09 中国科学院信息工程研究所 Failure detection and fault tolerance method and failure detection and fault tolerance system for real-time cloud platform
CN104008007A (en) * 2014-06-12 2014-08-27 深圳先进技术研究院 Interoperability data processing system and method based on streaming calculation and batch processing calculation

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110019369A (en) * 2017-12-31 2019-07-16 中国移动通信集团福建有限公司 Method, apparatus, equipment and the medium of shared data stream process topology
CN110019369B (en) * 2017-12-31 2022-06-07 中国移动通信集团福建有限公司 Method, apparatus, device and medium for sharing data stream processing topology

Also Published As

Publication number Publication date
WO2017032212A1 (en) 2017-03-02
CN106487694B (en) 2020-03-27

Similar Documents

Publication Publication Date Title
Yao et al. AI routers & network mind: A hybrid machine learning paradigm for packet routing
US11272396B2 (en) Frame aggregation method, network setting frame sending method, and device
CN110009201B (en) Electric power data link system and method based on block chain technology
CN111565113B (en) Flexible Ethernet network topology abstraction method and system for SDN controller
WO2023082596A1 (en) Edge computing gateway-based model distribution application design method and system
CN105607954A (en) Stateful container online migration method and apparatus
CN108881158A (en) Data interaction system and method
US20230042747A1 (en) Message Processing Method and Device, Storage Medium, and Electronic Device
CN105052076B (en) Network element management system and network element management method based on cloud computing
CN107959715A (en) Remote terminal information recognition software system and recognition methods based on wireless telecommunications
CN104038957A (en) 4G base station operation maintenance information analysis process method based on integration structure
CN105763297A (en) Cloud computing system-based remote data optimized transmission method and device
Rath et al. MAQ system development in mobile ad-hoc networks using mobile agents
Wette et al. DCT2Gen: A traffic generator for data centers
CN101741745A (en) Method and system for identifying application traffic of peer-to-peer network
CN114945032A (en) Electric power internet of things terminal data access system, method, device, equipment and medium
CN105142035A (en) Video stream slice transmitting system and method
CN106487694A (en) A kind of data flow processing method and device
CN101753456B (en) Method and system for detecting flow of peer-to-peer network
Porter et al. DeSARM: A Decentralized Mechanism for Discovering Software Architecture Models at Runtime in Distributed Systems.
CN104038364B (en) The fault-tolerance approach of distributed stream treatment system, node and system
CN107493254B (en) TCP message forwarding method, device and system
CN112202896A (en) Edge calculation method, frame, terminal and storage medium
CN104184729A (en) Message processing method and device
CN107707598A (en) The method and Internet of Things communication equipment of a kind of remote control Internet of Things communication equipment

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant