CN104580322A - Distributed data stream processing method and device - Google Patents
Distributed data stream processing method and device Download PDFInfo
- Publication number
- CN104580322A CN104580322A CN201310513394.6A CN201310513394A CN104580322A CN 104580322 A CN104580322 A CN 104580322A CN 201310513394 A CN201310513394 A CN 201310513394A CN 104580322 A CN104580322 A CN 104580322A
- Authority
- CN
- China
- Prior art keywords
- node
- state information
- section point
- data flow
- working node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the field of data processing, particularly to a distributed data stream processing method and device, and aims to solve the problem that load balancing processing cannot be performed on data streams with the same key value by the distributed stream processing technique. The method disclosed by the embodiment of the invention comprises the following steps: according to key values of data streams which need to be distributed to the next working node, a first node determines a second node to be used as the next working node for processing the data streams corresponding to the key values; after the second node is determined to satisfy set load transfer conditions, the data streams which need to be distributed to the next working node are transferred from the second node to the third node to be processed, and the second node and the third node are indicated to realize synchronization of state information of the data streams corresponding to the key values. Through the use of the method, after the downstream second node is determined to satisfy the set load transfer conditions, the first node can transfer the data streams corresponding to the key values which are responsible for being processed by the second node to the third node of which the current cumulated loads are smaller to be processed, so that the data streams with the same key value can be treated by load balancing processing.
Description
Technical field
The present invention relates to data processing field, particularly relate to a kind of distributed data method for stream processing and device.
Background technology
Research On The Key Technology In Data Stream is widely used in various fields, such as Financial Management, network monitoring, communication data management, Web application, sensor network data process etc.Have a typical feature in these application: Data Stream Processing amount is very big, have quite high sudden, when the speed that data arrive exceeds the disposal ability of system, system there will be overload and hydraulic performance decline.So load management all becomes the focus and emphasis studied in Data Stream Processing.
Research On The Key Technology In Data Stream comprises centralized Research On The Key Technology In Data Stream and distributed traffic treatment technology; In centralized Research On The Key Technology In Data Stream, when detecting that overload occurs system, optionally can lose some data tuple to ensure the normal operation of system, obviously, adopting is sacrifice data tuple for cost in this way, larger on systematic function impact.Because flow data source and application itself exist distributed feature, at present, distributed stream treatment technology becomes the focus of stream treatment research, adopts distributed stream treatment technology, can by load Distribution to each processing node, the load balance in keeping system between each node.
In distributed stream treatment technology, for stateless object, data tuple is assigned on corresponding processing node by the main polling mode that adopts, and belongs to disposable operation to the processing procedure of stateless object, can not preserve any state information, processing procedure is fairly simple; For there being status object, usually the distribution of flows with same key assignments (Key) is processed to same node.
In real network, the distribution of flow has very large obliquity, the data flow of corresponding same key assignments likely can contribute sizable flow, this also carries out load balance process with regard to needing to the data flow with same key assignments, and above-mentioned distributed stream treatment technology obviously cannot carry out load balance process for this situation.
Summary of the invention
The embodiment of the present invention provides a kind of distributed data method for stream processing and device, cannot carry out the problem of load balance process in order to solve distributed stream treatment technology to the data flow with same key assignments.
First aspect, provides a kind of distributed data method for stream processing, comprising:
First node distributes to the key assignments of the data flow of down hop working node as required, determines that Section Point is the down hop working node of the data flow that the described key-value pair of process is answered;
Described first node is after determining that described Section Point meets the load migration condition of setting, need the data flow distributing to down hop working node to migrate to the 3rd node from described Section Point to process by described, and indicate the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes;
Wherein, described Section Point and the 3rd node belong to the down hop working node set of described first node, and the accumulative load of described 3rd node is less than the accumulative load of described Section Point.
In conjunction with first aspect, in the implementation that the first is possible, described 3rd node is the working node in the down hop working node set of current described first node with minimum accumulative load.
In conjunction with first aspect, or in the first possible implementation of first aspect, in the implementation that the second is possible, the load migration condition of described setting comprises:
The accumulative load of described Section Point exceedes setting threshold; And/or,
Ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold.
In conjunction with the implementation that the second of first aspect is possible, in the implementation that the third is possible, the load migration condition of described setting also comprises:
Current time and described first node the last time carry out the time interval of the time of data flow migration process and are not less than setting threshold.
In conjunction with first aspect, or the first to three of first aspect kind of any one possible implementation, in the 4th kind of possible implementation, described first node is according to the accumulative load W of any one down hop working node of described first node in following formula determination setting-up time section:
W＝c×W′+(1-c)×y；
Wherein, y is the load that described first node distributes to any one down hop working node described in described setting-up time section, W ' is the accumulative load of any one down hop working node described at the end of the previous time period of described setting-up time section, c is constant, and 0 < c < 1.
In conjunction with first aspect, or any one possible implementation of first to fourth kind of first aspect, in the 5th kind of possible implementation, described first node indicates the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes, comprising:
Described first node sends to described Section Point the state information comprising described key assignments and to move out instruction, and, send to described 3rd node the state information comprising described key assignments and to move into instruction;
Wherein, described state information the first state information that instruction is used to indicate the data flow that described key-value pair that this locality generates by described Section Point is answered of moving out sends to the telegon managing each working node; Described state information is moved into indicate and is used to indicate the first state information that described 3rd node obtains the data flow that described key-value pair is answered from the telegon of each working node of management, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
In conjunction with first aspect, or the first to five of first aspect kind of any one possible implementation, in the 6th kind of possible implementation, described first node, before determining that described Section Point meets the load migration condition of setting, also comprises:
Described first node, after the information of the described Section Point overload of the instruction receiving the telegon transmission managing each working node, adjusts the described load migration condition of setting.
In conjunction with first aspect, or the first to six of first aspect kind of any one possible implementation, in the 7th kind of possible implementation, described method also comprises:
First node distributes to the key assignments of the data flow of down hop working node as required, determine current there is not the down hop working node of this key-value pair of process data flow of answering time, the described distribution of flows distributing to down hop working node that needs is processed to the working node in the down hop working node set of current described first node with minimum accumulative load.
Second aspect, provides a kind of distributed data method for stream processing, comprising:
Section Point receives the state information comprising setting key value that first node sends and to move out instruction;
Described Section Point to be moved out instruction according to described state information, determines the first state information of the data flow that the local described key-value pair generated is answered, and described first state information determined is sent to the telegon managing each working node.
The third aspect, provides a kind of distributed data method for stream processing, comprising:
3rd node receives the state information comprising setting key value that first node sends and to move into instruction;
Described 3rd node to be moved into instruction according to described state information, from the telegon of each working node of management, obtain the first state information that described Section Point is sent to the data flow that the described key-value pair in telegon is answered, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
Fourth aspect, provide a kind of distributed data current processing device, this device belongs to first node, comprising:
Determination module, for distributing to the key assignments of the data flow of the down hop working node of described first node as required, determines that Section Point is the down hop working node of the data flow that the described key-value pair of process is answered;
Transferring module, for after determining that described Section Point meets the load migration condition of setting, need the data flow distributing to down hop working node to migrate to the 3rd node from described Section Point to process by described, and indicate the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes;
Wherein, described Section Point and the 3rd node belong to the down hop working node set of described first node, and the accumulative load of described 3rd node is less than the accumulative load of described Section Point.
In conjunction with fourth aspect, in the implementation that the first is possible, described 3rd node is the working node in the down hop working node set of current described first node with minimum accumulative load.
In conjunction with fourth aspect, or the first possible implementation of fourth aspect, in the implementation that the second is possible, the load migration condition of described setting comprises:
The accumulative load of described Section Point exceedes setting threshold; And/or,
Ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold.
In conjunction with the implementation that the second of fourth aspect is possible, in the implementation that the third is possible, the load migration condition of described setting also comprises:
Current time and described device the last time carry out the time interval of the time of data flow migration process and are not less than setting threshold.
In conjunction with fourth aspect, or the first to three of fourth aspect kind of any one possible implementation, in the 4th kind of possible implementation, described transferring module is specifically for the accumulative load W according to any one down hop working node of described first node in following formula determination setting-up time section:
W＝c×W′+(1-c)×y；
Wherein, y is the load that described device distributes to any one down hop working node described in described setting-up time section, W ' is the accumulative load of any one down hop working node described at the end of the previous time period of described setting-up time section, c is constant, and 0 < c < 1.
In conjunction with fourth aspect, or any one possible implementation of first to fourth kind of fourth aspect, in the 5th kind of possible implementation, described transferring module specifically for, send to described Section Point the state information comprising described key assignments to move out instruction, and, send to described 3rd node the state information comprising described key assignments and to move into instruction;
Wherein, described state information the first state information that instruction is used to indicate the data flow that described key-value pair that this locality generates by described Section Point is answered of moving out sends to the telegon managing each working node; Described state information is moved into indicate and is used to indicate the first state information that described 3rd node obtains the data flow that described key-value pair is answered from the telegon of each working node of management, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
In conjunction with fourth aspect, or the first to five of fourth aspect kind of any one possible implementation, in the 6th kind of possible implementation, described transferring module also for, before determining that described Section Point meets the load migration condition of setting, the information of the described Section Point overload of the instruction that the telegon managing each working node if receive sends, then adjust the described load migration condition of setting.
In conjunction with fourth aspect, or the first to six of fourth aspect kind of any one possible implementation, in the 7th kind of possible implementation, described transferring module is also for the key assignments of distributing to the data flow of down hop working node as required, determine current there is not the down hop working node of this key-value pair of process data flow of answering time, the described distribution of flows distributing to down hop working node that needs is processed to the working node in the down hop working node set of current described first node with minimum accumulative load.
5th aspect, provides a kind of distributed data current processing device, comprising:
Receiver module, the state information comprising setting key value sent for receiving first node is moved out instruction, and instruction of described state information being moved out transfers to sending module;
Sending module, the state information for receiving according to described receiver module is moved out instruction, determines the first state information of the data flow that the local described key-value pair generated is answered, and described first state information determined is sent to the telegon managing each working node.
6th aspect, provides a kind of distributed data current processing device, comprising:
Receiver module, the state information comprising setting key value sent for receiving first node is moved into instruction, and instruction of described state information being moved into transfers to acquisition module;
Acquisition module, state information for receiving according to described receiver module is moved into instruction, from the telegon of each working node of management, obtain the first state information that described Section Point is sent to the data flow that the described key-value pair in telegon is answered, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
Adopt the distributed data method for stream processing that above-mentioned first aspect provides, first node can after determining that the Section Point in downstream meets the load migration condition of setting, the data flow migration that key-value pair Section Point being responsible for process is answered processes to the 3rd node that current accumulative load is less, and indicate Section Point and the 3rd nodes for state synchronizing information, thus load balance process can be carried out for the data flow of same key assignments.
Accompanying drawing explanation
Fig. 1 is the method flow diagram that the embodiment of the present invention one carries out distributed traffic process;
Fig. 2 is the method flow diagram that the embodiment of the present invention two carries out distributed traffic process;
Fig. 3 is the method flow diagram that the embodiment of the present invention three carries out distributed traffic process;
Fig. 4 is the functional unit distribution schematic diagram of each working node and each working node in the distributed system of the embodiment of the present invention;
Fig. 5 is the method flow diagram that in the embodiment of the present invention, first node carries out load migration;
Fig. 6 carries out the synchronous schematic diagram of state information in the embodiment of the present invention between working node;
Fig. 7 carries out the synchronous method flow diagram of state information in the embodiment of the present invention corresponding with Fig. 6;
In the distributed system of Fig. 8 embodiment of the present invention, working node carries out the schematic diagram of overload feedback;
Fig. 9 is the method flow diagram that in the distributed system of the embodiment of the present invention corresponding with Fig. 8, working node carries out overload feedback;
The distributed data current processing device schematic diagram that Figure 10 provides for the embodiment of the present invention one;
The distributed data current processing device schematic diagram that Figure 11 provides for the embodiment of the present invention two;
The distributed data current processing device schematic diagram that Figure 12 provides for the embodiment of the present invention three;
The distributed data current processing device schematic diagram that Figure 13 provides for the embodiment of the present invention four;
The distributed data current processing device schematic diagram that Figure 14 provides for the embodiment of the present invention five;
The distributed data current processing device schematic diagram that Figure 15 provides for the embodiment of the present invention six.
Embodiment
Below in conjunction with Figure of description, the embodiment of the present invention is described in further detail.
As shown in Figure 1, for the embodiment of the present invention one carries out the method flow diagram of distributed traffic process, comprise the following steps:
S101: first node distributes to the key assignments of the data flow of down hop working node as required, determines that Section Point is the down hop working node of the data flow that the described key-value pair of process is answered;
S102: described first node is after determining that described Section Point meets the load migration condition of setting, need the data flow distributing to down hop working node to migrate to the 3rd node from described Section Point to process by described, and indicate the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes;
Wherein, described Section Point and the 3rd node belong to the down hop working node set of described first node, and the accumulative load of described 3rd node is less than the accumulative load of described Section Point.
Adopt said method, first node can after determining that the Section Point in downstream meets the load migration condition of setting, the data flow migration that key-value pair Section Point being responsible for process is answered processes to the 3rd node that current accumulative load is less, and indicate Section Point and the 3rd nodes for state synchronizing information, thus load balance process can be carried out for the data flow of same key assignments.
Alternatively, described 3rd node is the working node in the down hop working node set of current described first node with minimum accumulative load.
Alternatively, described method also comprises:
First node distributes to the key assignments of the data flow of down hop working node as required, determine current there is not the down hop working node of this key-value pair of process data flow of answering time, the described distribution of flows distributing to down hop working node that needs is processed to the working node in the down hop working node set of current described first node with minimum accumulative load.
In specific implementation process, the data flow that first node distributes to down hop working node can be the local initial data stream generated, and also can be the data flow that other working node forwarded generates, first node distributes to the key assignments of the data flow of down hop working node as required, judge whether the data flow that this key-value pair is answered has been assigned to certain down hop working node, if determine, the data flow that this key-value pair is answered is assigned to Section Point, then judge whether Section Point has met load migration condition, if meet, the data flow migration of then being answered by this key-value pair processes to the 3rd node that the accumulative load of current accumulative duty ratio first node is little, here the 3rd node can be specifically the working node in all down hop working nodes of current first node with minimum accumulative load, if do not meet, then the distribution of flows continued this key-value pair is answered processes to Section Point, if the data flow that this key-value pair is answered also is not assigned to any one down hop working node, then process this distribution of flows to the current working node with minimum accumulative load, wherein, the data flow that each key-value pair is answered can be called a data items.
Alternatively, the load migration condition of described setting comprises one or more in following condition:
First condition: the accumulative load of described Section Point exceedes setting threshold;
Second condition: ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold;
Except this, load migration condition, except comprising in above-mentioned two conditions except one or two, can also comprise:
3rd condition: current time and described first node the last time carry out the time interval of the time of data flow migration process and be not less than setting threshold.
In the embodiment of the present invention, can when above first and/or second condition meet, the data flow of being answered by described key-value pair migrates to current 3rd node with minimum accumulative load from described Section Point and processes; In concrete enforcement, in order to take into account overhead, can when above first and/or second condition meet, check that current time and described first node the last time carry out the time interval of the time of data flow migration process and whether be less than setting threshold, if meet or exceed setting threshold, then the data flow of being answered by described key-value pair migrates to current 3rd node with minimum accumulative load from described Section Point and processes.
The accumulative load of above-mentioned Section Point and the 3rd node can calculate according to following formula:
Alternatively, described first node is according to the accumulative load W of any one down hop working node of described first node in following formula determination setting-up time section:
W＝c×W′+(1-c)×y；
Wherein, y is the load that described first node distributes to any one down hop working node described in described setting-up time section, W ' is the accumulative load of any one down hop working node described at the end of the previous time period of described setting-up time section, c is constant, and 0 < c < 1.
Such as, for above-mentioned Section Point, in above-mentioned formula, y is the load that first node distributes to Section Point in setting-up time section, and this load can comprise the data flow that a key-value pair is answered, and also can comprise the data flow that multiple key-value pair is answered; For wherein any one key assignments, also be any one data items, suppose that the load that this data items is corresponding when this setting-up time section starts is w ', in this setting-up time section, the load that this data items increases newly is v, then at the end of this setting-up time section, load corresponding for this data items is updated to w=c × w '+(1-c) × v.
Alternatively, described first node indicates the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes, comprising:
Described first node sends to described Section Point the state information comprising described key assignments and to move out instruction, and, send to described 3rd node the state information comprising described key assignments and to move into instruction;
Wherein, described state information the first state information that instruction is used to indicate the data flow that described key-value pair that this locality generates by described Section Point is answered of moving out sends to the telegon managing each working node; Described state information is moved into indicate and is used to indicate the first state information that described 3rd node obtains the data flow that described key-value pair is answered from the telegon of each working node of management, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
In specific implementation process, first node is in the data flow of being answered when the described key-value pair of pre-treatment by Section Point, and also i.e. data items, migrates to while the 3rd node carries out processing, need for this data items, synchronous first node and the 3rd internodal state information; Particularly, adopt the telegon of each working node of management as the synchronous terminal of state information between two nodes in the embodiment of the present invention, first state information of this data items stored before is sent to telegon by Section Point, this the first state information can be obtained from this telegon to make the 3rd node, 3rd node is after this first state information of acquisition, can by this first state information and the 3rd node newly-generated to should the second state information of data items merge, so complete the state information of the data items of migration synchronous.In concrete enforcement, first node can send move out instruction and state information of state information respectively to Section Point and the 3rd node and to move into instruction, this instruction can explicit instruction Section Point and the 3rd nodes for state synchronizing information, also implicit expression Section Point and the 3rd nodes for state synchronizing information can be indicated, such as, first node directly can notify that the first state information of data flow corresponding for setting key value is moved out by Section Point, and send to telegon, also can notify that data flow that described key-value pair has been answered by first node described in Section Point has moved to other working node from this Section Point and processed, Section Point is after receiving this notice, according to making an appointment between each working node, first state information of the data flow of being answered by this key-value pair that this locality generates is sent to telegon, correspondingly, first node directly can notify that the 3rd node is moved into from telegon the first state information of data flow corresponding to setting key value, and this first state information and newly-generated the second state information of the 3rd node are merged, also can notify that data flow that described in the 3rd node, described key-value pair has been answered by first node has moved to the 3rd node from other working node and processed, 3rd node is after receiving this notice, according to making an appointment between each working node, described first state information is obtained from telegon, and the second state information of the data flow of described first state information being answered with local this key-value pair generated merges,
Alternatively, described first node, before determining that described Section Point meets the load migration condition of setting, also comprises:
Described first node, after the information of the described Section Point overload of the instruction receiving the telegon transmission managing each working node, adjusts the described load migration condition of setting.
In specific implementation process, except except the loading condition of each early-stage work node monitors downstream working node, overall situation monitoring can also be carried out by the telegon managing each working node to each working node, particularly, the data receiver amount of each working node monitoring self, after the data receiver amount of self exceedes setting threshold, then think that own load transships, start overload feedback mechanism, self overload messages is fed back to telegon, overall scheduling is carried out by telegon, notify all or part of early-stage work node of this overload work node, these early-stage work nodes can adjust above-mentioned load migration condition according to these overload messages, such as, if first node is before determining that Section Point meets the load migration condition of setting, receive the information of the instruction Section Point overload of the telegon transmission managing each working node, then adjust the load migration condition of above-mentioned setting, load migration condition is met to make Section Point, particularly, the setting threshold of three conditions in above-mentioned load migration condition can be reduced, or the c value etc. adjusted in above-mentioned accumulative load computing formula, this overload work node is made to meet the load migration condition of setting.
Introduce the processing mode of lower above-mentioned Section Point and the 3rd node response first node below respectively, carry out the method flow of Data Stream Processing;
As shown in Figure 2, for the embodiment of the present invention two carries out the method flow diagram of distributed traffic process, comprising:
S201: Section Point receives the state information comprising setting key value that first node sends and to move out instruction;
S202: described Section Point to be moved out instruction according to described state information, determines the first state information of the data flow that the local described key-value pair generated is answered, and described first state information determined is sent to the telegon managing each working node;
Alternatively, described 3rd node is the working node in all down hop working nodes of current described first node with minimum accumulative load.
In specific implementation process, Section Point is moved out according to the state information that first node sends and is indicated the key assignments that comprises, the first state information of the data flow that described key-value pair is answered is extracted from the state information that this locality stores, and this first state information extracted is sent to the telegon managing each working node, 3rd node extracts this first state information from telegon, and this first state information and newly-generated the second state information of the 3rd node are merged, complete the merging of the state information for migration data project.
As shown in Figure 3, for the embodiment of the present invention three carries out the method flow diagram of distributed traffic process, comprising:
S301: the three node receives the state information comprising setting key value that first node sends and to move into instruction;
S302: described 3rd node to be moved into instruction according to described state information, from the telegon of each working node of management, obtain the first state information that described Section Point is sent to the data flow that the described key-value pair in telegon is answered, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges;
Alternatively, described 3rd node is the working node in all down hop working nodes of current described first node with minimum accumulative load.
Carry out the method for Data Stream Processing to further illustrate the embodiment of the present invention, below by some comparatively specific embodiment be described in detail;
As shown in Figure 4, be the functional unit distribution schematic diagram of working node each in the distributed system of the embodiment of the present invention and each working node; Distributed system in the embodiment of the present invention comprises a telegon (coordinator) and multiple working node, and wherein telegon is responsible for connecting each work node, and manages each working node; Each working node comprises multiple functional unit, particularly, comprise a container (container), a distributor (dispatcher), a gatherer (collector), and one or more analyzer (analyzer), wherein, distributor and analyzer, between analyzer and gatherer, have data channel, between container and other each functional unit, have control channel; The effect of each functional unit is respectively: distributor is responsible for receiving the outside data flow sent, the data flow received is decoded, and by decoded distribution of flows in different analyzers, the data flow wherein received can be the initial data stream that this working node produces, and also can be the data flow that other working node produces; Analyzer is responsible for the data execution analysis operation to receiving, and by the transfer of data after analysis to gatherer; The result gathering the output of each analyzer is responsible for by gatherer, and is responsible for the result exporting analyzer output, and gatherer also can produce new output stream, and is distributed in down hop working node; Container is in charge of distributor, analyzer and gatherer, is also the interface that this working node is connected with telegon simultaneously.
In order to ensure the reliability of telegon, in the embodiment of the present invention, telegon can be structured in a distributed collaboration work system of increasing income, as on Zookeeper cluster; Although telegon belongs to stateless object, may occur crashing or other fault, as long as Zookeeper cluster is normal, just can restart telegon fast, wherein, the high-performance communication storehouse that the transfer of data between working node can use to increase income, as ZeroMQ; Except this, telegon in the embodiment of the present invention also can adopt single-point framework, but, this mode may make telegon fault-tolerant ability reduce, at this moment, directly TCP(Transmission Control Protocol can be passed through, transmission control protocol between working node) or UDP(User Data Protocol, User Datagram Protoco (UDP)) carry out transfer of data.
As shown in Figure 5, for first node in the embodiment of the present invention carries out the method flow diagram of load migration, comprising:
S501: first node determines the key assignments needing the data flow distributing to down hop working node;
S502: first node, according to the key assignments determined, judges the current Section Point that whether there is the data flow that this key-value pair of process is answered, if exist, then enters step S503, otherwise enter step S507;
S503: first node, according to the load migration condition of setting, judges whether to need to carry out load migration, if so, then enters step S504, otherwise, enter step S505;
Described load migration condition comprises: the accumulative load of described Section Point exceedes setting threshold; Ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold; Current time and described first node the last time carry out the time interval of the time of data flow migration process and are not less than setting threshold; Wherein, the computing formula of accumulative load see the above-mentioned description about accompanying drawing 1, can repeat no more here.
S504: data flow migration to current 3rd node with minimum accumulative load that this key-value pair is answered by first node processes, and indicate Section Point and the 3rd node to carry out state information synchronously for this key assignments, and enter step S508;
S505: the distribution of flows that described key-value pair is answered by first node proceeds process to Section Point, and enters step S506;
S506: first node upgrades the accumulative load of the Section Point stored;
S507: the distribution of flows that described key-value pair is answered by first node processes to current 3rd node with minimum accumulative load, and enters step S508;
S508: first node upgrades the accumulative load of the 3rd node stored.
As shown in Figure 6, for carrying out the synchronous schematic diagram of state information between working node in the embodiment of the present invention, as shown in Figure 7, for carrying out the synchronous method flow diagram of state information in the embodiment of the present invention corresponding with Fig. 6, comprising:
S701: the gatherer of first node is determined to need data flow migration corresponding for the setting key value of Section Point process to process to the 3rd node, proceeds to step S702 respectively, and step S706;
S702: the gatherer of first node to be moved out instruction to the state information that the distributor of Section Point sends corresponding described key assignments;
S703: the distributor of Section Point finds the analyzer being responsible for the data flow that this key-value pair of process is answered, instruction of the described state information received being moved out is transmitted to this analyzer;
S704: the analyzer of Section Point extracts the first state information that the local described key-value pair stored is answered, and described first state information is sent to the container of described Section Point;
S705: described first state information is sent to telegon by the container of Section Point;
S706: the gatherer of first node to be moved into instruction to the state information that the distributor of the 3rd node sends corresponding described key assignments;
The distributor of the S707: the three node finds the analyzer being responsible for the data flow that this key-value pair of process is answered, and instruction of described state information being moved into is transmitted to this analyzer;
The analyzer of the S708: the three node sends the state information request of moving into of corresponding described key assignments to the container of the 3rd node;
In concrete enforcement, the analyzer of the 3rd node, after sending the state information request of moving into container, continues data streams, and the data flow of process comprises follow-up data stream that described key-value pair answers and the data flow that other key-value pair is answered;
The container of the S709: the three node obtains from telegon the first state information that described key-value pair answers;
The first state information that first state information of moving into sends to the analyzer processing the data flow that this key-value pair is answered, this analyzer to be answered by this key-value pair by the container of the S710: the three node merges with the second state information newly-generated on the 3rd node.
As shown in Figure 8, in the distributed system of the embodiment of the present invention, working node carries out the schematic diagram of overload feedback, as shown in Figure 9, for in the distributed system of the embodiment of the present invention corresponding with Fig. 8, working node carries out the method flow diagram of overload feedback, this embodiment can be used for the auxiliary above-mentioned load migration flow process corresponding with accompanying drawing 4, realize carrying out overall situation monitoring and adjustment to the load of working node each in system, comprising:
S901: the distributor of Section Point sends overload messages to the container of this Section Point;
In concrete enforcement, the distributor of working node A can judge whether overload according to the utilization rate of data cube computation between distributor and analyzer, such as, the data channel between distributor and analyzer can continue the time that takes when exceeding setting threshold, determine that working node A transships; Particularly, the distributor of working node A can measure the size of the Circular buffer (ring buffer) sending data to analyzer in real time, if all ring buffer are full within the setting-up time cycle, then determines that working node A transships;
S902: the information that the container of Section Point reports Section Point to transship to telegon;
S903: the message notice that described Section Point transships by telegon is to all or part of early-stage work node of Section Point, and wherein this all or part of early-stage work node comprises first node;
S904: after the container of first node obtains the information of described Section Point overload from telegon, the message notice of being transshipped by described Section Point is to the gatherer of first node;
S905: the gatherer adjustment load migration condition of first node, makes described Section Point meet the load migration condition of setting.
Based on same inventive concept, a kind of data stream processing device corresponding with data flow processing method is additionally provided in the embodiment of the present invention, the principle of dealing with problems due to this device is similar to the data flow processing method in the embodiment of the present invention, therefore the enforcement of this device see the enforcement of method, can repeat part and repeats no more.
As shown in Figure 10, be the distributed data current processing device schematic diagram that the embodiment of the present invention one provides, this device belongs to first node, comprising:
Determination module 101, for distributing to the key assignments of the data flow of the down hop working node of described first node as required, determines that Section Point is the down hop working node of the data flow that the described key-value pair of process is answered;
Transferring module 102, for after determining that described Section Point meets the load migration condition of setting, need the data flow distributing to down hop working node to migrate to the 3rd node from described Section Point to process by described, and indicate the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes;
Wherein, described Section Point and the 3rd node belong to the down hop working node set of described first node, and the accumulative load of described 3rd node is less than the accumulative load of described Section Point.
Alternatively, described 3rd node is the working node in the down hop working node set of current described first node with minimum accumulative load.
Alternatively, the load migration condition of described setting comprises:
The accumulative load of described Section Point exceedes setting threshold; And/or,
Ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold.
Alternatively, the load migration condition of described setting also comprises: current time and described device the last time carry out the time interval of the time of data flow migration process and be not less than setting threshold.
Alternatively, described transferring module 102 is specifically for the accumulative load W according to any one down hop working node of described first node in following formula determination setting-up time section:
W＝c×W′+(1-c)×y；
Wherein, y is the load that described device distributes to any one down hop working node described in described setting-up time section, W ' is the accumulative load of any one down hop working node described at the end of the previous time period of described setting-up time section, c is constant, and 0 < c < 1.
Alternatively, described transferring module 102 specifically for, send to described Section Point the state information comprising described key assignments and to move out instruction, and, send to described 3rd node the state information comprising described key assignments and to move into instruction;
Wherein, described state information the first state information that instruction is used to indicate the data flow that described key-value pair that this locality generates by described Section Point is answered of moving out sends to the telegon managing each working node; Described state information is moved into indicate and is used to indicate the first state information that described 3rd node obtains the data flow that described key-value pair is answered from the telegon of each working node of management, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
Alternatively, described transferring module 102 also for, before determining that described Section Point meets the load migration condition of setting, the information of the described Section Point overload of the instruction that the telegon managing each working node if receive sends, then adjust the described load migration condition of setting.
Alternatively, described determination module 101, also for the key assignments distributing to the data flow of down hop working node as required, determines the current down hop working node that there is not the data flow that this key-value pair of process is answered; Described transferring module 102 also for, the described distribution of flows needing to distribute to down hop working node is processed to the working node in the down hop working node set of current described first node with minimum accumulative load.
As shown in figure 11, be the distributed data current processing device schematic diagram that the embodiment of the present invention two provides, this device belongs to above-mentioned Section Point, comprising:
Receiver module 111, the state information comprising setting key value sent for receiving first node is moved out instruction, and instruction of described state information being moved out transfers to sending module 112;
Sending module 112, described state information for receiving according to receiver module 111 is moved out instruction, determine the first state information of the data flow that the local described key-value pair generated is answered, and described first state information determined is sent to the telegon managing each working node.
As shown in figure 12, be the distributed data current processing device schematic diagram that the embodiment of the present invention three provides, this device belongs to above-mentioned 3rd node, comprising:
Receiver module 121, the state information comprising setting key value sent for receiving first node is moved into instruction, and instruction of described state information being moved into transfers to acquisition module 122;
Acquisition module 122, described state information for receiving according to receiver module 121 is moved into instruction, from the telegon of each working node of management, obtain the first state information that described Section Point is sent to the data flow that the described key-value pair in telegon is answered, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
As shown in figure 13, be the distributed data current processing device schematic diagram that the embodiment of the present invention four provides, this device belongs to first node, comprising:
Processor 131, for distributing to the key assignments of the data flow of the down hop working node of described first node as required, determine that Section Point is the down hop working node of the data flow that the described key-value pair of process is answered, after determining that described Section Point meets the load migration condition of setting, determine that the described data flow needing to distribute to down hop working node is migrated to the 3rd node from described Section Point to be processed, and determine the indication information of the state information indicating the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes, the indication information determined is transferred to transceiver 132, wherein, described Section Point and the 3rd node belong to the down hop working node set of described first node, and the accumulative load of described 3rd node is less than the accumulative load of described Section Point,
Transceiver 132, for sending described indication information to described Section Point and the 3rd node.
Alternatively, described 3rd node is the working node in the down hop working node set of current described first node with minimum accumulative load.
Alternatively, the load migration condition of described setting comprises:
The accumulative load of described Section Point exceedes setting threshold; And/or,
Ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold.
Alternatively, the load migration condition of described setting also comprises: current time and described device the last time carry out the time interval of the time of data flow migration process and be not less than setting threshold.
Alternatively, described processor 131 is specifically for the accumulative load W according to any one down hop working node of described first node in following formula determination setting-up time section:
W＝c×W′+(1-c)×y；
Wherein, y is the load that described device distributes to any one down hop working node described in described setting-up time section, W ' is the accumulative load of any one down hop working node described at the end of the previous time period of described setting-up time section, c is constant, and 0 < c < 1.
Alternatively, described transceiver 132 specifically for, send to described Section Point the state information comprising described key assignments and to move out instruction, and, send to described 3rd node the state information comprising described key assignments and to move into instruction;
Wherein, described state information the first state information that instruction is used to indicate the data flow that described key-value pair that this locality generates by described Section Point is answered of moving out sends to the telegon managing each working node; Described state information is moved into indicate and is used to indicate the first state information that described 3rd node obtains the data flow that described key-value pair is answered from the telegon of each working node of management, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
Alternatively, described processor 131 also for, before determining that described Section Point meets the load migration condition of setting, if the information of the described Section Point overload of the instruction being received the telegon transmission managing each working node by transceiver 132, then adjust the described load migration condition of setting.
Alternatively, described processor 131 is also for the key assignments distributing to the data flow of down hop working node as required, determine current there is not the down hop working node of this key-value pair of process data flow of answering time, the described distribution of flows distributing to down hop working node that needs is processed to the working node in the down hop working node set of current described first node with minimum accumulative load.
As shown in figure 14, be the distributed data current processing device schematic diagram that the embodiment of the present invention five provides, this device belongs to above-mentioned Section Point, comprising:
Transceiver 141, the state information comprising setting key value sent for receiving first node is moved out instruction, and instruction of described state information being moved out transfers to processor 142;
Processor 142, described state information for receiving according to transceiver 141 is moved out instruction, determine the first state information of the data flow that the local described key-value pair generated is answered, and by transceiver 141, described first state information determined is sent to the telegon managing each working node.
As shown in figure 15, be the distributed data current processing device schematic diagram that the embodiment of the present invention six provides, this device belongs to above-mentioned 3rd node, comprising:
Transceiver 151, the state information comprising setting key value sent for receiving first node is moved into instruction, and instruction of described state information being moved into transfers to processor 152;
Processor 152, described state information for receiving according to transceiver 151 is moved into instruction, from the telegon of each working node of management, obtain the first state information that described Section Point is sent to the data flow that the described key-value pair in telegon is answered by transceiver 151, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
Those skilled in the art should understand, embodiments of the invention can be provided as method, system or computer program.Therefore, the present invention can adopt the form of complete hardware embodiment, completely software implementation or the embodiment in conjunction with software and hardware aspect.And the present invention can adopt in one or more form wherein including the upper computer program implemented of computer-usable storage medium (including but not limited to magnetic disc store, CD-ROM, optical memory etc.) of computer usable program code.
The present invention describes with reference to according to the flow chart of the method for the embodiment of the present invention, device (system) and computer program and/or block diagram.Should understand can by the combination of the flow process in each flow process in computer program instructions realization flow figure and/or block diagram and/or square frame and flow chart and/or block diagram and/or square frame.These computer program instructions can being provided to the processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing device to produce a machine, making the instruction performed by the processor of computer or other programmable data processing device produce device for realizing the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
These computer program instructions also can be stored in can in the computer-readable memory that works in a specific way of vectoring computer or other programmable data processing device, the instruction making to be stored in this computer-readable memory produces the manufacture comprising command device, and this command device realizes the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
These computer program instructions also can be loaded in computer or other programmable data processing device, make on computer or other programmable devices, to perform sequence of operations step to produce computer implemented process, thus the instruction performed on computer or other programmable devices is provided for the step realizing the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
Although describe the preferred embodiments of the present invention, those skilled in the art once obtain the basic creative concept of cicada, then can make other change and amendment to these embodiments.So claims are intended to be interpreted as comprising preferred embodiment and falling into all changes and the amendment of the scope of the invention.
Obviously, those skilled in the art can carry out various change and modification to the present invention and not depart from the spirit and scope of the present invention.Like this, if these amendments of the present invention and modification belong within the scope of the claims in the present invention and equivalent technologies thereof, then the present invention is also intended to comprise these change and modification.
Claims (20)
1. a distributed data method for stream processing, is characterized in that, the method comprises:
First node distributes to the key assignments of the data flow of down hop working node as required, determines that Section Point is the down hop working node of the data flow that the described key-value pair of process is answered;
Described first node is after determining that described Section Point meets the load migration condition of setting, need the data flow distributing to down hop working node to migrate to the 3rd node from described Section Point to process by described, and indicate the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes;
Wherein, described Section Point and the 3rd node belong to the down hop working node set of described first node, and the accumulative load of described 3rd node is less than the accumulative load of described Section Point.
2. the method for claim 1, is characterized in that, described 3rd node is the working node in the down hop working node set of current described first node with minimum accumulative load.
3. method as claimed in claim 1 or 2, it is characterized in that, the load migration condition of described setting comprises:
The accumulative load of described Section Point exceedes setting threshold; And/or,
Ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold.
4. method as claimed in claim 3, it is characterized in that, the load migration condition of described setting also comprises:
Current time and described first node the last time carry out the time interval of the time of data flow migration process and are not less than setting threshold.
5. the method as described in as arbitrary in Claims 1 to 4, it is characterized in that, described first node is according to the accumulative load W of any one down hop working node of described first node in following formula determination setting-up time section:
W＝c×W′+(1-c)×y；
Wherein, y is the load that described first node distributes to any one down hop working node described in described setting-up time section, W ' is the accumulative load of any one down hop working node described at the end of the previous time period of described setting-up time section, c is constant, and 0 < c < 1.
6. the method as described in as arbitrary in Claims 1 to 5, it is characterized in that, described first node indicates the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes, comprising:
Described first node sends to described Section Point the state information comprising described key assignments and to move out instruction, and, send to described 3rd node the state information comprising described key assignments and to move into instruction;
Wherein, described state information the first state information that instruction is used to indicate the data flow that described key-value pair that this locality generates by described Section Point is answered of moving out sends to the telegon managing each working node; Described state information is moved into indicate and is used to indicate the first state information that described 3rd node obtains the data flow that described key-value pair is answered from the telegon of each working node of management, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
7. the method as described in as arbitrary in claim 1 ~ 6, it is characterized in that, described first node, before determining that described Section Point meets the load migration condition of setting, also comprises:
Described first node, after the information of the described Section Point overload of the instruction receiving the telegon transmission managing each working node, adjusts the described load migration condition of setting.
8. the method as described in as arbitrary in claim 1 ~ 7, it is characterized in that, described method also comprises:
First node distributes to the key assignments of the data flow of down hop working node as required, determine current there is not the down hop working node of this key-value pair of process data flow of answering time, the described distribution of flows distributing to down hop working node that needs is processed to the working node in the down hop working node set of current described first node with minimum accumulative load.
9. a distributed data method for stream processing, is characterized in that, the method comprises:
Section Point receives the state information comprising setting key value that first node sends and to move out instruction;
Described Section Point to be moved out instruction according to described state information, determines the first state information of the data flow that the local described key-value pair generated is answered, and described first state information determined is sent to the telegon managing each working node.
10. a distributed data method for stream processing, is characterized in that, the method comprises:
3rd node receives the state information comprising setting key value that first node sends and to move into instruction;
Described 3rd node to be moved into instruction according to described state information, from the telegon of each working node of management, obtain the first state information that described Section Point is sent to the data flow that the described key-value pair in telegon is answered, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
11. 1 kinds of distributed data current processing devices, is characterized in that, this device belongs to first node, comprising:
Determination module, for distributing to the key assignments of the data flow of the down hop working node of described first node as required, determines that Section Point is the down hop working node of the data flow that the described key-value pair of process is answered;
Transferring module, for after determining that described Section Point meets the load migration condition of setting, need the data flow distributing to down hop working node to migrate to the 3rd node from described Section Point to process by described, and indicate the state information of the data flow that key-value pair is answered described in described Section Point and the 3rd synchronisation of nodes;
Wherein, described Section Point and the 3rd node belong to the down hop working node set of described first node, and the accumulative load of described 3rd node is less than the accumulative load of described Section Point.
12. devices as claimed in claim 11, is characterized in that, described 3rd node is the working node in the down hop working node set of current described first node with minimum accumulative load.
13. devices as described in claim 11 or 12, it is characterized in that, the load migration condition of described setting comprises:
The accumulative load of described Section Point exceedes setting threshold; And/or,
Ratio and/or the difference of the accumulative load of described Section Point and the accumulative load of the 3rd node exceed setting threshold.
14. devices as claimed in claim 13, it is characterized in that, the load migration condition of described setting also comprises: current time and described device the last time carry out the time interval of the time of data flow migration process and be not less than setting threshold.
15. as arbitrary in claim 11 ~ 14 as described in device, it is characterized in that, described transferring module is specifically for the accumulative load W according to any one down hop working node of described first node in following formula determination setting-up time section:
W＝c×W′+(1-c)×y；
Wherein, y is the load that described device distributes to any one down hop working node described in described setting-up time section, W ' is the accumulative load of any one down hop working node described at the end of the previous time period of described setting-up time section, c is constant, and 0 < c < 1.
16. as arbitrary in claim 11 ~ 15 as described in device, it is characterized in that, described transferring module specifically for, send to described Section Point the state information comprising described key assignments and to move out instruction, and, send to described 3rd node the state information comprising described key assignments and to move into instruction;
Wherein, described state information the first state information that instruction is used to indicate the data flow that described key-value pair that this locality generates by described Section Point is answered of moving out sends to the telegon managing each working node; Described state information is moved into indicate and is used to indicate the first state information that described 3rd node obtains the data flow that described key-value pair is answered from the telegon of each working node of management, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
17. as arbitrary in claim 11 ~ 16 as described in device, it is characterized in that, described transferring module also for, before determining that described Section Point meets the load migration condition of setting, the information of the described Section Point overload of the instruction that the telegon managing each working node if receive sends, then adjust the described load migration condition of setting.
18. as arbitrary in claim 11 ~ 17 as described in device, it is characterized in that, described transferring module is also for the key assignments of distributing to the data flow of down hop working node as required, determine current there is not the down hop working node of this key-value pair of process data flow of answering time, the described distribution of flows distributing to down hop working node that needs is processed to the working node in the down hop working node set of current described first node with minimum accumulative load.
19. 1 kinds of distributed data current processing devices, is characterized in that, this device comprises:
Receiver module, the state information comprising setting key value sent for receiving first node is moved out instruction, and instruction of described state information being moved out transfers to sending module;
Sending module, described state information for receiving according to described receiver module is moved out instruction, determine the first state information of the data flow that the local described key-value pair generated is answered, and described first state information determined is sent to the telegon managing each working node.
20. 1 kinds of distributed data current processing devices, is characterized in that, this device comprises:
Receiver module, the state information comprising setting key value sent for receiving first node is moved into instruction, and instruction of described state information being moved into transfers to acquisition module;
Acquisition module, state information for receiving according to described receiver module is moved into instruction, from the telegon of each working node of management, obtain the first state information that described Section Point is sent to the data flow that the described key-value pair in telegon is answered, and the second state information of the data flow of the corresponding described key assignments described first state information obtained generated with this locality merges.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310513394.6A CN104580322B (en) | 2013-10-25 | 2013-10-25 | A kind of distributed traffic processing method and processing device |
| PCT/CN2014/078654 WO2015058525A1 (en) | 2013-10-25 | 2014-05-28 | Distributed method and device for processing data stream |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310513394.6A CN104580322B (en) | 2013-10-25 | 2013-10-25 | A kind of distributed traffic processing method and processing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104580322A true CN104580322A (en) | 2015-04-29 |
| CN104580322B CN104580322B (en) | 2019-02-12 |
Family
ID=52992197
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310513394.6A Active CN104580322B (en) | 2013-10-25 | 2013-10-25 | A kind of distributed traffic processing method and processing device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104580322B (en) |
| WO (1) | WO2015058525A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016206567A1 (en) * | 2015-06-26 | 2016-12-29 | 阿里巴巴集团控股有限公司 | Distributed stream computing system, method and device |
| CN108063731A (en) * | 2018-01-03 | 2018-05-22 | 烟台大学 | Load balance distribution method based on data distribution in a kind of distributed traffic |
| CN108632144A (en) * | 2017-03-17 | 2018-10-09 | 华为数字技术（苏州）有限公司 | The method and apparatus for transmitting flow |
| CN112527841A (en) * | 2020-12-17 | 2021-03-19 | 上海数依数据科技有限公司 | Stream data merging processing method and device |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107454013A (en) * | 2017-06-08 | 2017-12-08 | 国家计算机网络与信息安全管理中心 | A kind of method and apparatus of flow data processing system data partition |
| CN110839086A (en) * | 2019-12-23 | 2020-02-25 | 吉林省民航机场集团公司 | High-concurrency load balancing processing method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080049753A1 (en) * | 2006-08-22 | 2008-02-28 | Heinze John M | System and method for load balancing network resources using a connection admission control engine |
| CN101442435A (en) * | 2008-12-25 | 2009-05-27 | 华为技术有限公司 | Method and apparatus for managing business data of distributed system and distributed system |
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
-
2013
- 2013-10-25 CN CN201310513394.6A patent/CN104580322B/en active Active
-
2014
- 2014-05-28 WO PCT/CN2014/078654 patent/WO2015058525A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080049753A1 (en) * | 2006-08-22 | 2008-02-28 | Heinze John M | System and method for load balancing network resources using a connection admission control engine |
| CN101442435A (en) * | 2008-12-25 | 2009-05-27 | 华为技术有限公司 | Method and apparatus for managing business data of distributed system and distributed system |
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016206567A1 (en) * | 2015-06-26 | 2016-12-29 | 阿里巴巴集团控股有限公司 | Distributed stream computing system, method and device |
| CN106293892A (en) * | 2015-06-26 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Distributed stream calculates system, method and apparatus |
| CN106293892B (en) * | 2015-06-26 | 2019-03-19 | 阿里巴巴集团控股有限公司 | Distributed stream computing system, method and apparatus |
| CN108632144A (en) * | 2017-03-17 | 2018-10-09 | 华为数字技术（苏州）有限公司 | The method and apparatus for transmitting flow |
| CN108063731A (en) * | 2018-01-03 | 2018-05-22 | 烟台大学 | Load balance distribution method based on data distribution in a kind of distributed traffic |
| CN112527841A (en) * | 2020-12-17 | 2021-03-19 | 上海数依数据科技有限公司 | Stream data merging processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104580322B (en) | 2019-02-12 |
| WO2015058525A1 (en) | 2015-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104580322A (en) | Distributed data stream processing method and device | |
| CN107729147B (en) | Data processing method in stream computing system, control node and stream computing system | |
| CN109905259B (en) | Communication connection maintaining method, system and related equipment | |
| US10270726B2 (en) | Selective distribution of messages in a scalable, real-time messaging system | |
| CN109600388B (en) | Data transmission method and device, computer readable medium and electronic equipment | |
| CN103580966A (en) | Wireless link and wire link parallel data transmission method and system | |
| CN103516744A (en) | A data processing method, an application server and an application server cluster | |
| CN102447636A (en) | Load sharing method and device of data center communication equipment | |
| CN103812949A (en) | Task scheduling and resource allocation method and system for real-time cloud platform | |
| CN101873005A (en) | Method for realizing balanced acquisition of electric energy | |
| CN103544285A (en) | Data loading method and device | |
| CN103297543A (en) | Job scheduling method based on computer cluster | |
| WO2016095410A1 (en) | Link traffic distributing method and device | |
| CN105471700B (en) | A kind of methods, devices and systems of Message Processing | |
| CN105786447A (en) | Method and apparatus for processing data by server and server | |
| US11172018B2 (en) | Method and apparatus for controlling traffic | |
| CN105610992A (en) | Task allocation load balancing method of distributive flow computing system | |
| CN105528366A (en) | A data synchronization control method and device | |
| CN113364701A (en) | Congestion control method and device based on RTT (round trip time) and combined with proportional-integral-derivative control | |
| WO2021057068A1 (en) | Rdma data flow control method and system, electronic device and readable storage medium | |
| CN101893876A (en) | Control network design for large-scale distributed control system | |
| JP2013235515A (en) | Data distribution management system, apparatus, method and program | |
| CN112866985A (en) | Flow control method, resource downloading method, device, equipment and storage medium | |
| US10684875B2 (en) | Synchronization of a virtual machine across mobile devices | |
| WO2023005809A1 (en) | Message queuing telemetry transport network access method, and controller and gateway |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |