CN103188345A - Distributive dynamic load management system and distributive dynamic load management method - Google Patents
Distributive dynamic load management system and distributive dynamic load management method Download PDFInfo
- Publication number
- CN103188345A CN103188345A CN201310065185XA CN201310065185A CN103188345A CN 103188345 A CN103188345 A CN 103188345A CN 201310065185X A CN201310065185X A CN 201310065185XA CN 201310065185 A CN201310065185 A CN 201310065185A CN 103188345 A CN103188345 A CN 103188345A
- Authority
- CN
- China
- Prior art keywords
- load
- processing node
- node
- processing
- event
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a distributive dynamic load management system and distributive dynamic load management method. The system consists of distributive processing clusters with dynamic load management function and P2P (Peer to Peer) complete equal frameworks. The processing clusters are not provided with central nodes controlled intensively but are provided with a plurality of processing nodes. Each processing node can independently provide distributive arbitral dynamic load management capacity and the nodes are completely same in right and dynamic load management function but same or different in event processing capacity. Each processing node is a control core of the dynamic load management system, and is provided with four modules, namely an event processing module, an event monitoring module, an event issuing module and a dynamic load management module. According to the system and the method, distributive dynamic load management can be realized under a distributive network environment, so that system load transfer fluctuation caused by change of data processing request hot spots and system load balance during node dynamic change can be effectively solved. The robustness of the system can be effectively ensured when the processing nodes of the distributive cluster system are overloaded under the P2P environment.
Description
Technical field
The present invention relates to a kind of distributed dynamic load management system and method, belong to the technical field of computer.
Background technology
Along with the quick growth of network traffic, visit capacity and data traffic, end-to-end P2P(Peer-to-Peer) load management technology under the peer to peer technology environment become urgent problem.At present, the load-balancing technique in the P2P architecture environment becomes the focus of research already, and existing a lot of technical schemes can be optimized the load-balancing performance of P2P system.
As everyone knows, the P2P system is divided into structuring and destructuring two big classes, has solved the directed problem of resource of destructuring system based on the structured P 2 P system of distributed hash table (DHT).But in structured P 2 P system, data processing request and node balancedly are mapped on the logic box at random by the consistency hash function; Because the problem of load balancing that some characteristics of data processing request task are brought becomes increasingly conspicuous.Studies show that, in real time mass data processing request information, there is hot issue in most request tasks, roughly obeying Zipfp distributes: namely in certain time period, 60% ~ 80% data processing request can be mapped on certain or certain several nodes by the consistency hash function, cause in the P2P system some node load very high, and other node load is very light; And the focus of different time sections data processing request task has modificability, thereby causes the overall laod unbalance of whole system, and the treatment effeciency of system and stability reduce.
In order effectively to solve data processing request hot issue in the structured P 2 P system, existing program is devoted to solve communication overhead (communication cost of Hash ring value and the change of the processing node contrast relationship) problem that how to reduce in the load transfer at present.But, because the data processing request focus has modifiability, they do not consider the fluctuation of the system load transfer that the change of data processing request focus brings, especially in number of nodes environment of certain scale, fluctuation can be stronger, the communication cost of bringing can become greatly, thereby influences the load balancing rate of convergence of system, and the GSLB of system.Again, these schemes are not all considered the overload control problem of structured P 2 P system under the full overload situations of node, when system data processing request amount surpasses the maximal workload of system, can cause the integral body collapse of system, influence the stability of system and the consequence of bringing on a disaster property.Therefore, the dynamic management approach that how to improve load also becomes the research focus.
In order effectively to solve data processing request hot issue and system load balancing problem in the structured P 2 P system, present prior art scheme mainly contains following several:
" load balancing implementation method and the device of distributed hashtable network " (Chinese patent application publication number: CN102457429A) provide a kind of dynamic load equilibrium technology that shifts based on dummy node.This method is at first according to the node planning of described DHT network and the unbalanced degree threshold value of presetting of load, the quantity Z(Z that determines first virtual identifying is natural number), and the whole load space of this DHT network is divided into Z part, each part load space identifies with mutually different described first virtual identifying respectively; According to the ability of each described node, Z first virtual identifying distributed each node again, be responsible for distributing to the load space of its first virtual identifying correspondence by each node.The present invention can reduce the workload of virtual identifying management and virtual identifying migration when realizing the equilibrium of DHT network data memory load.This method can effectively reduce the workload of virtual identifying management and virtual identifying migration, but, the system load transfer fluctuation problem that the change of request focus brings that solves is not provided, and the system robustness problem under the full overload situations of system, application has certain limitation under the distributed type assemblies environment.
" SiteServer LBS and method " (Chinese patent application publication number: CN 102436401A) provide a kind of load-balancing technique of multistage load strategy.The SiteServer LBS of its proposition comprises: first order load equalizer, a plurality of second level load equalizer and a plurality of server.Wherein, first order load equalizer is used for receiving a plurality of tasks, and uses first strategy that described a plurality of tasks are dispensed to a plurality of second level load equalizer; A plurality of second level load equalizer is used for using second strategy a plurality of tasks to be dispensed to a plurality of servers.Wherein, introduce the stratification concept and adopt multistage load-balancing method, can reduce the possibility that the load balancing center becomes performance bottleneck, support simultaneously to have promoted flexibility, accuracy, robustness and the fail safe of load balancing process by more complicated load balancing strategy.But the load strategy change overhead issues that it brings when not considering the processing node dynamic change does not provide the technical scheme that how to solve the system robustness problem under the full overload situations of system yet.
Summary of the invention
In view of this, the purpose of this invention is to provide a kind of improved distributed dynamic load management system and method based on the DHT principle, the present invention is the system that realizes the distributed dynamic load management under distributed network environment, can effectively solve system load that the change of data processing request focus brings and shift the problem that guarantees system load balancing when fluctuation problem and node dynamically change; Also support effectively to guarantee the robustness problem of system under each processing node of distributed cluster system all transships under the P2P environment the situation.
In order to achieve the above object, the invention provides a kind of distributed dynamic load management system, it is characterized in that: described system is by having the dynamic load management function, and having end-to-end P2P(Peer-to-Peer) the distributed treatment cluster of complete reciprocity framework forms, this Processing Cluster does not have central controlled Centroid, only be provided with a plurality of processing nodes, wherein each processing node can both independently provide the dynamic load managerial ability of distributed arbitration program, and function of authority and dynamic load management is identical separately for it, but the ability of the event of processing is identical or different; Each processing node is the control core of this dynamic load management system, and all is provided with following four functional modules: event processing module, action listener module, event distribution module, dynamic load administration module; The function of each module is as follows:
Event processing module, event for the treatment of some particular type and attribute, produce intermediate treatment result or final result, and the new events that produces is transferred to this processing node or mail to other processing node continuation processing by the event distribution module, or final process result is returned to client; This event processing module is default in this system, or is developed and be deployed on the processing node by the third party programmer;
The action listener module be responsible for to receive the data processing request event from client, and from the event of the processing node of self or other; And accept the dynamic load administrative decision control command of dynamic load administration module: when receiving the load transfer instruction, event is sent to the processing node of appointment in this instruction respectively; When receiving the overload control command, produce the response message of system overload, and send to client; Perhaps event being transferred to event processing module handles again;
The dynamic load administration module, be provided with two parts of load monitoring unit and load decision package, wherein: the load monitoring unit is under the action listener module is assisted, the information of survival condition, pay(useful) load rate and the comprehensive treatment capability of each processing node in collection and the distributed storage Processing Cluster, and real-time update in real time and dummy node mapping table and the dynamic load information table of storage after the load decision package is created or safeguarded; The load decision package is responsible for the information based on survival condition, pay(useful) load rate and the comprehensive treatment capability of each processing node, with and the dummy node mapping table of storage, load balancing, the load that formulate to be used for event handling and event distribution procedure shifted and the dynamic load administrative decision control command of overload control;
The event distribution module, be responsible for accepting the dynamic load administrative decision control command that the dynamic load administration module sends, its new events or final process result of being produced by event processing module that receives is carried out the event distribution processor: when receiving the load balancing instruction, action listener module or other processing node or client that its new events that is produced by event processing module that receives or final process result are sent to this specified processing node of this instruction respectively.
In order to achieve the above object, the present invention also provides a kind of and has adopted distributed dynamic load management system of the present invention based on the distributed dynamic load management method of P2P technology, it is characterized in that: when client sends the mass data processing request in real time to this distributed treatment cluster, after certain processing node is received request, whether earlier judge according to the pay(useful) load rate of this node and overloading threshold that this processing node is current transships, if overload is then carried out load transfer or overload control; Otherwise, handle this request; After the request finished is handled, just the intermediate object program that produces is encapsulated as event, and carries out load-balancing decision according to dummy node mapping table and this event determinant attribute value, determine this event still to mail to other processing node by self node and continue processing; And follow-up each processing node is when handling and distribute intermediate event, all be according to pay(useful) load rate and overloading threshold or dummy node mapping table and event determinant attribute value execution load transfer, overload control or the balanced operation of distributing of event, up to the final result of generation, and return to client.
The innovative technology characteristics of system of the present invention are: realize distributed load equalizing by increasing the dummy node layer based on the DHT principle, and realize that by the numbering of data processing request or event being carried out probability ground re-orientation processes node load shifts, and abandon by the probability overload and to realize overload control.
The Hash ring value of consistency Hash has similarity in dummy node concept in the system of the present invention and the DHT algorithm, all be evenly to distribute pro rata to the mapping of handling node with according to the processing node comprehensive treatment capability by dummy node, realize the load management on first aspect of system of the present invention thus; Be redirected by probability processing node numbering again, can solve the unbalanced problem of the caused processing node load of data processing request hot issue, thus the load management on second aspect of realization system of the present invention.And abandon by the probability overload, can solve distributed treatment cluster overall situation load overload problem, thereby realize the load management on the 3rd aspect of the present invention.
Load-balancing technique in the P2P architecture environment is the focus of research already, and existing a lot of technical schemes are optimized the load-balancing performance of P2P system.Than these schemes, the present invention has more advantage, and its reason is:
In solving structured P 2 P system in the data processing request hot issue; the scheme of prior art is devoted to solve communication overhead (being the communication cost of Hash ring value and the processing node contrast relationship change) problem that how to reduce in the load transfer at present, does not consider the fluctuation problem that the system load that change causes of data processing request focus is shifted.System of the present invention is in the load transfer process, do not change the mapping relations between dummy node and the processing node, just probability ground is redirected the processing node numbering, namely according to setting probability from the set of underload processing node, selecting this K numerical value of K(at random is a less natural number of setting in system) small set of individual processing node composition, from this small set, elect a processing node of pay(useful) load rate minimum then, and it is numbered as the re-orientation processes node serial number.Thereby system of the present invention can solve the system load that the change of data processing request hot issue and focus causes in the structured P 2 P system effectively and shifts the fluctuation problem.
Moreover, in the scheme of the load-balancing performance of optimizing the P2P system, also there not be discovery that system robustness problem under the full overload situations of technical scheme taking into account system is arranged at present.System of the present invention adopts probability ground overload drop policy under the full overload situations of system, can effectively solve the robustness problem of system under the full overload situations.Thereby the distributed dynamic load management method of system of the present invention is a kind of improvement at present existing DHT implementation method.Therefore, popularizing application prospect of the present invention is good.
Description of drawings
Fig. 1 is that distributed terminator management system general structure of the present invention is formed schematic diagram.
Fig. 2 is that the processing node functional module in the distributed terminator management system of the present invention is formed schematic diagram.
Fig. 3 is that distributed terminator management system of the present invention is based on the distributed dynamic load management method operating procedure flow chart of P2P technology.
Fig. 4 is the flow chart that load management system of the present invention is carried out the distribution of distributed dynamic load management event.
Fig. 5 is the establishment renewal flow chart that load management system of the present invention is carried out the distributed load equalizing table
Fig. 6 is that load management system of the present invention is carried out the event distribution flow figure that the distributed dynamic load is shifted.
Fig. 7 is the event distribution flow figure that load management system of the present invention is carried out the control of distributed dynamic overload.
Embodiment
For making the purpose, technical solutions and advantages of the present invention clearer, the present invention is described in further detail below in conjunction with accompanying drawing.
Referring to Fig. 1, introduce the structure composition that the present invention has the dynamic load management function and has the distributed processing system(DPS) of the complete reciprocity framework of end-to-end P2P: this system does not have Centroid (being centralized control node), but be provided with a plurality of processing nodes, each processing node can both independently provide the load management ability of distributed arbitration program, and function of authority and dynamic load management is identical separately for it, so equity fully between each processing node; But it is identical or different that each processing node is handled the ability of event; Each processing node is the control core of this dynamic load management system, and is divided on the server of different server.In the actual motion environment was disposed, same physical server can arrange and move a plurality of processing nodes.
Referring to Fig. 2, introduce four functional modules in inside of processing node:
(1) action listener module: be responsible for to receive the data processing request event from client, and from the event of the processing node of self or other; And accept the dynamic load administrative decision control command of dynamic load administration module: when receiving the load transfer instruction, event is sent to the processing node of appointment in this instruction respectively; When receiving the overload control command, produce the response message of system overload, and send to client; Perhaps event being transferred to event processing module handles again.
(2) event processing module: for the treatment of the event of some particular type and attribute, produce intermediate treatment result or final result, and the new events that produces is transferred to this processing node or mail to other processing nodes by the event distribution module and continue to handle, or final process result is returned to client; This event processing module is default in this system, or is developed and be deployed on the processing node by the third party programmer.
(3) dynamic load administration module: be responsible for the required load information of establishment and maintenance event monitoring module and event distribution module and the load decision information is provided, be provided with two parts of load monitoring unit and load decision package, wherein: the load monitoring unit is under the action listener module is assisted, the information of survival condition, pay(useful) load rate and the comprehensive treatment capability of each processing node in collection and the distributed storage Processing Cluster, and real-time update in real time and dummy node mapping table and the dynamic load information table of storage after the load decision package is created or safeguarded; The load decision package is responsible for the information based on survival condition, pay(useful) load rate and the comprehensive treatment capability of each processing node, with and the dummy node mapping table of storage, load balancing, the load that formulate to be used for event handling and event distribution procedure shifted and the dynamic load administrative decision control command of overload control.
The dynamic load administrative decision function declaration of load decision package wherein is as follows:
Survival condition and the comprehensive treatment capability information of each processing node of distributed treatment cluster that provides according to the load monitoring unit, when system initialization or processing node generation dynamic change, real-time servicing dummy node mapping table and dynamic load information table, adjust the load of each processing node in the distributed treatment cluster, be the dummy node number, so that respectively handle the node load equilibrium;
When the hot issue because of processing and distributing data processing request or event causes that processing node pay(useful) load rate is unbalanced, take the load transition strategy, so that respectively handle the node load equilibrium;
When each processing node pay(useful) load rate is all transshipped, take to transship control strategy, so that the distributed treatment cluster keeps the stability of a system and robustness.
(4) event distribution module: be responsible for accepting the dynamic load administrative decision control command that the dynamic load administration module sends, its new events or final process result of being produced by event processing module that receives is carried out the event distribution processor: when receiving the load balancing instruction, action listener module or other processing node or client that its new events that is produced by event processing module that receives or final process result are sent to this specified processing node of this instruction respectively.
The event distribution module has defined following Same of Important for distributed dynamic load management method can be provided:
Processing node pay(useful) load rate: be the pay(useful) load rate of the last load information collection period of processing node and the value after the real-time load factor weighting in this cycle.The actual negative carrying capacity that this real-time load factor is processing node and the ratio of its comprehensive treatment capability, wherein, the actual negative carrying capacity of processing node comprises the consumption of CPU and internal memory, or the length of pending event queue; The comprehensive treatment capability of processing node comprises the spatial content of cpu function, internal memory or the maximum permissible value of pending event queue length; Utilize the pay(useful) load rate of processing node can effectively handle isomerism problem in the distributed system, and processing node is divided into following three ranks according to the pay(useful) load rate: underload processing node, normal load processing node and overload processing node.
Dummy node: the concept that provides distributed load equalizing method to introduce is provided.Dummy node is necessary being not, just the logical symbol with the processing node mutual mapping.Each dummy node can only be mapped to a processing node, but each processing node can be mapped to a plurality of dummy nodes.The configuration when quantity of dummy node starts in system, its quantity is much larger than the number of processing node; And after system's operation, the quantity of dummy node remains unchanged, and when the processing node in the system increased or shifts out, the mapping relations between processing node and dummy node were carried out corresponding change according to distributed dynamic load management method.
Dummy node mapping table: be a kind of important data structures that system of the present invention realizes the distributed dynamic load management, be used for the mapping relations of each dummy node symbol of storage and its processing node numbering.Each processing node has dummy node mapping table separately, in distributed load equalizing method of the present invention, system realizes distributed load equalizing by searching the processing node (physical node) that the dummy node mapping table can search out its correspondence mappings from dummy node.
Dynamic load information table: be another important data structures that system of the present invention realizes the distributed dynamic load management, be used for storage each processing node management and dummy node symbol and the number thereof controlled, the information in this dynamic load information table and the information of dummy node mapping table are one to one.
The application scenarios of system of the present invention includes following four kinds:
The Distribution Events scene: client sends the mass data processing request in real time to the distributed treatment cluster, in the data processing request transmitting-receiving just often, this cluster returns result to client in real time, and guarantees load balancing and the stability of each processing node in this cluster; And in the dynamic load management process, each processing node equal, i.e. no centralized load management node in this cluster, each processing node can both independently be finished the dynamic load management of distributed arbitration program;
The scene of processing node initialization and dynamic change: when client sends the mass data processing request in real time to the distributed treatment cluster, real-time start-up system or increase new processing node or shift out processing node to this cluster; And in short time as far as possible, adjust its load in proportion according to the comprehensive treatment capability of each processing node again, make that each processing node in this cluster reaches load balancing again;
Scene is shifted in the processing node load: client sends in the mass data processing request process in real time to the distributed treatment cluster, because there is hot issue in data processing request, cause receiving the pay(useful) load rate overload of processing of request node, and when in the set of underload processing node, also having processing node, this cluster with probability be redirected data processing request and mail to certain or some processing node in the underload processing node set, in order to dynamically adjust the load balancing of this cluster in real time;
Processing node overload control scene: client sends the mass data processing request in real time to the distributed treatment cluster, because of data processing request quantity too huge, when causing each processing node pay(useful) load rate in this cluster all to be transshipped, each processing node in this cluster with probability abandon data processing request, and to the response message of client answering system overload, so that the stability of this cluster of real-time ensuring and reliability.
Introduce distributed dynamic load management system of the present invention below based on the distributed dynamic load management method of P2P technology: when client sends the mass data processing request in real time to this distributed treatment cluster, after certain processing node receives the data processing request or event that needs to handle, whether earlier judge according to the pay(useful) load rate of this node and overloading threshold that this processing node is current transships, if overload, then carry out load transfer or overload control, guarantee the stability of a system and load balancing; Otherwise, handle this request.After finishing request or the processing of event, just the intermediate object program that produces is encapsulated as event, and carries out load-balancing decision according to dummy node mapping table and this event determinant attribute value, determine this event still to mail to other processing node by self node and continue processing; And follow-up each processing node is when handling and distribute intermediate event, all be according to pay(useful) load rate and overloading threshold or dummy node mapping table and event determinant attribute value execution load transfer, overload control or the balanced operation of distributing of event, up to the final result of generation, and return to client.
Referring to Fig. 3, introduce the concrete operations step of the inventive method:
Step 1, the dynamic load administration module of processing node are created or renewal dummy node mapping table and dynamic load information table; Simultaneously, the dynamic load administration module is gathered existing state and the real-time load information of each processing node in the distributed type assemblies according to setting cycle under the action listener module is assisted.This step content of operation is as follows:
(11) system initially sets up, when creating dummy node mapping table and dynamic load information table, and the method initialization dummy node mapping table that adopts the WRR that is directly proportional according to comprehensive treatment capability or Weighted random to distribute; Perhaps adopt Random assignment and guarantee equally distributed method initialization dummy node mapping table; At this moment, the dummy node number of distributing for each different processing node of processing event ability should be differentiated.
(12) when increasing in this system or shifting out processing node, reach load balancing as early as possible again for making system, the dynamic load administration module will carry out real-time update to dummy node mapping table and dynamic load information table.
This step (12) comprises following content of operation:
When (12a) the dynamic load administration module of each processing node detects processing node in this system change is arranged, this situation is informed all processing nodes in this cluster.
(12b) for reduce the system topological maintenance workload as far as possible, the dynamic load administration module of each processing node adopts consistency hash method real-time update dynamic load information table and dummy node mapping table according to the initialization of processing node, the modification information that increases or shift out.
(13) after system starts operation, load monitoring unit in the dynamic load administration module is under the action listener module is assisted, gather existing state and the real-time load information of each processing node in the distributed type assemblies according to setting cycle, and calculate its pay(useful) load rate according to the real-time load information of processing node, and processing node is included into respectively in the set of underload processing node, the set of normal load processing node or the set of overload processing node.
This step (13) comprises following content of operation:
(13a) the load monitoring unit is under the action listener module is assisted, and the cycle is gathered the real-time load information of each processing node; And in order to solve the isomerism of processing node, the real-time load information of processing node all is normalized to real-time load factor, i.e. the real-time load factor of calculating this this cycle of processing node according to real-time load information and the comprehensive treatment capability thereof of each processing node.
(13b) the load monitoring unit is according to the pay(useful) load rate of last one-period and the real-time load factor in this cycle, adopt weighting scheme to calculate the pay(useful) load rate in this this cycle of processing node, and this processing node pay(useful) load rate score that will obtain is stored in the load monitoring unit of dynamic load administration module.
(13c) the load monitoring unit compares judgement with the pay(useful) load rate score of this processing node and the load overloading threshold of default earlier, if the pay(useful) load rate score is greater than the load overloading threshold, then this processing node is included in the set of overload processing node, finishes this step operation; Otherwise namely the pay(useful) load rate score is not more than the load overloading threshold, then carries out subsequent step (13d).
(13d) the load monitoring unit compares judgement with the pay(useful) load rate score of this processing node and the underload threshold value of default again, if the pay(useful) load rate less than the underload threshold value, then is included into this processing node in the set of underload processing node; Otherwise namely the pay(useful) load rate score is not more than the load overloading threshold, just this processing node is included in the set of normal load processing node.
(21) after in a single day the action listener module listens to the event that needs to handle, load decision package in the dynamic load administration module and load monitoring unit carry out alternately, know the pay(useful) load rate score of each processing node, and whether the pay(useful) load rate of judging this processing node transships: if it does not transship, then carry out subsequent step (22); If its overload, then redirect execution in step (23).
(22) the load decision package need not to carry out load transfer or overload control strategy, directly gives event processing module with this event by the action listener module and handles, and finishes this step operation.
(23) the load decision package judges whether other processing node all transships, if, then take probabilistic overload drop policy to abandon this event: namely the processing node pay(useful) load rate that makes preparation mail to is more high, then its probability of carrying out the operation of overload drop policy is more big, to realize overload control, the working stability of real-time ensuring distributed treatment cluster and reliable; If not, namely also there is the lighter processing node of effective load factor, then takes probability to be redirected the processing node of this event, shift to realize load; And when transferring load, do not change the mapping relations of dummy node and processing node, just probability is reselected the processing node numbering of this event, and this event handled without event processing module just directly mail to new processing node, namely the processing node numbering is redirected according to the probability that is directly proportional with this processing node pay(useful) load rate, and make that this processing node pay(useful) load rate is more high, the probability that this processing node is carried out event re-orientation processes nodal operation is more big.
This step (23) comprises following content of operation:
(23a) the load decision package judges whether other processing node all transships, if then carry out subsequent step (23b); If not, namely also there is the lighter processing node of effective load factor, then redirect execution in step (23c);
(23b) the load decision package produces the decimal at random in (0, a 1) scope, and with this at random decimal and this processing node pay(useful) load rate score compare judgement: this at random decimal whether less than this processing node pay(useful) load rate score; If then the load decision package sends overload to the action listener module and abandons instruction, the action listener module directly abandons this event, and the response message of answering system overload, under the event distribution module is assisted this response message is sent to client then; If not, then carry out subsequent step (23e);
(23c) the load decision package produces the decimal at random in (0, a 1) scope, and with this at random decimal and this processing node pay(useful) load rate score compare judgement: this at random decimal whether less than this processing node pay(useful) load rate score; If then carry out subsequent step (23d); If not, then redirect execution in step (23e);
(23d) the load decision package takes out one or more processing nodes in the load node set on the lenient side at random, and one or more processing nodes of therefrom seeking out pay(useful) load rate score minimum are numbered the destination of being redirected as this event, then, this processing node numbering and load transfer instruction are returned to the action listener module; The action listener module is under the assistance of event distribution module, and the processing node that directly this event is mail to this processing node numbering representative is handled, in order to dynamically adjust the load balancing of distributed treatment cluster in real time;
(23e) the action listener module is directly given event processing module with this event and is handled.
After step 3, event processing module are finished processing to this event, produce intermediate event or final process result, send to the event distribution module; Carry out the dynamic load administrative decision by load decision package in the dynamic load administration module according to dummy node mapping table, intermediate event determinant attribute value, and formulation dynamic load administrative decision control command, the event distribution module is according to this instruction execution event distribution operation, intermediate event or final process result are distributed to corresponding processing node or client, and realize respectively handling the node load equilibrium
The content of operation of this step is as described below:
(31) event processing module is finished dealing with after this event, the intermediate event that produces is carried out consistency Hash evaluate operation according to its determinant attribute value, and mould is got the dummy node number of this system, obtaining preparation with the dummy node numbering of this intermediate event distribution, in order to this intermediate event is mail to self or other processing node continues to handle.
(32) the event distribution module carries out alternately with the dynamic load administration module according to the dummy node numbering of the preparation distribution that obtains, and obtains the processing node numbering that this intermediate event will mail to according to the dummy node mapping table in the dynamic load administration module.
(33) the event distribution module processing node number information that will mail to according to this intermediate event mails to the respective handling node with it and continues to handle, and up to the event of generation final process result, directly returns to client.
The present invention has carried out repeatedly implementing test, and the implementing procedure of system embodiment execution distributed dynamic load management method of the present invention under Distribution Events scene, processing node initialization, dynamic change scene, processing node load transfer scene and processing node overload control scene is described respectively below.
Referring to Fig. 4, introduce the distributed dynamic load management event distribution flow of Distribution Events scene, its correspondence be the conventional scene that native system carries out mass data processing, namely data processing request does not exist focus and quantity not to exceed in the scene of maximal workload of system.Its operating procedure is as described below:
(1) the action listener module of processing node listens to the event that new needs are handled.Need to prove: each processing node in this distributed processing system(DPS) is all listening to the event that needs processing at any time.
(2) the action listener module is obtained the decision instruction request to the load decision-making module.
(3) load decision-making module and load monitoring module are mutual, obtain the pay(useful) load rate information of each processing node of system.
(4) the load decision-making module learns that according to each the processing node pay(useful) load rate information that obtains each processing node pay(useful) load rate of distributed treatment cluster is normal, need not to carry out load transfer or overload control strategy.
(5) the action listener module sends to event processing module with this event and handles;
(6) after event processing module is finished this event handling, the event that newly produces waited for mail to other processing nodes or self processing node is proceeded to handle.
(7) the event distribution module carries out Hash operation according to the new events property value, and mould gets the virtual processing node number (its numerical value is the constant of configuration when starting) of system, obtains the dummy node numbering.
(8) the event distribution module is numbered according to the dummy node that obtains, and is mutual with the load decision-making module, dummy node mapping table in the inquiry dynamic load administration module, the processing node numbering that the event of obtaining will be distributed.
(9) the event distribution module mails to other processing nodes or self processing node continuation processing with new events, namely repeats from the process of step (1) beginning, up to the event of generation final process result, returns to client.
It should be noted that, the operating process of above-mentioned steps (2) ~ (8) is the core process of invention distributed dynamic load management method, can see that each processing node carries out the load management strategy respectively, no central loading management processing node is a kind of typical distributed dynamic load management method based on the p2p technology.
Referring to Fig. 5, when introducing system of the present invention and carrying out mass data processing, during corresponding to processing node initialization or processing node dynamic change, the load number of each processing node of Adjustment System makes the distributed treatment cluster reach the load balancing process again, is the establishment and the flow chart that upgrades operation of distributed load equalizing table.
(1) the load monitoring module of processing node detects the processing node change in the distributed processing system(DPS).
(2) the load decision-making module of all processing nodes in the load monitoring module distribution of notifications formula cluster: have processing node to change.
(3) respectively handle the node load decision-making module and upgrade dynamic load information table and dummy node mapping table according to processing node modification information (comprising: initialization information, increase node and shift out nodal information).The load number of namely distributing each processing node according to the comprehensive treatment capability of each processing node in the distributed processing system(DPS) pro rata.The dynamic load information table of processing node of the present invention and dummy node mapping table update method have been used for reference the consistency Hash and reduced the thought that system topological is safeguarded as far as possible when node are added or shifts out system.
Referring to Fig. 6, introduce the distribution flow figure that shifts the corresponding distributed dynamic load failover events of scene with the processing node load.Its correspondence be that native system is when carrying out mass data processing, after causing the processing node load overload that will mail to because of the data processing request hot issue, the load balancing module of processing node is taked load to shift, is the content of operation of the failover events distribution flow that is redirected of processing node numbering:
(1) the action listener module listens to event or the data processing request that needs processing.
(2) action listener module and load decision-making module are mutual, send the decision instruction request to it.
(3) load decision-making module and load monitoring module are mutual, obtain the pay(useful) load rate information of this each processing node of system.
(4) the load decision-making module learns that according to each the processing node pay(useful) load rate information that obtains processing node pay(useful) load rate overload and the set of underload processing node that will mail to also have processing node, and the load decision-making module carries out probability load transfer operation; Taking out K(K at random in the load node set on the lenient side is a less natural number of setting in the system) individual processing node, and from this K processing node, find out the processing node of pay(useful) load rate minimum, and this processing node numbering returned to the event distribution module, namely carry out the processing node numbering and be redirected.
(5) the event distribution module mails to the processing node that this processing node numbering refers to new events and handles, in order to dynamically adjust the load balancing of distributed treatment cluster in real time.
Send in real time in the mass data processing request process to the distributed treatment cluster in client, there is hot issue in processing node in the system of the present invention because of data processing request, thereby the load that causes mailing to processing node is unbalanced, and this system carries out probability ground and mails to the operation that the processing node numbering is redirected; Adjust the load balancing of distributed treatment cluster real-time dynamicly.
Referring to Fig. 7, introduce the present invention corresponding to the distributed dynamic overload control event distribution flow figure of processing node overload control scene.Its correspondence be native system when carrying out mass data processing, after the load of each processing node of distributed treatment cluster was all transshipped, the load balancing module of processing node assisted the overload control module to transship the content of operation of process of drop policy:
(1) the action listener module listens to event or the data processing request that needs processing.
(2) action listener module and load decision-making module are mutual, send the decision instruction request to it.
(3) load decision-making module and load monitoring module are mutual, obtain the pay(useful) load rate information of this each processing node of system.
(4) the load decision-making module is according to each the processing node pay(useful) load rate information that obtains, learn that processing node pay(useful) load rate overload and the underload node set that will mail to are sky, be that each processing node of distributed treatment cluster all transships, the load decision-making module then starts the overload controlling mechanism and carries out overload control, namely carries out probabilistic event and abandons.
(5) the action listener module event of receiving abandons information, event is abandoned, and to client answering system overload response message.
Send in real time in the mass data processing request process to the distributed treatment cluster in client, because of data processing request quantity too huge, thereby cause each processing node in the distributed treatment cluster all to transship, each processing node of distributed treatment group system abandons data processing request with probability; Stability and the reliability of real-time ensuring distributed treatment cluster.
When introducing load transfer and overload control here, the concrete operations content of processing node reorientation method, probabilistic event discarding method and load factor information collecting method.Need to prove: following these method detailed just for example, its objective is in the statement load factor information collecting method, emphasis is the thinking of multi-load factor weighted calculation pay(useful) load rate, the processing node reorientation method then stresses, and probability more big in processing node pay(useful) load rate, that its execution load is shifted is also more big, and the redirected thinking of selecting at random, in the probabilistic event discarding method, emphasis is also more big thinking just of more big in processing node pay(useful) load rate, that it carries out overload control probability.
The content of operation of load factor information collecting method is as follows:
(a) be effective execution that proof load is shifted and overload is controlled, the collection of each processing node pay(useful) load rate information is basis and prerequisite.Each processing node is in the setting-up time period T, the real-time load factor loadCur of collection single treatment node (i, j): loadCur (i, j)=α * cpu (i, j)+β * mem (i, j)+and γ * queLen (i, j), in the formula, loadCur (i, j) be to be numbered the processing node of i in the load factor of j time cycle, (i j) is the utilance that is numbered processing node cpu when j time cycle of i to cpu; (i j) is the utilance that is numbered processing node internal memory when j time cycle of i to mem; (i j) is processing node the data processing request queue length and the maximum ratio that allows queue length when j time cycle that is numbered i to queLen; α, β, γ are respectively the weighting proportion of each load factor factor, and alpha+beta+γ=1.Need to prove that the load factor of considering in the above-mentioned load factor collection is an example, other load factor also is adapted to load factor information collecting method of the present invention.
(b) (i j), obtains the current pay(useful) load rate loadAfter (i of processing node to the real-time load factor loadCur of processing node that obtains according to step (a), j): loadAfter (i, j)=and α * loadCur (i, j)+(1-α) * loadAfter (i, j-1); In the formula, (i is to be numbered the pay(useful) load rate of the processing node of i j time cycle j) to loadAfter; α is weight ratio, and α is more big, in the expression pay(useful) load rate in real time load factor to occupy proportion more big; And 0＜α＜1.
(c) the processing node pay(useful) load rate loadAfter (i that obtains according to step (b), j), can processing node be included in the set of underload processing node, the set of normal load processing node and the set of overload processing node according to load overload threshhold loadMax and the underload threshold values loadMin of default.
Just, if loadAfter (i, j)＜loadMin, then processing node is included into the set of underload processing node; If loadAfter (i, j)〉loadMax, then learn the processing node overload, and processing node be included into the set of overload processing node; Otherwise, be included into the set of normal load processing node.
Processing node of the present invention numbering reorientation method is to be based upon on the basis of load factor information collecting method.According to the load factor information collecting method can obtain each processing node of distributed treatment cluster pay(useful) load rate loadAfter (i, j).Its concrete operations content is:
(a) be the proof load equilibrium, the unbalanced problem of processing node load that solution brings because of the data processing request hot issue, the dynamic load administration module of processing node is in carrying out the redirected operating process of processing node numbering, calculate earlier at j moment processing node i and carry out the probability feasibility probability (i that the processing node numbering is redirected, j): probability (i, j)=random (), in the formula, random () can produce interval decimal of (0,1).
(b) with the probability feasibility probability (i of processing node i, j) with the pay(useful) load rate loadAfter (i of processing node, j) compare, if probability is (i, j)≤loadAfter (i, j), then the order execution in step (c), otherwise, with regard to this end process node serial number reorientation method.
(c) according to the pay(useful) load rate loadAfter (i of each processing node, j), selecting K(K at random from the set of underload processing node is a less natural number of setting in the system) individual processing node, then from a selected K processing node, seek out a processing node of pay(useful) load rate minimum, and give the event distribution module with its numbering as the processing node numbering that will mail to and handle.
Probabilistic event discarding method of the present invention is to be based upon under the prerequisite of load factor information collecting method.According to the load factor information collecting method can obtain each processing node in this system pay(useful) load rate loadAfter (i, j).Its content of operation is:
(a) be the proof load equilibrium, solution causes each processing node of distributed treatment cluster all to transship problem because data processing request quantity is too huge, carrying out probabilistic event at the dynamic load administration module of processing node abandons in the process, calculate earlier at j moment processing node i and carry out the probability feasibility probability (i that the processing node numbering is redirected, j): probability (i, j)=random (); In the formula, random () can produce interval decimal of (0,1).
(b) with the probability feasibility probability of processing node i (i, j) (i j) compares with the pay(useful) load rate loadAfter of processing node, if probability is (i, j)≤and loadAfter (i, j), then the notification event distribution module abandons event; Otherwise, finish the probabilistic event discarding method at this point.
The test of the embodiment of the invention is successful, has realized goal of the invention.
Claims (13)
1. distributed dynamic load management system, it is characterized in that: described system is made up of the distributed treatment cluster that has the dynamic load management function and have a complete reciprocity framework of end-to-end P2P, this Processing Cluster does not have central controlled Centroid, only be provided with a plurality of processing nodes, wherein each processing node can both independently provide the dynamic load managerial ability of distributed arbitration program, and function of authority and dynamic load management is identical separately for it, but the ability of the event of processing is identical or different; Each processing node is the control core of this dynamic load management system, and all is provided with following four functional modules: event processing module, action listener module, event distribution module, dynamic load administration module; The function of each module is as follows:
Event processing module, event for the treatment of some particular type and attribute, produce intermediate treatment result or final result, and the new events that produces is transferred to this processing node or mail to other processing node continuation processing by the event distribution module, or final process result is returned to client; This event processing module is default in this system, or is developed and be deployed on the processing node by the third party programmer;
The action listener module be responsible for to receive the data processing request event from client, and from the event of the processing node of self or other; And accept the dynamic load administrative decision control command of dynamic load administration module: when receiving the load transfer instruction, event is sent to the processing node of appointment in this instruction respectively; When receiving the overload control command, produce the response message of system overload, and send to client; Perhaps event being transferred to event processing module handles again;
The dynamic load administration module, be provided with two parts of load monitoring unit and load decision package, wherein: the load monitoring unit is under the action listener module is assisted, the information of survival condition, pay(useful) load rate and the comprehensive treatment capability of each processing node in collection and the distributed storage Processing Cluster, and real-time update in real time and dummy node mapping table and the dynamic load information table of storage after the load decision package is created or safeguarded; The load decision package is responsible for the information based on survival condition, pay(useful) load rate and the comprehensive treatment capability of each processing node, with and the dummy node mapping table of storage, load balancing, the load that formulate to be used for event handling and event distribution procedure shifted and the dynamic load administrative decision control command of overload control;
The event distribution module, be responsible for accepting the dynamic load administrative decision control command that the dynamic load administration module sends, its new events or final process result of being produced by event processing module that receives is carried out the event distribution processor: when receiving the load balancing instruction, action listener module or other processing node or client that its new events that is produced by event processing module that receives or final process result are sent to this specified processing node of this instruction respectively.
2. distributed dynamic load management system according to claim 1, it is characterized in that: described each processing node is distributed on the different servers; In the actual motion environment, same physical server can arrange and move a plurality of processing nodes.
3. distributed dynamic load management system according to claim 1, it is characterized in that: the dynamic load management function of the load decision package in the described dynamic load administration module is as follows:
Survival condition and the comprehensive treatment capability information of each processing node of distributed treatment cluster that provides according to the load monitoring unit, when system initialization or processing node generation dynamic change, real-time servicing dummy node mapping table and dynamic load information table, adjust the load of each processing node in the distributed treatment cluster, be the dummy node number, so that respectively handle the node load equilibrium;
When the hot issue because of processing and distributing data processing request or event causes that processing node pay(useful) load rate is unbalanced, take the load transition strategy, so that respectively handle the node load equilibrium;
When each processing node pay(useful) load rate is all transshipped, take to transship control strategy, so that the distributed treatment cluster keeps the stability of a system and robustness.
4. distributed dynamic load management system according to claim 3, it is characterized in that: described processing node pay(useful) load rate is the pay(useful) load rate of the last load information collection period of this processing node and the value after the real-time load factor weighting in this cycle, the actual negative carrying capacity that described real-time load factor is processing node and the ratio of its comprehensive treatment capability, wherein, the actual negative carrying capacity of processing node comprises the consumption of CPU and internal memory, or the length of pending event queue; The comprehensive treatment capability of processing node comprises the spatial content of cpu function, internal memory or the maximum permissible value of pending event queue length; Utilize the pay(useful) load rate of processing node, can effectively handle the isomerism problem in the distributed system, and processing node is divided into following three ranks according to the pay(useful) load rate: underload processing node, normal load processing node and overload processing node;
Described dummy node be non-necessary being and with the logical symbol of processing node mutual mapping, each dummy node can only be mapped to a processing node, but each processing node is mapped to a plurality of dummy nodes; The configuration when quantity of dummy node starts in system, its quantity is much larger than the number of processing node; And after system's operation, the quantity of dummy node remains unchanged, and when the processing node in the system increased or shifts out, the mapping relations between processing node and dummy node were carried out corresponding change according to distributed dynamic load management method;
Described dummy node mapping table is a kind of data structure that this system realizes the distributed dynamic load management: the mapping relations that are used for each dummy node symbol of storage and its processing node numbering, each processing node has dummy node mapping table separately, system realizes distributed load equalizing by searching the processing node that the dummy node mapping table can search out its correspondence mappings from dummy node;
Described dynamic load information table is the another kind of data structure that this system realizes the distributed dynamic load management: be used for dummy node symbol and the number thereof of each processing node management of storage and control, the information in this dynamic load information table and the information of dummy node mapping table are one to one.
5. distributed dynamic load management system according to claim 1, it is characterized in that: the application scenarios of this system is as follows:
The Distribution Events scene: client sends the mass data processing request in real time to the distributed treatment cluster, in the data processing request transmitting-receiving just often, this cluster returns result to client in real time, and guarantees load balancing and the stability of each processing node in this cluster; And in the dynamic load management process, each processing node equal, i.e. no centralized load management node in this cluster, each processing node can both independently be finished the dynamic load management of distributed arbitration program;
The scene of processing node initialization and dynamic change: when client sends the mass data processing request in real time to the distributed treatment cluster, real-time start-up system or increase new processing node or shift out processing node to this cluster; And in short time as far as possible, adjust its load in proportion according to the comprehensive treatment capability of each processing node again, make that each processing node in this cluster reaches load balancing again;
Scene is shifted in the processing node load: client sends in the mass data processing request process in real time to the distributed treatment cluster, because there is hot issue in data processing request, cause receiving the pay(useful) load rate overload of processing of request node, and when in the set of underload processing node, also having processing node, this cluster with probability be redirected data processing request and mail to certain or some processing node in the underload processing node set, in order to dynamically adjust the load balancing of this cluster in real time;
Processing node overload control scene: client sends the mass data processing request in real time to the distributed treatment cluster, because of data processing request quantity too huge, when causing each processing node pay(useful) load rate in this cluster all to be transshipped, each processing node in this cluster with probability abandon data processing request, and to the response message of client answering system overload, so that the stability of this cluster of real-time ensuring and reliability.
6. one kind is adopted the described distributed dynamic load management system of claim 1 based on the distributed dynamic load management method of P2P technology, it is characterized in that: when client sends the mass data processing request in real time to this distributed treatment cluster, after certain processing node is received request, whether earlier judge according to the pay(useful) load rate of this node and overloading threshold that this processing node is current transships, if overload is then carried out load transfer or overload control; Otherwise, handle this request; After the request finished is handled, just the intermediate object program that produces is encapsulated as event, and carries out load-balancing decision according to dummy node mapping table and this event determinant attribute value, determine this event still to mail to other processing node by self node and continue processing; And follow-up each processing node is when handling and distribute intermediate event, all be according to pay(useful) load rate and overloading threshold or dummy node mapping table and event determinant attribute value execution load transfer, overload control or the balanced operation of distributing of event, up to the final result of generation, and return to client.
7. distributed dynamic load management method according to claim 6, it is characterized in that: described method comprises following operating procedure:
Step 1, the dynamic load administration module of processing node are created or renewal dummy node mapping table and dynamic load information table; Simultaneously, the dynamic load administration module is gathered existing state and the real-time load information of each processing node in the distributed type assemblies according to setting cycle under the action listener module is assisted;
Step 2, in a single day the action listener module of processing node listens to the event that needs processing, by load decision package in the dynamic load administration module according to the dummy node mapping table, and the pay(useful) load rate information of each processing node of providing of load monitoring unit is carried out the dynamic load administrative decision, and formulation dynamic load administrative decision control command, the action listener module carries out according to this control command that load is shifted, overload control, or event is sent to event processing module handles;
After step 3, event processing module are finished processing to this event, produce intermediate event or final process result, send to the event distribution module; Carry out the dynamic load administrative decision by load decision package in the dynamic load administration module according to dummy node mapping table, intermediate event determinant attribute value, and formulation dynamic load administrative decision control command, the event distribution module is according to this instruction execution event distribution operation, intermediate event or final process result are distributed to corresponding processing node or client, and realize respectively handling the node load equilibrium.
8. distributed dynamic load management method according to claim 6, it is characterized in that: described step 1 comprises following content of operation:
(11) system initially sets up, when creating dummy node mapping table and dynamic load information table, and the method initialization dummy node mapping table that adopts the WRR that is directly proportional according to comprehensive treatment capability or Weighted random to distribute; Perhaps adopt Random assignment and guarantee equally distributed method initialization dummy node mapping table; At this moment, the dummy node number of distributing for each different processing node of processing event ability should be differentiated;
(12) when increasing in this system or shifting out processing node, reach load balancing as early as possible again for making system, the dynamic load administration module will carry out real-time update to dummy node mapping table and dynamic load information table;
(13) after system starts operation, load monitoring unit in the dynamic load administration module is under the action listener module is assisted, gather existing state and the real-time load information of each processing node in the distributed type assemblies according to setting cycle, and calculate its pay(useful) load rate according to the real-time load information of processing node, and processing node is included into respectively in the set of underload processing node, the set of normal load processing node or the set of overload processing node.
9. distributed dynamic load management method according to claim 8, it is characterized in that: described step (12) comprises following content of operation:
When (12a) the dynamic load administration module of each processing node detects processing node in this system change is arranged, this situation is informed all processing nodes in this cluster;
(12b) for reduce the system topological maintenance workload as far as possible, the dynamic load administration module of each processing node adopts consistency hash method real-time update dynamic load information table and dummy node mapping table according to the initialization of processing node, the modification information that increases or shift out.
10. distributed dynamic load management method according to claim 8, it is characterized in that: described step (13) comprises following content of operation:
(13a) the load monitoring unit is under the action listener module is assisted, and the cycle is gathered the real-time load information of each processing node; And in order to solve the isomerism of processing node, the real-time load information of processing node all is normalized to real-time load factor, i.e. the real-time load factor of calculating this this cycle of processing node according to real-time load information and the comprehensive treatment capability thereof of each processing node;
(13b) the load monitoring unit is according to the pay(useful) load rate of last one-period and the real-time load factor in this cycle, adopt weighting scheme to calculate the pay(useful) load rate in this this cycle of processing node, and this processing node pay(useful) load rate score that will obtain is stored in the load monitoring unit of dynamic load administration module;
(13c) the load monitoring unit compares judgement with the pay(useful) load rate score of this processing node and the load overloading threshold of default earlier, if the pay(useful) load rate score is greater than the load overloading threshold, then this processing node is included in the set of overload processing node, finishes this step operation; Otherwise namely the pay(useful) load rate score is not more than the load overloading threshold, then carries out subsequent step (13d);
(13d) the load monitoring unit compares judgement with the pay(useful) load rate score of this processing node and the underload threshold value of default again, if the pay(useful) load rate less than the underload threshold value, then is included into this processing node in the set of underload processing node; Otherwise namely the pay(useful) load rate score is not more than the load overloading threshold, just this processing node is included in the set of normal load processing node.
11. distributed dynamic load management method according to claim 7, it is characterized in that: described step 2 comprises following content of operation:
(21) after in a single day the action listener module listens to the event that needs to handle, load decision package in the dynamic load administration module and load monitoring unit carry out alternately, know the pay(useful) load rate score of each processing node, and whether the pay(useful) load rate of judging this processing node transships: if it does not transship, then carry out subsequent step (22); If its overload, then redirect execution in step (23);
(22) the load decision package need not to carry out load transfer or overload control strategy, directly gives event processing module with this event by the action listener module and handles, and finishes this step operation;
(23) the load decision package judges whether other processing node all transships, if, then take probabilistic overload drop policy to abandon this event: namely the processing node pay(useful) load rate that makes preparation mail to is more high, then its probability of carrying out the operation of overload drop policy is more big, to realize overload control, the working stability of real-time ensuring distributed treatment cluster and reliable; If not, namely also there is the lighter processing node of effective load factor, then takes probability to be redirected the processing node of this event, shift to realize load; And when transferring load, do not change the mapping relations of dummy node and processing node, just probability is reselected the processing node numbering of this event, and this event handled without event processing module just directly mail to new processing node, namely the processing node numbering is redirected according to the probability that is directly proportional with this processing node pay(useful) load rate, and make that this processing node pay(useful) load rate is more high, the probability that this processing node is carried out event re-orientation processes nodal operation is more big.
12. distributed dynamic load management method according to claim 11 is characterized in that: comprise following content of operation in the described step (23):
(23a) the load decision package judges whether other processing node all transships, if then carry out subsequent step (23b); If not, namely also there is the lighter processing node of effective load factor, then redirect execution in step (23c);
(23b) the load decision package produces the decimal at random in (0, a 1) scope, and with this at random decimal and this processing node pay(useful) load rate score compare judgement: this at random decimal whether less than this processing node pay(useful) load rate score; If then the load decision package sends overload to the action listener module and abandons instruction, the action listener module directly abandons this event, and the response message of answering system overload, under the event distribution module is assisted this response message is sent to client then; If not, then carry out subsequent step (23e);
(23c) the load decision package produces the decimal at random in (0, a 1) scope, and with this at random decimal and this processing node pay(useful) load rate score compare judgement: this at random decimal whether less than this processing node pay(useful) load rate score; If then carry out subsequent step (23d); If not, then redirect execution in step (23e);
(23d) the load decision package takes out one or more processing nodes in the load node set on the lenient side at random, and one or more processing nodes of therefrom seeking out pay(useful) load rate score minimum are numbered the destination of being redirected as this event, then, this processing node numbering and load transfer instruction are returned to the action listener module; The action listener module is under the assistance of event distribution module, and the processing node that directly this event is mail to this processing node numbering representative is handled, in order to dynamically adjust the load balancing of distributed treatment cluster in real time;
(23e) the action listener module is directly given event processing module with this event and is handled.
13. distributed dynamic load management method according to claim 7, it is characterized in that: described step 3 comprises following content of operation:
(31) event processing module is finished dealing with after this event, the intermediate event that produces is carried out consistency Hash evaluate operation according to its determinant attribute value, and mould is got the dummy node number of this system, obtaining preparation with the dummy node numbering of this intermediate event distribution, in order to this intermediate event is mail to self or other processing node continues to handle;
(32) the event distribution module carries out alternately with the dynamic load administration module according to the dummy node numbering of the preparation distribution that obtains, and obtains the processing node numbering that this intermediate event will mail to according to the dummy node mapping table in the dynamic load administration module;
(33) the event distribution module processing node number information that will mail to according to this intermediate event mails to the respective handling node with it and continues to handle, and up to the event of generation final process result, directly returns to client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310065185.XA CN103188345B (en) | 2013-03-01 | 2013-03-01 | Distributed dynamic load management system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310065185.XA CN103188345B (en) | 2013-03-01 | 2013-03-01 | Distributed dynamic load management system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103188345A true CN103188345A (en) | 2013-07-03 |
| CN103188345B CN103188345B (en) | 2016-05-18 |
Family
ID=48679307
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310065185.XA Expired - Fee Related CN103188345B (en) | 2013-03-01 | 2013-03-01 | Distributed dynamic load management system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103188345B (en) |
Cited By (36)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103716381A (en) * | 2013-12-12 | 2014-04-09 | 华为技术有限公司 | Control method and management node of distributed system |
| CN103746838A (en) * | 2013-12-26 | 2014-04-23 | 华北计算技术研究所 | Task scheduling method of computer network without center node |
| CN103945005A (en) * | 2014-05-06 | 2014-07-23 | 江苏物联网研究发展中心 | Multiple evaluation indexes based dynamic load balancing framework |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN104519125A (en) * | 2013-09-27 | 2015-04-15 | 英特尔公司 | Distributed in-order load spreading resilient to topology changes |
| CN104572307A (en) * | 2015-01-30 | 2015-04-29 | 无锡华云数据技术服务有限公司 | Method for flexibly scheduling virtual resources |
| WO2015090248A1 (en) * | 2013-12-16 | 2015-06-25 | 腾讯科技（深圳）有限公司 | Server overload protection method and device |
| CN104951427A (en) * | 2015-06-30 | 2015-09-30 | 深圳清华大学研究院 | Distributed computation framework with matrix as center |
| CN106209665A (en) * | 2015-04-30 | 2016-12-07 | 中兴通讯股份有限公司 | A kind of realize the method for load sharing, interface message processor (IMP), service processor and system |
| CN106302628A (en) * | 2015-12-29 | 2017-01-04 | 北京典赞科技有限公司 | ARM architecture network cluster calculates the unified management dispatching method of resource |
| CN106559448A (en) * | 2015-09-28 | 2017-04-05 | 北京国双科技有限公司 | Server load balancing method and apparatus |
| CN106610870A (en) * | 2016-12-28 | 2017-05-03 | 北京奇艺世纪科技有限公司 | Method and device for adjusting quantity of processing nodes |
| CN106797485A (en) * | 2014-10-02 | 2017-05-31 | 恩特里克丝有限公司 | High in the clouds stream media service system, high in the clouds stream media service method and its device using optimal GPU |
| CN107040476A (en) * | 2017-03-10 | 2017-08-11 | 浙江工业大学 | A kind of dynamic towards real-time stream calculation counter-pressure method step by step |
| CN107341055A (en) * | 2017-07-03 | 2017-11-10 | 杭州知物数据科技有限公司 | The real-time dynamic management approach and system of distributed memory file system |
| WO2018000991A1 (en) * | 2016-06-30 | 2018-01-04 | 华为技术有限公司 | Data balancing method and device |
| CN107733961A (en) * | 2016-10-19 | 2018-02-23 | 北京奥斯达兴业科技有限公司 | Resource allocation methods and device |
| CN108282516A (en) * | 2017-12-21 | 2018-07-13 | 创新科存储技术（深圳）有限公司 | A kind of distributed storage cluster load balancing method and device based on iSCSI |
| CN108829511A (en) * | 2018-05-07 | 2018-11-16 | 中山大学 | Load balancing adjusting method based on state machine replica management model |
| CN109218341A (en) * | 2017-06-29 | 2019-01-15 | 北京京东尚科信息技术有限公司 | For monitoring server and for the load-balancing method and device of server |
| CN109408219A (en) * | 2017-08-16 | 2019-03-01 | 中国电信股份有限公司 | Distributed data method of reseptance, system and be used for the received device of distributed data |
| CN109510855A (en) * | 2017-09-15 | 2019-03-22 | 腾讯科技（深圳）有限公司 | Event distribution system, method and device |
| CN109903178A (en) * | 2019-04-04 | 2019-06-18 | 腾讯科技（深圳）有限公司 | It is a kind of to determine the method, apparatus, system of social object jointly and calculate equipment |
| CN109951394A (en) * | 2019-03-11 | 2019-06-28 | 中国银联股份有限公司 | A kind of method and device of the dummy node quantity of determining transaction system |
| CN110049084A (en) * | 2018-12-10 | 2019-07-23 | 阿里巴巴集团控股有限公司 | Current-limiting method, device and the equipment of distributed system |
| CN110120973A (en) * | 2019-04-28 | 2019-08-13 | 华为技术有限公司 | A kind of request control method, relevant device and computer storage medium |
| CN110213351A (en) * | 2019-05-17 | 2019-09-06 | 北京航空航天大学 | A kind of dynamic self-adapting I/O load equalization methods towards wide area high-performance computing environment |
| CN110855738A (en) * | 2019-09-25 | 2020-02-28 | 安徽四创电子股份有限公司 | Communication processing system for multi-source equipment |
| CN110908590A (en) * | 2018-09-17 | 2020-03-24 | 中国电力科学研究院有限公司 | Distributed storage method and system for transformer substation data |
| CN111338793A (en) * | 2020-02-14 | 2020-06-26 | 拉扎斯网络科技（上海）有限公司 | Cluster task distribution method, device, equipment and storage medium |
| CN111339196A (en) * | 2020-02-24 | 2020-06-26 | 深圳点链科技有限公司 | Data processing method and system based on block chain and computer readable storage medium |
| CN111522649A (en) * | 2019-02-01 | 2020-08-11 | 北京京东尚科信息技术有限公司 | Distributed task allocation method, device and system |
| US10901805B2 (en) | 2018-12-18 | 2021-01-26 | At&T Intellectual Property I, L.P. | Distributed load balancing for processing of high-volume data streams |
| US11418417B2 (en) | 2020-12-21 | 2022-08-16 | Red Hat, Inc. | Managing stateful workloads executing on temporarily available resources of a cloud computing system |
| CN115587222A (en) * | 2022-12-12 | 2023-01-10 | 阿里巴巴（中国）有限公司 | Distributed graph calculation method, system and equipment |
| CN115766737A (en) * | 2023-01-09 | 2023-03-07 | 北京微吼时代科技有限公司 | Load balancing method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102457428A (en) * | 2010-10-27 | 2012-05-16 | 中兴通讯股份有限公司 | Load balancing realization method and device for DHT (distributed Hash table) network |
| CN102457429A (en) * | 2010-10-27 | 2012-05-16 | 中兴通讯股份有限公司 | Method and device for realizing load balance of DHT (Distributed Hash Table) network |
| CN102882973A (en) * | 2012-10-11 | 2013-01-16 | 北京邮电大学 | Distributed load balancing system and distributed load balancing method based on peer to peer (P2P) technology |
-
2013
- 2013-03-01 CN CN201310065185.XA patent/CN103188345B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102457428A (en) * | 2010-10-27 | 2012-05-16 | 中兴通讯股份有限公司 | Load balancing realization method and device for DHT (distributed Hash table) network |
| CN102457429A (en) * | 2010-10-27 | 2012-05-16 | 中兴通讯股份有限公司 | Method and device for realizing load balance of DHT (Distributed Hash Table) network |
| CN102882973A (en) * | 2012-10-11 | 2013-01-16 | 北京邮电大学 | Distributed load balancing system and distributed load balancing method based on peer to peer (P2P) technology |
Cited By (57)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104519125A (en) * | 2013-09-27 | 2015-04-15 | 英特尔公司 | Distributed in-order load spreading resilient to topology changes |
| CN104519125B (en) * | 2013-09-27 | 2019-05-07 | 英特尔公司 | Distributed load distribution in order flexible for change in topology |
| CN103716381A (en) * | 2013-12-12 | 2014-04-09 | 华为技术有限公司 | Control method and management node of distributed system |
| CN103716381B (en) * | 2013-12-12 | 2017-04-12 | 华为技术有限公司 | Control method and management node of distributed system |
| WO2015090248A1 (en) * | 2013-12-16 | 2015-06-25 | 腾讯科技（深圳）有限公司 | Server overload protection method and device |
| CN103746838B (en) * | 2013-12-26 | 2016-12-07 | 华北计算技术研究所 | A kind of method for scheduling task of non-stop layer node computer network |
| CN103746838A (en) * | 2013-12-26 | 2014-04-23 | 华北计算技术研究所 | Task scheduling method of computer network without center node |
| CN103945005B (en) * | 2014-05-06 | 2017-06-13 | 山西睿信智达传媒科技股份有限公司 | Dynamic load leveling framework based on many evaluation indexes |
| CN103945005A (en) * | 2014-05-06 | 2014-07-23 | 江苏物联网研究发展中心 | Multiple evaluation indexes based dynamic load balancing framework |
| CN106797485B (en) * | 2014-10-02 | 2019-10-25 | Sk科技有限公司 | Cloud stream media service system, cloud stream media service method and its device using best GPU |
| CN106797485A (en) * | 2014-10-02 | 2017-05-31 | 恩特里克丝有限公司 | High in the clouds stream media service system, high in the clouds stream media service method and its device using optimal GPU |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN104486445B (en) * | 2014-12-30 | 2017-03-22 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system based on cloud platform |
| CN104572307A (en) * | 2015-01-30 | 2015-04-29 | 无锡华云数据技术服务有限公司 | Method for flexibly scheduling virtual resources |
| CN104572307B (en) * | 2015-01-30 | 2019-03-05 | 无锡华云数据技术服务有限公司 | The method that a kind of pair of virtual resource carries out flexible scheduling |
| CN106209665A (en) * | 2015-04-30 | 2016-12-07 | 中兴通讯股份有限公司 | A kind of realize the method for load sharing, interface message processor (IMP), service processor and system |
| CN104951427A (en) * | 2015-06-30 | 2015-09-30 | 深圳清华大学研究院 | Distributed computation framework with matrix as center |
| CN106559448A (en) * | 2015-09-28 | 2017-04-05 | 北京国双科技有限公司 | Server load balancing method and apparatus |
| CN106302628A (en) * | 2015-12-29 | 2017-01-04 | 北京典赞科技有限公司 | ARM architecture network cluster calculates the unified management dispatching method of resource |
| CN106302628B (en) * | 2015-12-29 | 2019-12-27 | 北京典赞科技有限公司 | Unified management scheduling method for computing resources in ARM architecture network cluster |
| WO2018000991A1 (en) * | 2016-06-30 | 2018-01-04 | 华为技术有限公司 | Data balancing method and device |
| CN107733961A (en) * | 2016-10-19 | 2018-02-23 | 北京奥斯达兴业科技有限公司 | Resource allocation methods and device |
| CN107733961B (en) * | 2016-10-19 | 2021-06-22 | 北京奥斯达兴业科技有限公司 | Resource allocation method and device |
| CN106610870A (en) * | 2016-12-28 | 2017-05-03 | 北京奇艺世纪科技有限公司 | Method and device for adjusting quantity of processing nodes |
| CN106610870B (en) * | 2016-12-28 | 2019-10-15 | 北京奇艺世纪科技有限公司 | A kind of processing number of nodes method of adjustment and device |
| CN107040476B (en) * | 2017-03-10 | 2020-05-05 | 浙江工业大学 | Dynamic step-by-step back pressure method for real-time stream calculation |
| CN107040476A (en) * | 2017-03-10 | 2017-08-11 | 浙江工业大学 | A kind of dynamic towards real-time stream calculation counter-pressure method step by step |
| CN109218341A (en) * | 2017-06-29 | 2019-01-15 | 北京京东尚科信息技术有限公司 | For monitoring server and for the load-balancing method and device of server |
| CN109218341B (en) * | 2017-06-29 | 2022-02-25 | 北京京东尚科信息技术有限公司 | Load balancing method and device for monitoring server and server |
| CN107341055A (en) * | 2017-07-03 | 2017-11-10 | 杭州知物数据科技有限公司 | The real-time dynamic management approach and system of distributed memory file system |
| CN109408219A (en) * | 2017-08-16 | 2019-03-01 | 中国电信股份有限公司 | Distributed data method of reseptance, system and be used for the received device of distributed data |
| CN109408219B (en) * | 2017-08-16 | 2021-04-02 | 中国电信股份有限公司 | Distributed data receiving method, system and device for distributed data receiving |
| CN109510855A (en) * | 2017-09-15 | 2019-03-22 | 腾讯科技（深圳）有限公司 | Event distribution system, method and device |
| CN109510855B (en) * | 2017-09-15 | 2020-07-28 | 腾讯科技（深圳）有限公司 | Event distribution system, method and device |
| CN108282516B (en) * | 2017-12-21 | 2021-09-10 | 深圳创新科技术有限公司 | Distributed storage cluster load balancing method and device based on iSCSI |
| CN108282516A (en) * | 2017-12-21 | 2018-07-13 | 创新科存储技术（深圳）有限公司 | A kind of distributed storage cluster load balancing method and device based on iSCSI |
| CN108829511A (en) * | 2018-05-07 | 2018-11-16 | 中山大学 | Load balancing adjusting method based on state machine replica management model |
| CN110908590A (en) * | 2018-09-17 | 2020-03-24 | 中国电力科学研究院有限公司 | Distributed storage method and system for transformer substation data |
| CN110049084B (en) * | 2018-12-10 | 2021-08-24 | 创新先进技术有限公司 | Current limiting method, device and equipment of distributed system |
| CN110049084A (en) * | 2018-12-10 | 2019-07-23 | 阿里巴巴集团控股有限公司 | Current-limiting method, device and the equipment of distributed system |
| US10901805B2 (en) | 2018-12-18 | 2021-01-26 | At&T Intellectual Property I, L.P. | Distributed load balancing for processing of high-volume data streams |
| US11416308B2 (en) | 2018-12-18 | 2022-08-16 | At&T Intellectual Property I, L.P. | Distributed load balancing for processing of high-volume data streams |
| CN111522649A (en) * | 2019-02-01 | 2020-08-11 | 北京京东尚科信息技术有限公司 | Distributed task allocation method, device and system |
| CN109951394B (en) * | 2019-03-11 | 2021-04-27 | 中国银联股份有限公司 | Method and device for determining number of virtual nodes of transaction system |
| CN109951394A (en) * | 2019-03-11 | 2019-06-28 | 中国银联股份有限公司 | A kind of method and device of the dummy node quantity of determining transaction system |
| CN109903178A (en) * | 2019-04-04 | 2019-06-18 | 腾讯科技（深圳）有限公司 | It is a kind of to determine the method, apparatus, system of social object jointly and calculate equipment |
| CN109903178B (en) * | 2019-04-04 | 2021-08-20 | 腾讯科技（深圳）有限公司 | Method, device and system for determining common social contact object and computing equipment |
| CN110120973A (en) * | 2019-04-28 | 2019-08-13 | 华为技术有限公司 | A kind of request control method, relevant device and computer storage medium |
| CN110213351A (en) * | 2019-05-17 | 2019-09-06 | 北京航空航天大学 | A kind of dynamic self-adapting I/O load equalization methods towards wide area high-performance computing environment |
| CN110855738A (en) * | 2019-09-25 | 2020-02-28 | 安徽四创电子股份有限公司 | Communication processing system for multi-source equipment |
| CN110855738B (en) * | 2019-09-25 | 2022-06-07 | 安徽四创电子股份有限公司 | Communication processing system for multi-source equipment |
| CN111338793A (en) * | 2020-02-14 | 2020-06-26 | 拉扎斯网络科技（上海）有限公司 | Cluster task distribution method, device, equipment and storage medium |
| CN111339196B (en) * | 2020-02-24 | 2022-02-25 | 深圳点链科技有限公司 | Data processing method and system based on block chain and computer readable storage medium |
| CN111339196A (en) * | 2020-02-24 | 2020-06-26 | 深圳点链科技有限公司 | Data processing method and system based on block chain and computer readable storage medium |
| US11418417B2 (en) | 2020-12-21 | 2022-08-16 | Red Hat, Inc. | Managing stateful workloads executing on temporarily available resources of a cloud computing system |
| CN115587222A (en) * | 2022-12-12 | 2023-01-10 | 阿里巴巴（中国）有限公司 | Distributed graph calculation method, system and equipment |
| CN115766737A (en) * | 2023-01-09 | 2023-03-07 | 北京微吼时代科技有限公司 | Load balancing method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103188345B (en) | 2016-05-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103188345A (en) | Distributive dynamic load management system and distributive dynamic load management method | |
| CN109617992B (en) | Block chain-based dynamic election method for edge computing nodes | |
| CN102882973B (en) | Distributed load balancing system and distributed load balancing method based on peer to peer (P2P) technology | |
| CN102244685B (en) | Distributed type dynamic cache expanding method and system for supporting load balancing | |
| CN103368864A (en) | Intelligent load balancing method based on c/s (Client/Server) architecture | |
| CN104461740A (en) | Cross-domain colony computing resource gathering and distributing method | |
| CN110109756A (en) | A kind of network target range construction method, system and storage medium | |
| CN102164184A (en) | Computer entity access and management method for cloud computing network and cloud computing network | |
| CN102622275A (en) | Load balancing realization method in cloud computing environment | |
| CN102110014A (en) | Method for balancing loads of virtual machine (VM) | |
| CN103441918A (en) | Self-organizing cluster server system and self-organizing method thereof | |
| CN104239555A (en) | MPP (massively parallel processing)-based parallel data mining framework and MPP-based parallel data mining method | |
| WO2020134133A1 (en) | Resource allocation method, substation, and computer-readable storage medium | |
| CN104580503A (en) | Efficient dynamic load balancing system and method for processing large-scale data | |
| CN107786460A (en) | A kind of management of electricity transaction system request and current-limiting method based on token bucket algorithm | |
| CN109960586A (en) | A kind of appreciable four-layer load-equalizing device of server state and equalization methods | |
| WO2021120633A1 (en) | Load balancing method and related device | |
| CN108322525A (en) | A kind of industry multi-core network establishing method | |
| CN111404818A (en) | Routing protocol optimization method for general multi-core network processor | |
| CN116684418B (en) | Calculation power arrangement scheduling method, calculation power network and device based on calculation power service gateway | |
| CN105337832A (en) | Resource distribution method for on-line multicast virtual network | |
| Wang et al. | Research on load balancing technology for microservice architecture | |
| CN101778135A (en) | Server system of rural general message platform and operation method thereof | |
| Xu et al. | A service redundancy strategy and ant colony optimization algorithm for multiservice fog nodes | |
| CN111092821A (en) | SDN network load balancing method based on dynamic migration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160518 Termination date: 20170301 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |