CN106484528A

CN106484528A - It is used in Distributed Architecture realizing the method and device of cluster dynamic retractility

Info

Publication number: CN106484528A
Application number: CN201610809555.XA
Authority: CN
Inventors: 周恺; 王倩; 肖远昊; 王家兴; 张发恩
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2016-09-07
Filing date: 2016-09-07
Publication date: 2017-03-08
Anticipated expiration: 2036-09-07
Also published as: CN106484528B

Abstract

The invention provides being used in Distributed Architecture realizing method and the device of cluster dynamic retractility, method therein includes：Determine there is cluster dynamic retractility demand, wherein, described cluster includes：Multiple from nodes, and the plurality of from node is divided into computing resource group and storage resource group according to the resource service that it provides；From node quantity in described computing resource group and/or storage resource group is adjusted according to described cluster dynamic retractility demand.The technique scheme that the present invention provides can preferably make cluster scale and disposal ability with practical application scene, the demand of storage resource and computing resource be mated, thus effectively avoiding the phenomenon that cluster resource deficiency and cluster resource waste, and then while improve the motility of cluster dynamic retractility, improve the performance of cluster.

Description

It is used in Distributed Architecture realizing the method and device of cluster dynamic retractility

Technical field

The present invention relates to network technology, especially relate to be used in a kind of Distributed Architecture realizing the side of cluster dynamic retractility It is used in method and Distributed Architecture realizing the device of cluster dynamic retractility.

Background technology

In distributed technical field, because the system bottom details of Hadoop is transparent and to become current application relatively broad Distributed Architecture.Cluster (i.e. Hadoop cluster) based on Hadoop stretches and typically refers to the tune of cluster scale and disposal ability Whole.

At present, the implementation method that Hadoop cluster stretches is usually：Hadoop cluster needs increase new from node When, the server first this from node being located configures, and then, interrupts the service in each node in Hadoop cluster, and After each node in notifying Hadoop cluster increased new from node, start the service in each node of Hadoop cluster, So that new from node adds Hadoop cluster；When needing to reduce from node in Hadoop cluster, interrupt Hadoop cluster In each node in service, and after the from node that is contracted by of each node in notifying Hadoop cluster, start Hadoop collection Service in each node of group, so that the from node being contracted by exits Hadoop cluster.

Inventor finds in realizing process of the present invention, and the mode of existing adjustment cluster scale and disposal ability is due to needing Interrupt the service in each node of Hadoop cluster, and arrange the operation such as server needing artificial treatment, so that cluster is stretched The cost of implementation of contracting is higher, and intelligence degree is relatively low.In addition, adjustment after Hadoop cluster be difficult to practical application scene in The demand of storage resource and computing resource is matched, such as Hadoop cluster scale and disposal ability were typically according to peak period The demand of cluster resource is arranged, therefore, necessarily occurs, in offpeak period, the phenomenon that cluster resource wastes.

Content of the invention

It is an object of the invention to provide being used in a kind of Distributed Architecture realizing the method and device of cluster dynamic retractility.

According to an aspect of the present invention, a kind of method being used in Distributed Architecture realizing cluster dynamic retractility is provided, And methods described mainly includes the following steps that：Determine there is cluster dynamic retractility demand, wherein, described cluster includes：Multiple From node, and the plurality of from node is divided into computing resource group and storage resource group according to the resource service that it provides；Root Adjust the from node quantity in described computing resource group and/or storage resource group according to described cluster dynamic retractility demand.

According to another aspect of the present invention, provide and be used in a kind of Distributed Architecture realizing the dress of cluster dynamic retractility Put, including：For determining the device that there is cluster dynamic retractility demand, wherein, described cluster includes：Multiple from nodes, and The plurality of from node is divided into computing resource group and storage resource group according to the resource service that it provides；For according to described Cluster dynamic retractility demand adjusts the device of the from node quantity in described computing resource group and/or storage resource group.

Compared with prior art, the present invention has advantages below：The present invention pass through by the multiple from nodes in cluster according to Resource service (i.e. computing resource service, storage resource service) that it is each provided and be divided in computing resource group and storage money In the group of source, so that the resource service that the from node in computing resource group and storage resource group is provided is had differences, so, calculate money From node in the group of source can be the from node of the light storage of re-computation, and the from node in storage resource group can attach most importance to storage gently The from node calculating, thus the present invention is when cluster stretches, can be targetedly to computing resource group and/or storage resource group Stretched, and then the present invention can be made the adjustment of cluster scale and disposal ability more targeted, makes cluster scale and place Reason ability preferably can be mated to the demand of storage resource and computing resource with practical application scene, such as in peak period, Cluster can provide the computing resource of abundance in time, and in low-valley interval, cluster can discharge unnecessary computing resource etc. in time； It follows that the technical scheme that the present invention provides can effectively avoid cluster resource not enough and cluster resource wastes shows As while improve the motility of cluster dynamic retractility, improve the performance of cluster.

Brief description

By reading the detailed description that non-limiting example is made made with reference to the following drawings, other of the present invention Feature, objects and advantages will become more apparent upon：

Fig. 1 is for being used in the Distributed Architecture of the embodiment of the present invention one realizing the method flow diagram of cluster dynamic retractility；

Fig. 2 is that the utilization being used for realizing in the method for cluster dynamic retractility in the Distributed Architecture of the embodiment of the present invention two is empty The flow chart that a specific example of Hadoop cluster built by plan machine；

Fig. 3 is for being used in the Distributed Architecture of the embodiment of the present invention two realizing the Hadoop in the method for cluster dynamic retractility The flow chart of one specific example of the dilatation of cluster；

Fig. 4 is for being used in the Distributed Architecture of the embodiment of the present invention two realizing the Hadoop in the method for cluster dynamic retractility The flow chart of one specific example of the capacity reducing of cluster；

Fig. 5 is used for realizing cluster dynamic retractility in the remotely control node execution Distributed Architecture of the embodiment of the present invention three One of method specific example flow chart；

Fig. 6 is for being used in the Distributed Architecture of the embodiment of the present invention four realizing the schematic device of cluster dynamic retractility；

Fig. 7 is the schematic diagram of a specific example of determination demand device of the embodiment of the present invention four；

Fig. 8 is the schematic diagram of a specific example of adjustresources group device of the embodiment of the present invention four；

Fig. 9 is the schematic diagram of another specific example of adjustresources group device of the embodiment of the present invention four；

Figure 10 is the schematic diagram of a specific example of register device of the embodiment of the present invention four；

Figure 11 is the schematic diagram of another specific example of adjustresources group device of the embodiment of the present invention four.

In accompanying drawing, same or analogous reference represents same or analogous part.

Specific embodiment

It should should be mentioned that before exemplary embodiment is discussed in greater detail, some exemplary embodiments are described Become the process described as flow chart or method.Although operations are described as the process of order by flow chart, therein Many operations can be implemented concurrently, concomitantly or simultaneously.Additionally, the execution sequence of operations can be pacified again Row.Described process can be terminated when its operations are completed, it is also possible to have the additional step being not included in accompanying drawing.Institute State process and can correspond to method, function, code, subroutine, subprogram etc..

Alleged within a context " computer equipment ", also referred to as " computer ", refer to by running preset program or to refer to Order executing the intelligent electronic device of the predetermined process process such as numerical computations and/or logical calculated, its can include processor with Memorizer, the survival being prestored in memory by computing device instructs and to execute predetermined process process, or by ASIC, The hardware such as FPGA, DSP execute predetermined process process, or combine to realize by said two devices.Computer equipment include but not It is limited to server, PC and notebook computer etc..

Described computer equipment includes user equipment and the network equipment.Wherein, described user equipment includes but is not limited to electricity Brain, smart mobile phone, PDA etc.；The described network equipment includes but is not limited to single network server, multiple webserver forms Server group or the cloud being made up of a large amount of computers or the webserver Ji Yu cloud computing (Cloud Computing), wherein, Cloud computing is one kind of Distributed Calculation, a super virtual computer being made up of a group loosely-coupled computer collection.Its In, described computer equipment can isolated operation realizing the present invention, also can access network and by with network in other calculating The interactive operation of machine equipment is realizing the present invention.Wherein, the network residing for described computer equipment include but is not limited to the Internet, Wide area network, Metropolitan Area Network (MAN), LAN, VPN etc..

It should be noted that described user equipment, the network equipment and network etc. are only for example, other are existing or from now on may be used The computer equipment that can occur or network are such as applicable to the present invention, within also should being included in the scope of the present invention, and to draw It is incorporated herein with mode.

Method (some of them are illustrated by flow process) discussed hereafter can by hardware, software, firmware, middleware, Microcode, hardware description language or its combination in any are implementing.When with software, firmware, middleware or microcode to implement, Program code or code segment in order to implement necessary task can be stored in machine or computer-readable medium (such as storage Jie Matter) in.(one or more) processor can implement necessary task.

Concrete structure disclosed herein and function detail are only representational, and are for describing showing of the present invention The purpose of example property embodiment.But, the present invention can be implemented by many alternative forms, and is not interpreted as It is limited only by the embodiments set forth herein.

Although it should be appreciated that may have been used term " first ", " second " etc. here to describe unit, But these units should not be limited by these terms.It is used for the purpose of a unit and another unit using these terms Make a distinction.For example, in the case of the scope without departing substantially from exemplary embodiment, it is single that first module can be referred to as second Unit, and similarly second unit can be referred to as first module.Term "and/or" used herein above include one of or Any and all combination of more listed associated item.

It should be appreciated that when a unit is referred to as " connection " or during " coupled " to another unit, it can be straight Connect and be connected or coupled to described another unit, or there may be temporary location.On the other hand, when a unit is referred to as " directly Connect in succession " or " direct-coupling " arrive another unit when, then there is not temporary location.Should explain in a comparable manner by with Relation between description unit other words (for example " between being in ... " compared to " between being directly in ... ", " with ... neighbouring " compared to " with ... it is directly adjacent to " etc.).

Term used herein above is used for the purpose of description specific embodiment and is not intended to limit exemplary embodiment.Unless Context clearly refers else, and otherwise singulative " one " used herein above, " one " also attempt to including plural number.Also should When being understood by, term " inclusion " used herein above and/or "comprising" specify stated feature, integer, step, operation, Unit and/or the presence of assembly, and do not preclude the presence or addition of other features one or more, integer, step, operation, unit, Assembly and/or a combination thereof.

It should further be mentioned that in some replaces realization modes, the function/action being previously mentioned can be according to different from attached The order that in figure indicates occurs.For example, depending on involved function/action, the two width figures in succession illustrating actually may be used Substantially simultaneously to execute or sometimes can execute in a reverse order.

Below in conjunction with the accompanying drawings the present invention is described in further detail.

The method being used in embodiment one, Distributed Architecture realizing cluster dynamic retractility.

Fig. 1 is for being used for the flow chart realizing the method for cluster dynamic retractility in the Distributed Architecture of the present embodiment, and Fig. 1 institute The method shown mainly includes：Step S100 and step S110.Method described in the present embodiment is typically in computer equipment In be performed it is preferred that the method described in the present embodiment can be in server, desk computer and other network equipments In be performed, such as the method is performed in the server being remotely connected with cluster, desk computer and other network equipments. Below each step in Fig. 1 is illustrated respectively.

S100, determine there is cluster dynamic retractility demand.

Cluster in the present embodiment can be Hadoop cluster or the cluster based on other Distributed Architecture.This Cluster in embodiment includes：Multiple from nodes, i.e. Slave node.From node in the present embodiment is the master with respect to cluster For node (i.e. Master node).From node in the present embodiment do not imply that for company-data copy from section Point.In addition, the from node in the present embodiment is generally by the logic being easy to create and be easy to destroy of virtual machine or other forms Equipment is realizing.

As an example, all from nodes in cluster are divided into computing resource group and storage resource group, certainly, this enforcement The part from node that example is not precluded from cluster is divided into computing resource group and the probability of storage resource group.Normal conditions Under, computing resource group should include at least one from node (be properly termed as calculate from node), and storage resource group also should include to A few from node (being properly termed as storing from node), that is, computing resource group and storage resource group are not in generally for empty feelings Condition.

As an example, the foundation that from node is divided in computing resource group or storage resource group generally includes： The resource service that this from node is provided by cluster；For example, a from node provides storage resource service (as divided if cluster Cloth storage service), then this from node can be divided in storage resource group, and if a from node does not provide for cluster and deposits Storage resource service, but but provide computing resource service (as distributed computing services) for cluster, then and this from node can be divided To in computing resource group.

As an example, each from node in storage resource group generally has the disk of larger memory space (as Large Copacity is hard Disk etc.), and the computing capability of its CPU (Central Processing Unit, central processing unit) is generally weaker；Contrary, meter Calculate the CPU that each from node in resource group generally has stronger computing capability, and the usual very little of the memory space of its disk (is such as joined Put low capacity hard disk etc.).

From node in cluster is divided into computing resource group and storage resource group by the present embodiment, and computing resource group and depositing Configuration in terms of storage resource and computing resource for the from node in storage resource group there may be larger difference, such that it is able to make collection The adjustment of group's scale and disposal ability is more targeted, and can enable cluster scale and disposal ability and practical application field In scape, the demand of storage resource and computing resource is preferably mated, store light calculating as cluster can preferably be applied to again In practical application scene, cluster can be preferably be applied to the practical application scene of light storage re-computation for another example.

As an example, the from node in the storage resource group of the present embodiment provides storage resource service for cluster and calculates money Source services, and such as the from node in storage resource group provides distributed storage service and distributed computing services for cluster；And this reality Apply the from node in the computing resource group of example and cluster offer computing resource service is provided, that is, the from node in computing resource group is only Cluster provides distributed computing services, and does not provide distributed storage service for cluster.Certainly, the present embodiment is not precluded from storing From node in resource group is only the probability that cluster provides storage resource service and do not provide computing resource service for cluster.By From node in cluster needs to take certain computing resource when executing data read operation, but its shared calculating Resource is very limited, and therefore, the present embodiment provides storage resource service and calculating by making the from node in storage resource group Resource service, it is possible to achieve to making full use of of the from node in storage resource group.

As an example, the cluster dynamic retractility demand in the present embodiment can include：Expand computing resource group in from section From node demand in point demand, Reduction Computation resource group, from node demand and the reduction storage expanded in storage resource group One of from node demand in resource group or multiple.

As an example, in the case that cluster dynamic retractility demand in the present embodiment includes above-mentioned four kinds of demands, this reality The cluster dynamic retractility demand applying the presence that example is determined can be any one demand in above-mentioned four kinds of demands；Certainly, The cluster dynamic retractility demand of the presence that the present embodiment is determined can also comprise more than a kind of demand but include two simultaneously The demand of kind, such as expands the from node demand in computing resource group and expands the from node demand in storage resource group；For another example expand Fill the from node demand in computing resource group and the from node demand in reduction storage resource group；For another example Reduction Computation resource group In from node demand and expand storage resource group in from node demand；For another example the from node in Reduction Computation resource group needs Seek and reduce the from node demand in storage resource group.

As an example, the present embodiment can excessively form the situation of accumulation or harmful competition etc. in the calculating task of cluster Under, determine there is cluster dynamic retractility demand, and the present embodiment can be in the memory space inadequate of cluster or memory space In the case of there is crisis, determine there is cluster dynamic retractility demand；More specifically, the present embodiment can be according to getting Cluster performance information determine there is cluster dynamic retractility demand.Cluster performance information accessed by the present embodiment can wrap Include：At least one of utilization ratio of storage resources of the computing resource utilization rate of cluster and cluster；And under normal conditions, this reality Apply the cluster performance information that example gets to include simultaneously：The computing resource utilization rate of cluster and the utilization ratio of storage resources of cluster. The computing resource utilization rate of cluster is usually the computing resource shared by all calculating tasks and current cluster in current cluster In the ratio of total computing resource that can provide of all from nodes, namely shared by all calculating tasks in current cluster Computing resource account for the percentage ratio of total computing resource that all from nodes in current cluster can provide.The storage money of cluster Source utilization rate is usually in the memory space shared by data of all from nodes storage and the current cluster in current cluster The ratio of total storage resource that can provide of all from nodes, namely the number of all from nodes storage in current cluster Account for the percentage ratio of total storage resource that all from nodes in current cluster can provide according to shared memory space.When So, the cluster performance information accessed by the present embodiment can also include：The computing resource being not used by cluster and cluster In unappropriated storage resource etc..The present embodiment does not limit the concrete manifestation form of cluster performance information.

As an example, the present embodiment can add the process of cluster to know each of cluster by all from nodes in cluster Total computing resource (as the data-handling capacity of CPU) that from node each can provide and total storage resource are (as disk Storage size), thus the present embodiment can know total computing resource that all from nodes of cluster can provide and total Storage resource.

As an example, all from nodes in the cluster are all divided in computing resource group and storage resource group, and count The all from nodes calculated in resource group only provide computing resource service, and all from nodes in storage resource group only provide storage In the case of resource service, the computing resource utilization rate of the obtained cluster of the present embodiment its actually in computing resource group all from The computing resource utilization rate of node, the utilization ratio of storage resources of the cluster acquired in the present embodiment its actually in storage resource group The utilization ratio of storage resources of all from nodes.

As an example, all from nodes in the cluster are all divided in computing resource group and storage resource group, and count Calculating all from nodes in resource group only provides computing resource service, and all from nodes in storage resource group provide storage money In the case of source service and computing resource service, its actually computing resource of the computing resource utilization rate of the obtained cluster of the present embodiment The computing resource utilization rate of all from nodes in group and storage resource group, the storage resource profit of the cluster that the present embodiment is obtained Utilization ratio of storage resources with its actually all from node in storage resource group of rate.

As an example, the present embodiment can obtain computing resource utilization rate and the cluster of cluster at the host node of cluster Utilization ratio of storage resources；Send request as the present embodiment carries out telecommunication by the host node with cluster to host node, and Receive host node and transmit the computing resource utilization rate of current cluster come and the storage resource utilization of cluster according to this request Rate；For another example host node actively regularly reports the computing resource utilization rate of current cluster and depositing of cluster by telecommunication mode Storage resource utilization；For another example host node reaches first threshold or less than the in the computing resource utilization rate monitoring current cluster When the utilization ratio of storage resources of two threshold values or cluster reaches the 3rd threshold value or is less than four threshold values, by telecommunication mode The computing resource utilization rate of active reporting current cluster and the utilization ratio of storage resources of cluster.

As an example, the present embodiment can obtain shared by all calculating tasks in cluster at the host node of cluster The memory space shared by data of all from nodes storage in computing resource and cluster, and to the occupancy getting Total computing resource that all from nodes in the cluster of computing resource, the memory space taking and local maintenance can provide The total storage resource that can provide with all from nodes in cluster is calculated, thus the computing resource obtaining cluster utilizes Rate and the utilization ratio of storage resources of cluster；

One specific example, the present embodiment carries out telecommunication by the host node with cluster please to host node transmission Ask, and receive host node according to this request transmit come current cluster in all calculating tasks shared by computing resource with And the memory space shared by data of all from nodes storage in current cluster, then, calculate the current collection receiving All in the computing resource shared by all calculating tasks and the current computing resource group and storage resource group safeguarded in group The ratio of total computing resource that from node provides, thus obtaining the computing resource utilization rate of current cluster, likewise, the present embodiment Also need to calculate the memory space shared by data of all from nodes storage and the leading dimension in the current cluster receiving The ratio of total memory space that all from nodes in the storage resource group of shield provide, thus obtain the storage resource of current cluster Utilization rate；

Another specific example, host node actively regularly reports all meters in current cluster by telecommunication mode The memory space shared by data of all from nodes storage in the computing resource shared by calculation task and current cluster, Then, the computing resource shared by all calculating tasks in the current cluster reporting and the current computing resource group safeguarded are calculated The ratio of the total computing resource providing with all from nodes in storage resource group, thus obtain the computing resource profit of current cluster With rate, likewise, the present embodiment also needs to calculate shared by the data of all from nodes storage in the current cluster reporting Memory space and the current storage resource group safeguarded in the ratio of total memory space that provides of all from nodes, thus obtaining The utilization ratio of storage resources of current cluster；

Another specific example, the calculating money shared by all calculating tasks in monitoring current cluster for the host node Source reaches the default memory space shared by data processing all from nodes storage in threshold value or current cluster and reaches During default storage threshold value, report the institute in the computing resource shared by all calculating tasks and the current cluster in current cluster There is the memory space shared by the data of from node storage, then, calculate all calculating tasks in the current cluster reporting Total calculating that all from nodes in shared computing resource and the current computing resource group and storage resource group safeguarded provide The ratio of resource, thus obtaining the computing resource utilization rate of current cluster, likewise, the present embodiment also needs to calculate working as of reporting Institute in the memory space shared by data of all from nodes storage in front cluster and the current storage resource group safeguarded There is the ratio of total memory space of from node offer, thus obtaining the utilization ratio of storage resources of current cluster.

As an example, using the computing resource utilization rate of cluster, the present embodiment can determine that cluster whether there is expansion meter Calculate the from node demand in resource group or the from node demand in Reduction Computation resource group；And utilize the storage resource profit of cluster Can determine that cluster whether there is in from node demand or the reduction storage resource group expanding in storage resource group with rate From node demand.

As an example, using the computing resource utilization rate of cluster, the present embodiment determines that cluster whether there is expansion and calculates money One specific example of the from node demand in the group of source or the from node demand in Reduction Computation resource group is to judge In the case that the computing resource utilization rate of cluster exceedes first threshold, determine that there is the from node expanding in computing resource group needs Ask；And in the case that the computing resource utilization rate judging cluster is less than Second Threshold, determine there is Reduction Computation resource From node demand in group；And above-mentioned first threshold is typically much deeper than Second Threshold.

As an example, using the utilization ratio of storage resources of cluster, the present embodiment can determine that cluster whether there is expansion and deposits One specific example of the from node demand in storage resource group or the from node demand in reduction storage resource group is to sentence Break the utilization ratio of storage resources storage resource group more than the 3rd threshold value in the case of, determine presence expand storage resource group in From node demand；And in the case that the utilization ratio of storage resources judging computing resource group is less than the 4th threshold value, determine There is the from node demand in reduction storage resource group；And above-mentioned 3rd threshold value is typically much deeper than the 4th threshold value.

As an example, according to the resource adjustment control information receiving, the present embodiment can determine that there is cluster dynamically stretches Contracting demand；As determined there is cluster dynamic retractility demand when receiving the resource adjustment control information that host node transmission comes； For another example determine there is cluster dynamic retractility demand when the resource receiving user input adjusts control information.

As an example, the resource adjustment control information in the present embodiment can be：Expand the from node in computing resource group Control information, the control information of from node in Reduction Computation resource group, the control of the from node expanding in storage resource group Any one in the control information of from node in information and reduction storage resource group；Can also be for expanding computing resource group In the control information of from node and Reduction Computation resource group in the control information of from node in any one and expand Any in the control information of from node in the control information of the from node in storage resource group and reduction storage resource group One.

S110, according to the from node quantity in cluster dynamic retractility demand Adjustable calculation resource group and/or storage resource group.

As an example, the present embodiment can first determine need in computing resource group and/or storage resource group adjust from The quantity of node, then, increases in computing resource group and/or storage resource group further according to the quantity determined or reduces phase Answer the from node of quantity.

As an example, the present embodiment determine need in computing resource group one of the quantity of from node adjusted specific Example is：It is previously provided with the computing resource utilization rate (i.e. the computing resource utilization rate of preferable cluster) of preferably cluster, In the case of needing to add new from node in computing resource group, can be according to shared by all calculating tasks in current cluster Computing resource determines the total computing resource needed for the computing resource utilization rate reaching above-mentioned preferably cluster, calculates required The difference of total computing resource that provided of total computing resource and current cluster, then, newly increased according to this difference and one The computing resource that can provide of from node determine the from node quantity needing to expand in computing resource group.Another is specific Example, in the case of the from node in needing Reduction Computation resource group, can be according to all calculating tasks in current cluster Shared computing resource determines the total computing resource needed for the computing resource utilization rate reaching above-mentioned preferably cluster, calculates The difference of total computing resource that required total computing resource and current cluster are provided, then, according to this difference and calculating money The computing resource that in the group of source, from node can provide determines the from node quantity needing to reduce in computing resource group.

As an example, the present embodiment determine need in storage resource group one of the quantity of from node adjusted specific Example is：It is previously provided with the utilization ratio of storage resources (i.e. the utilization ratio of storage resources of preferable cluster) of preferably cluster, In the case of needing to add new from node in storage resource group, can be according to shared by the data storage in current cluster Storage resource determine total storage resource needed for the utilization ratio of storage resources reaching above-mentioned preferably cluster, calculate required The difference of total storage resource that total storage resource and current cluster are provided, and according to this difference and one newly increase from section The storage resource that point can provide determines the from node quantity needing to expand in storage resource group.Another specific example, In the case of needing to reduce the from node in storage resource group, can be according to shared by all data storages in current cluster Storage resource determines the total storage resource needed for the utilization ratio of storage resources reaching above-mentioned preferably cluster, calculates required The difference of total storage resource that provided of total storage resource and current cluster, then, according to this difference and storage resource group In the storage resource that can provide of from node determine the from node quantity needing to reduce in storage resource group.

It should be strongly noted that needing the from node quantity in computing resource group and storage resource group to enter respectively In the case that row adjusts, and the from node in storage resource group can provide storage resource and computing resource, the present embodiment leads to Chang Yingxian determine need in storage resource group adjust from node quantity, then, then determine need in computing resource group adjust The quantity of whole from node, and determine computing resource group in need adjust from node quantity during, generally take an examination Consider and in storage resource group, need to expand/the impact of the computing resource that provided of from node of the reduction total computing resource to cluster.

As an example, the from node in computing resource group by virtual machine to realize in the case of, the present embodiment is according to collection Expand in computing resource group one of from node of group's dynamic retractility demand implements process and is：First, determine calculating money Need the from node quantity expanding in the group of source, then, create the virtual machine of equivalent amount according to this quantity, and new establishment is each Virtual machine is registered respectively in the cluster (as being added on the log-on message including the host name of the new virtual machine creating respectively In host node in cluster and the cluster configuration file in each from node), and start the distribution of each virtual machine of successful registration Formula calculates service, afterwards, each virtual machine is divided in computing resource group, such as by the relevant information of each virtual machine newly increasing (configuration information such as computing resource information of the virtual machine such as newly increasing) is maintained in computing resource group information.

It should be strongly noted that above-mentioned determination needs the operation of from node quantity expanded, creates new virtual machine The registration in the cluster of operation, new virtual machine operates, starts the distributed computing services in the new virtual machine of successful registration Operation and all will not be in current cluster by successfully starting up the operation that the virtual machine after service is divided in computing resource group Task performed by each node produce impact, and distributed computing services in having successfully started up virtual machine for the present embodiment Afterwards, this virtual machine is formally come into operation the from node becoming in computing resource group, and the host node in cluster can be according to it Current distribution of computation tasks strategy distributes corresponding calculating task for this virtual machine, thus the present embodiment can be in computing resource The new from node of the increase of smooth no breakpoint in group.It follows that the present embodiment can not interrupt the clothes in each node of cluster In the case of business, computing resource group increases new from node, thus avoiding the service disruption institute during cluster is stretched The cluster bringing stretches the higher problem of cost of implementation；In addition, the present embodiment to be realized in computing resource group by using virtual machine From node, additions and deletions easily can be carried out to the from node in computing resource group with remote controlled manner, thus improve reality The intelligence degree that existing cluster is stretched.In addition, the computing resource that the different virtual machine that the present embodiment creates is provided may be identical, There may be difference.

As an example, the from node in storage resource group by virtual machine to realize in the case of, the present embodiment is according to collection Expand in storage resource group one of from node quantity of group's dynamic retractility demand implements process and is：First, determine and deposit Need the from node quantity expanding in storage resource group, then, create the virtual machine of equivalent amount according to this quantity, then, will be new The each virtual machine creating is registered respectively in the cluster (as divided the log-on message including the host name of the new virtual machine creating Do not add in the cluster configuration information in host node in the cluster and each from node), and start each virtual of successful registration The distributed computing services of machine and distributed storage service, afterwards, each virtual machine are divided in storage resource group, such as will increase newly Plus each virtual machine relevant information (the computing resource information of the virtual machine such as newly increasing and storage resource information etc. configuration letter Breath) it is maintained in storage resource group information.

It should be strongly noted that above-mentioned determination needs the operation of from node quantity expanded, creates new virtual machine The registration in the cluster of operation, new virtual machine operates, starts the distributed storage service in the new virtual machine of successful registration Operation with distributed computing services and equal by successfully starting up the operation that the virtual machine after service is divided in storage resource group Impact will not be produced on the task performed by each node in current cluster, and the present embodiment is in having successfully started up virtual machine Distributed storage service and distributed computing services after, this virtual machine formally come into operation become in storage resource group from Node, the host node in cluster can according to its current store tasks allocation strategy and distribution of computation tasks be measured as this from The corresponding store tasks of node distribution and calculating task, thus the present embodiment can smooth the increasing of no breakpoint in storage resource group Plus new from node.It follows that in the case that the present embodiment can not interrupt the service in each node of cluster, in storage money Increase new from node, thus avoiding the cluster that the service disruption during cluster is stretched brought to stretch cost of implementation in the group of source Higher problem；In addition, the present embodiment to realize the from node in storage resource group by using virtual machine, can be controlled with long-range Mode processed easily carries out additions and deletions to the from node in storage resource group, thus improve the intelligence degree realizing that cluster is stretched. In addition, the computing resource that provided of different virtual machine that creates of the present embodiment may identical it is also possible to have differences.

As an example, virtual machine is registered in one of cluster specific example and is by the present embodiment：By Telnet The new virtual machine creating of the continuous logon attempt of mode, after the virtual machine of this new establishment of successful Telnet, can be in this void In plan machine, code entry authority is exempted from the Telnet side's setting for this Telnet, in order in follow-up deletion from node During easily remote operation can be carried out to this from node；Then, the log-on messages such as the host name of this virtual machine are joined Put in the host node and each from node of cluster, make the host node in cluster and each from node is all known in its cluster being located and added Add this from node.

As an example, the from node in computing resource group by virtual machine to realize in the case of, the present embodiment is according to collection One of from node in group's dynamic retractility cutback in demand computing resource group implements process and is：First, determine calculating money Need the from node quantity reduced in the group of source, then, corresponding from node is chosen from computing resource group according to this quantity, for The each from node selecting, notifies the host node in cluster and each from node to delete each from node selecting from cluster (will select as controlled host node in cluster and each from node each to execute space between two tasks at it respectively The log-on message of each from node is deleted respectively from cluster configuration file), all delete in all of host node and from node After the log-on message of the from node being picked, delete the virtual machine of the from node realizing being picked, and the void that will be deleted The relevant information (as deleted configuration information of virtual machine etc.) of plan machine is deleted from computing resource group information.

It should be strongly noted that because the present embodiment can execute the gap of two tasks to the note in node in node Volume information carries out delete processing, therefore, during the from node in Reduction Computation resource group, generally will not be to current cluster In other tasks performed by each node produce impact, and the present embodiment, after deleting from node from cluster, has divided The calculating task of this from node of dispensing and the unsuccessful execution of this from node generally can be avoided by the disaster tolerance mechanism of cluster itself This task finally executes failure, thus the present embodiment can balance the from node in the deletion computing resource group of no breakpoint.Thus Understand, the present embodiment can in the case of the service in each node not interrupting cluster, reduce computing resource group in from section Point.Further, since the computing resource that can be provided by of the different from nodes in computing resource group may identical it is also possible to exist poor Different, therefore, in the case that the computing resource that the different from nodes in computing resource group can be provided by has differences, need determining The quantity of from node to be reduced and during choosing the from node that is contracted by, is considered as each from node in computing resource group The computing resource that can be provided by.

As an example, the from node in storage resource group by virtual machine to realize in the case of, the present embodiment is according to collection One of from node quantity in group's dynamic retractility cutback in demand storage resource group implements process and is：First, determine and deposit Need the from node quantity reduced in storage resource group, then, corresponding from node chosen from storage resource group according to this quantity, For each from node selecting, notify host node in cluster and each from node by each from node selecting from cluster Delete and (such as control the host node in cluster and each from node each to execute the space between two tasks at it and will choose respectively The log-on message of each from node going out is deleted respectively from cluster configuration file), all delete in all of host node and from node After the log-on message of the from node being picked, delete the virtual machine of the from node realizing being picked, and will be deleted The relevant information (configuration information of virtual machine etc. as being deleted) of virtual machine delete from storage resource group information.

It should be strongly noted that because the present embodiment can execute the gap of two tasks to the note in node in node Volume information carries out delete processing, therefore, during the from node in reduction storage resource group, generally will not be to current cluster In other tasks performed by each node produce impact, and the present embodiment, after deleting from node from cluster, has divided The calculating task of this from node of dispensing and the unsuccessful execution of this from node generally can be kept away by the disaster tolerance mechanism of cluster itself Exempt from this calculating task and finally execute the loss of data storing in failure, and this from node leading to due to deleting this from node now As generally being recovered by the synchronizing process between the data trnascription of cluster；Thus the present embodiment can balance no breakpoint Delete the from node in computing resource group.It follows that the present embodiment can service in each node not interrupting cluster In the case of, the from node in storage resource group is reduced by remotely control.Further, since the different from nodes in storage resource group The storage resource that can be provided by may be identical, also by there may be difference, therefore, different from node institutes in storage resource group In the case that the storage resource being provided that has differences, it is contracted by the quantity and selection determining the from node needing reduction During from node, it is considered as the storage resource that in storage resource group, each from node can be provided by.Further, in reduction storage money During from node in the group of source, it is considered as the impact of the computing resource to cluster.

As an example, during being reduced for storage resource group, under normal conditions, selected by the present embodiment All from nodes that the from node quantity being contracted by going out should be less than in storage resource group and the difference of company-data copy amount；As In the case that data trnascription quantity in the cluster is 3, after the from node in storage resource group is reduced, storage From node quantity in resource group should be not less than 3, to reduce the loss of data risk in cluster as far as possible.

As an example, during being reduced for computing resource group, under normal conditions, selected by the present embodiment Total calculating money that all from nodes in total computing resource that the from node being contracted by going out can provide and cluster are provided The ratio in source is less than predetermined ratio (as 5%), the computation delay leading to reduce re-computation as far as possible.

Embodiment two, the method being used for realizing cluster dynamic retractility in Hadoop cluster.

Hadoop cluster in the present embodiment is built by virtual machine.Taken using virtual machine by remote controlled manner Build a detailed process of Hadoop cluster as shown in Fig. 2 and the method shown in Fig. 2 comprise the steps：

S200, the quantity of the from node being comprised based on presetting Hadoop cluster create the virtual machine of respective numbers, During creating each virtual machine, it is that the essential information of virtual machine distribution generally includes：LAN IP address is (such as 192.168.0.62), the login account (as root) of virtual machine and the login password of virtual machine；

S210, after successfully creating each virtual machine, by each virtual machine of the continuous logon attempt of Telnet mode, with Confirm each virtual machine network all connect available；And after each virtual machine of long-range Successful login, difference pin in each virtual machine Password login authority is exempted to the setting of this Telnet side, subsequently Hadoop cluster dynamic retractility is controlled with facilitating；

S220, distribute a Hos tname (host name), and the Hos tname by each virtual machine for each virtual machine It is respectively arranged in the respective profiles of other all virtual machines (in such as/etc/hos ts file)；

S230, in addition to the host node in Hadoop cluster, storage resource group and calculating are carried out to all of virtual machine The division of resource group, and arrange the nodemanager's in the yarn service of each virtual machine according to the hardware configuration of each virtual machine Storage size (i.e. storage resource information) in Cpu information (i.e. computing resource information) and hdfs service, makes Telnet Side can easily adjust the quantity of the from node in storage resource group and computing resource group；In addition, host node also should be known respectively Storage size in the Cpu information of nodemanager in the yarn service of virtual machine and hdfs service, in order to main section Point can carry out the distribution of calculating task and store tasks；

S240, the assembly being managed by remote controlled manner in each virtual machine are installed and are run, and assembly here includes Hdfs service and yarn service；Specifically, the present embodiment can be started in each virtual machine by Telnet mode Ambari services, and is installed and activated each assembly based on Hadoop in each virtual machine using Restful Api, such as starts and deposits Hdfs service in each virtual machine in storage resource group and yarn service, start the yarn in each virtual machine in computing resource group Service etc., so that virtual machine becomes the from node in Hadoop cluster.

After successfully building Hadoop cluster using virtual machine, if there is the dilatation demand of Hadoop cluster, held One specific example of the operation of row is as shown in figure 3, the method in Fig. 3 comprises the steps：

S300, determine it is that computing resource group needs to increase new from node, or storage resource group need to increase new from Node, such as in the case of the memory space inadequate of Hadoop cluster it may be determined that go out only storage resource group need to increase new From node, only counts it may be determined that going out when the calculating task of Hadoop cluster excessively forms accumulation or harmful competition for another example Calculate resource group need to increase new from node, if above-mentioned two situations occur simultaneously, can determine that storage resource group and Computing resource group is required to increase new from node.

S310, determine required the increased new from node quantity of computing resource group and storage resource group, and according to this quantity Create the new virtual machine of respective numbers, be each new virtual machine distribution Hostname, and be respectively in each new virtual machine Password login authority is exempted from remotely control node (i.e. above-mentioned Telnet side) setting.Because virtual machine now does not also add Hadoop cluster, therefore, this step will not produce any impact to the task of from node execution each in cluster.

(such as will wrap in S320, the Ambari service that each new virtual machine is registered in all nodes of Hadoop cluster The Hos tname including each new virtual machine is arranged in the configuration file of Ambari service in interior log-on message), and start each Corresponding assembly (as started the hdfs service in virtual machine and/or yarn service etc.) on new virtual machine, so that new void Plan machine is added in computing resource group or the storage resource group of Hadoop cluster, and then improves scale and the place of Hadoop cluster Reason ability.

After successfully building Hadoop cluster using virtual machine, if there is the capacity reducing demand of Hadoop cluster, held One specific example of the operation of row is as shown in figure 4, the method in Fig. 4 comprises the steps：

S400, determination are that computing resource group needs to reduce from node, or storage resource group needs to reduce from node, such as exists Need to reduce from node it may be determined that going out only storage resource group in the case that the idle memory space of Hadoop cluster is too high, then As Hadoop cluster calculating task very few and formed idle computing resource too high when need it may be determined that going out only computing resource group From node to be reduced, if above-mentioned two situations occur simultaneously, can determine that storage resource group and computing resource group are both needed to From node to be reduced.

S410, determine the from node quantity of the required reduction of computing resource group and storage resource group, and according to this quantity from meter Calculate resource group and the storage resource group from node that is contracted by of selection, such as can be using from node minimum for data storage amount as being contracted The from node subtracting, for another example can be using from node minimum for the calculating task being undertaken as the from node being contracted by.

In the case of S420, all services in not interrupting Hadoop cluster, in Hadoop cluster except being contracted Other all nodes outside the from node subtracting are executing the gap of first latter two task, respectively by the note of the from node being contracted by Volume information is deleted in the configuration file from Ambari service, (as being contracted by configuration file that Ambari is serviced from The Hos tname of node deletes).After each node has been performed both by information deletion operation, to S430.

The virtual machine of the from node that S430, deletion realization are contracted by.

The method being used in embodiment three, Distributed Architecture realizing cluster dynamic retractility.

It is used in the Distributed Architecture of the present embodiment realizing the method for cluster dynamic retractility by remote with cluster telecommunication Process control node executes, and the flow process of the method for this embodiment is as shown in Figure 5.

All calculating tasks in Fig. 5, in the current cluster that S500, the host node of remotely control node reception cluster report The memory space A2 shared by data of all from nodes storage in shared computing resource A1 and current cluster.

S510, remotely control node obtain currently from the computing resource group information and storage resource group information of local maintenance All from nodes in total computing resource Z1 that all from nodes in cluster can provide and current cluster can carry For total storage resource Z2, and calculate the ratio X 1 of A1 and Z1 and the ratio X 2 of A2 and Z2.

S520, remotely control node judge whether X1 exceedes whether whether first threshold Y1, X1 be less than Second Threshold Y2, X2 Whether it is less than the 4th threshold value Y4 more than the 3rd threshold value Y3 and X2；

If it is judged that X1 exceed first threshold Y1 (as 0.9) it is determined that go out computing resource group need to add new from section Point, to step S531；

If it is judged that X1 be less than Second Threshold Y2 (as 0.4) it is determined that go out computing resource group need to reduce existing from Node, to step S532；

If it is judged that X2 more than the 3rd threshold value Y3 (as 0.8) it is determined that go out storage resource group need to add new from section Point, to step S533；

If it is judged that X2 be less than the 4th threshold value Y4 (as 0.5) it is determined that go out storage resource group need to reduce existing from Node, to step S534.

S531, remotely control node determines needs to add the quantity of new from node, and creates the virtual machine of respective numbers, It is respectively allocated essential information (as IP address and host name etc.) for each virtual machine, and each virtual machine is registered in the cluster, such as The log-on message of each virtual machine is arranged in the configuration file of all nodes of cluster.

S541, each virtual machine of remotely control node control start its distributed computing services, make virtual machine as in cluster Calculating from node and come into operation, the from node realized by virtual machine is divided in computing resource group remotely control node, As the configuration informations such as the new computing resource information of each from node expanding are maintained in local computing resource group information.

S532, remotely control node determines the quantity of the from node needing reduction, and selects to calculate from computing resource group The from node of task respective numbers the lightest；The log-on message of the from node needing reduction is deleted from each node of cluster.

S542, remotely control knot removal are realized needing each virtual machine of the from node of reduction, and from computing resource group Delete the from node needing reduction, such as will need the configuration informations such as the computing resource information of each from node of reduction from local Delete in computing resource group information.

S533, remotely control node determines needs to add the quantity of new from node, and creates the virtual machine of respective numbers, It is respectively allocated essential information (as IP address and host name etc.) for each virtual machine, and each virtual machine is registered in the cluster, such as The log-on message of each virtual machine is arranged in the configuration file of all nodes of cluster.

S543, each virtual machine of remotely control node control start its distributed computing services and distributed storage service, make Virtual machine comes into operation as the storage from node in cluster, and the from node realized by virtual machine is divided by remotely control node In storage resource group, such as by the configuration information dimension such as the computing resource information of the new each from node expanding and storage resource information Shield is in local storage resource group information.

S534, remotely control node determines the quantity of the from node needing reduction, and selects storage from storage resource group The from node of task respective numbers the lightest；The log-on message of the from node needing reduction is deleted from each node of cluster.

S544, remotely control knot removal are realized needing each virtual machine of the from node of reduction, and from storage resource group Delete the from node needing reduction, such as computing resource information and storage resource information of each from node of reduction etc. will be needed to join Confidence breath is from deletion local storage resource group information.

It is used in example IV, Distributed Architecture realizing the device of cluster dynamic retractility.

The device being used for realizing cluster dynamic retractility in the Distributed Architecture of the present embodiment would generally be arranged at computer It is preferred that can arrange for realizing the device of cluster dynamic retractility in Distributed Architecture described in the present embodiment in equipment In server, desk computer and other network equipments.In addition, using in the Distributed Architecture of the present embodiment of the present embodiment Can be with cluster telecommunication in the computer equipment that the device realizing cluster dynamic retractility is located.Cluster in the present embodiment can Think Hadoop cluster or the cluster based on other Distributed Architecture.Cluster in the present embodiment comprised from section Point is divided into computing resource group and storage resource group, and the foundation of division waits such as the description in above-described embodiment one, and here is no longer Repeat specification.

In the Distributed Architecture of the present embodiment, the primary structure of the device for realizing cluster dynamic retractility is as shown in Figure 6. Referring to specific embodiment, the device being used for realizing cluster dynamic retractility in Distributed Architecture is illustrated.

In figure 6, the device being used for realizing cluster dynamic retractility in the Distributed Architecture of the present embodiment mainly includes：For Determine the device (referred to as following " determining demand device 600 ") that there is cluster dynamic retractility demand and for according to cluster Device (following referred to as " adjustment of the from node quantity in dynamic retractility demand Adjustable calculation resource group and/or storage resource group Resource group device 610 ").

Determine that demand device 600 is mainly used in determining there is cluster dynamic retractility demand, and the demand determined is permissible Including the from node demand expanding in computing resource group, the from node demand in Reduction Computation resource group, expand storage resource group In from node demand and one of from node demand in reduction storage resource group or multiple.

As an example, determine that demand device 600 can excessively form accumulation or harmful competition etc. in the calculating task of cluster In the case of, determine there is cluster dynamic retractility demand, and determine that demand device 600 can be in the memory space inadequate of cluster Or in the case that memory space has crisis, determine there is cluster dynamic retractility demand.

As an example, above-mentioned determination demand device 600 can include：For being determined according to the cluster performance information getting Go out to exist the device (following referred to as " the first determination demand device 601 ") of cluster dynamic retractility demand and for according to reception To resource adjust control information and determine device (following referred to as " the second determination demands that there is cluster dynamic retractility demand Device 602 ") (as shown in Figure 7).

Cluster performance information accessed by first determination demand device 601 can include：The computing resource of cluster utilizes At least one of utilization ratio of storage resources of rate and cluster；And under normal conditions, the first determination demand device 601 gets Cluster performance information include simultaneously：The computing resource utilization rate of cluster and the utilization ratio of storage resources of cluster.First determination needs Ask device 601 can obtain the computing resource utilization rate of cluster and the storage resource utilization of cluster at the host node of cluster Rate, the description in detailed process such as above-described embodiment one, it is not repeated.

As an example, using the computing resource utilization rate of cluster, the first determination demand device 601 can determine that cluster is No have the from node demand in expansion computing resource group or the from node demand in Reduction Computation resource group；And the first determination Demand device 601 using the utilization ratio of storage resources of cluster can determine that cluster whether there is expand storage resource group in from From node demand in node demand or reduction storage resource group.

As an example, using the computing resource utilization rate of cluster, the first determination demand device 601 determines whether cluster is deposited From node demand in expanding computing resource group or a specific example of the from node demand in Reduction Computation resource group Son is that first determines that demand device 601, in the case that the computing resource utilization rate judging cluster exceedes first threshold, determines Go out to there is the from node demand expanding in computing resource group；And the first determination demand device 601 provides in the calculating judging cluster In the case that source utilization rate is less than Second Threshold, determine the from node demand existing in Reduction Computation resource group；And above-mentioned One threshold value is typically much deeper than Second Threshold.

As an example, using the utilization ratio of storage resources of cluster, the first determination demand device 601 can determine that cluster is No there is the from node demand in the from node demand expanding in storage resource group or reduction storage resource group one is specifically Example be, first determination demand device 601 judging the utilization ratio of storage resources of storage resource group more than the 3rd threshold value In the case of, determine there is the from node demand expanding in storage resource group；And the first determination demand device 601 is being judged to count Calculate resource group utilization ratio of storage resources be less than the 4th threshold value in the case of, determine exist reduction storage resource group in from section Point demand；And above-mentioned 3rd threshold value is typically much deeper than the 4th threshold value.

As an example, the second determination demand device 602 can be determined according to the resource adjustment control information receiving and deposit In cluster dynamic retractility demand；As the second determination demand device 602 is receiving the resource adjustment control letter that host node transmission comes During breath, determine there is cluster dynamic retractility demand；For another example the second determination demand device 602 is receiving the resource of user input During adjustment control information, determine there is cluster dynamic retractility demand.

As an example, the resource that the second determination demand device 602 receives adjusts control information and can be：Expand computing resource In the control information of from node in the control information of the from node in group, Reduction Computation resource group, expansion storage resource group Any one in the control information of from node in the control information of from node and reduction storage resource group；Can also be for expanding Fill any in the control information of from node in the control information of from node and Reduction Computation resource group in computing resource group One and expand storage resource group in the control information of from node and reduction storage resource group in from node control Any one in information.

Adjustresources group device 610 is mainly used according to cluster dynamic retractility demand Adjustable calculation resource group and/or storage From node quantity in resource group.

As an example, adjustresources group device 610 can first be determined in computing resource group and/or storage resource group and need The quantity of the from node of adjustment, then, adjustresources group device 610 further according to the quantity determined in computing resource group and/or Increase or reduce the from node of respective numbers in storage resource group.

As an example, the adjustresources group device 610 of the present embodiment can optionally include：Calculate money for determining Source group need the device (following referred to as " the first quantification device 611 ") of the from node quantity expanding, be used for creating above-mentioned The device (following referred to as " create virtual machine 612 ") of the virtual machine of quantity, for by the log-on message configuration of virtual machine Device in the cluster (following referred to as " register device 613 "), the device for starting the distributed computing services of virtual machine (following referred to as " the first startup service unit 614 ") and for using virtual machine as the from node in computing resource group dress Put (referred to as following " the first attending device 615 ") (as shown in Figure 8).That is, from node in computing resource group by In the case that virtual machine is to realize, adjustresources group device 610 expands a specific example of the from node in computing resource group For：First, the first quantification device 611 determines the from node quantity needing to expand in computing resource group, then, creates empty Plan machine device 612 creates the virtual machine of equivalent amount according to this quantity, and the new each virtual machine creating is noted by register device 613 respectively Volume is in the cluster (as the log-on message including the host name of the new virtual machine creating is added on by register device 613 respectively In host node in cluster and the cluster configuration file in each from node), first starts service unit 614 starts successful registration Each virtual machine distributed computing services, afterwards, each virtual machine is divided in computing resource group the first attending device 615, If the first attending device 615 is by relevant information (the computing resource information of the virtual machine such as newly increasing of each virtual machine newly increasing Deng configuration information) it is maintained in computing resource group information.

It is previously provided with the computing resource utilization rate of preferably cluster in first quantification device 611, needing in meter Calculate in the case of adding new from node in resource group, the first quantification device 611 can be according to all in current cluster Computing resource shared by calculating task determines the total calculating needed for the computing resource utilization rate reaching above-mentioned preferably cluster Resource, calculates the difference of total computing resource that required total computing resource is provided, then, the first quantification with current cluster Device 611 determines need in computing resource group according to the computing resource that this difference and from node newly increasing can provide From node quantity to be expanded.Another specific example, in the case of the from node in needing Reduction Computation resource group, the One quantification device 611 can be determined according to the computing resource shared by all calculating tasks in current cluster and reach State the total computing resource needed for the computing resource utilization rate of preferably cluster, the first quantification device 611 calculates required total The difference of total computing resource that computing resource and current cluster are provided, then, the first quantification device 611 is according to this difference And the computing resource that in computing resource group, from node can provide determine need in computing resource group to reduce from section Point quantity.

As another example, the adjustresources group device 610 of the present embodiment can optionally include：For determining storage Resource group needs the device (referred to as following " the second quantification device 616 ") of the from node quantity expanding, for creating State the device (following referred to as " establishment virtual machine 612 ") of the virtual machine of quantity, be used for joining the log-on message of virtual machine Put device in the cluster (following referred to as " register device 613 "), for start virtual machine distributed computing services and point The device (following referred to as " the second startup service unit 617 ") of cloth storage service and for using virtual machine as storage money The device (referred to as following " the second attending device 618 ") (as shown in Figure 9) of the from node in the group of source.That is, in storage From node in resource group by virtual machine to realize in the case of, adjustresources group device 610 expand storage resource group in from One specific example of number of nodes is：First, the second quantification device 616 is determined needs expansion in storage resource group From node quantity, then, creates the virtual machine that virtual machine 612 creates equivalent amount according to this quantity, then, register device 613 register the new each virtual machine creating respectively in the cluster (as register device 613 will include the main frame of the new virtual machine creating Name is added in the cluster configuration information in host node in the cluster and each from node respectively in interior log-on message), second Start the distributed computing services of each virtual machine and the distributed storage service that service unit 617 starts successful registration, afterwards, the Each virtual machine is divided in storage resource group two attending devices 618, and such as the second attending device 618 is by each virtual machine newly increasing Relevant information (configuration information such as the computing resource information of the virtual machine such as newly increasing and storage resource information) be maintained in and deposit In storage resource group information.

As an example, it is previously provided with the utilization ratio of storage resources of preferably cluster in the second quantification device 616, In the case of needing to add new from node in storage resource group, the second quantification device 616 can be according to current cluster In the storage resource shared by data storage determine needed for the utilization ratio of storage resources reaching above-mentioned preferably cluster Total storage resource, the second quantification device 616 calculates total storage money that required total storage resource is provided with current cluster The difference in source, and storage resource group is determined according to the storage resource that this difference and from node newly increasing can provide The middle from node quantity needing to expand.Another specific example, is needing to reduce the situation of the from node in storage resource group Under, the second quantification device 616 can determine according to the storage resource shared by all data storages in current cluster Go out the total storage resource needed for the utilization ratio of storage resources reaching above-mentioned preferably cluster, the second quantification device 616 calculates The difference of total storage resource that required total storage resource and current cluster are provided, then, the second 616, quantification device Determining according to the storage resource that from node in this difference and storage resource group can provide needs in storage resource group to contract The from node quantity subtracting.

As an example, above-mentioned register device 613 can optionally include：Device for Telnet virtual machine is (following Referred to as " Telnet device 6131 "), exempt from the dress of code entry authority for being directed to the setting of Telnet side in virtual machine Put (following referred to as " setting authority device 6132 ") and for the log-on message including host name is joined by virtual machine Put device in the host node of cluster and each from node (following referred to as " configuration information device 6133 ") (as Figure 10 institute Show).That is, virtual machine is registered in one of cluster specific example by register device 613 being：Telnet device 6131, by the new virtual machine creating of the continuous logon attempt of Telnet mode, successfully remotely step in Telnet device 6131 After recording the virtual machine of this new establishment, setting authority device 6132 can be in this virtual machine, remotely stepping on for this Telnet The setting of record side exempts from code entry authority, in order to can easily enter to this from node during follow-up deletion from node Row remote operation；Then, the log-on messages such as the host name of this virtual machine are configured the main section in cluster by configuration information device 6133 In point and each from node, make the host node in cluster and each from node is all known in its cluster being located and be with the addition of this from node.

As another example, the adjustresources group device 610 of the present embodiment can optionally include：Calculate for determining Device (following referred to as " the 3rd quantification devices of the from node quantity of reduction are needed in resource group and/or storage resource group 619 "), it is used for choosing device (following referred to as " the selection node apparatus being contracted by from node of respective numbers from respective sets 620 "), it is used for that any one is configured with the node of the log-on message being contracted by from node in cluster, execute two in node The gap of task, device (following referred to as " deletion log-on message dresses of the log-on message being contracted by from node in deletion of node Put 621 ") and in the case of all deleting, for all nodes in the cluster, the log-on message being contracted by from node, delete Realize being contracted by the virtual machine of from node, and will be contracted by what from node was deleted from computing resource group and/or storage resource group Device (referred to as following " third dimension protection unit 622 ") (as shown in figure 11).

As an example, the from node in computing resource group by virtual machine to realize in the case of, adjustresources group device One specific example of the from node in 610 Reduction Computation resource groups is：First, the 3rd quantification device 619 determines meter Calculate the from node quantity needing in resource group to reduce, then, choose node apparatus 620 and selected from computing resource group according to this quantity Take corresponding from node, for each from node selecting, delete log-on message device 621 notify host node in cluster and The each from node selecting is deleted from cluster and (is controlled respectively in cluster as deleted log-on message device 621 by each from node Host node and each from node each execute space between two tasks by the log-on message of each from node selecting at it Delete from cluster configuration file respectively), all delete, in all of host node and from node, the from node being picked After log-on message, third dimension protection unit 622 deletes the virtual machine of the from node realizing being picked, and the virtual machine that will be deleted Relevant information (configuration information of virtual machine etc. as being deleted) delete from computing resource group information.

As an example, the from node in storage resource group by virtual machine to realize in the case of, adjustresources group device Specific examples of from node quantity in 610 reduction storage resource groups are：First, the 3rd quantification device 619 determines Go out the from node quantity needing to reduce in storage resource group, then, choose node apparatus 620 according to this quantity from storage resource group The corresponding from node of middle selection, for each from node selecting, deletes log-on message device 621 and notifies the host node in cluster And each from node selecting deleted from cluster and (controls cluster respectively as deleted log-on message device 621 by each from node In host node and each from node each execute space between two tasks by the registration of each from node selecting at it Information respectively from cluster configuration file delete), all of host node and from node all delete be picked from section After the log-on message of point, third dimension protection unit 622 deletes the virtual machine of the from node realizing being picked, third dimension protection unit 622 by the relevant information (configuration information of virtual machine etc. as being deleted) of the virtual machine being deleted from storage resource group information Delete.

As an example, during adjustresources group device 610 is reduced for storage resource group, in normal conditions Under, choose the selected from node quantity being contracted by taken out of node apparatus 620 and should be less than all from nodes in storage resource group Difference with company-data copy amount；In the case of being 3 as data trnascription quantity in the cluster, in adjustresources group device After from node in 610 pairs of storage resource groups is reduced, the from node quantity in storage resource group should be not less than 3, with Reduce the loss of data risk in cluster as far as possible.

As an example, during adjustresources group device 610 is reduced for computing resource group, in normal conditions Under, choose the institute in the total computing resource and cluster that the selected from node being contracted by taken out of node apparatus 620 can provide The ratio having total computing resource that from node provided is less than predetermined ratio (as 5%), reducing re-computation as far as possible The computation delay leading to.

It is used in embodiment five, Distributed Architecture realizing a concrete application of the device of cluster dynamic retractility.

First, in the current cluster that the host node of the determination demand device 600 reception cluster in remotely control node reports Computing resource A1 shared by all calculating tasks and the data of all from nodes storage in current cluster shared by Memory space A2.

Secondly, determine demand device 600 from the computing resource group information of remotely control node local maintenance and storage resource Institute in total computing resource Z1 that can provide of all from nodes and current cluster in current cluster is provided in group information There is total storage resource Z2 that from node can provide, and calculate the ratio X 1 of A1 and Z1 and the ratio X 2 of A2 and Z2.

Determine that demand device 600 judges whether X1 exceedes whether whether first threshold Y1, X1 surpass less than Second Threshold Y2, X2 Cross the 3rd threshold value Y3 and whether X2 is less than the 4th threshold value Y4；

If it is determined that demand device 600 judges that X1 exceedes first threshold Y1 (as 0.9) it is determined that demand device 600 is true Making computing resource group needs to add new from node；Adjustresources group device 610 in remotely control node determines that needs add Plus the quantity of new from node, and creating the virtual machine of respective numbers, adjustresources group device 610 is respectively allocated for each virtual machine Essential information (as IP address and host name etc.), and each virtual machine is registered in the cluster, such as adjustresources group device 610 will The log-on message of each virtual machine is arranged in the configuration file of all nodes of cluster.Adjustresources group device 610 controls each void Plan machine starts its distributed computing services, so that virtual machine is come into operation as the calculating from node in cluster, adjustresources group The from node realized by virtual machine is divided in computing resource group device 610, as adjustresources group device 610 expands new The configuration informations such as the computing resource information of each from node are maintained in the local computing resource group information of remotely control node.

If it is determined that demand device 600 judges that X1 is less than Second Threshold Y2 (as 0.4) it is determined that demand device 600 is true Making computing resource group needs to reduce existing from node；Adjustresources group device 610 determines the number of the from node needing reduction Amount, and select the from node of calculating task respective numbers the lightest from computing resource group；Adjustresources group device 610 is from cluster Each node in delete need reduction from node log-on message.Adjustresources group device 610 is deleted and is realized needing reduction Each virtual machine of from node, and delete the from node needing to reduce from computing resource group, such as adjustresources group device 610 will need The configuration informations such as the computing resource information of each from node to be reduced are from deletion local computing resource group information.

If it is determined that demand device 600 judge X2 more than the 3rd threshold value Y3 (as 0.8) it is determined that demand device 600 is true Making storage resource group needs to add new from node；Adjustresources group device 610 determines to be needed to add the number of new from node Amount, and create the virtual machine of respective numbers, adjustresources group device 610 is respectively allocated essential information (as IP for each virtual machine Location and host name etc.), and each virtual machine is registered in the cluster, such as adjustresources group device 610 is by the registration of each virtual machine Information is arranged in the configuration file of all nodes of cluster.Adjustresources group device 610 controls each virtual machine to start its distribution Formula calculates service and distributed storage service, so that virtual machine is come into operation as the storage from node in cluster, adjustresources The from node realized by virtual machine is divided in storage resource group group device 610, and such as adjustresources group device 610 will newly expand The computing resource information of each from node and the configuration information such as storage resource information be maintained in local storage resource group information In.

If it is determined that demand device 600 judges that X2 is less than the 4th threshold value Y4 (as 0.5) it is determined that demand device 600 is true Making storage resource group needs to reduce existing from node.Adjustresources group device 610 determines the number of the from node needing reduction Amount, and select the from node of store tasks respective numbers the lightest from storage resource group；Adjustresources group device 610 is from cluster Each node in delete need reduction from node log-on message.Adjustresources group device 610 is deleted and is realized needing reduction Each virtual machine of from node, and delete the from node needing to reduce from storage resource group, such as adjustresources group device 610 will need The configuration information such as the computing resource information of each from node to be reduced and storage resource information is from local storage resource group Delete in information.

It is used in embodiment six, Distributed Architecture realizing another concrete application of the device of cluster dynamic retractility.

In the present embodiment, the host node in cluster calculates the calculating money shared by all calculating tasks in current cluster The ratio X 1 of total computing resource Z1 that all from nodes in source A1 and current cluster can provide, and calculate in current cluster The memory space A2 shared by data of all from nodes storage can provide with all from nodes in current cluster Total storage resource Z2 ratio X 2, and the ratio X calculating 1 and X2 are reported determination demand device 600, determine that demand fills Put 600 and judge whether X1 exceedes whether whether first threshold Y1, X1 exceed less than Second Threshold Y2, X2 according to the information receiving Whether the 3rd threshold value Y3 and X2 are less than the 4th threshold value Y4；Follow-up determination demand device 600 and adjustresources group device 610 institute The device of execution is identical with the description in above-described embodiment five, is not repeated.

It should be noted that the present invention can be carried out in software and/or software with the assembly of hardware, for example, Each device of the present invention can be realized using special IC (ASIC) or any other similar hardware device.At one In embodiment, the software program of the present invention can realize steps described above or function by computing device.Similarly, originally The software program of invention can be stored in computer readable recording medium storing program for performing (including related data structure), and for example, RAM deposits Reservoir, magnetic or CD-ROM driver or floppy disc and similar devices.In addition, some steps of the present invention or function can employ hardware to Realizing, for example, coordinating thus executing the circuit of each step or function as with processor.

It will be apparent to a person skilled in the art that the invention is not restricted to the details of above-mentioned one exemplary embodiment, Er Qie In the case of the spirit or essential attributes of the present invention, the present invention can be realized in other specific forms.Therefore, no matter From the point of view of in terms of which, all embodiment should be regarded as exemplary, and be nonrestrictive, the scope of the present invention is by institute Attached claim rather than described above are limiting, accordingly, it is intended to will fall in the implication of equivalency of claim and scope Interior all changes are included in the present invention.Any reference in claim should not be considered as limiting involved power Profit requires.Furthermore, it is to be understood that " inclusion " one word is not excluded for other units or step, odd number is not excluded for plural number.In system claims Multiple units of statement or device can also be realized by software or hardware by a unit or device.First and second It is used for representing title Deng word, and be not offered as any particular order.

Although above specifically shown and describe exemplary embodiment, it will be understood to those of skill in the art that It is, in the case of the spirit and scope without departing substantially from claims, can be varied from terms of its form and details.Here Sought protection illustrates in the dependent claims.

Claims

1. a kind of method being used in Distributed Architecture realizing cluster dynamic retractility, wherein, the method comprising the steps of：

Determine there is cluster dynamic retractility demand, wherein, described cluster includes：Multiple from nodes, and the plurality of from node Computing resource group and storage resource group are divided into according to the resource service that it provides；

From node quantity in described computing resource group and/or storage resource group is adjusted according to described cluster dynamic retractility demand.

2. method according to claim 1, wherein, the from node in described computing resource group provides Distributed Calculation clothes Business, the from node in described storage resource group provides distributed storage service and distributed computing services.

3. method according to claim 1, wherein, described cluster dynamic retractility demand includes：Expand in computing resource group From node demand, the from node demand in Reduction Computation resource group, from node demand and the contracting expanded in storage resource group Subtract at least one of from node demand in storage resource group.

4. method according to claim 1, wherein, described determines that the step that there is cluster dynamic retractility demand includes：

Cluster performance information according to getting determines there is cluster dynamic retractility demand；And/or

Resource adjustment control information according to receiving determines there is cluster dynamic retractility demand.

5. method according to claim 4, wherein, described cluster performance packet includes：The computing resource utilization rate of cluster And/or the utilization ratio of storage resources of cluster.

6. method according to claim 1, described according to described cluster dynamic retractility demand adjust described computing resource group And/or the step of the from node quantity in storage resource group includes：

Determine that computing resource group needs the from node quantity expanding；

Create the virtual machine of described quantity；

The log-on message of described virtual machine is configured in the cluster；

Start the distributed computing services of described virtual machine；

Using virtual machine as the from node in computing resource group.

7. method according to claim 1, described according to described cluster dynamic retractility demand adjust described computing resource group And/or the step of the from node quantity in storage resource group includes：

Determine that storage resource group needs the from node quantity expanding；

Create the virtual machine of described quantity；

The log-on message of described virtual machine is configured in the cluster；

Start distributed computing services and the distributed storage service of described virtual machine；

Using virtual machine as the from node in storage resource group.

8. the method according to claim 6 or 7, the described log-on message configuration by described virtual machine step in the cluster Including：

Virtual machine described in Telnet；

It is directed to the setting of Telnet side and exempt from code entry authority in described virtual machine；

Described virtual machine is configured in the host node of cluster and each from node inclusion the log-on message including host name.

9. method according to claim 1, wherein, described according to described cluster dynamic retractility demand adjustment described calculate money The step of the from node quantity in source group and/or storage resource group includes：

Determine the from node quantity needing to reduce in computing resource group and/or storage resource group；

That chooses respective numbers from respective sets is contracted by from node；

For in cluster any one be configured with described in be contracted by from node log-on message node, described node execute two The gap of individual task, is contracted by the log-on message of from node described in deletion of node；

In the case that all nodes in the cluster all delete the log-on message being contracted by from node, delete realization be contracted by from The virtual machine of node, and from node will be contracted by delete from computing resource group and/or storage resource group.

10. method according to claim 9, wherein：

The total computing resource of from node offer reduced and from nodes all in cluster are needed to provide in described computing resource group The ratio of total computing resource is less than predetermined ratio；

The from node quantity reduced is needed to be less than all from node quantity and cluster in storage resource group in described storage resource group The difference of data trnascription quantity.

It is used in a kind of 11. Distributed Architecture realizing the device of cluster dynamic retractility, wherein, including：

For determining the device that there is cluster dynamic retractility demand, wherein, described cluster includes：Multiple from nodes, and described Multiple from nodes are divided into computing resource group and storage resource group according to the resource service that it provides；

For adjusting the from node number in described computing resource group and/or storage resource group according to described cluster dynamic retractility demand The device of amount.

12. devices for realizing cluster dynamic retractility according to claim 11, wherein, in described computing resource group From node provides distributed computing services, and the from node in described storage resource group provides distributed storage service and distributed meter Calculate service.

13. devices for realizing cluster dynamic retractility according to claim 11, wherein, described cluster dynamic retractility needs Ask including：Expand the from node demand in computing resource group, the from node demand in Reduction Computation resource group, expand storage resource At least one of from node demand in group and the from node demand in reduction storage resource group.

14. devices for realizing cluster dynamic retractility according to claim 11, wherein, described are used for determining presence The device of cluster dynamic retractility demand includes：

For the device that there is cluster dynamic retractility demand is determined according to the cluster performance information getting；

And/or

For the device that there is cluster dynamic retractility demand is determined according to the resource adjustment control information receiving.

15. devices for realizing cluster dynamic retractility according to claim 14, wherein, described cluster performance packet Include：The computing resource utilization rate of cluster and/or the utilization ratio of storage resources of cluster.

16. devices for realizing cluster dynamic retractility according to claim 11, described for according to described collection group motion The device that the flexible demand of state adjusts the from node quantity in described computing resource group and/or storage resource group includes：

The device of the from node quantity expanding for determining computing resource group to need；

For creating the device of the virtual machine of described quantity；

For the device in the cluster by the log-on message configuration of described virtual machine；

For starting the device of the distributed computing services of described virtual machine；

For using virtual machine as the from node in computing resource group device.

17. devices for realizing cluster dynamic retractility according to claim 11, described for according to described collection group motion The device that the flexible demand of state adjusts the from node quantity in described computing resource group and/or storage resource group includes：

The device of the from node quantity expanding for determining storage resource group to need；

For creating the device of the virtual machine of described quantity；

For starting the distributed computing services of described virtual machine and the device of distributed storage service；

For using virtual machine as the from node in storage resource group device.

18. devices for realizing cluster dynamic retractility according to claim 16 or 17, described for will be described virtual The log-on message configuration of machine device in the cluster includes：

Device for virtual machine described in Telnet；

Exempt from the device of code entry authority for being directed to the setting of Telnet side in described virtual machine；

For by log-on message configuration including host name for the described virtual machine in the host node of cluster and each from node In device.

19. devices for realizing cluster dynamic retractility according to claim 11, wherein, described for according to described collection The device that group's dynamic retractility demand adjusts the from node quantity in described computing resource group and/or storage resource group includes：

For determining the device of the from node quantity needing to reduce in computing resource group and/or storage resource group；

For choosing the device being contracted by from node of respective numbers from respective sets；

For in cluster any one be configured with described in be contracted by from node log-on message node, hold in described node The gap of two tasks of row, is contracted by the device of the log-on message of from node described in deletion of node；

All delete for all nodes in the cluster in the case of being contracted by the log-on message of from node, delete and realize being contracted Subtract the virtual machine of from node, and the device that from node is deleted from computing resource group and/or storage resource group will be contracted by.

20. devices for realizing cluster dynamic retractility according to claim 19, wherein：