CN110535898A - Copy storage, completion, node selecting method and management system in big data storage - Google Patents

Abstract

The invention relates to a method and management system for replica storage, completion, node selection and management system in big data storage. The node selection method is as follows: according to the real-time status information and historical fault information of each data node server, the evaluation index of the replica storage node is selected, wherein the data The predicted value of the probability of failure is included in the evaluation index, the weight of each evaluation index is determined, and the data node is obtained by the calculation of the weight for copy storage. Based on the above node selection method, a suitable node is selected for copy storage according to the three-copy scheme. When a replica failure needs to be completed, first complete the replica according to the active node on the rack where the faulty node is located. When the rack where the replica faulty node is located cannot work normally, select the active node with a similar failure rate to complete the replica. The invention effectively improves the writing efficiency and load balancing degree during storage without affecting the copy safety, and fundamentally solves the problem that the cluster needs load balancing after running for a long time.

Description

Copy storage, completion, node selection method and management system in big data storage

technical field

The invention belongs to the technical field of big data storage and cloud computing, and in particular relates to a method and management system for copy storage, completion, node selection in big data storage.

Background technique

Existing big data storage systems are generally distributed storage (such as HDFS), and node failure and hardware failure are issues that must be considered. The application of replica technology ensures the reliability and availability of the system. Ensuring proper placement and selection of copies of data also ensures more efficient access to data. The big data storage strategy generally adopts the three-copy strategy, which ensures the security of data and can effectively support distributed computing. However, when considering the localized access of data, the distribution of copies is unreasonable, which will require high data localization. The impact is that tasks will be distributed to machines with replicas but with lower performance, resulting in performance degradation of the entire cluster.

The load balancing mentioned in the currently commonly used copy storage strategy mainly solves the load imbalance from the perspective of data volume and then balances the existing data. At this time, the essence of load balancing is the transfer of copies. Load balancing can only be regarded as a remedy for an unreasonable copy storage strategy. The ideal method should be to choose or adjust the location where the replica is stored according to the performance of the current cluster when placing the replica.

The existing copy storage strategy also proposes a method of placing according to the rack, hard disk usage, and load conditions. However, different methods refer to a single influencing factor when selecting them, and still cannot take into account the load balancing and storage efficiency issues. Especially in heterogeneous clusters, such problems are more obvious. For example, there may be excessive load on some machines with poor performance, while some machines with high performance are idle. Replication and transfer of replicas in big data storage clusters are generally caused by server hardware failures. Generally, servers are designed to be used for 5-7 years. The actual service life of a single server is related to server batches, usage intensity, and usage environment. are related. The current copy storage method does not consider the factors of server failure or aging, such as the reference "Improvement Strategy of HDFS Copy Placement Based on Support Vector Machine" (author: Luo Jun et al., Computer Engineering, Vol. 41, No. 11, Nov. 2015) Period) It only considers the relative load rate, network distance, disk performance, CPU performance and memory 5 factors, when the server fails, the data replication and migration actions are random, which will cause the disorder of replica placement.

From the perspective of operations research, the copy storage strategy of big data can be regarded as a decision-making problem, and it is a decision-making problem that is difficult to quantitatively analyze. In response to such problems, there is an analytic hierarchy process, which compares the evaluation indicators involved, and can quantitatively analyze the relative importance of several groups of relationships. Finally, the most ideal effect can be obtained by weighting various influencing factors. As long as the appropriate weights can be given, the ideal results can be obtained.

SUMMARY OF THE INVENTION

The present invention provides a method and management system for copy storage, completion, node selection in big data storage, so as to solve the problem that the existing cluster copy storage strategy does not effectively perceive the server status of each data node and select the influencing factors for reference It is relatively simple and cannot take into account the issues of load balancing and storage efficiency.

In order to solve the above technical problems, the method for selecting a replica storage node in the big data storage of the present invention includes the following four unit schemes:

Unit scheme 1: Select the evaluation indicators of the replica storage node according to the real-time status information and historical fault information of each data node server, including disk usage, disk I/O load rate, CPU load rate, memory load rate, read and write task connection rate and the node failure rate, the read and write task connection rate is the ratio of the number of connections of the current server read and write tasks to the maximum number of connections of read and write tasks allowed by the file system; determine the weights of each evaluation index, and then calculate according to the following formula: Refer to the value to select the data node as the replica storage location:

ω=λ ₀ ω _{disk_used} +λ ₁ ω _{disk_io} +λ ₂ ω _cpu +λ ₃ ω _mem +λ ₄ ω _process +λ ₅ ω _fr

Among them, ω is the data node selection reference value, ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr are the disk usage, disk I/O load rate, CPU load rate, memory load rate, read and write rate, respectively Task connection rate and node failure rate, λ ₀ +λ ₁ +λ ₂ +λ ₃ +λ ₄ +λ ₅ =1, λ ₀ , λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr ∈[0,1].

Unit plan 2: On the basis of unit plan 1, the weights of each evaluation index are determined by the AHP method, the weights of each evaluation index are described as a judgment matrix, and each evaluation index is stratified, and the relationship between the layers Quantitative analysis is realized, and a normalized eigenvector is finally obtained as a judgment matrix; when it is perceived that the cluster is processing different tasks, the corresponding matrix is adaptively matched to correct the appropriate replica placement position.

Unit scheme three, on the basis of unit scheme one, the node failure rate is the ratio of the data node failure time to the online running time or the ratio of the data node's service life to the design service life.

Unit scheme 4, on the basis of unit scheme 2, the node failure rate is the ratio of the data node failure time to the online running time or the ratio of the data node's service life to the design service life.

The method for storing copies in big data storage of the present invention includes the following four unit schemes:

Unit scheme 1, the default number of copies in this method is 3, two copies are stored on different nodes on the same rack, and the other is stored on nodes in different racks. When the copy starts to be stored, if the client is a data node , place the first copy on the node, if the client is not a data node, select a node for placing the first copy in the nodes on all racks according to the method for selecting copy storage nodes in the big data storage; then In a node with a different rack from the first replica, select a node for placing the second replica according to the method for selecting replica storage nodes in the big data storage, and select a node for placing the second replica in the node on the same rack as the second replica but on a different node. Selection method of replica storage node in big data storage Select the node to place the third replica.

Unit plan 2: On the basis of unit plan 1, the weights of each evaluation index are determined by the AHP method, the weights of each evaluation index are described as a judgment matrix, and each evaluation index is stratified, and the relationship between the layers Quantitative analysis is realized, and a normalized eigenvector is finally obtained as a judgment matrix; when it is perceived that the cluster is processing different tasks, the corresponding matrix is adaptively matched to correct the appropriate replica placement position.

Unit scheme three, on the basis of unit scheme one, the node failure rate is the ratio of the data node failure time to the online running time or the ratio of the data node's service life to the design service life.

Unit scheme 4, on the basis of unit scheme 2, the node failure rate is the ratio of the data node failure time to the online running time or the ratio of the data node's service life to the design service life.

The copy completion method in big data storage of the present invention includes the following five unit schemes:

Unit scheme 1: When the number of replicas to be completed is less than 3, obtain the rack where each lost replica is located, and determine whether there are active nodes on the rack where each faulty node is located. The specified node selection method selects data nodes for replica completion. If there is no active node, the nodes with the same failure rate as the faulty node are selected for replica completion according to the set node selection method.

Unit scheme 2, on the basis of unit scheme 1, the node selection method of the setting is: according to the real-time status information and historical fault information of each data node server, select the evaluation index of the copy storage node, including the disk usage rate, disk I /O load rate, CPU load rate, memory load rate, read-write task connection rate and node failure rate, the read-write task connection rate is the current server read-write task connection number and the maximum number of read-write tasks allowed by the file system. Determine the weight of each evaluation index, and then select the data node as the copy storage location according to the reference value calculated by the following formula:

ω=λ ₀ ω _{disk_used} +λ ₁ ω _{disk_io} +λ ₂ ω _cpu +λ ₃ ω _mem +λ ₄ ω _process +λ ₅ ω _fr

Among them, ω is the data node selection reference value, ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr are the disk usage, disk I/O load rate, CPU load rate, memory load rate, read and write rate, respectively Task connection rate and node failure rate, λ ₀ +λ ₁ +λ ₂ +λ ₃ +λ ₄ +λ ₅ =1, λ ₀ , λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr ∈[0,1].

Unit scheme 3, on the basis of unit scheme 2, adopts AHP to determine the weights of each evaluation index, describes the weights of each evaluation index as a judgment matrix, stratifies each evaluation index, and establishes the relationship between the layers. Quantitative analysis is realized, and a normalized eigenvector is finally obtained as a judgment matrix; when it is perceived that the cluster is processing different tasks, the corresponding matrix is adaptively matched to correct the appropriate replica placement position.

Unit scheme 4, on the basis of unit scheme 2, the node failure rate is the ratio of the data node failure time to the online running time or the ratio of the data node's service life to the design service life.

The fifth unit scheme, based on the third unit scheme, the node failure rate is the ratio of the failure time of the data node to the online running time or the ratio of the service life of the data node to the designed service life.

The copy management system in the big data storage of the present invention includes the following four unit schemes:

Unit scheme 1, the system can realize the following functions: select the evaluation indicators of the replica storage node according to the real-time status information and historical fault information of each data node server, including disk usage, disk I/O load rate, CPU load rate, memory load rate, read/write task connection rate and node failure rate, the read/write task connection rate is the ratio of the current server read/write task connection number to the maximum number of read/write task connections allowed by the file system; determine the weight of each evaluation index, Then select the data node as the replica storage location according to the reference value calculated by the following formula:

ω=λ ₀ ω _{disk_used} +λ ₁ ω _{disk_io} +λ ₂ ω _cpu +λ ₃ ω _mem +λ ₄ ω _process +λ ₅ ω _fr

Among them, ω is the data node selection reference value, ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr are the disk usage, disk I/O load rate, CPU load rate, memory load rate, read and write rate, respectively Task connection rate and node failure rate, λ ₀ +λ ₁ +λ ₂ +λ ₃ +λ ₄ +λ ₅ =1, λ ₀ , λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr ∈[0,1].

Unit plan 2: On the basis of unit plan 1, the weights of each evaluation index are determined by the AHP method, the weights of each evaluation index are described as a judgment matrix, and each evaluation index is stratified, and the relationship between the layers Quantitative analysis is realized, and a normalized eigenvector is finally obtained as a judgment matrix; when it is perceived that the cluster is processing different tasks, the corresponding matrix is adaptively matched to correct the appropriate replica placement position.

Unit scheme three, on the basis of unit scheme one, the node failure rate is the ratio of the data node failure time to the online running time or the ratio of the data node's service life to the design service life.

Unit scheme 4, on the basis of unit scheme 2, the node failure rate is the ratio of the data node failure time to the online running time or the ratio of the data node's service life to the design service life.

The beneficial effects of the present invention are: the present invention perceives the real-time status information and historical fault information of each data node server, provides a more reliable data node for copy storage for the management node, and effectively improves the storage time without affecting the copy security. The writing efficiency and load balancing degree of the cluster are basically solved, which fundamentally solves the problem of load balancing after the cluster runs for a long time.

In the present invention, the server failure information is included in the evaluation index, and the predicted value of the probability of data failure is used as a reference factor for storing data copies. When a data node actually fails, the data copies that need to be completed are completed according to the setting method. , to avoid the disorder of data storage, and minimize the overhead of copy completion. To a certain extent, it replaces the function of rack awareness.

When a replica fails and needs to be completed, the active node on the rack where the faulty node resides is given priority to complete the replica. When the rack where the replica faulty node is located cannot work normally, the active node with a similar failure rate is preferentially selected for replica completion. Taking into account the batch factor during server deployment, this can ensure that the node where the completed replica occurs is as close as possible to the faulty node. in the same batch, thus ensuring that copies are stored in close proximity.

Description of drawings

Fig. 1 is a design reference model diagram of a method for selecting a replica storage node in big data storage according to the present invention;

Fig. 2 is the flow chart of copy storage in big data storage of the present invention;

FIG. 3 is a flow chart of copy completion in big data storage according to the present invention.

Detailed ways

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings.

Embodiments of the method for selecting replica storage nodes in big data storage of the present invention

The method of this embodiment is to sense the real-time status information and historical fault information of each data node server, and select the evaluation index of the replica storage node according to the real-time status information and historical fault information of each data node server, including disk usage, disk I/O load rate, CPU load rate, memory load rate, read and write task connection rate and node failure rate; determine the weights of each evaluation index, and then select the data node as the replica storage location according to the reference value calculated by the following formula:

ω=λ ₀ ω _{disk_used} +λ ₁ ω _{disk_io} +λ ₂ ω _cpu +λ ₃ ω _mem +λ ₄ ω _process +λ ₅ ω _fr

Among them, ω is the data node selection reference value, ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr are the disk usage, disk I/O load rate, CPU load rate, memory load rate, read and write rate, respectively Task connection rate and node failure rate, λ ₀ +λ ₁ +λ ₂ +λ ₃ +λ ₄ +λ ₅ =1, λ ₀ , λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem , ω _process , ω _fr ∈[0,1].

Among them, ω _{disk_used} , ω _{disk_io} , ω _cpu , ω _mem can be obtained from the corresponding operating system in the cluster server, and the read and write task connection rate ω _process is the number of connections of the current server read and write tasks and the maximum number of read and write tasks allowed by the file system. The ω _fr parameter can be calculated by combining the ratio of the server's failure time to the online running time. In some clusters that do not record failures, this parameter can also be designed as the ratio of the service life to the design service life. Therefore, for the same data center, the model design of this parameter is related to the production time and deployment time of the server.

Once the weight of each evaluation index is determined, the placement strategy of the replica is basically determined. The weights of the evaluation indicators can be modified as needed. After modification, the scheme can be contracted to a method of placing copies according to a single evaluation indicator to suit special work situations. The weights of each evaluation index can be described as a judgment matrix. When it is perceived that the cluster is dealing with different tasks, the corresponding matrix is adaptively matched to correct the appropriate replica position.

The weights of the evaluation indicators are represented by a matrix [A B C D E F], in which A, B, C, D, E, and F represent the above six evaluation indicators, respectively. For example, the lowercase letter ab represents the relative relationship between the two layers of AB. The weight of each evaluation index can be determined according to the relative relationship between the evaluation indicators in Table 1, and finally a normalized eigenvector is obtained as the judgment matrix. Table 1 Calculated. For example, when it is perceived that the cluster is dealing with different tasks, the evaluation index A is given priority, then the evaluation index A accounts for the largest proportion, and then the other five evaluation indexes are set according to the relationship between the evaluation index A and the other five evaluation indexes. Weights. After the weight is set, the replica placement position is obtained according to the reference value calculated from the information of each node. The design idea is shown in Figure 1.

Table 1 Relative relationship matrix of evaluation indicators

A B C D E F A 1 ab ac ad ae af B 1/ab 1 bc bd be bf C 1/ac 1/bc 1 cd ce cf D 1/ad 1/bd 1/cd 1 de df E 1/ae 1/be 1/ce 1/de 1 ef F 1/af 1/bf 1/cf 1/df 1/ef 1

Embodiments of the copy storage method in big data storage of the present invention

The copy storage method of this embodiment is stored according to the three-copy scheme, and the reliability of the copies is ensured according to the principle of placing copies on two racks, that is, two copies are stored on different nodes on the same rack, and the other is stored on the same rack. on nodes in different racks. When the copy starts to be stored, if the client is a data node, the first copy is placed on the node. If the client is not a data node, the copies are stored in the nodes on all racks according to the big data storage in the above embodiment. The node selection method selects a node for placing the first copy; then selects a node for placing the second copy in a different node on the same rack as the first copy according to the method for selecting a copy storage node in the big data storage in the above-mentioned embodiment. In a node on a rack different from the rack where the first and second copies are located, the node is selected for placing the third copy according to the method for selecting a copy storage node in the large data storage in the foregoing embodiment.

The specific storage process is shown in Figure 2, when the number of copies to be stored is greater than 0, step 1) judges whether the first copy is to be stored, if so, enter step 2), otherwise enter step 3);

Step 2) judge whether the client is a data node, if so, go to step 4), otherwise go to step 5);

Step 3) judge whether what is to be stored is the second copy, if so, enter step 6), otherwise it is the third copy, enter step 7);

Step 4) Select the data node for placing the first copy;

Step 5) In the nodes on all racks, select a node for placing the first copy according to the method for selecting a copy storage node in the big data storage in the above-mentioned embodiment;

Step 6) In a node with a different rack from the first copy, select a node for placing the second copy according to the method for selecting a copy storage node in the big data storage in the above-mentioned embodiment;

Step 7) judge whether the first copy and the second copy are stored on the same rack, if so, enter step 8), otherwise enter step 9);

Step 8) In a node on a rack different from the rack where the first and second copies are located, select a node for placing the third copy according to the method for selecting a copy storage node in the large data storage in the above-mentioned embodiment, and enter step 10 );

Step 9) In a different node in the same rack as the second copy, select a node for placing the third copy according to the method for selecting a copy storage node in the large data storage in the above-mentioned embodiment, and enter step 10);

Step 10) The placement of the three copies is completed, and the process ends.

Embodiments of the Copy Completion Method in Big Data Storage of the Present Invention

The completion method of this embodiment is: when the number of copies to be completed is less than 3, obtain the rack where each lost copy is located, and determine whether there is an active node on the rack where each faulty node is located, and if there is an active node, in these In the active node, the data node is selected for the completion of the copy according to the set node selection method. If there is no active node, the node with the same failure rate as the faulty node is selected for the copy according to the set node selection method. Completion.

The specific completion process is shown in Figure 3. When the number of copies to be completed is greater than 0, it is judged whether the number of copies to be completed is equal to 3, if it is equal to 3, go to step 4), if it is less than 3, go to step 1) ;

Step 1) Obtain the rack where each lost copy is located, and determine whether there is an active node on the rack where each faulty node is located, if there is an active node, go to step 2), otherwise go to step 3);

Step 2) enumerate the active nodes of the rack where the faulty node is located, and calculate suitable nodes according to the set node selection method to complete the copy;

Step 3) enumerate the nodes with the same failure rate as the faulty node, and calculate the appropriate node according to the set node selection method to complete the copy;

Step 4) A bad block is generated, and an alarm is issued.

The set node selection method described in the above steps 2) and 3) may adopt the above-mentioned selection method of the replica storage node in the big data storage, or may adopt other node selection methods in the prior art, which will not be described in detail here. .

The embodiment of the copy management system in the big data storage of the present invention

The copy management system in big data storage in this embodiment is a management platform capable of implementing a method for selecting a node for copy storage in big data storage. For the method for selecting a replica storage node in the big data storage, reference may be made to the above-mentioned embodiment, which will not be described in detail here.

The specific embodiments to which the present invention relates are given above, but the present invention is not limited to the described embodiments. Under the idea given by the present invention, the technical means in the above-mentioned embodiments are transformed, replaced and modified in a manner that is easy for those skilled in the art to imagine, and the functions played are basically the same as those of the corresponding technical means in the present invention. The purpose of the invention is basically the same. The technical solution formed in this way is formed by fine-tuning the above embodiment, and this technical solution still falls within the protection scope of the present invention.

Claims (10)

1. the selection method of copy storage node in big data storage, which is characterized in that this method are as follows: taken according to each back end The real time status information and historical failure information of business device choose the evaluation index of copy storage node, including disk utilization rate, magnetic Disk I/O load factor, cpu load rate, memory load factor, read-write task bonding ratio and node failure rate, the read-write task connection Rate is the ratio for the read-write task maximum connection number that current server reads and writes the connection number of task and file system allows；It determines each The weight of a evaluation index, then calculated reference value selects the back end as copy storage position according to the following formula It sets:

ω=λ₀ω_{disk_used}+λ₁ω_{disk_io}+λ₂ω_cpu+λ₃ω_mem+λ₄ω_process+λ₅ω_fr

Wherein, ω is that back end selects reference value, ω_{disk_used}、ω_{disk_io}、ω_cpu、ω_mem、ω_process、ω_frRespectively magnetic Disk utilization rate, magnetic disc i/o load factor, cpu load rate, memory load factor, read-write task bonding ratio and node failure rate, λ₀+λ₁+ λ₂+λ₃+λ₄+λ₅=1, λ₀、λ₁、λ₂、λ₃、λ₄、λ₅、ω_{disk_used}、ω_{disk_io}、ω_cpu、ω_mem、ω_process、ω_fr∈[0,1]。

2. the selection method of copy storage node in big data storage according to claim 1, which is characterized in that use layer Fractional analysis determines the weight of each evaluation index, and the weight of each evaluation index is described as judgment matrix, refers to each evaluation Mark is layered, and quantitative analysis is realized in the connection between each layer, finally seeks a normalization characteristic vector as judgment matrix； When perceiving cluster when handling different task, corresponding matrix is adaptively matched, to correct suitable Replica placement position.

3. the selection method of copy storage node in big data storage according to claim 1 or 2, which is characterized in that institute State the ratio or back end years already spent and design that node failure rate is back end fault time and on-line operation time The ratio of service life.

4. using copy is deposited in the big data storage of the selection method of copy storage node in the storage of big data described in claim 1 Put method, default number of copies is 3 in this method, and two of them copy is stored on the different nodes in same rack, in addition one On a node for being stored in different racks, which is characterized in that when copy starts storage, if client is back end, by first On a Replica placement node, if client is not back end, deposited in the node on institute's organic frame according to the big data The selection method selection node of copy storage node is for placing the first authentic copy in storage；Then with first copy difference rack Node according to the big data storage in copy storage node selection method selection node for place second copy, It is selected from triplicate same machine frame and different nodes according to the selection method of copy storage node in big data storage Node is selected for placing triplicate.

5. copy deposit method in big data storage according to claim 4, which is characterized in that true using analytic hierarchy process (AHP) The weight of fixed each evaluation index, is described as judgment matrix for the weight of each evaluation index, is layered to each evaluation index, Quantitative analysis is realized in connection between each layer, finally seeks a normalization characteristic vector as judgment matrix；Collect when perceiving Group matches corresponding matrix, adaptively when handling different task to correct suitable Replica placement position.

6. copy complementing method in big data storage, which is characterized in that when needing the number of copies of completion less than 3, acquisition is respectively lost The rack where copy is lost, and judges that each malfunctioning node whether there is active node on the rack, active node is then if it exists The completion of copy is used for according to the node selecting method selection back end of setting in these active nodes, it is movable if it does not exist Node, then the node selecting method selection node from node identical with malfunctioning node failure rate according to setting is for copy Completion.

7. copy complementing method in big data storage according to claim 6, which is characterized in that the node of the setting selects Method are as follows: the evaluation of copy storage node is chosen according to the real time status information of each back end server and historical failure information Index, including disk utilization rate, magnetic disc i/o load factor, cpu load rate, memory load factor, read-write task bonding ratio and node event Barrier rate, the read-write task that the read-write task bonding ratio reads and writes the connection number of task for current server and file system allows is most The ratio of big connection number；Determine the weight of each evaluation index, then calculated reference value selects number according to the following formula According to node as copy storage position:

ω=λ₀ω_{disk_used}+λ₁ω_{disk_io}+λ₂ω_cpu+λ₃ω_mem+λ₄ω_process+λ₅ω_fr

Wherein, ω is that back end selects reference value, ω_{disk_used}、ω_{disk_io}、ω_cpu、ω_mem、ω_process、ω_frRespectively magnetic Disk utilization rate, magnetic disc i/o load factor, cpu load rate, memory load factor, read-write task bonding ratio and node failure rate, λ₀+λ₁+ λ₂+λ₃+λ₄+λ₅=1, λ₀、λ₁、λ₂、λ₃、λ₄、λ₅、ω_{disk_used}、ω_{disk_io}、ω_cpu、ω_mem、ω_process、ω_fr∈[0,1]。

8. copy complementing method in big data storage according to claim 7, which is characterized in that true using analytic hierarchy process (AHP) The weight of fixed each evaluation index, is described as judgment matrix for the weight of each evaluation index, is layered to each evaluation index, Quantitative analysis is realized in connection between each layer, finally seeks a normalization characteristic vector as judgment matrix；Collect when perceiving Group matches corresponding matrix, adaptively when handling different task to correct suitable Replica placement position.

9. replica management system in big data storage, which is characterized in that the system can be realized following functions: according to each data section The real time status information and historical failure information of point server choose the evaluation index of copy storage node, including disk uses Rate, magnetic disc i/o load factor, cpu load rate, memory load factor, read-write task bonding ratio and node failure rate, the read-write task Bonding ratio is the ratio for the read-write task maximum connection number that current server reads and writes the connection number of task and file system allows；Really The weight of fixed each evaluation index, then calculated reference value selects the back end as copy storage according to the following formula Position:

ω=λ₀ω_{disk_used}+λ₁ω_{disk_io}+λ₂ω_cpu+λ₃ω_mem+λ₄ω_process+λ₅ω_fr

Wherein, ω is that back end selects reference value, ω_{disk_used}、ω_{disk_io}、ω_cpu、ω_mem、ω_process、ω_frRespectively magnetic Disk utilization rate, magnetic disc i/o load factor, cpu load rate, memory load factor, read-write task bonding ratio and node failure rate, λ₀+λ₁+ λ₂+λ₃+λ₄+λ₅=1, λ₀、λ₁、λ₂、λ₃、λ₄、λ₅、ω_{disk_used}、ω_{disk_io}、ω_cpu、ω_mem、ω_process、ω_fr∈[0,1]。

10. replica management system in big data storage according to claim 9, which is characterized in that use analytic hierarchy process (AHP) The weight of each evaluation index is described as judgment matrix, divided each evaluation index by the weight for determining each evaluation index Layer, quantitative analysis is realized in the connection between each layer, finally seeks a normalization characteristic vector as judgment matrix；When perceiving Cluster adaptively matches corresponding matrix when handling different task, to correct suitable Replica placement position.