TW201732591A - Disk failure prediction method and apparatus - Google Patents

Abstract

Disclosed are a disk failure prediction method and apparatus. The method comprises: acquiring sample disk data of a disk through disk monitoring technology, the sample disk data comprising sample data on a plurality of dimensions; performing sample training on the sample disk data by using a GBDT algorithm, to obtain a disk prediction model consisting of a plurality of decision-making trees; and after disk data of a disk to be predicted is received, processing the disk data of the disk to be predicted by using the disk prediction model consisting of the plurality of decision-making trees, and determining whether the disk to be predicted is a failed disk. The method solves the technical problem in the prior art of an inaccurate prediction result caused by the fact that some factors resulting in hard disk failures cannot be collected or quantized in a hard disk failure prediction system.

Description

Disk failure prediction method and device

The present invention relates to the field of magnetic disks, and more particularly to a method and apparatus for predicting failure of a magnetic disk.

At present, the hard disk is the main medium for storing data, and if the hard disk fails, it will cause huge data loss. Therefore, how to ensure the stability of the hard disk can be very important. Under normal conditions, the probability of a hard disk error in 24 hours is about one ten thousandth. When a server has ten hard disks, the chance of a server hard disk error will rise to one thousandth. With the development of services such as the current website, the number of hard disks that the server needs to use will increase, and the probability of multiple hard disks simultaneously failing will increase.

Usually, the data storage usually has multiple backups, such as the mysql main standby library, and the GFS file defaults to 3 backups. On a large data storage platform, if multiple hard disks fail at the same time, the chances of storing the same file on these hard disks will be high, that is, if multiple hard disks fail at the same time, it will lead to some files. Loss, for some online services, mostly rely on a large amount of data stored in the server. If the hard disk fails, it will cause the above-mentioned online service to be abnormal or even suspended.

For the above reasons, systems that need to predict whether a hard disk will go wrong need a system that can tell us in advance which hard disk will be wrong. There are many reasons why the data may be lost. The most common ones are: external vibration , temperature and humidity, electrical component damage, sound and dust, some of the above factors can be collected, such as temperature and humidity, some component data, but more data can not be collected and quantified, thus leading to predictions The result is not accurate.

In view of the problem that some factors in the prior art hard disk failure prediction system that are likely to cause a hard disk failure cannot be inaccurate due to acquisition or quantification, an effective solution has not been proposed yet.

Embodiments of the present invention provide a method and apparatus for predicting a failure of a magnetic disk, so as to at least solve the technical problem that some prediction factors in the prior art hard disk failure prediction system that cause the failure of the hard disk cannot be collected or quantized are inaccurate. .

According to an aspect of an embodiment of the present invention, a method for predicting a failure of a magnetic disk includes: acquiring a magnetic disk data of a magnetic disk by using a disk monitoring technology, wherein the sample disk data includes sample data in multiple dimensions. The GBDT algorithm is used to perform sample training on the sample disk data to obtain a disk prediction model composed of multiple decision trees. After receiving the disk data of the disk to be tested, a disk composed of multiple decision trees is used. The prediction model processes the disk data of the disk to be determined to determine whether the disk to be tested is a failed disk.

According to another aspect of the embodiments of the present invention, a fault prediction apparatus for a magnetic disk is provided, comprising: acquiring a magnetic disk data of a magnetic disk by using a disk monitoring technology, wherein the sample disk data includes multiple dimensions. Sample data; using the GBDT algorithm to sample the sample disk data to obtain a disk prediction model composed of multiple decision trees; after receiving the disk data of the disk to be tested, using a plurality of decision trees The disk prediction model processes the disk data of the disk to be determined to determine whether the disk to be tested is a failed disk.

In the embodiment of the present invention, the sample disk data of the disk is obtained by using a disk monitoring technology, wherein the sample disk data includes sample data in multiple dimensions; and the sample disk disk data is sampled by using the GBDT algorithm. Obtaining a disk prediction model method composed of a plurality of decision trees, after processing the disk data of the disk to be tested, using a disk prediction model composed of a plurality of decision trees to process the disk data of the disk to be tested The purpose of determining whether the disk to be tested is a faulty disk is achieved, thereby realizing the technical effect of predicting the disk fault state, thereby solving some factors in the prior art hard disk fault prediction system that can easily cause the hard disk failure to be Collect or quantify technical problems that result in inaccurate predictions.

10‧‧‧Computer terminal

102‧‧‧Processor

104‧‧‧ memory

106‧‧‧Transmission module

60‧‧‧Getting module

62‧‧‧ training module

64‧‧‧Processing module

70‧‧‧ Computing Module

80‧‧‧ initial module

82‧‧‧ extraction module

84‧‧‧First Computing Module

90‧‧‧Reading module

92‧‧‧Comparative Module

94‧‧‧Determining modules

96‧‧‧Processing submodules

100‧‧‧ receiving module

102‧‧‧Second calculation module

104‧‧‧ traversal module

111‧‧‧ Processor

113‧‧‧ memory

115‧‧‧Transportation device

The drawings described herein are intended to provide a further understanding of the invention, and are intended to be a part of the invention. In the drawing: 1 is a block diagram of a hardware structure of a computer terminal for predicting a failure of a disk according to an embodiment of the present invention; FIG. 2 is a flowchart of a method for predicting a failure of a disk according to an embodiment of the present invention; A schematic diagram of training a sample disk data using a GBDT algorithm according to an embodiment of the present invention; FIG. 4 is a schematic diagram of calculating a disk prediction value using a GBDT algorithm according to an embodiment of the present invention; FIG. 5 is a schematic diagram of a disk according to an embodiment of the present invention; FIG. 6 is a schematic structural diagram of a failure prediction apparatus for a magnetic disk according to an embodiment of the present invention; FIG. 7 is an optional magnetic disk according to an embodiment of the present invention; FIG. 8 is a schematic structural diagram of an optional disk fault prediction apparatus according to an embodiment of the present invention; FIG. 9 is an optional disk fault prediction apparatus according to an embodiment of the present invention; FIG. 10 is a schematic structural diagram of an optional disk fault prediction apparatus according to an embodiment of the present invention; and FIG. 11 is an embodiment of the present invention. Species block diagram of the computer terminal.

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is an embodiment of the invention, but not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of the present invention.

It should be noted that the terms "first", "second" and the like in the specification and claims of the present invention are used to distinguish similar items, and are not necessarily used to describe a specific order or order. It is to be understood that the materials so used are interchangeable, where appropriate, so that the embodiments of the invention described herein can be carried out in a sequence other than those illustrated or described herein. In addition, the terms "comprises" and "comprises" and "the" and "the" are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to Those steps or units may include other steps or units not explicitly listed or inherent to such processes, methods, products or devices.

Example 1

According to an embodiment of the present invention, there is also provided an embodiment of a method for predicting a failure of a disk, and it is to be noted that the steps shown in the flowchart of the drawing may be performed in a computer system such as a set of computer executable instructions, and Although the logical order is shown in the flowchart, in some cases, The steps shown or described are performed in a different order than here.

The method embodiment provided in Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or the like. Taking a computer terminal as an example, FIG. 1 is a block diagram showing a hardware structure of a computer terminal for predicting a failure of a disk according to an embodiment of the present invention. As shown in FIG. 1, computer terminal 10 may include one or more (only one shown) processor 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc. ), a memory 104 for storing data, and a transmission module 106 for communication functions. It will be understood by those skilled in the art that the structure shown in FIG. 1 is merely illustrative and does not limit the structure of the above electronic device. For example, computer terminal 10 may also include more or fewer components than those shown in FIG. 1, or have a different configuration than that shown in FIG.

The memory 104 can be used to store software programs and modules of the application software, such as the program instructions/modules corresponding to the method for predicting the failure of the disk in the embodiment of the present invention, and the processor 102 runs the software program stored in the memory 104. And modules to perform various functional applications and data processing, that is, to implement the vulnerability detection method of the above application. The memory 104 can include high speed random memory and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, memory 104 can further include memory disposed remotely from processor 102, which can be connected to computer terminal 10 via a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and groups thereof. Hehe.

The transmission module 106 is configured to receive or transmit data via a network. The above specific network example may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transport module 106 includes a network interface controller (NIC) that can be connected to other network devices through the base station to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module for communicating wirelessly with the Internet.

In the above operating environment, the present application provides a method for predicting the failure of a disk as shown in FIG. 2. 2 is a flow chart of a method for predicting a failure of a magnetic disk according to an embodiment of the present invention.

It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the present invention is not limited by the described action sequence. Because certain steps may be performed in other sequences or concurrently in accordance with the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of a software plus a necessary universal hardware platform, and of course, can also be through hardware, but in many cases The former is a better implementation. Based on such understanding, the technical solution of the present invention may be essential in nature or contribution to the prior art. In the form of a software product, the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), and includes a number of instructions for making a terminal device (can be a mobile phone, a computer, a server) , or a network device, etc.) performs the methods described in various embodiments of the present invention.

In the above operating environment, the present application provides a method of processing the decompiled data as shown in FIG. 2. 2 is a flowchart of a method for processing decompiled data according to Embodiment 1 of the present invention. As shown in FIG. 2, the method includes: Step 21: acquiring a sample disk data of a disk by using a disk monitoring technology, where The sample disk data includes sample data in multiple dimensions.

In the above steps, the disk monitoring technology is used to monitor the disk data generated during the use of the disk after the factory to predict the fault state of the disk, so that the disk user can before the disk fails. Know that the disk is about to fail, so that the data in the disk is copied and stored to avoid data loss.

In an optional embodiment, the sample disk data may include: an underlying data read error rate, a start/stop count, a number of remapping magnetic regions, an energization time accumulation, a spindle spin retry count, and a disk calibration weight. The number of trials, the number of disk power-ons, the temperature, and the write error rate allow you to obtain sample disk data based on disk history failures. For example, the sample acquisition can be performed at a ratio of positive to negative sample ratio of 1:5, wherein the positive sample is a disk with a fault, and the negative sample is a disk with no fault.

What needs to be explained here is that the disk is obtained by the disk monitoring technology. In the case of sample disk data, the disks used by the various organizations that predict disk failure are not necessarily the same, and because of the influence of environmental factors such as temperature and humidity of various mechanisms on the disk, the ratio of the disk of different organizations is good or bad. Differently, in order to provide more reliable sample disk data for the training of sample disk data, it is also possible to obtain sample disk data according to the actual disk damage of the mechanism.

Step S23: Perform sample training on the sample disk data by using a GBDT algorithm to obtain a disk prediction model composed of multiple decision trees.

In the above steps, GBDT (Gradient Boosting Decision Tree) is a decision tree algorithm of repeated operations. The algorithm consists of multiple decision trees and accumulates the conclusions of all decision trees to obtain the final result. As a kind of prediction model, the above decision tree is based on the result of the previous decision, and makes the next decision, including decision points, state nodes, result nodes, etc. Each node in the tree represents the predicted object. Each of the bifurcation paths represents a possible attribute of the object.

In an optional embodiment, in the case that the sample disk is the original value of the SMART of the disk, the sample disk is sample-trained, for example, the original value is greater than or equal to the preset original value, and the sample magnetic is considered as The chance of a disk failure is large. When the original value is less than the preset value, the probability that the sample disk is faulty is considered to be small. Therefore, when the disk prediction model is determined, the original value of the sample disk is greater than or equal to the preset. In the case of the original value, it is confirmed that the attribute of the sample disk is a failure, and if the original value of the sample disk is less than the preset original value, it is confirmed that the attribute of the sample disk is non-faulty. Establish a disk prediction model with the above decision-making ability If the original value of the disk to be detected is greater than or equal to the preset original value, the decision tree automatically confirms that the disk to be detected is faulty, and confirms that the attribute of the sample disk is faulty. If the original value of the sample disk is less than the preset original value, confirm that the attribute of the sample disk is non-faulty.

Step S25, after receiving the disk data of the disk to be tested, processing the disk data of the disk to be tested by using the disk prediction model composed of a plurality of decision trees to determine the magnetic field to be tested Whether the disc is a failed disk.

In an optional embodiment, the values of the plurality of dimensions of the sample disk are used as evaluation indexes of the decision tree to obtain a plurality of decision trees, and then a plurality of decision trees form a disk prediction model, and the disk to be detected is performed. Detection.

It is worth noting here that the decision trees obtained according to each dimension of the disk may be the same and may not be the same. Therefore, when using multiple decision trees to construct the disk prediction model, it is necessary to be important in the evaluation system according to each decision tree. To determine the weight value of each decision tree, and to obtain a disk prediction model.

What needs to be explained here is that when the disk disk data of the disk is obtained through the disk monitoring technology, the disk detecting technology is adopted, so that the process of acquiring the sample disk data is simpler and the acquired data is more comprehensive. A wealth of disk sample data is provided for the training of sample disk data. In the above steps, the sample training of the sample disk data by using the GBDT algorithm may be performed in two or more times to improve the accuracy and recall of the disk prediction model composed of the decision tree corresponding to the training result. rate.

Therefore, the solution of the above-mentioned Embodiment 1 provided by the present application solves the existing solution. Some of the technical hard disk failure prediction systems that are susceptible to hard disk failure cannot be collected or quantized to cause inaccurate prediction results.

According to the above embodiment of the present application, in a preferred solution, the sample disk data includes at least sample data in four dimensions: original value, standard value, worst value, and cumulative value.

The above original value is the current parameter of the disk running; the above standard value is the value of each parameter of the normal disk running; the above difference is when the disk is running, the various detecting parameters of the disk have appeared and normal values. The abnormal value with the largest deviation; the above cumulative value is the cumulative result of each detection parameter of the disk from the use of the disk to the current time.

In an optional embodiment, the parameters of the disk may be information describing various attributes of the disk, and may include an error reading rate, a power-on frequency, a redistributed magnetic area number, and a rotating retry number. One or more of the number of disk calibration retries and the parity error rate may also include other attribute information of the disk.

The above steps of the present application can respectively obtain a plurality of different decision trees by using the sample data in the above four dimensions.

In an optional embodiment, the file disk data can be obtained by using software such as HDTune or CrystalDiskInfo.

According to the above embodiment of the present application, in a preferred solution, after acquiring the sample disk data of the disk by using the disk monitoring technology, the method further includes: step S211, performing sample data in each dimension Any one or more of the following operations: differential operation, square operation, and distribution summation Calculate, so that the sample data in any dimension is expanded to the sample data in the new dimension.

In the above steps, the decision result is further calculated, and the decision tree can be expanded into a new dimension according to the operation result, and the sample data in this dimension is obtained.

When it is worth noting here, the sample data of each dimension can be subjected to various operations to obtain more dimensional sample data on the basis of this dimension. On the basis of four dimensions, each dimension is separately differentiated. By computing, squaring, and distributed summation, it is possible to obtain sample data in sixteen dimensions, and the focus of decision making through sample data for each dimension is different.

In an optional embodiment, the sample data of the original value is still taken as an example, and the sample data of the original value is subjected to a difference operation, a square operation, and a distribution sum operation, thereby obtaining a sample of the new four dimensions. Data, using the new four dimensions of sample data to train the most decision-making indicators, and get a new four decision trees.

According to the above embodiment of the present application, in a preferred solution, the sample disk data is sample-trained by using a GBDT algorithm to obtain a disk prediction model composed of a plurality of decision trees, including: step S231, with all magnetics The sample disk data of the disk is used as training data, and the classification model parameters of the training data are initialized by using preset values.

In the above steps, the classification model parameter of the initialization training data may be preset the number of the above decision trees and the number of layers of each decision tree, that is, the initial setting of the attributes of the decision tree.

Step S233, extracting a plurality of feature data in the training data, creating each of the plurality of decision trees as a root node, and using the feature value corresponding to each feature data as a leaf node of the corresponding decision tree. .

Step S235: Calculate an optimal partition of all current leaf nodes and a gain thereof, and perform splitting with the leaf node with the largest gain and the corresponding split point, so that the sample disk data is divided into the child nodes.

In the above steps, the gain may be the minimum mean square error of the label value, that is, the square of the difference between the label value of each sample and the predicted label value, and calculate the sum of the squares of all the differences, which may be considered to be predicted. The more samples that are in error, the greater the mean square error, so the best branching basis can be found by minimizing the mean square error.

The decision tree may be a binary tree with each feature data as a root node, and each special data corresponds to a feature value, and the feature value is a leaf node of a decision tree with the feature data as a root node. After determining the leaf nodes of the decision tree, the leaf nodes are further divided. It is worth noting that when the leaf nodes are further divided, the gain is maximized when the gains of the plurality of leaf nodes are different. Leaf nodes, so that all sample data can be divided into corresponding leaf nodes.

In an optional embodiment, the sample disks are four disks A, B, C, and D, wherein the A disk and the B disk are normal disks, and the C disk and the D disk are The damaged disk, in this example, the normal disk corresponds to 0, and the failed disk corresponds to 1, so the four disks A, B, C, and D correspond to 0, 0, 1, 1, respectively. Get the above disk in the first dimension The upper eigenvalue is A, and the sample disk data is trained using the GBDT algorithm. FIG. 3 is a schematic diagram of training the sample disk data using the GBDT algorithm according to an embodiment of the present invention. The default initial value is 0.5, that is, the probability of each disk being a failed disk is 0.5, the threshold of the first dimension is A0, and the disk with the feature value greater than A0 is divided into a child node, and the feature in the first dimension is A disk whose value is less than or equal to A0 is divided into another child node, and the probability that the disk of the two child nodes is a failed disk is 0.5.

It should be noted that, in the above embodiment, for convenience of description, only four sample data are selected for description, so only two leaf nodes are obtained. In practical applications, after the root node is divided into two leaf nodes, You can continue to divide, the larger the sample data, the more the level of division.

According to the above embodiment of the present application, in a preferred solution, a plurality of feature data in the training material are extracted, each feature data is used as a root node to create the plurality of decision trees, and each feature data is correspondingly The feature value is used as a leaf node of the corresponding decision tree, and includes: Step S2331, reading a threshold corresponding to any feature data.

Step S2333, comparing the feature value of any one of the feature data with the threshold value, and obtaining the entropy of the two branches according to the comparison result.

Step S2335, determining two new nodes as two leaf nodes of the arbitrary one of the feature data according to the entropy of the two branches.

In step S2337, each feature data is processed by the above steps until each feature data obtains two predetermined unique leaf nodes.

In the above steps, exhaust each threshold of each feature, find So that the feature and the threshold with the minimum entropy of the two branches whose characteristics are less than or equal to the threshold and the feature is greater than the threshold are obtained, according to the standard branch, two new nodes are obtained, and the branching is continued in the same way until all the samples are divided into only A normal disk or a leaf node with only a failed disk, or a preset termination condition is reached. If the final leaf node does not have only a normal disk or a failed disk, the average tag value of all samples on the node is used as the leaf node. The predicted tag value.

It should be noted here that the tag value is the probability that the disk is a failed disk.

It should be noted here that the minimum entropy means that as far as possible, the ratio of positive and negative samples in each branch is far from 1:1, and the case of minimum entropy is that there are only positive or negative samples on the branch. That is, there is only a normal disk on the branch, or a failed disk.

In an optional embodiment, in the example where the decision tree is a regression tree, each node obtains a predicted value equal to the average of all the tag values belonging to the node, and when the node is divided, Exhausting each threshold of each feature, finding the best segmentation point, until the tag value of each sample on each leaf node is unique or reaches the preset termination condition, if the label value of the sample on the final leaf node Non-unique, the average tag value of all samples on the node is used as the predicted tag value of the leaf node.

It should be noted here that in the above embodiment, the optimal partitioning criterion is no longer to minimize the entropy, but to minimize the mean square error, that is, the difference between the label value of each sample and the predicted label value. Squared and calculate all differences The sum of the squares can be considered as the more samples that are predicted to be erroneous, the greater the mean square error, so the best branching basis can be found by minimizing the mean square error.

It should also be noted here that it is difficult to achieve the unique label value of each sample on each leaf node when performing the partitioning, so a termination condition can be preset in order to obtain the prediction result closest to the real situation. The termination condition can be the upper limit of the leaf.

According to the above embodiment of the present application, in a preferred solution, after obtaining a disk prediction model composed of a plurality of decision trees, the method further comprises: adjusting the classification model parameters, wherein the classification In the case where the model parameters include a failed disk sample and a non-faulty disk sample, if it is determined whether the disk to be tested is a failed disk, the proportion of the failed disk sample in the classification model parameter is increased.

According to the above embodiment of the present application, in a preferred solution, the disk data of the disk to be tested is processed by using the disk prediction model composed of a plurality of decision trees to determine whether the disk to be tested is For the faulty disk, the method includes the following steps: Step S251: After receiving the disk data of the disk to be tested, assign an initial value to the disk data of the disk to be tested.

Step S253, traversing each decision tree according to the initial value of the disk to be tested, calculating a prediction result and a first residual determined by the first decision tree, and assigning the first residual to the initial Value, get the updated initial value.

Step S255, calculating, by using the updated initial value, a prediction result determined by the second decision tree and a second residual, and the second residual is assigned The updated initial value is used to traverse all the decision trees to obtain a result of predicting whether the disk to be tested is a failed disk.

In step S257, each tree learns the residual of all previous tree conclusions, and the residual is an accumulated amount that can obtain the true value after adding the predicted value.

In an optional embodiment, the four disks of the above A, B, C, and D are still taken as an example. The feature A can divide the four disks A, B, C, and D into two parts, respectively A, B and C, D, each part uses the average tag value as the predicted value. At this time, the residual is calculated, wherein the residual is the difference between the predicted value of the disk and the actual value of the disk, so the residual of A is 1-0.5=0.5 and the residuals of A, B, C, and D are respectively 0.5, -0.5, 0.5, -0.5. Then, as shown in FIG. 4, FIG. 4 is a schematic diagram of calculating a disk prediction value using a GBDT algorithm according to an embodiment of the present invention, using the residual to replace the original values of A, B, C, and D, and inputting to the second decision. The tree is trained and divided into two leaf nodes according to the comparison result with feature B. If the predicted values and their residuals are equal, then only the conclusion of the second tree is added to the first tree to obtain The actual value of the disk. The second tree has only two values of 0.5 and -0.5, so it is split directly into two nodes. At this point everyone's residual is 0, that is, everyone gets real predictions.

It should be noted that the above embodiment is for the purpose of description, so there are only two decision trees. In practical applications, a decision tree can be obtained according to the sample data amount, and the predicted value refers to the sum of all the previous trees. Since in this embodiment, the decision tree has only one decision tree before, it is directly 0.5. If there is a strange decision tree, it needs to be added up as the predicted value of A.

FIG. 5 is a flow chart of an optional method for predicting a failure of a magnetic disk according to an embodiment of the present invention. A preferred embodiment of the present application is described in detail below with reference to FIG. 5.

As shown in FIG. 5, a method for predicting a failure of a magnetic disk is provided. The method may include the following steps S51 to S57: S51: acquiring sample data of a sample disk.

Specifically, in the above steps, the sample disk data can be obtained through software such as HDTune and CrystalDiskInfo.

S52, performing differential operations on the sample data.

Specifically, in the above steps, the difference operation refers to a value obtained by performing a difference operation between the feature data of the disk at a certain time and the feature data of the disk before 24 hours.

S53, performing a distribution summation and/or a square operation on the result obtained by the difference operation.

S54, obtaining training and prediction data.

S55, the first step of training and forecasting, makes the recall rate larger.

S56, the second step of training and forecasting, balances recall and accuracy.

Specifically, in the above steps, since the proportion of negative samples in the training data is large, the proportion of positive samples is small. For example, when the ratio of the two is 1000:1, if all the training materials are used for training, the prediction can be accurately predicted. Positive samples are rare. Because there are few positive samples in the training data, many data with negative real samples may be misjudged as positive samples. Therefore, the first step is to make the positive sample recall rate higher during training. The second step is During training, the training data predicted as the positive sample in the first step is used as the training data of the second step, that is, the positive and negative samples are selected. Those samples that are close to each other are used as training samples. Therefore, when training, the trained model will be more conducive to predicting positive samples, so that the results obtained by the second step prediction will have a higher accuracy than the first step. , so that the accuracy and recall rate reach a certain degree of balance.

Example 2

According to an embodiment of the present invention, there is also provided a processing apparatus for implementing decompilation data for implementing the processing method of the decompiled data, and FIG. 6 is a schematic structural diagram of a fault prediction apparatus for a magnetic disk according to an embodiment of the present invention. As shown in FIG. 6, the device includes: an acquisition module 60, a training module 62, and a processing module 64.

The module 60 is configured to acquire the sample disk data of the disk by using a disk monitoring technology, wherein the sample disk data includes sample data in multiple dimensions; and the training module 62 is configured to use the GBDT algorithm. Performing sample training on the sample disk data to obtain a disk prediction model composed of a plurality of decision trees; and processing module 64, after receiving the disk data of the disk to be tested, using the plurality of decision trees The disk prediction model processes the disk data of the disk to be tested to determine whether the disk to be tested is a failed disk.

It should be noted that the above-mentioned acquisition module 60, the training module 62, and the processing module 64 are the same as the application scenario of the step S21 to the step S25 of the embodiment, but are not limited to the foregoing embodiment. Public content. It should be noted that the above module can be operated as part of the device in the computer terminal 10 provided in the first embodiment.

According to the above embodiment of the present application, in a preferred embodiment, the sample disk data is SMART disk data, wherein the sample disk data includes at least sample data in the following four dimensions: original value, standard value , the worst value and the cumulative value.

According to the above embodiment of the present application, in a preferred solution, as shown in FIG. 7, the apparatus further includes: an operation module 70, configured to perform any one or more of the following operations on the sample data in each dimension: The difference operation, the square operation, and the distribution sum operation are such that the sample data in any one dimension is expanded to the sample data in the new dimension.

It should be noted that the above-mentioned operation module 770 is the same as the example and application scenario implemented in step S21 to step S25 in the first embodiment, but is not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as part of the device in the computer terminal 10 provided in the first embodiment.

According to the above embodiment of the present application, in a preferred solution, as shown in FIG. 8 , the training module 62 further includes: an initial module 80 for using the sample disk data of all the disks as training materials, and adopting The preset value initializes the classification model parameter of the training data; the extraction module 82 is configured to extract a plurality of feature data in the training data, and create each of the plurality of decisions by using each feature data as a root node. a tree, and the feature value corresponding to each feature data is used as a leaf node of the corresponding decision tree; the first calculation module 84 is configured to calculate an optimal partition of all current leaf nodes and a gain thereof, and the leaf node with the largest gain And dividing the corresponding dividing points to divide the sample disk data into the child nodes.

It should be noted that the initial module 80, the extraction module 82, and the first computing module 84 are the same as the application scenarios implemented in steps S231 to S235 of the embodiment, but are not limited to the above embodiments. A public content. It should be noted that the above module can be operated as part of the device in the computer terminal 10 provided in the first embodiment.

According to the above embodiment of the present application, in a preferred solution, as shown in FIG. 9, the extraction module 82 includes: a reading module 90 for reading a threshold corresponding to any feature data; and a comparison module 92. And comparing the feature value of the any one of the feature data with the threshold, and obtaining the entropy of the two branches according to the comparison result; the determining module 94 is configured to determine two new according to the entropy of the two branches. The node serves as two leaf nodes of the arbitrary feature data; the processing sub-module 96 is configured to process each feature data by using the above steps until each feature data obtains two predetermined unique leaf nodes.

It should be noted here that the above reading module 90, the comparison module The example and the application scenario implemented by the step S2331 to the step S2337 are the same as those of the first embodiment, but are not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as part of the device in the computer terminal 10 provided in the first embodiment.

According to the above embodiment of the present application, in a preferred solution, after obtaining a disk prediction model composed of a plurality of decision trees, the method further comprises: adjusting the classification model parameters, wherein the classification In the case where the model parameters include a failed disk sample and a non-faulty disk sample, if it is determined whether the disk to be tested is a failed disk, the proportion of the failed disk sample in the classification model parameter is increased.

According to the above embodiment of the present application, in a preferred solution, as shown in FIG. 10, the processing module 64 includes: a receiving module 100, configured to receive the disk data of the disk to be tested, and then The disk data of the disk to be tested is given an initial value; the second computing module 102 is configured to traverse each of the decision trees according to the initial value of the disk to be tested, and calculate the prediction determined by the first decision tree. a result and a first residual, and assigning the first residual to the initial value to obtain an updated initial value; the traversing module 104, configured to calculate a second decision by using the updated initial value Determining the prediction result and the second residual determined by the tree, and assigning the updated residual value to the second residual, thereby traversing all the decision trees to obtain whether the predicted disk is a faulty disk. result.

It should be noted here that the receiving module 100, the second computing module The example and application scenario implemented by the group 102 and the traversal module 104 corresponding to the steps S251 to S255 of the embodiment are the same, but are not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as part of the device in the computer terminal 10 provided in the first embodiment.

Example 3

An embodiment of the present invention may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.

In this embodiment, the computer terminal can execute the code of the following steps in the method for predicting the fault of the disk: acquiring the sample disk data of the disk by using a disk monitoring technology, wherein the sample disk data includes multiple dimensions. Sample data; using the GBDT algorithm to sample the sample disk data to obtain a disk prediction model composed of multiple decision trees; after receiving the disk data of the disk to be tested, using a plurality of decision trees The disk prediction model processes the disk data of the disk to be determined to determine whether the disk to be tested is a failed disk.

Optionally, FIG. 11 is a structural block diagram of a computer terminal according to an embodiment of the present invention. As shown in FIG. 11, the computer terminal A may include one or more (only one shown in the figure) processor 111, memory 113, and transmission device 115.

The memory can be used to store software programs and modules, such as the method for predicting the failure of the disk in the embodiment of the present invention and the program instructions/modules corresponding to the device. The processor runs the software program and the module stored in the memory. , thereby performing various functional applications and data processing, that is, implementing the above-described method for predicting the failure of the disk. The memory may include high speed random memory and may also include non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, the memory can further include memory disposed remotely from the processor, the remote registers being connectable to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor can call the information and application stored in the memory through the transmission device to perform the following steps: the sample disk data is SMART disk data, wherein the sample disk data includes at least the sample data in the following four dimensions: original Value, standard value, worst value, and cumulative value.

Optionally, the processor may further execute the following code: performing any one or more of the following operations on the sample data in each dimension: a differential operation, a square operation, and a distributed summation operation, so that samples in any one dimension The data is expanded to sample data in a new dimension.

Optionally, the processor may further execute the following steps: using the sample disk data of all the disks as training data, and initializing the classification model parameters of the training data by using preset values; and extracting multiple features in the training data. Data, each feature data is used as a root node to create multiple decision trees, and the feature values corresponding to each feature data are used as corresponding decision trees. The leaf node calculates the optimal partition of all current leaf nodes and its gain, and splits with the leaf node with the largest gain and the corresponding dividing point, so that the sample disk data is divided into the child nodes.

Optionally, the processor may further execute the following steps: reading a threshold corresponding to any one of the feature data; comparing the feature value of any one of the feature data with a threshold, and obtaining an entropy of the two branches according to the comparison result; Two new nodes are determined as two leaf nodes of any one of the feature data according to the entropy of the two branches; each feature data is processed by the above steps until each feature data obtains two predetermined unique leaf nodes.

Optionally, the processor may further execute the following code: after obtaining the disk prediction model composed of multiple decision trees, the method further includes: adjusting the classification model parameters, wherein the classification model parameters include faults In the case of a disk sample and a non-faulty disk sample, if it is determined whether the disk to be tested is a failed disk, the proportion of the failed disk sample in the classification model parameter is increased.

Optionally, the processor may further execute the following code: after receiving the disk data of the disk to be tested, the disk data of the disk to be tested is given an initial value; traversing according to the initial value of the disk to be tested. For each decision tree, the prediction result determined by the first decision tree and the first residual are calculated, and the first residual is assigned to the initial value to obtain the updated initial value; and the updated initial value is calculated. The prediction result determined by the two decision trees and the second residual, and the second residual is assigned an updated initial value, thereby traversing all the decision trees to obtain a knot predicting whether the disk to be tested is a failed disk fruit.

In the embodiment of the present invention, the sample disk data of the disk is obtained by using a disk monitoring technology, wherein the sample disk data includes sample data in multiple dimensions; and the sample disk disk data is sampled by using the GBDT algorithm. Obtaining a disk prediction model method composed of a plurality of decision trees, after processing the disk data of the disk to be tested, using a disk prediction model composed of a plurality of decision trees to process the disk data of the disk to be tested The purpose of determining whether the disk to be tested is a faulty disk is achieved, thereby realizing the technical effect of predicting the disk fault state, thereby solving some factors in the prior art hard disk fault prediction system that can easily cause the hard disk failure to be Collect or quantify technical problems that result in inaccurate predictions.

A person skilled in the art can understand that the structure shown in FIG. 11 is only an illustration, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (MID). ), PAD and other terminal devices. FIG. 11 does not limit the structure of the above electronic device. For example, computer terminal A may also include more or fewer components (such as a network interface, display device, etc.) than shown in FIG. 11, or have a different configuration than that shown in FIG.

A person skilled in the art can understand that all or part of the steps of the foregoing embodiments can be completed by using a program to instruct a terminal device related hardware, and the program can be stored in a computer readable storage medium, and the storage medium can be Including: flash memory disk, read-only memory (ROM), random access memory (Random) Access Memory, RAM), disk or CD.

Example 4

Embodiments of the present invention also provide a storage medium. Optionally, in the embodiment, the storage medium may be used to save the code executed by the fault prediction method of the disk provided in the first embodiment.

Optionally, in this embodiment, the storage medium may be located in any one of the computer terminal groups in the computer network, or in any one of the mobile terminal groups.

Optionally, in this embodiment, the storage medium is configured to store a code for performing the following steps: acquiring the magnetic disk data of the magnetic disk by using a disk monitoring technology, wherein the sample disk data includes multiple dimensions. Sample data; sample training of sample disk data using GBDT algorithm to obtain a disk prediction model composed of multiple decision trees; after receiving the disk data of the disk to be tested, using a plurality of decision trees The disk prediction model processes the disk data of the disk to be determined to determine whether the disk to be tested is a failed disk.

Optionally, the storage medium is further configured to store a code for performing the following steps: performing one or more of the following operations on the sample data in each dimension: a difference operation, a square operation, and a distribution sum operation, so that Sample data in one dimension is extended to sample data in a new dimension.

Optionally, the storage medium is further configured to store a code for performing the following steps: using sample disk data of all disks as training resources Material, and initialize the classification model parameters of the training data by using preset values; extract multiple feature data in the training data, create each decision tree as a root node, and associate the feature values corresponding to each feature data As the leaf node of the corresponding decision tree; calculate the optimal partition of all current leaf nodes and its gain, and split with the leaf node with the largest gain and the corresponding partition point, so that the sample disk data is divided into the child nodes.

Optionally, the storage medium is further configured to store a code for performing the following steps: reading a threshold corresponding to any one of the feature data; comparing the feature value of any one of the feature data with a threshold, and obtaining two according to the comparison result Entropy of branches; two new nodes are determined as two leaf nodes of any one feature data according to the entropy of the two branches; each feature data is processed by the above steps until each feature data is obtained by two predetermined unique Leaf node.

Optionally, the storage medium is further configured to store a code for performing the following steps: after obtaining a disk prediction model composed of a plurality of decision trees, the method further comprises: adjusting the classification model parameters, wherein In the case where the classification model parameters include a failed disk sample and a non-faulty disk sample, if it is determined whether the disk to be tested is a failed disk, the proportion of the failed disk sample in the classification model parameter is increased.

Optionally, the storage medium is further configured to store a code for performing the following steps: after receiving the disk data of the disk to be tested, the disk data of the disk to be tested is given an initial value; The initial value of the disc traverses each decision tree, calculates the prediction result and the first residual determined by the first decision tree, and assigns the first residual to the initial value to obtain an update. After the initial value; the predicted result and the second residual determined by the second decision tree are calculated by the updated initial value, and the updated initial value is assigned to the second residual, thereby traversing all the decision trees and obtaining Predict whether the disk to be tested is the result of a failed disk.

The serial numbers of the above embodiments of the present invention are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the descriptions of the various embodiments are different, and the parts that are not detailed in a certain embodiment can be referred to the related descriptions of other embodiments.

In the several embodiments provided by the present application, it should be understood that the disclosed technical contents may be implemented in other manners. The device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or integrated. Go to another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, unit or module, and may be electrical or otherwise.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in various embodiments of the present invention can be integrated In one processing unit, each unit may be physically present alone, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of a hardware or a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, can be stored in a computer readable storage medium. Based on such an understanding, the technical solution of the present invention may contribute to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium. A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage media include: flash flash drive, read-only memory (ROM), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disc, etc. The code of the media.

The above description is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can also make several improvements and retouchings without departing from the principles of the present invention. It should be considered as the scope of protection of the present invention.

Claims (14)

A method for predicting a fault of a magnetic disk, comprising: acquiring a sample disk data of a magnetic disk by using a disk monitoring technology, wherein the sample disk data includes sample data in a plurality of dimensions; using a GBDT algorithm The sample disk data is sample-trained to obtain a disk prediction model composed of a plurality of decision trees; after receiving the disk data of the disk to be tested, the disk prediction model composed of a plurality of decision trees is used for the disk disk prediction model. The disk data of the disk is processed to determine whether the disk to be tested is a failed disk. The method of claim 1, wherein the sample disk data is SMART disk data, wherein the sample disk data includes at least sample data in the following four dimensions: original value, standard value, and most Difference and cumulative value. The method of claim 2, wherein after obtaining the sample disk data of the disk by the disk monitoring technology, the method further comprises: performing one or more of the following sample data in each dimension; Operations: differential operations, square operations, and distributed summation operations, so that sample data in any dimension is extended to sample data in a new dimension. The method according to any one of claims 1 to 3, wherein the sample disk data is sample-trained by using a GBDT algorithm to obtain a disk prediction model composed of a plurality of decision trees, including: The sample disk data of all the disks is used as training data, and the classification model parameters of the training materials are initialized by using preset values; Extracting a plurality of feature data in the training data, creating each of the plurality of decision trees as a root node, and using the feature value corresponding to each feature data as a leaf node of the corresponding decision tree; calculating all current leaves The optimal partitioning of the node and its gain are split by the leaf node with the largest gain and the corresponding dividing point, so that the sample disk data is divided into the child nodes. The method of claim 4, wherein extracting a plurality of feature data in the training material, using each feature data as a root node to create the plurality of decision trees, and corresponding features of each feature data The value is a leaf node of the corresponding decision tree, including: reading a threshold corresponding to any one of the feature data; comparing the feature value of the any one of the feature data with the threshold, and obtaining the entropy of the two branches according to the comparison result; The entropy of the two branches determines two new nodes as the two leaf nodes of the arbitrary feature data; each of the feature data is processed by the above steps until each feature data obtains two predetermined unique leaf nodes. . The method of claim 4, wherein after obtaining a disk prediction model composed of a plurality of decision trees, the method further comprises: adjusting the classification model parameters, wherein the classification model parameters In the case of a failed disk sample and a non-faulty disk sample, if it is determined whether the disk to be tested is a failed disk, the proportion of the failed disk sample in the classification model parameter is increased. According to the method of claim 1, wherein And processing the disk data of the disk to be tested by using the disk prediction model consisting of multiple decision trees to determine whether the disk to be tested is a fault disk, including: receiving the magnetic disk of the disk to be tested After the disc data, an initial value is given to the disc data of the disc to be tested; and each decision tree is traversed according to the initial value of the disc to be tested, and the prediction result determined by the first decision tree and the first stub are calculated. Poor, and assigning the first residual to the initial value to obtain an updated initial value; calculating the predicted result and the second residual determined by the second decision tree by using the updated initial value, and The second residual is assigned the updated initial value, thereby traversing all the decision trees to obtain a result of predicting whether the disk to be tested is a failed disk. A device for predicting a failure of a disk, comprising: an acquisition module, configured to acquire a sample disk data of a disk by using a disk monitoring technology, wherein the sample disk data includes sample data in multiple dimensions; The training module is configured to perform sample training on the sample disk data by using the GBDT algorithm to obtain a disk prediction model composed of a plurality of decision trees; after receiving the disk data of the disk to be tested, the processing module The disk data of the disk to be tested is processed by using the disk prediction model composed of a plurality of decision trees to determine whether the disk to be tested is a failed disk. The device of claim 8, wherein the sample disk data is SMART disk data, wherein the sample disk data includes at least sample data in the following four dimensions: original value, standard Value, worst value, and cumulative value. The device according to claim 9, wherein the device further comprises: an operation module, configured to perform any one or more of the following operations on the sample data in each dimension: a difference operation, a square operation, and a distribution summation. The operation is such that the sample data in any one dimension is expanded to the sample data in the new dimension. The device of any one of claims 8 to 10, wherein the training module further comprises: an initial module for using the sample disk data of all the disks as training materials, and adopting a preset The value initializes the classification model parameter of the training data; the extraction module is configured to extract a plurality of feature data in the training data, and each feature data is used as a root node to create the plurality of decision trees, and each feature data is corresponding The eigenvalue is used as the leaf node of the corresponding decision tree; the first computing module is configured to calculate the optimal partition of all current leaf nodes and the gain thereof, and split with the leaf node with the largest gain and the corresponding dividing point, so that The sample disk data is divided into child nodes. The device of claim 11, wherein the extraction module comprises: a reading module for reading a threshold corresponding to any one of the feature data; and a comparison module for using the any one of the feature data The eigenvalue is compared with the threshold, and the entropy of the two branches is obtained according to the comparison result; the determining module is configured to determine two new nodes as the two leaf nodes of the arbitrary feature data according to the entropy of the two branches ; The processing sub-module is configured to process each feature data by using the above steps until each feature data obtains two predetermined unique leaf nodes. The apparatus of claim 11, wherein after obtaining a disk prediction model composed of a plurality of decision trees, the apparatus further comprises: adjusting the classification model parameters, wherein the classification model parameters include In the case of a failed disk sample and a non-faulty disk sample, if it is determined whether the disk to be tested is a failed disk, the proportion of the failed disk sample in the classification model parameter is increased. The device of claim 8, wherein the processing module comprises: a receiving module, configured to receive the disk data of the disk to be tested after receiving the disk data of the disk to be tested An initial value is given; a second computing module is configured to traverse each decision tree according to the initial value of the disk to be tested, calculate a prediction result and a first residual determined by the first decision tree, and A residual is assigned to the initial value to obtain an updated initial value; a traversal module is configured to calculate a prediction result and a second residual determined by the second decision tree by using the updated initial value, and The second residual is assigned the updated initial value, thereby traversing all the decision trees to obtain a result of predicting whether the disk to be tested is a failed disk.