CN115617604A

CN115617604A - Disk failure prediction method and system based on image pattern matching

Info

Publication number: CN115617604A
Application number: CN202211102546.9A
Authority: CN
Inventors: 欧阳京; 刘攀; 周泽湘; 文中领; 尹微
Original assignee: Beijing Toyou Feiji Electronics Co ltd
Current assignee: Beijing Toyou Feiji Electronics Co ltd
Priority date: 2022-09-09
Filing date: 2022-09-09
Publication date: 2023-01-17

Abstract

The invention provides a disk failure prediction method and system based on image pattern matching. The method comprises the following steps: collecting a characteristic image of a predicted disk; matching the collected characteristic image with a failure prediction matching legend so as to predict the probability of future failure of the disk; and carrying out tracking verification on the probability obtained through prediction. The disk failure prediction method and system based on image pattern matching can accurately predict the occurrence of a failure disk with low calculation amount cost.

Description

Disk failure prediction method and system based on image pattern matching

Technical Field

The invention relates to the technical field of disk fault monitoring, in particular to a disk fault prediction method and system based on image pattern matching.

Background

Today's enterprise server performance is increasing more and more powerful, and data is expanding more and more, and the disk quantity on the storage system, capacity all exponentially increase. In such a high-load working background, the probability of the occurrence of the disk failure is more frequent, and the frequency of the maintenance of the disk in the whole operation and maintenance process is increased. In order to improve the high quality of service, the failure prediction of the disk becomes more important, the sub-health disk is replaced in advance, the risk of data loss or service downtime is reduced, and the beneficial support is provided for the benefit and the operation and maintenance cost of enterprises.

The mainstream disk failure prediction is a prediction method based on AI machine learning, which utilizes known classified disk historical data to perform model training, obtains a trained model and then utilizes unknown disk data to perform prediction. Machine-learned models are generally classified into "classification" and "regression," as well as "offline" and "online" approaches. A relatively straightforward approach is to model "offline" + "classification", meaning that historical disk data that is "known to be class failed" is used for training, e.g., mass data downloaded from the backslaze website, "classification" refers to an algorithm whose prediction conclusion is a discrete-valued class, e.g., K-neighborhood, decision tree, support vector machine, naive bayes, etc., and whose prediction conclusion is "whether the probability of the disk failure exceeds 90% after 30 days". "regression" is a linear model that fits training data to a continuous "curve", e.g., Y = a X + b, and solves the best solution for a, b using known N (Xi, yi) data (least squares yields the minimum loss function). Where X is an abstraction of all features, simplifying M features to 1-dimension, and Y is the predicted conclusion. If a "regression" model is used for disk prediction, Y is the "probability value of disk failure after 30 days". For "regression" models, the data of features require more preprocessing, because most regression models rely on "numeric" type as input parameters for their formula, and normalization of data is greatly introduced. By "online" it is meant that the production environment collects data from a real working disk and the model is trained using the real data, which is obviously an "unsupervised training" because it is not known whether the disk will fail in the next 30 days when the data is collected, and there is no "known" conclusion for all the training data. Compared with the schemes of supervised training and unsupervised training of the prior offline model, more system overhead is required to be occupied, and formula operation and model training can be effectively carried out only after a certain amount of data is required to be collected by the system before starting prediction. More complex schemes such as deep learning and neural networks have recently been introduced into failure prediction, not only for disks, but also for other components such as motherboards, CPUs, network cards, and the like. The failure prediction is the most critical loop of an AI intelligent operation and maintenance system and is one of conditions for triggering AI self-repair.

Machine learning is an existing failure prediction scheme, and the first step of the scheme is to observe whether a failure disk presents a certain rule or accords with clear classification of boundaries aiming at certain characteristics (including physical attributes, operation indexes, real-time states and the like) through visual processing of disk SMART data. After assuming that it fits the category of a certain model, an appropriate training model is selected.

And secondly, importing a large amount of magnetic disk SMART historical data with balanced positive and negative samples, selecting TOP n characteristic values with strong correlation, preprocessing the historical data, removing or filling invalid characteristic columns, and normalizing the characteristic values regularly. And (4) labeling the training data (whether the training data belongs to a fault disk), wherein the positive and negative balance of the sample is paid attention to, and otherwise, the training data is predicted to be biased to the party with a large proportion.

And thirdly, an algorithm of a machine model is realized through coding, the block can use a module packaged by mainstream sklern, training data is divided into a training set and a testing set, after cross training is carried out for multiple times, an optimal prediction model (actually, a value of a weight variable in a model formula is solved) is obtained, the calculation amount required by the step is large, and exponential rise can be realized due to the increase of the amount of the training data.

And fourthly, transmitting the SMART characteristic parameters of the disks which are unknown whether to have faults on line into the model, and solving to obtain fault probability. If it is greater than the preset value (90%), the prediction hit is flagged.

And fifthly, sending a prediction report in the form of an alarm module of the system, or printing a log, or interface reminding, or mail or short message notification, and selecting whether to replace the disk in advance after the IT maintainer operates the computer again to confirm the disk state.

The data volume of the offline historical disk data is large, certain difficulty is caused in finding and extracting key features, and the expressive force among different industries, different environments and different disk products is insufficient in the representativeness of the disks of the offline historical disk data to the self products. Besides the SMART parameters, the electrical characteristics of the host machine for the operation of the magnetic disk, the temperature and the humidity of the machine room and the stability of the power supply also need to be considered. The higher the complexity of the model is, the higher the accuracy of the model cannot be represented completely, but the computation loss and the computation cost caused by the complexity of the model and the size of the data training amount are inevitably increased exponentially. And some characteristics can obviously represent whether the system is in a 'failure' state or not, but belong to the types of jitter-free before failure and abrupt change after failure, and the reference provided for prediction by the characteristics is insufficient. When the unknown disk is predicted, the characteristic performance of the unknown disk is the same as that of a normal disk.

In the scheme of the distributed storage system, a server for system management and maintenance is not specially configured, and for machine learning with more consumed resources, if the machine learning is frequently operated on a storage node, unnecessary loss is inevitably brought to a client, and the input-output ratio is more satisfactory.

Disclosure of Invention

The invention aims to solve the technical problem of providing a disk failure prediction method and system based on image pattern matching, which can accurately predict the occurrence of a failed disk with low calculation amount cost.

In order to solve the technical problem, the invention provides a disk failure prediction method based on image pattern matching, which comprises the following steps: collecting a characteristic image of a predicted disk; matching the collected characteristic image with a failure prediction matching legend so as to predict the probability of future failure of the disk; and carrying out tracking verification on the probability obtained through prediction.

In some embodiments, further comprising: before collecting the characteristic image of the predicted disk, selecting historical final fault data of the disk with the same product model as the predicted disk; selecting valid features from historical final fault data; and distinguishing the service scene and the production environment for the selected effective characteristics to form a failure prediction matching legend.

In some embodiments, the feature image comprises: and (4) a characteristic image of the number of failed address selection times of the magnetic head.

In some embodiments, acquiring a feature image of a predicted disk comprises: and acquiring characteristic images of the predicted disk within n days.

In some embodiments, the characteristic image is a timing diagram.

In some embodiments, matching the collected feature images with a failure prediction matching legend to predict the probability of future failure of the disk includes: and carrying out pattern matching on the collected characteristic image and the failure prediction matching legend so as to predict the probability of failure of the disk in the next n days.

In some embodiments, selecting the valid signature from the historical final fault data includes: as long as the time-series variation characteristics different from those of the disks working in a healthy state are represented, the effective characteristics are selected.

In some embodiments, differentiating the service scenario and the production environment for the selected valid features to form a failure prediction matching illustration includes: distinguishing different service scenes, and predicting whether the disk has rules in the aspects of reading, writing and IOPS; and performing subjective correction on the fault trend graph according to the estimation result, and improving the matching degree.

In some embodiments, further comprising: after the failure prediction matching legend is formed, two-dimensional images such as time-series trend graphs of a plurality of characteristics are combined into one pattern graph by adopting different attributes of image gray scale and color.

In addition, the invention also provides a disk failure prediction system based on image pattern matching, which comprises: one or more processors; storage means for storing one or more programs; when executed by the one or more processors, cause the one or more processors to implement a disk failure prediction method based on image pattern matching as described above.

After adopting such design, the invention has at least the following advantages:

and an image matching algorithm is introduced, the calculated amount of a training model is reduced, the magnetic disk characteristic attribute with the most trend change is mined, the occurrence of a fault magnetic disk is predicted by a method for capturing the characteristic trend on line in real time, and the product competitiveness is improved.

Drawings

The foregoing is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and the detailed description.

FIG. 1 is a schematic flow diagram of one embodiment of the present invention;

fig. 2 is a schematic flow diagram of another embodiment of the present invention.

Detailed Description

The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.

Aiming at the defects existing in the traditional machine learning, the matching method based on the image characteristics uses the simplest and easily understood mode to estimate the possibility of the disk failure. The method does not need derivation of a large number of complex formulas, does not need adjustment of hyper-parameters to reduce loss functions, does not need evaluation of prediction accuracy and generalization capability among various models, and only needs to select the most representative magnetic disk characteristics and comprehensively evaluate the optimal fault trend image characteristics by combining production environment, application field and the like.

Selecting historical data of the disks on the Backblaze only needs to select the data of the disks with the same model or the same manufacturer, drawing a change trend graph of all characteristics by using time series of the data in the historical N days of the disks with the faults. And simply evaluating three or four characteristics according with the 'data before and after the fault is in a gradual change trend'. The data characteristics of steady or regular fluctuation before the fault and sudden change or rigor after the fault are directly ignored. The feature atlas generated in the way can be used in the feature image pattern matching of an unknown disk, and when the matching reaches a certain degree, the conclusion of N days of future failures of the disk is obtained. The absolute value of the data is not critical in image pattern matching, the key is the trend of data jitter and change, and the human health is predicted by examining the changes of electrocardiogram, blood pressure and blood sugar similar to the medical doctor of a human body.

And the service scenes (road monitoring, live broadcast platform, cloud computing service and the like) of the server to which the disk belongs, and fixed indexes of the voltage and the temperature and the humidity of the production environment can be used as class 'super parameters' for weighting or adjusting the characteristic weight.

Two-dimensional images such as time-series trend graphs of a plurality of features can also be combined into one pattern graph by adopting different attributes such as image gray scale, color and the like. A multi-dimensional deep fault disk index trend legend predicts the current index of an unknown disk by pattern matching in a production environment, and the calculation cost is much lower than that of a common machine learning model.

Referring to fig. 1, in an embodiment of the present invention, the technical solution specifically includes the following steps:

and S11, selecting historical final fault data of the disk with the same product model as the predicted disk.

And S12, selecting effective characteristics from the historical final fault data.

And S13, distinguishing the service scene and the production environment for the selected effective characteristics to form a fault prediction matching legend.

And S14, collecting the characteristic image of the predicted disk.

And S15, matching the collected characteristic image with a failure prediction matching legend, so as to predict the probability of future failure of the disk.

And S16, tracking and verifying the probability obtained through prediction.

An image matching algorithm is introduced, the calculated amount of a training model is reduced, the magnetic disk characteristic attribute with the most trend change is mined, the occurrence of a fault magnetic disk is predicted by a method for capturing the characteristic trend on line in real time, the settings of the magnetic disk and other peers are differentiated, and the product competitiveness is improved.

Referring to fig. 2, in another embodiment of the present invention, the technical solution specifically includes the following steps:

s21, selecting the history data of the magnetic disk. And collecting the disk data of the same product model as far as possible to serve as an input condition of characteristic analysis, and if the disk data of the same model does not exist, selecting disks of other models of the same manufacturer. The characteristics exhibited by disks made by different manufacturers may not be the same. And as much as possible, the historical data of the magnetic disk of the final fault is collected, more fault cases can enrich the image library resources for matching, and the success rate of prediction is improved.

And S22, selecting effective characteristics. And importing and drawing a result of each data characteristic changing on a time axis by using a software tool. For example, "the number of times of head addressing failures" is counted from a sample n days before the failure occurs (assuming n = 90) to the time of the failure occurrence. If the trend graphs of the disks of the final failure and the final healthy are compared, the conclusion of obvious difference can be obtained, the number of times of failed head addressing can be used as an effective characteristic, and the trend graph drawn by historical data can be filed as a failure prediction matching legend. Since there are more than 100 features contained in the disk data, the time consumed for the time chart drawing at this stage is large, but the time chart drawing does not involve complicated mathematical calculation, but the time chart drawing and the calculation of basic statistics (variance, mean, median, etc.) are only performed. The timing diagram of the active feature is expected to be a gradual rise, or fall, and may not be linear (allowing for jitter), but may not be smooth to abrupt changes. Disk signatures that are mutated after a failure occurs do not provide a reference value in the prediction. Although the disk may be suddenly broken due to the sudden change of the external environment (burnt by sudden power failure, damage caused by misoperation of people and the like), the disk is not a range accident of the fault prediction, and like the situation that a doctor cannot predict that a patient encounters a traffic accident, the disk is predicted to be abnormal only due to the change of indexes related to the service life of the disk. As long as the time sequence change characteristics with different trends from the disk working healthily are shown, the time sequence change characteristics need to be archived as a fault prediction matching legend.

And S23, selecting a service scene and a production environment. For a disk, different service scenarios may affect the read-write frequency, the size of a file object, the upper limit of capacity, and the like. For example, the terminal writes frequent road surface monitoring, and for the cloud disk, the update frequency of the small file is higher. For example, while the monitoring system writes in a linear input pattern (a constant regular record of uniform file size), the mail system does not display an irregular input pattern (presence or absence of attachments varies greatly with mail size). The distinction of different service scenarios is mainly to predict whether the disk has rules in the read, write, and IOPS aspects. And carrying out subjective correction on the fault trend graph and improving the matching degree. The environment of the disk operation depends on the comprehensive consideration of the storage server chassis, the cabinet and even the machine room. The heat dissipation of the chassis, the stability of the power supply, the firmness of the cabinet, the humidity control of the machine room, etc. all need to be taken as correction parameters. If an index shows a "negative" effect, it can be speculated that the lifetime of its disk is more likely to "decrease" and most likely to be concluded "failure trend acceleration". For example, the main control power supply voltage of the machine room fluctuates greatly and cannot be recovered in a short period. As the 'correction hyper-parameter' of the image pattern matching, an empirical value which needs to be corrected continuously along with the operation of the system is used, so that gradient descent continuous exploration similar to a least square method is not needed, the final loss function calculation amount is greatly reduced, and the calculation resource needed by a fault prediction system is reduced.

And S24, collecting the characteristics of the unknown disk. With the output of the second and third steps, the prediction can be performed in the storage system, and the feature collection of the unknown disk needs a certain time to complete. It cannot be predicted within n days (n may equal 90) of the predicted system start-up. n can be adjusted in the management page, and the user does not need to be less than 30 as much as possible for accuracy. In addition, it should be noted that the characteristic sampling time, preferably the time period when the storage traffic is busy, may be sampled once every 30 minutes, and then the data is finally merged to form a sample record of 1 day when the traffic is busy according to the data analysis. After the sample records of n days are leveled, a time sequence chart can be drawn. Since the features to be analyzed may be m, m is generally larger than 3 and smaller than 10 (too much of which does not represent the performance advantage of the present invention due to machine learning), a certain amount of memory resources are required for the system to buffer the data amount of m × n.

And S25, matching the characteristic image patterns. And drawing a timing chart of the unknown disk sample extracted in the fourth step, and performing pattern matching in a 'failure prediction matching legend' database. Pattern matching is to compare the recent performance of a certain feature of a disk to the performance of a "failed disk". The matching degree determines whether the probability of failure is judged after n days in the future. After the "super-parameter" adjustment of the third step, the original probability is concluded by the whole prediction system. For the original feature timing graph, only image boundary matching is needed. If a "high-dimensional feature legend" is used for multi-feature merging, the image schema involves more elements, such as layers, depths, etc. There is currently no more detailed solution to the aspect of feature data conversion to image attributes involved here. The purpose of merging feature legends is to reduce the number of image pattern matches and to take full advantage of the unique properties of image patterns instead of numerical computation methods.

And S26, tracking the prediction result. And fifthly, after a certain disk is predicted, if the disk is predicted to be in failure, tracking the disk for n days to judge whether the disk is actually in failure. And if the conclusion is correct, increasing the weight of the relevant characteristics of the prediction matching library so as to better predict the same-class faults in the future. When a disk failure occurs in a certain chassis, the probability weight of the failure of other disks in the host is also increased. Because of concerns that poor system software, SAS cards, motherboard performance, or power may be the origin of a failure, disks on the same host should be more prone to failure than other good hosts. The online prediction result can continuously correct and optimize the 'super parameter' of the prediction system, and the prediction accuracy can also be continuously improved along with time.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the present invention in any way, and it will be apparent to those skilled in the art that the above description of the present invention can be applied to various modifications, equivalent variations or modifications without departing from the spirit and scope of the present invention.

Claims

1. A disk failure prediction method based on image pattern matching is characterized by comprising the following steps:

collecting a characteristic image of a predicted disk;

matching the collected characteristic image with a failure prediction matching legend so as to predict the probability of future failure of the disk;

and carrying out tracking verification on the probability obtained through prediction.

2. The disk failure prediction method based on image pattern matching according to claim 1, further comprising:

before collecting the characteristic image of the predicted disk, selecting historical final fault data of the disk with the same product model as the predicted disk;

selecting valid features from historical final fault data;

and distinguishing the service scene and the production environment for the selected effective characteristics to form a failure prediction matching legend.

3. The disk failure prediction method based on image pattern matching according to claim 1, wherein the feature image comprises: and (4) a characteristic image of the number of failed address selection times of the magnetic head.

4. The disk failure prediction method based on image pattern matching according to claim 1, wherein the collecting of the feature image of the predicted disk comprises:

and acquiring characteristic images of the predicted disk within n days.

5. The disk failure prediction method based on image pattern matching as claimed in claim 4, wherein the characteristic image is a timing chart.

6. The disk failure prediction method based on image pattern matching according to claim 4, wherein the step of matching the collected feature image with the failure prediction matching legend so as to predict the probability of future failure of the disk comprises the following steps:

and carrying out pattern matching on the collected characteristic image and the failure prediction matching legend so as to predict the probability of failure of the disk in the next n days.

7. The disk failure prediction method based on image pattern matching according to claim 2, wherein selecting valid features from historical final failure data comprises:

as long as the time-series variation characteristics different from those of the disks working in a healthy state are represented, the effective characteristics are selected.

8. The disk failure prediction method based on image pattern matching according to claim 2, wherein the distinguishing of the service scene and the production environment for the selected valid features to form a failure prediction matching illustration comprises:

distinguishing different service scenes, and predicting whether the disk has rules in the aspects of reading, writing and IOPS;

and performing subjective correction on the fault trend graph according to the estimation result, and improving the matching degree.

9. The disk failure prediction method based on image pattern matching according to claim 2, further comprising:

after a failure prediction matching legend is formed, two-dimensional images such as time sequence trend graphs of a plurality of characteristics are combined into one pattern graph by adopting different attributes of image gray scale and color.

10. A disk failure prediction system based on image pattern matching, comprising:

one or more processors;

storage means for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the method for disk failure prediction based on image pattern matching according to any one of claims 1 to 9.