CN110413703B - Method for classifying monitoring index data based on artificial intelligence and related equipment - Google Patents
Method for classifying monitoring index data based on artificial intelligence and related equipment Download PDFInfo
- Publication number
- CN110413703B CN110413703B CN201910540031.9A CN201910540031A CN110413703B CN 110413703 B CN110413703 B CN 110413703B CN 201910540031 A CN201910540031 A CN 201910540031A CN 110413703 B CN110413703 B CN 110413703B
- Authority
- CN
- China
- Prior art keywords
- classification
- index data
- monitoring index
- classification result
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention relates to the field of artificial intelligence and discloses a classification method and related equipment for monitoring index data based on artificial intelligence. The classification method of the monitoring index data based on the artificial intelligence comprises the following steps: acquiring monitoring index data; preprocessing and clustering calculation are carried out on the monitoring index data to obtain a first classification result; correcting the deviation and carrying out classification numbering on the first classification result to obtain a second classification result; and classifying the second classification result through the trained neural network model to obtain a final classification result. The invention well solves the problem that the practical applicability and the universality of the existing monitoring index data are limited due to the dependence on a large number of training models, the classification of the monitoring index data is beneficial to reducing the development work of the training models and shortening the research and development period, and the monitoring index data with the same classification can share a data set, thereby being beneficial to establishing the monitoring resource portraits.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a classification method and related equipment of monitoring index data based on artificial intelligence.
Background
With the development of network technology, various big data applications have become particularly widespread. Some basic service monitoring is to monitor the CPU operation, the memory operation and the like of the host computer in real time through performance and operation environment, and provide utilization rate data, so that the method is relatively backward, and in the age of conforming to trend and following technological progress, how to better complete intelligent service monitoring by utilizing big data and cloud computing environment should be learned.
At present, most of hosts for providing services for cloud computing service platforms, including physical machines and virtual machines, basically reach tens of thousands of levels, and each host contains hundreds of indexes to be monitored, so that the monitored index instances are measured in tens of millions. For example, a cloud service manufacturer has 10000 hosts, and each host needs to be monitored indexes, such as memory usage, CPU usage, disk usage, etc., for a total of 100, the monitored instance needs to be monitored reaches 1000000 (100×1000). Currently, most factories introduce artificial intelligence (artificial intelligence, AI) technology for monitoring business, which is called intelligent operation and maintenance (artificial intelligence for IT operations, AIOps), wherein AI is applied to the operation and maintenance field, and the problem of automatic operation and maintenance is further solved by a machine learning mode based on the existing operation and maintenance data such as logs, monitoring information, application information and the like, and AIOps do not depend on manually specified rules.
If the anomaly detection processing is performed on each monitoring index instance through a machine learning training model, the model to be trained is millions, the development period is too long, and the cost is too high, which is unacceptable to enterprises.
Disclosure of Invention
The invention mainly aims to solve the technical problems that each piece of monitoring index data is subjected to exception handling through a machine learning training model, the required training model is up to millions, the development period is long, and the cost is too high.
To achieve the above object, a first aspect of the present invention provides a method for classifying monitoring index data based on artificial intelligence, including: acquiring monitoring index data; preprocessing and clustering calculation are carried out on the monitoring index data to obtain a first classification result; correcting the deviation and carrying out classification numbering on the first classification result to obtain a second classification result; and classifying the second classification result through the trained neural network model to obtain a final classification result.
Optionally, in a first implementation manner of the first aspect of the present invention, the preprocessing and clustering the monitoring index data to obtain a first classification result includes: preprocessing the monitoring index data to obtain a distance numerical index; and carrying out clustering calculation according to the distance numerical index to obtain a first classification result.
Optionally, in a second implementation manner of the first aspect of the present invention, preprocessing the monitoring index data to obtain a distance numerical index includes: performing standardized processing on the monitoring index data to obtain an original data sequence; smoothing the original data sequence to obtain a smoothed sequence; performing difference operation according to the original data sequence and the smooth sequence to obtain a residual sequence; calculating according to the smooth sequence and the residual sequence to obtain the correlation of the monitoring index data, wherein the value range of the correlation is greater than or equal to-1 and less than or equal to 1; and calculating the distance numerical index according to the correlation of the monitoring index data, wherein the range of the distance numerical index is larger than or equal to 0 and smaller than or equal to 2.
Optionally, in a third implementation manner of the first aspect of the present invention, performing cluster calculation according to the distance numerical indicator, to obtain a first classification result includes: and carrying out clustering calculation through a density-based clustering DBSCAN algorithm with noise according to the distance numerical index to obtain a first classification result.
Optionally, in a fourth implementation manner of the first aspect of the present invention, performing deviation rectification and class numbering processing on the first classification result to obtain a second classification result includes: performing deviation rectifying treatment on the first classification result through a preset deviation rectifying algorithm to obtain a first classification deviation rectifying result; and carrying out classification numbering processing according to the first classification deviation correcting result to obtain a second classification result.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the classifying, by using the trained neural network model, the second classification result to obtain a final classification result includes: inputting the second classification result into a preset one-dimensional convolutional neural network model, wherein the preset one-dimensional convolutional neural network model is a trained neural network model; and classifying the second classification result through the preset one-dimensional convolutional neural network model to obtain a target classification label, and setting the target classification label as a final classification result, wherein the target classification label is used for indicating the classification type of the monitoring index data marked according to a preset label value.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after performing deviation rectification and classification numbering processing on the first classification result to obtain a second classification result, the classification method based on the monitoring index data of artificial intelligence further includes: training a neural network model according to a preset training sample to obtain a trained neural network model, and taking the value of a parameter when the training frequency of the neural network model reaches a preset threshold value as the value of the parameter of the trained neural network model; when detecting newly added different types of monitoring index data, setting the newly added different types of monitoring index data as a target training sample, and carrying out iterative optimization on the trained neural network model according to the target training sample.
The second aspect of the present invention provides a classification device for monitoring index data based on artificial intelligence, comprising: the acquisition unit is used for acquiring the monitoring index data; the first classification unit is used for preprocessing and clustering calculation of the monitoring index data to obtain a first classification result; the second classification unit is used for carrying out deviation correction and classification numbering treatment on the first classification result to obtain a second classification result; and the final classification unit is used for classifying the second classification result through the trained neural network model to obtain a final classification result.
Optionally, in a first implementation manner of the second aspect of the present invention, the first classification unit includes: the preprocessing subunit is used for preprocessing the monitoring index data to obtain a distance numerical index; and the clustering subunit is used for carrying out clustering calculation according to the distance numerical index to obtain a first classification result.
Optionally, in a second implementation manner of the second aspect of the present invention, the preprocessing subunit is specifically configured to: performing standardized processing on the monitoring index data to obtain an original data sequence; smoothing the original data sequence to obtain a smoothed sequence; performing difference operation according to the original data sequence and the smooth sequence to obtain a residual sequence; calculating according to the smooth sequence and the residual sequence to obtain the correlation of the monitoring index data, wherein the value range of the correlation is greater than or equal to-1 and less than or equal to 1; and calculating the distance numerical index according to the correlation of the monitoring index data, wherein the range of the distance numerical index is larger than or equal to 0 and smaller than or equal to 2.
Optionally, in a third implementation manner of the second aspect of the present invention, the clustering subunit is specifically configured to: and carrying out clustering calculation through a density-based clustering DBSCAN algorithm with noise according to the distance numerical index to obtain a first classification result.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the second classification unit is specifically configured to: performing deviation rectifying treatment on the first classification result through a preset deviation rectifying algorithm to obtain a first classification deviation rectifying result; and carrying out classification numbering processing according to the first classification deviation correcting result to obtain a second classification result.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the final classification unit is specifically configured to: inputting the second classification result into a preset one-dimensional convolutional neural network model, wherein the preset one-dimensional convolutional neural network model is a trained neural network model; and classifying the second classification result through the preset one-dimensional convolutional neural network model to obtain a target classification label, and setting the target classification label as a final classification result, wherein the target classification label is used for indicating the classification type of the monitoring index data marked according to a preset label value.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the classification device based on the monitoring index data of artificial intelligence further includes: the training unit is used for training the neural network model according to a preset training sample to obtain a trained neural network model, and taking the value of the parameter when the training times of the neural network model reach a preset threshold value as the value of the parameter of the trained neural network model; and the iteration unit is used for setting the newly added monitoring index data of different types as a target training sample when detecting the newly added monitoring index data of different types, and carrying out iterative optimization on the trained neural network model according to the target training sample.
A third aspect of the present invention provides an artificial intelligence based classification apparatus for monitoring index data, comprising: the system comprises a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through a line; the at least one processor invokes the instructions in the memory to cause the classification device based on the artificial intelligence monitoring metric data to perform the method of the first aspect described above.
A fourth aspect of the invention provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the method of the first aspect described above.
From the above technical scheme, the invention has the following advantages:
in the technical scheme provided by the invention, monitoring index data are acquired; preprocessing and clustering calculation are carried out on the monitoring index data to obtain a first classification result; correcting the deviation and carrying out classification numbering on the first classification result to obtain a second classification result; and classifying the second classification result through the trained neural network model to obtain a final classification result. In the embodiment of the invention, after preprocessing each monitoring index, clustering calculation, deviation correction processing, classification numbering processing and classification processing are performed on the trained neural network model, so that a final classification result is obtained. The invention well solves the problem that the practical applicability and the universality of the existing monitoring index data are limited due to the dependence on a large number of training models, the classification of the monitoring index data is beneficial to reducing the development work of the training models and shortening the research and development period, and the monitoring index data with the same classification can share a data set, thereby being beneficial to establishing the monitoring resource portraits.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a classification method of monitoring index data based on artificial intelligence in an embodiment of the invention;
FIG. 2 is a schematic diagram of another embodiment of a classification method of monitoring index data based on artificial intelligence in an embodiment of the invention;
FIG. 3 is a schematic diagram of an embodiment of an artificial intelligence based classification device for monitoring index data according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another embodiment of an artificial intelligence based classification device for monitoring index data according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an embodiment of an artificial intelligence based classification device for monitoring metric data in an embodiment of the invention.
Detailed Description
The embodiment of the invention provides a classification method and related equipment for monitoring index data based on artificial intelligence, which are used for preprocessing each monitoring index, clustering calculation, deviation rectifying treatment, classification numbering treatment and classification treatment of a trained neural network model to obtain a final classification result. The problem that the practical applicability and the universality of the existing monitoring index data are limited due to dependence on a large number of training models is well solved, classification of the monitoring index data is beneficial to reducing development work of the training models and shortening development period, and the monitoring index data with the same classification can share a data set, so that the monitoring resource portraits can be built.
In order to enable those skilled in the art to better understand the present invention, embodiments of the present invention will be described below with reference to the accompanying drawings.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, a specific flow of an embodiment of the present invention is described below with reference to fig. 1, and an embodiment of a method for classifying monitoring index data based on artificial intelligence in an embodiment of the present invention includes:
101. Acquiring monitoring index data;
the server acquires the monitoring index data. Specifically, the server may read the monitoring index data from the preset database, and the server may also collect the monitoring index data in real time, which is not limited herein.
It should be noted that, the monitoring index data is time-varying, and has a continuous output, and also has a monitoring index data output under a specific condition, where a value related to the system, such as the number of users currently logged into the Web application, needs to be collected at a specific time. Thus, the metrics are typically collected at regular intervals, such as once every second and once every minute. The server obtains monitoring index data with sufficient granularity to show significant peaks and dips. The specific granularity is related to the monitored system, the cost of acquisition, and the duration between index changes. Different monitoring index data may have different acquisition granularity, the memory or the CPU may use second as granularity statistics, and the energy consumption may use minute as granularity statistics to obtain monitoring index data of multiple ranges.
102. Preprocessing and clustering calculation are carried out on the monitoring index data to obtain a first classification result;
And the server performs preprocessing and clustering calculation on the monitoring index data to obtain a first classification result. Specifically, the server preprocesses the monitoring index data to obtain a distance numerical index; and the server performs clustering calculation through a clustering algorithm according to the distance numerical index to obtain a first classification result. The clustering algorithm is to measure the distance between two pieces of monitoring index data, that is, based on the distance numerical index, and the clustering algorithm is that the two pieces of monitoring index data with smaller distance numerical index are more likely to be aggregated into the same class.
It should be noted that, the range of the distance numerical indicator is greater than or equal to 0 and less than or equal to 2, for example, if the distance numerical indicators of the two pieces of monitoring indicator data a and B are both 0.1, a and B are clustered into the same class, and the first classification results of a and B are marked as 0.
103. Correcting the deviation and carrying out classification numbering on the first classification result to obtain a second classification result;
and the server performs deviation correction and classification numbering processing on the first classification result to obtain a second classification result. Specifically, the server performs deviation rectifying processing on the first classification result through a preset deviation rectifying algorithm to obtain a first classification deviation rectifying result. For example, the first class classification result includes three kinds of tag values-1, 0 and 1, if the monitoring index data with the tag value-1 is actually similar to the monitoring index data with the tag value-1 and should be combined into the same class, the server modifies the tag value from-1 to 1 according to a preset deviation correction algorithm, and finally the server adjusts the first classification result output by aggregation to 0 and 1, which is not limited herein.
And the server performs classification numbering processing according to the first classification deviation correcting result to obtain a second classification result. For example, the classification numbers of the first classification deviation correcting result are sequentially numbered from class 0, class 1 and class 2, and the numbers may be different from the labels of the first classification result, but the numbers of the monitoring index data of the same class are the same, and are not limited herein.
It should be noted that, the server has an error according to the first classification result obtained by the clustering algorithm, so the server adjusts the monitoring index data by adopting a deviation rectifying and classification numbering processing mode to obtain a more accurate classification result of the monitoring index data.
104. And classifying the second classification result through the trained neural network model to obtain a final classification result.
And the server trains the second classification result through the trained neural network model to obtain a final classification result. Specifically, the server classifies the second classification result through a preset one-dimensional convolutional neural network model to obtain a final classification result, wherein the final classification result is a group of classification labels, and the server marks the final classification result according to a preset label value. For example, the label value of the second classification result of the monitor index data A, B, C, D and the second classification result of the monitor index data E are labeled as [1,0,0,1,0], wherein the classification results of a and D are the same, the classification results of B, C and E are the same, and after the server performs classification processing on the second classification result of A, B, C, D and the second classification result of E through the trained neural network model, the preset label value of the final classification result is [1, 0], that is, the classification results of B, C, D and the classification result of E are the same.
It should be noted that, the server adjusts the value of the parameter of the preset one-dimensional convolutional neural network model according to a preset gradient descent algorithm to perform iterative optimization, the gradient descent algorithm is commonly used in machine learning and artificial intelligence to recursively approach the minimum deviation model, and the calculation process is to solve the minimum value along the gradient descent direction. The one-dimensional convolutional neural network model is used for further recalculating the classification of the second classification result, so that the accuracy of algorithm classification is improved; meanwhile, the one-dimensional convolution calculation is used for reducing the calculation time consumption of a program and improving the classification performance of the monitoring index data.
In the embodiment of the invention, after preprocessing each monitoring index, clustering calculation, deviation correction processing, classification numbering processing and classification processing are performed on the trained neural network model, so that a final classification result is obtained. The problem that the practical applicability and the universality of the existing monitoring index data are limited due to dependence on a large number of training models is well solved, classification of the monitoring index data is beneficial to reducing development work of the training models, the research and development period is shortened, the monitoring index data with the same classification can share a data set, and the establishment of monitoring resource portraits is facilitated.
Referring to fig. 2, another embodiment of the classification method for monitoring index data based on artificial intelligence in the embodiment of the invention includes:
201. acquiring monitoring index data;
the server acquires the monitoring index data. Specifically, the server may read the monitoring index data from the preset database, and the server may also collect the monitoring index data in real time, which is not limited herein.
It should be noted that, the monitoring index data is time-varying, and has a continuous output, and also has a monitoring index data output under a specific condition, where a value related to the system, such as the number of users currently logged into the Web application, needs to be collected at a specific time. Thus, the metrics are typically collected at regular intervals, such as once every second and once every minute. The server obtains monitoring index data with sufficient granularity to show significant peaks and dips. Specific granularity and monitoring systems, the cost of acquisition and the duration between index changes are related. Different monitoring index data may have different acquisition granularity, the memory or the CPU may use second as granularity statistics, and the energy consumption may use minute as granularity statistics to obtain monitoring index data of multiple ranges.
202. Performing standardized processing on the monitoring index data to obtain an original data sequence;
and the server performs standardization processing on the monitoring index data to obtain an original data sequence. Specifically, before data analysis, the server performs standardization processing on the monitoring index data according to a standard deviation standardization algorithm to obtain an original data sequence. Standard deviation normalization, also called z-score normalization, is to normalize data based on the mean and standard deviation of the original data, and the original data x is normalized by z-score to obtain x', where the conversion formula is:
wherein, the liquid crystal display device comprises a liquid crystal display device,for the mean value of the raw data, +.>Is the standard deviation of the original data.
It should be noted that the z-score normalization algorithm is applicable to the case where the maximum value and the minimum value of the data sequence are unknown, or the case where there is outlier data out of the range of the values.
203. Smoothing the original data sequence to obtain a smoothed sequence;
the server performs smoothing processing on the original data sequence to obtain a smoothed sequence so as to remove abnormal values and noise data of the original data sequence. Specifically, the server performs smoothing on the original data sequence by using a sequence data smoothing algorithm to obtain a smoothed sequence, wherein the used sequence data smoothing algorithm is an exponential weighted moving average method (exponentially weighted moving average, EWMA), and means that the weighting coefficient of each numerical value decreases exponentially with time, and the numerical value weighting coefficient is larger when the numerical value is closer to the current moment, and the calculation formula is as follows:
EWMA(t)＝λY(t)+(1-λ)EWMA(t-1)；
Wherein t=1, 2, 3..n., EWMA (t) is an estimate of time t, Y (t) is the actual observed or measured value at time t, λ (0 < λ < 1) is a weight coefficient, and the closer the value is to 1, the lower the weight on the history measurement value.
204. Performing difference operation according to the original data sequence and the smooth sequence to obtain a residual sequence;
and the server performs difference operation according to the original data sequence and the smooth sequence to obtain a residual sequence. The residual error refers to the difference between the actual observed value and the fitting value in mathematical statistics, and the server calculates the residual error sequence according to the original data sequence and the smooth sequence, for example, as follows:
original data sequence: [1,2,9,2,1],
smoothing sequence: [1,2,3,2,1],
residual sequence: [0,0,6,0,0].
205. Calculating according to the smooth sequence and the residual sequence to obtain the correlation of the monitoring index data, wherein the value range of the correlation is more than or equal to-1 and less than or equal to 1;
and the server calculates according to the smooth sequence and the residual sequence to obtain the correlation of the monitoring index data, wherein the value range is greater than or equal to-1 and less than or equal to 1. Correlation refers to the degree of correlation of two variables. In general, two variables can be observed from the scatter plot in one of three relationships: the two variables are positively correlated, negatively correlated and uncorrelated. The algorithm for calculating the correlation is a normalized cross correlation (normalized cross correlation, NCC) algorithm which is generally an algorithm for comparing whether two pictures are identical or not in image processing, but which is still applicable to monitoring index data, assuming two items of monitoring index data S ε R 1×n ，S'∈R 1×n Comparing whether the two monitoring index data are the same, and calculating the following formula:
the server calculates the correlation between the smoothed sequence and the residual sequence as R by NCC algorithm, wherein the value range of R is [ -1,1].
206. Calculating a distance numerical index according to the correlation of the monitoring index data, wherein the value range of the distance numerical index is more than or equal to 0 and less than or equal to 2;
and the server performs difference operation according to the correlation R of the value 1 and the monitoring index data to obtain a distance value index, wherein the value range is greater than or equal to 0 and less than or equal to 2. The conversion formula is as follows:
D＝1-R；
the server calculates the value range of the correlation R of the smooth sequence and the residual sequence to be [ -1,1] through NCC algorithm, and the value range of the distance numerical index D is [0,2]. It should be noted that, the distance value index is used to calculate the similarity of two pieces of monitoring index data, when the distance value index is 0, it represents that two pieces of monitoring index data are similar, otherwise, when the distance value index is 2, it represents that two pieces of monitoring index data are dissimilar, and the smaller the distance value index value is, the higher the similarity of two pieces of monitoring index data is.
207. Clustering calculation is carried out according to the distance numerical index, and a first classification result is obtained;
and the server performs clustering calculation according to the distance numerical index to obtain a first classification result. Specifically, the server performs cluster calculation through a density-based clustering DBSCAN algorithm with noise according to the distance numerical index to obtain a first classification result. The clustering algorithm is to measure the distance between two pieces of monitoring index data, that is, based on the distance numerical index, and the clustering algorithm is that the two pieces of monitoring index data with smaller distance numerical index are more likely to be aggregated into the same class. The clustering algorithm uses a density-based clustering with noise (DBSCAN) algorithm. The algorithm divides regions of sufficient density into clusters and finds arbitrarily shaped clusters in the noisy spatial database, which are defined as the largest set of densely connected points.
It should be noted that, for example, after 40 pieces of monitoring index data are processed by the DBSCAN clustering algorithm, 24 pieces of monitoring index data are divided into the same class, the algorithm output label value is marked as 0, other 16 pieces of monitoring index data cannot be clustered, the label value output by the algorithm is marked as-1, 40 pieces of monitoring index data are aggregated into two classes of-1 and 0, and the server sets-1 and 0 as the first classification result. Wherein the tag value may also be marked as other integers, without limitation herein.
208. Correcting the deviation and carrying out classification numbering on the first classification result to obtain a second classification result;
and the server performs deviation correction and classification numbering processing on the first classification result to obtain a second classification result. Specifically, the server performs deviation rectifying processing on the first classification result according to a preset deviation rectifying algorithm to obtain a first classification deviation rectifying result, for example, the first classification result comprises three kinds of label values-1, 0 and 1, if the monitoring index data with the label value-1 is similar to the monitoring index data with the label value-1 in practice and should be combined into the same type, the label value is modified from-1 to 1, and finally the server adjusts the first classification result output by aggregation into two types; and the server performs classification numbering processing according to the first classification deviation correcting result to obtain a second classification result. For example, the classification numbers of the first classification deviation correcting result are sequentially numbered from class 0, class 1 and class 2, and the numbers may be different from the tag values of the first classification result, but the numbers of the monitoring index data of the same class are the same, and are not limited herein.
It should be noted that, the server has an error according to the first classification result obtained by the clustering algorithm, so the server adjusts the monitoring index data by adopting a deviation rectifying and classification numbering processing mode to obtain a more accurate classification result of the monitoring index data. For example, the first classification result is [0, -1, 0], and the server performs correction and classification numbering processing on the first classification result to obtain a second classification result is [0,0,1,1,0].
209. And classifying the second classification result through the trained neural network model to obtain a final classification result.
And the trained neural network model of the server carries out classification processing on the second classification result to obtain a final classification result. Specifically, the server inputs a second classification result into a preset one-dimensional convolutional neural network model, wherein the preset one-dimensional convolutional neural network model is a trained neural network model; the server classifies the second classification result through a preset one-dimensional convolutional neural network model to obtain a target classification label, the target classification label is set as a final classification result, and the target classification label is used for indicating the classification type of the monitoring index data according to the preset label value. The one-dimensional convolutional neural network model is commonly used for training and learning the sequence model. For example, the label value of the second classification result of the monitor index data A, B, C, D and E is labeled as [1,0,0,1,0], where the label values of a and D are the same, and the label values of B, C and E are the same, and after the server classifies the second classification result of A, B, C, D and E by using the trained neural network model, the target classification label is obtained as [1, 0], and the target classification label is labeled with a preset label value, where the preset label values of B, C, D and E are 0, B, C, D and E are the same classification, a is another classification, and the final classification result includes two preset label values.
Optionally, training the neural network model according to the target training sample to obtain a trained neural network model, and taking the value of the parameter when the training frequency of the neural network model reaches a preset threshold value as the value of the parameter of the trained neural network model; when the newly added monitoring index data of different types are detected, setting the newly added monitoring index data of different types as target training samples, and carrying out iterative optimization on the trained neural network model according to the target training samples.
It should be noted that, the server adjusts the value of the parameter of the preset one-dimensional convolutional neural network model according to a preset gradient descent algorithm to perform iterative optimization, where the gradient descent algorithm is commonly used in machine learning and artificial intelligence to recursively approach the minimum deviation model, and the calculation process is to solve the minimum value along the gradient descent direction. The server further calculates and classifies the second classification result by using a preset one-dimensional convolutional neural network model, so that the algorithm classification accuracy is improved; meanwhile, the one-dimensional convolution calculation is used for reducing the calculation time consumption of a program and improving the classification performance of the monitoring index data.
In the embodiment of the invention, after preprocessing each monitoring index, clustering calculation, deviation correction processing, classification numbering processing and classification processing are performed on the trained neural network model, so that a final classification result is obtained. The problem that the practical applicability and the universality of the existing monitoring index data are limited due to dependence on a large number of training models is well solved, classification of the monitoring index data is beneficial to reducing development work of the training models, the research and development period is shortened, the monitoring index data with the same classification can share a data set, and the establishment of monitoring resource portraits is facilitated.
The method for classifying monitoring index data based on artificial intelligence in the embodiment of the present invention is described above, and the device for classifying monitoring index data based on artificial intelligence in the embodiment of the present invention is described below, referring to fig. 3, one embodiment of the device for classifying monitoring index data based on artificial intelligence in the embodiment of the present invention includes:
an acquisition unit 301, configured to acquire monitoring index data;
the first classification unit 302 is configured to perform preprocessing and clustering calculation on the monitoring index data to obtain a first classification result;
a second classification unit 303, configured to perform deviation rectification and classification numbering processing on the first classification result, to obtain a second classification result;
and the final classification unit 304 is configured to classify the second classification result through the trained neural network model, so as to obtain a final classification result.
In the embodiment of the invention, after preprocessing each monitoring index, clustering calculation, deviation correction processing, classification numbering processing and processing of a trained neural network model are performed to obtain a final classification result. The problem that the practical applicability and the universality of the existing monitoring index data are limited due to dependence on a large number of training models is well solved, classification of the monitoring index data is beneficial to reducing development work of the training models, the research and development period is shortened, the monitoring index data with the same classification can share a data set, and the establishment of monitoring resource portraits is facilitated.
Referring to fig. 4, another embodiment of the classification apparatus for monitoring index data based on artificial intelligence according to the embodiment of the present invention includes:
an acquisition unit 301, configured to acquire monitoring index data;
the first classification unit 302 is configured to perform preprocessing and clustering calculation on the monitoring index data to obtain a first classification result;
a second classification unit 303, configured to perform deviation rectification and classification numbering processing on the first classification result, to obtain a second classification result;
and the final classification unit 304 is configured to classify the second classification result through the trained neural network model, so as to obtain a final classification result.
Optionally, the first classification unit 302 may further include:
a preprocessing subunit 3021, configured to preprocess the monitor indicator data to obtain a distance numerical indicator;
and the clustering subunit 3022 is configured to perform clustering calculation according to the distance numerical index to obtain a first classification result.
Optionally, the preprocessing subunit 3021 may further be specifically configured to:
performing standardized processing on the monitoring index data to obtain an original data sequence;
smoothing the original data sequence to obtain a smoothed sequence;
performing difference operation according to the original data sequence and the smooth sequence to obtain a residual sequence;
Calculating according to the smooth sequence and the residual sequence to obtain the correlation of the monitoring index data, wherein the value range of the correlation is more than or equal to-1 and less than or equal to 1;
and calculating a distance numerical index according to the correlation of the monitoring index data, wherein the value range of the distance numerical index is larger than or equal to 0 and smaller than or equal to 2.
Optionally, the clustering subunit 3022 may be further specifically configured to:
and carrying out clustering calculation through a density-based clustering DBSCAN algorithm with noise according to the distance numerical index to obtain a first classification result.
Optionally, the second classification unit 303 may be further specifically configured to:
performing deviation rectifying treatment on the first classification result according to a preset deviation rectifying algorithm to obtain a first classification deviation rectifying result;
and carrying out classification numbering processing according to the first classification deviation correcting result to obtain a second classification result.
Optionally, the final classification unit 304 may be further specifically configured to:
inputting the second classification result into a preset one-dimensional convolutional neural network model, wherein the preset one-dimensional convolutional neural network model is a trained neural network model;
and classifying the second classification result through a preset one-dimensional convolutional neural network model to obtain a target classification label, setting the target classification label as a final classification result, and marking the classification type of the monitoring index data according to a preset label value by the target classification label.
Optionally, the classification device based on the monitoring index data of the artificial intelligence may further include:
the training unit 305 is configured to train the neural network model according to a preset training sample, obtain a trained neural network model, and use a value of a parameter when the training frequency of the neural network model reaches a preset threshold value as a value of the parameter of the trained neural network model;
and the iteration unit 306 is used for setting the newly added monitoring index data of different types as target training samples when the newly added monitoring index data of different types are detected, and carrying out iterative optimization on the trained neural network model according to the target training samples.
In the embodiment of the invention, after preprocessing each monitoring index, the final classification result is obtained through clustering calculation, deviation correction processing, classification numbering processing and neural network model training. The problem that the practical applicability and the universality of the existing monitoring index data are limited due to dependence on a large number of training models is well solved, classification of the monitoring index data is beneficial to reducing development work of the training models, the research and development period is shortened, the monitoring index data with the same classification can share a data set, and the establishment of monitoring resource portraits is facilitated.
The classifying device based on the monitoring index data of the artificial intelligence in the embodiment of the present invention is described in detail from the point of view of the modularized functional entity in the above fig. 3 and 4, and the classifying device based on the monitoring index data of the artificial intelligence in the embodiment of the present invention is described in detail from the point of view of hardware processing in the following.
Fig. 5 is a schematic structural diagram of an artificial intelligence-based monitoring index data classifying device 500 according to an embodiment of the present invention, where the artificial intelligence-based monitoring index data classifying device 500 may have relatively large differences according to configuration or performance, and may include one or more processors (central processing units, CPU) 501 (e.g., one or more processors) and a memory 509, and one or more storage media 508 (e.g., one or more mass storage devices) storing application programs 509 or data 509. Wherein the memory 509 and storage medium 508 may be transitory or persistent storage. The program stored on the storage medium 508 may include one or more modules (not shown), each of which may include a series of instruction operations in classifying the monitoring metric data based on artificial intelligence. Still further, the processor 501 may be configured to communicate with the storage medium 508 to execute a series of instruction operations in the storage medium 508 on the classification device 500 based on the monitored metric data of the artificial intelligence.
The artificial intelligence based classification device 500 may also include one or more power sources 502, one or more wired or wireless network interfaces 503, one or more input/output interfaces 504, and/or one or more operating systems 505, such as Windows Serve, mac OS X, unix, linux, freeBSD, etc. It will be appreciated by those skilled in the art that the classification device structure based on artificial intelligence monitoring metric data shown in FIG. 5 does not constitute a limitation of the classification device based on artificial intelligence monitoring metric data, and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in the present invention, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A method for classifying monitoring index data based on artificial intelligence, comprising the steps of:
acquiring monitoring index data;
preprocessing and clustering calculation are carried out on the monitoring index data to obtain a first classification result;
correcting the deviation and carrying out classification numbering on the first classification result to obtain a second classification result;
classifying the second classification result through a trained neural network model to obtain a final classification result;
the preprocessing and clustering calculation are carried out on the monitoring index data, and the obtaining of the first classification result comprises the following steps:
preprocessing the monitoring index data to obtain a distance numerical index;
Performing clustering calculation according to the distance numerical index to obtain a first classification result;
the preprocessing of the monitoring index data to obtain a distance numerical index comprises the following steps:
performing standardized processing on the monitoring index data to obtain an original data sequence;
smoothing the original data sequence to obtain a smoothed sequence;
performing difference operation according to the original data sequence and the smooth sequence to obtain a residual sequence;
calculating according to the smooth sequence and the residual sequence to obtain the correlation of the monitoring index data, wherein the value range of the correlation is greater than or equal to-1 and less than or equal to 1;
calculating the distance numerical index according to the correlation of the monitoring index data, wherein the value range of the distance numerical index is larger than or equal to 0 and smaller than or equal to 2;
performing cluster calculation according to the distance numerical index to obtain a first classification result, wherein the step of obtaining the first classification result comprises the following steps:
performing clustering calculation through a density-based clustering DBSCAN algorithm with noise according to the distance numerical index to obtain a first classification result;
the classification method based on the monitoring index data of the artificial intelligence further comprises the following steps of:
Training a neural network model according to a preset training sample to obtain a trained neural network model, and taking the value of a parameter when the training frequency of the neural network model reaches a preset threshold value as the value of the parameter of the trained neural network model;
when detecting the newly added different types of monitoring index data, setting the newly added different types of monitoring index data as a target training sample, and carrying out iterative optimization on the trained neural network model according to the target training sample.
2. The method for classifying monitoring index data based on artificial intelligence according to claim 1, wherein the performing correction and classification numbering processing on the first classification result to obtain a second classification result comprises:
performing deviation rectifying treatment on the first classification result through a preset deviation rectifying algorithm to obtain a first classification deviation rectifying result;
and carrying out classification numbering processing according to the first classification deviation correcting result to obtain a second classification result.
3. The method for classifying monitoring index data based on artificial intelligence according to claim 1, wherein classifying the second classification result by the trained neural network model to obtain a final classification result comprises:
Inputting the second classification result into a preset one-dimensional convolutional neural network model, wherein the preset one-dimensional convolutional neural network model is a trained neural network model;
and classifying the second classification result through the preset one-dimensional convolutional neural network model to obtain a target classification label, and setting the target classification label as a final classification result, wherein the target classification label is used for indicating the classification type of the monitoring index data marked according to a preset label value.
4. A classification device for monitoring index data based on artificial intelligence, which is characterized in that the classification device comprises:
the acquisition unit is used for acquiring the monitoring index data;
the first classification unit is used for preprocessing and clustering calculation of the monitoring index data to obtain a first classification result;
the second classification unit is used for carrying out deviation correction and classification numbering treatment on the first classification result to obtain a second classification result;
the final classification unit is used for classifying the second classification result through a trained neural network model to obtain a final classification result;
the first classification unit includes: the preprocessing subunit is used for preprocessing the monitoring index data to obtain a distance numerical index; the clustering subunit is used for carrying out clustering calculation according to the distance numerical index to obtain a first classification result;
The preprocessing subunit is specifically configured to: performing standardized processing on the monitoring index data to obtain an original data sequence; smoothing the original data sequence to obtain a smoothed sequence; performing difference operation according to the original data sequence and the smooth sequence to obtain a residual sequence; calculating according to the smooth sequence and the residual sequence to obtain the correlation of the monitoring index data, wherein the value range of the correlation is greater than or equal to-1 and less than or equal to 1; calculating the distance numerical index according to the correlation of the monitoring index data, wherein the value range of the distance numerical index is larger than or equal to 0 and smaller than or equal to 2;
the clustering subunit is specifically configured to: performing clustering calculation through a density-based clustering DBSCAN algorithm with noise according to the distance numerical index to obtain a first classification result;
the training unit is used for training the neural network model according to a preset training sample to obtain a trained neural network model, and taking the value of the parameter when the training times of the neural network model reach a preset threshold value as the value of the parameter of the trained neural network model; and the iteration unit is used for setting the newly added monitoring index data of different types as a target training sample when detecting the newly added monitoring index data of different types, and carrying out iterative optimization on the trained neural network model according to the target training sample.
5. The classification device based on artificial intelligence according to claim 4, wherein the second classification unit is specifically configured to:
performing deviation rectifying treatment on the first classification result through a preset deviation rectifying algorithm to obtain a first classification deviation rectifying result;
and carrying out classification numbering processing according to the first classification deviation correcting result to obtain a second classification result.
6. The classification device based on artificial intelligence according to claim 4, wherein the final classification unit is specifically configured to:
inputting the second classification result into a preset one-dimensional convolutional neural network model, wherein the preset one-dimensional convolutional neural network model is a trained neural network model;
and classifying the second classification result through the preset one-dimensional convolutional neural network model to obtain a target classification label, and setting the target classification label as a final classification result, wherein the target classification label is used for indicating the classification type of the monitoring index data marked according to a preset label value.
7. An artificial intelligence based classification device for monitoring index data, wherein the artificial intelligence based classification device for monitoring index data comprises: the system comprises a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through a line;
The at least one processor invokes the instructions in the memory to cause the artificial intelligence based monitoring metric data classification device to perform the artificial intelligence based monitoring metric data classification method of any of claims 1-3.
8. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program, when executed by a processor, implements the artificial intelligence based classification method of monitoring metric data as claimed in any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910540031.9A CN110413703B (en) | 2019-06-21 | 2019-06-21 | Method for classifying monitoring index data based on artificial intelligence and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910540031.9A CN110413703B (en) | 2019-06-21 | 2019-06-21 | Method for classifying monitoring index data based on artificial intelligence and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413703A CN110413703A (en) | 2019-11-05 |
| CN110413703B true CN110413703B (en) | 2023-07-25 |
Family
ID=68359543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910540031.9A Active CN110413703B (en) | 2019-06-21 | 2019-06-21 | Method for classifying monitoring index data based on artificial intelligence and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413703B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111459695A (en) * | 2020-03-12 | 2020-07-28 | 平安科技（深圳）有限公司 | Root cause positioning method and device, computer equipment and storage medium |
| CN112559742A (en) * | 2020-12-08 | 2021-03-26 | 北京伟杰东博信息科技有限公司 | Classified storage method and system thereof |
| CN113283502B (en) * | 2021-05-24 | 2023-04-28 | 平安国际融资租赁有限公司 | Device state threshold determining method and device based on clustering |
| US11928009B2 (en) | 2021-08-06 | 2024-03-12 | International Business Machines Corporation | Predicting a root cause of an alert using a recurrent neural network |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104091035A (en) * | 2014-07-30 | 2014-10-08 | 中国科学院空间应用工程与技术中心 | Health monitoring method for effective loads of space station based on data-driven algorithm |
| CN108923952A (en) * | 2018-05-31 | 2018-11-30 | 北京百度网讯科技有限公司 | Method for diagnosing faults, equipment and storage medium based on service monitoring index |
| CN109495296A (en) * | 2018-11-02 | 2019-03-19 | 国网四川省电力公司电力科学研究院 | Intelligent substation communication network state evaluation method based on clustering and neural network |
| CN109872003A (en) * | 2019-03-06 | 2019-06-11 | 中国科学院软件研究所 | Obj State prediction technique, system, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10261851B2 (en) * | 2015-01-23 | 2019-04-16 | Lightbend, Inc. | Anomaly detection using circumstance-specific detectors |
-
2019
- 2019-06-21 CN CN201910540031.9A patent/CN110413703B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104091035A (en) * | 2014-07-30 | 2014-10-08 | 中国科学院空间应用工程与技术中心 | Health monitoring method for effective loads of space station based on data-driven algorithm |
| CN108923952A (en) * | 2018-05-31 | 2018-11-30 | 北京百度网讯科技有限公司 | Method for diagnosing faults, equipment and storage medium based on service monitoring index |
| CN109495296A (en) * | 2018-11-02 | 2019-03-19 | 国网四川省电力公司电力科学研究院 | Intelligent substation communication network state evaluation method based on clustering and neural network |
| CN109872003A (en) * | 2019-03-06 | 2019-06-11 | 中国科学院软件研究所 | Obj State prediction technique, system, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413703A (en) | 2019-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110413703B (en) | Method for classifying monitoring index data based on artificial intelligence and related equipment | |
| US9275353B2 (en) | Event-processing operators | |
| CN110806954B (en) | Method, device, equipment and storage medium for evaluating cloud host resources | |
| WO2021043140A1 (en) | Method, apparatus and system for determining label | |
| EP1958034B1 (en) | Use of sequential clustering for instance selection in machine condition monitoring | |
| CN108228684B (en) | Method and device for training clustering model, electronic equipment and computer storage medium | |
| EP3200076A1 (en) | System and method for load estimation of virtual machines in a cloud environment and serving node | |
| CN111984442A (en) | Method and device for detecting abnormality of computer cluster system, and storage medium | |
| CN114270332A (en) | Method and system for automatically detecting anomalies in data | |
| CN114978956B (en) | Method and device for detecting abnormal mutation points of performance of intelligent city network equipment | |
| WO2018211721A1 (en) | Abnormal information estimation device, abnormal information estimation method, and program | |
| CN111666187A (en) | Method and apparatus for detecting abnormal response time | |
| CN111753875A (en) | Power information system operation trend analysis method and device and storage medium | |
| CN108280007B (en) | Method and device for evaluating equipment resource utilization rate | |
| CN109784403B (en) | Method for identifying risk equipment and related equipment | |
| CN113810792A (en) | Edge data acquisition and analysis system based on cloud computing | |
| US9397921B2 (en) | Method and system for signal categorization for monitoring and detecting health changes in a database system | |
| CN112070180B (en) | Power grid equipment state judging method and device based on information physical bilateral data | |
| CN110837953A (en) | Automatic abnormal entity positioning analysis method | |
| CN114202009A (en) | Medical equipment performance index abnormity detection method and device based on PU learning | |
| CN113568952A (en) | Internet of things resource data analysis method | |
| CN113592218A (en) | Photovoltaic user baseline load estimation method and device and terminal equipment | |
| CN113705920A (en) | Generation method of water data sample set for thermal power plant and terminal equipment | |
| CN113506266A (en) | Method, device and equipment for detecting tongue greasy coating and storage medium | |
| CN113032225A (en) | Monitoring data processing method, device and equipment of data center and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |