CN112506996A - Data anomaly detection method and device, computer equipment and readable storage medium - Google Patents
Data anomaly detection method and device, computer equipment and readable storage medium Download PDFInfo
- Publication number
- CN112506996A CN112506996A CN202011451616.2A CN202011451616A CN112506996A CN 112506996 A CN112506996 A CN 112506996A CN 202011451616 A CN202011451616 A CN 202011451616A CN 112506996 A CN112506996 A CN 112506996A
- Authority
- CN
- China
- Prior art keywords
- industrial data
- sample
- data
- detected
- industrial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 86
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 230000002159 abnormal effect Effects 0.000 claims abstract description 51
- 238000012549 training Methods 0.000 claims description 13
- 238000000034 method Methods 0.000 claims description 10
- 230000000694 effects Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 230000005856 abnormality Effects 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 1
- 230000002547 anomalous effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005553 drilling Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The embodiment of the application provides a data anomaly detection method, a data anomaly detection device, computer equipment and a readable storage medium, and relates to the technical field of industrial data monitoring, wherein the data anomaly detection method comprises the following steps: acquiring industrial data to be detected, wherein the industrial data to be detected comprises time sequence characteristics; constructing and obtaining a to-be-detected industrial data sequence corresponding to the to-be-detected industrial data according to the time sequence characteristics; inputting the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value; and determining an abnormal data segment in the industrial data to be detected according to the reconstruction value, and efficiently realizing the abnormal detection of the industrial data through the steps.
Description
Technical Field
The application relates to the technical field of industrial data monitoring, in particular to a data anomaly detection method and device, computer equipment and a readable storage medium.
Background
With the development of industrial intelligence, more and more industrial devices have been operated automatically. Due to the characteristics of time sequence, no label and the like of industrial data generated in related industrial processes, the conventional abnormal data detection scheme based on the traditional neural network cannot accurately detect abnormal data in the industrial data.
In view of this, it is necessary for those skilled in the art to provide a solution for accurately detecting the industrial data anomaly.
Disclosure of Invention
The application provides a data anomaly detection method and device, computer equipment and a readable storage medium.
The embodiment of the application can be realized as follows:
in a first aspect, the present application provides a data anomaly detection method, including:
acquiring industrial data to be detected, wherein the industrial data to be detected comprises time sequence characteristics;
constructing and obtaining a to-be-detected industrial data sequence corresponding to the to-be-detected industrial data according to the time sequence characteristics;
inputting an industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value;
and determining an abnormal data segment in the industrial data to be detected according to the reconstruction value.
In an alternative embodiment, the anomaly detection model is trained by:
obtaining sample industrial data, wherein the sample industrial data comprises time sequence characteristics;
constructing a sample industrial data sequence corresponding to the obtained sample industrial data according to the time sequence characteristics;
inputting a sample industrial data sequence into a pre-constructed anomaly detection model to obtain a sample reconstruction value;
and training the abnormal detection model according to the sample reconstruction value, and obtaining the trained abnormal detection model under the condition that the sample reconstruction value achieves the preset fitting effect.
In an alternative embodiment, the anomaly detection model includes an encoding layer and a decoding layer;
inputting a sample industrial data sequence into a pre-constructed anomaly detection model to obtain a sample reconstruction value, wherein the step comprises the following steps:
acquiring a plurality of sample industrial data subsequences from the sample industrial data sequence by utilizing a pre-constructed sliding window;
inputting a plurality of sample industrial data subsequences into an encoding layer to obtain a sample hidden variable corresponding to each sample industrial data subsequence;
inputting all sample hidden variables into a decoding layer to obtain a sample sub-reconstruction value corresponding to each sample industrial data subsequence;
and taking all the sample sub-reconstruction values as sample reconstruction values.
In an optional embodiment, the step of inputting a plurality of sample industrial data subsequences into the coding layer to obtain a sample hidden variable corresponding to each sample industrial data subsequence includes:
inputting a plurality of sample industrial data subsequences into an encoding layer to obtain a sample hidden variable mean value and a sample hidden variable variance corresponding to each sample industrial data subsequence;
and splicing the mean value of each sample hidden variable and the variance of each sample hidden variable, and adding Gaussian noise to obtain each sample hidden variable.
In an optional embodiment, the reconstruction value includes a plurality of sub-reconstruction values, the industrial data sequence to be detected includes a plurality of industrial data sub-sequences to be detected, and the plurality of sub-reconstruction values and the plurality of industrial data sub-sequences to be detected are in one-to-one correspondence based on the time sequence characteristics;
the method comprises the following steps of determining an abnormal data segment in industrial data to be detected according to a reconstruction value, wherein the steps comprise:
under the condition that the target sub-reconstruction value exceeds a preset threshold value, judging that a target industrial data sub-sequence to be detected corresponding to the target sub-reconstruction value is an abnormal data sub-sequence, wherein the target sub-reconstruction value is any one of a plurality of sub-reconstruction values;
repeating the steps until all abnormal data subsequences in the industrial data to be detected are determined;
and determining an abnormal data segment from the industrial data to be detected according to all the abnormal data subsequences based on the time sequence characteristics.
In an optional embodiment, the step of obtaining the industrial data to be measured includes:
preprocessing original industrial data to remove a white noise wave band in the original industrial data;
and carrying out standardization processing on the preprocessed original industrial data to obtain the industrial data to be detected.
In a second aspect, the present application provides a data anomaly detection apparatus, including:
the acquisition module is used for acquiring industrial data to be detected, wherein the industrial data to be detected comprises time sequence characteristics;
the construction module is used for constructing and obtaining a to-be-detected industrial data sequence corresponding to the to-be-detected industrial data according to the time sequence characteristics;
the detection module is used for inputting the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value; and determining an abnormal data segment in the industrial data to be detected according to the reconstruction value.
In an alternative embodiment, the building module is further configured to:
obtaining sample industrial data, wherein the sample industrial data comprises time sequence characteristics; constructing a sample industrial data sequence corresponding to the obtained sample industrial data according to the time sequence characteristics; inputting a sample industrial data sequence into a pre-constructed anomaly detection model to obtain a sample reconstruction value; and training the abnormal detection model according to the sample reconstruction value, and obtaining the trained abnormal detection model under the condition that the sample reconstruction value achieves the preset fitting effect.
In a third aspect, the present application provides a computer device, where the computer device includes a processor and a non-volatile memory storing computer instructions, and when the computer instructions are executed by the processor, the computer device executes the data anomaly detection method in any one of the foregoing embodiments.
In a fourth aspect, the present application provides a readable storage medium, where the readable storage medium includes a computer program, and the computer program controls a computer device in the readable storage medium to execute the data anomaly detection method in any one of the foregoing embodiments when running.
The beneficial effects of the embodiment of the application include, for example: the embodiment of the application provides a data anomaly detection method, a data anomaly detection device, computer equipment and a readable storage medium, wherein industrial data to be detected are obtained, and the industrial data to be detected comprise time sequence characteristics; then, according to the time sequence characteristics, constructing and obtaining an industrial data sequence to be detected corresponding to the industrial data to be detected; then inputting the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value; and finally, determining an abnormal data section in the industrial data to be detected according to the reconstruction value, and skillfully utilizing an abnormal detection model to realize efficient industrial data abnormal detection.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a schematic flow chart illustrating a step of a data anomaly detection method according to an embodiment of the present application;
fig. 2 is a schematic flow chart illustrating another step of a data anomaly detection method according to an embodiment of the present application;
fig. 3 is a schematic block diagram of a data anomaly detection apparatus according to an embodiment of the present application;
fig. 4 is a block diagram schematically illustrating a structure of a computer device according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
It should be noted that the features of the embodiments of the present application may be combined with each other without conflict.
Industrial automation has become a trend, not only the task progress is greatly improved, but also more precise and accurate operation than manual operation can be realized. In order to maintain the normal operation of industrial automation, in the prior art, the industrial data generated in the industrial automation process needs to be detected abnormally manually to ensure the normal operation of industrial operation, and because the industrial data often has a time-sequence characteristic, the conventional detection scheme based on the traditional neural network has low efficiency and high false detection rate.
To solve the above-mentioned problems, please refer to fig. 1, and fig. 1 is a schematic flow chart illustrating steps of a data anomaly detection method according to an embodiment of the present application. The data abnormality detection method is described in detail below.
Step 201, acquiring industrial data to be measured.
The industrial data to be detected are all related data with time sequence characteristics, such as offshore drilling platform motion data (data of speed, acceleration, angular velocity and the like of a drill bit), and due to the characteristics of large volume, multi-source isomerism, no label, potential sequence association, high dynamics and the like of the industrial data, the existing detection scheme based on the neural network is as follows: algorithms such as K-Nearest Neighbor (KNN), Local anomaly Factor (LOF), and high-bandwidth-based anomaly detection (HBOS) distinguish anomalous data by using the similarity between samples, which all have the problems of high computational complexity and high omission Factor.
And 202, constructing and obtaining a to-be-detected industrial data sequence corresponding to the to-be-detected industrial data according to the time sequence characteristics.
In the embodiment of the present application, the time sequence characteristic may refer to all data of the industrial data to be measured in a certain continuous time range. The to-be-measured industrial data sequence corresponding to the to-be-measured industrial data is constructed according to the time sequence characteristics, namely the to-be-measured industrial data is divided into a plurality of sections according to a preset time interval so as to form the to-be-measured industrial data sequence corresponding to the to-be-measured industrial data, and the to-be-measured industrial data sequence can be displayed in a one-dimensional vector form.
And 203, inputting the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value.
In the embodiment of the application, the anomaly detection model can be constructed on the basis of an LSTM-VAE (Long Short-Term Memory-variable Auto-Encoder), and the reconstruction value can represent the anomaly condition of the industrial data to be detected corresponding to the industrial data sequence to be detected. It should be understood that, in the embodiment of the present application, the variational self-encoder is combined with the LSTM network sensitive to the time sequence characteristics, so as to obtain the variational self-encoder based on the LSTM, which can achieve the purpose of capturing the high-dimensional and time-dependent industrial data characteristics to be measured and performing the anomaly identification and alarm mechanism.
And step 204, determining an abnormal data segment in the industrial data to be detected according to the reconstruction value.
On the basis of determining the reconstruction value, the abnormal data section can be determined from the industrial data to be detected, the whole process does not need manual intervention, and the abnormal data section can be rapidly and accurately determined, so that subsequent debugging, overhauling and other operations of a user can be conveniently carried out.
On this basis, the abnormality detection model is trained in the following manner.
Step 301, sample industrial data is obtained.
Wherein the sample industrial data includes a timing characteristic.
And step 302, constructing and obtaining a sample industrial data sequence corresponding to the sample industrial data according to the time sequence characteristics.
Step 303, inputting the sample industrial data sequence into a pre-constructed anomaly detection model to obtain a sample reconstruction value.
And 304, training the abnormal detection model according to the sample reconstruction value, and obtaining the trained abnormal detection model under the condition that the sample reconstruction value achieves a preset fitting effect.
In the embodiment of the present application, the anomaly detection model may be obtained through the training in the above steps, and it should be understood that the sample industrial data, the sample industrial data sequence, and the corresponding sample reconstruction value are all known, and the training of the anomaly detection model may be considered to be completed under the condition that the sample reconstruction value output by the anomaly detection model obtained through the training based on the LSTM-VAE deep neural network and the sample obtained in advance are reconstructed to reach the preset fitting effect, that is, the confidence coefficient meets the requirement.
On this basis, please refer to fig. 2 in combination, the anomaly detection model includes an encoding layer and a decoding layer as an alternative embodiment, and the foregoing step 303 can be implemented by the following detailed steps.
Substep 303-1, obtaining a plurality of sample industrial data subsequences from the sample industrial data sequence using a pre-constructed sliding window.
And a substep 303-2, inputting the plurality of sample industrial data subsequences into an encoding layer to obtain a sample hidden variable corresponding to each sample industrial data subsequence.
And a substep 303-3, inputting all the sample hidden variables into a decoding layer to obtain a sample sub-reconstruction value corresponding to each sample industrial data sub-sequence.
Sub-step 303-4, taking all sample sub-reconstruction values as sample reconstruction values.
Optionally, the length of the pre-constructed sliding window may be 2800, a plurality of sample industrial data subsequences in the sample industrial data sequence may be obtained through continuous sliding, and the plurality of sample industrial data subsequences may be input into the coding layer as a training set to generate corresponding sample hidden variables, and then the sample hidden variables are input into the decoding layer to be decoded, so as to obtain sample reconstruction values of the sample industrial data sequence corresponding to the plurality of sample industrial data subsequences.
Through the steps, the training sample with higher reference value can be obtained based on the sliding window with the length of 2800 determined in the actual experimental process.
On the basis of the foregoing, in order to express the scheme of the present application more clearly, the foregoing sub-step 303-2 may be implemented by the following embodiments.
(1) And inputting the plurality of sample industrial data subsequences into the coding layer to obtain a sample implicit variable mean value and a sample implicit variable variance corresponding to each sample industrial data subsequence.
(2) And splicing the mean value of each sample hidden variable and the variance of each sample hidden variable, and adding Gaussian noise to obtain each sample hidden variable.
Optionally, a plurality of sample industrial data subsequences may be input into the coding layer to obtain a sample hidden variable mean and a sample hidden variable variance corresponding to each sample industrial data subsequences, and it should be understood that both the forms of the sample hidden variable mean and the sample hidden variable variance may exist in a vector form, so that each sample hidden variable mean and each sample hidden variable variance may be subjected to splicing processing, and gaussian noise may be added to obtain a sample hidden variable so as to obtain a model with better robustness.
On the basis, the reconstruction value comprises a plurality of sub-reconstruction values, the industrial data sequence to be detected comprises a plurality of industrial data sub-sequences to be detected, and the plurality of sub-reconstruction values and the plurality of industrial data sub-sequences to be detected are in one-to-one correspondence based on time sequence characteristics; as an alternative embodiment, the foregoing step 204 can be implemented by the following specific steps.
And a substep 204-1, under the condition that the target sub-reconstruction value exceeds a preset threshold value, judging that the target industrial data subsequence to be detected corresponding to the target sub-reconstruction value is an abnormal data subsequence.
Wherein the target sub-reconstruction value is any one of the plurality of sub-reconstruction values.
And a substep 204-2, repeating the steps until all abnormal data subsequences in the industrial data to be detected are determined.
And a substep 204-3 of determining an abnormal data segment from the industrial data to be detected according to all the abnormal data subsequences based on the time sequence characteristics.
In the embodiment of the application, after determining whether each industrial data subsequence to be tested is an abnormal data subsequence, it can be known whether each data segment corresponding to the industrial data subsequence to be tested according to a preset time period (i.e., based on a time sequence characteristic) has an abnormality. Through the steps, the abnormal data section in the industrial data to be detected can be determined quickly and accurately.
On this basis, as an alternative embodiment, the foregoing step 201 may be implemented in the following manner.
Substep 201-1, pre-processes the raw industrial data to remove white noise bands in the raw industrial data.
And a substep 201-2, performing standardization processing on the preprocessed original industrial data to obtain industrial data to be detected.
In order to ensure the accuracy of the data, the original industrial data can be preprocessed, so that a complete and non-white noise wave band is reserved, meanwhile, the original industrial data can be subjected to standardization processing to eliminate dimension influence, and preparation is made for subsequent exploration and analysis of the trend, the stationarity and the like of an original sequence (namely an industrial data sequence to be detected).
Except for the above scheme, the fitting effect of network training constructed based on the traditional AE (auto encoder for short) is not good, and cannot be completely matched with an actual value, and the detection of abnormal data by a corresponding obtained detection model is also incomplete, so that the scheme can be described more clearly by referring to table one.
Watch 1
As is clear from the above table, the training set fitting effect (i.e., the maximum mean absolute error) of the LSTM-VAE-based anomaly data detection model proposed in the present application is more accurate than that of the prior art self-encoder or variational self-encoder.
An embodiment of the present application provides a data anomaly detection apparatus 110, which may refer to fig. 3, and includes:
the obtaining module 1101 is configured to obtain industrial data to be measured, where the industrial data to be measured includes a time sequence feature.
The constructing module 1102 is configured to construct and obtain a to-be-detected industrial data sequence corresponding to the to-be-detected industrial data according to the time sequence characteristics.
The detection module 1103 is configured to input the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value; and determining an abnormal data segment in the industrial data to be detected according to the reconstruction value.
Further, the building module 1102 is further configured to:
obtaining sample industrial data, wherein the sample industrial data comprises time sequence characteristics; constructing a sample industrial data sequence corresponding to the obtained sample industrial data according to the time sequence characteristics; inputting a sample industrial data sequence into a pre-constructed anomaly detection model to obtain a sample reconstruction value; and training the abnormal detection model according to the sample reconstruction value, and obtaining the trained abnormal detection model under the condition that the sample reconstruction value achieves the preset fitting effect.
Further, the anomaly detection model includes an encoding layer and a decoding layer, and the building module 1102 is specifically configured to:
acquiring a plurality of sample industrial data subsequences from the sample industrial data sequence by utilizing a pre-constructed sliding window; inputting a plurality of sample industrial data subsequences into an encoding layer to obtain a sample hidden variable corresponding to each sample industrial data subsequence; inputting all sample hidden variables into a decoding layer to obtain a sample sub-reconstruction value corresponding to each sample industrial data subsequence; and taking all the sample sub-reconstruction values as sample reconstruction values.
Further, the building module 1102 is further specifically configured to:
inputting a plurality of sample industrial data subsequences into an encoding layer to obtain a sample hidden variable mean value and a sample hidden variable variance corresponding to each sample industrial data subsequence; and splicing the mean value of each sample hidden variable and the variance of each sample hidden variable, and adding Gaussian noise to obtain each sample hidden variable.
Further, the reconstruction value includes a plurality of sub-reconstruction values, the industrial data sequence to be detected includes a plurality of industrial data sub-sequences to be detected, the plurality of sub-reconstruction values and the plurality of industrial data sub-sequences to be detected are in one-to-one correspondence based on the time sequence characteristics, and the detection module 1103 is specifically configured to:
under the condition that the target sub-reconstruction value exceeds a preset threshold value, judging that a target industrial data sub-sequence to be detected corresponding to the target sub-reconstruction value is an abnormal data sub-sequence, wherein the target sub-reconstruction value is any one of a plurality of sub-reconstruction values; repeating the steps until all abnormal data subsequences in the industrial data to be detected are determined; and determining an abnormal data segment from the industrial data to be detected according to all the abnormal data subsequences based on the time sequence characteristics.
Further, the obtaining module 1101 is specifically configured to:
preprocessing original industrial data to remove a white noise wave band in the original industrial data; and carrying out standardization processing on the preprocessed original industrial data to obtain the industrial data to be detected.
The embodiment of the present application provides a computer device 100, where the computer device 100 includes a processor and a non-volatile memory storing computer instructions, and when the computer instructions are executed by the processor, the computer device 100 executes the foregoing data anomaly detection method. As shown in fig. 4, fig. 4 is a block diagram of a computer device 100 according to an embodiment of the present disclosure. The computer apparatus 100 includes a data abnormality detection device 110, a memory 111, a processor 112, and a communication unit 113.
To facilitate the transfer or interaction of data, the elements of the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other, directly or indirectly. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The data anomaly detection device 110 includes at least one software function module which can be stored in the memory 111 in the form of software or firmware (firmware) or solidified in an Operating System (OS) of the computer apparatus 100. The processor 112 is used for executing executable modules stored in the memory 111, such as software functional modules and computer programs included in the data anomaly detection device 110.
The embodiment of the present application provides a readable storage medium, where the readable storage medium includes a computer program, and when the computer program runs, the computer device 100 where the readable storage medium is located is controlled to execute the foregoing data anomaly detection method.
In summary, the embodiment of the present application provides a data anomaly detection method, an apparatus, a computer device, and a readable storage medium, by acquiring industrial data to be detected, where the industrial data to be detected includes a time sequence characteristic; then, according to the time sequence characteristics, constructing and obtaining an industrial data sequence to be detected corresponding to the industrial data to be detected; then inputting the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value; and finally, determining an abnormal data section in the industrial data to be detected according to the reconstruction value, and skillfully utilizing an abnormal detection model to realize efficient industrial data abnormal detection.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A data anomaly detection method is characterized by comprising the following steps:
acquiring industrial data to be detected, wherein the industrial data to be detected comprises time sequence characteristics;
constructing and obtaining a to-be-detected industrial data sequence corresponding to the to-be-detected industrial data according to the time sequence characteristics;
inputting the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value;
and determining an abnormal data segment in the industrial data to be detected according to the reconstruction value.
2. The method of claim 1, wherein the anomaly detection model is trained by:
obtaining sample industrial data, wherein the sample industrial data comprises the timing feature;
constructing and obtaining a sample industrial data sequence corresponding to the sample industrial data according to the time sequence characteristics;
inputting the sample industrial data sequence into the pre-constructed anomaly detection model to obtain a sample reconstruction value;
and training the abnormal detection model according to the sample reconstruction value, and obtaining the trained abnormal detection model under the condition that the sample reconstruction value achieves a preset fitting effect.
3. The method of claim 2, wherein the anomaly detection model comprises an encoding layer and a decoding layer;
the step of inputting the sample industrial data sequence into the pre-constructed anomaly detection model to obtain a sample reconstruction value includes:
acquiring a plurality of sample industrial data subsequences from the sample industrial data sequence by utilizing a pre-constructed sliding window;
inputting the plurality of sample industrial data subsequences into the coding layer to obtain a sample hidden variable corresponding to each sample industrial data subsequence;
inputting all the sample hidden variables into the decoding layer to obtain a sample sub-reconstruction value corresponding to each sample industrial data subsequence;
taking all of the sample sub-reconstruction values as the sample reconstruction values.
4. The method according to claim 3, wherein the step of inputting the plurality of sample industrial data subsequences into the coding layer to obtain a sample hidden variable corresponding to each sample industrial data subsequence comprises:
inputting the plurality of sample industrial data subsequences into the coding layer to obtain a sample implicit variable mean value and a sample implicit variable variance corresponding to each sample industrial data subsequences;
and splicing each sample hidden variable mean value and each sample hidden variable variance, and adding Gaussian noise to obtain each sample hidden variable.
5. The method of claim 1, wherein the reconstruction values comprise a plurality of sub-reconstruction values, the industrial data sequence to be tested comprises a plurality of industrial data sub-sequences to be tested, and the plurality of sub-reconstruction values and the plurality of industrial data sub-sequences to be tested are in one-to-one correspondence based on the time sequence characteristics;
the step of determining the abnormal data segment in the industrial data to be detected according to the reconstruction value comprises the following steps:
under the condition that the target sub-reconstruction value exceeds a preset threshold value, judging that a target industrial data sub-sequence to be detected corresponding to the target sub-reconstruction value is an abnormal data sub-sequence, wherein the target sub-reconstruction value is any one of the plurality of sub-reconstruction values;
repeating the steps until all the abnormal data subsequences in the industrial data to be detected are determined;
and determining the abnormal data segment from the industrial data to be detected according to all the abnormal data subsequences based on the time sequence characteristics.
6. The method of claim 1, wherein the step of obtaining industrial data to be measured comprises:
preprocessing original industrial data to remove a white noise wave band in the original industrial data;
and carrying out standardization processing on the preprocessed original industrial data to obtain the industrial data to be detected.
7. A data abnormality detection apparatus, characterized by comprising:
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring industrial data to be detected, and the industrial data to be detected comprises time sequence characteristics;
the construction module is used for constructing and obtaining a to-be-detected industrial data sequence corresponding to the to-be-detected industrial data according to the time sequence characteristics;
the detection module is used for inputting the industrial data sequence to be detected into a pre-trained anomaly detection model to obtain a reconstruction value; and determining an abnormal data segment in the industrial data to be detected according to the reconstruction value.
8. The apparatus of claim 7, wherein the build module is further configured to:
obtaining sample industrial data, wherein the sample industrial data comprises the timing feature; constructing and obtaining a sample industrial data sequence corresponding to the sample industrial data according to the time sequence characteristics; inputting the sample industrial data sequence into the pre-constructed anomaly detection model to obtain a sample reconstruction value; and training the abnormal detection model according to the sample reconstruction value, and obtaining the trained abnormal detection model under the condition that the sample reconstruction value achieves a preset fitting effect.
9. A computer device comprising a processor and a non-volatile memory having computer instructions stored thereon, wherein the computer instructions, when executed by the processor, cause the computer device to perform the data anomaly detection method of any one of claims 1-6.
10. A readable storage medium, characterized in that the readable storage medium comprises a computer program, and the computer program controls a computer device in which the readable storage medium is located to execute the data anomaly detection method according to any one of claims 1-6 when running.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011451616.2A CN112506996A (en) | 2020-12-10 | 2020-12-10 | Data anomaly detection method and device, computer equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011451616.2A CN112506996A (en) | 2020-12-10 | 2020-12-10 | Data anomaly detection method and device, computer equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112506996A true CN112506996A (en) | 2021-03-16 |
Family
ID=74971952
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011451616.2A Withdrawn CN112506996A (en) | 2020-12-10 | 2020-12-10 | Data anomaly detection method and device, computer equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112506996A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113420070A (en) * | 2021-06-24 | 2021-09-21 | 平安国际智慧城市科技股份有限公司 | Pollution discharge monitoring data processing method and device, electronic equipment and storage medium |
| CN113505344A (en) * | 2021-07-16 | 2021-10-15 | 长鑫存储技术有限公司 | Anomaly detection method, repair method and anomaly detection system for machine slot |
| CN115834174A (en) * | 2022-11-15 | 2023-03-21 | 北京天融信网络安全技术有限公司 | Network security situation prediction method and device based on timing diagram neural network |
-
2020
- 2020-12-10 CN CN202011451616.2A patent/CN112506996A/en not_active Withdrawn
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113420070A (en) * | 2021-06-24 | 2021-09-21 | 平安国际智慧城市科技股份有限公司 | Pollution discharge monitoring data processing method and device, electronic equipment and storage medium |
| CN113420070B (en) * | 2021-06-24 | 2023-06-30 | 平安国际智慧城市科技股份有限公司 | Pollution discharge monitoring data processing method and device, electronic equipment and storage medium |
| CN113505344A (en) * | 2021-07-16 | 2021-10-15 | 长鑫存储技术有限公司 | Anomaly detection method, repair method and anomaly detection system for machine slot |
| CN113505344B (en) * | 2021-07-16 | 2023-08-29 | 长鑫存储技术有限公司 | Abnormality detection method, repair method and abnormality detection system for machine slot |
| CN115834174A (en) * | 2022-11-15 | 2023-03-21 | 北京天融信网络安全技术有限公司 | Network security situation prediction method and device based on timing diagram neural network |
| CN115834174B (en) * | 2022-11-15 | 2023-06-09 | 北京天融信网络安全技术有限公司 | Network security situation prediction method and device based on time sequence diagram neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112506996A (en) | Data anomaly detection method and device, computer equipment and readable storage medium | |
| US10743819B2 (en) | System and method for determining information and outliers from sensor data | |
| CN110148285B (en) | Intelligent oil well parameter early warning system based on big data technology and early warning method thereof | |
| US10228994B2 (en) | Information processing system, information processing method, and program | |
| US20200143292A1 (en) | Signature enhancement for deviation measurement-based classification of a detected anomaly in an industrial asset | |
| US20160202228A1 (en) | Water quality monitoring and early event detection | |
| CN107942956B (en) | Information processing device, information processing method, information processing program, and recording medium | |
| US10810508B1 (en) | Methods and apparatus for classifying and discovering historical and future operational states based on Boolean and numerical sensor data | |
| CN113518011A (en) | Abnormality detection method and apparatus, electronic device, and computer-readable storage medium | |
| CN113469300B (en) | Equipment state detection method and related device | |
| US20220004163A1 (en) | Apparatus for predicting equipment damage | |
| CN116204770B (en) | Training method and device for detecting abnormality of bridge health monitoring data | |
| US20220011760A1 (en) | Model fidelity monitoring and regeneration for manufacturing process decision support | |
| CN115903741A (en) | Data anomaly detection method for industrial control system | |
| CN115470717A (en) | Method, device, equipment and storage medium for predicting remaining life of robot | |
| US11715284B2 (en) | Anomaly detection apparatus, anomaly detection method, and program | |
| CN113898334A (en) | Intelligent parameter analysis method and system for multifunctional comprehensive tester of pumping well | |
| US20230281186A1 (en) | Explainable anomaly detection for categorical sensor data | |
| KR20210142864A (en) | Apparatus and method for recognizing number of measuring intrument | |
| CN114445917A (en) | Network training method and system for face living body recognition and electronic equipment | |
| CN111883226A (en) | Information processing and model training method, device, equipment and storage medium | |
| CN114255373B (en) | Sequence anomaly detection method, device, electronic equipment and readable medium | |
| CN116543538B (en) | Internet of things fire-fighting electrical early warning method and early warning system | |
| CN116381429B (en) | Method and system for correcting online partial discharge detection result | |
| CN115309736B (en) | Time sequence data anomaly detection method based on self-supervision learning multi-head attention network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 303-309, No.3, Pazhou Avenue East Road, Haizhu District, Guangzhou City, Guangdong Province 510000 Applicant after: Shugen Internet Co.,Ltd. Address before: Unit 12-30, 4th floor, Xigang office building, Guangzhou international media port, 218 and 220 Yuejiang West Road, Haizhu District, Guangzhou City, Guangdong Province 510000 Applicant before: IROOTECH TECHNOLOGY Co.,Ltd. |
|
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210316 |