CN114925731B - Method for detecting abnormal value of monitoring data of flexible inclinometer - Google Patents
Method for detecting abnormal value of monitoring data of flexible inclinometer Download PDFInfo
- Publication number
- CN114925731B CN114925731B CN202210628562.5A CN202210628562A CN114925731B CN 114925731 B CN114925731 B CN 114925731B CN 202210628562 A CN202210628562 A CN 202210628562A CN 114925731 B CN114925731 B CN 114925731B
- Authority
- CN
- China
- Prior art keywords
- data
- sample data
- data set
- mid
- training sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 68
- 238000012544 monitoring process Methods 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000012549 training Methods 0.000 claims abstract description 54
- 238000012360 testing method Methods 0.000 claims abstract description 45
- 238000001514 detection method Methods 0.000 claims abstract description 25
- 238000005070 sampling Methods 0.000 claims abstract description 12
- 238000013139 quantization Methods 0.000 claims abstract description 9
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 8
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000010606 normalization Methods 0.000 claims abstract description 4
- 239000013256 coordination polymer Substances 0.000 claims description 24
- 238000000926 separation method Methods 0.000 claims description 10
- 238000010276 construction Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 125000004122 cyclic group Chemical group 0.000 claims description 2
- 238000011897 real-time detection Methods 0.000 abstract description 2
- 230000001133 acceleration Effects 0.000 description 8
- 238000006073 displacement reaction Methods 0.000 description 7
- 230000005484 gravity Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/30—Assessment of water resources
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Testing Or Calibration Of Command Recording Devices (AREA)
Abstract
The invention provides a method for detecting abnormal values of monitoring data of a flexible inclinometer, which comprises the following steps: s1, acquiring time sequence data monitored during the operation of a flexible inclinometer, carrying out normalization processing on the time sequence data, and setting sampling frequency for sampling; establishing a training sample data set and a test sample data set; s2, based on the training sample data set A, searching a cutting point C by calculating a quantization index of the attribute value of the cutting point after the training sample data set A is cut; s3, constructing a flexible inclinometer monitoring data anomaly detection model based on an isolated forest algorithm; s4, substituting the test sample data set B into the constructed flexible inclinometer monitoring data anomaly detection model to obtain anomaly scores of each time sequence data to be detected; and judging normal sample data and abnormal sample data in the time sequence data to be detected by setting an abnormal score threshold value, and marking. The invention can realize real-time detection and abnormal value identification of the monitoring data of the flexible inclinometer, and has high detection efficiency and high detection precision.
Description
Technical Field
The invention relates to a method for detecting abnormal values of monitoring data of a flexible inclinometer, and belongs to the technical field of displacement monitoring and data processing.
Background
The flexible inclinometer is used as a high-precision and automatic monitoring instrument and is widely applied to the scenes of concrete dam monitoring, slope monitoring, foundation settlement monitoring and the like. The flexible inclinometer is a static inclination measuring instrument combining quasi-distributed and pure accelerometers, and comprises a strip or column measuring and controlling unit, wherein embedded acceleration sensors and a processor are distributed in the strip or column measuring and controlling unit, the acceleration sensors sense three-axis gravity acceleration components of the measuring and controlling unit, the acceleration sensors are connected to the processor through data wires, and the processor transmits the measured three-axis gravity acceleration components and displacement values to a monitoring system in a wired or wireless mode.
When the flexible inclinometer is used for monitoring displacement and triaxial gravity acceleration components, the flexible inclinometer has high precision and low anti-interference capability on complex field environments, so that a large amount of data deviating from real acceleration and displacement values often appear during real-time monitoring, and even blank values sometimes appear. In order to prevent abnormal data from interfering with the measurement of the monitoring object by the flexible inclinometer and realize the accurate measurement of the monitoring object by the flexible inclinometer and the judgment of the real property of the monitoring object, the data monitored by the flexible inclinometer needs to be monitored in real time, and the abnormal data is removed.
The traditional method for detecting the abnormal value of the monitoring data of the flexible inclinometer comprises the following steps: and detecting abnormal value methods based on statistics, proximity, density and clustering. Because the acceleration value or displacement value monitored by the flexible inclinometer has large data volume, various data and complex abnormal data types, the traditional method for detecting the abnormal value of the data monitored by the flexible inclinometer still has the following problems: 1. the efficiency of detecting the abnormal value of the monitoring data is low, and the occupancy rate of system resources is high. The traditional method for detecting the abnormal value of the data is single in processing object, when the processed monitoring data volume is large, the abnormal value cannot be effectively identified, the problem of processing or insufficient processing can occur, the efficiency of detecting the abnormal value of the monitoring data is low, and the system resource occupancy rate is high. 2. The detection accuracy is low, and global errors are easy to generate. Because the selection of the cutting points used in the traditional method for detecting the abnormal value of the data monitored by the flexible inclinometer is random, the local optimal solution is easy to generate, global errors are caused, and the fluctuation range of abnormal value detection is enlarged for long-period detection.
Disclosure of Invention
In view of the foregoing, an object of the present invention is to provide a method for detecting abnormal values of monitoring data of a flexible inclinometer. The method has high detection efficiency and high detection precision on the abnormal value of the monitoring data of the flexible inclinometer, can realize real-time detection and identification of the abnormal value of the monitoring data of the flexible inclinometer, and improves the accuracy of the monitoring data.
In order to achieve the above purpose, the present invention adopts the following technical methods: a method for detecting abnormal values of monitoring data of a flexible inclinometer, comprising the steps of:
S1, acquiring time sequence data monitored during operation of a flexible inclinometer; normalizing the acquired time sequence data to be detected; setting a sampling frequency based on the demand frequency, and sampling the time sequence data to be detected after normalization processing; establishing a training sample data set and a test sample data set, and renumbering the two data sets respectively to obtain a renumbered training sample data set A and a renumbered test sample data set B;
S2, cutting the training sample data set A at a median x mid for the first time based on the training sample data set A, calculating a cutting point attribute value quantization index of the cut training sample data set, and searching a cutting point C;
S3, constructing a flexible inclinometer monitoring data anomaly detection model based on an isolated forest algorithm (iForest), and introducing a training sample data set A and a cutting point C into the constructed flexible inclinometer monitoring data anomaly detection model to complete the construction of the model;
S4, substituting the test sample data set B into the constructed flexible inclinometer monitoring data anomaly detection model to obtain anomaly scores of each time sequence data to be detected; and judging normal sample data and abnormal sample data in the time sequence data to be detected by setting an abnormal fraction threshold value, and marking the detected abnormal data.
Drawings
FIG. 1 is a flow chart of a method of detecting anomaly values of monitoring data of a flexible inclinometer in accordance with the present invention;
FIG. 2 is a flow chart of a method of selecting a cutting point C in the present invention;
FIG. 3 is a flow chart of an improved isolated forest algorithm of the present invention;
FIG. 4 is a graph showing time-series data of displacement monitored by a flexible inclinometer to be tested in an embodiment of the present invention;
FIG. 5 is a graph showing the results of anomaly detection of data monitored by a flexible inclinometer in an embodiment of the present invention;
FIG. 6 is a graph of flexible inclinometer monitoring data after anomaly data is eliminated in an embodiment of the invention.
Detailed Description
The structure and features of the present invention will be described in detail below with reference to the accompanying drawings and examples. It should be noted that various modifications can be made to the embodiments disclosed herein, and thus, the embodiments disclosed in the specification should not be taken as limiting the invention, but merely as exemplifications of embodiments, which are intended to make the features of the invention apparent.
The invention provides a method for detecting abnormal values of monitoring data of a flexible inclinometer, which comprises the following steps:
S1, acquiring time sequence data which are monitored during the operation of a flexible inclinometer, namely time sequence data to be detected and contain abnormal values; normalizing the acquired time sequence data to be detected; setting a sampling frequency based on the required frequency, namely how many time sequence data need to be acquired within one hour, and sampling the time sequence data to be detected after normalization processing; the method comprises the steps of establishing a training sample data set and a test sample data set, and renumbering the two data sets respectively, wherein the specific method is as follows:
S1.1, referring to FIG. 4, acquiring time sequence data X (t) to be detected, which contains abnormal values, of a monitoring object through a data acquisition system of a flexible inclinometer:
X(t)＝[x(t),x(2t),……,x(nt)] (1)
Wherein: x (t) represents time sequence data to be detected, and t is a monitoring time interval; n represents the number of time series data; nt represents a time corresponding to the nth time series data; x (nt) represents time series data corresponding to the nth time series data;
s1.2, normalizing the time sequence data X (t) to be detected by adopting the following formula (2):
wherein: xz (t) is normalized time series data to be detected, mu is the mean value of the time series data to be detected, and sigma is the standard deviation of the time series data to be detected.
S1.3, setting sampling rates i and j based on a required frequency f, sampling normalized time sequence data to be detected, and establishing a training sample data set A 0 and a test sample data set B 0:
A0＝[x(t),x(ft),……,x(jft)] (3)
B0＝[x(jft),x((j+1)ft),……,x(ift)] (4)
Where f is the required frequency, j is the frequency used by the training sample data set, i is the frequency used by the test sample data set, j and i are constants, j < i < n/f.
S1.4, the training sample data set A 0 and the test sample data set B 0 are re-labeled, and the re-labeled training sample data set A and test sample data set B are obtained, wherein the expression is as follows:
A＝[x(1),x(2),……,x(l)] (5)
B＝[x(l+1),x(l+2),……,x(m)] (6)
Where l represents the length of the training sample data set A at the required frequency f and m represents the length of the test sample data set B at the required frequency f.
S2, cutting the training sample data set A at the position of a median x mid for the first time based on the training sample data set A, and searching a cutting point (Cpoint) C by calculating an attribute value quantization index of a training sample sub-data set after sample cutting.
In general, one skilled in the art selects a point between the maximum value and the minimum value of the training sample data set as the cut point C, so that the conventional method for detecting the abnormal value is easy to generate a locally optimal solution rather than a globally optimal solution. The invention searches the cutting point (Cpoint) C by calculating the attribute value quantization index of the training sample sub-data set after sample cutting, so the cutting point selected by the invention is applicable to all data, has the advantages of optimal global error and high accuracy, and can search the corresponding cutting point in each time sequence interval, thereby achieving the purpose of detecting abnormal data in real time.
Step S2 specifically includes the following steps, see fig. 2:
S2.1, time sequence data in a training sample data set A are called, and the data size is jf;
S2.2, setting initial positions of the extracted training sample data sets x left and x right, x left＝x(1),xright =x (l), and confirming the median x mid of the extracted training sample data sets x left to x right, and calculating a cut point attribute index CP (a, x mid), wherein the expression of the cut point attribute index CP (a, x mid) is as follows:
Wherein A is a training sample data set, x mid is the median of the training sample data set, EC (A, x mid) is the mean quantization index of the training sample data set, DC (A, x mid) is the discrete quantization index of the training sample data set, and the expression is as follows:
EC(A,xmid)＝|E(x|x∈A,x<xmid)-E(x|x∈A,x>xmid)| (8)
DC(A,xmid)＝D(x|x∈A,x<xmid)+D(x|x∈A,x>xmid) (9)
where E (x) is the mean function and D (x) is the variance function.
Calculating median x left 'of x (1) =x left to x mid in the training sample dataset, median x right' of x mid to x (l) =x right in the training sample dataset, and calculating cut point attribute indices CP (a, x left ') and CP (a, x right');
S2.4: comparing the sizes of CP (a, x left ') and CP (a, x right');
Let x right′＝xmid if CP (a, x left′)>CP(A,xright'), otherwise let x left′＝xmid;
S2.5: repeating S2.3-S2.4 until one of the following conditions is satisfied:
x left′＝xright', or
CP(A,xleft′)＜CP(A,xmid)and CP(A,xright′)＜CP(A,xmid)
The output cut point c=x mid;
S3, constructing a flexible inclinometer monitoring data anomaly detection model based on an isolated forest algorithm (iForest), and introducing the training sample data set A and the cutting point C into the constructed flexible inclinometer monitoring data anomaly detection model to complete the construction of the model.
The specific method is as follows, see fig. 3:
S3.1: randomly selecting m points from the training sample data set A as a subsampled set A 1;
S3.2: searching for a cut point C 1 in the subsampled set A 1 according to the method of step S2;
S3.3: the selection of the cut point C 1 cuts the current sub-sample set a 1 into 2 sub-sample sets: placing a point smaller than C 1 under the current selected dimension on the left branch of the current node, and placing a point larger than or equal to C 1 on the right branch of the current node;
S3.4: repeating steps S3.2 and S3.3 on the left branch and the right branch of the sub-sample set to construct a new sub-sample set until only one leaf data x i of the separation tree exists in the final sub-sample set;
s3.5: repeating the steps S3.1-S3.4 to obtain a plurality of separation trees of the training sample data set A, namely a monitoring data anomaly detection model.
S4, substituting the test sample data set B into the constructed flexible inclinometer monitoring data anomaly detection model to obtain anomaly scores of each time sequence data to be detected; and judging normal sample data and abnormal sample data in the time sequence data to be detected by setting an abnormal fraction threshold value, and marking the detected abnormal data.
The traditional method for detecting the abnormal value of the monitoring data of the flexible inclinometer judges the abnormal data through the Laida criterion, the weight of the abnormal score threshold adopted by the method can be designed, the threshold is set according to the actual requirement, and the abnormal value in the monitoring data is detected, so that the detection accuracy is higher.
The specific method comprises the following steps:
S4.1: the test sample data set B is called, each test sample data is traversed through each separation tree obtained in the step S3, the height of each test sample data on each separation tree, namely the path length H i, is finally obtained, and the abnormal score S of the monitoring time sequence data of the flexible inclinometer is obtained through calculation according to the path length;
Wherein the anomaly score S expression is as follows:
wherein: e (h) is the average of the path lengths of the test sample data in all the separation trees, expressed as follows:
Wherein: i represents the ith separate tree; hi represents the distance of the test sample data from the ith separation tree;
c (m) is the average path length of the isolated forest of test sample data, and the expression is as follows:
s4.2: solving an anomaly score threshold t for judging anomaly data;
S4.2.1: setting an initial threshold t 0 =0.4 and an initial inter-class variance sigma m =0, wherein the test sample data with the anomaly score S being greater than t 0 is classified as the anomaly sample data and is classified as the anomaly data sample set D F, and the test sample data with the anomaly score S being less than or equal to t 0 is classified as the normal sample data and is classified as the normal data sample set D R;
s4.2.2: calculating average value of integrated anomaly score of anomaly data and normal data And/>The expression is as follows:
Wherein D F is the data volume of the abnormal sample, and D R is the data volume of the normal sample; si is the i-th test sample data anomaly score.
S4.2.3: determining an anomaly score average for test sample dataset BThe expression is as follows:
s4.2.4: calculating an inter-class variance sigma of the test sample data set B, wherein the expression is as follows;
S4.2.5: the cyclic threshold t=0.40-0.70 retains two decimal places), and the inter-class variance σ is calculated according to S4.2.1-S4.2.4, if σ > σ m, then t 0 =t, otherwise, t 0 =t+0.01.
S4.2.6: and outputting an anomaly score threshold t 0 of the final judgment anomaly value data.
S4.3, judging normal sample data and abnormal sample data in the time sequence data to be detected by setting an abnormal score threshold value, and marking the detected abnormal data.
Fig. 5 is a graph showing the result of abnormality detection on the data monitored by the flexible inclinometer in the embodiment of the present invention. When the anomaly score S of the test sample data is greater than t 0, the test sample data is the anomaly sample data; when the anomaly score S is less than or equal to t 0, the test sample data is normal sample data. In fig. 5 x is the abnormal points of the monitor data of the flexible inclinometer detected by the present invention, and the normal data indicates the apparent points of the majority of the time series data. FIG. 6 is a graph of monitoring data from a flexible inclinometer after exception data is removed in an embodiment of the present invention.
Compared with the prior art, the invention has the advantages that:
1. Because the invention adopts a statistical method and a cutting point identification method to improve an isolated forest algorithm, the invention has better abnormality identification capability, can effectively identify abnormal values in the monitored data when the processed monitored data volume is larger, and can effectively distinguish abnormal values influenced by noise from abnormal values of actual response of a monitored object.
2. According to the invention, by improving an isolated forest algorithm, the optimal cutting point C and the abnormal score threshold t 0 are selected, so that the situation of local optimal solution is avoided, and the error of long-period detection is greatly reduced.
3. According to the method, the characteristics of time sequence data are extracted, and the optimal cutting point C is selected to identify abnormal value dividing limits; constructing an anomaly detection model through an isolated forest algorithm; judging an abnormal result through an abnormal score threshold; the monitoring data abnormal value of the flexible inclinometer can be accurately detected in real time.
4. The invention has small error and can better detect most abnormal values in the data. The invention has a wide application range and can be used on different displacement monitoring devices.
Finally, it should be noted that: the embodiments described above are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced with equivalents; such modifications and substitutions do not depart from the spirit of the invention.
Claims (3)
1. A method for detecting abnormal values of monitoring data of a flexible inclinometer, comprising the steps of:
S1, acquiring time sequence data monitored during operation of a flexible inclinometer; normalizing the acquired time sequence data to be detected; setting a sampling frequency based on the demand frequency, and sampling the time sequence data to be detected after normalization processing; establishing a training sample data set and a test sample data set, and renumbering the two data sets respectively to obtain a renumbered training sample data set A and a renumbered test sample data set B;
s2, cutting the training sample data set A at a median x mid for the first time based on the training sample data set A, calculating a cutting point attribute value quantization index of the cut training sample data set, and searching a cutting point C; the specific method comprises the following steps:
S2.1, time sequence data in a training sample data set A are called, and the data size is jf;
S2.2, setting initial positions of the called training sample data sets x left and x right, confirming median x mid of the called training sample data sets x left to x right, and calculating a cutting point attribute index CP (A, x mid), wherein the expression of the cutting point attribute index CP (A, x mid) is as follows:
Wherein A is a training sample data set, x mid is the median of the training sample data set, EC (A, x mid) is the mean quantization index of the training sample data set, DC (A, x mid) is the discrete quantization index of the training sample data set, and the expression is as follows:
EC(A，xmid)＝|E(x|x∈A,x＜xmid)-E(x|x∈A,x＞xmid)|
DC(A，xmid)＝D(x|x∈A,x＜xmid)+D(x|x∈A,x＞xmid)
Wherein E (x) is a mean function and D (x) is a variance function;
S2.3, calculating median x left 'of x (1) =x left to x mid in the training sample data set, median x right' of x mid to x (l) =x right in the training sample data set, and calculating cut point attribute indexes CP (a, x left ') and CP (a, x right');
S2.4, comparing the sizes of the CP (A, x left ') and the CP (A, x right');
Let x right′＝xmid if CP (a, x left′)＞CP(A,xright'), otherwise let x left′＝xmid;
S2.5, repeating S2.3-S2.4 until one of the following conditions is met:
x left′＝xright', or
CP(A，xleft′)＜CP(A,xmid)and CP(A,xright′)＜CP(A,xmid)
The output cut point c=x mid;
s3, constructing a flexible inclinometer monitoring data anomaly detection model based on an isolated forest algorithm (iForest), and introducing a training sample data set A and a cutting point C into the constructed flexible inclinometer monitoring data anomaly detection model to complete the construction of the model; the specific method comprises the following steps:
S3.1, randomly selecting m points from the training sample data set A to serve as a subsampled set A 1;
S3.2, searching a cutting point C 1 in the subsampled set A 1 according to the method of the step S2;
S3.3, the current sub-sample set A 1 is segmented into 2 sub-sample sets by selecting the cutting point C 1: placing a point smaller than C 1 under the current selected dimension on the left branch of the current node, and placing a point larger than or equal to C 1 on the right branch of the current node;
S3.4, repeating the steps S3.2 and S3.3 on the left branch and the right branch of the sub-sample set, and constructing a new sub-sample set until only one leaf data x i of the separation tree exists in the final sub-sample set;
S3.5, repeating the steps S3.1-S3.4 to obtain a plurality of separation trees of the training sample data set A, namely a monitoring data anomaly detection model;
S4, substituting the test sample data set B into the constructed flexible inclinometer monitoring data anomaly detection model to obtain anomaly scores of each time sequence data to be detected; and judging normal sample data and abnormal sample data in the time sequence data to be detected by setting an abnormal fraction threshold value, and marking the detected abnormal data.
2. The method for detecting abnormal values of monitoring data of a flexible inclinometer according to claim 1, wherein: the step S4 includes the steps of:
S4.1, a test sample data set B is called, each test sample data is traversed through each separating tree obtained in the step S3, the height of each test sample data on each separating tree, namely the path length H i, is finally obtained, and the abnormal score S of the monitoring time sequence data of the flexible inclinometer is obtained through calculation according to the path length;
Wherein the anomaly score S expression is as follows:
wherein: e (h) is the average of the path lengths of the test sample data in all the separation trees, expressed as follows:
Wherein: i represents the ith separate tree; hi represents the distance of the test sample data from the ith separation tree;
c (m) is the average path length of the isolated forest of test sample data, and the expression is as follows:
S4.2, solving an anomaly score threshold t for judging anomaly data;
S4.2.1, setting an initial threshold value t 0 =0.4, setting an initial inter-class variance sigma m =0, and classifying test sample data with an anomaly score S larger than t 0 as abnormal sample data into an abnormal data sample set D F, and classifying test sample data with an anomaly score S smaller than or equal to t 0 as normal sample data into a normal data sample set D R;
S4.2.2 calculating an average value of the integrated anomaly score of the anomaly data and the normal data And/>The expression is as follows:
Wherein D F is the data volume of the abnormal sample, and D R is the data volume of the normal sample; s i is the abnormal score of the ith test sample data;
s4.2.3 determining an outlier score average for test sample dataset B The expression is as follows:
s4.2.4, calculating an inter-class variance sigma of the test sample dataset B, wherein the expression is as follows;
s4.2.5, the cyclic threshold t=0.40-0.70 is reserved with two decimal places, the inter-class variance sigma is calculated according to S4.2.1-S4.2.4, if sigma is larger than sigma m, t 0 =t, otherwise, t 0 =t+0.01;
S4.2.6, outputting an anomaly score threshold t 0 for finally judging anomaly value data;
s4.3, judging normal sample data and abnormal sample data in the time sequence data to be detected by setting an abnormal score threshold value, and marking the detected abnormal data.
3. The method for detecting abnormal values of monitoring data of a flexible inclinometer according to claim 2, wherein: the step S1 includes the steps of:
S1.1, acquiring time sequence data X (t) to be detected, which contains abnormal values, of a monitoring object through a data acquisition system of a flexible inclinometer:
X(t)＝[x(t)，x(2t)，……，x(nt)]
Wherein: x (t) represents time sequence data to be detected, and t is a monitoring time interval; n represents the number of time series data; nt represents a time corresponding to the nth time series data; x (nt) represents time series data corresponding to the nth time series data;
s1.2, normalizing the time sequence data X (t) to be detected by adopting the following formula (2):
Wherein: xz (t) is normalized time sequence data to be detected, mu is the mean value of the time sequence data to be detected, and sigma is the standard deviation of the time sequence data to be detected;
S1.3, setting sampling rates i and j based on a required frequency f, sampling normalized time sequence data to be detected, and establishing a training sample data set A 0 and a test sample data set B 0:
A0＝[x(t)，x(ft)，……，x(jft)]
B0＝[x(jft)，x((j+1)ft)，……，x(ift)]
Wherein f is the required frequency, j is the frequency adopted by the training sample data set, i is the frequency adopted by the test sample data set, j and i are constants, and j is less than i and less than n/f;
S1.4, the training sample data set A 0 and the test sample data set B 0 are re-labeled, and the re-labeled training sample data set A and test sample data set B are obtained, wherein the expression is as follows:
A＝[x(1)，x(2)，……，x(l)]
B＝[x(l+1)，x(l+2)，……，x(m)]
where l represents the length of the training sample data set A at the required frequency f and m represents the length of the test sample data set B at the required frequency f.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210628562.5A CN114925731B (en) | 2022-06-06 | 2022-06-06 | Method for detecting abnormal value of monitoring data of flexible inclinometer |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210628562.5A CN114925731B (en) | 2022-06-06 | 2022-06-06 | Method for detecting abnormal value of monitoring data of flexible inclinometer |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114925731A CN114925731A (en) | 2022-08-19 |
| CN114925731B true CN114925731B (en) | 2024-05-31 |
Family
ID=82811671
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210628562.5A Active CN114925731B (en) | 2022-06-06 | 2022-06-06 | Method for detecting abnormal value of monitoring data of flexible inclinometer |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114925731B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117970168B (en) * | 2024-03-29 | 2024-05-28 | 国网山东省电力公司莱芜供电公司 | High-efficiency processing method for monitoring data of dual-power conversion device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108777873A (en) * | 2018-06-04 | 2018-11-09 | 江南大学 | The wireless sensor network abnormal deviation data examination method of forest is isolated based on weighted blend |
| WO2020155752A1 (en) * | 2019-01-28 | 2020-08-06 | 平安科技（深圳）有限公司 | Outlier detection model verification method and apparatus, and computer device and storage medium |
| CN111666169A (en) * | 2020-05-13 | 2020-09-15 | 云南电网有限责任公司信息中心 | Improved isolated forest algorithm and Gaussian distribution-based combined data anomaly detection method |
| KR20210031094A (en) * | 2019-09-11 | 2021-03-19 | 한국전력공사 | Tree-based outlier detection apparutus and method, computer program |
| CN114493023A (en) * | 2022-02-07 | 2022-05-13 | 中国水利水电科学研究院 | Diagnosis and interpolation method in abnormal water regime data based on RF-Adaboost model |
-
2022
- 2022-06-06 CN CN202210628562.5A patent/CN114925731B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108777873A (en) * | 2018-06-04 | 2018-11-09 | 江南大学 | The wireless sensor network abnormal deviation data examination method of forest is isolated based on weighted blend |
| WO2019233189A1 (en) * | 2018-06-04 | 2019-12-12 | 江南大学 | Method for detecting sensor network abnormal data |
| WO2020155752A1 (en) * | 2019-01-28 | 2020-08-06 | 平安科技（深圳）有限公司 | Outlier detection model verification method and apparatus, and computer device and storage medium |
| KR20210031094A (en) * | 2019-09-11 | 2021-03-19 | 한국전력공사 | Tree-based outlier detection apparutus and method, computer program |
| CN111666169A (en) * | 2020-05-13 | 2020-09-15 | 云南电网有限责任公司信息中心 | Improved isolated forest algorithm and Gaussian distribution-based combined data anomaly detection method |
| CN114493023A (en) * | 2022-02-07 | 2022-05-13 | 中国水利水电科学研究院 | Diagnosis and interpolation method in abnormal water regime data based on RF-Adaboost model |
Non-Patent Citations (3)
| Title |
|---|
| A Survey of outlier detection methodologies;VICTORIA J H等;《Artificial intelligence review：An international Scienence and Engineering Journal》;20041231;第22卷(第2期);85-126 * |
| 基于孤立森林算法的取用水量异常数据检测方法;赵臣啸;薛惠锋;王磊;万毅;;中国水利水电科学研究院学报;20200215(第01期);全文 * |
| 立森林算法在大坝监测数据异常识别中的应用;张海龙等;《人民黄河》;20201231;第42卷(第08期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114925731A (en) | 2022-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103473540B (en) | The modeling of intelligent transportation system track of vehicle increment type and online method for detecting abnormality | |
| CN117238058B (en) | Starter monitoring method for automobile based on data analysis | |
| US20210348985A1 (en) | Abnormality diagnostic device, abnormality diagnostic method, and program | |
| CN117171604B (en) | Sensor-based insulation board production line abnormality monitoring system | |
| CN114925731B (en) | Method for detecting abnormal value of monitoring data of flexible inclinometer | |
| CN117436005B (en) | Abnormal data processing method in automatic ambient air monitoring process | |
| CN117289778B (en) | Real-time monitoring method for health state of industrial control host power supply | |
| CN117573667B (en) | Optimized cleaning method and system for coal mine inter-well data | |
| CN112949735A (en) | Liquid hazardous chemical substance volatile concentration abnormity discovery method based on outlier data mining | |
| CN106935038B (en) | Parking detection system and detection method | |
| CN117454283A (en) | State evaluation method for wind turbine generator operation detection data | |
| CN115809435A (en) | Simulator-based automobile operation fault identification method | |
| CN117435972A (en) | Abnormal monitoring method of electric energy metering box based on internet of things sensing | |
| CN117688498B (en) | Ship comprehensive safety state monitoring system based on ship-shore cooperation | |
| CN116304963A (en) | Data processing system suitable for geological disaster early warning | |
| CN113553319A (en) | LOF outlier detection cleaning method, device and equipment based on information entropy weighting and storage medium | |
| CN117572837A (en) | Intelligent power plant AI active operation and maintenance method and system | |
| CN117590388A (en) | Side slope radar monitoring and early warning method based on DBSCAN clustering algorithm | |
| CN117310829A (en) | Optimized identification method based on geomagnetic anomaly data | |
| CN113255771B (en) | Fault diagnosis method and system based on multi-dimensional heterogeneous difference analysis | |
| CN113687411B (en) | Earth stress azimuth prediction method based on microseism | |
| CN113487141A (en) | Bearing state evaluation method based on multi-source information clustering fusion | |
| CN112541516A (en) | Energy spectrum-based bridge static monitoring data principal component clustering method | |
| CN117854622B (en) | Construction method and system of layered rock strength calculation model | |
| US20240271990A1 (en) | Vibration measurement error determination method and vibration error discernment system using the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |