CN111121946A - Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area - Google Patents
Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area Download PDFInfo
- Publication number
- CN111121946A CN111121946A CN201911307015.1A CN201911307015A CN111121946A CN 111121946 A CN111121946 A CN 111121946A CN 201911307015 A CN201911307015 A CN 201911307015A CN 111121946 A CN111121946 A CN 111121946A
- Authority
- CN
- China
- Prior art keywords
- standard deviation
- dynamic range
- point
- sample
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01H—MEASUREMENT OF MECHANICAL VIBRATIONS OR ULTRASONIC, SONIC OR INFRASONIC WAVES
- G01H9/00—Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves by using radiation-sensitive means, e.g. optical means
- G01H9/004—Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves by using radiation-sensitive means, e.g. optical means using fibre optic sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B10/00—Transmission systems employing electromagnetic waves other than radio-waves, e.g. infrared, visible or ultraviolet light, or employing corpuscular radiation, e.g. quantum communication
- H04B10/07—Arrangements for monitoring or testing transmission systems; Arrangements for fault measurement of transmission systems
- H04B10/075—Arrangements for monitoring or testing transmission systems; Arrangements for fault measurement of transmission systems using an in-service signal
- H04B10/079—Arrangements for monitoring or testing transmission systems; Arrangements for fault measurement of transmission systems using an in-service signal using measurements of the data signal
- H04B10/0791—Fault location on the transmission path
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Electromagnetism (AREA)
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)
Abstract
The invention discloses a method for accurately determining abnormal values in a large dynamic range and a large discrete single area at multiple points. Collecting a single-area sample with a large dynamic range and a large dispersion, extracting the standard deviation and the dispersion difference of each point in the single-area sample, and carrying out threshold processing on the standard deviation and the dispersion difference; processing the standard deviation of each point in the single-area sample through standard deviation mean forward translation operation; calculating the confidence rates of all points in the single-region sample; and comparing the confidence rates of all the points with the threshold values of the confidence rates respectively, judging abnormal points in the sample, and accurately determining abnormal values by using the large dynamic range and the large discrete single-region multipoint. The invention overcomes the problems of large data volume and unobvious abnormality and can quickly and accurately determine the abnormal numerical value.
Description
Technical Field
The invention belongs to the technical field of computer data processing, and particularly relates to a method for determining abnormal values of large dynamic range and large discrete data.
Background
In computer modeling, it is important to clean up data samples to ensure that observations adequately represent problems. Sometimes, a data set may contain extreme values outside of the expected range, which are often referred to as outliers. By understanding and even removing these outliers, modeling and model skills can be improved.
Currently proposed methods of determining data outliers include distance-based methods, bias-based methods, and density-based methods, among others. Although the methods can find out abnormal values, the methods are complicated and require a large amount of machine learning, and particularly when the dynamic range is large and abnormal data are not obvious, the abnormal values are difficult to find out by the existing methods.
For example, in the distributed optical fiber vibration sensing positioning technology, the false alarm rate is a prominent problem, and when the system makes a false alarm, the vibration point measured by the system is different from the actual vibration point by hundreds of meters. Especially, when the transmission distance is long, the signal and the background noise cannot be distinguished because the back scattering signal is weak, and the false alarm of vibration positioning is caused, so a method for accurately positioning the vibration point is needed.
Disclosure of Invention
In order to solve the technical problems in the background art, the invention provides a method for accurately determining abnormal values at multiple points in a large dynamic range and a large discrete single area, which overcomes the problems of large data volume and unobvious abnormality and can quickly and accurately determine abnormal values.
In order to achieve the technical purpose, the technical scheme of the invention is as follows:
the method for accurately determining the abnormal value of the large dynamic range and the large discrete single area multipoint comprises the following steps:
(1) collecting a single-area sample with a large dynamic range and a large dispersion, extracting the standard deviation and the dispersion difference of each point in the single-area sample, and carrying out threshold processing on the standard deviation and the dispersion difference;
(2) processing the standard deviation of each point in the single-area sample through standard deviation mean forward translation operation;
(3) calculating the confidence rates of all points in the single-region sample;
(4) and comparing the confidence rates of all the points with the threshold values of the confidence rates respectively, judging abnormal points in the sample, and accurately determining abnormal values by using the large dynamic range and the large discrete single-region multipoint.
Further, in step (ii)(1) In the method, N single-region samples are collected at one time, and the average value of each point is calculatedStandard deviation sigmaiAnd a difference of dispersion dij:
Wherein x isijThe ith sample is represented as ith point value, i is 1,2, …, M is the number of sampling points, and j is 1,2, …, N.
Further, in step (1), the method of thresholding the standard deviation and the discrete deviation is as follows:
setting a threshold t1And t2;
Standard deviation sigma of the ith sample pointiSatisfy sigmai≤t1When, will σiIs set to t1;
Discrete difference d of ith sampling point of jth sampleijSatisfy dij≤t2When d is greater than dijIs set to 0.
Further, the process of step (2) is as follows:
setting T1And T2The ith-T1~i-T2The standard deviation of all the internal sampling points is weighted and averaged to be used as the standard deviation sigma of the ith sampling pointi。
Further, in step (3), the confidence rates D of the respective points are calculated as followsij:
Dij=dij-g*σi
Wherein g is a preset multiple; if D isijIf greater than 0, then orderDijIf not, let Dij=0;
Will DijStored in an array D, DijI.e. the element in the ith row and the jth column of the array D.
Further, in step (4), a confidence threshold D is set, and the sum D of each row element of the array D is calculatediIf d isiIf > d, the ith sampling point is considered as an abnormal point.
Drawings
FIG. 1 is a flow chart of an embodiment distributed optical fiber vibration sensing positioning method;
FIG. 2 is a graph of signal versus disturbance location at 10km of disturbance provided by an embodiment;
FIG. 3 is a graph of standard deviation at 10km of perturbation versus perturbation position provided by an embodiment;
FIG. 4 is a graph of the dispersion difference at 10km of perturbation versus the location of the perturbation provided by the embodiment;
FIG. 5 is a ring diagram of the criterion and disturbance position at 10km of disturbance provided by the embodiment;
FIG. 6 is a graph of confidence rate at 10km versus disturbance location for an embodiment;
FIG. 7 is a graph of location dispersion at a perturbation of 10km provided by an embodiment;
FIG. 8 is a diagram illustrating the relationship between the maximum localization dispersion and the perturbation position according to the embodiment.
Detailed Description
The technical scheme of the invention is explained in detail in the following with the accompanying drawings.
In reality, many scenes need to locate abnormal values of large data, such as the distributed optical fiber vibration sensing location technology mentioned in the aforementioned background art. In this embodiment, a method for accurately determining an abnormal value at multiple points in a large dynamic range and a large discrete single area is applied to a distributed optical fiber vibration sensing positioning technology, so as to obtain a distributed optical fiber vibration sensing positioning method, as shown in fig. 1, the steps of which are as follows:
step 1: taking N curves collected by the distributed optical fiber vibration sensing system as a group, calculating the average value, the standard deviation and the dispersion difference of M points on the N curves, and carrying out threshold processing on the standard deviation and the dispersion difference.
The following preferred scheme is adopted to realize the step 1:
average value of each pointStandard deviation sigmaiAnd a difference of dispersion dijThe calculation formula of (a) is as follows:
xijthe ith point value in the jth curve is represented, i is 1,2, …, M is the number of sampling points, and j is 1,2, …, N. N-20, M-65536.
The method for thresholding the standard deviation and the dispersion deviation is as follows:
setting a threshold t1And t2Standard deviation σ of the ith sample pointiSatisfy sigmai≤t1When, will σiIs set to t1(ii) a Discrete difference d of ith sampling point of jth sampleijSatisfy dij≤t2When d is greater than dijIs set to 0. Further, set t1=30,t2=72。
Step 2: and processing the standard deviation of each sampling point through a standard deviation mean forward translation operation.
The following preferred scheme is adopted to realize the step 2:
setting T1And T2The ith-T1~i-T2The standard deviation of all the internal sampling points is weighted and averaged to be used as the standard deviation sigma of the ith sampling pointi. Further, T is set1=1000,T2When 500, then:
and step 3: the confidence rates of all the sample points are calculated.
The following preferred scheme is adopted to realize the step 3:
the confidence rate D of each point is calculated according to the following formulaij:
Dij=dij-g*σi
Wherein g is a preset multiple; if D isijIf greater than 0, let DijIf not, let Dij=0;
Will DijStored in an array D, DijI.e. the element in the ith row and the jth column of the array D.
In distributed fiber optic vibration sensing, the preferred value of the multiple g is 2.4.
And 4, step 4: and comparing the confidence rates of all the points with threshold values of the confidence rates respectively, thereby judging the vibration points.
The following preferred scheme is adopted to realize the step 4:
setting a confidence rate threshold value D, and calculating the sum D of each row element of the array DiIf d isiIf the sampling point is greater than d, the ith sampling point is considered to be possibly disturbed, the number of all the possible disturbed points is counted to be t, and the average of all the possible disturbed points is taken as the real vibrated point. Further, d is set to 6.
To verify the present invention, the following experiment was performed.
Continuously applying disturbance at 10km, continuously acquiring 2000 curves, and performing method processing on the acquired 2000 curves, wherein fig. 2 is an acquired signal original curve diagram, and because a disturbance signal is small and a dynamic range is large, it is difficult to directly obtain a disturbance position from fig. 2. By adopting the invention, the original data of FIG. 2 is subjected to feature extraction, FIG. 3 is a standard deviation graph thereof, and the standard deviation of the disturbance position and the subsequent point can be obviously changed; FIG. 4 is a graph of the dispersion difference of 20 samples taken, from above, it can be seen that the dispersion difference of the disturbance position and the point after the disturbance position has a sudden change; FIG. 5 is a graph of 20 samples taken after each point criterion, and it can be seen from the graph that only the criterion of the disturbance point position is distinguished from other position points, and the characteristics are obvious; FIG. 6 shows the confidence rates of 20 samples taken at each point, and it can be seen that the confidence rate at the perturbed point is greater than 6, as distinguished from the un-perturbed point. Fig. 7 shows the positioning dispersion of 100 sets of data at 10km of disturbance points by using the method of the present invention, and it can be seen that the maximum dispersion is 4 and the minimum dispersion is 0.
In addition, in this embodiment, only the disturbance position is changed under the condition that the laboratory conditions are kept unchanged, and the data of 2000 curves at different positions are collected and processed, so as to obtain the maximum dispersion at 2.2km, 5km, 10km, 15km and 20 km. Fig. 8 is a relationship diagram between the positioning dispersion and the disturbance position, and it can be seen that, along with the increase of the disturbance position, the positioning dispersion is always kept within 4.8 meters, which indicates that the positioning method can achieve accurate positioning, i.e. the abnormal point can be accurately found.
The above experiment shows that the method for accurately determining the abnormal value at the large dynamic range and the large discrete single area and the multiple points is applied to the distributed optical fiber vibration sensing system for locating the disturbance point, when the optical fiber is subjected to perturbation, the signal at the disturbance position is not obviously changed, the threshold value comparison method cannot determine the vibration point, but the standard deviation, the discrete difference and the confidence rate at the position are obviously changed due to the small change of the signal, so that the abnormal value can be accurately determined through the method, and the possibility of misjudgment is reduced.
The embodiments are only for illustrating the technical idea of the present invention, and the technical idea of the present invention is not limited thereto, and any modifications made on the basis of the technical scheme according to the technical idea of the present invention fall within the scope of the present invention.
Claims (6)
1. The method for accurately determining the abnormal value of the large dynamic range and the large discrete single area multipoint is characterized by comprising the following steps of:
(1) collecting a single-area sample with a large dynamic range and a large dispersion, extracting the standard deviation and the dispersion difference of each point in the single-area sample, and carrying out threshold processing on the standard deviation and the dispersion difference;
(2) processing the standard deviation of each point in the single-area sample through standard deviation mean forward translation operation;
(3) calculating the confidence rates of all points in the single-region sample;
(4) and comparing the confidence rates of all the points with the threshold values of the confidence rates respectively, judging abnormal points in the sample, and accurately determining abnormal values by using the large dynamic range and the large discrete single-region multipoint.
2. The method for accurately determining abnormal values of large dynamic range and large discrete single-area multipoint according to claim 1, wherein in step (1), N single-area samples are collected at a time, and the average value of each point is calculatedStandard deviation sigmaiAnd a difference of dispersion dij:
Wherein x isijThe ith sample is represented as ith point value, i is 1,2, …, M is the number of sampling points, and j is 1,2, …, N.
3. The method for accurately determining the abnormal values of the large dynamic range and the large discrete single-area multipoint according to claim 2, wherein in the step (1), the method for thresholding the standard deviation and the discrete difference is as follows:
setting a threshold t1And t2;
Standard deviation of the ith sample pointσiSatisfy sigmai≤t1When, will σiIs set to t1;
Discrete difference d of ith sampling point of jth sampleijSatisfy dij≤t2When d is greater than dijIs set to 0.
4. The method for accurately determining the abnormal value of the large dynamic range and the large discrete single-area multipoint according to claim 2, wherein the process of the step (2) is as follows:
setting T1And T2The ith-T1~i-T2The standard deviation of all the internal sampling points is weighted and averaged to be used as the standard deviation sigma of the ith sampling pointi。
5. The method for accurately determining outliers of large dynamic range and large discrete single-area multi-point as claimed in claim 2 wherein in step (3), the confidence rates D of the points are calculated according to the following formulaij:
Dij=dij-g*σi
Wherein g is a preset multiple; if D isijIf greater than 0, let DijIf not, let Dij=0;
Will DijStored in an array D, DijI.e. the element in the ith row and the jth column of the array D.
6. The method for accurately determining abnormal values at large dynamic range and large discrete single-area and multi-point as claimed in claim 5, wherein in step (4), a confidence threshold D is set, and the sum D of each row of elements in the array D is calculatediIf d isiIf > d, the ith sampling point is considered as an abnormal point.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911307015.1A CN111121946B (en) | 2019-12-18 | 2019-12-18 | Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911307015.1A CN111121946B (en) | 2019-12-18 | 2019-12-18 | Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111121946A true CN111121946A (en) | 2020-05-08 |
CN111121946B CN111121946B (en) | 2021-09-28 |
Family
ID=70499428
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911307015.1A Active CN111121946B (en) | 2019-12-18 | 2019-12-18 | Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111121946B (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111735528A (en) * | 2020-06-05 | 2020-10-02 | 南京曦光信息科技有限公司 | KNN (K nearest neighbor) voting point accurate positioning method |
CN115112866A (en) * | 2022-08-29 | 2022-09-27 | 山东省鲁南地质工程勘察院(山东省地质矿产勘查开发局第二地质大队) | Hydrothermal deposit investigation system based on electromagnetic radiation |
Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5689067A (en) * | 1995-02-03 | 1997-11-18 | Daimler-Benz Ag | Diagnostic method and apparatus for monitoring the wear of at least an engine timing chain |
US6131013A (en) * | 1998-01-30 | 2000-10-10 | Motorola, Inc. | Method and apparatus for performing targeted interference suppression |
CN102288286A (en) * | 2011-06-16 | 2011-12-21 | 中国科学院沈阳自动化研究所 | Method for analyzing and evaluating measure point precision of gearbox in vibration acceleration sensor |
CN104217513A (en) * | 2014-09-02 | 2014-12-17 | 浙江中欣动力测控技术有限公司 | Method for improving intrusion event identification accuracy based on phase-sensitive optical time domain reflectrometer (phi-OTDR) |
CN104240455A (en) * | 2014-08-07 | 2014-12-24 | 北京航天控制仪器研究所 | Method for identifying disturbance event in distributed type optical fiber pipeline security early-warning system |
KR101696789B1 (en) * | 2015-10-28 | 2017-01-16 | 한국생산기술연구원 | Method for judging abnormal condition of waterjet nozzle and apparatus for monitering abnormal condition of waterjet nozzle |
CN106803137A (en) * | 2017-01-25 | 2017-06-06 | 东南大学 | Urban track traffic AFC system enters the station volume of the flow of passengers method for detecting abnormality in real time |
CN107092582A (en) * | 2017-03-31 | 2017-08-25 | 江苏方天电力技术有限公司 | One kind is based on the posterior exceptional value on-line checking of residual error and method for evaluating confidence |
CN108509850A (en) * | 2018-02-24 | 2018-09-07 | 华南理工大学 | A kind of invasion signal Recognition Algorithm based on distribution type fiber-optic system |
JP2018159319A (en) * | 2017-03-22 | 2018-10-11 | 大阪瓦斯株式会社 | Abnormality discrimination device, and abnormality discrimination method |
CN109711551A (en) * | 2018-12-28 | 2019-05-03 | 北京旷视科技有限公司 | Data processing method and device |
CN110010202A (en) * | 2017-09-22 | 2019-07-12 | 北京蓝标一成科技有限公司 | A kind of foundation, judgment criteria and the judgment method of the method judging the purebred phase recency of Dendrobium fimbriatum Hook |
-
2019
- 2019-12-18 CN CN201911307015.1A patent/CN111121946B/en active Active
Patent Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5689067A (en) * | 1995-02-03 | 1997-11-18 | Daimler-Benz Ag | Diagnostic method and apparatus for monitoring the wear of at least an engine timing chain |
US6131013A (en) * | 1998-01-30 | 2000-10-10 | Motorola, Inc. | Method and apparatus for performing targeted interference suppression |
CN102288286A (en) * | 2011-06-16 | 2011-12-21 | 中国科学院沈阳自动化研究所 | Method for analyzing and evaluating measure point precision of gearbox in vibration acceleration sensor |
CN104240455A (en) * | 2014-08-07 | 2014-12-24 | 北京航天控制仪器研究所 | Method for identifying disturbance event in distributed type optical fiber pipeline security early-warning system |
CN104217513A (en) * | 2014-09-02 | 2014-12-17 | 浙江中欣动力测控技术有限公司 | Method for improving intrusion event identification accuracy based on phase-sensitive optical time domain reflectrometer (phi-OTDR) |
KR101696789B1 (en) * | 2015-10-28 | 2017-01-16 | 한국생산기술연구원 | Method for judging abnormal condition of waterjet nozzle and apparatus for monitering abnormal condition of waterjet nozzle |
CN106803137A (en) * | 2017-01-25 | 2017-06-06 | 东南大学 | Urban track traffic AFC system enters the station volume of the flow of passengers method for detecting abnormality in real time |
JP2018159319A (en) * | 2017-03-22 | 2018-10-11 | 大阪瓦斯株式会社 | Abnormality discrimination device, and abnormality discrimination method |
CN107092582A (en) * | 2017-03-31 | 2017-08-25 | 江苏方天电力技术有限公司 | One kind is based on the posterior exceptional value on-line checking of residual error and method for evaluating confidence |
CN110010202A (en) * | 2017-09-22 | 2019-07-12 | 北京蓝标一成科技有限公司 | A kind of foundation, judgment criteria and the judgment method of the method judging the purebred phase recency of Dendrobium fimbriatum Hook |
CN108509850A (en) * | 2018-02-24 | 2018-09-07 | 华南理工大学 | A kind of invasion signal Recognition Algorithm based on distribution type fiber-optic system |
CN109711551A (en) * | 2018-12-28 | 2019-05-03 | 北京旷视科技有限公司 | Data processing method and device |
Non-Patent Citations (2)
Title |
---|
孙进明 等: "分布式光纤振动传感信号的采集与处理", 《工业控制计算机》 * |
张红民 等: "一种改进的相邻概率随机抽样一致性算法", 《激光杂志》 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111735528A (en) * | 2020-06-05 | 2020-10-02 | 南京曦光信息科技有限公司 | KNN (K nearest neighbor) voting point accurate positioning method |
CN115112866A (en) * | 2022-08-29 | 2022-09-27 | 山东省鲁南地质工程勘察院(山东省地质矿产勘查开发局第二地质大队) | Hydrothermal deposit investigation system based on electromagnetic radiation |
Also Published As
Publication number | Publication date |
---|---|
CN111121946B (en) | 2021-09-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108491861A (en) | Power transmission and transformation equipment state abnormal patterns recognition methods based on multi-source multi-parameter fusion and device | |
CN109359690B (en) | Vehicle travel track identification method based on checkpoint data | |
CN111121946B (en) | Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area | |
CN113344134B (en) | Low-voltage distribution monitoring terminal data acquisition abnormality detection method and system | |
CN103473540B (en) | The modeling of intelligent transportation system track of vehicle increment type and online method for detecting abnormality | |
CN112911627B (en) | Wireless network performance detection method, device and storage medium | |
EP1958034B1 (en) | Use of sequential clustering for instance selection in machine condition monitoring | |
CN106815639A (en) | The abnormal point detecting method and device of flow data | |
CN113657737B (en) | Water quality evaluation method and system based on automatic monitoring data, equipment and storage medium | |
CN116308305B (en) | Bridge health monitoring data management system | |
CN106935038B (en) | Parking detection system and detection method | |
CN113037595A (en) | Abnormal device detection method and device, electronic device and storage medium | |
CN111949496A (en) | Data detection method and device | |
CN113723467A (en) | Sample collection method, device and equipment for defect detection | |
CN106033643B (en) | A kind of data processing method and device | |
CN114093171A (en) | Traffic running state monitoring method and device based on multi-source data fusion | |
CN112100037B (en) | Alarm level identification method, device, electronic equipment and storage medium | |
CN112329810B (en) | Image recognition model training method and device based on significance detection | |
CN115932144B (en) | Chromatograph performance detection method, chromatograph performance detection device, chromatograph performance detection equipment and computer medium | |
CN112711052A (en) | GNSS coordinate sequence step detection improvement method and system based on continuous t inspection | |
Candy et al. | Receiver operating characteristic (roc) curves: An analysis tool for detection performance | |
CN112149546B (en) | Information processing method, device, electronic equipment and storage medium | |
CN114579519A (en) | File system abnormality detection method and device, storage medium and electronic device | |
CN113702931B (en) | External parameter calibration method and device for vehicle-mounted radar and storage medium | |
CN112529299B (en) | Short traffic flow prediction method based on ARIMA and LSTM mixed neural network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |