CN108733624A - A kind of water quality anomaly data detection and reconstructing method - Google Patents
A kind of water quality anomaly data detection and reconstructing method Download PDFInfo
- Publication number
- CN108733624A CN108733624A CN201810318841.5A CN201810318841A CN108733624A CN 108733624 A CN108733624 A CN 108733624A CN 201810318841 A CN201810318841 A CN 201810318841A CN 108733624 A CN108733624 A CN 108733624A
- Authority
- CN
- China
- Prior art keywords
- signal
- water quality
- time series
- value
- abnormal data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/02—Preprocessing
- G06F2218/04—Denoising
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/14—Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
- G06F2218/10—Feature extraction by analysing the shape of a waveform, e.g. extracting parameters relating to peaks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A20/00—Water conservation; Efficient water supply; Efficient water use
- Y02A20/152—Water filtration
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention discloses a kind of water quality anomaly data detection and reconstructing methods.VMD (Variational Mode Decomposition) algorithm is applied to the detection of water quality abnormal data by the present invention, overcome the shortcomings that EMD (Empirical Mode Decomposition) algorithm easy tos produce modal overlap phenomenon in signal decomposition, optimized parameter k is found using Newton iteration method, improve the deficiency that Conventional wisdom method determines parameter k, and piecewise curve-fitting method is combined to reconstruct water quality abnormal data, so that the method for the present invention is had more applicability in water quality exception monitoring field.
Description
Technical field
The invention belongs to water quality monitoring fields, and in particular to a kind of water quality anomaly data detection and reconstructing method.
Background technology
Anomaly data detection is one important research direction of environmental monitoring with reconstruct, is especially led in water quality monitoring
Domain.Water monitoring data it is normal whether weight is implemented with to the law enforcement of water quality supervision department and related water environment protection measure
It is big to influence.Therefore, the anomaly data detection in water monitoring data is come out and is reconstructed and be of great significance.
Numerous scholars expand further investigation to the detection of abnormal data, common method have based on statistical method,
Method based on clustering, the rejecting outliers method based on SVR, the mode decomposition method etc. based on EMD.It is any of the above
Detection method has preferable detection result to different types of abnormal data, but there are also inevitable defects.Such as
The variation tendency of entire data set is had ignored based on statistical rejecting outliers method, it sometimes appear that normal data is differentiated
For abnormal data and the phenomenon that abnormal data is determined as normal data;Mode decomposition method based on EMD to signal into
Be susceptible to serious modal overlap phenomenon during row mode decomposition, the anomaly data detection directly influenced it is accurate
Rate.
Above-mentioned various detection methods, have only only completed the detection to abnormal data, and there is no the abnormal numbers to detecting
According to being reconstructed, this is a prodigious defect.In order to solve the deficiency of existing detection method, it is necessary to propose water quality exception number
According to the new method of detection, this method is made to have more applicability in water quality monitoring field.
Invention content
The main object of the present invention is to overcome EMD (Empirical Mode Decomposition) algorithm in water quality
The drawbacks of modal overlap phenomenon is easy tod produce in anomaly data detection, the accuracy rate for improving water quality anomaly data detection, it is proposed that
Water quality abnormal data inspection of the one kind based on VMD (Variational Mode Decomposition) and subsection curve drafting algorithm
Survey and reconstructing method.
Concrete scheme of the present invention is as follows:
Obtain water quality monitoring parameter time series signal p (t), water quality abnormal data is detected using VMD algorithms, using point
Section curve-fitting method reconstructs water quality abnormal data.Specifically include following five steps:
Step 1:It is converted using Hilbert, time series signal is resolved into k IMF modal components, obtain k decomposition
Signal.
Step 2:Solve the optimal value of k.Related coefficient (COR) is introduced, it represents mode and the degree of correlation of original signal.
If two time series x (n), the correlation coefficient ρ of y (n)xyIt is defined as follows:
K optimal values are solved using Newton iteration method, it is assumed that original signal is decomposed into k mode, k-th of mode and original letter
Number related coefficient be denoted as ρxy(k), the average value of this k related coefficient is sought:
Provide correlation coefficient ρ0Satisfied mode decomposition effect can be reached when=0.8.Construct object function
An assigned error ε (ε=0.01) is set, in an iterative process, when f (k)≤ε, stops iteration, k values at this time
As optimal solution.
Step 3:The unilateral frequency spectrum for calculating k decomposed signal carries out spectrum analysis to k modal components, removes pseudo- component
Mode.Remaining modal components are superimposed, one group of new time series signal p ' (t) is obtained.
Step 4:On the basis of the 1/10 of p ' (t) absolute values, a dynamic threshold T (t) is set, p (t) and p ' (t) are done
Difference takes absolute value, obtain error e (t)=| p (t)-p ' (t) |.As e (t) >=T (t), which is considered as abnormal data.
Step 5:The abnormal data in time series signal p (t) is rejected, one group of new time series signal p " is obtained
(t)." (t) carries out segmentation Cubic Curve Fitting to the concavity and convexity for judging time series signal p " (t), to p, reconstructs abnormal data.Tool
Body process is as follows:
Set the cubic fit curvilinear function of certain segment data (assuming that having m data) as:Y (t)=a+bt+ct2+dt3, y
(t) it is the match value for rejecting time series signal p " (t), a, b, c, d are fitted polynomial coefficients, the variance of this section of curve matching
For the mean value of time series signal p " (t).Construct object function
E is deviation, solves a, b, c, d by the principle of sum of square of deviations minimum, obtains matched curve y (t).By abnormal data
Time sequential value substitute into y (t), obtain the reconstruction value of abnormal data.
The beneficial benefit of the present invention:The present invention overcomes EMD algorithms, and modal overlap phenomenon is easy tod produce in signal decomposition
The shortcomings that.In terms of water quality anomaly data detection, VMD algorithm ratio EMD algorithms are more advantageous.It is found most using Newton iteration method
Excellent parameter k improves the deficiency that Conventional wisdom method determines parameter k, and piecewise curve-fitting method is combined to reconstruct water quality abnormal data,
The method of the present invention is set to have more applicability in water quality exception monitoring field.This method is equally applicable to other type abnormal datas
Detection and reconstruct are suitble to promote in other anomaly data detections and reconstruction field.
Description of the drawings
The following content is the simple declarations to attached drawing used in the method for the present invention:
Fig. 1 is the flow diagram of the method for the present invention;
Fig. 2 is VMD algorithm flow charts of the present invention;
Fig. 3 is the original emulation signal time-domain diagram of the present invention and spectrogram;
Fig. 4 is Newton iteration method flow chart of the present invention;
The time-domain diagram and spectrogram of Fig. 5 each modal components when being k=4 of the present invention;
Fig. 6 is the time-domain diagram that the present invention rejects remaining modal components superposition after pseudo- modal components;
Fig. 7 is anomaly data detection result of the present invention;
Fig. 8 is piecewise curve-fitting method flow chart of the present invention;
Fig. 9 is abnormal data restoration result of the present invention.
Specific implementation mode
Detailed specific description is done to technical scheme of the present invention presently in connection with attached drawing.
As shown in Figure 1, the method for the present invention implementing procedure is as follows:
Water quality monitoring value is pre-processed using the abnormal deviation data examination method based on normal distribution first, is rejected more
Then apparent exception monitoring value uses VMD algorithms to carry out mode decomposition to residue monitoring value sequence, and Newton iteration is combined to calculate
Method Optimal Parameters k obtains one group of optimal IMF decomposed signal, after rejecting puppet IMF components, is overlapped to remaining IMF components,
Detect the abnormal data in water quality monitoring value.Finally, thundering observed data is reconstructed using piecewise curve-fitting method.
As shown in Fig. 2, the step of VMD algorithms detection abnormal data used in the present invention, is as follows:
Step 1:It is converted using Hilbert, time series signal is resolved into k IMF component, obtained k and decompose letter
Number.
Step 2:The unilateral frequency spectrum for calculating k decomposed signal carries out spectrum analysis to k modal components, removes pseudo- component
Mode.Remaining modal components are superimposed, one group of new time series signal p ' (t) is obtained.
Step 3:On the basis of the 1/10 of p ' (t) absolute values, a dynamic threshold T (t) is set, p (t) and p ' (t) are done
Difference takes absolute value, obtain error e (t)=| p (t)-p ' (t) |.As e (t) >=T (t), which is considered as abnormal data.
As shown in figure 3, being verified to the method for the present invention using emulation signal, when which is original emulation signal of the invention
Domain figure and spectrogram.Emulation signal is using three cosine signals and one group of noise sequence that frequency is respectively 55Hz, 266Hz, 580Hz
Row A is formed by stacking, and emulation signal expression is:
The expression formula of A is as follows:
A=zeros (1, N);
A([8,16,30,67,25,69,78,97,101,134,150,170,210,245,289,310,330,400,
420,440,506])=0.6;
A([536,562,581,602,635,665,693,726,742,771,800,825,847,862,879,893,
1005])=- 0.6;
As shown in figure 4, it is as follows to solve optimal Decomposition mode number k methods using Newton iteration method:
Related coefficient (COR) is introduced, it represents mode and the degree of correlation of original signal.If two time series x (n), y
(n) correlation coefficient ρxyIt is defined as follows:
K optimal values are solved using Newton iteration method, it is assumed that original signal is decomposed into k mode, k-th of mode and original letter
Number related coefficient be denoted as ρxy(k), the average value of this k related coefficient is sought:
Provide correlation coefficient ρ0Satisfied mode decomposition effect can be reached when=0.8.Construct object function
An assigned error ε (ε=0.01) is set, in an iterative process, as f (k)≤ε, stops iteration, k at this time
Value is optimal solution.
Fig. 5 is that the optimal value of k is equal to the time-domain diagram and spectrogram of each modal components when 4, it can be seen from the figure that passing through
VMD algorithms have decomposited 4 IMF components, and frequency is respectively 55Hz, 266Hz, 580Hz and one group of noise.Believe with original emulation
Number effective frequency component for including is completely the same.
Fig. 6 is the time-domain diagram that the present invention rejects remaining modal components superposition after pseudo- modal components.
Fig. 7 is anomaly data detection result figure of the present invention.Remaining modal components are superimposed, one group of new time sequence is obtained
Column signal p ' (t).On the basis of the 1/10 of p ' (t) absolute values, a dynamic threshold T (t) is set, p (t) and p ' (t) are made the difference,
Take absolute value, obtain error e (t)=| p (t)-p ' (t) |.As e (t) >=T (t), which is considered as abnormal data.Figure
It is middle with the part of cross mark be detected it is original emulation signal in abnormal data.The abnormal data sequence detected and original
The sequence of noise is consistent in beginning signal.
Fig. 8 is piecewise curve-fitting method flow chart of the present invention.After rejecting the abnormal data detected by VMD algorithms, obtain
One group of new time series signal.Judge the concavity and convexity of the time series signal, carry out segmentation Cubic Curve Fitting, reconstruct is abnormal
Data.Detailed process is as follows:
The cubic fit curvilinear function of certain segment data (assuming that having m data) is enabled to be:Y (t)=a+bt+ct2+dt3, y (t)
To reject the match value of time series signal p " (t), a, b, c, d are fitted polynomial coefficients, the variance of this section of curve matching
For the mean value of time series signal p " (t).Construct object function
E is deviation, solves a, b, c, d by the principle of sum of square of deviations minimum, obtains matched curve y (t).By abnormal data
Time sequential value substitute into y (t), obtain the reconstruction value of abnormal data.
Fig. 9 is abnormal data restoration result figure of the present invention.The part marked with dot in figure is different in original emulation signal
The reconstruction value of regular data.
Claims (1)
1. a kind of water quality anomaly data detection and reconstructing method, it is characterised in that:This approach includes the following steps:
Step 1:It is converted using Hilbert, by water quality monitoring parameter time series signalResolve into k IMF mode point
Amount, obtains k decomposed signal;
Step 2:Solve the optimal value of k;Related coefficient is introduced, it represents mode and the degree of correlation of original signal;If two times
Sequence signal,Related coefficientIt is defined as follows:
K optimal values are solved using Newton iteration method, it is assumed that original signal is decomposed into k mode, k-th of mode and original signal
Related coefficient is denoted as, seek the average value of this k related coefficient:
Provide related coefficientSatisfied mode decomposition effect can be reached when=0.8;Construct object function
Set an assigned error, in an iterative process,When, stop iteration, k values at this time are optimal solution;
Step 3:The unilateral frequency spectrum for calculating k decomposed signal carries out spectrum analysis to k modal components, removes pseudo- component mould
State;Remaining modal components are superimposed, one group of new time series signal is obtained;
Step 4:With time series signalOn the basis of the 1/10 of absolute value, a dynamic threshold is set,
WithIt makes the difference, takes absolute value, obtain error;WhenWhen, the monitoring
Value is considered as abnormal data;
Step 5:Reject time series signalIn abnormal data, obtain one group of new time series signal;Sentence
Disconnected time series signalConcavity and convexity, it is rightSegmentation Cubic Curve Fitting is carried out, abnormal data is reconstructed;Specific mistake
Journey is as follows:
Set the cubic fit curvilinear function of certain segment data as:,When to reject
Between sequence signalMatch value,a、b、c、dFor fitted polynomial coefficients, segment data hypothesis has m data, curve quasi-
The variance of conjunction
,
For time series signalMean value;Construct object function
,
For deviation, solved by the principle of sum of square of deviations minimuma、b、c、d, obtain matched curve;By abnormal data when
Between sequential value substitute into, obtain the reconstruction value of abnormal data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810318841.5A CN108733624B (en) | 2018-04-11 | 2018-04-11 | Water quality abnormal data detection and reconstruction method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810318841.5A CN108733624B (en) | 2018-04-11 | 2018-04-11 | Water quality abnormal data detection and reconstruction method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN108733624A true CN108733624A (en) | 2018-11-02 |
CN108733624B CN108733624B (en) | 2021-11-30 |
Family
ID=63940777
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201810318841.5A Expired - Fee Related CN108733624B (en) | 2018-04-11 | 2018-04-11 | Water quality abnormal data detection and reconstruction method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN108733624B (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110617927A (en) * | 2019-09-20 | 2019-12-27 | 长安大学 | Structural settlement deformation prediction method based on EMD-SVR-WNN |
CN112331259A (en) * | 2020-11-26 | 2021-02-05 | 浙江大学 | Tissue metabolite information evaluation method, device and medium based on Bloch-McConnell equation simulation |
CN112819082A (en) * | 2021-02-09 | 2021-05-18 | 南京邮电大学 | Satellite spectrum sensing data reconstruction method based on deep learning |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101281187A (en) * | 2008-04-08 | 2008-10-08 | 杭州电子科技大学 | Water environment monitoring node based on ZigBee wireless technique |
CN103732132A (en) * | 2011-06-13 | 2014-04-16 | 安吉奥梅特里克斯公司 | Multifunctional guidewire assemblies and system for analyzing anatomical and functional parameters |
TW201519862A (en) * | 2013-11-28 | 2015-06-01 | Nat Univ Chin Yi Technology | Method and apparatus for monitoring heartbeat signal based on empirical mode decomposition |
CN107273234A (en) * | 2017-05-26 | 2017-10-20 | 中国航天系统科学与工程研究院 | A kind of time series data rejecting outliers and bearing calibration based on EEMD |
CN107463633A (en) * | 2017-07-17 | 2017-12-12 | 中国航天系统科学与工程研究院 | A kind of real time data rejecting outliers method based on EEMD neutral nets |
CN107702908A (en) * | 2017-10-12 | 2018-02-16 | 国网山东省电力公司莱芜供电公司 | GIS mechanical oscillation signal Time-Frequency Analysis Methods based on VMD self adapting morphologies |
CN107884190A (en) * | 2017-11-01 | 2018-04-06 | 北京信息科技大学 | The Method for Bearing Fault Diagnosis decomposed based on variation mode decomposition and wavelet singular |
-
2018
- 2018-04-11 CN CN201810318841.5A patent/CN108733624B/en not_active Expired - Fee Related
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101281187A (en) * | 2008-04-08 | 2008-10-08 | 杭州电子科技大学 | Water environment monitoring node based on ZigBee wireless technique |
CN103732132A (en) * | 2011-06-13 | 2014-04-16 | 安吉奥梅特里克斯公司 | Multifunctional guidewire assemblies and system for analyzing anatomical and functional parameters |
TW201519862A (en) * | 2013-11-28 | 2015-06-01 | Nat Univ Chin Yi Technology | Method and apparatus for monitoring heartbeat signal based on empirical mode decomposition |
CN107273234A (en) * | 2017-05-26 | 2017-10-20 | 中国航天系统科学与工程研究院 | A kind of time series data rejecting outliers and bearing calibration based on EEMD |
CN107463633A (en) * | 2017-07-17 | 2017-12-12 | 中国航天系统科学与工程研究院 | A kind of real time data rejecting outliers method based on EEMD neutral nets |
CN107702908A (en) * | 2017-10-12 | 2018-02-16 | 国网山东省电力公司莱芜供电公司 | GIS mechanical oscillation signal Time-Frequency Analysis Methods based on VMD self adapting morphologies |
CN107884190A (en) * | 2017-11-01 | 2018-04-06 | 北京信息科技大学 | The Method for Bearing Fault Diagnosis decomposed based on variation mode decomposition and wavelet singular |
Non-Patent Citations (3)
Title |
---|
刘尚坤 等: "基于改进变分模态分解的旋转机械故障时频分析方法", 《振动工程学报》 * |
姜海旭 等: "基于形态变分模态分解和JRD的航天器异常状态识别", 《西北工业大学学报》 * |
孙光培: "流域水环境监测终端的研发", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110617927A (en) * | 2019-09-20 | 2019-12-27 | 长安大学 | Structural settlement deformation prediction method based on EMD-SVR-WNN |
CN110617927B (en) * | 2019-09-20 | 2022-04-05 | 长安大学 | Structural settlement deformation prediction method based on EMD-SVR-WNN |
CN112331259A (en) * | 2020-11-26 | 2021-02-05 | 浙江大学 | Tissue metabolite information evaluation method, device and medium based on Bloch-McConnell equation simulation |
CN112331259B (en) * | 2020-11-26 | 2024-04-12 | 浙江大学 | Tissue metabolite information evaluation method, device and medium based on Bloch-McConnell equation simulation |
CN112819082A (en) * | 2021-02-09 | 2021-05-18 | 南京邮电大学 | Satellite spectrum sensing data reconstruction method based on deep learning |
Also Published As
Publication number | Publication date |
---|---|
CN108733624B (en) | 2021-11-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108733624A (en) | A kind of water quality anomaly data detection and reconstructing method | |
CN107505652B (en) | A kind of mine microquake signal discrimination method based on energy-distributing feature | |
Chevillard et al. | Lagrangian velocity statistics in turbulent flows: Effects of dissipation | |
CN101063660B (en) | Method for detecting textile defect and device thereof | |
Feng et al. | Design and implementation of a self-validating pressure sensor | |
CN104460654B (en) | A kind of gear imperfection failure diagnosis Rules extraction method based on quantization characteristic relation | |
CN101887407B (en) | Hilbert-Huang transform-based equipment or system built-in test signal characteristic extraction method | |
CN116226646B (en) | Method, system, equipment and medium for predicting health state and residual life of bearing | |
CN109288508A (en) | A kind of pressure value intelligent measurement method based on CRNN-BP | |
CN105571645A (en) | Automatic dam monitoring method | |
CN110457979A (en) | The Diagnosis Method of Diesel Fault of fuzzy control is decomposed based on tensor Tucker | |
CN109459399A (en) | A kind of spectral water quality COD, turbidity detection method | |
Ji et al. | A novel method to identify the scaling region for chaotic time series correlation dimension calculation | |
CN109297928B (en) | A kind of method and system detecting soy sauce quality | |
CN108761202B (en) | Harmonic detection method combining pole symmetric modal decomposition and Hilbert transform | |
CN117314020B (en) | Wetland carbon sink data monitoring system of plankton | |
CN109490776A (en) | A kind of good substandard products detection method of mobile phone vibrating motor based on machine learning | |
CN109992748A (en) | A kind of river mouth tidal level harmonic analysis method | |
CN201081763Y (en) | Textile defect detector | |
CN106770861B (en) | The evaluation method of oil-filled transformer on-line monitoring availability of data | |
CN110231165B (en) | Mechanical equipment fault diagnosis method based on expectation difference constraint confidence network | |
CN106815434A (en) | Noisiness time frequency analysis denoising effect evaluation method based on small echo and EMD | |
CN104316220B (en) | Automatic mercury thermometer calibration method based on machine vision | |
Al-Sobky et al. | A generalized piecewise regression for transportation models | |
Sun et al. | Identification method of gas-liquid two-phase flow regime based on distance evaluation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20211130 |