CN108733624A - A kind of water quality anomaly data detection and reconstructing method - Google Patents

A kind of water quality anomaly data detection and reconstructing method Download PDF

Info

Publication number
CN108733624A
CN108733624A CN201810318841.5A CN201810318841A CN108733624A CN 108733624 A CN108733624 A CN 108733624A CN 201810318841 A CN201810318841 A CN 201810318841A CN 108733624 A CN108733624 A CN 108733624A
Authority
CN
China
Prior art keywords
signal
water quality
time series
value
abnormal data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810318841.5A
Other languages
Chinese (zh)
Other versions
CN108733624B (en
Inventor
蒋鹏
孙光培
许欢
余善恩
林广�
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Dianzi University
Original Assignee
Hangzhou Dianzi University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Dianzi University filed Critical Hangzhou Dianzi University
Priority to CN201810318841.5A priority Critical patent/CN108733624B/en
Publication of CN108733624A publication Critical patent/CN108733624A/en
Application granted granted Critical
Publication of CN108733624B publication Critical patent/CN108733624B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/02Preprocessing
    • G06F2218/04Denoising
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10Complex mathematical operations
    • G06F17/14Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/08Feature extraction
    • G06F2218/10Feature extraction by analysing the shape of a waveform, e.g. extracting parameters relating to peaks
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02ATECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A20/00Water conservation; Efficient water supply; Efficient water use
    • Y02A20/152Water filtration

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Computational Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Testing And Monitoring For Control Systems (AREA)

Abstract

The invention discloses a kind of water quality anomaly data detection and reconstructing methods.VMD (Variational Mode Decomposition) algorithm is applied to the detection of water quality abnormal data by the present invention, overcome the shortcomings that EMD (Empirical Mode Decomposition) algorithm easy tos produce modal overlap phenomenon in signal decomposition, optimized parameter k is found using Newton iteration method, improve the deficiency that Conventional wisdom method determines parameter k, and piecewise curve-fitting method is combined to reconstruct water quality abnormal data, so that the method for the present invention is had more applicability in water quality exception monitoring field.

Description

A kind of water quality anomaly data detection and reconstructing method
Technical field
The invention belongs to water quality monitoring fields, and in particular to a kind of water quality anomaly data detection and reconstructing method.
Background technology
Anomaly data detection is one important research direction of environmental monitoring with reconstruct, is especially led in water quality monitoring Domain.Water monitoring data it is normal whether weight is implemented with to the law enforcement of water quality supervision department and related water environment protection measure It is big to influence.Therefore, the anomaly data detection in water monitoring data is come out and is reconstructed and be of great significance.
Numerous scholars expand further investigation to the detection of abnormal data, common method have based on statistical method, Method based on clustering, the rejecting outliers method based on SVR, the mode decomposition method etc. based on EMD.It is any of the above Detection method has preferable detection result to different types of abnormal data, but there are also inevitable defects.Such as The variation tendency of entire data set is had ignored based on statistical rejecting outliers method, it sometimes appear that normal data is differentiated For abnormal data and the phenomenon that abnormal data is determined as normal data;Mode decomposition method based on EMD to signal into Be susceptible to serious modal overlap phenomenon during row mode decomposition, the anomaly data detection directly influenced it is accurate Rate.
Above-mentioned various detection methods, have only only completed the detection to abnormal data, and there is no the abnormal numbers to detecting According to being reconstructed, this is a prodigious defect.In order to solve the deficiency of existing detection method, it is necessary to propose water quality exception number According to the new method of detection, this method is made to have more applicability in water quality monitoring field.
Invention content
The main object of the present invention is to overcome EMD (Empirical Mode Decomposition) algorithm in water quality The drawbacks of modal overlap phenomenon is easy tod produce in anomaly data detection, the accuracy rate for improving water quality anomaly data detection, it is proposed that Water quality abnormal data inspection of the one kind based on VMD (Variational Mode Decomposition) and subsection curve drafting algorithm Survey and reconstructing method.
Concrete scheme of the present invention is as follows:
Obtain water quality monitoring parameter time series signal p (t), water quality abnormal data is detected using VMD algorithms, using point Section curve-fitting method reconstructs water quality abnormal data.Specifically include following five steps:
Step 1:It is converted using Hilbert, time series signal is resolved into k IMF modal components, obtain k decomposition Signal.
Step 2:Solve the optimal value of k.Related coefficient (COR) is introduced, it represents mode and the degree of correlation of original signal. If two time series x (n), the correlation coefficient ρ of y (n)xyIt is defined as follows:
K optimal values are solved using Newton iteration method, it is assumed that original signal is decomposed into k mode, k-th of mode and original letter Number related coefficient be denoted as ρxy(k), the average value of this k related coefficient is sought:
Provide correlation coefficient ρ0Satisfied mode decomposition effect can be reached when=0.8.Construct object function
An assigned error ε (ε=0.01) is set, in an iterative process, when f (k)≤ε, stops iteration, k values at this time As optimal solution.
Step 3:The unilateral frequency spectrum for calculating k decomposed signal carries out spectrum analysis to k modal components, removes pseudo- component Mode.Remaining modal components are superimposed, one group of new time series signal p ' (t) is obtained.
Step 4:On the basis of the 1/10 of p ' (t) absolute values, a dynamic threshold T (t) is set, p (t) and p ' (t) are done Difference takes absolute value, obtain error e (t)=| p (t)-p ' (t) |.As e (t) >=T (t), which is considered as abnormal data.
Step 5:The abnormal data in time series signal p (t) is rejected, one group of new time series signal p " is obtained (t)." (t) carries out segmentation Cubic Curve Fitting to the concavity and convexity for judging time series signal p " (t), to p, reconstructs abnormal data.Tool Body process is as follows:
Set the cubic fit curvilinear function of certain segment data (assuming that having m data) as:Y (t)=a+bt+ct2+dt3, y (t) it is the match value for rejecting time series signal p " (t), a, b, c, d are fitted polynomial coefficients, the variance of this section of curve matching
For the mean value of time series signal p " (t).Construct object function
E is deviation, solves a, b, c, d by the principle of sum of square of deviations minimum, obtains matched curve y (t).By abnormal data Time sequential value substitute into y (t), obtain the reconstruction value of abnormal data.
The beneficial benefit of the present invention:The present invention overcomes EMD algorithms, and modal overlap phenomenon is easy tod produce in signal decomposition The shortcomings that.In terms of water quality anomaly data detection, VMD algorithm ratio EMD algorithms are more advantageous.It is found most using Newton iteration method Excellent parameter k improves the deficiency that Conventional wisdom method determines parameter k, and piecewise curve-fitting method is combined to reconstruct water quality abnormal data, The method of the present invention is set to have more applicability in water quality exception monitoring field.This method is equally applicable to other type abnormal datas Detection and reconstruct are suitble to promote in other anomaly data detections and reconstruction field.
Description of the drawings
The following content is the simple declarations to attached drawing used in the method for the present invention:
Fig. 1 is the flow diagram of the method for the present invention;
Fig. 2 is VMD algorithm flow charts of the present invention;
Fig. 3 is the original emulation signal time-domain diagram of the present invention and spectrogram;
Fig. 4 is Newton iteration method flow chart of the present invention;
The time-domain diagram and spectrogram of Fig. 5 each modal components when being k=4 of the present invention;
Fig. 6 is the time-domain diagram that the present invention rejects remaining modal components superposition after pseudo- modal components;
Fig. 7 is anomaly data detection result of the present invention;
Fig. 8 is piecewise curve-fitting method flow chart of the present invention;
Fig. 9 is abnormal data restoration result of the present invention.
Specific implementation mode
Detailed specific description is done to technical scheme of the present invention presently in connection with attached drawing.
As shown in Figure 1, the method for the present invention implementing procedure is as follows:
Water quality monitoring value is pre-processed using the abnormal deviation data examination method based on normal distribution first, is rejected more Then apparent exception monitoring value uses VMD algorithms to carry out mode decomposition to residue monitoring value sequence, and Newton iteration is combined to calculate Method Optimal Parameters k obtains one group of optimal IMF decomposed signal, after rejecting puppet IMF components, is overlapped to remaining IMF components, Detect the abnormal data in water quality monitoring value.Finally, thundering observed data is reconstructed using piecewise curve-fitting method.
As shown in Fig. 2, the step of VMD algorithms detection abnormal data used in the present invention, is as follows:
Step 1:It is converted using Hilbert, time series signal is resolved into k IMF component, obtained k and decompose letter Number.
Step 2:The unilateral frequency spectrum for calculating k decomposed signal carries out spectrum analysis to k modal components, removes pseudo- component Mode.Remaining modal components are superimposed, one group of new time series signal p ' (t) is obtained.
Step 3:On the basis of the 1/10 of p ' (t) absolute values, a dynamic threshold T (t) is set, p (t) and p ' (t) are done Difference takes absolute value, obtain error e (t)=| p (t)-p ' (t) |.As e (t) >=T (t), which is considered as abnormal data.
As shown in figure 3, being verified to the method for the present invention using emulation signal, when which is original emulation signal of the invention Domain figure and spectrogram.Emulation signal is using three cosine signals and one group of noise sequence that frequency is respectively 55Hz, 266Hz, 580Hz Row A is formed by stacking, and emulation signal expression is:
The expression formula of A is as follows:
A=zeros (1, N);
A([8,16,30,67,25,69,78,97,101,134,150,170,210,245,289,310,330,400, 420,440,506])=0.6;
A([536,562,581,602,635,665,693,726,742,771,800,825,847,862,879,893, 1005])=- 0.6;
As shown in figure 4, it is as follows to solve optimal Decomposition mode number k methods using Newton iteration method:
Related coefficient (COR) is introduced, it represents mode and the degree of correlation of original signal.If two time series x (n), y (n) correlation coefficient ρxyIt is defined as follows:
K optimal values are solved using Newton iteration method, it is assumed that original signal is decomposed into k mode, k-th of mode and original letter Number related coefficient be denoted as ρxy(k), the average value of this k related coefficient is sought:
Provide correlation coefficient ρ0Satisfied mode decomposition effect can be reached when=0.8.Construct object function
An assigned error ε (ε=0.01) is set, in an iterative process, as f (k)≤ε, stops iteration, k at this time Value is optimal solution.
Fig. 5 is that the optimal value of k is equal to the time-domain diagram and spectrogram of each modal components when 4, it can be seen from the figure that passing through VMD algorithms have decomposited 4 IMF components, and frequency is respectively 55Hz, 266Hz, 580Hz and one group of noise.Believe with original emulation Number effective frequency component for including is completely the same.
Fig. 6 is the time-domain diagram that the present invention rejects remaining modal components superposition after pseudo- modal components.
Fig. 7 is anomaly data detection result figure of the present invention.Remaining modal components are superimposed, one group of new time sequence is obtained Column signal p ' (t).On the basis of the 1/10 of p ' (t) absolute values, a dynamic threshold T (t) is set, p (t) and p ' (t) are made the difference, Take absolute value, obtain error e (t)=| p (t)-p ' (t) |.As e (t) >=T (t), which is considered as abnormal data.Figure It is middle with the part of cross mark be detected it is original emulation signal in abnormal data.The abnormal data sequence detected and original The sequence of noise is consistent in beginning signal.
Fig. 8 is piecewise curve-fitting method flow chart of the present invention.After rejecting the abnormal data detected by VMD algorithms, obtain One group of new time series signal.Judge the concavity and convexity of the time series signal, carry out segmentation Cubic Curve Fitting, reconstruct is abnormal Data.Detailed process is as follows:
The cubic fit curvilinear function of certain segment data (assuming that having m data) is enabled to be:Y (t)=a+bt+ct2+dt3, y (t) To reject the match value of time series signal p " (t), a, b, c, d are fitted polynomial coefficients, the variance of this section of curve matching
For the mean value of time series signal p " (t).Construct object function
E is deviation, solves a, b, c, d by the principle of sum of square of deviations minimum, obtains matched curve y (t).By abnormal data Time sequential value substitute into y (t), obtain the reconstruction value of abnormal data.
Fig. 9 is abnormal data restoration result figure of the present invention.The part marked with dot in figure is different in original emulation signal The reconstruction value of regular data.

Claims (1)

1. a kind of water quality anomaly data detection and reconstructing method, it is characterised in that:This approach includes the following steps:
Step 1:It is converted using Hilbert, by water quality monitoring parameter time series signalResolve into k IMF mode point Amount, obtains k decomposed signal;
Step 2:Solve the optimal value of k;Related coefficient is introduced, it represents mode and the degree of correlation of original signal;If two times Sequence signal,Related coefficientIt is defined as follows:
K optimal values are solved using Newton iteration method, it is assumed that original signal is decomposed into k mode, k-th of mode and original signal Related coefficient is denoted as, seek the average value of this k related coefficient:
Provide related coefficientSatisfied mode decomposition effect can be reached when=0.8;Construct object function
Set an assigned error, in an iterative process,When, stop iteration, k values at this time are optimal solution;
Step 3:The unilateral frequency spectrum for calculating k decomposed signal carries out spectrum analysis to k modal components, removes pseudo- component mould State;Remaining modal components are superimposed, one group of new time series signal is obtained
Step 4:With time series signalOn the basis of the 1/10 of absolute value, a dynamic threshold is set, WithIt makes the difference, takes absolute value, obtain error;WhenWhen, the monitoring Value is considered as abnormal data;
Step 5:Reject time series signalIn abnormal data, obtain one group of new time series signal;Sentence Disconnected time series signalConcavity and convexity, it is rightSegmentation Cubic Curve Fitting is carried out, abnormal data is reconstructed;Specific mistake Journey is as follows:
Set the cubic fit curvilinear function of certain segment data as:,When to reject Between sequence signalMatch value,abcdFor fitted polynomial coefficients, segment data hypothesis has m data, curve quasi- The variance of conjunction
,
For time series signalMean value;Construct object function
,
For deviation, solved by the principle of sum of square of deviations minimumabcd, obtain matched curve;By abnormal data when Between sequential value substitute into, obtain the reconstruction value of abnormal data.
CN201810318841.5A 2018-04-11 2018-04-11 Water quality abnormal data detection and reconstruction method Expired - Fee Related CN108733624B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810318841.5A CN108733624B (en) 2018-04-11 2018-04-11 Water quality abnormal data detection and reconstruction method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810318841.5A CN108733624B (en) 2018-04-11 2018-04-11 Water quality abnormal data detection and reconstruction method

Publications (2)

Publication Number Publication Date
CN108733624A true CN108733624A (en) 2018-11-02
CN108733624B CN108733624B (en) 2021-11-30

Family

ID=63940777

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810318841.5A Expired - Fee Related CN108733624B (en) 2018-04-11 2018-04-11 Water quality abnormal data detection and reconstruction method

Country Status (1)

Country Link
CN (1) CN108733624B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110617927A (en) * 2019-09-20 2019-12-27 长安大学 Structural settlement deformation prediction method based on EMD-SVR-WNN
CN112331259A (en) * 2020-11-26 2021-02-05 浙江大学 Tissue metabolite information evaluation method, device and medium based on Bloch-McConnell equation simulation
CN112819082A (en) * 2021-02-09 2021-05-18 南京邮电大学 Satellite spectrum sensing data reconstruction method based on deep learning

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101281187A (en) * 2008-04-08 2008-10-08 杭州电子科技大学 Water environment monitoring node based on ZigBee wireless technique
CN103732132A (en) * 2011-06-13 2014-04-16 安吉奥梅特里克斯公司 Multifunctional guidewire assemblies and system for analyzing anatomical and functional parameters
TW201519862A (en) * 2013-11-28 2015-06-01 Nat Univ Chin Yi Technology Method and apparatus for monitoring heartbeat signal based on empirical mode decomposition
CN107273234A (en) * 2017-05-26 2017-10-20 中国航天系统科学与工程研究院 A kind of time series data rejecting outliers and bearing calibration based on EEMD
CN107463633A (en) * 2017-07-17 2017-12-12 中国航天系统科学与工程研究院 A kind of real time data rejecting outliers method based on EEMD neutral nets
CN107702908A (en) * 2017-10-12 2018-02-16 国网山东省电力公司莱芜供电公司 GIS mechanical oscillation signal Time-Frequency Analysis Methods based on VMD self adapting morphologies
CN107884190A (en) * 2017-11-01 2018-04-06 北京信息科技大学 The Method for Bearing Fault Diagnosis decomposed based on variation mode decomposition and wavelet singular

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101281187A (en) * 2008-04-08 2008-10-08 杭州电子科技大学 Water environment monitoring node based on ZigBee wireless technique
CN103732132A (en) * 2011-06-13 2014-04-16 安吉奥梅特里克斯公司 Multifunctional guidewire assemblies and system for analyzing anatomical and functional parameters
TW201519862A (en) * 2013-11-28 2015-06-01 Nat Univ Chin Yi Technology Method and apparatus for monitoring heartbeat signal based on empirical mode decomposition
CN107273234A (en) * 2017-05-26 2017-10-20 中国航天系统科学与工程研究院 A kind of time series data rejecting outliers and bearing calibration based on EEMD
CN107463633A (en) * 2017-07-17 2017-12-12 中国航天系统科学与工程研究院 A kind of real time data rejecting outliers method based on EEMD neutral nets
CN107702908A (en) * 2017-10-12 2018-02-16 国网山东省电力公司莱芜供电公司 GIS mechanical oscillation signal Time-Frequency Analysis Methods based on VMD self adapting morphologies
CN107884190A (en) * 2017-11-01 2018-04-06 北京信息科技大学 The Method for Bearing Fault Diagnosis decomposed based on variation mode decomposition and wavelet singular

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
刘尚坤 等: "基于改进变分模态分解的旋转机械故障时频分析方法", 《振动工程学报》 *
姜海旭 等: "基于形态变分模态分解和JRD的航天器异常状态识别", 《西北工业大学学报》 *
孙光培: "流域水环境监测终端的研发", 《中国优秀硕士学位论文全文数据库 信息科技辑》 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110617927A (en) * 2019-09-20 2019-12-27 长安大学 Structural settlement deformation prediction method based on EMD-SVR-WNN
CN110617927B (en) * 2019-09-20 2022-04-05 长安大学 Structural settlement deformation prediction method based on EMD-SVR-WNN
CN112331259A (en) * 2020-11-26 2021-02-05 浙江大学 Tissue metabolite information evaluation method, device and medium based on Bloch-McConnell equation simulation
CN112331259B (en) * 2020-11-26 2024-04-12 浙江大学 Tissue metabolite information evaluation method, device and medium based on Bloch-McConnell equation simulation
CN112819082A (en) * 2021-02-09 2021-05-18 南京邮电大学 Satellite spectrum sensing data reconstruction method based on deep learning

Also Published As

Publication number Publication date
CN108733624B (en) 2021-11-30

Similar Documents

Publication Publication Date Title
CN108733624A (en) A kind of water quality anomaly data detection and reconstructing method
CN107505652B (en) A kind of mine microquake signal discrimination method based on energy-distributing feature
Chevillard et al. Lagrangian velocity statistics in turbulent flows: Effects of dissipation
CN101063660B (en) Method for detecting textile defect and device thereof
Feng et al. Design and implementation of a self-validating pressure sensor
CN104460654B (en) A kind of gear imperfection failure diagnosis Rules extraction method based on quantization characteristic relation
CN101887407B (en) Hilbert-Huang transform-based equipment or system built-in test signal characteristic extraction method
CN116226646B (en) Method, system, equipment and medium for predicting health state and residual life of bearing
CN109288508A (en) A kind of pressure value intelligent measurement method based on CRNN-BP
CN105571645A (en) Automatic dam monitoring method
CN110457979A (en) The Diagnosis Method of Diesel Fault of fuzzy control is decomposed based on tensor Tucker
CN109459399A (en) A kind of spectral water quality COD, turbidity detection method
Ji et al. A novel method to identify the scaling region for chaotic time series correlation dimension calculation
CN109297928B (en) A kind of method and system detecting soy sauce quality
CN108761202B (en) Harmonic detection method combining pole symmetric modal decomposition and Hilbert transform
CN117314020B (en) Wetland carbon sink data monitoring system of plankton
CN109490776A (en) A kind of good substandard products detection method of mobile phone vibrating motor based on machine learning
CN109992748A (en) A kind of river mouth tidal level harmonic analysis method
CN201081763Y (en) Textile defect detector
CN106770861B (en) The evaluation method of oil-filled transformer on-line monitoring availability of data
CN110231165B (en) Mechanical equipment fault diagnosis method based on expectation difference constraint confidence network
CN106815434A (en) Noisiness time frequency analysis denoising effect evaluation method based on small echo and EMD
CN104316220B (en) Automatic mercury thermometer calibration method based on machine vision
Al-Sobky et al. A generalized piecewise regression for transportation models
Sun et al. Identification method of gas-liquid two-phase flow regime based on distance evaluation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20211130