CN114298281A - Method, device, equipment and medium for detecting abnormal data in water level measurement data - Google Patents
Method, device, equipment and medium for detecting abnormal data in water level measurement data Download PDFInfo
- Publication number
- CN114298281A CN114298281A CN202111638348.XA CN202111638348A CN114298281A CN 114298281 A CN114298281 A CN 114298281A CN 202111638348 A CN202111638348 A CN 202111638348A CN 114298281 A CN114298281 A CN 114298281A
- Authority
- CN
- China
- Prior art keywords
- water level
- measurement data
- level measurement
- data
- fitting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005259 measurement Methods 0.000 title claims abstract description 260
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 251
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 96
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000013528 artificial neural network Methods 0.000 claims abstract description 84
- 238000001514 detection method Methods 0.000 claims abstract description 21
- 238000012549 training Methods 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 25
- 238000004422 calculation algorithm Methods 0.000 claims description 21
- 230000002068 genetic effect Effects 0.000 claims description 20
- 210000000349 chromosome Anatomy 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 10
- 238000005215 recombination Methods 0.000 claims description 10
- 230000006798 recombination Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 4
- 230000002547 anomalous effect Effects 0.000 claims 1
- 230000008030 elimination Effects 0.000 abstract description 5
- 238000003379 elimination reaction Methods 0.000 abstract description 5
- 239000002609 medium Substances 0.000 description 20
- 238000012545 processing Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000035772 mutation Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000012120 mounting media Substances 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Landscapes
- Measurement Of Levels Of Liquids Or Fluent Solid Materials (AREA)
Abstract
The embodiment of the invention discloses a method, a device, equipment and a medium for detecting abnormal data in water level measurement data. The method comprises the following steps: acquiring water level measurement data to be detected; respectively inputting each water level measurement data into a corresponding trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data, and calculating a residual value of each water level measurement data according to the fitting value; respectively calculating adjacent correlation coefficients of the measured data before the moment of each water level measured data; and respectively comparing the residual value of each water level measurement data with a preset gross error detection threshold, comparing the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold, and determining whether each water level measurement data is abnormal data according to the comparison result. Therefore, independent and sliced abnormal data are effectively eliminated, the elimination rate of the abnormal data is improved, and the model has better stability and reliability.
Description
Technical Field
The embodiment of the invention relates to the technical field of data preprocessing, in particular to a method, a device, equipment and a medium for detecting abnormal data in water level measurement data.
Background
In the measurement process of many complex systems, due to the limitation of environment or other various conditions, the influence factors are more, and various interference factors have a larger influence on the data, so that the measurement data contains a large amount of abnormal data. Therefore, there is a need to improve the quality and accuracy of data processing results through data preprocessing. The water level is an important parameter in various hydrological parameters, accurate measurement of the water level is realized, and the method has important significance for realizing the modernization of the hydrological technology.
The preprocessing of the measured data is mostly carried out by manual processing at present, namely, abnormal data is judged manually, and bad values are removed and repaired one by one. The measurement process has large data volume, low manual processing efficiency and time and labor waste. On the other hand, due to the influence of factors such as environment, instruments and manual operation in the measuring process, the measured data is inevitably low in accuracy, and normal values are easy to eliminate. Increasingly, algorithms such as median filtering or wavelet detection analysis are also used to preprocess data. However, the median filtering method is insensitive to small abnormal values and has certain influence on details, so that in a flat terrain, when the abnormal values are distributed more densely, namely in a cluster distribution, the detection effect of the abnormal values is poor. The wavelet detection analysis method decomposes signals according to the frequency, so that the detection effect on continuous multipoint abnormal values is poor, and the abnormal values cannot be positioned, so that the effect of post-judgment processing is influenced.
Disclosure of Invention
The embodiment of the invention provides a method, a device, equipment and a medium for detecting abnormal data in water level measurement data, which are used for effectively detecting independent and fragmented abnormal data and improving the rate of distinguishing the abnormal data.
In a first aspect, an embodiment of the present invention provides a method for detecting abnormal data in water level measurement data, where the method includes:
acquiring water level measurement data to be detected;
respectively inputting each water level measurement data into a correspondingly trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data, and calculating a residual value of each water level measurement data according to the fitting value;
respectively calculating adjacent correlation coefficients of the measured data before the moment of each water level measured data;
and respectively comparing the residual value of each water level measurement data with a preset gross error detection threshold, comparing the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold, and determining whether each water level measurement data is abnormal data according to a comparison result.
Optionally, before the step of inputting each water level measurement data into the corresponding trained inverse propagation neural network for fitting, the method further includes:
and constructing the inverse propagation neural network, and optimizing the inverse propagation neural network based on a genetic algorithm to obtain the optimal weight and the optimal threshold of the inverse propagation neural network.
Optionally, the constructing the inverse propagation neural network includes:
and aiming at each water level measurement data, determining training sample data of the corresponding inverse propagation neural network by adopting a de-self method, so as to train the inverse propagation neural network by using the training sample data.
Optionally, the optimizing the inverse propagation neural network based on the genetic algorithm to obtain the optimal weight and the optimal threshold of the inverse propagation neural network includes:
initializing the reverse propagation neural network, and encoding the weight and the threshold value of the reverse propagation neural network into a genetic algorithm chromosome which is marked as an initial population;
substituting the coded inverse propagation neural network into corresponding training sample data to carry out cross recombination and variation so as to form a new population;
and establishing a fitness function according to the chromosome coding and the neural network error function, terminating the genetic algorithm if the result of the fitness function meets a preset fitness condition, selecting the best individual to decode to obtain the optimal weight and the optimal threshold value, and continuing to perform cross recombination and variation if the result of the fitness function does not meet the preset fitness condition until the result meets the preset fitness condition.
Optionally, before the step of inputting each water level measurement data into the corresponding trained inverse propagation neural network for fitting, the method further includes:
and segmenting the water level measurement data according to the height of the water level measurement data, and removing the water level measurement data with the deviation larger than a preset deviation threshold value in each segment of data by adopting a polynomial fitting method.
Optionally, the removing the water level measurement data with the deviation greater than the preset deviation threshold in each segment of data by respectively adopting a polynomial fitting method includes:
for each segment of data, performing polynomial fitting on the water level measurement data to obtain a coefficient matrix and a fitting polynomial, and calculating to obtain a corresponding fitting value sequence and a fitting residual value sequence;
and according to the fitting residual value sequence, if the residual value of the water level measurement data is smaller than a preset threshold value, judging that the water level measurement data is normal data, if the residual value of the water level measurement data is larger than or equal to the preset threshold value, judging that the water level measurement data is abnormal data, and replacing the water level measurement data with the median of the measurement data before the moment of the water level measurement data.
Optionally, after determining whether each of the water level measurement data is abnormal data according to the comparison result, the method further includes:
and if the water level measurement data are determined to be abnormal data, removing the water level measurement data, and performing time-by-time fitting on normal data by adopting a polynomial interpolation method so as to repair and compensate the removed water level measurement data.
In a second aspect, an embodiment of the present invention further provides an apparatus for detecting abnormal data in water level measurement data, where the apparatus includes:
the measurement data acquisition module is used for acquiring water level measurement data to be detected;
the residual value calculation module is used for inputting each water level measurement data into a corresponding trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data and calculating a residual value of each water level measurement data according to the fitting value;
the correlation coefficient calculation module is used for respectively calculating adjacent correlation coefficients of the measured data before the moment of each water level measured data;
and the abnormal data judgment module is used for respectively comparing the residual value of each water level measurement data with a preset gross error detection threshold, comparing the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold, and determining whether each water level measurement data is abnormal data according to a comparison result.
In a third aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the method for detecting abnormal data in the water level measurement data provided by any embodiment of the invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for detecting abnormal data in water level measurement data provided in any embodiment of the present invention.
The embodiment of the invention provides a method for detecting abnormal data in water level measurement data, which comprises the steps of firstly obtaining the water level measurement data to be detected, then respectively inputting each water level measurement data into a corresponding trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data, calculating a residual value of each water level measurement data according to the fitting value, and respectively calculating adjacent correlation coefficients of the measurement data before the moment of each water level measurement data, so that the residual value and the corresponding adjacent correlation coefficient of each water level measurement data are respectively compared with a preset gross error detection threshold and a preset correlation threshold, and whether each water level measurement data is abnormal data or not is determined according to a comparison result. According to the method for detecting the abnormal data in the water level measurement data, provided by the embodiment of the invention, aiming at the complex measurement environment, the residual value is calculated by fitting the water level measurement data by using the inverse propagation neural network, the continuity of the water level is considered, the correlation between the measurement data at adjacent moments is analyzed, and the abnormal data is longitudinally detected, so that the independent and flaky abnormal data can be effectively identified, the removing rate of the abnormal data is improved, and meanwhile, the model has better stability and reliability.
Drawings
Fig. 1 is a flowchart of a method for detecting abnormal data in water level measurement data according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a device for detecting abnormal data in water level measurement data according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1 is a flowchart of a method for detecting abnormal data in water level measurement data according to an embodiment of the present invention. The embodiment is applicable to the situation that abnormal data existing in water level measurement data of various water flows are detected and removed, and the method can be executed by the detection device for the abnormal data in the water level measurement data provided by the embodiment of the invention, and the device can be realized by hardware and/or software and can be generally integrated in computer equipment. As shown in fig. 1, the method specifically comprises the following steps:
and S11, acquiring water level measurement data to be detected.
Specifically, any feasible manner may be adopted to obtain the water level measurement data in any scene, and the method provided in this embodiment may be adopted to preprocess the water level measurement data after obtaining the water level measurement data, so as to detect abnormal data therein, without specific limitation in this embodiment. The water level measurement data may be time series data, that is, there is a corresponding time series, and each water level measurement data corresponds to a sampling time point in the time series.
And S12, inputting the water level measurement data into a correspondingly trained inverse propagation neural network for fitting to obtain a fitting value of the water level measurement data, and calculating a residual value of the water level measurement data according to the fitting value.
Specifically, the corresponding inverse propagation neural network can be trained for each water level measurement data in advance, after the water level measurement data to be detected are obtained, the water level measurement data can be sequentially input into the corresponding trained inverse propagation neural network for fitting, so that fitting values of each water level measurement data are respectively obtained, and then the fitting values and corresponding actual observation values can be differenced, so that residual values of each water level measurement data can be obtained.
Optionally, before the step of inputting each water level measurement data into the corresponding trained inverse propagation neural network for fitting, the method further includes: and constructing the inverse propagation neural network, and optimizing the inverse propagation neural network based on a genetic algorithm to obtain the optimal weight and the optimal threshold of the inverse propagation neural network. Specifically, the reverse propagation neural networks corresponding to the water level measurement data can be respectively constructed according to the time sequence order of the water level measurement data, and the reverse propagation neural networks are optimized based on the genetic algorithm by using respective training sample data, specifically, the input weights and the threshold values of the neurons in each layer are optimized, so that the output of the reverse propagation neural networks is as close to the expected output as possible.
Further optionally, the constructing the inverse propagation neural network includes: and aiming at each water level measurement data, determining training sample data of the corresponding inverse propagation neural network by adopting a de-self method, so as to train the inverse propagation neural network by using the training sample data. Specifically, for each piece of water level measurement data, the current water level measurement data may be removed, and then the water level measurement data before the time of the water level measurement data is used as training sample data of the corresponding inverse propagation neural network, that is, the training sample data is determined by a de-self method, so that the obtained training sample data is used for training the corresponding inverse propagation neural network. After the training sample data is determined, normalization processing can be performed on the training sample data first to facilitate subsequent data processing, and the training sample data is assumed to be { x }1,x2,…,xnAnd then, the normalized data is as follows:
wherein x isminRepresenting the minimum value, x, in the training sample datamaxRepresenting the maximum value in the training sample data.
Further optionally, the optimizing the inverse propagation neural network based on the genetic algorithm to obtain an optimal weight and an optimal threshold of the inverse propagation neural network includes: initializing the reverse propagation neural network, and encoding the weight and the threshold value of the reverse propagation neural network into a genetic algorithm chromosome which is marked as an initial population; substituting the coded inverse propagation neural network into corresponding training sample data to carry out cross recombination and variation so as to form a new population; and establishing a fitness function according to the chromosome coding and the neural network error function, terminating the genetic algorithm if the result of the fitness function meets a preset fitness condition, selecting the best individual to decode to obtain the optimal weight and the optimal threshold value, and continuing to perform cross recombination and variation if the result of the fitness function does not meet the preset fitness condition until the result meets the preset fitness condition. Specifically, for each water level measurement data, after determining training sample data, fitting training can be performed on a corresponding inverse propagation neural network by using the training sample data, firstly, the weight and the threshold of the inverse propagation neural network can be initialized, a weight matrix and a threshold vector of the inverse propagation neural network are encoded into a genetic algorithm chromosome and recorded as an initial population, then the encoded inverse propagation neural network can be substituted into the training sample data to perform operations such as cross recombination and variation, a new population is formed, a fitness function f is established according to chromosome encoding and a neural network error function, wherein E is a neural network error function value, so that corresponding fitness can be calculated, if the fitness meets a preset fitness condition, the genetic algorithm can be terminated, and the optimal individual can be selected for decoding to obtain the optimal weight and the optimal threshold, if the fitness does not meet the preset fitness condition, the operations of cross recombination, mutation and the like can be continuously carried out, a new population is formed, whether the preset fitness condition is met or not is judged in the same way, if so, the genetic algorithm can be stopped, and if not, the process can be continuously repeated until the preset fitness condition is met. Wherein an error function can be defined as:
wherein d isk(i) Representing the desired output, xk(i) Representing the actual observed value, i.e. training sample data, m represents the number of output layer neurons of the inverse propagation neural network, and n represents the number of training sample data. Further, the weight matrix and the threshold vector may be updated using the following equations:
wherein,
a matrix of weights is represented by a matrix of weights,
representing a threshold vector and alpha a learning rate.
And S13, respectively calculating the adjacent correlation coefficients of the measured data before the moment of each water level measured data.
Specifically, according to the continuity of the water level measurement data, if the correlation between the water level measurement data measured at adjacent times is high, the adjacent correlation coefficient corresponding to each water level measurement data can be respectively calculated, so that the water level measurement data can be longitudinally checked according to the adjacent correlation coefficient, specifically, for each water level measurement data, the current water level measurement data is firstly removed, and then the adjacent correlation coefficient between the water level measurement data before the time at which the current water level measurement data is located is calculated as the corresponding adjacent correlation coefficient. The adjacent correlation coefficient can be calculated by adopting the following formula:
wherein r represents an adjacent correlation coefficient, XiIndicating water level measurement data before the time of the current water level measurement data, n indicating the total number of water level measurement data before the time of the current water level measurement data,
and an average water level value representing the water level measurement data before the time when the current water level measurement data is present.
S14, respectively comparing the residual value of each water level measurement data with a preset gross error detection threshold value, comparing the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold value, and determining whether each water level measurement data is abnormal data according to the comparison result.
Specifically, after the residual value and the corresponding adjacent correlation coefficient of each water level measurement data are determined, the residual value and the preset coarse difference check threshold value may be compared, the corresponding adjacent correlation coefficient and the preset correlation threshold value may be compared, and if the residual value of a certain water level measurement data exceeds the preset coarse difference check threshold value or the corresponding adjacent correlation coefficient does not reach the preset correlation threshold value, the water level measurement data may be determined as abnormal data, so as to be processed subsequently.
On the basis of the above technical solution, optionally, before the respectively inputting each water level measurement data into the corresponding trained inverse propagation neural network for fitting, the method further includes: and segmenting the water level measurement data according to the height of the water level measurement data, and removing the water level measurement data with the deviation larger than a preset deviation threshold value in each segment of data by adopting a polynomial fitting method. Specifically, before the abnormal data detection method is adopted, a polynomial fitting method can be adopted to eliminate water level measurement data with large deviation from the obtained water level measurement data so as to avoid interference of gross errors, and the data volume can be compressed, so that the efficiency of subsequent abnormal data judgment is improved. The water level measurement data can be segmented according to different heights, the data in each segment are relatively close, and at the moment, a polynomial fitting method can be adopted to detect the data in each segment so as to eliminate the water level measurement data with the deviation larger than a preset deviation threshold value.
Further optionally, the removing the water level measurement data with the deviation greater than the preset deviation threshold from each segment of data by respectively adopting a polynomial fitting method includes: for each segment of data, performing polynomial fitting on the water level measurement data to obtain a coefficient matrix and a fitting polynomial, and calculating to obtain a corresponding fitting value sequence and a fitting residual value sequence; and according to the fitting residual value sequence, if the residual value of the water level measurement data is smaller than a preset threshold value, judging that the water level measurement data is normal data, if the residual value of the water level measurement data is larger than or equal to the preset threshold value, judging that the water level measurement data is abnormal data, and replacing the water level measurement data with the median of the measurement data before the moment of the water level measurement data. Specifically, for each section of data, firstly, the water level measurement data in the section of data is subjected to n-order polynomial fitting, wherein n can be preset according to the requirement to obtain a coefficient matrix and a fitting polynomial, then a corresponding fitting value sequence is obtained by calculation according to the coefficient matrix and the fitting polynomial, and a fitting residual value sequence is further obtained by calculation, i.e. including the residual values of the respective water level measurement data, so that the respective residual values can be compared with a preset threshold value, and if the residual values are smaller than the preset threshold value, judging that the corresponding water level measurement data is normal data, if the residual error value is greater than or equal to a preset threshold value, then the corresponding water level measurement data is judged to be abnormal data, and the abnormal data can be removed, and the median of the measurement data before the moment of the water level measurement data can be used to replace the water level measurement data so as to repair the water level measurement data.
On the basis of the above technical solution, optionally, after determining whether each of the water level measurement data is abnormal data according to the comparison result, the method further includes: and if the water level measurement data are determined to be abnormal data, removing the water level measurement data, and performing time-by-time fitting on normal data by adopting a polynomial interpolation method so as to repair and compensate the removed water level measurement data. Specifically, after the abnormal data is determined, the abnormal data may be removed first, and then the abnormal data is repaired and compensated by a polynomial interpolation method, and specifically, the compensated data may be obtained by performing time-to-time fitting on the remaining normal data, and specifically, the following formula may be adopted:
according to the technical scheme provided by the embodiment of the invention, water level measurement data needing to be detected are firstly obtained, then, the water level measurement data are respectively input into a corresponding trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data, a residual error value of each water level measurement data is calculated according to the fitting value, adjacent correlation coefficients of the measurement data before the moment of each water level measurement data are respectively calculated, and therefore, the residual error value and the corresponding adjacent correlation coefficient of each water level measurement data are respectively compared with a preset gross error detection threshold value and a preset correlation threshold value, and whether each water level measurement data is abnormal data or not is determined according to a comparison result. Aiming at the complex situation of the measurement environment, the inverse propagation neural network is used for fitting the water level measurement data to calculate the residual value, the continuity of the water level is considered, the correlation between the measurement data at adjacent moments is analyzed, and the abnormal data is longitudinally checked, so that the independent and flaky abnormal data can be effectively identified, the elimination rate of the abnormal data is improved, and meanwhile, the model has better stability and reliability.
Example two
Fig. 2 is a schematic structural diagram of a device for detecting abnormal data in water level measurement data according to a second embodiment of the present invention, where the device may be implemented by hardware and/or software, and may be generally integrated in a computer device. As shown in fig. 2, the apparatus includes:
the measurement data acquisition module 21 is used for acquiring water level measurement data to be detected;
a residual value calculation module 22, configured to input each of the water level measurement data into a corresponding trained inverse propagation neural network for fitting, so as to obtain a fitting value of each of the water level measurement data, and calculate a residual value of each of the water level measurement data according to the fitting value;
a correlation coefficient calculation module 23, configured to calculate adjacent correlation coefficients of measurement data before the time of each water level measurement data;
an abnormal data determining module 24, configured to compare the residual value of each water level measurement data with a preset coarse difference detection threshold, compare the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold, and determine whether each water level measurement data is abnormal data according to a comparison result.
According to the technical scheme provided by the embodiment of the invention, water level measurement data needing to be detected are firstly obtained, then, the water level measurement data are respectively input into a corresponding trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data, a residual error value of each water level measurement data is calculated according to the fitting value, adjacent correlation coefficients of the measurement data before the moment of each water level measurement data are respectively calculated, and therefore, the residual error value and the corresponding adjacent correlation coefficient of each water level measurement data are respectively compared with a preset gross error detection threshold value and a preset correlation threshold value, and whether each water level measurement data is abnormal data or not is determined according to a comparison result. Aiming at the complex situation of the measurement environment, the inverse propagation neural network is used for fitting the water level measurement data to calculate the residual value, the continuity of the water level is considered, the correlation between the measurement data at adjacent moments is analyzed, and the abnormal data is longitudinally checked, so that the independent and flaky abnormal data can be effectively identified, the elimination rate of the abnormal data is improved, and meanwhile, the model has better stability and reliability.
On the basis of the above technical solution, optionally, the apparatus for detecting abnormal data in water level measurement data further includes:
and the neural network training module is used for constructing the inverse propagation neural network before the water level measurement data are respectively input into the correspondingly trained inverse propagation neural network for fitting, and optimizing the inverse propagation neural network based on a genetic algorithm so as to obtain the optimal weight and the optimal threshold of the inverse propagation neural network.
On the basis of the above technical solution, optionally, the neural network training module includes:
and the sample data determining unit is used for determining training sample data of the corresponding inverse propagation neural network by adopting a de-self method aiming at each water level measurement data so as to train the inverse propagation neural network by using the training sample data.
On the basis of the above technical solution, optionally, the neural network training module further includes:
the network parameter coding unit is used for initializing the inverse propagation neural network, coding the weight and the threshold value of the inverse propagation neural network into a genetic algorithm chromosome and recording the genetic algorithm chromosome as an initial population;
the cross mutation unit is used for substituting the coded reverse propagation neural network into the corresponding training sample data to carry out cross recombination and mutation so as to form a new population;
and the network parameter determining unit is used for establishing a fitness function according to the chromosome coding and the neural network error function, terminating the genetic algorithm if the result of the fitness function meets a preset fitness condition, selecting the best individual for decoding to obtain the optimal weight and the optimal threshold value, and continuing to perform cross recombination and mutation if the result of the fitness function does not meet the preset fitness condition until the result meets the preset fitness condition.
On the basis of the above technical solution, optionally, the apparatus for detecting abnormal data in water level measurement data further includes:
and the pre-elimination module is used for segmenting the water level measurement data according to the height of the water level measurement data before inputting the water level measurement data into the corresponding trained inverse propagation neural network for fitting, and eliminating the water level measurement data with the deviation larger than a preset deviation threshold value in each segment of data by adopting a polynomial fitting method.
On the basis of the above technical solution, optionally, the pre-elimination module includes:
the polynomial fitting unit is used for performing polynomial fitting on the water level measurement data in each section of data to obtain a coefficient matrix and a fitting polynomial, and calculating to obtain a corresponding fitting value sequence and a fitting residual value sequence;
and the abnormal data removing unit is used for judging that the water level measurement data is normal data if the residual value of the water level measurement data is smaller than a preset threshold value according to the fitting residual value sequence, judging that the water level measurement data is abnormal data if the residual value of the water level measurement data is larger than or equal to the preset threshold value, and replacing the water level measurement data with the median of the measurement data before the moment of the water level measurement data.
On the basis of the above technical solution, optionally, the apparatus for detecting abnormal data in water level measurement data further includes:
and the removing and repairing module is used for removing the water level measurement data and performing time-by-time fitting on the normal data by adopting a polynomial interpolation method to perform repairing compensation on the removed water level measurement data after determining whether each water level measurement data is abnormal data according to the comparison result and if the water level measurement data is determined to be abnormal data.
The detection device for the abnormal data in the water level measurement data provided by the embodiment of the invention can execute the detection method for the abnormal data in the water level measurement data provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
It should be noted that, in the embodiment of the apparatus for detecting abnormal data in the water level measurement data, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a computer device provided in the third embodiment of the present invention, and shows a block diagram of an exemplary computer device suitable for implementing the embodiment of the present invention. The computer device shown in fig. 3 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present invention. As shown in fig. 3, the computer apparatus includes a processor 31, a memory 32, an input device 33, and an output device 34; the number of the processors 31 in the computer device may be one or more, one processor 31 is taken as an example in fig. 3, the processor 31, the memory 32, the input device 33 and the output device 34 in the computer device may be connected by a bus or in other ways, and the connection by the bus is taken as an example in fig. 3.
The memory 32 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the detection method of abnormal data in water level measurement data in the embodiment of the present invention (for example, the measurement data acquisition module 21, the residual value calculation module 22, the correlation coefficient calculation module 23, and the abnormal data judgment module 24 in the detection device of abnormal data in water level measurement data). The processor 31 executes various functional applications and data processing of the computer device by running software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned method for detecting abnormal data in the water level measurement data.
The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the computer device, and the like. Further, the memory 32 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 32 may further include memory located remotely from the processor 31, which may be connected to a computer device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 33 may be used to acquire water level measurement data to be detected, and to generate key signal inputs and the like relating to user settings and function control of the computer apparatus. The output device 34 may include a display screen or the like, and may be used to display the abnormal data to the user and the water level measurement data after the abnormal data is removed.
Example four
An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a method for detecting abnormal data in water level measurement data, where the method includes:
acquiring water level measurement data to be detected;
respectively inputting each water level measurement data into a correspondingly trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data, and calculating a residual value of each water level measurement data according to the fitting value;
respectively calculating adjacent correlation coefficients of the measured data before the moment of each water level measured data;
and respectively comparing the residual value of each water level measurement data with a preset gross error detection threshold, comparing the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold, and determining whether each water level measurement data is abnormal data according to a comparison result.
The storage medium may be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system connected to the computer system through a network (such as the internet). The second computer system may provide the program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations, such as in different computer systems that are connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the method for detecting abnormal data in water level measurement data provided by any embodiment of the present invention.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for detecting abnormal data in water level measurement data is characterized by comprising the following steps:
acquiring water level measurement data to be detected;
respectively inputting each water level measurement data into a correspondingly trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data, and calculating a residual value of each water level measurement data according to the fitting value;
respectively calculating adjacent correlation coefficients of the measured data before the moment of each water level measured data;
and respectively comparing the residual value of each water level measurement data with a preset gross error detection threshold, comparing the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold, and determining whether each water level measurement data is abnormal data according to a comparison result.
2. The method for detecting abnormal data in water level measurement data according to claim 1, wherein before the step of inputting each water level measurement data into the corresponding trained inverse propagation neural network for fitting, the method further comprises:
and constructing the inverse propagation neural network, and optimizing the inverse propagation neural network based on a genetic algorithm to obtain the optimal weight and the optimal threshold of the inverse propagation neural network.
3. The method for detecting abnormal data in water level measurement data according to claim 2, wherein the constructing the inverse propagation neural network comprises:
and aiming at each water level measurement data, determining training sample data of the corresponding inverse propagation neural network by adopting a de-self method, so as to train the inverse propagation neural network by using the training sample data.
4. The method for detecting abnormal data in water level measurement data according to claim 3, wherein the optimizing the back propagation neural network based on the genetic algorithm to obtain the optimal weight and the optimal threshold of the back propagation neural network comprises:
initializing the reverse propagation neural network, and encoding the weight and the threshold value of the reverse propagation neural network into a genetic algorithm chromosome which is marked as an initial population;
substituting the coded inverse propagation neural network into corresponding training sample data to carry out cross recombination and variation so as to form a new population;
and establishing a fitness function according to the chromosome coding and the neural network error function, terminating the genetic algorithm if the result of the fitness function meets a preset fitness condition, selecting the best individual to decode to obtain the optimal weight and the optimal threshold value, and continuing to perform cross recombination and variation if the result of the fitness function does not meet the preset fitness condition until the result meets the preset fitness condition.
5. The method for detecting abnormal data in water level measurement data according to claim 1, wherein before the step of inputting each water level measurement data into the corresponding trained inverse propagation neural network for fitting, the method further comprises:
and segmenting the water level measurement data according to the height of the water level measurement data, and removing the water level measurement data with the deviation larger than a preset deviation threshold value in each segment of data by adopting a polynomial fitting method.
6. The method for detecting abnormal data in water level measurement data according to claim 5, wherein the removing the water level measurement data with deviation greater than a preset deviation threshold value in each section of data by respectively adopting a polynomial fitting method comprises:
for each segment of data, performing polynomial fitting on the water level measurement data to obtain a coefficient matrix and a fitting polynomial, and calculating to obtain a corresponding fitting value sequence and a fitting residual value sequence;
and according to the fitting residual value sequence, if the residual value of the water level measurement data is smaller than a preset threshold value, judging that the water level measurement data is normal data, if the residual value of the water level measurement data is larger than or equal to the preset threshold value, judging that the water level measurement data is abnormal data, and replacing the water level measurement data with the median of the measurement data before the moment of the water level measurement data.
7. The method for detecting abnormal data in water level measurement data according to claim 1, wherein after determining whether each of the water level measurement data is abnormal data according to the comparison result, the method further comprises:
and if the water level measurement data are determined to be abnormal data, removing the water level measurement data, and performing time-by-time fitting on normal data by adopting a polynomial interpolation method so as to repair and compensate the removed water level measurement data.
8. An apparatus for detecting abnormal data in water level measurement data, comprising:
the measurement data acquisition module is used for acquiring water level measurement data to be detected;
the residual value calculation module is used for inputting each water level measurement data into a corresponding trained inverse propagation neural network for fitting to obtain a fitting value of each water level measurement data and calculating a residual value of each water level measurement data according to the fitting value;
the correlation coefficient calculation module is used for respectively calculating adjacent correlation coefficients of the measured data before the moment of each water level measured data;
and the abnormal data judgment module is used for respectively comparing the residual value of each water level measurement data with a preset gross error detection threshold, comparing the adjacent correlation coefficient corresponding to each water level measurement data with a preset correlation threshold, and determining whether each water level measurement data is abnormal data according to a comparison result.
9. A computer device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method of detecting anomalous data in water level measurement data as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method of detecting abnormal data in water level measurement data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111638348.XA CN114298281A (en) | 2021-12-29 | 2021-12-29 | Method, device, equipment and medium for detecting abnormal data in water level measurement data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111638348.XA CN114298281A (en) | 2021-12-29 | 2021-12-29 | Method, device, equipment and medium for detecting abnormal data in water level measurement data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114298281A true CN114298281A (en) | 2022-04-08 |
Family
ID=80972115
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111638348.XA Pending CN114298281A (en) | 2021-12-29 | 2021-12-29 | Method, device, equipment and medium for detecting abnormal data in water level measurement data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114298281A (en) |
-
2021
- 2021-12-29 CN CN202111638348.XA patent/CN114298281A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111734961B (en) | Natural gas pipeline leakage detection method | |
| CN110852515B (en) | Water quality index prediction method based on mixed long-time and short-time memory neural network | |
| CN107153874B (en) | Water quality prediction method and system | |
| CN108710974B (en) | Water ammonia nitrogen prediction method and device based on deep belief network | |
| CN112508243B (en) | Training method and device for multi-fault prediction network model of power information system | |
| CN109784528A (en) | Water quality prediction method and device based on time series and support vector regression | |
| CN111967535B (en) | Fault diagnosis method and device for temperature sensor of grain storage management scene | |
| CN112100574A (en) | Resampling-based AAKR model uncertainty calculation method and system | |
| CN113988210A (en) | Method and device for restoring distorted data of structure monitoring sensor network and storage medium | |
| CN114553681B (en) | Device state abnormality detection method and device and computer device | |
| CN116677569A (en) | Fan tower stress monitoring and early warning method and system | |
| CN117171702A (en) | Multi-mode power grid fault detection method and system based on deep learning | |
| CN115687322A (en) | Water quality time series missing data completion method based on encoder-decoder and autoregressive generated countermeasure network | |
| CN118114118A (en) | CNDT-based fault diagnosis method for typical weapon equipment | |
| CN114818817A (en) | Weak fault recognition system and method for capacitive voltage transformer | |
| CN117150445B (en) | Settlement monitoring and evaluating method for section tunnel short-distance downward river | |
| CN116680639A (en) | Deep-learning-based anomaly detection method for sensor data of deep-sea submersible | |
| CN114298281A (en) | Method, device, equipment and medium for detecting abnormal data in water level measurement data | |
| CN116701844A (en) | Full-waveform-based high-density resistivity data screening and processing method and system | |
| CN115409245A (en) | Prediction auxiliary state estimation method, device, equipment and medium for power system | |
| CN116012634A (en) | Strip steel shape classification method, device, computer medium and equipment | |
| CN113239075A (en) | Construction data self-checking method and system | |
| CN113552444A (en) | Online setting method and device for leakage current characteristic threshold | |
| CN113159114A (en) | High-dimensional data dimension reduction cross validation analysis method based on application in NIR data | |
| CN117349770B (en) | Structural health monitoring multi-strain sensor data anomaly detection and repair method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |