CN116522086A - Data recovery and water quality detection method and device based on variation self-encoder - Google Patents
Data recovery and water quality detection method and device based on variation self-encoder Download PDFInfo
- Publication number
- CN116522086A CN116522086A CN202310493139.3A CN202310493139A CN116522086A CN 116522086 A CN116522086 A CN 116522086A CN 202310493139 A CN202310493139 A CN 202310493139A CN 116522086 A CN116522086 A CN 116522086A
- Authority
- CN
- China
- Prior art keywords
- data
- water quality
- training
- data set
- recovery
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 141
- 238000011084 recovery Methods 0.000 title claims abstract description 96
- 238000001514 detection method Methods 0.000 title claims abstract description 32
- 238000012549 training Methods 0.000 claims abstract description 126
- 238000000034 method Methods 0.000 claims abstract description 61
- 239000010865 sewage Substances 0.000 claims abstract description 53
- 230000008569 process Effects 0.000 claims abstract description 18
- 230000002779 inactivation Effects 0.000 claims abstract description 17
- 238000013145 classification model Methods 0.000 claims description 30
- 230000006870 function Effects 0.000 claims description 23
- 230000015654 memory Effects 0.000 claims description 19
- 238000003860 storage Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012372 quality testing Methods 0.000 claims 2
- 230000000694 effects Effects 0.000 abstract description 6
- 238000004519 manufacturing process Methods 0.000 abstract description 5
- 238000012545 processing Methods 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 3
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 2
- ZAMOUSCENKQFHK-UHFFFAOYSA-N Chlorine atom Chemical compound [Cl] ZAMOUSCENKQFHK-UHFFFAOYSA-N 0.000 description 2
- 239000000460 chlorine Substances 0.000 description 2
- 229910052801 chlorine Inorganic materials 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 241000894006 Bacteria Species 0.000 description 1
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 229910052782 aluminium Inorganic materials 0.000 description 1
- XAGFODPZIPBFFR-UHFFFAOYSA-N aluminium Chemical compound [Al] XAGFODPZIPBFFR-UHFFFAOYSA-N 0.000 description 1
- XKMRRTOUMJRJIA-UHFFFAOYSA-N ammonia nh3 Chemical compound N.N XKMRRTOUMJRJIA-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 description 1
- 238000011001 backwashing Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 230000009849 deactivation Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 229910052751 metal Inorganic materials 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003647 oxidation Effects 0.000 description 1
- 238000007254 oxidation reaction Methods 0.000 description 1
- 239000001301 oxygen Substances 0.000 description 1
- 229910052760 oxygen Inorganic materials 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000004062 sedimentation Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000008399 tap water Substances 0.000 description 1
- 235000020679 tap water Nutrition 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
- G06F18/15—Statistical pre-processing, e.g. techniques for normalisation or restoring missing data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A20/00—Water conservation; Efficient water supply; Efficient water use
- Y02A20/152—Water filtration
Abstract
The invention discloses a data recovery and water quality detection method and device based on a variation self-encoder, wherein the method comprises the following steps: acquiring sensor data of each link of sewage treatment, and constructing a training data set; establishing a variational self-encoder network model; inputting a training data set into the variation self-encoder network model for training to obtain a data recovery model, and performing inactivation treatment on each data node of an input layer and a first hidden layer in the process of inputting the training data set into the variation self-encoder network model for training; and inputting the first target data set of the lost part of the sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, so that the lost sensor data can be effectively recovered, production activities of all links can be guided according to the recovery data, and the accuracy of water quality detection is improved.
Description
Technical Field
The invention relates to the technical field of sewage treatment, in particular to a data recovery and water quality detection method and device based on a variation self-encoder.
Background
With the rapid development of society and economy, serious pollution problems of water resources are brought. The sewage discharge of industry and agriculture increases year by year, and the sewage treatment is related to basic folk life and is closely related to everyone. The water quality detection technology used at present in sewage plants only depends on that relevant sensors are installed in each link, and after sensor data are acquired by a data acquisition device, the sensor data are directly sent to monitoring staff, and the monitoring staff have two main works: firstly, judging whether the processing of each link meets the standard or not according to the sensors of each link, and timely carrying out compensation measures when the processing of each link does not meet the standard; and secondly, evaluating the current water quality output outwards according to the sensor data installed in the final water outlet link.
However, the sensitivity of the sensor is reduced or damaged, and part of sensor signal loss is a common problem of sewage treatment plants, and the loss of sensor data greatly influences the monitoring and evaluation of the working state of the step and the accuracy of the final water quality result.
Disclosure of Invention
Therefore, the invention aims to overcome the defect that the accuracy of the water quality detection result is affected by the data loss of the existing sensor, and further provides a data recovery and water quality detection method and device based on a variation self-encoder.
According to a first aspect, an embodiment of the present invention discloses a method for recovering data based on a variation self-encoder, the method comprising: acquiring sensor data of each link of sewage treatment, and constructing a training data set; establishing a variational self-encoder network model; inputting the training data set into the variation self-encoder network model for training to obtain a data recovery model, wherein the variation self-encoder network model comprises an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer which are sequentially connected, and in the process of inputting the training data set into the variation self-encoder network model for training, each data node of the input layer and the first hidden layer is subjected to inactivation treatment in any batch of data training in the data set; and inputting a first target data set of the lost partial sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, wherein the recovery data comprises the lost partial sensor data.
Optionally, before the training data set is input into the variational self-encoder network model for training, the method further includes: and normalizing all the sensor data in the training data set.
Optionally, the inputting the training data set into the variational self-encoder network model for training includes: inputting the training data set into the variational self-encoder network model to obtain corresponding prediction recovery data; according to the actual data corresponding to the inactivation treatment in the prediction recovery data and training data set, calculating a loss function corresponding to the variation self-encoder network model; and judging whether the loss function is smaller than a preset threshold value, if so, carrying out parameter adjustment on the variation self-encoder network model, and training the variation self-encoder network model by using a training data set again until the loss function is smaller than the preset threshold value.
According to a second aspect, the embodiment of the invention also discloses a water quality detection method, which comprises the following steps: acquiring a data set corresponding to the water quality of each link of sewage treatment, wherein the data set comprises grade labels corresponding to the water quality after sewage treatment and various parameters of the water quality; inputting the data set into a preset classification model for training to obtain a water quality classification model; and inputting the second target data set after the loss of the sensor data is recovered into the water quality classification model to obtain the water quality grade corresponding to the second target data set, wherein the second target data set after the loss of the sensor data is recovered is obtained by the data recovery method based on the variation self-encoder according to the first aspect or any optional implementation mode of the first aspect.
Optionally, determining a grade label corresponding to the water quality after sewage treatment by the following steps: obtaining scores corresponding to various parameters of water quality, and adding the scores based on weights corresponding to the various parameters of the water quality to obtain a total score of the water quality; a grade label of the water quality is determined based on the total score of the water quality.
Optionally, the method further comprises: when the water quality grade corresponding to the second target data set is lower than a preset grade threshold, respectively calculating the mean value and the variance of the sensor data corresponding to each sewage treatment link output by the data recovery model at different moments; and comparing the calculated mean value and variance corresponding to each sewage treatment link with a preset reasonable interval respectively, and determining whether each sewage treatment link is normal or not.
According to a third aspect, the embodiment of the invention also discloses a data recovery device based on the variation self-encoder, which comprises: the data acquisition module is used for acquiring sensor data of each link of sewage treatment and constructing a training data set; the model building module is used for building a variational self-encoder network model; the model training module is used for inputting the training data set into the variable self-encoder network model for training to obtain a data recovery model, the variable self-encoder network model comprises an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer which are sequentially connected, and in the process of inputting the training data set into the variable self-encoder network model for training, in any batch of data training process in the data set, each data node of the input layer and the first hidden layer is subjected to inactivation treatment; the data recovery module is used for inputting a first target data set of the lost partial sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, wherein the recovery data comprises the lost partial sensor data.
According to a fourth aspect, the embodiment of the invention also discloses a water quality detection device, which comprises: the water quality data acquisition module is used for acquiring a data set corresponding to the water quality of each link of sewage treatment, wherein the data set comprises grade labels corresponding to the water quality after sewage treatment and various parameters of the water quality; the model training module is used for inputting the data set into a preset classification model for training to obtain a water quality classification model; the water quality detection module is used for inputting a second target data set after the loss of the sensor data is recovered into the water quality classification model to obtain a water quality grade corresponding to the second target data set, and the second target data set after the loss of the sensor data is recovered is obtained by the data recovery method based on the variation self-encoder according to the first aspect or any optional implementation mode of the first aspect.
According to a fifth aspect, an embodiment of the present invention further discloses an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of recovering variable-component-self-encoder-based data as described in the first aspect or any of the alternative embodiments of the first aspect, or the steps of the method of water quality detection as described in the second aspect or any of the alternative embodiments of the second aspect.
According to a sixth aspect, the embodiment of the present invention further discloses a computer readable storage medium, on which a computer program is stored, the computer program implementing the method for recovering data based on a variation self-encoder according to the first aspect or any optional embodiment of the first aspect, or the steps of the method for detecting water quality according to the second aspect or any optional embodiment of the second aspect, when the computer program is executed by a processor.
The technical scheme of the invention has the following advantages:
according to the data recovery method based on the variation self-encoder, a training data set is constructed by acquiring sensor data of each link of sewage treatment; establishing a variational self-encoder network model; inputting a training data set into the variation self-encoder network model for training to obtain a data recovery model, and performing inactivation treatment on each data node of an input layer and a first hidden layer in the process of inputting the training data set into the variation self-encoder network model for training; and inputting the first target data set of the lost part of the sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, so that the lost sensor data can be effectively recovered, production activities of all links can be guided according to the recovery data, and the accuracy of water quality detection is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a specific example of a data recovery method based on a variation self-encoder in an embodiment of the present invention;
FIG. 2 is a diagram of one embodiment of a model of a variable self-encoder network in accordance with embodiments of the present invention;
FIG. 3 is a flowchart showing a specific example of a water quality detection method according to an embodiment of the present invention;
FIG. 4 is a functional block diagram of one specific example of a variation-from-encoder based data recovery apparatus in an embodiment of the present invention;
FIG. 5 is a schematic block diagram of a water quality detection apparatus according to an embodiment of the present invention;
fig. 6 is a diagram illustrating an embodiment of an electronic device according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; the two components can be directly connected or indirectly connected through an intermediate medium, or can be communicated inside the two components, or can be connected wirelessly or in a wired way. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
The embodiment of the invention discloses a data recovery method based on a variation self-encoder, which is shown in fig. 1 and comprises the following steps:
and 101, acquiring sensor data of each link of sewage treatment, and constructing a training data set.
Illustratively, the sewage treatment links from sewage to drainable water and even to resident usable tap water generally comprise a series of links of starting, running, adjusting, stopping and maintaining such as an adjusting tank, a hair collector, a biological contact oxidation tank, a sedimentation tank, an intermediate tank, a secondary lifting pump, a mechanical filter, an activated carbon filter, a backwashing water pump, a clean water tank and the like, and the corresponding sensors comprise a water flow sensor, a PH sensor, a water level altimeter, a thermometer, an oxygen content detector, a residual chlorine analyzer, a Bod/Cod analyzer, an ammonia nitrogen detector, a phosphide detector and the like, and the data acquired by the sensors can be acquired in real time by way of example only, or historical data in a sensor database can be directly acquired, and the acquired sensor data can form a one-dimensional vector, such as 137 sensors are involved in the sewage treatment links, namely 137 corresponding data are acquired, and the one-dimensional vector dimension is 137 by way of example only.
Step 102, establishing a variational self-encoder network model.
Illustratively, the variational self-encoder network model VAE is capable of actively generating and inputting very similar data by extracting and observing features of input data and adding normal distribution, in this embodiment, the variational self-encoder network model is composed of two parts of an encoder and a decoder, as shown in fig. 2, the encoder part is composed of an input layer, a first hidden layer and a second hidden layer, the decoder part is composed of a third hidden layer, a fourth hidden layer and an output layer, and the decoder part and the encoder part belong to a symmetrical structure, in the specific embodiment, the obtained one-dimensional vector 137 is input to the input layer, then the dimension of the input layer is [137,1], the dimension of the first hidden layer is [1024,1], the dimension of the second hidden layer is [256,1], the dimension of the intermediate coding result is [10,1], the dimension of the third hidden layer is [256,1], the dimension of the fourth hidden layer is [1024,1], and the dimension of the output layer is [137,1], which can be built by using, by way of example only, PYTHON and PYToch as an example.
And step 103, inputting the training data set into the variational self-encoder network model for training to obtain a data recovery model.
The variable self-encoder network model comprises an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer which are sequentially connected.
And in the training process of inputting the training data set into the variational self-encoder network model for training, carrying out inactivation treatment on each data node of the input layer and the first hidden layer in any batch of data training process in the data set.
The method for performing the inactivation treatment on the data nodes is not limited, for example, the random partial area inactivation of the convolution layer can be performed by using a SpatialDropout method, or the inactivation of a block can be performed by selecting DropBlock, so that the inactivation treatment on the image can be performed, and the method can be determined by itself based on practical situations, wherein the probability that each data node of the neural network has P is abandoned by adopting dropoout, the probability of 1-P is reserved, the output of the node is regarded as zero, the error information of partial neurons can be corrected by adopting dropoout, the overfitting problem of a model can be effectively relieved, the regularization effect is achieved, and the co-adaptability among the nodes is prevented.
In the training process of inputting the training data set into the variable self-encoder network model, for example, the number of samples of the training data set is 300, when each batch of data is trained, dropout can be adopted to randomly discard the data node at the input layer with lower probability, meanwhile, a probability can be set at the first hidden layer to randomly discard the data node, the specific probability setting can be automatically determined according to actual conditions, the output data set corresponding to the training data set can be obtained, and the data recovery model can be obtained by carrying out multiple batches of training, for example, training of 200 rounds of models to achieve convergence.
And 104, inputting a first target data set of the lost partial sensor data into the data recovery model to obtain recovery data corresponding to the first target data set.
Wherein the recovery data includes the lost portion of sensor data.
In an exemplary embodiment, when a part of sensor data is lost in the first target data set obtained in the embodiment of the present application, when the first target data set is input into the data recovery model, a data node corresponding to the lost sensor data may be set to 0, and because Dropout has undergone a great deal of training on the condition that information is lost to the data node in the training process, the information of the current data node has less influence on the result of encoding and decoding the data recovery model, and finally, the result output by the data recovery model decoder, that is, the lost part of sensor data is effectively recovered, so that recovery data corresponding to the first target data set may be obtained.
According to the data recovery method based on the variation self-encoder, a training data set is constructed by acquiring sensor data of each link of sewage treatment; establishing a variational self-encoder network model; inputting a training data set into the variation self-encoder network model for training to obtain a data recovery model, and performing inactivation treatment on each data node of an input layer and a first hidden layer in the process of inputting the training data set into the variation self-encoder network model for training; and inputting the first target data set of the lost part of the sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, so that the lost sensor data can be effectively recovered, production activities of all links can be guided according to the recovery data, and the accuracy of water quality detection is improved.
As an optional embodiment of the invention, before the training data set is input into the variational self-encoder network model for training, the method further includes: and normalizing all the sensor data in the training data set.
For example, the data in the training data set is collected by different sensors, and the range of the data may have a great gap due to different ranges of the different sensors, and the embodiment of the application can normalize all the sensor data in the training data set by the following formula to ensure that all the sensor data are between [0,1 ]:
wherein x represents a certain data value of any one sensor; x_min represents the minimum of all data in any one sensor; x_max represents the maximum value among all data of any one sensor; x_norm represents a normalized data value corresponding to a certain data value.
According to the embodiment of the application, the professional can directly define the numerical range of the sensor which accords with the actual for the relevant sensor of each link according to the actual production experience of the water plant, the sensor can directly acquire the data in the preset range when performing data acquisition, and the training difficulty of the model can be reduced by normalizing the data of the input variation self-encoder network model by taking the data as an example only, so that the model is easier to converge, and the efficiency of obtaining the data recovery model is improved.
As an optional embodiment of the present invention, the inputting the training data set into the variational self-encoder network model for training includes: inputting the training data set into the variational self-encoder network model to obtain corresponding prediction recovery data; according to the actual data corresponding to the inactivation treatment in the prediction recovery data and training data set, calculating a loss function corresponding to the variation self-encoder network model; and judging whether the loss function is smaller than a preset threshold value, if so, carrying out parameter adjustment on the variation self-encoder network model, and training the variation self-encoder network model by using a training data set again until the loss function is smaller than the preset threshold value.
Illustratively, the loss function used to measure the inconsistency of the predicted recovery value and the true value of the model may include two parts: the method comprises the steps of firstly measuring the direct distance between input and output, using mean square loss, secondly, evaluating the loss obtained by the fact that the intermediate coding result accords with normal distribution, and obtaining the integral loss function by adding the distribution of the coding result and KL divergence between standard gauss (zero mean and unit variance).
According to the embodiment of the application, the training data set is input into the variable self-encoder network model for training, prediction recovery data corresponding to the training data set can be obtained, calculation is carried out according to the obtained prediction recovery data and the actual data before discarding, a loss function corresponding to the batch of training process is obtained, whether the loss function obtained in the batch of training process is smaller than a preset threshold value is judged, wherein the preset threshold value can be determined based on human experience, if the loss function is smaller than the preset threshold value, the result obtained through training meets the requirement, if the loss function is larger than the preset threshold value, parameters of the variable self-encoder network model also need to be adjusted, the variable self-encoder network model is trained by utilizing the training set again until the loss function is smaller than the preset threshold value, and whether the variable self-encoder network model is trained or not is judged by calculating the loss function, so that the accuracy of data recovery of the data recovery model can be improved.
The embodiment of the invention also discloses a water quality detection method, which is shown in figure 3 and comprises the following steps:
step 201, acquiring a data set corresponding to the water quality of each link of sewage treatment.
The data set comprises grade labels corresponding to the water quality after sewage treatment and various parameters of the water quality.
By way of example, the water quality design parameters of the sewage treatment plant may include total bacteria number, total chlorine, chromaticity, turbidity, metal content such as aluminum and copper, reflectivity index and limit value, etc., and only by way of example, the data set corresponding to the water quality of each link of sewage treatment may be obtained in real time through the sensor, where the data set includes each parameter of the water quality of each link, the grade label corresponding to the treated water quality may be set to 1-7 seven grades, and the grade of the water quality may be determined by detecting the water quality through artificial experience, and only by way of example.
As an alternative embodiment of the present invention, the grade label corresponding to the water quality after sewage treatment is determined by the steps of: obtaining scores corresponding to various parameters of water quality, and adding the scores based on weights corresponding to the various parameters of the water quality to obtain a total score of the water quality; a grade label of the water quality is determined based on the total score of the water quality.
For example, the grade label corresponding to the water quality may be different weights corresponding to each parameter given by a monitoring person with abundant experience in a water plant based on actual conditions, the corresponding scoring may be performed on each parameter of the water quality, the total score of the water quality may be obtained by adding the weights corresponding to each parameter of the water quality, and the corresponding grade may be determined according to the total score, for example, in a range of scores within a range of 1-7.
Step 202, inputting the data set into a preset classification model for training to obtain a water quality classification model.
The preset classification model in the embodiment of the present application may be a multi-layer perceptron network classification model, and may utilize a ReLU (activating function) as a nonlinear unit to improve network expression capability, where the network structure may be 4 layers, the dimensions are 137, 1024, 128, and 7, where 7 represents 7 levels of final output, and 7 levels are probabilities of levels corresponding to input water quality parameters calculated by using a Softmax function, where the level corresponding to the highest probability node is the input water quality level, the obtained dataset is input into the preset classification model to perform training, whether the level obtained after training of the dataset is consistent with a pre-determined water quality level is determined, if consistent, the training of the preset classification model is finished, and if inconsistent, parameters of the preset classification model may be adjusted, retrained until the level obtained after training of the dataset is consistent with the pre-determined water quality level, which is merely taken as an example.
And 203, inputting the second target data set after the loss of the sensor data is recovered into the water quality classification model to obtain a water quality grade corresponding to the second target data set, wherein the second target data set after the loss of the sensor data is recovered is obtained by the data recovery method based on the variation self-encoder.
Illustratively, in the embodiment of the present application, the second target data set after the loss sensor data recovery is input into the trained water quality classification model, so as to obtain a water quality grade corresponding to the second target data set.
According to the water quality detection method provided by the invention, the data set corresponding to the water quality of each link of sewage treatment is obtained, the data set is input into the preset classification model for training, the water quality classification model is obtained, the second target data set after the sensor data is recovered is input into the water quality classification model, the water quality grade corresponding to the second target data set is obtained, the automatic 24-hour real-time monitoring of the water quality grade can be realized, the condition that whether each index is normal or not is not needed to be frequently paid attention to by manpower is not needed, the labor cost is low, and the accuracy of water quality detection can be improved.
As an optional embodiment of the present invention, the method further comprises: when the water quality grade corresponding to the second target data set is lower than a preset grade threshold, respectively calculating the mean value and the variance of the sensor data corresponding to each sewage treatment link output by the data recovery model at different moments; and comparing the calculated mean value and variance corresponding to each sewage treatment link with a preset reasonable interval respectively, and determining whether each sewage treatment link is normal or not.
The embodiment of the application can preset a normal level threshold value of water quality treatment according to human experience, compare the water quality level corresponding to the second target data set with the preset level threshold value after obtaining the water quality level corresponding to the second target data set, if the output water quality level is lower than the preset level threshold value, indicate that a problem occurs in a certain link of sewage treatment, automatically send out an alarm, output sensor data corresponding to each sewage treatment link output by a data recovery model at different moments, for example, the data recovery model outputs the sensor data of each link 30 times to perform mean value and variance calculation, assume that a group of detection data contains random errors according to a 3 sigma rule, calculate and process the detection data to obtain standard deviation, determine a reasonable interval of a mean value and a reasonable interval of variance according to a certain probability, and if the calculated variance exceeds the reasonable interval or the calculated mean value exceeds the reasonable interval, indicate that the sewage treatment link is abnormal, independently control each treatment link, and can quickly find out which link of sewage treatment has the problem, so that an operation and maintenance personnel can quickly adjust according to the data.
The embodiment of the invention also discloses a data recovery device based on the variation self-encoder, as shown in fig. 4, which comprises:
the data acquisition module 301 is configured to acquire sensor data of each link of sewage treatment, and construct a training data set;
a model building module 302, configured to build a variational self-encoder network model;
the model training module 303 is configured to input the training dataset into the variable self-encoder network model for training to obtain a data recovery model, where the variable self-encoder network model includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer, and an output layer that are sequentially connected, and in the process of inputting the training dataset into the variable self-encoder network model for training, in any batch of data training in the dataset, each data node of the input layer and the first hidden layer is subjected to deactivation processing;
the data recovery module 304 is configured to input a first target data set of the lost portion of sensor data into the data recovery model, to obtain recovery data corresponding to the first target data set, where the recovery data includes the lost portion of sensor data.
The data recovery device based on the variation self-encoder constructs a training data set by acquiring sensor data of each link of sewage treatment; establishing a variational self-encoder network model; inputting a training data set into the variation self-encoder network model for training to obtain a data recovery model, and performing inactivation treatment on each data node of an input layer and a first hidden layer in the process of inputting the training data set into the variation self-encoder network model for training; and inputting the first target data set of the lost part of the sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, so that the lost sensor data can be effectively recovered, production activities of all links can be guided according to the recovery data, and the accuracy of water quality detection is improved.
As an alternative embodiment of the present invention, the apparatus further comprises: and the data processing module is used for carrying out normalization processing on all the sensor data in the training data set.
As an alternative embodiment of the present invention, the model training module includes: the data recovery sub-module is used for inputting the training data set into the variational self-encoder network model to obtain corresponding prediction recovery data; the loss function calculation sub-module is used for calculating the loss function corresponding to the variation self-encoder network model according to the actual data corresponding to the inactivation treatment in the prediction recovery data and training data set; and the model adjustment sub-module is used for judging whether the loss function is smaller than a preset threshold value, if so, carrying out parameter adjustment on the variation self-encoder network model, and training the variation self-encoder network model by using a training data set again until the loss function is smaller than the preset threshold value.
The embodiment of the invention also discloses a water quality detection device, which is characterized in that as shown in fig. 5, the device comprises:
the water quality data acquisition module 501 is configured to acquire a data set corresponding to water quality in each link of sewage treatment, where the data set includes a grade label corresponding to water quality after sewage treatment and various parameters of water quality;
the model training module 502 is configured to input the data set into a preset classification model for training, so as to obtain a water quality classification model;
the water quality detection module 503 is configured to input a second target data set after the loss of the sensor data is recovered into the water quality classification model, to obtain a water quality level corresponding to the second target data set, where the second target data set after the loss of the sensor data is recovered is obtained by the data recovery method based on the variable-component self-encoder described in the foregoing embodiment.
According to the water quality detection device provided by the invention, the data set corresponding to the water quality of each link of sewage treatment is obtained, the data set is input into the preset classification model for training, the water quality classification model is obtained, the second target data set after the sensor data is recovered is input into the water quality classification model, the water quality grade corresponding to the second target data set is obtained, the automatic 24-hour real-time monitoring of the water quality grade can be realized, the condition that whether each index is normal or not is not needed to be frequently paid attention to by manpower is not needed, the labor cost is low, and the accuracy of water quality detection can be improved.
As an alternative embodiment of the present invention, the water quality data acquisition module includes: the total score determining submodule is used for obtaining scores corresponding to various parameters of the water quality, and adding the scores based on weights corresponding to the various parameters of the water quality to obtain the total score of the water quality; the grade determining submodule is used for determining grade labels of the water quality based on the total score of the water quality.
As an alternative embodiment of the present invention, the apparatus further comprises: the data calculation module is used for respectively calculating the mean value and the variance of the sensor data corresponding to each sewage treatment link output by the data recovery model at different moments when the water quality level corresponding to the second target data set is lower than a preset level threshold; the treatment link judging module is used for comparing the calculated mean value and the calculated variance corresponding to each sewage treatment link with a preset reasonable interval respectively to determine whether each sewage treatment link is normal or not.
The embodiment of the present invention further provides an electronic device, as shown in fig. 6, which may include a processor 401 and a memory 402, where the processor 401 and the memory 402 may be connected by a bus or other means, and in fig. 6, the connection is exemplified by a bus.
The processor 401 may be a central processing unit (Central Processing Unit, CPU). The processor 401 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.
The memory 402, as a non-transitory computer readable storage medium, may be used to store a non-transitory software program, a non-transitory computer executable program, and a module, such as a program instruction/module corresponding to a data recovery method or a water quality detection method based on a variation self-encoder in an embodiment of the present invention. The processor 401 executes various functional applications of the processor and data processing, i.e., implements the variable self-encoder-based data recovery method or the water quality detection method in the above-described method embodiments by running non-transitory software programs, instructions, and modules stored in the memory 402.
Memory 402 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created by the processor 401, or the like. In addition, memory 402 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, such remote memory being connectable to processor 401 through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 402, which when executed by the processor 401, perform a variation-from-encoder based data recovery method as in the embodiment of fig. 1, or a water quality detection method as in the embodiment of fig. 3.
The specific details of the electronic device may be understood correspondingly with reference to the corresponding related descriptions and effects in the embodiments shown in fig. 1 or fig. 3, which are not repeated herein.
It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, where the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. The storage medium may be a magnetic Disk, an optical disc, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a flash Memory (flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope as defined.
Claims (10)
1. A method of data recovery based on a variational self-encoder, the method comprising:
acquiring sensor data of each link of sewage treatment, and constructing a training data set;
establishing a variational self-encoder network model;
inputting the training data set into the variation self-encoder network model for training to obtain a data recovery model, wherein the variation self-encoder network model comprises an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer which are sequentially connected, and in the process of inputting the training data set into the variation self-encoder network model for training, each data node of the input layer and the first hidden layer is subjected to inactivation treatment in any batch of data training in the data set;
and inputting a first target data set of the lost partial sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, wherein the recovery data comprises the lost partial sensor data.
2. The method of claim 1, wherein prior to said inputting the training dataset into the variational self-encoder network model for training, the method further comprises:
and normalizing all the sensor data in the training data set.
3. The method of claim 1, wherein said inputting the training data set into the variational self-encoder network model for training comprises:
inputting the training data set into the variational self-encoder network model to obtain corresponding prediction recovery data;
according to the actual data corresponding to the inactivation treatment in the prediction recovery data and training data set, calculating a loss function corresponding to the variation self-encoder network model;
and judging whether the loss function is smaller than a preset threshold value, if so, carrying out parameter adjustment on the variation self-encoder network model, and training the variation self-encoder network model by using a training data set again until the loss function is smaller than the preset threshold value.
4. A water quality testing method, the method comprising:
acquiring a data set corresponding to the water quality of each link of sewage treatment, wherein the data set comprises grade labels corresponding to the water quality after sewage treatment and various parameters of the water quality;
inputting the data set into a preset classification model for training to obtain a water quality classification model;
inputting a second target data set after the loss of the sensor data is recovered into the water quality classification model to obtain a water quality grade corresponding to the second target data set, wherein the second target data set after the loss of the sensor data is recovered is obtained by the data recovery method based on the variation self-encoder as set forth in any one of claims 1-3.
5. The method according to claim 4, wherein the grade label corresponding to the water quality after sewage treatment is determined by:
obtaining scores corresponding to various parameters of water quality, and adding the scores based on weights corresponding to the various parameters of the water quality to obtain a total score of the water quality;
a grade label of the water quality is determined based on the total score of the water quality.
6. The method according to claim 4, wherein the method further comprises:
when the water quality grade corresponding to the second target data set is lower than a preset grade threshold, respectively calculating the mean value and the variance of the sensor data corresponding to each sewage treatment link output by the data recovery model at different moments;
and comparing the calculated mean value and variance corresponding to each sewage treatment link with a preset reasonable interval respectively, and determining whether each sewage treatment link is normal or not.
7. A data recovery apparatus based on a variational self-encoder, the apparatus comprising:
the data acquisition module is used for acquiring sensor data of each link of sewage treatment and constructing a training data set;
the model building module is used for building a variational self-encoder network model;
the model training module is used for inputting the training data set into the variable self-encoder network model for training to obtain a data recovery model, the variable self-encoder network model comprises an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer which are sequentially connected, and in the process of inputting the training data set into the variable self-encoder network model for training, in any batch of data training process in the data set, each data node of the input layer and the first hidden layer is subjected to inactivation treatment;
the data recovery module is used for inputting a first target data set of the lost partial sensor data into the data recovery model to obtain recovery data corresponding to the first target data set, wherein the recovery data comprises the lost partial sensor data.
8. A water quality testing device, the device comprising:
the water quality data acquisition module is used for acquiring a data set corresponding to the water quality of each link of sewage treatment, wherein the data set comprises grade labels corresponding to the water quality after sewage treatment and various parameters of the water quality;
the model training module is used for inputting the data set into a preset classification model for training to obtain a water quality classification model;
the water quality detection module is used for inputting a second target data set after the loss of the sensor data is recovered into the water quality classification model to obtain a water quality grade corresponding to the second target data set, and the second target data set after the loss of the sensor data is recovered is obtained by the data recovery method based on the variation self-encoder as set forth in any one of claims 1-3.
9. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the steps of the variation-from-encoder-based data recovery method of any one of claims 1-3 or the steps of the water quality detection method of any one of claims 4-6.
10. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the variation self-encoder based data recovery method of any of claims 1-3 or the steps of the water quality detection method of any of claims 4-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310493139.3A CN116522086A (en) | 2023-04-25 | 2023-04-25 | Data recovery and water quality detection method and device based on variation self-encoder |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310493139.3A CN116522086A (en) | 2023-04-25 | 2023-04-25 | Data recovery and water quality detection method and device based on variation self-encoder |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116522086A true CN116522086A (en) | 2023-08-01 |
Family
ID=87391705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310493139.3A Pending CN116522086A (en) | 2023-04-25 | 2023-04-25 | Data recovery and water quality detection method and device based on variation self-encoder |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116522086A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111157694A (en) * | 2020-01-14 | 2020-05-15 | 长沙理工大学 | Diagnostic method for quality abnormity of blowdown water of steam generator of nuclear power plant |
| CN112101789A (en) * | 2020-09-16 | 2020-12-18 | 清华大学合肥公共安全研究院 | Water pollution alarm grade identification method based on artificial intelligence |
| CN112395382A (en) * | 2020-11-23 | 2021-02-23 | 武汉理工大学 | Ship abnormal track data detection method and device based on variational self-encoder |
| CN112735541A (en) * | 2021-01-15 | 2021-04-30 | 杭州中科先进技术研究院有限公司 | Sewage treatment water quality prediction method based on simple circulation unit neural network |
| CN112884056A (en) * | 2021-03-04 | 2021-06-01 | 河北工程大学 | Optimized LSTM neural network-based sewage quality prediction method |
| CN114611336A (en) * | 2022-05-11 | 2022-06-10 | 中国农业大学 | Circulating water aquaculture dissolved oxygen prediction control method, device, equipment and medium |
| US20220292301A1 (en) * | 2021-03-10 | 2022-09-15 | Mitsubishi Electric Research Laboratories, Inc. | System and a Method for Training a Neural Network Having Autoencoder Architecture to Recover Missing Data |
| CN115469227A (en) * | 2022-09-14 | 2022-12-13 | 中南大学 | Set variational self-encoder and dynamic regular lithium battery abnormity detection method |
-
2023
- 2023-04-25 CN CN202310493139.3A patent/CN116522086A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111157694A (en) * | 2020-01-14 | 2020-05-15 | 长沙理工大学 | Diagnostic method for quality abnormity of blowdown water of steam generator of nuclear power plant |
| CN112101789A (en) * | 2020-09-16 | 2020-12-18 | 清华大学合肥公共安全研究院 | Water pollution alarm grade identification method based on artificial intelligence |
| CN112395382A (en) * | 2020-11-23 | 2021-02-23 | 武汉理工大学 | Ship abnormal track data detection method and device based on variational self-encoder |
| CN112735541A (en) * | 2021-01-15 | 2021-04-30 | 杭州中科先进技术研究院有限公司 | Sewage treatment water quality prediction method based on simple circulation unit neural network |
| CN112884056A (en) * | 2021-03-04 | 2021-06-01 | 河北工程大学 | Optimized LSTM neural network-based sewage quality prediction method |
| US20220292301A1 (en) * | 2021-03-10 | 2022-09-15 | Mitsubishi Electric Research Laboratories, Inc. | System and a Method for Training a Neural Network Having Autoencoder Architecture to Recover Missing Data |
| CN114611336A (en) * | 2022-05-11 | 2022-06-10 | 中国农业大学 | Circulating water aquaculture dissolved oxygen prediction control method, device, equipment and medium |
| CN115469227A (en) * | 2022-09-14 | 2022-12-13 | 中南大学 | Set variational self-encoder and dynamic regular lithium battery abnormity detection method |
Non-Patent Citations (1)
| Title |
|---|
| 黄晓祥 等: "基于变分自编码器的异常颈动脉早期识别和预测", 《计算机应用》, vol. 41, no. 10, pages 3082 - 3088 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111813084B (en) | Mechanical equipment fault diagnosis method based on deep learning | |
| CN112529240B (en) | Atmospheric environment data prediction method, system, device and storage medium | |
| CN115333678B (en) | Unmanned ship water quality monitoring data transmission method and system | |
| CN111210128B (en) | Wetland early warning method based on artificial intelligence and random self-adaptive threshold | |
| CN113177646B (en) | Power distribution equipment online monitoring method and system based on self-adaptive edge proxy | |
| CN115470850A (en) | Water quality abnormal event recognition early warning method based on pipe network water quality time-space data | |
| CN114385614A (en) | Water quality early warning method based on Informmer model | |
| CN111191855B (en) | Water quality abnormal event identification and early warning method based on pipe network multi-element water quality time sequence data | |
| CN115099512A (en) | Freshwater fish culture intelligent management system and management method | |
| CN114023399A (en) | Air particulate matter analysis early warning method and device based on artificial intelligence | |
| CN110119165B (en) | Aquaculture pond dissolved oxygen detection device | |
| CN116451142A (en) | Water quality sensor fault detection method based on machine learning algorithm | |
| CN116680643A (en) | Abnormality detection method combining deep contrast learning and density clustering | |
| CN109211564B (en) | Self-adaptive threshold detection method for health assessment of ball screw pair | |
| CN116522086A (en) | Data recovery and water quality detection method and device based on variation self-encoder | |
| CN113408659A (en) | Building energy consumption integrated analysis method based on data mining | |
| CN115792165B (en) | Intelligent environmental water quality monitoring method and system | |
| CN113836813B (en) | Blast furnace tuyere water leakage detection method based on data analysis | |
| CN113723670B (en) | Photovoltaic power generation power short-term prediction method with variable time window | |
| CN115994692B (en) | Intelligent river and lake management platform based on 5G and big data | |
| CN117035230B (en) | Sewage treatment equipment running state evaluation method based on big data analysis | |
| Hu et al. | Optimized convolutional neural networks for fault diagnosis in wastewater treatment processes | |
| CN116597355B (en) | Small micro water body state real-time monitoring method, system and device based on vision system | |
| CN117370919B (en) | Remote monitoring system for sewage treatment equipment | |
| CN117713688B (en) | Low-efficiency identification and electric quantity lifting method of photovoltaic module under multi-orientation and inclination angles |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |