CN110567905B - Water quality detection method and device - Google Patents
Water quality detection method and device Download PDFInfo
- Publication number
- CN110567905B CN110567905B CN201910831416.0A CN201910831416A CN110567905B CN 110567905 B CN110567905 B CN 110567905B CN 201910831416 A CN201910831416 A CN 201910831416A CN 110567905 B CN110567905 B CN 110567905B
- Authority
- CN
- China
- Prior art keywords
- training
- sample
- spectrum data
- absorption spectrum
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 55
- 238000001514 detection method Methods 0.000 title abstract description 12
- 238000012549 training Methods 0.000 claims abstract description 89
- 238000000862 absorption spectrum Methods 0.000 claims abstract description 60
- 238000012360 testing method Methods 0.000 claims abstract description 57
- 230000009467 reduction Effects 0.000 claims abstract description 43
- 238000000034 method Methods 0.000 claims abstract description 29
- 238000007781 pre-processing Methods 0.000 claims abstract description 24
- 238000001228 spectrum Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 11
- 230000004913 activation Effects 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 6
- 238000012372 quality testing Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 4
- 238000007476 Maximum Likelihood Methods 0.000 claims description 2
- 238000011478 gradient descent method Methods 0.000 claims description 2
- 238000011160 research Methods 0.000 claims description 2
- 238000005259 measurement Methods 0.000 abstract description 4
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 238000002835 absorbance Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 230000003595 spectral effect Effects 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 239000000126 substance Substances 0.000 description 3
- 238000002211 ultraviolet spectrum Methods 0.000 description 3
- 238000003911 water pollution Methods 0.000 description 3
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 229910052799 carbon Inorganic materials 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
- G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
- G01N21/33—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using ultraviolet light
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/01—Arrangements or apparatus for facilitating the optical investigation
- G01N2021/0106—General arrangement of respective parts
- G01N2021/0118—Apparatus with remote processing
- G01N2021/0137—Apparatus with remote processing with PC or the like
Landscapes
- Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
Abstract
The embodiment of the invention provides a water quality detection method and device, and belongs to the technical field of water quality detection. Comprising the following steps: collecting absorption spectrum data of a water quality sample; preprocessing the absorption spectrum data to obtain a training sample and a test sample; training the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model; using the DBN dimension reduction model to reduce the dimension of the test sample to obtain a dimension reduced test sample; inputting the dimension reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample. Compared with the prior art, the method has the advantages of low requirement on training samples, simple measurement and more accurate result.
Description
Technical Field
The embodiment of the invention relates to the technical field of water quality detection, in particular to a water quality detection method and device.
Background
The good ecological environment is the basis on which the terrestrial living things depend on, water pollution is one of the most important environmental problems in the world, and how to effectively control water pollution and reasonably utilize water resources has become the focus of common attention in countries of the world. In various water pollution problems faced by China, organic pollution is a non-negligible factor. The organic pollution degree of water quality is mainly characterized by total organic carbon (Total Organic Carbon, TOC), which represents the total sum of organic substances contained in the water body and directly reflects the pollution degree of the water body by the organic substances. In the prior art, a dimension reduction model is mainly adopted to reduce the dimension of absorbance data of a water body, but a counter propagation network (Back Propagation Network, BP) model is then established to invert the TOC concentration of the water quality to be measured.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
1. building a BP model requires a large amount of training data, and it is impractical to obtain sufficient training data in an industrial setting.
2. The absorbance data of the water body directly obtained by the spectrometer has the characteristics of high dimension, large quantity and the like, and the inversion effect is poor when the obtained low micro absorbance data is used for measuring the pollution degree of the water body by organic substances because the traditional dimension reduction method only can extract the shallow layer characteristics of the absorbance data.
Therefore, how to provide a more accurate and simple modeling method to overcome the above problems when inverting the TOC concentration of water is a urgent problem in water quality detection technology.
Disclosure of Invention
Therefore, one of the technical problems to be solved by the embodiments of the present invention is to provide a water quality detection method and apparatus, which are used for overcoming the defects of high training data requirement, poor inversion effect and the like in the prior art, and achieving the effects of simple modeling and accurate inversion result.
The embodiment of the invention provides a water quality detection method, which comprises the following steps:
collecting absorption spectrum data of a water quality sample;
preprocessing the absorption spectrum data to obtain a training sample and a test sample;
training the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model;
using the DBN dimension reduction model to reduce the dimension of the test sample to obtain a dimension reduced test sample;
inputting the dimension reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample.
Optionally, in an embodiment of the present invention, the collecting absorption spectrum data of the water quality sample includes:
and collecting the absorption spectrum data of the water quality sample with the known TOC concentration, and collecting the absorption spectrum data of the water quality sample with the unknown TOC concentration.
Optionally, in an embodiment of the present invention, the preprocessing the absorption spectrum data to obtain the training sample and the test sample includes:
intercepting a waveband with separated TOC concentration characteristics in the absorption spectrum data;
subtracting the dark spectrum data from the intercepted absorption spectrum data and taking the logarithm to obtain preprocessed absorption spectrum data;
optionally, in an embodiment of the present invention, the preprocessing the absorption spectrum data to obtain the training sample and the test sample includes:
and preprocessing the absorption spectrum data to obtain a first training sample, a second training sample and a test sample.
Optionally, in an embodiment of the present invention, the training using the training samples to obtain the DBN dimension reduction model and the elastic net inversion model includes:
training by using the first training sample to obtain a DBN dimension reduction model and an elastic Net inversion model;
and evaluating the DBN dimension reduction model and the elastic Net inversion model performance by using the second training sample.
The embodiment of the invention provides a water quality detection device, which comprises:
the data acquisition module is configured to acquire absorption spectrum data of a water quality sample;
the preprocessing module is configured to preprocess the absorption spectrum data to obtain a training sample and a test sample;
the modeling module is configured to train the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model;
the dimension reduction module is configured to reduce the dimension of the test sample by using the DBN dimension reduction model to obtain a dimension reduced test sample;
and the prediction module is configured to input the dimension-reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample.
Optionally, in a specific embodiment of the present invention, the method includes:
a first spectrum acquisition module configured to acquire absorption spectrum data of a water quality sample of known TOC concentration;
and the second spectrum acquisition module is configured to acquire absorption spectrum data of a water quality sample with unknown TOC concentration.
Optionally, in a specific embodiment of the present invention, the preprocessing module includes:
the spectrum interception module is configured to intercept wave bands with separated TOC concentration characteristics in the absorption spectrum data;
the spectrum separation module is configured to subtract the dark spectrum data from the intercepted absorption spectrum data and take the logarithm to obtain preprocessed absorption spectrum data;
optionally, in a specific embodiment of the present invention, the preprocessing module further includes:
the first output module is configured to output a first training sample obtained by preprocessing the absorption spectrum data;
the second output module is configured to output a second training sample obtained by preprocessing the absorption spectrum data;
and the third output module is configured to output the test sample obtained by preprocessing the absorption spectrum data.
Optionally, in a specific embodiment of the present invention, the method further includes:
and the performance evaluation module is configured to evaluate the first training sample by using the second training sample to train to obtain a DBN dimension reduction model and an elastic Net inversion model.
As can be seen from the above technical solutions, the embodiment of the present invention includes: collecting absorption spectrum data of a water quality sample; preprocessing the absorption spectrum data to obtain a training sample and a test sample; training the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model; using the DBN dimension reduction model to reduce the dimension of the test sample to obtain a dimension reduced test sample; inputting the dimension reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample. Compared with the prior art, the method has the advantages of low requirement on training samples, simple measurement and more accurate result.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present invention, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.
FIG. 1 is a flow chart of an embodiment of a water quality testing method of the present invention;
FIG. 2 is a schematic view of RBM structure of an embodiment of a water quality testing method of the present invention;
FIG. 3 is a flow chart of another embodiment of a water quality testing method of the present invention;
FIG. 4 is a schematic diagram of a water quality testing apparatus according to an embodiment of the present invention;
FIG. 5 is a schematic view of another embodiment of a water quality testing apparatus according to the present invention;
Detailed Description
The embodiment of the invention comprises the following steps: collecting absorption spectrum data of a water quality sample; preprocessing the absorption spectrum data to obtain a training sample and a test sample; training the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model; using the DBN dimension reduction model to reduce the dimension of the test sample to obtain a dimension reduced test sample; inputting the dimension reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample. Compared with the prior art, the method has the advantages of low requirement on training samples, simple measurement and more accurate result.
Of course, it is not necessary for any of the embodiments of the invention to be practiced with all of the advantages described above.
In order to better understand the technical solutions in the embodiments of the present invention, the following description will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which are derived by a person skilled in the art based on the embodiments of the present invention, shall fall within the scope of protection of the embodiments of the present invention.
The implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.
Referring to fig. 1, an embodiment of the present invention provides a water quality detection method, including:
s1, collecting absorption spectrum data of a water quality sample;
s2, preprocessing the absorption spectrum data to obtain a training sample and a test sample;
s3, training the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model;
s4, reducing the dimension of the test sample by using the DBN dimension reduction model to obtain a dimension reduced test sample;
s5, inputting the dimension-reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample.
Specifically, a TOC solution sample is placed in a room for standing for 1h, and ultraviolet spectrum data of a water sample is collected at room temperature (25+/-1) DEG C. To reduce the effect of single random errors, each water sample was scanned 1000 times in total, thus obtaining 1000 sets of spectral data for each water sample. The spectrum data acquisition process is that light emitted by a light source is firstly transmitted into a sample cell through an incident optical fiber, and after being absorbed by a water sample in the sample cell, transmitted light enters an online spectrometer through an emergent optical fiber for measurement, and spectrum data in the spectrometer enters a computer system for processing through a USB interface.
The obtained spectral data is separated into training samples of known concentration and test samples of unknown concentration. The DBN dimension reduction model is trained using training samples of known concentration, and the elastic Net inversion model is trained using the post-dimension reduction training samples.
The DBN is a probabilistic generating model. The system comprises a plurality of layers of limited Boltzmann machines (Restrict Boltzmann Machine, RBM), and is a deep network structure. By stacking the RBMs, underlying features in the absorption spectrum data of the TOC can be combined to form more abstract high-level features, thereby achieving better effects in the data reconstruction process. The RBM is used as an undirected graph model based on bipartite graph, and has a two-layer structure, full connection among layers and no connection among layers, and the structural model is shown in figure 2. The RBM model includes a visible layer v, an implied layer h, and a connection weight W between the visible layer and the implied layer. The absorption spectrum data of TOC is input into the visible layer v, and the activation state of each unit of the hidden layer h can be obtained by connecting the weight W with the activation function. Depending on the activation state of the network at this time, an energy function E (v, h|θ) can be established.
Further, a joint probability distribution of (v, h) can be established according to equation (1)
The distribution P (v|θ) of the input data v can be determined by the boundary distribution of P (v, h|θ).
The process of training the RBM using the training samples is to determine the parameter θ= { W i,j ,a i ,b j Process of }. If the training sample set contains T samples, the parameter θ is obtained by its maximum likelihood function.
A loss function is defined as in equation (5) and then the parameters are optimized by a random gradient descent method.
The parameters are then chosen in the optimization of each layer to minimize their loss function and to expect to produce a globally optimal solution by doing a local optimization based on greedy thinking. Each RBM layer needs to learn the entire input, with the previous layer detecting simple features and the later layer recombining them. Just as a convolutional neural network is used to detect a face, the front layer detects edges in the image and the rear layer will use these results to form facial features.
The training set data is input into the first RBM, then the hidden layer of the RBM of the upper layer serves as the input of the RBM of the lower layer, and the best characteristic of the research problem is automatically found through the non-supervision learning of the RBM. By training layer by layer, the output of the DBN shows deep features of the input data more accurately.
And finally, performing dimension reduction on the test sample with unknown concentration by using the DBN dimension reduction model obtained by training the training sample, inputting the dimension reduced test sample into an elastic Net inversion model obtained by training the dimension reduced training sample, and measuring the TOC concentration of the test sample.
The TOC concentration in the water sample can be accurately measured when the training sample is small by using the DBN dimension reduction model and the elastic Net inversion model. As can be seen from Table 1
Table 1 predictive values and standard deviations of various methods
| Modeling method | 20mg/L | 30mg/L | 50mg/L | 60mg/L | 80mg/L | 90mg/L |
| PCA+OLS | 19.60±0.0005 | 30.55±0.0005 | 49.38±0.0013 | 59.52±0.0011 | 78.18±0.0043 | 87.80±0.0042 |
| PCA+BP | 20.06±0.0004 | 30.71±0.0005 | 49.60±0.0014 | 60.03±0.0011 | 79.66±0.0048 | 89.98±0.0048 |
| PCA+SVR | 20.48±0.0004 | 31.65±0.0006 | 50.91±0.0014 | 61.28±0.0011 | 80.35±0.0044 | 90.18±0.0044 |
| PCA+ElasticNet | 20.36±0.0005 | 31.05±0.0004 | 50.55±0.0011 | 61.15±0.0012 | 80.45±0.0034 | 90.23±0.0035 |
| DBN+OLS | 19.68±0.0004 | 30.51±0.0005 | 49.30±0.0013 | 59.46±0.0009 | 78.18±0.0037 | 87.92±0.0036 |
| DBN+BP | 20.02±0.0004 | 30.73±0.0005 | 49.86±0.0014 | 60.39±0.0010 | 79.81±0.0040 | 90.05±0.0038 |
| DBN+SVR | 20.38±0.0004 | 31.41±0.0005 | 50.58±0.0013 | 60.95±0.0009 | 80.04±0.0038 | 89.97±0.0037 |
| DBN+ElasticNet | 20.16±0.0003 | 30.24±0.0004 | 50.16±0.0010 | 60.55±0.0008 | 80.03±0.0031 | 90.01±0.0001 |
Referring to fig. 3, in another embodiment of the present invention, the preprocessing the absorption spectrum data to obtain a training sample and a test sample includes:
s21, intercepting a wave band with separated TOC concentration characteristics in the absorption spectrum data;
s22, subtracting the dark spectrum data from the intercepted absorption spectrum data and taking the logarithm to obtain preprocessed absorption spectrum data;
specifically, when ultraviolet spectrum data of a water sample are collected, ultraviolet spectrum absorption data of thousands of wave bands are collected, at the moment, because water sample absorption data of different TOC concentrations can be aliased in some wave bands, in order to reduce the influence of the wave bands and obtain better experimental results, the distinguishable wave bands are intercepted during pretreatment, and absorption spectrum data in the wave bands are obtained.
The intercepted absorption spectrum data can be distinguished, but in order to obtain better processing inversion results later, the embodiment of the invention further increases the distinguishing degree between the absorption spectrum data of water samples with different TOC concentrations by subtracting the dark spectrum data and taking the logarithm. Thereby further reducing the requirements on training samples and instrumentation.
In yet another embodiment of the present invention, the preprocessing the absorption spectrum data to obtain the training sample and the test sample includes: and preprocessing the absorption spectrum data to obtain a first training sample, a second training sample and a test sample.
Training by using the first training sample to obtain a DBN dimension reduction model and an elastic Net inversion model; and evaluating the DBN dimension reduction model and the elastic Net inversion model performance by using the second training sample.
The DBN dimensionality reduction model and the elastic Net inversion model performance are evaluated by using a second training sample. The accuracy of the method provided by the invention can be seen exactly, the adjustment direction of each parameter in the model is defined in the process of establishing the model, the establishment of the model is accelerated, and in addition, the effectiveness of each model can be explained in detail.
Referring to fig. 4, an embodiment of the present invention provides a water quality detection apparatus, including: the data acquisition module 1 is configured to acquire absorption spectrum data of a water quality sample; a preprocessing module 2 configured to preprocess the absorption spectrum data to obtain a training sample and a test sample; the modeling module 3 is configured to train the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model; the dimension reduction module 4 is configured to reduce the dimension of the test sample by using the DBN dimension reduction model to obtain a dimension reduced test sample; and the prediction module 5 is configured to input the dimension-reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample.
Specifically, each water sample is scanned 1000 times through the data acquisition module 1, so that 1000 groups of spectrum data of each water sample are obtained. The preprocessing module 2 divides the obtained spectral data into training samples of known concentration and test samples of unknown concentration. The modeling module 3 trains the DBN dimensionality reduction model using training samples of known concentration and trains the elastic net inversion model using the post-dimensionality reduction training samples.
The dimension reduction module 4 uses the DBN dimension reduction model obtained by training the training sample to reduce the dimension of the test sample with unknown concentration. The prediction module 5 measures the TOC concentration of the test sample according to the dimension-reduced test sample.
The device can accurately measure TOC concentration in the water sample when the training sample is smaller.
Referring to fig. 5, in another embodiment of the present invention, the preprocessing module includes: a spectrum interception module 21 configured to intercept a band in which TOC concentration characteristics in the absorption spectrum data are separated; a spectrum separation module 22 configured to subtract the dark spectrum data from the intercepted absorption spectrum data and take the logarithm to obtain preprocessed absorption spectrum data;
specifically, in order to reduce the influence of invalid wave bands and obtain better experimental results, the invention is provided with a spectrum interception module 21 in the preprocessing module 2, and can intercept distinguishable wave bands to obtain absorption spectrum data in the wave bands.
Although the intercepted absorption spectrum data can be distinguished, in order to obtain better processing inversion results later, the embodiment of the invention further increases the distinguishing degree between the absorption spectrum data of water samples with different TOC concentrations by subtracting the dark spectrum data from the spectrum separation module 22 and taking the logarithm. Thereby further reducing the requirements on training samples and instrumentation.
In yet another embodiment of the present invention, the method further includes: and the performance evaluation module is configured to evaluate the first training sample by using the second training sample to train to obtain a DBN dimension reduction model and an elastic Net inversion model.
Training by using the first training sample to obtain a DBN dimension reduction model and an elastic Net inversion model; and evaluating the DBN dimension reduction model and the elastic Net inversion model performance by using the second training sample.
And evaluating the performance of the DBN dimension reduction model and the elastic Net inversion model by a performance evaluation module by using a second training sample. The accuracy of the method provided by the invention can be seen from the definition, the adjustment direction of each parameter in the model is defined in the process of establishing the model, the establishment of the model is accelerated, and the effectiveness of each model can be explained in detail.
It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus (device), or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the invention. It will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the present invention without departing from the spirit or scope of the embodiments of the invention. Thus, the embodiments of the present invention are intended to include such modifications and alterations insofar as they come within the scope of the embodiments of the invention as claimed and the equivalents thereof.
Claims (1)
1. A water quality testing method, comprising:
collecting absorption spectrum data of a water quality sample;
preprocessing the absorption spectrum data to obtain a training sample and a test sample;
training the training sample to obtain a DBN dimension reduction model and an elastic Net inversion model;
the DBN is a probability generation model and comprises a plurality of layers of RBMs of a limited Boltzmann machine, and is a deep network structure, and the RBMs are stacked to combine bottom layer features in absorption spectrum data of TOC to form more abstract high-level features, so that a better effect is obtained in the data reconstruction process;
the RBM is an undirected graph model based on bipartite graph, and has a two-layer structure, wherein the layers are fully connected, and the layers are not connected; the RBM model comprises a visible layer v, an implicit layer h and a connection weight W between the visible layer and the implicit layer; inputting absorption spectrum data of TOC into a visible layer v, and obtaining the activation state of each unit of an hidden layer h by connecting a weight W with an activation function; according to the activation state of the network at this time, an energy function E (v, h|theta) is established:
further, a joint probability distribution of (v, h) is established according to equation (1):
the distribution P (v|θ) of the input data v is determined by the boundary distribution of P (v, h|θ):
the process of training the RBM using the training samples is to determine the parameter θ= { W i,j ,a i ,b j A process of }; if the training sample set contains T samples, the parameter θ can be obtained by its maximum likelihood function:
defining a loss function as in (5), and optimizing parameters by a random gradient descent method:
then, according to greedy thought, the parameters are selected to minimize the loss function in the optimization of each layer, and a global optimal solution is expected to be generated through the local optimization; each RBM layer needs to learn the entire input, the previous layer detects simple features, and the latter layer reassembles them;
inputting training set data into a first RBM, then using a hidden layer of the RBM of the upper layer as the input of the RBM of the lower layer, and automatically finding out the best characteristic of a research problem through the non-supervision learning of the RBM; by training layer by layer, the deep features of the input data are accurately displayed by the output of the DBN;
using the DBN dimension reduction model to reduce the dimension of the test sample to obtain a dimension reduced test sample;
inputting the dimension reduced test sample into the elastic Net inversion model to obtain TOC concentration of the test sample;
the absorption spectrum data of the collected water quality sample comprises the following steps:
collecting absorption spectrum data of a water sample with known TOC concentration, and collecting absorption spectrum data of a water sample with unknown TOC concentration;
the preprocessing of the absorption spectrum data to obtain a training sample and a test sample comprises the following steps:
intercepting a waveband with separated TOC concentration characteristics in the absorption spectrum data;
subtracting the dark spectrum data from the intercepted absorption spectrum data and taking the logarithm to obtain preprocessed absorption spectrum data;
preprocessing the absorption spectrum data to obtain a first training sample, a second training sample and a test sample;
the training by using the training sample to obtain the DBN dimension reduction model and the elastic Net inversion model comprises the following steps:
training by using the first training sample to obtain a DBN dimension reduction model and an elastic Net inversion model;
and evaluating the DBN dimension reduction model and the elastic Net inversion model performance by using the second training sample.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910831416.0A CN110567905B (en) | 2019-09-04 | 2019-09-04 | Water quality detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910831416.0A CN110567905B (en) | 2019-09-04 | 2019-09-04 | Water quality detection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110567905A CN110567905A (en) | 2019-12-13 |
| CN110567905B true CN110567905B (en) | 2023-12-29 |
Family
ID=68777768
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910831416.0A Active CN110567905B (en) | 2019-09-04 | 2019-09-04 | Water quality detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110567905B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111678869A (en) * | 2020-05-29 | 2020-09-18 | 山东省科学院海洋仪器仪表研究所 | Water quality ecological parameter detection device and detection method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105911003A (en) * | 2016-04-11 | 2016-08-31 | 重庆大学 | RBM regression-based water TOC concentration analysis method |
| CN107036974A (en) * | 2016-11-18 | 2017-08-11 | 中国水利水电科学研究院 | Inversion method is cooperateed with based on the water quality parameter multi-model that certainty set is modeled |
| CN107421895A (en) * | 2017-06-30 | 2017-12-01 | 中国水利水电科学研究院 | A kind of water quality parameter retrieving concentration method and apparatus of multiband optimum organization |
| CN108876650A (en) * | 2018-06-25 | 2018-11-23 | 仲恺农业工程学院 | Characteristic extraction method and device for waterfowl healthy breeding environment parameters |
| CN109959634A (en) * | 2019-04-15 | 2019-07-02 | 中科谱光科技（天津）有限公司 | A kind of water quality monitoring system and method |
-
2019
- 2019-09-04 CN CN201910831416.0A patent/CN110567905B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105911003A (en) * | 2016-04-11 | 2016-08-31 | 重庆大学 | RBM regression-based water TOC concentration analysis method |
| CN107036974A (en) * | 2016-11-18 | 2017-08-11 | 中国水利水电科学研究院 | Inversion method is cooperateed with based on the water quality parameter multi-model that certainty set is modeled |
| CN107421895A (en) * | 2017-06-30 | 2017-12-01 | 中国水利水电科学研究院 | A kind of water quality parameter retrieving concentration method and apparatus of multiband optimum organization |
| CN108876650A (en) * | 2018-06-25 | 2018-11-23 | 仲恺农业工程学院 | Characteristic extraction method and device for waterfowl healthy breeding environment parameters |
| CN109959634A (en) * | 2019-04-15 | 2019-07-02 | 中科谱光科技（天津）有限公司 | A kind of water quality monitoring system and method |
Non-Patent Citations (1)
| Title |
|---|
| 近红外光谱elasticnet建模方法与应用;郑年年 等;《光谱学与光谱分析》;20181031;第38卷(第10期);引言、章节1、4-5 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110567905A (en) | 2019-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103983595B (en) | A kind of water turbidity calculation method processed based on ultraviolet-visible spectrum | |
| CN110321959B (en) | Multispectral image information and CNN coal and gangue identification method | |
| CN104729965B (en) | PM2.5 concentration detection method based on interzone radial basis function nerve network | |
| CN105486655A (en) | Rapid detection method for organic matters in soil based on infrared spectroscopic intelligent identification model | |
| CN104316491A (en) | Method for detecting urea doped in milk based on synchronous-asynchronous two-dimensional near-infrared correlation spectrum | |
| Sabti et al. | Galaxy luminosity function pipeline for cosmology and astrophysics | |
| CN111047073B (en) | Aquaculture water quality prediction method and system based on neural network | |
| CN102158486A (en) | Method for rapidly detecting network invasion | |
| CN105784672A (en) | Drug detector standardization method based on dual-tree complex wavelet algorithm | |
| CN110567905B (en) | Water quality detection method and device | |
| CN104237235A (en) | Rapid detection method based on near-infrared imaging technology for food-borne pathogens | |
| Ying et al. | Automatic detection of photovoltaic module cells using multi-channel convolutional neural network | |
| CN110070220A (en) | A kind of ammonia nitrogen index flexible measurement method based on neural network algorithm | |
| CN115601572A (en) | Ultrasonic phased array image optimization reconstruction method and system based on semi-supervised CycleGan network | |
| Zhang et al. | Automatic microseismic detection and location via the deep-convolutional neural network | |
| WO2022134579A1 (en) | Method and apparatus for detecting abnormalities of service transaction data, and computer device | |
| CN104063459B (en) | Cell differential gene data sorting system based on influenza virus stimulation | |
| CN117607120A (en) | Food additive Raman spectrum detection method and device based on improved Resnext model | |
| CN114019082B (en) | Soil organic matter content monitoring method and system | |
| CN114462127A (en) | Structural damage identification method and system based on deep extreme learning machine | |
| CN112697745A (en) | Method for measuring alcohol content of white spirit | |
| CN103514288A (en) | Identification method and system of classes of clients | |
| CN113376107B (en) | Water quality monitoring system and method based on cloud platform | |
| CN104268386B (en) | A kind of method that testability virtual test data is converted to actual loading test data | |
| CN117288903B (en) | Building block type test method and system for aircraft composite material |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231128 Address after: 418000, Zone 123, Phase II A, JD Cloud (Huaihua) Digital Economy Industrial Park, Economic Development Zone, Huaihua City, Hunan Province Applicant after: Huaihua Water Direct to Shanquan Drinking Water Co.,Ltd. Address before: 401, venture building, Hunan University Science Park, 186 Guyuan Road, Changsha high tech Zone, Changsha City, Hunan Province, 410205 Applicant before: HUNAN KUANGCHU TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |