CN111198165A - Method for measuring water quality parameters based on spectral data standardization - Google Patents
Method for measuring water quality parameters based on spectral data standardization Download PDFInfo
- Publication number
- CN111198165A CN111198165A CN202010037852.3A CN202010037852A CN111198165A CN 111198165 A CN111198165 A CN 111198165A CN 202010037852 A CN202010037852 A CN 202010037852A CN 111198165 A CN111198165 A CN 111198165A
- Authority
- CN
- China
- Prior art keywords
- data
- spectral data
- water quality
- spectrometer
- spectral
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 85
- 230000003595 spectral effect Effects 0.000 title claims abstract description 77
- 238000000034 method Methods 0.000 title claims abstract description 31
- 239000011159 matrix material Substances 0.000 claims abstract description 53
- 238000001228 spectrum Methods 0.000 claims abstract description 47
- 238000001514 detection method Methods 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000010606 normalization Methods 0.000 claims abstract description 16
- 238000001914 filtration Methods 0.000 claims abstract description 6
- 230000009467 reduction Effects 0.000 claims abstract description 6
- 238000002835 absorbance Methods 0.000 claims description 28
- 238000012546 transfer Methods 0.000 claims description 14
- 238000005259 measurement Methods 0.000 claims description 7
- 238000002371 ultraviolet--visible spectrum Methods 0.000 claims description 6
- 238000000513 principal component analysis Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 230000007423 decrease Effects 0.000 claims description 3
- 229910052757 nitrogen Inorganic materials 0.000 claims description 3
- XKMRRTOUMJRJIA-UHFFFAOYSA-N ammonia nh3 Chemical compound N.N XKMRRTOUMJRJIA-UHFFFAOYSA-N 0.000 claims description 2
- 150000001875 compounds Chemical class 0.000 claims description 2
- 230000004044 response Effects 0.000 abstract description 3
- 238000012360 testing method Methods 0.000 abstract description 3
- 230000009466 transformation Effects 0.000 abstract description 2
- 239000000523 sample Substances 0.000 description 29
- 239000012488 sample solution Substances 0.000 description 22
- 238000004458 analytical method Methods 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 5
- 238000000862 absorption spectrum Methods 0.000 description 4
- 230000000052 comparative effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 239000000126 substance Substances 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- IWZKICVEHNUQTL-UHFFFAOYSA-M potassium hydrogen phthalate Chemical compound [K+].OC(=O)C1=CC=CC=C1C([O-])=O IWZKICVEHNUQTL-UHFFFAOYSA-M 0.000 description 2
- 230000004083 survival effect Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 238000000870 ultraviolet spectroscopy Methods 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003153 chemical reaction reagent Substances 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000003651 drinking water Substances 0.000 description 1
- 235000020188 drinking water Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005316 response function Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000010183 spectrum analysis Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
- G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
- G01N21/27—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands using photo-electric detection ; circuits for computing concentration
- G01N21/274—Calibration, base line adjustment, drift correction
- G01N21/278—Constitution of standards
Landscapes
- Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Biochemistry (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Engineering & Computer Science (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
Abstract
The invention provides a method for measuring water quality parameters based on spectral data standardization, which comprises the steps of firstly filtering sample water quality spectral data by using EWMA (enhanced wavelet transform algorithm), and performing dimensionality reduction on the data to extract principal components; and performing matrix transformation on the two groups of data of the source machine and the target machine by using a direct standardization algorithm, then performing Z-score standardization and normalization data processing, and finally fitting regression spectrum data to obtain the water quality parameter value of the water sample to be detected. According to the invention, the spectral data are standardized, so that the problems that the unit resolution, the precision and the response range of multi-parameter detection spectral detection of different water qualities are not uniform, the comparison of the test data among different instruments and the fitting of multi-parameter data are difficult to perform and the like are solved, and therefore, the spectrums scanned by different instruments can be analyzed by using a universal model after being transferred.
Description
Technical Field
The invention belongs to the technical field of spectral processing analysis, and particularly relates to a method for measuring water quality parameters based on spectral data standardization.
Background
Water resources are the material basis for the survival of all living things, and along with the rapid increase of the global population and the rapid development of industrialization in recent years, the water resources are the material basis for the survival of all living things. The demand of human beings on water resources is increasing day by day, and the deterioration of water environment has become a great problem facing the water resources and the environment in the world. With the improvement of the attention of people to daily drinking water safety and environmental protection, the water quality parameter detection becomes a necessary means for monitoring the quality of social water resources. At present, chemical methods and spectrum analysis methods are mainly adopted for water quality detection. Although the traditional chemical method has high water quality detection precision, the detection period is long, the operation of professional personnel is needed, and the used chemical reagent is easy to generate secondary pollution. The ultraviolet-visible spectroscopy is a method for analyzing and measuring radiation in a spectral region of 200-800 nm absorbed by molecules of certain substances, and a relation model between sample spectral information and component content of the sample spectral information can be established for predicting the component content of an unknown sample to be measured. The ultraviolet-visible spectrum method has the advantages of rapidness, no secondary pollution, fingerprint detection, pollution traceability and the like, is widely applied to water quality detection, and becomes a research hotspot in the field of water quality detection.
However, the uv-vis spectroscopy is an indirect analysis method, and when a spectrum calibration model is established, the detection result of the same sample may be different based on different manufacturing processes of different instrument manufacturers, different product batches of the same manufacturer, different aging degrees of detection equipment, and other influence factors. The model built in one instrument will not be applicable to another instrument. The "failure" of the model is mainly caused by the inconsistency of the measured signals caused by the difference of the response functions of the sample and the instruments. Therefore, the actual monitoring environment needs to be respectively modeled aiming at the ultraviolet-visible water quality spectrum data measured by different monitoring points and different spectrometers, so that the efficiency is low, the model maintenance cost is increased, and the problems can be effectively solved by model transmission. An effective way to address model delivery is to standardize the instruments or data.
Therefore, how to effectively overcome the existing defects in the water quality parameter detection, achieve high measurement precision and system stability, and reduce measurement cost and operation difficulty becomes a problem to be solved urgently at present.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a method for measuring water quality parameters based on spectral data standardization, which solves the problems that in the existing water quality detection, the resolution, the precision and the response range of a unit are not uniform, and the comparison of test data among different instruments and the fitting of multi-parameter data are difficult to perform, and the like.
In order to achieve the purpose, the invention adopts the following scheme: a method for measuring water quality parameters based on spectral data standardization comprises the following steps:
s1: obtaining a plurality of samples with different water sample parameters;
s2: collecting and denoising the ultraviolet-visible spectrum of the sample in the step S1 by using a multi-source spectrometer, and performing weighted average moving filtering processing on the denoised detection spectrum to obtain preprocessed detection spectrum data;
s3: performing feature extraction on the preprocessed detection spectrum obtained in step S2 by Principal Component Analysis (PCA), and performing input water quality detection spectrum data an×mDimensionality reduction is carried out so as to output a spectral matrix A 'containing k characteristic dimensions'n×m；
S4: randomly selecting a standard multi-source spectrometer as a source machine, and determining a sample by the source machine according to the steps S2-S3 to obtain a spectrum data matrix measured by the source machine; selecting a multi-source spectrometer to be standardized as a target machine, determining the same sample by the target machine according to the steps S2-S3 to obtain a spectral data matrix measured by the target machine, and determining a transfer matrix by the spectral data matrix measured by the source machine and the spectral data matrix measured by the target machine; therefore, because the source machine has high data measuring precision, but the instrument cost is high, a standard instrument is used as the source machine, in addition, the instrument with low cost is used as the target machine, and after a functional relation of the measured data of the two instruments is found, the purpose that the standard multi-source spectrometer is replaced by the multi-source spectrometer with low cost can be realized.
S5: detecting water quality parameters of a water sample to be detected by using a target machine to obtain spectral data of a spectrometer to be detected, then substituting the spectral data into a standardized model, and calculating to obtain standardized spectral data of the water sample to be detected;
the standardized model adopts the following formula:
Astd＝Aunknown*F
in the formula uunknownNormalized spectrum, AstdRepresenting the spectrum of the spectrometer to be tested, and F representing a transfer matrix;
s6: and (5) carrying out normalized data processing on the normalized spectral data obtained in the step (S5), carrying out fitting regression, and then obtaining the water quality parameter value of the water sample to be detected according to the regressed water quality spectral model.
Preferably, the water sample parameters are COD, turbidity, ammonia nitrogen, TDS or TOC.
Preferably, the multi-source spectrometer is a DH2000 light source and Hamamatsu C10082CAH spectrometer, an American ocean optics Maya2000Pro spectrometer or a Xiamen spectral space-time ATP2000 spectrometer.
Preferably, the weighted average moving filter process is calculated using the following formula:
EWMA(N)＝λY(N)+(1-λ)EWMA(N-1),i＝1,2,…,n；
where ewma (N) represents an estimated value of spectral data at a band point N, y (N) represents a measured value of spectral data at a band point N, λ is a weighting factor, 0< λ <1 and decreases exponentially as the band point N increases, and N represents the sum of the number of measured band values. Thus, EWMA (explicit Weighted Moving-Average) is an Exponentially Weighted Average Moving algorithm. The algorithm can smooth short-term fluctuation, retain the long-term development trend of the waveform, introduce a weight factor lambda to carry out maximum likelihood estimation on data, search a phylogenetic tree which generates and observes ultraviolet-visible water quality spectrum data with higher probability, and restore theoretical ultraviolet-visible water quality spectrum data under the condition of maximum probability.
Preferably, taking into account the non-synchronous setting of the sampling points of the spectrometer adopted in the experiment, cubic polynomial interpolation is adopted herein to obtain the absorbance values of the same wavelength points. The wave band range selects the intersection of the measuring ranges of the three spectrometers. The measurement waveband ranges of the three spectrometers are respectively set as L1 ═ b1, b2]，L2＝[m1,m2]，L3＝[a1,a2]. Compare the range values if b1<m1<a1,b2>m2>a2, the wavelength measurement range is [ b1, a2 ]]I.e. in [ b1, a2 ]]And carrying out interpolation processing in the wavelength range to obtain the absorbance values of the same wavelength point. Defining the matrix after interpolation of the source machine as AmDefining the matrix obtained after interpolation of the target machine as AtEstablishing A by transforming the matrix FmAnd AtAssociation of the two sets of matrices: a. them＝AtF；
Wherein
Is AtThe transition matrix F can thus be:
target machine matrix AtConversion to A by transfer matrix FmI.e. the source machine matrix. It can be obtained that when the target machine instrument is used to measure an unknown sample solution, the obtained spectral data matrix AunknownOr can be converted into a spectral data matrix A of the source machine instrument for the unknown sample solution through a transfer matrix FstdTheoretically, the spectral matrix A 'measured by the source machine instrument for the unknown sample solution is analyzed'stdShould be consistent with AstdThe same, namely: a'std＝Astd＝Aunknown*F
Preferably, the normalization data processing adopts the following steps:
adopting Z-score standardization to average the value of each characteristic into 0 and change the standard value into 1, and returning the absorbance value beyond the value range to the normal value A*：
In the formula, mu is the mean value of all absorbance data, and sigma is the standard deviation of all absorbance data;
the Z-score normalized data was then subjected to normalization data processing to map absorbance values between [0-1 ]:
in the formula (I), the compound is shown in the specification,
is the maximum value of the absorbance, and,
is the minimum value of absorbance.
Therefore, the abnormal values are pulled back to be average by the Z-score, and then the data are mapped to the interval by the second normalization, so that the solving speed of gradient descent can be accelerated in the subsequent model transmission process, the convergence speed of the model is improved, the initialization of water quality spectrum data is facilitated, the calculation can be simplified, the quantity value is reduced, and the expressive ability of the water quality spectrum data is improved.
Compared with the prior art, the invention has the following beneficial effects:
1. firstly, filtering sample water quality spectral data by using EWMA (equivalent weighted average) and performing dimensionality reduction on the data to extract principal components; and performing matrix transformation on the two groups of data of the source machine and the target machine by using a direct standardization algorithm, then performing Z-score standardization and normalization data processing, and finally obtaining the water quality parameter value of the water sample to be detected according to the spectral data. According to the invention, the spectral data are standardized, so that the problems that the unit resolution, the precision and the response range are not uniform, the comparison of the test data among different instruments and the fitting of multi-parameter data are difficult to perform and the like in the multi-parameter detection spectral detection of different water qualities are solved, and therefore, the spectrums scanned by different instruments can be analyzed by using a universal model after being transferred. The invention has very important significance for the standardization work of the water quality detecting instrument.
2. The method for determining the water quality parameter value is established based on the ultraviolet-visible spectrum method, and the characteristic peak shift phenomenon can occur during standardization, but the problem is usually ignored in the existing method, and the characteristic peak shift problem is not processed. The invention adopts the normalization algorithm of EWMA-PCA to process the water quality spectrum data, improves the utilization rate of model transmission, improves the water quality detection precision and the stability of the system, and can provide the water quality parameter value according to the input spectrum data by fitting the spectrum data model established by regression. The method is simple, reduces the measurement cost and the operation difficulty, and has good application prospect.
Drawings
FIG. 1 is a schematic flow chart of a method for measuring water quality parameters based on spectral data standardization according to the present invention.
FIG. 2 is water quality spectrum data detected by three spectrometers for eight sets of potassium hydrogen phthalate solutions; (a) sample solution of COD20mg/L, (b) sample solution of COD40 mg/L, (c) sample solution of COD60mg/L, (d) sample solution of COD80mg/L, (e) sample solution of COD160mg/L, (f) sample solution of COD200mg/L, (g) sample solution of COD400mg/L, and (h) sample solution of COD 800 mg/L.
FIG. 3 is a comparison graph of water absorption spectrum standardized data of different concentrations measured by three sets of comparison experiments; a is comparative group 1; b is comparative group 2; c is comparative group 3.
FIG. 4 is a diagram of the standardized evaluation and analysis of the absorption spectra of water with different concentrations measured in three sets of comparison experiments; a is a correlation coefficient; b is a variance; and c is the peak offset.
Detailed Description
The present invention will be described in further detail with reference to specific examples.
As shown in fig. 1, the invention provides a method for measuring water quality parameters based on spectral data standardization, which comprises the following steps:
s1: obtaining a plurality of samples with different water sample parameters;
s2: collecting and denoising the ultraviolet-visible spectrum of the sample in the step S1 by using a multi-source spectrometer, and performing weighted average moving filtering processing on the denoised detection spectrum to obtain preprocessed detection spectrum data;
s3: performing feature extraction on the preprocessed detection spectrum obtained in the step S2 by adopting principal component analysis, and performing dimensionality reduction on the input water quality detection spectrum data to output a spectrum matrix X 'containing k characteristic dimensionalities'n×m；
S4: randomly selecting a standard multi-source spectrometer as a source machine, and determining a sample by the source machine according to the steps S2-S3 to obtain a spectrum data matrix measured by the source machine; selecting a multi-source spectrometer to be standardized as a target machine, determining the same sample by the target machine according to the steps S2-S3 to obtain a spectral data matrix measured by the target machine, and determining a transfer matrix by the spectral data matrix measured by the source machine and the spectral data matrix measured by the target machine;
s5: detecting water quality parameters of a water sample to be detected by using a target machine to obtain spectral data of a spectrometer to be detected, then substituting the spectral data into a standardized model, and calculating to obtain standardized spectral data of the water sample to be detected;
the standardized model adopts the following formula:
Astd＝Aunknown*F
in the formula, AunknownNormalized spectrum, AstdRepresenting the spectrum of the spectrometer to be tested, and F representing a transfer matrix;
s6: and (5) carrying out normalized data processing on the normalized spectral data obtained in the step (S5), carrying out fitting regression, and then obtaining the water quality parameter value of the water sample to be detected according to the regressed water quality spectral model.
Example 1
A method for measuring water quality parameters based on spectral data standardization comprises the following steps:
s1: eight groups of to-be-detected potassium hydrogen phthalate solutions with different COD concentrations are obtained to be used as water quality samples, and the COD concentrations are respectively 20mg/L, 40mg/L, 60mg/L, 120mg/L, 160mg/L, 200mg/L, 400mg/L and 800 mg/L;
s2: respectively using a Maya2000Pro spectrometer, an ATP2000 spectrometer and a C10082CAH spectrometer to carry out ultraviolet-visible spectrum collection and denoising on the sample in the step S1; the results are shown in FIG. 2.
As can be seen from fig. 2, for 8 groups of sample solutions, the absorbance of the original spectrum data measured by the three spectrometers is relatively high in the range of 210-300nm, and the trend of the wave deformation is obvious in this range: for 160mg/L and below sample solutions, two obvious characteristic peaks can be seen, the wave band is in the range of about 250nm and 280nm, for sample solutions above 400nm/L, it is clear that a third characteristic peak is newly added at 300nm (for 200mg/L solutions, the Maya2000Pro spectrometer and the ATP2000 spectrometer have no obvious absorption peak, and the analysis reason may be caused by the instrument states of the three spectrometers, such as different detection accuracies and the like); the change range of the absorbance value is smaller in the range of 315-430 nm. However, the three spectrometers are still different for the same sample solution: for the first characteristic peak, the absorbance value measured by a C10082CAH spectrometer can reach 2.4-3.25; however, the absorbance values of the first characteristic peak measured by the Maya2000Pro spectrometer and the ATP2000 spectrometer are only 1.3-2.2 and 1.3-2.4, and the difference of the absorbance values of the same sample solution is more obvious, for example, as shown in fig. 3(a) of a sample solution with COD20mg/L, the absorbance measured by the C10082CAH spectrometer can reach 2.4, but the absorbance values measured by the Maya2000Pro spectrometer and the ATP2000 spectrometer can only reach 1.3, and the like; for the second characteristic peak, the absorbance value measured by C10082CAH can reach 0.4-3.1, the absorbance values measured by Maya2000Pro spectrometer and ATP2000 spectrometer are respectively 0.2-2 and 0.3-2.3, and the corresponding wave bands are different. Meanwhile, the difference is more obvious for the absorbance value of the same sample solution (for example, as shown in fig. 2(C), the absorbance value measured by a COD60mg/L and a C10082CAH spectrometer is far more than 1.5, but the absorbance value measured by a Maya2000Pro spectrometer and an ATP2000 spectrometer is far less than 1, and the absorbance values measured by the Maya2000Pro spectrometer and the ATP2000 spectrometer are different from each other. just for the condition that the absorbance value difference of the same sample solution is large and the waveform trends are different (for example, as shown in fig. 2(f), COD200mg/L), the water quality spectral data needs to be standardized and normalized for the accuracy of single sample data, so that the instrument standardization is realized, a basis is provided for subsequent stable data processing, and model transfer is realized.
S3: carrying out weighted average moving filtering processing on the detection spectrum subjected to denoising in the step S2 to obtain preprocessed detection spectrum data; the weighted average moving filter process is calculated using the following formula:
EWMA(N)＝λY(N)+(1-λ)EWMA(N-1),i＝1,2,…,n；
where ewma (N) represents an estimated value of spectral data at a band point N, y (N) represents a measured value of spectral data at a band point N, λ is a weighting factor, 0< λ <1 and decreases exponentially as the band point N increases, and N represents the sum of the number of measured band values.
S3: preprocessing obtained in step S2Performing feature extraction on the processed detection spectrum by adopting principal component analysis, and performing dimensionality reduction on input water quality detection spectrum data to output a spectrum matrix X' containing k characteristic dimensionsn×m；
S4: randomly selecting a standard multi-source spectrometer as a source machine, and determining a sample by the source machine according to the steps S2-S3 to obtain a spectrum data matrix measured by the source machine; selecting a multi-source spectrometer to be standardized as a target machine, determining the same sample by the target machine according to the steps S2-S3 to obtain a spectral data matrix measured by the target machine, and determining a transfer matrix by the spectral data matrix measured by the source machine and the spectral data matrix measured by the target machine;
the transition matrix is calculated by adopting the following formula:
Am＝AtF
in the formula, AmRepresenting the spectral data matrix measured by the source machine, AtA matrix of spectral data measured by the target machine is represented,
is AtThe generalized inverse matrix of (1) is an n × n matrix (n is the number of points in the spectral wavelength measurement range), and F denotes a transfer matrix.
The comparison group 1 selects a source machine hamamatsu C10082CAH spectrometer and a target machine ocean Maya2000Pro spectrometer, the comparison group 2 selects a source machine hamamatsu C10082CAH spectrometer and a target machine Olympic spectrum space ATP2000 spectrometer, and the comparison group 3 selects a source machine ocean Maya2000Pro spectrometer and a target machine Olympic spectrum space ATP2000 spectrometer.
S5: the water sample to be detected is detected for the water quality parameter COD by using a target machine to obtain the spectral data (spectral data before standardization) of the spectrometer to be detected, then the spectral data is brought into a standardization model, and the standardized spectral data of the water sample to be detected is obtained through calculation, wherein the result is shown in figure 3.
The standardized model adopts the following formula:
Astd＝Aunknown*F
in the formula, AunknownNormalized spectrum, AstdRepresenting the spectrum of the spectrometer to be tested, and F representing a transfer matrix;
the results of the standardized evaluation of the water absorption spectrum of each concentration measured by three sets of comparison experiments are shown in FIG. 4.
As can be seen from fig. 4, in the evaluation of the ultraviolet-visible absorption spectrum data standardization index of eight groups of sample solutions corresponding to 3 groups of instruments, except that the correlation coefficient of 20mg/L in the comparison group 1 is kept unchanged, in the comparison groups 2 and 3, the concentration distribution is from 20mg/L to 800mg/L, the correlation coefficient of the standardized data is increased compared with that of the original data, and as is apparent from fig. 4(a), the value after standardization is higher than the value before standardization, that is, the correlation coefficients of the two curves are improved, that is, the algorithm standardization improves the similarity of the curves; from the variance index, the concentration distribution is from 20-800mg/L, the variance of 8 groups of sample solution is greatly reduced after standardization compared with that before standardization, and as is obvious from the graph (b) in FIG. 4, the algorithm standardization is shown to reduce the variance between two curves, so that the fluctuation of the data of the corresponding wave band points of the two curves is reduced, namely the standardization effect is better; according to analysis from the peak offset Po, the peak offsets of the 8 groups of sample solutions in the three groups of comparative experiments are all about 1, and as is apparent from fig. 4(c), the error between the group 1 and the "1" is 0.0028% to 0.0736%, the error between the group 2 and the "1" is 0.0005% to 0.054%, and the error between the group 3 and the "1" is 0.004% to 0.045%, which are small, and thus it is indicated that the curve characteristics are not sacrificed when the normalization algorithm is used for curve normalization. Through calculation, after normalization by adopting an EWMA-PCA-based normalization algorithm, the correlation coefficient can reach 99.5765%, the variance can reach 0.0823%, and the peak offset can be reduced to 0.0005%, all of the three indicate that the algorithm is in good development in three groups of instruments of a C10082CAH spectrometer and a Maya2000Pro spectrometer, a C10082CAH spectrometer and an ATP2000 spectrometer, and a Maya2000Pro spectrometer and an ATP2000 spectrometer, and in a data normalization process, the normalization algorithm is effective, better in performance and good in feasibility.
S6: and (5) carrying out normalized data processing on the normalized spectral data obtained in the step (S5), carrying out fitting regression, and then obtaining the water quality parameter value of the water sample to be detected according to the regressed water quality spectral model.
In conclusion, the method for measuring the water quality parameters can be well applied to different comparison spectrometers, after the method is adopted, the correlation coefficient reaches 99.5765%, the variance reaches 0.0823%, and the peak offset can be reduced to 0.0005%.
Finally, the above embodiments are only used for illustrating the technical solutions of the present invention and not for limiting, although the applicant has described the present invention in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention and shall be covered by the claims of the present invention.
Claims (6)
1. A method for measuring water quality parameters based on spectral data standardization is characterized by comprising the following steps:
s1: obtaining a plurality of samples with different water sample parameters;
s2: collecting and denoising the ultraviolet-visible spectrum of the sample in the step S1 by using a multi-source spectrometer, and performing weighted average moving filtering processing on the denoised detection spectrum to obtain preprocessed detection spectrum data;
s3: performing feature extraction on the preprocessed detection spectrum obtained in the step S2 by adopting principal component analysis, and performing dimensionality reduction on the input water quality detection spectrum data to output a spectrum matrix A 'containing k characteristic dimensionalities'n×m；
S4: randomly selecting a standard multi-source spectrometer as a source machine, and determining a sample by the source machine according to the steps S2-S3 to obtain a spectrum data matrix measured by the source machine; selecting a multi-source spectrometer to be standardized as a target machine, determining the same sample by the target machine according to the steps S2-S3 to obtain a spectral data matrix measured by the target machine, and determining a transfer matrix by the spectral data matrix measured by the source machine and the spectral data matrix measured by the target machine;
s5: detecting water quality parameters of a water sample to be detected by using a target machine to obtain spectral data of a spectrometer to be detected, substituting the spectral data into a standardized model, and calculating to obtain standardized spectral data of the water sample to be detected;
the standardized model adopts the following formula:
Astd＝Aunknown*F
in the formula, AunknownNormalized spectrum, AstdRepresenting the spectrum of the spectrometer to be tested, and F representing a transfer matrix;
s6: and (5) carrying out normalized data processing on the normalized spectral data obtained in the step (S5), carrying out fitting regression, and then obtaining the water quality parameter value of the water sample to be detected according to the regressed water quality spectral model.
2. The method for measuring water quality parameters based on the spectral data standardization of claim 1, wherein the water sample parameters are COD, turbidity, ammonia nitrogen, TDS or TOC.
3. The method of claim 1, wherein the source machine and the target machine are independently selected from a DH2000 light source and Hamamatsu C10082CAH spectrometer, an American ocean optics Maya2000Pro spectrometer or a Xiamen spectral power ATP2000 spectrometer.
4. The method for measuring a water quality parameter based on the spectral data normalization according to claim 1, wherein the weighted-average moving filter process is calculated by using the following formula:
EWMA(N)＝λY(N)+(1-λ)EWMA(N-1),i＝1,2,…,n；
where ewma (N) represents an estimated value of spectral data at a band point N, y (N) represents a measured value of spectral data at a band point N, λ is a weighting factor, 0< λ <1 and decreases exponentially as the band point N increases, and N represents the sum of the number of measured band values.
5. The method for measuring water quality parameters based on spectral data normalization of claim 1, wherein the transfer matrix is calculated using the following formula:
Am＝AtF
in the formula, AmRepresenting the spectral data matrix measured by the source machine, AtA matrix of spectral data measured by the target machine is represented,
is AtThe generalized inverse matrix of (1) is an n × n matrix, n is the number of points in the spectral wavelength measurement range, and F represents a transfer matrix.
6. The method for measuring the water quality parameter based on the spectral data standardization according to claim 1, wherein the normalization data processing comprises the following steps:
adopting Z-score standardization to average the value of each characteristic into 0 and change the standard value into 1, and returning the absorbance value beyond the value range to the normal value A*：
In the formula, mu is the mean value of all absorbance data, and sigma is the standard deviation of all absorbance data;
the Z-score normalized data was then subjected to normalization data processing to map absorbance values between [0-1 ]:
in the formula (I), the compound is shown in the specification,
is the maximum value of the absorbance, and,
is the minimum value of absorbance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010037852.3A CN111198165A (en) | 2020-01-14 | 2020-01-14 | Method for measuring water quality parameters based on spectral data standardization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010037852.3A CN111198165A (en) | 2020-01-14 | 2020-01-14 | Method for measuring water quality parameters based on spectral data standardization |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111198165A true CN111198165A (en) | 2020-05-26 |
Family
ID=70744856
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010037852.3A Pending CN111198165A (en) | 2020-01-14 | 2020-01-14 | Method for measuring water quality parameters based on spectral data standardization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111198165A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111811998A (en) * | 2020-09-01 | 2020-10-23 | 中国人民解放军国防科技大学 | Method for determining strongly-absorbable biological particle component under target waveband |
| CN112082962A (en) * | 2020-09-04 | 2020-12-15 | 安徽思环科技有限公司 | Water quality ultraviolet-visible spectrum denoising and correcting method based on compressed sensing |
| CN113077019A (en) * | 2021-06-07 | 2021-07-06 | 芯视界（北京）科技有限公司 | Pollution type identification method and device and storage medium |
| CN113283072A (en) * | 2021-05-20 | 2021-08-20 | 重庆理工大学 | Water body COD detection method suitable for multi-scene conditions |
| CN113376114A (en) * | 2021-06-24 | 2021-09-10 | 北京市生态环境监测中心 | Water pollution tracing method based on ultraviolet-visible spectrum data |
| CN113466130A (en) * | 2021-06-30 | 2021-10-01 | 青岛崂应环境科技有限公司 | Ultraviolet gas analyzer model transfer method and system and ultraviolet gas analyzer |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102590132A (en) * | 2012-02-14 | 2012-07-18 | 浙江大学 | Method for measuring methanol content in methanol gasoline |
| CN103983595A (en) * | 2014-05-27 | 2014-08-13 | 重庆大学 | Water quality turbidity calculating method based on ultraviolet-visible spectroscopy treatment |
| CN105352898A (en) * | 2015-10-14 | 2016-02-24 | 浙江大学 | Turbidity compensation method for COD detection based on spectrometry |
| CN106990060A (en) * | 2017-03-24 | 2017-07-28 | 四川碧朗科技有限公司 | Water quality index monitor, cloud data center and system, Forecasting Methodology and water sample recognition methods |
| CN107643265A (en) * | 2016-07-22 | 2018-01-30 | 贵州中烟工业有限责任公司 | Spectrum standardization method |
| CN109086547A (en) * | 2018-08-23 | 2018-12-25 | 华南理工大学 | A kind of ageing of metal level measurement method |
| CN109444066A (en) * | 2018-10-29 | 2019-03-08 | 山东大学 | Model transfer method based on spectroscopic data |
| CN109444072A (en) * | 2018-10-12 | 2019-03-08 | 南京富岛信息工程有限公司 | The solution at deceitful peak in a kind of transfer of near-infrared spectroscopy |
-
2020
- 2020-01-14 CN CN202010037852.3A patent/CN111198165A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102590132A (en) * | 2012-02-14 | 2012-07-18 | 浙江大学 | Method for measuring methanol content in methanol gasoline |
| CN103983595A (en) * | 2014-05-27 | 2014-08-13 | 重庆大学 | Water quality turbidity calculating method based on ultraviolet-visible spectroscopy treatment |
| CN105352898A (en) * | 2015-10-14 | 2016-02-24 | 浙江大学 | Turbidity compensation method for COD detection based on spectrometry |
| CN107643265A (en) * | 2016-07-22 | 2018-01-30 | 贵州中烟工业有限责任公司 | Spectrum standardization method |
| CN106990060A (en) * | 2017-03-24 | 2017-07-28 | 四川碧朗科技有限公司 | Water quality index monitor, cloud data center and system, Forecasting Methodology and water sample recognition methods |
| CN109086547A (en) * | 2018-08-23 | 2018-12-25 | 华南理工大学 | A kind of ageing of metal level measurement method |
| CN109444072A (en) * | 2018-10-12 | 2019-03-08 | 南京富岛信息工程有限公司 | The solution at deceitful peak in a kind of transfer of near-infrared spectroscopy |
| CN109444066A (en) * | 2018-10-29 | 2019-03-08 | 山东大学 | Model transfer method based on spectroscopic data |
Non-Patent Citations (1)
| Title |
|---|
| DAVID A. WHITE 等: "Low Open-Area Endpoint Detection Using a PCA-Based T2 Statistic and Q Statistic on Optical Emission Spectroscopy Measurements", 《IEEE TRANSACTIONS ON SEMICONDUCTOR MANUFACTURING》, 31 May 2000 (2000-05-31) * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111811998A (en) * | 2020-09-01 | 2020-10-23 | 中国人民解放军国防科技大学 | Method for determining strongly-absorbable biological particle component under target waveband |
| CN112082962A (en) * | 2020-09-04 | 2020-12-15 | 安徽思环科技有限公司 | Water quality ultraviolet-visible spectrum denoising and correcting method based on compressed sensing |
| CN113283072A (en) * | 2021-05-20 | 2021-08-20 | 重庆理工大学 | Water body COD detection method suitable for multi-scene conditions |
| CN113077019A (en) * | 2021-06-07 | 2021-07-06 | 芯视界（北京）科技有限公司 | Pollution type identification method and device and storage medium |
| CN113376114A (en) * | 2021-06-24 | 2021-09-10 | 北京市生态环境监测中心 | Water pollution tracing method based on ultraviolet-visible spectrum data |
| CN113466130A (en) * | 2021-06-30 | 2021-10-01 | 青岛崂应环境科技有限公司 | Ultraviolet gas analyzer model transfer method and system and ultraviolet gas analyzer |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111198165A (en) | Method for measuring water quality parameters based on spectral data standardization | |
| CN107219188B (en) | A method of based on the near-infrared spectrum analysis textile cotton content for improving DBN | |
| WO2021232757A1 (en) | Method for improving mixture component identification precision by using raman spectra of known mixtures | |
| CN103175805B (en) | Method for determining indexes of COD and BOD5 in sewage through near infrared spectrometry | |
| CN117589741B (en) | Indocyanine green intelligent detection method based on optical characteristics | |
| CN114646606A (en) | Spectrum water quality detection method | |
| CN103018178A (en) | LPP (Local Preserving Projection)-based Infrared spectrometer calibration method | |
| CN114611582B (en) | Method and system for analyzing substance concentration based on near infrared spectrum technology | |
| CN109214423B (en) | Food quality discrimination analysis method based on dynamic and static data fusion | |
| CN116793979A (en) | Method for detecting chemical oxygen demand in water sample | |
| CN113686811A (en) | Spectral data processing method based on double sensors | |
| CN115420726A (en) | Method for rapidly identifying target object by using reconstructed SERS spectrum | |
| CN117194902B (en) | Noise data filtering method in spectrum measurement process | |
| CN111366573B (en) | Evaluation method based on LIBS spectral component analysis result | |
| Zhou et al. | Detection of chemical oxygen demand in water based on UV absorption spectroscopy and PSO-LSSVM algorithm | |
| CN110887798B (en) | Nonlinear full-spectrum water turbidity quantitative analysis method based on extreme random tree | |
| CN111896497A (en) | Spectral data correction method based on predicted value | |
| CN115078281B (en) | Water body substance component detection and calculation method based on picture spectral similarity | |
| CN112683816B (en) | Spectrum identification method for spectrum model transmission | |
| CN111795742B (en) | Dimension reduction method for single RGB image reconstruction spectrum | |
| CN115144362A (en) | Spectrum model self-adaption method based on standard calibration plate | |
| CN113252641A (en) | Substance identification method based on residual analysis under Raman spectrum | |
| CN112229817A (en) | Method for establishing soda saline-alkali soil heavy metal quantitative inversion model | |
| CN109507120A (en) | Solution cathode glow discharging emission spectrographic analysis method based on standard addition method combination wavelet transformation | |
| CN114235715B (en) | Full phosphorus content nondestructive monitoring method for karst plant leaves |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200526 |