CN111307764A - Transmission type turbidity measuring method and device based on partial least square method - Google Patents

Transmission type turbidity measuring method and device based on partial least square method Download PDF

Info

Publication number
CN111307764A
CN111307764A CN202010157242.7A CN202010157242A CN111307764A CN 111307764 A CN111307764 A CN 111307764A CN 202010157242 A CN202010157242 A CN 202010157242A CN 111307764 A CN111307764 A CN 111307764A
Authority
CN
China
Prior art keywords
spectral data
turbidity
partial
data
transmission
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010157242.7A
Other languages
Chinese (zh)
Inventor
付晓琳
王鸿
孙霞
程丽宁
王致杰
王扬
李占辉
陶梦琳
张智禹
渠省委
刘衡
张思佳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Dianji University
Original Assignee
Shanghai Dianji University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Dianji University filed Critical Shanghai Dianji University
Priority to CN202010157242.7A priority Critical patent/CN111307764A/en
Publication of CN111307764A publication Critical patent/CN111307764A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G01MEASURING; TESTING
    • G01NINVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/59Transmissivity

Abstract

The invention relates to a transmission type turbidity measuring method and device based on a partial least square method, wherein the method comprises the following steps: a spectral data acquisition step: collecting spectral data of a turbidity substance; preprocessing the spectral data: removing accidental errors of the collected spectral data; and (3) data analysis step: and (3) performing principal component extraction on the preprocessed spectral data by adopting a partial least square method, and then performing turbidity measurement. Compared with the prior art, the method provided by the invention can be used for analyzing the spectral data of the turbidity substance by the partial least square method, can effectively remove invalid data, and has the advantages of high turbidity measurement precision, reliable measurement result and the like.

Description

Transmission type turbidity measuring method and device based on partial least square method
Technical Field
The invention relates to the field of transmission-type turbidimeters, in particular to a transmission-type turbidity measuring method and device based on a partial least square method.
Background
The turbidity measuring method mainly comprises a transmitted light method, a scattered light method and a scattering transmission ratio measuring method at present, but compared with the scattered light measuring method and the scattering transmission ratio measuring method, the transmitted light turbidity measuring method has the problems of low measuring precision and poor anti-interference capability under low turbidity, the transmission measuring method mainly comprises a multiple linear regression method and a principal component analysis method, the multiple linear regression method can realize regression modeling at the same time, but multiple regression needs to be carried out on dependent variables one by one, and the efficiency is low; the principal component analysis method has a simple data structure, but cannot effectively solve the complex problem of high dimensionality, so that the method has important significance in researching the transmission type turbidity measurement method.
Disclosure of Invention
The invention aims to overcome the defect that the transmission type turbidity measuring method in the prior art has low measuring precision under low turbidity, and provides a transmission type turbidity measuring method and a transmission type turbidity measuring device based on a partial least square method.
The purpose of the invention can be realized by the following technical scheme:
a transmission type turbidity measuring method based on partial least square method comprises the following steps:
a spectral data acquisition step: collecting spectral data of a turbidity substance;
preprocessing the spectral data: removing accidental errors of the collected spectral data;
and (3) data analysis step: and (3) performing principal component extraction on the preprocessed spectral data by adopting a partial least square method, and then performing turbidity measurement.
Further, in the spectral data acquisition step, the spectral data of the turbidity material is acquired by adopting a multi-optical-path length method.
Further, the spectral data preprocessing step specifically includes performing centering processing and compression processing on the acquired spectral data at the same time.
Further, the spectral data is a data set { X, Y }, where X is an n × m input matrix composed of m wavelength point parameters and n times of measurement values, Y is an n × 2 output matrix composed of n times of measurement values of fat emulsion and n times of measurement values of turbid water, m is a positive integer, n is a positive integer, and the measurement values are obtained by the spectral data acquisition step.
Further, the m wavelength point parameters are 512 wavelengths in total, which are output in the dynamic range of the turbidity meter, and the optical path length.
Further, in the data analysis step, the turbidity measurement is specifically performed by a graphical analysis method.
Further, the data analysis step further comprises modeling the preprocessed spectral data by a partial least squares method, thereby performing turbidity measurement.
Further, the main component extraction of the preprocessed spectral data by the partial least square method is specifically that the partial least square method is used for extracting a component t from X and Y respectively1And u1And the following conditions are required to be satisfied:
a)t1and u1Should carry as much information as possible about the variations in their respective data tables;
b)t1and u1Can be maximized.
These two requirements indicate that t1And u1Data X and Y should be represented as much as possible; component t of simultaneous independent variables1For the component u of the dependent variable1Has strong dissolving ability. The optimization conditions are as follows:
max<E0w1,F0c1
the constraint conditions are as follows:
Figure BDA0002404486620000021
Figure BDA0002404486620000022
in the formula: e0Is a data matrix after X is processed by standardization; f0Is a data matrix of Y after standardized processing; w is a1Is E0Is a unit vector, i.e. | w1‖=1;c1Is F0Is a unit vector, i.e. iic1| ═ 1. Finding w1And c1Then, the following can be obtained:
t1=E0w1
u1=F0c1
residual matrix E of regression equation1And F1Comprises the following steps:
Figure BDA0002404486620000023
Figure BDA0002404486620000024
in the formula: p is a radical of1And q is1Is the corresponding regression coefficient vector, for the residual matrix E1、F1Similar decomposition is performed until the kth principal element, at which time the residual matrix EkAnd FkAlmost no more valid information is contained. Determining the number k of the pivot elements by a cross-checking method, and obtaining the relation between X and Y as follows:
Figure BDA0002404486620000031
Figure BDA0002404486620000032
therefore, the principal component extracted in the partial least square method is the result of the original data fusion processing, invalid data can be effectively eliminated, and the method is a comprehensive variable with interpretation capability and reflecting the characteristics of the described object, and is suitable for turbidity measurement of a multi-optical path transmission type turbidity meter.
The invention also provides a transmission type turbidity measuring device based on the partial least square method, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the method.
Compared with the prior art, the invention has the following advantages:
(1) the invention adopts partial least square method to extract the main component of the spectrum data after removing the accidental error, then carries out the turbidity measurement, the extracted main component is the result after the fusion processing of the original data, it can effectively eliminate the invalid data, and is a comprehensive variable with interpretation ability and reflecting the described object characteristics, compared with the traditional measurement method, the invention has better analysis effect, and the turbidity measurement result is more reliable, the integrity is stronger, and the precision of the turbidity measurement is improved.
(2) The invention measures the spectra of different turbidity substances by a multi-optical-path length method, and the prediction error of the multi-optical-path length modeling is generally smaller than that of the single-optical-path length modeling according to the complete interactive verification result, and generally, the concentration residual value measured by the multi-optical-path length method is one order of magnitude smaller than that of the single-optical-path length method.
(3) The invention adopts partial least square method to model the preprocessed spectral data, the modeling method can better solve many problems which can not be solved by common multiple regression in the past, realizes the comprehensive application of various data analysis methods, and has higher integrating degree with the infrared multi-optical path transmission type turbidity measuring method.
(4) The coring treatment of the present invention can bring many technical conveniences; the compression treatment can eliminate the parameter dimension effect, so that each variable has equal expressive force; and the centralization treatment and the compression treatment are carried out simultaneously, so that the requirement of a partial least square method is met.
Drawings
FIG. 1 is a schematic flow diagram of a transmission turbidity measurement method of the present invention;
FIG. 2 is a diagram showing the result of PLS extraction of the principal components from the experimental data with multiple optical path lengths;
FIG. 3 is a graph showing comparison between experimental data and fitting data of multi-optical path long fat emulsion;
FIG. 4 is a graph showing comparison between experimental data and fitting data of multi-optical path long turbid water;
FIG. 5 is a schematic diagram showing a comparison of single optical path length and multiple optical path length residuals of fat emulsion;
FIG. 6 is a graph showing a comparison of single and multiple optical path length residuals for turbid water.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, but the scope of the present invention is not limited to the following embodiments.
Example 1
The embodiment provides a transmission type turbidity measuring method based on a partial least square method, which comprises the following steps:
a spectral data acquisition step S1: collecting spectral data of a turbidity substance by adopting a multi-optical path length method;
spectral data preprocessing step S2: removing accidental errors of the collected spectral data, and performing centralized processing and compression processing on the collected spectral data simultaneously in the embodiment;
data analysis step S3: and (3) performing principal component extraction and modeling on the preprocessed spectral data by adopting a partial least square method, and then performing turbidity measurement.
The spectral data, partial least squares and specific examples are described in detail below, respectively.
1. Spectral data
The spectral data is a data set { X, Y }, wherein X is an n multiplied by m input matrix formed by m wavelength point parameters and n measured values, Y is an n multiplied by 2 output matrix formed by n measured values of fat milk and n measured values of turbid water, m is a positive integer, n is a positive integer, and the measured values are obtained through the spectral data acquisition step.
2. Partial least squares method (PLS)
In general, actual data all have some degree of co-linearity, and PLS considers the redundancy of data by first performing orthogonal decomposition on X and Y simultaneously (assuming that X and Y have been subjected to normalization preprocessing), i.e.
PLS extracts component t in X and Y, respectively1And u1And the following conditions are required to be satisfied:
a)t1and u1Should carry as much information as possible about the variations in their respective data tables;
b)t1and u1Can be maximized.
These two requirements indicate that t1And u1Data X and Y should be represented as much as possible; component t of simultaneous independent variables1For the component u of the dependent variable1Has strong dissolving ability. The optimization conditions are as follows:
max<E0w1,F0c1>(1)
the constraint conditions are as follows:
Figure BDA0002404486620000051
Figure BDA0002404486620000052
in the formula: e0Is a data matrix after X is processed by standardization; f0Is a data matrix of Y after standardized processing; w is a1Is E0Is a unit vector, i.e. | w1‖=1;c1Is F0Is a unit vector, i.e. iic1| ═ 1. Finding w1And c1Then, the following can be obtained:
t1=E0w1(4)
u1=F0c1(5)
residual matrix E of regression equation1And F1Comprises the following steps:
Figure BDA0002404486620000053
Figure BDA0002404486620000054
in the formula: p is a radical of1And q is1Is the corresponding regression coefficient vector, for the residual matrix E1、F1Similar decomposition is performed until the kth principal element, at which time the residual matrix EkAnd FkAlmost no more valid information is contained. Determining the number k of the pivot elements by a cross-checking method, and obtaining the relation between X and Y as follows:
Figure BDA0002404486620000055
Figure BDA0002404486620000056
therefore, the main components extracted in the PLS method are the result of the fusion processing of the original data, invalid data can be effectively eliminated, and the PLS method is a comprehensive variable with interpretability and reflecting the characteristics of the described object, and is suitable for turbidity measurement of a multi-optical path transmission type turbidity meter.
The embodiment also provides a transmission type turbidity measuring device based on the partial least square method, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the transmission type turbidity measuring method based on the partial least square method.
3. Detailed description of the preferred embodiments
1) Multi-optical path length experiment data preprocessing
The data is centralized-compressed at the same time, and the centralized processing can bring many technical conveniences; the compression process can eliminate parametric dimension effects, making each variable equal in expressive power.
2) Modeling by PLS method
The total 512 wavelengths of the wavelength and the optical path length output in the dynamic range of the turbidity meter are used as data columns, and the data measured by the optical path length at each turbidity point are used as data rows to form an input matrix of 210 multiplied by 512. The actual values of the turbidity of fat emulsion and turbid water at different optical paths constitute a 210 x 2 output matrix. According to the component extraction mode adopted by PLS, 10 principal components t are extracted1,t2,…,tmThe requirements are met. The extracted results are analyzed and explained by the graph analysis function of PLS, as shown in FIG. 2. The curves of the actual measured values and the fitting values are shown in fig. 3 and 4, and it can be seen from the graphs that the model fitting effect is better. The single-optical-path and multi-optical-path residual error comparison of fat emulsion and turbid water is shown in fig. 5 and 6, wherein the abscissa is the serial numbers of samples with different concentrations, and the ordinate is the residual error value between the actual value and the fitting value.
Since determination of turbidity of a liquid plays an extremely important role in quality detection of industrial and agricultural products, food safety, and the like, it is extremely important to accurately measure turbidity of a liquid and also an important basis for quantitative analysis of a liquid. In the embodiment, spectra of different turbidity substances are measured by a multi-optical path length method, and accidental errors of experimental data are removed by a superposition average method. And modeling by using a PLS method to realize comprehensive application of various data analysis methods. The experimental result shows that the multi-optical path length residual value is obviously smaller than the single optical path length, and particularly for turbid water, the multi-optical path length residual value is one order of magnitude shorter than the single optical path length. Therefore, the PLS modeling is utilized under the multi-optical path measurement method, the regression modeling of multiple dependent variables to multiple independent variables is realized, and the method has important significance for improving the measurement accuracy under low turbidity.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.

Claims (8)

1. A transmission type turbidity measuring method based on partial least square method is characterized by comprising the following steps:
a spectral data acquisition step: collecting spectral data of a turbidity substance;
preprocessing the spectral data: removing accidental errors of the collected spectral data;
and (3) data analysis step: and (3) performing principal component extraction on the spectral data subjected to the spectral data preprocessing step by adopting a partial least square method, and then performing turbidity measurement.
2. The transmission-type turbidity measuring method according to claim 1, wherein the spectral data collection step is a multi-optical path length method for collecting the spectral data of the turbidity material.
3. The transmission-type turbidity measurement method based on partial least squares of claim 1, wherein the spectral data preprocessing step is to perform the centering processing and the compressing processing on the collected spectral data at the same time.
4. The transmission-based turbidity measurement method according to claim 1, wherein the spectral data is a data set { X, Y }, where X is an n X m input matrix consisting of m wavelength point parameters and n measurements, Y is an n X2 output matrix consisting of n measurements of fat emulsion and n measurements of turbid water, m is a positive integer, and n is a positive integer, and the measurements are obtained by the spectral data acquisition step.
5. The transmission-based turbidity measurement method according to claim 4, wherein said m wavelength point parameters are 512 wavelengths in total, which are outputted in the dynamic range of the turbidity meter, and the optical path length.
6. The transmission-based turbidity measurement method according to claim 1, wherein in the data analysis step, the turbidity measurement is performed by a graphical analysis method.
7. The method of claim 1, wherein the step of analyzing the data further comprises modeling the preprocessed spectral data using partial least squares to perform turbidity measurements.
8. A transmission-type turbidity measuring apparatus based on partial least squares, comprising a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the method according to any one of claims 1 to 7.
CN202010157242.7A 2020-03-09 2020-03-09 Transmission type turbidity measuring method and device based on partial least square method Pending CN111307764A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010157242.7A CN111307764A (en) 2020-03-09 2020-03-09 Transmission type turbidity measuring method and device based on partial least square method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010157242.7A CN111307764A (en) 2020-03-09 2020-03-09 Transmission type turbidity measuring method and device based on partial least square method

Publications (1)

Publication Number Publication Date
CN111307764A true CN111307764A (en) 2020-06-19

Family

ID=71149508

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010157242.7A Pending CN111307764A (en) 2020-03-09 2020-03-09 Transmission type turbidity measuring method and device based on partial least square method

Country Status (1)

Country Link
CN (1) CN111307764A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1982003460A1 (en) * 1981-03-31 1982-10-14 Coogan Clive Keith Application of optical fibre probes
JP2004177122A (en) * 2002-11-22 2004-06-24 Kurita Water Ind Ltd Water quality detection method and operation method of water purifying apparatus
CN102053083A (en) * 2010-11-09 2011-05-11 清华大学 Method for on-line measurement of coal quality characteristics based on partial least squares method
JP4986196B1 (en) * 2011-09-21 2012-07-25 秀樹 相澤 Method for evaluating the stability of a liquid in an emulsified state
US20150369727A1 (en) * 2014-06-24 2015-12-24 Shimadzu Corporation Method and apparatus for analyzing the concentration of materials in suspension

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1982003460A1 (en) * 1981-03-31 1982-10-14 Coogan Clive Keith Application of optical fibre probes
JP2004177122A (en) * 2002-11-22 2004-06-24 Kurita Water Ind Ltd Water quality detection method and operation method of water purifying apparatus
CN102053083A (en) * 2010-11-09 2011-05-11 清华大学 Method for on-line measurement of coal quality characteristics based on partial least squares method
JP4986196B1 (en) * 2011-09-21 2012-07-25 秀樹 相澤 Method for evaluating the stability of a liquid in an emulsified state
US20150369727A1 (en) * 2014-06-24 2015-12-24 Shimadzu Corporation Method and apparatus for analyzing the concentration of materials in suspension

Non-Patent Citations (10)

* Cited by examiner, † Cited by third party
Title
中国物资再生协会编, 中国财富出版社 *
张德涛等: "近红外光谱结合不同偏最小二乘法快速检测镇江香醋的浑浊度", 《中国酿造》 *
张德涛等: "近红外光谱结合不同偏最小二乘法快速检测镇江香醋的浑浊度", 《中国酿造》, 31 December 2012 (2012-12-31), pages 169 - 172 *
方坷昊等: "紫外-可见光谱法水质监测中浊度影响的非线性校正", 《传感器与微系统》 *
方坷昊等: "紫外-可见光谱法水质监测中浊度影响的非线性校正", 《传感器与微系统》, no. 10, 26 September 2018 (2018-09-26) *
曹引等: "基于离散粒子群和偏最小二乘的水源地浊度高光谱反演", 《农业机械学报》 *
曹引等: "基于离散粒子群和偏最小二乘的水源地浊度高光谱反演", 《农业机械学报》, no. 01, 25 January 2018 (2018-01-25) *
李刚: "尿微量白蛋白的多波段多光程光谱检测", 《分析化学》, no. 04, pages 157 - 160 *
林凌: "多光程长测量高散射物质浓度", 《纳米技术与精密工程》, no. 01, pages 52 - 56 *
王焱: "采用多光程长建模方法检测血液成分含量", 《分析化学》, no. 10, pages 106 - 109 *

Similar Documents

Publication Publication Date Title
Shao et al. Visible/near-infrared spectra for linear and nonlinear calibrations: a case to predict soluble solids contents and pH value in peach
CN107219188B (en) A method of based on the near-infrared spectrum analysis textile cotton content for improving DBN
CN105067531A (en) Mango quality nondestructive detection method and mango quality nondestructive detection apparatus
CN104965973B (en) A kind of Apple Mould Core multiple-factor Non-Destructive Testing discrimination model and method for building up thereof
CN113008805B (en) Radix angelicae decoction piece quality prediction method based on hyperspectral imaging depth analysis
CN106841083A (en) Sesame oil quality detecting method based on near-infrared spectrum technique
CN112200770A (en) Tumor detection method based on Raman spectrum and convolutional neural network
CN110749565A (en) Method for rapidly identifying storage years of Pu&#39; er tea
CN104596979A (en) Method for measuring cellulose of reconstituted tobacco by virtue of near infrared reflectance spectroscopy technique
CN109001182B (en) Raman spectrum nondestructive testing method for alcohol content in closed container
CN104596976A (en) Method for determining protein of paper-making reconstituted tobacco through ear infrared reflectance spectroscopy technique
CN107247033B (en) Identify the method for Huanghua Pear maturity based on rapid decay formula life cycle algorithm and PLSDA
CN113310929A (en) Soybean powder doped in high-temperature sterilized milk and spectral identification method of doping proportion thereof
CN113324940A (en) Spectrum grading method for super-high-quality milk, high-protein special milk, high-milk-fat special milk and common milk
CN101231270B (en) Method for determining index composition content of Qingkailing injection intermediate body and finished product
CN111307764A (en) Transmission type turbidity measuring method and device based on partial least square method
CN105158178B (en) Navel orange pol detection rapid modeling method based on EO-1 hyperion through-transmission technique spectral peak area
CN116858822A (en) Quantitative analysis method for sulfadiazine in water based on machine learning and Raman spectrum
CN116399836A (en) Cross-talk fluorescence spectrum decomposition method based on alternating gradient descent algorithm
CN104596982A (en) Method for measuring pectin of paper-making reconstituted tobacco by near-infrared diffuse reflection spectrum technology
CN113933247A (en) Construction method of fruit maturity detection model
CN115420726A (en) Method for rapidly identifying target object by using reconstructed SERS spectrum
CN114550843A (en) Model for predicting monosaccharide composition and content in traditional Chinese medicine polysaccharide and construction method and application thereof
CN106442376B (en) Assessment method for quality of assay data
CN111912823A (en) Multi-component pesticide residue fluorescence detection analysis method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20200619