CN109186474A - A kind of aerosol optical depth inverting bias correction method based on ridge regression - Google Patents
A kind of aerosol optical depth inverting bias correction method based on ridge regression Download PDFInfo
- Publication number
- CN109186474A CN109186474A CN201811184427.6A CN201811184427A CN109186474A CN 109186474 A CN109186474 A CN 109186474A CN 201811184427 A CN201811184427 A CN 201811184427A CN 109186474 A CN109186474 A CN 109186474A
- Authority
- CN
- China
- Prior art keywords
- inverting
- model
- ridge regression
- algorithm
- optical depth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
- G01B11/06—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness for measuring thickness ; e.g. of sheet material
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N2201/00—Features of devices classified in G01N21/00
- G01N2201/12—Circuits of general importance; Signal processing
- G01N2201/127—Calibration; base line adjustment; drift compensation
Abstract
The present invention discloses a kind of aerosol optical depth inverting bias correction method based on ridge regression, belongs to computer meteorology applied technical field.First with " dark target " algorithm inverting aerosol optical depth and as initialization result；Inverting bias correction frame is constructed again corrects inversion result, in series system, the aerosol optical depth that " dark target " algorithm obtains corrects inversion result as a validity feature of ridge regression model, in parallel way, inversion result is corrected with the residual error between the inversion result and true value of the direct learning physics model of ridge regression model.The present invention has sufficiently merged the advantages of physical model and machine learning model, aerosol optical depth value in series system using physical model inverting is as the feature of ridge regression model, the residual error between the inversion result and true value of the direct learning physics model of ridge regression model is utilized in parallel way, preferably inverting aerosol optical depth improves inverting performance and inverting accuracy rate.
Description
Technical field
The aerosol optical depth inverting bias correction method based on ridge regression that the present invention relates to a kind of, belongs to computer gas
As applied technical field.
Background technique
Atmospheric aerosol refers to the multiphase that suspend solid in an atmosphere and liquid particle and carrier gas collectively constitute
System is the important component of atmosphere.It is generally believed that the diameter of atmospheric aerosol particle several nanometers to tens microns it
Between.The main source of atmospheric aerosol include the release (flue dust of such as anthropogenic discharge, the sand and dust etc. of desert area) of each provenance with
And (nitrogen dioxide and sulfur dioxide in such as atmosphere can be converted to corresponding nitrate to gas-and sulfate gas is molten for grain conversion
Glue), characteristic outstanding is the height space-time changeability of its physicochemical properties.Aerosol is in the earth-atmospheric radiation revenue and expenditure
Important role is play in balance and Global climate change, and mainly weather is become by directly or indirectly two kinds of mechanism of effect
Change has an impact.Direct effect refers to that particulate can scatter and absorb solar radiation, in turn results in and reaches ground too
The variation of positive radiation energy, big pneumatic jack solar radiant energy and inner-atmopshere solar radiant energy, to influence the radiation of ground vapour system
Revenue and expenditure；Indirect effect refers to that the presence of particulate can change the physics and microphysical property of cloud, and changes cloud in turn
Radiation characteristic, to indirectly influence distribution of the solar radiant energy in ground vapour system.
In addition to this, aerosol also drastically influences daily life.For China, end up to the present,
Measurement of the aerosol to municipal pollution, usually used three relevant to aerosol important environmental pollution indexs: total suspension
Grain object, PM10(particulate matter of the partial size between 2.5 microns and 10 microns), PM2.5(particulate matter of the partial size less than 2.5 microns).Gas
Colloidal sol particle is sucked by human body, can be accumulated in respiratory system, and a variety of diseases are caused, especially with fine particle PM2.5To the mankind
It endangers bigger.Compared with thicker Atmospheric particulates, fine particle PM2.5Partial size is small, and is rich in a large amount of noxious material.Due to
Its partial size is small, therefore after being sucked by human body, can be deep into bronchiole and alveolar, directly affect the ventilatory function of lung, make body
It is easy to be in anaerobic condition, to cause a variety of diseases.In addition, when the atmospheric aerosol particle in atmosphere sharply increases, just
Haze weather can be caused, atmospheric visibility is reduced, to cause serious air environmental pollution.
How to quantify aerosol it is break-even on the earth-atmospheric radiation influence be current climate research ultimate challenge it
One.Aerosol optical depth (AOD) is the important measure index for describing optical properties of aerosol, refers generally to disappearing for flood aerosol
The integral of backscatter extinction logarithmic ratio in vertical direction.AOD has become many earth observation satellites (such as intermediate resolution imager
MODIS, multi-angle imaging spectrometer MISR etc.) main Atmosphere Product.The radiation data that these passings of satelline obtain carrys out inverting
Corresponding AOD value.Existing satellite Retrieval algorithm is mostly based on physical kinetics model.This class model needs to consider to influence radiation
The physical descriptor (such as atmospheric conditions, solar azimuth and zenith angle, the azimuth of sensor and zenith angle etc.) of characteristic, and benefit
Complicated mathematical formulae is constructed, with radiation transfer equation to indicate the relationship between variable.In order to simplify calculate, usually it is assumed that
In the case where aerosol model, atmospheric model and corresponding geometrical relationship, utilize atmospheric radiation transmission (such as 6S model)
Look-up table is constructed, the apparent reflectance of the Reflectivity for Growing Season of moonscope, corresponding geometric parameter and corresponding wave band is then passed through
Corresponding AOD value is searched from look-up table.However, being difficult to consider all relevant due to the complicated earth-atmosphere reciprocation
Physical descriptor simultaneously accurately establishes the relationship between them.In addition, search look-up table calculation amount is larger, it usually needs expend very much
Time.Currently, MODIS satellite is the 6th generation " dark target " algorithm in the algorithm of Beijing and Xianghe website inverting AOD.Although the 6th
Certain enhancing and code reparation are carried out on the basis of five generations for algorithm, but they are based on same principle.Firstly, right
The reflectivity of atmosphere top layer is screened within the scope of 10 × 10 square kilometres, is removed by the unavailable of the coverings such as cloud, desert, ice and snow
Pixel, and " dark target " pixel is identified using 2.12 micron spectral wave bands；Then, before removing in 0.66 micron spectral wave band
50% bright pixel and 20% dark pixel, the reflectivity of residual pixel is averaged；Secondly, big according to 2.12 micron wavebands
The reflectivity of pneumatic jack layer derives the Reflectivity for Growing Season of visible light wave range (0.47 micron and 0.66 micron)；Finally, by thin mode
Aerosol model is weighted with the aerosol model of roughcast state and combines, in the hope of the value of AOD.The mode of aerosol is by observation station
The factors such as position, the season of point determine." dark target " algorithm cannot be used for bright earth surface area, such as desert, snowfield etc., because can
The relationship of light-exposed wave band Reflectivity for Growing Season and 2.12 micron waveband atmosphere top layer reflectivity is invalid in these regions.
A kind of alternative solution of physical model is machine learning model, which can be regarded as returning by data-driven
Return method.Atmospheric aerosol inverting can be regarded as the regression problem in machine learning, and study spectral value and aerosol optical are thick
Mapping function between degree.Firstly, utilizing matched Satellite Observations and ground based observa tion data one regression model of training；So
Afterwards, the AOD value of moonscope is given using trained forecast of regression model.Neural network and support vector machines are this class models
In most commonly used two kinds of models, because of the non-linear relation that they can be complicated between approximate moonscope and ground based observa tion.
Different from physical model, machine learning model does not need to pre-suppose that relationship, functional form etc. between different variables, calculation amount
It is small, it can be applied to different inverting scenes.More importantly enough training samples are given, machine learning model energy
It is enough to obtain inversion accuracy more higher than physical model.
It is well known that ridge regression is a kind of multiple linear regression model, 2 norms are dissolved into least square model by it.
Having benefited from the simplicity and flexibility of model, ridge regression model has been widely used in every field, such as recognition of face,
Bioinformatics, Chemoinformatics etc..In recent years, some scholars are attempted to utilize ridge regression model, directly from remotely-sensed data inverting
Biophysics parameter (such as chlorophyll concentration, leaf area index etc.).Remotely-sensed data, which generally comprises, does not largely mark sample,
In order to sufficiently excavate the effective information wherein contained, it is contemplated that using ridge regression model, to capture the non-linear of remotely-sensed data
Distribution character.
For machine learning model, how the pass that effective expression is the application of its success is carried out to remotely-sensed data
Key, existing model often directly use the spectral value of subband as feature.However, due to the influence of calibration error etc.,
Remotely-sensed data would generally be by various degenerations, such as noise pollution, stripe interference, shortage of data.No any pre-
In the case where processing, the data for directly using these to be disturbed can reduce the inverting performance of model as feature.In addition, engineering
Model is practised to be difficult from physical significance to explain its result.In recent years, some scholars are using physical model as priori knowledge,
It is dissolved into machine learning model, growing process is modeled.By the inspiration of these work, it is proposed that a kind of base
It is respective excellent to combine physical model and machine learning model in the aerosol optical depth inverting bias correction model of ridge regression
Point.This inverting bias correction model first derives initial inversion result with the physical model based on " dark target " algorithm, so
The deviation for correcting " dark target " algorithm inversion result with ridge regression model afterwards, to improve the inverting accuracy of model.
Summary of the invention
The technical problem to be solved by the present invention is to need to preset correlated variables for existing physical model to establish aerosol relationship
Function and amount of calculation is big, and machine learning directly uses and is disturbed data and reduces the inverting of model as Characteristics of The Aerosol
The deficiencies of performance, proposes a kind of aerosol optical depth inverting bias correction method based on ridge regression, first using based on " dark mesh
The physical model of mark " algorithm derives initial inversion result, and ridge regression model is recycled to correct " dark target " algorithm inverting knot
The deviation of fruit, the advantage of combination physical model and machine learning model respectively, improves the inverting accuracy of model.
It is entangled in order to solve the above technical problems, the present invention provides a kind of aerosol optical depth inverting deviation based on ridge regression
Correction method, first using " dark target " algorithm inverting aerosol optical depth and as initialization result；Then building is anti-
Bias correction frame is drilled, inversion result is corrected by series connection and two ways in parallel, in series system, by " dark mesh
A validity feature of the aerosol optical depth that mark " algorithm obtains as ridge regression model, corrects inversion result；?
In parallel way, with the residual error between the inversion result and true value of the direct learning physics model of ridge regression model, to inverting knot
Fruit is corrected.
The specific steps of the aerosol optical depth inverting bias correction method based on ridge regression are as follows:
(1) it first collects the reflectivity data of atmosphere top layer and it is screened, removal is covered by cloud, desert, ice and snow etc.
Unavailable pixel, then by " dark target " algorithm inverting aerosol optical depth AOD value, and tied as initialization
Fruit；
(2) by " dark target " algorithm and ridge regression model f (x；W) it is coupled, forms following inverting bias correction model:
In formula,Indicate the AOD value of estimation, ymodIndicate " dark target " algorithm inverting as a result, x indicates that mode input is special
Sign, w is the parameter vector in ridge regression model, symbolIndicate coupling operation；Under normal conditions, not general coupling behaviour
Make, actual design depends on the existence form of domain knowledge；
(3) by the inverting bias correction model of series connection and two ways in parallel, inverting bias correction is carried out；In series connection side
In formula, the model of step (2) is rewritten as following form:
Wherein,The transposition of subscript T representing matrix, the optimal value of parameter w are denoted as w*；
Series system is by the result y of " dark target " algorithm invertingmodRidge regression model f (x is input to as a kind of feature；w)
In, i.e., ridge regression model f (x is input to using all AOD values of " dark target " algorithm inverting as a kind of feature；W) it in, is returned by ridge
Return model to carry out automatic screening to feature, the inversion result of " dark target " algorithm is corrected.
In the model measurement stage, first with one initial value of " dark target " algorithm inverting, then by formulaIt pushes away
Lead the AOD value of correction, w*For the optimal value of parameter w.In series system, ridge regression model plays the role of bias correction；More
It is important that using all spectral values as feature, and automatic screening is carried out to feature with ridge regression model, enables model
Enough obtain better inverting performance.
In parallel way, the model of step (2) is rewritten as following form；
At this point, ridge regression model f (x；W) label becomes y-y from ymod；
Parallel way utilizes ridge regression model f (x；W) between the inversion result and true value for learning " dark target " algorithm
Residual error corrects the inversion result of " dark target " algorithm.
In the model measurement stage, " dark target " residual error between algorithm and true value is calculated first with ridge regression model, then lead to
Cross formulaTo estimate the AOD value of correction, w*For the optimal value of parameter w.In parallel way, ridge is returned
Model is returned to be used to residual error of the study " dark target " between algorithm and true value, the residual error learnt is anti-plus " dark target " algorithm
The AOD value drilled, to correct the AOD value of inverting；Physical model and machine learning model are combined.
" dark target " algorithm for inverting aerosol optical depth uses the 6th generation " dark target " of MODIS satellite
Algorithm.
Inverting bias correction frame in the method for the present invention has used ridge regression model to correct by " dark target " algorithm
Caused deviation.In series system, ridge regression model carries out automatic screening to feature, and model is enabled to obtain better inverting
Performance；In parallel way, ridge regression model is used to directly learn the residual error of " dark target " between algorithm and true value, so that mould
Type can obtain better inverting performance.
The ridge regression model in the method for the present invention is briefly described below.
Give l mark sampleSample is not marked with uWherein,D is feature sky
Between dimension.For ease of calculation, data can be usually expressed as to the form of matrix.Such as with X=[x1..., xl+u]TIt indicates
Entire data set, Xt=[x1..., xl]TIndicate training set, the transposition of subscript " T " representing matrix.
The core concept of linear regression is fitting function f (x；W)=xTW+b, so that predicted value and true value error is flat
Side and (loss function) are minimum, it may be assumed thatIn order to facilitate derivation, Ke Yi
Each variable xiOne element 1 of middle addition, so that parameter b is put into vector w.Assuming that Yt=[y1..., yl]T, then objective function
It can be write as the form of matrix:The solution of the function isHave however, in many practical applications, between sample different characteristic stronger correlation or
The dimension of feature is greater than the number of training sample, so that covariance matrixOften singular matrix.A kind of common solution
Scheme is that penalty term is added on w:
Wherein, α | | w | |2For lucky big vast promise husband's regular terms, statistically referred to as ridge regression.For the solution of this formula, " I " indicates the unit matrix of d × d dimension.
The present invention has sufficiently merged physical model and the advantage of machine learning model respectively, proposes a kind of based on ridge regression
Aerosol optical depth inverting bias correction frame, in series system utilize physical model inverting aerosol optical depth
Be worth feature as ridge regression model, in parallel way using the inversion result of the direct learning physics model of ridge regression model and
Residual error between true value corrects deviation, preferably inverting as caused by " dark target " algorithm by inverting bias correction frame
Aerosol optical depth, substantially increases inverting performance and inverting accuracy rate, has preferable inverting performance.Same real
Under the conditions of testing, accuracy rate of the inverting accuracy rate of the method for the present invention than " dark target " algorithm, Ridge Regression Modeling Method is high.
Detailed description of the invention
Fig. 1 is that the present invention is based on the schematic diagrams of the aerosol optical depth inverting bias correction method of ridge regression.
Fig. 2 is the series system schematic diagram of the inverting bias correction frame in the method for the present invention.
Fig. 3 is the parallel way schematic diagram of the inverting bias correction frame in the method for the present invention.
Fig. 4 is influence of the regularization parameter of ridge regression model in the method for the present invention to inverting performance.
Fig. 5 be the method for the present invention in by " dark target " algorithm, ridge regression algorithm, series system and parallel way these four not
The Pearson correlation coefficient and its standard deviation obtained on the training sample of different number is applied with method.
Fig. 6 be the method for the present invention in by " dark target " algorithm, ridge regression algorithm, series system and parallel way these four not
The root-mean-square error obtained on the training sample of different number and its standard deviation are applied with method.
Fig. 7 is that two methods of " dark target " algorithm and series system are applied the training sample 50% in the method for the present invention
On the obtained scatter plot of inversion result.
Fig. 8 is that two methods of " dark target " algorithm and parallel way are applied the training sample 50% in the method for the present invention
On the obtained scatter plot of inversion result.
Specific embodiment
A specific embodiment of the invention is further described in detail with reference to the accompanying drawing, the skill being not specified in embodiment
The conventional products that art or product are the prior art or can be obtained by purchase.
Embodiment 1: as shown in figures 1-8, it is based on the aerosol optical depth inverting bias correction method of ridge regression: first
First, using " dark target " algorithm inverting aerosol optical depth and as initialization result；Then, then building inverting is inclined
Difference corrects frame, is corrected by series connection and two ways in parallel to inversion result, and in series system, " dark target " is calculated
A validity feature of the aerosol optical depth that method obtains as ridge regression model, corrects inversion result；In parallel connection
In mode, with the residual error between the inversion result and true value of the direct learning physics model of ridge regression model, to inversion result into
Row is corrected.The following steps are included:
(1) " dark target " algorithm inverting aerosol optical depth is selected and as initialization result；It first collects big
The reflectivity data of pneumatic jack layer simultaneously screens it, removes by the unavailable pixel of the coverings such as cloud, desert, ice and snow, then
By " dark target " algorithm inverting aerosol optical depth (AOD), and as initialization result.
When it is implemented, the 6th generation " dark target " algorithm inverting aerosol optical depth of selection MODIS satellite.It is this
" dark target " algorithm cannot be used for bright earth surface area, such as cloud, desert, snowfield etc. because visible light wave range Reflectivity for Growing Season with
The relationship of 2.12 micron waveband atmosphere top layer reflectivity is invalid in these regions.In order to avoid these area influence, we
It needs the reflectivity of atmosphere top layer to be screened, removes by the unavailable pixel of the coverings such as cloud, desert, ice and snow, it is then anti-again
Aerosol optical depth is drilled, and as initialization result.
(2) " dark target " algorithm and ridge regression model are coupled to obtain inverting bias correction model, is then corrected again
The AOD value of inverting.
Aerosol optical depth inverting bias correction uses the inverting bias correction model in the method for the present invention.Such as figure
Shown in 1, inverting bias correction model is made of two sub- models: " dark target " algorithm and ridge regression model.Model coupling process
It can be indicated with following form:
In formula,Indicate the AOD value of estimation, x indicates mode input feature, ymodIndicate the knot of " dark target " algorithm inverting
Fruit, w are the parameter vector in ridge regression model, symbolIndicate coupling operation.Under normal conditions, not general coupling behaviour
Make, actual design depends on the existence form of domain knowledge.
For optics aerosol thickness inverting, domain knowledge is included in " dark target " algorithm.Therefore, side of the present invention
Method proposes a kind of aerosol optical depth inverting bias correction frame based on ridge regression, by physical model and machine learning model
Combine.Inverting bias correction model includes series connection and two ways in parallel.Fig. 2 is series system, and Fig. 3 is parallel way.
In series system, the result y of " dark target " algorithm invertingmodIt is input in ridge regression model as a kind of feature, to correct
The inversion result of " dark target " algorithm；In parallel way, ridge regression model is used to study " dark target " algorithm and true value
Between residual error, to correct the inversion result of " dark target " algorithm.
In series system, the AOD value of " dark target " algorithm inverting is input in ridge regression model as a kind of feature, this
When, inverting bias correction model above-mentioned can be written as follow form:
Wherein,In formula (2), the optimal value of parameter w is denoted as w*.It is sharp first in the model measurement stage
With one initial value of " dark target " method inverting, then by formulaDerive final AOD value.
In series system, ridge regression model plays the role of bias correction.More importantly this method is using all
Spectral value carries out automatic screening to feature as feature, and with ridge regression model, and model is enabled to obtain better inverting
Energy.
In parallel way, ridge regression model is used to residual error of the study " dark target " between algorithm and true value, at this point, preceding
The inverting bias correction model stated can be written as follow form:
Therefore, the label of ridge regression model becomes y-y from ymod, the optimal value of parameter w is denoted as w*。
In the model measurement stage, " dark target " residual error between algorithm and true value is calculated first with ridge regression model, so
After pass through formulaTo estimate final AOD value.
Inverting bias correction model in the method for the present invention includes series connection and two ways in parallel, has used ridge regression mould
Type corrects the deviation as caused by " dark target " algorithm.In series system, ridge regression model carries out automatic screening to feature, so that
Model can obtain better inverting performance；In parallel way, ridge regression model be used to directly to learn " dark target " algorithm with
Residual error between true value enables model to obtain better inverting performance.Ridge regression model is briefly described below.
Give l mark sampleSample is not marked with uWherein,D is feature sky
Between dimension.For ease of calculation, data can be usually expressed as to the form of matrix.Such as with X=[x1..., xl+u]TIt indicates
Entire data set, Xt=[x1..., xl]TIndicate training set, the transposition of subscript " T " representing matrix.
The core concept of linear regression is fitting function f (x；W)=xTW+b, so that predicted value and true value error is flat
Side and (loss function) are minimum, it may be assumed thatIn order to facilitate derivation, Ke Yi
Each variable xiOne element 1 of middle addition, so that parameter b is put into vector w.Assuming that Yt=[y1..., yl]T, then objective function
It can be write as the form of matrix:The solution of the function isHave however, in many practical applications, between sample different characteristic stronger correlation or
The dimension of feature is greater than the number of training sample, so that covariance matrixOften singular matrix.A kind of common solution
Scheme is that penalty term is added on w:
Wherein, α | | w | |2For lucky big vast promise husband's regular terms, statistically referred to as ridge regression.
For the solution of this formula, " I " indicates the unit matrix of d × d dimension.
The present invention is in Beijing observation station (39.98 ° of N, 116.38 ° of E), Xianghe observation station (39.75 ° of N, 116.96 ° of E), Hangzhoupro
State agricultural university observation station ZFU (30.26 ° of N, 119.73 ° of E), Hefei observation station (31.91 ° of N, 117.76 ° of E), Hong Kong science and engineering are big
Learn observation station (22.30 ° of N, 114.18 ° of E), Hong Kong observation station (22.48 ° of N, 114.12 ° of E), Lanzhou University's semiarid climate with
Environment survey station SACOL (35.95 ° of N, 104.14 ° of E), Taihu Lake observation station (31.42 ° of N, 120.22 ° of E), prosperous observation station
It is answered on 10 Chinese aerosol observation websites of (40.40 ° of N, 117.58 ° of E), Yulin observation station (38.28 ° of N, 109.72 ° of E)
With, using method proposed by the invention, inverting bias correction model is used to correct the deviation as caused by " dark target " algorithm,
To improve the inverting accuracy of model.True tag using the inverting value of ground based observa tion net (AERONET) as sample, i.e.,
Model output；Using the spectral value of intermediate resolution imaging spectrometer (MODIS) all wave bands as mode input.
AERONET is one established jointly by National Aeronautics and Space Administration NASA and Centre National de la Recherche Scientifique CNRS
Ground aerosol remote sensing network.Global about 250 ground-based devices are observed aerosol, they use CIMEL spectrum
Instrument measures the radiation of radiation and sky scattering from the sun, so that the AOD value of different spectral bands is calculated, these wave bands
Central wavelength includes 340 nanometers, 440 nanometers, 670 nanometers etc..For the ease of the inversion result between more different observation devices,
It is generally necessary to which the inversion result of AERONET is interpolated into the wave band that central wavelength is 550 nanometers.AERONET provides 3 kinds of differences
The product data of credit rating are respectively as follows: Level 1.0, the data for not filtering by cloud and finally verifying；Level 1.5,
The data for filtering by cloud but not verifying finally；And Level 2.0, it is filtered by cloud and finally verifies, has quality assurance
Data.We use 2.0 data of Level.
MODIS is a sensor being loaded on TERRA satellite, for collecting the information of aerosol and cloud.It sweeps
Retouching width is 2330 kms, can from the spectral reflectivity of 36 different spectrum channels observation atmosphere top layers, including visible light,
Near-infrared and infrared band.There are three different spatial resolutions, respectively 250 meters, 500 meters and 1000 meters for MODIS tool.We
Spatial resolution is had collected to be 1000 meters and include the Level-1B calibration radiation product of 10 AERONET station datas
MOD021KM。
Due to the difference of observation device, MODIS product needs to be matched on room and time with AERONET product,
Judge some to be observed the condition effectively observed that there are three aspects: first, centered on each observation website, periphery 30 × 30 is flat
At least one cloudless pixel in square kilometer range；Second, at least one MODIS aerosol retrieval products confidence level (QA) is super
Cross 1；Third, MODIS pass by least one available AERONET aerosol product in the half an hour of front and back.According to the above principle,
It is collected into 3093 matchings altogether from Chinese 10 websites using multisensor aerosol product sampling system AERONET and MODIS
Effective observation data.Take the flat of all data spectral bands within the scope of each effective observation 30 × 30 sq-km of data periphery
Input (i.e. input feature vector dimension be 36) of the mean value as model, the inverting average value of AERONET is as mould in the half an hour of front and back
The output of type.
It is specific as follows using process of the invention in the present embodiment:
1, " dark target " algorithm inverting aerosol optical depth is selected and as initialization result
The 6th generation " dark target " algorithm inverting aerosol optical depth of MODIS satellite is selected, and by this physical model
It is used to obtain initialization result as benchmark.Firstly, being sieved to the reflectivity of atmosphere top layer within the scope of 10 × 10 square kilometres
Choosing, removal utilize 2.12 micron spectral wave bands identification " dark target " by the unavailable pixel of the coverings such as cloud, desert, ice and snow
Pixel；Then, preceding 50% bright pixel and 20% dark pixel in 0.66 micron spectral wave band are removed, by the reflection of residual pixel
Rate is averaged；Secondly, deriving visible light wave range (0.47 micron and 0.66 according to the reflectivity of 2.12 micron waveband atmosphere top layers
Micron) Reflectivity for Growing Season；It is combined finally, the aerosol model of thin mode is weighted with the aerosol model of roughcast state,
In the hope of the value of AOD.
2, inverting bias correction frame is constructed
To correct aerosol optical depth inverting deviation, it is inclined to establish a kind of aerosol optical depth inverting based on ridge regression
Difference corrects model.Inverting bias correction model includes series connection and two ways in parallel.In series system, by " dark target " algorithm
Inversion result ymodIt is input in ridge regression model as a kind of feature, inversion result can be write asWherein,The optimal value w of regularization parameter w*It can be obtained with reference to the solution of ridge regression model:
Wherein, α | | w | |2For penalty term.For aerosol inverting, China only has several surface observation websites, and can
Obtained aerosol true value is limited.In addition, surface observation has to be matched on space-time with moonscope.In addition, respectively
It is strong relevant between a spectral reflectivity, therefore, so that covariance matrixOften singular matrix is needed to add and be punished
Penalize a α | | w | |2It could solve.
In the model measurement stage, first with one initial value of " dark target " algorithm inverting, then by formula
Derive final AOD value.In series system, due to containing the domain knowledge of aerosol inverting in " dark target " algorithm, because
And the inversion result of " dark target " algorithm is a kind of effective feature, is conducive to the inverting for improving aerosol optical depth.More
Importantly, series system uses the inversion result of all spectral value and " dark target " algorithm as feature, and use ridge regression
Model carries out automatic screening to feature, and model is enabled to obtain higher inverting performance.
In order to analyze influence of the regularization parameter to inverting performance in ridge regression model, by taking Beijing and Xianghe website as an example
It is tested, using 50% sample, remaining sample is as test set, from { 10 as training set-3, 10-2..., 102, 103}
Middle selection regularization parameter.As shown in figure 4, with the increase of regularization parameter then RMSE, which first reduces, to be increased.Therefore, most
Excellent regularization parameter is 10-1。
In parallel way, ridge regression model is used to residual error of the study " dark target " between algorithm and true value, at this point,
Inversion result can be write asTherefore, the label (model output) of ridge regression model becomes y- from y
ymod, i.e. f (x；W)=y-ymod.Wherein, the optimal value w of regularization parameter w*It can be according to formula?
It arrives.
In the model measurement stage, " dark target " residual error between algorithm and true value is calculated first with ridge regression model, so
After pass through formulaTo estimate final AOD value.
Parallel way directly learns the residual error of " dark target " between algorithm and true value using ridge regression model, to correct
The inversion result of " dark target " algorithm is equally beneficial for improving the inverting performance of aerosol optical depth.
In order to verify effectiveness of the invention, in the present embodiment to " dark target " algorithm, ridge regression algorithm, series system and
Four kinds of distinct methods of parallel way are compared, and " dark target " algorithm is obtained initialization result as benchmark, using being based on
The inverting bias correction model of ridge regression model corrects the deviation as caused by " dark target " algorithm, to improve the inverting essence of model
Exactness.
Experimental data used in the present embodiment is divided into training set and test set two parts by random.Training set is for training
Different machine learning models, test set are then used to assess the performance of each model.For the influence for reducing stochastical sampling, all calculations
Method is repeatedly executed 10 times, and last result is the average value of 10 experiments.Without loss of generality, to use Pearson correlation coefficient
(r) and root-mean-square error (RMSE) two kinds of evaluation indexes.For convenience of description, " dark target " inversion algorithm, ridge regression algorithm, series connection
Mode and parallel way are briefly referred to as DT, RR, Serial and Parallel.
Fig. 5 compares four kinds of distinct methods and applies the average r value for carrying out 10 experiments on the training sample of different number
And its standard deviation.We can observe that the following from figure: first, with the increase of training samples, in addition to DT algorithm
Outside r value between acquisition inversion result and true value keeps relative stability, other three kinds of methods r values obtained are all increasing,
Illustrate that the inverting performance of physical model does not depend on training data.Second, when the ratio of training sample is 10%, the inverting of RR
DT can be lower than, because a small amount of sample is difficult one ideal machine learning model of training.Similarly, RR is difficult accurately to learn DT anti-
The residual error between AOD value and true AOD value drilled.Different from RR and Parallel, Serial is using the AOD value of DT inverting as one
A feature, this feature will be endowed biggish weight in the training process, and other feature weight is smaller, to solve indirectly small
The problem of sample training, obtains precision more higher than DT.Third, when the ratio of training sample is more than 10%, DT is most of
In the case of can obtain precision more higher than RR because linear model RR be unable to accurate simulation go out spectral reflectance value and AOD value it
Between non-linear relation.On the contrary, Serial and Parallel can accurately correct it is inclined between true value and DT inversion result
Difference, and allow them to obtain inversion result more better than DT.These sufficiently demonstrate having for inverting bias correction model
Effect property.4th, compared with parallel way, series system perhaps more has prospect, because it can obtain preferably inverting performance.With
Upper conclusion can be obtained from another Measure Indexes RMSE, as shown in fig. 6, RMSE value is smaller, inversion accuracy is higher.
In addition, Fig. 7 illustrates the AOD value of Serial and DT inverting and the distribution situation of true AOD value, abscissa indicates true
Real AOD value y, ordinate indicate the AOD value that inverting obtains"+" is labeled as the distribution of results of DT inverting, and " * " is labeled as
The distribution of results of Serial inverting, solid black lines are optimal Data Position, and two dotted lines indicate acceptable error range.
It can be seen from the figure that the value of the AOD value ratio DT inverting of Serial inverting is closer to ideal line, especially when true AOD value
When smaller.In addition, thering are more data points to fall within the scope of two dotted lines in the result of Serial inverting.Fig. 8 is illustrated
The distribution situation of the AOD value of Parallel and DT inverting and true AOD value, abscissa indicate true AOD value y, ordinate table
Show the AOD value that inverting obtains"+" is labeled as the distribution of results of DT inverting, the result point of " * " labeled as Parallel inverting
Cloth, solid black lines are optimal Data Position, and two dotted lines indicate acceptable error range.It can be seen from the figure that
The value of the AOD value ratio DT inverting of Parallel inverting is closer to ideal line, especially when true AOD value is smaller.In addition,
In the result of Parallel inverting, there are more data points to fall within the scope of two dotted lines.
Model is also demonstrated in the present embodiment in ground surface type and observes seasonal robustness.Still with Beijing and Xianghe
It is tested for website.Table 1 lists the distribution situation of Various Seasonal sample, is surveyed using the data in a season as test set
The data of the inversion accuracy of die trial type, remaining three seasons are trained as training the set pair analysis model.Table 2 and table 3 are shown respectively
Different models RMSE value obtained and r value, best result under font representation the same terms of overstriking.It can from these tables
To find out, Serial is higher than other three models in three spring, summer, autumn seasonal inversion accuracies, and Parallel is at four
Slightly it is better than Serial in the average behavior in season, because it there can be better prediction result in winter.In addition, table 4 is shown
Different models fall in Various Seasonal the percentage (PAR) of acceptance region, and by observation, discovery Parallel and Serial can be with
Obtain more acceptable inversion results.Similar, respectively from RMSE value, three aspects of r value and PAR value compare table 5,6,7
Different models different websites inversion result, with the data training pattern of a website, the data test mould of another website
Type, Parallel obtain best inversion result again in the average behavior in four seasons.
Distribution of the table 1 in Beijing and Xianghe website Various Seasonal sample number
| Season | Month | Sample number |
| Spring | The 3-5 month | 443 |
| Summer | The 6-8 month | 441 |
| Autumn | The 9-11 month | 578 |
| Winter | The 12-2 month | 55 |
| Summation | - | 1517 |
Table 2 Beijing and Xianghe website difference model Various Seasonal inverting performance comparison (RMSE)
Table 3 Beijing and Xianghe website difference model Various Seasonal inverting performance comparison (r)
| Model | Spring | Summer | Autumn | Winter | It is average |
| DT | 0.9285 | 0.9230 | 0.9478 | 0.9298 | 0.9323 |
| RR | 0.8747 | 0.8998 | 0.9164 | 0.5609 | 0.8130 |
| Parallel | 0.9337 | 0.9322 | 0.9612 | 0.9461 | 0.9433 |
| Serial | 0.9381 | 0.9366 | 0.9621 | 0.8459 | 0.9207 |
Table 4 Beijing and Xianghe website difference model Various Seasonal inverting performance comparison (PAR)
| Model | Spring | Summer | Autumn | Winter | It is average |
| DT | 0.4944 | 0.4989 | 0.6765 | 0.8182 | 0.6220 |
| RR | 0.4673 | 0.4739 | 0.4983 | 0.3636 | 0.4508 |
| Parallel | 0.6682 | 0.5964 | 0.7751 | 0.8727 | 0.7281 |
| Serial | 0.6275 | 0.6009 | 0.6972 | 0.6364 | 0.6405 |
Comparison (RMSE) of the table 5 in Beijing and Xianghe website difference model inversion performance
| Website | DT | RR | Parallel | Serial |
| Beijing | 0.2406 | 0.2861 | 0.2199 | 0.2160 |
| Xianghe | 0.1726 | 0.2764 | 0.1601 | 0.2086 |
| It is average | 0.2066 | 0.2812 | 0.1900 | 0.2123 |
Comparison (r) of the table 6 in Beijing and Xianghe website difference model inversion performance
| Website | DT | RR | Parallel | Serial |
| Beijing | 0.9103 | 0.8593 | 0.9147 | 0.9217 |
| Xianghe | 0.9592 | 0.8701 | 0.9603 | 0.9411 |
| It is average | 0.9348 | 0.8647 | 0.9375 | 0.9314 |
Comparison (PAR) of the table 7 in Beijing and Xianghe website difference model inversion performance
| Website | DT | RR | Parallel | Serial |
| Beijing | 0.4958 | 0.4069 | 0.5836 | 0.5065 |
| Xianghe | 0.6780 | 0.3724 | 0.7404 | 0.5718 |
| It is average | 0.5869 | 0.3897 | 0.6620 | 0.5392 |
In summary, it is compared with " dark target " algorithm, Ridge Regression Modeling Method, the inverting bias correction frame that the method for the present invention proposes
Frame has apparent advantage, can effectively correct the deviation as caused by " dark target " algorithm from inverting performance.This
Outside, the inverting performance of series system is in most cases better than parallel mode, but parallel mode is in ground surface type and observing season
Robustness on section is stronger.
Technology contents of the invention are described above in conjunction with attached drawing, but protection scope of the present invention be not limited to it is described
Content within the knowledge of one of ordinary skill in the art can also be in the premise for not departing from present inventive concept
Under technology contents of the invention are made a variety of changes, all within the spirits and principles of the present invention, any modification for being made, etc.
With replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (5)
1. a kind of aerosol optical depth inverting bias correction method based on ridge regression, it is characterised in that: use " dark mesh first
Mark " algorithm inverting aerosol optical depth and as initialization result；Then inverting bias correction frame is constructed, string is passed through
Connection and two ways in parallel correct inversion result, in series system, aerosol light that " dark target " algorithm is obtained
A validity feature of the thickness as ridge regression model is learned, inversion result is corrected；In parallel way, with ridge regression mould
Residual error between the inversion result and true value of the direct learning physics model of type, corrects inversion result.
2. the aerosol optical depth inverting bias correction method according to claim 1 based on ridge regression, feature exist
In: the specific steps of the method are as follows:
(1) it first collects the reflectivity data of atmosphere top layer and it is screened, remove by coverings such as cloud, desert, ice and snow not
Available pixel point, then by " dark target " algorithm inverting aerosol optical depth (AOD), and as initialization result；
(2) by " dark target " algorithm and ridge regression model f (x；W) it is coupled, forms following inverting bias correction model:
In formula,Indicate the AOD value of estimation, ymodIndicate " dark target " algorithm inverting as a result, x indicates mode input feature, w is
Parameter vector in ridge regression model, symbolIndicate coupling operation；
(3) by the inverting bias correction model of series connection and two ways in parallel, inverting bias correction is carried out；In series system
In, the model of step (2) is rewritten as following form:
Wherein,The transposition of subscript T representing matrix, the optimal value of parameter w are denoted as w*；
Series system is by the result y of " dark target " algorithm invertingmodRidge regression model f (x is input to as a kind of feature；W) in,
Will all AOD values of " dark target " algorithm inverting be input to ridge regression model f (x as a kind of feature；W) in, by ridge regression
Model carries out automatic screening to feature, corrects to the inversion result of " dark target " algorithm；
In parallel way, the model of step (2) is rewritten as following form；
At this point, ridge regression model f (x；W) label becomes y-y from ymod；
Parallel way utilizes ridge regression model f (x；W) learn the residual error between the inversion result and true value of " dark target " algorithm,
The inversion result of " dark target " algorithm is corrected.
3. the aerosol optical depth inverting bias correction method according to claim 2 based on ridge regression, feature exist
In: in the series system of the step (3), in the model measurement stage, first with one initial value of " dark target " algorithm inverting, then
By formulaDerive the AOD value corrected, w*For the optimal value of parameter w.
4. the aerosol optical depth inverting bias correction method according to claim 2 based on ridge regression, feature exist
In: in the parallel way of the step (3), in the model measurement stage, first with ridge regression model calculate " dark target " algorithm with
Residual error between true value, then pass through formulaTo estimate the AOD value of correction, w*For the optimal value of parameter w.
5. the aerosol optical depth inverting bias correction method according to claim 1-4 based on ridge regression,
It is characterized by: " dark target " algorithm for inverting aerosol optical depth uses the 6th generation " dark mesh of MODIS satellite
Mark " algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811184427.6A CN109186474A (en) | 2018-10-11 | 2018-10-11 | A kind of aerosol optical depth inverting bias correction method based on ridge regression |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811184427.6A CN109186474A (en) | 2018-10-11 | 2018-10-11 | A kind of aerosol optical depth inverting bias correction method based on ridge regression |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109186474A true CN109186474A (en) | 2019-01-11 |
Family
ID=64947800
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811184427.6A Withdrawn CN109186474A (en) | 2018-10-11 | 2018-10-11 | A kind of aerosol optical depth inverting bias correction method based on ridge regression |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109186474A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110567385A (en) * | 2019-09-19 | 2019-12-13 | 廊坊师范学院 | Hyperspectral technology-based construction thickness detection method for building reflective insulation coating |
| CN111191380A (en) * | 2020-01-08 | 2020-05-22 | 北京大学 | Atmospheric aerosol optical thickness estimation method and device based on measurement data of foundation spectrometer |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1727844A (en) * | 2005-07-05 | 2006-02-01 | 华东师范大学 | Surficial contrast method for inverting optical thickness of aerosol at boundary layer from aeronautic high spectrum remote sensing |
| CN101598543A (en) * | 2009-07-29 | 2009-12-09 | 中国科学院对地观测与数字地球科学中心 | A kind of atmospheric correction method for remote sensing images of practicality |
| CN104279967A (en) * | 2014-05-23 | 2015-01-14 | 中国科学院光电研究院 | Aerosol optical depth inversion method based on hyperspectral image |
| CN104751003A (en) * | 2015-04-15 | 2015-07-01 | 华北电力大学 | Dip angle predicting and calculating method of ice-coated power transmission lead |
-
2018
- 2018-10-11 CN CN201811184427.6A patent/CN109186474A/en not_active Withdrawn
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1727844A (en) * | 2005-07-05 | 2006-02-01 | 华东师范大学 | Surficial contrast method for inverting optical thickness of aerosol at boundary layer from aeronautic high spectrum remote sensing |
| CN101598543A (en) * | 2009-07-29 | 2009-12-09 | 中国科学院对地观测与数字地球科学中心 | A kind of atmospheric correction method for remote sensing images of practicality |
| CN104279967A (en) * | 2014-05-23 | 2015-01-14 | 中国科学院光电研究院 | Aerosol optical depth inversion method based on hyperspectral image |
| CN104751003A (en) * | 2015-04-15 | 2015-07-01 | 华北电力大学 | Dip angle predicting and calculating method of ice-coated power transmission lead |
Non-Patent Citations (1)
| Title |
|---|
| RENLONG HANG等: "Correcting MODIS aerosol optical depth products using a ridge regression model", 《INTERNATIONAL JOURNAL OF REMOTE SENSING》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110567385A (en) * | 2019-09-19 | 2019-12-13 | 廊坊师范学院 | Hyperspectral technology-based construction thickness detection method for building reflective insulation coating |
| CN111191380A (en) * | 2020-01-08 | 2020-05-22 | 北京大学 | Atmospheric aerosol optical thickness estimation method and device based on measurement data of foundation spectrometer |
| CN111191380B (en) * | 2020-01-08 | 2020-11-20 | 北京大学 | Atmospheric aerosol optical thickness estimation method and device based on measurement data of foundation spectrometer |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103994976B (en) | Agriculture Drought remote-sensing monitoring method based on MODIS data | |
| Li et al. | Road network extraction via deep learning and line integral convolution | |
| CN110186820A (en) | Multisource data fusion and environomental pollution source and pollutant distribution analysis method | |
| Castillo et al. | LIDAR remote sensing for secondary Tropical Dry Forest identification | |
| Du et al. | Impact of urban expansion on land surface temperature in Fuzhou, China using Landsat imagery | |
| CN102175209B (en) | Effective sampling method for crop cultivated area measurement under support of historical remote sensing product data | |
| CN115356249B (en) | Satellite polarization PM2.5 estimation method and system based on machine learning fusion model | |
| CN105608293A (en) | Forest aboveground biomass inversion method and system fused with spectrum and texture features | |
| CN105224960A (en) | Based on the corn seed classification hyperspectral imagery model of cognition update method of clustering algorithm | |
| Yan et al. | Assimilating multi-source remotely sensed data into a light use efficiency model for net primary productivity estimation | |
| CN112270291B (en) | Automatic detection method for illegal construction land development based on multi-source optical remote sensing image | |
| CN103400364A (en) | Monitoring method for forest resource change | |
| CN113252583B (en) | Method for calculating alpine hay coverage based on hay vegetation index | |
| CN114819737B (en) | Method, system and storage medium for estimating carbon reserves of highway road vegetation | |
| CN104680151A (en) | High-resolution panchromatic remote-sensing image change detection method considering snow covering effect | |
| CN109186474A (en) | A kind of aerosol optical depth inverting bias correction method based on ridge regression | |
| An et al. | Monitoring grassland degradation and restoration using a novel climate use efficiency (NCUE) index in the Tibetan Plateau, China | |
| Lou et al. | An effective method for canopy chlorophyll content estimation of marsh vegetation based on multiscale remote sensing data | |
| Yu et al. | Spatio-temporal monitoring of urban street-side vegetation greenery using Baidu Street View images | |
| Liu et al. | An algorithm for early rice area mapping from satellite remote sensing data in southwestern Guangdong in China based on feature optimization and random Forest | |
| Zhang et al. | Spatial and temporal variation of ecological quality in northeastern China and analysis of influencing factors | |
| Shi et al. | An improved framework for assessing the impact of different urban development strategies on land cover and ecological quality changes-A case study from Nanjing Jiangbei New Area, China | |
| Cao et al. | Investigating mangrove canopy phenology in coastal areas of China using time series Sentinel-1/2 images | |
| Zhang et al. | Enhancing model performance in detecting lodging areas in wheat fields using UAV RGB Imagery: Considering spatial and temporal variations | |
| Lou et al. | Recognition of thaw slumps based on machine learning and UAVs: A case study in the Qilian Mountains, northeastern Qinghai-Tibet Plateau |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 210044 No. 219 Ningliu Road, Jiangbei New District, Nanjing City, Jiangsu Province Applicant after: Nanjing University of Information Science and Technology Address before: 211500 Yuting Square, 59 Wangqiao Road, Liuhe District, Nanjing City, Jiangsu Province Applicant before: Nanjing University of Information Science and Technology |
|
| CB02 | Change of applicant information | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190111 |
|
| WW01 | Invention patent application withdrawn after publication |