CN115166871A - Microwave imager rainfall inversion method based on hybrid neural network - Google Patents
Microwave imager rainfall inversion method based on hybrid neural network Download PDFInfo
- Publication number
- CN115166871A CN115166871A CN202210497652.5A CN202210497652A CN115166871A CN 115166871 A CN115166871 A CN 115166871A CN 202210497652 A CN202210497652 A CN 202210497652A CN 115166871 A CN115166871 A CN 115166871A
- Authority
- CN
- China
- Prior art keywords
- rainfall
- precipitation
- layer
- data set
- inversion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 36
- 238000001556 precipitation Methods 0.000 claims abstract description 95
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 23
- 230000003595 spectral effect Effects 0.000 claims abstract description 8
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 239000011159 matrix material Substances 0.000 claims description 42
- 238000010606 normalization Methods 0.000 claims description 34
- 230000004913 activation Effects 0.000 claims description 30
- 210000002569 neuron Anatomy 0.000 claims description 26
- 238000004364 calculation method Methods 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 13
- 238000012549 training Methods 0.000 claims description 13
- 230000000694 effects Effects 0.000 claims description 10
- 238000012952 Resampling Methods 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 239000004744 fabric Substances 0.000 claims 1
- 238000005457 optimization Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 58
- 230000009286 beneficial effect Effects 0.000 description 10
- 238000009826 distribution Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000010287 polarization Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000005855 radiation Effects 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 230000033228 biological regulation Effects 0.000 description 1
- 210000004027 cell Anatomy 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 238000002759 z-score normalization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01W—METEOROLOGY
- G01W1/00—Meteorology
- G01W1/14—Rainfall or precipitation gauges
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Environmental & Geological Engineering (AREA)
- Hydrology & Water Resources (AREA)
- Engineering & Computer Science (AREA)
- Atmospheric Sciences (AREA)
- Biodiversity & Conservation Biology (AREA)
- Ecology (AREA)
- Environmental Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a microwave imager precipitation inversion method based on a hybrid neural network, which comprises the following steps: establishing a data set, wherein the data set comprises observation data of a Fengyun No. three B star microwave imager and passive microwave precipitation products of an American ice and snow data center; preprocessing a data set, and matching the observation data with the passive microwave precipitation product according to a matching principle to obtain a matching data set meeting the matching principle; performing rainfall classification operation, clustering the matched data set by adopting an approximate spectral clustering algorithm with dense cores and density peak values, determining the dense cores, obtaining pixel-based rainfall probability according to the dense cores, and classifying rainfall and rainfall absence to obtain a rainfall sample data set; and executing a precipitation estimation operation, constructing a precipitation inversion model based on the depth neural network, and determining the rainfall rate by using the rainfall sample data set according to the precipitation inversion model. The method can realize global precipitation inversion optimization and improve precipitation inversion precision.
Description
Technical Field
The invention relates to the technical field of precipitation inversion, in particular to a precipitation inversion method of a microwave imager based on a hybrid neural network.
Background
Precipitation is one of important parameters of climate characteristics, and plays an important role in research in the fields of weather analysis, climate change, numerical weather forecast and the like. One of the most difficult reasons for monitoring precipitation is time and space change, and satellite remote sensing has the advantage of high time and space resolution, is not limited by areas and earth surface environments, can be used for detecting data of global precipitation climate and performing precipitation inversion by observing the data.
China's meteorological satellite- ' Feng Yun III ' satellite (Feng Yun-3, FY-3) is used for carrying on the climate observation, the Microwave Imager (Microwave Radiation Imager, MWRI) is one of the important loads on FY-3, one totally has 5 frequencies: 10.65GHz, 18.7GHz, 23.8GHz, 36.5GHz and 89.0GHz, with two polarization modes for each frequency: vertical and horizontal polarization (v.h). The MWRI is carried on the in-orbit operation satellites of the FY-3 series, the MWRI of the FY-3D satellite continues the design of the FY-3A, the FY-3B and the FY-3C satellite, and observation data such as ocean, land and atmospheric parameters and the like all day long and all weather can be provided through different channels or channel combinations.
The rainfall inversion is a process of estimating the rainfall value through the MWRI observation data, and the rainfall inversion precision is improved. Therefore, the MWRI design inversion method has important significance for processing the observation data of the same series or other series of in-orbit running satellites.
The current precipitation inversion method mainly comprises an empirical algorithm and a physical algorithm, and the inversion accuracy of the empirical algorithm has limitations and is related to sample data, such as whether the data is representative or not and whether the data is enough. The physical algorithm has the defects of large calculation amount and long calculation time in the processes of selecting the initial parameters in the previous stage and solving the radiation equation, and the forward model is complex to construct.
Therefore, the existing inversion algorithm has the problem of slow calculation speed, and an optimal nonlinear regression equation needs to be found through repeated iteration; although global optimization can be achieved, it is also prone to getting into the dilemma of local optimization.
Disclosure of Invention
The invention provides a microwave imager precipitation inversion algorithm based on a hybrid neural network, which can realize global precipitation inversion optimization, simplify input parameters and processing performance, reduce precipitation inversion calculation time and improve precipitation inversion precision.
A microwave imager precipitation inversion method based on a hybrid neural network comprises the following steps:
s1: establishing a data set, wherein the data set comprises observation data of a Fengyun No. three B star microwave imager and passive microwave precipitation products of an American ice and snow data center;
s2: preprocessing a data set, and matching the observation data with the passive microwave precipitation product according to a matching principle to obtain a matching data set meeting the matching principle;
s3: performing rainfall classification operation, clustering the matched data set by adopting an approximate spectral clustering algorithm with dense cores and density peak values, determining the dense cores, obtaining pixel-based rainfall probability according to the dense cores, and classifying rainfall and rainfall absence to obtain a rainfall sample data set;
s4: and executing a precipitation estimation operation, constructing a precipitation inversion model based on the depth neural network, and determining the rainfall rate by using the rainfall sample data set according to the precipitation inversion model.
Further, in S2, the method includes the following steps:
s201: extracting brightness temperature, geographical position and time information of 5 channels according to the observation data; the observation data comprises dual-polarization observation brightness temperature data, an instrument height angle and an instrument incidence angle;
s202: extracting rainfall rate, geographical position and time information of the passive microwave rainfall product;
s203: the matching principle comprises the following steps: and selecting data with the matching data time difference not exceeding 30 minutes and the longitude and latitude difference not exceeding 0.05 degrees, resampling the passive microwave precipitation product, extracting data meeting the matching principle, and forming a matching data set by the extracted data.
Further, in S3, the method includes the following steps:
s301: constructing a graph structure according to the matching data set, wherein graph structure nodes correspond to matching data set data, and setting a graph structure node dense core according to the graph structure node dense degree; the dense core is a point with a local maximum density;
s302: defining new distances according to the public neighborhoods of the dense cores of the nodes of the graph structure, and calculating the distances among the dense cores according to the newly defined distances;
s303: constructing a non-directional weight graph and selecting an initial center;
s304: constructing a similarity matrix according to the number of the dense cores, and calculating the similarity between the initial center and the dense cores;
s305: constructing a Laplace matrix according to the similarity matrix, performing characteristic decomposition on the Laplace matrix, and selecting a characteristic vector to form a characteristic space;
s306: and outputting a clustering result by utilizing a normalized clustering algorithm in the feature space to obtain the rainfall probability based on the pixels, and classifying rainfall and rainfall-free to obtain a sample data set with rainfall.
Further, in S306, the output clustering result is a continuous function with a value in an interval range of 0 to 1, and the continuous function value is set as a rainfall probability; the continuous function value 0.5 is set as a threshold value, and if the continuous function value is greater than 0.5, it indicates rainfall, and if the continuous function value is less than or equal to 0.5, it indicates no rainfall.
Further, in S4, constructing a precipitation inversion model includes the following steps:
s401: constructing four-layer network structures which are connected in sequence, wherein the four-layer network structures comprise: an input layer, two hidden layers and an output layer;
s402: the input layer is a sample data set with rainfall obtained in the step S3 and is processed;
s403: the hidden layer processes the data input by the input layer to determine the rainfall rate;
s404: and the output layer outputs the rainfall rate.
Further, the S402 includes:
s4021: the input layer is a sample data set with rainfall obtained in the S3;
s4022: adding a normalization layer behind an input layer, wherein the normalization layer is used for performing normalization processing on each input data item by adopting a Z-score method to normalize the data items to a standard normal distribution with the mean value of 0 and the variance of 1;
further, the S403 includes:
s4031: adding a full connection layer after the normalization layer, wherein the parameter of the full connection layer is the number of neurons;
s4032: adding a nonlinear activation unit layer behind the full connection layer, wherein the parameter of the nonlinear activation unit layer is an activation function ReLU;
s4033: adding a normalization layer after the nonlinear activation unit layer, wherein the normalization layer is used for carrying out normalization processing on each input data item by adopting a Z-score method;
s4034: and adding a dropout layer after the normalization layer, wherein the dropout layer is used for randomly discarding some neurons in each batch training process, preventing overfitting and determining the rainfall rate.
Further, the S404 includes:
s4041: adding a full connection layer behind the dropout layer, wherein the parameter of the full connection layer is the number of neurons;
s4042: adding an activation function layer after the full connection layer, wherein the parameter of the activation function layer is an activation function Sigmoid;
s4043: and outputting the rainfall rate.
Further, after the step S4, the method further includes:
s5: evaluating the effect of the precipitation inversion model;
the method for evaluating the effect of the precipitation inversion model comprises the following steps:
s501: calculating the difference between the result of a forward calculation formula of each iteration of the precipitation inversion model and an actual value by adopting an average absolute error loss function, guiding the precipitation inversion model to train, and then calculating the error of a predicted value and the actual value by adopting a root mean square error loss function:
s502: initializing parameters of a forward calculation formula by using random values;
s503: substituting the samples into a forward calculation formula, and calculating to obtain a predicted value;
s504: calculating the error between the predicted value and the actual value by using an average absolute error loss function;
s505: according to the derivative of the average absolute error loss function, returning the error along the minimum direction of the gradient, and correcting each weight value in the forward calculation formula;
s506: performing the actions of S502 to S505, and stopping the iteration until the average absolute error loss function value reaches an optimal value;
s507: and calculating errors of the predicted value and the actual value by adopting a root-mean-square error loss function, and evaluating the effect of the precipitation inversion model based on the errors of the predicted value and the actual value.
Further, after the step S4, the method further includes step S6: verifying and analyzing the precipitation inversion result and the passive microwave precipitation product;
the method for performing verification analysis comprises the following steps:
s601: comparing the inversion rainfall rate output by the rainfall inversion result with the predicted rainfall rate in the passive microwave rainfall product; when the deviation between the inversion rainfall rate and the predicted rainfall rate is smaller than a preset deviation threshold value, the inversion rainfall rate is reserved; when the deviation between the inversion rainfall rate and the predicted rainfall rate is larger than a preset deviation threshold value, abandoning the inversion rainfall rate;
s602: estimating rainfall and rainfall absence according to preset conditions by using the reserved inversion rainfall rate; the preset conditions include a certain future time and a certain region.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic diagram of a microwave imager precipitation inversion algorithm method based on a hybrid neural network according to the present invention;
FIG. 2 is a flow chart of steps of a method of performing precipitation classification operations in accordance with the present invention;
fig. 3 is a flow chart of the steps of the precipitation estimation operation method of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
The embodiment of the invention provides a microwave imager precipitation inversion method based on a hybrid neural network, and as shown in fig. 1, the method comprises the following steps:
s1: establishing a data set, wherein the data set comprises observation data of a Fengyun No. three B-star microwave imager and passive microwave precipitation products of an American ice and snow data center;
s2: preprocessing a data set, and matching the observation data with the passive microwave precipitation product according to a matching principle to obtain a matching data set meeting the matching principle;
s3: performing rainfall classification operation, clustering the matched data set by adopting an approximate spectral clustering algorithm with dense cores and density peak values, determining the dense cores, obtaining pixel-based rainfall probability according to the dense cores, and classifying rainfall and rainfall absence to obtain a rainfall sample data set;
s4: and executing precipitation estimation operation, constructing a precipitation inversion model based on the depth neural network, and determining the rainfall rate by using the sample data set with rainfall according to the precipitation inversion model.
The working principle of the technical scheme is as follows: the microwave imager rainfall inversion method based on the hybrid neural network is characterized in that firstly, data matching is carried out on observation Data of a wind cloud three-number B-star microwave imager and a passive microwave rainfall product of a National Snow and Ice Data Center (NSIDC), and the NSIDC passive microwave rainfall product needs to be resampled; then, inversion operation, namely precipitation classification operation and precipitation estimation operation, is carried out through two different modules based on a neural network, the precipitation classification operation clusters the matched data sets through an approximate spectrum clustering algorithm with dense cores and density peak values, a similarity matrix and a Laplace matrix are calculated according to data of the matched data sets, and finally clustering results are output to obtain pixel-based rainfall probability, and rainfall-free classification is carried out to obtain a sample data set with rainfall; and (4) rainfall estimation operation, namely constructing a machine learning model with multiple hidden layers and a large amount of training data through a deep neural network, and determining the rainfall rate by using a sample data set with rainfall.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the global precipitation inversion optimization can be realized, the input parameters and the processing performance are simplified, the precipitation inversion calculation time is shortened, and the precipitation inversion accuracy is improved. In addition, according to the scheme provided by the embodiment, an optimal nonlinear regression equation does not need to be found through a repeated iteration scheme, calculation steps and calculation time are saved, the rainfall rate capable of estimating the rainfall is determined through rainfall classification operation and rainfall estimation operation combined with a deep neural network, and then the rainfall is estimated.
In one embodiment, when a data set is established, two types of data materials are required to be used, wherein the two types of data materials comprise observation data provided by a microwave imager on a wind cloud three-star B and NSIDC passive microwave precipitation products as 'input variables' of precipitation classification operation, so that the data set is required to be preprocessed, and the preprocessing comprises three steps, namely, in the first step, brightness temperature, geographic position and time information of 5 channels of the data set are extracted according to the observation data; the observation data comprises dual-polarization observation brightness temperature data, an instrument height angle and an instrument incidence angle; step two, extracting rainfall rate, geographical position and time information of NSIDC passive microwave precipitation products; and thirdly, according to preset regulation that the time difference of the matched data does not exceed 30 minutes and the difference of longitude and latitude does not exceed 0.05 degrees, resampling the NSIDC passive microwave precipitation product, extracting data meeting requirements, and forming a matched data set by the extracted data.
The working principle of the technical scheme is as follows: according to the requirement of precipitation inversion operation, extracting corresponding types of data information from observation data and NSIDC passive microwave precipitation products, matching the data information according to fixed time difference, longitude difference and latitude difference, resampling the NSIDC passive microwave precipitation products, and classifying the matched data to form a matched data set.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the observation data and the NSIDC passive microwave precipitation product can be further matched, so that the data set meets the data input requirement of the hybrid neural network.
In one embodiment, when the rainfall classification operation is performed, as shown in fig. 2, the matching data set is clustered by using an approximate spectral clustering algorithm with dense cores and density peaks to determine the dense cores, and rainfall probability based on pixels is obtained according to the dense cores, so that rainfall and rainfall absence are classified to obtain a sample data set with rainfall. The method comprises the following specific steps:
s301, constructing a graph structure according to the matching data sets, wherein graph structure nodes correspond to the matching data sets, and setting graph structure node dense cores according to the graph structure node dense degree; the dense core is a point with a local maximum density;
s302, defining new distances according to the public neighborhoods of the dense cores of the nodes of the graph structure, and calculating the distances among the dense cores according to the newly defined distances;
s303, constructing a undirected weight graph and selecting an initial center;
s304, constructing a similarity matrix according to the quantity of the dense cores, and calculating the similarity between the initial center and the dense cores;
s305, constructing a Laplace matrix according to the similarity matrix, performing characteristic decomposition on the Laplace matrix, and selecting a characteristic vector to form a characteristic space;
and S306, outputting a clustering result by utilizing a normalized clustering algorithm in the feature space to obtain the rainfall probability based on the pixels, and classifying rainfall and rainfall-free to obtain a sample data set with rainfall.
The working principle of the technical scheme is as follows: constructing a graph structure according to the matching data set, wherein graph structure nodes correspond to matching data set data, and setting a graph structure node dense core according to the graph structure node dense degree; defining new distances according to the public neighborhoods of the dense cores of the nodes of the graph structure, calculating the distances among the dense cores according to the newly defined distances, constructing a non-directional weight graph G, and selecting an initial center; constructing a Laplace matrix according to the similarity matrix, performing characteristic decomposition on the Laplace matrix, and selecting a characteristic vector to form a characteristic space; outputting a clustering result by utilizing a normalized clustering algorithm in a feature space to obtain pixel-based rainfall probability, and classifying rainfall and rainfall-free to obtain a sample data set with rainfall; the basic idea of the spectral clustering algorithm is that a similarity matrix is calculated according to sample points, then a degree matrix and a Laplace matrix are calculated, then eigenvectors corresponding to k eigenvalues in front of the Laplace matrix are calculated, finally, each row of a matrix M consisting of the eigenvectors corresponding to the k eigenvalues becomes a newly generated sample point, the newly generated sample points are clustered by adopting a k-means clustering algorithm to form k classes, and a clustering result is output, wherein the algorithm comprises the following specific processes:
step 1: defining input n sample points X = { X = ×) 1 ，x 2 ，…,x n And the number of dense cores k; output as dense core number A 1 ，A 2 ，…,A k . A typical data set consists of samples (rows) and their features (columns), but spectral clustering algorithms can only be applied to a graph of node connections. Therefore, data must be converted in order to convert from rows and columns to graphics.
Step 2: calculating a similarity matrix S of n x n; where n is the number of samples, each point of the matrix is the euclidean distance between each pair of points, then an adjacency matrix W is created by the similarity matrix S, the magnitude relationship of the similarity matrix S and the threshold is compared by setting a threshold, and is set to 0 if the distance is greater than the threshold, and is otherwise 1. The adjacency matrix W can then be used to construct a graph, and if there is a 1 in the cell of the adjacency matrix W, we draw an edge between the nodes of the columns and rows.
And 3, step 3: calculating a degree matrix D; when the adjacency matrix W is constructed, the degree matrix D is constructed. For each row of the degree matrix D, the diagonal of the degree matrix D is represented by summing all elements of the corresponding row in the adjacency matrix W, and then the laplacian matrix is calculated by subtracting the adjacency matrix W from the degree matrix D.
And 4, step 4: calculating a Laplace matrix, wherein the calculation formula is as follows:
L rw ＝D -1 (D-W)
in the above formula, D is a degree matrix, W is an adjacent matrix, and L is rw Sorting the eigenvalues from small to large for the eigenvalues, taking the top k eigenvalues, and calculating eigenvectors m of the top k eigenvalues 1 ，m 2 ，…,m n ；
The upper k columns vector composition matrix M = { M 1 ,m 2 ,…,m n }，M∈R n*k ；
Let y i ∈R k Is a vector of the ith row of M, where i =1, 2.., n;
new sample points Y = { Y using k-means clustering algorithm 1 ,y 2 ,…,y n Cluster-forming C 1 ,C 2 ,…,C k ；
Output dense core number A 1 ，A 2 ，…,A k Wherein A is i ＝{j|y j ∈C i }
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, a density core concept is introduced, each density core is a predicted cluster, each dense core is a point with local maximum density, only a local core with high density is reserved and is not limited to local optimum, the dense cores of the data set are searched, the dense cores are divided by adopting a normalized spectral clustering method, clustering is realized in the whole data set, the rainfall probability based on pixels is obtained after a clustering structure is output, and rainfall-absence classification is carried out to obtain a sample data set with rainfall.
In one embodiment, a normalized clustering algorithm is used in the feature space, the output clustering result is a continuous function with the value in the range of 0 to 1, and the continuous function value is set as the precipitation probability; the continuous function value 0.5 is set as a threshold value, and if the continuous function value is greater than 0.5, it indicates rainfall, and if the continuous function value is less than or equal to 0.5, it indicates no rainfall.
The working principle of the technical scheme is as follows: output dense core number A 1 ，A 2 ，…,A k Wherein A is i ＝j|y j ∈C i The value is in the interval of 0 to 1 when A i >When the water content is 0.5, it indicates rainfall, and when A is in the water content i When the rainfall is less than or equal to 0.5, the rainfall is absent.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the output value of the continuous function can be better distinguished by setting the threshold value, and rainfall and no rainfall can be visually seen through the peer-to-peer relationship between the output value and the rainfall probability.
In one embodiment, constructing the precipitation inversion model comprises the steps of: as shown in fig. 3, the construction method is as follows:
s401: the deep neural network of four layer network structure who establishes connection gradually, four layer network structure includes: an input layer, two hidden layers and an output layer;
s402: the input layer is a sample data set with rainfall obtained in the S3 and is processed;
s403: the hidden layer processes the data input by the input layer to determine the rainfall rate;
s404: and the output layer outputs the rainfall rate.
The working principle of the technical scheme is as follows: a deep neural network is understood to be a neural network with many hidden layers, which are divided into: the input layer, the hidden layer and the output layer, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. The layers are all connected, namely any neuron of the ith layer is necessarily connected with any neuron of the (i + 1) th layer. From the small local model, a deep neural network is a model with several inputs and one output, i.e. a linear relationship plus an activation function σ (z). The linear relation formula is as follows:
wherein t =1, 2., q, w t Is the coefficient of the t-th linear relationship, b is the bias, p t For the t-th input variable, z is the variable of the activation function σ (z).
The deep neural network has more layers, more parameters and a linear relation coefficient w t And the definition of bias b requires certain rules. Coefficient of linear relationship w t Definition of (c): the linear coefficients of the 4 th neuron of the second layer to the 2 nd neuron of the third layer are defined as
Superscript 3 represents the linear coefficient w t The index corresponds to the output level 2 and the index corresponds to the input level 4. Definition of bias b: the bias corresponding to the third neuron in the second layer is defined as
The deep neural network performs a series of linear operations and activation operations from an input layer by using an input vector t, a plurality of weight coefficient matrixes W and a bias vector b, calculates the output of a next layer by using the output of a previous layer, calculates backwards layer by layer until the operation is performed to an output layer to obtain an output result.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the performance of instantaneous precipitation detection and precipitation rate estimation at each initial input can be obviously improved through the deep neural network, the establishment of a physical forward model is not needed, and the problem caused by analyzing the physical forward model is avoided.
In one embodiment, the input layer is a sample data set with rainfall, and the processing is performed, specifically adopting the following scheme:
s4021: the input layer is a sample data set with rainfall acquired in S3;
s4022: adding a normalization layer behind an input layer, wherein the normalization layer is used for performing normalization processing on each input data item by adopting a Z-score method to normalize the data items to a standard normal distribution with the mean value of 0 and the variance of 1;
before data analysis, it is generally necessary to normalize the data and perform data analysis using the normalized data. And adding a normalization layer after the input layer, wherein the normalization layer is used for carrying out normalization processing on each input data item by adopting a Z-score method, and Z-score normalization is used for carrying out normalization on data based on the mean value (mean) and standard deviation (standard deviation) of the original data to obtain a standard normal distribution with the mean value of 0 and the variance of 1. The calculation formula is as follows:
in the above two formulae, l =1,2,3,4 l Is the l original value of the input data, v is the number of the input data, std (a) is the standard deviation of the input data, mean (a) is the mean value of the input data, normalized (a) 1 ) Indicating normalization of input data raw value a to a using Z-score 1 ；
The working principle of the technical scheme is as follows: the Z-score processing method is a common method for data processing in the data preparation stage of the whole framework, and is used for converting obtained rainfall-containing multiple groups of data into unitless Z-score scores for comparison, wherein the mean value of the processed data is 0, and the variance of the processed data is 1.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the data standard is unified through the Z-score processing method, and the calculated Z-score value is used for measurement, so that the data comparability is improved, and the data interpretability is weakened.
In one embodiment, the hidden layer processes data input by the input layer to determine the rainfall rate, and the specific implementation scheme is as follows:
s4031: adding a full connection layer after the normalization layer, wherein the parameter of the full connection layer is the number of neurons;
s4032: adding a nonlinear activation unit layer after the full connection layer, wherein the parameter of the nonlinear activation unit layer is an activation function ReLU;
s4033: adding a normalization layer after the nonlinear activation unit layer, wherein the normalization layer is used for carrying out normalization processing on each input data item by adopting a Z-score method;
s4034: and adding a dropout layer after the normalization layer, wherein the dropout layer is used for randomly discarding some neurons in each batch training process, preventing overfitting and determining the rainfall rate.
The working principle of the technical scheme is as follows: in the neural network of two hidden layers in this embodiment, a functional relationship exists between the output of an upper node and the input of a lower node, and this function adopts a modified linear unit ReLU activation function, and its expression is:
H(g)＝max(0，g)
h (g) is an activation function, and g is a variable of the activation function;
it is clear from the expression that the Relu activation function is in fact a function that takes the maximum value.
After the nonlinear activation unit layer processes data through a ReLU activation function, an output value is transmitted to a normalization layer, and the normalization layer performs normalization processing on each input data item by adopting a Z-score method;
the data processed by the normalization layer continuously pass through a dropout layer, and the dropout layer comprises the following steps when carrying out first batch processing training:
step 1, setting a probability of dropout;
step 2, taking off a part of neurons according to the corresponding probability, then starting training, updating the parameters of the neurons which are not taken off and the weight, and keeping the parameters;
and 3, after all the parameters are updated, taking off a part of neurons again according to the corresponding probability, then starting training, if the neuron newly used for training is trained in the first time, continuously updating the parameters of the neuron, cutting off the neurons for the second time, and simultaneously updating the parameters for the first time, keeping the weights of the neurons, and not modifying the neurons until the neurons are not deleted when the neurons are subjected to dropout in the last batch processing.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the normalization layer aims to standardize the distribution of the input x in the training process and has the effect of stabilizing the learning process; reLU as an excitation function can overcome the problem of gradient disappearance or saturation; the dropout layer randomly discards some neurons in each batch of training, which can prevent overfitting of the deep neural network.
In one embodiment, the operation of the output layer includes:
s4041: adding a full connection layer behind the dropout layer, wherein the parameter of the full connection layer is the number of neurons;
s4042: adding an activation function layer after the full connection layer, wherein the parameter of the activation function layer is an activation function Sigmoid;
s4043: and outputting the rainfall rate.
The working principle of the technical scheme is as follows: the Sigmoid function, also known as an S-shaped growth curve, is used for hidden layer neuron outputs, mapping the data to the interval of (0, 1). It is defined by the following formula:
wherein S (u) is a Sigmoid function, and u is a variable of the Sigmoid function;
the Sigmoid function is continuously smooth, strictly monotonic, centered at (0, 0.5), and a very good threshold function.
The beneficial effects of the above technical scheme are: with the solution provided in this embodiment, the Sigmoid function value domain is limited to (0, 1), and (0, 1) corresponds to the range of probability values, and the Sigmoid function value is associated with the pixel-based precipitation probability distribution. In addition, the Sigmoid function is used as a classifier of the logistic regression model, the derivation calculation of the Sigmoid function is very convenient, and the calculation time is also very saved.
In one embodiment, a method of evaluating the effect of precipitation inversion models comprises:
s501: calculating the difference between the result of a forward calculation formula of each iteration of the precipitation inversion model and an actual value by adopting an average absolute error loss function, guiding the precipitation inversion model to train, and then calculating the error of a predicted value and the actual value by adopting a root mean square error loss function:
s502: initializing parameters of a forward calculation formula by using random values;
s503: substituting the samples into a forward calculation formula, and calculating to obtain a predicted value;
s504: calculating the error between the predicted value and the actual value by using an average absolute error loss function;
s505: according to the derivative of the average absolute error loss function, returning the error along the minimum gradient direction, and correcting each weight value in the forward calculation formula;
s506: performing the actions of S502 to S505, and stopping the iteration until the average absolute error loss function value reaches an optimal value;
s507: and calculating errors of the predicted value and the actual value by adopting a root-mean-square error loss function, and evaluating the effect of the precipitation inversion model based on the errors of the predicted value and the actual value.
The working principle of the technical scheme is as follows: the mean absolute error loss function is the most intuitive one, and the Euclidean distance between the predicted value and the true value is calculated. The closer the predicted value and the true value are, the smaller the mean square error of the two. The effectiveness of the model is measured by using an average absolute error loss function, and then the errors of the predicted value and the actual value are calculated by using a root-mean-square error loss function.
The average absolute error loss function is calculated as:
the root mean square error loss function is calculated as:
in the above two formulas, MAE is the mean absolute error loss function, RMSE is the root mean square error loss function, r =1,2,3,4 r Is the model predicted value, y r Is the actual value.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the average absolute error is the average value of the absolute errors, so that the actual situation of the error of the predicted value can be better reflected, and the effectiveness of the precipitation estimation model is measured; the root mean square error is often used as a standard for measuring the prediction result of the machine learning model, and can ensure the training effect of the rainfall estimation model as a loss function in actual training.
In one embodiment, after the step S4, the method further includes S6: verifying and analyzing the precipitation inversion result and the passive microwave precipitation product; the method for performing verification analysis comprises the following steps:
s601: comparing the inversion rainfall rate output by the rainfall inversion result with the predicted rainfall rate in the passive microwave rainfall product; when the deviation of the inversion rainfall rate and the predicted rainfall rate is smaller than a preset deviation threshold value, the inversion rainfall rate is reserved; when the deviation between the inversion rainfall rate and the predicted rainfall rate is larger than a preset deviation threshold value, abandoning the inversion rainfall rate;
s602: estimating rainfall and non-rainfall according to preset conditions by using the reserved inversion rainfall rate; the preset conditions include a certain future time and a certain region.
The working principle of the technical scheme is as follows: estimating the precipitation condition in a certain region and a certain period of time according to the precipitation probability inverted by the precipitation estimation model; the specific method comprises the following steps: comparing the inversion rainfall rate output by the rainfall inversion result with the predicted rainfall rate in the passive microwave rainfall product; when the deviation of the inversion rainfall rate and the predicted rainfall rate is smaller than a preset deviation threshold value, the inversion rainfall rate is reserved; when the deviation between the inversion rainfall rate and the predicted rainfall rate is larger than a preset deviation threshold value, abandoning the inversion rainfall rate; estimating rainfall and non-rainfall according to preset conditions by using the reserved inversion rainfall rate; the preset conditions include a certain future time and a certain region.
The beneficial effects of the above technical scheme are: by adopting the scheme provided by the embodiment, the rainfall rate reversed by the rainfall estimation model can be estimated in advance, and the work of preventing flood disasters can be done in advance.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A microwave imager precipitation inversion method based on a hybrid neural network is characterized by comprising the following steps:
s1: establishing a data set, wherein the data set comprises observation data of a Fengyun No. three B star microwave imager and passive microwave precipitation products of an American ice and snow data center;
s2: preprocessing a data set, and matching the observation data with the passive microwave precipitation product according to a matching principle to obtain a matching data set meeting the matching principle;
s3: performing rainfall classification operation, clustering the matched data set by adopting an approximate spectral clustering algorithm with dense cores and density peak values, determining the dense cores, obtaining pixel-based rainfall probability according to the dense cores, and classifying rainfall and rainfall absence to obtain a rainfall sample data set;
s4: and executing a precipitation estimation operation, constructing a precipitation inversion model based on the depth neural network, and determining the rainfall rate by using the rainfall sample data set according to the precipitation inversion model.
2. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 1, wherein in the step S2, the method comprises the following steps:
s201: extracting brightness temperature, geographical position and time information of 5 channels according to the observation data; the observation data comprises dual-polarization observation brightness temperature data, an instrument height angle and an instrument incidence angle;
s202: extracting rainfall rate, geographical position and time information of the passive microwave rainfall product;
s203: the matching principle comprises the following steps: and selecting data with the matching data time difference not exceeding 30 minutes and the longitude and latitude difference not exceeding 0.05 degrees, resampling the passive microwave precipitation product, extracting data meeting the matching principle, and forming a matching data set by the extracted data.
3. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 1, wherein in S3, the following steps are included:
s301: constructing a graph structure according to the matching data set, wherein graph structure nodes correspond to matching data set data, and setting a graph structure node dense core according to the graph structure node dense degree; the dense core is a point with a local maximum density;
s302: defining new distances according to the public neighborhoods of the dense cores of the nodes of the graph structure, and calculating the distances among the dense cores according to the newly defined distances;
s303: constructing a non-directional weight graph and selecting an initial center;
s304: constructing a similarity matrix according to the quantity of the dense cores, and calculating the similarity between the initial center and the sum of the dense cores;
s305: constructing a Laplace matrix according to the similarity matrix, performing characteristic decomposition on the Laplace matrix, and selecting a characteristic vector to form a characteristic space;
s306: and outputting a clustering result by utilizing a normalized clustering algorithm in the feature space to obtain the rainfall probability based on the pixels, and classifying rainfall and rainfall-free to obtain a sample data set with rainfall.
4. The microwave imager rainfall inversion method based on the hybrid neural network of claim 3, wherein in the step S306, the output clustering result is a continuous function with a value in an interval range of 0 to 1, and the continuous function value is set as a rainfall probability; the continuous function value 0.5 is set as a threshold value, and if the continuous function value is greater than 0.5, it indicates rainfall, and if the continuous function value is less than or equal to 0.5, it indicates no rainfall.
5. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 1, wherein in S4, the construction of the precipitation inversion model comprises the following steps:
s401: constructing four-layer network structures which are connected in sequence, wherein the four-layer network structures comprise: an input layer, two hidden layers and an output layer;
s402: the input layer is a sample data set with rainfall obtained in the S3 and is processed;
s403: the hidden layer processes the data input by the input layer to determine the rainfall rate;
s404: and the output layer outputs the rainfall rate.
6. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 5, wherein said S402 comprises:
s4021: the input layer is a sample data set with rainfall obtained in the S3;
s4022: adding a normalization layer after the input layer, wherein the normalization layer is used for carrying out normalization processing on each input data item by adopting a Z-score method to normalize the data items to standard normal scores with the mean value of 0 and the variance of 1
And (4) cloth.
7. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 5, wherein the step S403 comprises:
s4031: adding a full connection layer after the normalization layer, wherein the parameter of the full connection layer is the number of neurons;
s4032: adding a nonlinear activation unit layer after the full connection layer, wherein the parameter of the nonlinear activation unit layer is an activation function ReLU;
s4033: adding a normalization layer after the nonlinear activation unit layer, wherein the normalization layer is used for carrying out normalization processing on each input data item by adopting a Z-score method;
s4034: and adding a dropout layer after the normalization layer, wherein the dropout layer is used for randomly discarding some neurons in each batch training process, preventing overfitting and determining the rainfall rate.
8. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 5, wherein said S404 comprises:
s4041: adding a full connection layer behind the dropout layer, wherein the parameter of the full connection layer is the number of neurons;
s4042: adding an activation function layer after the full connection layer, wherein the parameter of the activation function layer is an activation function Sigmoid;
s4043: and outputting the rainfall rate.
9. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 1, further comprising, after the step S4:
s5: evaluating the effect of the precipitation inversion model;
the method for evaluating the effect of the precipitation inversion model comprises the following steps:
s501: calculating the difference between the result of a forward calculation formula of each iteration of the precipitation inversion model and an actual value by adopting an average absolute error loss function, guiding the training of the precipitation inversion model, and then calculating the error between a predicted value and the actual value by adopting a root mean square error loss function:
s502: initializing parameters of a forward calculation formula by using random values;
s503: substituting the samples into a forward calculation formula, and calculating to obtain a predicted value;
s504: calculating the error between the predicted value and the actual value by using an average absolute error loss function;
s505: according to the derivative of the average absolute error loss function, returning the error along the minimum gradient direction, and correcting each weight value in the forward calculation formula;
s506: executing the actions from S502 to S505, and stopping iteration until the average absolute error loss function value reaches an optimal value;
s507: and calculating errors of the predicted value and the actual value by adopting a root-mean-square error loss function, and evaluating the effect of the precipitation inversion model based on the errors of the predicted value and the actual value.
10. The microwave imager precipitation inversion method based on the hybrid neural network as claimed in claim 1, wherein after the step S4, the method further comprises the step S6: verifying and analyzing precipitation inversion results and passive microwave precipitation products;
the method for performing verification analysis comprises the following steps:
s601: comparing the inversion rainfall rate output by the rainfall inversion result with the predicted rainfall rate in the passive microwave rainfall product; when the deviation between the inversion rainfall rate and the predicted rainfall rate is smaller than a preset deviation threshold value, the inversion rainfall rate is reserved; when the deviation of the inversion rainfall rate and the predicted rainfall rate is larger than a preset deviation threshold value, abandoning the inversion rainfall rate;
s602: estimating rainfall and rainfall absence according to preset conditions by using the reserved inversion rainfall rate; the preset conditions include a certain future time and a certain region.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210497652.5A CN115166871A (en) | 2022-05-09 | 2022-05-09 | Microwave imager rainfall inversion method based on hybrid neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210497652.5A CN115166871A (en) | 2022-05-09 | 2022-05-09 | Microwave imager rainfall inversion method based on hybrid neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115166871A true CN115166871A (en) | 2022-10-11 |
Family
ID=83484000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210497652.5A Pending CN115166871A (en) | 2022-05-09 | 2022-05-09 | Microwave imager rainfall inversion method based on hybrid neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115166871A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107239788A (en) * | 2017-04-20 | 2017-10-10 | 浙江工业大学 | The optimal selection Spectral Clustering of characteristic vector group based on degree adaptive |
| CN108874734A (en) * | 2018-04-25 | 2018-11-23 | 中国科学院国家空间科学中心 | A kind of Global Land Surface Precipitation inversion method |
| CN111652303A (en) * | 2020-05-28 | 2020-09-11 | 辽宁工程技术大学 | Outlier detection method based on spectral clustering under non-independent same distribution |
| CN111935747A (en) * | 2020-08-17 | 2020-11-13 | 南昌航空大学 | Method for predicting link quality of wireless sensor network by adopting GRU (generalized regression Unit) |
| CN113591387A (en) * | 2021-08-05 | 2021-11-02 | 安徽省气象台 | Huber norm constraint-based satellite data inversion precipitation method and system |
| CN114169502A (en) * | 2021-12-08 | 2022-03-11 | 桂林理工大学 | Rainfall prediction method and device based on neural network and computer equipment |
-
2022
- 2022-05-09 CN CN202210497652.5A patent/CN115166871A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107239788A (en) * | 2017-04-20 | 2017-10-10 | 浙江工业大学 | The optimal selection Spectral Clustering of characteristic vector group based on degree adaptive |
| CN108874734A (en) * | 2018-04-25 | 2018-11-23 | 中国科学院国家空间科学中心 | A kind of Global Land Surface Precipitation inversion method |
| CN111652303A (en) * | 2020-05-28 | 2020-09-11 | 辽宁工程技术大学 | Outlier detection method based on spectral clustering under non-independent same distribution |
| CN111935747A (en) * | 2020-08-17 | 2020-11-13 | 南昌航空大学 | Method for predicting link quality of wireless sensor network by adopting GRU (generalized regression Unit) |
| CN113591387A (en) * | 2021-08-05 | 2021-11-02 | 安徽省气象台 | Huber norm constraint-based satellite data inversion precipitation method and system |
| CN114169502A (en) * | 2021-12-08 | 2022-03-11 | 桂林理工大学 | Rainfall prediction method and device based on neural network and computer equipment |
Non-Patent Citations (3)
| Title |
|---|
| 李娜: ""基于FY-3C MWHTS 的台风降水反演算法研究"", 《遥感技术与应用》, vol. 34, no. 5, pages 1091 - 1100 * |
| 翟召坤: ""基于NSIDC海冰产品的FY北极海冰数据集优化"", 《地球信息科学》, vol. 19, no. 2, pages 143 - 151 * |
| 薛丽霞: "《基于密度峰值优化的谱聚类算法》", 《计算机应用研究》, vol. 36, no. 7, pages 1948 - 1983 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Han et al. | Convolutional neural network for convective storm nowcasting using 3-D Doppler weather radar data | |
| CN109492822B (en) | Air pollutant concentration time-space domain correlation prediction method | |
| CN110533631B (en) | SAR image change detection method based on pyramid pooling twin network | |
| CN107607954B (en) | FNN precipitation particle phase state identification method based on T-S model | |
| CN114254561A (en) | Waterlogging prediction method, waterlogging prediction system and storage medium | |
| CN104794730B (en) | SAR image segmentation method based on super-pixel | |
| Tabatabaei et al. | Optimization of sediment rating curve coefficients using evolutionary algorithms and unsupervised artificial neural network | |
| CN116307291B (en) | Distributed photovoltaic power generation prediction method and prediction terminal based on wavelet decomposition | |
| CN116152678A (en) | Marine disaster-bearing body identification method based on twin neural network under small sample condition | |
| CN114973019A (en) | Deep learning-based geospatial information change detection classification method and system | |
| Liu et al. | Soil water content forecasting by ANN and SVM hybrid architecture | |
| CN110826526A (en) | Method for cloud detection radar to identify clouds | |
| CN114882373A (en) | Multi-feature fusion sandstorm prediction method based on deep neural network | |
| CN111242028A (en) | Remote sensing image ground object segmentation method based on U-Net | |
| CN116525135B (en) | Method for predicting epidemic situation development situation by space-time model based on meteorological factors | |
| CN111707999A (en) | Sea surface floating small target detection method based on combination of multiple features and ensemble learning | |
| CN116758349A (en) | Hyperspectral image classification method based on multi-scale super-pixel node aggregate graph convolution residual network | |
| CN115166871A (en) | Microwave imager rainfall inversion method based on hybrid neural network | |
| CN113253275B (en) | Rainfall estimation method based on improved RBF neural network | |
| CN115345245A (en) | Tropical cyclone rapid reinforcement forecasting method based on random forest and transfer learning | |
| CN112464997B (en) | Multi-sensor collaborative monitoring method and system based on fuzzy-greedy search decision | |
| CN114444763A (en) | Wind power prediction method based on AFSA-GNN | |
| Zhang et al. | A three-dimensional convolutional-recurrent network for convective storm nowcasting | |
| CN108776968B (en) | SAR image change detection method based on deep forest | |
| Prasad et al. | A novel decision tree approach for the prediction of precipitation using entropy in sliq |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221011 |
|
| RJ01 | Rejection of invention patent application after publication |