CN114970743A - Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning - Google Patents
Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning Download PDFInfo
- Publication number
- CN114970743A CN114970743A CN202210684570.1A CN202210684570A CN114970743A CN 114970743 A CN114970743 A CN 114970743A CN 202210684570 A CN202210684570 A CN 202210684570A CN 114970743 A CN114970743 A CN 114970743A
- Authority
- CN
- China
- Prior art keywords
- data
- modal
- remote sensing
- rainfall data
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning, which comprises the following steps: the method comprises the following steps: collecting data of a research area; step two: preprocessing data; step three: constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method; step four: and evaluating the fusion effect of the multi-source rainfall data. The method can effectively capture the multi-modal information characteristics of different remote sensing rainfall data, including the data characteristics of an image grid space mode, a data matrix mode and a time sequence mode, and eliminates the redundancy among the modes by utilizing the complementarity among the multi-modalities so as to obtain more useful remote sensing rainfall data change characteristics. Meanwhile, the method disclosed by the invention is based on the fusion of multi-modal information characteristics of various deep learning algorithms, has stronger generalization capability, can process and associate information characteristics from various modes, further fuses and obtains rainfall data with higher precision, and has important significance for the key problems of drought and flood disaster forecast and early warning, reasonable water resource development, sustainable utilization and the like in China.
Description
Technical Field
The invention relates to the technical field of multi-source remote sensing rainfall product fusion, in particular to a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning.
Background
Rainfall is an important component of water circulation, and is affected by global warming and high-intensity human activities, and the spatial and temporal distribution characteristics of rainfall change remarkably. Acquiring high-precision regional rainfall data gradually becomes a problem of wide attention in the fields of atmospheric science, geography, hydrology and the like.
Ground station actual measurement, weather radar detection and satellite remote sensing monitoring are the current mainstream rainfall observation modes. The ground meteorological station has high observation precision on the point scale, has a direct observation mode, and is easily restricted by the density of the station network and the spatial distribution of the station network. The weather radar detection can be interfered by the external environment and self signals to generate system errors. The satellite remote sensing can obtain continuous time sequences and rainfall in a large space range, can reflect the spatial distribution characteristics of the rainfall, and has good real-time performance and coverage. Due to the difference of remote sensors and inversion model algorithms, the precision of various satellite remote sensing rainfall products in the same area is different, and how to fuse the advantages of different satellite rainfall products to obtain high-precision rainfall data is of great importance to water resource management and hydrological simulation.
At present, although the multi-source remote sensing rainfall data fusion has made some progress, the existing method mostly fuses single modal information characteristics in different satellite rainfall data, and neglects the difference of different modal sample characteristics of different satellite rainfall products. The multi-modal deep learning has great application potential in the multidisciplinary cross field, and the multi-modal deep learning is applied to a plurality of aspects such as audio-visual speech recognition, image-text emotion analysis, collaborative annotation, matching and classification, alignment expression learning and the like. The method for constructing the multi-source remote sensing rainfall data fusion based on the multi-modal deep learning effectively captures the dominant modal characteristics of different satellite rainfall products, and obtains high-precision rainfall data by utilizing the complementarity among the multiple modes is a problem which needs to be solved by technical personnel in the field at present.
Disclosure of Invention
The invention aims to provide a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning, aiming at fusing multi-modal information of different rainfall products through the multi-modal deep learning method and effectively acquiring modal information of different satellite rainfall products by utilizing complementarity among multiple modes, so that better information characteristics are learned, and rainfall data precision is improved through fusion.
In order to achieve the purpose, the invention provides the following technical scheme:
a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning comprises the following steps:
step one, collecting research area data: collecting multi-source remote sensing rainfall data and actually measured rainfall data of a ground meteorological station; the multi-source remote sensing rainfall data comprises GPM remote sensing rainfall data, CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data; the rainfall data of the ground meteorological station is actually measured rainfall data of daily scale;
step two, data preprocessing: the method comprises the steps of spatial resampling, multi-mode sample data extraction and spatial relationship matching; the spatial resampling is to process the multi-source remote sensing rainfall data to make the spatial resolution of the multi-source remote sensing rainfall data consistent; the multi-modal sample data extraction is to extract image grid space modal sample data, data matrix modal sample data and time sequence modal sample data from the multi-source remote sensing rainfall data; the spatial relationship matching is to match the extracted multi-modal sample data with rainfall data of the ground meteorological station through geographic point location coordinates to generate a multi-modal sample data set;
step three, constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method: the method comprises the steps of multi-modal sample data feature learning and multi-source rainfall data decision fusion.
The multi-mode sample data feature learning refers to learning sample data features of different modes for the multi-mode sample data set in the step two based on different deep learning methods; the different deep learning method for learning the data characteristics of different modes comprises the following steps: learning image grid space modal sample data characteristics by using a convolutional neural network model, learning data matrix modal sample data characteristics by using a multilayer perceptron, and learning time sequence modal sample data characteristics by using a long-short term memory network model;
and the multi-source rainfall data decision fusion refers to fusing the multi-modal sample data characteristics obtained by the multi-modal sample data characteristic learning by utilizing a multilayer fully-connected neural network.
Step four, evaluating the fusion effect of the multi-source rainfall data: and performing precision evaluation on the fused rainfall data by using the precision evaluation index.
Further, the data space resampling in the step two is to process CMORPH satellite remote sensing rainfall data with a spatial resolution of 0.25 °, PERSIANN satellite remote sensing rainfall data, and TRMM satellite remote sensing rainfall data into rainfall data with a spatial resolution of 0.1 ° by using a bilinear interpolation method, and the rainfall data is consistent with the spatial resolution of 0.1 ° of the GPM remote sensing rainfall data.
Further, the extracting of the sample data of the image grid spatial mode in the second step is specifically: based on daily-scale GPM remote sensing rainfall data, a 3 x 3 grid data set with a meteorological site as a center is extracted, and the spatial resolution of each grid is 0.3 degrees x 0.3 degrees.
Further, the extracted matrix modal sample data in the second step is CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data after joint space resampling, a satellite grid numerical value closest to the longitude and latitude where the meteorological site is located is selected as satellite product rainfall data corresponding to the site by taking the grid central point as a reference, and three different satellite product rainfall data corresponding to each meteorological site can form 1 × 3 matrix data.
Further, the extracting time series modal sample data in the second step is specifically GPM remote sensing rainfall data with a time resolution of 0.5 hour, and daily rainfall time series sample data of a satellite grid closest to the longitude and latitude where the meteorological site is located is extracted to form a 48 × 1 time series.
Further, the convolutional neural network model in the third step includes a convolutional layer, a pooling layer and a Flatten layer, the specific process of learning the image grid space modal sample data features by using the convolutional neural network model is that after the convolutional layer is subjected to modal feature extraction, an extracted feature map is transferred to the pooling layer for feature selection, and finally the Flatten layer converts a high-dimensional feature vector into a one-dimensional feature vector, and the specific formula is as follows:
in the formula (1)
A value representing the jth feature map of the ith layer at the (x, y) position; ReLU is the activation function of each layer; b ij Deviation of jth characteristic diagram of ith layer; m is an index of the feature map of layer i-1;
is the convolution kernel at position (x, y) connecting the mth feature map of the i-1 th layer and the jth feature map of the ith layer; p i ,Q i Height and width of the convolution kernel, respectively; p and q represent the convolution step size respectively.
Further, the specific formula for learning matrix modal sample data characteristics by using the multi-layer perceptron described in the third step is as follows:
featurevetor in formula (2) t Characteristic vectors in the t-th-day matrix modal sample data;
connecting weights between the kth neuron from the l-1 layer and the z-th neuron from the l layer;
bias for the z-th neuron of layer i; x is the number of Z And L is the number of layers of the neural network, which is the z-th input variable.
Further, the specific process of learning the time series modal sample data characteristics by using the long and short term memory model described in the third step is to input the time series samples of the time series modal into the long and short term memory model memory neural unit with a plurality of hidden layers for cyclic connection, then connect a full connection layer and output characteristic values, and the specific formula of the long and short term memory model memory neural unit is as follows:
in formula (3) i t Is an input gate; f. of t To forget the door; o t Is an output gate; sigma is a sigmoidal function; x is the number of t Inputting variables for the time step t; u. of i ,u f ,u c ,u o ,v i ,v f ,v c ,v o Is the weight of the neural network; k is a radical of i ,k f ,k c ,k o A neural network deviation vector is obtained; c. C t ,c t-1 The states of the memory units are respectively the current time step t and the last time step t-1; h is t ,h t-1 Hidden states of the current time step t and the last time step t-1 are respectively;
the specific formula for connecting a full connection layer and outputting a characteristic value is as follows:
featurevector t ＝f(h t ,h t-1 ,h t-2 …) (4)
formula (4)Middle featurevatector t And the feature vector is the feature vector of the modal sample data of the time series of the t day. f is the fully-connected layer function, h t ,h t-1 ,h t-2 The hidden states of the long-short term memory network model at t, t-1 and t-2 are respectively.
Further, the multi-source rainfall data fusion described in the third step is specifically that a Concatenate layer is adopted to splice a plurality of feature vectors extracted from an image grid mode, a matrix mode and a time sequence mode, then the feature vectors are input into a full-connection layer to be fused, fused rainfall data is output, and a formalized formula of decision fusion is as follows:
P i,t ＝f(featuremap(model1),featurevector(model2),featurevector(model3)) (5)
in the formula (5), P i,t The rainfall data of the ith station and the t day is obtained; f is a function of a conditioner layer, and featuremap (model1) is a feature vector extracted by model1 modal data; featurevaluectror (model2) is a feature vector extracted for model2 modal data; featurevaluectror (model3) is a feature vector extracted for model3 modal data; model1 is an image grid space modality; model2 is a data matrix modality; model3 is a time series modality.
Further, the evaluation index described in step four includes one or more of pearson correlation coefficient R, root mean square error RMSE, mean absolute error MAE, detection rate POD, false alarm rate FAR, and fair foreboding score ETS.
The specific formula of the Pearson correlation coefficient R is as follows:
the specific formula of the root mean square error RMSE is:
the specific formula of the mean absolute error MAE is:
the specific formula of the detection rate POD is:
the specific formula of the false alarm rate FAR is as follows:
the concrete formula of the fair premonitory score ETS is as follows:
wherein n is the number of weather stations in the research area, G i Representing measured rainfall data of meteorological sites, the average value of which is
y i Represents the corresponding rainfall data of the satellite rainfall product, and the average value is
Taking 0.1mm/d, 10mm/d, 25mm/d and 50mm/d as thresholds epsilon for "producing rainfall", "light rain", "medium rain" and "heavy rain", respectively, H represents that rainfall is observed and the satellite product detects a rainfall event, M represents a rainfall event that the satellite rainfall product does not detect, F represents that the satellite rainfall product detects a rainfall event but is not observed, and Z represents an event that the satellite rainfall product does not detect and the site does not observe rainfall, wherein the rainfall event refers to rainfall exceeding the threshold.
The invention has the beneficial effects that:
the invention provides a multi-mode deep learning-based multi-source remote sensing rainfall data fusion method which can effectively capture multi-mode information characteristics of different remote sensing rainfall data, including data characteristics of an image grid space mode, a data matrix mode and a time sequence mode, and better feature expression is learned by utilizing complementarity among the multi-modes and eliminating redundancy among the modes. Meanwhile, the method is based on various deep learning algorithms, has stronger generalization capability, further improves the precision of rainfall data obtained after multi-source rainfall data fusion, and has important significance for early warning of drought and flood disaster forecast and water resource development and reasonable utilization in China.
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Drawings
The invention is further illustrated by the following figures and examples.
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a technical roadmap for an embodiment of the method of the present invention;
FIG. 3 is a schematic diagram of multi-modal sample data extraction in an embodiment of the invention;
FIG. 4 is a schematic diagram of multi-source remote sensing rainfall data fusion in the embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings and tables in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
In this embodiment, a certain watershed is taken as a specific case area, and high-precision daily rainfall data of the watershed is taken as a research target, so as to further explain the specific application of the method of the present invention.
The embodiment discloses a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning, which comprises the following steps (as shown in figure 1):
step one, collecting research area data: collecting basin multi-source remote sensing rainfall data and ground meteorological station actual rainfall data; as shown in fig. 2, the multi-source remote sensing rainfall data includes GPM remote sensing rainfall data, CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data, and TRMM satellite remote sensing rainfall data; the rainfall data of the ground meteorological station is GPM IMERG Final Run V06B Level 3 data products, the time resolution is 0.5 hour and day respectively, and the spatial resolution is 0.1 degree; CMORPH satellite remote sensing rainfall data is a CMORPH Version 1.0 rainfall data product, the time resolution is a daily scale, and the spatial resolution is 0.25 degrees; the PERSIANN satellite remote sensing rainfall data is a PERSIANN _ CDR Version 1.0 rainfall data product, the time resolution is the daily scale, and the spatial resolution is 0.25 degrees; the TRMM satellite remote sensing rainfall data is a Level 3TRMM3B42V7 data product, and the time resolution day and the spatial resolution day are 0.25 degrees.
Step two, data preprocessing: the method comprises the steps of spatial resampling, multi-mode sample data extraction and spatial relationship matching; the spatial resampling is to process CMORPH satellite remote sensing rainfall data with spatial resolution of 0.25 degrees, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data into rainfall data with spatial resolution of 0.1 degrees by adopting a bilinear interpolation method; as shown in fig. 2, the multi-modal sample data extraction is to extract sample data of an image raster space modality (modality 1), sample data of a data matrix modality (modality 2) and sample data of a time series modality (modality 3) from the multi-source remote sensing rainfall data; and the spatial relationship matching is to match the extracted multi-modal sample data with rainfall data of the ground meteorological station through the geographic point location coordinates to obtain a multi-modal sample data set.
As shown in fig. 3, a specific way to extract the sample data of the image grid spatial mode is as follows: and extracting sub-grid data taking the meteorological site as the center for each meteorological site by using GPM (general purpose computer) remote sensing rainfall data with time resolution as a daily scale, wherein the spatial resolution of each sub-grid is 0.3 degrees multiplied by 0.3 degrees.
The specific way of extracting the matrix modal sample data is as follows: combining CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data after spatial resampling, taking a grid central point as a reference, selecting a satellite grid numerical value closest to the longitude and latitude where a meteorological site is located as satellite product rainfall data corresponding to the site, and combining three different satellite product rainfall data corresponding to each meteorological site to form 1 x 3 matrix data.
The specific way of extracting the time series modal sample data is as follows: based on GPM remote sensing rainfall data with the time resolution of 0.5 hour, sample data of a daily rainfall time sequence of a satellite grid closest to the longitude and latitude where the meteorological site is located is extracted, and a 48 multiplied by 1 time sequence is formed.
Step three, constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method, as shown in fig. 4: learning image grid mode sample data characteristics by using a convolutional neural network model (CNN), learning matrix mode sample data characteristics by using a multilayer perceptron (MLP) and learning time sequence mode sample data characteristics by using a long-short term memory model (LSTM);
specifically, the convolutional neural network model comprises a convolutional layer, a pooling layer and a Flatten layer, the specific process of learning the image grid space modal sample data features by using the convolutional neural network model is that after modal feature extraction is performed on the convolutional layer, an extracted feature map is transmitted to the pooling layer for feature selection, and finally the Flatten layer converts high-dimensional feature vectors into one-dimensional feature vectors, and the specific formula is as follows:
in the formula (1)
A value representing the jth feature map of the ith layer at the (x, y) position; ReLU is the activation function of each layer; b ij Deviation of jth characteristic diagram of ith layer; m is an index of the feature map of layer i-1;
is the convolution kernel at position (x, y) connecting the mth feature map of the i-1 th layer and the jth feature map of the ith layer; p i ,Q i Height and width of the convolution kernel, respectively; p and q represent the convolution step size respectively.
The specific formula for learning matrix modal sample data characteristics by using the multilayer perceptron is as follows:
featurevetor in formula (2) t Characteristic vectors in the t-th-day matrix modal sample data;
connecting weights between the kth neuron from the l-1 layer and the z-th neuron from the l layer;
bias for the z-th neuron of layer i; x is the number of Z And L is the number of layers of the neural network, which is the z-th input variable.
The specific process of learning the time series modal sample data characteristics by using the long and short term memory model (LSTM) is to input the time series modal sample data into a long and short term memory model memory neural unit with a plurality of hidden layers for cyclic connection, then connect a full connection layer and output characteristic values, and the specific formula of the long and short term memory model memory neural unit is as follows:
in formula (3) i t Is an input gate; f. of t To forget the door; o t Is an output gate; sigma is a sigmoidal function; x is the number of t Inputting variables for the time step t; u. of i ,u f ,u c ,u o ,v i ,v f ,v c ,v o Is the weight of the neural network; k is a radical of i ,k f ,k c ,k o A neural network deviation vector is obtained; c. C t ,c t-1 The states of the memory units are respectively the current time step t and the last time step t-1; h is t ,h t-1 Hidden states of the current time step t and the last time step t-1 are respectively;
the specific formula for connecting a full connection layer and outputting a characteristic value is as follows:
featurevector t ＝f(h t ,h t-1 ,h t-2 …) (4)
featurevector in formula (4) t And the feature vector is the feature vector of the modal sample data of the time series of the t day. f is the fully-connected layer function, h t ,h t-1 ,h t-2 The hidden states of the long-short term memory network model at t, t-1 and t-2 are respectively.
The decision fusion of this embodiment specifically includes splicing a plurality of feature vectors extracted from an image grid modality, a matrix modality, and a time series modality by using a concatemate layer, then inputting the spliced feature vectors to a full-connection layer for fusion, and outputting fused rainfall data, where a formalized formula of the decision fusion is:
P i,t ＝f(featuremap(model1),featurevector(model2),featurevector(model3)) (5)
in the formula (5), P i,t The rainfall data of the ith station and the t day is obtained; f is a function of a conditioner layer, and featuremap (model1) is a feature vector extracted by model1 modal data; featurevaluectror (model2) is a feature vector extracted for model2 modal data; featurevaluectror (model3) is a feature vector extracted for model3 modal data; model1 is an image grid space modality; model2 is a data matrix modality; model3 is a time series modality.
Step four, evaluating the fusion effect of the multi-source rainfall data: and fusing the multisource satellite remote sensing rainfall data of the research area by using the trained model, acquiring the rainfall data fused in the research area, and performing precision evaluation on the fused rainfall data through the evaluation index.
In this embodiment, the correlation coefficient R, the mean absolute error MAE, and the root mean square error RMSE are used as evaluation indexes to evaluate the precision of the fused rainfall data obtained by the method of the present invention, and a calculation formula of statistical indexes is selected as follows:
the specific formula of the pearson correlation coefficient R is:
the specific formula of the mean absolute error MAE is:
the specific formula of the root mean square error RMSE is:
the accuracy evaluation result of the watershed scale multi-source rainfall data fusion method in the embodiment is shown in table 1. From table 1, the correlation coefficient between the fused rainfall data obtained by the multi-source remote sensing rainfall data fusion method and the rainfall data measured at the meteorological site in the drainage basin is higher than that of a single remote sensing rainfall product and can reach 0.78, and the average absolute error and the root mean square error are lower than those of a single original remote sensing rainfall product.
Table 1 embodiment multi-source remote sensing rainfall data fusion result precision evaluation of certain watershed
Finally, it should be noted that the above-mentioned is only for illustrating the technical solution of the present invention and not for limiting, although the present invention has been described in detail with reference to the preferred arrangement, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (10)
1. A multi-source remote sensing rainfall data fusion method based on multi-modal deep learning is characterized by comprising the following steps:
step one, collecting research area data: collecting multi-source remote sensing rainfall data and actually measured rainfall data of a ground meteorological station; the multi-source remote sensing rainfall data comprises GPM remote sensing rainfall data, CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data; selecting actual rainfall data of a daily scale from the actual rainfall data of the ground meteorological station;
step two, data preprocessing: data space resampling, multi-mode sample information extraction and data space relation matching are included; the data space resampling is to process the remote sensing rainfall data, so that the spatial resolution of different remote sensing rainfall data is kept consistent; the multi-modal sample information extraction is to extract image grid space modal sample data, data matrix modal sample data and time sequence modal sample data from remote sensing rainfall data; the spatial relationship matching is to match the extracted multimodal sample information with rainfall data of the ground meteorological station through geographic coordinates to generate a multimodal sample data set;
step three, constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method: the method comprises the steps of multi-modal sample data feature learning and multi-source rainfall data decision fusion;
the multi-mode sample data feature learning refers to learning sample data features of different modes for the multi-mode sample data set in the step two based on different deep learning methods, and comprises the following steps: learning image grid space modal sample data characteristics by using a convolutional neural network model, learning mathematical matrix modal sample data characteristics by using a multilayer perceptron, and learning time sequence modal sample data characteristics by using a long-short term memory network model;
the multi-source rainfall data decision fusion means that multi-modal sample data features obtained through multi-modal sample data feature learning are fused by utilizing a multi-layer fully-connected neural network;
step four, evaluating the fusion effect of the multi-source rainfall data: and performing precision evaluation on the fused rainfall data by using the precision evaluation index.
2. The multi-source remote sensing rainfall data fusion method based on the multi-modal deep learning according to claim 1, wherein the data space resampling in the step two is to resample the remote sensing rainfall data by using a bilinear interpolation method.
3. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, wherein the extracting of the image grid space modal sample data in the second step is specifically: and extracting a raster data set taking a meteorological site as a center based on the daily scale GPM remote sensing rainfall data.
4. The multi-source remote sensing rainfall data fusion method based on the multi-modal deep learning, which is characterized in that the extracted data matrix modal sample data in the second step is CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and satellite TRMM remote sensing rainfall data after joint space resampling, a satellite grid numerical value closest to the longitude and latitude where a meteorological site is located is selected as satellite product rainfall data corresponding to the site by taking a grid central point as a reference, and three different satellite product rainfall data corresponding to each meteorological site can form 1 x 3 matrix data.
5. The multi-source remote sensing rainfall data fusion method based on the multi-modal deep learning according to claim 1, wherein the time series modal sample data extracted in the second step is a 48 x 1 time series obtained by extracting the daily rainfall time series sample data of a satellite grid closest to the longitude and latitude where the meteorological site is located based on GPM remote sensing rainfall data with a time resolution of 0.5 hour.
6. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, characterized in that the convolutional neural network model in step three comprises a convolutional layer, a pooling layer and a Flatten layer, and the specific process of learning image grid space modal sample data features by using the convolutional neural network model is that after modal feature extraction is performed on the convolutional layer, extracted feature image features are transferred to the pooling layer for feature selection, and finally high-dimensional feature vectors are converted into one-dimensional feature vectors in the Flatten layer, and the specific formula is as follows:
in the formula (I), the compound is shown in the specification,
a value representing the jth feature map of the ith layer at the (x, y) position; ReLU is the activation function of each layer; b ij Deviation of jth characteristic diagram of ith layer; m is an index of the feature map of layer i-1;
is the convolution kernel at position (x, y) connecting the mth feature map of the i-1 th layer and the jth feature map of the ith layer; p i ,Q i Height and width of the convolution kernel, respectively; p and q represent the convolution step size respectively.
7. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, wherein the specific formula for learning data matrix modal sample data characteristics by using a multi-layered perceptron in the third step is as follows:
in the formula, featurevoeter t Characteristic vectors in the data matrix modal sample data of the t day are obtained;
connecting weights between the kth neuron from the l-1 layer and the z-th neuron from the l layer;
bias for the z-th neuron of layer i; x is the number of Z And L is the number of layers of the neural network, which is the z-th input variable.
8. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, wherein the specific process of learning the time series modal sample data characteristics by using the long and short term memory network model in step three is that the time series samples of the time series modal are input into a memory neural unit of the long and short term memory network model comprising a plurality of hidden layers for cyclic connection, and then are connected to a full connection layer and output characteristic values, wherein the specific formula of the long and short term memory neural unit is as follows:
in the formula i t Is an input gate; f. of t To forget the door; o t Is an output gate; sigma is a sigmoidal function; x is the number of t Inputting variables for the time step t; u. of i ,u f ,u c ,u o ,v i ,v f ,v c ,v o Weights for neural networks；k i ,k f ,k c ,k o Is a neural network deviation vector; c. C t ,c t-1 The states of the memory units are respectively the current time t and the last time t-1; h is a total of t ,h t-1 Hidden states of the current time t and the last time t-1 are respectively;
the specific formula for connecting a full connection layer and outputting a characteristic value is as follows:
featurevector t ＝f(h t ,h t-1 ,h t-2 …) (4)
in the formula, featurevatervector t Is the characteristic vector of the t-th day time sequence modal sample data characteristic, f is the full-continuous layer function, h t ,h t-1 ,h t-2 The hidden states of the long-short term memory network model at t, t-1 and t-2 are respectively.
9. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, characterized in that the multi-source rainfall data fusion in step three specifically comprises a Concatenate layer to splice a plurality of feature vectors extracted from an image grid space mode, a data matrix mode and a time sequence mode, then input to a full connection layer for fusion and output fused rainfall data, and a formalized formula of decision fusion is as follows:
P i,t ＝f(featuremap(model1),featurevector(model2),featurevector(model3))(5)
in the formula, P i,t The rainfall data of the ith station and the t day is obtained; f is a function of a conditioner layer, and featuremap (model1) is a feature vector extracted by model1 modal data; featurevaluectror (model2) is a feature vector extracted for model2 modal data; featurevaluectror (model3) is a feature vector extracted for model3 modal data; model1 is an image grid space modality; model2 is a data matrix modality; model3 is a time series modality.
10. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning of claim 1, wherein the evaluation index in step four comprises one or more of Pearson correlation coefficient R, Root Mean Square Error (RMSE), Mean Absolute Error (MAE), detection rate (POD), False Alarm Rate (FAR) and fair foreboding score (ETS).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210684570.1A CN114970743B (en) | 2022-06-17 | 2022-06-17 | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210684570.1A CN114970743B (en) | 2022-06-17 | 2022-06-17 | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114970743A true CN114970743A (en) | 2022-08-30 |
| CN114970743B CN114970743B (en) | 2022-11-08 |
Family
ID=82962804
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210684570.1A Active CN114970743B (en) | 2022-06-17 | 2022-06-17 | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114970743B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116070792A (en) * | 2023-03-28 | 2023-05-05 | 中国科学院地理科学与资源研究所 | Fusion method, device, storage medium and equipment of multi-source precipitation data |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295714A (en) * | 2016-08-22 | 2017-01-04 | 中国科学院电子学研究所 | A kind of multi-source Remote-sensing Image Fusion based on degree of depth study |
| CN107992904A (en) * | 2017-12-22 | 2018-05-04 | 重庆邮电大学 | Forest Ecology man-machine interaction method based on Multi-source Information Fusion |
| CN108761574A (en) * | 2018-05-07 | 2018-11-06 | 中国电建集团北京勘测设计研究院有限公司 | Rainfall evaluation method based on Multi-source Information Fusion |
| CN111401602A (en) * | 2019-12-31 | 2020-07-10 | 河海大学 | Assimilation method for satellite and ground rainfall measurement values based on neural network |
| CN111665560A (en) * | 2020-04-23 | 2020-09-15 | 中国石油天然气股份有限公司 | Oil-gas reservoir identification method and device, computer equipment and readable storage medium |
| CN111859800A (en) * | 2020-07-15 | 2020-10-30 | 河海大学 | Method for spatio-temporal estimation and prediction of PM2.5 concentration distribution |
| CN112308029A (en) * | 2020-11-24 | 2021-02-02 | 国网湖南省电力有限公司 | Rainfall station and satellite rainfall data fusion method and system |
| US20210264197A1 (en) * | 2020-02-25 | 2021-08-26 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Point cloud data processing method, apparatus, electronic device and computer readable storage medium |
| CN113657472A (en) * | 2021-08-02 | 2021-11-16 | 中国空间技术研究院 | Multi-source remote sensing data fusion method based on subspace learning |
| CN114418038A (en) * | 2022-03-29 | 2022-04-29 | 北京道达天际科技有限公司 | Space-based information classification method and device based on multi-mode fusion and electronic equipment |
-
2022
- 2022-06-17 CN CN202210684570.1A patent/CN114970743B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295714A (en) * | 2016-08-22 | 2017-01-04 | 中国科学院电子学研究所 | A kind of multi-source Remote-sensing Image Fusion based on degree of depth study |
| CN107992904A (en) * | 2017-12-22 | 2018-05-04 | 重庆邮电大学 | Forest Ecology man-machine interaction method based on Multi-source Information Fusion |
| CN108761574A (en) * | 2018-05-07 | 2018-11-06 | 中国电建集团北京勘测设计研究院有限公司 | Rainfall evaluation method based on Multi-source Information Fusion |
| CN111401602A (en) * | 2019-12-31 | 2020-07-10 | 河海大学 | Assimilation method for satellite and ground rainfall measurement values based on neural network |
| US20210264197A1 (en) * | 2020-02-25 | 2021-08-26 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Point cloud data processing method, apparatus, electronic device and computer readable storage medium |
| CN111665560A (en) * | 2020-04-23 | 2020-09-15 | 中国石油天然气股份有限公司 | Oil-gas reservoir identification method and device, computer equipment and readable storage medium |
| CN111859800A (en) * | 2020-07-15 | 2020-10-30 | 河海大学 | Method for spatio-temporal estimation and prediction of PM2.5 concentration distribution |
| CN112308029A (en) * | 2020-11-24 | 2021-02-02 | 国网湖南省电力有限公司 | Rainfall station and satellite rainfall data fusion method and system |
| CN113657472A (en) * | 2021-08-02 | 2021-11-16 | 中国空间技术研究院 | Multi-source remote sensing data fusion method based on subspace learning |
| CN114418038A (en) * | 2022-03-29 | 2022-04-29 | 北京道达天际科技有限公司 | Space-based information classification method and device based on multi-mode fusion and electronic equipment |
Non-Patent Citations (4)
| Title |
|---|
| FRANCESCOPICCIALLI等: "Artificial intelligence and healthcare: Forecasting of medical bookings through multi-source time-series fusion", 《INFORMATION FUSION》 * |
| ISAAC KOFI NTI等: "A novel multi-source information-fusion predictive framework based on deep neural networks for accuracy enhancement in stock market prediction", 《JOURNAL OF BIG DATA》 * |
| 周康辉: "基于深度卷积神经网络的强对流天气预报方法研究", 《中国博士学位论文全文数据库基础科学辑》 * |
| 黄彤辉等: "GIS和RS技术应用于流域水生生物时空分析的研究进展", 《环境科学研究》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116070792A (en) * | 2023-03-28 | 2023-05-05 | 中国科学院地理科学与资源研究所 | Fusion method, device, storage medium and equipment of multi-source precipitation data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114970743B (en) | 2022-11-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Dinda et al. | An integrated simulation approach to the assessment of urban growth pattern and loss in urban green space in Kolkata, India: A GIS-based analysis | |
| CN105243435B (en) | A kind of soil moisture content prediction technique based on deep learning cellular Automation Model | |
| CN111310968A (en) | LSTM neural network circulation hydrological forecasting method based on mutual information | |
| CN111639787A (en) | Spatio-temporal data prediction method based on graph convolution network | |
| Kovordányi et al. | Cyclone track forecasting based on satellite images using artificial neural networks | |
| Saxena et al. | A review study of weather forecasting using artificial neural network approach | |
| CN112507861A (en) | Pedestrian detection method based on multilayer convolution feature fusion | |
| Argany et al. | Impact of the quality of spatial 3D city models on sensor networks placement optimization | |
| Li et al. | An object-based approach for verification of precipitation estimation | |
| Abdi et al. | Regional frequency analysis using Growing Neural Gas network | |
| CN114970743B (en) | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning | |
| CN116720156A (en) | Weather element forecasting method based on graph neural network multi-mode weather data fusion | |
| Faruq et al. | Deep Learning-Based Forecast and Warning of Floods in Klang River, Malaysia. | |
| WO2023105294A1 (en) | Methods, systems, devices and neural networks for forecasting a time series | |
| CN114881286A (en) | Short-time rainfall prediction method based on deep learning | |
| Wu et al. | A novel bayesian additive regression trees ensemble model based on linear regression and nonlinear regression for torrential rain forecasting | |
| CN116884222B (en) | Short-time traffic flow prediction method for bayonet nodes | |
| Al Kindhi et al. | Sensor and internet of things based integrated inundation mitigation for smart city. | |
| Ramadas et al. | Predictor selection for streamflows using a graphical modeling approach | |
| CN115438841A (en) | Training method and prediction method based on artificial intelligence accurate prediction rainfall model | |
| Escalante-Sandoval et al. | Regional monthly runoff forecast in southern Canada using ANN, K-means, and L-moments techniques | |
| Luo et al. | Automatic floor map construction for indoor localization | |
| CN113011657A (en) | Method and device for predicting typhoon water level, electronic equipment and storage medium | |
| Özbilge et al. | Modelling and analysis of IoT technology using neural networks in agriculture environment | |
| Sharma et al. | Hydrologic simulation approach for El Niño Southern Oscillation (ENSO)-affected watershed with limited raingauge stations |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |