CN114970743B - Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning - Google Patents
Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning Download PDFInfo
- Publication number
- CN114970743B CN114970743B CN202210684570.1A CN202210684570A CN114970743B CN 114970743 B CN114970743 B CN 114970743B CN 202210684570 A CN202210684570 A CN 202210684570A CN 114970743 B CN114970743 B CN 114970743B
- Authority
- CN
- China
- Prior art keywords
- data
- remote sensing
- modal
- rainfall data
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a multi-source remote sensing rainfall data fusion method based on multi-mode deep learning, which comprises the following steps: the method comprises the following steps: collecting data of a research area; step two: preprocessing data; step three: constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method; step four: and evaluating the fusion effect of the multi-source rainfall data. The method can effectively capture the multi-modal information characteristics of different remote sensing rainfall data, including the data characteristics of an image grid space mode, a data matrix mode and a time sequence mode, and eliminates the redundancy among the modes by utilizing the complementarity among the multi-modalities so as to obtain more useful remote sensing rainfall data change characteristics. Meanwhile, the method disclosed by the invention is based on the fusion of multi-modal information characteristics of various deep learning algorithms, has stronger generalization capability, can process and associate information characteristics from various modes, further fuses and obtains rainfall data with higher precision, and has important significance for the key problems of drought and flood disaster forecast and early warning, reasonable water resource development, sustainable utilization and the like in China.
Description
Technical Field
The invention relates to the technical field of multi-source remote sensing rainfall product fusion, in particular to a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning.
Background
Rainfall is an important component of water circulation, and is affected by global warming and high-intensity human activities, and the spatial and temporal distribution characteristics of rainfall change remarkably. Acquiring high-precision regional rainfall data has gradually become a problem of wide attention in the fields of atmospheric science, geography, hydrology and the like.
Ground station actual measurement, weather radar detection and satellite remote sensing monitoring are the current mainstream rainfall observation modes. The ground meteorological station has high observation precision on the point scale, has a direct observation mode, and is easily restricted by the density of the station network and the spatial distribution of the station network. The weather radar detection is interfered by external environment and self signals to generate system errors. The satellite remote sensing can obtain continuous time sequences and rainfall in a large space range, can reflect the spatial distribution characteristics of the rainfall, and has good real-time performance and coverage. Due to the difference of remote sensors and inversion model algorithms, the precision of various satellite remote sensing precipitation products in the same area is different, and how to fuse the advantages of different satellite precipitation products and obtain high-precision rainfall data is very important for water resource management and hydrological simulation.
Currently, although multi-source remote sensing rainfall data fusion has made some progress, the existing method mostly fuses single-modal information characteristics in different satellite rainfall data, and ignores the difference of different modal sample characteristics of different satellite rainfall products. The multi-mode deep learning has great application potential in the multidisciplinary crossing field, and the multi-mode deep learning is applied to a plurality of aspects such as audio-visual speech recognition, image-text emotion analysis, collaborative labeling, matching and classification, alignment expression learning and the like. The method for constructing the multi-source remote sensing rainfall data fusion based on the multi-modal deep learning effectively captures the dominant modal characteristics of different satellite rainfall products, and obtains high-precision rainfall data by utilizing the complementarity among the multiple modes is a problem which needs to be solved by technical personnel in the field at present.
Disclosure of Invention
The invention aims to provide a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning, aiming at fusing multi-modal information of different rainfall products through the multi-modal deep learning method and effectively acquiring modal information of different satellite rainfall products by utilizing complementarity among multiple modes, so that better information characteristics are learned, and rainfall data precision is improved through fusion.
In order to achieve the purpose, the invention provides the following technical scheme:
a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning comprises the following steps:
step one, collecting research area data: collecting multi-source remote sensing rainfall data and actually measured rainfall data of a ground meteorological station; the multi-source remote sensing rainfall data comprises GPM remote sensing rainfall data, CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data; the rainfall data of the ground meteorological station is actually measured rainfall data of daily scale;
step two, data preprocessing: the method comprises the steps of spatial resampling, multi-modal sample data extraction and spatial relationship matching; the spatial resampling is to process the multi-source remote sensing rainfall data to make the spatial resolution of the multi-source remote sensing rainfall data consistent; the multi-mode sample data extraction is to extract image grid space modal sample data, data matrix modal sample data and time series modal sample data from the multi-source remote sensing rainfall data; the spatial relationship matching is to match the extracted multimodal sample data with rainfall data of the ground meteorological station through geographic point location coordinates to generate a multimodal sample data set;
step three, constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method: the method comprises the steps of multi-modal sample data feature learning and multi-source rainfall data decision fusion.
The multi-mode sample data feature learning refers to learning sample data features of different modes for the multi-mode sample data set in the step two based on different deep learning methods; the different deep learning method for learning the data characteristics of different modes comprises the following steps: learning image grid space modal sample data characteristics by using a convolutional neural network model, learning data matrix modal sample data characteristics by using a multilayer perceptron, and learning time sequence modal sample data characteristics by using a long-short term memory network model;
and the multi-source rainfall data decision fusion refers to fusing the multi-modal sample data characteristics obtained by the multi-modal sample data characteristic learning by utilizing a multilayer fully-connected neural network.
Step four, evaluating the fusion effect of the multi-source rainfall data: and performing precision evaluation on the fused rainfall data by using the precision evaluation index.
Further, the data space resampling in the step two is to process CMORPH satellite remote sensing rainfall data with a spatial resolution of 0.25 °, PERSIANN satellite remote sensing rainfall data, and TRMM satellite remote sensing rainfall data into rainfall data with a spatial resolution of 0.1 ° by using a bilinear interpolation method, and the rainfall data is consistent with the spatial resolution of 0.1 ° of the GPM remote sensing rainfall data.
Further, the extracting of the sample data of the image grid spatial mode in the second step is specifically: based on daily scale GPM remote sensing rainfall data, a 3 x 3 grid data set with a meteorological site as a center is extracted, and the spatial resolution of each grid is 0.3 degrees x 0.3 degrees.
Further, the matrix mode sample data extracted in the second step is CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data after joint space resampling, a satellite grid value closest to the longitude and latitude where a meteorological site is located is selected as satellite product rainfall data corresponding to the site by taking a grid central point as a reference, and three different satellite product rainfall data corresponding to each meteorological site can form 1 x 3 matrix data.
Further, the time series modal sample data extracted in the second step is GPM remote sensing rainfall data with a time resolution of 0.5 hour, and the satellite grid daily rainfall time series sample data closest to the latitude and longitude where the meteorological site is located is extracted to form a 48 × 1 time series.
Further, the convolutional neural network model described in the third step includes a convolutional layer, a pooling layer and a scatter layer, the specific process of learning the image grid space modal sample data features by using the convolutional neural network model is that after the convolutional layer performs modal feature extraction, the extracted feature map is transferred to the pooling layer for feature selection, and finally the scatter layer converts the high-dimensional feature vector into a one-dimensional feature vector, and the specific formula is as follows:
in the formula (1)
A value representing the jth feature map of the ith layer at the (x, y) position; reLU is the activation function of each layer; b ij Deviation of jth characteristic diagram of ith layer; m is an index of the feature map of layer i-1;
is the convolution kernel at position (x, y) connecting the mth feature map of the i-1 th layer and the jth feature map of the ith layer; p i ,Q i Height and width of the convolution kernel, respectively; p and q represent the convolution step size respectively.
Further, the specific formula for learning matrix modal sample data characteristics by using the multi-layer perceptron described in the third step is as follows:
featurevetor in the formula (2) t For the characteristic direction in the t-th matrix modal sample dataAn amount;
connecting weights between the kth neuron from the l-1 layer and the z-th neuron from the l layer;
bias for the z-th neuron of layer i; x is the number of Z And L is the number of neural network layers as the z input variable.
Further, the specific process of learning the time series modal sample data characteristics by using the long and short term memory model described in the third step is to input the time series samples of the time series modal into the long and short term memory model memory neural unit with a plurality of hidden layers for cyclic connection, then connect a full connection layer and output characteristic values, and the specific formula of the long and short term memory model memory neural unit is as follows:
in formula (3) i t Is an input gate; f. of t To forget the door; o t Is an output gate; sigma is a sigmoidal function; x is the number of t Inputting variables for the time step t; u. of i ,u f ,u c ,u o ,v i ,v f ,v c ,v o Is the weight of the neural network; k is a radical of formula i ,k f ,k c ,k o A neural network deviation vector is obtained; c. C t ,c t-1 The memory unit states of the current time step t and the last time step t-1 are respectively; h is a total of t ,h t-1 Hidden states of the current time step t and the last time step t-1 are respectively;
the specific formula for connecting a full connection layer and outputting a characteristic value is as follows:
featurevector t ＝f(h t ,h t-1 ,h t-2 …) (4)
featurevector in formula (4) t And the characteristic vector is the characteristic of the data of the time series modal sample at the t day. f is the fully-connected layer function, h t ,h t-1 ,h t-2 The hidden states of the long-short term memory network model at t, t-1 and t-2 are respectively.
Furthermore, the multi-source rainfall data fusion described in the third step is specifically to adopt a conditioner layer to splice a plurality of feature vectors extracted from an image grid mode, a matrix mode and a time sequence mode, then input the feature vectors into a full-connection layer for fusion and output the fused rainfall data, and the formalized formula of decision fusion is as follows:
P i,t ＝f(featuremap(model1),featurevector(model2),featurevector(model3)) (5)
in the formula (5), P i,t The rainfall data of the ith station and the t day is obtained; f is a function of a conditioner layer, and featuremap (model 1) is a feature vector extracted by model1 modal data; a featurevaluever (model 2) is a feature vector extracted by model2 modal data; a feature vector extracted by a featurever (model 3) for model3 modal data; model1 is an image grid spatial mode; model2 is a data matrix modality; model3 is a time series modality.
Further, the evaluation index described in the fourth step includes one or more of pearson correlation coefficient R, root mean square error RMSE, mean absolute error MAE, detection rate POD, false positive rate FAR, and fair forebodice score ETS.
The specific formula of the Pearson correlation coefficient R is as follows:
the specific formula of the root mean square error RMSE is:
the specific formula of the mean absolute error MAE is:
the specific formula of the detection rate POD is:
the specific formula of the false alarm rate FAR is as follows:
the specific formula of the just premonitory score ETS is as follows:
wherein n is the number of weather stations in the research area, G i The measured rainfall data of the representative meteorological site is the average value
y i Represents the corresponding rainfall data of the satellite rainfall product, and the average value is
With 0.1mm/d,10mm/d,25mm/d and 50mm/d as thresholds ε for "producing rain", "light rain", "medium rain" and "heavy rain", respectively, H represents that rain was observed and the satellite product detected a rain event, M represents that the satellite rain product did not detect a rain event, F represents that the satellite rain product detected a rain event, but was not observed, Z represents that the satellite rain product was not detected and the site did not observe a rain event, where a rain event refers to rain event that exceeded a threshold.
The beneficial effects of the invention are:
the invention provides a multi-mode deep learning-based multi-source remote sensing rainfall data fusion method which can effectively capture multi-mode information characteristics of different remote sensing rainfall data, including data characteristics of an image grid space mode, a data matrix mode and a time sequence mode, and better feature expression is learned by utilizing complementarity among the multi-modes and eliminating redundancy among the modes. Meanwhile, the method is based on various deep learning algorithms, has stronger generalization capability, further improves the precision of rainfall data obtained after multi-source rainfall data fusion, and has important significance for early warning of drought and flood disaster forecast and water resource development and reasonable utilization in China.
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Drawings
The invention is further illustrated by the following figures and examples.
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a technical roadmap for an embodiment of the method of the present invention;
FIG. 3 is a schematic diagram of multi-modal sample data extraction in an embodiment of the invention;
FIG. 4 is a schematic diagram of multi-source remote sensing rainfall data fusion in the embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings and tables in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
In the embodiment, a certain watershed is taken as a specific case area, and high-precision daily-scale rainfall data of the watershed is taken as a research target, so that specific application of the method provided by the invention is further explained.
The embodiment discloses a multi-source remote sensing rainfall data fusion method based on multi-modal deep learning, which comprises the following steps (as shown in figure 1):
step one, collecting research area data: collecting basin multi-source remote sensing rainfall data and ground meteorological station actual rainfall data; as shown in fig. 2, the multi-source remote sensing rainfall data comprises GPM remote sensing rainfall data, CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data; the rainfall data of the ground meteorological station comprises GPM remote sensing rainfall data which are GPM IMERG Final Run V06B Level 3 data products, time resolution is 0.5 hour and day respectively, and spatial resolution is 0.1 degree; CMORPH satellite remote sensing rainfall data is a CMORPH Version 1.0 rainfall data product, the time resolution is a daily scale, and the spatial resolution is 0.25 degrees; the remote sensing rainfall data of the PERSIANN satellite is a PERSIANN _ CDR Version 1.0 rainfall data product, the time resolution is a daily scale, and the spatial resolution is 0.25 degrees; the TRMM satellite remote sensing rainfall data is a Level 3TRMM3B42V7 data product, and the time resolution is 0.25 degrees.
Step two, data preprocessing: the method comprises the steps of spatial resampling, multi-modal sample data extraction and spatial relationship matching; the spatial resampling is to process CMORPH satellite remote sensing rainfall data with spatial resolution of 0.25 degrees, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data into rainfall data with spatial resolution of 0.1 degrees by adopting a bilinear interpolation method; as shown in fig. 2, the multi-modal sample data extraction is to extract image raster space mode (mode 1) sample data, data matrix mode (mode 2) sample data and time series mode (mode 3) sample data from the multi-source remote sensing rainfall data; and the spatial relationship matching is to match the extracted multimodal sample data with rainfall data of the ground meteorological station through geographical point location coordinates to obtain a multimodal sample data set.
As shown in fig. 3, a specific way to extract the sample data of the image grid spatial mode is as follows: and extracting sub-grid data taking the meteorological site as the center for each meteorological site by using GPM (general purpose computer) remote sensing rainfall data with time resolution as a daily scale, wherein the spatial resolution of each sub-grid is 0.3 degrees multiplied by 0.3 degrees.
The specific way of extracting the sample data of the matrix mode is as follows: combining CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data after spatial resampling, taking a grid central point as a reference, selecting a satellite grid numerical value closest to the longitude and latitude where a meteorological site is located as satellite product rainfall data corresponding to the site, and combining three different satellite product rainfall data corresponding to each meteorological site to form 1 x 3 matrix data.
The specific mode for extracting the time series modal sample data is as follows: based on GPM remote sensing rainfall data with the time resolution of 0.5 hour, sample data of a satellite grid daily rainfall time sequence closest to the longitude and latitude where the meteorological site is located are extracted to form a 48 multiplied by 1 time sequence.
Step three, constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method, as shown in fig. 4: learning image grid mode sample data characteristics by using a convolutional neural network model (CNN), learning matrix mode sample data characteristics by using a multilayer perceptron (MLP) and learning time sequence mode sample data characteristics by using a long-short term memory model (LSTM);
specifically, the convolutional neural network model comprises a convolutional layer, a pooling layer and a Flatten layer, the specific process of learning the image grid space modal sample data features by using the convolutional neural network model is that after modal feature extraction is performed on the convolutional layer, an extracted feature map is transmitted to the pooling layer for feature selection, and finally the Flatten layer converts high-dimensional feature vectors into one-dimensional feature vectors, and the specific formula is as follows:
in the formula (1)
A value representing the jth feature map of the ith layer at the (x, y) position; reLU is the activation function of each layer; b ij Deviation of jth characteristic diagram of ith layer; m is an index of the feature map of layer i-1;
is in position (x, y)The convolution kernel of (a), the convolution kernel connects the mth feature map of the i-1 st layer and the jth feature map of the ith layer; p is i ,Q i Height and width of the convolution kernel, respectively; p and q represent the convolution step size respectively.
The specific formula for learning matrix modal sample data characteristics by using the multilayer perceptron is as follows:
featurevetor in formula (2) t Characteristic vectors in the t-th-day matrix modal sample data;
connecting weights between the kth neuron from the l-1 layer and the z-th neuron from the l layer;
bias for the z-th neuron of layer i; x is a radical of a fluorine atom Z And L is the number of layers of the neural network, which is the z-th input variable.
The specific process of learning time series modal sample data characteristics by utilizing a long-short term memory model (LSTM) is that time series samples of a time series modal are input into a long-short term memory model memory neural unit with a plurality of hidden layers for cyclic connection, then a full connection layer is connected, and characteristic values are output, wherein the specific formula of the long-short term memory model memory neural unit is as follows:
in formula (3) i t Is an input gate; f. of t To forget the door; o. o t Is an output gate; sigma is a sigmoidal function; x is a radical of a fluorine atom t Inputting a variable for the time step t; u. u i ,u f ,u c ,u o ,v i ,v f ,v c ,v o Is the weight of the neural network; k is a radical of i ,k f ,k c ,k o A neural network deviation vector is obtained; c. C t ,c t-1 The states of the memory units are respectively the current time step t and the last time step t-1; h is t ,h t-1 Hidden states of the current time step t and the last time step t-1 are respectively;
the specific formula for connecting a full connection layer and outputting a characteristic value is as follows:
featurevector t ＝f(h t ,h t-1 ,h t-2 …) (4)
featurevector in formula (4) t And the characteristic vector is the characteristic of the data of the time series modal sample at the t day. f is the fully-connected layer function, h t ,h t-1 ,h t-2 The hidden states of the long-short term memory network model at t, t-1 and t-2 are respectively.
The decision fusion of this embodiment specifically includes splicing a plurality of feature vectors extracted from an image grid modality, a matrix modality, and a time series modality by using a concatemate layer, then inputting the spliced feature vectors to a full-connection layer for fusion, and outputting fused rainfall data, where a formalized formula of the decision fusion is:
P i,t ＝f(featuremap(model1),featurevector(model2),featurevector(model3)) (5)
in formula (5), P i,t The rainfall data of the ith station and the t day is obtained; f is a function of a corresponding layer, and featuremap (model 1) is a feature vector extracted from model1 modal data; a featurevaluever (model 2) is a feature vector extracted by model2 modal data; a feature vector extracted by a featurever (model 3) for model3 modal data; model1 is an image grid spatial mode; model2 is a data matrix modality; model3 is a time series modality.
Step four, evaluating the fusion effect of the multi-source rainfall data: and fusing the multisource satellite remote sensing rainfall data of the research area by using the trained model, acquiring the rainfall data fused in the research area, and performing precision evaluation on the fused rainfall data through the evaluation index.
In this embodiment, the correlation coefficient R, the mean absolute error MAE, and the root mean square error RMSE are used as evaluation indexes to evaluate the precision of the fused rainfall data obtained by the method of the present invention, and a calculation formula of statistical indexes is selected as follows:
the specific formula of the pearson correlation coefficient R is:
the specific formula of the mean absolute error MAE is:
the specific formula of the root mean square error RMSE is:
the accuracy evaluation result of the watershed scale multi-source rainfall data fusion method in the embodiment is shown in table 1. From table 1, the correlation coefficient between the fused rainfall data obtained by the multi-source remote sensing rainfall data fusion method and the rainfall data measured at the meteorological site in the drainage basin is higher than that of a single remote sensing rainfall product and can reach 0.78, and the average absolute error and the root mean square error are lower than those of a single original remote sensing rainfall product.
Table 1 embodiment multi-source remote sensing rainfall data fusion result precision evaluation of certain watershed
Finally, it should be noted that the above-mentioned is only for illustrating the technical solution of the present invention and not for limiting, although the present invention has been described in detail with reference to the preferred arrangement, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (10)
1. A multi-source remote sensing rainfall data fusion method based on multi-modal deep learning is characterized by comprising the following steps:
step one, collecting research area data: collecting multi-source remote sensing rainfall data and actually measured rainfall data of a ground meteorological station; the multi-source remote sensing rainfall data comprises GPM remote sensing rainfall data, CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and TRMM satellite remote sensing rainfall data; selecting actual rainfall data of a daily scale from the actual rainfall data of the ground meteorological station;
step two, data preprocessing: data space resampling, multi-mode sample information extraction and data space relation matching are included; the data space resampling is to process the remote sensing rainfall data, so that the spatial resolution of different remote sensing rainfall data is kept consistent; the multi-modal sample information extraction is to extract image grid space modal sample data, data matrix modal sample data and time sequence modal sample data from remote sensing rainfall data; the spatial relationship matching is to match the extracted multimodal sample information with rainfall data of the ground meteorological station through geographic coordinates to generate a multimodal sample data set;
step three, constructing a multi-source remote sensing rainfall data fusion model based on a multi-modal deep learning method: the method comprises the steps of multi-mode sample data feature learning and multi-source rainfall data decision fusion;
the multi-mode sample data feature learning refers to learning sample data features of different modes for the multi-mode sample data set in the step two based on different deep learning methods, and comprises the following steps: learning image grid space modal sample data characteristics by using a convolutional neural network model, learning mathematical matrix modal sample data characteristics by using a multilayer perceptron, and learning time sequence modal sample data characteristics by using a long-short term memory network model;
the multi-source rainfall data decision fusion means that multi-modal sample data features obtained through multi-modal sample data feature learning are fused by utilizing a multi-layer fully-connected neural network;
step four, evaluating the fusion effect of the multi-source rainfall data: and performing precision evaluation on the fused rainfall data by using the precision evaluation index.
2. The multi-source remote sensing rainfall data fusion method based on the multi-modal deep learning according to claim 1, wherein the data space resampling in the step two is to resample the remote sensing rainfall data by using a bilinear interpolation method.
3. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, wherein the extracting of the image grid space modal sample data in the second step is specifically: and extracting a raster data set taking a meteorological site as a center based on the daily scale GPM remote sensing rainfall data.
4. The multi-source remote sensing rainfall data fusion method based on the multi-modal deep learning, which is characterized in that the extracted data matrix modal sample data in the second step is CMORPH satellite remote sensing rainfall data, PERSIANN satellite remote sensing rainfall data and satellite TRMM remote sensing rainfall data after joint space resampling, a satellite grid numerical value closest to the longitude and latitude where a meteorological site is located is selected as satellite product rainfall data corresponding to the site by taking a grid central point as a reference, and three different satellite product rainfall data corresponding to each meteorological site can form 1 x 3 matrix data.
5. The multi-source remote sensing rainfall data fusion method based on the multi-modal deep learning, as claimed in claim 1, wherein the time series modal sample data extracted in step two is a 48 x 1 time series based on GPM remote sensing rainfall data with a time resolution of 0.5 hour, and the daily rainfall time series sample data of a satellite grid closest to the latitude and longitude of a meteorological site is extracted.
6. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, characterized in that the convolutional neural network model in step three comprises a convolutional layer, a pooling layer and a scatter layer, and the specific process of learning image grid space modal sample data features by using the convolutional neural network model is that after modal feature extraction is performed on the convolutional layer, extracted feature image features are transferred to the pooling layer for feature selection, and finally, high-dimensional feature vectors are converted into one-dimensional feature vectors on the scatter layer, and the specific formula is as follows:
in the formula (I), the compound is shown in the specification,
a value representing the jth feature map of the ith layer at the (x, y) position; reLU is the activation function of each layer; b ij Deviation of jth characteristic diagram of ith layer; m is an index of the feature map of layer i-1;
is the convolution kernel at position (x, y) connecting the mth feature map of the i-1 th layer and the jth feature map of the ith layer; p i ,Q i Height and width of the convolution kernel, respectively; p and q represent the convolution step size respectively.
7. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, wherein the specific formula for learning data matrix modal sample data characteristics by using a multi-layered perceptron in step three is as follows:
in the formula, featurevotor t Characteristic vectors in the data matrix modal sample data of the t day are obtained;
is the weight of the connection between the kth neuron from layer l-1 and the z-th neuron from layer l;
bias for the z-th neuron of layer l; x is the number of Z And L is the number of layers of the neural network, which is the z-th input variable.
8. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, wherein the specific process of learning time series modal sample data characteristics by using the long and short term memory network model in step three is to input time series samples of a time series modal into a memory neural unit of the long and short term memory network model including a plurality of hidden layers for cyclic connection, and then connect to a full connection layer and output characteristic values, wherein the specific formula of the memory neural unit of the long and short term memory network model is as follows:
in the formula i t Is an input gate; f. of t To forget the door; o. o t Is an output gate; sigma is a sigmoidal function; x is the number of t Inputting variables for the time step t; u. of i ,u f ,u c ,u o ,v i ,v f ,v c ,v o Is the weight of the neural network; k is a radical of i ,k f ,k c ,k o A neural network deviation vector; c. C t ,c t-1 Are respectively asThe memory cell states at the current time t and the last time t-1; h is t ,h t-1 Hidden states of the current time t and the last time t-1 are respectively;
the specific formula for connecting a full connection layer and outputting a characteristic value is as follows:
featurevector t ＝f(h t ,h t-1 ,h t-2 …) (4)
in the formula, featurevatervector t Is the characteristic vector of the t-th day time series modal sample data characteristic, f is the full-link layer function, h t ,h t-1 ,h t-2 The hidden states of the long-short term memory network model at t, t-1 and t-2 are respectively.
9. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning according to claim 1, characterized in that the multi-source rainfall data fusion in step three specifically comprises a Concatenate layer to splice a plurality of feature vectors extracted from an image grid space mode, a data matrix mode and a time sequence mode, then input to a full connection layer for fusion and output fused rainfall data, and a formalized formula of decision fusion is as follows:
P i,t ＝f(featuremap(model1),featurevector(model2),featurevector(model3)) (5)
in the formula, P i,t The rainfall data of the ith station and the t day is obtained; f is a function of a conditioner layer, and featuremap (model 1) is a feature vector extracted by model1 modal data; a featurevaluever (model 2) is a feature vector extracted by model2 modal data; a featurevaluector (model 3) is a feature vector extracted by model3 modal data; model1 is an image grid spatial modality; model2 is a data matrix modality; model3 is a time series modality.
10. The multi-source remote sensing rainfall data fusion method based on multi-modal deep learning of claim 1, wherein the evaluation index in step four comprises one or more of Pearson correlation coefficient R, root Mean Square Error (RMSE), mean Absolute Error (MAE), detection rate POD, false alarm rate FAR and fair foreboding score (ETS).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210684570.1A CN114970743B (en) | 2022-06-17 | 2022-06-17 | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210684570.1A CN114970743B (en) | 2022-06-17 | 2022-06-17 | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114970743A CN114970743A (en) | 2022-08-30 |
| CN114970743B true CN114970743B (en) | 2022-11-08 |
Family
ID=82962804
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210684570.1A Active CN114970743B (en) | 2022-06-17 | 2022-06-17 | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114970743B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116070792B (en) * | 2023-03-28 | 2023-07-25 | 中国科学院地理科学与资源研究所 | Fusion method, device, storage medium and equipment of multi-source precipitation data |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295714A (en) * | 2016-08-22 | 2017-01-04 | 中国科学院电子学研究所 | A kind of multi-source Remote-sensing Image Fusion based on degree of depth study |
| CN107992904A (en) * | 2017-12-22 | 2018-05-04 | 重庆邮电大学 | Forest Ecology man-machine interaction method based on Multi-source Information Fusion |
| CN108761574A (en) * | 2018-05-07 | 2018-11-06 | 中国电建集团北京勘测设计研究院有限公司 | Rainfall evaluation method based on Multi-source Information Fusion |
| CN111401602A (en) * | 2019-12-31 | 2020-07-10 | 河海大学 | Assimilation method for satellite and ground rainfall measurement values based on neural network |
| CN111665560A (en) * | 2020-04-23 | 2020-09-15 | 中国石油天然气股份有限公司 | Oil-gas reservoir identification method and device, computer equipment and readable storage medium |
| CN111859800A (en) * | 2020-07-15 | 2020-10-30 | 河海大学 | Method for spatio-temporal estimation and prediction of PM2.5 concentration distribution |
| CN112308029A (en) * | 2020-11-24 | 2021-02-02 | 国网湖南省电力有限公司 | Rainfall station and satellite rainfall data fusion method and system |
| CN113657472A (en) * | 2021-08-02 | 2021-11-16 | 中国空间技术研究院 | Multi-source remote sensing data fusion method based on subspace learning |
| CN114418038A (en) * | 2022-03-29 | 2022-04-29 | 北京道达天际科技有限公司 | Space-based information classification method and device based on multi-mode fusion and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111369602B (en) * | 2020-02-25 | 2023-10-27 | 阿波罗智能技术(北京)有限公司 | Point cloud data processing method and device, electronic equipment and readable storage medium |
-
2022
- 2022-06-17 CN CN202210684570.1A patent/CN114970743B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295714A (en) * | 2016-08-22 | 2017-01-04 | 中国科学院电子学研究所 | A kind of multi-source Remote-sensing Image Fusion based on degree of depth study |
| CN107992904A (en) * | 2017-12-22 | 2018-05-04 | 重庆邮电大学 | Forest Ecology man-machine interaction method based on Multi-source Information Fusion |
| CN108761574A (en) * | 2018-05-07 | 2018-11-06 | 中国电建集团北京勘测设计研究院有限公司 | Rainfall evaluation method based on Multi-source Information Fusion |
| CN111401602A (en) * | 2019-12-31 | 2020-07-10 | 河海大学 | Assimilation method for satellite and ground rainfall measurement values based on neural network |
| CN111665560A (en) * | 2020-04-23 | 2020-09-15 | 中国石油天然气股份有限公司 | Oil-gas reservoir identification method and device, computer equipment and readable storage medium |
| CN111859800A (en) * | 2020-07-15 | 2020-10-30 | 河海大学 | Method for spatio-temporal estimation and prediction of PM2.5 concentration distribution |
| CN112308029A (en) * | 2020-11-24 | 2021-02-02 | 国网湖南省电力有限公司 | Rainfall station and satellite rainfall data fusion method and system |
| CN113657472A (en) * | 2021-08-02 | 2021-11-16 | 中国空间技术研究院 | Multi-source remote sensing data fusion method based on subspace learning |
| CN114418038A (en) * | 2022-03-29 | 2022-04-29 | 北京道达天际科技有限公司 | Space-based information classification method and device based on multi-mode fusion and electronic equipment |
Non-Patent Citations (4)
| Title |
|---|
| A novel multi-source information-fusion predictive framework based on deep neural networks for accuracy enhancement in stock market prediction;Isaac Kofi Nti等;《Journal of Big Data》;20210109;第8卷(第1期);1-28 * |
| Artificial intelligence and healthcare: Forecasting of medical bookings through multi-source time-series fusion;FrancescoPiccialli等;《Information Fusion》;20210401;第74卷;1-16 * |
| GIS和RS技术应用于流域水生生物时空分析的研究进展;黄彤辉等;《环境科学研究》;20210827;第34卷(第11期);2674-2685 * |
| 基于深度卷积神经网络的强对流天气预报方法研究;周康辉;《中国博士学位论文全文数据库基础科学辑》;20220215(第(2022)02期);A009-18 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114970743A (en) | 2022-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Feng et al. | Modeling urban growth with GIS based cellular automata and least squares SVM rules: a case study in Qingpu–Songjiang area of Shanghai, China | |
| CN101413806B (en) | Mobile robot grating map creating method of real-time data fusion | |
| Wang et al. | Tropical cyclone intensity estimation from geostationary satellite imagery using deep convolutional neural networks | |
| CN105243435B (en) | A kind of soil moisture content prediction technique based on deep learning cellular Automation Model | |
| CN111639787A (en) | Spatio-temporal data prediction method based on graph convolution network | |
| Lian et al. | A novel data-driven tropical cyclone track prediction model based on CNN and GRU with multi-dimensional feature selection | |
| Kovordányi et al. | Cyclone track forecasting based on satellite images using artificial neural networks | |
| Saxena et al. | A review study of weather forecasting using artificial neural network approach | |
| CN108304536A (en) | A kind of geographical environmental simulation of the geographical environmental element of coupling and predicting platform | |
| Li et al. | An object-based approach for verification of precipitation estimation | |
| CN112507861A (en) | Pedestrian detection method based on multilayer convolution feature fusion | |
| CN114970743B (en) | Multi-source remote sensing rainfall data fusion method based on multi-modal deep learning | |
| CN114936691A (en) | Temperature forecasting method integrating relevance weighting and space-time attention | |
| Faruq et al. | Deep Learning-Based Forecast and Warning of Floods in Klang River, Malaysia. | |
| CN116720156A (en) | Weather element forecasting method based on graph neural network multi-mode weather data fusion | |
| Wong et al. | Automatic tropical cyclone eye fix using genetic algorithm | |
| CN117314006A (en) | Intelligent data analysis method and system | |
| Wang et al. | Relative importance of radar variables for nowcasting heavy rainfall: A machine learning approach | |
| Wu et al. | A novel bayesian additive regression trees ensemble model based on linear regression and nonlinear regression for torrential rain forecasting | |
| Suparta et al. | Modeling of precipitable water vapor using an adaptive neuro-fuzzy inference system in the absence of the GPS network | |
| CN116884222B (en) | Short-time traffic flow prediction method for bayonet nodes | |
| Al Kindhi et al. | Sensor and internet of things based integrated inundation mitigation for smart city. | |
| Escalante-Sandoval et al. | Regional monthly runoff forecast in southern Canada using ANN, K-means, and L-moments techniques | |
| CN113011657A (en) | Method and device for predicting typhoon water level, electronic equipment and storage medium | |
| Kesavavarthini et al. | Bias correction of CMIP6 simulations of precipitation over Indian monsoon core region using deep learning algorithms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |