CN116484189A - ERA5 precipitation product downscaling method based on deep learning - Google Patents
ERA5 precipitation product downscaling method based on deep learning Download PDFInfo
- Publication number
- CN116484189A CN116484189A CN202310454771.7A CN202310454771A CN116484189A CN 116484189 A CN116484189 A CN 116484189A CN 202310454771 A CN202310454771 A CN 202310454771A CN 116484189 A CN116484189 A CN 116484189A
- Authority
- CN
- China
- Prior art keywords
- data
- precipitation
- era5
- downscaling
- branch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001556 precipitation Methods 0.000 title claims abstract description 108
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000013135 deep learning Methods 0.000 title claims abstract description 27
- 230000006870 function Effects 0.000 claims abstract description 22
- 238000004458 analytical method Methods 0.000 claims abstract description 21
- 238000012549 training Methods 0.000 claims abstract description 15
- 238000012368 scale-down model Methods 0.000 claims abstract description 10
- 238000011160 research Methods 0.000 claims abstract description 8
- 238000005520 cutting process Methods 0.000 claims abstract description 6
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 238000011176 pooling Methods 0.000 claims description 17
- 230000004927 fusion Effects 0.000 claims description 16
- 238000010586 diagram Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000010606 normalization Methods 0.000 claims description 7
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 6
- 238000013527 convolutional neural network Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000012360 testing method Methods 0.000 claims description 5
- 239000013598 vector Substances 0.000 claims description 4
- 238000002790 cross-validation Methods 0.000 claims description 3
- 230000009191 jumping Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000002759 z-score normalization Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 description 5
- 238000004088 simulation Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000013213 extrapolation Methods 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01W—METEOROLOGY
- G01W1/00—Meteorology
- G01W1/10—Devices for predicting weather conditions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/761—Proximity, similarity or dissimilarity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Environmental & Geological Engineering (AREA)
- Medical Informatics (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Atmospheric Sciences (AREA)
- Biodiversity & Conservation Biology (AREA)
- Ecology (AREA)
- Environmental Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses an ERA5 precipitation product downscaling method based on deep learning, which comprises the steps of firstly obtaining ERA5 re-analysis data, GPM satellite data and ground station daily precipitation observation data, and preprocessing; based on the Pearson correlation coefficient method, measuring the correlation between the meteorological factors and ground observation precipitation, and selecting the meteorological factors with larger correlation as effective characteristics; cutting each data to a research area and normalizing; building a precipitation scale-down model, and designing a loss function and model parameters; training to obtain an optimal model, and evaluating a downscaling result; adopting a double-branch structure, taking GPM satellite data as a tag, taking low-resolution precipitation as input by one branch, and taking effective meteorological features as input by the other branch to learn auxiliary guidance; the trained model captures the data change in time and space, well captures the extreme situation, and can effectively reduce ERA5 precipitation data from 25km to 10km.
Description
Technical Field
The invention relates to the technical field of downscaling, in particular to an ERA5 precipitation product downscaling method based on deep learning.
Background
Precipitation estimation is the basis of weather forecast, the purpose of which is to accurately and timely predict the intensity of precipitation in a local area, which has been an important problem, and typical precipitation estimation methods rely on numerical weather forecast and radar echo extrapolation. In recent years, with the development of artificial intelligence technology, methods have also emerged that focus on learning potential precipitation representations under a machine learning framework. However, these methods mainly focus on precipitation estimation in local areas, and the spatial resolution is relatively coarse, so that the requirements of practical applications cannot be met, and the spatial-temporal distribution of regional precipitation cannot be accurately reflected. The high-resolution precipitation data is helpful for describing the space-time diversity characteristic of precipitation, and has important effect on accurate simulation of the hydrologic, meteorological and ecological processes of a current domain. Therefore, the spatial downscaling is carried out on the low-resolution precipitation product, and the resolution is very necessary to be improved.
The key idea of precipitation downscaling is to train a neural network using a combination of low-resolution and high-resolution simulations, mapping from the former to the latter. At present, a dynamic downscaling and statistical downscaling method is generally adopted, the dynamic downscaling is mainly carried out according to a physical environment generated by precipitation, and regional climate modes are embedded into a global climate mode to generate regional scale precipitation data with high resolution, but the method requires a large amount of calculation resources and has errors and uncertainty due to the influence of a plurality of factors; the statistical downscaling method realizes downscaling by establishing the relation among products with different resolutions, and the common method is to interpolate the low-resolution precipitation products onto the high-resolution grids firstly, then, the interpolation result is corrected point by combining with the historical high-resolution grid point information, and the common correction method comprises a linear function, a support vector machine and an artificial neural network. However, the conventional statistical downscaling method cannot generally learn such a mapping relationship for those areas without high-resolution observation data, and is often poor for extreme events.
Disclosure of Invention
In order to solve the technical problems, the invention provides an ERA5 precipitation product downscaling method based on deep learning, which comprises the following steps of
S1, acquiring ERA5 analysis meteorological data, GPM satellite rainfall data and ground rainfall station observation data;
s2, preprocessing data;
s3, calculating the correlation between the meteorological data and the ground rainfall station observation data by using a Pearson correlation coefficient method, and selecting meteorological factors influencing precipitation as effective characteristics;
s4, cutting each data to a research area range to obtain cut data, and carrying out normalization processing on the cut data to construct a training data set;
s5, constructing a precipitation scale-down model based on deep learning, wherein the model comprises a first branch, a second branch and a residual error attention network, the first branch takes original ERA5 precipitation data as input, the second branch takes weather factors influencing precipitation as input, and outputs of the first branch and the second branch are fused and then input into the residual error attention network for precipitation scale-down;
s6, dividing a training set sample and a test set sample, setting each parameter and a loss function of a precipitation scale-down model, and continuously training and adjusting until an optimal parameter combination is obtained;
and S7, obtaining an optimal water-reducing and scale-reducing model, and carrying out precision analysis on a scale-reducing result by using ground rainfall station observation data.
The technical scheme of the invention is as follows:
further, in step S2, the data preprocessing includes the steps of
S2.1, detecting a missing value by using an isnull () function in a pythonpandas library;
s2.2, detecting abnormal values by using a box graph method;
s2.3, replacing by using the missing value and the average value of the area before and after the position of the abnormal value or the adjacent area according to the similarity in space and time.
In the foregoing method for reducing the scale of ERA5 precipitation products based on deep learning, in step S3, weather factors influencing precipitation are selected as effective features, the pearson correlation coefficient method is utilized to calculate the correlation between each weather factor and the ground rainfall station observation data, a set threshold is preset, weather factors with correlation larger than the set threshold are used as weather factors influencing precipitation, and a pearson correlation coefficient formula is as follows:
wherein Cov (X, Y) is the covariance of the eigenvalues X, Y,and->The variances of features X, Y are shown, respectively.
In the aforementioned ERA5 precipitation product downscaling method based on deep learning, in step S4, cutting each data to the range of the research area comprises the following steps
S4.1.1, uniformly converting ERA5 analysis meteorological data and GPM satellite precipitation data into Beijing time;
s4.1.2, aligning the site coordinates with the grid data according to longitude and latitude coordinates of each ground observation time;
s4.1.3 ERA5 analysis meteorological data and GPM satellite data are segmented into study areas and spread around.
In the aforementioned method for reducing the scale of ERA5 precipitation product based on deep learning, in step S4, the normalization processing of the segmentation data comprises the following steps of
S4.2.1, carrying out Min-Max standardization operation on the segmentation data once, wherein the calculation formula is as follows:
wherein y is i Representing normalized data, x i Representing raw data, min 1≤j≤n {x j The minimum value, max, in the data 1≤j≤n {x j The maximum value in the data is indicated,representing the normalized data;
s4.2.2, performing a Z-SCORE normalization method on the normalized features, normalizing the data features to the same dimension, and adopting a calculation formula:
wherein u is i Is the overall mean value, sigma, of the ith element feature i Is the total standard deviation of the ith element feature.
In the foregoing method for reducing the size of the ERA5 precipitation product based on deep learning, in step S5, a first branch adopts a convolutional neural network to extract the spatial characteristics of original ERA5 precipitation data, and the first branch comprises 3 convolutional layers and a maximum pooling layer; the second branch adopts a dense network structure, and comprises 1 convolution layer and 3 dense modules;
the dense module comprises 3 convolution layers and 3 ReLU functions, wherein one ReLU function is connected behind each convolution layer, and the dense module inputs the input characteristics of each convolution layer to all subsequent convolution layers through jump connection.
In the foregoing method for reducing the size of ERA5 precipitation products based on deep learning, in step S5, in the residual attention network, the input sequentially passes through a convolution layer, a residual feature fusion module, a spatial and channel attention module, an up-sampling layer and a convolution layer;
the residual feature fusion module comprises 3 residual group modules, the tail output of each residual group module is connected in a jumping manner, elements and operations are carried out at the tail of the whole residual feature fusion module, and the residual group module comprises 2 convolution layers 3*3;
the space and channel attention module comprises a space attention module and a channel attention module, wherein in the space attention module, the input is subjected to parallel maximum pooling layer and average pooling layer, the pooled vectors are sent to a full-connection layer for calculation and then added, and the channel attention feature is obtained by multiplying the initial fusion feature after the Sigmoid activation function; and in the channel attention module, carrying out maximum pooling and average pooling according to space, splicing channels, carrying out convolution operation, and multiplying the channel attention module with the channel attention force diagram through a Sigmoid activation function to obtain a final characteristic diagram.
In the foregoing method for reducing the size of ERA5 precipitation products based on deep learning, in step S6, a 10-fold cross validation method is adopted to divide the training set and the testing set.
In the foregoing method for reducing the size of the ERA5 precipitation product based on deep learning, in step S6, each parameter of the precipitation size-reducing model includes an adjustable learning rate, a momentum parameter, a total iteration number and a weight attenuation parameter;
an initial value of the adjustable learning rate is set to 10 -4 Through 2X 10 5 The learning rate is halved after the reverse iteration, and the weight attenuation parameter is set to 10 -7 The momentum parameter is set to be 0.5, and the precipitation scale-down model is trained by using an Adam optimization method, wherein the first moment attenuation coefficient beta 1 =0.9, second moment attenuation coefficient β 2 Epsilon parameter epsilon=10 in adam optimization method=0.99 -8 The method comprises the steps of carrying out a first treatment on the surface of the The loss function of the water-reducing scale-reducing model is set to be L 1 The loss is given by:
wherein n represents the number of samples, E i Representing corresponding pixel values of ERA5 precipitation data after downscaling, G i And representing the pixel value corresponding to the GPM satellite precipitation data.
In the foregoing method for downscaling the ERA5 precipitation product based on deep learning, in step S7, accuracy analysis is performed on the downscaling result through the correlation coefficient, the deviation and the root mean square error, and the formula is as follows
Wherein X is i And Y i Respectively representing the high-resolution rainfall data obtained after the downscaling and the ground rainfall station observation data,and->And respectively representing the average value of the high-resolution precipitation data obtained after the downscaling and the average value of the ground rainfall station observation data.
The beneficial effects of the invention are as follows:
(1) According to the invention, the element correlation and the space correlation in the precipitation process are fully considered, a precipitation scale model is constructed by selecting meteorological factors with potential influence on precipitation and adopting a double-branch network structure, GPM high-resolution precipitation data is used as a label, and the optimal network parameters are obtained by continuously training and optimizing the model, so that the optimal ERA5 precipitation product scale model based on deep learning is obtained;
(2) In the invention, a double-branch network structure is used, one branch extracts the spatial precipitation characteristics of original coarse resolution precipitation through a Convolutional Neural Network (CNN), in order to extract strong auxiliary guidance of precipitation downscaling so as to obtain a more accurate rainfall characteristic diagram, the other branch constructs a dense network based on the CNN, weather factors related to precipitation are taken as input, all auxiliary weather elements share a common structure with shared parameters, and the strategy enables the downscaling network to learn more specific characteristics with weather correlation;
(3) In the invention, the characteristics of two branches in a double-branch network structure are fused and input into a residual attention network to finish precipitation and downscaling, the residual attention network fuses a residual characteristic fusion mechanism, a space and a channel attention mechanism, the network fuses the training processed characteristics in each residual block through jump connection to form a group of new characteristics, the precipitation characteristics of the group contain more abundant and complete information, and the characteristics can be allocated with weights by the space and the channel attention module without missing the characteristics to a great extent; the result shows that the downscaling result of ERA5 has a finer spatial structure than ERA5, and the precipitation spatial pattern revealed by high-resolution simulation can be reproduced, and the precipitation spatial characteristics are greatly improved; the downscaled ERA5 retains the temporal characteristics of the original ERA5, which is more consistent with ground rainfall station data than high resolution simulation.
Drawings
FIG. 1 is a schematic overall flow chart of the present invention;
FIG. 2 is a schematic diagram of a study area in an embodiment of the invention;
FIG. 3 is a schematic structural diagram of a deep learning-based precipitation downscaling model in the invention;
FIG. 4 is a schematic view of the dense module structure of the present invention;
FIG. 5 is a schematic diagram of a residual attention network according to the present invention;
FIG. 6 is a schematic diagram of a residual block module according to the present invention;
FIG. 7 is a schematic diagram of a space and channel attention module according to the present invention.
Detailed Description
The embodiment provides an ERA5 precipitation product downscaling method based on deep learning, as shown in fig. 1, comprising the following steps of
S1, acquiring ERA5 analysis meteorological data, GPM satellite rainfall data and ground rainfall station observation data;
in this embodiment, the study area is southeast China, as shown in fig. 2, the data set mainly selects ERA5 to analyze meteorological data for precipitation downscaling, ERA5 is a fifth generation atmospheric analysis data set of global climate by ECMWF, and provides estimated values of a large number of atmospheric, terrestrial and marine climate variables per hour, the spatial resolution is 25km, and the time resolution is 1h.
And a comprehensive multi-satellite retrieval GPM_IMERG (Final) daily scale data set is selected as a label, the data source has been widely proven to be more accurate in the disclosed satellite precipitation data set, the spatial resolution is 10km, and the time resolution is 24 hours.
S2, re-analyzing meteorological data, GPM satellite rainfall data and ground rainfall station observation data according to the acquired ERA5, and preprocessing before inputting a neural network model, wherein the specific flow is as follows: firstly, detecting a missing value by using isnull () function in the pythonpandas library, detecting an abnormal value by using a box graph method, and replacing by using the average value of the area before and after the position of the value or the adjacent area according to the similarity in the space and time dimensions.
S3, calculating the correlation between ERA5 re-analysis meteorological data and ground rainfall station observation data by using a Pearson correlation coefficient method, presetting a set threshold value to be 0.2, and recognizing all the features larger than the set threshold value as effective features, namely weather factors influencing precipitation (ERA 5 re-analysis meteorological data comprise precipitation, wind, temperature and other meteorological factors in ERA5 re-analysis meteorological data except precipitation), so as to remove any unnecessary noise input which can negatively influence the performance, wherein the temperature, the relative humidity and the wind are calculated as the weather factors influencing the precipitation, and the Pearson correlation coefficient formula is as follows:
wherein Cov (X, Y) is the covariance of the eigenvalues X, Y,and->The variances of features X, Y are shown, respectively.
S4, cutting the data to a research area range, wherein the range contains the context information of the spatial characteristics of the research area, so that the spatial correlation can be improved, and the spatial information distribution can be better extracted; cutting ERA5 analysis meteorological data and GPM satellite data into a research area and diffusing the data to the periphery to consider the influence of the surrounding environment on precipitation of a target area; accumulating collected ERA5 re-analysis meteorological data into daily precipitation scales, keeping the time resolution of the data consistent, and converting ERA5 re-analysis meteorological data and GPM satellite data into Beijing time after 8 hours;
considering that the dimensions of different meteorological features in GPM satellite precipitation data and ERA5 analysis meteorological data are different, the network training may be affected, normalization processing is needed for the data, and in order to avoid introducing larger outliers, min-Max normalization operation is firstly carried out on the segmentation data, wherein the calculation formula is as follows:
wherein y is i Representing normalized data, x i Representing raw data, min }≤j≤n {x j The minimum value, max, in the data 1≤j≤n {x j The maximum value in the data is indicated,representing the normalized data;
and then, performing a Z-SCORE normalization method on the normalized characteristics, wherein the calculation formula is as follows:
normalization of data features to the sameIn one dimension, the difference in feature scale is eliminated in this way, where u i Is the overall mean value, sigma, of the ith element feature i Is the total standard deviation of the ith element feature.
S5, constructing a precipitation scale-down model based on a deep learning pyrach framework, wherein the model structure is shown in a figure 3, the model structure is divided into two branches, the first branch is used for extracting the spatial characteristics of original coarse resolution ERA5 precipitation data based on a convolutional neural network, and the branch consists of a convolutional layer conv1 of 5*5, a convolutional layer conv2 of 3*3, a 2 x 2 convolutional layer and a 2 x 2 maximum pooling layer;
the other branch adopts a dense network structure, takes weather factors (relative humidity, temperature and wind) which are selected in the step S3 and have larger correlation with precipitation as input to learn auxiliary guidance so as to accurately downscale, so that the downscale network can learn more specific characteristics with weather correlation, and the branch mainly comprises a convolution layer of 5*5 and three dense modules.
The structure of the dense modules is shown in fig. 4, each dense module is composed of three convolution layers and three ReLU functions, one convolution layer and one ReLU function are sequentially connected into a group, the dense network inputs the input features of each convolution layer to all subsequent convolution layers through jump connection, the calculation complexity of the overall convolution neural network parameters can be reduced, and the information of meteorological factor feature loss caused by building a deep convolution neural network model can be compensated.
The output characteristics of the two branches are fused and then used as the input of a residual attention network, as shown in fig. 5, the residual attention network comprises a convolution layer, a residual characteristic fusion module (RFAM), a space and channel attention module (CBAM), an up-sampling layer and a convolution layer, the characteristics of input data are fully extracted, and the residual cascade structure can construct a very deep characteristic mapping network for improving the performance of super-resolution so as to finish precipitation downscaling.
The residual attention network can be divided into three stages, namely a shallow feature extraction stage, a deep feature mapping stage and a reconstruction stage, wherein in the shallow feature extraction stage, only one convolution layer exists, the size of a convolution kernel is 3*3, and the obtained shallow features serve as the input of deep features.
The deep feature mapping module mainly comprises a residual feature fusion module and a space and channel attention module, wherein the residual feature fusion module mainly comprises 3 residual group modules (RB), as shown in fig. 6, the residual group modules are composed of two 3*3 convolution layers, and for integrating the features and distributing weights, the tail output of each residual group module is connected in a jumping manner, elements and operations are carried out at the tail of the whole residual feature fusion module, and the obtained precipitation features have the fusion of low-frequency information and high-frequency information.
The residual feature fusion module is further added with a space and channel dual attention module, the trained rainfall features are screened out to obtain features with higher layers, the results of the space and channel attention module are shown in fig. 7, firstly, the channel attention module is used for transmitting the vector after the pooling into a full connection layer for calculation and adding, and the channel attention feature is obtained by multiplying the vector after the pooling with the initial fusion feature after the Sigmoid activation function through a channel attention module and a parallel maximum pooling layer and an average pooling layer;
then, through a space attention module, carrying out maximum pooling and average pooling according to space, carrying out convolution operation after channel splicing, and multiplying the channel attention diagram by a Sigmoid activation function to obtain a final feature diagram;
and finally, obtaining high-resolution precipitation after upsampling and convolution of the reconstruction part.
The model comprises a first branch, a second branch and a residual error attention network, wherein the first branch takes original ERA5 precipitation data as input, the second branch takes meteorological factors influencing precipitation as input, and outputs of the first branch and the second branch are fused and then input into the residual error attention network for precipitation scale reduction;
s6, dividing a training set sample and a test set sample by adopting a 10-fold cross validation method, setting each parameter and a loss function of a precipitation scale model, updating the model parameters according to a counter-propagation algorithm, and continuously training until the loss error reaches the minimum; the parameters of the precipitation scale model comprise adjustable learning rate, momentum parameters, total iteration times and weight attenuation parameters;
through repeated experiment training models, specific parameters of the models are set as follows: the initial value of the adjustable learning rate is set to be 10 < -4 >, the learning rate is halved after 2X 105 reverse iterations, the weight attenuation parameter is set to be 10 < -7 >, the momentum parameter is set to be 0.5, and the precipitation scale-down model is trained by using an Adam optimization method, wherein the first moment attenuation coefficient beta 1 =0.9, second moment attenuation coefficient β 2 Epsilon parameter epsilon=10 in adam optimization method=0.99 -8 The method comprises the steps of carrying out a first treatment on the surface of the The loss function of the water-reducing scale-reducing model is set to be L 1 The loss is given by:
wherein n represents the number of samples, E i Representing corresponding pel values of the downscaled ERA5 precipitation data (high resolution), G i And representing the pixel value corresponding to the GPM satellite precipitation data.
S7, obtaining an optimal water-reducing and scale-reducing model, and carrying out precision evaluation and analysis on a scale-reducing result based on ground rainfall station observation data to illustrate the superiority of the scale-reducing method, wherein the precision evaluation and analysis indexes mainly adopt a correlation coefficient (R), a Root Mean Square Error (RMSE) and a deviation (BIAS), and the formulas of the correlation coefficient, the deviation and the root mean square error are as follows
Wherein X is i And Y i Respectively representing the high-resolution rainfall data obtained after the downscaling and the ground rainfall station observation data,and->And respectively representing the average value of the high-resolution precipitation data obtained after the downscaling and the average value of the ground rainfall station observation data.
In addition to the embodiments described above, other embodiments of the invention are possible. All technical schemes formed by equivalent substitution or equivalent transformation fall within the protection scope of the invention.
Claims (10)
1. An ERA5 precipitation product downscaling method based on deep learning is characterized by comprising the following steps of: comprises the following steps
S1, acquiring ERA5 analysis meteorological data, GPM satellite rainfall data and ground rainfall station observation data;
s2, preprocessing data;
s3, calculating the correlation between the meteorological data and the ground rainfall station observation data by using a Pearson correlation coefficient method, and selecting meteorological factors influencing precipitation as effective characteristics;
s4, cutting each data to a research area range to obtain cut data, and carrying out normalization processing on the cut data to construct a training data set;
s5, constructing a precipitation scale-down model based on deep learning, wherein the model comprises a first branch, a second branch and a residual error attention network, the first branch takes original ERA5 precipitation data as input, the second branch takes weather factors influencing precipitation as input, and outputs of the first branch and the second branch are fused and then input into the residual error attention network for precipitation scale-down;
s6, dividing a training set sample and a test set sample, setting each parameter and a loss function of a precipitation scale-down model, and continuously training and adjusting until an optimal parameter combination is obtained;
and S7, obtaining an optimal water-reducing and scale-reducing model, and carrying out precision analysis on a scale-reducing result by using ground rainfall station observation data.
2. The ERA5 precipitation product downscaling method based on deep learning of claim 1, wherein: in the step S2, the data preprocessing includes the following steps
S2.1, detecting a missing value by using an isnull () function in a pythonpandas library;
s2.2, detecting abnormal values by using a box graph method;
s2.3, replacing by using the missing value and the average value of the area before and after the position of the abnormal value or the adjacent area according to the similarity in space and time.
3. The ERA5 precipitation product downscaling method based on deep learning of claim 1, wherein: in the step S3, weather factors affecting precipitation are selected as effective features, the pearson correlation coefficient method is used to calculate the correlation between each weather factor and the ground rainfall station observation data, a set threshold is preset, weather factors with correlation larger than the set threshold are used as weather factors affecting precipitation, and the pearson correlation coefficient formula is:
wherein CoV (X, Y) is the covariance of the eigenvalues X, Y,and->The variances of features X, Y are shown, respectively.
4. The ERA5 precipitation product downscaling method based on deep learning of claim 1, wherein: in the step S4, clipping the data to the range of the research area comprises the following steps
S4.1.1, uniformly converting ERA5 analysis meteorological data and GPM satellite precipitation data into Beijing time;
s4.1.2, aligning the site coordinates with the grid data according to longitude and latitude coordinates of each ground observation time;
s4.1.3 ERA5 analysis meteorological data and GPM satellite data are segmented into study areas and spread around.
5. The ERA5 precipitation product downscaling method based on deep learning of claim 1, wherein: in the step S4, the normalization processing of the segmentation data includes the following steps
S4.2.1, carrying out Min-Max standardization operation on the segmentation data once, wherein the calculation formula is as follows:
wherein y is i Representing normalized data, x i Representing raw data, min 1≤j≤n {x j The minimum value, max, in the data 1≤j≤n {x j The maximum value in the data is indicated,representing the normalized data;
s4.2.2, performing a Z-SCORE normalization method on the normalized features, normalizing the data features to the same dimension, and adopting a calculation formula:
wherein u is i Is the overall mean value, sigma, of the ith element feature i Is the total standard deviation of the ith element feature.
6. The ERA5 precipitation product downscaling method based on deep learning of claim 1, wherein: in the step S5, a first branch adopts a convolutional neural network to extract the spatial characteristics of original ERA5 precipitation data, and comprises 3 convolutional layers and a maximum pooling layer; the second branch adopts a dense network structure, and comprises 1 convolution layer and 3 dense modules;
the dense module comprises 3 convolution layers and 3 ReLU functions, wherein one ReLU function is connected behind each convolution layer, and the dense module inputs the input characteristics of each convolution layer to all subsequent convolution layers through jump connection.
7. The ERA5 precipitation product downscaling method based on deep learning of claim 1, wherein: in the step S5, in the residual attention network, the input sequentially passes through a convolution layer, a residual feature fusion module, a space and channel attention module, an up-sampling layer and a convolution layer;
the residual feature fusion module comprises 3 residual group modules, the tail output of each residual group module is connected in a jumping manner, elements and operations are carried out at the tail of the whole residual feature fusion module, and the residual group module comprises 2 convolution layers 3*3;
the space and channel attention module comprises a space attention module and a channel attention module, wherein in the space attention module, the input is subjected to parallel maximum pooling layer and average pooling layer, the pooled vectors are sent to a full-connection layer for calculation and then added, and the channel attention feature is obtained by multiplying the initial fusion feature after the Sigmoid activation function; and in the channel attention module, carrying out maximum pooling and average pooling according to space, splicing channels, carrying out convolution operation, and multiplying the channel attention module with the channel attention force diagram through a Sigmoid activation function to obtain a final characteristic diagram.
8. The ERA5 precipitation product downscaling method based on deep learning of claim 1, wherein: in the step S6, a 10-fold cross-validation method is used to divide the training set and the testing set.
9. A method as claimed in claim 1, wherein: in the step S6, each parameter of the precipitation scale-down model includes an adjustable learning rate, a momentum parameter, a total iteration number and a weight attenuation parameter;
the initial value of the adjustable learning rate is set to be 10 < -4 >, the learning rate is halved after 2X 105 reverse iterations, the weight attenuation parameter is set to be 10 < -7 >, the momentum parameter is set to be 0.5, the precipitation scale-down model is trained by using an Adam optimization method, wherein the first moment attenuation coefficient beta 1=0.9, the second moment attenuation coefficient beta 2=0.99, and epsilon parameter epsilon in the Adam optimization method is 10 < -8 >; the loss function of the water-reducing scale model is set as follows 1 The loss is given by:
wherein n represents the number of samples, E i Representing corresponding pixel values of ERA5 precipitation data after downscaling, G i And representing the pixel value corresponding to the GPM satellite precipitation data.
10. A method as claimed in claim 1, wherein: in the step S7, the downscaling result is analyzed with accuracy by the correlation coefficient, the deviation and the root mean square error, respectively, as follows
Wherein X is i And Y i Respectively representing the high-resolution rainfall data obtained after the downscaling and the ground rainfall station observation data,and->And respectively representing the average value of the high-resolution precipitation data obtained after the downscaling and the average value of the ground rainfall station observation data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310454771.7A CN116484189A (en) | 2023-04-25 | 2023-04-25 | ERA5 precipitation product downscaling method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310454771.7A CN116484189A (en) | 2023-04-25 | 2023-04-25 | ERA5 precipitation product downscaling method based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116484189A true CN116484189A (en) | 2023-07-25 |
Family
ID=87226457
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310454771.7A Pending CN116484189A (en) | 2023-04-25 | 2023-04-25 | ERA5 precipitation product downscaling method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116484189A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117313026A (en) * | 2023-09-18 | 2023-12-29 | 宁夏大学 | Soil moisture prediction method based on double-branch hybrid model |
| CN117633449A (en) * | 2024-01-25 | 2024-03-01 | 南京信息工程大学 | DE-DOA improved RRDBNet precipitation data downscaling method based on Spark-Cassandra framework |
| CN117892064A (en) * | 2024-03-13 | 2024-04-16 | 南京信息工程大学 | Method, device and memory for reducing daily average air temperature of global climate model |
-
2023
- 2023-04-25 CN CN202310454771.7A patent/CN116484189A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117313026A (en) * | 2023-09-18 | 2023-12-29 | 宁夏大学 | Soil moisture prediction method based on double-branch hybrid model |
| CN117633449A (en) * | 2024-01-25 | 2024-03-01 | 南京信息工程大学 | DE-DOA improved RRDBNet precipitation data downscaling method based on Spark-Cassandra framework |
| CN117633449B (en) * | 2024-01-25 | 2024-04-12 | 南京信息工程大学 | DE-DOA improved RRDBNet precipitation data downscaling method based on Spark-Cassandra framework |
| CN117892064A (en) * | 2024-03-13 | 2024-04-16 | 南京信息工程大学 | Method, device and memory for reducing daily average air temperature of global climate model |
| CN117892064B (en) * | 2024-03-13 | 2024-05-14 | 南京信息工程大学 | Method, device and memory for reducing daily average air temperature of global climate model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114254561B (en) | Waterlogging prediction method, system and storage medium | |
| CN116484189A (en) | ERA5 precipitation product downscaling method based on deep learning | |
| Wang et al. | On deep learning-based bias correction and downscaling of multiple climate models simulations | |
| CN113255972B (en) | Short-term rainfall prediction method based on Attention mechanism | |
| CN111105068B (en) | Numerical mode correction method based on sequential regression learning | |
| CN112668615B (en) | Satellite cloud picture prediction method based on depth cross-scale extrapolation fusion | |
| CN114120101B (en) | Multi-scale comprehensive sensing method for soil moisture | |
| CN112215393B (en) | Rainfall numerical prediction post-processing correction method based on adaptive space-time scale selection | |
| CN117492110B (en) | Numerical mode forecasting method based on artificial intelligence enhancement | |
| CN115062527A (en) | Geostationary satellite sea temperature inversion method and system based on deep learning | |
| CN114330641A (en) | Method for establishing short-term wind speed correction model based on deep learning of complex terrain | |
| CN113984198B (en) | Shortwave radiation prediction method and system based on convolutional neural network | |
| CN117076738A (en) | Medium-long term weather prediction method, system and medium based on variable grid model | |
| CN116152206A (en) | Photovoltaic output power prediction method, terminal equipment and storage medium | |
| Xu et al. | On vision transformer for ultra-short-term forecasting of photovoltaic generation using sky images | |
| CN118245746A (en) | Deep learning-based middle-long term sea surface temperature prediction method | |
| CN112285808B (en) | Method for reducing scale of APHRODITE precipitation data | |
| CN117233869A (en) | Site short-term wind speed prediction method based on GRU-BiTCN | |
| CN117273200A (en) | Runoff interval forecasting method based on convolution optimization algorithm and Pyraformer neural network | |
| Ni et al. | Monthly precipitation prediction at regional scale using deep convolutional neural networks | |
| CN117911883A (en) | GEE time sequence analysis-based large-area ecological vulnerability remote sensing dynamic assessment method | |
| Kaparakis et al. | WF-UNet: Weather data fusion using 3d-unet for precipitation nowcasting | |
| CN114971041A (en) | Sea ice area prediction method and system based on residual error network | |
| CN115980885A (en) | Rainfall forecast deviation correction method based on ensemble forecast | |
| de Baar et al. | Recent improvements in spatial regression of climate data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |