CN112819698A - Data assimilation method, device, equipment and storage medium - Google Patents
Data assimilation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112819698A CN112819698A CN202110246219.XA CN202110246219A CN112819698A CN 112819698 A CN112819698 A CN 112819698A CN 202110246219 A CN202110246219 A CN 202110246219A CN 112819698 A CN112819698 A CN 112819698A
- Authority
- CN
- China
- Prior art keywords
- data
- picture data
- picture
- assimilation
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000012545 processing Methods 0.000 claims description 33
- 238000012549 training Methods 0.000 claims description 23
- 238000003062 neural network model Methods 0.000 claims description 15
- 238000005516 engineering process Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 6
- 238000009499 grossing Methods 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 2
- 238000001556 precipitation Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000012795 verification Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000013136 deep learning model Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013178 mathematical model Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000010355 oscillation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a data assimilation method, a device, equipment and a storage medium, wherein first picture data are determined based on current discrete station data, the first picture data are processed to obtain second picture data, the size of the second picture data is smaller than that of the first picture data, and the second picture data are input into a pre-trained data assimilation model to obtain assimilated precision field picture data. According to the technical scheme, the problem that the resolution of the element field after assimilation is too low due to sparsity of a traditional observation station is solved by referring to a pre-trained data assimilation model, and high-fineness element space distribution is obtained under the condition that station data are sparse.
Description
Technical Field
The embodiment of the invention relates to the technical field of weather prediction, in particular to a data assimilation method, a data assimilation device, data assimilation equipment and a storage medium.
Background
Data assimilation is a process of comprehensively utilizing data from different sources through a series of processing and adjustment. In numerical weather forecasting, data assimilation was initially thought to be a process of analytically processing spatially and temporally distributed observations to provide an initial field for numerical forecasting.
The data assimilation technology can convert discrete and isolated meteorological observation data into spatial distribution of the element field, and provides a better element initial field for numerical prediction. For example, the precipitation distribution at a specific time and in a specific area is deduced through continuous monitoring of precipitation elements at observation sites.
However, for observation sites which are too sparse, the conventional data assimilation technology is difficult to obtain high-precision element space distribution.
Disclosure of Invention
The embodiment of the invention provides a data assimilation method, a device, equipment and a storage medium, which are used for obtaining high-fineness element space distribution under the condition of sparse site data.
In a first aspect, an embodiment of the present invention provides a data assimilation method, including:
determining first picture data based on current discrete site data;
processing the first picture data to obtain second picture data, wherein the size of the second picture data is smaller than that of the first picture data;
and inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
In a second aspect, an embodiment of the present invention further provides a data assimilation device, including:
the first picture data determining module is used for determining first picture data based on the current discrete site data;
the second picture data determining module is used for processing the first picture data to obtain second picture data, wherein the size of the second picture data is smaller than that of the first picture data;
and the assimilation module is used for inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
In a third aspect, an embodiment of the present invention further provides a data assimilation apparatus, including:
one or more processors;
a memory for storing one or more programs;
the one or more programs are executable by the one or more processors to cause the one or more processors to implement a data assimilation method as provided above in the first aspect of embodiments of the present invention.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which one or more computer programs are stored, and when the computer programs are executed by a processor, the computer programs implement the data assimilation method provided in the first aspect.
In the data assimilation method, the data assimilation device, the data assimilation equipment and the storage medium, first picture data are determined based on current discrete station data, the first picture data are processed to obtain second picture data, the size of the second picture data is smaller than that of the first picture data, the second picture data are input into a pre-trained data assimilation model, and assimilated precision field picture data are obtained. According to the technical scheme, the problem that the resolution of the element field after assimilation is too low due to sparsity of a traditional observation station is solved by referring to a pre-trained data assimilation model, and high-fineness element space distribution is obtained under the condition that station data are sparse.
Drawings
FIG. 1 is a flow chart of a data assimilation method according to an embodiment of the present invention;
FIG. 2 is a flowchart of a data assimilation model training method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a Cycle GAN provided in this embodiment;
FIG. 4 is a block diagram of a data assimilation device according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a hardware structure of an apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Data assimilation can be understood as weather forecast or climate prediction and analysis performed by inputting weather data of different time, different regions and different properties into a mathematical model. Data assimilation is a very important step in numerical weather forecasting. If the initial data is directly applied to the mathematical model to make the initial field, high-frequency oscillation is particularly easily caused, which results in failure of prediction. The data assimilation method in the embodiment is to extract optimal input data and input a value mathematical model so as to avoid the problem of prediction failure caused by high-frequency oscillation and improve the success rate of prediction.
FIG. 1 is a flow chart of a data assimilation method, which is applicable to data assimilation of a weather observation station, according to an embodiment of the present invention, and can be executed by a data assimilation device, which can be implemented by hardware and/or software. The data assimilation device can be formed by two or more physical entities or can be formed by one physical entity and is generally integrated in a computer device.
It should be noted that the data assimilation method provided in this embodiment can be specifically used in a computer device, and can be considered to be specifically executed by a data assimilation apparatus integrated on the computer device, where the computer device specifically may be a computer device including a processor, a memory, an input apparatus, and an output apparatus. Such as notebook computers, desktop computers, tablet computers, intelligent terminals, and the like.
Specifically, as shown in fig. 1, the data assimilation method provided by the embodiment of the invention specifically includes the following steps S11, S12, and S13.
And S11, determining first picture data based on the current discrete site data.
In this embodiment, the station refers to a monitoring station that can detect weather conditions. The station data refers to data of meteorological observation characteristics in a station range collected by the monitoring station. The site data may be any one or more of: precipitation amount, precipitation range, high air pressure, dew point temperature, humidity, wind direction, wind speed and the like.
Further, the site data may be data collected in a time dimension. The site data may be data collected by one observation station or data collected by a plurality of observation stations. Since the site data is collected in the time dimension, the site data is in a discrete form. For example: the data collection may be performed once at intervals of 30 minutes, or may be performed once at intervals of 1 hour. Different data may also employ different time dimensions. This embodiment is not limited.
Further, the current discrete site data does not refer to the weather characteristic data at the current moment. May be the meteorological feature data within a preset time range which can be collected currently. For example: it can be a month precipitation data, or a half month temperature data, etc.
In this embodiment, the first picture data is large-size picture data, the first picture data refers to large-size picture data formed by discrete site data, the large-size picture data may be in a picture format with a resolution of 1km × 1km, and the picture size is 4500 × 7000.
In this embodiment, the current discrete site data may be directly converted into the first picture data by an interpolation technique. The interpolation method can adopt any one of the following methods: polynomial interpolation, linear interpolation, Cubic interpolation, lagrange polynomial interpolation, newton interpolation, hermitian interpolation, spline interpolation, and the like.
In an exemplary embodiment, determining the first picture data based on the current discrete site data includes: and converting the current discrete station data into first picture data through a linear interpolation technology.
Linear interpolation is an interpolation method for one-dimensional data, and numerical values are estimated according to two data points adjacent to the left and right of a point to be interpolated in a one-dimensional data sequence. Specifically, the left and right adjacent data points are assigned specific weights according to their distances. The linear interpolation method can reduce the calculation amount of data.
In the present embodiment, only the interpolation method is described, but not limited. The first picture data may be obtained by any interpolation method.
And S12, processing the first picture data to obtain second picture data.
In this embodiment, the size of the second picture data is smaller than the size of the first picture data. Further, the second picture data may be obtained by dividing or reducing the first picture data.
In this embodiment, a sliding frame may slide on the first picture data to obtain a plurality of second picture data, or the first picture data may be directly reduced to one second picture data in a picture resize manner; the first picture data can be cut to obtain a plurality of second picture data by adopting a picture cutting mode.
It should be noted that, in the present embodiment, only the processing manner of the first picture data is illustrated by way of example, and is not limited.
And S13, inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
The data assimilation model may be obtained by a pre-training method, and may be trained by any machine learning method, which is not limited in this embodiment.
Furthermore, if the first picture data is cut to obtain a plurality of second picture data, the plurality of second picture data are sequentially input into the pre-trained data assimilation model to obtain a plurality of output pictures, and the precision field picture data with the same size as the first picture can be obtained only after the output pictures are spliced.
It should be noted that, as the stitching method of the multiple output pictures, any existing image stitching method may be adopted, and this embodiment is not limited.
Further, if a plurality of second picture data are obtained after the sliding of the sliding frame with the overlapping rate, the plurality of output pictures need to be smoothed for the overlapping portion after the splicing.
In another embodiment, the first picture data is not processed by sliding frame and dividing, but is directly reduced to the second picture data by using picture resize. And inputting the second picture data into a pre-trained data assimilation model to obtain the assimilated precision field picture data, wherein the assimilated precision field picture data is obtained without additional splicing and smoothing.
The data assimilation method provided by the embodiment of the invention comprises the steps of firstly determining first picture data based on current discrete site data, processing the first picture data to obtain second picture data, inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data, wherein the size of the second picture data is smaller than that of the first picture data. According to the technical scheme, the problem that the resolution of the element field after assimilation is too low due to sparsity of a traditional observation station is solved by referring to a pre-trained data assimilation model, and high-fineness element space distribution is obtained under the condition that station data are sparse.
In one embodiment, processing the first picture data to obtain the second picture data includes: the first picture data is overlapped and slidably divided into a plurality of second picture data according to a preset step length.
In this embodiment, the size and dimension of the sliding frame may be set in advance, wherein the size and shape of the sliding frame may be set according to actual conditions. The size of the slide frame coincides with the size of the second picture data.
Further, the sliding frame sequentially slides on the first picture data according to a preset sliding track to obtain a plurality of second picture data. The preset sliding trajectory may be from top to bottom, left to right, etc. Other sliding tracks are also possible, and this embodiment is not limited.
Further, the overlapping rate of the position of the former sliding frame and the position of the latter sliding frame can be set. The overlapping rate is the ratio of the overlapping area of the previous sliding frame position and the next sliding frame position to the whole sliding frame area. As such, the detail information in each second picture data may be included.
Further, the size of the first picture data is (x1, y1), and the size of the second picture data is (x2, y 2). If the overlap ratio of the previous slide frame position and the next slide frame position is 0, the slide step (s _ x, s _ y) is equal to the size of the second picture data, i.e., (x2, y 2). The number of the second picture data is (x1/x2) × (y1/y 2).
If the overlap ratio of the previous slide frame position and the next slide frame position is (r _ x, r _ y), the slide step (s _ x, s _ y) is smaller than the size of the second picture data, i.e., the slide step (s _ x, s _ y) is (x2/r _ x, y2/r _ y). Wherein r _ x and r _ y are both smaller than 1. And r _ x is the overlap ratio in the x-direction and r _ y is the overlap ratio in the y-direction.
Further, the size of the first picture data cannot be divided exactly by the sliding step size, i.e., (x1/s _ x) the remainder is taken to be other than 0, or (y1/s _ y) the remainder is taken to be other than 0. At this time, zero padding may be performed on the bottom row and the right column of the first picture data, which may specifically be calculated according to the following formula:
num_line＝ceil(x1/s_y)×s_y-x1；
num_col＝ceil(y1/s_x)×s_x-y1；
where num _ line and num _ col represent the number of rows and columns to be complemented by 0, respectively, and ceil () represents rounding up.
After the above operations, a first picture data with size (x1, y1) is divided into ceil (x1/s _ y) × ceil (y1/s _ x) second picture data with size (x2, y2) and stored.
Further, in addition to the above embodiment, the method for obtaining assimilated precision field picture data by inputting the second picture data into a pre-trained data assimilation model includes: inputting a plurality of second picture data into a pre-trained data assimilation model to obtain a plurality of output pictures; and splicing and smoothing the output pictures to obtain assimilated precision field picture data.
In the embodiment, the first picture data is divided into a plurality of second picture data, the plurality of second picture data are sequentially input into the pre-trained data assimilation model to obtain a plurality of output pictures, and the precision field picture data with the same size as the first picture can be obtained only after the plurality of output pictures are spliced.
It should be noted that, as the stitching method of the multiple output pictures, any existing image stitching method may be adopted, and this embodiment is not limited.
Further, if a plurality of second picture data are obtained after the sliding of the sliding frame with the overlapping rate, the plurality of output pictures need to be smoothed for the overlapping portion after the splicing.
In another embodiment, processing the first picture data to obtain second picture data includes: and processing the first picture data by using an image size transformation resize mode to obtain second picture data.
In this embodiment, the processing of the first picture data is not performed by sliding the sliding frame, but is performed by directly reducing the first picture data to the second picture data by directly using the picture resize. It should be noted that, by using the picture resize method, only one piece of second picture data can be obtained from the first picture data. By adopting the picture resize mode, the detail information of the whole picture can be sensed.
In addition to the present embodiment, the second picture data is input into a pre-trained data assimilation model to obtain the assimilated precision field picture data, which is the picture data of one precision field, and no additional splicing or smoothing is required.
In addition to the above embodiments, there is provided a training method of a data assimilation model, as shown in fig. 2, the training method of the data assimilation model mainly includes the following steps:
and S21, determining third picture data and fourth picture data based on the historical discrete site data.
In this embodiment, the historical discrete site data may be discrete site data in a certain period of time before, and the historical discrete site data may be data of one site in a plurality of time periods, or data of a plurality of sites in a plurality of times. In order to ensure the accuracy of the training, the third image data may be one image or a set of multiple images.
In one embodiment, determining the third picture data based on the historical discrete site data comprises: converting the historical discrete site data into lattice point data; converting the lattice point data into third picture data; carrying out downsampling processing on the lattice point data to obtain downsampled picture data; and the downsampled picture data is subjected to linear interpolation processing to obtain the third picture data.
In the present embodiment, conventional data assimilation techniques are used to convert discrete site data into dense grid data, also referred to as element spatial distribution information, i.e., precision fields. For example: the meteorological station observes the temperature data of the xu-hui area, and the existing data assimilation technology can deduce the temperature condition of Shanghai city according to the information. The above-mentioned factors are precipitation, air temperature and wind speed, etc.
Further, the lattice point data is converted into third picture data, and the third picture data is used as target data of neural network model learning.
Furthermore, downsampling is carried out on the lattice point data, and sparsity of a station is simulated. The down-sampling resolution can be arbitrarily selected, such as 10km × 10km, 50km × 50km, and 100km × 100 km. The novel data assimilation model constructed based on the method can deal with observation site data under various resolutions, and is beneficial to data restoration under the condition that few historical observation sites exist.
The technical scheme of the embodiment can also obtain high-fineness element space distribution by training low-resolution data of a sparse observation station generated by downsampling lattice data, and a high-resolution element field is difficult to obtain under the condition of the traditional data assimilation technology. The method can be used for repairing the problem of low resolution of the assimilated element field caused by the sparsity of the traditional observation station.
And finally, converting the downsampled sparse site into fourth picture data through linear interpolation, and storing the fourth picture data as input data of the neural network model.
And S22, processing the third picture data and/or the fourth picture data to obtain a data set.
In this embodiment, the third picture data and/or the fourth picture data are/is divided to obtain a plurality of small-sized pictures. Or reducing the third picture data and/or the fourth picture data by adopting a picture resize technology to obtain a small-size picture. The data set is a set comprising a plurality of small-sized pictures.
Further, the dividing processing on the third picture data and/or the fourth picture data is the same as the dividing processing on the first picture data in the foregoing embodiment, and the specific method may refer to the description in the foregoing embodiment, and is not repeated in this embodiment.
The reducing of the third picture data and/or the fourth picture data by using the picture resize technology is the same as the reducing of the first picture data by using the picture resize technology in the foregoing embodiment, and the specific method may refer to the description in the foregoing embodiment, and is not described in detail in this embodiment.
Further, the data set is divided into a training set, a verification set and a test set. Wherein the number ratio of the small-size pictures in the training set, the verification set and the test set is 8:1: 1. The training set, the verification set and the test set are used for training and effect verification of the neural network model.
And S23, constructing a neural network model by taking the third picture data as target data and the fourth picture data as input data.
In this embodiment, the neural network model may select various deep learning models for solving the super-resolution task, such as: Super-Resolution Convolutional Neural Network (SRCNN), Deep Convolutional Neural Network (VDSR), Super-Resolution general adaptive Network (SRGAN), pix2pixHD, cyclic adaptive Network (cyclic GAN), and the like.
In this embodiment, a Cycle GAN is taken as an example for explanation, and the Cycle GAN is an improvement on a GAN network, and is used for solving the problem of image conversion and generating a better image. Fig. 3 is a schematic structural diagram of a Cycle GAN provided in this embodiment, and as shown in fig. 3, generators G1 and G2 serve as an important component in the GAN to convert an input low-resolution picture into an expected high-resolution picture, and discriminators D1 and D2 serve as an important component in the GAN to discriminate a generated high-resolution picture from a real picture. U-Net is a name given by the structural shape of a neural network in a U-shape, and is mainly applied to a generator part in GAN.
And S24, training the neural network model by using the data set to obtain a data assimilation model.
The data assimilation model may be obtained by a pre-training method, and may be trained by any machine learning method, which is not limited in this embodiment.
In the data assimilation process is introduced to the deep neural network model in this embodiment, through a large amount of data sample training, approximate fitting traditional data assimilation process, the efficiency of data assimilation can be promoted greatly to this scheme, shortens the meticulous field distribution response time to meteorological specific element greatly.
In an exemplary embodiment, the present disclosure provides an off-line construction method of a data assimilation model. The implementation provides an off-line construction method of a data assimilation model, which mainly comprises the following steps:
the method comprises the following steps: and converting the site data into lattice point data.
The discrete site data is converted into grid data, i.e., precision field, using conventional data assimilation techniques. For example: the meteorological station observes the temperature data of the xu-hui area, and the existing data assimilation technology can deduce the temperature condition of Shanghai city according to the information, but the step is long in time consumption and needs a large amount of computing resources. The grid point data is also referred to as element spatial distribution information, and the elements may be parameters representing weather conditions, such as precipitation, temperature, wind speed, and the like.
The first step is intended for data generation, and any existing data assimilation technology can be used to convert the site data into the lattice point data.
Step two: and converting the lattice point data into a large-size picture.
The large-size picture is a picture format having a resolution of 1km × 1km in a chinese region, and has a picture size of 4500 × 7000.
First, the lattice data obtained by the data assimilation technology in the first step is converted into picture data to be stored, and the picture is used as a target for deep learning model learning and can be also called as target data.
And then, downsampling the lattice point data obtained by the data assimilation technology in the step one, simulating the sparsity of a station, and randomly selecting the downsampling resolution. For example, 10km × 10km, 50km × 50km, 100km × 100 km. The novel data assimilation model constructed based on the method can deal with observation site data under various resolutions, and is beneficial to data restoration under the condition that few historical observation sites exist.
And finally, converting the down-sampled sparse site into a large-size picture through linear interpolation, and storing the large-size picture as input data of the neural network model.
Step three: and segmenting the large-size picture.
The purpose of this step is to reduce the large-size picture in step two, in order to facilitate the neural network training.
In one embodiment, the target data and the input data generated in step two are cut into small-sized pictures in an overlapping and sliding manner, for example, the size of the small-sized picture is set to (scale _ x, scale _ y), then the sliding step size (sliding amplitude) in two directions is (stride _ x, stride _ y), and the constraint: scale _ x <4500 (size of large size picture), and scale _ y < 7000.
Sliding amplitude:
stride_x＝scale_x/r_x
stride_y＝scale_y/r_y
wherein, 1/r _ x and 1/r _ y are the overlapping rates of two directions respectively.
It should be noted that if the size of the large size picture cannot be divided by the sliding amplitude, that is, 4500 does not take the remainder of stride _ x to be 0, or 7000 does not take the remainder of stride _ y to be 0; zero padding operations may be performed on the bottom row and right column of a large size picture.
The number of rows and columns of zero padding can be calculated according to the following formula:
num_line＝ceil(4500/stride_y)×stride_y-4500
num_col＝ceil(7000/stride_x)×stride_x-7000
where num _ line and num _ col represent the number of rows and columns to be complemented by 0, respectively, and ceil () represents rounding up.
After the above operations, a large 4500 × 7000 picture is cut into ceil (4500/stride _ y) × ceil (7000/stride _ x) small pictures with scale _ x × scale _ y size, and stored.
In another embodiment, the step of reducing the large-size picture may also be performed not by sliding cut, but by using a picture resize, that is, by directly converting the large-size picture into the small-size picture by the resize. By the picture reduction method, only one small-size picture can be generated from one large-size picture.
The method comprises the steps that a cutting mode is adopted to obtain a small-size picture, and detail information in a small image can be reserved; and the small-size picture is acquired by adopting a resize mode, so that the detail information of the whole picture can be sensed.
The size pictures obtained in the two embodiments are divided into a training set, a verification set and a test set according to the proportion of 8:1:1, and the training set, the verification set and the test set are used for training and effect verification of a neural network model.
Step four: and (5) training a neural network model.
And training a neural network model according to the data set generated in the first step to the third step to obtain a data assimilation model.
Various deep learning models for solving the super-resolution task may be selected. Such as SRCNN, VDSR, SRGAN, pix2pixHD, SRGAN, Cycle GAN, etc. The trained data assimilation model can realize the conversion of the low-resolution pictures generated in the second step and the third step into the high-resolution element distribution field.
In one illustrative embodiment, the present embodiment provides an online working method of a data assimilation model. The implementation provides an on-line working method of a data assimilation model, which mainly comprises the following steps:
the method comprises the following steps: and converting the observation site data into a large-size picture.
And directly performing linear interpolation on the thinned station data, converting the station data into a large-size picture, and storing the large-size picture as input data of the data assimilation model.
Step two: and segmenting the large-size picture.
The large-size segmentation method in this embodiment is consistent with the large-size image segmentation method in the model offline construction embodiment, and specifically, the large-size image segmentation method in the model offline construction embodiment may be referred to, and details are not repeated in this embodiment.
Step three: the data model is assimilated on line.
And the trained data assimilation model can realize the conversion of the low-resolution pictures generated in the step two into the high-resolution element distribution field.
Furthermore, if the element fine field is generated in the second step by adopting a cutting mode, mean value smoothing is required to be performed on the overlapped part to form a large-size picture, so that data assimilation is completed.
Fig. 4 is a block diagram of a data assimilation device according to an embodiment of the present invention. The data assimilation device is suitable for the data assimilation of a meteorological observation station, can be realized by hardware and/or software, and is generally integrated in intelligent equipment.
As shown in fig. 4, the data assimilation device includes: a first picture data determination module 41, a second picture data determination module 42, and an assimilation module 43.
Wherein the content of the first and second substances,
a first picture data determining module 41, configured to determine first picture data based on current discrete site data;
a second picture data determining module 42, configured to process the first picture data to obtain second picture data, where a size of the second picture data is smaller than a size of the first picture data;
and the assimilation module 43 is used for inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
The data assimilation device provided by the embodiment of the invention firstly determines first picture data based on current discrete station data, processes the first picture data to obtain second picture data, wherein the size of the second picture data is smaller than that of the first picture data, and inputs the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data. According to the technical scheme, the problem that the resolution of the element field after assimilation is too low due to sparsity of a traditional observation station is solved by referring to a pre-trained data assimilation model, and high-fineness element space distribution is obtained under the condition that station data are sparse.
Further, the first picture data determining module 41 is specifically configured to convert the current discrete site data into the first picture data through a linear interpolation technique.
Further, the second picture data determining module 42 is specifically configured to slidably divide the first picture data into a plurality of second picture data according to a preset step length, wherein the first picture data is overlapped.
Further, the assimilation module 43, including the input unit and the processing unit,
the input unit is used for inputting the second picture data into a pre-trained data assimilation model to obtain a plurality of output pictures;
a processing unit for splicing and smoothing the output pictures to obtain assimilated precision field picture data
Further, the second picture data determining module 42 is specifically configured to process the first picture data in a manner of transforming resize of the image size to obtain second picture data.
Further, the training step of the data assimilation model comprises the following steps:
determining third picture data and fourth picture data based on historical discrete site data;
processing the third picture data and/or the fourth picture data to obtain a data set;
establishing a neural network model by taking the third picture data as target data and the fourth picture data as input data;
and training the neural network model by using the data set to obtain a data assimilation model.
Specifically, determining the third picture data and the fourth picture data based on the historical discrete site data includes:
converting the historical discrete site data into lattice point data;
converting the lattice point data into third picture data;
carrying out downsampling processing on the lattice point data to obtain downsampled picture data;
and the downsampled picture data is subjected to linear interpolation processing to obtain the second picture data.
The data assimilation device provided by the embodiment of the invention can execute the data assimilation method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 5 is a schematic diagram of a hardware structure of an apparatus according to an embodiment of the present invention, as shown in fig. 5, the apparatus includes a processor 501, a memory 502, an input device 503, and an output device 504; the number of the processors 501 in the device may be one or more, and one processor 501 is taken as an example in fig. 5; the processor 501, the memory 502, the input device 503 and the output device 504 of the apparatus may be connected by a bus or other means, and fig. 5 illustrates the connection by a bus as an example.
The memory 502 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data assimilation method in the embodiment of the present invention (for example, the modules in the data assimilation apparatus shown in fig. 4 include the first picture data determination module 41, the second picture data determination module 42, and the assimilation module 43). The processor 501 executes the software programs, instructions and modules stored in the memory 502 to execute various functional applications of the device and data processing, so as to realize the data assimilation method.
The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like. Further, the memory 502 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 502 may further include memory located remotely from processor 501, which may be connected to devices through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
And when the one or more programs included in the above-described apparatus are executed by the one or more processors 501, the programs perform the following operations:
determining first picture data based on current discrete site data;
processing the first picture data to obtain second picture data, wherein the size of the second picture data is smaller than that of the first picture data;
and inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
The input device 503 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the apparatus. The output device 504 may include a display device such as a display screen.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processing device, implements a data assimilation method provided by an embodiment of the present invention, and the method includes:
determining first picture data based on current discrete site data;
processing the first picture data to obtain second picture data, wherein the size of the second picture data is smaller than that of the first picture data;
and inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
Of course, the storage medium provided by the embodiments of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also execute the related operations in the data assimilation method provided by any embodiments of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the data assimilation device, the included units and modules are only divided according to the functional logic, but not limited to the above division as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for data assimilation, comprising:
determining first picture data based on current discrete site data;
processing the first picture data to obtain second picture data, wherein the size of the second picture data is smaller than that of the first picture data;
and inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
2. The method of claim 1, wherein determining the first picture data based on current discrete site data comprises:
and converting the current discrete station data into first picture data through a linear interpolation technology.
3. The method of claim 2, wherein processing the first picture data to obtain second picture data comprises:
and the first picture data is divided into a plurality of second picture data in a sliding manner according to a preset step length, wherein the first picture data is overlapped.
4. The method of claim 3, wherein inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data comprises:
inputting a plurality of second picture data into a pre-trained data assimilation model to obtain a plurality of output pictures;
and splicing and smoothing the output pictures to obtain assimilated precision field picture data.
5. The method of claim 2, wherein processing the first picture data to obtain second picture data comprises:
and processing the first picture data by using an image size transformation resize mode to obtain second picture data.
6. The method of claim 1, wherein the step of training the data assimilation model comprises:
determining third picture data and fourth picture data based on historical discrete site data;
processing the third picture data and/or the fourth picture data to obtain a data set;
establishing a neural network model by taking the third picture data as target data and the fourth picture data as input data;
and training the neural network model by using the data set to obtain a data assimilation model.
7. The method of claim 6, wherein determining the third picture data and the fourth picture data based on historical discrete site data comprises:
converting the historical discrete site data into lattice point data;
converting the lattice point data into third picture data;
carrying out downsampling processing on the lattice point data to obtain downsampled picture data;
and the downsampled picture data is subjected to linear interpolation processing to obtain the fourth picture data.
8. A data assimilation device, comprising:
the first picture data determining module is used for determining first picture data based on the current discrete site data;
the second picture data determining module is used for processing the first picture data to obtain second picture data, wherein the size of the second picture data is smaller than that of the first picture data;
and the assimilation module is used for inputting the second picture data into a pre-trained data assimilation model to obtain assimilated precision field picture data.
9. A data assimilation apparatus, comprising:
one or more processors;
a memory for storing one or more programs;
the one or more programs being executable by the one or more processors to cause the one or more processors to implement the data assimilation method of any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data assimilation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110246219.XA CN112819698A (en) | 2021-03-05 | 2021-03-05 | Data assimilation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110246219.XA CN112819698A (en) | 2021-03-05 | 2021-03-05 | Data assimilation method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112819698A true CN112819698A (en) | 2021-05-18 |
Family
ID=75862933
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110246219.XA Pending CN112819698A (en) | 2021-03-05 | 2021-03-05 | Data assimilation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112819698A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024071377A1 (en) * | 2022-09-29 | 2024-04-04 | 国立大学法人東京工業大学 | Information processing device, information processing method, and program |
-
2021
- 2021-03-05 CN CN202110246219.XA patent/CN112819698A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024071377A1 (en) * | 2022-09-29 | 2024-04-04 | 国立大学法人東京工業大学 | Information processing device, information processing method, and program |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Guo et al. | Data‐driven flood emulation: Speeding up urban flood predictions by deep convolutional neural networks | |
| KR102143108B1 (en) | Lane recognition modeling method, device, storage medium and device, and recognition method, device, storage medium and device | |
| CN112116155B (en) | Population flow prediction method and device based on intelligent decision and computer equipment | |
| JP6384065B2 (en) | Information processing apparatus, learning method, and program | |
| CN110738252A (en) | Space autocorrelation machine learning satellite precipitation data downscaling method and system | |
| KR102063358B1 (en) | Learning method and testing method for generating high-resolution weather and climate data, and learning device and testing device using the same | |
| CN109657839B (en) | Wind power prediction method based on deep convolutional neural network | |
| CN110991497B (en) | BSVC (binary sequence video coding) -method-based urban land utilization change simulation cellular automaton method | |
| US11836605B2 (en) | Meteorological big data fusion method based on deep learning | |
| KR20220125719A (en) | Method and equipment for training target detection model, method and equipment for detection of target object, electronic equipment, storage medium and computer program | |
| Guo et al. | Data-driven rapid flood prediction mapping with catchment generalizability | |
| Dimitriou et al. | A deep learning framework for simulation and defect prediction applied in microelectronics | |
| CN112241784A (en) | Training generative model and discriminant model | |
| CN115457492A (en) | Target detection method and device, computer equipment and storage medium | |
| CN112819698A (en) | Data assimilation method, device, equipment and storage medium | |
| CN106599172A (en) | Lossless fidelity visualization optimization method for mass data | |
| CN110532868B (en) | Method for predicting free space semantic boundary | |
| Son et al. | Partial convolutional LSTM for spatiotemporal prediction of incomplete data | |
| CN107464272A (en) | The interpolation method of central diffusion type meteorological causes isopleth based on key point | |
| CN111105093A (en) | Near precipitation prediction method based on CNN and BilSTM | |
| CN116051699A (en) | Dynamic capture data processing method, device, equipment and storage medium | |
| US20230130199A1 (en) | Semiconductor device simulation system and semiconductor device simulation method | |
| CN109409497B (en) | Road condition prediction method and device | |
| Bosma et al. | Estimating solar and wind power production using computer vision deep learning techniques on weather maps | |
| CN116166642A (en) | Spatio-temporal data filling method, system, equipment and medium based on guide information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |