CN110991685A

CN110991685A - Meteorological temperature prediction method

Info

Publication number: CN110991685A
Application number: CN201910796220.2A
Authority: CN
Inventors: 马贞立; 高艺恬; 陈千千
Original assignee: Hohai University HHU
Current assignee: Hohai University HHU
Priority date: 2019-08-27
Filing date: 2019-08-27
Publication date: 2020-04-10

Abstract

The invention discloses a meteorological temperature prediction method, which comprises the following steps: s1: collecting meteorological data; s2: selecting effective data from the meteorological data collected in the step S1, and then forming the effective data into a data set; s3: dividing a data set into a training set, a testing set and a verification set; s4: setting parameters of a neural network model; s5: setting input function parameters; s6: training the neural network model; s7: checking whether the neural network model is over-fitted: if so, return to step S5; otherwise, proceed to step S8; s8: evaluating the trained neural network model by using the test set; s9: and outputting a prediction result: and the prediction result is the fitting degree between the predicted value and the actual value of the test set after the prediction by the neural network model. The invention can process a large amount of meteorological data and has simple method.

Description

Meteorological temperature prediction method

Technical Field

The invention relates to deep learning data processing, in particular to a meteorological temperature prediction method.

Background

As an important component of the natural environment, climate affects human life in all aspects, ranging from small to personal life to large as agriculture, economy, military. In the development process of many years, there are three common methods for analyzing and calculating meteorological data, namely, a weather method, namely, weather is analyzed on the basis of the weather, and available data mainly include weather patterns, and then weather satellite cloud pictures, radars and the like; secondly, a statistical method, namely a meteorological statistical analysis and calculation method, which takes probability theory statistics as a means to forecast weather; thirdly, a dynamic numerical method, namely a dynamic meteorology analysis and numerical calculation method, which is based on three subjects of dynamic meteorology, thermodynamics and hydrodynamics, forms a forecast equation and uses a computer as a tool to form weather forecast.

The first method focuses on the distribution characteristics of data, the second method focuses on mining data relevance, and the third method focuses on forming a complex numerical model based on the combination of the results of the first two methods, wherein the model can accurately forecast the atmospheric motion state. Regarding the analysis method of numerical weather forecast, bayesian probabilistic argument is used to derive the idealized equations to find the best analysis of numerical weather prediction. Three weather forecasting methods, which promote, penetrate and combine each other, are continuously developed forward.

These methods can be combined, integrated, and generally implemented using a computer.

In recent years, the wave of deep learning is raised in the field of artificial intelligence, the situation is greatly increased from the academic world to the industrial world, and the deep learning is emphasized by researchers and high-tech companies in related fields of all countries in the world, so that the deep learning has obvious advantages in feature extraction and model fitting compared with a shallow model. Deep learning is adept at mining abstract distributed feature representations, and these abstract representations have good generalization capability. The previously difficult problem in artificial intelligence has been solved with the advent of deep learning [13 ]. Moreover, as the processing capability of the chip is rapidly developed and the data volume of the training set is greatly increased, deep learning is greatly successful from the aspects of theoretical analysis and application; the applicable range of deep learning in signal processing is very wide: voice, image, video, text, language, and semantic information. Likewise, challenges and opportunities are presented to weather forecasting techniques.

In foreign countries, a series of researches are carried out on temperature attributes and data sequence dependencies, such as Singh S, Bhambri P and the like, so that a temperature prediction model combining GA and a neural network based on a time sequence is provided; s Singh and J Gill consider different meteorological factors and provide a real-time temperature prediction model based on combination of a genetic algorithm and a BP neural network. The Aznarte J L and the Siebert N jointly use four machine learning algorithms and numerical weather forecast to perform modeling and prediction, and a dynamic linear evaluation experimental method is provided.

In China, the application of data mining technology in meteorology is proposed by Malting Huai and Muchun, etc. The field of the technology is wide, the Weak army and the like propose to apply a BP neural network to implement refined temperature prediction and the like, and deep learning is increasingly applied to meteorological prediction.

Disclosure of Invention

The purpose of the invention is as follows: the invention aims to provide a meteorological temperature prediction method which can solve the technical problems of large meteorological data volume and complex processing in the prior art.

The technical scheme is as follows: the meteorological temperature prediction method comprises the following steps:

s1: collecting meteorological data;

s2: selecting effective data from the meteorological data collected in the step S1, and then forming the effective data into a data set;

s3: dividing a data set into a training set, a testing set and a verification set;

s4: setting parameters of a neural network model;

s5: setting input function parameters;

s6: training the neural network model;

s7: checking whether the neural network model is over-fitted: if so, return to step S5; otherwise, proceed to step S8;

s8: evaluating the trained neural network model by using the test set;

s9: and outputting a prediction result: and the prediction result is the fitting degree between the predicted value and the actual value of the test set after the prediction by the neural network model.

Further, in step S2, the process of selecting valid data is: evaluating the quality of the meteorological data collected in the step S1, deleting uninteresting related attributes, adding derivative features to be evaluated, unifying data types, and processing extreme abnormal values and missing data values.

Further, the processing of the extreme abnormal value is to generate a two-dimensional table data structure (DataFrame) by using a descriptor function, wherein the two-dimensional table comprises a count, an average value, a standard deviation, a minimum value, a 25 th percentile, a 50 th percentile, a 75 th percentile and a maximum value, and if the meteorological data is less than the 25 th percentile or greater than the 75 th percentile, the meteorological data is determined to be the extreme abnormal value.

Further, the processing of the missing data value is to fill the missing value with an interpolated value, where the interpolated value is an average value of two data before and after the missing value or an average value of the average value over a period of time.

Further, in step S3, the data in the training set accounts for 80% of the data in the data set, the data in the testing set accounts for 10% of the data in the data set, and the data in the verification set accounts for 10% of the data in the data set.

Further, the step S4 specifically includes the following steps: the depth of the neural network model is set to be two layers, the width is set to be 50 nodes, the storage position of data in the neural network model is specified, and a ReLU function is adopted as an activation function.

Further, the input function parameters in the step S5 include the following five parameters: input interface type, target value, number of times the entire data set has been executed, whether a batch of subsets is randomly selected before each execution, number of samples participating in each execution.

Further, the step S6 specifically includes the following steps: training a neural network model through a training set, iterating for 50 times, selecting training records of random batches, pushing the training records through a network, recording a loss function for each iteration, and adjusting the association weight between two connected neurons on the basis of evaluating whether loss is reduced and logic of a neural network optimizer.

Has the advantages that: the invention discloses a meteorological temperature prediction method which can process a large amount of meteorological data and is simple.

Drawings

FIG. 1 is a flow chart of a method in accordance with an embodiment of the present invention;

FIG. 2 illustrates a specific neuron model in accordance with an embodiment of the present invention;

FIG. 3 is a diagram illustrating a neural network model according to an embodiment of the present invention;

FIG. 4(a) is a diagram illustrating a step function according to an embodiment of the present invention;

FIG. 4(b) is a diagram illustrating a sigmoid function according to an embodiment of the present invention;

FIG. 4(c) is an image of the Tanh function in an embodiment of the present invention;

FIG. 4(d) is an image of a ReLU function according to an embodiment of the present invention;

FIG. 5(a) is a minimum pressure profile in an embodiment of the present invention;

FIG. 5(b) is a distribution of rainfall features in an embodiment of the present invention;

FIG. 6 is an output of the data miss processing according to an embodiment of the present invention;

FIG. 7 is a Loss record for each loop in an embodiment of the present invention.

Detailed Description

The specific embodiment discloses a meteorological temperature prediction method, as shown in fig. 1, comprising the following steps:

s1: collecting meteorological data;

s4: setting parameters of a neural network model;

s5: setting input function parameters;

s6: training the neural network model; the neural network model is shown in fig. 2;

s8: evaluating the trained neural network model by using the test set;

In step S2, the process of selecting valid data is: and (4) evaluating the quality of the meteorological data collected in the step (S1), deleting uninteresting related attributes, adding derivative features to be evaluated, unifying data types, and processing extreme abnormal values and data missing values.

The extreme abnormal value is processed by generating a two-dimensional table data structure (DataFrame) by using a descriptor function, wherein the two-dimensional table comprises a count, an average value, a standard deviation, a minimum value, a 25 th percentile, a 50 th percentile, a 75 th percentile and a maximum value, and if the meteorological data is less than the 25 th percentile or more than the 75 th percentile, the meteorological data is judged to be the extreme abnormal value.

The missing data value is processed by filling the missing value with an interpolated value, which is an average value of two data before and after the missing value or an average value of the average value over a period of time.

In step S3, the data in the training set accounts for 80% of the data in the data set, the data in the testing set accounts for 10% of the data in the data set, and the data in the verification set accounts for 10% of the data in the data set.

Step S4 specifically includes the following processes: the depth of the neural network model is set to be two layers, the width is set to be 50 nodes, the storage position of data in the neural network model is specified, and a ReLU function is adopted as an activation function.

The input function parameters in step S5 include the following five: input interface type, target value, number of times the entire data set is executed in its entirety, whether a batch of subsets is randomly selected before each execution, number of samples participating in each execution.

Step S6 specifically includes the following processes: training a neural network model through a training set, iterating for 50 times, selecting training records of random batches, pushing the training records through a network, recording a loss function for each iteration, and adjusting the correlation weight between two connected neurons on the basis of evaluating whether loss is reduced and the logic of a neural network optimizer.

Analysis of simulation experiment results

1. Data pre-processing

The data used in this experiment were 3000 sets of Weather data from 2008.1.1 to 2016.3.18, available from Weather underwriter, Airport, boston, usa, General Edward Lawrence Logan International Airport.

(1) Selection of the relevant attributes: since the goal here is to predict the future average temperature from weather measurements of the last three days. Only the maximum, minimum and average temperatures per day, and all new derivative variables added (i.e. data from the first three days) need be retained. The original characteristics are maximum temperature (maxtempm), minimum temperature (mintempm), average temperature (meatempm), average dew point (meanfwtm), average pressure (meanpressure), maximum humidity (maxhumidity), minimum humidity (minhumurity), maximum dew point (maxdewtm), minimum dew point (minfwtm), maximum pressure (maxpressure), minimum pressure (minpressurem), rainfall (preconpm) and their new derivative variables (meatempm _1 before one day, mean temperature before two days (meatempm _2), mean temperature before three days (meatempm _3), derivative variables of other characteristics, and so on. Wherein the unit of each feature: temperature (F), dew point (F), pressure (in Hg), humidity (%), rainfall (in).

(2) Data normalization: the built-in function realizes data standardization.

(3) Processing of extreme outliers: counting, average value, standard deviation, minimum value, 25 th percentile, 50% percentile (or median), 75 th percentile and maximum value are output through a descriptor function of a DataFrame method, and a characteristic column with an extreme abnormal value is judged and finally output according to a quartile method.

(4) Missing value processing: and if the element row in the output data has a deficiency according to the non-null value of the element row, supplementing the element row according to the introduction of the third chapter, and finally excluding the data of the first three days.

FIG. 4(a) is a schematic diagram of a step function; FIG. 4(b) is a schematic diagram of a sigmoid function; FIG. 4(c) is an image of the Tanh function; fig. 4(d) is an image of the ReLU function.

2. Building neural network model

(1) Determining neural network parameters: from the flow chart created by the neural network model, mintempm and maxtempm columns in the data are deleted, and data about future temperatures are obviously not available because they are predicted. Specifying a neural network with a depth of two layers, wherein the width of the two layers is 50 nodes; indicating a storage location of the model data; the activation function defaults to ReLU.

(2) Defining an input function, determining its parameters: after the experiment tests some data, the batch size is determined to be 200; circulating for 50 times; num epochs are not specified;

3. inputting training data into neural network for training model

To avoid the over-fitting phenomenon, a simple training loop is defined, the model is trained for 50 iterations, the model is trained on training data, the evaluation data is periodically evaluated by using a verification set and drawn as a Loss image, and the model is stopped from being further trained before the evaluation Loss never shows a significant change in the direction of the added value.

4. Test model

And on the premise of not fitting, transferring the data of the test set by using the input function to test the accuracy of prediction.

5. Analysis of Experimental results

(1) Processing result output of correlation attributes

Index(['meantempm','maxtempm','mintempm','meantempm_1', 'meantempm_2','meantempm_3','meandewptm_1','meandewptm_2', 'meandewptm_3','meanpressurem_1', 'meanpressurem_2','meanpressurem_3','maxhumidity_1','maxhumidity_2','maxhum idity_3','minhumidity_1','minhumidity_2','minhumidity_3','maxtempm_1', 'maxtempm_2', 'maxtempm_3','mintempm_1','mintempm_2','mintempm_3','maxdewptm_1','maxd ewptm_2','maxdewptm_3','mindewptm_1','mindewptm_2','mindewptm_3', 'maxpressurem_1', 'maxpressurem_2','maxpressurem_3','minpressurem_1','minpressurem_2','minpress urem_3','precipm_1','precipm_2','precipm_3'],

dtype＝'object')

(2) Data normalized type output

The object data type is converted to the required float type herein.

(3) Result of processing extreme outliers

As shown in Table 1, it can be seen that the minimum pressure quartile range is 0.33 (75 percent and 25 percent difference), and the minimum value (28.72) is less than the 3 quartile range (0.99) of the 25 th percentile (29.72); the quartile range for precipitation was 0.03, with the maximum value (3.40) being greater than the 3 quartile ranges (0.09) for the 75 th percentile (0.03). There are abnormal values for the characteristic categories of minimum pressure and precipitation, and for the convenience of research, the form of histogram will be used herein. As shown in fig. 5(a) and 5 (b).

It can be seen from fig. 5(a) that the minimum pressure profile data is multimodal, so it can be concluded that there are two very different sets of environmental conditions in this data, which can be met between seasons of the year, so the occurrence of these extreme outliers can be considered reasonable, and this profile is not deleted for the time being.

As can be seen from the precipitation characteristic data distribution in fig. 5(b), most of the data values are close to 0; it can be understood that since the number of days to dry (i.e. no precipitation) is more frequent in this region, most of the days precipitation approaches 0, so it is normal to see outliers that overall favor 0, so this feature need not be deleted either.

TABLE 1 correlation attributes containing extreme outliers

(4) Processing missing data values

The results are shown in FIG. 6.

It can be seen from the type output of the data that since the data is completely saved and there is no missing value, only the first three days need to be excluded from the data set. As can be seen from fig. 6, 2997 characteristic data were obtained. The final result of data preprocessing is obtained.

(5) Overfitting model

As seen in fig. 7, the experiment did not over-configure the model after all loop iterations, since the overall trend is a decline in evaluation loss despite the increasing and decreasing variation in intermediate volatility. So that the model can continue to be predicted and evaluated from the remaining test data sets.

(6) Results of the prediction model

Interpreting the variance: 0.90;

mean absolute error: 2.32 degrees celsius (4.17 degrees fahrenheit);

median absolute error: 1.85 degrees celsius (3.33 degrees fahrenheit).

Where a value of 0.90 is interpreted such that the final model herein explains about 90% of the observed resulting variable mearemempm, the model fits well.

The experimental result shows that the fitting degree of the neural network is better, the error value is smaller, and the prediction accuracy of the neural network model is higher.

Claims

1. A meteorological temperature prediction method is characterized in that: the method comprises the following steps:

s1: collecting meteorological data;

s4: setting parameters of a neural network model;

s5: setting input function parameters;

s6: training the neural network model;

s8: evaluating the trained neural network model by using the test set;

2. The meteorological temperature prediction method according to claim 1, wherein: in step S2, the process of selecting valid data is as follows: and (4) evaluating the quality of the meteorological data collected in the step (S1), deleting uninteresting related attributes, adding derivative features to be evaluated, unifying data types, and processing extreme abnormal values and data missing values.

3. The meteorological temperature prediction method according to claim 2, wherein: the processing of the extreme abnormal value is to generate a two-dimensional table data structure (DataFrame) by using a descriptor function, wherein the two-dimensional table comprises a count, an average value, a standard deviation, a minimum value, a 25 th percentile, a 50 th percentile, a 75 th percentile and a maximum value, and if the meteorological data is less than the 25 th percentile or greater than the 75 th percentile, the meteorological data is judged to be the extreme abnormal value.

4. The meteorological temperature prediction method according to claim 2, wherein: the processing of the missing data value is to fill in the missing value by using an interpolation value, wherein the interpolation value is an average value of two data before and after the missing value or an average value of the average value in a period of time.

5. The meteorological temperature prediction method according to claim 1, wherein: in step S3, the data in the training set accounts for 80% of the data in the data set, the data in the testing set accounts for 10% of the data in the data set, and the data in the verification set accounts for 10% of the data in the data set.

6. The meteorological temperature prediction method according to claim 1, wherein: the step S4 specifically includes the following steps: the depth of the neural network model is set to be two layers, the width is set to be 50 nodes, the storage position of data in the neural network model is specified, and a ReLU function is adopted as an activation function.

7. The meteorological temperature prediction method according to claim 1, wherein: the input function parameters in the step S5 include the following five: input interface type, target value, number of times the entire data set has been executed, whether a batch of subsets is randomly selected before each execution, number of samples participating in each execution.

8. The meteorological temperature prediction method according to claim 1, wherein: the step S6 specifically includes the following steps: training a neural network model through a training set, iterating for 50 times, selecting training records of random batches, pushing the training records through a network, recording a loss function for each iteration, and adjusting the association weight between two connected neurons on the basis of evaluating whether loss is reduced and the logic of a neural network optimizer.