CN116796894A - Construction method of efficient deep learning weather prediction model - Google Patents

Construction method of efficient deep learning weather prediction model Download PDF

Info

Publication number
CN116796894A
CN116796894A CN202310720829.8A CN202310720829A CN116796894A CN 116796894 A CN116796894 A CN 116796894A CN 202310720829 A CN202310720829 A CN 202310720829A CN 116796894 A CN116796894 A CN 116796894A
Authority
CN
China
Prior art keywords
data
value
weather
weather prediction
deep learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310720829.8A
Other languages
Chinese (zh)
Inventor
晏星
蒋一泽
罗娜娜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Normal University
Original Assignee
Beijing Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Normal University filed Critical Beijing Normal University
Priority to CN202310720829.8A priority Critical patent/CN116796894A/en
Publication of CN116796894A publication Critical patent/CN116796894A/en
Pending legal-status Critical Current

Links

Landscapes

  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a construction method of a high-efficiency deep learning weather prediction model, which belongs to the technical field of weather prediction and comprises the following specific steps: (1) Preprocessing original meteorological data and carrying out data preprocessing on the original meteorological data; (2) Analyzing and recording the correlation of the processed original meteorological data; (3) Constructing a predictive weather model according to each group of processed data; (4) Collecting weather prediction model prediction data and performing block storage; (5) Collecting and analyzing an operation log stored by a management platform; according to the method, the weather big data can be efficiently operated under the condition of limited system resources, irrelevant attributes are effectively removed, data is reduced, a high-quality data source is provided for subsequent model construction, the workload of workers can be reduced, the model construction efficiency is improved, the operation difficulty is reduced, the use experience is improved, meanwhile, the parameter accuracy is improved, and the weather prediction accuracy is ensured.

Description

Construction method of efficient deep learning weather prediction model
Technical Field
The invention relates to the technical field of weather prediction, in particular to a method for constructing a high-efficiency deep learning weather prediction model.
Background
With the rapid development of computer technology, deep learning opens a new era of artificial intelligence. Represented by deep learning, with breakthrough progress in the fields of computer vision, voice recognition, natural language processing and the like, the new technology innovation brings challenges and opportunities for the development of meteorological prediction technology. Weather phenomenon is closely related to aspects of production and life of human beings, social economy, military operations and the like, and the weather directly influences the health of each organism; with the continuous improvement of information storage, sensor detection and other technologies, meteorological equipment is increasingly abundant, meteorological data materials which can be collected by a meteorological department are more and more, and contents and types are also more and more abundant, so that a higher standard is provided for a meteorological big data analysis processing technology. The research on the high-resolution short-time refined weather forecast technology is the development direction of future weather forecast; therefore, it becomes particularly important to invent a construction method of a high-efficiency deep learning weather prediction model.
The existing method for constructing the weather prediction model cannot enable weather big data to run efficiently under limited system resources, and meanwhile, the data is easy to have redundancy and cannot guarantee the quality of the data; in addition, the existing method for constructing the weather prediction model has low model construction efficiency, higher operation difficulty and low parameter accuracy; therefore, we propose a method for constructing a weather prediction model by high-efficiency deep learning.
Disclosure of Invention
The invention aims to solve the defects in the prior art, and provides a method for constructing a high-efficiency deep learning weather prediction model.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a construction method of a high-efficiency deep learning weather prediction model comprises the following specific steps:
(1) Preprocessing original meteorological data and carrying out data preprocessing on the original meteorological data;
(2) Analyzing and recording the correlation of the processed original meteorological data;
(3) Constructing a predictive weather model according to each group of processed data;
(4) Collecting weather prediction model prediction data and performing block storage;
(5) And collecting and analyzing the operation log stored by the management platform.
As a further aspect of the present invention, the specific steps of the data preprocessing in the step (1) are as follows:
step one: carding different interfaces, acquiring reasons for different original meteorological data of each group, acquiring and judging data source information of each group of data according to a carding result, and classifying the acquired original meteorological data of each group according to different data sources;
step two: when a defect value exists, replacing the defect value with an average value of a front value and a rear value or an average value of a certain time interval; when a singular value exists, resetting the value to a front-to-back value or an average value over a period of time; when the non-numerical number exists, carrying out data format coding on the data;
step three: and carrying out data cleaning on the original meteorological data through code conversion and format processing, carrying out characteristic dimension reduction processing on the cleaned data, and carrying out standardization processing on the original meteorological data through a Z-SCORE standardization method to obtain sample data.
As a further scheme of the invention, the specific calculation formula of the Z-SCORE normalization method in the step three is as follows:
wherein sigma is the standard deviation of the original meteorological data; v n Data deviation of original meteorological data; x represents the characteristic parameters of the original meteorological data;representing the average of all raw meteorological data.
As a further scheme of the invention, the correlation analysis of the original meteorological data in the step (2) comprises the following specific steps:
step I: initializing a multidimensional candidate variable set and an equal output conversion characteristic set, and then screening characteristic attributes of each group of sample data through priori knowledge and mutual information method;
step II: and comparing the maximum information coefficient of each group of characteristic attributes to determine the association degree between the attributes, selecting the characteristic attributes with higher association degree for reservation, continuously repeating the steps until the final result meets the MMIS evaluation standard, and then outputting the filtered equal output conversion characteristic set.
As a further scheme of the invention, the specific calculation formula of the maximum information coefficient in the step II is as follows:
wherein U and V respectively represent different characteristic attributes, and p (U, V) is the joint probability of two groups of edge probabilities p (U) and p (V);
the specific calculation formula of the MMFS evaluation criteria in the step II is as follows:
where β represents a metric parameter for evaluating correlation and redundancy, and G is an output value.
As a further scheme of the invention, the specific construction steps of the predictive weather model in the step (3) are as follows:
step (1): setting a group of convolutional neural networks, dividing the screened characteristic attributes into a training set and a testing set, transmitting the training set into the convolutional neural networks as input, acquiring linear combinations with hidden nodes as output layers, and defining energy functions of the convolutional neural networks after multiple rounds of learning by adopting a least square method;
step (2): when the energy function is smaller than the target error, the training process is finished, a predicted meteorological model is output, a test set is led into the predicted meteorological model for testing, the loss value of the predicted meteorological model is calculated, if the loss value does not meet the expected value, a network connection weight is initialized in a specified interval of the predicted meteorological model, then training samples are submitted from a set of input and output pairs during training, the output of the predicted meteorological model is calculated, the expected network output and the actual network output are compared, and meanwhile, the local error of all neurons is calculated;
step (3): training and updating the weight of the predicted meteorological model according to a learning rule equation after the local error exceeds a preset threshold value of a worker, listing all possible data results according to a preset learning rate and step length, selecting any subset as a test set for each group of data, taking the rest subset as a training set, detecting the test set after the test model is trained, and counting the root mean square error of the detection results;
step (4): and replacing the test set with another subset, taking the rest subset as a training set, counting root mean square errors again until all data are predicted once, and selecting the corresponding combined parameter with the minimum root mean square error as the optimal parameter in the data interval and replacing the original parameters of the predicted meteorological model.
As a further aspect of the present invention, the specific steps of the blocking storage in the step (4) are as follows:
the first step: preprocessing the predicted data into a block meeting the condition, generating a local public and private key pair by each node in a block chain network as an identification code in the network when the block is accessed to the network, and broadcasting a leader application to other nodes in the network and transmitting the leader application when one node waits for the local role to become a candidate node;
and a second step of: when the candidate node becomes a leading node, the other nodes become following nodes, the leading node broadcasts the block record information, the following nodes broadcast the received information to the other following nodes after receiving the information, record the repetition times, and generate a block head by using the information with the largest repetition times;
and a third step of: and after the verification is passed, the leader node sends an addition command and enters a sleep stage, and after the follow-up node receives the confirmation information, each newly generated block group is added to the block chain and returns to the candidate identity.
As a further aspect of the present invention, the specific analysis steps of the operation log in the step (5) are as follows:
step I: disposing related log acquisition plug-ins on management platforms of different systems or acquiring operation logs recorded in the management platforms through a syslog server, and screening the operation logs meeting preset conditions;
step II: processing the screened operation log into log data in a unified format, matching the operation behavior recorded in the processed operation log with abnormal behavior characteristics, and generating corresponding alarm information according to a matching result;
and III, step III: and calculating the risk scores of the alarm information, outputting a calculation result, feeding the alarm information back to related maintenance personnel, and interrupting related operation processes.
Compared with the prior art, the invention has the beneficial effects that:
1. compared with the traditional construction method, the construction method of the high-efficiency deep learning weather prediction model classifies the acquired original weather data of each group according to different data sources, and carries out corresponding processing on the data with defect values, singular values and non-numerical digits. And then carrying out data cleaning on the original meteorological data through code conversion and format processing, carrying out feature dimension reduction processing and standardization processing on the cleaned data, initializing a multi-dimensional candidate variable set and an equal output conversion feature set, screening the feature attributes of each group of sample data through priori knowledge and a mutual information method, comparing the maximum information coefficient of each group of feature attributes to determine the association degree among the attributes, selecting the feature attributes with higher association degree for reservation, continuously repeating the steps until a final result meets MMIS evaluation standards, and outputting the screened equal output conversion feature set, so that the meteorological big data can efficiently operate under limited system resources, simultaneously effectively eliminating irrelevant attributes, finely reducing data, and providing a high-quality data source for subsequent model construction.
2. According to the method for constructing the efficient deep learning weather prediction model, the screened characteristic attribute is divided into the training set and the testing set, the training set is transmitted to the convolutional neural network to obtain relevant linear combinations, after an energy function meets preset conditions, the training process is finished, the predicted weather model is output, the testing set is led into the predicted weather model to conduct testing, the predicted weather model loss value is calculated, if the loss value does not meet an expected value, network connection weight is initialized in a specified interval of the predicted weather model, expected network output and actual network output are compared, local errors of all neurons are calculated, after the local errors exceed a preset threshold value, the weight of the predicted weather model is trained and updated, all possible data results are obtained, corresponding combination parameters are selected as optimal parameters in a data interval through a cross verification method, the original parameters of the predicted weather model are replaced, the working load of workers is reduced, the model construction efficiency is improved, the use experience is improved, and meanwhile accuracy of the parameters is guaranteed.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention.
FIG. 1 is a block flow diagram of a method for constructing a high-efficiency deep learning weather prediction model according to the present invention.
Detailed Description
Example 1
Referring to fig. 1, a method for constructing a weather prediction model by high-efficiency deep learning specifically includes the following steps:
the original meteorological data is subjected to data preprocessing.
Specifically, carding different interfaces, acquiring reasons for different types of original meteorological data, acquiring and judging data source information of each group of data according to carding results, classifying each group of acquired original meteorological data according to different data sources, and replacing a defect value with an average value of a front value and a rear value or an average value of a certain time interval when the defect value exists; when a singular value exists, resetting the value to a front-to-back value or an average value over a period of time; when the non-numerical digits exist, the data are subjected to data format coding, data cleaning is carried out on the original meteorological data through code conversion and format processing, then feature dimension reduction processing is carried out on the cleaned data, and the original meteorological data are subjected to standardization processing through a Z-SCORE standardization method so as to obtain sample data.
Specifically, the specific calculation formula of the Z-SCORE standardization method is as follows:
wherein sigma is the standard deviation of the original meteorological data; v n Data deviation of original meteorological data; x represents the characteristic parameters of the original meteorological data;representing the average of all raw meteorological data.
And analyzing and recording the correlation of the processed original meteorological data.
Specifically, initializing a multi-dimensional candidate variable set and an equal output conversion feature set, screening the feature attributes of each group of sample data through priori knowledge and a mutual information method, comparing the maximum information coefficient of each group of feature attributes to determine the association degree among the attributes, selecting the feature attributes with higher association degree for reservation, continuously repeating the steps until a final result meets MMIS evaluation criteria, and outputting the screened equal output conversion feature set.
In this embodiment, the specific calculation formula of the maximum information coefficient is as follows:
wherein U and V respectively represent different characteristic attributes, and p (U, V) is the joint probability of two groups of edge probabilities p (U) and p (V);
the specific calculation formula of the MMFS evaluation criteria is as follows:
where β represents a metric parameter for evaluating correlation and redundancy, and G is an output value.
And constructing a predictive weather model according to each group of processed data.
Specifically, a set of convolutional neural network is set, the screened characteristic attribute is divided into a training set and a test set, the training set is used as input to the convolutional neural network, linear combination with hidden nodes at the output layer is obtained, a least square method is adopted to define energy functions of the convolutional neural network after multiple rounds of learning, when the energy functions are smaller than target errors, the training process is finished, the predicted weather model is output, the test set is imported into the predicted weather model for testing, the loss value of the predicted weather model is calculated, if the loss value does not meet the expected value, network connection weight is initialized in a specified interval of the predicted weather model, training samples are submitted from a set of input and output pairs during training, the output of the predicted weather model is calculated, the expected network output and the actual network output are compared, local errors of all neurons are calculated, after the local errors exceed a preset threshold value of staff, training and updating are carried out on the predicted weather model according to a learning rule equation, all possible data are listed according to preset learning rate and step length, and the root mean square error is selected as the optimal test parameter set, the rest of the test set is replaced by the corresponding test set, the test set is replaced by the optimal test set, the rest of the test set is used as the rest of the test parameter set, and the rest of the test set is replaced by the optimal test set, and the rest of the test set is used as the rest of the test parameter set, and the rest of the test set is replaced by the optimal test set.
Example 2
Referring to fig. 1, a method for constructing a weather prediction model by high-efficiency deep learning specifically includes the following steps:
and collecting weather prediction model prediction data and performing block storage.
Specifically, the prediction data is preprocessed into blocks meeting the conditions, when the blocks enter the network, each node in the blockchain network generates a local public and private key pair as an identification code in the network, when one node waits for a local role to become a candidate node, a leader application is broadcast to other nodes in the network and sent, after the candidate node becomes the leader node, the other nodes become follower nodes, the leader node broadcasts the block record information, the follower nodes broadcast the received information to the other follower nodes after receiving the information and record the repetition times, a block head is generated by using the information with the maximum repetition times, a verification application is sent to the leader node, after verification is passed, the leader node sends an addition command and enters a falling period, and after the follower node receives the confirmation information, each newly generated block is added to the blockchain and returns to the candidate identity.
And collecting and analyzing the operation log stored by the management platform.
Specifically, relevant log acquisition plug-ins are deployed on management platforms of different systems, or operation logs recorded in the management platforms are obtained through a syslog server, operation logs meeting preset conditions are screened out, the screened operation logs are processed into log data in a unified format, operation behaviors recorded in the processed operation logs are matched with abnormal behavior features, corresponding alarm information is generated according to the matching results, risk scores of the alarm information are calculated, calculation results are output, and the alarm information is fed back to relevant maintenance personnel and related operation processes are interrupted.

Claims (8)

1. The method for constructing the efficient deep learning weather prediction model is characterized by comprising the following specific steps of:
(1) Preprocessing original meteorological data and carrying out data preprocessing on the original meteorological data;
(2) Analyzing and recording the correlation of the processed original meteorological data;
(3) Constructing a predictive weather model according to each group of processed data;
(4) Collecting weather prediction model prediction data and performing block storage;
(5) And collecting and analyzing the operation log stored by the management platform.
2. The method for constructing a highly efficient deep learning weather prediction model according to claim 1, wherein the specific steps of data preprocessing in the step (1) are as follows:
step one: carding different interfaces, acquiring reasons for different original meteorological data of each group, acquiring and judging data source information of each group of data according to a carding result, and classifying the acquired original meteorological data of each group according to different data sources;
step two: when a defect value exists, replacing the defect value with an average value of a front value and a rear value or an average value of a certain time interval; when a singular value exists, resetting the value to a front-to-back value or an average value over a period of time; when the non-numerical number exists, carrying out data format coding on the data;
step three: and carrying out data cleaning on the original meteorological data through code conversion and format processing, carrying out characteristic dimension reduction processing on the cleaned data, and carrying out standardization processing on the original meteorological data through a Z-SCORE standardization method to obtain sample data.
3. The method for constructing a highly efficient deep learning weather prediction model according to claim 2, wherein the specific calculation formula of the Z-SCORE normalization method in the third step is as follows:
wherein sigma is the standard deviation of the original meteorological data; v n Data deviation of original meteorological data; x represents the characteristic parameters of the original meteorological data;representing the average of all raw meteorological data.
4. The method for constructing a highly efficient deep learning weather prediction model according to claim 2, wherein the step (2) of analyzing the correlation of the raw weather data is specifically as follows:
step I: initializing a multidimensional candidate variable set and an equal output conversion characteristic set, and then screening characteristic attributes of each group of sample data through priori knowledge and mutual information method;
step II: and comparing the maximum information coefficient of each group of characteristic attributes to determine the association degree between the attributes, selecting the characteristic attributes with higher association degree for reservation, continuously repeating the steps until the final result meets the MMIS evaluation standard, and then outputting the filtered equal output conversion characteristic set.
5. The method for constructing a highly efficient deep learning weather prediction model according to claim 4, wherein the specific calculation formula of the maximum information coefficient in step ii is as follows:
wherein U and V respectively represent different characteristic attributes, and p (U, V) is the joint probability of two groups of edge probabilities p (U) and p (V);
the specific calculation formula of the MMFS evaluation criteria in the step II is as follows:
where β represents a metric parameter for evaluating correlation and redundancy, and G is an output value.
6. The method for constructing a highly efficient deep learning weather prediction model according to claim 4, wherein the specific construction steps of the prediction weather model in the step (3) are as follows:
step (1): setting a group of convolutional neural networks, dividing the screened characteristic attributes into a training set and a testing set, transmitting the training set into the convolutional neural networks as input, acquiring linear combinations with hidden nodes as output layers, and defining energy functions of the convolutional neural networks after multiple rounds of learning by adopting a least square method;
step (2): when the energy function is smaller than the target error, the training process is finished, a predicted meteorological model is output, a test set is led into the predicted meteorological model for testing, the loss value of the predicted meteorological model is calculated, if the loss value does not meet the expected value, a network connection weight is initialized in a specified interval of the predicted meteorological model, then training samples are submitted from a set of input and output pairs during training, the output of the predicted meteorological model is calculated, the expected network output and the actual network output are compared, and meanwhile, the local error of all neurons is calculated;
step (3): training and updating the weight of the predicted meteorological model according to a learning rule equation after the local error exceeds a preset threshold value of a worker, listing all possible data results according to a preset learning rate and step length, selecting any subset as a test set for each group of data, taking the rest subset as a training set, detecting the test set after the test model is trained, and counting the root mean square error of the detection results;
step (4): and replacing the test set with another subset, taking the rest subset as a training set, counting root mean square errors again until all data are predicted once, and selecting the corresponding combined parameter with the minimum root mean square error as the optimal parameter in the data interval and replacing the original parameters of the predicted meteorological model.
7. The method for constructing a highly efficient deep learning weather prediction model according to claim 1, wherein the block storage in step (4) comprises the following specific steps:
the first step: preprocessing the predicted data into a block meeting the condition, generating a local public and private key pair by each node in a block chain network as an identification code in the network when the block is accessed to the network, and broadcasting a leader application to other nodes in the network and transmitting the leader application when one node waits for the local role to become a candidate node;
and a second step of: when the candidate node becomes a leading node, the other nodes become following nodes, the leading node broadcasts the block record information, the following nodes broadcast the received information to the other following nodes after receiving the information, record the repetition times, and generate a block head by using the information with the largest repetition times;
and a third step of: and after the verification is passed, the leader node sends an addition command and enters a sleep stage, and after the follow-up node receives the confirmation information, each newly generated block group is added to the block chain and returns to the candidate identity.
8. The method for constructing a highly efficient deep learning weather prediction model according to claim 1, wherein the specific analysis steps of the operation log in the step (5) are as follows:
step I: disposing related log acquisition plug-ins on management platforms of different systems or acquiring operation logs recorded in the management platforms through a syslog server, and screening the operation logs meeting preset conditions;
step II: processing the screened operation log into log data in a unified format, matching the operation behavior recorded in the processed operation log with abnormal behavior characteristics, and generating corresponding alarm information according to a matching result;
and III, step III: and calculating the risk scores of the alarm information, outputting a calculation result, feeding the alarm information back to related maintenance personnel, and interrupting related operation processes.
CN202310720829.8A 2023-06-16 2023-06-16 Construction method of efficient deep learning weather prediction model Pending CN116796894A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310720829.8A CN116796894A (en) 2023-06-16 2023-06-16 Construction method of efficient deep learning weather prediction model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310720829.8A CN116796894A (en) 2023-06-16 2023-06-16 Construction method of efficient deep learning weather prediction model

Publications (1)

Publication Number Publication Date
CN116796894A true CN116796894A (en) 2023-09-22

Family

ID=88049102

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310720829.8A Pending CN116796894A (en) 2023-06-16 2023-06-16 Construction method of efficient deep learning weather prediction model

Country Status (1)

Country Link
CN (1) CN116796894A (en)

Similar Documents

Publication Publication Date Title
CN113378990B (en) Flow data anomaly detection method based on deep learning
CN110636066B (en) Network security threat situation assessment method based on unsupervised generative reasoning
CN112910690A (en) Network traffic prediction method, device and equipment based on neural network model
CN114757432A (en) Future execution activity and time prediction method and system based on flow log and multi-task learning
CN116843955A (en) Microorganism classification and identification method and system based on computer vision
CN113328755A (en) Compressed data transmission method facing edge calculation
CN117421684B (en) Abnormal data monitoring and analyzing method based on data mining and neural network
CN117828539B (en) Intelligent data fusion analysis system and method
CN117349782A (en) Intelligent data early warning decision tree analysis method and system
CN111506635A (en) System and method for analyzing residential electricity consumption behavior based on autoregressive naive Bayes algorithm
CN113485863A (en) Method for generating heterogeneous unbalanced fault samples based on improved generation countermeasure network
CN115758908A (en) Alarm online prediction method under alarm flooding condition based on deep learning
CN117453764A (en) Data mining analysis method
CN111221915B (en) Online learning resource quality analysis method based on CWK-means
CN109635008B (en) Equipment fault detection method based on machine learning
CN116681556A (en) Intelligent remote teaching cloud platform system and method based on education big data
CN116883128A (en) Method and device for excavating money laundering bulk, electronic equipment and computer storage medium
CN116545679A (en) Industrial situation security basic framework and network attack behavior feature analysis method
CN116796894A (en) Construction method of efficient deep learning weather prediction model
CN115392710A (en) Wind turbine generator operation decision method and system based on data filtering
CN115438101B (en) Data feature construction system and method based on feature morphology and data relationship
CN112312454A (en) Method for judging KPI (Key Performance indicator) and weak coverage relation of LTE (Long term evolution) user
CN116993307B (en) Collaborative office method and system with artificial intelligence learning capability
CN117667495B (en) Association rule and deep learning integrated application system fault prediction method
CN117057777A (en) Comprehensive management method for operation and maintenance of power distribution network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination