CN116860824A - Meteorological data knowledge discovery method and system - Google Patents
Meteorological data knowledge discovery method and system Download PDFInfo
- Publication number
- CN116860824A CN116860824A CN202310623030.7A CN202310623030A CN116860824A CN 116860824 A CN116860824 A CN 116860824A CN 202310623030 A CN202310623030 A CN 202310623030A CN 116860824 A CN116860824 A CN 116860824A
- Authority
- CN
- China
- Prior art keywords
- data
- weather
- knowledge
- algorithm
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 77
- 238000005516 engineering process Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 20
- 238000012549 training Methods 0.000 claims abstract description 15
- 238000004458 analytical method Methods 0.000 claims abstract description 12
- 238000013135 deep learning Methods 0.000 claims abstract description 12
- 238000013473 artificial intelligence Methods 0.000 claims abstract description 11
- 238000004140 cleaning Methods 0.000 claims abstract description 11
- 230000002159 abnormal effect Effects 0.000 claims abstract description 10
- 238000003908 quality control method Methods 0.000 claims abstract description 10
- 238000010801 machine learning Methods 0.000 claims abstract description 8
- 230000007613 environmental effect Effects 0.000 claims abstract description 6
- 238000005070 sampling Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 16
- 238000012937 correction Methods 0.000 claims description 15
- 238000007781 pre-processing Methods 0.000 claims description 12
- 238000010606 normalization Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 5
- 238000002759 z-score normalization Methods 0.000 claims description 5
- 238000013461 design Methods 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 abstract description 3
- 238000013527 convolutional neural network Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000007418 data mining Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000015654 memory Effects 0.000 description 4
- 238000003860 storage Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012952 Resampling Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- Fuzzy Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a meteorological data knowledge discovery method and system, which are based on multi-source heterogeneous meteorological data including space-time observation data, historical analysis data and other data, carry out data cleaning and data quality control processing on missing data and abnormal values through statistics and a machine learning algorithm, and form a complete and accurate intelligent algorithm training data set by combining with a unified space-time technology of the meteorological data; constructing a plurality of algorithm models based on artificial intelligence and deep learning technology to form an algorithm pool, enabling a data set to form knowledge such as strong convection weather identification, accurate weather forecast and the like through the algorithm pool, and storing the knowledge in a weather characteristic knowledge data set to support environmental information application; and applying the obtained knowledge product to data, and helping a decision maker to make corresponding decisions according to a weather space-time evolution rule generated by the knowledge product, so as to reduce safety uncertainty related to weather. The invention focuses more on the conversion from data to knowledge, and has higher accuracy than the traditional machine learning.
Description
Technical Field
The invention relates to a weather data knowledge discovery method and system, and belongs to the field of weather service.
Background
The data volume in the meteorological field is increasingly larger, and with the continuous development of technologies such as meteorological instruments, sensors, meteorological satellites and the like, the data volume in the meteorological field also has a tendency of explosive growth. These data, including observations of various weather parameters, pattern forecast results, etc., make efficient use of these data a challenge for weather research. Despite the continual advances in meteorological science and technology, there is room for improvement in accuracy and efficiency of weather forecast. The accuracy and efficiency of weather forecast are not only related to the production and life of people. With the continuous development of artificial intelligence technology, the application of the artificial intelligence technology in the meteorological field is also becoming wider and wider. The artificial intelligence technology can automatically discover the association, the mode and the rule in the data, thereby improving the accuracy and the efficiency of weather forecast.
The data formats and data structures of different meteorological data sources in the traditional meteorological data mining system are different, and the lack of unified standards and semantics among different meteorological data leads to difficulties in data integration and analysis. And the traditional meteorological data mining system often uses a single algorithm or a simple combination algorithm, lacks flexibility and expandability, and cannot meet the complex data mining requirements in different scenes.
Disclosure of Invention
The invention provides a meteorological data knowledge discovery method and system, which solve the problems disclosed in the background technology.
In order to solve the technical problems, the invention adopts the following technical scheme:
the weather data knowledge discovery method is characterized by comprising the following steps of:
1) Based on multi-source heterogeneous meteorological data, including space-time observation data and historical analysis data, performing data cleaning and data quality control processing on missing data and abnormal values through statistics and a machine learning algorithm, and combining with a unified space-time technology of the meteorological data to form an intelligent algorithm training data set;
2) Constructing a plurality of algorithm models based on artificial intelligence and deep learning technology to form an algorithm pool, carrying out strong convection weather identification and weather forecast on the intelligent algorithm training data set obtained in the step 1) through the algorithm pool to form knowledge, and storing the knowledge in a pre-constructed weather feature knowledge data set for supporting environmental information application;
3) And (3) carrying out data application on the weather characteristic knowledge data set obtained in the step (2) and generating a weather space-time evolution rule according to a knowledge product.
Further, in the step 1), the data cleaning process is as follows: and carrying out de-duplication, de-noising, error correction and filtering on the acquired meteorological data.
Further, in the step 1), the process of processing the missing data and the outlier by the data quality control is as follows: converting the data of different dimensions into uniform dimensions based on the maximum-minimum normalization and the Z-score normalization;
filling the missing values by using linear interpolation, polynomial interpolation and spline interpolation;
for data which is too dense, reducing the data volume through random sampling, hierarchical sampling and clustering sampling;
the raw data is converted into an analyzed and mined form.
Further, the weather data unified space-time technology in the step 1) comprises the following steps:
s31: unifying space-time coordinates;
s32: the time resolution is uniform;
s33: the spatial resolution is uniform;
s34: the data format is uniform.
Further, the algorithm model of the weather space-time evolution rule generated in the step 2) comprises a pattern correction algorithm model, a time sequence prediction algorithm model and an intelligent recognition algorithm model.
Accordingly, a weather data knowledge discovery system, comprising:
the data preprocessing module is used for processing missing data and abnormal values through data cleaning and data quality control by means of statistics and machine learning algorithms based on multi-source heterogeneous meteorological data, wherein the multi-source heterogeneous meteorological data comprises space-time observation data and historical analysis data, and an intelligent algorithm training data set is formed by combining meteorological data with a unified space-time technology;
the knowledge discovery module is used for constructing a plurality of algorithm models based on artificial intelligence and deep learning technology to form an algorithm pool, carrying out strong convection weather identification and weather forecast on an intelligent algorithm training data set through the algorithm pool to form knowledge, and storing the knowledge in a pre-constructed weather characteristic knowledge data set for supporting environmental information application;
and the data application module is used for carrying out data application on the weather characteristic knowledge data set and generating a weather space-time evolution rule according to the knowledge product.
Further, in the data preprocessing module, the data cleaning process is as follows: and carrying out de-duplication, de-noising, error correction and filtering on the acquired meteorological data.
Further, the process of processing missing data and abnormal values by data quality control in the data preprocessing module is as follows: converting the data of different dimensions into uniform dimensions based on the maximum-minimum normalization and the Z-score normalization;
filling the missing values by using linear interpolation, polynomial interpolation and spline interpolation;
for data which is too dense, reducing the data volume through random sampling, hierarchical sampling and clustering sampling;
the raw data is converted into an analyzed and mined form.
Further, the unified space-time technology of meteorological data in the data preprocessing module comprises the following steps:
s31: unifying space-time coordinates;
s32: the time resolution is uniform;
s33: the spatial resolution is uniform;
s34: the data format is uniform.
10. The weather data knowledge discovery hierarchy design system of claim 6, wherein:
the algorithm model of the weather space-time evolution rule generated in the knowledge discovery module comprises a mode correction algorithm model, a time sequence prediction algorithm model and an intelligent recognition algorithm model.
The invention has the beneficial effects that: compared with the prior art, the technical scheme solves the problems of different data formats, single algorithm use and the like in the traditional meteorological data mining system, can effectively improve the accuracy of acquiring knowledge products, and has the main innovation points that:
1) Different meteorological data are integrated into a unified space-time data set by utilizing a unified space-time technology of the meteorological data, and a data base is provided for meteorological analysis and application.
2) The intelligent algorithm component library is provided, a plurality of algorithm models are constructed based on artificial intelligence and deep learning technology to form an algorithm pool, and flexibility and expandability are improved.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a diagram of a meteorological feature knowledge data set construction process in accordance with the present invention;
FIG. 3 is a flow chart of the application of the knowledge product of the present invention in track planning;
FIG. 4 is a detailed flow chart of the weather data knowledge discovery system of the invention for application.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and are not intended to limit the scope of the present invention.
As shown in FIG. 1, the weather data knowledge discovery method of the invention comprises the following steps:
s1: the data preprocessing module is used for processing missing data and abnormal values through data cleaning and data quality control by means of statistics and machine learning algorithms based on data such as multi-source heterogeneous meteorological data including space-time observation data, historical analysis data and the like, and forming a complete and accurate intelligent algorithm training data set by combining a unified space-time technology of the meteorological data;
s2: the knowledge discovery module constructs a plurality of algorithm models based on artificial intelligence and deep learning technology to form an algorithm pool, and the intelligent algorithm training data set obtained in the step S1 can be used for forming knowledge such as strong convection weather identification, accurate weather forecast and the like through the algorithm pool and is stored in a weather feature knowledge data set so as to support environmental information application;
s3: and the data application module is used for applying the weather characteristic knowledge data set obtained in the step S2 to data, and helping a decision maker to make a corresponding decision according to a weather space-time evolution rule generated by a knowledge product, so that safety uncertainty related to weather is reduced.
Steps S1, S2 and S3 are sequentially executed;
further, the data preprocessing module processes the multi-source heterogeneous original meteorological data to improve availability and reliability of the multi-source heterogeneous original meteorological data, lays a foundation for subsequent mining and analysis, and comprises the following steps:
s21: the data is cleaned, the collected meteorological data is subjected to processing such as de-duplication, de-noising, error correction and filtering, abnormal values, error values and the like in the data are avoided, and the reliability and usability of the data are improved;
s22: data normalization, namely converting data with different dimensions into uniform dimensions based on methods such as maximum-minimum normalization, Z-score normalization and the like so as to facilitate data analysis and comparison;
s23: data interpolation, the condition that missing values often exist in meteorological data, and interpolation methods such as linear interpolation, polynomial interpolation, spline interpolation and the like are required to be used for filling the missing values;
s24: data sampling, namely reducing the data volume of data which is too dense through sampling methods such as random sampling, layered sampling, clustering sampling and the like so as to improve the efficiency of data processing and analysis;
s25: data conversion, converting raw data into a form suitable for analysis and mining, such as converting meteorological data into time series data, frequency data, and the like.
Furthermore, the unified space-time technology of meteorological data is mainly used for integrating data with different meteorological data sources, different time resolutions and different spatial resolutions in a unified way, and different meteorological data can be integrated into a unified space-time data set, so that a more convenient and reliable data base is provided for subsequent meteorological analysis and application. The main technology comprises the following aspects:
s31: the space-time coordinates are unified. The meteorological data originate from different meteorological sites and satellites, and the time and space coordinates of the meteorological data are different. For the time coordinates, UTC or Greenwich mean time can be used as a unified time reference; for the space coordinates, WGS84 or chinese 2000 geodetic coordinate system or the like may be adopted as a unified coordinate reference.
S32: the time resolution is uniform. There may be differences in the time resolution of the different sources of meteorological data, for example, the observation time interval for a meteorological observation station may be 10 minutes or 1 hour, while the time interval for satellite telemetry may be 30 minutes or 1 day. For this case, interpolation or time resampling techniques may be used to unify the meteorological data at different time resolutions to the same time resolution.
S33: the spatial resolution is uniform. The spatial resolution of different sources of meteorological data varies, for example, the observation radius of a meteorological observation station may be hundreds of meters, while the resolution of satellite remote sensing data may be hundreds or thousands of meters. For this case, technologies such as interpolation or spatial resampling may be used to unify meteorological data with different spatial resolutions to the same spatial resolution.
S34: the data format is uniform. The data formats of different meteorological data sources differ, for example, the data format of a meteorological observation station may be ASCII or binary format, while the data format of satellite telemetry data may be HDF or NetCDF format. For this case, the weather data in different formats may be unified into the same format by adopting techniques such as data format conversion or data format standardization.
Further, the intelligent algorithm trains the data set, according to the user's demand, obtain the definition of the problem goal, choose the relevant element characteristic of the relevant weather phenomenon (such as wind, rain, visibility, thunderstorm, etc.), delete the unnecessary element in the original data set, get the data set after the dimension reduction; and according to the input requirements of different intelligent algorithm models in the knowledge discovery component, performing data preprocessing operation before training on the data set containing the required element characteristics to form an intelligent algorithm training data set capable of being directly input into a data mining model. The following table lists the weather support base elements and related physical quantities and their knowledge products available after processing:
further, the knowledge discovery module combines the definition of users and experts on the problem targets, selects relevant characteristic elements, selects a proper data mining algorithm from an algorithm library, performs a knowledge discovery key flow, forms knowledge and stores the knowledge in a knowledge library, and enumerates partial model models in the algorithm library:
s41: the model correction algorithm model refers to a process of correcting meteorological model output by using a deep learning technology. The weather mode refers to the description of the atmospheric phenomenon obtained by computer simulation by utilizing basic theories such as aerodynamics, thermodynamics and the like, observation data and a numerical calculation method, and is one of important tools for weather forecast and weather research. Because the weather patterns have errors with different degrees, the errors may be caused by the uncertainty of an approximation method, an initial value and a boundary condition of numerical calculation, the imperfection of a model and the like, and therefore correction of weather pattern output is needed to improve the prediction accuracy of the weather pattern output. The traditional weather pattern correction method is usually based on a machine learning method such as statistical regression, needs to manually select features and build models, and is difficult to process for complex weather phenomena and high-dimensional data space. The weather pattern correction method based on deep learning can automatically learn the mapping relation between the weather pattern output and the actual observation through training the deep neural network, so as to correct the weather pattern output. The deep neural network can adaptively extract characteristics in meteorological mode output, so that the problem of manually selecting the characteristics is avoided, meanwhile, a high-dimensional data space can be processed, and the accuracy and the efficiency of meteorological mode correction are improved;
s42: the time sequence prediction algorithm model is used for learning the historical characteristics of meteorological data by establishing a neural network model and predicting future meteorological data according to the learned rule. Deep learning models are typically composed of multiple levels of neurons, including an input layer, a hidden layer, and an output layer. In weather timing prediction, past time series data are generally taken as input, the rules and features of these data are learned through a neural network, and then weather data for a future time period are predicted from the learned knowledge. Specifically, the weather timing prediction may be implemented using a deep learning model such as a Recurrent Neural Network (RNN) or a Convolutional Neural Network (CNN). In the RNN model, by introducing a cyclic unit, information of past time is transferred to the current time so that the model can take into consideration the correlation in time, thereby being suitable for time series prediction. In meteorological timing predictions, various RNN models, such as models based on LSTM (long short term memory) or GRU (gated loop units), may be used. In the CNN model, local features in an input sequence can be extracted through convolution operation and pooling operation, and integration is carried out at an output layer, so that time sequence prediction is realized. In weather timing prediction, models such as a one-dimensional convolutional neural network (1D CNN) or a Time Convolutional Network (TCN) can be used;
s43: the intelligent recognition algorithm model predicts possible extreme weather events such as typhoons, storm, tornadoes and the like by analyzing meteorological data so as to be capable of recognizing the mode and the characteristic of the possible extreme weather events, and can be realized by using a convolutional neural network (Convolutional Neural Network, CNN) or a cyclic neural network (Recurrent Neural Network, RNN) and other models. The convolutional neural network is mainly used for processing images or sequence data, and can capture spatial features in the data, such as textures and shapes in the images and trends and periodic patterns in time sequences. In extreme weather intelligent identification, convolutional neural networks can be used to extract spatio-temporal features in meteorological data, such as barometric pressure, temperature, humidity, wind speed, etc., and relationships between these features, for determining whether an extreme weather event has occurred.
Further, the weather feature knowledge data set is obtained by extracting new data from the original weather ocean data set by utilizing a feature extraction algorithm, for example, the adiabatic attenuation rate can be extracted according to the temperature and the height, the data are integrated to form the weather ocean feature data set, knowledge is formed after training and processing the original data set based on methods such as artificial intelligence and deep learning, for example, strong convection weather data can be obtained according to weather data such as temperature, wind and rain, the obtained knowledge is integrated to form the knowledge data set, and the features and the knowledge data set are provided for an intelligent application module for guaranteeing application. FIG. 2 is a diagram of a meteorological feature knowledge data set construction process.
Further, the data application module is used for managing and evaluating knowledge in a knowledge base, mainly in the aspects of prediction, diagnosis, planning, decision support and the like, and is used for carrying out weather threat index visualization, flight path planning and the like by combining a Cesium3D earth technology and a path generation algorithm. FIG. 3 is a flow chart of the application of a knowledge product in the planning of a flight path.
Example 1
FIG. 4 is a detailed flow chart of the weather data knowledge discovery system for application;
s1: the data preprocessing module is used for cleaning and processing data before the meteorological data are stored in the three-dimensional hypercube data warehouse so as to ensure that the obtained data are of high quality. The processing component may include a variety of algorithms, such as a random forest algorithm, a KNN algorithm, and the like. The algorithms can process abnormal values and missing values, and more accurate and reliable data can be obtained through filtered data;
s2: the knowledge discovery module is the core of the data mining flow, can be combined with the definition of a user and an expert on a problem target, selects relevant characteristic elements, selects a proper deep learning algorithm from an algorithm library, and performs a knowledge discovery key flow in the face of artificial intelligent algorithms such as intelligent recognition, pattern correction, time sequence prediction and the like to form knowledge and store the knowledge in a knowledge library. Deep learning algorithms can discover correlations and patterns between data and discover underlying knowledge based on these patterns and correlations. Thus, the knowledge discovery module may provide more accurate and reliable knowledge for subsequent data application modules.
S3: the data application module mainly relates to the aspects of prediction, diagnosis, planning, decision support and the like, and needs to be realized by utilizing the knowledge base obtained in the first two stages. These applications may be weather threat index visualization and aircraft track planning by combining the Cesium3D earth technology and path generation algorithms. These applications can provide visual weather information and planning for weather workers and related personnel to better address weather threats.
The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.
A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform a weather data knowledge discovery method.
A computing device comprising one or more processors, one or more memories, and one or more programs, wherein one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing a weather data knowledge discovery method.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing is illustrative of the present invention and is not to be construed as limiting thereof, but rather as providing for the use of additional embodiments and advantages of all such modifications, equivalents, improvements and similar to the present invention are intended to be included within the scope of the present invention as defined by the appended claims.
Claims (10)
1. The weather data knowledge discovery method is characterized by comprising the following steps of:
1) Based on multi-source heterogeneous meteorological data, including space-time observation data and historical analysis data, performing data cleaning and data quality control processing on missing data and abnormal values through statistics and a machine learning algorithm, and combining with a unified space-time technology of the meteorological data to form an intelligent algorithm training data set;
2) Constructing a plurality of algorithm models based on artificial intelligence and deep learning technology to form an algorithm pool, carrying out strong convection weather identification and weather forecast on the intelligent algorithm training data set obtained in the step 1) through the algorithm pool to form knowledge, and storing the knowledge in a pre-constructed weather feature knowledge data set for supporting environmental information application;
3) And (3) carrying out data application on the weather characteristic knowledge data set obtained in the step (2) and generating a weather space-time evolution rule according to a knowledge product.
2. The weather data knowledge discovery method as claimed in claim 1, wherein in the step 1), the data cleaning process is as follows: and carrying out de-duplication, de-noising, error correction and filtering on the acquired meteorological data.
3. The method according to claim 1, wherein in the step 1), the process of processing missing data and abnormal values by data quality control is as follows: converting the data of different dimensions into uniform dimensions based on the maximum-minimum normalization and the Z-score normalization;
filling the missing values by using linear interpolation, polynomial interpolation and spline interpolation;
for data which is too dense, reducing the data volume through random sampling, hierarchical sampling and clustering sampling;
the raw data is converted into an analyzed and mined form.
4. The method for designing a weather data knowledge discovery system according to claim 1, wherein: the weather data unified space-time technology in the step 1) comprises the following steps:
s31: unifying space-time coordinates;
s32: the time resolution is uniform;
s33: the spatial resolution is uniform;
s34: the data format is uniform.
5. The method for designing a weather data knowledge discovery system according to claim 1, wherein:
the algorithm model of the weather space-time evolution rule generated in the step 2) comprises a mode correction algorithm model, a time sequence prediction algorithm model and an intelligent recognition algorithm model.
6. A weather data knowledge discovery system, comprising:
the data preprocessing module is used for processing missing data and abnormal values through data cleaning and data quality control by means of statistics and machine learning algorithms based on multi-source heterogeneous meteorological data, wherein the multi-source heterogeneous meteorological data comprises space-time observation data and historical analysis data, and an intelligent algorithm training data set is formed by combining meteorological data with a unified space-time technology;
the knowledge discovery module is used for constructing a plurality of algorithm models based on artificial intelligence and deep learning technology to form an algorithm pool, carrying out strong convection weather identification and weather forecast on an intelligent algorithm training data set through the algorithm pool to form knowledge, and storing the knowledge in a pre-constructed weather characteristic knowledge data set for supporting environmental information application;
and the data application module is used for carrying out data application on the weather characteristic knowledge data set and generating a weather space-time evolution rule according to the knowledge product.
7. The weather data knowledge discovery system of claim 6, wherein the data preprocessing module performs the process of cleaning data: and carrying out de-duplication, de-noising, error correction and filtering on the acquired meteorological data.
8. The weather data knowledge discovery system of claim 6, wherein the data quality control process of the data preprocessing module processes missing data and outliers by: converting the data of different dimensions into uniform dimensions based on the maximum-minimum normalization and the Z-score normalization;
filling the missing values by using linear interpolation, polynomial interpolation and spline interpolation;
for data which is too dense, reducing the data volume through random sampling, hierarchical sampling and clustering sampling;
the raw data is converted into an analyzed and mined form.
9. The weather data knowledge discovery hierarchy design system of claim 6, wherein: the weather data unified space-time technology in the data preprocessing module comprises the following steps:
s31: unifying space-time coordinates;
s32: the time resolution is uniform;
s33: the spatial resolution is uniform;
s34: the data format is uniform.
10. The weather data knowledge discovery hierarchy design system of claim 6, wherein:
the algorithm model of the weather space-time evolution rule generated in the knowledge discovery module comprises a mode correction algorithm model, a time sequence prediction algorithm model and an intelligent recognition algorithm model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310623030.7A CN116860824A (en) | 2023-05-30 | 2023-05-30 | Meteorological data knowledge discovery method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310623030.7A CN116860824A (en) | 2023-05-30 | 2023-05-30 | Meteorological data knowledge discovery method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116860824A true CN116860824A (en) | 2023-10-10 |
Family
ID=88227532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310623030.7A Pending CN116860824A (en) | 2023-05-30 | 2023-05-30 | Meteorological data knowledge discovery method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116860824A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117556197A (en) * | 2024-01-11 | 2024-02-13 | 中国气象科学研究院 | Typhoon vortex initialization method based on artificial intelligence |
-
2023
- 2023-05-30 CN CN202310623030.7A patent/CN116860824A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117556197A (en) * | 2024-01-11 | 2024-02-13 | 中国气象科学研究院 | Typhoon vortex initialization method based on artificial intelligence |
| CN117556197B (en) * | 2024-01-11 | 2024-03-22 | 中国气象科学研究院 | Typhoon vortex initialization method based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10909446B2 (en) | Systems and methods for selecting global climate simulation models for training neural network climate forecasting models | |
| US10871594B2 (en) | Methods and systems for climate forecasting using artificial neural networks | |
| CN111639787B (en) | Spatio-temporal data prediction method based on graph convolution network | |
| Nguyen et al. | PM2. 5 prediction using genetic algorithm-based feature selection and encoder-decoder model | |
| CN112598180A (en) | Distributed regional wind power prediction method | |
| Li et al. | Deep spatio-temporal wind power forecasting | |
| CN115730684A (en) | Air quality detection system based on LSTM-CNN model | |
| CN116860824A (en) | Meteorological data knowledge discovery method and system | |
| CN115271181A (en) | Typhoon probability forecasting intelligent method and device based on multi-mode data fusion | |
| CN117172509B (en) | Construction project distribution system based on decoration construction progress analysis | |
| CN114943365A (en) | Rainfall estimation model establishing method fusing multi-source data and rainfall estimation method | |
| CN116451848A (en) | Satellite telemetry data prediction method and device based on space-time attention mechanism | |
| CN114498619A (en) | Wind power prediction method and device | |
| CN116720156A (en) | Weather element forecasting method based on graph neural network multi-mode weather data fusion | |
| Jin et al. | Bayesian hierarchical model for change point detection in multivariate sequences | |
| CN114418243B (en) | Distributed new energy cloud grid prediction method and system | |
| Naoui et al. | Integrating iot devices and deep learning for renewable energy in big data system | |
| Li et al. | Deepphysinet: Bridging deep learning and atmospheric physics for accurate and continuous weather modeling | |
| CN111325384B (en) | NDVI prediction method combining statistical characteristics and convolutional neural network model | |
| Zhang | Deep learning-based hybrid short-term solar forecast using sky images and meteorological data | |
| Nazemi et al. | A Novel Two-Dimensional Convolutional Neural Network-Based an Hour-Ahead Wind Speed Prediction Method | |
| Fredyan et al. | Spatiotemporal convolutional LSTM with attention mechanism for monthly rainfall prediction | |
| CN117236571B (en) | Planning method and system based on Internet of things | |
| Xiao et al. | Prediction of Monthly Rainfall in Plateau Area Based on Convolutional Neural Network | |
| CN114662274B (en) | Snow water equivalent modeling and predicting method based on zero-expansion space-time regression model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |