CN110019343A - A kind of new energy meteorological data management method and system - Google Patents
A kind of new energy meteorological data management method and system Download PDFInfo
- Publication number
- CN110019343A CN110019343A CN201711353102.1A CN201711353102A CN110019343A CN 110019343 A CN110019343 A CN 110019343A CN 201711353102 A CN201711353102 A CN 201711353102A CN 110019343 A CN110019343 A CN 110019343A
- Authority
- CN
- China
- Prior art keywords
- meteorological data
- data
- node
- new energy
- mpp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01W—METEOROLOGY
- G01W1/00—Meteorology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Environmental & Geological Engineering (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Environmental Sciences (AREA)
- Atmospheric Sciences (AREA)
- Ecology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biodiversity & Conservation Biology (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention provides a kind of new energy meteorological data management method and system, comprising: obtains meteorological data；The meteorological data of the acquisition is stored on the node of massive parallel processing MPP；Establish the meteorological data query table for being directed to the node.Technical solution provided by the invention utilizes the distributed storage ability of MPP, ten PB grades of weather bureau's metric data of support sizes；Magnanimity timing big data can be handled with efficient storage and quickly；Hash storage and MPP parallelization processing capacity on Shared nothing framework based on the Hash in the library MPP, maximum have played intelligence index and column storage capacity in single node, have greatly accelerated the scanning speed to big table data.
Description
Technical field
The invention belongs to new energy meteorological fields, big data field, and in particular to a kind of new energy meteorological data management side
Method and system.
Background technique
New energy power prediction is to predict future time period new energy power swing rule in advance by founding mathematical models at present
The technology of rule.Due to can not integrally hold the huge complicated data of weather forecast of the scale of construction, meteorological measuring, power prediction
The Research Thinking of data and the operation data of other new energy stations, new energy power prediction is confined to for single wind-powered electricity generation more
Prediction on the independent time section of field, according to physical significance theoretic in cognitive range, artificial carries out data information
Segmentation and dimension-reduction treatment, have ignored the associate feature of wind energy resources and other elements in time scale and space scale, with
Two-dimensional model carrys out the data correlation in descriptive analysis hyperspace, has limited to the cognition dimension to new energy power, so as to cause
Precision of prediction room for promotion is very limited.
Numerical weather forecast is the most important input data of the new energy power prediction such as wind-powered electricity generation, photovoltaic, according to statistics, new energy
There are about 60% in the error of prediction power to be originated from numerical weather forecast, expands the estimation range of numerical weather forecast and improves numerical value
The precision of data of weather forecast is to realize the most effective means of new energy power prediction precision improvement.Currently, towards new energy function
The spatial resolution of the mesoscale NWP of rate prediction is mostly 10km × 10km, and typical installed capacity is 5MW's
Wind power plant occupied area is about 3km × 3km, and the occupied area of photovoltaic plant is smaller, and the demand to computing resource is with spatial discrimination
The increase of rate exponentially increases.
For the meteorological data of 3km × 3km, object storage system is stored in a distributed manner at present, and meteorological data passes through pre-
Following 3 days prediction data can be generated by surveying model, and since these data volumes are bigger, warehouse-in efficiency is relatively low, thus often by
It abandons without storage, to cause the loss with break-up value data.
Summary of the invention
In view of the deficiencies of the prior art, the present invention proposes a kind of new energy meteorological data management method and system, including is situated between
Continue the general frames of MPP+ time series databases, proposes that data store scalability scheme, provides for basic electric power weather prognosis
Efficiently, accurately data foundation, effectively support power network safety operation.
A kind of new energy meteorological data management method, comprising:
Obtain meteorological data；
The meteorological data of the acquisition is stored on the node of massive parallel processing MPP；
Establish the meteorological data query table for being directed to the node.
Further, the meteorological data by the acquisition is stored on the node of massive parallel processing MPP,
Include:
The meteorological data of the acquisition is ranked up according to the time；
Average packet is carried out to the data after sequence；
Data after grouping are stored in by the way of distributed storage on the node of MPP.
It is further, described to establish the meteorological data query table for being directed to the node, comprising:
Data in the node are subjected to region ordering, establish the meteorological data query table.
Further, the data by the node carry out region ordering, before establishing the meteorological data query table
Further include:
If the node memory, in data, the meteorological data that will acquire establishes interim table.
Further, further includes: the inquiry of data is carried out using the meteorological data query table, comprising:
According to hash storage method, temporally field is inquired；And/or
It is inquired according to hash storage method by zone number.
Further, the meteorological data is PB grades of meteorological datas.
Further, the node uses Share-nothing framework.
A kind of new energy meteorological data management system, comprising:
Module is obtained, for obtaining meteorological data；
Memory module, for the meteorological data of the acquisition to be stored in the node of massive parallel processing MPP；
Module is established, for establishing the meteorological data query table for being directed to the node.
Further, the memory module, is used for,
The meteorological data of the acquisition is ranked up according to the time；
Average packet is carried out to the data after sequence；
Data after grouping are stored in by the way of distributed storage on the node of MPP.
Further, the module of establishing includes:
Sorting sub-module establishes the meteorological data query table for the data in the node to be carried out region ordering.
It is further, described to establish module further include:
Interim submodule, if the meteorological data that will acquire establishes interim table for the node memory in data.
Further, further includes: enquiry module, comprising:
Time inquiring submodule, for temporally field to be inquired according to hash storage method；And/or
Site polling submodule, for being inquired according to hash storage method by zone number.
Compared with the latest prior art, technical solution provided by the invention has the advantages that
1, technical solution provided by the invention utilizes the distributed storage ability of MPP, and meteorological data is stored, and builds
Vertical meteorological data query table, the convenient inquiry to data later.
2, technical solution provided by the invention utilizes the distributed storage ability of MPP, and ten PB grades of weather bureau of support sizes measures
Data can handle magnanimity timing big data with efficient storage and quickly.
3, technical solution provided by the invention is deposited based on the hash on the Shared nothing framework of the Hash in the library MPP
Storage and MPP parallelization processing capacity, maximum have played intelligence index and column storage capacity in single node, have greatly accelerated pair
The scanning speed of big table data, theoretically scanning speed=500DC × 65536 × m intelligently indexes filterability × n node.
4, the coarseness index that intelligently indexes of the technical solution provided by the invention based on MPP, significantly control meteorological data
Volume expansions rate, and prevent with data volume promoted and cause index generate cost promoted, to greatly improve data
Warehouse-in efficiency can achieve the loading velocity of 3.6G between the second.
Detailed description of the invention
Fig. 1 is flow chart of the present invention；
Fig. 2 is data memory node MPP configuration diagram；
Fig. 3 is data store organisation schematic diagram；
Fig. 4 is that memory node extends schematic diagram；
Fig. 5 is back end distribution schematic diagram；
Fig. 6 is using data copy schematic diagram.
Specific embodiment
The present invention is described in further details with reference to the accompanying drawing.For purpose, the technical solution for making the embodiment of the present invention
Clearer with advantage, following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out
It clearly and completely describes, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Base
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts it is all its
Its embodiment, shall fall within the protection scope of the present invention.
Embodiment 1, the present invention provides a kind of new energy meteorological data management methods, as shown in Figure 1, comprising:
Obtain meteorological data；
The meteorological data of the acquisition is stored on the node of massive parallel processing MPP；
Establish the meteorological data query table for being directed to the node.
Using the distributed storage ability of MPP, ten PB grades of weather bureau's metric data of support sizes can with efficient storage and quickly
Handle magnanimity timing big data.
Embodiment 2, the present invention provides a kind of new energy meteorological data management systems, comprising:
Module is obtained, for obtaining meteorological data；
Memory module, for the meteorological data of the acquisition to be stored in the node of massive parallel processing MPP；
Module is established, for establishing the meteorological data query table for being directed to the node.
Further, the memory module, is used for,
The meteorological data of the acquisition is ranked up according to the time；
Average packet is carried out to the data after sequence；
Data after grouping are stored in by the way of distributed storage on the node of MPP.
Further, the module of establishing includes:
Sorting sub-module establishes the meteorological data query table for the data in the node to be carried out region ordering.
It is further, described to establish module further include:
Interim submodule, if the meteorological data that will acquire establishes interim table for the node memory in data.
Further, further includes: enquiry module, comprising:
Time inquiring submodule, for temporally field to be inquired according to hash storage method；And/or
Site polling submodule, for being inquired according to hash storage method by zone number.
Embodiment 3 is required, the technical side merged using MPP+ time series database according to big data quantity demand and scalability
Case.
1, the statement of requirements
Meteorological data is typical temporal information, carries out information to electric power meteorological data and derives and predict have important work
With.It acts not only as historical data long-term preservation, can also reflect variation tendency of the data in application environment.When reasonable
State storage model can simplify the design of application system, reduce the redundancy of data storage, improve the execution efficiency of program and be
System stability.
Weather information need to be stored to be described as follows:
Range: phase 20000, a somewhere collection point information, succeeding ranges are by further expansion；
Data generate frequency: generating 1 time within every 6 hours, generating 72 hours every time, (15 minutes intervals generate one group of data, altogether
288 acquisition moment) data；
Size of data: (72 hours, interval generated one group of number to the data that each node generates every time about 57K byte within 15 minutes
According to totally 288 acquisition moment, each 200 byte of moment)；
Data storage period: 10 years；
Data query mode: 1) a certain 1 year data in collection point (time series approach) are inquired；2) sometime point is inquired
All data (when discontinuity surface mode) of 20000 points；
Storage requires: entry time is no more than 30 points；
Query requirement: any point extraction time is less than 20s, and profile extraction time time is less than 30s；
Scalability requirement: framework can dynamic linear expansion, future is applicable to national sampling.
2, Technical Architecture
Data memory node uses MPP framework, as shown in Figure 2:
Database node uses Share Nothing framework, parallel memorizing and processing data in MPP, without coupling between data,
Cross-node affairs will not be generated.When needing to carry out global query, it is distributed to after each node obtains data and is combined by MPP management program
The query result of node.
Each node is an independent database, and the storage of time profile data and time series data are substantially carried out in database
Storage, data store organisation are as shown in Figure 3.
3, write performance
By taking the back end of somewhere as an example:
Time series data is additional every time to be written 20000 blob, about 50 blob of write-in per second, it is contemplated that control is in 7 minutes.
1 blob, size 1.15G is written in time profile data every time, and disk writing speed is about 30M/ seconds, it is contemplated that control
System is in 1 minute.
The above performance is single node performance, and when multinode distributed data, performance can be promoted according to number of nodes, or main is utilized
MPP stores the data of a logical collection, reaches PB grades of storages using multinode to the linear expansion of storage, each node.
4, reading performance
By taking the back end of somewhere as an example:
Time series data reads 1 annual data of 1 collection point, reads each a part of 12 blob every time, each
Size is about 7MB, total 84M, it is contemplated that control is in 10 seconds.
Time profile data read every time 20000 collection points some when discontinuity surface total data, read 1 blob,
Size is about 1.15G, and disk writing speed is about 40M/ seconds, it is contemplated that control is in 30 seconds.
5, scalability
Data volume single machine database can satisfy at present, if the following construction whole nation or global metadata, it is contemplated that MPP frame
Structure, since data dependence is low, no coupling, MPP can obtain the performance of linear increase.
It when MPP increases node, needs to do fast resampling in MPP management node, uniformly to utilize the processing energy of each node
Power, it is as shown in Figure 4 that memory node extends schematic diagram.
6, safety
High Availabitity mode can be used in each back end, ensures Information Security and operation duration, back end point
Cloth schematic diagram is as shown in Figure 5.
The quantity of equipment can also be reduced, using data copy mode according to resource situation, by the way of data copy
Schematic diagram is as shown in Figure 6.
The application is referring to method, the process of equipment (system) and computer program product according to the embodiment of the present application
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Finally it should be noted that: the above embodiments are merely illustrative of the technical scheme of the present invention and are not intended to be limiting thereof, to the greatest extent
Invention is explained in detail referring to above-described embodiment for pipe, it should be understood by those ordinary skilled in the art that: still
It can be with modifications or equivalent substitutions are made to specific embodiments of the invention, and without departing from any of spirit and scope of the invention
Modification or equivalent replacement, are intended to be within the scope of the claims of the invention.
Claims (12)
1. a kind of new energy meteorological data management method characterized by comprising
Obtain meteorological data；
The meteorological data of the acquisition is stored on the node of massive parallel processing MPP；
Establish the meteorological data query table for being directed to the node.
2. a kind of new energy meteorological data management method as described in claim 1, which is characterized in that described by the acquisition
Meteorological data is stored on the node of massive parallel processing MPP, comprising:
The meteorological data of the acquisition is ranked up according to the time；
Average packet is carried out to the data after sequence；
Data after grouping are stored in by the way of distributed storage on the node of MPP.
3. a kind of new energy meteorological data management method as described in claim 1, which is characterized in that described to establish for described
The meteorological data query table of node, comprising:
Data in the node are subjected to region ordering, establish the meteorological data query table.
4. a kind of new energy meteorological data management method as claimed in claim 3, which is characterized in that it is described will be in the node
Data carry out region ordering, before establishing the meteorological data query table further include:
If the node memory, in data, the meteorological data that will acquire establishes interim table.
5. a kind of new energy meteorological data management method as described in claim 1, which is characterized in that further include: described in utilization
The inquiry of meteorological data query table progress data, comprising:
According to hash storage method, temporally field is inquired；And/or
It is inquired according to hash storage method by zone number.
6. a kind of new energy meteorological data management method a method as claimed in any one of claims 1 to 5, which is characterized in that the meteorology number
According to for PB grades of meteorological datas.
7. a kind of new energy meteorological data management method as described in claim 1, which is characterized in that the node uses
Share-nothing framework.
8. a kind of new energy meteorological data management system characterized by comprising
Module is obtained, for obtaining meteorological data；
Memory module, for the meteorological data of the acquisition to be stored in the node of massive parallel processing MPP；
Module is established, for establishing the meteorological data query table for being directed to the node.
9. a kind of new energy meteorological data management system as claimed in claim 8, which is characterized in that the memory module is used
In,
The meteorological data of the acquisition is ranked up according to the time；
Average packet is carried out to the data after sequence；
Data after grouping are stored in by the way of distributed storage on the node of MPP.
10. a kind of new energy meteorological data management system as claimed in claim 8, which is characterized in that described to establish module packet
It includes:
Sorting sub-module establishes the meteorological data query table for the data in the node to be carried out region ordering.
11. a kind of new energy meteorological data management system as claimed in claim 10, which is characterized in that described to establish module also
Include:
Interim submodule, if the meteorological data that will acquire establishes interim table for the node memory in data.
12. a kind of new energy meteorological data management system as claimed in claim 8, which is characterized in that further include: inquiry mould
Block, comprising:
Time inquiring submodule, for temporally field to be inquired according to hash storage method；And/or
Site polling submodule, for being inquired according to hash storage method by zone number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711353102.1A CN110019343A (en) | 2017-12-15 | 2017-12-15 | A kind of new energy meteorological data management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711353102.1A CN110019343A (en) | 2017-12-15 | 2017-12-15 | A kind of new energy meteorological data management method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110019343A true CN110019343A (en) | 2019-07-16 |
Family
ID=67186944
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711353102.1A Pending CN110019343A (en) | 2017-12-15 | 2017-12-15 | A kind of new energy meteorological data management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019343A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11386089B2 (en) | 2020-01-13 | 2022-07-12 | The Toronto-Dominion Bank | Scan optimization of column oriented storage |
| CN116881241A (en) * | 2023-09-06 | 2023-10-13 | 深圳市银河系科技有限公司 | Safety management method and system applied to meteorological data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182438A (en) * | 2014-02-25 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Message counting method and device |
| CN105045929A (en) * | 2015-08-31 | 2015-11-11 | 国家电网公司 | MPP architecture based distributed relational database |
-
2017
- 2017-12-15 CN CN201711353102.1A patent/CN110019343A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182438A (en) * | 2014-02-25 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Message counting method and device |
| CN105045929A (en) * | 2015-08-31 | 2015-11-11 | 国家电网公司 | MPP architecture based distributed relational database |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11386089B2 (en) | 2020-01-13 | 2022-07-12 | The Toronto-Dominion Bank | Scan optimization of column oriented storage |
| CN116881241A (en) * | 2023-09-06 | 2023-10-13 | 深圳市银河系科技有限公司 | Safety management method and system applied to meteorological data |
| CN116881241B (en) * | 2023-09-06 | 2023-11-07 | 深圳市银河系科技有限公司 | Safety management method and system applied to meteorological data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10229129B2 (en) | Method and apparatus for managing time series database | |
| Shyam et al. | Apache spark a big data analytics platform for smart grid | |
| CN106777093B (en) | Skyline inquiry system based on space time sequence data flow application | |
| Lamba et al. | Uses of cluster computing techniques to perform big data analytics for smart grid automation system | |
| CN109800898A (en) | A kind of intelligence short-term load forecasting method and system | |
| CN107194533B (en) | Power distribution network full information model construction method and system | |
| CN107798059B (en) | NCO meteorological data structured storage method and device | |
| CN111586091A (en) | Edge computing gateway system for realizing computing power assembly | |
| CN111178587A (en) | Spark framework-based short-term power load rapid prediction method | |
| CN109446230A (en) | A kind of big data analysis system and method for photovoltaic power generation influence factor | |
| Jach et al. | Application of HADOOP to store and process big data gathered from an urban water distribution system | |
| CN110019343A (en) | A kind of new energy meteorological data management method and system | |
| CN117372201A (en) | Rapid construction method of intelligent water conservancy digital twin model applied to reservoir | |
| Huang et al. | Parallel map matching on massive vehicle gps data using mapreduce | |
| Niu et al. | Parallel grid-based density peak clustering of big trajectory data | |
| CN112614207A (en) | Contour line drawing method, device and equipment | |
| CN111144629A (en) | Method and system for predicting water inflow of hydroelectric power station | |
| KR20170069396A (en) | Very short range microscale weather forecast model for real-time forecasting | |
| CN109816149A (en) | A kind of wind power plant is contributed scene generating method and device at random | |
| Colosi et al. | Time series data management optimized for smart city policy decision | |
| CN114123190A (en) | Method and device for determining target region to which ammeter belongs, electronic equipment and storage medium | |
| CN109217367B (en) | Wind power generation prediction method, device and equipment | |
| CN114021833A (en) | Line loss prediction method, system, storage medium and computing device | |
| Zou et al. | An integrated disaster rapid cloud service platform using remote sensing data | |
| CN114710481B (en) | Flow ticket analysis method, device, equipment and storage medium based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |