CN106202378A - The immediate processing method of a kind of streaming meteorological data and system - Google Patents
The immediate processing method of a kind of streaming meteorological data and system Download PDFInfo
- Publication number
- CN106202378A CN106202378A CN201610534963.9A CN201610534963A CN106202378A CN 106202378 A CN106202378 A CN 106202378A CN 201610534963 A CN201610534963 A CN 201610534963A CN 106202378 A CN106202378 A CN 106202378A
- Authority
- CN
- China
- Prior art keywords
- data
- meteorological
- station
- streaming
- website
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses the immediate processing method of a kind of streaming meteorological data, concrete implementation process is: streaming meteorological data described in Real-time Collection, and described streaming meteorological data includes country's station data and regional station data；Be sent to described streaming meteorological data to specify respectively calculates node port；Described each calculating node port is monitored, when listening to described streaming meteorological data by described each calculating node port, described streaming meteorological data is stored in each calculating node memory and described streaming meteorological data is backuped in distributed file system, it is then based on described streaming meteorological data and carries out structuring process, the list structure that design is corresponding, set up index, inquire about for user.The present invention has taken into full account the feature of the streaming meteorological data of typical industry, it is possible to realizes the Real-time Collection of streaming meteorological data, statistical analysis, and then is supplied to the purpose of user's inquiry.
Description
Technical field
The application relates to weather service process field, particularly relates to the immediate processing method of a kind of streaming meteorological data and is
System.
Background technology
In recent years, along with the progress of Meteorological Modernization cause, automatic Weather Station is relied on increasing, to automatic Weather Station by weather service
Data storage management, use require more and more higher.
At present, the increase rapidly of surface weather observation station quantity, surface weather observation station, the Yangtze river basin has reached about more than 20,000
Individual.The increase of surface weather observation station quantity and the increase of the observation frequency so that ground observation data become " streaming meteorology number
According to " typical case.The high efficiency of the streaming weather data analysis historical data to never going offline and processing capability in real time proposes more
High demand,
At present, big data processing technique is broadly divided into batch system and two kinds of forms of Stream Processing technology, industry about
It is relative with discussion fully that big data batch processing calculates systematic research, and Stream Processing technology is still also in the process tried to explore
In.It is various passively that the calculating of high amount of traffic formula is currently mainly applied to monitor Sensor Network, Internet of Things, social media, financial transaction etc.
Produce data, actively produce data and automatically generate the distributed environment of data.According to the accuracy requirement of application scenarios, it is entered
Go and calculate in real time and timely feedback result, and in the effective time of data, obtain its value, excavate and predict its Changing Pattern,
It it is the core objective of streaming meteorological data calculating system.And Meteorological Automatic Station data are as a kind of natural streaming meteorological data,
Application can be promoted with service by streaming process meteorological data technology and the integration of ground automatic Weather Station observation corresponding service platform,
Limits ground plays weather information to business development and the supporting role of scientific and technical innovation.But, this target to be realized, it is necessary to solve
Immediate processing method centered by streaming meteorological data memory architecture in a distributed manner.
Summary of the invention
The present invention proposes the immediate processing method of a kind of streaming meteorological data, cannot accomplish for existing streaming meteorological data
The present situation of real-time statistic analysis, by Real-time Collection regional station data (that is: this locality automatic Weather Station live file) and country's station data
(that is: CIMISS data), and use storage based on distributed memory, transmission and parallel processing technique, it is achieved business personnel couple
The display analysis of local all automatic Weather Stations observation data, pushes forward basic unit's weather service modernization construction comprehensively, constantly strengthens gas
Comprehensive strength and meteorological work influence power as mechanism.The present invention can be province, city, the Information Assurance personnel in county and weather forecast clothes
The related personnel such as business personnel, operational control provide service, provide information reference for relevant units such as meteorological observatory, weather centers, carry
For the refreshing in real time of automatic Weather Station data, show in real time, inquire about, add up, analytic function.
For solving above-mentioned technical problem, the invention provides the immediate processing method of a kind of streaming meteorological data, described side
Method is applied in the quick processing system of described streaming meteorological data, and described method includes:
Streaming meteorological data described in Real-time Collection, described streaming meteorological data includes country's station data and regional station data；
Be sent to described streaming meteorological data to specify respectively calculates node port；
Described each calculating node port is monitored, is saved by described each calculating when listening to described streaming meteorological data
During point port, described streaming meteorological data is stored in each calculating node memory and described streaming meteorological data is backuped to distributed
In file system；
Carry out structuring process, the list structure that design is corresponding based on described streaming meteorological data, set up index, for user
Inquiry.
Preferably, when described streaming meteorological data is described country station data, streaming described in described Real-time Collection is meteorological
Data, particularly as follows:
Send the first data acquisition request extremely described comprehensive national Meteorological Information Sharing Platform in the first moment, and receive institute
State first country's station data that comprehensive national Meteorological Information Sharing Platform returns；Described first data acquisition request carries website
Set and meteorological community set, wherein, enumerate the N number of website needing to share data in described Website Hosting, described meteorology belongs to
Property set in enumerate the meteorological attribute that each website in N number of website is corresponding so that described comprehensive national meteorological information share put down
Stylobate obtains, in described first data acquisition request, the gas that in described N number of website corresponding to described first moment, each website is corresponding
As property value, described first country's station data are the gas that in described N number of website corresponding to described first moment, each website is corresponding
As property value；N is positive integer；
The second moment after described first moment sends the second data acquisition request and believes to described comprehensive national meteorology
Breath shared platform, and receive second country's station data that described comprehensive national Meteorological Information Sharing Platform returns, described second state
Family station data be described second moment corresponding country station data；Described second data acquisition request carries described Website Hosting
With described meteorological community set so that described comprehensive national Meteorological Information Sharing Platform obtains based on described second data acquisition request
The meteorological property value that each website is corresponding in described N number of website corresponding to described second moment, described second country's station data
It it is the meteorological property value that in described N number of website corresponding to described second moment, each website is corresponding；
If described first country's station data are different with described second country's station data, then use described second country's station data
Update described first country's station data, so that described country station data keep up-to-date.
Preferably, when described streaming meteorological data is described country station data, described described streaming meteorological data is sent out
That delivers to specify respectively calculates node port, including:
Described second country's station data are resolved, it is thus achieved that each in described N number of website that described second moment is corresponding
The meteorological property value that website is corresponding；
Specify each calculating node port by socket, send corresponding described N number of of described second moment with appointed interval
The meteorological property value that in website, each website is corresponding to described specify respectively calculate node port.
Preferably, described based on described streaming meteorological data carry out structuring process and set up index, for user inquire about,
Including:
Leave meteorological property value correspondence corresponding for each website in described N number of website corresponding described second moment in unit
In tables of data, to form country's station hour metadata table, country's station minute metadata table, tourism station metadata table, traffic station unit
One or more of tables of data；By described country station hour metadata table, station minute metadata table, the described tourism of described country
Stand metadata table, described traffic station metadata table one or more be stored in each calculating node memory preservation；
With day, hour, minute, in year, the moon, the character string of site number composition indexes as the first order；
To country's station hour metadata table, country's station minute metadata table, tourism station metadata table, traffic station metadata table
One or more list structures each create concordance list, the literary name section in described concordance list include observation time, province, point
Clock, hour, site number, type of site, as the second level index.
Preferably, when described streaming meteorological data is described regional station data, streaming described in described Real-time Collection is meteorological
Data, particularly as follows:
From locally located appointment data sharing catalogue, call described regional station data in real time, described regional station data by
Provincial Meteorological Information Sharing Platform is sent in described appointment data sharing catalogue in real time；Wherein, described regional station data are provinces
The meteorological property value of each website in M the website that level Meteorological Information Sharing Platform collects, M is positive integer.
Preferably, when described streaming meteorological data is described regional station data, described described streaming meteorological data is sent out
That delivers to specify respectively calculates node port, including:
Described regional station data are resolved, it is thus achieved that the meteorological property value of each website in described M website；
With appointed interval send the meteorological property value of each website in described M website to described specify respectively calculate node
Port.
Preferably, state and carry out structuring process based on described streaming meteorological data and set up index, inquire about for user, bag
Include:
The meteorological property value correspondence of each website in described M website is left in metadata table, to form regional station
Hour metadata table, regional station minute metadata table, tourism station metadata table, traffic station metadata table one or more；
By described regional station hour metadata table, described regional station minute metadata table, described tourism station metadata table, institute
One or more stating traffic station metadata table are stored in each calculating node memory preservation；
With day, hour, minute, in year, the moon, the character string of site number composition indexes as the first order；
To regional station hour metadata table, regional station minute metadata table, tourism station metadata table, traffic station metadata table
One or more list structures each create concordance list, the literary name section in described concordance list include observation time, province, point
Clock, hour, site number, type of site, as the second level index.
Preferably, described based on described streaming meteorological data carry out structuring process and set up index, for user inquire about
Afterwards, described method also includes:
Receive the inquiry request of user, the described streaming meteorological data in each calculating node memory is carried out distributed query
And show.
Preferably, described based on described streaming meteorological data carry out structuring process and set up index, for user inquire about
Afterwards, including:
Judge whether the threshold value of each calculating node memory reaches predetermined threshold；
The described streaming meteorological data in the most each calculating node memory is dumped in distributed data base, and delete
Except the described streaming meteorological data in each calculating node memory.
The invention discloses the quick processing system of a kind of streaming meteorological data, including:
Real time data acquisition module, for streaming meteorological data described in Real-time Collection, described streaming meteorological data includes state
Family's station data and regional station data；
Data transmission blocks, respectively calculates node port for be sent to described streaming meteorological data to specify；
Monitor module, for described each calculating node port is monitored, lead to when listening to described streaming meteorological data
When crossing described each calculating node port, described streaming meteorological data is stored in each calculating node memory and by described streaming meteorology number
According to backuping in distributed file system；
Processing module, for carrying out structuring process, the list structure that design is corresponding based on described streaming meteorological data, sets up
Index, inquires about for user.
By one or more technical scheme of the present invention, the invention have the advantages that or advantage:
The invention discloses the immediate processing method of a kind of streaming meteorological data, concrete implementation process is: Real-time Collection
Described streaming meteorological data, described streaming meteorological data includes country's station data and regional station data；By described streaming meteorology number
Node port is respectively calculated according to be sent to specify；Described each calculating node port is monitored, when listening to described streaming gas
When image data is by described each calculating node port, described streaming meteorological data is stored in each calculating node memory and by described stream
Formula meteorological data backups in distributed file system, is then based on described streaming meteorological data and carries out structuring process, design
Corresponding list structure, sets up index, inquires about for user.The present invention has taken into full account the spy of the streaming meteorological data of typical industry
Point, it is possible to realize the Real-time Collection of streaming meteorological data, statistical analysis, and then be supplied to the purpose of user's inquiry.
Further, the present invention by data and result of calculation are buffered in there is higher access rate respectively calculate node
In internal memory, in combination with the parallel computation framework of multimachine network cluster, calculating task is expanded to multiple cooperated computing node, gram
Take traditional calculations using disk as storage medium and the limited local resource of unit serial computing, and the computational efficiency caused is low
Problem, it is achieved being substantially improved of streaming process meteorological data speed.
Further, the upper strata of quoting of the present invention provides class SQL formula user interface, it is achieved data and result of calculation quick
Inquiry and visualization.The present invention can be applied directly to typical case's industries such as weather forecast, the analysis of public opinion, disaster monitoring, improves it big
The service feature that data process.
Accompanying drawing explanation
Fig. 1 is the structure chart of the quick processing system of a kind of streaming meteorological data in the embodiment of the present invention；
Fig. 2 is the flow chart of the immediate processing method of a kind of streaming meteorological data in the embodiment of the present invention.
Detailed description of the invention
In order to make the application the technical staff in the technical field be more clearly understood that the application, below in conjunction with the accompanying drawings,
By specific embodiment, technical scheme is described in detail.
The present invention mainly uses ' Stream Processing technology ' convection type meteorological data to process, and ' Stream Processing technology ' can
Make data stream arrive in real time, process in real time, drastically increase the efficiency of big Data Analysis Services.And Meteorological Automatic Station data are made
For a kind of natural streaming meteorological data, can be put down by streaming process meteorological data technology and ground automatic Weather Station observation corresponding service
The integration of platform, plays weather information to greatest extent to business development and the supporting role of scientific and technical innovation.
Streaming meteorological data, be the automatic monitor station that is geographically distributed gather according to specified time interval to meteorological data
The time series data comprising the key elements such as temperature, rainfall, visibility that center timing sends, has real-time, volatibility, unlimited
Property, randomness.
The Concrete facts principle of the first quick processing system specifically introducing a kind of streaming meteorological data of the present invention below.
Real time data acquisition module 1, for streaming meteorological data described in Real-time Collection.
Introduce country's station data and the Real-time Collection mode of regional station data separately below.
The number that CIMISS data are mainly provided from comprehensive national Meteorological Information Sharing Platform by real time data acquisition module 1
Obtain according to access interface is downloaded, comprehensive national Meteorological Information Sharing Platform contains the national surface weather observation station
Meteorological data.After setting Website Hosting, meteorological community set.Real time data acquisition module 1, for meteorological to comprehensive national
Information sharing platform sends data acquisition request, obtains the automatic Weather Station meteorological data that national interface can obtain, and with JSON
Form returns.
When Real-time Collection CIMISS data, concrete Real-time Collection process is as follows:
In real time data acquisition module 1, include the first timing acquiring module, the second timing acquiring module.
Wherein, the first timing acquiring module, it is mainly used in sending the first data acquisition request in the first moment the most described entirely
The comprehensive Meteorological Information Sharing Platform of state, and receive first country's station number that described comprehensive national Meteorological Information Sharing Platform returns
According to.
First data acquisition request carries parameter: Website Hosting, meteorological community set.Website Hosting is enumerated N number of website,
N is positive integer.This N number of website is the website needing to share each ground weather station of CIMISS data.Meteorological community set
In enumerate N number of website each self-corresponding meteorological attribute, such as wind direction, wind scale, rainfall etc..Certainly, each website needs
The concrete meteorological attribute shared can be identical, it is also possible to different, and the meteorological attribute that such as certain website one is shared is wind direction, rainfall
Amount.The meteorological attribute that certain website two is shared is wind scale, wind direction.Certainly, in order to ensure that website shares the integrity of CIMISS data,
All shared meteorological attribute can be set out in elements combination.
After comprehensive national Meteorological Information Sharing Platform receives the first data acquisition request, then send based on the first moment
The Website Hosting that carries from it of the first data acquisition request and elements combination, obtain in N number of website corresponding to the first moment
The meteorological property value that each website is corresponding, is then returned to the first timing acquiring module.Therefore, described first country's station data are
The meteorological property value that in described N number of website that described first moment is corresponding, each website is corresponding.
Described first timing acquiring module, is additionally operable to receive the first country that comprehensive national Meteorological Information Sharing Platform returns
Stand data.
Owing to streaming meteorological data is real-time change, in order to ensure to obtain up-to-date meteorological data, the present invention have also been devised
Second timing acquiring module.
Described second timing acquiring module, sends the second data acquisition for the second moment after described first moment
Request is to described comprehensive national Meteorological Information Sharing Platform, and receive that described comprehensive national Meteorological Information Sharing Platform returns the
Two country's station data；Described second data acquisition request is also carried described Website Hosting and described meteorological community set.
After comprehensive national Meteorological Information Sharing Platform receives the second data acquisition request, then it it is the moment based on second
The second data acquisition request sent, the Website Hosting carried from it and elements combination, obtains the N number of website in Website Hosting
In meteorological property value corresponding to each website, and return to the second timing acquiring module.Therefore, described second country's station data are
The meteorological property value that in described N number of website that described second moment is corresponding, each website is corresponding.
Second timing acquiring module, is additionally operable to receive described second country's station data, described second country's station data is passed
Give the first timing acquiring module.
Described first timing acquiring module, is used for contrasting described first country's station data and described second country's station data.
If described first country's station data are identical with described second country's station data, then explanation the first country station data are up-to-date countries
Stand data, can update.If described first country's station data are different with described second country's station data, then explanation the second country
Data of standing are up-to-date, then use described second country's station data to update described first country's station data, so that described country station
Data keep up-to-date.
Illustrate underneath with concrete example.
1, Website Hosting, meteorological community set, data code are previously set (such as ground hour, country station data, country station
Ground minute data).Data code is mainly used in controlling described first timing acquiring module and described second timing acquiring module
Send moment and the number of times of data acquisition request.Certainly, if desired quick more new nation station data, then can also use other
Timing acquiring module, the quantity of timing acquiring module is not limited by the present invention, certainly, even if there being multiple timing acquiring module,
Process also with the second country station data that it updates are consistent with the renewal process of first country's station data.
2, owing to meteorological data is out-of-sequence transmission, transmitting procedure, by web influence, simultaneously takes account of the readwrite performance of system,
When processing country's station ground hour data, arranging two timing acquiring modules, the first timing acquiring module is the hourly 0th
Minute perform once obtain country station data task, from comprehensive national Meteorological Information Sharing Platform download obtain N number of website
First country station data.Second timing acquiring module with the 0th second of the per hour the 2nd, 3,4,5,6,7,8,9,10,20 minute and
Within 30th second, perform CIMISS data interface calls to obtain second country's station data, and by second country's station data incoming first
Timing acquiring module carries out the renewal of country's station data, if the country's station data in first timing acquiring module and second fixed
Time acquisition module country station data identical, do not update, otherwise utilize second timing acquiring module country station data update
Country's station data of first timing acquiring module.
3, when processing country's station ground minute data, owing to minute data was spaced more new data with 5 minutes, data update
Frequency is fast, arranges three timing acquiring modules when processing, the first timing acquiring module the 0th, 5,10,15,20,25,30,
0th second execution one subtask of 35,40,45,50,55 minutes, downloads from comprehensive national Meteorological Information Sharing Platform and obtains task
First country's station data of the website to be shared of middle requirement.Second timing acquiring module then the 2nd, 4,6,8,11,13,16,18,
The 10th second execution CIMISS data interface of 21,23,26,28,31,33,36,38,41,43,46,48,51,53,56,58 minutes
Call to obtain second country's station data.And incoming first timing acquiring module carries out country's station data and updates, the process of renewal
Above-mentioned have been introduced, do not repeat them here.3rd timing acquiring module is then at number of execution in the 0th second of the per hour the 32nd minute
According to traversal task, it is thus achieved that the 3rd country's station data, and incoming first timing acquiring module carries out country's station data and updates, with amended record
Data.The process of renewal is above-mentioned to be had been introduced, and does not repeats them here.
4, comprehensive national Meteorological Information Sharing Platform returns with JSON form when returning country's station data, sends for data
Module 2 is called.
It is above the Real-time Collection process of country's station data, the Real-time Collection process of regional station data is described below.
Regional station data (also referred to as: local automatic Weather Station live file) utilize provincial Meteorological Information Sharing Platform to provide
The system that FTP (File Transfer Protocol, file transfer protocol (FTP)) service is sent to the present invention in real time is locally created
Meteorological data file in the FTP data share directory specified, collection is the automatic Weather Station meteorological datas such as the most at county level, city-level.
When Real-time Collection, real time data acquisition module 1 only need to be utilized to call regional station data in real time from FTP data share directory i.e.
Can.Wherein, the meteorological genus of each website during described regional station data are M the website that provincial Meteorological Information Sharing Platform collects
Property value, M is positive integer.
Specifically, system needs the most locally created FTP data share directory, and sets the access right of this catalogue
Limit.And provincial Meteorological Information Sharing Platform enables data router in transmission, from the original storehouse of the meteorological data of shared platform
Automatic Weather Station meteorological data is packaged into TXT (text) file, and the FTP data that TXT file is sent to local setting shares mesh
In record.
Local monitor FTP data share directory, when provincial shared platform sends TXT file under this catalogue, then enable the
Two parsing modules.
The formalized description of local live file is: L={L1, L2..., Ln| n=15}, wherein, n represents the row of file
Number, the 1st row record L1One-dimensional vector is used to be expressed as L1=district's station number, and observation time, longitude, latitude, observation field height above sea level,
Baroceptor height above sea level, observed pattern, quality control identifies, file corrigendum mark }；L2～L15Then use identification code gentle
As community set identifies, i.e. Li={ Pi, Ti| i >=2, i≤15}, wherein PiRepresent the identification code of the i-th row, TiRepresent the i-th row
Meteorological community set.
Such as, L2=PP, and station pressure, sea-level pressure, 3 hours transformations, 24 hours transformations, the highest station pressure,
The highest station pressure time of occurrence, minimum station pressure, minimum station pressure time of occurrence }, wherein PP represents air pressure attribute mark
Know.
Real time data acquisition module 1, only need to call regional station data from local FTP data share directory.
After introducing the Real-time Collection process of complete streaming meteorological data, being transmitted across of streaming meteorological data is described below
Journey.
Data transmission blocks 2, respectively calculates node port for be sent to described streaming meteorological data to specify.
Each node that calculates belongs to multimachine network cluster, has multiple calculating node in this cluster, and the present invention is to calculate node
As a example by A, calculating node B, calculating node C, calculating node D, streaming meteorological data can be sent to calculate joint by data transmission blocks 2
The port of some A, the port of calculating node B, the port of calculating node C, the port of calculating node D.
In concrete implementation process, country's station data are different due to data form with regional station data, it is therefore desirable to divide
Do not resolve to different data forms.Generally speaking, when sending, country's station data need to resolve JSON form, obtain correspondence
Property value, and to specify time interval to be sent to the port that user sets.Regional station data then need to resolve file
After with appointed interval to certain port send data.
For country's station data:
Owing to the latest data of country's station data is second country's station data, therefore data transmission blocks 2 also includes:
First parsing module, for resolving described second country's station data, it is thus achieved that described second moment is corresponding
The meteorological property value that in described N number of website, each website is corresponding.Specifically, second country's station data carry out resolving to JSON
(a kind of data form) form, to obtain corresponding meteorological property value.
Send submodule, for specifying each calculating node port by SOCKET socket, send with appointed interval described
The meteorological property value that in described N number of website that second moment is corresponding, each website is corresponding to described specify respectively calculate node side
Mouthful.The time interval present invention specified does not limits, can be by system or be manually set time arbitrary interval.
For regional station data:
Owing to regional station data are TXT (text) file, it is therefore desirable to first resolve text, then with above-mentioned same side
Formula sends.
Data transmission blocks 2 includes:
Second parsing module, for resolving described regional station data, it is thus achieved that each website in described M website
Meteorological property value；
Concrete analysis mode is as follows:
1, judge whether regional station data have 15 row, the most then forward the 2nd step to, otherwise return format error；
2, travel through often row record, obtain property value, and be assigned to the attribute field of correspondence；
3, the attribute field of value is packaged into Attribute class, sends data for sending submodule.
Send submodule, be used for the meteorological property value of each website in appointed interval described M the website of transmission to described
That specifies respectively calculates node port.
After sending, native system is additionally provided with monitoring module 3, for described calculating node port is monitored, when
When listening to described streaming meteorological data by described each calculating node port, described streaming meteorological data is stored in each calculating and saves
Described streaming meteorological data is also backuped in distributed file system 5 by some internal memory.That is: the data flowed in real time are preserved respectively
In the internal memory and distributed file system 5 of each calculating node (respectively calculating node).Such as, streaming meteorological data is stored in calculating
The internal memory of node A, the internal memory of calculating node B, the internal memory of calculating node C, the internal memory of calculating node D.
User calculates node name and port numbers by configuring connection SOCKET, and when external program is set up and connected, data are sent out
Send module 2 to start to send data, and place the data in each calculating node memory, back up data to distributed field system simultaneously
In system 5, to realize the fault-tolerance of system.
The snoop procedure monitoring module 3 is implemented as follows:
1, a newly-built background thread, it is intended that connect calculating node name and the port numbers of SOCKET；
2, set an intervalometer, from the beginning of the 0th minute, the port numbers to specifying per minute is monitored once, if no data,
Then thread is in blocked state；If there being data, then obtain data and be stored in internal memory in corresponding metadata table.
After real time data acquisition module 1 Real-time Collection to country's station data, then carried out sending out by data transmission blocks 2
Sending, and be stored in each calculating node memory by data decryptor module 3 data of country being stood, therefore country station data only have data conversion storage
Record, and do not store in locally created catalogue, and then need not delete.And regional station data are being stored in distribution
After formula file system 5 realizes redundancy backup, the file processed in local FTP catalogue can be deleted, to prevent a large amount of literary composition simultaneously
Part is piled up.
Processing module 4, for carrying out structuring process, the list structure that design is corresponding based on described streaming meteorological data, builds
Lithol draws, and inquires about for user.When providing inquiry service, mainly provide the data access interface of class SQL.
Specifically, according to Website Hosting and meteorological community set, the unit that design is corresponding of the streaming meteorological data obtained
Tables of data, its Chinese Home station data leave in country's station metadata table, and regional station data leave in regional station metadata table.
Metadata table is sent to each and calculates nodal cache in internal memory so that the data in each nodes sharing internal memory.Basis simultaneously
Weather service query demand, design country station index of metadata table and regional station index of metadata table.Such as, metadata table is sent out
Deliver to calculate node A internal memory in, calculate node B internal memory in, calculate node C internal memory in, calculate node D internal memory in.
Introduce the design of memory metadata list structure and the establishment process of distributed index in detail below.
The layout strategy of metadata table structure is as follows:
Design a bivariate table, with day, hour, minute, in year, the moon, the character string of site number composition is as major key, so
Can deposit data according to the sequencing of time, user, when inquiry, need not travel through full table and can get up-to-date data；
Other row of table then arrange row name according to the field name of Attribute class；
Table is divided into multiple row race, automatic Weather Station Back ground Information is divided into a Ge Lie race, during as inserted time, observation
Between, longitude, latitude, type of site etc..Other attribute information be then divided into another row race, as barometric minimum, hyperbar, 3
Hourly rainfall depth, visibility etc.；So when inquiry, may specify that row race is inquired about；
Table is carried out pre-subregion, divides according to the time, as monthly No. 1 00 time data leave a subregion in, No. 1
There is another subregion in data when 01, so when carrying out statistical analysis, only need to inquire about the data of specified partition, greatly
Decrease greatly the visit capacity of data；
It is respectively created country's station hour metadata table, country's station minute metadata table, unit of regional station hour according to this strategy
Tables of data, regional station minute metadata table, tourism station metadata table, traffic station metadata table.
For country's station data, by gas corresponding for each website in described N number of website corresponding described second moment
As property value correspondence leaves in metadata table, to form country's station hour metadata table, country's station minute metadata table, tourism
Stand metadata table, traffic station metadata table one or more；By described country station hour metadata table, described country station point
Clock metadata table, described tourism station metadata table, described traffic station metadata table one or more be stored in each calculating node
Internal memory preserves.
For regional station data in, the meteorological property value correspondence of each website in described M website is left unit's number
According in table, to form regional station hour metadata table, regional station minute metadata table, tourism station metadata table, traffic station unit number
According to one or more of table.By described regional station hour metadata table, described regional station minute metadata table, described tourism station
Metadata table, described traffic station metadata table one or more be stored in each calculating node memory preservation.
According to business demand, user is when inquiry, and querying condition is bound to include query time, so at distributed index
Establishment time, use secondary index strategy, be implemented as follows:
With day, hour, minute, in year, the moon, the character string of site number composition is as major key, when inquiry as the first order
Index；
To country's station hour metadata table, country station minute metadata table, regional station hour metadata table, regional station minute
Metadata table, tourism station metadata table, traffic station metadata table one or more be each respectively created a concordance list,
In each concordance list, literary name section include observation time, province, minute, hour, site number, type of site, as second level rope
Draw；
The first search concordance list when inquiry, if data exist, then returns；Otherwise inquiry first order index.
During inquiry:
Enquiry module, for receiving the inquiry request of user, to the described streaming meteorological data in each calculating node memory
Carry out distributed query and show.
Specifically, owing to the invention provides data access interface, and then user can pass through JDBC (Java Data
Base Connectivity, java data base connects) client connection data access interface, send inquiry request.Enquiry module
Receiving, inquiry request is converted into class SQL (SQL, Structured Query Language) inquiry language
Sentence, and be further converted to parallel task and be distributed to each nodal parallel that calculates of multimachine network cluster and calculate, to each calculating node
Data in internal memory carry out distributed query, by the meteorological data information that inquires with WEBGIS, figure, form and listed files
Show etc. mode.
As an example it is assumed that calculate node A be master computing node, calculate node B, calculate node C, calculate node D be from
Calculate node.When having query task, calculate node A, calculating task is expanded to multiple cooperated computing node, perform task
Selectivity indicating gage operator node B, calculating node C, calculating node D carry out parallel computation inquiry and obtain Query Result.
Specifically introduce data visualization service function:
Meteorological data visualization function utilizes the browser drawing techniques such as HTML5, Flash, WebGL, it is achieved conventional gas
As attribute data graphics process serviced component, support real time graph output and interactive drawing, as important precipitation, temperature, air pressure,
The mottle figure of the conventional key element such as wind speed, humidity, isogram, variation tendency, multistation comparison diagram, sounding curve chart, Wind outline figure
Deng.Simultaneously according to the display needs of different user, it is provided that the visual presentation of various ways, user is possible not only to carry out general
Graph image browses, it is also possible to combine the function (map (GIS) service utilizing publicly-owned cloud service to provide) of GIS, it is achieved multiple
The Overlapping display of Meteorological Services graphic product, such as scattergram and the mottle figure of significant weather phenomenon, typhoon forecast path and reality
Path stacking chart, the hodograph of different layers, the mottle figure of conventional key element, the equivalence such as precipitation, temperature, air pressure, wind speed, humidity
Line chart, thunder and lightning figure, radar mosaic, Model Products isogram, mottle figure, the meansigma methods figure of many numerical forecasting product, the torrid zone
Trajectory of cyclone figure etc..Meanwhile, this function also provides for online interactive operation, and user can be by changing the display of visualization data
Parameter, the organizational form etc. of data, obtain different data display effects.(as selected different image enhaucament parameters to show not
Same image).
Memory module, for when judging whether the threshold value of each calculating node memory reaches predetermined threshold；The most then by each meter
Described streaming meteorological data in operator node internal memory dumps in distributed data base, and deletes the institute in each calculating node memory
State streaming meteorological data, it is ensured that internal memory has enough spaces deposit real-time streaming data.
Specifically, the historical data in internal memory is dumped in distributed data base, needs to design corresponding list structure,
According to system configuration settings memory threshold, reach upper threshold when system monitor monitors internal memory, then by the number in internal memory
According in the metadata table being stored in distributed data base, build distributed index simultaneously, with realize historical data quick search,
Display is analyzed.
The list structure of distributed data base and Index Design, similar with the design of concordance list with memory metadata table, below
Specifically introduce internal memory Replacement Strategy:
A newly-built background thread, and the upper threshold of internal memory is set, such as 200GB；
One timed task is set, respectively calculates the internal memory of node remain within the per hour the 0th minute, to start to monitor
Complementary space, if not up to upper threshold, then thread is in blocked state；If reaching the internal memory upper limit, then
Data in internal memory are inserted in distributed data base, and delete corresponding data in memory metadata table.
The immediate processing method of a kind of streaming meteorological data is described below, and described method is applied to described streaming meteorological data
Quick processing system in.
Specifically, refer to Fig. 2, the implementation process of the method is as follows:
S101, streaming meteorological data described in Real-time Collection.
Dividing according to source, described streaming meteorological data includes country's station data and regional station data.Country's station data refer to
Be comprehensive national Meteorological Information Sharing Platform obtain automatic Weather Station meteorological data (referred to as: CIMISS data).And regional station number
According to referring to the automatic Weather Station meteorological data that provincial comprehensive Meteorological Information Sharing Platform is shared, (referred to as: local live file).
Specifically, the data access interface that country's station data are provided by comprehensive national Meteorological Information Sharing Platform,
User sets after needing to obtain the Website Hosting of data, elements combination, by timing to comprehensive national Meteorological Information Sharing Platform
Send data acquisition request and obtain up-to-date country's station data that national interface can obtain, return with JSON form.Region
Data of standing (local live file) are then to obtain up-to-date meteorological data file from provincial interface, are sent in real time by FTP service
In the local FTP catalogue that user specifies.
Carry out concrete introduction below.
For country's station data:
Send the first data acquisition request extremely described comprehensive national Meteorological Information Sharing Platform in the first moment, and receive institute
State first country's station data that comprehensive national Meteorological Information Sharing Platform returns；Described first data acquisition request carries website
Set and meteorological community set, wherein, enumerate the N number of website needing to share data in described Website Hosting, described meteorology belongs to
Property set in enumerate the meteorological attribute that each website in N number of website is corresponding so that described comprehensive national meteorological information share put down
Stylobate obtains, in described first data acquisition request, the gas that in described N number of website corresponding to described first moment, each website is corresponding
As property value, described first country's station data are the gas that in described N number of website corresponding to described first moment, each website is corresponding
As property value；N is positive integer；
The second moment after described first moment sends the second data acquisition request and believes to described comprehensive national meteorology
Breath shared platform, and receive second country's station data that described comprehensive national Meteorological Information Sharing Platform returns, described second state
Family station data be described second moment corresponding country station data；Described second data acquisition request carries described Website Hosting
With described meteorological community set so that described comprehensive national Meteorological Information Sharing Platform obtains based on described second data acquisition request
The meteorological property value that each website is corresponding in described N number of website corresponding to described second moment, described second country's station data
It it is the meteorological property value that in described N number of website corresponding to described second moment, each website is corresponding；
If described first country's station data are different with described second country's station data, then use described second country's station data
Update described first country's station data, so that described country station data keep up-to-date.
Citing below illustrates:
(1) user sets country's station Website Hosting, property value set, data code (such as ground hour, country station data, state
Ground minute, family station data)；
(2) being out-of-sequence transmission due to meteorological data, transmitting procedure, by web influence, simultaneously takes account of the read-write of system
Can, when processing country's station ground hour data, timing performed a subtask at 0th minute hourly, from comprehensive national gas
Image information shared platform is downloaded the first country's station data obtaining N number of website.Timing is the the the the per hour the 2nd, 3,4,5,6,7 again,
Within the 0th second of 8,9,10,20 minutes and the 30th second, perform CIMISS data interface calls to obtain second country's station data, and general
Second country's station incoming first timing acquiring module of data carries out the renewal of country's station data, if in first timing acquiring module
Country's station data of country station data and second timing acquiring module identical, do not update, otherwise utilize second timing to adopt
Country's station data of collection module update country's station data of first timing acquiring module.
(3) when processing country's station ground minute data, owing to minute data was spaced more new data with 5 minutes, data are more
New frequency is fast, sends three timed tasks when processing, and timing is at the 0th, 5,10,15,20,25,30,35,40,45,50,55 point
0th second execution one subtask of clock, downloads from comprehensive national Meteorological Information Sharing Platform and obtains the station to be shared required task
First country's station data of point.Timing is the 2nd again, and 4,6,8,11,13,16,18,21,23,26,28,31,33,36,38,41,
The 10th second execution CIMISS data interface of 43,46,48,51,53,56,58 minutes is called to obtain second country's station data, with
More new data.Timing carries out data renewal in data traversal task of execution in the 0th second of the per hour the 32nd minute again, if upper one
Hour data do not capture completely, amended record data the most again.
(4) return for calling with JSON form by result.
For regional station data:
Regional station data (also referred to as: local automatic Weather Station live file) utilize provincial Meteorological Information Sharing Platform to provide
FTP service is sent to the meteorological data file in the FTP data share directory specified that the system of the present invention is locally created in real time.
And then regional station data only need to call from FTP data share directory in real time.Wherein, described regional station data are provincial gas
Image information shared platform Real-time Collection to M website in the meteorological property value of each website, M is positive integer.
Specifically, system needs the most locally created FTP data share directory, and sets the access right of this catalogue
Limit.And provincial Meteorological Information Sharing Platform enables data router in transmission, from the original storehouse of the meteorological data of shared platform
Automatic Weather Station meteorological data is packaged into TXT (text) file, and the FTP data that TXT file is sent to local setting shares mesh
In record.
Local monitoring FTP data share directory, when under provincial shared platform transmission TXT file to this catalogue, then solves
Analysis.
The formalized description of local live file is: L={L1, L2..., Ln| n=15}, wherein, n represents the row of file
Number, the 1st row record L1One-dimensional vector is used to be expressed as L1=district's station number, and observation time, longitude, latitude, observation field height above sea level,
Baroceptor height above sea level, observed pattern, quality control identifies, file corrigendum mark }；L2～L15Then use identification code gentle
As community set identifies, i.e. Li={ Pi, Ti| i >=2, i≤15}, wherein PiRepresent the identification code of the i-th row, TiRepresent the i-th row
Meteorological community set.
Such as, L2=PP, and station pressure, sea-level pressure, 3 hours transformations, 24 hours transformations, the highest station pressure,
The highest station pressure time of occurrence, minimum station pressure, minimum station pressure time of occurrence }, wherein PP represents air pressure attribute mark
Know.
S102, be sent to described streaming meteorological data to specify respectively calculates node port.
In concrete implementation process, owing to the latest data of country's station data is second country's station data, therefore sending out
Before sending, described second country's station data are resolved, it is thus achieved that each station in described N number of website that described second moment is corresponding
The meteorological property value that point is corresponding.Specifically, second country's station data carry out resolving to JSON form, to obtain corresponding meteorology
Property value.Then specify each calculating node port by SOCKET socket, send described second moment with appointed interval corresponding
Described N number of website in meteorological property value corresponding to each website to described specify respectively calculate node port.The time specified
Interval is the present invention do not limit, can be by system or be manually set time arbitrary interval.
For regional station data:
Owing to regional station data are TXT (text) file, it is therefore desirable to first resolve text, then with above-mentioned same side
Formula sends.
Concrete analysis mode is as follows:
1, judge whether regional station data have 15 row, the most then forward the 2nd step to, otherwise return format error；
2, travel through often row record, obtain property value, and be assigned to the attribute field of correspondence；
3, the attribute field of value is packaged into Attribute class, sends data for sending submodule.
After parsing, with the meteorological property value of each website in appointed interval described M the website of transmission to described appointment
Respectively calculate node port.
S103, monitors described calculating node port, when listening to described streaming meteorological data by described each meter
During operator node port, described streaming meteorological data is stored in each calculating node memory and described streaming meteorological data is backuped to point
In cloth file system 5.That is: the data flowed in real time are saved in respectively the internal memory of each calculating node (respectively calculating node) and divide
In cloth file system 5.
User calculates node name and port numbers by configuring connection SOCKET, starts when external program is set up and connected to send
Data, and place the data in each calculating node memory, back up data in distributed file system 5, to realize system simultaneously
The fault-tolerance of system.
Snoop procedure is implemented as follows:
1, a newly-built background thread, it is intended that connect calculating node name and the port numbers of SOCKET；
2, set an intervalometer, from the beginning of the 0th minute, the port numbers to specifying per minute is monitored once, if no data,
Then thread is in blocked state；If there being data, then obtain data and be stored in internal memory in corresponding metadata table.
Owing to country's station data are to be directly stored in each calculating node memory, therefore country station data only have the note of data conversion storage
Record, and do not store in locally created catalogue, and then need not delete.And regional station data are being stored in distributed literary composition
After part system 5 realizes redundancy backup, the file processed in local FTP catalogue can be deleted, to prevent heap file heap simultaneously
Long-pending.
S104, carries out structuring process, the list structure that design is corresponding based on described streaming meteorological data, sets up index, with
Inquire about for user.
Specifically, according to Website Hosting and meteorological community set, the unit that design is corresponding of the streaming meteorological data obtained
Tables of data, its Chinese Home station data leave in country's station metadata table, and regional station data leave in regional station metadata table.
Metadata table is sent to each and calculates nodal cache in internal memory so that the data in each nodes sharing internal memory.Basis simultaneously
Weather service query demand, design country station index of metadata table and regional station index of metadata table.
The design of metadata table structure and the establishment process of distributed index refer to above-described embodiment, in this present invention not
Repeat again.
And when inquiry, then receive the inquiry request of user, to the described streaming meteorological data in each calculating node memory
Carry out distributed query and show.
Specifically, owing to the invention provides data access interface, and then user can connect number by JDBC client
According to access interface, send inquiry request.Inquiry request is converted into class SQL query statement receiving by enquiry module, goes forward side by side one
Step is converted into parallel task and is distributed to each calculating node, the data in each calculating node memory is carried out distributed query, will look into
The meteorological data information ask shows in modes such as WEBGIS, figure, form and listed files.
Meteorological data visualization function utilizes the browser drawing techniques such as HTML5, Flash, WebGL, it is achieved conventional gas
As attribute data graphics process serviced component, support real time graph output and interactive drawing, as important precipitation, temperature, air pressure,
The mottle figure of the conventional key element such as wind speed, humidity, isogram, variation tendency, multistation comparison diagram, sounding curve chart, Wind outline figure
Deng.Simultaneously according to the display needs of different user, it is provided that the visual presentation of various ways, user is possible not only to carry out general
Graph image browses, it is also possible to combine the function (map (GIS) service utilizing publicly-owned cloud service to provide) of GIS, it is achieved multiple
The Overlapping display of Meteorological Services graphic product, such as scattergram and the mottle figure of significant weather phenomenon, typhoon forecast path and reality
Path stacking chart, the hodograph of different layers, the mottle figure of conventional key element, the equivalence such as precipitation, temperature, air pressure, wind speed, humidity
Line chart, thunder and lightning figure, radar mosaic, Model Products isogram, mottle figure, the meansigma methods figure of many numerical forecasting product, the torrid zone
Trajectory of cyclone figure etc..Meanwhile, this function also provides for online interactive operation, and user can be by changing the display of visualization data
Parameter, the organizational form etc. of data, obtain different data display effects.(as selected different image enhaucament parameters to show not
Same image).
The list structure of distributed data base and Index Design, similar with the design of concordance list with memory metadata table, below
Specifically introduce internal memory Replacement Strategy:
A newly-built background thread, and the upper threshold of internal memory is set, such as 200GB；
One timed task is set, respectively calculates the internal memory of node remain within the per hour the 0th minute, to start to monitor
Complementary space, if not up to upper threshold, then thread is in blocked state；If reaching the internal memory upper limit, then
Data in internal memory are inserted in distributed data base, and delete corresponding data in memory metadata table.
Further below with Hubei country station and real-time streaming meteorological data citing during regional station 20 days 12 May in 2016
Describe the actual application of the embodiment of the present invention in detail, comprise the following steps:
1, when obtaining Hubei country station and regional station 20 days 12 May in 2016 in real time by interface and FTP service hour
And minute data；
2, resolve this data, and transmit data to calculating node and the port specified；
3, oracle listener monitors calculating node and the port that step 302 sets, and data is stored in each calculating node
Deposit with in distributed file system 5；
4, the size of system oracle listener detection free memory；
5, if not up to threshold value, then by the metadata table structure being pre-designed, data are stored in table, update index simultaneously
Table；Otherwise, step 206 is forwarded to；
6, user submits query statement to, shows meteorological data information by the mode specified；
7, by the data conversion storage in internal memory to distributed data base, update the index of metadata table in internal memory and pin simultaneously
Concordance list to distributed data base table.
Visible, the immediate processing method of the embodiment of the present invention a kind of streaming meteorological data, can be Meteorological Automatic Station data
Refreshes in real time, show in real time, inquire about, add up, analyze a kind of new technical solution is provided.The present invention can be applied directly to
Typical case's industry such as weather forecast, the analysis of public opinion, disaster monitoring, improves the service feature that its big data process.
It should be noted that the method for the embodiment of the present invention is suitable for all of streaming meteorological data；Intervalometer in the present invention
Setting be not limited solely to the method employed in specific embodiment；The present invention is not intended to distributed data base prototype；The present invention
It is not limited solely to the building mode of secondary index in specific embodiment.
Above the embodiment of the present invention is described in detail, the detailed description of the invention of application in the present invention is explained
Stating, the explanation of above example is only applicable to help and understands the inventive method；Simultaneously, it should be appreciated that common to this area
For technical staff, can modifications and variations the most in addition, and these modifications and variations all should belong to institute of the present invention
Attached scope of the claims.
By one or more embodiment of the present invention, the invention have the advantages that or advantage:
The invention discloses the immediate processing method of a kind of streaming meteorological data, concrete implementation process is: Real-time Collection
Described streaming meteorological data, described streaming meteorological data includes country's station data and regional station data；By described streaming meteorology number
Node port is respectively calculated according to be sent to specify；Described each calculating node port is monitored, when listening to described streaming gas
When image data is by described each calculating node port, described streaming meteorological data is stored in each calculating node memory and by described stream
Formula meteorological data backups in distributed file system, is then based on described streaming meteorological data and carries out structuring process, design
Corresponding list structure, sets up index, inquires about for user.The present invention has taken into full account the spy of the streaming meteorological data of typical industry
Point, it is possible to realize the Real-time Collection of streaming meteorological data, statistical analysis, and then be supplied to the purpose of user's inquiry.
Although having been described for the preferred embodiment of the application, but one of ordinary skilled in the art once knowing substantially
Creative concept, then can make other change and amendment to these embodiments.So, claims are intended to be construed to bag
Include preferred embodiment and fall into all changes and the amendment of the application scope.
Obviously, those skilled in the art can carry out various change and the modification essence without deviating from the application to the application
God and scope.So, if these amendments of the application and modification belong to the scope of the application claim and equivalent technologies thereof
Within, then the application is also intended to comprise these change and modification.
Claims (10)
1. an immediate processing method for streaming meteorological data, described method is applied to the quick process of described streaming meteorological data
In system, it is characterised in that described method includes:
Streaming meteorological data described in Real-time Collection, described streaming meteorological data includes country's station data and regional station data；
Be sent to described streaming meteorological data to specify respectively calculates node port；
Described each calculating node port is monitored, when listening to described streaming meteorological data by described each calculating node side
During mouth, described streaming meteorological data is stored in each calculating node memory and described streaming meteorological data is backuped to distributed document
In system；
Carry out structuring process, the list structure that design is corresponding based on described streaming meteorological data, set up index, look into for user
Ask.
2. the method for claim 1, it is characterised in that when described streaming meteorological data is described country station data,
Streaming meteorological data described in described Real-time Collection, particularly as follows:
Send the first data acquisition request to described comprehensive national Meteorological Information Sharing Platform in the first moment, and receive described entirely
First country's station data that the comprehensive Meteorological Information Sharing Platform of state returns；Described first data acquisition request carries Website Hosting
With meteorological community set, wherein, described Website Hosting enumerates the N number of website needing to share data, described meteorological property set
Conjunction enumerates the meteorological attribute that each website in N number of website is corresponding, so that described comprehensive national Meteorological Information Sharing Platform base
The meteorological genus that each website is corresponding in described N number of website that described first data acquisition request acquisition described first moment is corresponding
Property value, described first country's station data are the meteorological genus that in described N number of website corresponding to described first moment, each website is corresponding
Property value；N is positive integer；
The second moment after described first moment sends the second data acquisition request to described comprehensive national weather information altogether
Enjoy platform, and receive second country's station data that described comprehensive national Meteorological Information Sharing Platform returns, described second country station
Data be described second moment corresponding country station data；Described second data acquisition request is carried described Website Hosting and institute
State meteorological community set, so that described comprehensive national Meteorological Information Sharing Platform obtains institute based on described second data acquisition request
Stating the meteorological property value that in described N number of website corresponding to the second moment, each website is corresponding, described second country's station data are institutes
State the meteorological property value that in described N number of website corresponding to the second moment, each website is corresponding；
If described first country's station data are different with described second country's station data, then described second country's station data are used to update
Described first country's station data, so that described country station data keep up-to-date.
3. method as claimed in claim 2, it is characterised in that when described streaming meteorological data is described country station data,
Described be sent to described streaming meteorological data to specify respectively calculate node port, including:
Described second country's station data are resolved, it is thus achieved that each website in described N number of website that described second moment is corresponding
Corresponding meteorological property value；
Each calculating node port is specified, with described N number of website that appointed interval transmission described second moment is corresponding by socket
In meteorological property value corresponding to each website to described specify respectively calculate node port.
4. method as claimed in claim 3, it is characterised in that described carry out structuring process based on described streaming meteorological data
And set up index, inquire about for user, including:
Leave meteorological property value correspondence corresponding for each website in described N number of website corresponding described second moment in metadata
In table, to form country's station hour metadata table, country's station minute metadata table, tourism station metadata table, traffic station metadata
One or more of table；By described country station hour metadata table, described country station minute metadata table, described tourism station unit
Tables of data, described traffic station metadata table one or more be stored in each calculating node memory preservation；
With day, hour, minute, in year, the moon, the character string of site number composition indexes as the first order；
To country station hour metadata table, country station minute metadata table, tourism station metadata table, the one of traffic station metadata table
Kind or multiple list structure each create concordance list, the literary name section in described concordance list include observation time, province, minute, little
Time, site number, type of site, as the second level index.
5. the method for claim 1, it is characterised in that when described streaming meteorological data is described regional station data,
Streaming meteorological data described in described Real-time Collection, particularly as follows:
Calling described regional station data from locally located appointment data sharing catalogue in real time, described regional station data are by provincial
Meteorological Information Sharing Platform is sent in described appointment data sharing catalogue in real time；Wherein, described regional station data are provincial gas
The meteorological property value of each website in M the website that image information shared platform collects, M is positive integer.
6. method as claimed in claim 5, it is characterised in that when described streaming meteorological data is described regional station data,
Described be sent to described streaming meteorological data to specify respectively calculate node port, including:
Described regional station data are resolved, it is thus achieved that the meteorological property value of each website in described M website；
With appointed interval send the meteorological property value of each website in described M website to described specify respectively calculate node side
Mouthful.
7. method as claimed in claim 6, it is characterised in that described carry out structuring process based on described streaming meteorological data
And set up index, inquire about for user, including:
The meteorological property value correspondence of each website in described M website is left in metadata table, to form regional station hour
Metadata table, regional station minute metadata table, tourism station metadata table, traffic station metadata table one or more；
By described regional station hour metadata table, described regional station minute metadata table, described tourism station metadata table, described friendship
One or more of logical station metadata table are stored in each calculating node memory preservation；
With day, hour, minute, in year, the moon, the character string of site number composition indexes as the first order；
To regional station hour metadata table, regional station minute metadata table, tourism station metadata table, the one of traffic station metadata table
Kind or multiple list structure each create concordance list, the literary name section in described concordance list include observation time, province, minute, little
Time, site number, type of site, as the second level index.
8. the method for claim 1, it is characterised in that described carry out structuring process based on described streaming meteorological data
And set up index, after inquiring about for user, described method also includes:
The inquiry request receiving user carries out parallel computation to each calculating node, and then to the described stream in each calculating node memory
Formula meteorological data carries out distributed query and shows.
9. the method for claim 1, it is characterised in that described carry out structuring process based on described streaming meteorological data
And set up index, after inquiring about for user, including:
Judge whether the threshold value of each calculating node memory reaches predetermined threshold；
The described streaming meteorological data in the most each calculating node memory is dumped in distributed data base, and delete each
Calculate the described streaming meteorological data in node memory.
10. the quick processing system of a streaming meteorological data, it is characterised in that including:
Real time data acquisition module, for streaming meteorological data described in Real-time Collection, described streaming meteorological data includes country station
Data and regional station data；
Data transmission blocks, respectively calculates node port for be sent to described streaming meteorological data to specify；
Monitor module, for described each calculating node port is monitored, pass through institute when listening to described streaming meteorological data
When stating each calculating node port, described streaming meteorological data is stored in each calculating node memory and by standby for described streaming meteorological data
Part in distributed file system；
Processing module, for carrying out structuring process, the list structure that design is corresponding based on described streaming meteorological data, sets up rope
Draw, inquire about for user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610534963.9A CN106202378A (en) | 2016-07-08 | 2016-07-08 | The immediate processing method of a kind of streaming meteorological data and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610534963.9A CN106202378A (en) | 2016-07-08 | 2016-07-08 | The immediate processing method of a kind of streaming meteorological data and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106202378A true CN106202378A (en) | 2016-12-07 |
Family
ID=57472749
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610534963.9A Pending CN106202378A (en) | 2016-07-08 | 2016-07-08 | The immediate processing method of a kind of streaming meteorological data and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106202378A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106802961A (en) * | 2017-02-15 | 2017-06-06 | 北京佳格天地科技有限公司 | Meteorological managing device, method and computer system based on plot |
| CN107102382A (en) * | 2017-03-20 | 2017-08-29 | 南京信息工程大学 | A kind of dynamic analysis meteorological data retransmission unit |
| CN107403458A (en) * | 2017-07-27 | 2017-11-28 | 郑州云海信息技术有限公司 | A kind of wind direction data processing method and processing device |
| CN108416022A (en) * | 2018-03-07 | 2018-08-17 | 电信科学技术第五研究所有限公司 | A kind of real-time fidelity Drawing of Curve model realization system and method for stream data |
| CN108763323A (en) * | 2018-05-03 | 2018-11-06 | 华风象辑（北京）气象科技有限公司 | Meteorological lattice point file application process based on resource set and big data technology |
| CN110703256A (en) * | 2019-10-24 | 2020-01-17 | 上海眼控科技股份有限公司 | Radar data display method and device, computer equipment and storage medium |
| CN112527880A (en) * | 2020-12-16 | 2021-03-19 | 平安壹钱包电子商务有限公司 | Method, device, equipment and medium for collecting metadata information of big data cluster |
| CN113688100A (en) * | 2021-09-06 | 2021-11-23 | 北京普睿德利科技有限公司 | Meteorological data processing method, device, terminal and storage medium |
| CN113986911A (en) * | 2021-12-29 | 2022-01-28 | 中国气象局气象探测中心 | Information acquisition method and system for monitoring racing field operation |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104506373A (en) * | 2015-01-07 | 2015-04-08 | 国家计算机网络与信息安全管理中心 | Device and method for collecting and processing network information |
| CN104750749A (en) * | 2013-12-31 | 2015-07-01 | 阿里巴巴集团控股有限公司 | Data processing method and data processing device |
-
2016
- 2016-07-08 CN CN201610534963.9A patent/CN106202378A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104750749A (en) * | 2013-12-31 | 2015-07-01 | 阿里巴巴集团控股有限公司 | Data processing method and data processing device |
| CN104506373A (en) * | 2015-01-07 | 2015-04-08 | 国家计算机网络与信息安全管理中心 | Device and method for collecting and processing network information |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106802961B (en) * | 2017-02-15 | 2020-06-02 | 北京佳格天地科技有限公司 | Meteorological management device and method based on land parcel and computer system |
| CN106802961A (en) * | 2017-02-15 | 2017-06-06 | 北京佳格天地科技有限公司 | Meteorological managing device, method and computer system based on plot |
| CN107102382A (en) * | 2017-03-20 | 2017-08-29 | 南京信息工程大学 | A kind of dynamic analysis meteorological data retransmission unit |
| CN107102382B (en) * | 2017-03-20 | 2019-12-06 | 南京信息工程大学 | dynamic analytic meteorological data forwarding device |
| CN107403458A (en) * | 2017-07-27 | 2017-11-28 | 郑州云海信息技术有限公司 | A kind of wind direction data processing method and processing device |
| CN108416022A (en) * | 2018-03-07 | 2018-08-17 | 电信科学技术第五研究所有限公司 | A kind of real-time fidelity Drawing of Curve model realization system and method for stream data |
| CN108416022B (en) * | 2018-03-07 | 2020-06-09 | 电信科学技术第五研究所有限公司 | System and method for realizing streaming data real-time fidelity curve drawing model |
| CN108763323A (en) * | 2018-05-03 | 2018-11-06 | 华风象辑（北京）气象科技有限公司 | Meteorological lattice point file application process based on resource set and big data technology |
| CN108763323B (en) * | 2018-05-03 | 2022-03-15 | 华风象辑（北京）气象科技有限公司 | Meteorological grid point file application method based on resource set and big data technology |
| CN110703256A (en) * | 2019-10-24 | 2020-01-17 | 上海眼控科技股份有限公司 | Radar data display method and device, computer equipment and storage medium |
| CN112527880A (en) * | 2020-12-16 | 2021-03-19 | 平安壹钱包电子商务有限公司 | Method, device, equipment and medium for collecting metadata information of big data cluster |
| CN112527880B (en) * | 2020-12-16 | 2023-08-08 | 平安壹钱包电子商务有限公司 | Method, device, equipment and medium for collecting metadata information of big data cluster |
| CN113688100A (en) * | 2021-09-06 | 2021-11-23 | 北京普睿德利科技有限公司 | Meteorological data processing method, device, terminal and storage medium |
| CN113688100B (en) * | 2021-09-06 | 2023-07-18 | 北京普睿德利科技有限公司 | Meteorological data processing method, device, terminal and storage medium |
| CN113986911A (en) * | 2021-12-29 | 2022-01-28 | 中国气象局气象探测中心 | Information acquisition method and system for monitoring racing field operation |
| CN113986911B (en) * | 2021-12-29 | 2022-04-01 | 中国气象局气象探测中心 | Information acquisition method and system for monitoring racing field operation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106202378A (en) | The immediate processing method of a kind of streaming meteorological data and system | |
| CN103345508B (en) | A kind of date storage method being applicable to community network figure and system | |
| CN102023983B (en) | Managing method of statistical space-time database | |
| CN103019728B (en) | Effective complex report parsing engine and parsing method thereof | |
| CN102693306A (en) | On-line linkage update service method of multi-scale geographic data | |
| CN103268542B (en) | A kind of machining equipment resource informationization method that facing cloud manufactures | |
| CN109871418A (en) | A kind of space index method and system of space-time data | |
| CN116089555A (en) | CIM platform-based three-dimensional space data acquisition and light weight system and method | |
| CN201853252U (en) | Wetland resource and ecological environment supervising system | |
| CN111708778B (en) | Big data management method and system | |
| CN106202447A (en) | A kind of possess the digital cartography system produced with management integration | |
| CN102855332A (en) | Graphic configuration management database based on graphic database | |
| CN111488420B (en) | Flood early warning water information system for decentralized micro-service area and integration method thereof | |
| CN108595611A (en) | Based on one point one scape technology space-time big data on-line acquisition system and its method | |
| CN104199955B (en) | information management system and method | |
| CN113570275A (en) | Water resource real-time monitoring system based on BIM and digital elevation model | |
| CN104008209B (en) | Reading-writing method for MongoDB cluster geographic data stored with GeoJSON format structuring method | |
| CN102158533B (en) | Distributed web service selection method based on QoS (Quality of Service) | |
| CN101719148A (en) | Three-dimensional spatial information saving method, device, system and dispatching system | |
| CN112132954B (en) | Distributed management method and system for three-dimensional laser point cloud entity object | |
| CN107870949A (en) | Data analysis job dependence relation generation method and system | |
| CN113674135A (en) | Calculation method for realizing CALPUFF high performance based on workflow | |
| Wei et al. | Discover, visualize, and deliver geospatial data through OGC standards-based WebGIS system | |
| Wu et al. | Research on data sharing architecture for ecological monitoring using Iot streaming data | |
| CN112231301A (en) | Yellow river water sand change data warehouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161207 |