CN107992584A - A kind of ocean big data classification parsing and gridding storage method - Google Patents
A kind of ocean big data classification parsing and gridding storage method Download PDFInfo
- Publication number
- CN107992584A CN107992584A CN201711293286.7A CN201711293286A CN107992584A CN 107992584 A CN107992584 A CN 107992584A CN 201711293286 A CN201711293286 A CN 201711293286A CN 107992584 A CN107992584 A CN 107992584A
- Authority
- CN
- China
- Prior art keywords
- data
- ocean
- storage
- oceanographic
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of ocean big data classification parsing and gridding storage method, can be applied to oceanographic data storage processing.This method builds data grids mapping ruler model, the data of marine exploration key element is regarded as to the data grids formed in certain time and space, are suitable for the multiresolution requirement of different types of data according to oceanographic observation data characteristics；The data of front end sensors collection are parsed, after classification parsing, picking different, duplicate removal, obtaining basic data after the pretreatment such as fill a vacancy, is mapped in data grids, realizes and represented according to ocean big data gridding；Realize that buffering area is interacted with data in magnetic disk using cache technology, Data Transference Technology, the storage and fast dispatch for realizing ocean big data use；And then to oceanographic data rapid tissue and retrieval, it can further realize ocean big data visualization.
Description
Technical field
The present invention relates to a kind of ocean big data classification parsing, data prediction and storage method.
Background technology
As ocean power strategy is constantly implemented and deepened, China relates to that extra large industrial economy is continual and steady, develops in a healthy way, to sea
Foreign converging information, storage propose increasingly strong demand with shared.Particularly existing oceanographic data is converged, handled and deposited again
Storage, helps to be formed three-dimensional, continuous, real-time, more key element comprehensive integration observing capacity, convergence magnanimity, multi-source, multiclass, various dimensions,
The marine information key element of multiresolution, meets to relate to sea-run industry, access of the specialty to the multi-source heterogeneous marine information of different scale and altogether
Enjoy demand；On the other hand as oceanographic data constantly converges, increases, the quantitative change of data scale can turn by multiple technologies approach
Information service qualitative change is turned to, is excavated using ocean big data, visualization technology means can improve the science for relating to sea-run industry decision-making
Property and specific aim.
Ocean big data message is complicated, and passes sensor back without unified storage standard, most of oceanographic observation systems
Message data be directly locally stored or parse after store, do not carry out classification parsing, pick different, duplicate removal, pretreatment of filling a vacancy, no
It is adapted to be uniformly processed on a large scale；The classification series of preprocessing such as parsing is carried out to initial data, facilitates data post storage and fast
Speed retrieval.The data of different waters different platform are separate, do not connect mutually, and data sharing is low with service efficiency, information products
Service guarantee ability is weak, it is difficult to meets growing ocean business demand, it is necessary to build unified storage model, data are carried out
Gridding storage, convergence, realize the interconnection, intercommunication, interoperability of oceanographic data.Therefore, carry out the classification of ocean big data and deposit parsing
Technical research is stored up with gridding, tamps ocean big data service infrastructure, data supporting can be provided for diversified ocean application service,
It is of great significance simultaneously for marine informatization construction.
With reference to concerned countries and professional standard, oceanographic data is divided into nine major classes according to its perceptual elements：Targets in ocean, sea
Foreign activity, marine hydrology, maritime meteorology, Oceanic disasters, Marine Chemistry, marine organisms, ocean acoustic photoelectromagnetic, ocean geography；Often
A marine information key element includes multiple variables, collectively constitutes ocean big data pond.The data class obtained according to different perception means
Type is different, can be divided into oceanographic data：Video data, view data, text data etc..It is big according to data processing degree, ocean
Data can divide three classes：Initial data, a data and secondary data.Wherein, initial data is that sensor gets to obtain most original
Data, such as oceanographic station data；Initial data obtains carrying after real time parsing after hierarchical classification standardization reorganization, pretreatment
A data for providing element information for ocean service；Data are associated with analysis, Transition Evaluation Data Assimilation, connection
Close the data for the face facies analysis application that the exemplary process such as inverting, data mining, environmental warning report are formed, it is possible to provide secondary data is produced
Product service and typical case service.Efficient storage has most important theories and practice significance with management ocean big data.
The present invention obtains marine information key element by carrying out classification parsing to sensor raw data, and according to nine class oceans
Data and the feature for perceiving variable, propose data grids concept, and structure mesh mapping rule model realizes the high speed of mass data
Effectively storage, realizes that buffering area is interacted with data in magnetic disk using cache technology, Data Transference Technology, realizes ocean big data
Storage and fast dispatch use；And then to oceanographic data rapid tissue and retrieval, complete the visualization of ocean big data.Lifting
Ocean big data utilization ratio.
The content of the invention
It is an object of the invention to provide a kind of technical method of classification, parsing, modeling and the storage of magnanimity oceanographic data,
Data-handling capacity is lifted, the extensive use for ocean big data provides support.
Realizing the solution of the present invention is：The data characteristics perceived according to ocean, by each ocean essential of oceanographic observation
Variable regard as in the field of spatio-temporal distribution, establish oceanographic data mesh mapping rule model.Front end sensors collection
Data first carry out dissection process, and sort data into, by going to refetch oceanographic data after the pretreatments such as essence are integrated, according to net
Lattice mapping ruler model, using information such as the position of gathered data, elevation, times, maps the data into corresponding save mesh, raw
Into ocean big data grid, easy to the rapid tissue of oceanographic data, the data resource of oceanographic observation key element can be extracted soon.Utilize height
Fast caching technology, Data Transference Technology realize that buffering area is interacted with data in magnetic disk, realize the storage of ocean big data and quick
Scheduling uses；And then can be further to realize data visualization to oceanographic data rapid tissue and retrieval.
Compared with traditional oceanographic data storage and scheduling use technology, the present invention has remarkable advantage：1. mesh mapping
Rule model can meet different pieces of information object, the multiresolution memory requirement of different space-times, be easy to data visualization.It is 2. opposite
It is locally stored in original message data, after this programme collection front-end sensor data, by classification parsing, goes to refetch precision processing
Afterwards, fine effective basic data can be provided for top service by realizing that the cleaning to initial data filters to be formed, and it is superfluous to reduce data
Remaining, and can trace to the source.3. by data fields and the mapping relations of oceanographic observation key element, can more efficiently obtain interested
A variety of marine environment variables, reduce related data extraction complexity.4. oceanographic data amount is huge, traditional method is by physics
Memory size limits and the limitation of data in magnetic disk reading speed is difficult to meet requirement of real-time, and dispatching efficiency is relatively low, this
The data Real-Time Scheduling between memory and disk is realized in invention using cache policy.
The present invention is described in further detail below in conjunction with the accompanying drawings.
Brief description of the drawings
Fig. 1 is oceanographic data gridding storage schematic diagram.
Fig. 2 is mesh mapping rule model.
Fig. 3 is ocean big data classification parsing with depositing gridding storage method flow chart.
Embodiment
Specific implementation method of the present invention is：
1. marine environment perceptive object is regarded as to the variable being distributed in different time and space, spatially can be with number
The variable of diverse location is stored according to the form of grid, is considered as the state change of data grids in time.Any given
Point on (x, y, h, t) all can obtain corresponding observation.Establish the mesh mapping rule model of environmental objects P：
P=f (x, y, h, t, id)
Wherein, x, y are the longitudes and latitudes of mesh point, and h is height above sea level, and id is the mark for identifying different ocean perceptive objects.And build
The data field model of multiple environmental objects is found, corresponding data is stored respectively, can be selected according to the different adaptabilities of object
With vector model and Scalar Model.The optional scalar fields such as such as optional scalar field of air pressure, ocean temperature, course, wind direction.Root
Different from data source according to the Variation Features of environmental objects, the change resolution of Various types of data is larger, mesh mapping rule model
It can meet multiresolution requirement.
2. generation is with longitude and latitude, time and the multi-level tree-like mesh of information element, storage is by classification parsing, duplicate removal
Essence etc. is taken to obtain multi-source heterogeneous basic data after pre-processing.
3. according to mesh mapping rule model, using information such as the position of gathered data, height above sea level, times, data are mapped
To corresponding save mesh, ocean big data is shown as into data grids.According to each perceptive object feature, two dimension can be established, it is three-dimensional
Grid data space, if h is constant, data are stored as two-dimensional grid；When h changes, then three-dimensional grid is shown as.
4. grid data is put in storage, it can create, inquire about, be inserted into, update and delete using SQL syntax model realization tables of data
Remove, response user's request, realizes the aggregation and storage of marine environment data collection, and can fast dispatch, easy to data manipulation.
5. creating shared drive region using relational database high-speed buffer technology, united according to user data requests history
Meter, realizes that buffering area is quickly interacted with disk, the quick response of request of data and buffering area.Optimize data exchange algorithm, improve
Data migration method, lifts data access speed.
6. data visualization.Mesh mapping rule model can significant increase data visualization efficiency, according to mesh point attribute,
Data grids are drawn, are suitable for different resolution requirement, realize the visualization of Various types of data.
Claims (4)
1. a kind of ocean big data classification parsing and gridding storage method, it is characterized in that：It is primarily based on marine environment perception pair
The time-space relationship and key element intension reduction of elephant, build the mesh mapping rule model of ocean big data, and formation meets more points of isomery
The storage mapping descriptive power of resolution data；Generation has longitude and latitude, time and the multi-level tree-like storage grid of information element,
After the pretreatment for complete classification parsing, picking different, duplicate removal, filling a vacancy, isomerous multi-source data will be mapped using mapping model, storage is arrived
In two dimension or three-dimensional grid；Shared drive finally is created using non-relational database cache technology, improves request of data
With exchange efficiency, meet data access and visual demand.
2. a kind of ocean big data classification parsing according to claim 1 and gridding storage method, it is characterised in that：Pin
There is the feature of polyphyly, regionality, magnanimity, multidimensional, dynamic, space correlation, tense to oceanographic observation data, by oceanographic observation
Key element variable regards continuously distributed vector on room and time as, establishes the mesh mapping model P=f of oceanographic observation key element P
(x,y,h,t,id)；The storage need of corresponding data grids adaptation magnanimity oceanographic observation data are generated according to different types of data
Ask.
3. a kind of ocean big data classification parsing according to claim 1 and gridding storage method, it is characterised in that：Root
According to the main object of oceanographic observation, oceanographic data is divided into nine major classes：Targets in ocean, Activities of Ocean, marine hydrology, ocean gas
As, Oceanic disasters, Marine Chemistry, marine organisms, ocean acoustic photoelectromagnetic, ocean geography；After the parsing of front end sensors message data
Be divided into nine class of the above, per class data in include some variables again, according to types of variables in mesh mapping rule model, position, when
Between, the parameter such as resolution ratio, the corresponding site of tree-like storage structure is arrived into the mapping of corresponding variate-value, storage.
4. a kind of ocean big data classification parsing according to claim 1 and gridding storage method, it is characterised in that：Profit
Shared drive region is established with relevant database caching techniques, according to user's acess control, establishes shared drive region,
Realize that buffer area disk quickly interacts, optimization data access, switching channel, complete the aggregation of marine environment data collection and deposit
Storage, lifts data sharing and visualization capability.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711293286.7A CN107992584A (en) | 2017-12-08 | 2017-12-08 | A kind of ocean big data classification parsing and gridding storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711293286.7A CN107992584A (en) | 2017-12-08 | 2017-12-08 | A kind of ocean big data classification parsing and gridding storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107992584A true CN107992584A (en) | 2018-05-04 |
Family
ID=62036757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711293286.7A Pending CN107992584A (en) | 2017-12-08 | 2017-12-08 | A kind of ocean big data classification parsing and gridding storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107992584A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108961074A (en) * | 2018-06-22 | 2018-12-07 | 武汉欣网创业科技开发有限公司 | A kind of insurance gridding system |
| CN109684388A (en) * | 2018-12-29 | 2019-04-26 | 成都信息工程大学 | A kind of meteorological data index and visual analysis method based on hypercube lattice tree |
| CN109828988A (en) * | 2019-01-25 | 2019-05-31 | 重庆科技学院 | A kind of big data statistical method and the system for big data statistics |
| CN110516129A (en) * | 2019-08-30 | 2019-11-29 | 吉林大学 | A kind of data processing method and device |
| TWI761799B (en) * | 2020-04-01 | 2022-04-21 | 麥克海卓科技有限公司 | Integrated weather forecast system and method thereof for coastal waters |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103412962A (en) * | 2013-09-04 | 2013-11-27 | 国家测绘地理信息局卫星测绘应用中心 | Storage method and reading method for mass tile data |
-
2017
- 2017-12-08 CN CN201711293286.7A patent/CN107992584A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103412962A (en) * | 2013-09-04 | 2013-11-27 | 国家测绘地理信息局卫星测绘应用中心 | Storage method and reading method for mass tile data |
Non-Patent Citations (1)
| Title |
|---|
| 袁立成 等: "海洋网格——海洋环境信息存储与交换的数据网格", 《计算机应用》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108961074A (en) * | 2018-06-22 | 2018-12-07 | 武汉欣网创业科技开发有限公司 | A kind of insurance gridding system |
| CN109684388A (en) * | 2018-12-29 | 2019-04-26 | 成都信息工程大学 | A kind of meteorological data index and visual analysis method based on hypercube lattice tree |
| CN109684388B (en) * | 2018-12-29 | 2023-07-25 | 成都信息工程大学 | Meteorological data index and visual analysis method based on super-cubic grid tree |
| CN109828988A (en) * | 2019-01-25 | 2019-05-31 | 重庆科技学院 | A kind of big data statistical method and the system for big data statistics |
| CN110516129A (en) * | 2019-08-30 | 2019-11-29 | 吉林大学 | A kind of data processing method and device |
| CN110516129B (en) * | 2019-08-30 | 2022-07-01 | 吉林大学 | Data processing method and device |
| TWI761799B (en) * | 2020-04-01 | 2022-04-21 | 麥克海卓科技有限公司 | Integrated weather forecast system and method thereof for coastal waters |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107992584A (en) | A kind of ocean big data classification parsing and gridding storage method | |
| CN103425772B (en) | A kind of mass data inquiry method with multidimensional information | |
| CN103049549B (en) | A kind of island data management method and system | |
| CN103023970B (en) | Method and system for storing mass data of Internet of Things (IoT) | |
| CN108446293A (en) | A method of based on urban multi-source isomeric data structure city portrait | |
| CN106909644A (en) | A kind of multistage tissue and indexing means towards mass remote sensing image | |
| CN109635068A (en) | Mass remote sensing data high-efficiency tissue and method for quickly retrieving under cloud computing environment | |
| CN107193967A (en) | A kind of multi-source heterogeneous industry field big data handles full link solution | |
| CN108804602A (en) | A kind of distributed spatial data storage computational methods based on SPARK | |
| CN109299298A (en) | Construction method, device, application method and the system of image fusion model | |
| CN111275065A (en) | Aquaculture space partitioning method based on marine environment multiple attributes | |
| Qian et al. | Data science for oceanography: from small data to big data | |
| CN104699826B (en) | A kind of the pyramid laminar storage method and Spatial Database Systems of image data | |
| CN102646116A (en) | Remote visualization method for ocean current vector fields | |
| CN113742505A (en) | Mass synthetic aperture radar interferometric measurement (InSAR) data online visualization method | |
| CN115129795A (en) | Data space-time storage method based on geospatial grid | |
| Alkathiri et al. | Geo-spatial big data mining techniques | |
| CN107679126A (en) | Laser three-D cloud data stores and management method and its system | |
| CN104572648B (en) | A kind of storage statistical system and method based on high-performance calculation | |
| Tian et al. | Marine information sharing and publishing system: a WebGIS approach | |
| Yang et al. | Data resolution improvement for ocean of things based on improved FCM | |
| Chen et al. | Internet of things technology in ecological security assessment system of intelligent land | |
| CN108959420A (en) | The method of spatio-temporal data visualization interface rapidly locating | |
| Sun et al. | Regular Flow Field Sparse Processing and Blue Finance of Ocean City Research Based on GIS System | |
| CN116204511A (en) | Big data analysis management system based on cultural tourism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180504 |