CN111309252A - Intelligent space model under big data and construction method - Google Patents
Intelligent space model under big data and construction method Download PDFInfo
- Publication number
- CN111309252A CN111309252A CN202010071345.1A CN202010071345A CN111309252A CN 111309252 A CN111309252 A CN 111309252A CN 202010071345 A CN202010071345 A CN 202010071345A CN 111309252 A CN111309252 A CN 111309252A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- cluster
- resources
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 title abstract description 5
- 238000003860 storage Methods 0.000 claims abstract description 140
- 238000007726 management method Methods 0.000 claims abstract description 53
- 238000004519 manufacturing process Methods 0.000 claims abstract description 9
- 239000007787 solid Substances 0.000 claims abstract description 9
- 238000013499 data model Methods 0.000 claims abstract description 8
- 230000006870 function Effects 0.000 claims abstract description 7
- 230000005055 memory storage Effects 0.000 claims abstract description 5
- 238000000034 method Methods 0.000 claims description 11
- 238000013500 data storage Methods 0.000 claims description 10
- 230000007423 decrease Effects 0.000 claims 1
- 230000003247 decreasing effect Effects 0.000 claims 1
- 235000019633 pungent taste Nutrition 0.000 claims 1
- 238000003491 array Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an intelligent space model under big data and a construction method, a uniform storage resource pool is established by adopting a mixed storage medium and mixed storage frame mode, a hierarchical storage mode matching space data resources and storage resources is realized according to the calling heat of the space data resources, namely PC cluster and memory storage resources are adopted in data production, PC cluster and NAS storage are adopted in a service system, SAN array and solid state disk are adopted in cluster storage, SAN array and disk are adopted in archive management, and storage frames of a relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster (HDFS) facing spatial data are formed, and a standardized data access interface is provided, and the interface provides data model management, data catalog management, data resource management and data inquiry and browsing functions, so that the access and management of data in various storage modes are realized.
Description
Technical Field
The invention relates to the technical field of spatial information storage management, in particular to an intelligent spatial model under big data and a construction method.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
In the big data era of space, the storage resource is an important infrastructure resource as a physical location for storing multi-source heterogeneous multi-temporal space data at a large scale. With the continuous increase of the amount of spatial data, the user concurrency amount is continuously increased, which causes an I/O performance bottleneck of the conventional storage, and the conventional storage resources have weak capacity in the aspects of expansibility and high availability and relatively high management and maintenance costs.
The inventor finds that the traditional spatial data resource management usually adopts centralized relational database storage management or file data storage backup by using a disk array and a discretization hard disk. Under the structure, the storage management of the spatial data resources of a plurality of service systems is relatively independent and dispersed, which directly causes the dispersed management of the corresponding storage resources and low value discrimination, and the conditions of the flow direction, the application direction, the use frequency and the like of the data resources cannot be mastered in time.
On the aspect of the use heat of the space data and the filing management requirement, the problem of poor matching degree exists between the space data resources and the storage resources, the unused heat data exist on the same type of disk, the performance of the stored disk is reduced, the system efficiency is further influenced, and the filing data are stored in high-performance storage, so that the waste of the storage resources is caused.
Disclosure of Invention
In order to solve the problems, the invention provides an intelligent spatial model under big data and a construction method thereof, wherein a uniform storage resource pool is established by adopting a mixed storage medium and mixed storage frame mode, a hierarchical storage mode that the spatial data resources are matched with the storage resources is realized according to the calling heat of the spatial data resources, and storage frames of a relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster (HDFS) facing the spatial data are formed.
In some embodiments, the following technical scheme is adopted:
a big-data down smart space model, comprising: a resource storage pool consisting of a hybrid storage media and a hybrid storage framework; the resource storage pool is respectively connected with an external relational database, a NoSQL database, a shared file system and a distributed file system through a data storage access interface; and realizing the hierarchical storage of the spatial data resources and the storage resources matched according to the calling heat of the spatial data resources, and forming a storage frame of a spatial data-oriented relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster.
A unified storage resource pool is established by adopting a mixed storage medium and mixed storage frame mode, a hierarchical storage mode that the space data resources are matched with the storage resources is realized according to the calling heat of the space data resources, and the problem that the matching degree between the space data resources and the storage resources is poor is solved.
In other embodiments, the following technical solutions are adopted:
a method for constructing an intelligent space model under big data comprises the following steps:
constructing a resource storage pool by the hybrid storage media and the hybrid storage framework; and realizing the hierarchical storage of the spatial data resources and the storage resources matched according to the calling heat of the spatial data resources, and forming a storage frame of a spatial data-oriented relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster.
Compared with the prior art, the invention has the beneficial effects that:
(1) the invention provides an effective solution for the storage resource management of the space data resources in the big data era, realizes the centralized management of the storage management, and timely masters the flow direction, the application direction, the use frequency and other conditions of the data resources.
(2) The scheme of the invention has strong operability and easy operation and implementation, adopts a mode of a mixed storage medium and a mixed storage frame to set up a uniform storage resource pool, realizes a hierarchical storage mode of matching the space data resources and the storage resources according to the calling heat of the space data resources, and solves the problem of poor matching degree between the space data resources and the storage resources.
Drawings
FIG. 1 is a diagram of a smart space model under big data according to an embodiment of the present invention.
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
The embodiments and features of the embodiments of the present invention may be combined with each other without conflict.
The terms are explained below:
SAN: is a stored area network; the SAN card storage system is composed of optical fibers and SAN card storage of an SAN switch. Forming a storage network.
NAS (Network Attached Storage) is a device which is connected to a Network in a literal sense and has a data Storage function, and is also called a "Network Storage".
NoSQL: broadly refers to a non-relational database.
Example one
In one or more embodiments, a big data down smart space model is disclosed, with reference to fig. 1, comprising: a resource storage pool consisting of a hybrid storage media and a hybrid storage framework; the resource storage pool is respectively connected with an external relational database, a NoSQL database, a shared file system and a distributed file system through a data storage access interface; the method comprises the steps of calling heat according to spatial data resources, realizing hierarchical storage of the spatial data resources and storage resources in a matching mode, forming storage frames of a relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster which are oriented to the spatial data, supporting storage management of various data by the storage frames, and dividing the storage management modes of the relational database cluster, the NoSQL database cluster, the shared file system cluster and the distributed file system cluster (HDFS) corresponding to the spatial data according to data types.
The resource storage pool is composed of a mixed storage medium and a mixed storage framework, the mixed storage medium mainly comprises a disk, a solid state disk and a memory, and the mixed storage framework mainly comprises an SAN array, an NAS storage and a PC cluster.
Specifically, the calling heat of the spatial data resources is divided into data production, a service system, cluster storage and archive management, and the calling heat is sequentially reduced;
the method comprises the steps of realizing a hierarchical storage mode that space data resources are matched with storage resources, calling the hierarchical storage resources matched with the space data resources according to the space data resources, namely, PC clusters and memory storage resources are adopted for data production, PC clusters and NAS storage are adopted for a service system, SAN arrays and solid state disks are adopted for cluster storage, and SAN arrays and disks are adopted for archive management.
In this embodiment, the intelligent space model provides a standardized data access interface to the outside, that is, provides a unified data storage access interface, provides data model management, data catalog management, data resource management, and data query and browsing functions, and implements access and management of data in multiple storage modes.
Specifically, data model management provides modeling capacity for a data organization structure and a data model management function, a logical structure of a database is expanded by using a data model, and access and management of wider data resources are realized by matching with a warehousing plug-in tool;
the data catalog management realizes the configuration of a unified data catalog facing the data management, and supports the management, identification and positioning of data resources from a plurality of angles such as classification, application and the like by hierarchically organizing the data resources;
the data resource management realizes the comprehensive management of various data, process data, result data and map and data services, and organizes data tables or data sets in various data sources (a relational database, a spatial database and a file database) and various map services and data services accessed spontaneously or externally based on a tree structure;
the data query browsing realizes the query browsing of various data, process data and result data and supports the comprehensive query browsing of various types of data based on a data catalog.
The intelligent space model provides an effective solution for the storage resource management of the space data resources in the big data era, realizes the centralized management of the storage management, and timely grasps the flow direction, the application direction, the use frequency and other conditions of the data resources.
A unified storage resource pool is established by adopting a mixed storage medium and mixed storage frame mode, a hierarchical storage mode that the space data resources are matched with the storage resources is realized according to the calling heat of the space data resources, and the problem that the matching degree between the space data resources and the storage resources is poor is solved.
Example two
In one or more embodiments, a method for constructing an intelligent space model under big data is disclosed, which comprises the following specific steps:
firstly, a storage resource pool composed of a mixed storage medium and a mixed storage frame is constructed, wherein the mixed storage medium mainly comprises a disk, a solid state disk and a memory, the mixed storage frame mainly comprises an SAN array, an NAS storage and a PC cluster, and the existing storage resources are planned to the same storage resource pool for uniform distribution and utilization.
Secondly, according to the calling heat of the spatial data resources, a hierarchical storage mode that the spatial data resources are matched with the storage resources is realized, and a storage frame of a spatial data-oriented relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster (HDFS) is formed, and the method specifically comprises the following steps:
(1) dividing space data resources to call heat, and arranging data production, a service system, cluster storage and archive management in sequence from high to low according to the heat;
(2) allocating storage resources, namely corresponding to a hierarchical storage mode according to the heat degree, namely adopting a PC cluster and memory storage resources for data production, adopting a PC cluster and NAS storage for a service system, adopting an SAN array and a solid state disk for cluster storage, and adopting an SAN array and a disk for archive management;
(3) and (3) constructing a storage management mode, namely establishing a plurality of spatial data resource storage management modes of a relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster (HDFS) which are adaptive to the spatial data resources according to the storage management mode adopted by the spatial data resources.
And finally, constructing a standardized data access interface, providing a unified data storage access interface based on a storage mode, namely the unified data storage access interface, providing data model management, data catalog management, data resource management and data inquiry and browsing functions, and realizing the access and management of data in various storage modes.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts by those skilled in the art based on the technical solution of the present invention.
Claims (10)
1. An intelligent spatial model under big data, comprising: a resource storage pool consisting of a hybrid storage media and a hybrid storage framework; the resource storage pool is respectively connected with an external relational database, a NoSQL database, a shared file system and a distributed file system through a data storage access interface; and realizing the hierarchical storage of the spatial data resources and the storage resources matched according to the calling heat of the spatial data resources, and forming a storage frame of a spatial data-oriented relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster.
2. The big data down smart space model according to claim 1, wherein the hybrid storage medium comprises a disk, a solid state drive, and a memory.
3. The big-data down smart space model as recited in claim 1, wherein the hybrid storage framework comprises: SAN array, NAS storage, and PC cluster.
4. The big-data down intelligent space model according to claim 1, wherein the space data resources specifically include: data production, a service system, cluster storage and archive management; the call heat thereof decreases in turn.
5. The big-data intelligent spatial model of claim 4, wherein the matching hierarchical storage resources are allocated according to spatial data resource calling hotness, namely: the data production adopts PC cluster and memory storage resources, the service system adopts PC cluster and NAS storage, the cluster storage adopts SAN array and solid state disk, and the archive management adopts SAN array and disk.
6. The big data down smart space model as claimed in claim 1, wherein the resource storage pool implements data model management, data catalog management, data resource management and data query browsing functions through a data storage access interface.
7. A method for constructing an intelligent space model under big data is characterized by comprising the following steps:
constructing a resource storage pool by the hybrid storage media and the hybrid storage framework; and realizing the hierarchical storage of the spatial data resources and the storage resources matched according to the calling heat of the spatial data resources, and forming a storage frame of a spatial data-oriented relational database cluster, a NoSQL database cluster, a shared file system cluster and a distributed file system cluster.
8. The method for constructing the smart space model under the big data according to claim 7, wherein the hybrid storage medium comprises a disk, a solid state disk and a memory; the hybrid storage frame includes: SAN array, NAS storage, and PC cluster.
9. The method for constructing the intelligent spatial model under the big data as claimed in claim 7, wherein the hierarchical storage of the spatial data resources and the storage resources matching is realized according to the calling heat of the spatial data resources, specifically:
the calling heat of the space data resources is decreased in sequence according to data production, a service system, cluster storage and archive management;
the data production adopts PC cluster and memory storage resources, the service system adopts PC cluster and NAS storage, the cluster storage adopts SAN array and solid state disk, and the archive management adopts SAN array and disk.
10. The method for constructing the smart space model under big data as claimed in claim 7, wherein the resource storage pool is connected to the external relational database, the NoSQL database, the shared file system and the distributed file system through data storage access interfaces; the functions of data model management, data catalog management, data resource management and data inquiry and browsing are provided, and the access and management of data in various storage modes are realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010071345.1A CN111309252A (en) | 2020-01-21 | 2020-01-21 | Intelligent space model under big data and construction method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010071345.1A CN111309252A (en) | 2020-01-21 | 2020-01-21 | Intelligent space model under big data and construction method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111309252A true CN111309252A (en) | 2020-06-19 |
Family
ID=71144930
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010071345.1A Pending CN111309252A (en) | 2020-01-21 | 2020-01-21 | Intelligent space model under big data and construction method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111309252A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
| CN103237046A (en) * | 2013-02-25 | 2013-08-07 | 中国科学院深圳先进技术研究院 | Distributed file system supporting mixed cloud storage application and realization method thereof |
| CN106973119A (en) * | 2017-05-17 | 2017-07-21 | 国网山东省电力公司信息通信公司 | A kind of electric power enterprise storage resource management system |
| CN109302319A (en) * | 2018-10-26 | 2019-02-01 | 许继集团有限公司 | Message pool distributed type assemblies and its management method |
| US10326837B1 (en) * | 2016-09-28 | 2019-06-18 | EMC IP Holding Company LLC | Data storage system providing unified file/block cloud access |
-
2020
- 2020-01-21 CN CN202010071345.1A patent/CN111309252A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
| CN103237046A (en) * | 2013-02-25 | 2013-08-07 | 中国科学院深圳先进技术研究院 | Distributed file system supporting mixed cloud storage application and realization method thereof |
| US10326837B1 (en) * | 2016-09-28 | 2019-06-18 | EMC IP Holding Company LLC | Data storage system providing unified file/block cloud access |
| CN106973119A (en) * | 2017-05-17 | 2017-07-21 | 国网山东省电力公司信息通信公司 | A kind of electric power enterprise storage resource management system |
| CN109302319A (en) * | 2018-10-26 | 2019-02-01 | 许继集团有限公司 | Message pool distributed type assemblies and its management method |
Non-Patent Citations (1)
| Title |
|---|
| "《计算机系统应用》", pages: 35 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113468232B (en) | Scalable database system for querying time-series data | |
| CN106709067B (en) | Multisource heterogeneous space data circulation method based on Oracle database | |
| US11314708B2 (en) | Heterogeneous type database storage system based on optical disc, and method for using system | |
| CN102541990B (en) | Database redistribution method and system utilizing virtual partitions | |
| CN101566981A (en) | Method for establishing dynamic virtual data base in analyzing and processing system | |
| CN107291889A (en) | A kind of date storage method and system | |
| CN102937964B (en) | Intelligent data service method based on distributed system | |
| CN103020204A (en) | Method and system for carrying out multi-dimensional regional inquiry on distribution type sequence table | |
| CN102243660A (en) | Data access method and device | |
| CN101916290B (en) | Managing method of internal memory database and device | |
| CN109918450B (en) | Distributed parallel database based on analysis type scene and storage method | |
| CN114647716B (en) | System suitable for generalized data warehouse | |
| CN112632025A (en) | Power grid enterprise management decision support application system based on PAAS platform | |
| CN105516284A (en) | Clustered database distributed storage method and device | |
| US10095738B1 (en) | Dynamic assignment of logical partitions according to query predicate evaluations | |
| CN102779138A (en) | Hard disk access method of real time data | |
| CN105843955A (en) | Data migration system | |
| CN116089414B (en) | Time sequence database writing performance optimization method and device based on mass data scene | |
| CN107273462A (en) | One kind builds HBase cluster full-text index methods, method for reading data and method for writing data | |
| Bhagat et al. | Comparative study of row and column oriented database | |
| CN111309252A (en) | Intelligent space model under big data and construction method | |
| CN116431635A (en) | Lake and warehouse integrated-based power distribution Internet of things data real-time processing system and method | |
| Li et al. | SP-phoenix: a massive spatial point data management system based on phoenix | |
| CN113284573A (en) | Method and device for searching document database | |
| CN204102026U (en) | Large database concept all-in-one |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200619 |
|
| RJ01 | Rejection of invention patent application after publication |