CN110147348A - A kind of long data block structured storage method and system - Google Patents
A kind of long data block structured storage method and system Download PDFInfo
- Publication number
- CN110147348A CN110147348A CN201910258616.1A CN201910258616A CN110147348A CN 110147348 A CN110147348 A CN 110147348A CN 201910258616 A CN201910258616 A CN 201910258616A CN 110147348 A CN110147348 A CN 110147348A
- Authority
- CN
- China
- Prior art keywords
- big data
- file system
- data
- tree file
- range
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/185—Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof
Abstract
The present invention relates to a kind of long data block structured storage method and system, method includes the following steps, S1, carries out range to big data by time to live by the way of Data Migration；S2 establishes tree file system, and the big data for carrying out range by time to live is imported into the tree file system；The big data is stored in the corresponding node of the tree file system by S3 using the principle of classification storage.A kind of long data block structured storage method of the present invention carries out range to big data by time to live by the way of Data Migration, and on this basis, classification storage is carried out to big data using tree file system, can solve the huge problem of storage data quantity and the effective access efficiency for improving big data.
Description
Technical field
The present invention relates to big data fields, and in particular to a kind of long data block structured storage method and system.
Background technique
In recent years, big data field develops a large amount of technologies and product, become big data obtain, storage, processing analysis or
Visual effective means.But the storage method of big data is to collect big data running log number by deployment O&M monitoring software
Plain text format storage is all difficult to solve big data storing data to local file system or system R accordingly
Huge problem is measured, traditional scheme can only periodically delete journal file, and the access efficiency of mass data is extremely slow.
Summary of the invention
Technical problem to be solved by the invention is to provide a kind of long data block structured storage method and system, can solve
The certainly huge problem of storage data quantity and raising data access efficiency.
The technical scheme to solve the above technical problems is that a kind of long data block structured storage method, including
Following steps,
S1 carries out range to big data by time to live by the way of Data Migration；
S2 establishes tree file system, and the big data for carrying out range by time to live is imported into described tree-shaped
File system；
The big data is stored in the corresponding node of the tree file system by S3 using the principle of classification storage.
The beneficial effects of the present invention are: a kind of long data block structured storage method of the present invention is by the way of Data Migration
Range is carried out to big data by time to live, and on this basis, big data is classified using tree file system
Storage can solve the huge problem of storage data quantity and the effective access efficiency for improving big data.
Based on the above technical solution, the present invention can also be improved as follows.
It further, further include being filtered to the big data, and the filtered big data is adopted in the S1
Range is carried out by time to live with the mode of Data Migration.
Further, in the S2, the tree file system is established with specific reference to the range series of the big data
System.
Further, the number of nodes of the tree file system and the range series of the big data match.
Further, the tree file system is specially the database of HBase type.
Based on a kind of above-mentioned long data block structured storage method, the present invention also provides a kind of long data block structured storages
System.
A kind of long data block structured storage system, comprises the following modules,
Range module is used to by the way of Data Migration carry out range to big data by time to live；
Tree file system establishes module, is used to establish tree file system, and will carry out classification by time to live and return
The big data of class imported into the tree file system；
It is classified memory module, is used to that the big data to be stored in the tree file system using the principle of classification storage
In the corresponding node of system.
The beneficial effects of the present invention are: a kind of long data block structured storage system of the present invention is by the way of Data Migration
Range is carried out to big data by time to live, and on this basis, big data is classified using tree file system
Storage can solve the huge problem of storage data quantity and the effective access efficiency for improving big data.
Based on the above technical solution, the present invention can also be improved as follows.
Further, the range module is also used to be filtered the big data, and will be filtered described big
Data carry out range by time to live by the way of Data Migration.
Further, the tree file system establishes module and is specifically used for being built according to the range series of the big data
Found the tree file system.
Further, the number of nodes of the tree file system and the range series of the big data match.
Further, the tree file system is specially the database of HBase type.
Detailed description of the invention
Fig. 1 is a kind of flow chart of long data block structured storage method of the present invention；
Fig. 2 is a kind of structural block diagram of long data block structured storage system of the present invention.
Specific embodiment
The principle and features of the present invention will be described below with reference to the accompanying drawings, and the given examples are served only to explain the present invention, and
It is non-to be used to limit the scope of the invention.
As shown in Figure 1, a kind of big data quick development method, includes the following steps,
A kind of long data block structured storage method, includes the following steps,
S1 carries out range to big data by time to live by the way of Data Migration；
S2 establishes tree file system, and the big data for carrying out range by time to live is imported into described tree-shaped
File system；
The big data is stored in the corresponding node of the tree file system by S3 using the principle of classification storage.
A kind of long data block structured storage method of the present invention is by the way of Data Migration by time to live to big data
Range is carried out, and on this basis, classification storage is carried out to big data using tree file system, can solve storage number
According to the access efficiency for measuring huge problem and effective raising big data.
In this embodiment:
It further include being filtered to the big data, and the filtered big data is used into data in the S1
The mode of migration carries out range by time to live.
In the S2, the tree file system is established with specific reference to the range series of the big data.
The number of nodes of the tree file system and the range series of the big data match.
The tree file system is specially the database of HBase type.
Based on a kind of above-mentioned long data block structured storage method, the present invention also provides a kind of long data block structured storages
System.
A kind of long data block structured storage system, comprises the following modules,
Range module is used to by the way of Data Migration carry out range to big data by time to live；
Tree file system establishes module, is used to establish tree file system, and will carry out classification by time to live and return
The big data of class imported into the tree file system；
It is classified memory module, is used to that the big data to be stored in the tree file system using the principle of classification storage
In the corresponding node of system.
A kind of long data block structured storage system of the present invention is by the way of Data Migration by time to live to big data
Range is carried out, and on this basis, classification storage is carried out to big data using tree file system, can solve storage number
According to the access efficiency for measuring huge problem and effective raising big data.
In this specific embodiment of circle:
The range module is also used to be filtered the big data, and the filtered big data is used
The mode of Data Migration carries out range by time to live.
The tree file system is established module and is specifically used for according to the foundation of the range series of the big data
Tree file system.
The number of nodes of the tree file system and the range series of the big data match.
The tree file system is specially the database of HBase type.
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of long data block structured storage method, it is characterised in that: include the following steps,
S1 carries out range to big data by time to live by the way of Data Migration；
S2 establishes tree file system, and the big data for carrying out range by time to live is imported into the tree file
System；
The big data is stored in the corresponding node of the tree file system by S3 using the principle of classification storage.
2. a kind of long data block structured storage method according to claim 1, it is characterised in that: in the S1, also
Including being filtered to the big data, and by the filtered big data by the way of Data Migration by time to live into
Row range.
3. a kind of long data block structured storage method according to claim 1 or 2, it is characterised in that: in the S2,
The tree file system is established with specific reference to the range series of the big data.
4. a kind of long data block structured storage method according to claim 3, it is characterised in that: the tree file system
The number of nodes of system and the range series of the big data match.
5. a kind of long data block structured storage method according to claim 1 or 2, it is characterised in that: the tree-shaped text
Part system is specially the database of HBase type.
6. a kind of long data block structured storage system, it is characterised in that: it comprises the following modules,
Range module is used to by the way of Data Migration carry out range to big data by time to live；
Tree file system establishes module, is used to establish tree file system, and will carry out range by time to live
Big data imported into the tree file system；
It is classified memory module, is used to that the big data to be stored in the tree file system using the principle of classification storage
In corresponding node.
7. a kind of long data block structured storage system according to claim 6, it is characterised in that: the range mould
Block is also used to be filtered the big data, and the filtered big data is all by survival by the way of Data Migration
Phase carries out range.
8. a kind of long data block structured storage system according to claim 6 or 7, it is characterised in that: the tree-shaped text
Part system establishes module and is specifically used for establishing the tree file system according to the range series of the big data.
9. a kind of long data block structured storage system according to claim 8, it is characterised in that: the tree file system
The number of nodes of system and the range series of the big data match.
10. a kind of long data block structured storage system according to claim 6 or 7, it is characterised in that: the tree-shaped text
Part system is specially the database of HBase type.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910258616.1A CN110147348A (en) | 2019-04-01 | 2019-04-01 | A kind of long data block structured storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910258616.1A CN110147348A (en) | 2019-04-01 | 2019-04-01 | A kind of long data block structured storage method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110147348A true CN110147348A (en) | 2019-08-20 |
Family
ID=67589616
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910258616.1A Pending CN110147348A (en) | 2019-04-01 | 2019-04-01 | A kind of long data block structured storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110147348A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114995772A (en) * | 2022-08-08 | 2022-09-02 | 南京三百云信息科技有限公司 | Customer data migration and storage method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030145041A1 (en) * | 2001-10-05 | 2003-07-31 | Dunham Douglas Paul | Storage area network methods and apparatus for display and management of a hierarchical file system extension policy |
| CN101419575A (en) * | 2007-10-23 | 2009-04-29 | 华硕电脑股份有限公司 | Data protection method |
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
| CN104794190A (en) * | 2015-04-16 | 2015-07-22 | 成都睿峰科技有限公司 | Method and device for effectively storing big data |
| CN105808770A (en) * | 2016-03-22 | 2016-07-27 | 北京北方微电子基地设备工艺研究中心有限责任公司 | File management method and device |
| CN107453950A (en) * | 2017-08-09 | 2017-12-08 | 郑州云海信息技术有限公司 | A kind of information processing method and monitoring system |
-
2019
- 2019-04-01 CN CN201910258616.1A patent/CN110147348A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030145041A1 (en) * | 2001-10-05 | 2003-07-31 | Dunham Douglas Paul | Storage area network methods and apparatus for display and management of a hierarchical file system extension policy |
| CN101419575A (en) * | 2007-10-23 | 2009-04-29 | 华硕电脑股份有限公司 | Data protection method |
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
| CN104794190A (en) * | 2015-04-16 | 2015-07-22 | 成都睿峰科技有限公司 | Method and device for effectively storing big data |
| CN105808770A (en) * | 2016-03-22 | 2016-07-27 | 北京北方微电子基地设备工艺研究中心有限责任公司 | File management method and device |
| CN107453950A (en) * | 2017-08-09 | 2017-12-08 | 郑州云海信息技术有限公司 | A kind of information processing method and monitoring system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114995772A (en) * | 2022-08-08 | 2022-09-02 | 南京三百云信息科技有限公司 | Customer data migration and storage method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106777351A (en) | Computing system and its method are stored based on ART tree distributed systems figure | |
| Wang et al. | Review on community detection algorithms in social networks | |
| CN103412889B (en) | The data storage and query method and its system of intelligent electric meter | |
| CN111178587B (en) | Spark framework-based short-term power load rapid prediction method | |
| CN109190672A (en) | Operation of Electric Systems operating condition unsupervised clustering and device | |
| CN105071994B (en) | A kind of mass data monitoring system | |
| CN106708989A (en) | Spatial time sequence data stream application-based Skyline query method | |
| CN106055697A (en) | Unstructured event log data classification and storage method and device | |
| CN103279505B (en) | A kind of based on semantic mass data processing method | |
| CN105912601A (en) | Partition storage method for distributed real-time memory database of energy management system | |
| CN105471670A (en) | Flow data classification method and device | |
| CN105373620A (en) | Mass battery data exception detection method and system for large-scale battery energy storage power stations | |
| CN110147348A (en) | A kind of long data block structured storage method and system | |
| CN109446243A (en) | A method of it is abnormal based on big data analysis detection photovoltaic power station power generation | |
| CN102801548A (en) | Intelligent early warning method, device and information system | |
| CN109711658A (en) | A kind of industrial production optimizing detection system and method | |
| CN105787090A (en) | Index building method and system of OLAP system of electric data | |
| CN103905276A (en) | Fast network topology detecting method | |
| CN115934856A (en) | Method and system for constructing comprehensive energy data assets | |
| CN103530328A (en) | Data statistic analysis system and method | |
| CN105930380A (en) | Chart monitor method and device based on HADOOP | |
| CN111639060A (en) | Thermal power plant time sequence data processing method, device, equipment and medium | |
| CN110865329A (en) | Electric energy metering method and system based on big data self-diagnosis | |
| CN112052372B (en) | Automatic crawler algorithm for communication network gridding splitting and recombination | |
| CN114662939A (en) | Social information gridding management system and management method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190820 |