CN107291390A - A kind of data classification storage and device - Google Patents
A kind of data classification storage and device Download PDFInfo
- Publication number
- CN107291390A CN107291390A CN201710473539.2A CN201710473539A CN107291390A CN 107291390 A CN107291390 A CN 107291390A CN 201710473539 A CN201710473539 A CN 201710473539A CN 107291390 A CN107291390 A CN 107291390A
- Authority
- CN
- China
- Prior art keywords
- memory block
- data
- storage
- lustre
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000015654 memory Effects 0.000 claims abstract description 52
- VQLYBLABXAHUDN-UHFFFAOYSA-N bis(4-fluorophenyl)-methyl-(1,2,4-triazol-1-ylmethyl)silane;methyl n-(1h-benzimidazol-2-yl)carbamate Chemical compound C1=CC=C2NC(NC(=O)OC)=NC2=C1.C=1C=C(F)C=CC=1[Si](C=1C=CC(F)=CC=1)(C)CN1C=NC=N1 VQLYBLABXAHUDN-UHFFFAOYSA-N 0.000 claims abstract description 26
- 238000013500 data storage Methods 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 8
- 238000007726 management method Methods 0.000 claims description 10
- 238000012546 transfer Methods 0.000 claims description 8
- 238000013508 migration Methods 0.000 claims description 5
- 230000005012 migration Effects 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of data classification storage and device, and the above method comprises the following steps：Data storage is respectively used to using first order memory block and second level memory block, storage performance is improved and reduces carrying cost, the problem of solving the problems, such as lustre single node failures and too high Nvme ssd costs.
Description
Technical field
The invention belongs to computer realm, more particularly to a kind of data classification storage and device.
Background technology
Lustre is HP, Intel, the Linux clusters of joint USDOE of Cluster File System companies exploitation
Parallel file system.Lustre realizes con current control using distributed lock management mechanism, metadata and file data it is logical
Link separate management is interrogated, it, to solve the problems, such as brand-new file system that mass memory is designed, is follow-on group document system to be
System, can support 10,000 node, PB amount of storage, 100GB/S transmission speed, perfect security and manageability.
At 2011, Non-Volatile Memory Express nonvolatile memory standards are that NVMe specifications are formal
Come out of the stove, this is specific to the specification that NVND flash memories and Next generation storage devices are proposed.It is based on flash memory the characteristics of and research and develop,
The purpose is to reduce the gap between storage system and memory bandwidth as far as possible.
Lustre eliminates the problem of legacy network file system is in scalability, availability and performance, NVMe and
PCI-E is especially suitable for the high performance demands of SSD, and NVM is led to as the agreement exclusively for flash memory design by the PCI-E of high bandwidth
Road, can give full play of SSD performance.But there is lustre single node failures, and Nvme SSD costs in the program
It is too high, be not suitable for large-capacity storage media.
Therefore, in the urgent need to providing a kind of efficient data storage scheme, to solve the problems, such as lustre single node failures.
The content of the invention
The present invention provides a kind of data classification storage and device, to solve the above problems.
The present invention provides a kind of data classification storage, applied to based on Lustre file system.The above method includes
Following steps：
Data storage is respectively used to using first order memory block and second level memory block.
The present invention also provides a kind of data staging storage device, including：Applied to based on Lustre file system, including：
First order memory block and second level memory block, are respectively used to data storage.
Pass through following scheme：Data storage is respectively used to using first order memory block and second level memory block, improves and deposits
Storage performance simultaneously reduces carrying cost, the problem of solving the problems, such as lustre single node failures and too high Nvme ssd costs.
Brief description of the drawings
Accompanying drawing described herein is used for providing a further understanding of the present invention, constitutes the part of the application, this hair
Bright schematic description and description is used to explain the present invention, does not constitute inappropriate limitation of the present invention.In the accompanying drawings：
Fig. 1 show the data classification storage process chart of the embodiment of the present invention 1；
Fig. 2 show the data staging memory device structure figure of the embodiment of the present invention 2；
Fig. 3 show the data staging memory device structure figure of the embodiment of the present invention 3.
Embodiment
Describe the present invention in detail below with reference to accompanying drawing and in conjunction with the embodiments.It should be noted that not conflicting
In the case of, the feature in embodiment and embodiment in the application can be mutually combined.
Fig. 1 show the data classification storage process chart of the embodiment of the present invention 1, comprises the following steps：
Step 102：Data storage is respectively used to using first order memory block and second level memory block.
Further, the first order memory block includes single node lustre file system and based on Nvme host-host protocols
Hard disk, second memory block includes HA lustre file system and mechanical hard disk.
Further, two sets of network connections, described two sets are used between first memory block and second memory block
Network is respectively used to transfer management data and calculates data.
Further, the network for transfer management data is Ethernet, and the network that data are calculated for transmitting is IB nets
Network.
Further, Data Migration between second level memory block described in the first order storage region is controlled.
Therefore, high-performance memory block uses single node lustre, and hard disk uses Nvme ssd, improves real-time to greatest extent
Storage performance；Secondary storage area is guaranteed data security using HA lustre frameworks, and hard disk uses standard machinery hard disk, and satisfaction is deposited
Store up the demand of capacity；High-performance memory block carries out Data Migration backup with secondary storage by backup software.
Fig. 2 show the data staging memory device structure figure of the embodiment of the present invention 2.
As shown in Fig. 2 a kind of data staging storage device according to embodiments of the present invention, applied to based on Lustre files
System, including：
First order memory block 202 and second level memory block 204, are respectively used to data storage.
Further, the first order memory block 202 includes single node lustre file system and transmitted based on Nvme to assist
The hard disk of view, second memory block 204 includes HA lustre file system and mechanical hard disk.
Further, two sets of network connections, institute are used between first memory block 202 and second memory block 204
Two sets of networks are stated to be respectively used to transfer management data and calculate data.
Wherein, the network for transfer management data is Ethernet, and the network that data are calculated for transmitting is IB networks.
Data staging storage device also includes：
Control module 206, for controlling data between second level memory block 204 described in the first order storage region 202
Migration.
The present embodiment meets IO high-speed requests by being classified storage with high-performance memory block, is provided and deposited with secondary storage
Capacity and data safety guarantee are stored up, formulating backup policy by backup software meets backup requirements.
Fig. 3 show the data staging memory device structure figure of the embodiment of the present invention 3.
As shown in figure 3, a set of lustre+Nvme storage systems are disposed in high-performance memory block；Secondary storage area deployment is a set of
HA lustre file system, hard disk ensures memory capacity and safety using standard machinery hard disk.High-performance memory block in this programme
Two sets of networks are used between secondary storage area：The common Ethernet of Web vector graphic is managed, calculating network uses High Speed I B networks.This
Outside, backup software is deployed in lustre clients, and real-time number is completed by accessing high-performance memory block and secondary storage area simultaneously
According to backup, backup in full and incremental backup, data recovery, stale data deletion etc. are completed by formulating respective backup strategy.
The program improves storage performance and reduces carrying cost；Solve the problem of Nvme ssd costs are too high and biography
The relatively low present situation of hard-disc storage performance of uniting.
Pass through following scheme：Data storage is respectively used to using first order memory block and second level memory block, improves and deposits
Storage performance simultaneously reduces carrying cost, the problem of solving the problems, such as lustre single node failures and too high Nvme ssd costs.
The preferred embodiments of the present invention are the foregoing is only, are not intended to limit the invention, for the skill of this area
For art personnel, the present invention can have various modifications and variations.Within the spirit and principles of the invention, that is made any repaiies
Change, equivalent substitution, improvement etc., should be included in the scope of the protection.
Claims (10)
1. a kind of data classification storage, it is characterised in that applied to based on Lustre file system, comprise the following steps：
Data storage is respectively used to using first order memory block and second level memory block.
2. according to the method described in claim 1, it is characterised in that the first order memory block includes single node lustre files
System and the hard disk based on Nvme host-host protocols, second memory block include HA lustre file system and mechanical hard disk.
3. according to the method described in claim 1, it is characterised in that make between first memory block and second memory block
With two sets of network connections, two sets of networks are respectively used to transfer management data and calculate data.
4. method according to claim 3, it is characterised in that the network for transfer management data is Ethernet, is used for
The network that transmission calculates data is IB networks.
5. method according to any one of claim 1 to 4, it is characterised in that the control first order storage region institute
State Data Migration between the memory block of the second level.
6. a kind of data staging storage device, it is characterised in that applied to based on Lustre file system, including：
First order memory block and second level memory block, are respectively used to data storage.
7. device according to claim 6, it is characterised in that the first order memory block includes single node lustre files
System and the hard disk based on Nvme host-host protocols, second memory block include HA lustre file system and mechanical hard disk.
8. device according to claim 6, it is characterised in that make between first memory block and second memory block
With two sets of network connections, two sets of networks are respectively used to transfer management data and calculate data.
9. device according to claim 8, it is characterised in that the network for transfer management data is Ethernet, is used for
The network that transmission calculates data is IB networks.
10. the device according to any one of claim 6 to 9, it is characterised in that also include：Control module, for controlling
Data Migration between second level memory block described in the first order storage region.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710473539.2A CN107291390B (en) | 2017-06-20 | 2017-06-20 | Data hierarchical storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710473539.2A CN107291390B (en) | 2017-06-20 | 2017-06-20 | Data hierarchical storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107291390A true CN107291390A (en) | 2017-10-24 |
| CN107291390B CN107291390B (en) | 2020-05-15 |
Family
ID=60098118
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710473539.2A Active CN107291390B (en) | 2017-06-20 | 2017-06-20 | Data hierarchical storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107291390B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108306717A (en) * | 2018-01-30 | 2018-07-20 | 平安科技（深圳）有限公司 | Data transmission method, server and storage medium |
| CN113742290A (en) * | 2021-11-04 | 2021-12-03 | 上海闪马智能科技有限公司 | Data storage method and device, storage medium and electronic device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
| CN103095837A (en) * | 2013-01-18 | 2013-05-08 | 浪潮电子信息产业股份有限公司 | Method achieving lustre metadata server redundancy |
| CN103605728A (en) * | 2013-11-18 | 2014-02-26 | 浪潮(北京)电子信息产业有限公司 | Method and system for hierarchically storing data |
| CN104023061A (en) * | 2014-06-10 | 2014-09-03 | 浪潮电子信息产业股份有限公司 | High availability cluster scheme of OSS (Open Storage service) for LUSTRE |
| CN104598568A (en) * | 2015-01-12 | 2015-05-06 | 浪潮电子信息产业股份有限公司 | Efficient and low-power-consumption offline storage system and method |
| CN105573673A (en) * | 2015-12-11 | 2016-05-11 | 芜湖乐锐思信息咨询有限公司 | Database based data cache system |
| CN105634813A (en) * | 2016-01-04 | 2016-06-01 | 浪潮电子信息产业股份有限公司 | Method for automatically switching nodes under double-computer environment based on network |
| CN106502742A (en) * | 2016-10-31 | 2017-03-15 | 郑州云海信息技术有限公司 | A kind of deployment Lustre file system methods, device and client |
-
2017
- 2017-06-20 CN CN201710473539.2A patent/CN107291390B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
| CN103095837A (en) * | 2013-01-18 | 2013-05-08 | 浪潮电子信息产业股份有限公司 | Method achieving lustre metadata server redundancy |
| CN103605728A (en) * | 2013-11-18 | 2014-02-26 | 浪潮(北京)电子信息产业有限公司 | Method and system for hierarchically storing data |
| CN104023061A (en) * | 2014-06-10 | 2014-09-03 | 浪潮电子信息产业股份有限公司 | High availability cluster scheme of OSS (Open Storage service) for LUSTRE |
| CN104598568A (en) * | 2015-01-12 | 2015-05-06 | 浪潮电子信息产业股份有限公司 | Efficient and low-power-consumption offline storage system and method |
| CN105573673A (en) * | 2015-12-11 | 2016-05-11 | 芜湖乐锐思信息咨询有限公司 | Database based data cache system |
| CN105634813A (en) * | 2016-01-04 | 2016-06-01 | 浪潮电子信息产业股份有限公司 | Method for automatically switching nodes under double-computer environment based on network |
| CN106502742A (en) * | 2016-10-31 | 2017-03-15 | 郑州云海信息技术有限公司 | A kind of deployment Lustre file system methods, device and client |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108306717A (en) * | 2018-01-30 | 2018-07-20 | 平安科技（深圳）有限公司 | Data transmission method, server and storage medium |
| CN113742290A (en) * | 2021-11-04 | 2021-12-03 | 上海闪马智能科技有限公司 | Data storage method and device, storage medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107291390B (en) | 2020-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6661696B2 (en) | Database system with database engine and separate distributed storage service | |
| CN111183420B (en) | Log structured storage system | |
| TWI737395B (en) | Log-structured storage systems and method | |
| CN106688217B (en) | Method and system for converged networking and storage | |
| CN103150347B (en) | Based on the dynamic replication management method of file temperature | |
| US9628299B2 (en) | Method and system for data transfer between compute clusters and file system | |
| CN102307221A (en) | Cloud storage system and implementation method thereof | |
| JP6264666B2 (en) | Data storage method, data storage device, and storage device | |
| TW202111585A (en) | Log-structured storage systems | |
| CN103366016A (en) | Electronic file concentrated storing and optimizing method based on HDFS | |
| US9860317B1 (en) | Throughput throttling for distributed file storage services with varying connection characteristics | |
| CN103929454A (en) | Load balancing storage method and system in cloud computing platform | |
| TW202113580A (en) | Log-structured storage systems | |
| CN105025053A (en) | Distributed file upload method based on cloud storage technology and system | |
| CN105593828A (en) | Method for managing files, distributed storage system and management node | |
| TW202111564A (en) | Log-structured storage systems | |
| US10230544B1 (en) | Efficient data forwarding in a networked device | |
| CN106484322A (en) | A kind of method of carry file system, device and equipment | |
| CN102880658A (en) | Distributed file management system based on seismic data processing | |
| CN104184812A (en) | Multi-point data transmission method based on private cloud | |
| CN104145468A (en) | File access authority control method and device thereof | |
| CN103067488A (en) | Implement method of unified storage | |
| Xu et al. | Rethink the storage of virtual machine images in clouds | |
| CN106066890A (en) | A kind of distributed high-performance data storehouse integrated machine system | |
| CN104536702A (en) | Storage array system and data writing request processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200420 Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601 Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |