CN107291390B - Data hierarchical storage method and device - Google Patents
Data hierarchical storage method and device Download PDFInfo
- Publication number
- CN107291390B CN107291390B CN201710473539.2A CN201710473539A CN107291390B CN 107291390 B CN107291390 B CN 107291390B CN 201710473539 A CN201710473539 A CN 201710473539A CN 107291390 B CN107291390 B CN 107291390B
- Authority
- CN
- China
- Prior art keywords
- storage area
- data
- level storage
- network
- lustre
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 16
- VQLYBLABXAHUDN-UHFFFAOYSA-N bis(4-fluorophenyl)-methyl-(1,2,4-triazol-1-ylmethyl)silane;methyl n-(1h-benzimidazol-2-yl)carbamate Chemical compound C1=CC=C2NC(NC(=O)OC)=NC2=C1.C=1C=C(F)C=CC=1[Si](C=1C=CC(F)=CC=1)(C)CN1C=NC=N1 VQLYBLABXAHUDN-UHFFFAOYSA-N 0.000 claims abstract description 26
- 238000007726 management method Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000013508 migration Methods 0.000 claims description 5
- 230000005012 migration Effects 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data hierarchical storage method and a data hierarchical storage device, wherein the method comprises the following steps: the first-level storage area and the second-level storage area are respectively used for storing data, so that the storage performance is improved, the storage cost is reduced, and the problems of single lustre node failure and overhigh Nvmesd cost are solved.
Description
Technical Field
The invention belongs to the field of computers, and particularly relates to a data hierarchical storage method and device.
Background
Lustre is a Linux Cluster parallel File System developed by HP, Intel, Cluster File System company in conjunction with the U.S. department of energy. Lustre adopts a distributed lock management mechanism to realize concurrency control, separately manages communication links of metadata and file data, is a brand new file system designed for solving the problem of mass storage, is a next generation cluster file system, and can support 10,000 nodes, the storage capacity of PB, the transmission speed of 100GB/S, and perfect safety and manageability.
In 2011, the Non-Volatile Memory Express (NVMe) specification was officially released, which is a specification specifically proposed for NVND flash Memory and next generation storage devices. It is developed based on the characteristics of flash memory, and aims to reduce the gap between the storage system and the memory bandwidth as much as possible.
Lustre eliminates the problems of expandability, usability and performance of the traditional network file system, NVMe and PCI-E are very suitable for the requirement of high performance of the SSD, and the NVM is used as a protocol specially designed for a flash memory and can fully exert the performance of the SSD through a high-bandwidth PCI-E channel. But the scheme has the problem of lustre single-node failure, and the cost of the Nvme SSD is too high, so that the scheme is not suitable for a large-capacity storage medium.
Therefore, it is urgently needed to provide an efficient data storage scheme to solve the problem of single node failure of lustre.
Disclosure of Invention
The invention provides a data hierarchical storage method and a data hierarchical storage device, which are used for solving the problems.
The invention provides a data hierarchical storage method which is applied to a Lustre-based file system. The method comprises the following steps:
the first-level storage area and the second-level storage area are respectively used for storing data.
The present invention also provides a data hierarchical storage apparatus, comprising: the method is applied to a Lustre-based file system and comprises the following steps: and the first-level storage area and the second-level storage area are respectively used for storing data.
By the following scheme: the first-level storage area and the second-level storage area are respectively used for storing data, so that the storage performance is improved, the storage cost is reduced, and the problems of single lustre node failure and overhigh Nvmesd cost are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flowchart of a data hierarchical storage method according to embodiment 1 of the present invention;
FIG. 2 is a block diagram of a data staging memory device according to embodiment 2 of the present invention;
fig. 3 is a structural diagram of a data hierarchical storage device according to embodiment 3 of the present invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
Fig. 1 is a processing flow chart of a data hierarchical storage method according to embodiment 1 of the present invention, which includes the following steps:
step 102: the first-level storage area and the second-level storage area are respectively used for storing data.
Further, the first-stage storage area comprises a single-node lustre file system and a hard disk based on the Nvme transmission protocol, and the second storage area comprises an HA lustre file system and a mechanical hard disk.
Furthermore, the first storage area and the second storage area are connected by two sets of networks, and the two sets of networks are respectively used for transmitting management data and calculation data.
Further, the network for transmitting management data is an ethernet network, and the network for transmitting calculation data is an IB network.
And further controlling data migration between the first-level storage area and the second-level storage area.
Therefore, the high-performance storage area adopts single-node lustre, and the hard disk uses the Nvmesd, so that the real-time storage performance is improved to the maximum extent; the secondary storage area adopts an HA lustre framework to ensure data security, and the hard disk adopts a common mechanical hard disk to meet the requirement of storage capacity; and the high-performance storage area and the secondary storage area carry out data migration backup through backup software.
Fig. 2 is a structural diagram of a data hierarchical storage device according to embodiment 2 of the present invention.
As shown in fig. 2, a hierarchical data storage device according to an embodiment of the present invention is applied to a Lustre-based file system, and includes:
a first level memory area 202 and a second level memory area 204, respectively, for storing data.
Further, the first-level storage area 202 includes a single-node lustre file system and a hard disk based on the Nvme transport protocol, and the second storage area 204 includes an HA lustre file system and a mechanical hard disk.
Further, two sets of network connections are used between the first storage area 202 and the second storage area 204, and the two sets of networks are respectively used for transmitting management data and calculation data.
Wherein the network for transmitting the management data is an ethernet network and the network for transmitting the calculation data is an IB network.
The data hierarchical storage device further comprises:
a control module 206, configured to control data migration between the first-level storage area 202 and the second-level storage area 204.
According to the embodiment, the high-speed request of IO is met by the high-performance storage area for hierarchical storage, the storage capacity and data safety guarantee are provided by the secondary storage, and the backup strategy is formulated by the backup software to meet the backup requirement.
Fig. 3 is a structural diagram of a data hierarchical storage device according to embodiment 3 of the present invention.
As shown in fig. 3, a set of lustre + Nvme storage systems is deployed in the high-performance storage area; a set of HA lustre file system is deployed in the secondary storage area, and the storage capacity and safety of the hard disk are guaranteed by using a common mechanical hard disk. In the scheme, two sets of networks are used between the high-performance storage area and the secondary storage area: the management network uses a normal ethernet network and the computing network uses a high-speed IB network. In addition, the backup software is deployed at the lustre client, real-time data backup is completed by simultaneously accessing the high-performance storage area and the secondary storage area, and full backup, incremental backup, data recovery, outdated data deletion and the like are completed by formulating a corresponding backup strategy.
The scheme improves the storage performance and reduces the storage cost; the problem that the cost of the Nvme ssd is too high and the current situation that the storage performance of the traditional hard disk is lower are solved.
By the following scheme: the first-level storage area and the second-level storage area are respectively used for storing data, so that the storage performance is improved, the storage cost is reduced, and the problems of single lustre node failure and overhigh Nvmesd cost are solved.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A data hierarchical storage method is applied to a Lustre-based file system and comprises the following steps:
the method comprises the steps that a first-level storage area and a second-level storage area are respectively used for storing data;
the first-level storage area comprises a single-node lustre file system and a hard disk based on an Nvme transmission protocol, and the second-level storage area comprises an HA lustre file system and a mechanical hard disk.
2. The method of claim 1 wherein two sets of network connections are used between the primary storage area and the secondary storage area, the two sets of networks being used to transmit management data and computing data, respectively.
3. Method according to claim 2, characterized in that the network for transmitting management data is an ethernet network and the network for transmitting calculation data is an IB network.
4. A method according to any one of claims 1 to 3, wherein data migration between the first level storage area and the second level storage area is controlled.
5. A hierarchical data storage device is applied to a Lustre-based file system and comprises the following components:
the first-level storage area and the second-level storage area are used for storing data respectively;
the first-level storage area comprises a single-node lustre file system and a hard disk based on an Nvme transmission protocol, and the second-level storage area comprises an HA lustre file system and a mechanical hard disk.
6. The apparatus of claim 5 wherein said primary storage area and said secondary storage area are connected using two sets of networks for transmitting management data and computing data, respectively.
7. The apparatus of claim 6, wherein the network for transmitting management data is an ethernet network and the network for transmitting calculation data is an IB network.
8. The apparatus of any of claims 5 to 7, further comprising: and the control module is used for controlling data migration between the first-level storage area and the second-level storage area.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710473539.2A CN107291390B (en) | 2017-06-20 | 2017-06-20 | Data hierarchical storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710473539.2A CN107291390B (en) | 2017-06-20 | 2017-06-20 | Data hierarchical storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107291390A CN107291390A (en) | 2017-10-24 |
| CN107291390B true CN107291390B (en) | 2020-05-15 |
Family
ID=60098118
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710473539.2A Active CN107291390B (en) | 2017-06-20 | 2017-06-20 | Data hierarchical storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107291390B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108306717B (en) * | 2018-01-30 | 2019-03-15 | 平安科技（深圳）有限公司 | Data transmission method, server and storage medium |
| CN113742290B (en) * | 2021-11-04 | 2022-03-15 | 上海闪马智能科技有限公司 | Data storage method and device, storage medium and electronic device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103095837A (en) * | 2013-01-18 | 2013-05-08 | 浪潮电子信息产业股份有限公司 | Method achieving lustre metadata server redundancy |
| CN103605728A (en) * | 2013-11-18 | 2014-02-26 | 浪潮(北京)电子信息产业有限公司 | Method and system for hierarchically storing data |
| CN104023061A (en) * | 2014-06-10 | 2014-09-03 | 浪潮电子信息产业股份有限公司 | High availability cluster scheme of OSS (Open Storage service) for LUSTRE |
| CN104598568A (en) * | 2015-01-12 | 2015-05-06 | 浪潮电子信息产业股份有限公司 | High-efficiency and low-power-consumption offline storage system and method |
| CN105573673A (en) * | 2015-12-11 | 2016-05-11 | 芜湖乐锐思信息咨询有限公司 | Database based data cache system |
| CN105634813A (en) * | 2016-01-04 | 2016-06-01 | 浪潮电子信息产业股份有限公司 | Method for automatically switching nodes under dual-computer environment based on network |
| CN106502742A (en) * | 2016-10-31 | 2017-03-15 | 郑州云海信息技术有限公司 | A kind of deployment Lustre file system methods, device and client |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102291450B (en) * | 2011-08-08 | 2014-01-15 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
-
2017
- 2017-06-20 CN CN201710473539.2A patent/CN107291390B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103095837A (en) * | 2013-01-18 | 2013-05-08 | 浪潮电子信息产业股份有限公司 | Method achieving lustre metadata server redundancy |
| CN103605728A (en) * | 2013-11-18 | 2014-02-26 | 浪潮(北京)电子信息产业有限公司 | Method and system for hierarchically storing data |
| CN104023061A (en) * | 2014-06-10 | 2014-09-03 | 浪潮电子信息产业股份有限公司 | High availability cluster scheme of OSS (Open Storage service) for LUSTRE |
| CN104598568A (en) * | 2015-01-12 | 2015-05-06 | 浪潮电子信息产业股份有限公司 | High-efficiency and low-power-consumption offline storage system and method |
| CN105573673A (en) * | 2015-12-11 | 2016-05-11 | 芜湖乐锐思信息咨询有限公司 | Database based data cache system |
| CN105634813A (en) * | 2016-01-04 | 2016-06-01 | 浪潮电子信息产业股份有限公司 | Method for automatically switching nodes under dual-computer environment based on network |
| CN106502742A (en) * | 2016-10-31 | 2017-03-15 | 郑州云海信息技术有限公司 | A kind of deployment Lustre file system methods, device and client |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107291390A (en) | 2017-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108228082B (en) | Storage system and method for storage control | |
| US9678686B2 (en) | Managing sequentiality of tracks for asynchronous PPRC tracks on secondary | |
| JP6714614B2 (en) | Receive buffer credits by multiple channels of one or more host computing devices to transmit data to the control unit | |
| CN102098342B (en) | Transaction level-based data synchronizing method, device thereof and system thereof | |
| EP3206128B1 (en) | Data storage method, data storage apparatus, and storage device | |
| US20190163371A1 (en) | Next generation storage controller in hybrid environments | |
| CN108062285A (en) | A kind of access method of NVMe storage devices and NVMe storage devices | |
| CN106708653B (en) | Mixed tax big data security protection method based on erasure code and multiple copies | |
| CN105740248B (en) | A kind of method of data synchronization, apparatus and system | |
| KR101983208B1 (en) | Data management method, node and system for database cluster | |
| CN110119329B (en) | Data replication disaster recovery method and disaster recovery system | |
| US11093141B2 (en) | Method and apparatus for caching data | |
| CN105760519A (en) | Cluster file system and file lock allocation method thereof | |
| CN103209210A (en) | Method for improving erasure code based storage cluster recovery performance | |
| CN102014153A (en) | Data storage system and method thereof | |
| CN107291390B (en) | Data hierarchical storage method and device | |
| CN111045602A (en) | Cluster system control method and cluster system | |
| Herold et al. | An introduction to BeeGFS | |
| CN102780769A (en) | Cloud computing platform-based disaster recovery storage method | |
| CN103428288B (en) | Based on the copies synchronized method of subregion state table and coordinator node | |
| CN102404201A (en) | Method of realizing maximum bandwidth of Lustre concurrent file system | |
| CN104915153A (en) | Method for double control cache synchronous design based on SCST | |
| CN102681790A (en) | Data dump device and method and system for data dump by same | |
| CN105760398A (en) | Log recording system and log record operating method | |
| CN103631903A (en) | System for synchronizing data of database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200420 Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601 Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |