CN107291390B - Data hierarchical storage method and device - Google Patents

Data hierarchical storage method and device Download PDF

Info

Publication number
CN107291390B
CN107291390B CN201710473539.2A CN201710473539A CN107291390B CN 107291390 B CN107291390 B CN 107291390B CN 201710473539 A CN201710473539 A CN 201710473539A CN 107291390 B CN107291390 B CN 107291390B
Authority
CN
China
Prior art keywords
storage area
data
level storage
network
lustre
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710473539.2A
Other languages
Chinese (zh)
Other versions
CN107291390A (en
Inventor
魏桂宝
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN201710473539.2A priority Critical patent/CN107291390B/en
Publication of CN107291390A publication Critical patent/CN107291390A/en
Application granted granted Critical
Publication of CN107291390B publication Critical patent/CN107291390B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/062Securing storage systems
    • G06F3/0623Securing storage systems in relation to content
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1458Management of the backup or restore process
    • G06F11/1464Management of the backup or restore process for networked environments
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1458Management of the backup or restore process
    • G06F11/1469Backup restoration techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • G06F3/0643Management of files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647Migration mechanisms
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a data hierarchical storage method and a data hierarchical storage device, wherein the method comprises the following steps: the first-level storage area and the second-level storage area are respectively used for storing data, so that the storage performance is improved, the storage cost is reduced, and the problems of single lustre node failure and overhigh Nvmesd cost are solved.

Description

Data hierarchical storage method and device
Technical Field
The invention belongs to the field of computers, and particularly relates to a data hierarchical storage method and device.
Background
Lustre is a Linux Cluster parallel File System developed by HP, Intel, Cluster File System company in conjunction with the U.S. department of energy. Lustre adopts a distributed lock management mechanism to realize concurrency control, separately manages communication links of metadata and file data, is a brand new file system designed for solving the problem of mass storage, is a next generation cluster file system, and can support 10,000 nodes, the storage capacity of PB, the transmission speed of 100GB/S, and perfect safety and manageability.
In 2011, the Non-Volatile Memory Express (NVMe) specification was officially released, which is a specification specifically proposed for NVND flash Memory and next generation storage devices. It is developed based on the characteristics of flash memory, and aims to reduce the gap between the storage system and the memory bandwidth as much as possible.
Lustre eliminates the problems of expandability, usability and performance of the traditional network file system, NVMe and PCI-E are very suitable for the requirement of high performance of the SSD, and the NVM is used as a protocol specially designed for a flash memory and can fully exert the performance of the SSD through a high-bandwidth PCI-E channel. But the scheme has the problem of lustre single-node failure, and the cost of the Nvme SSD is too high, so that the scheme is not suitable for a large-capacity storage medium.
Therefore, it is urgently needed to provide an efficient data storage scheme to solve the problem of single node failure of lustre.
Disclosure of Invention
The invention provides a data hierarchical storage method and a data hierarchical storage device, which are used for solving the problems.
The invention provides a data hierarchical storage method which is applied to a Lustre-based file system. The method comprises the following steps:
the first-level storage area and the second-level storage area are respectively used for storing data.
The present invention also provides a data hierarchical storage apparatus, comprising: the method is applied to a Lustre-based file system and comprises the following steps: and the first-level storage area and the second-level storage area are respectively used for storing data.
By the following scheme: the first-level storage area and the second-level storage area are respectively used for storing data, so that the storage performance is improved, the storage cost is reduced, and the problems of single lustre node failure and overhigh Nvmesd cost are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flowchart of a data hierarchical storage method according to embodiment 1 of the present invention;
FIG. 2 is a block diagram of a data staging memory device according to embodiment 2 of the present invention;
fig. 3 is a structural diagram of a data hierarchical storage device according to embodiment 3 of the present invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
Fig. 1 is a processing flow chart of a data hierarchical storage method according to embodiment 1 of the present invention, which includes the following steps:
step 102: the first-level storage area and the second-level storage area are respectively used for storing data.
Further, the first-stage storage area comprises a single-node lustre file system and a hard disk based on the Nvme transmission protocol, and the second storage area comprises an HA lustre file system and a mechanical hard disk.
Furthermore, the first storage area and the second storage area are connected by two sets of networks, and the two sets of networks are respectively used for transmitting management data and calculation data.
Further, the network for transmitting management data is an ethernet network, and the network for transmitting calculation data is an IB network.
And further controlling data migration between the first-level storage area and the second-level storage area.
Therefore, the high-performance storage area adopts single-node lustre, and the hard disk uses the Nvmesd, so that the real-time storage performance is improved to the maximum extent; the secondary storage area adopts an HA lustre framework to ensure data security, and the hard disk adopts a common mechanical hard disk to meet the requirement of storage capacity; and the high-performance storage area and the secondary storage area carry out data migration backup through backup software.
Fig. 2 is a structural diagram of a data hierarchical storage device according to embodiment 2 of the present invention.
As shown in fig. 2, a hierarchical data storage device according to an embodiment of the present invention is applied to a Lustre-based file system, and includes:
a first level memory area 202 and a second level memory area 204, respectively, for storing data.
Further, the first-level storage area 202 includes a single-node lustre file system and a hard disk based on the Nvme transport protocol, and the second storage area 204 includes an HA lustre file system and a mechanical hard disk.
Further, two sets of network connections are used between the first storage area 202 and the second storage area 204, and the two sets of networks are respectively used for transmitting management data and calculation data.
Wherein the network for transmitting the management data is an ethernet network and the network for transmitting the calculation data is an IB network.
The data hierarchical storage device further comprises:
a control module 206, configured to control data migration between the first-level storage area 202 and the second-level storage area 204.
According to the embodiment, the high-speed request of IO is met by the high-performance storage area for hierarchical storage, the storage capacity and data safety guarantee are provided by the secondary storage, and the backup strategy is formulated by the backup software to meet the backup requirement.
Fig. 3 is a structural diagram of a data hierarchical storage device according to embodiment 3 of the present invention.
As shown in fig. 3, a set of lustre + Nvme storage systems is deployed in the high-performance storage area; a set of HA lustre file system is deployed in the secondary storage area, and the storage capacity and safety of the hard disk are guaranteed by using a common mechanical hard disk. In the scheme, two sets of networks are used between the high-performance storage area and the secondary storage area: the management network uses a normal ethernet network and the computing network uses a high-speed IB network. In addition, the backup software is deployed at the lustre client, real-time data backup is completed by simultaneously accessing the high-performance storage area and the secondary storage area, and full backup, incremental backup, data recovery, outdated data deletion and the like are completed by formulating a corresponding backup strategy.
The scheme improves the storage performance and reduces the storage cost; the problem that the cost of the Nvme ssd is too high and the current situation that the storage performance of the traditional hard disk is lower are solved.
By the following scheme: the first-level storage area and the second-level storage area are respectively used for storing data, so that the storage performance is improved, the storage cost is reduced, and the problems of single lustre node failure and overhigh Nvmesd cost are solved.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (8)

1. A data hierarchical storage method is applied to a Lustre-based file system and comprises the following steps:
the method comprises the steps that a first-level storage area and a second-level storage area are respectively used for storing data;
the first-level storage area comprises a single-node lustre file system and a hard disk based on an Nvme transmission protocol, and the second-level storage area comprises an HA lustre file system and a mechanical hard disk.
2. The method of claim 1 wherein two sets of network connections are used between the primary storage area and the secondary storage area, the two sets of networks being used to transmit management data and computing data, respectively.
3. Method according to claim 2, characterized in that the network for transmitting management data is an ethernet network and the network for transmitting calculation data is an IB network.
4. A method according to any one of claims 1 to 3, wherein data migration between the first level storage area and the second level storage area is controlled.
5. A hierarchical data storage device is applied to a Lustre-based file system and comprises the following components:
the first-level storage area and the second-level storage area are used for storing data respectively;
the first-level storage area comprises a single-node lustre file system and a hard disk based on an Nvme transmission protocol, and the second-level storage area comprises an HA lustre file system and a mechanical hard disk.
6. The apparatus of claim 5 wherein said primary storage area and said secondary storage area are connected using two sets of networks for transmitting management data and computing data, respectively.
7. The apparatus of claim 6, wherein the network for transmitting management data is an ethernet network and the network for transmitting calculation data is an IB network.
8. The apparatus of any of claims 5 to 7, further comprising: and the control module is used for controlling data migration between the first-level storage area and the second-level storage area.
CN201710473539.2A 2017-06-20 2017-06-20 Data hierarchical storage method and device Active CN107291390B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710473539.2A CN107291390B (en) 2017-06-20 2017-06-20 Data hierarchical storage method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710473539.2A CN107291390B (en) 2017-06-20 2017-06-20 Data hierarchical storage method and device

Publications (2)

Publication Number Publication Date
CN107291390A CN107291390A (en) 2017-10-24
CN107291390B true CN107291390B (en) 2020-05-15

Family

ID=60098118

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710473539.2A Active CN107291390B (en) 2017-06-20 2017-06-20 Data hierarchical storage method and device

Country Status (1)

Country Link
CN (1) CN107291390B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108306717B (en) * 2018-01-30 2019-03-15 平安科技(深圳)有限公司 Data transmission method, server and storage medium
CN113742290B (en) * 2021-11-04 2022-03-15 上海闪马智能科技有限公司 Data storage method and device, storage medium and electronic device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103095837A (en) * 2013-01-18 2013-05-08 浪潮电子信息产业股份有限公司 Method achieving lustre metadata server redundancy
CN103605728A (en) * 2013-11-18 2014-02-26 浪潮(北京)电子信息产业有限公司 Method and system for hierarchically storing data
CN104023061A (en) * 2014-06-10 2014-09-03 浪潮电子信息产业股份有限公司 High availability cluster scheme of OSS (Open Storage service) for LUSTRE
CN104598568A (en) * 2015-01-12 2015-05-06 浪潮电子信息产业股份有限公司 High-efficiency and low-power-consumption offline storage system and method
CN105573673A (en) * 2015-12-11 2016-05-11 芜湖乐锐思信息咨询有限公司 Database based data cache system
CN105634813A (en) * 2016-01-04 2016-06-01 浪潮电子信息产业股份有限公司 Method for automatically switching nodes under dual-computer environment based on network
CN106502742A (en) * 2016-10-31 2017-03-15 郑州云海信息技术有限公司 A kind of deployment Lustre file system methods, device and client

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102291450B (en) * 2011-08-08 2014-01-15 浪潮电子信息产业股份有限公司 Data online hierarchical storage method in cluster storage system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103095837A (en) * 2013-01-18 2013-05-08 浪潮电子信息产业股份有限公司 Method achieving lustre metadata server redundancy
CN103605728A (en) * 2013-11-18 2014-02-26 浪潮(北京)电子信息产业有限公司 Method and system for hierarchically storing data
CN104023061A (en) * 2014-06-10 2014-09-03 浪潮电子信息产业股份有限公司 High availability cluster scheme of OSS (Open Storage service) for LUSTRE
CN104598568A (en) * 2015-01-12 2015-05-06 浪潮电子信息产业股份有限公司 High-efficiency and low-power-consumption offline storage system and method
CN105573673A (en) * 2015-12-11 2016-05-11 芜湖乐锐思信息咨询有限公司 Database based data cache system
CN105634813A (en) * 2016-01-04 2016-06-01 浪潮电子信息产业股份有限公司 Method for automatically switching nodes under dual-computer environment based on network
CN106502742A (en) * 2016-10-31 2017-03-15 郑州云海信息技术有限公司 A kind of deployment Lustre file system methods, device and client

Also Published As

Publication number Publication date
CN107291390A (en) 2017-10-24

Similar Documents

Publication Publication Date Title
CN108228082B (en) Storage system and method for storage control
US9678686B2 (en) Managing sequentiality of tracks for asynchronous PPRC tracks on secondary
JP6714614B2 (en) Receive buffer credits by multiple channels of one or more host computing devices to transmit data to the control unit
CN102098342B (en) Transaction level-based data synchronizing method, device thereof and system thereof
EP3206128B1 (en) Data storage method, data storage apparatus, and storage device
US20190163371A1 (en) Next generation storage controller in hybrid environments
CN108062285A (en) A kind of access method of NVMe storage devices and NVMe storage devices
CN106708653B (en) Mixed tax big data security protection method based on erasure code and multiple copies
CN105740248B (en) A kind of method of data synchronization, apparatus and system
KR101983208B1 (en) Data management method, node and system for database cluster
CN110119329B (en) Data replication disaster recovery method and disaster recovery system
US11093141B2 (en) Method and apparatus for caching data
CN105760519A (en) Cluster file system and file lock allocation method thereof
CN103209210A (en) Method for improving erasure code based storage cluster recovery performance
CN102014153A (en) Data storage system and method thereof
CN107291390B (en) Data hierarchical storage method and device
CN111045602A (en) Cluster system control method and cluster system
Herold et al. An introduction to BeeGFS
CN102780769A (en) Cloud computing platform-based disaster recovery storage method
CN103428288B (en) Based on the copies synchronized method of subregion state table and coordinator node
CN102404201A (en) Method of realizing maximum bandwidth of Lustre concurrent file system
CN104915153A (en) Method for double control cache synchronous design based on SCST
CN102681790A (en) Data dump device and method and system for data dump by same
CN105760398A (en) Log recording system and log record operating method
CN103631903A (en) System for synchronizing data of database

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20200420

Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province

Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd.

Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601

Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd.

GR01 Patent grant
GR01 Patent grant