CN111309245A - Layered storage writing method and device, reading method and device and system - Google Patents

Layered storage writing method and device, reading method and device and system Download PDF

Info

Publication number
CN111309245A
CN111309245A CN201811521157.3A CN201811521157A CN111309245A CN 111309245 A CN111309245 A CN 111309245A CN 201811521157 A CN201811521157 A CN 201811521157A CN 111309245 A CN111309245 A CN 111309245A
Authority
CN
China
Prior art keywords
data
storage medium
metadata
storage
area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201811521157.3A
Other languages
Chinese (zh)
Other versions
CN111309245B (en
Inventor
刘攀
王正勇
汪堃
刘飞
李舒
谢可可
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201811521157.3A priority Critical patent/CN111309245B/en
Publication of CN111309245A publication Critical patent/CN111309245A/en
Application granted granted Critical
Publication of CN111309245B publication Critical patent/CN111309245B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061Improving I/O performance
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614Improving the reliability of storage systems
    • G06F3/0616Improving the reliability of storage systems in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671In-line storage system
    • G06F3/0683Plurality of storage devices
    • G06F3/0685Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the application provides a layered storage writing method and device, and a reading method, device and system. The layered storage writing method comprises the following steps: acquiring target data; generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into a data storage area; writing metadata corresponding to the target data in the data storage area into a metadata area of a second storage medium; and writing the target data in the data storage area to the second storage medium. And establishing a second storage medium metadata area in the first storage medium for storing metadata of the second storage medium. The second storage medium only stores the target data, and does not store the metadata corresponding to the target data of the second storage medium. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is solved, metadata and target data do not need to be stored in different address spaces in the second storage medium, and the overall write-in speed of layered storage is improved.

Description

Layered storage writing method and device, reading method and device and system
Technical Field
The present application relates to the field of storage technologies, and in particular, to a layered storage writing method and apparatus, and a layered storage reading method and apparatus and system.
Background
The hierarchical storage technology refers to storing data in different hierarchical media, and performing automatic or manual data migration, copying and other operations between different media. High speed storage media with fast access speeds typically have high storage unit cost and low individual media capacity. And low-speed storage media with slow access speed generally have low storage unit cost and high single-medium capacity. Therefore, a hierarchical storage technique can be adopted, and a high-speed storage medium and a low-speed storage medium are matched with each other to obtain better storage performance in lower cost.
In the prior art, a high-speed storage medium and a low-speed storage medium in a layered storage architecture respectively have a file system to manage own storage space. Generally, after data of an application is issued to a fast storage medium, a background thread is responsible for transferring the data of the application from the fast storage medium to a low-speed storage medium. When writing operation is performed in the low-speed storage medium, metadata such as bitmap and inode in a file system of the low-speed storage medium needs to be updated, and writing amplification in the low-speed storage medium is caused. The write life and performance of the low-speed storage medium are reduced rapidly. Secondly, when the metadata and the data in the low-speed storage medium are updated, the data are respectively stored in different address spaces, so that when the low-speed storage medium is updated and stored, the data are written randomly, the performance of the second storage medium of the low-speed storage medium is further influenced, and the layered storage architecture is difficult to achieve the expected storage speed. In addition, the capacity of a single low-speed storage medium is high, and once the metadata part of the low-speed storage medium is damaged, all data of the single low-speed storage medium cannot be read, so that full-disk recovery is required, and the security is low.
Disclosure of Invention
In view of the above problems, embodiments of the present application are proposed to provide a layered storage writing method and apparatus, a reading method and apparatus, and a system, which overcome the above problems or at least partially solve the above problems.
A layered storage write method is applied to a layered storage architecture object, the layered storage architecture object comprises a first storage medium and a second storage medium, the first storage medium comprises a second storage medium metadata area and a data storage area, and the method comprises the following steps:
acquiring target data;
generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into the data storage area;
writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium; and writing the target data in the data storage area into the second storage medium.
Preferably, the data storage area comprises a first storage medium metadata area and a target data cache area;
the step of writing the target data and its corresponding metadata into the data storage area comprises:
writing metadata corresponding to the target data into the first storage medium metadata area;
and writing the target data into the target data cache region.
Preferably, the step of writing the target data and the corresponding metadata into the data storage area further comprises:
determining a target data storage area in the data storage areas by adopting a preset load balancing algorithm;
storing the target data in the target data store.
Preferably, the step of writing the metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium includes:
and when a preset transfer period is reached or the preset service storage capacity of the first storage medium reaches an upper limit, writing the metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium.
Preferably, the step of writing the target data in the data storage area to the second storage medium includes:
when all metadata corresponding to the target data are written into the metadata area of the second storage medium, writing the target data in the data storage area into the second storage medium;
and deleting the target data in the data storage area.
Preferably, all the second storage medium metadata areas in the first storage medium constitute a redundant array of independent hard disks.
A layered storage reading method is applied to a layered storage architecture object, wherein the layered storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area;
the method comprises the following steps:
receiving a data calling request, wherein the data calling request comprises a to-be-called data list;
searching target metadata from the metadata area of the second storage medium aiming at the data list to be called, wherein the target metadata is metadata corresponding to the data to be called in the data list to be called;
when the target metadata exists in the metadata area of the second storage medium, acquiring the data to be called from the second storage medium by adopting the target metadata;
and returning the data to be called.
Preferably, the first storage medium further comprises a data storage area therein;
the step of searching target metadata from the second storage medium metadata area aiming at the data list to be called comprises the following steps:
searching target metadata from the data storage area aiming at the data list to be called;
when the target metadata does not exist in the data storage area, searching the target metadata from the second storage medium metadata area.
A layered storage apparatus applied in a layered storage architecture object, the layered storage architecture object comprising a first storage medium and a second storage medium, the first storage medium comprising a second storage medium metadata area and a data storage area, the apparatus comprising:
the target data acquisition module is used for acquiring target data;
the target data and metadata writing module is used for generating metadata corresponding to the target data and writing the target data and the metadata corresponding to the target data into the data storage area;
a second storage medium writing module, configured to write metadata corresponding to target data in the data storage area into a second storage medium metadata area; and writing the target data in the data storage area into the second storage medium.
Preferably, the data storage area comprises a first storage medium metadata area and a target data cache area;
the target data and metadata writing module comprises:
a first storage medium metadata writing sub-module, configured to write metadata corresponding to the target data in the first storage medium metadata area;
and the target data writing submodule is used for writing the target data into the target data cache region.
Preferably, the target data and metadata writing module further comprises:
the target data storage area determining submodule is used for determining a target data storage area in the data storage area by adopting a preset load balancing algorithm;
and the target data storage submodule is used for storing the target data in the target data storage area.
Preferably, the low-speed storage medium writing module includes:
and the second storage medium metadata writing submodule is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area when a preset transfer period is reached or the preset business storage volume of the first storage medium reaches an upper limit.
Preferably, the low-speed storage medium writing module further comprises:
the second storage medium target data writing sub-module is used for sending the target data to the second storage medium when the metadata corresponding to the target data is successfully transferred to the second storage medium metadata area;
and the target data deleting submodule is used for deleting the target data in the first storage medium.
Preferably, all the second storage medium metadata areas in the first storage medium constitute a redundant array of independent hard disks.
A hierarchical storage reading device is applied to a hierarchical storage architecture, the hierarchical storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area; the device comprises the following modules:
the data calling request receiving module is used for receiving a data calling request, and the data calling request comprises a to-be-called data list;
the second storage medium metadata searching module is used for searching target metadata from the second storage medium metadata area aiming at the data list to be called, wherein the target metadata is metadata corresponding to the data to be called in the data list to be called;
a to-be-called data acquisition module, configured to acquire, when the target metadata exists in the second storage medium metadata area, the to-be-called data from a second storage medium by using the target metadata;
and the data to be called returning module is used for returning the data to be called.
Preferably, the first storage medium further comprises a data storage area therein;
the second storage medium metadata lookup module includes:
the data to be called searching submodule of the data storage area is used for searching target metadata from the data storage area aiming at the data list to be called;
and the second storage medium metadata searching sub-module is used for searching the target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
A tiered storage system comprising:
the high-speed storage module is used for caching target data;
the low-speed storage module is used for storing the target data sent by the first storage medium;
the high-speed storage module comprises a second storage medium metadata area used for storing metadata of a second storage medium.
Preferably, the second storage medium metadata areas in all the high-speed storage modules form an independent hard disk redundancy array for data redundancy protection.
The embodiment of the application has the following advantages:
according to the embodiment of the application, a second storage medium metadata area is established in the first storage medium and is used for storing metadata of the second storage medium. The second storage medium only stores the target data, and does not store the metadata corresponding to the target data of the second storage medium. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is avoided, and meanwhile, metadata and target data do not need to be stored in different address spaces in the second storage medium at the moment, so that the problem that the write speed is reduced by random writing of the second storage medium is avoided, and the overall write speed of layered storage is effectively improved.
Drawings
FIG. 1 is a flowchart illustrating steps of an embodiment of a method for write to a hierarchical storage according to the present application;
FIG. 2 is a flowchart illustrating steps of an embodiment of a hierarchical memory read method according to the present application;
FIG. 3 is a block diagram of an embodiment of a layered storage writer of the present application;
FIG. 4 is a block diagram of a hierarchical storage read apparatus according to an embodiment of the present application;
FIG. 5 is a block diagram of a hierarchical storage system embodiment of the present application.
Detailed description of the preferred embodiments
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and embodiments.
One of the core concepts of the embodiments of the present application is to establish a second storage medium metadata area in the first storage medium for storing metadata of the second storage medium. The second storage medium only stores the target data, and does not store the metadata corresponding to the target data of the second storage medium. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is avoided, and meanwhile, metadata and target data do not need to be stored in different address spaces in the second storage medium at the moment, so that the problem that the write speed is reduced by random writing of the second storage medium is avoided, and the overall write speed of layered storage is effectively improved.
Referring to fig. 1, a flowchart illustrating steps of an embodiment of a layered storage writing method according to the present application is shown, where the method is applied to a layered storage architecture object, where the layered storage architecture object includes a first storage medium and a second storage medium, and the first storage medium includes a second storage medium metadata area and a data storage area.
In the embodiments of the present application, a storage medium refers to a carrier storing data. The first storage medium has a higher read-write speed than the second storage medium. For example, when the first storage medium is an SCM, the low-speed storage medium may be a storage medium with a lower read/write speed than the SCM, such as NVMESSD, optanesd, or a mechanical hard disk. When the first storage medium is NVMESSD or OptaneSSD, the second storage medium may be a mechanical hard disk, a magnetic tape, or the like.
In a preferred embodiment of the present application, the first storage medium may be an SCM storage medium. The SCM storage medium has the characteristics of both persistence (StorageClass) and fast direct access (Memory), the access delay of the SCM storage medium can be less than 1 mu s, and the SCM storage medium is excellent in terms of service life and data retention capability. When the method is applied to the SCM storage medium, compared with storage media such as NVMESSD and OptaneSSD, the method can obviously improve the reading and writing speed of the metadata of the second storage medium in the first storage medium, thereby more obviously improving the reading and writing speed of the object of the layered storage architecture.
In the embodiment of the present application, the second storage medium may be any storage medium having a lower read/write speed than the first storage medium. For example, when
In the embodiment of the present application, a second storage medium metadata area is established in the first storage medium and is used for storing metadata such as bitmaps and inodes in the second storage medium, and the second storage medium does not store metadata of itself any more and only stores target data.
In an embodiment of the present application, the data storage area is a data partition provided in the first storage medium for storing the target data.
The hierarchical storage writing method may include the steps of:
step 101, acquiring target data;
in this embodiment of the present application, the target data may be obtained from an interface provided by a background processing process for managing all storage media in the hierarchical storage architecture object for the application process. And the application process sends the target data to the hierarchical storage structure through an interface provided by the background processing process.
102, generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into the data storage area;
in the embodiment of the present application, the metadata may be an index of target data, and is used to describe information of a data property (property), and may implement functions of indicating a storage location, history data, resource lookup, file recording, and the like. And establishing metadata based on the target data, so that the target data can be conveniently and quickly searched based on the metadata.
In this embodiment of the application, after the background processing process receives the target data from the interface, the target data and the metadata corresponding to the target data are written into a preset data storage area in the first storage medium.
103, writing the metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium; and writing the target data in the data storage area into the second storage medium.
In this embodiment of the present application, when it is necessary to write target data in the data storage area in the first storage medium into the second storage medium, a background processing process writes metadata corresponding to the target data into a metadata area of the second storage medium in the first storage medium.
In an embodiment of the application, the background processing process transfers the target data to the second storage medium. The second storage medium only stores the target data, and does not store the metadata corresponding to the target data of the second storage medium.
According to the layered storage writing method, a second storage medium metadata area is established in the first storage medium and is used for storing metadata of the second storage medium. The second storage medium only stores the target data, and does not store the metadata corresponding to the target data of the second storage medium. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is avoided, and meanwhile, metadata and target data do not need to be stored in different address spaces in the second storage medium at the moment, so that the problem that the write speed is reduced by random writing of the second storage medium is avoided, and the overall write speed of layered storage is effectively improved.
In another embodiment of the present application, the data storage area includes a first storage medium metadata area and a target data cache area;
the step of writing the target data and its corresponding metadata into the data storage area comprises:
writing metadata corresponding to the target data into the first storage medium metadata area;
and writing the target data into the target data cache region.
In this embodiment, the data storage area is a data storage area of the first storage medium itself, and is divided into a first storage medium metadata area and a target data cache area, where the first storage medium metadata area is used to store metadata of target data stored in the first storage medium itself, and the target data cache area is used to store target data cached in the first storage medium. The first storage medium has higher read-write speed, so that the target data sent from the application process is written into the first storage medium, and the application process can complete the data writing process quickly.
By the layered storage writing method, the target data sent by the application process can be rapidly stored in the first storage medium, and the writing efficiency of the layered storage is improved.
In another embodiment of the present application, the step of writing the target data and the corresponding metadata thereof into the data storage area further comprises:
determining a target data storage area in the data storage areas by adopting a preset load balancing algorithm;
storing the target data in the target data store.
In an embodiment of the present application, at least one of the first storage medium and at least one of the second storage medium may exist in the hierarchical storage architecture object. Each of the first storage media may be connected to at least one of the second storage media, or each of the second storage media may be connected to at least one of the first storage media.
In a preferred embodiment of the present application, each of the first storage media is provided with data storage areas having the same number as the number of the second storage media corresponding to the number of the second storage media connected thereto. For example, when 1 second storage medium is connected to the single first storage medium, 1 data storage area is provided in the high-speed storage area. When 2 second storage media are connected to the single first storage medium, 2 data storage areas are provided in the first storage medium. By establishing the data storage areas corresponding to the number of the second storage media in a one-to-one manner in the first storage media, the problem that target data of different second storage media are mixed in high-speed storage can be avoided, and the reading and writing speed of the first storage media is further improved.
In an embodiment of the present application, at least one first storage medium exists in the hierarchical storage architecture object, and at least one data storage area exists in the first storage medium. A preset load balancing algorithm may be adopted to reasonably distribute the data load pressure to each data storage area in the hierarchical storage architecture object. After the target data sent by the application process is obtained, the background processing process determines the data storage area with the lower current load as a target data storage area based on a load balancing algorithm, and stores the target data into the target data storage area.
According to the layered writing method, the target data storage areas are determined in the plurality of data storage areas through the load balancing algorithm, the data load pressure is reasonably distributed to each data storage area in the layered storage architecture object, and the storage performance of the layered storage architecture object is further improved.
In another embodiment of the present application, the writing the metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium includes:
and when a preset transfer period is reached or the preset service storage capacity of the first storage medium reaches an upper limit, writing the metadata corresponding to the target data into the metadata area of the second storage medium.
In this embodiment of the application, when a preset transfer period is reached or a preset traffic storage amount of the first storage medium reaches an upper limit, the target data should be transferred from the first storage medium to the second storage medium. Therefore, the background processing process may write the metadata corresponding to the target data into the second storage medium metadata area first, so as to prepare for subsequent target data transfer.
In this embodiment of the application, the preset transfer period is a preset target data transfer period, and the preset transfer period may be 1s, 1min, 5min, 3h, 1day, and the like, which is not limited in this application.
In this embodiment of the application, the preset traffic storage amount of the first storage medium may specifically reach an upper limit, where the data storage area in the first storage medium reaches the preset traffic storage amount. The preset service storage capacity may be equivalent to the data storage capacity, or may be equivalent to 4/5 of the data storage capacity, and the application is not limited thereto.
According to the layered writing method, when a preset transfer period is reached or the preset service storage capacity of the first storage medium reaches an upper limit, the metadata corresponding to the target data is transferred to the metadata area of the second storage medium. The method and the device realize the regular transfer of the target data, more fully utilize the storage spaces of the first storage medium and the second storage medium, and further improve the storage performance of the hierarchical storage architecture object.
In another embodiment of the present application, the writing the target data in the data storage area to the second storage medium includes:
when the metadata corresponding to the target data are successfully and completely written into the metadata area of the second storage medium, writing the target data in the data storage area into the second storage medium;
deleting the target data in the first storage medium.
In this embodiment of the present application, when the metadata corresponding to the target data is successfully written into the metadata area of the second storage medium, the preparation work for writing the target data into the second storage medium is completed, the target data is written into the second storage medium, and the target data in the first storage medium and the metadata corresponding to the target data in the data storage area are deleted, so that the transfer of the target data is completed. And if the metadata corresponding to the target data cannot be completely written into the metadata area of the second storage medium, and the storage space of the current second storage medium is full, the background processing process returns failure information, and the target data transfer process is stopped.
In a preferred embodiment of the present application, the sending the target data to the second storage medium may specifically be: and writing the target data into the second storage medium in a large data block mode in sequence, and writing the target data into the second storage medium in sequence, so that compared with the random writing of the target data into the second storage medium, the writing speed of the second storage medium can be further improved, and the subsequent further management of the target data in the second storage medium is facilitated.
By the layered writing method of the embodiment of the application, when the metadata corresponding to the target data is successfully transferred to the metadata area of the second storage medium, the target data is written into the second storage medium, so that the safety of the transfer process of the target data can be further ensured; and the low-speed storage directly stores only the target data, thereby avoiding the problem of write amplification of data transferred from the first storage medium to the second storage medium.
In another embodiment of the present application, the method further comprises:
and all the second storage medium metadata areas in the first storage medium form an independent hard disk redundant array.
In this embodiment of the present application, the second storage medium generally has a higher storage capacity, and if the metadata of the second storage medium is damaged, all data in the second storage medium may not be readable, and the performance of hierarchical storage is affected. The redundant array of the independent hard disk can enhance the integration degree of data, enhance the fault-tolerant function, increase the processing capacity or capacity and carry out the redundancy protection of data. When a certain second storage medium metadata area in the redundant array of the independent hard disk has data failure, the data can still be read out and reconstructed, so that the problem that all data in the second storage medium cannot be read due to metadata damage is avoided.
In a specific implementation, the second storage medium metadata area may form an RAID 1 type redundant array of independent hard disks, and the redundant array of independent hard disks may have two or more groups of N storage media as mirror images, and may have a good reading speed in some multithreading operating systems, where the reading speed is theoretically equal to a multiple of the number of the storage media, and is the same as RAID 0. In addition, the writing speed is slightly reduced. The operation can be maintained as long as one disk is normal, and the reliability is highest. The second storage medium metadata area can also form an RAID5 type redundant array of independent hard disks, and the redundant array of independent hard disks is a storage solution with storage performance, data safety and storage cost taken into consideration. It uses Disk Striping (hard Disk partitioning) technology. RAID5 requires at least three storage media, which, instead of backing up the stored data, stores the data and corresponding parity information on the individual storage media that make up RAID5, and the parity information and corresponding data are stored separately on different storage media. When a storage medium of RAID5 is corrupted, the remaining data and corresponding parity information may be used to recover the corrupted data.
Referring to fig. 2, a flowchart illustrating steps of an embodiment of a layered storage reading method according to the present application is shown, and is applied to a layered storage architecture object, where the layered storage architecture object includes a first storage medium and a second storage medium, and may include the following steps:
step 201, receiving a data calling request, wherein the data calling request comprises a to-be-called data list;
in this embodiment of the application, the application process may send a data retrieval request to the background process through an interface provided by the background process, and retrieve data to be retrieved.
Step 202, searching target metadata from the metadata area of the second storage medium for the to-be-called data list, wherein the target metadata is metadata corresponding to the to-be-called data in the to-be-called data list;
in this embodiment of the application, the background processing process searches whether target metadata corresponding to data to be called in the data list to be called exists in a metadata area of a second storage medium in the first storage medium according to the data list to be called in the data calling request. Since the metadata of the second storage medium is stored in the first storage medium, the speed of searching for the data to be called can be higher than before.
Step 203, when the target metadata exists in the metadata area of the second storage medium, acquiring the data to be called from the second storage medium by using the target metadata;
in this embodiment of the present application, when target metadata corresponding to the data to be called exists in the metadata area of the second storage medium, a storage location of the data to be called may be determined, and the data to be called may be acquired from the second storage medium. Since the target data in the second storage medium are all written sequentially, the reading speed can be higher than that of the conventional method of randomly writing the target data when the data is called.
In this embodiment of the application, when the target metadata corresponding to the data to be retrieved does not exist in the metadata area of the second storage medium, the background processing process returns read failure information and stops processing the data retrieval request.
And step 204, returning the data to be called.
In this embodiment of the application, the background processing process may directly retrieve data to be retrieved from the low-speed storage, and return the data to be retrieved to the application process. And when the same target data is called for more times, the target data can be transferred to the first storage medium, so that the reading speed of the data to be called is improved.
According to the layered storage reading method, the metadata of the second storage medium is stored in the first storage medium, so that the searching speed and the calling speed of the data to be called are improved, and the storage performance of the layered storage architecture object is effectively improved.
In another embodiment of the present application, the first storage medium further includes a data storage area therein;
the step of searching whether the metadata corresponding to the data to be called corresponding to the data list to be called exists in the second storage medium metadata area based on the data list to be called comprises:
searching target metadata from the data storage area aiming at the data list to be called;
in the embodiment of the application, the data to be called can be still stored in the high-speed storage directly because the time for writing the data into the first storage medium is short and the data is not transferred into the second storage medium. The number of times of calling the data to be called is large. Therefore, the data to be called is reserved in the first storage medium, and the calling speed of the data to be called is further improved.
When the target metadata does not exist in the data storage area, searching the target metadata from the second storage medium metadata area.
In this embodiment of the application, the background processing process may determine whether to-be-called data exists in the first storage medium, and then determine whether to-be-called data corresponds to target metadata in a metadata area of the second storage medium. Because the data storage area and the second storage medium metadata area are both in the first storage medium, the speed of searching the data to be called is obviously improved.
According to the layered storage reading method, whether the data to be called exists in the data storage area is searched first, and whether the data to be called exists in the metadata area of the second storage medium is searched, so that the storage position of the data to be called can be determined efficiently, and the reading efficiency of layered storage is improved.
It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the embodiments. Further, those skilled in the art will also appreciate that the embodiments described in the specification are presently preferred and that no particular act is required of the embodiments of the application.
Referring to fig. 3, a block diagram of a layered storage writing apparatus of the present application is shown, which is applied to a layered storage architecture object, where the layered storage architecture object includes a first storage medium and a second storage medium, the first storage medium includes a second storage medium metadata area and a data storage area, and may include the following modules:
a target data acquiring module 301, configured to acquire target data;
a target data and metadata writing module 302, configured to generate metadata corresponding to the target data, and write the target data and the metadata corresponding to the target data into the data storage area;
a second storage medium writing module 303, configured to write metadata corresponding to the target data in the data storage area into the second storage medium metadata area; and writing the target data in the data storage area into the second storage medium.
In another embodiment of the present application, the data storage area includes a first storage medium metadata area and a target data cache area;
the target data and metadata writing module comprises:
a first storage medium metadata writing sub-module, configured to write metadata corresponding to the target data in the first storage medium metadata area;
and the target data writing submodule is used for writing the target data into the target data cache region.
In another embodiment of the present application, the target data and metadata writing module further includes:
the target data storage area determining submodule is used for determining a target data storage area in the data storage area by adopting a preset load balancing algorithm;
and the target data storage submodule is used for storing the target data in the target data storage area.
In another embodiment of the present application, the low speed storage medium writing module includes:
and the second storage medium metadata writing submodule is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area when a preset transfer period is reached or the preset business storage volume of the first storage medium reaches an upper limit.
In another embodiment of the present application, the low speed storage medium writing module further includes:
the second storage medium target data writing sub-module is used for sending the target data to the second storage medium when the metadata corresponding to the target data is successfully transferred to the second storage medium metadata area;
and the target data deleting submodule is used for deleting the target data in the first storage medium.
In another embodiment of the present application, all of the second storage medium metadata areas in the first storage medium constitute a redundant array of independent hard disks.
Referring to fig. 4, a block diagram of an embodiment of a layered storage and reading apparatus according to the present application is shown, which is applied to a layered storage architecture object, where the layered storage architecture object includes a first storage medium and a second storage medium, and the first storage medium includes a second storage medium metadata area; the hierarchical storage reading device can comprise the following modules:
a data retrieval request receiving module 401, configured to receive a data retrieval request, where the data retrieval request includes a to-be-retrieved data list;
a second storage medium metadata searching module 402, configured to search, for the to-be-called data list, target metadata from the second storage medium metadata area, where the target metadata is metadata corresponding to the to-be-called data in the to-be-called data list;
a to-be-called data obtaining module 403, configured to obtain, when the target metadata exists in the second storage medium metadata area, the to-be-called data from a second storage medium by using the target metadata;
and a to-be-called data returning module 404, configured to return the to-be-called data.
In another embodiment of the present application, the first storage medium further includes a data storage area therein;
the second storage medium metadata lookup module includes:
the data to be called searching submodule of the data storage area is used for searching target metadata from the data storage area aiming at the data list to be called;
and the second storage medium metadata searching sub-module is used for searching the target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
Referring to fig. 5, a block diagram of a hierarchical storage system embodiment of the present application is shown, the system comprising:
a high-speed storage module 501 for caching target data;
a low-speed storage module 502, configured to store the target data sent by the first storage medium;
the high-speed storage module comprises a second storage medium metadata area used for storing metadata of a second storage medium.
In another embodiment of the present application, the metadata areas of the second storage media in all the high-speed storage modules form a redundant array of independent hard disks for data redundancy protection.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one of skill in the art, embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
In a typical configuration, the computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (fransitory media), such as modulated data signals and carrier waves.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the true scope of the embodiments of the application.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The foregoing describes in detail a layered storage writing method and apparatus, a reading method and apparatus, and a system provided by the present application, and a principle and an implementation of the present application are described herein by applying an example, and the description of the foregoing embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (18)

1. A layered storage write method is applied to a layered storage architecture object, wherein the layered storage architecture object comprises a first storage medium and a second storage medium, the first storage medium comprises a second storage medium metadata area and a data storage area, and the method comprises the following steps:
acquiring target data;
generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into the data storage area;
writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium; and writing the target data in the data storage area into the second storage medium.
2. The tiered storage write method of claim 1 wherein said data storage area includes a first storage medium metadata area and a target data cache area;
the step of writing the target data and its corresponding metadata into the data storage area comprises:
writing metadata corresponding to the target data into the first storage medium metadata area;
and writing the target data into the target data cache region.
3. The tiered storage writing method of claim 1 or 2, wherein said step of writing said target data and its corresponding metadata in said data storage area further comprises:
determining a target data storage area in the data storage areas by adopting a preset load balancing algorithm;
storing the target data in the target data store.
4. The layered writing method of claim 1, wherein the step of writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area comprises:
and when a preset transfer period is reached or the preset service storage capacity of the first storage medium reaches an upper limit, writing the metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium.
5. The layered writing method according to claim 1 or 4, wherein the step of writing the target data in the data storage area to the second storage medium comprises:
when all metadata corresponding to the target data are written into the metadata area of the second storage medium, writing the target data in the data storage area into the second storage medium;
and deleting the target data in the data storage area.
6. The layered writing method of claim 1,
and all the second storage medium metadata areas in the first storage medium form an independent hard disk redundant array.
7. A layered storage reading method is applied to a layered storage architecture object, wherein the layered storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area;
the method comprises the following steps:
receiving a data calling request, wherein the data calling request comprises a to-be-called data list;
searching target metadata from the metadata area of the second storage medium aiming at the data list to be called, wherein the target metadata is metadata corresponding to the data to be called in the data list to be called;
when the target metadata exists in the metadata area of the second storage medium, acquiring the data to be called from the second storage medium by adopting the target metadata;
and returning the data to be called.
8. The tiered storage read method of claim 7 wherein said first storage medium further includes a data storage area therein;
the step of searching target metadata from the second storage medium metadata area aiming at the data list to be called comprises the following steps:
searching target metadata from the data storage area aiming at the data list to be called;
when the target metadata does not exist in the data storage area, searching the target metadata from the second storage medium metadata area.
9. A layered storage architecture object, wherein the layered storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area and a data storage area, the apparatus comprising:
the target data acquisition module is used for acquiring target data;
the target data and metadata writing module is used for generating metadata corresponding to the target data and writing the target data and the metadata corresponding to the target data into the data storage area;
a second storage medium writing module, configured to write metadata corresponding to target data in the data storage area into a second storage medium metadata area; and writing the target data in the data storage area into the second storage medium.
10. The tiered storage device of claim 9 wherein said data storage areas include a first storage media metadata area and a target data cache area;
the target data and metadata writing module comprises:
a first storage medium metadata writing sub-module, configured to write metadata corresponding to the target data in the first storage medium metadata area;
and the target data writing submodule is used for writing the target data into the target data cache region.
11. The tiered storage device of claim 9 or 10, wherein said target data and metadata write module further comprises:
the target data storage area determining submodule is used for determining a target data storage area in the data storage area by adopting a preset load balancing algorithm;
and the target data storage submodule is used for storing the target data in the target data storage area.
12. The tiered storage device of claim 9, wherein the low speed storage medium writing module comprises:
and the second storage medium metadata writing submodule is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area when a preset transfer period is reached or the preset business storage volume of the first storage medium reaches an upper limit.
13. The tiered storage device of claim 9 or 12, wherein the low speed storage medium writing module further comprises:
the second storage medium target data writing sub-module is used for sending the target data to the second storage medium when the metadata corresponding to the target data is successfully transferred to the second storage medium metadata area;
and the target data deleting submodule is used for deleting the target data in the first storage medium.
14. The tiered storage device of claim 9 wherein all of said second storage medium metadata regions in said first storage medium form a redundant array of independent hard disks.
15. The layered storage reading device is applied to a layered storage architecture, the layered storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area; the device comprises the following modules:
the data calling request receiving module is used for receiving a data calling request, and the data calling request comprises a to-be-called data list;
the second storage medium metadata searching module is used for searching target metadata from the second storage medium metadata area aiming at the data list to be called, wherein the target metadata is metadata corresponding to the data to be called in the data list to be called;
a to-be-called data acquisition module, configured to acquire, when the target metadata exists in the second storage medium metadata area, the to-be-called data from a second storage medium by using the target metadata;
and the data to be called returning module is used for returning the data to be called.
16. The tiered storage read device of claim 15 wherein said first storage medium further includes a data storage area therein;
the second storage medium metadata lookup module includes:
the data to be called searching submodule of the data storage area is used for searching target metadata from the data storage area aiming at the data list to be called;
and the second storage medium metadata searching sub-module is used for searching the target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
17. A tiered storage system, comprising:
the high-speed storage module is used for caching target data;
the low-speed storage module is used for storing the target data sent by the first storage medium;
the high-speed storage module comprises a second storage medium metadata area used for storing metadata of a second storage medium.
18. The tiered storage system of claim 17 wherein the second storage media metadata regions in all of said high speed storage modules form redundant arrays of independent hard disks for data redundancy protection.
CN201811521157.3A 2018-12-12 2018-12-12 Hierarchical storage writing method and device, reading method and device and system Active CN111309245B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811521157.3A CN111309245B (en) 2018-12-12 2018-12-12 Hierarchical storage writing method and device, reading method and device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811521157.3A CN111309245B (en) 2018-12-12 2018-12-12 Hierarchical storage writing method and device, reading method and device and system

Publications (2)

Publication Number Publication Date
CN111309245A true CN111309245A (en) 2020-06-19
CN111309245B CN111309245B (en) 2024-04-09

Family

ID=71149232

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811521157.3A Active CN111309245B (en) 2018-12-12 2018-12-12 Hierarchical storage writing method and device, reading method and device and system

Country Status (1)

Country Link
CN (1) CN111309245B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112035065A (en) * 2020-08-28 2020-12-04 北京浪潮数据技术有限公司 Data writing method, device and equipment and computer readable storage medium
CN112269540A (en) * 2020-10-16 2021-01-26 苏州浪潮智能科技有限公司 Data storage method, system, equipment and computer readable storage medium
CN112463079A (en) * 2020-12-17 2021-03-09 北京浪潮数据技术有限公司 Data storage control method, device, equipment and readable storage medium
CN117574446A (en) * 2024-01-16 2024-02-20 苏州元脑智能科技有限公司 Data protection method, device, equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104182353A (en) * 2014-08-21 2014-12-03 段超 Data storage system adopting different storage media
WO2016086649A1 (en) * 2014-12-04 2016-06-09 中兴通讯股份有限公司 Method, client, and storage medium for reading, writing, deleting, and querying a file
CN107704208A (en) * 2017-10-13 2018-02-16 郑州云海信息技术有限公司 A kind of restorative procedure of metadata, device and medium
CN108664218A (en) * 2018-04-23 2018-10-16 浙江大华技术股份有限公司 A kind of SSD storage methods based on data characteristics, apparatus and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104182353A (en) * 2014-08-21 2014-12-03 段超 Data storage system adopting different storage media
WO2016086649A1 (en) * 2014-12-04 2016-06-09 中兴通讯股份有限公司 Method, client, and storage medium for reading, writing, deleting, and querying a file
CN107704208A (en) * 2017-10-13 2018-02-16 郑州云海信息技术有限公司 A kind of restorative procedure of metadata, device and medium
CN108664218A (en) * 2018-04-23 2018-10-16 浙江大华技术股份有限公司 A kind of SSD storage methods based on data characteristics, apparatus and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
于耳;: "分层存储:一种经济的存储方式", 中国教育网络, no. 11, 5 November 2012 (2012-11-05) *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112035065A (en) * 2020-08-28 2020-12-04 北京浪潮数据技术有限公司 Data writing method, device and equipment and computer readable storage medium
CN112035065B (en) * 2020-08-28 2022-06-07 北京浪潮数据技术有限公司 Data writing method, device and equipment and computer readable storage medium
CN112269540A (en) * 2020-10-16 2021-01-26 苏州浪潮智能科技有限公司 Data storage method, system, equipment and computer readable storage medium
CN112463079A (en) * 2020-12-17 2021-03-09 北京浪潮数据技术有限公司 Data storage control method, device, equipment and readable storage medium
CN112463079B (en) * 2020-12-17 2023-12-22 北京浪潮数据技术有限公司 Data storage control method, device, equipment and readable storage medium
CN117574446A (en) * 2024-01-16 2024-02-20 苏州元脑智能科技有限公司 Data protection method, device, equipment and storage medium
CN117574446B (en) * 2024-01-16 2024-04-05 苏州元脑智能科技有限公司 Data protection method, device, equipment and storage medium

Also Published As

Publication number Publication date
CN111309245B (en) 2024-04-09

Similar Documents

Publication Publication Date Title
US9916176B2 (en) Method and apparatus of accessing data of virtual machine
CN106407040B (en) A kind of duplicating remote data method and system
CN111309245B (en) Hierarchical storage writing method and device, reading method and device and system
CN107807794B (en) Data storage method and device
US11243706B2 (en) Fragment management method and fragment management apparatus
CN102024044B (en) Distributed file system
CN109947363B (en) Data caching method of distributed storage system
US10126946B1 (en) Data protection object store
US7933938B2 (en) File storage system, file storing method and file searching method therein
CN106776130B (en) Log recovery method, storage device and storage node
US9189493B2 (en) Object file system
US6907507B1 (en) Tracking in-progress writes through use of multi-column bitmaps
US10095585B1 (en) Rebuilding data on flash memory in response to a storage device failure regardless of the type of storage device that fails
US7844776B2 (en) RAID capacity expansion handling method and system with concurrent data access capability
US10642508B2 (en) Method to limit impact of partial media failure of disk drive and detect/report the loss of data for objects due to partial failure of media
CN112379825A (en) Distributed data storage method and device based on data feature sub-pools
CN109144403B (en) Method and equipment for switching cloud disk modes
US10534558B1 (en) Relocating paged metadata
CN105068896B (en) Data processing method and device based on RAID backup
CN109508140B (en) Storage resource management method and device, electronic equipment and system
CN107422990B (en) Dynamic migration method and device
CN116048878A (en) Business service recovery method, device and computer equipment
US20160371192A1 (en) Apparatus and method for performing cache management in a storage system
CN112328181A (en) Metadata reading and writing method and device of distributed storage system
CN117539690B (en) Method, device, equipment, medium and product for merging and recovering multi-disk data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant