CN111309245B - Hierarchical storage writing method and device, reading method and device and system - Google Patents
Hierarchical storage writing method and device, reading method and device and system Download PDFInfo
- Publication number
- CN111309245B CN111309245B CN201811521157.3A CN201811521157A CN111309245B CN 111309245 B CN111309245 B CN 111309245B CN 201811521157 A CN201811521157 A CN 201811521157A CN 111309245 B CN111309245 B CN 111309245B
- Authority
- CN
- China
- Prior art keywords
- storage medium
- data
- storage
- metadata
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 519
- 238000000034 method Methods 0.000 title claims abstract description 85
- 238000013500 data storage Methods 0.000 claims abstract description 128
- 238000012546 transfer Methods 0.000 claims description 16
- 230000003321 amplification Effects 0.000 abstract description 6
- 238000003199 nucleic acid amplification method Methods 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 28
- 238000010586 diagram Methods 0.000 description 14
- 238000012545 processing Methods 0.000 description 14
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000013404 process transfer Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0616—Improving the reliability of storage systems in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a hierarchical storage writing method and device, and a reading method, device and system. The hierarchical storage writing method comprises the following steps: acquiring target data; generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into a data storage area; writing metadata corresponding to the target data in the data storage area into a metadata area of the second storage medium; and writing the target data in the data storage area into the second storage medium. And establishing a metadata area of the second storage medium in the first storage medium for storing metadata of the second storage medium. Only the target data is stored in the second storage medium, and metadata corresponding to the target data is not stored. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is avoided, metadata and target data are not needed to be stored in different address spaces in the second storage medium, and the overall writing speed of layered storage is improved.
Description
Technical Field
The present disclosure relates to the field of storage technologies, and in particular, to a hierarchical storage writing method and device, and a reading method, device, and system.
Background
Hierarchical storage refers to storing data in different levels of media and performing operations such as automatic or manual data migration, copying, etc. between different media. High-speed storage media with fast access speeds typically have high storage unit costs and low single media capacities. Whereas low-speed storage media with slow access speeds typically have low storage unit costs and a single medium capacity. Thus, layered storage techniques may be employed, using high-speed storage media in conjunction with low-speed storage media, to achieve better storage performance at lower cost.
In the related art, a high-speed storage medium and a low-speed storage medium in a hierarchical storage architecture each have a file system to manage own storage space. Generally, after data of an application is transferred to a fast storage medium, a background thread is responsible for transferring the data of the application from the fast storage medium to a slow storage medium. Each time a write operation is performed in a low-speed storage medium, metadata such as bitmaps and inodes in a file system of the low-speed storage medium need to be updated, so that write amplification in the low-speed storage medium is caused. The write life and performance of the low-speed storage medium are both reduced faster. In addition, since the metadata and the data in the low-speed storage medium are stored in different address spaces when updating, the low-speed storage medium is equivalent to randomly writing data when updating and storing, and the performance of the second storage medium of the low-speed storage medium is further affected, so that the layered storage architecture is difficult to achieve the expected storage speed. In addition, the single medium capacity of the low-speed storage medium is high, and once the metadata part of the low-speed storage medium is damaged, all data of the single low-speed storage medium cannot be read, full-disk recovery is required, and the security is low.
Disclosure of Invention
In view of the foregoing, embodiments of the present application have been presented to provide a hierarchical storage writing method and apparatus, a reading method and apparatus, and a system that overcome or at least partially solve the foregoing problems.
A hierarchical storage writing method applied to a hierarchical storage architecture object, the hierarchical storage architecture object including a first storage medium and a second storage medium, the first storage medium including a second storage medium metadata area and a data storage area, the method comprising:
acquiring target data;
generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into the data storage area;
writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium; and writing the target data in the data storage area into the second storage medium.
Preferably, the data storage area includes a first storage medium metadata area and a target data cache area;
the step of writing the target data and the corresponding metadata thereof into the data storage area comprises the following steps:
Writing metadata corresponding to the target data into the first storage medium metadata area;
and writing the target data into the target data cache area.
Preferably, the step of writing the target data and the corresponding metadata thereof into the data storage area further comprises:
determining a target data storage area in the data storage areas by adopting a preset load balancing algorithm;
storing the target data in the target data storage area.
Preferably, the step of writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium includes:
and when a preset transfer period is reached or the preset business storage amount of the first storage medium reaches the upper limit, writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium.
Preferably, the step of writing the target data in the data storage area to the second storage medium includes:
when all metadata corresponding to the target data are written into the metadata area of the second storage medium, writing the target data in the data storage area into the second storage medium;
And deleting the target data in the data storage area.
Preferably, all the metadata areas of the second storage medium in the first storage medium form a redundant array of independent hard disks.
A hierarchical storage reading method is applied to a hierarchical storage architecture object, wherein the hierarchical storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area;
the method comprises the following steps:
receiving a data calling request, wherein the data calling request comprises a data list to be called;
searching target metadata from the second storage medium metadata area aiming at the data list to be called, wherein the target metadata are metadata corresponding to the data to be called in the data list to be called;
when the target metadata exists in the metadata area of the second storage medium, acquiring the data to be called from the second storage medium by adopting the target metadata;
and returning the data to be called.
Preferably, the first storage medium further includes a data storage area therein;
the step of searching the target metadata from the second storage medium metadata area for the data list to be called includes:
Searching target metadata from the data storage area aiming at the data list to be called;
and searching for target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
A tiered storage device for use in a tiered storage architecture object, the tiered storage architecture object including a first storage medium and a second storage medium, the first storage medium including a second storage medium metadata area and a data storage area, the device comprising:
the target data acquisition module is used for acquiring target data;
the target data and metadata writing module is used for generating metadata corresponding to the target data and writing the target data and the metadata corresponding to the target data into the data storage area;
the second storage medium writing module is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area; and writing the target data in the data storage area into the second storage medium.
Preferably, the data storage area includes a first storage medium metadata area and a target data cache area;
The target data and metadata writing module comprises:
the first storage medium metadata writing sub-module is used for writing metadata corresponding to the target data into the first storage medium metadata area;
and the target data writing sub-module is used for writing the target data into the target data cache area.
Preferably, the target data and metadata writing module further includes:
the target data storage area determining sub-module is used for determining a target data storage area in the data storage area by adopting a preset load balancing algorithm;
and the target data storage sub-module is used for storing the target data in the target data storage area.
Preferably, the low-speed storage medium writing module includes:
and the second storage medium metadata writing sub-module is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area when a preset transfer period is reached or the preset service storage amount of the first storage medium reaches an upper limit.
Preferably, the low-speed storage medium writing module further includes:
the second storage medium target data writing sub-module is used for sending the target data to the second storage medium when the metadata corresponding to the target data is successfully transferred to the second storage medium metadata area;
And the target data deleting sub-module is used for deleting the target data in the first storage medium.
Preferably, all the metadata areas of the second storage medium in the first storage medium form a redundant array of independent hard disks.
A tiered storage reader device for use in a tiered storage architecture, the tiered storage architecture object including a first storage medium and a second storage medium, the first storage medium including a second storage medium metadata area therein; the device comprises the following modules:
the data calling request receiving module is used for receiving a data calling request, wherein the data calling request comprises a data list to be called;
the second storage medium metadata searching module is used for searching target metadata from the second storage medium metadata area aiming at the data list to be called, wherein the target metadata is metadata corresponding to the data to be called in the data list to be called;
the data to be called is obtained from the second storage medium by adopting the target metadata when the target metadata exists in the metadata area of the second storage medium;
and the data to be called return module is used for returning the data to be called.
Preferably, the first storage medium further includes a data storage area therein;
the second storage medium metadata search module includes:
the data storage area to-be-called data searching sub-module is used for searching target metadata from the data storage area aiming at the to-be-called data list;
and the second storage medium metadata searching sub-module is used for searching the target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
A tiered storage system, comprising:
the high-speed storage module is used for caching target data;
the low-speed storage module is used for storing the target data sent by the first storage medium;
the high-speed storage module comprises a second storage medium metadata area and is used for storing metadata of a second storage medium.
Preferably, the metadata areas of the second storage media in all the high-speed storage modules form an independent hard disk redundant array for data redundancy protection.
Embodiments of the present application include the following advantages:
according to the embodiment of the application, the metadata area of the second storage medium is built in the first storage medium and is used for storing metadata of the second storage medium. Only target data is stored in the second storage medium, and metadata corresponding to the target data is not stored. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is avoided, meanwhile, metadata and target data are not needed to be stored in different address spaces in the second storage medium, the problem that the write speed is reduced when the second storage medium is randomly written is avoided, and the overall write speed of layered storage is effectively improved.
Drawings
FIG. 1 is a flow chart of steps of an embodiment of a tiered storage write method of the present application;
FIG. 2 is a flow chart of steps of an embodiment of a tiered storage read method of the present application;
FIG. 3 is a block diagram of an embodiment of a tiered storage write device of the present application;
FIG. 4 is a block diagram of an embodiment of a tiered storage read device of the present application;
FIG. 5 is a block diagram of an embodiment of a tiered storage system of the present application.
Description of the embodiments
In order to make the above objects, features and advantages of the present application more comprehensible, the present application is described in further detail below with reference to the accompanying drawings and embodiments.
One of the core ideas of the embodiments of the present application is that metadata of a second storage medium is stored by creating a metadata area of the second storage medium in the first storage medium. Only target data is stored in the second storage medium, and metadata corresponding to the target data is not stored. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is avoided, meanwhile, metadata and target data are not needed to be stored in different address spaces in the second storage medium, the problem that the write speed is reduced when the second storage medium is randomly written is avoided, and the overall write speed of layered storage is effectively improved.
Referring to FIG. 1, a flowchart of steps of an embodiment of a tiered storage writing method of the present application is shown, where the method is applied to a tiered storage architecture object that includes a first storage medium and a second storage medium, where the first storage medium includes a second storage medium metadata area and a data storage area.
In the embodiments of the present application, a storage medium refers to a carrier storing data. The first storage medium has a higher read-write speed than the second storage medium. For example, when the first storage medium is an SCM, the low-speed storage medium may be a storage medium having a read/write speed lower than that of the SCM, such as NVMESSD, optaneSSD, a mechanical hard disk, or the like. When the first storage medium is NVMESSD, optaneSSD, the second storage medium may be a mechanical hard disk, a magnetic tape, or the like.
In a preferred embodiment of the present application, the first storage medium may be an SCM storage medium. The SCM storage medium has both features of persistence (Storageclass) and quick direct access (Memory), and the access delay of the SCM storage medium can be less than 1 mus, which is excellent in terms of lifetime and data retention capability. When the method and the device are applied to SCM storage media, compared with storage media such as NVMESSD, optaneSSD, the method and the device can obviously improve the read-write speed of metadata of the second storage media in the first storage media, so that the read-write speed of the hierarchical storage architecture object is obviously improved.
In this embodiment of the present application, the second storage medium may be any storage medium having a lower read/write speed than the first storage medium. For example, when
In this embodiment of the present application, a second storage medium metadata area is established in the first storage medium, and is used to store metadata such as bitmap and inode in the second storage medium, where the second storage medium does not store metadata of itself, and only stores target data.
In this embodiment of the present application, the data storage area is a data partition set in the first storage medium for storing the target data.
The hierarchical storage writing method may include the steps of:
step 101, obtaining target data;
in the embodiment of the application, the target data may be obtained from an interface provided by a background processing process for managing all storage media in the hierarchical storage architecture object for an application process. And the application process sends the target data to the hierarchical storage structure through an interface provided by the background processing process.
102, generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into the data storage area;
in this embodiment of the present application, the metadata may be an index of target data, which is used to describe information of a data attribute (property), and may implement functions of indicating a storage location, historical data, searching resources, recording a file, and the like. And establishing metadata based on the target data, so that the target data can be conveniently and quickly searched based on the metadata.
In this embodiment of the present application, after the background process receives the target data from the interface, the target data and the metadata corresponding to the target data are written into a data storage area preset in the first storage medium.
Step 103, writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium; and writing the target data in the data storage area into the second storage medium.
In this embodiment of the present application, when target data in the data storage area in the first storage medium needs to be written into the second storage medium, a background processing process writes metadata corresponding to the target data into the second storage medium metadata area in the first storage medium.
In this embodiment, the background process transfers the target data to the second storage medium. Only target data is stored in the second storage medium, and metadata corresponding to the target data is not stored.
According to the hierarchical storage writing method, the metadata area of the second storage medium is built in the first storage medium and is used for storing metadata of the second storage medium. Only target data is stored in the second storage medium, and metadata corresponding to the target data is not stored. Therefore, the problem of write amplification of data transferred from the first storage medium to the second storage medium is avoided, meanwhile, metadata and target data are not needed to be stored in different address spaces in the second storage medium, the problem that the write speed is reduced when the second storage medium is randomly written is avoided, and the overall write speed of layered storage is effectively improved.
In another embodiment of the present application, the data storage area includes a first storage medium metadata area and a target data cache area;
the step of writing the target data and the corresponding metadata thereof into the data storage area comprises the following steps:
writing metadata corresponding to the target data into the first storage medium metadata area;
and writing the target data into the target data cache area.
In this embodiment of the present application, the data storage area is a data storage area of the first storage medium itself, and is divided into a first storage medium metadata area and a target data cache area, where the first storage medium metadata area is used to store metadata of target data stored in the first storage medium itself, and the target data cache area is used to store target data cached in the first storage medium. Because the first storage medium has higher read-write speed, the target data sent from the application process is written into the first storage medium, so that the application process can conveniently and quickly complete the data writing process.
By the hierarchical storage writing method, the target data sent by the application process can be quickly stored in the first storage medium, and the writing efficiency of hierarchical storage is improved.
In another embodiment of the present application, the step of writing the target data and the metadata corresponding thereto into the data storage area further includes:
determining a target data storage area in the data storage areas by adopting a preset load balancing algorithm;
storing the target data in the target data storage area.
In an embodiment of the present application, at least one of the first storage medium and at least one of the second storage medium may be present in the hierarchical storage architecture object. Each of the first storage media may be connected to at least one of the second storage media, or each of the second storage media may be connected to at least one of the first storage media.
In a preferred embodiment of the present application, each of the first storage media is provided with the same number of data storage areas as the number of second storage media corresponding to the number of second storage media connected thereto. For example, when the single first storage medium is connected to 1 second storage medium, 1 data storage area is provided in the high-speed storage area. When the single first storage medium is connected with 2 second storage media, 2 data storage areas are arranged in the first storage medium. By establishing the data storage areas which are in one-to-one correspondence with the number of the second storage media in the first storage media, the problem that target data of different second storage media are mixed in high-speed storage direct can be avoided, and the read-write speed of the first storage media is further improved.
In an embodiment of the present application, at least one first storage medium exists in the hierarchical storage architecture object, and at least one data storage area exists in the first storage medium. A preset load balancing algorithm may be employed to reasonably distribute data load pressure to each data store in the tiered storage architecture object. After target data sent by an application process is obtained, the background processing process determines the data storage area with lower current load as a target data storage area based on a load balancing algorithm, and stores the target data into the target data storage area.
According to the hierarchical writing method, the target data storage areas are determined in the plurality of data storage areas through the load balancing algorithm, and the data load pressure is reasonably distributed to each data storage area in the hierarchical storage architecture object, so that the storage performance of the hierarchical storage architecture object is further improved.
In another embodiment of the present application, the step of writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium includes:
and when a preset transfer period is reached or the preset business storage amount of the first storage medium reaches an upper limit, writing the metadata corresponding to the target data into the metadata area of the second storage medium.
In the embodiment of the present application, when a preset transfer period is reached or a preset traffic storage amount of the first storage medium reaches an upper limit, the target data should be transferred from the first storage medium to the second storage medium. For this purpose, the background process may first write metadata corresponding to the target data into the metadata area of the second storage medium, and prepare for subsequent target data transfer.
In this embodiment of the present application, the preset transfer period is a preset target data transfer period, and the preset transfer period may be 1s, 1min, 5min, 3h, 1day, or the like, which is not limited in this application.
In this embodiment of the present application, the reaching of the preset service storage amount of the first storage medium to the upper limit may specifically be that the data storage area in the first storage medium reaches the preset service storage amount. The preset service storage amount may be equal to the data storage area capacity, or may be equal to 4/5 of the data storage area capacity, which is not limited in this application.
According to the layered writing method, when a preset transfer period is reached or the preset service storage amount of the first storage medium reaches an upper limit, metadata corresponding to the target data is transferred to the metadata area of the second storage medium. The method realizes the periodic transfer of the target data, more fully utilizes the storage space of the first storage medium and the second storage medium, and further improves the storage performance of the hierarchical storage architecture object.
In another embodiment of the present application, the step of writing the target data in the data storage area to the second storage medium includes:
when the metadata corresponding to the target data are successfully written into the metadata area of the second storage medium, writing the target data in the data storage area into the second storage medium;
deleting the target data in the first storage medium.
In this embodiment of the present application, when metadata corresponding to the target data is successfully written into the metadata area of the second storage medium, a preparation task of writing the target data into the second storage medium is completed, and the target data is written into the second storage medium, and metadata corresponding to the target data in the first storage medium and metadata corresponding to the target data in the data storage area are deleted, so that transfer of the target data is completed. If the metadata corresponding to the target data cannot be completely written into the metadata area of the second storage medium, the storage space of the second storage medium may be full, and the background processing process returns failure information to stop the target data transfer process.
In a preferred embodiment of the present application, the sending the target data to the second storage medium may specifically be: and writing the target data into the second storage medium in sequence in a large data block mode, and writing the target data into the second storage medium in sequence.
According to the hierarchical writing method, when the metadata corresponding to the target data is successfully transferred to the metadata area of the second storage medium, the target data is written into the second storage medium, so that the safety of a target data transfer process can be further ensured; and the low-speed storage directly stores only target data, thereby avoiding the problem of write amplification of data transferred from the first storage medium to the second storage medium.
In another embodiment of the present application, the method further comprises:
and the metadata areas of the second storage media in all the first storage media form an independent hard disk redundant array.
In this embodiment of the present application, the second storage medium generally has a higher storage capacity, if metadata of the second storage medium is damaged, all data in the second storage medium may not be readable, which affects performance of hierarchical storage, and in order to avoid this situation, metadata areas of the second storage medium in all first storage media may be formed into a redundant array of independent hard disks (Raid, redundant Array of Independent Disks). The redundant array of independent hard disk can enhance the data integration degree, enhance the fault tolerance function, increase the processing capacity or capacity and carry out the redundant protection of data. After a data failure occurs in a certain second storage medium metadata area in the redundant array of independent hard disks, the data can still be read out and reconstructed, so that the problem that all data in the second storage medium cannot be read due to metadata damage is avoided.
In a specific implementation, the second storage medium metadata area may form a RAID 1 type redundant array of independent hard disks, where the redundant array of independent hard disks may mirror N storage media of more than two groups, and in some multithreaded operating systems, the second storage medium metadata area may have a very good reading speed, where the theoretical reading speed is equal to a multiple of the number of storage media, and is the same as RAID 0. In addition, there is a slight decrease in the writing speed. The operation can be maintained as long as one magnetic disk is normal, and the reliability is highest. The second storage medium metadata area can also form a RAID5 type redundant array of independent hard disks, and the redundant array of independent hard disks is a storage solution with the advantages of storage performance, data security and storage cost. It uses Disk Striping technology. RAID5 requires at least three storage media, which do not back up the stored data, but rather store the data and corresponding parity information on each storage medium that makes up RAID5, and the parity information and corresponding data are stored on different storage media, respectively. When one of the storage medium data of RAID5 is corrupted, the remaining data and corresponding parity information may be utilized to recover the corrupted data.
Referring to fig. 2, a step flow diagram of an embodiment of a hierarchical storage read method of the present application is shown, which is applied to a hierarchical storage architecture object, where the hierarchical storage architecture object includes a first storage medium and a second storage medium, and may include the following steps:
step 201, receiving a data calling request, wherein the data calling request comprises a data list to be called;
in this embodiment of the present application, the application process may send a data call request to the background process through an interface provided by the background process, so as to call the data to be called.
Step 202, for the data list to be called, searching target metadata from the second storage medium metadata area, where the target metadata is metadata corresponding to the data to be called in the data list to be called;
in this embodiment of the present application, the background processing process searches whether there is target metadata corresponding to data to be called in the data to be called list in the metadata area of the second storage medium in the first storage medium according to the data to be called list in the data to be called request. Since the metadata of the second storage medium is stored in the first storage medium, the speed of searching the data to be called can have a higher processing speed than before.
Step 203, when the target metadata exists in the metadata area of the second storage medium, acquiring the data to be called from the second storage medium by adopting the target metadata;
in this embodiment of the present application, when target metadata corresponding to the data to be called exists in the metadata area of the second storage medium, a storage location of the data to be called may be determined, and the data to be called may be obtained from the second storage medium. Because the target data in the second storage medium are all written sequentially, the method has higher reading speed when the data is called as compared with the prior method for randomly writing the target data.
In this embodiment of the present application, when the target metadata corresponding to the data to be called does not exist in the metadata area of the second storage medium, the background processing process returns the reading failure information, and stops processing the data call request.
Step 204, returning the data to be called.
In this embodiment of the present application, the background process may directly call the data to be called from the low-speed storage directly, and return the data to be called to the application process. And when the same target data is called for more times, transferring the target data to the first storage medium, so as to facilitate the improvement of the reading speed of the data to be called.
By means of the hierarchical storage reading method, metadata of the second storage medium are stored in the first storage medium, so that the searching speed and the fetching speed of data to be fetched are improved, and the storage performance of the hierarchical storage architecture object is effectively improved.
In another embodiment of the present application, the first storage medium further includes a data storage area therein;
the step of searching whether metadata corresponding to the data to be called list exists in the metadata area of the second storage medium based on the data to be called list includes:
searching target metadata from the data storage area aiming at the data list to be called;
in this embodiment of the present application, the data to be fetched may not be transferred to the second storage medium due to the shorter time of writing into the first storage medium, so that the data to be fetched is still stored in the high-speed storage direct. And the data to be called can be called for more times. Therefore, the data to be called is reserved in the first storage medium, and the calling speed of the data to be called is further improved.
And searching for target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
In this embodiment of the present application, the background process may determine whether data to be called exists in the first storage medium, and then determine whether target metadata corresponding to the data to be called exists in the metadata area of the second storage medium. Because the data storage area and the second storage medium metadata area are both in the first storage medium, the speed of searching the data to be called is obviously improved.
According to the hierarchical storage reading method, the storage position of the data to be called can be determined efficiently by searching whether the data storage area has the data to be called or not and then searching whether the metadata area of the second storage medium has the data to be called or not, and the reading efficiency of the hierarchical storage is improved.
It should be noted that, for simplicity of description, the method embodiments are shown as a series of acts, but it should be understood by those skilled in the art that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently in accordance with the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments and that the acts referred to are not necessarily required by the embodiments of the present application.
Referring to fig. 3, a block diagram of an embodiment of a hierarchical storage writing device of the present application is shown, where the block diagram is applied to a hierarchical storage architecture object, where the hierarchical storage architecture object includes a first storage medium and a second storage medium, and the first storage medium includes a metadata area of the second storage medium and a data storage area, and may include the following modules:
a target data acquisition module 301, configured to acquire target data;
a target data and metadata writing module 302, configured to generate metadata corresponding to the target data, and write the target data and metadata corresponding to the target data into the data storage area;
a second storage medium writing module 303, configured to write metadata corresponding to the target data in the data storage area into the second storage medium metadata area; and writing the target data in the data storage area into the second storage medium.
In another embodiment of the present application, the data storage area includes a first storage medium metadata area and a target data cache area;
the target data and metadata writing module comprises:
the first storage medium metadata writing sub-module is used for writing metadata corresponding to the target data into the first storage medium metadata area;
And the target data writing sub-module is used for writing the target data into the target data cache area.
In another embodiment of the present application, the target data and metadata writing module further includes:
the target data storage area determining sub-module is used for determining a target data storage area in the data storage area by adopting a preset load balancing algorithm;
and the target data storage sub-module is used for storing the target data in the target data storage area.
In another embodiment of the present application, the low-speed storage medium writing module includes:
and the second storage medium metadata writing sub-module is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area when a preset transfer period is reached or the preset service storage amount of the first storage medium reaches an upper limit.
In another embodiment of the present application, the low-speed storage medium writing module further includes:
the second storage medium target data writing sub-module is used for sending the target data to the second storage medium when the metadata corresponding to the target data is successfully transferred to the second storage medium metadata area;
And the target data deleting sub-module is used for deleting the target data in the first storage medium.
In another embodiment of the present application, the metadata areas of the second storage medium in all the first storage media form a redundant array of independent hard disks.
Referring to FIG. 4, there is shown a block diagram of an embodiment of a tiered storage reader of the present application as applied to a tiered storage architecture object that includes a first storage medium and a second storage medium, the first storage medium including a second storage medium metadata area therein; the hierarchical storage reading device may include the following modules:
a data call request receiving module 401, configured to receive a data call request, where the data call request includes a data list to be called;
a second storage medium metadata searching module 402, configured to search, for the to-be-called data list, target metadata from the second storage medium metadata area, where the target metadata is metadata corresponding to-be-called data in the to-be-called data list;
a data to be called acquisition module 403, configured to acquire the data to be called from the second storage medium by using the target metadata when the target metadata exists in the metadata area of the second storage medium;
The data to be called returns the module 404, which is configured to return the data to be called.
In another embodiment of the present application, the first storage medium further includes a data storage area therein;
the second storage medium metadata search module includes:
the data storage area to-be-called data searching sub-module is used for searching target metadata from the data storage area aiming at the to-be-called data list;
and the second storage medium metadata searching sub-module is used for searching the target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
Referring to FIG. 5, a block diagram of an embodiment of a tiered storage system of the present application is shown, the system comprising:
a cache module 501 for caching target data;
a low-speed storage module 502, configured to store the target data sent by the first storage medium;
the high-speed storage module comprises a second storage medium metadata area and is used for storing metadata of a second storage medium.
In another embodiment of the present application, the metadata area of the second storage medium in all the high-speed storage modules forms a redundant array of independent hard disks for data redundancy protection.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, the present embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
In a typical configuration, the computer device includes one or more processors (CPUs), an input/output interface, a network interface, and memory. The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash memory (flashRAM). Memory is an example of computer-readable media. Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include non-transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present embodiments have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the present application.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The above description is provided in detail of a hierarchical storage writing method and device, and a reading method and device and system, and examples are applied to illustrate the principles and embodiments of the present application, and the above description of the examples is only used to help understand the method and core idea of the present application; meanwhile, as those skilled in the art will have variations in embodiments and application ranges based on the ideas of the present application, the present disclosure should not be construed as limiting the present application in view of the above.
Claims (18)
1. The method is applied to a layered storage architecture object, wherein the layered storage architecture object comprises a first storage medium and a second storage medium, the first storage medium comprises a second storage medium metadata area and a data storage area, and the first storage medium is provided with the same number of data storage areas as the second storage medium corresponding to the number of the second storage medium connected with the first storage medium; the method comprises the following steps:
acquiring target data;
generating metadata corresponding to the target data, and writing the target data and the metadata corresponding to the target data into the data storage area;
Writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium; and writing the target data in the data storage area into the second storage medium.
2. The hierarchical storage writing method of claim 1, wherein the data storage area includes a first storage medium metadata area and a target data cache area;
the step of writing the target data and the corresponding metadata thereof into the data storage area comprises the following steps:
writing metadata corresponding to the target data into the first storage medium metadata area;
and writing the target data into the target data cache area.
3. The hierarchical storage writing method according to claim 1 or 2, wherein the step of writing the target data and its corresponding metadata into the data storage area further comprises:
determining a target data storage area in the data storage areas by adopting a preset load balancing algorithm;
storing the target data in the target data storage area.
4. The layered writing method as claimed in claim 1, wherein the step of writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area comprises:
And when a preset transfer period is reached or the preset business storage amount of the first storage medium reaches the upper limit, writing metadata corresponding to the target data in the data storage area into the metadata area of the second storage medium.
5. The layered writing method as claimed in claim 1 or 4, wherein the step of writing the target data in the data storage area to the second storage medium comprises:
when all metadata corresponding to the target data are written into the metadata area of the second storage medium, writing the target data in the data storage area into the second storage medium;
and deleting the target data in the data storage area.
6. The layered writing method of claim 1, wherein,
and the metadata areas of the second storage media in all the first storage media form an independent hard disk redundant array.
7. The hierarchical storage reading method is characterized by being applied to a hierarchical storage architecture object, wherein the hierarchical storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area; wherein, the first storage medium is provided with data storage areas with the same number as the second storage medium corresponding to the number of the second storage medium connected with the first storage medium;
The method comprises the following steps:
receiving a data calling request, wherein the data calling request comprises a data list to be called;
searching target metadata from the second storage medium metadata area aiming at the data list to be called, wherein the target metadata are metadata corresponding to the data to be called in the data list to be called;
when the target metadata exists in the metadata area of the second storage medium, acquiring the data to be called from the second storage medium by adopting the target metadata;
and returning the data to be called.
8. The hierarchical storage read method of claim 7 wherein said first storage medium further comprises a data storage area therein;
the step of searching the target metadata from the second storage medium metadata area for the data list to be called includes:
searching target metadata from the data storage area aiming at the data list to be called;
and searching for target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
9. A hierarchical storage device, characterized in that the hierarchical storage device is applied to a hierarchical storage architecture object, the hierarchical storage architecture object comprises a first storage medium and a second storage medium, the first storage medium comprises a second storage medium metadata area and a data storage area, and the first storage medium is provided with the same number of data storage areas as the second storage medium corresponding to the number of the second storage medium connected with the first storage medium; the device comprises:
The target data acquisition module is used for acquiring target data;
the target data and metadata writing module is used for generating metadata corresponding to the target data and writing the target data and the metadata corresponding to the target data into the data storage area;
the second storage medium writing module is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area; and writing the target data in the data storage area into the second storage medium.
10. The tiered storage device of claim 9 wherein the data storage area includes a first storage medium metadata area and a target data cache area;
the target data and metadata writing module comprises:
the first storage medium metadata writing sub-module is used for writing metadata corresponding to the target data into the first storage medium metadata area;
and the target data writing sub-module is used for writing the target data into the target data cache area.
11. The tiered storage device of claim 9 or 10 wherein the target data and metadata writing module further comprises:
The target data storage area determining sub-module is used for determining a target data storage area in the data storage area by adopting a preset load balancing algorithm;
and the target data storage sub-module is used for storing the target data in the target data storage area.
12. The tiered storage device of claim 9 wherein the low speed storage medium write module includes:
and the second storage medium metadata writing sub-module is used for writing metadata corresponding to the target data in the data storage area into the second storage medium metadata area when a preset transfer period is reached or the preset service storage amount of the first storage medium reaches an upper limit.
13. The tiered storage device of claim 9 or 12 wherein the low speed storage medium write module further comprises:
the second storage medium target data writing sub-module is used for sending the target data to the second storage medium when the metadata corresponding to the target data is successfully transferred to the second storage medium metadata area;
and the target data deleting sub-module is used for deleting the target data in the first storage medium.
14. The tiered storage device of claim 9 wherein all of the second storage medium metadata areas in the first storage medium constitute a redundant array of independent hard disks.
15. A hierarchical storage reading device, characterized in that the hierarchical storage reading device is applied to a hierarchical storage architecture, wherein the hierarchical storage architecture object comprises a first storage medium and a second storage medium, and the first storage medium comprises a second storage medium metadata area; wherein, the first storage medium is provided with data storage areas with the same number as the second storage medium corresponding to the number of the second storage medium connected with the first storage medium; the device comprises the following modules:
the data calling request receiving module is used for receiving a data calling request, wherein the data calling request comprises a data list to be called;
the second storage medium metadata searching module is used for searching target metadata from the second storage medium metadata area aiming at the data list to be called, wherein the target metadata is metadata corresponding to the data to be called in the data list to be called;
the data to be called is obtained from the second storage medium by adopting the target metadata when the target metadata exists in the metadata area of the second storage medium;
and the data to be called return module is used for returning the data to be called.
16. The tiered storage read apparatus of claim 15 wherein said first storage medium further includes a data storage area therein;
the second storage medium metadata search module includes:
the data storage area to-be-called data searching sub-module is used for searching target metadata from the data storage area aiming at the to-be-called data list;
and the second storage medium metadata searching sub-module is used for searching the target metadata from the second storage medium metadata area when the target metadata does not exist in the data storage area.
17. A tiered storage system, comprising:
the high-speed storage module is used for caching target data;
the low-speed storage module is used for storing the target data sent by the first storage medium;
the high-speed storage module comprises a second storage medium metadata area and is used for storing metadata of a second storage medium; the first storage medium is provided with data storage areas corresponding to the number of the second storage mediums connected with the first storage medium, and the number of the data storage areas is the same as that of the second storage mediums.
18. The tiered storage system of claim 17 wherein the second storage medium metadata areas in all of the high-speed storage modules form a redundant array of independent hard disks for data redundancy protection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811521157.3A CN111309245B (en) | 2018-12-12 | 2018-12-12 | Hierarchical storage writing method and device, reading method and device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811521157.3A CN111309245B (en) | 2018-12-12 | 2018-12-12 | Hierarchical storage writing method and device, reading method and device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111309245A CN111309245A (en) | 2020-06-19 |
| CN111309245B true CN111309245B (en) | 2024-04-09 |
Family
ID=71149232
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811521157.3A Active CN111309245B (en) | 2018-12-12 | 2018-12-12 | Hierarchical storage writing method and device, reading method and device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111309245B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035065B (en) * | 2020-08-28 | 2022-06-07 | 北京浪潮数据技术有限公司 | Data writing method, device and equipment and computer readable storage medium |
| CN112269540A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Data storage method, system, equipment and computer readable storage medium |
| CN112463079B (en) * | 2020-12-17 | 2023-12-22 | 北京浪潮数据技术有限公司 | Data storage control method, device, equipment and readable storage medium |
| CN117574446B (en) * | 2024-01-16 | 2024-04-05 | 苏州元脑智能科技有限公司 | Data protection method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182353A (en) * | 2014-08-21 | 2014-12-03 | 段超 | Data storage system adopting different storage media |
| WO2016086649A1 (en) * | 2014-12-04 | 2016-06-09 | 中兴通讯股份有限公司 | Method, client, and storage medium for reading, writing, deleting, and querying a file |
| CN107704208A (en) * | 2017-10-13 | 2018-02-16 | 郑州云海信息技术有限公司 | A kind of restorative procedure of metadata, device and medium |
| CN108664218A (en) * | 2018-04-23 | 2018-10-16 | 浙江大华技术股份有限公司 | A kind of SSD storage methods based on data characteristics, apparatus and system |
-
2018
- 2018-12-12 CN CN201811521157.3A patent/CN111309245B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182353A (en) * | 2014-08-21 | 2014-12-03 | 段超 | Data storage system adopting different storage media |
| WO2016086649A1 (en) * | 2014-12-04 | 2016-06-09 | 中兴通讯股份有限公司 | Method, client, and storage medium for reading, writing, deleting, and querying a file |
| CN107704208A (en) * | 2017-10-13 | 2018-02-16 | 郑州云海信息技术有限公司 | A kind of restorative procedure of metadata, device and medium |
| CN108664218A (en) * | 2018-04-23 | 2018-10-16 | 浙江大华技术股份有限公司 | A kind of SSD storage methods based on data characteristics, apparatus and system |
Non-Patent Citations (3)
| Title |
|---|
| Xiao Liu.DMS: A Dynamic Multi-tiered Storage with deduplication based on variable-sized chunks.IEEE.2018,全文. * |
| 于耳 ; .分层存储:一种经济的存储方式.中国教育网络.2012,(第11期),全文. * |
| 分层存储:一种经济的存储方式;于耳;;中国教育网络;20121105(第11期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111309245A (en) | 2020-06-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210056074A1 (en) | File System Data Access Method and File System | |
| CN111309245B (en) | Hierarchical storage writing method and device, reading method and device and system | |
| US9916176B2 (en) | Method and apparatus of accessing data of virtual machine | |
| US10671290B2 (en) | Control of storage of data in a hybrid storage system | |
| CN106407040B (en) | A kind of duplicating remote data method and system | |
| US10289315B2 (en) | Managing I/O operations of large data objects in a cache memory device by dividing into chunks | |
| US11307776B2 (en) | Method for accessing distributed storage system, related apparatus, and related system | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| CN106951375B (en) | Method and device for deleting snapshot volume in storage system | |
| CN103152395A (en) | Storage method and device of distributed file system | |
| CN111400083B (en) | Data storage method and system and storage medium | |
| US10628298B1 (en) | Resumable garbage collection | |
| CN115794669A (en) | Method, device and related equipment for expanding memory | |
| CN111444114B (en) | Method, device and system for processing data in nonvolatile memory | |
| CN110895545B (en) | Shared data synchronization method and device | |
| CN107102898B (en) | Memory management and data structure construction method and device based on NUMA (non Uniform memory Access) architecture | |
| CN105068896B (en) | Data processing method and device based on RAID backup | |
| CN109508140B (en) | Storage resource management method and device, electronic equipment and system | |
| CN114328018B (en) | Snapshot creation method, computing device and storage medium | |
| CN107422990B (en) | Dynamic migration method and device | |
| CN114428681A (en) | Method and device for expanding computing capacity of database system | |
| US20160371192A1 (en) | Apparatus and method for performing cache management in a storage system | |
| US11645333B1 (en) | Garbage collection integrated with physical file verification | |
| US10606497B1 (en) | Systems and methods for managing replication path direction in storage networks | |
| CN111352590A (en) | File storage method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |