WO2023246240A1 - 一种数据重构方法及装置 - Google Patents

一种数据重构方法及装置 Download PDF

Info

Publication number
WO2023246240A1
WO2023246240A1 PCT/CN2023/087558 CN2023087558W WO2023246240A1 WO 2023246240 A1 WO2023246240 A1 WO 2023246240A1 CN 2023087558 W CN2023087558 W CN 2023087558W WO 2023246240 A1 WO2023246240 A1 WO 2023246240A1
Authority
WO
WIPO (PCT)
Prior art keywords
block
data
blocks
damaged
copy
Prior art date
Application number
PCT/CN2023/087558
Other languages
English (en)
French (fr)
Inventor
蒲贵友
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023246240A1 publication Critical patent/WO2023246240A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0689—Disk arrays, e.g. RAID, JBOD
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks

Definitions

  • the present invention relates to the field of data storage technology, and in particular to a data reconstruction method and device.
  • Redundant array of independent disks is a basic technology widely used in the storage field to improve the reliability of storage media. It is widely used in both storage arrays and solid state drives. SSD), one of the basic technologies of storage.
  • the basic principle of RAID is that by using additional parity block storage, when a data block fails, the undamaged data block and parity block are used to calculate the data block of the damaged column through the RAID algorithm, thereby realizing the damaged data block. repair and write the recovered data blocks into free storage space.
  • the data blocks D 0 , D 1 ...D n , and the corresponding check block is P.
  • Data blocks D 0 , D 1 ...D n and parity block P constitute a RAID data protection group.
  • D 1 can be calculated based on D 0 ,...D n and parity block P through the RAID algorithm, and the calculated D 1 is stored in the hot spare storage space.
  • the number of data blocks is N and the number of parity blocks is M.
  • N the number of data blocks
  • M the number of parity blocks
  • the number of data blocks corresponding to RAID in the storage array has reached 20, or even more than 100.
  • the number of data blocks corresponding to RAID in an SSD disk usually reaches 60 or even exceeds 128.
  • Embodiments of the present invention provide a data reconstruction method and device, which can reduce the size of data read and processed for data reconstruction in a data protection group (such as a RAID group), save computing resources, and improve the efficiency of data reconstruction. .
  • a data protection group such as a RAID group
  • embodiments of the present invention provide a data reconstruction method, which includes obtaining a copy of a damaged block in a data protection group; restoring the damaged block in the data protection group based on the copy of the damaged block; wherein, in the data protection group
  • the blocks are protected using Erasure Coding (EC) algorithm or Redundant Array of Independent Disks RAID.
  • EC Erasure Coding
  • the data reconstruction method provided by the embodiment of the present invention reconstructs damaged blocks in the data protection group through copies.
  • the data reconstruction method is Reconstruction requires reading N times the data.
  • the embodiment of the present invention only needs to read 1 times the amount of data for processing through copy reconstruction, which saves computing resources and improves the efficiency of data reconstruction.
  • the damaged block mentioned above is a block generated by a garbage collection (GC) operation to move valid data;
  • the copy of the damaged block is a block generated after the garbage collection operation completes the movement of valid data. Mark the blocks stored in the storage location where the damaged blocks are located before moving as invalid blocks.
  • GC garbage collection
  • the invalid data (which can also be called garbage data) generated by the GC operation is used as a copy of the block stored in the storage location where the damaged block was located before the move, and the data of the damaged block is reconstructed.
  • Most storage systems mostly use append writing to write data.
  • GC will exist to organize space. There will be a large amount of garbage data remaining due to GC operations in the storage system. Therefore, The garbage data generated by GC can be used as a copy of the damaged block to reconstruct the data of the damaged block in the data protection group.
  • the storage system will store reverse description information while storing data.
  • This reverse description information can be used to obtain the copy. For example, obtain the reverse description information of a damaged block.
  • the reverse description information includes the logical block address (LBA) information of the damaged block; read the reverse description information of the undamaged blocks in the data protection group to obtain the logical block address information corresponding to each data block in the data protection group. ; Based on the logical block address information, match the damaged block with the undamaged block; use the successfully matched data block as a copy of the damaged block.
  • LBA logical block address
  • the reverse description information also includes verification information of the data block, which is used to verify the correctness of the data block; the matching result of the damaged block and the undamaged block is also matched with the verification information.
  • verification information of the data block which is used to verify the correctness of the data block
  • the matching result of the damaged block and the undamaged block is also matched with the verification information.
  • the logical block address and check information of the damaged block are the same as the logical block address and check information of the undamaged block, it can be determined that the two data blocks successfully matched, and then the undamaged block can be determined. Can be used as a copy of the damaged block to ensure the correctness of the data block.
  • the copy can come from many sources.
  • the copy can be the garbage data left over from the GC operation, or the data generated by prefetching, such as the storage system cache data and the copy brought by the system hierarchical storage move.
  • prefetching or caching is a copy of the data that is read in advance according to certain rules and placed in some caches.
  • Hierarchical storage is similar, and the data is stored on different speed media based on whether the data is hot or cold. Even if the deletion is not timely, there will be duplicates.
  • the data reconstruction method provided by the embodiment of the present invention also includes: obtaining the probability of damage of each block in the storage system; configuring the garbage collection operation to move the valid data blocks that perform the garbage operation to the location where the failure occurs.
  • the storage address has the highest probability or the probability of failure is greater than the preset threshold. That is to say, adjust the GC algorithm to write the data moved by GC to a more vulnerable storage address. In this way, when a block is damaged, a large number of GC residual copies can be used for reconstruction, which saves computing resources and improves data duplication. structural efficiency.
  • the data reconstruction method provided by the embodiment of the present invention also includes: when there is no copy of the damaged block, reconstructing the damaged block according to the traditional RAID reconstruction method.
  • the blocks in the data protection group are blocks in the same hard disk, that is, the data protection group is implemented in the same hard disk.
  • EC or RAID protection is used in the SSD disk to protect the data.
  • the blocks in the data protection group are blocks in the storage system.
  • the data protection group is not limited to being implemented in the same hard disk.
  • a storage system such as a storage array or a distributed storage system, uses EC or RAID protection to protect data.
  • the data reconstruction method provided by the embodiment of the present invention can be applied to SSD, and RAID or EC technology is applied in the SSD to provide a data protection group inside the SSD.
  • the blocks in the data protection group are distributed in different channels of the SSD (Channel ) or on physical blocks (Blocks) on different flash memory particles (DIE).
  • embodiments of the present invention also provide a data reconstruction device, including an acquisition module and a reconstruction module, wherein the acquisition module is used to obtain a copy of a damaged block in the data protection group; the reconstruction module is used to based on the damaged block The copy recovers the corrupted blocks in the data protection group; where the blocks in the data protection group are protected using erasure coding EC algorithm or redundant array of independent disks RAID.
  • the damaged block mentioned above is a block generated by a garbage collection (GC) operation to move valid data;
  • the copy of the damaged block is a block generated after the garbage collection operation completes the movement of valid data. , mark the block stored in the storage location where the damaged block is located as an invalid block.
  • GC garbage collection
  • the copy of the corrupted block is a copy of the corrupted block in a cache or a block generated by a hierarchical storage operation of the corrupted block.
  • the blocks in the data protection group are blocks on the same hard disk.
  • the blocks in the data protection group are blocks in the storage system.
  • the storage system writes data using append writing.
  • the data reconstruction apparatus provided by the embodiment of the present invention further includes a verification module, which is used to verify copies of damaged blocks.
  • the acquisition module is specifically configured to: obtain the reverse description information of the damaged block; the reverse description information includes the logical address of the damaged block; and determine a copy of the damaged block based on the logical address.
  • the present invention provides a data processing device, including an interface and a processor.
  • the interface is communicatively connected to the processor, and the processor is used to implement the method provided in the first aspect of the present invention.
  • the present invention provides a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed in a computer, the computer is caused to execute the method provided in the first aspect of the present invention.
  • the present invention provides a computer program or computer program product.
  • the computer program or computer program product includes instructions. When the instructions are executed, the method provided by the first aspect of the present invention is implemented.
  • Figure 1 is a schematic diagram of the traditional RAID reconstruction method
  • Figure 2 shows a schematic structural diagram of a storage system
  • Figure 3 shows a schematic structural diagram of an SSD
  • Figure 4 shows a schematic diagram of data storage in SSD
  • Figure 5 is a schematic diagram of data reconstruction according to an embodiment of the present invention.
  • Figure 6 is a schematic structural diagram of a data reconstruction device provided by an embodiment of the present invention.
  • Figure 7 is a schematic structural diagram of a data processing device provided by an embodiment of the present invention.
  • RAID or Erasure Coding (EC) technology In order to achieve the reliability of stored data, storage systems usually use RAID or Erasure Coding (EC) technology to protect data.
  • EC Erasure Coding
  • the data is divided into blocks according to the RAID or EC algorithm, and each Data blocks are stored on different storage media blocks, and parity blocks are calculated for these data blocks to effectively protect these data blocks.
  • the data blocks and parity blocks form a RAID group or EC group, collectively referred to as data protection. Groups, data blocks and parity blocks are collectively called blocks. This ensures that when some storage media blocks fail or data blocks are damaged due to other reasons, the damaged blocks can be obtained by calculating the undamaged blocks in the data protection group based on the RAID or EC algorithm.
  • embodiments of the present invention provide a data reconstruction method, which can be especially applied to storage systems or hard disks.
  • the data protection group in the embodiment of the present invention can be based on RAID technology, such as RAID5, RAID6, etc.
  • the data protection group in the embodiment of the present invention may also be based on EC technology.
  • the embodiment of the present invention takes the data protection group as a RAID group as an example and applies it to a storage system as an example for description.
  • the storage system in the embodiment of the present invention may be a storage array or distributed storage, which is not limited in the embodiment of the present invention.
  • the damaged data block when a data block in a data protection group, that is, a RAID group, is damaged, the damaged data block is calculated using other undamaged blocks in the RAID group without following the traditional RAID reconstruction method. Instead, it is beneficial to copy the damaged data blocks existing in the storage system. Reading the copy enables data reconstruction, which reduces the size of data read and processed by RAID reconstruction, saves computing resources, and improves reconstruction efficiency.
  • FIG. 2 shows a schematic structural diagram of a storage system.
  • the RAID storage system includes a processor and an interface.
  • the interface can be a physical interface card, and the processor communicates with the interface.
  • the processor may be a central processing unit (CPU) or other hardware processing device, such as a RAID card, a data processing unit (DPU), etc.
  • the storage system contains one or more RAID groups, and each RAID group contains multiple blocks, such as N data blocks and M parity blocks of the N data blocks.
  • the storage system obtains a copy of data block A.
  • the copy of data block A may be that when the storage system performs a garbage collection operation, data block A is moved to a new storage location as valid data. After the garbage collection operation completes the move of data block A, the data before the move is Data block A stored in the storage location where block A is located is marked as invalid, that is, the block before data block A is moved. Since data is not erased or overwritten immediately after it is marked as invalid data during the garbage collection operation, the corresponding block before data block A is moved is used as a copy of data block A.
  • the copy of data block A may also be data block A in the cache.
  • the copy of data block A can also be a block generated by data block A during a hierarchical storage operation. For example, after data block A is migrated from the first performance storage layer to the second performance storage layer, data block A occurs. If it is damaged, data block A in the first performance storage layer will be used as a copy of data block A before being deleted.
  • the RAID group mentioned in the embodiment of the present invention can also be implemented in a hard disk, such as an SSD.
  • a hard disk such as an SSD.
  • the specific implementation of reconstructing the damaged blocks when a block in the RAID group is damaged is introduced below.
  • FIG 3 shows a schematic structural diagram of an SSD.
  • the SSD includes a controller and multiple physical blocks, such as physical block 1, physical block 2, physical block 3...physical block N in Figure 3.
  • the controller communicates with multiple physical blocks and controls operations such as data reading and writing of multiple physical blocks.
  • the controller uses RAID technology to form multiple physical blocks into a RAID group, that is, a data protection group.
  • Each RAID group includes multiple blocks, such as N data blocks and M parity blocks corresponding to the N data blocks.
  • the SSD gets a copy of data block A.
  • the copy of data block A can be that when the SSD performs a garbage collection operation, data block A is moved to a new storage location as valid data. After the garbage collection operation completes the movement of data block A, the previous storage location is moved.
  • the stored data block A is marked as an invalid block, that is, the block before data block A is moved. Since data is not erased immediately after it is marked as invalid during the garbage collection operation, the corresponding block before data block A is moved is used as a copy of data block A.
  • the copy of data block A can also be data block A in the cache, it can be a block in the SSD cache, or it can be data block A cached in other locations in the storage system.
  • the copy of data block A can also be a block generated by data block A during a hierarchical storage operation. For example, after data block A is migrated from the first performance storage layer to the second performance storage layer, data block A occurs. If it is damaged, data block A in the first performance storage layer will be used as a copy of data block A before being deleted.
  • Copies from different sources can be obtained through different methods. For example, for a copy from a cache or hierarchical storage source, you can pass the metadata of the corrupted block and then obtain a copy of the corrupted block based on the information in that metadata.
  • the metadata of the data block records the physical address information of the data block and the cache/hierarchical storage address information of the data block.
  • the controller reads a copy of the data block based on the cache/hierarchical storage address information of the data block.
  • the SSD Since data is written internally using the append write method, garbage collection operations are required to organize the data inside the SSD.
  • the SSD generally has about 15% of the over-provisioned space inside.
  • the SSD is generally not completely full, and most of the written data is about 80%. Therefore, there is a large amount of data in the SSD disk in the form of unerased garbage.
  • the general write amplification in SSD disks is around 3, which is mainly caused by garbage collection in the disk, which also reflects the storage of more data copies. Therefore, the garbage data blocks generated by garbage collection can be used as copies of the damaged blocks to reconstruct the damaged blocks.
  • the SSD performs a garbage collection operation, and after moving the valid data blocks in the physical blocks to new storage locations, the unerased data blocks remaining in the physical blocks are called garbage data blocks.
  • garbage data blocks When the valid data blocks moved by the SSD during garbage collection operations are damaged, the garbage data blocks generated by the garbage operation can be used as copies of the damaged valid data blocks.
  • the SSD disk will store reverse description information when storing data to support its own judgment on data movement and correctness.
  • the reverse description information includes the reverse mapping information and verification information of each data block in the physical block. Using this information, you can judge whether the garbage data block corresponds to a certain block of data and whether the data is correct, thereby ensuring the integrity of the data. Correctness.
  • the reverse mapping information includes the mapping between the data physical address and the logical block address (Logical Block Address, LBA). Through the reverse mapping information, the LBA corresponding to the data stored at the physical address can be identified.
  • LBA Logical Block Address
  • Figure 4 shows a schematic diagram of data storage in an SSD disk.
  • the SSD disk includes multiple physical blocks, such as physical block 1, physical block 2, physical block 3, physical block 4 and physical block 5. Multiple physical blocks form a RAID group, where physical block 1, physical block 5 Data blocks are stored in block 2, physical block 3, and physical block 4, and check blocks are stored in physical block 5.
  • data block A in physical block 6 is moved to physical block 3, and the SSD marks data block A in physical block 6 as invalid.
  • the data blocks marked as invalid are not immediately erased, so the data blocks in physical block 6 have a copy of the data blocks in physical block 3.
  • Figure 5 is a schematic diagram of data reconstruction using data blocks marked as invalid.
  • data block A in physical block 3 when data block A in physical block 3 is damaged, data block A in physical block 6 is used to restore data block A in physical block 3.
  • data block A marked as invalid in physical block 6 is obtained as a copy of data block A stored in physical block 3, and the physical block is modified based on this copy.
  • the damaged blocks in 3 are reconstructed, thereby avoiding the use of the RAID algorithm for reconstruction calculations, saving computing resources, and improving data reconstruction efficiency.
  • the reverse description information of the data block can be obtained. If the reverse description information is lost, because the data volume of the reverse description information is small, the RAID algorithm can be used to restore the reverse description information, and the LBA of the damaged data block can be determined based on the reverse description information.
  • Read the reverse description information of other physical blocks obtain the LBA of each data block stored in other physical blocks, and match it with the LBA corresponding to the damaged data block.
  • determine the two data Block matching records the location of the successfully matched data block, including the location of the successfully matched damaged data block and the location of the undamaged data block, and reads a copy of the damaged data block based on the location information.
  • the reverse description information also includes check information of each data block stored in the physical block, such as cyclic redundancy check (cyclic redundancy check, CRC).
  • CRC cyclic redundancy check
  • the process of obtaining a copy of the data block includes reading the LBA and CRC of each damaged data block in the physical block, establishing this information in a table or other organizational form, and storing it with the LBA as an index to facilitate subsequent matching. Comparison.
  • Read the reverse description information of other physical blocks obtain the LBA and CRC of each undamaged data block, and match it with the index information corresponding to the damaged data block.
  • LBA and CRC of the two data blocks are the same, determine Two data blocks match, and the location of the successfully matched data block is recorded, including the location of the damaged data block and the location of the undamaged data block.
  • a copy of the damaged data block is obtained, as well as the location of the data block.
  • CRC According to the CRC, the correctness of the data block is verified. If the verification passes, the data block is used as a copy to restore the damaged data block.
  • the data reconstruction method provided by the embodiment of the present invention also includes: obtaining the probability of failure of each physical block; configuring the garbage collection operation to perform the garbage operation on the data Blocks are moved to physical blocks with the highest probability of failure or with a probability of failure greater than a preset threshold.
  • the probability of failure of each medium in the SSD disk is predicted based on self-monitoring analysis and reporting technology (SMART), and the probability of failure is greater than the preset threshold or the physical medium with the highest probability of failure is block as the target physical block, adjust the garbage collection algorithm of the SSD, and write the data blocks moved by garbage collection into the target physical block.
  • SMART self-monitoring analysis and reporting technology
  • the probability is that it is in the target physical block.
  • the data blocks in the physical block need to be reconstructed, a large number of invalid data blocks remaining from garbage collection can be used as copies, further improving the efficiency of data reconstruction.
  • the reconstruction method of damaged blocks is similar to the reconstruction method of damaged blocks in the data protection group protected by RAID. In the same way, the corresponding copy of the damaged block is first obtained and used This copy recovers corrupted blocks which will not be described here.
  • the embodiment of the present invention also provides a data reconstruction device 600.
  • the data reconstruction device 600 includes a method for implementing the above-mentioned data reconstruction method. units or modules for each step in the process.
  • Figure 6 is a schematic structural diagram of a data reconstruction device provided by an embodiment of the present invention.
  • the data reconstruction device 600 at least includes an acquisition module 601 and a reconstruction module 602.
  • the acquisition module 601 is used to acquire a copy of a damaged block in the data protection group; the reconstruction module 602 is used to based on the damaged block.
  • the copy recovers the corrupted blocks in the data protection group; where the blocks in the data protection group are protected using erasure coding EC algorithm or redundant array of independent disks RAID.
  • the damaged block mentioned above is a block generated by the GC operation to move valid data; the copy of the damaged block is the damaged block before the garbage collection operation completes the movement of valid data.
  • the storage location where the block is stored is marked as an invalid block.
  • the copy of the corrupted block is a copy of the corrupted block in a cache or a block generated by a hierarchical storage operation of the corrupted block.
  • the blocks in the data protection group are blocks on the same hard disk.
  • the blocks in the data protection group are blocks in the storage system.
  • the storage system writes data using append writing.
  • the data reconstruction apparatus provided by the embodiment of the present invention further includes a verification module, which is used to verify copies of damaged blocks.
  • the acquisition module is specifically configured to: obtain the reverse description information of the damaged block; the reverse description information includes the logical address of the damaged block; and determine a copy of the damaged block based on the logical address.
  • the data reconstruction device 600 provided by the embodiment of the present invention also includes a configuration module 604, which configures the garbage collection operation to move the data blocks performing the garbage operation to the highest probability of failure or failure.
  • the probability is greater than the preset threshold.
  • the GC algorithm is adjusted to write the data moved by GC into more vulnerable blocks. In this way, when the block is damaged, a large number of GC residual copies can be reconstructed, which saves computing resources and improves the efficiency of data reconstruction. efficiency.
  • the data reconstruction device 600 may correspond to performing the method described in the embodiment of the present invention, and the above and other operations and/or functions of each module in the data reconstruction device 600 are respectively to achieve the above mentioned
  • For the data reconstruction method please refer to the above description for specific implementation. For the sake of brevity, we will not go into details here.
  • the data reconstruction device provided in the embodiment of Figure 6 can be executed by the processor of the storage system shown in Figure 2 .
  • the data reconstruction device may be executed by a controller of the hard disk, where the hard disk includes a controller and a storage medium, and the controller communicates with the storage medium.
  • the storage system shown in Figure 2 may be a storage array or a distributed storage system, which is not limited in this embodiment of the present invention.
  • embodiments of the present invention also provide a data processing device.
  • the electronic device at least includes a processor and an interface.
  • the processor executes a program and can realize the data reprocessing mentioned above. construction method.
  • Figure 7 is a schematic structural diagram of a data processing device provided by an embodiment of the present invention.
  • the data processing device 700 includes at least one processor 701 and an interface 702 .
  • the processor 701 and the interface 702 are connected through communication, and the communication connection can be realized through a bus.
  • the interface 702 is used to receive data sent by other devices (such as IO data sent by upper-layer applications); the processor 701 executes the foregoing method embodiments. Methods.
  • the processor 701 can be a central processing unit CPU, and the processor 701 can also be other general-purpose processors, digital signal processors (digital signal processors, DSPs), application specific integrated circuits (application specific integrated circuits). specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor can be a microprocessor or any conventional processor, etc.
  • the data processing device shown in Figure 7 can be a hard disk, a storage system, a server, or a device provided by a cloud service.
  • the storage system can be a storage array or a server.
  • the data device 700 can perform the method mentioned in the embodiment of the present invention.
  • the detailed description of the implementation of the method please refer to the above.
  • the details will not be described again.
  • Embodiments of the present invention provide a computer-readable storage medium on which a computer program is stored. When the computer instructions are executed by a processor, the above-mentioned method is implemented.
  • An embodiment of the present invention provides a chip, which includes at least one processor and an interface.
  • the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to Implement the methods mentioned above.
  • Embodiments of the present invention provide a computer program or computer program product, which includes instructions that, when executed, cause the computer to perform the above-mentioned method.
  • RAM random access memory
  • ROM read-only memory
  • electrically programmable ROM electrically erasable programmable ROM
  • registers hard disks, removable disks, CD-ROMs, or anywhere in the field of technology. any other known form of storage media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本发明提供一种数据重构方法，包括获取数据保护组中损坏的块的副本；基于损坏的块的副本恢复数据保护组中损坏的块；其中，数据保护组中的块使用纠删码EC算法或独立磁盘冗余阵列RAID保护。本发明提供的数据重构方法，通过副本对数据保护组中损坏的坏进行重构(例如垃圾回收操作残留的垃圾数据)，节省了计算资源，提高了数据重构的效率。

Description

一种数据重构方法及装置 技术领域
本发明涉及数据存储技术领域，尤其涉及一种数据重构方法及装置。
背景技术
独立冗余磁盘阵列(redundant array of independent disks，RAID)是存储领域里边广泛使用的提升存储介质可靠性能力的基础技术，既被广泛用于存储阵列，也被用于固态硬盘(solid state drive，SSD)内，属于存储的基础技术之一。
RAID的基础原理为，通过使用额外的校验块存放，在数据块发生故障时，利用未损坏的数据块和校验块，通过RAID算法计算出损坏列的数据块，从而实现损坏的数据块的修复，并将恢复出来的数据块写入空闲的存储空间。如图1所示，数据块D0,D1…Dn,对应的校验块为P。数据块D0,D1…Dn和校验块P构成一个RAID数据保护组。当数据块,D1损坏，可以通过RAID算法根据D0,…Dn和校验块P计算出D1，并将计算出来的D1存储到热备存储空间。
基于RAID的基本原理，数据块的数量为N，校验块的数量为M，要重构一个损坏的数据块，至少需要从RAID中读取N个块，才能计算出损坏的数据块。
现在存储阵列中RAID对应的数据块的数量达到20，甚至已经超过100。而SSD盘内的RAID对应的数据块的数量也通常达到60，甚至超过128。
现有技术，使用RAID恢复损坏的数据块，需要涉及大量的数据计算，从而消耗了大量的计算资源，同时数据恢复效率低。
发明内容
本发明的实施例提供一种数据重构方法及装置，可以降低数据保护组(例如RAID组)中数据重构所读取和处理的数据规模，节省了计算资源，提高了数据重构的效率。
第一方面，本发明实施例提供了一种数据重构方法，包括获取数据保护组中损坏的块的副本；基于损坏的块的副本恢复数据保护组中损坏的块；其中，数据保护组中的块使用纠删码(Erasure Coding,EC)算法或独立磁盘冗余阵列RAID保护。
本发明实施例提供的数据重构方法，通过副本对数据保护组中损坏的块进行重构，相比于现有的数据保护组(例如背景技术中提及的RAID保护)中损坏的块的重构要读取N倍的数据，本发明实施例通过副本重构只需读取1倍数据量处理，节省了计算资源，提高了数据重构的效率。
在一个可能的实现中，上述提及的损坏的块为垃圾回收(garbage collection，GC)操作搬移有效数据生成的块；所述损坏的块的副本为所述垃圾回收操作完成有效数据搬移后，将搬移前所述损坏的块所在的存储位置存储的块标记为无效的块。
也就是说，利用GC操作产生的无效数据(也可称之为垃圾数据)，即将搬移前损坏的块所在的存储位置存储的块作为副本，对损坏的块进行数据重构。而大多数存储系统多采用追加写的方式写入数据，在采用追加写的存储系统中均会存在GC以实现空间整理，存储系统中会存在大量因GC操作而残留存在的垃圾数据，因此，可以将GC产生的垃圾数据作为损坏的块的副本对数据保护组中损坏的块进行数据重建。
为了支持垃圾回收GC的有效运行，存储系统会在存储数据的同时，存放反向描述信息，可以利用该反向描述信息实现对副本的获取。例如获取损坏的块的反向描述信息，该反向描 述信息包括损坏块的逻辑区块地址(logical block address，LBA)信息；读取数据保护组中未损坏的块的反向描述信息，得到数据保护组中各个数据块对应的逻辑区块地址信息；基于逻辑区块地址信息，将损坏的块与未损坏的块进行匹配；将匹配成功的数据块作为损坏的块的副本。
在一个可能的实现中，反向描述信息还包括数据块的校验信息，该校验信息用于校验数据块的正确性；损坏的块与未损坏的块的匹配结果还与校验信息相关。例如，当损坏的块的逻辑区块地址和校验信息与未损坏的块的逻辑区块地址和校验信息都相同的情况下，才能确定两个数据块匹配成功，进而确定未损坏的块可以作为损坏的块的副本，以保证数据块的正确性。
副本的来源可以有多种，可选的，副本可以是GC操作残留的垃圾数据、还有可能是预取等产生的数据，例如存储系统缓存数据、系统分级存储搬移带来的副本。例如，预取或缓存是将数据按照一定规则预先读取出来，放在某些高速缓存上产生的副本，而分级存储也类似，会根据数据冷热进行数据存放在不同的速度介质上，此时删除也会不及时，进而有副本的存在。
在另一个可能的实现中，本发明实施例提供的数据重构方法还包括：获取存储系统中各个块损坏的概率；将垃圾回收操作配置为，将执行垃圾操作的有效数据块搬移至发生故障的概率最高或发生故障的概率大于预设阈值的存储地址中。也就是说，调整GC算法，将GC搬移的数据写入更易损坏的存储地址中，这样当块损坏时，可以有大量的GC残留的副本用于重构，节省了计算资源，提高了数据重构的效率。
在另一个可能的实现中，本发明实施例提供的数据重构方法还包括：当损坏的块不存在副本时，则根据传统的RAID重构方法，对损坏的块进行重构。
在一个示例中，数据保护组中的块为同一块硬盘中的块，也就是说，数据保护组在同一硬盘中实现，例如，SSD盘内采用EC或RAID保护实现对数据的保护。
在另一个示例中，数据保护组中的块为存储系统中的块。换言之，数据保护组不限于在同一硬盘中实现，例如，存储系统，例如存储阵列或分布式存储系统，采用EC或RAID保护实现对数据的保护。
在一个示例中，本发明实施例提供的数据重构方法可以应用于SSD，并且SSD内应用了RAID或EC技术在SSD内部提供数据保护组，数据保护组中的块分布在SSD不同通道(Channel)或不同闪存颗粒(DIE)上的物理块(Block)上。
第二方面，本发明实施例还提供一种数据重构装置，包括获取模块和重构模块，其中获取模块用于获取数据保护组中损坏的块的副本；重构模块用于基于损坏的块的副本恢复数据保护组中损坏的块；其中，数据保护组中的块使用纠删码EC算法或独立磁盘冗余阵列RAID保护。
在另一个可能的实现中，上述提及的损坏的块为垃圾回收(garbage collection，GC)操作搬移有效数据生成的块；所述损坏的块的副本为所述垃圾回收操作完成有效数据搬移后，将搬移前所述损坏的块所在的存储位置存储的块标记为无效的块。
在另一个可能的实现中，损坏的块的副本为损坏的块在缓存中的副本或损坏的块在分级存储操作产生的块。
在另一个可能的实现中，数据保护组中的块为同一块硬盘中的块。
在另一个可能的实现中，数据保护组中的块为存储系统中的块。
在另一个可能的实现中，存储系统采用追加写的方式写入数据。
在另一个可能的实现中，本发明实施例提供的数据重构装置还包括校验模块，该校验模块用于对损坏的块的副本进行校验。
在另一个可能的实现中，获取模块具体用于：获取损坏的块的反向描述信息；反向描述信息包含损坏的块的逻辑地址；根据逻辑地址确定损坏的块的副本。
第三方面，本发明提供了一种数据处理设备，包括接口和处理器，所述接口与处理器通信连接，所述处理器用于实现本发明第一方面提供的方法。
第四方面，本发明提供了一种计算机可读存储介质，其上存储有计算机程序，当所述计算机程序在计算机中执行时，令计算机执行本发明第一方面提供的方法。
第五方面，本发明提供了一种计算机程序或计算机程序产品，所述计算机程序或计算机程序产品包括指令，当所述指令被执行时，实现本发明第一方面提供的方法。
附图说明
图1传统RAID重构的方法的示意图；
图2示出了一种存储系统的结构示意图；
图3示出了一种SSD的结构示意图；
图4示出了SSD中一种数据存放示意图；
图5为本发明实施例一种数据重构示意图；
图6为本发明实施例提供的数据重构装置的结构示意图；
图7为本发明实施例提供的一种数据处理设备的结构示意图。
具体实施方式
下面通过附图和实施例，对本发明的技术方案做进一步的详细描述。
为了实现存储数据的可靠性，存储系统通常会使用RAID或纠删码(Erasure Coding,EC)技术对数据进行保护，例如，在数据存储时，按照RAID或EC算法对数据进行分块，将每个数据块存储到不同的存储介质块上，并对这些数据块计算得到校验块，从而对这些数据块进行有效保护，数据块和校验块组成一个RAID组或EC组，统称为数据保护组，数据块和校验块统称为块。这样可以保证部分存储介质块出现故障或其他原因导致数据块损坏时，基于RAID或EC算法可以通过数据保护组中未损坏的块进行运算得到损坏的块。
针对此问题，本发明实施例提供一种数据重构方法，尤其可以应用于存储系统或硬盘，本发明实施例中的数据保护组可以基于RAID技术，例如RAID5、RAID6等。本发明实施例的数据保护组还可以基于EC技术。本发明实施例以数据保护组为RAID组为例，应用于存储系统为例进行说明。本发明实施例中的存储系统可以是存储阵列，也可以是分布式存储，本发明实施例对此不作限定。本发明实施例中，当一个数据保护组，即一个RAID组中数据块损坏时，不按照传统的RAID重构的方式，利用RAID组中其他未损坏的块计算出损坏的数据块。而是利于存储系统中存在的损坏的数据块的副本，读取该副本实现数据重构，降低了RAID重构所读取和处理的数据规模，节省了计算资源，提高了重构效率。
图2示出了一种存储系统的结构示意图。如图2所示，该RAID存储系统包括处理器和接口。接口可以是物理接口卡，处理器与接口通信。处理器可以是中央处理器(Central Processing Unit,CPU)，或者其他硬件处理能力的器件，例如RAID卡、数据处理单元(Data Processing Unit,DPU)等。存储系统中包含一个或多个RAID组，每个RAID组中包含多个块，例如N个数据块以及N个数据块的M个校验块。
当RAID组中有损坏的块时，例如数据块A发生损坏，存储系统获取数据块A的副本。在本发明实施例中，数据块A的副本可以是存储系统进行垃圾回收操作时，将数据块A作为有效数据搬移到新的存储位置，垃圾回收操作完成数据块A搬移后，将搬移前数据块A所在的存储位置存储的数据块A标记为无效，即数据块A搬移前的块。由于垃圾回收操作中，数据被标记为无效数据后，并不是立即进行擦除或覆盖，所以数据块A搬移前对应的块作为数据块A的副本。本发明另一实现方式中，数据块A的副本还可以为缓存中数据块A。本发明另一实施方式中，数据块A的副本还可以为数据块A在分级存储操作产生的块，例如数据块A从第一性能存储层迁移到第二性能存储层后，数据块A发生损坏，则在第一性能存储层的数据块A在没有被删除前，作为数据块A的副本。
本发明实施例提及的RAID组还可以在硬盘中实现，例如SSD中，下面介绍RAID组在SSD中实现时，RAID组中有块损坏时，对损坏的块进行重构的具体实现。
图3示出了一种SSD的结构示意图。如图3所示，SSD中包括控制器和多个物理块，多个物理块例如图3中的物理块1、物理块2、物理块3…物理块N。控制器与多个物理块通信连接，控制多个物理块的数据读写等操作。控制器采用RAID技术将多个物理块组成RAID组，即数据保护组，每个RAID组包括多个块，例如N个数据块以及N个数据块对应的M个校验块。
当RAID组中有损坏的块时，例如数据块A发生损坏，SSD获取数据块A的副本。在本发明实施例中，数据块A的副本可以是SSD进行垃圾回收操作时，将数据块A作为有效数据搬移到新的存储位置，垃圾回收操作完成数据块A搬移后，将搬移前存储位置存储的数据块A标记为无效的块，即数据块A搬移前的块。由于垃圾回收操作中，数据被标记为无效后，并不是立即进行擦除，所以数据块A搬移前对应的块作为数据块A的副本。本发明另一实现方式中，数据块A的副本还可以为缓存中数据块A，可以是SSD缓存中的块，还可以是存储系统中其他位置缓存的数据块A。本发明另一实施方式中，数据块A的副本还可以为数据块A在分级存储操作产生的块，例如数据块A从第一性能存储层迁移到第二性能存储层后，数据块A发生损坏，则在第一性能存储层的数据块A在没有被删除前，作为数据块A的副本。
针对不同来源的副本，可以通过不同的方法获取。例如，针对缓存或分级存储来源的副本，可以通过损坏的块的元数据，然后根据该元数据的信息获取损坏的块的副本。数据块的元数据中记载有数据块的物理地址信息以及数据块的缓存/分级存储地址信息，控制器根据数据块的缓存/分级存储地址信息读取数据块的副本。
针对垃圾回收产生的副本，可以通过获取损坏的块的反向描述信息的方式来获取损坏的块的副本。
SSD内部由于采用追加写方式写入数据，需要进行垃圾回收操作以对数据进行整理。而SSD内部一般具有约15％左右的超配空间，同时SSD一般不会完全写满，大部分写入数据约在80％左右。所以在SSD盘内具有大量以未擦除的垃圾形式存在于盘内的数据。同时SSD盘内一般写入放大在3左右，主要由盘内垃圾回收引起，这也反应了较多的数据副本的存证。因此，可以利用垃圾回收产生的垃圾数据块作为损坏的块的副本，来进行损坏的块的重构。
需要说明的是，本发明实施例将SSD执行垃圾回收操作，将物理块中的有效数据块搬移到新的存储位置后，残留在物理块中未擦除的数据块称之为垃圾数据块。当SSD执行垃圾回收操作搬移的有效数据块出现损坏时，垃圾操作产生的垃圾数据块可以作为损坏的有效数据块的副本。
同时，SSD盘为了有效支撑自身的垃圾回收运行，在存储数据时，会存放反向描述信息，来支撑自己对数据搬移以及正确性的判断。如图3所示，通常这些反向描述信息固定存放于 SSD物理块的尾部。反向描述信息包括物理块中各个数据块的反向映射信息和校验信息，利用这些信息，则可以对垃圾数据块是否和某块数据对应，以及数据是否正确做出判断，从而保证数据的正确性。例如，反向映射信息包括数据物理地址和逻辑块地址(Logical Block Address,LBA)的映射，通过反向映射信息，可以识别存储在物理地址上的数据对应的LBA。这些反向描述是另一种形式存在的盘上元数据，通常SSD的元数据占比＜存储数据容量的1％，整体占比较少。
图4示出了SSD盘中一种数据存放示意图。如图4所示，SSD盘中包括多个物理块，例如物理块1、物理块2、物理块3、物理块4和物理块5，多个物理块组成RAID组，其中物理块1、物理块2、物理块3、物理块4中存储数据块，物理块5中存储校验块。例如，在垃圾回收操作过程中，物理块6中的数据块A被搬移到物理块3中，SSD将物理块6中的数据块A标记为无效。在垃圾回收操作中，被标记为无效的数据块不会被立刻擦除，因此物理块6中的数据块存在一个在物理块3中的数据块副本。
图5为利用被标记为无效的数据块进行数据重构的示意图。在图5中，当物理块3中的数据块A损坏，利用物理块6中的数据块A恢复物理块3中的数据块A。如图5所示，当物理块3中的数据块A损坏时，获取物理块6中被标记为无效的数据块A作为物理块3中存储的数据块A的副本，基于该副本对物理块3中的损坏的块进行重构，从而避免使用RAID算法进行重构计算，节省了计算资源，提高了数据重构效率。
具体的，在物理块中存储的数据块发生故障，可以获取数据块的反向描述信息。如果反向描述信息丢失，因为反向描述信息的数据量小，可以使用RAID算法对反向描述信息进行恢复，从而根据反向描述信息确定损坏的数据块的LBA。
读取其他物理块的反向描述信息，得到其他物理块中存储的各个数据块的LBA，并与损坏的数据块对应的LBA进行匹配，当两个数据块的LBA相同时，确定两个数据块匹配，记录匹配成功的数据块的位置，包括匹配成功的损坏的数据块的位置以及未损坏的数据块的位置，根据该位置信息读取得到损坏的数据块的副本。
在另一个示例中，反向描述信息还包括物理块中存储的各个数据块的校验信息，例如循环冗余校验(cyclic redundancy check，CRC)。
在该示例中，获取数据块的副本的过程包括，读取物理块中损坏的各个数据块的LBA和CRC，并将这些信息建立表格或其他组织形式，以LBA为索引进行存放，便于后续匹配比对。
读取其他物理块的反向描述信息，得到未损坏的各个数据块的LBA和CRC，并与损坏的数据块对应的索引信息进行匹配，当两个数据块的LBA和CRC都相同时，确定两个数据块匹配，记录匹配成功的数据块的位置，包括损坏的数据块的位置以及未损坏的数据块的位置，根据该位置信息读取得到损坏的数据块的副本，以及该数据块的CRC，根据该CRC校验数据块的正确性，验证通过，则将该数据块作为副本对损坏的数据块进行恢复。
为了进一步提升数据重构的效率，在另一个示例中，本发明实施例提供的数据重构方法还包括，获取各个物理块发生故障的概率；将垃圾回收操作配置为，将执行垃圾操作的数据块搬移至于发生故障的概率最高或发生故障的概率大于预设阈值的物理块中。
例如，对于SSD的故障进行分析预测，得到SSD中大概率会发生故障的物理块。比如根据自我检测、分析和报告技术(self-monitoring analysis and reporting technology，SMART)预测SSD盘内的各个介质发生故障的概率，将发生故障的概率大于预设阈值的或者发生故障的概率最大的物理块作为目标物理块，调整SSD的垃圾回收算法，将垃圾回收搬移的数据块写入该目标物理块中。如此，当SSD发生故障大概率是在该目标物理块中，当其发生故障时， 需要对该物理块中的数据块进行重构时，可以有大量的垃圾回收残留的无效数据块可以作为副本，进一步提升数据重构的效率。
针对EC算法实现保护的数据保护组，对损坏的块的重构方法与RAID保护的数据保护组中的损坏的块的重构方法类似，同样是对损坏的块先获取其对应的副本，利用该副本恢复损坏的块这里不再赘述。
本发明实施例上述方法还可以应用在服务器或者云服务提供的设备，本发明实施例对此不作限定。
与前述的数据重构方法的实施例基于相同的构思，本发明实施例中还提供了一种数据重构装置600，该数据重构装置600包括用以实现上文提及的数据重构方法中的各个步骤的单元或模块。
图6为本发明实施例提供的数据重构装置的结构示意图。
如图6所示，该数据重构装置600至少包括获取模块601和重构模块602，其中获取模块601用于获取数据保护组中损坏的块的副本；重构模块602用于基于损坏的块的副本恢复数据保护组中损坏的块；其中，数据保护组中的块使用纠删码EC算法或独立磁盘冗余阵列RAID保护。
在另一个可能的实现中，上述提及的损坏的块为GC操作搬移有效数据生成的块；所述损坏的块的副本为所述垃圾回收操作完成有效数据搬移后，将搬移前损坏的块所在的存储位置存储的块标记为无效的块。
在另一个可能的实现中，损坏的块的副本为损坏的块在缓存中的副本或损坏的块在分级存储操作产生的块。
在另一个可能的实现中，数据保护组中的块为同一块硬盘中的块。
在另一个可能的实现中，数据保护组中的块为存储系统中的块。
在另一个可能的实现中，存储系统采用追加写的方式写入数据。
在另一个可能的实现中，本发明实施例提供的数据重构装置还包括校验模块，该校验模块用于对损坏的块的副本进行校验。
在另一个可能的实现中，获取模块具体用于：获取损坏的块的反向描述信息；反向描述信息包含损坏的块的逻辑地址；根据逻辑地址确定损坏的块的副本。
在另一个可能的实现中，本发明实施例提供的数据重构装置600还包括配置模块604，将垃圾回收操作配置为，将执行垃圾操作的数据块搬移至于发生故障的概率最高或发生故障的概率大于预设阈值的块中。也就是说，调整GC算法，将GC搬移的数据写入更易损坏的块中，这样当块损坏时，可以有大量的GC残留的副本进行重构，节省了计算资源，提高了数据重构的效率。
根据本发明实施例的数据重构装置600可对应于执行本发明实施例中描述的方法，并且数据重构装置600中的各个模块的上述和其它操作和/或功能分别为了实现上文提及的数据重构方法，具体实现可以参考上文描述，为了简洁，在此不再赘述。
其中一种实现，图6实施例提供的数据重构装置可以由图2所示的存储系统的处理器执行。另一种实现，数据重构装置可以由硬盘的控制器执行，其中，硬盘包含控制器和存储介质，控制器与存储介质通信。图2所示的存储系统可以由存储阵列，也可以是分布式存储系统，本发明实施例对此不作限定。
与前述的方法的实施例基于相同的构思，本发明实施例中还提供了一种数据处理设备，该电子设备至少包括处理器和接口，处理器执行程序，可以实现上文提及的数据重构方法。
图7为本发明实施例提供的一种数据处理设备的结构示意图。
如图7所示，所述数据处理设备700包括至少一个处理器701和接口702。其中，处理器701和接口702通信连接，可以通过总线的方式实现通信连接，该接口702用于接收其他设备发送的数据(例如上层应用发送的IO数据)；处理器701执行前述方法实施例中的方法。
应理解，在本发明实施例中，该处理器701可以是中央处理单元CPU，该处理器701还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者是任何常规的处理器等。
图7所示的数据处理设备可以是硬盘、存储系统、服务器或者云服务提供的设备。其中，存储系统可以是存储阵列或服务器。
应理解，根据本发明实施例的数据设备700可以执行实现本发明实施例提及的方法，该方法实现的详细描述参见上文，为了简洁，在此不再赘述。
本发明的实施例提供了一种计算机可读存储介质，其上存储有计算机程序，当所述计算机指令在被处理器执行时，使得上文提及的方法被实现。
本发明的实施例提供了一种芯片，该芯片包括至少一个处理器和接口，所述至少一个处理器通过所述接口确定程序指令或者数据；该至少一个处理器用于执行所述程序指令，以实现上文提及的方法。
本发明的实施例提供了一种计算机程序或计算机程序产品，该计算机程序或计算机程序产品包括指令，当该指令执行时，令计算机执行上文提及的方法。
本领域普通技术人员应该还可以进一步意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执轨道，取决于技术方案的特定应用和设计约束条件。本领域普通技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。
结合本文中所公开的实施例描述的方法或算法的步骤可以用硬件、处理器执轨道的软件模块，或者二者的结合来实施。软件模块可以置于随机存储器(RAM)、内存、只读存储器(ROM)、电可编程ROM、电可擦除可编程ROM、寄存器、硬盘、可移动磁盘、CD-ROM、或技术领域内所公知的任意其它形式的存储介质中。
以上所述的具体实施方式，对本发明的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上所述仅为本发明的具体实施方式而已，并不用于限定本发明的保护范围，凡在本发明的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (18)

  1. 一种数据重构方法，其特征在于，所述方法包括：
    获取数据保护组中损坏的块的副本；
    基于所述损坏的块的副本恢复所述数据保护组中损坏的块；其中，数据保护组中的块使用纠删码EC算法或独立磁盘冗余阵列RAID保护。
  2. 根据权利要求1所述的方法，其特征在于，所述损坏的块为垃圾回收操作搬移有效数据生成的块；所述损坏的块的副本为所述垃圾回收操作完成有效数据搬移后，将搬移前所述损坏的块所在的存储位置存储的块标记为无效的块。
  3. 根据权利要求1所述的方法，其特征在于，所述损坏的块的副本为所述损坏的块在缓存中的副本或所述损坏的块在分级存储操作产生的块。
  4. 根据权利要求1至3任一所述的方法，其特征在于，所述数据保护组中的块为同一块硬盘中的块。
  5. 根据权利要求1至3任一所述的方法，其特征在于，所述数据保护组中的块为存储系统中的块。
  6. 根据权利要求5所述的方法，其特征在于，所述存储系统采用追加写的方式写入数据。
  7. 根据权利要求1至3任一所述的方法，其特征在于，所述方法还包括：对所述损坏的块的副本进行校验。
  8. 根据权利要求1至3任一所述的方法，其特征在于，获取数据保护组中损坏的块的副本，具体包括：
    获取所述损坏的块的反向描述信息；所述反向描述信息包含所述损坏的块的逻辑地址；
    根据所述逻辑地址确定所述损坏的块的副本。
  9. 一种数据重构装置，其特征在于，包括：
    获取模块，用于获取数据保护组中损坏的块的副本；
    重构模块，用于基于所述损坏的块的副本恢复所述数据保护组中损坏的块；其中，数据保护组中的块使用纠删码EC算法或独立磁盘冗余阵列RAID保护。
  10. 根据权利要求9所述的装置，其特征在于，所述损坏的块为垃圾回收操作搬移有效数据生成的块；所述损坏的块的副本为所述垃圾回收操作完成有效数据搬移后，将搬移前所述损坏的块所在的存储位置存储的块标记为无效的块。
  11. 根据权利要求9所述的装置，其特征在于，所述损坏的块的副本为所述损坏的块在缓存中的副本或所述损坏的块在分级存储操作产生的块。
  12. 根据权利要求9至11任一项所述的装置，其特征在于，所述数据保护组中的块为同一块硬盘中的块。
  13. 根据权利要求9至11任一项所述的装置，其特征在于所述数据保护组中的块为存储系统中的块。
  14. 根据权利要求13所述的装置，其特征在于，所述存储系统采用追加写的方式写入数据。
  15. 根据权利要求9至11任一项所述的装置，其特征在于，还包括：
    校验模块，用于对所述损坏的块的副本进行校验。
  16. 根据权利要求9至11任一项所述的装置，其特征在于，所述获取模块具体用于：
    获取所述损坏的块的反向描述信息；所述反向描述信息包含所述损坏的块的逻辑地址；
    根据所述逻辑地址确定所述损坏的块的副本。
  17. 一种数据处理设备，包括接口和处理器，其特征在于，所述接口与所述处理器通信连接，所述处理器用于实现权利要求1至8任一项所述的方法。
  18. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，当所述计算机程序在计算机中执行时，令计算机执行权利要求1至8任一项所述的方法。
PCT/CN2023/087558 2022-06-20 2023-04-11 一种数据重构方法及装置 WO2023246240A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202210701616.6 2022-06-20
CN202210701616 2022-06-20
CN202211378245.9A CN117270758A (zh) 2022-06-20 2022-11-04 一种数据重构方法及装置
CN202211378245.9 2022-11-04

Publications (1)

Publication Number Publication Date
WO2023246240A1 true WO2023246240A1 (zh) 2023-12-28

Family

ID=89214804

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/087558 WO2023246240A1 (zh) 2022-06-20 2023-04-11 一种数据重构方法及装置

Country Status (2)

Country Link
CN (1) CN117270758A (zh)
WO (1) WO2023246240A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108170555A (zh) * 2017-12-21 2018-06-15 浙江大华技术股份有限公司 一种数据恢复方法及设备
US20200241770A1 (en) * 2019-01-30 2020-07-30 Xilinx, Inc. Retaining memory during partial reconfiguration
CN112596673A (zh) * 2020-12-18 2021-04-02 南京道熵信息技术有限公司 具有双重raid数据保护的多活多控存储系统
CN114155906A (zh) * 2021-10-29 2022-03-08 广东浪潮智慧计算技术有限公司 一种数据块修复方法、装置、设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108170555A (zh) * 2017-12-21 2018-06-15 浙江大华技术股份有限公司 一种数据恢复方法及设备
US20200241770A1 (en) * 2019-01-30 2020-07-30 Xilinx, Inc. Retaining memory during partial reconfiguration
CN112596673A (zh) * 2020-12-18 2021-04-02 南京道熵信息技术有限公司 具有双重raid数据保护的多活多控存储系统
CN114155906A (zh) * 2021-10-29 2022-03-08 广东浪潮智慧计算技术有限公司 一种数据块修复方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN117270758A (zh) 2023-12-22

Similar Documents

Publication Publication Date Title
US11941257B2 (en) Method and apparatus for flexible RAID in SSD
US9798620B2 (en) Systems and methods for non-blocking solid-state memory
CN104035830B (zh) 一种数据恢复方法和装置
JP4916033B2 (ja) データ格納方法、データ・ストレージ・システムおよびプログラム（ストレージ・システムにおけるデータ完全性の検証）（著作権および商標登録表示本特許文書の開示の一部は、著作権保護を受ける内容を含む。本所有権者は、特許文書または特許開示書のいずれか一つによるファクシミリ複写物には、複写物が特許商標庁の特許ファイルまたは記録として世に出現している限り異論はないが、他の場合に全ての著作権は完全に留保する。）（本明細書で参照するある種のマークについては、出願人またはその譲受人と提携しまたは提携しない第三者の、慣習法上の、または登録された商標である可能性がある。これらのマークを使用するのは、例示によって実施可能な開示を提供するためであり、そのようなマークに関連するもののみに本発明の範囲を制限するように解釈されるべきではない。）
US8788876B2 (en) Stripe-based memory operation
US9430329B2 (en) Data integrity management in a data storage device
US8051367B2 (en) Storage sub-system and method for controlling the same
US11531590B2 (en) Method and system for host-assisted data recovery assurance for data center storage device architectures
TWI461901B (zh) 資料儲存與重建的方法與系統
US20140351526A1 (en) Data storage controller with multiple pipelines
TWI451257B (zh) 保護在直接附加儲存（ｄａｓ）系統中快取資料的完整性之裝置及方法
US10474527B1 (en) Host-assisted error recovery
WO2013159503A1 (zh) 一种硬盘数据恢复方法、装置及系统
US10261705B2 (en) Efficient data consistency verification for flash storage
CN103049222A (zh) 一种raid5的写io优化处理方法
CN111124258B (zh) 全闪存阵列的数据存储方法、装置、设备及可读存储介质
WO2024113685A1 (zh) 一种raid阵列的数据恢复方法及相关装置
US20160026408A1 (en) Storage device metadata synchronization
US8938641B2 (en) Method and apparatus for synchronizing storage volumes
US20150134875A1 (en) Maintaining at least one journal and/or at least one data structure by circuitry
US20150347224A1 (en) Storage control apparatus and method therefor
KR101532283B1 (ko) Ｓｓｄ 기반 ｒａｉｄ 스토리지에서 데이터 및 패리티 디스크의 복합적 중복제거 방법
US11604697B2 (en) Distributed erasure encoding in shared file system
WO2023246240A1 (zh) 一种数据重构方法及装置
WO2015100639A1 (zh) 一种去重方法装置与系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23825895

Country of ref document: EP

Kind code of ref document: A1