WO2021248863A1 - 数据处理方法及存储设备 - Google Patents

数据处理方法及存储设备 Download PDF

Info

Publication number
WO2021248863A1
WO2021248863A1 PCT/CN2020/136106 CN2020136106W WO2021248863A1 WO 2021248863 A1 WO2021248863 A1 WO 2021248863A1 CN 2020136106 W CN2020136106 W CN 2020136106W WO 2021248863 A1 WO2021248863 A1 WO 2021248863A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
granularity
storage device
block
metadata
Prior art date
Application number
PCT/CN2020/136106
Other languages
English (en)
French (fr)
Inventor
任仁
刘中全
刘宏伟
朱芳芳
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202010784929.3A external-priority patent/CN113806341A/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP20939891.6A priority Critical patent/EP4030310A4/en
Publication of WO2021248863A1 publication Critical patent/WO2021248863A1/zh
Priority to US17/741,079 priority patent/US12001703B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques

Definitions

  • This application relates to the field of storage technology, and in particular to a data processing method and storage device.
  • Deduplication and compression are key technologies in the storage industry. Through deduplication and compression, the storage device can reduce the size of the actual stored data, save the storage space occupied by the data in the storage device, and improve the storage efficiency of the storage device.
  • the storage device presets a fixed granularity, performs deduplication based on this granularity, and performs compression based on this granularity. For example, if the granularity is set to 8 kilobytes (Kilobyte, KB) in advance, the storage device will determine whether every 8KB data block is a duplicate block when performing deduplication. If a certain 8KB data block is a duplicate block, The storage device deletes the 8KB data block; and when the storage device compresses, the 8KB data block is compressed every time.
  • the granularity is set to 8 kilobytes (Kilobyte, KB) in advance
  • the storage device will determine whether every 8KB data block is a duplicate block when performing deduplication. If a certain 8KB data block is a duplicate block, The storage device deletes the 8KB data block; and when the storage device compresses, the 8KB data block is compressed every time.
  • the embodiments of the present application provide a data processing method and storage device, which solves the limitations of the data processing method to a certain extent.
  • the technical solution is as follows:
  • a data processing method which is executed by a storage device and includes: acquiring data; performing deduplication processing on the data based on a first granularity; performing compression processing on the data based on a second granularity, The size of the second granularity is greater than the size of the first granularity; the data that has undergone the deduplication processing and the compression processing is stored in the hard disk of the storage device.
  • the storage device uses different granularity for deduplication and compression, a large granularity is used for deduplication and a small granularity is used for compression, thus getting rid of the deduplication granularity.
  • the same restrictions as the compression granularity must be avoided to a certain extent to avoid the reduction of the deduplication rate due to the excessively large granularity and the decrease of the compression rate due to the excessively small granularity, which helps to improve the overall reduction rate of deduplication and compression.
  • this application does not limit the sequence of the deduplication processing and the compression processing.
  • you can perform compression processing first and then perform deduplication processing and in some scenarios you can perform deduplication processing first and then perform compression processing.
  • the deduplication process first and then the compression process as an example, the duplicate blocks and non-duplicate blocks are obtained after the deduplication process, and this application may only perform the compression process on the non-duplicate blocks. If the compression process is performed first and then the deduplication process is performed, the compressed block is obtained after the data is compressed, and the deduplication process is performed on the compressed block.
  • metadata is stored in the storage device, the metadata is managed based on metadata management granularity, and the size of the metadata management granularity is less than or equal to a set maximum value and greater than or equal to a set value.
  • the size of the first granularity is equal to an integer multiple of the minimum value.
  • the minimum value of the metadata management granularity is used as the granularity used in the deduplication process, it helps to obtain a better granularity during the deduplication process, thereby increasing the deduplication rate and saving storage resources.
  • the size of the second granularity is the product of the minimum value and the compression ratio.
  • the performing deduplication processing on the data based on the first granularity includes: dividing the data into a plurality of data blocks; acquiring a fingerprint of each data block; according to the fingerprint, from the plurality of data blocks; In the data block, repeating blocks and non-repeating blocks are determined.
  • the performing compression processing on the data based on the second granularity includes: performing compression processing on the non-repeated blocks based on the second granularity to obtain compressed blocks;
  • the compressed data includes the compressed block.
  • the method further includes: recording metadata of the compressed block.
  • the recording the metadata of the compressed block includes: if the number of the compressed blocks is multiple and the addresses of the multiple compressed blocks are continuous, recording one piece of metadata for the multiple compressed blocks.
  • addresses of the multiple compressed blocks are continuous refers to that the physical addresses of the multiple compressed blocks are continuous and the logical addresses are continuous.
  • the piece of metadata includes the address of the first compressed block in the multiple compressed blocks and the length of each compressed block.
  • the data is compressed based on a third granularity before undergoing the deduplication processing and the compression processing, and the size of the third granularity is smaller than the size of the second granularity.
  • the storage device finds that the original compression granularity (third granularity) of the compressed block is not the better compression granularity (second granularity), it re-compresses according to the better compression granularity (second granularity). , So that the compression granularity of the compressed block is optimized, thus helping to improve the compression rate.
  • the storage device is a storage array.
  • the storage device is a storage node in a distributed storage system.
  • the first granularity is 4KB
  • the second granularity is 32KB.
  • a data processing method which is executed by a storage device, and includes: acquiring data; determining a first granularity based on a metadata management granularity, where the metadata management granularity is used to manage the storage of the storage device
  • the size of the metadata management granularity is less than or equal to the set maximum value and greater than or equal to the set minimum value, and the size of the first granularity is equal to an integer multiple of the minimum value;
  • the first granularity performs data deduplication processing on the data; the data that has undergone the data deduplication processing is stored in the hard disk of the storage device.
  • the granularity used in the deduplication processing of the storage device is determined based on the granularity of metadata management, it is helpful to use the minimum value of the metadata management granularity as the granularity used in the deduplication processing. In the deduplication process, a better granularity is obtained, thereby increasing the deduplication rate and saving storage resources.
  • the performing deduplication processing on the data based on the first granularity includes: dividing the data into a plurality of data blocks; acquiring a fingerprint of each data block; according to the fingerprint, from the plurality of data blocks; In the data block, repeating blocks and non-repeating blocks are determined.
  • the first granularity is 4KB.
  • a data processing method is provided.
  • the method is executed by a storage device and includes: acquiring data; determining a second granularity based on a metadata management granularity, where the metadata management granularity is used to manage the storage of the storage device.
  • the size of the metadata of the metadata, the size of the metadata management granularity is less than or equal to the set maximum value and greater than or equal to the set minimum value, and the size of the second granularity is the minimum value and the set compression ratio
  • the data is compressed based on the second granularity; the compressed data is stored in the hard disk of the storage device.
  • the granularity used in the compression processing of the storage device is no longer a fixed value, it is dynamically determined according to the metadata management granularity and the compression ratio, so as to ensure that the data reading performance does not decrease.
  • the compression rate is better.
  • the size of the second granularity is the product of the minimum value and the compression ratio.
  • the compressing the data based on the second granularity includes:
  • the method further includes: recording metadata of the compressed block.
  • the recording the metadata of the compressed block includes:
  • the piece of metadata includes the address of the first compressed block in the multiple compressed blocks and the length of each compressed block.
  • the data is compressed based on a third granularity before undergoing the deduplication processing and the compression processing, and the size of the third granularity is smaller than the size of the second granularity.
  • the method further includes: saving the fingerprint of the repeated block.
  • the method further includes: recording the metadata of the non-repeated block, and save the fingerprint of the non-repeated block in a fingerprint table.
  • the second granularity is 32KB.
  • a storage device in a fourth aspect, includes at least one processor and a hard disk, and the at least one processor is configured to execute instructions so that the storage device executes any one of the foregoing first aspect and the first aspect.
  • the data processing method provided by at least one of the second aspect, any one of the optional manners in the second aspect, and any one of the third aspect and the third aspect, the hard disk is used to store data .
  • the storage device provided by the fourth aspect please refer to the above-mentioned first aspect, any one of the optional manners in the first aspect, the second aspect, any one of the optional manners in the second aspect, the third aspect, and the third aspect. At least one of any of the optional methods is not repeated here.
  • the at least one processor includes a first processor, a second processor, and a third processor
  • the first processor is used to obtain data
  • the second processor is configured to perform data deduplication processing on the data based on a first granularity
  • the third processor is configured to compress the data based on a second granularity, where the size of the second granularity is larger than the size of the first granularity;
  • the first processor is further configured to store the data that has undergone the deduplication processing and the compression processing in the hard disk.
  • a storage device in a fifth aspect, has an optional way to implement any one of the first aspect, the first aspect, the second aspect, any one of the optional ways in the second aspect, and the third aspect.
  • the storage device includes at least one module, and at least one module is used to implement the above-mentioned first aspect, any one of the optional manners in the first aspect, the second aspect, any one of the optional manners in the second aspect, the third aspect, and the third aspect.
  • the data processing method provided by at least one of the optional methods in any of the aspects.
  • the modules in the storage device are implemented by software, and the modules in the storage device are program modules. In other embodiments, the modules in the storage device are implemented by hardware or firmware.
  • the modules in the storage device are implemented by hardware or firmware.
  • a computer-readable storage medium stores at least one instruction, and the instruction is read by a processor to enable the storage device to execute any one of the foregoing first aspect and the first aspect.
  • the data processing method provided by at least one of the optional manners in the second aspect and the second aspect, and the third aspect and the optional manner in the third aspect.
  • a computer program product in a seventh aspect, includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the storage device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the storage device executes any one of the above-mentioned first aspect, the first aspect, the second aspect, and the first aspect.
  • a data processing method provided in at least one of the optional manner in the second aspect, the third aspect, and any optional manner in the third aspect.
  • a chip when the chip runs on a storage device, the storage device executes any one of the foregoing first aspect, any one of the optional methods of the first aspect, the second aspect, and the second aspect.
  • the data processing method provided in at least one of the optional manner, the third aspect, and any one of the third aspect.
  • FIG. 1 is a schematic diagram of a system architecture of a distributed storage system provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of storing metadata according to an embodiment of the present application
  • Fig. 3 is a flowchart of a data processing method provided by an embodiment of the present application.
  • Fig. 4 is a schematic diagram of a pre-deduplication provided by an embodiment of the present application.
  • FIG. 5 is a flowchart of a data processing method provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of post-duplication provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of recording a piece of metadata for multiple compressed blocks according to an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a storage device provided by an embodiment of the present application.
  • Deduplication short for data deduplication, is a data reduction technology. A lot of duplicate data is stored in the storage system, which takes up a lot of hard disk space. Using deduplication technology can delete duplicate data. Only one copy of the same data is stored, thereby saving data storage space.
  • the technical principles of deduplication include: first perform block division, that is, divide the data to be written into multiple data blocks; then perform fingerprint calculation, that is, calculate the fingerprint of the data block according to each divided data block; then perform Fingerprint search, that is, use fingerprints as an index to search and compare in the fingerprint table. If the same fingerprint exists in the fingerprint table, it means that it is a duplicate block, and the data block is not saved, but the fingerprint index of the data block is saved.
  • deduplication is divided into pre-deduplication and post-deduplication, which are introduced in (2) and (3) below.
  • deduplication is divided into fixed-length deduplication and similar deduplication, which are introduced in (4) and (5) below.
  • Pre-deduplication refers to deduplication before data is written to the hard disk. Pre-deduplication is also called online deduplication.
  • Post-deduplication refers to deduplication after data is written to the hard disk. Post deduplication is also called offline deduplication. There are many ways to implement post-deduplication. In some embodiments, after writing the data to the hard disk, read the data from the hard disk to the cache; perform fingerprint calculation for the data in the cache, and determine duplicate data by comparing whether the fingerprints are the same; if duplicate data is found, perform deduplication , And then rewrite the deduplicated data to the hard disk. In some embodiments, when the data to be stored is obtained, the fingerprint of the data is calculated; the data is written to the hard disk, and the fingerprint is saved in the opportunity table. When deduplication is required, fingerprints are read from the opportunity table, and different fingerprints are compared to determine duplicate data; if duplicate data is found, deduplication is performed.
  • the data blocks are not required to be completely matched. If two data blocks are similar, it can be determined that the data block is a duplicate block. In addition, when performing block division, the data is also divided according to a preset granularity.
  • the fingerprint is the essential feature of the data block.
  • the data block itself is often large, so the goal of the fingerprint is to use a smaller data representation (such as 16, 32, 64, 128 bytes) to distinguish different data blocks.
  • a hash algorithm is used to calculate the fingerprint of the data block, and the fingerprint of the data block is the hash value of the database.
  • each data block has a unique fingerprint, and different data blocks have different fingerprints.
  • different data blocks may also have the same fingerprint.
  • Compression is a byte-level data reduction technology.
  • the idea is to use encoding technology to express longer data in a shorter, encoded format, so as to achieve the purpose of reducing data size.
  • the compression ratio is a positive integer greater than or equal to 1.
  • the compression rate represents the ratio between the amount of data before compression and the amount of data after compression. For example, if 32KB of data is compressed into 8KB, the compression ratio is 4:1.
  • Deduplication compression refers to a reduction technique that uses deduplication and compression together.
  • deduplication compression scheme when duplicate blocks and non-duplicate blocks are found, multiple non-duplicate blocks will be compressed first, and the compressed blocks will be saved.
  • the compressed block When reading data, the compressed block will be decompressed. Since the non-repetitive blocks are compressed, the data reduction effect is the superposition of the deduplication effect and the compression effect, and more data is reduced.
  • Read amplification refers to the situation where the data granularity actually read from the hard disk is larger than the data granularity corresponding to the read request. Reading amplification will consume network bandwidth resources and affect the performance of data reading.
  • the granularity of hard disk storage data is 8KB.
  • the storage device receives the read request, and the read request instructs the storage device to read 4KB data; the storage device reads the 8KB data block where the 4KB data is located from the hard disk, determines the 4KB data requested to be read from the 8KB data block, and returns the 4KB data to The initiator of the read request.
  • the data granularity corresponding to the read request is 4KB
  • the actual read data granularity is 8KB. Because the extra 4KB data is read, it will take up too much bandwidth resources and affect the data read performance.
  • Fingerprint index fingerprint index, FPI
  • the fingerprint index is the index of the fingerprint of the data block.
  • the fingerprint index is used to query the corresponding fingerprint.
  • the following embodiments of this application use the form of "FPI numbers" to simplify the representation of specific FPI without introducing difficulties in understanding.
  • the meaning of the number is the identification of the corresponding data block, such as FPI 4 for the data block 4 Fingerprint index.
  • a storage unit refers to the smallest unit in a storage device that can perform storage operations, and storage operations include data write operations or data read and write operations.
  • storage operations include data write operations or data read and write operations.
  • a storage unit is a sector of a storage device.
  • the physical address refers to the actual address where the data is stored in the hard disk. Specifically, each area on the hard disk is determined by the head (Heads), cylinder (track), and sector (Sector) where the area is located.
  • the physical address includes three parameters, namely the head parameter, the cylinder parameter and the sector parameter.
  • the head parameters are used to identify the head where the data is located.
  • the cylinder parameter is used to identify the cylinder where the data is located, and the sector parameter is used to identify the sector where the data is located.
  • the logical address is different from the physical address.
  • the full name of the logical address is the logical block address (Logical Block Address, LBA).
  • LBA Logical Block Address
  • the physical address which is a three-dimensional addressing method based on magnetic heads, cylinders, and sectors, is transformed into one-dimensional linear addressing, thereby improving the addressing efficiency.
  • the logical address is the address of the logical space presented by the storage device to the host.
  • the host sends a write request or read request to the storage device, it will carry the logical address in the write request or read request.
  • the storage device receives a write request or read request, it will obtain the logical address carried in the write request or read request, and determine the physical address after one or more address conversions to the logical address, write data to the physical address or read from the physical address data.
  • the logical address continuity is, for example, LBA continuity.
  • the LBA of data block 1 is 201
  • the LBA of data block 2 is 202
  • the LBA of data block 3 is 203. It can be said that the logical addresses of data block 1, data block 2, and data block 3 are continuous.
  • Metadata is data that describes the attributes of business data. For example, metadata describes fingerprints, logical addresses, physical addresses, the mapping relationship between logical addresses and physical addresses, the mapping relationship between fingerprints and logical addresses, and so on.
  • the storage method of metadata is different from the storage method of business data.
  • Metadata is generally stored in a certain data structure. Data structures for storing metadata are, for example, binary trees, B+ trees, etc. These data structures require a certain granularity to manage metadata. In some embodiments of the present application, the granularity of managing metadata is referred to as metadata management granularity.
  • the fingerprint table is used to save the fingerprint of each data block stored in the storage device.
  • the opportunity table is used to save the fingerprint of the data block written in the storage device in the most recent period of time.
  • the opportunity table is different from the fingerprint table.
  • the opportunity table can be understood as a temporary window for finding data blocks with deduplication opportunities.
  • the storage device will store the fingerprint of the data block generated in the most recent period in the opportunity table.
  • the deduplication trigger condition for example, the load is lower than the threshold
  • the storage device will find the duplicate block according to the fingerprint in the opportunity table.
  • the fingerprint of the duplicate block is saved in the fingerprint table.
  • the method provided in this embodiment can be applied to a distributed storage system or a centralized storage device, and these two application scenarios are respectively introduced below.
  • Application scenario 1 Distributed storage system
  • this embodiment provides a distributed storage system 100.
  • the system 100 includes a plurality of storage nodes 101 and at least one host 102.
  • Each host 102 and the storage node 101 establish a communication connection through a wired or wireless network.
  • each host 102 and the storage node 101 establish a communication connection through an Internet protocol (IP) network or other networks.
  • IP Internet protocol
  • Each storage node 101 includes a network card 1011, one or more hard disks 1012, a processor 1013, a processor 1014, and a memory 1015.
  • the network card 1011 is also called a network interface card (NIC).
  • NIC network interface card
  • the network card 1011 is used to communicate with the host 102.
  • the hard disk 1012 is, for example, a solid state disk (SSD) or a hard disk drive (HDD). This embodiment does not limit the positional relationship between the storage node 101 and the hard disk 1012.
  • the hard disk 1012 is located inside the storage node 101.
  • the storage node 101 is a server, and multiple hard disks are provided in the server.
  • the hard disk 1012 is not inside the storage node 101, but is located in a hard disk enclosure coupled to the storage node 101, and the hard disk enclosure includes a plurality of hard disks 1012.
  • the processor 1013 is, for example, a central processing unit (CPU).
  • the processor 1013 is configured with one or more processors, for example.
  • the processor 1014 is used to undertake the function of compression and/or deduplication, thereby reducing the computational burden of the processor 1013.
  • the processor 1014 is a processor having the same physical form as the processor 1013.
  • the processor 1014 and the processor 1013 have a different physical form.
  • the processor 1014 is a processing chip with computing capabilities.
  • the processor 1014 is an accelerator card, a coprocessor, a graphics processing unit (Graphics Processing Unit, GPU), or a neural network processor (Neural-network Processing Unit, NPU), etc.
  • One or more processors 1014 are configured, for example.
  • the processor 1014 and the processor 1013 may optionally cooperate together to complete the data processing operation.
  • the processor 1013 is configured to receive data from the host, send the data to the processor 1014, and instruct the processor 1014 to compress and/or deduplicate the data.
  • the processor 1014 performs compression and/or deduplication when triggered by receiving an instruction from the processor 1013.
  • the processor 1013 when the storage node 101 has multiple processors 1014, the processor 1013 is used to schedule the multiple processors 1014. For example, the processor 1013 divides the compression task and/or the deduplication task into multiple subtasks, and assigns each subtask to the corresponding processor 1014.
  • the storage node 101 further includes a communication bus (not shown in FIG. 1), and the processor 1014 and the processor 1013, for example, both access the memory 1015 through the communication bus to obtain instructions or codes cached in the memory 1015.
  • the processor 1014 is an optional component of the storage node 101.
  • the storage node 101 does not have a processor 1014 but has a processor 1013.
  • the processor 1013 independently completes the operations of obtaining data, compressing and/or deduplicating the data.
  • the host 102 includes an application 1031 and a client 1032.
  • the storage node 101 can provide data storage services for the host 102. For example, when the host 102 needs to store data in the storage node 101, an application 1031 (also referred to as an upper-layer application) in the host 102 generates a write request and sends the write request to the storage node 101.
  • the storage node receives the write request through the network card 1011, writes the data indicated by the write request to the hard disk 1012, and saves the metadata of the data.
  • the storage node 101 can provide the host 102 with data access services. For example, when the host 102 needs to access data stored in the storage node 101, the application 1031 in the host 102 generates a read request and sends the read request to the storage node 101. The storage node receives the read request through the network card 1011. The storage node determines the address of the data in the hard disk 1012 according to the data indicated by the read request and the saved metadata, reads the data from the corresponding address in the hard disk 1012, and sends the data to the host 102 through the network card 1011. The host 102 receives the data, thereby obtaining the data stored by the storage node 101.
  • the centralized storage device is, for example, a storage array.
  • the storage array includes one or more controllers, also known as storage controllers and one or more hard disks.
  • the centralized storage device may also be a storage node, as shown in the storage node 101 in FIG. 1.
  • the controller in the storage device is also called the storage controller.
  • the centralized storage device is connected to the host through a wired network or a wireless network.
  • the storage device uses different granularities for deduplication and compression. That is, the deduplication granularity and the compression granularity are different, so as to avoid the situation that the deduplication rate is reduced due to the excessively large granularity and the compression rate is reduced due to the excessively small granularity.
  • the deduplication granularity and the compression granularity are different, so as to avoid the situation that the deduplication rate is reduced due to the excessively large granularity and the compression rate is reduced due to the excessively small granularity.
  • several particle sizes involved in this application and the relationship between different particle sizes are introduced through (a) to (g).
  • Granularity is also called the size of the data, the size of the data, or the length of the data. The larger the granularity, the larger the size of a piece of data.
  • the unit of granularity includes but is not limited to KB, Mega (MB, Mega, M for short), etc.
  • the granularity is 4KB, which means that the size of a piece of data is 4KB.
  • granularity is an important parameter. Granularity affects many businesses of storage devices, including but not limited to data reading, data storage, deduplication, compression, metadata management, etc. In some embodiments of the present application, the description will focus on the granularity involved in several types of services such as deduplication, compression, and metadata management, the relationship between different granularities, and the impact of granularity on storage devices.
  • the deduplication granularity is used to indicate the granularity used when the storage device queries duplicate data.
  • the deduplication granularity is equal to the granularity of the block in the deduplication process. For example, when the size of the deduplication granularity is 4KB, the storage device will divide the data into multiple 4KB data blocks; the storage device will determine whether each 4KB data block is a duplicate block; if a 4KB data block is a duplicate block, The storage device will delete the 4KB data block. In some embodiments, if fixed-length deduplication is used for deduplication, the size of the deduplication granularity is 4KB. If a similar deduplication method is used for deduplication, the size of the deduplication granularity is 8KB.
  • the compression granularity is used to indicate the granularity used when the storage device performs data compression.
  • the storage device will determine how much data to compress at a time according to the compression granularity. For example, when the size of the compression granularity is 32KB, the storage device will compress 32KB of data.
  • the size of the compression granularity affects the compression ratio. It is found through experiments that the compression rate is directly proportional to the size of the compressed particle size. When the size of the compressed particle size is less than 32KB, the larger the size of the compressed particle size, the greater the compression rate; when the size of the compressed particle size exceeds 32KB, the compression rate tends to be stable.
  • the size of the compression granularity is determined based on the compression ratio. In some embodiments, the size of the compression granularity is set to 32KB.
  • the metadata management granularity is used to indicate the granularity of the metadata recorded by the storage device.
  • the value of the metadata management granularity is not a fixed value, but can be dynamically changed within a certain range.
  • the metadata management granularity is an interval. This interval has a minimum and maximum value.
  • the metadata granularity recorded by the storage device is, for example, the minimum value of the interval, the maximum value of the interval, or a value between the minimum value and the maximum value.
  • the storage device can optionally record a piece of metadata of 4KB, or record a piece of metadata of 1M, or record a piece of metadata with a granularity between 4KB and 1M.
  • the size of the metadata management granularity is an integer multiple of the size of the storage unit.
  • metadata is stored through at least one storage unit.
  • the quantitative relationship between the metadata and the storage unit is a one-to-one relationship or a one-to-many relationship.
  • one storage unit stores one piece of metadata.
  • multiple storage units collectively store one piece of metadata, for example, multiple storage units with consecutive physical addresses and consecutive logical addresses collectively store one piece of metadata.
  • the minimum size of the metadata management granularity is the size of a storage unit.
  • Figure 2 shows 8 storage units in the storage device.
  • the 8 storage units are respectively the storage unit 201, the storage unit 202, the storage unit 203, the storage unit 204, the storage unit 205, and the storage unit.
  • the minimum size of the metadata management granularity is the size of one storage unit in 8 storage units, which is represented by 1 grid in FIG. 2.
  • the maximum size of the metadata management granularity is the sum of the sizes of 8 storage units, which is represented by 8 grids in FIG. 2.
  • the size of the deduplication granularity is inversely proportional to the size of the compression granularity.
  • the size of the deduplication granularity is smaller than the size of the compression granularity.
  • the size of the deduplication granularity is 4KB, and the size of the compression granularity is 32KB. Since the smaller the deduplication granularity, the better the deduplication rate, and the larger the compression rate, the better the compression rate. Therefore, the storage device adopts a large-granularity compression and small-granularity deduplication scheme to help deduplication and compression rates. Both have been improved to achieve an overall better reduction rate.
  • both the deduplication granularity and the compression granularity are determined according to the metadata management granularity.
  • the storage device selects the granularity for deduplication and compression according to the granularity of metadata management, which helps deduplication and compression to operate according to their respective better granularity, so that the deduplication rate and compression rate are both better.
  • the deduplication granularity is determined according to the metadata management granularity.
  • the size of the deduplication granularity is equal to the minimum value of the metadata management granularity. For example, when the metadata management granularity is [4KB, 1M], the size of the deduplication granularity is equal to 4KB. For example, when the metadata management granularity is [8KB, 2M], the size of the deduplication granularity is equal to 8KB.
  • the size of the deduplication granularity is, for example, the size of a storage unit.
  • the size of the deduplication granularity is, for example, the size of the storage unit 201.
  • the granularity of deduplication will affect the deduplication rate. If the deduplication granularity is too large, the deduplication rate will decrease. For example, if the deduplication granularity is 32KB, and the 32KB data is duplicate data, the storage device will treat the 32KB data as a duplicate block and delete it. If only part of the 32KB data is repeated data, for example, only 24KB of data is repeated data, and the other 8KB of data is not repeated data, the storage device will not perform deduplication. From this example, it can be seen that the deduplication granularity is too large and the deduplication effect is not good.
  • the granularity of deduplication will affect the storage overhead of metadata. Since the storage device records a piece of metadata for each duplicate block, the smaller the deduplication granularity, the more metadata the storage device will record. Therefore, if the deduplication granularity is too small and excessive duplicate blocks are generated, the storage device needs to record too much metadata, which causes the metadata to occupy too much storage resources.
  • the minimum value of the metadata management granularity is used as the deduplication granularity. Since the deduplication granularity is small enough, it helps to increase the deduplication rate. In addition, during deduplication, the storage device does not need to record metadata for data smaller than the minimum value of the metadata management granularity, thus avoiding resource waste caused by recording multiple data.
  • the size of the deduplication granularity is not the minimum value of the metadata management granularity, but an integer multiple of the minimum value of the metadata management granularity.
  • the size of the deduplication granularity is any multiple of 4KB between 4KB and 1M.
  • the size of the deduplication granularity is 2 or 3 times the minimum value of the metadata management granularity.
  • the size of the deduplication granularity is, for example, an integer multiple of the size of the storage unit.
  • the size of the deduplication granularity is, for example, an integer multiple of the size of the storage unit 201.
  • the size of the compression granularity is, for example, the product of the size of a storage unit and the compression rate.
  • the compression granularity is, for example, the product of the size of the storage unit 201 and the compression rate.
  • the compression granularity is 4 times the size of the storage unit 201, and the compression granularity corresponds to four grids in FIG. 2. Since the size of the compression granularity is no longer a fixed value, it is dynamically selected according to the compression ratio, so that the compression ratio is better when the data reading performance does not decrease.
  • the various granularity and granularity relationship features introduced above are, for example, applied to the process of writing data to the storage device.
  • the process of writing data involves a pre-deduplication process and a post-deduplication process.
  • the method 300 is used to illustrate how the storage device uses the various granularities described above to perform pre-deduplication
  • the method 400 is used to illustrate how the storage device uses the various granularities described above to perform post-deduplication.
  • the method 300 and method 400 described below are executed by a storage device.
  • the method 300 or the method 400 is applied in a distributed storage system, and the storage device that executes the method 300 or the method 400 is one or more storage nodes in the distributed storage system.
  • the storage device that executes the method 300 or the method 400 is the storage node 101 in the system 100 shown in FIG. 1, and the data processed by the method 300 or the method 400 is data of the host 102 in the system 100.
  • the method 300 or the method 400 is applied to a centralized storage device, and the storage device that executes the method 300 or the method 400 is a storage array.
  • the method 300 or the method 400 is executed by the CPU. In other embodiments, the method 300 or the method 400 is executed by a CPU and a dedicated processor for deduplication and compression in cooperation, and the dedicated processor for deduplication and compression is, for example, a hardware accelerator card.
  • the CPU is the processor 1013 shown in FIG. 1
  • the dedicated processor is the processor 1014 shown in FIG. 1.
  • the process of deduplication and compression involves tasks such as data partitioning, fingerprint calculation, fingerprint search, data compression, and data storage.
  • the task of fingerprint calculation and data compression is performed by a dedicated processor, and other tasks involved in the deduplication and compression process are performed by the CPU, thereby offloading the task of fingerprint calculation and data compression from the CPU to the dedicated processor, reducing repetition.
  • Deletion and compression takes up CPU computing resources and accelerates the execution of the deduplication and compression process.
  • FIG. 3 is a flowchart of a data processing method 300 provided by an embodiment of the present application.
  • the method 300 includes S310 to S360.
  • the storage device obtains multiple data blocks.
  • S310 specifically includes the following steps S311 to S313.
  • the storage device receives a write request from the host.
  • the write request is used to request the storage device to store data, and the write request includes the data to be stored and the logical address of the data.
  • the storage device obtains data from the write request.
  • the storage device divides the data according to the first granularity to obtain multiple data blocks, and the size of each data block is equal to the size of the first granularity.
  • the first granularity is the deduplication granularity described above, that is, the granularity used when the storage device performs data deduplication processing.
  • the storage device determines the first granularity according to the metadata management granularity. For example, the storage device determines the minimum value of the metadata management granularity, and takes an integer multiple of the minimum value of the metadata management granularity as the first granularity. Optionally, the minimum value of the metadata management granularity is taken as the first granularity.
  • the storage device determines that the minimum metadata management granularity is 4KB, then determines that the size of the first granularity is 4KB, and divides the data into multiple 4KB data blocks .
  • the storage device determines the fingerprint of each data block in the multiple data blocks.
  • the storage device performs fingerprint calculation on each data block to obtain the fingerprint of each data block.
  • the fingerprint of the data block is the hash value of the data block, and the storage device performs a hash calculation on each data block to obtain the hash value of each data block.
  • the storage device determines repeated blocks and non-repeated blocks among multiple data blocks according to the fingerprint of each data block.
  • the fingerprint of the duplicate block is the same as the fingerprint of the data block already stored in the storage device.
  • the data contained in the repeated block is exactly the same as the data contained in the data block stored by the storage device; or the data contained in the repeated block is different from the same part of the data contained in the data block stored in the storage device. If the fixed-length deduplication method is used, the data contained in the duplicate block is exactly the same as the data contained in the data block stored by the storage device. If the similar deduplication method is adopted, only the data contained in the duplicate block must be stored in the storage device.
  • the data block contains the same data part.
  • Non-repeated blocks refer to data blocks other than repeated blocks in multiple data blocks. Non-duplicate blocks are also weighed as unsuccessful data blocks.
  • the storage device determines whether the data block is a repeated block or a non-repetitive block by executing the step of querying the fingerprint table.
  • step S330 is also called fingerprint query.
  • the fingerprint table is used to save the fingerprints of the data blocks stored in the storage device.
  • step S330 includes: the storage device queries the fingerprint table, and compares the fingerprint of the first data block with the fingerprint in the fingerprint table; if the fingerprint of the first data block is It is the same as a fingerprint in the fingerprint table (that is, the first data block hits the fingerprint table), the storage device determines that the first data block is a duplicate block; if the fingerprint of the first data block is different from each fingerprint in the fingerprint table (that is, the first data block is different from each fingerprint in the fingerprint table). If the data block misses the fingerprint table), the storage device determines that the first data block is a non-repetitive block.
  • S330 is performed locally in the storage device. In other embodiments, S330 is performed by the storage device and a dedicated server in cooperation.
  • the dedicated server is an independent device that is coupled to a storage device or connected via a network.
  • the server stores a fingerprint table and can be specifically responsible for fingerprint query work. Specifically, the storage device sends the fingerprint of each data block to the server according to preset rules, and the server determines to query the fingerprint table according to the fingerprint of each data block, thereby sharing the task of fingerprint query to the server, avoiding fingerprint query The amount of computation brings performance bottlenecks to storage devices.
  • step S330 is executed in real time after step S320 is completed.
  • the storage device will determine whether the deduplication trigger condition is currently met after S320 is executed, and if the deduplication trigger condition is met, then step S330 will be executed. For example, when the deduplication trigger condition is that the load is lower than the threshold, the storage device will determine whether the load is lower than the threshold after performing step S320; if the load is higher than the threshold, cache the obtained fingerprints and wait, if the load is lower than the threshold , Go to S330.
  • the operations performed by the storage device on duplicate blocks are different from the operations performed on non-duplicate blocks.
  • the storage device performs a deduplication operation on duplicate blocks and a compression operation on non-duplicate blocks.
  • the following is an example of how to perform a deduplication operation on a storage device through S340, and an example of how to perform a compression operation on a storage device through S350 and S360.
  • the pre-deduplication includes the case where the pre-deduplication is successful and the case where the pre-deduplication fails, and the case where the pre-deduplication fails refers to the case where the storage device does not perform deduplication but performs compression.
  • the storage device executes S340 instead of S350 and S360; in the case of successful pre-deduplication, if the storage device determines to store One part of the data block is a duplicate block and the other part is a non-duplicate block.
  • the storage device executes S340 and executes S350 and S360; in the case where the previous deduplication fails, the storage device executes S350 and S360 instead of S340.
  • FIG. 4 is an example of a case where the previous deduplication is successful.
  • FIG. 4 is an example of a case where the previous deduplication fails (without deduplication and compression).
  • the process involved in the scenario shown in FIG. 4 will be cited as an example.
  • the scenario shown in (a) in Figure 4 is specifically: the host initiates a write request for the 8 data blocks of data block 1, data block 2 to data block 8; after the storage device receives the write request, it responds to the 8 data blocks Perform fingerprint calculations on each block to obtain 8 fingerprints.
  • the 8 fingerprints are the fingerprint FP 1 of data block 1, the fingerprint FP 2 of data block 2 to the fingerprint FP 8 of data block 8.
  • the storage device searches the fingerprint table according to the 8 fingerprints, and finds fingerprints FP 4 and FP 7
  • the fingerprint table is hit, that is, the storage device finds the same fingerprint as the fingerprint FP 4 of the data block 4 in the fingerprint table, and finds the same fingerprint as the fingerprint FP 7 of the data block 7 in the fingerprint table.
  • the storage device determines that data block 4 and data block 7 are duplicate blocks; the storage device does not store data block 4 and data block 7, and records the fingerprint index FPI 4 corresponding to the fingerprint FP 4 of data block 4 in the storage unit 204, and The storage unit 207 records the fingerprint index FPI 7 corresponding to the fingerprint FP 7 of the data block 7.
  • the storage device determines that data blocks other than data block 4 and data block 7 are non-repetitive blocks.
  • the storage device compresses data block 1, data block 2, and data block 3 to obtain compressed block 1, and uses a piece of metadata to represent compressed block 1, and obtains metadata 1 of compressed block 1.
  • the storage device stores the metadata 1 of the compressed block 1 in three physically continuous and logically continuous storage units: the storage unit 201, the storage unit 202, and the storage unit 203.
  • the storage device compresses data block 5, data block 6, and data block 8 to obtain compressed block 2.
  • the metadata 2 of the compressed block 2 is stored in the three physically continuous but logically discontinuous storage units 206 and the storage unit 208.
  • the metadata 2 includes two parts, one part is the metadata 21, and the other part is the metadata 22.
  • the storage unit 205 and the storage unit 206 store the metadata 21, and the storage unit 208 stores the metadata 22.
  • the storage device will write compressed block 1 and compressed block 2 to the hard disk.
  • the scenario shown in (b) in Figure 4 is specifically: the host initiates a write request for the eight data blocks of data block 1, data block 2 to data block 8; after the storage device receives the write request, the storage device pairs 8 Perform fingerprint calculation on each data block to obtain 8 fingerprints. These 8 fingerprints are the fingerprint FP 1 of data block 1, the fingerprint FP 2 of data block 2 to the fingerprint FP 8 of data block 8.
  • the storage device has a load higher than the threshold or Deduplication failed due to other reasons. In this case, the storage device compresses data block 1, data block 2, and data block 3 to obtain compressed block 1, and uses a piece of metadata to represent compressed block 1, and obtains metadata 1 of compressed block 1.
  • the storage device stores the metadata 1 of the compressed block 1 in three physically continuous and logically continuous storage units: the storage unit 201, the storage unit 202, and the storage unit 203.
  • the storage device compresses data block 4, data block 5, and data block 6 to obtain compressed block 2, and uses a piece of metadata to represent compressed block 2 to obtain metadata 2 of compressed block 2.
  • the storage device stores the metadata 2 of the compressed block 2 in three physically continuous and logically continuous storage units, the storage unit 204, the storage unit 205, and the storage unit 206.
  • the storage device compresses the data block 7 and the data block 8 to obtain the compressed block 3, uses a piece of metadata to represent the compressed block 3, and obtains the metadata 3 of the compressed block 3.
  • the storage device stores the metadata 3 of the compressed block 3 in two physically and logically continuous storage units, the storage unit 207 and the storage unit 208.
  • the storage device writes compressed block 1, compressed block 2, and compressed block 3 to the hard disk.
  • the storage device records metadata for the repeated block.
  • the storage device records the metadata of the repeated block without saving the repeated block, thereby avoiding the repeated block from occupying the storage space of the hard disk, thereby saving the storage resources of the storage device.
  • the storage device also records the metadata of the non-repeated block, and saves the fingerprint of the non-repeated block to the fingerprint table, so that when the new block is subsequently deduplicated, the previous block can be found in the fingerprint table. The fingerprint of the saved non-duplicated block.
  • the storage device saves fingerprints of duplicate blocks. In some embodiments, the storage device saves the fingerprint index of the duplicate block.
  • the metadata recorded by the storage device for the repeated block is the fingerprint index (FPI) of the repeated block.
  • the storage device uses the fingerprint index of the repeated block as the metadata of the repeated block, and writes the fingerprint index of the repeated block to the storage unit of the metadata, thereby storing the fingerprint index in the storage unit of the metadata. For example, referring to Figure 4, after the storage device determines that data block 4 and data block 7 are duplicate blocks, it writes the fingerprint index FPI 4 of data block 4 to the storage unit 204, and writes the fingerprint index FPI 7 of data block 7 to the storage unit 204.
  • Storage unit 207 storage unit 207.
  • the size of the storage space occupied by the metadata of the repeated block is equal to the size of the first granularity.
  • the storage space occupied by the metadata of the repeated block is equal to the minimum value of the metadata management granularity.
  • the minimum value of metadata management granularity corresponds to a grid in FIG. 4, and the minimum value of metadata management granularity is, for example, the size of a storage unit.
  • the fingerprint index FPI 4 of the data block 4 occupies the storage unit 204, and the size of the storage space occupied by the fingerprint index FPI 4 of the data block 4 is equal to the size of the storage unit 204.
  • the storage device selects the first storage unit according to the metadata management granularity, and stores the metadata of the repeated block in the first storage unit.
  • the granularity of the first storage unit is the minimum value of the metadata management granularity. For example, if the metadata management granularity is [4KB, 1M], the storage device selects a storage unit with a size of 4KB, and stores the metadata of the repeated block in the 4KB storage unit.
  • the storage device compresses multiple non-repeated blocks to obtain compressed blocks.
  • a compressed block refers to a non-repetitive block after compression.
  • S350 specifically includes the following steps a to c.
  • Step a The storage device obtains the compression rate of the data.
  • How to obtain the compression ratio includes multiple implementation methods.
  • machine learning is used to perform model training based on samples to obtain a prediction model. After the data is obtained, the data is input to the prediction model, the data compression rate is predicted through the prediction model, and the compression rate is output.
  • the compression rate is preset by the user based on experience. In other embodiments, considering that there is a high probability that the compression rate of two consecutive data blocks is the same, the compression rate of the previous data is used as the compression rate of the current data to be compressed.
  • Step b The storage device divides the multiple non-repeated blocks into at least one data block group according to the predicted compression rate, and the number of non-repeated blocks contained in each data block group is equal to the compression rate.
  • the storage device divides 3 non-repeated blocks into a data block group, so that a data block group contains 3 non-repeated blocks, so that the 3 non-repeated blocks can be compressed together ;
  • the predicted compression ratio is 4:1, the storage device divides the 4 non-repeated blocks into a data block group, so that a data block group contains 4 non-repeated blocks, so that the 4 non-repeated blocks are compressed together.
  • the three grids in Figure 4 may correspond to one compressed block.
  • the storage device converts the data block 1.
  • Data block 2 and data block 3 are divided into one group, and data block 5, data block 6 and data block 8 are divided into another group.
  • the storage device divides data block 1, data block 2, and data block 3 into one group, and divides data block 4, data block 5, and data block 6 into another group. Divide data block 7 and data block 8 into one group.
  • the storage device not only considers the compression rate when grouping, but also considers whether different non-repeated blocks are continuous. For example, the storage device divides consecutive non-repeated blocks into the same data block group according to the compression rate and the address of each non-repeated block.
  • continuous includes but is not limited to at least one of physical continuous or logical continuous. Whether the physical is continuous is determined, for example, based on whether the physical address is continuous, and whether the logical is continuous, for example, is determined based on whether the logical address is continuous.
  • Step c The storage device compresses each data block group into a compressed block.
  • the storage device compresses data block 1, data block 2, and data block 3 into compressed block 1, and compresses data block 5, data block 6, and data block 8 into compressed block 2.
  • the storage device compresses data block 1, data block 2, and data block 3 into compressed block 1, and compresses data block 4, data block 5, and data block 6 into compressed block 2.
  • the second granularity is equal to the product of the number of non-repeated blocks contained in a data block group and the granularity of the non-repeated blocks. . Since the number of non-repeated blocks contained in a data block group is equal to the compression rate, and the granularity of the non-repeated blocks is equal to the minimum value of the metadata management granularity, the second granularity is equal to the product of the compression ratio and the minimum value of the metadata management granularity. For example, referring to Fig. 4, the compression ratio is equal to 3:1, the metadata management granularity corresponds to one grid in Fig. 4, and the second granularity corresponds to the three grids in Fig. 4. For example, the second granularity is equal to the sum of the sizes of the storage unit 201, the storage unit 202, and the storage unit 203.
  • the compression granularity (that is, the second granularity) provided in this embodiment is no longer the same fixed value as the deduplication granularity (that is, the first granularity), but is determined according to the minimum value of the metadata management granularity and the compression ratio. , So it supports the function of dynamic selection of compression granularity. Through the dynamic selection of compression granularity, it is helpful to ensure that the compression rate is better under the condition that the data reading performance does not decrease, and to obtain a balance between the maximum compression rate and the read amplification. The following describes the technical principles to achieve this technical effect in conjunction with a specific example.
  • the minimum value of the metadata management granularity is 8KB
  • the storage device obtains four data blocks of 8KB in size.
  • the four data blocks are data block a, data block b, data block c, and data block d.
  • the storage device predicts that the compression ratio of the 4 data blocks is 4:1, the storage device determines that the compression granularity is 32KB, and compresses the 32KB data of data block a, data block b, data block c, and data block d together to obtain one 8KB compressed block e is stored in the hard disk. After that, the storage device receives the read request, and the read request instructs the storage device to read the data block a.
  • the storage device In response to the read request, the storage device reads 8KB compressed block e from the hard disk; the storage device decompresses the compressed block e to obtain data block a, data block b, data block c, and data block d. Return to the initiator of the read request. It can be seen from this example that the data granularity corresponding to the read request is 8KB, and the data granularity actually read from the hard disk by the storage device (that is, the granularity of compressed block e) is also 8KB. It can be seen that the data granularity actually read from the hard disk corresponds to the read request. The data granularity is the same, and there is no read amplification, which avoids the degradation of read performance and bandwidth resource consumption caused by read amplification. In addition, since the compression granularity is a better 32KB, it helps to achieve a better compression rate.
  • the storage device stores the compressed block in the hard disk, and records metadata for the compressed block.
  • the storage device stores the data (compression block) that has undergone deduplication processing and compression processing in the hard disk of the storage device.
  • the metadata of the compressed block indicates the mapping relationship between the logical address of the data and the physical address of the compressed block.
  • the storage device stores the metadata of the compressed block in a storage unit whose size is [minimum value of metadata management granularity, compression granularity].
  • the minimum value of the granularity of the metadata of the compressed block is the minimum value of the metadata management granularity
  • the maximum value of the granularity of the metadata of the compressed block is the compression granularity (second granularity).
  • the granularity of the metadata of the compressed block is the product of the minimum value of the metadata management granularity and the compression rate.
  • the storage device selects N storage units, and the metadata of the compressed block is stored in the N storage units.
  • the compression ratio is 3: 1
  • the minimum metadata management granularity is the size of a storage unit
  • the storage device will select 3 storage units to record the metadata of a compressed block
  • the metadata of the compressed block The particle size is three times the size of the storage unit.
  • the storage device selects storage unit 201, storage unit 202, and storage unit 203 to record metadata 1 of compressed block 1.
  • the granularity of metadata 1 of compressed block 1 is one of the sizes of storage unit 201, storage unit 202, and storage unit 203. and.
  • the manner in which the storage device records metadata specifically includes at least one of the following manner 1 to manner 2.
  • Method A The storage device records a piece of metadata for multiple compressed blocks with consecutive addresses.
  • the storage device uses one piece of metadata to represent the multiple compressed blocks.
  • multiple compressed block addresses are continuous, for example, multiple compressed block physical addresses are continuous and multiple compressed block logical addresses are continuous.
  • a piece of metadata recorded by the storage device includes two parts, one part is the address of the first compressed block in the multiple compressed blocks with consecutive addresses, and the other part is the address of the first compressed block in the multiple compressed blocks with consecutive addresses. The length of a compressed block. For example, referring to FIG. 7, a small cell in FIG. 7 identifies a data block or metadata with the smallest granularity, such as a 4KB data block or a 4KB metadata. Fig.
  • the 7 is an example of how to use a piece of metadata to represent compressed block 1, compressed block 2, and compressed block 3.
  • the recorded piece of metadata contains compressed block 1 (the first The metadata 1 of the compressed block) and a jump table.
  • the metadata 1 indicates the address of the compressed block 1.
  • the jump table includes the length of the compressed block 1, the length of the compressed block 2, and the length of the compressed block 3. For example, if the length of compressed block 1 is 9KB, the length of compressed block 2 is 7KB, and the length of compressed block 3 is 4KB, the jump table includes 9KB, 7KB, and 4KB.
  • the offset of each compressed block relative to the first compressed block can be specified.
  • the offset of the second compressed block relative to the first compressed block is the length of the first compressed block
  • the offset of the third compressed block relative to the first compressed block is the offset of the first compressed block and The sum of the length of the second compressed block. Therefore, when the second compressed block needs to be read, the second compressed block can be addressed in the hard disk according to the address of the first compressed block and the offset of the second compressed block relative to the first compressed block.
  • the third compressed block when the third compressed block needs to be read, the third compressed block can be addressed in the hard disk according to the address of the first compressed block and the offset of the third compressed block relative to the first compressed block piece.
  • the storage device when the compressed block 2 needs to be read, the storage device offsets the address of the metadata 1 of the compressed block 1 by 9KB, and the compressed block 2 can be found from the hard disk.
  • Method B The storage device records multiple pieces of metadata for multiple compressed blocks with discontinuous addresses.
  • the storage device uses multiple pieces of metadata to respectively represent the multiple compressed blocks.
  • one piece of metadata is recorded for two or more compressed blocks with consecutive addresses.
  • the logical interval of the multiple compressed blocks with discontinuous addresses corresponds to the fingerprint index of the repeated block, for example.
  • compressed block 2 is obtained by compressing three non-repeated blocks of data block 5, data block 6, and data block 8.
  • data block 5 and data block 6 are two logically adjacent blocks.
  • Data block 6 and data block 8 are logically separated. Specifically, the logical address of the data block 5 and the logical address of the data block 6 are continuous, and the logical address of the data block 6 and the logical address of the data block 8 are not continuous.
  • the logical address of data block 5 is 205 and the length is 8KB
  • the logical address of data block 6 is 206 and the length is 8KB.
  • the logical address of data block 8 is 208, and the length is 8KB.
  • the storage device compresses data block 5, data block 6, and data block 8 into compressed block 2.
  • the metadata 2 recorded for compressed block 2 includes two metadata, metadata 21 and metadata 22.
  • the metadata 22 indicates that the starting logical address is 208 and the length is 8KB.
  • the same compressed block corresponds to two metadata (metadata 21 and metadata 22).
  • the metadata 21 and the metadata 22 correspond to different parts of the decompressed data block. Specifically, after compressed block 2 is decompressed, data block 5, data block 6 and data block 8 are obtained, metadata 21 corresponds to data block 5 and data block 6, and metadata 22 corresponds to data block 8.
  • the storage device uses different granularities for deduplication and compression, it gets rid of the restriction that the deduplication granularity and the compression granularity must be the same, and to a certain extent avoids the reduction of the deduplication rate due to excessive granularity.
  • the situation and the situation where the granularity is too small leads to a decrease in the compression rate, thereby increasing the overall reduction rate of deduplication and compression.
  • the deduplication granularity and the compression granularity are determined according to the metadata management granularity, the deduplication and compression are operated at a better granularity respectively, which helps to achieve better deduplication and compression ratios.
  • FIG. 5 is a flowchart of a data processing method 400 provided by an embodiment of the present application.
  • the method 400 includes S410 to S450.
  • the storage device stores multiple data blocks in the hard disk.
  • the storage device also calculates the fingerprint of each data block, and saves the fingerprint of the data block and the physical location of the data block in the opportunity table.
  • the opportunity table is, for example, in the form of key-value pairs, and the key of the opportunity table is the fingerprint of the data block.
  • the value of the opportunity table is the physical location of the data block.
  • the chance table is used to find the fingerprints of duplicate blocks.
  • the storage device first compresses multiple data blocks to obtain compressed blocks, and then stores the compressed blocks in the hard disk.
  • the compression process is similar to the compression process described in the above method 300. Specifically, the storage device predicts the compression rate of the data during the compression process; the storage device determines the second granularity according to the predicted compression ratio and the metadata management granularity, and performs compression according to the second granularity. For example, the storage device determines the product of the minimum value of the metadata management granularity and the compression rate as the second granularity.
  • the storage device determines duplicate blocks and non-duplicate blocks among the multiple data blocks.
  • the storage device reads the fingerprint of each data block from the opportunity table, and compares the fingerprint of the data block with the fingerprints already recorded in the fingerprint table. If the fingerprint of the data block is the same as a fingerprint recorded in the fingerprint table, the storage device determines that the data block is a duplicate block. If the fingerprint of the data block is different from each fingerprint recorded in the fingerprint table, the storage device determines that the data block is a non-repetitive block, and the storage device records the fingerprint of the data block in the fingerprint table.
  • the storage device updates the recorded metadata for the repeated block.
  • the storage device After the storage device finds a duplicate block that can be deduplicated, it updates the metadata of the duplicate block. In some embodiments, the storage device updates the metadata of the repeated block to the fingerprint of the repeated block. For example, the storage device determines the second storage unit, the second storage unit is used to store the metadata of the repeated block, and the granularity of the second storage unit is the minimum value of the metadata management granularity. The storage device writes the fingerprint of the repeated block to the second storage unit, so that the metadata stored in the second storage unit is overwritten from the previously recorded metadata to the fingerprint of the repeated block.
  • the metadata and data block of data block 4 are updated 7 metadata.
  • the data storage device using the data block fingerprint of an FPI index 4 4 4 as metadata block the data block is overwritten 4 fingerprint index storage unit 204 to an FPI 4, so that the storage unit 204 from the data storage block 4 metadata refreshes fingerprint index FPI 4; furthermore, the storage device with the data block fingerprint of an FPI index 7 7 7 as metadata of the data block, the data block is overwritten 7 to an FPI fingerprint index storage unit 2077, so that the storage unit 207 The saved data is refreshed from the metadata of the data block 4 to the fingerprint index FPI 7 .
  • Figure 6 (a) is an example of metadata before post-deduplication
  • Figure 6 (b) is an example of metadata to be updated
  • Figure 6 (c) is For an example of the merged metadata
  • (d) in Figure 6 is an example of the defragmented metadata.
  • the storage device performs garbage collection on the duplicate blocks.
  • the duplicate block can be called garbage data, and the storage device deletes the duplicate block to release the storage space occupied by the duplicate block.
  • the storage device updates the metadata stored in the storage unit 204 to the fingerprint index FPI 4 of data block 4, and updates the metadata stored in the storage unit 207 to the fingerprint index FPI 7 of data block 7, and then deletes the data Block 4 and Data Block 7.
  • the storage device reads both the duplicate block and the non-duplicate block from the hard disk, erases the duplicate block and the storage unit that the non-duplicate block originally occupied in the hard disk, and then rewrites the non-duplicate block To the hard disk for garbage collection.
  • repeated blocks include compressed blocks and ordinary data blocks. Therefore, in the garbage collection process, compressed blocks may be overwritten, or ordinary data blocks may be overwritten.
  • compressed blocks For ordinary data blocks, the storage device only needs to move out the valid data.
  • compressed blocks the storage device needs to first decompress the compressed blocks of variable length granularity, and then recompress the valid part of the decompressed data. As the granularity of the compressed block becomes smaller, the compression rate will decrease at this time, and the storage device will execute the following S450 to perform a defragmentation process to achieve maximum granularity compression.
  • the storage device defragments the non-repeated blocks.
  • the storage device stores the data (non-repeated blocks after defragmentation) that have undergone deduplication processing and compression processing in the hard disk of the storage device.
  • the storage device retains the fingerprint index of the duplicate block.
  • the data to be sorted includes, for example, data that has not been compressed according to the second granularity. If the data to be sorted is a compressed block, the storage device determines whether the compression granularity (ie, the second granularity) corresponding to the compressed block is smaller than the better compression granularity (for example, the product of the minimum metadata management granularity and the compression ratio).
  • the compression granularity (ie, the second granularity) corresponding to the block is smaller than the better compression granularity, and the multiple compressed blocks are recompressed according to the better compression granularity, thereby increasing the compression ratio.
  • the storage device first decompresses the compressed block to obtain non-repeated blocks. After that, the storage device determines the compression granularity according to the minimum metadata management granularity and the compression ratio in the same manner as in step SF50.
  • the storage device divides multiple non-repetitive blocks into at least one data block group, and compresses each data block group into a compressed block, thereby recompressing.
  • the compression ratio is 3:1
  • the minimum value of metadata management granularity is equal to the size of one storage unit
  • the better compression granularity is equal to the size of three storage units.
  • the compression granularity corresponding to compressed block 2 is the size of 2 storage units
  • the compression granularity corresponding to compression block 3 is determined to be the size of 1 storage unit.
  • the compression granularity corresponding to compressed block 2 and compressed block 3 are both smaller than the better compression granularity, and the storage device decompresses compressed block 2 to obtain data block 5 and data block 6, and decompresses compressed block 3 to obtain data block 8.
  • the storage device recompresses data block 5, data block 6, and data block 8 to obtain a new compressed block 2, and writes the new compressed block 2 to the hard disk.
  • the compression granularity corresponding to the new compression block 2 is the size of three storage units, which is better than the compression granularity corresponding to the compression block 2 and the compression granularity corresponding to the compression block 3, thereby helping to improve the compression ratio.
  • the storage device uses different granularities for deduplication and compression, which gets rid of the limitation that the first granularity and the second granularity must be the same, and to a certain extent avoids the reduction of the deduplication rate due to excessive granularity.
  • the situation and the situation that the granularity is too small leads to a decrease in the compression rate, thereby increasing the overall reduction rate of deduplication and compression.
  • the deduplication and compression are operated according to the better granularity respectively, which helps to achieve a better deduplication rate and compression rate.
  • FIG. 8 shows a schematic diagram of a possible structure of the storage device involved in the foregoing embodiment.
  • the storage device 600 shown in FIG. 8 implements the functions of the storage device in the method 300 or the method 400, for example.
  • the storage device 600 includes an acquisition module 601, a deduplication module 602, a compression module 603, and a storage module 604.
  • the obtaining module 601 is used to obtain data; the deduplication module 602 is used to perform deduplication processing on the data based on the first granularity; the compression module 603 is used to compress data based on the second granularity, and the size of the second granularity is greater than The size of the first granularity; the storage module 604 is used to store the data that has undergone deduplication processing and compression processing in the hard disk of the storage device.
  • the storage device further includes a recording module for recording metadata of the compressed block.
  • At least one module in the storage device 600 is integrated in a processor, the same chip, or the same board.
  • the acquisition module 601, the deduplication module 602, and the compression module 603 are all integrated in the same processor, and the functions of the acquisition module 601, the deduplication module 602, and the compression module 603 are realized by the processor.
  • different modules of the storage device 600 are implemented by different processors or other different hardware.
  • the acquisition module 601 is implemented by the network card 1011 shown in FIG. 1
  • the functions of the deduplication module 602 and the compression module 603 are respectively implemented by different dedicated processors
  • the functions of the storage module 604 are implemented by a central processing unit.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or modules, and may also be electrical, mechanical or other forms of connection.
  • modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments of the present application.
  • modules in the various embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or software modules.
  • the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of this application is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium It includes several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .
  • first, second and other words are used to distinguish the same or similar items with basically the same function and function. It should be understood that there is no logic or sequence between “first” and “second” The dependence relationship on the above does not limit the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another.
  • the first particle size may be referred to as the second particle size
  • the second particle size may be referred to as the first particle size.
  • Both the first particle size and the second particle size may be particle sizes, and in some cases, may be separate and different particle sizes.
  • the computer can be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
  • software it can be implemented in the form of a computer program product in whole or in part.
  • the computer program product includes one or more computer program instructions.
  • the computer program instructions When the computer program instructions are loaded and executed on the computer, the processes or functions in the embodiments of the present application are generated in whole or in part.
  • the computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.
  • the computer program instructions can be passed from a website, computer, server, or data center. Wired or wireless transmission to another website site, computer, server or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a digital video disc (DVD), or a semiconductor medium (for example, a solid state hard disk).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种数据处理方法及存储设备，属于存储技术领域。存储设备在重复数据删除处理和压缩处理时采用了不一样的粒度，采用小粒度对所述数据进行重复数据删除处理，采用大粒度对所述数据进行压缩处理，因此摆脱了重删粒度和压缩粒度必须相同的限制，一定程度上避免粒度过大而导致重删率下降以及粒度过小而导致压缩率下降，有助于提升重删压缩整体的缩减率。

Description

数据处理方法及存储设备
本申请要求于2020年06月11日提交的申请号为202010526840.7、发明名称为“一种存储系统、存储节点和数据存储方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
本申请要求于2020年08月06日提交的申请号为202010784929.3、发明名称为“数据处理方法及存储设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及存储技术领域，特别涉及一种数据处理方法及存储设备。
背景技术
重删和压缩是存储行业中的关键技术。存储设备通过进行重删和压缩，能够减少实际存储的数据规模，节约数据在存储设备中占用的存储空间，提高存储设备的存储效率。
目前，存储设备会预先设置一个固定的粒度，基于该粒度进行重删，并基于该粒度进行压缩。例如，预先将粒度设定为8千字节(Kilobyte，KB)，则存储设备在进行重删时，会判断每8KB的数据块是否是重复块，如果某个8KB的数据块是重复块，则存储设备删除该8KB的数据块；并且，存储设备在进行压缩时，每一次压缩8KB的数据块。
采用上述方法处理数据时，重删粒度和压缩粒度必须是相同的，可见该方法局限性强。
发明内容
本申请实施例提供了一种数据处理方法及存储设备，一定程度上解决数据处理方法面临的局限性。所述技术方案如下：
第一方面，提供了一种数据处理方法，该方法由存储设备执行，包括：获取数据；基于第一粒度对所述数据进行重复数据删除处理；基于第二粒度对所述数据进行压缩处理，所述第二粒度的尺寸大于所述第一粒度的尺寸；将经过所述重复数据删除处理和所述压缩处理的数据存储在所述存储设备的硬盘中。
在第一方面提供的方法中，由于存储设备在重复数据删除处理和压缩处理时采用了不一样的粒度，采用大粒度进行重复数据删除处理，采用小粒度进行压缩处理，因此摆脱了重删粒度和压缩粒度必须相同的限制，一定程度上避免粒度过大而导致重删率下降以及粒度过小而导致压缩率下降，有助于提升重删压缩整体的缩减率。
在第一方面中，本申请并不限定所述重复数据删除处理和所述压缩处理的先后顺序。某些场景下可以先执行压缩处理再执行重复数据删除处理，在某些场景下可以先执行重复数据删除处理再执行压缩处理。以先执行重复数据删除处理再执行压缩处理为例，在经过重复数据删除处理之后得到重复块和非重复块，本申请可以只对非重复块进行压缩处理。如果先执行压缩处理再执行重复数据删除处理，就是对所述数据进行压缩以后得到压缩块，再针对所述压缩块进行重复数据删除处理。
可选地，所述存储设备中存储有元数据，所述元数据是基于元数据管理粒度来管理的，所述元数据管理粒度的尺寸小于或等于设定的最大值，并且大于或等于设定的最小值，所述第一粒度的尺寸等于所述最小值的整数倍。
通过这种方式，由于将元数据管理粒度的最小值作为重复数据删除处理时使用的粒度，有助于重复数据删除处理时取到较优粒度，从而提高重删率并节省存储资源。
可选地，所述第二粒度的尺寸是所述最小值与压缩率的乘积。
通过这种方式，由于压缩处理时采用的粒度不再是固定的值，而是根据压缩率动态选择的，从而保证数据读取性能不下降的情况下压缩率较优。
可选地，所述基于第一粒度对所述数据进行重复数据删除处理包括：将所述数据划分为多个数据块；获取每个数据块的指纹；根据所述指纹，从所述多个数据块中确定重复块以及非重复块。
可选地，所述基于第二粒度对所述数据进行压缩处理包括：基于所述第二粒度对所述非重复块进行压缩处理，得到压缩块，所述经过所述重复数据删除处理和所述压缩处理的数据包括所述压缩块。
可选地，所述方法还包括：记录所述压缩块的元数据。
可选地，所述记录所述压缩块的元数据包括：如果所述压缩块的数量为多个且多个压缩块的地址连续，为所述多个压缩块记录一条元数据。
通过为多个地址连续的压缩块记录一条元数据，减少了记录的元数据的数量，从而节省元数据在存储设备中占用的存储资源。
可选地，所述多个压缩块的地址连续是指多个压缩块的物理地址连续且逻辑地址连续。
可选地，所述一条元数据包括所述多个压缩块中第一个压缩块的地址以及每个压缩块的长度。
通过采用这样的方式记录元数据，在能够通过元数据读取数据的基础上，达到压缩的元数据空间较优的效果。
可选地，所述数据在经过所述重复数据删除处理和所述压缩处理之前还基于第三粒度压缩过，所述第三粒度的尺寸小于所述第二粒度的尺寸。
通过这种方式，由于存储设备在发现压缩块原来的压缩粒度(第三粒度)并不是较优的压缩粒度(第二粒度)时，重新按照较优的压缩粒度(第二粒度)重新进行压缩，使得压缩块的压缩粒度得到优化，因此有助于提升压缩率。
可选地，所述存储设备是存储阵列。
可选地，所述存储设备是分布式存储系统中的一个存储节点。
可选地，所述第一粒度为4KB，所述第二粒度为32KB。
第二方面，提供了一种数据处理方法，该方法由存储设备执行，包括：获取数据；基于元数据管理粒度，确定第一粒度，所述元数据管理粒度是用于管理所述存储设备存储的元数据的粒度，所述元数据管理粒度的尺寸小于或等于设定的最大值并且大于或等于设定的最小值，所述第一粒度的尺寸等于所述最小值的整数倍；基于所述第一粒度对所述数据进行重复数据删除处理；将经过所述重复数据删除处理的数据存储在所述存储设备的硬盘中。
在第二方面提供的方法中，由于存储设备重复数据删除处理时采用的粒度是基于元数据 管理粒度确定的，通过将元数据管理粒度的最小值作为重复数据删除处理时使用的粒度，有助于重复数据删除处理时取到较优粒度，从而提高重删率并节省存储资源。
可选地，所述基于第一粒度对所述数据进行重复数据删除处理包括：将所述数据划分为多个数据块；获取每个数据块的指纹；根据所述指纹，从所述多个数据块中确定重复块以及非重复块。
可选地，所述第一粒度为4KB。
第三方面，提供了一种数据处理方法，该方法由存储设备执行，包括：获取数据；基于元数据管理粒度，确定第二粒度，所述元数据管理粒度是用于管理所述存储设备存储的元数据的粒度，所述元数据管理粒度的尺寸小于或等于设定的最大值并且大于或等于设定的最小值，所述第二粒度的尺寸是所述最小值与设定的压缩率的乘积；基于所述第二粒度对所述数据进行压缩处理；将经过所述压缩处理的数据存储在所述存储设备的硬盘中。
在第三方面提供的方法中，由于存储设备压缩处理时采用的粒度不再是固定的值，而是根据元数据管理粒度以及压缩率动态确定的，从而保证数据读取性能不下降的情况下压缩率较优。
可选地，所述第二粒度的尺寸是所述最小值与压缩率的乘积。
可选地，所述基于第二粒度对所述数据进行压缩处理包括：
基于所述第二粒度对所述非重复块进行压缩处理，得到压缩块，所述经过所述重复数据删除处理和所述压缩处理的数据包括所述压缩块。
可选地，所述方法还包括：记录所述压缩块的元数据。
可选地，所述记录所述压缩块的元数据包括：
如果所述压缩块的数量为多个且多个压缩块的地址连续，为所述多个压缩块记录一条元数据。
可选地，所述一条元数据包括所述多个压缩块中第一个压缩块的地址以及每个压缩块的长度。
可选地，所述数据在经过所述重复数据删除处理和所述压缩处理之前还基于第三粒度压缩过，所述第三粒度的尺寸小于所述第二粒度的尺寸。
可选地，所述方法还包括：保存所述重复块的指纹。
可选地，所述方法还包括：记录所述非重复块的元数据，并将所述非重复块的指纹保存至指纹表。
可选地，所述第二粒度为32KB。
第四方面，提供了一种存储设备，该存储设备包括至少一个处理器和硬盘，该至少一个处理器用于执行指令，使得该存储设备执行上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面、第三方面中任一种可选方式中的至少一项所提供的数据处理方法，所述硬盘用于存储数据。第四方面提供的存储设备的具体细节可参见上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面、第三方面中任一种可选方式中的至少一项，此处不再赘述。
在一些实施例中，该至少一个处理器包括第一处理器、第二处理器以及第三处理器；
所述第一处理器，用于获取数据；
所述第二处理器，用于基于第一粒度对所述数据进行重复数据删除处理；
所述第三处理器，用于基于第二粒度对所述数据进行压缩处理，所述第二粒度的尺寸大于所述第一粒度的尺寸；
所述第一处理器，还用于将经过所述重复数据删除处理和所述压缩处理的数据存储在所述硬盘中。
第五方面，提供了一种存储设备，该存储设备具有实现上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面、第三方面中任一种可选方式中的至少一项中数据处理的功能。存储设备包括至少一个模块，至少一个模块用于实现上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面、第三方面中任一种可选方式中的至少一项所提供的数据处理方法。
在一些实施例中，存储设备中的模块通过软件实现，存储设备中的模块是程序模块。在另一些实施例中，存储设备中的模块通过硬件或固件实现。第五方面提供的存储设备的具体细节可参见上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面或者第三方面中任一种可选方式，此处不再赘述。
第六方面，提供了一种计算机可读存储介质，该存储介质中存储有至少一条指令，该指令由处理器读取以使存储设备执行上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面、第三方面中任一种可选方式中的至少一项所提供的数据处理方法。
第七方面，提供了一种计算机程序产品，该计算机程序产品包括计算机指令，该计算机指令存储在计算机可读存储介质中。存储设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该存储设备执行上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面、第三方面中任一种可选方式中的至少一项所提供的数据处理方法。
第八方面，提供了一种芯片，当该芯片在存储设备上运行时，使得存储设备执行上述第一方面、第一方面中任一种可选方式、第二方面、第二方面中任一种可选方式、第三方面、第三方面中任一种可选方式中的至少一项所提供的数据处理方法。
附图说明
图1是本申请实施例提供的一种分布式存储系统的系统架构的示意图；
图2是本申请实施例提供的一种存储元数据的示意图；
图3是本申请实施例提供的一种数据处理方法的流程图；
图4是本申请实施例提供的一种前重删的示意图；
图5是本申请实施例提供的一种数据处理方法的流程图；
图6是本申请实施例提供的一种后重删的示意图；
图7是本申请实施例提供的一种为多个压缩块记录一条元数据的示意图；
图8是本申请实施例提供的一种存储设备的结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
由于本申请实施例涉及重删压缩技术的应用，为了便于理解，下面先对本申请实施例涉及的重删压缩技术中的术语相关概念进行介绍。
(1)重删(deduplicate)
重删，是重复数据删除的简称，它是一种数据缩减技术。在存储系统中存储了很多重复数据，这些数据占用了大量硬盘空间，利用重删技术，能够删除重复数据，对于相同数据仅存储一份，从而节约数据存储空间。重删的技术原理包括：首先执行分块，即，将待写入的数据划分为多个数据块；然后进行指纹计算，即，根据每个划分的数据块，计算数据块的指纹；然后进行指纹查找，即，以指纹为索引，在指纹表中进行查找比对。如果指纹表存在相同的指纹，说明为重复块，则不保存数据块，而是保存数据块的指纹索引。如果指纹表不存在相同的指纹，表明数据块是一个非重复块(也称唯一块)，则保存数据块并创建数据块的元数据。按照执行时机的不同，重删分为前重删和后重删，以下通过(2)和(3)进行介绍。按照查重方式的不同，重删分为定长重删和相似重删，以下通过(4)和(5)进行介绍。
(2)前重删
前重删是指在数据写入硬盘之前进行重删。前重删也称在线重删。
(3)后重删
后重删是指在数据写入硬盘之后进行重删。后重删也称离线重删。后重删的实现方式包括多种。在一些实施例中，将数据写入硬盘之后，将数据从硬盘读取到缓存中；为缓存中的数据进行指纹计算，通过比较指纹是否相同从而确定重复数据；如果发现重复数据则进行重删，再将重删后的数据重新写入硬盘中。在一些实施例中，当获取到待存储的数据时，计算数据的指纹；将数据写入到硬盘中，并将指纹保存至机会表中。当需要进行重删时，从机会表中读取指纹，对不同指纹进行比较从而确定重复数据；如果发现重复数据则进行重删。
(4)定长重删
采用定长重删的方式时，要求不同数据块必须完全相同才能将数据块作为重复块并重删。并且，在执行分块时，会按照预先设定的粒度对数据进行划分，在进行指纹查找时，数据是按粒度对齐的。
(5)相似重删
采用相似重删的方式时，不要求数据块完全匹配，两个数据块相似即可判定数据块为重复块。并且，在执行分块时，也会按照预先设定的粒度对数据进行划分。
(6)指纹(fingerprint，FP)
指纹是数据块的本质特征。数据块本身往往较大，因此指纹的目标是期望以较小的数据表示(如16、32、64、128字节)来区别不同数据块。在一些实施例中，采用哈希算法计算数据块的指纹，数据块的指纹为数据库的哈希值。在理想情况下，每个数据块具有唯一的指纹，不同的数据块具有不同的指纹。当然，在存在哈希碰撞的情况下，不同数据块也可能具有相同的指纹。
(7)压缩
压缩是一种字节级的数据缩减技术，其思想是采用编码技术，将较长的数据用较短的、 经过编码的格式来表示，以此达到减少数据大小的目的。
(8)压缩率
压缩率是一个大于或等于1的正整数。压缩率表示数据在压缩前的数据量与压缩后的数据量之间的比值。例如，将32KB的数据压缩为8KB，则压缩率为4:1。
(9)重删压缩
重删压缩是指重删和压缩搭配一起使用的缩减技术。在采用重删压缩的方案时，当查找到重复块以及非重复块后，会先对多个非重复块进行压缩，保存压缩后的块。当读取数据时，会对压缩后的块进行解压缩。由于非重复块经过了压缩，使得数据的缩减效果为重删效果和压缩效果的叠加，缩减掉的数据更多。
(10)读放大
读放大是指从硬盘中实际读取的数据粒度大于读请求对应的数据粒度的情况。读放大会带来网络带宽资源的消耗，影响数据读取的性能。例如，硬盘存储数据的粒度是8KB。存储设备接收读请求，读请求指示存储设备读取4KB数据；存储设备从硬盘中读取4KB数据所在的8KB数据块，从8KB数据块中确定被请求读取的4KB数据，将4KB数据返回给读请求的发起方。在这个例子中，读请求对应的数据粒度是4KB，而实际读取的数据粒度是8KB，由于额外读取4KB数据，会过多占用带宽资源并影响数据读取性能。
(11)指纹索引(fingerprint index，FPI)
指纹索引即数据块的指纹的索引。指纹索引用于查询对应的指纹。为了简明起见，本申请实施例后续在不至于引入理解困难的情况下用“FPI 数字”的形式来简化表示具体的FPI，数字的含义是对应数据块的标识，如FPI 4表示数据块4的指纹索引。
(12)存储单元
存储单元是指存储设备中能够进行存储操作的最小单元，存储操作包括数据写入操作或数据读写操作。例如，存储单元是存储设备的一个扇区。
(13)物理地址
物理地址是指数据在硬盘中存储的实际地址。具体地，硬盘上的每个区域是由区域所在的磁头(Heads)、柱面(Cylinder，即磁道)、扇区(Sector)确定的。物理地址包括三个参数，分别是磁头参数、柱面参数以及扇区参数。磁头参数用于标识数据所在的磁头。柱面参数用于标识数据所在的柱面，扇区参数用于标识数据所在的扇区。通过数据的物理地址，能够指明硬盘应当用哪个磁头、从哪个柱面上哪个扇区上读取数据。
(14)逻辑地址
逻辑地址和物理地址不同。逻辑地址的全称是逻辑区块地址(Logical Block Address，LBA)。通过采用LBA作为数据的地址，将物理地址这种基于磁头、柱面和扇区的三维寻址方式转变为一维的线性寻址，从而提高了寻址的效率。
逻辑地址是存储设备呈现给主机的逻辑空间的地址。主机在向存储设备发送写请求或读请求时，会将逻辑地址携带在写请求或读请求中。存储设备接收到送写请求或读请求时，会获取写请求或读请求携带的逻辑地址，对逻辑地址经过一次或多次地址转换确定物理地址，向物理地址写入数据或者从物理地址读取数据。
逻辑地址连续例如是LBA连续。例如，数据块1的LBA是201，数据块2的LBA是202，数据块3的LBA是203，可称数据块1、数据块2和数据块3的逻辑地址连续。
(15)元数据
元数据是描述业务数据属性的数据。例如，元数据描述指纹、逻辑地址、物理地址、逻辑地址与物理地址之间的映射关系，指纹与逻辑地址之间的映射关系等等。元数据的存储方式与业务数据的存储方式有所区别。元数据一般采用一定的数据结构来存储。存储元数据的数据结构例如为二叉树、B+树等，这些数据结构要求以一定的粒度来管理元数据。在本申请的一些实施例中，将管理元数据的粒度称为元数据管理粒度。
(16)指纹表
指纹表用于保存存储设备存储的每个数据块的指纹。
(17)机会表
机会表用于保存最近一段时间在存储设备写入的数据块的指纹。机会表与指纹表有所区别。机会表可以理解为一个临时的窗口，用于寻找具有重删机会的数据块。具体地，存储设备会将最近一段时间产生的数据块的指纹存放在机会表中，当满足重删触发条件(如负载低于阈值)时，存储设备会根据机会表中的指纹找到重复块，对重复块进行重删后，将重复块的指纹再保存至指纹表中。
下面介绍本申请实施例提供的应用场景。
本实施例提供的方法能够应用在分布式存储系统或集中式存储设备中，下面对这两种应用场景分别进行介绍。
应用场景一、分布式存储系统
参见附图1，本实施例提供了一种分布式存储系统100，如图1所示，系统100包括多个存储节点101以及至少一个主机102。每个主机102与存储节点101之间均通过有线或无线网络建立通信连接。例如，参考图1，各个主机102与存储节点101之间通过网际互连协议(Internet protocol，IP)网络或者其他网络建立通信连接。
每个存储节点101包括网卡1011、一个或多个硬盘1012、处理器1013、处理器1014以及内存1015。
网卡1011也称为网络接口卡(network interface card，NIC)。网卡1011用于与主机102通信。
硬盘1012例如为固态硬盘(solid state disk，SSD)或硬盘驱动器(hard disk drive，HDD)。本实施例对存储节点101与硬盘1012之间的位置关系不做限定。在一些实施例中，如附图1所示，硬盘1012位于存储节点101的内部。例如，存储节点101为服务器，该服务器内设置有多个硬盘。在另一些实施例中，硬盘1012不在存储节点101的内部，而是位于与存储节点101耦合的硬盘框内，该硬盘框包括多个硬盘1012。
处理器1013例如为中央处理器(central processing unit，CPU)。处理器1013例如配置一个或多个。
处理器1014用于承担压缩和/或重删的功能，从而减轻处理器1013的计算负担。在一些实施例中，处理器1014是和处理器1013具有相同物理形态的处理器。在另一些实施例中，处理器1014是和处理器1013具有不同物理形态的处理器。可选地，处理器1014是具备计算能力的处理芯片。例如，处理器1014是一个加速卡或协处理器或图形处理器(Graphics Processing Unit，GPU)或神经网络处理器(Neural-network Processing Unit，NPU)等。处理 器1014例如配置一个或多个。
在存储节点101既包括处理器1014，又包括处理器1013的情况下，处理器1014和处理器1013可选地共同配合完成数据处理操作。例如，处理器1013用于从主机接收数据，将数据发送给处理器1014，以及指示处理器1014对数据进行压缩和/或重删。处理器1014在接收到处理器1013的指令的触发下，进行压缩和/或重删。
在一些实施例中，在存储节点101具有多个处理器1014的情况下，处理器1013用于对多个处理器1014进行调度。例如，处理器1013将压缩任务和/或重删任务划分为多个子任务，将每个子任务分配至对应的处理器1014。
在一些实施例中，存储节点101还包括通信总线(在附图1未示出)，处理器1014和处理器1013例如均通过通信总线访问内存1015，得到内存1015缓存的指令或代码。
需要说明的一点是，处理器1014是存储节点101可选的组件。在另一些实施例中，存储节点101没有处理器1014而具有处理器1013。例如，由处理器1013独立完成获取数据、对数据进行压缩和/或重删的操作。
主机102包括应用1031以及客户端1032。
存储节点101能够为主机102提供数据存储服务。例如，当主机102需要将数据存储至存储节点101时，主机102中应用1031(也称为上层应用)生成写请求，向存储节点101发送写请求。存储节点通过网卡1011接收写请求，将写请求指示的数据写入至硬盘1012，并保存数据的元数据。
存储节点101能够为主机102提供数据访问服务。例如，当主机102需要访问存储节点101存储的数据时，主机102中应用1031生成读请求，向存储节点101发送读请求。存储节点通过网卡1011接收读请求。存储节点根据读请求指示的数据以及保存的元数据，确定数据在硬盘1012中的地址，从硬盘1012中的相应地址读取数据，通过网卡1011将数据发送至主机102。主机102接收数据，从而获得了存储节点101存储的数据。
应用场景二、集中式存储设备
集中式存储设备例如是存储阵列。存储阵列包括一个或多个控制器，又称存储控制器和一个或多个硬盘。集中式存储设备也可以是一个存储节点，如图1的存储节点101所示。存储设备中的控制器也称存储控制器。集中式存储设备通过有线网络或无线网络与主机相连。
在以上介绍的应用场景中，随着数据的爆炸式增长，主机的数据存储需求与日俱增，数据在存储系统中占用的空间也越来越大。为了缓解存储系统的空间增长问题，重删压缩技术已成为本领域的热门研究课题。通过重删压缩，能够缩减数据的规模，从而有效降低存储系统的开销。
时下的重删压缩方案中，由于元数据管理粒度是固定的，压缩和重删时使用的粒度也是保持一致的。
但是经过研究发现，在重删和压缩采用相同粒度的情况下，如果粒度过大，压缩率会提高，重删率会下降；如果粒度过小，重删率提高，压缩率下降。由此可见，如果重删与压缩使用相同的粒度，会导致要么重删率下降，要么压缩率下降，重删率和压缩率无法同时达到较优。
有鉴于此，本申请实施例中，存储设备在重删和压缩时会分别采用不同的粒度。即，重 删粒度和压缩粒度是不同的，从而避免粒度过大导致重删率下降的情况以及粒度过小导致压缩率下降的情况。下面，通过(a)至(g)对本申请涉及的几种粒度以及不同粒度之间的关系进行介绍。
(a)粒度(grain)
粒度也称数据的尺寸、数据的大小或数据的长度。粒度越大，表示一个数据的尺寸越大。粒度的单位包括而不限于KB、兆(MB，Mega，简称M)等。例如，粒度是4KB，含义是指一条数据的尺寸是4KB。对于存储设备而言，粒度是重要的参数。粒度会影响存储设备的很多业务，包括而不限于数据读取、数据存储、重删、压缩、元数据管理等。本申请的一些实施例中，将侧重描述重删、压缩、元数据管理这几类业务中涉及的粒度、不同粒度的关系以及粒度对于存储设备的影响。
(b)重删粒度
重删粒度用于指示存储设备查询重复数据时使用的粒度。重删粒度等于重删过程中分块的粒度。例如，在重删粒度的尺寸是4KB的情况下，存储设备会将数据划分为多个4KB数据块；存储设备会判断每个4KB数据块是否是重复块；如果一个4KB数据块是重复块，则存储设备会删除该4KB数据块。在一些实施例中，如果采用定长重删的方式进行重删，重删粒度的尺寸为4KB。如果采用相似重删的方式进行重删，重删粒度的尺寸为8KB。
(c)压缩粒度
压缩粒度用于指示存储设备执行数据压缩时使用的粒度。存储设备会根据压缩粒度，确定一次对多大的数据进行压缩。例如，在压缩粒度的尺寸是32KB的情况下，存储设备会对32KB的数据进行压缩。压缩粒度的尺寸会影响压缩率。经实验发现，压缩率与压缩粒度的尺寸成正比。当压缩粒度的尺寸小于32KB时，压缩粒度的尺寸越大，压缩率越大；当压缩粒度的尺寸超过32KB时，压缩率趋于稳定。在一些实施例中，压缩粒度的尺寸是根据压缩率确定的。在一些实施例中，压缩粒度的尺寸设置为32KB。
(d)元数据管理粒度
元数据管理粒度用于指示存储设备记录的元数据的粒度。例如，在一些实施例中，元数据管理粒度的取值不是一个固定值，而是能够在一定范围内动态变化。具体地，元数据管理粒度为一个区间。该区间具有最小值和最大值。存储设备记录的元数据粒度例如是区间的最小值、区间的最大值或者最小值与最大值之间的值。例如，元数据管理粒度的尺寸是[4KB，1M]，则存储设备可选地记录一条4KB的元数据，或者记录一条1M的元数据，或者记录一条粒度在4KB与1M之间的元数据。
在一些实施例中，元数据管理粒度的尺寸是存储单元的尺寸的整数倍。例如，元数据是通过至少一个存储单元存储的。元数据和存储单元之间的数量关系为一对一的关系或者一对多的关系。当元数据和存储单元之间的数量关系为一对一的关系时，一个存储单元存储一条元数据。当元数据和存储单元之间的数量关系为一对多的关系时，多个存储单元共同存储一条元数据，例如，多个物理地址连续且逻辑地址连续的存储单元共同存储一条元数据。在元数据采用这种方式存储时，元数据管理粒度的尺寸的最小值是一个存储单元的尺寸。
例如，参见附图2，附图2示出了存储设备中的8个存储单元，这8个存储单元分别是存储单元201、存储单元202、存储单元203、存储单元204、存储单元205、存储单元206、存储单元207和存储单元208。元数据管理粒度的尺寸的最小值是8个存储单元中一个存储 单元的尺寸，在附图2中通过1个格子表示。元数据管理粒度的尺寸的最大值是8个存储单元的尺寸之和，在附图2中通过8个格子表示。
(e)重删粒度与压缩粒度之间的关系
在一些实施例中，重删粒度的尺寸与压缩粒度的尺寸成反比。重删粒度的尺寸小于压缩粒度的尺寸。例如，重删粒度的尺寸是4KB，压缩粒度的尺寸是32KB。由于重删粒度越小越有助于提升重删率，压缩率越大越有助于提升压缩率，因此存储设备采用大粒度压缩、小粒度重删的方案，有助于重删率和压缩率都得到提升，达到缩减率整体较优的效果。
在一些实施例中，重删粒度以及压缩粒度均是根据元数据管理粒度确定的，具体参见下述(f)和(g)。存储设备通过根据元数据管理粒度分别为重删和压缩选择粒度，有助于重删和压缩均按照各自较优的粒度进行操作，使得重删率和压缩率均达到较优。
(f)重删粒度与元数据管理粒度之间的关系
在一些实施例中，重删粒度是根据元数据管理粒度确定的。在一些实施例中，重删粒度的尺寸等于元数据管理粒度的尺寸的最小值。例如，在元数据管理粒度是[4KB，1M]的情况下，重删粒度的尺寸等于4KB。例如，在元数据管理粒度是[8KB，2M]的情况下，重删粒度的尺寸等于8KB。其中，在元数据管理粒度的最小值是一个存储单元的尺寸的情况下，重删粒度的尺寸例如是一个存储单元的尺寸。例如，参见附图2，重删粒度的尺寸例如是存储单元201的尺寸。
通过将元数据管理粒度的最小值作为重删粒度，有助于重删粒度取到较优粒度，从而提高重删率并节省存储资源。以下对这一技术效果的技术原理进行论述。
一方面，重删粒度会影响重删率。如果重删粒度过大，则重删率会下降。例如，如果重删粒度为32KB，则32KB的数据均是重复数据时，存储设备才会将32KB的数据作为重复块并删除。如果32KB的数据中只有部分数据是重复数据，比如只有24KB的数据是重复数据，另外8KB的数据不是重复数据，存储设备是不会进行重删的。从这个例子可见，重删粒度过大会导致重删效果不佳。
另一方面，重删粒度会影响到元数据的存储开销。由于存储设备会为每个重复块记录一条元数据，重删粒度越小，存储设备记录的元数据就会越多。因此，如果重删粒度过小，产生过量的重复块，存储设备就需要记录过多的元数据，导致元数据占用过多的存储资源。
而本实施例通过将元数据管理粒度的最小值作为重删粒度，由于重删粒度足够小，因此有助于提高重删率。并且，存储设备在重删时，无需为小于元数据管理粒度的最小值的数据记录元数据，因此避免记录过多元数据带来的资源浪费。
在另一些实施例中，重删粒度的尺寸不是元数据管理粒度的最小值，而是元数据管理粒度的最小值的整数倍。例如，在元数据管理粒度是[4KB，1M]的情况下，重删粒度的尺寸为4KB至1M之间任意4KB的倍数。例如，重删粒度的尺寸是元数据管理粒度的最小值的2倍或3倍。其中，在元数据管理粒度的最小值是一个存储单元的尺寸的情况下，重删粒度的尺寸例如是存储单元的尺寸的整数倍。例如，参见附图2，重删粒度的尺寸例如是存储单元201的尺寸的整数倍。
(g)压缩粒度与元数据管理粒度之间的关系
在一些实施例中，压缩粒度是根据元数据管理粒度确定的。在一些实施例中，压缩粒度是根据元数据管理粒度和压缩率共同确定的。在一些实施例中，压缩粒度是根据元数据管理 粒度的最小值和压缩率确定的。例如，压缩粒度的尺寸是元数据管理粒度的最小值与压缩率的乘积。例如，如果压缩率是N：1，则压缩粒度的尺寸是元数据管理粒度的最小值的N倍，N为正整数。例如，在元数据管理粒度是[8KB，2M]且压缩率为4：1的情况下，压缩粒度的尺寸＝8KB*4＝32KB。
其中，在元数据管理粒度的最小值是一个存储单元的尺寸的情况下，压缩粒度的尺寸例如是一个存储单元的尺寸与压缩率的乘积。例如，参见附图2，压缩粒度例如是存储单元201的尺寸与压缩率的乘积。例如，在压缩率是4:1的情况下，压缩粒度是存储单元201的尺寸的4倍，压缩粒度在附图2中对应于四个格子。由于压缩粒度的尺寸不再是固定的值，而是根据压缩率动态选择的，从而保证数据读取性能不下降的情况下压缩率较优。
以上介绍的各种粒度以及粒度关系方面的特征例如应用在存储设备写数据的流程中。写数据的流程涉及前重删流程以及后重删流程。以下通过方法300，对存储设备如何利用上文描述的各种粒度进行前重删举例说明，通过方法400，对存储设备如何利用上文描述的各种粒度进行后重删举例说明。
以下介绍的方法300和方法400由存储设备执行。
在一些实施例中，方法300或方法400应用在分布式存储系统中，执行方法300或方法400的存储设备为分布式存储系统中的一个或多个存储节点。例如，执行方法300或方法400的存储设备为附图1所示系统100中的存储节点101，方法300或方法400处理的数据是系统100中主机102的数据。
在另一些实施例中，方法300或方法400应用在集中式存储设备中，执行方法300或方法400的存储设备为存储阵列。
在一些实施例中，方法300或方法400由CPU执行。在另一些实施例中，方法300或方法400由CPU和重删压缩的专用处理器协作执行，该重删压缩的专用处理器例如是硬件加速卡。例如，CPU为附图1所示的处理器1013，专用处理器为附图1所示的处理器1014。具体地，重删压缩的流程涉及数据分块、指纹计算、指纹查找、数据压缩、数据保存等任务。例如，由专用处理器执行指纹计算的任务以及数据压缩的任务，由CPU执行重删压缩流程涉及的其他任务，从而将指纹计算的任务以及数据压缩的任务从CPU卸载至专用处理器，减少重删压缩对CPU计算资源的占用，加速重删压缩流程的执行。
需要说明的一点是，方法400与方法300同理的内容还请参见方法300，在方法400中不做赘述。
参见附图3，附图3是本申请实施例提供的一种数据处理方法300的流程图。
示例性地，方法300包括S310至S360。
S310、存储设备获取多个数据块。
在一些实施例中，S310具体包括以下步骤S311至步骤S313。
S311、存储设备从主机接收写请求。
其中，写请求用于请求存储设备存储数据，写请求包括待存储的数据以及数据的逻辑地址。
S312、存储设备从写请求获取数据。
S313、存储设备按照第一粒度，对数据进行划分，得到多个数据块，每个数据块的尺寸 等于第一粒度的尺寸。
第一粒度为上文介绍的重删粒度，即存储设备进行重复数据删除处理时使用的粒度。在一些实施例中，存储设备根据元数据管理粒度，确定第一粒度。举例来说，存储设备确定元数据管理粒度的最小值，将元数据管理粒度的最小值的整数倍作为第一粒度。可选地，将元数据管理粒度的最小值作为第一粒度。例如，在元数据管理粒度是[4KB，1M]的情况下，存储设备确定元数据管理粒度的最小值是4KB，则确定第一粒度的尺寸是4KB，将数据划分为多个4KB的数据块。
S320、存储设备确定多个数据块中每个数据块的指纹。
在一些实施例中，存储设备对每个数据块进行指纹计算，得到每个数据块的指纹。在一些实施例中，数据块的指纹是数据块的哈希值，存储设备对每个数据块进行哈希计算，得到每个数据块的哈希值。
S330、存储设备根据每个数据块的指纹，在多个数据块中确定重复块以及非重复块。
重复块的指纹与存储设备已存储的数据块的指纹相同。重复块包含的数据与存储设备已存储的数据块包含的数据完全相同；或者，重复块包含的数据与存储设备已存储的数据块包含的数据部分相同部分不同。如果采用定长重删的方式，那么重复块包含的数据与存储设备已存储的数据块包含的数据完全相同，如果采用相似重删的方式，那么只要求重复块包含的数据与存储设备已存储的数据块包含的数据部分相同。非重复块是指多个数据块中重复块之外的其他数据块。非重复块也称重删不成功的数据块。
在一些实施例中，存储设备通过执行查询指纹表的步骤，从而判定数据块是重复块还是非重复块，在采用这种实现方式时，步骤S330也称指纹查询。其中，指纹表用于保存存储设备已存储的数据块的指纹。具体地，以多个数据块中的第一数据块为例，步骤S330包括：存储设备查询指纹表，对第一数据块的指纹与指纹表中的指纹进行比较；如果第一数据块的指纹与指纹表中一个指纹相同(即第一数据块命中指纹表)，则存储设备确定第一数据块是重复块；如果第一数据块的指纹与指纹表中每个指纹均不同(即第一数据块未命中指纹表)，则存储设备确定第一数据块是非重复块。
在一些实施例中，S330在存储设备本地执行。在另一些实施例中，S330由存储设备和专用服务器协作执行。该专用服务器是一个独立的设备，与存储设备耦合或通过网络连接，该服务器中保存有指纹表，可以专门负责指纹查询工作。具体来说，存储设备将每个数据块的指纹按照预设规则发送至服务器，由该服务器根据每个数据块的指纹确定查询指纹表，从而将指纹查询的任务分担至服务器，避免指纹查询的运算量对存储设备带来性能瓶颈。
执行S330的时机包括多种情况。在一些实施例中，步骤S330是在步骤S320完成后实时执行的。在另一些实施例中，存储设备在执行S320后会判断当前是否满足重删触发条件，若满足重删触发条件，再执行步骤S330。例如，在重删触发条件是负载低于阈值的情况下，存储设备执行步骤S320后，会判断负载是否低于阈值；若负载高于阈值，缓存得到的指纹并进行等待，若负载低于阈值，执行S330。
存储设备对重复块执行的操作与对非重复块执行的操作有所区别。具体地，存储设备会对重复块进行重删操作，对非重复块进行压缩操作。下面通过S340对存储设备如何进行重删操作举例说明，通过S350和S360对存储设备如何进行压缩操作举例说明。
值得说明的一点是，本实施例并不限定重删操作和压缩操作都要执行。在另一些实施例 中，重删操作和压缩操作择一执行。具体地，前重删包括前重删成功的情况和前重删失败的情况，前重删失败的情况是指存储设备没有进行重删而进行压缩的情况。在前重删成功的情况下，如果存储设备确定待存储的数据块全部是重复块，存储设备执行S340，而不执行S350和S360；在前重删成功的情况下，如果存储设备确定待存储的数据块中一部分是重复块而另一部分是非重复块，存储设备执行S340，且执行S350和S360；在前重删失败的情况下，存储设备执行S350和S360，而不执行S340。
例如，参见附图4，附图4中的(a)是对前重删成功的情况的举例说明。附图4中的(b)是对前重删失败(未重删有压缩)的情况的举例说明，在本文后续的描述中，将通过引用附图4所示的场景涉及的流程举例说明。
附图4中的(a)所示的场景具体为：主机发起对数据块1、数据块2至数据块8这8个数据块的写请求；存储设备接收到写请求后，对8个数据块分别进行指纹计算，得到8个指纹。其中，8个指纹分别是数据块1的指纹FP 1、数据块2的指纹FP 2至数据块8的指纹FP 8；存储设备分别根据8个指纹查询指纹表，发现指纹FP 4与指纹FP 7命中了指纹表，即，存储设备在指纹表中找到与数据块4的指纹FP 4相同的指纹，并在指纹表中找到与数据块7的指纹FP 7相同的指纹。因此，存储设备确定数据块4和数据块7均为重复块；存储设备不保存数据块4和数据块7，在存储单元204中记录数据块4的指纹FP 4对应的指纹索引FPI 4，在存储单元207中记录数据块7的指纹FP 7对应的指纹索引FPI 7。存储设备确定数据块4和数据块7之外的其他数据块是非重复块。存储设备对数据块1、数据块2和数据块3进行压缩，得到压缩块1，使用一条元数据表示压缩块1，得到压缩块1的元数据1。存储设备在存储单元201、存储单元202以及存储单元203这三个物理连续且逻辑连续的存储单元中，保存压缩块1的元数据1。存储设备对数据块5、数据块6和数据块8进行压缩，得到压缩块2，使用一条元数据2表示压缩块2，得到压缩块2的元数据2；存储设备在存储单元205、存储单元206以及存储单元208这三个物理连续而逻辑不连续的存储单元中保存压缩块2的元数据2。其中，元数据2包括两个部分，一个部分是元数据21，另一个部分是元数据22。存储单元205和存储单元206保存了元数据21，存储单元208保存了元数据22。此外，存储设备会将压缩块1和压缩块2写入至硬盘中。
附图4中的(b)所示的场景具体为：主机发起对数据块1、数据块2至数据块8这8个数据块的写请求；存储设备接收到写请求后，存储设备对8个数据块进行指纹计算，得到8个指纹，这8个指纹分别是数据块1的指纹FP 1、数据块2的指纹FP 2至数据块8的指纹FP 8；存储设备由于负载高于阈值或者其他原因导致重删失败。在这种情况下，存储设备对数据块1、数据块2和数据块3进行压缩，得到压缩块1，使用一条元数据表示压缩块1，得到压缩块1的元数据1。存储设备在存储单元201、存储单元202以及存储单元203这三个物理连续且逻辑连续的存储单元中，保存压缩块1的元数据1。存储设备对数据块4、数据块5和数据块6进行压缩，得到压缩块2，使用一条元数据表示压缩块2，得到压缩块2的元数据2。存储设备在存储单元204、存储单元205以及存储单元206这三个物理连续且逻辑连续的存储单元中，保存压缩块2的元数据2。存储设备对数据块7和数据块8进行压缩，得到压缩块3，使用一条元数据表示压缩块3，得到压缩块3的元数据3。存储设备在存储单元207、存储单元208这两个物理连续且逻辑连续的存储单元中，保存压缩块3的元数据3。此外，存储设备会将压缩块1、压缩块2和压缩块3写入至硬盘中。
S340、存储设备为重复块记录元数据。
存储设备通过记录重复块的元数据而不保存重复块，从而避免重复块占用硬盘的存储空间，因此节省了存储设备的存储资源。
在一些实施例中，存储设备还记录非重复块的元数据，并将非重复块的指纹保存至指纹表，以便后续对新的块进行重复数据删除处理时，在指纹表中能够查到之前保存的非重复块的指纹。
在一些实施例中，存储设备保存重复块的指纹。在一些实施例中，存储设备保存重复块的指纹索引。可选地，存储设备为重复块记录的元数据为重复块的指纹索引(FPI)。具体地，存储设备将重复块的指纹索引作为重复块的元数据，向元数据的存储单元写入重复块的指纹索引，从而将指纹索引保存在元数据的存储单元中。例如，参见附图4，存储设备确定数据块4和数据块7是重复块之后，将数据块4的指纹索引FPI 4写入至存储单元204，将数据块7的指纹索引FPI 7写入至存储单元207。
参见附图4，重复块的元数据占用的存储空间大小等于第一粒度的尺寸。例如，重复块的元数据占用的存储空间大小等于元数据管理粒度的最小值。例如，参见附图4，元数据管理粒度的最小值对应于附图4中的一个格子，元数据管理粒度的最小值例如是一个存储单元的大小。比如说，数据块4的指纹索引FPI 4占用存储单元204，数据块4的指纹索引FPI 4占用的存储空间大小等于存储单元204的尺寸。
在S340中存储设备如何记录元数据包括多种实现方式。在一些实施例中，存储设备根据元数据管理粒度，选取第一存储单元，在第一存储单元中存储重复块的元数据。其中，第一存储单元的粒度为元数据管理粒度的最小值。例如，元数据管理粒度是[4KB，1M]，则存储设备选取尺寸为4KB的存储单元，在4KB的存储单元中存储重复块的元数据。
S350、存储设备对多个非重复块进行压缩，得到压缩块。
压缩块是指压缩后的非重复块。在一些实施例中，S350具体包括以下步骤a至步骤c。
步骤a、存储设备获得数据的压缩率。
如何获得压缩率包括多种实现方式。在一些实施例中，采用机器学习的方式根据样本进行模型训练，得到预测模型，当获取数据后，将数据输入预测模型，通过预测模型对数据进行压缩率预测，输出压缩率。在另一些实施例中，由用户根据经验预先设定压缩率。在另一些实施例中，考虑到连续的两个数据块的压缩率相同的概率很大，将上一个数据的压缩率作为当前待压缩的数据的压缩率。
步骤b、存储设备根据预测出的压缩率，将多个非重复块划分为至少一个数据块组，每个数据块组包含的非重复块数量等于压缩率。
例如，如果预测压缩率是3:1，则存储设备将3个非重复块划分为一个数据块组，使得一个数据块组包含3个非重复块，以便将3个非重复块放在一起压缩；如果预测压缩率是4:1，则存储设备将4个非重复块划分为一个数据块组，使得一个数据块组包含4个非重复块，以便将4个非重复块放在一起压缩。例如，参见附图4，在压缩率为3:1的情况下，附图4中的三个格子可以对应于一个压缩块，比如，参见附图4中的(a)，存储设备将数据块1、数据块2和数据块3划分至一组，将数据块5、数据块6和数据块8划分至另一组。又如，参见附图4中的(b)，存储设备将数据块1、数据块2和数据块3划分至一组，将数据块4、数据块5和数据块6划分至另一组，将数据块7和数据块8划分至一组。
在一些实施例中，存储设备在分组时不仅考虑压缩率，还考虑不同非重复块是否连续。例如，存储设备根据压缩率以及每个非重复块的地址，将连续的多个非重复块划分至同一个数据块组中。其中，连续包括而不限于物理连续或者逻辑连续中的至少一项。物理是否连续例如根据物理地址是否连续确定，逻辑是否连续例如根据逻辑地址是否连续确定。
步骤c、存储设备将每个数据块组压缩为一个压缩块。
例如，参见附图4中的(a)，存储设备将数据块1、数据块2和数据块3压缩为压缩块1，将数据块5、数据块6和数据块8压缩为压缩块2。又如，参见附图4中的(b)，存储设备将数据块1、数据块2和数据块3压缩为压缩块1，将数据块4、数据块5和数据块6压缩为压缩块2，将数据块7和数据块8压缩为压缩块3。
通过执行上述压缩流程，由于一个压缩块是根据一个数据块组中的每个非重复块压缩得到的，因此第二粒度等于一个数据块组包含的非重复块数量与非重复块的粒度的乘积。由于一个数据块组包含的非重复块数量等于压缩率，非重复块的粒度等于元数据管理粒度的最小值，因此第二粒度等于压缩率与元数据管理粒度的最小值的乘积。比如，参见附图4，压缩率等于3：1，元数据管理粒度对应于附图4中的一个格子，第二粒度对应于附图4中的三个格子。比如，第二粒度等于存储单元201、存储单元202和存储单元203这三个存储单元的尺寸之和。
由此可见，本实施例提供的压缩粒度(即第二粒度)不再是与重删粒度(即第一粒度)相同的固定值，而是根据元数据管理粒度的最小值和压缩率来确定的，因此支持了压缩粒度动态选择的功能。通过压缩粒度的动态选择，有助于保证在数据读取性能不下降的情况下压缩率较优，获得最大压缩率和读放大的平衡。下面结合一个具体的例子，对达到这种技术效果的技术原理进行介绍。
例如，元数据管理粒度的最小值是8KB，存储设备获得了4个8KB大小的数据块，这4个数据块分别是数据块a、数据块b、数据块c和数据块d。存储设备预测4个数据块的压缩率均是4:1，则存储设备确定压缩粒度是32KB，将数据块a、数据块b、数据块c和数据块d这32KB的数据一起压缩，得到一个8KB的压缩块e，将压缩块e存储至硬盘中。之后，存储设备接收到了读请求，读请求指示存储设备读取数据块a。存储设备响应于读请求，从硬盘读取8KB的压缩块e；存储设备对压缩块e进行解压缩，得到数据块a、数据块b、数据块c和数据块d，将其中的数据块a返回给读请求的发起方。从这个例子可见，读请求对应的数据粒度是8KB，存储设备从硬盘中实际读取的数据粒度(即压缩块e的粒度)也是8KB，可见从硬盘中实际读取的数据粒度与读请求对应的数据粒度是相同的，没有出现读放大的情况，也就避免了读放大会带来的读取性能下降以及带宽资源消耗。并且，由于压缩粒度为较优的32KB，有助于压缩率达到较优。
S360、存储设备将压缩块存储至硬盘中，并为压缩块记录元数据。
通过执行S360，存储设备将经过重复数据删除处理和压缩处理的数据(压缩块)存储在存储设备的硬盘中。
压缩块的元数据例如表示数据的逻辑地址到压缩块的物理地址之间的映射关系。
在一些实施例中，存储设备在尺寸属于[元数据管理粒度的最小值，压缩粒度]的存储单元内，存储压缩块的元数据。采用这种方式时，压缩块的元数据的粒度的最小值是元数据管理粒度的最小值，压缩块的元数据的粒度的最大值是压缩粒度(第二粒度)。在一些实施例 中，压缩块的元数据的粒度为元数据管理粒度的最小值与压缩率的乘积。例如，压缩率为N：1，元数据管理粒度的最小值是一个存储单元的尺寸，则存储设备选取N个存储单元，则N个存储单元中存储压缩块的元数据。例如，参见附图4，压缩率为3：1，元数据管理粒度的最小值是一个存储单元的尺寸，存储设备会选取3个存储单元来记录一条压缩块的元数据，压缩块的元数据的粒度为存储单元的尺寸的三倍。比如说，存储设备选取存储单元201、存储单元202以及存储单元203记录压缩块1的元数据1，压缩块1的元数据1的粒度是存储单元201、存储单元202以及存储单元203的尺寸之和。
存储设备记录元数据的方式具体包括下述方式一至方式二中的至少一项。
方式A、存储设备为地址连续的多个压缩块记录一条元数据。
在一些实施例中，在多个压缩块的地址连续的情况下，存储设备使用一条元数据来表示多个压缩块。其中，多个压缩块地址连续例如是多个压缩块物理地址连续且多个压缩块逻辑地址连续。在一些实施例中，存储设备记录的一条元数据包括两个部分，一个部分是地址连续的多个压缩块中第一个压缩块的地址，另一个部分是地址连续的多个压缩块中每个压缩块的长度。例如，参见附图7，附图7中的一个小格标识一个最小粒度的数据块或元数据，如一条4KB的数据块或者一条4KB的的元数据。附图7是对如何使用一条元数据表示压缩块1、压缩块2以及压缩块3的举例说明。附图7所示，存储设备压缩处理得到压缩块1、压缩块2以及压缩块3这三个地址连续的压缩块后，记录一条元数据，记录的一条元数据包含压缩块1(第一个压缩块)的元数据1以及跳转表(jump table)，元数据1指明了压缩块1的地址，跳转表包含压缩块1的长度、压缩块2的长度以及压缩块3的长度。例如，压缩块1的长度是9KB，压缩块2的长度是7KB，压缩块3的长度是4KB，则跳转表包含9KB、7KB以及4KB。
通过采用这样的方式记录元数据，在能够通过元数据读取数据的基础上，达到压缩的元数据空间较优的效果，以下对实现这一技术效果的技术原理举例说明。
一方面，由于元数据中记录了每个压缩块的长度，能够指明每个压缩块相对于第一个压缩块的偏移量。例如，第二个压缩块相对于第一个压缩块的偏移量是第一个压缩块的长度，第三个压缩块相对于第一个压缩块的偏移量是第一个压缩块与第二个压缩块的长度之和。因此，当需要读取第二个压缩块时，能够根据第一个压缩块的地址以及第二个压缩块相对于第一个压缩块的偏移量，在硬盘中寻址到第二个压缩块，当需要读取第三个压缩块时，能够根据第一个压缩块的地址以及第三个压缩块相对于第一个压缩块的偏移量，在硬盘中寻址到第三个压缩块。比如说，在附图7的场景下，当需要读取压缩块2时，存储设备从压缩块1的元数据1的地址向后偏移9KB，即可从硬盘中找到压缩块2。当需要读取压缩块3时，存储设备从压缩块1的元数据1的地址向后偏移9KB+7KB＝16KB，即可从硬盘中找到压缩块3。由此可见，通过这种元数据的记录方式，能够在硬盘中分别定位到每个压缩块，保证每个压缩块都能被读取。
另一方面，由于为地址连续的多个压缩块记录一条元数据，减少了记录的元数据的数量，从而节省元数据在存储设备中占用的存储资源。比如说，在附图7的场景下，由于存储设备记录了压缩块1的元数据1，无需记录压缩块2的元数据2和压缩块3的元数据3，所以节省了压缩块2的元数据2和压缩块3的元数据3占用的存储空间。
方式B、存储设备为地址不连续的多个压缩块记录多条元数据。
在一些实施例中，在多个压缩块的地址不连续的情况下，存储设备使用多条元数据分别表示多个压缩块。可选地，如方式A同理，为两个或两个以上地址连续的压缩块记录一条元数据。其中，地址不连续的多个压缩块在逻辑上的间隔例如对应于重复块的指纹索引。
例如，参见附图4中的(a)，压缩块2是根据数据块5、数据块6和数据块8这3个非重复块压缩得到的。其中，数据块5和数据块6是两个逻辑上相邻的块。数据块6和数据块8是逻辑上相隔的。具体地，数据块5的逻辑地址与数据块6的逻辑地址连续，数据块6的逻辑地址与数据块8的逻辑地址不连续。例如，数据块5的逻辑地址是205，长度是8KB，数据块6的逻辑地址是206，长度是8KB。数据块8的逻辑地址是208，长度是8KB。而数据块6和数据块8之间在逻辑上本来存在数据块7，而数据块7被发现是重复块而被重删掉，使得存储单元207存储的是数据块7的指纹索引FPI4。存储设备将数据块5、数据块6和数据块8压缩为压缩块2，为压缩块2记录的元数据2包括两条元数据，分别是元数据21和元数据22。元数据21表示起始逻辑地址是205，长度是8KB*2＝12KB。元数据22表示起始逻辑地址是208，长度是8KB。
在这个例子中，同一个压缩块(压缩块2)对应了两条元数据(元数据21和元数据22)。在压缩块2解压缩后，元数据21和元数据22对应于解压后数据块的不同部分。具体地，压缩块2解压缩后会得到数据块5、数据块6和数据块8，元数据21对应于数据块5和数据块6，元数据22对应于数据块8。
本实施例提供的前重删方法中，存储设备由于使用不同的粒度进行重删和压缩，摆脱了重删粒度和压缩粒度必须相同的限制，一定程度上避免粒度过大导致重删率下降的情况以及粒度过小导致压缩率下降的情况，从而提升重删压缩整体的缩减率。进一步地，由于重删粒度和压缩粒度都是根据元数据管理粒度确定的，使得重删和压缩分别按照较优的粒度操作，有助于重删率和压缩率均达到较优。
参见附图5，附图5是本申请实施例提供的一种数据处理方法400的流程图。
示例性地，方法400包括S410至S450。
S410、存储设备将多个数据块存储至硬盘中。
在一些实施例中，存储设备还计算每个数据块的指纹，将数据块的指纹以及数据块的物理位置保存在机会表中。其中，机会表例如是键值对的形式，机会表的键是数据块的指纹。机会表的值是数据块的物理位置。机会表用于找到重复块的指纹。
在一些实施例中，存储设备会先对多个数据块进行压缩，得到压缩块，再将压缩块存储至硬盘中。压缩流程如上述方法300描述的压缩流程类似。具体地，存储设备在压缩过程中，会预测数据的压缩率；存储设备根据预测出的压缩率以及元数据管理粒度，确定第二粒度，根据该第二粒度进行压缩。例如，存储设备确定元数据管理粒度的最小值与压缩率的乘积，作为第二粒度。
S420、存储设备在多个数据块中确定重复块以及非重复块。
例如，存储设备从机会表中读取每个数据块的指纹，对数据块的指纹与指纹表中已记录的指纹进行比较。如果数据块的指纹和指纹表中已记录的一个指纹相同，存储设备确定数据块为重复块。如果数据块的指纹和指纹表中已记录的每个指纹均不同，存储设备确定数据块为非重复块，存储设备将数据块的指纹记录至指纹表中。
S430、存储设备为重复块更新已记录的元数据。
存储设备找到可重删的重复块后，会更新重复块的元数据。在一些实施例中，存储设备将重复块的元数据更新为重复块的指纹。例如，存储设备确定第二存储单元，第二存储单元用于保存重复块的元数据，第二存储单元的粒度为元数据管理粒度的最小值。存储设备向第二存储单元写入重复块的指纹，使得第二存储单元保存的元数据从之前记录的元数据覆盖为重复块的指纹。
例如，参见附图6，存储设备在硬盘中存储数据块1、数据块2至数据块8后，确定数据块4和数据块7均是重复块，则更新数据块4的元数据以及数据块7的元数据。具体地，存储设备使用数据块4的指纹索引FPI 4作为数据块4的元数据，向存储单元204覆盖写入数据块4的指纹索引FPI 4，使得存储单元204保存的数据从数据块4的元数据刷新为指纹索引FPI 4；此外，存储设备使用数据块7的指纹索引FPI 7作为数据块7的元数据，向存储单元207覆盖写入数据块7的指纹索引FPI 7，使得存储单元207保存的数据从数据块4的元数据刷新为指纹索引FPI 7。其中，附图6中的(a)是对后重删前的元数据的举例说明，附图6中的(b)是对待更新的元数据的举例说明，附图6中的(c)是对合并后的元数据的举例说明，附图6中的(d)是对碎片整理后的元数据的举例说明。
S440、存储设备对重复块进行垃圾回收。
在为重复块更新元数据后，重复块可称为垃圾数据，存储设备通过删除重复块，从而释放重复块占用的存储空间。例如，参见附图6，存储设备将存储单元204保存的元数据更新为数据块4的指纹索引FPI 4，将存储单元207保存的元数据更新为数据块7的指纹索引FPI 7后，删除数据块4和数据块7。在一些实施例中，存储设备将重复块以及非重复块均从硬盘中读取出来，对重复块以及非重复块原本在硬盘中占用的存储单元进行擦除，再将非重复块重新写入至硬盘中，从而进行垃圾回收。
其中，重复块包括压缩块以及普通的数据块。因此，在垃圾回收过程中，可能是压缩块被覆盖，也可能是普通的数据块被覆盖。对于普通的数据块，存储设备只需要将有效的数据搬出即可。对于压缩块而言，存储设备需要先将不定长粒度的压缩块进行解压，然后对解压后的数据中有效部分的数据重新压缩。由于被压缩块的粒度变小了，压缩率此时会下降，存储设备会通过执行以下S450，执行碎片整理流程，从而实现进行最大粒度压缩。
S450、存储设备对非重复块进行碎片整理。
通过执行S450，存储设备将经过重复数据删除处理和压缩处理的数据(碎片整理后的非重复块)存储在存储设备的硬盘中。
具体地，存储设备在碎片整理过程中，如果待整理的数据是重复块，则存储设备保留重复块的指纹索引。其中，待整理的数据例如包括尚未按照第二粒度进行压缩处理的数据。如果待整理的数据是压缩块，则存储设备判断压缩块对应的压缩粒度(即第二粒度)是否小于较优的压缩粒度(如元数据管理粒度的最小值与压缩率的乘积)，如果压缩块对应的压缩粒度(即第二粒度)小于较优的压缩粒度，则按照较优的压缩粒度对多个压缩块重新进行压缩，从而提升压缩率。例如，存储设备先对压缩块进行解压，得到非重复块，之后，存储设备按照与步骤SF50同理的方式，根据元数据管理粒度的最小值和压缩率确定压缩粒度。存储设备多个非重复块划分为至少一个数据块组，将每个数据块组压缩为一个压缩块，从而重新压缩。
例如，压缩率是3:1，元数据管理粒度的最小值等于一个存储单元的尺寸，较优的压缩粒 度等于三个存储单元的尺寸。参见附图6中的(c)，存储设备在碎片整理过程中，确定压缩块2对应的压缩粒度是2个存储单元的尺寸，确定压缩块3对应的压缩粒度是1个存储单元的尺寸，压缩块2和压缩块3对应的压缩粒度均小于较优的压缩粒度，则存储设备对压缩块2解压，得到数据块5和数据块6，并对压缩块3解压，得到数据块8。存储设备对数据块5、数据块6和数据块8重新进行压缩，得到新的压缩块2，向硬盘写入新的压缩块2。在这个例子中，新的压缩块2对应的压缩粒度为三个存储单元的尺寸，比压缩块2对应的压缩粒度和压缩块3对应的压缩粒度更优，从而有助于提升压缩率。
本实施例提供的后重删方法中，存储设备由于使用不同的粒度进行重删和压缩，摆脱了第一粒度和第二粒度必须相同的限制，一定程度上避免粒度过大导致重删率下降的情况以及粒度过小导致压缩率下降的情况，从而提升重删压缩整体的缩减率。进一步地，由于第一粒度和第二粒度都是根据元数据管理粒度确定的，使得重删和压缩分别按照较优的粒度操作，有助于重删率和压缩率均达到较优。
以上介绍了本申请实施例的方法300和方法400，以下从逻辑功能的角度介绍本申请实施例的存储设备。
请参考附图8，附图8示出了上述实施例中所涉及的存储设备的一种可能的结构示意图。附图8所示的存储设备600例如实现方法300或方法400中存储设备的功能。存储设备600包括获取模块601、重删模块602、压缩模块603和存储模块604。
获取模块601，用于获取数据；重删模块602，用于基于第一粒度对数据进行重复数据删除处理；压缩模块603，用于基于第二粒度对数据进行压缩处理，第二粒度的尺寸大于第一粒度的尺寸；存储模块604，用于将经过重复数据删除处理和压缩处理的数据存储在存储设备的硬盘中。
在一些实施例中，存储设备还包括：记录模块，用于记录压缩块的元数据。
本实施例对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可选地有另外的划分方式。
在一些实施例中，存储设备600中至少一个模块集成在一个处理器、同一个芯片或同一个单板中。例如，获取模块601、重删模块602、压缩模块603均集成在同一个处理器中，通过该处理器实现获取模块601、重删模块602、压缩模块603的功能。
在另一些实施例中，存储设备600不同模块通过不同的处理器或者其他不同的硬件实现。例如，获取模块601通过附图1所示的网卡1011实现，重删模块602和压缩模块603的功能分别通过不同的专用处理器实现，存储模块604的功能通过中央处理器实现。
本领域普通技术人员可以意识到，结合本文中所公开的实施例中描述的各方法步骤和模块，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各实施例的步骤及组成。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。本领域普通技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的系统、装置和模块的具体工作过程，可以参见前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，该模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个模块或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口、装置或模块的间接耦合或通信连接，也可以是电的，机械的或其它的形式连接。
该作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络模块上。可以根据实际的需要选择其中的部分或者全部模块来实现本申请实施例方案的目的。
另外，在本申请各个实施例中的各模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以是两个或两个以上模块集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件模块的形式实现。
该集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分，或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例中方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
本申请中术语“第一”“第二”等字样用于对作用和功能基本相同的相同项或相似项进行区分，应理解，“第一”、“第二”之间不具有逻辑或时序上的依赖关系，也不对数量和执行顺序进行限定。还应理解，尽管以下描述使用术语第一、第二等来描述各种元素，但这些元素不应受术语的限制。这些术语只是用于将一元素与另一元素区别分开。例如，在不脱离各种示例的范围的情况下，第一粒度可以被称为第二粒度，并且类似地，第二粒度可以被称为第一粒度。第一粒度和第二粒度都可以是粒度，并且在某些情况下，可以是单独且不同的粒度。
本申请中术语“至少一个”的含义是指一个或多个，本申请中术语“多个”的含义是指两个或两个以上，例如，多个压缩块是指两个或两个以上的压缩块。本文中术语“系统”和“网络”经常可互换使用。
还应理解，术语“如果”可被解释为意指“当...时”(“when”或“upon”)或“响应于确定”或“响应于检测到”。类似地，根据上下文，短语“如果确定...”或“如果检测到[所陈述的条件或事件]”可被解释为意指“在确定...时”或“响应于确定...”或“在检测到[所陈述的条件或事件]时”或“响应于检测到[所陈述的条件或事件]”。
以上描述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。该计算机程序产品包括 一个或多个计算机程序指令。在计算机上加载和执行该计算机程序指令时，全部或部分地产生按照本申请实施例中的流程或功能。该计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。
该计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，该计算机程序指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。该计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。该可用介质可以是磁性介质(例如软盘、硬盘、磁带)、光介质(例如，数字视频光盘(digital video disc，DVD)、或者半导体介质(例如固态硬盘)等。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，该程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
以上，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。

Claims (28)

  1. 一种数据处理方法，其特征在于，所述方法由存储设备执行，包括：
    获取数据；
    基于第一粒度对所述数据进行重复数据删除处理；
    基于第二粒度对所述数据进行压缩处理，所述第二粒度的尺寸大于所述第一粒度的尺寸；
    将经过所述重复数据删除处理和所述压缩处理的数据存储在所述存储设备的硬盘中。
  2. 根据权利要求1所述的方法，其特征在于，所述存储设备中存储有元数据，所述元数据是基于元数据管理粒度来管理的，所述元数据管理粒度的尺寸小于或等于设定的最大值，并且大于或等于设定的最小值，所述第一粒度的尺寸等于所述最小值的整数倍。
  3. 根据权利要求2所述的方法，其特征在于，所述第二粒度的尺寸是所述最小值与压缩率的乘积。
  4. 根据权利要求1所述的方法，其特征在于，所述基于第一粒度对所述数据进行重复数据删除处理包括：
    将所述数据划分为多个数据块；
    获取每个数据块的指纹；
    根据所述指纹，从所述多个数据块中确定重复块以及非重复块。
  5. 根据权利要求4所述的方法，其特征在于，所述基于第二粒度对所述数据进行压缩处理包括：
    基于所述第二粒度对所述非重复块进行压缩处理，得到压缩块，所述经过所述重复数据删除处理和所述压缩处理的数据包括所述压缩块。
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：记录所述压缩块的元数据。
  7. 根据权利要求6所述的方法，其特征在于，所述记录所述压缩块的元数据包括：
    如果所述压缩块的数量为多个且多个所述压缩块的地址连续，为所述多个压缩块记录一条元数据。
  8. 根据权利要求7所述的方法，其特征在于，所述一条元数据包括所述多个压缩块中第一个压缩块的地址以及每个压缩块的长度。
  9. 根据权利要求1所述的方法，其特征在于，所述数据在经过所述重复数据删除处理和所述压缩处理之前还基于第三粒度压缩过，所述第三粒度的尺寸小于所述第二粒度的尺寸。
  10. 根据权利要求1所述的方法，其特征在于，所述存储设备是存储阵列。
  11. 根据权利要求1所述的方法，其特征在于，所述存储设备是分布式存储系统中的一个存储节点。
  12. 一种存储设备，其特征在于，包括至少一个处理器和硬盘；
    所述至少一个处理器，用于获取数据；基于第一粒度对所述数据进行重复数据删除处理；基于第二粒度对所述数据进行压缩处理，所述第二粒度的尺寸大于所述第一粒度的尺寸；将经过所述重复数据删除处理和所述压缩处理的数据存储在所述硬盘中。
  13. 根据权利要求12所述的存储设备，其特征在于，所述存储设备中存储有元数据，所述元数据是基于元数据管理粒度来管理的，所述元数据管理粒度的尺寸小于或等于设定的最大值，并且大于或等于设定的最小值，所述第一粒度的尺寸等于所述最小值的整数倍。
  14. 根据权利要求13所述的存储设备，其特征在于，所述第二粒度的尺寸是所述最小值与压缩率的乘积。
  15. 根据权利要求12所述的存储设备，其特征在于，所述至少一个处理器，用于将所述数据划分为多个数据块；获取每个数据块的指纹；根据所述指纹，从所述多个数据块中确定重复块以及非重复块。
  16. 根据权利要求15所述的存储设备，其特征在于，所述至少一个处理器，用于基于所述第二粒度对所述非重复块进行压缩处理，得到压缩块，所述经过所述重复数据删除处理和所述压缩处理的数据包括所述压缩块。
  17. 根据权利要求16所述的存储设备，其特征在于，所述至少一个处理器，还用于记录所述压缩块的元数据。
  18. 根据权利要求17所述的存储设备，其特征在于，所述至少一个处理器，用于如果所述压缩块的数量为多个且多个压缩块的地址连续，为所述多个压缩块记录一条元数据。
  19. 根据权利要求18所述的存储设备，其特征在于，所述一条元数据包括所述多个压缩块中第一个压缩块的地址以及每个压缩块的长度。
  20. 根据权利要求12所述的存储设备，其特征在于，所述数据在经过所述重复数据删除处理和所述压缩处理之前还基于第三粒度压缩过，所述第三粒度的尺寸小于所述第二粒度的尺寸。
  21. 根据权利要求12所述的存储设备，其特征在于，所述存储设备是存储阵列。
  22. 根据权利要求12所述的存储设备，其特征在于，所述存储设备是分布式存储系统中的一个存储节点。
  23. 一种计算机可读存储介质，其特征在于，所述存储介质中存储有至少一条指令，所述指令由处理器读取以使存储设备执行如权利要求1至权利要求11中任一项所述的方法。
  24. 一种存储设备，其特征在于，所述存储设备包括：
    获取模块，用于获取数据；
    重删模块，用于基于第一粒度对所述数据进行重复数据删除处理；
    压缩模块，用于基于第二粒度对所述数据进行压缩处理，所述第二粒度的尺寸大于所述第一粒度的尺寸；
    存储模块，用于将经过所述重复数据删除处理和所述压缩处理的数据存储在所述存储设备的硬盘中。
  25. 根据权利要求24所述的存储设备，其特征在于，所述重删模块，用于将所述数据划分为多个数据块；获取每个数据块的指纹；根据所述指纹，从所述多个数据块中确定重复块以及非重复块。
  26. 根据权利要求25所述的存储设备，其特征在于，所述压缩模块，用于基于所述第二粒度对所述非重复块进行压缩处理，得到压缩块，所述经过所述重复数据删除处理和所述压缩处理的数据包括所述压缩块。
  27. 根据权利要求26所述的存储设备，其特征在于，所述存储设备还包括：记录模块，用于记录所述压缩块的元数据。
  28. 根据权利要求27所述的存储设备，其特征在于，所述记录模块，用于如果所述压缩块的数量为多个且多个所述压缩块的地址连续，为所述多个压缩块记录一条元数据。
PCT/CN2020/136106 2020-06-11 2020-12-14 数据处理方法及存储设备 WO2021248863A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20939891.6A EP4030310A4 (en) 2020-06-11 2020-12-14 DATA PROCESSING METHODS AND STORAGE DEVICE
US17/741,079 US12001703B2 (en) 2020-06-11 2022-05-10 Data processing method and storage device

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202010526840 2020-06-11
CN202010526840.7 2020-06-11
CN202010784929.3A CN113806341A (zh) 2020-06-11 2020-08-06 数据处理方法及存储设备
CN202010784929.3 2020-08-06

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/741,079 Continuation US12001703B2 (en) 2020-06-11 2022-05-10 Data processing method and storage device

Publications (1)

Publication Number Publication Date
WO2021248863A1 true WO2021248863A1 (zh) 2021-12-16

Family

ID=78846927

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/136106 WO2021248863A1 (zh) 2020-06-11 2020-12-14 数据处理方法及存储设备

Country Status (2)

Country Link
EP (1) EP4030310A4 (zh)
WO (1) WO2021248863A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103154950A (zh) * 2012-05-04 2013-06-12 华为技术有限公司 重复数据删除方法及装置
US20150347445A1 (en) * 2014-05-30 2015-12-03 International Business Machines Corporation Deduplication of file
CN107506153A (zh) * 2017-09-26 2017-12-22 深信服科技股份有限公司 一种数据压缩方法、数据解压方法及相关系统
CN108427539A (zh) * 2018-03-15 2018-08-21 深信服科技股份有限公司 缓存设备数据的离线去重压缩方法、装置及可读存储介质
CN111124259A (zh) * 2018-10-31 2020-05-08 深信服科技股份有限公司 一种基于全闪存阵列的数据压缩方法及系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9626373B2 (en) * 2012-10-01 2017-04-18 Western Digital Technologies, Inc. Optimizing data block size for deduplication
US10963171B2 (en) * 2017-10-16 2021-03-30 Red Hat, Inc. Compressibility instrumented dynamic volume provisioning
US11550755B2 (en) * 2018-11-15 2023-01-10 Red Hat, Inc. High performance space efficient distributed storage
US10635315B1 (en) * 2019-03-22 2020-04-28 EMC IP Holding Company LLC Performing compression and deduplication at different granularities

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103154950A (zh) * 2012-05-04 2013-06-12 华为技术有限公司 重复数据删除方法及装置
US20150347445A1 (en) * 2014-05-30 2015-12-03 International Business Machines Corporation Deduplication of file
CN107506153A (zh) * 2017-09-26 2017-12-22 深信服科技股份有限公司 一种数据压缩方法、数据解压方法及相关系统
CN108427539A (zh) * 2018-03-15 2018-08-21 深信服科技股份有限公司 缓存设备数据的离线去重压缩方法、装置及可读存储介质
CN111124259A (zh) * 2018-10-31 2020-05-08 深信服科技股份有限公司 一种基于全闪存阵列的数据压缩方法及系统

Also Published As

Publication number Publication date
US20220269431A1 (en) 2022-08-25
EP4030310A4 (en) 2022-12-21
EP4030310A1 (en) 2022-07-20

Similar Documents

Publication Publication Date Title
Dong et al. Optimizing Space Amplification in RocksDB.
US20200150890A1 (en) Data Deduplication Method and Apparatus
WO2022228458A1 (zh) 海量小文件存取优化方法、装置、设备及存储介质
CN106662981B (zh) 存储设备、程序和信息处理方法
JP6537214B2 (ja) 重複排除方法および記憶デバイス
KR20170054299A (ko) 메모리 관리 시의 중복 제거를 위해서 기준 세트로 기준 블록을 취합하는 기법
Nam et al. Assuring demanded read performance of data deduplication storage with backup datasets
US9569357B1 (en) Managing compressed data in a storage system
CN109189995B (zh) 基于mpi的云存储中数据消冗方法
US11580162B2 (en) Key value append
JP2017511521A (ja) フラッシュメモリ圧縮
CN113806341A (zh) 数据处理方法及存储设备
WO2015145647A1 (ja) ストレージ装置とデータ処理方法及びストレージシステム
CN111309720A (zh) 时序数据的存储、读取方法、装置、电子设备及存储介质
WO2021073635A1 (zh) 一种数据存储方法及装置
WO2018153051A1 (zh) 一种存储系统的控制装置
WO2017042978A1 (ja) 計算機システム、ストレージ装置、及びデータの管理方法
US9104726B2 (en) Columnar databases
JP7108784B2 (ja) データ記憶方法、データ取得方法、及び機器
US11461239B2 (en) Method and apparatus for buffering data blocks, computer device, and computer-readable storage medium
EP2811410B1 (en) Monitoring record management method and device
US9933971B2 (en) Method and system for implementing high yield de-duplication for computing applications
JP5444728B2 (ja) ストレージシステム、ストレージシステムにおけるデータ書込方法及びデータ書込プログラム
WO2021248863A1 (zh) 数据处理方法及存储设备
Liu et al. TSCache: an efficient flash-based caching scheme for time-series data workloads

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20939891

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020939891

Country of ref document: EP

Effective date: 20220412

NENP Non-entry into the national phase

Ref country code: DE