WO2022262381A1 - 一种数据压缩方法及装置 - Google Patents

一种数据压缩方法及装置 Download PDF

Info

Publication number
WO2022262381A1
WO2022262381A1 PCT/CN2022/085621 CN2022085621W WO2022262381A1 WO 2022262381 A1 WO2022262381 A1 WO 2022262381A1 CN 2022085621 W CN2022085621 W CN 2022085621W WO 2022262381 A1 WO2022262381 A1 WO 2022262381A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
compressed
data block
attribute
page
Prior art date
Application number
PCT/CN2022/085621
Other languages
English (en)
French (fr)
Inventor
俞超
陈宜
李桂付
邱歌
李志鹏
张代曰
钱璟
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP22823871.3A priority Critical patent/EP4336336A1/en
Publication of WO2022262381A1 publication Critical patent/WO2022262381A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0661—Format or protocol conversion arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0688—Non-volatile semiconductor memory arrays
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
    • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
    • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
    • H03M7/60—General implementation details not specific to a particular type of compression
    • H03M7/6047—Power optimization with respect to the encoder, decoder, storage or transmission

Definitions

  • the present application relates to the technical field of data compression, in particular to a data compression method and device.
  • the read-write file system of Linux such as: F2FS, the second generation flash file system (journalling Flash file system version2, JFFS2), B-tree file system (B-tree file system, BTRFS), etc.
  • the read-write file system of Windows for example: NTFS, etc. Since the metadata area in the file system accounts for a small proportion of the entire file system, the data area often occupies a relatively high device storage capacity. Therefore, compressing the data in the data area can reduce the size of the input and output IO and improve the overall read and write performance of the IO.
  • Existing data compression methods usually compress the original file data (or source data) that needs to be compressed according to the smallest compressible unit of fixed size, and the compressed file data (or compressed data) can include header data and compressed data. data.
  • the header data is used to represent the attribute information of the file data; the compressed data is used to represent the content of the file data. Then save the compressed file data to the storage medium.
  • the existing compression schemes for read-write file systems have the problem of random read amplification, and the read efficiency is low.
  • Embodiments of the present application provide a data compression method and device, which can solve the problem of random read amplification of a read-write file system and improve read efficiency.
  • the embodiment of the present application provides a data compression method.
  • the execution subject of the method may be an electronic device, or a component (for example, a chip, a chip system or a processor, etc.) located in the electronic device.
  • the subject is described as an example of an electronic device.
  • the method includes: the electronic device acquires m data blocks in the data area of the readable and writable file system, where m is a positive integer greater than or equal to 1.
  • the electronic device compresses m data blocks using a preset compression algorithm, and obtains n compressed data blocks in turn, wherein the first capacity of each compressed data block is the same, and the first capacity represents the compressed data that can be contained in the compressed data block
  • the number of bytes, n is a positive integer greater than or equal to 1.
  • the electronic device establishes a first index for each of the j data blocks corresponding to the i-th compressed data block among the n compressed data, and records a mapping relationship between the first index and the j data blocks.
  • i is a positive integer greater than or equal to 1 and less than or equal to n
  • j is a positive integer greater than or equal to 1 and less than or equal to m.
  • the first index is used to identify the storage location of each data block contained in the j data blocks in the storage medium, and the attribute information contained in each data block contained in the j data blocks.
  • the data compression method provided by the embodiment of the present application can effectively improve the reading efficiency when reading a data block, and can ensure that the random reading scenario completes data reading with a small read amplification factor.
  • the attributes contained in the index of the data block can be modified, so that the compressed file on the storage device can be modified. It can be seen that the embodiment of the present application solves the problem of random read amplification in existing read-write file system compression schemes, and at the same time solves the problem that existing file systems with fixed output compression methods cannot support data and metadata updates.
  • m data blocks are compressed using a preset compression algorithm, and n compressed data blocks are sequentially obtained, specifically: each data block in the m data blocks is assigned to the first gather.
  • each data block in the m data blocks is assigned to the first gather.
  • the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data is established, specifically: when the header of the i-th compressed data block When the sum of the total data length of the partial data and compressed data and the set compression threshold is less than or equal to the total data length of j data blocks, establish the first index of each data block in the j data blocks.
  • the attribute information includes at least one of the following: a first attribute, which is used to indicate whether the storage location of the compressed data block where the data block is compressed is pre-allocated.
  • the second attribute is used to indicate whether the data page of the data block is valid.
  • the third attribute is used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block.
  • the fourth attribute is used to represent whether the data page of the data block is included in the compressed data pages of the two compressed blocks.
  • the fifth attribute is used to indicate whether the data page of the data block is a compressed page of the compressed data block after the data block is compressed.
  • the sixth attribute is used to represent the index address of the compressed data block where the data page of the data block is located.
  • the seventh attribute is used to represent that when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the offset of the data block in the set corresponding to the compressed data block; when When the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the distance between the data page of the data block and the first compressed page of the compressed data block.
  • the attribute information includes a third attribute
  • the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data is established, specifically: when j When the data page of each data block in a data block is the first compressed page of the ith compressed data block, the attribute value of the third attribute is assigned a value of 1. When the data page of each data block in the j data blocks is not the first compressed page of the ith compressed data block, the attribute value of the third attribute is assigned a value of 0.
  • the attribute information includes the seventh attribute, and further includes: when the attribute value of the third attribute is 1, updating the offset of the data block in the set corresponding to the compressed data block when the attribute value of the seventh attribute is 1. shift.
  • the attribute value of the third attribute is 0, the distance between the data page of the data block and the first compressed page of the compressed data block is updated when the attribute value of the seventh attribute is 0.
  • the attribute information includes a fourth attribute
  • the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data is established, specifically: when j
  • the attribute value of the fourth attribute is assigned a value of 1.
  • the attribute value of the fourth attribute is assigned a value of 0.
  • the attribute information includes the second attribute, and the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data is established, specifically: when j When the data page of each data block in the data blocks is valid, the attribute value of the second attribute is assigned a value of 1. When the data page of each data block in the j data blocks is invalid, the attribute value of the second attribute is assigned a value of 0.
  • n compressed data blocks before compressing m data blocks using a preset compression algorithm to obtain n compressed data blocks sequentially, it also includes: obtaining a second set of data to be overwritten and written, the second set including p compressed data block, p is a positive integer greater than or equal to 1.
  • the data page of the first target data block is determined as the data page to be overwritten with data.
  • the first index is used to identify the storage location of the i-th compressed data block in the storage medium, and the attribute information contained in each of the j data blocks.
  • it also includes: reading the first index of the first data block to obtain the index address of the first compressed data block corresponding to the first data block, wherein the first index includes attribute information of the first data block .
  • An index of a first compressed data block corresponding to the first data block is read.
  • the first compressed data block is decompressed to obtain multiple data blocks corresponding to the first compressed data block, and the multiple data blocks include the first data block.
  • An offset of the first data block within the plurality of decompressed data blocks is determined. According to the offset of the first data block in the multiple decompressed data blocks, the data of the first data block can be obtained.
  • the embodiment of the present application provides a data compression device, which includes: a first acquisition unit configured to acquire m data blocks in a data area in a writable and writable file system, where m is a positive integer greater than or equal to 1.
  • the compression unit is used to compress m data blocks by using a preset compression algorithm to obtain n compressed data blocks in turn, wherein the first capacity of each compressed data block is the same, and the first capacity represents the post-compression processing that the compressed data block can contain The number of bytes of data, n is a positive integer greater than or equal to 1.
  • the update unit is configured to establish the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data, and record the mapping relationship between the first index and the j data blocks.
  • i is a positive integer greater than or equal to 1 and less than or equal to n
  • j is a positive integer greater than or equal to 1 and less than or equal to m.
  • the first index is used to identify the storage location of each data block contained in the j data blocks in the storage medium, and the attribute information contained in each data block contained in the j data blocks.
  • the data compression method provided by the embodiment of the present application can effectively improve the reading efficiency when reading a data block, and can ensure that the random reading scenario completes data reading with a small read amplification factor.
  • the attributes contained in the index of the data block can be modified, so that the compressed file on the storage device can be modified. It can be seen that the embodiment of the present application solves the problem of random read amplification in the existing read-write file system compression scheme, and at the same time solves the problem that the existing file system with fixed output compression mode cannot support data and metadata update.
  • the compression unit is configured to: sequentially allocate each data block in the m data blocks to the first set in a preset order.
  • the data capacity of the j data blocks in the first set is equal to the rated capacity of the first set, perform a compression operation on the j data blocks according to the set compression threshold, and obtain the i-th compressed data block.
  • the updating unit is used for: when the sum of the header data of the i-th compressed data block and the total data length of the compressed data and the set compression threshold is less than or equal to the total data length of the j data blocks When , establish the first index of each data block in the j data blocks.
  • the attribute information includes at least one of the following: a first attribute, which is used to indicate whether the storage location of the compressed data block where the data block is compressed is pre-allocated.
  • the second attribute is used to indicate whether the data page of the data block is valid.
  • the third attribute is used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block.
  • the fourth attribute is used to represent whether the data page of the data block is included in the compressed data pages of the two compressed blocks.
  • the fifth attribute is used to indicate whether the data page of the data block is a compressed page of the compressed data block after the data block is compressed.
  • the sixth attribute is used to represent the index address of the compressed data block where the data page of the data block is located.
  • the seventh attribute is used to indicate that when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the offset of the data block in the set corresponding to the compressed data block.
  • the attribute value of the seventh attribute is the distance between the data page of the data block and the first compressed page of the compressed data block.
  • the attribute information includes a third attribute
  • the update unit is further configured to: when the data page of each data block in the j data blocks is the first compressed page of the i-th compressed data block, The attribute value of the third attribute is assigned a value of 1. When the data page of each data block in the j data blocks is not the first compressed page of the ith compressed data block, the attribute value of the third attribute is assigned a value of 0.
  • the attribute information includes a seventh attribute
  • the updating unit is further configured to: when the attribute value of the third attribute is 1, update the attribute value of the seventh attribute to the set corresponding to the data block in the compressed data block offset within.
  • the attribute value of the third attribute is 0, the distance between the data page of the data block and the first compressed page of the compressed data block is updated when the attribute value of the seventh attribute is 0.
  • the attribute information includes a fourth attribute
  • the update unit is further configured to: when the data pages of each data block in the j data blocks are included in the compressed data pages of two compressed blocks, the first The attribute values of the four attributes are assigned a value of 1. When the data page of each data block in the j data blocks is not included in the compressed data pages of the two compressed blocks, the attribute value of the fourth attribute is assigned a value of 0.
  • the attribute information includes a second attribute
  • the updating unit is further configured to: assign a value of 1 to the attribute value of the second attribute when the data page of each data block in the j data blocks is valid.
  • the attribute value of the second attribute is assigned a value of 0.
  • the device further includes: a second acquiring unit, configured to acquire a second set of data to be overwritten and written, the second set includes p compressed data blocks, and p is a positive integer greater than or equal to 1.
  • the third obtaining unit is used to obtain the compressed page of the first target compressed data in the p compressed data blocks, and the q data blocks corresponding to the compressed page of the first target compressed data block, where q is a positive integer greater than or equal to 1.
  • the first determining unit is configured to determine the position offset of the first target data block among the q data blocks among the q data blocks.
  • the second determining unit is configured to determine that the data page of the first target data block is the data page to be overwritten with data.
  • the first index is used to identify the storage location of the i-th compressed data block in the storage medium, and the attribute information contained in each of the j data blocks.
  • it further includes: a first reading unit, configured to read the first index of the first data block, and obtain the index address of the first compressed data block corresponding to the first data block, wherein the first index It includes attribute information of the first data block.
  • the second reading unit is configured to read the index of the first compressed data block corresponding to the first data block.
  • the decompression unit is configured to decompress the first compressed data block according to the index of the first compressed data block to obtain multiple data blocks corresponding to the first compressed data block, the multiple data blocks including the first data block.
  • the third determining unit is configured to determine the offset of the first data block in the decompressed multiple data blocks.
  • the third obtaining unit is configured to obtain the data of the first data block according to the offset of the first data block in the decompressed multiple data blocks.
  • an embodiment of the present application provides a device, which includes: configured to execute the data compression method in the first aspect.
  • an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium includes computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the method described in the first aspect. Data compression method.
  • an embodiment of the present application provides a computer program, and when the program is called by a processor, the data compression method in the first aspect is executed.
  • an embodiment of the present application provides a chip system, which includes one or more processors, and when the one or more processors execute instructions, the one or more processors execute the data of the first aspect compression method.
  • FIG. 1a is a block diagram of an operating system provided by an embodiment of the present application.
  • FIG. 1b is a schematic structural diagram of a storage system provided by an embodiment of the present application.
  • Fig. 2 is a schematic structural diagram of a solid-state hard disk of the storage system in Fig. 1b;
  • Fig. 3 is the structural representation of the flash memory chip of solid-state hard disk in Fig. 2;
  • FIG. 4 is a schematic diagram of a flash memory translation layer corresponding to the flash memory chip in FIG. 3;
  • FIG. 5 is a schematic diagram of a fixed input compression mode
  • FIG. 6 is a schematic diagram of a fixed output compression mode provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a data block index provided by an embodiment of the present application.
  • Fig. 8 is a schematic diagram of a data block index in an existing scalable read-only file system
  • FIG. 9 is a schematic flow chart of a data compression method provided by an embodiment of the present application.
  • FIG. 10 is a schematic flow chart of updating a data block index provided by an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a data block index relationship during data compression provided by an embodiment of the present application.
  • FIG. 12 is a schematic flowchart of another data compression method provided by the embodiment of the present application.
  • Fig. 13 is a schematic diagram of data block index relationship during an overwrite writing or reading process provided by an embodiment of the present application
  • Fig. 14 is a schematic flow chart of data reading provided by the embodiment of the present application.
  • FIG. 15 is a schematic structural diagram of a data compression device provided by an embodiment of the present application.
  • FIG. 1a A block diagram of an operating system as shown in Figure 1a.
  • An operating system is a computer program that manages computer hardware and software resources. For example, unix operating system, windows operating system, linux operating system, etc.
  • the operating system needs to handle basic tasks such as managing and configuring memory, prioritizing the supply and demand of system resources, controlling input and output devices, operating the network, and managing the file system.
  • the operating system also provides an interface for the user to interact with the system.
  • the operating system kernel refers to the core part of most operating systems. It consists of those parts of the operating system used to manage memory, files, peripherals, and system resources. It is responsible for managing the system's processes, memory, device drivers, files, and network systems, and determines the performance and stability of the operating system.
  • the operating system kernel is a system software that provides functions such as hardware abstraction layer, disk and file system control, and multitasking. It provides secure access to computer hardware for many applications, and it can determine when and how long an application will operate on a certain part of the computer's hardware. Since it is very complicated to directly operate on computer hardware, the operating system kernel can provide a set of hardware abstraction methods to complete these operations.
  • the file system is the core module of the operating system kernel, that is, the main component.
  • the file system is a method of organizing files on the storage device, responsible for managing and storing file information, mainly for users to create files, store, read, modify, Dump files, control file access, revoke files when they are no longer used by users, etc.
  • the file system provides an abstract representation of files in the kernel, completes the mapping of files to physical storage devices (such as disks, hard disks, etc.), and maps the physical addresses of files on storage devices into user-visible path names and file names to facilitate file data. Fast reading, modification and persistence of data.
  • File systems include read-write file systems and read-only file systems.
  • a read-write file system is a file system that can write files to storage devices and read files from storage devices, such as: file allocation table (file allocation table, FAT), high performance file system (high performance file system, HPFS) , new technology file system (NTFS), fourth extended file system (fourth extended file system, EXT4), flash friendly file system (flash friendly file system, F2FS), etc.
  • a read-only file system is a file system that can only read files from a storage device, but cannot write files to a storage device, such as an extendable read-only file system (EROFS).
  • EROFS extendable read-only file system
  • FIG. 1b A schematic structural diagram of a storage system shown in FIG. 1b.
  • the application server 100 may be a physical machine or a virtual machine. Physical application servers include, but are not limited to, desktops, servers, laptops, and mobile devices.
  • the application server accesses the storage system through the optical fiber switch 110 to access data.
  • the switch 110 is only an optional device, and the application server 100 can also directly communicate with the storage system 120 through the network.
  • the optical fiber switch 110 can also be replaced with an Ethernet switch, an InfiniBand switch, a RoCE (RDMA over Converged Ethernet) switch, or the like.
  • the storage system 120 shown in FIG. 1b is a centralized storage system.
  • the so-called centralized storage system refers to a central node composed of one or more master devices, where data is stored centrally, and all data processing services of the entire system are centrally deployed on this central node.
  • the terminal or client is only responsible for the input and output of data, while the storage and control processing of data is completely handed over to the central node.
  • the characteristic of the centralized storage system is that there is a unified entrance, and all data from external devices must pass through this entrance, and this entrance is the engine 121 of the centralized storage system.
  • the engine 121 is the most core component in the centralized storage system, where many advanced functions of the storage system are implemented.
  • FIG. 1 b there are one or more controllers in the engine 121 .
  • FIG. 1 b takes the engine including two controllers as an example for illustration.
  • controller 0 and controller 1 There is a mirror channel between controller 0 and controller 1, so when controller 0 writes a piece of data into its memory 124, it can send a copy of the data to controller 1 through the mirror channel, and controller 1 Store the copy in its own local memory 124 . Therefore, controller 0 and controller 1 are mutual backups.
  • controller 0 fails, controller 1 can take over the business of controller 0.
  • controller 1 fails, controller 0 can take over the business of controller 1. business, so as to avoid the unavailability of the entire storage system 120 caused by hardware failure.
  • four controllers are deployed in the engine 121, there is a mirroring channel between any two controllers, so any two controllers are mutual backups.
  • the engine 121 also includes a front-end interface 125 and a back-end interface 126 , wherein the front-end interface 125 is used to communicate with the application server 100 to provide storage services for the application server 100 .
  • the back-end interface 126 is used to communicate with the hard disk 134 to expand the capacity of the storage system. Through the back-end interface 126, the engine 121 can be connected with more hard disks 134, thereby forming a very large storage resource pool.
  • the disk enclosure 130 may be a SAS disk enclosure, or an NVMe disk enclosure, an IP disk enclosure, or other types of disk enclosures.
  • the SAS hard disk enclosure adopts the SAS3.0 protocol, and each enclosure supports 25 SAS hard disks.
  • the engine 121 is connected to the hard disk enclosure 130 through an onboard SAS interface or a SAS interface module.
  • the NVMe disk enclosure is more like a complete computer system, and the NVMe disk is inserted into the NVMe disk enclosure. The NVMe disk enclosure is then connected to the engine 121 through the RDMA port.
  • the controller 0 includes at least a processor 123 and a memory 124 .
  • Processor 123 is a central processing unit (central processing unit, CPU), used for processing data access requests from outside the storage system (server or other storage systems), and also used for processing requests generated inside the storage system.
  • CPU central processing unit
  • the processor 123 receives the write data request sent by the application server 100 through the front-end port 125 , it will temporarily save the data in the write data request in the memory 124 .
  • the processor 123 sends the data stored in the memory 124 to the hard disk 134 for persistent storage through the back-end port.
  • the memory 124 refers to an internal memory directly exchanging data with the processor. It can read and write data at any time, and the speed is very fast. It is used as a temporary data storage for an operating system or other running programs.
  • the memory includes at least two kinds of memory, for example, the memory can be random access memory.
  • the random access memory is dynamic random access memory (Dynamic Random Access Memory, DRAM), or storage class memory (Storage Class Memory, SCM).
  • DRAM Dynamic Random Access Memory
  • SCM Storage Class Memory
  • DRAM is a semiconductor memory, which, like most Random Access Memory (RAM), is a volatile memory device.
  • SCM is a composite storage technology that combines the characteristics of traditional storage devices and memory.
  • Storage-class memory can provide faster read and write speeds than hard disks, but the access speed is slower than DRAM, and the cost is also cheaper than DRAM.
  • the DRAM and the SCM are only exemplary illustrations in this embodiment, and the memory may also include other random access memories, such as Static Random Access Memory (Static Random Access Memory, SRAM) and the like.
  • the memory 124 can also be a dual-in-line memory module or a dual-line memory module (Dual In-line Memory Module, DIMM for short), that is, a module composed of dynamic random access memory (DRAM), or a solid-state hard drive. (Solid State Disk, SSD).
  • DIMM Dual In-line Memory Module
  • SSD Solid State Disk
  • multiple memories 124 and different types of memories 124 may be configured in the controller 0 .
  • the memory 124 can be configured to have a power saving function.
  • the power saving function means that the data stored in the internal memory 124 will not be lost when the system is powered off and then powered on again.
  • Memory with a power saving function is called non-volatile memory.
  • both the memory 124 and the hard disk 134 may be a solid-state disk (English: Solid-state drive or Solid-state disk, SSD for short), which is a storage device mainly using flash memory (NAND Flash) as a permanent memory.
  • the SSD 200 includes a NAND flash memory and a main controller (referred to as main control) 201.
  • NAND flash memory includes multiple flash memory chips 205 for storing data.
  • the main control 201 is the brain center of the SSD, responsible for some complex tasks, such as managing data storage, maintaining SSD performance and service life, and so on.
  • the main control 201 is an embedded microchip, which includes a processor 202, and its function is like a command center, sending out all operation requests of the SSD.
  • the processor 202 can perform functions such as reading/writing data, garbage collection, and wear leveling through firmware in the buffer.
  • the SSD master 201 also includes a host interface 204 and several channel controllers. Wherein, the host interface 204 is used for communicating with the host.
  • the host here can refer to any device such as a server, a personal computer, or an array controller.
  • the main control 201 can operate multiple flash memory chips 205 in parallel, thereby increasing the underlying bandwidth. For example, assuming that there are 8 channels between the main control 201 and the FLASH particles, then the main control 201 reads and writes data to 8 flash memory chips 205 in parallel through these 8 channels.
  • a die is a package of one or more flash memory chips.
  • a die can contain multiple panels.
  • Multi-Plane NAND is a design that can effectively improve performance.
  • a die is divided into two Planes, and the block numbers in the two Planes are single and double crossover. Therefore, during operation, a single and double crossover operation can be performed to improve performance.
  • a panel contains multiple blocks (block).
  • a whole flash memory chip is composed of two panels. One panel stores blocks with odd numbers, and the other stores blocks with even numbers. Two planes can be parallelized. operate. This is just an example, and the size of the page, the capacity of the block, and the capacity of the flash memory chip may have different specifications, which are not limited in this embodiment.
  • the host writes data into the block, and when a block is full, the SSD master 201 will select the next block to continue writing.
  • a page is the smallest unit of data writing.
  • the master control 201 writes data into the block at the page granularity.
  • Block is the smallest unit of data erasure. When the master control erases data, it can only erase the entire block at a time.
  • the host accesses the SSD through the logical block address (Logical Block Address, LBA), each LBA represents a sector (take 512B as an example), and inside the SSD, the host accesses the SSD in units of pages (take 4KB as an example). Therefore, every time the application server writes a piece of data, the SSD master will find a Page to write the data into. The address of the page is called the Physical Block Address (PBA). SSD internally records a mapping from LBA to PBA. With such a mapping, the next time the host needs to read the data of a certain LBA, the SSD will know where to read the data from the flash memory chip.
  • LBA Logical Block Address
  • FIG. 4 is a schematic diagram of a flash translation layer (Flash Translation Layer, FTL), and the FTL is located in the firmware of the processor 202.
  • FTL Flash Translation Layer
  • every time the host writes a new data a new mapping relationship will be generated, and this mapping relationship will be added (first write) or changed (overwritten) to the FTL.
  • the SSD When reading a certain data, the SSD first searches for the PBA corresponding to the LBA of the data in the FTL, and then reads the corresponding data according to the PBA.
  • the flash memory chip cannot support overwriting, which means that when the host modifies the data on a certain LBA, it cannot be directly changed on the PBA corresponding to this LBA, but must be written to a new PBA, and a mapping is added in the FTL .
  • PBA E a new location
  • Invalid data also called garbage data refers to data that is not pointed to by any mapping relationship.
  • the memory 124 also stores a software program, and the processor 123 runs the software program in the memory 124 to manage the hard disk.
  • hard disks are abstracted into storage resource pools, and then divided into LUNs for use by servers.
  • the LUN here is actually the hard disk seen on the server.
  • some centralized storage systems are also file servers themselves, which can provide shared file services for servers.
  • Data stored in memory 124 may be represented by a file system.
  • the file system is a structured form of data file storage and organization. We know that all the data in the computer are 0 and 1, and a series of 01 combinations stored on the hardware media are completely indistinguishable and manageable for us. Therefore, we use the concept of "file" to organize these data, and the data used for the same purpose can be composed of different types of files according to the structure required by different applications. Usually different suffixes are used to refer to different types, and then we give each file a name that is easy to understand and remember. And when there are many files, we group these files according to a certain division method, and each group of files is placed in the same directory (or folder).
  • File System File System
  • FAT/FAT32/NTFS of Windows EXT2/EXT3/EXT4/XFS/BtrFS of Linux, etc.
  • a file path such as "/etc/systemd/system.conf” in Linux or "C: ⁇ Windows ⁇ System32 ⁇ taskmgr.exe” in Windows.
  • a path is a unique identifier for accessing a specific file. For example, D: ⁇ data ⁇ file.exe under Windows is the path of a file, which represents the file.exe file under the data directory under the D partition.
  • the file system is built on the block device.
  • the file system not only records the file path, but also records which blocks form a file, and which blocks record directory/subdirectory information.
  • Different file systems have different organizational structures.
  • a block device such as a hard disk can usually be divided into multiple logical block devices, that is, a hard disk partition (Partition).
  • Partition the capacity and performance of a single medium are limited, and multiple physical block devices can be combined into a logical block device through certain technical means, such as various levels of RAID, JBOD, etc.
  • File systems can also be built on top of these logical block devices.
  • the application server application does not need to care about the specific location of the underlying block device where the file to be accessed is located. It only needs to send the file name/ID of the file to the file system, and the file system will query the file according to the file name/ID. Just the path.
  • Common file access protocols are NFS, CIFS, or SMB, which are not limited in this embodiment.
  • the file system in this application is a read-write file system.
  • a read-write file system is a file system that can write files to storage devices and read files from storage devices, such as FAT, HPFS, NTFS, EXT4, F2FS, etc.
  • a file system generally includes a metadata area and a data area, and the metadata area includes a super block and an index node (inode) area.
  • the super block of the metadata area can include the control information of the file system, data structure, etc.
  • the inode area of the metadata area can include the description information of the file, such as file length, file type, etc., and the file type is, for example, a regular inode ), directory file (directory inode), soft link (symbol link inode), special file (special inode), etc.
  • the data stored in the data area may be data obtained after file-level compression processing based on a lossless compression technology.
  • the data in the data area is stored in the physical storage space of the storage medium (for example, disk, flash memory, etc.) according to a set of disk blocks.
  • the data of the same file may be stored in continuous disk blocks, or may also be interleavedly stored in discontinuous disk blocks.
  • the introduction of the concept of a disk block in the present application does not mean that the storage medium is only limited to a disk, and the disk block can be used to represent a small physical storage space obtained by dividing the physical storage space of the storage medium.
  • the storage system in this application may also include a distributed storage system.
  • the so-called distributed storage system refers to a system that stores data dispersedly on multiple independent storage nodes.
  • Traditional network storage systems use centralized storage arrays to store all data.
  • the performance of storage arrays is not only the bottleneck of system performance, but also the focus of reliability and security, which cannot meet the needs of large-scale storage applications.
  • the speed of the device is sorted according to the data read and write capabilities, and the order from strong to weak is as follows: central processing unit (central processing unit, CPU) >> double data rate synchronous DRAM (double data rate synchronous dynamic random access memory, DDR SDRAM) > flash memory chip falsh. It can be seen that the bottleneck of data access in the storage system is the IO (input output) time overhead of data between memory and flash.
  • CPU central processing unit
  • IO input output
  • the files in the memory need to be compressed. Since the metadata area in the file system accounts for a small proportion of the entire file system, the data area often occupies a relatively high device storage capacity. Therefore, when writing data to the falsh, compressing the data and writing the compressed data into the falsh can reduce the storage capacity of the falsh and prolong the service life of the falsh.
  • the read-write file system of Linux such as: F2FS, the second generation flash file system (journalling Flash file system version2, JFFS2), B-tree file system (B-tree file system, BTRFS), etc.
  • the read-write file system of Windows for example: NTFS, etc.
  • F2FS the second generation flash file system
  • JFFS2 journalling Flash file system version2, JFFS2
  • B-tree file system B-tree file system
  • NTFS NTFS
  • the original file data (or source data) that needs to be compressed is divided into the smallest compressible unit of fixed size
  • the compressed file data (or compressed data) includes header data and compressed data.
  • the header data is used to represent the attribute information of the file data
  • the compressed data is used to represent the content of the file data. Then save the compressed file data to flash and align it with a size of 4kb.
  • 4 data blocks (blocks) with continuous addresses are compressed as a cluster0 to obtain a cluster composed of header data (header)+compressed data (compressed data). Compress file data. If the compressed file data is less than 4kb, the compressed file data will be stored on the flash in 4kb size.
  • the size of the original file data (or source data) shown in Figure 5 is 4 blocks, each block is 4kb in size, one block is a logical page, and the logical pages of the original file data are numbered: 0, 1, 2, 3.
  • the original file data is compressed into compressed file data according to a compression ratio of 75%, and the size of the compressed file data is 12kb. Therefore, the data size of the compressed file is 3 blocks, so the actual page of the compressed file data is 3 pages, that is, the size of the actual falsh page that needs to be read to read a single logical page is shown in Table 1.
  • the compressed file data obtained through the data compression method shown in FIG. 5 has low reading efficiency in a random reading scenario.
  • the embodiment of the present application provides a data compression method, the method obtains m data blocks in the data area of the readable and writable file system. Compress m data blocks using a preset compression algorithm to obtain n compressed data blocks in sequence, wherein the first capacity of each compressed data block is the same, and the first capacity represents the bytes of compressed data that the compressed data block can contain number.
  • m and n are both positive integers greater than or equal to 1.
  • the preset compression algorithm may be a compression algorithm corresponding to a fixed output compression mode, such as (lempel-ziv 4, LZ4) compression algorithm.
  • the preset compression algorithm may also be other compression algorithms, which are not specifically limited in this embodiment of the present application.
  • the size of the source data is 16kb, taking the data of 4kb as a data block and also a logical page as an example, the logical pages of the source data are numbered: 0, 1, 2, 3, as shown in the first row of Table 2.
  • the continuous 16kb source data of the logical page is divided into three parts: 6kb, 7kb, and 5kb.
  • the three pieces of data are compressed using a preset compression algorithm (for example, LZ4) until the size of each piece of compressed data in the compressed data block is 4kb.
  • the data pages of the compressed data block are three pages, which are numbered respectively: compressed page 4, compressed page 5 and compressed page 6 as shown in FIG. 6 .
  • the source data of logical page 0 is compressed into compressed page 4, so it is compressed into 1 page.
  • a part of source data of logical page 1 is compressed in compressed page 4, and another part of source data of logical page 1 is compressed in compressed page 5, so it is compressed into 2 pages.
  • a part of source data of logical page 2 is compressed in compressed page 5, and another part of source data of logical page 2 is compressed in compressed page 6, so it is compressed into 2 pages.
  • the source data of logical page 3 is all compressed in compressed page 6, so it is compressed into 1 page.
  • any one or more of the above logical pages may be read.
  • the read efficiency can be calculated according to the following formula 2:
  • the read efficiency of reading logical page 3 is the same as that of reading logical page 0.
  • the read efficiency of reading logical page 2 is the same as that of reading logical page 1.
  • m data blocks in the data area of the readable and writable file system are compressed using a compression algorithm corresponding to a fixed output compression mode, and n compressed data blocks of data with the same number of bytes are sequentially obtained, so that the output Each compressed data block of is a fixed size.
  • the above data block index can find the corresponding data block.
  • the data block index is also an inode, that is, metadata.
  • the inode is the area used to store metadata, that is, the area used to store file-related attribute information, such as: the creator of the file, the date of creation, the size, the location of the data block, and so on.
  • each inode has a number, and the operating system uses different inode numbers to identify different files.
  • the user opens the file through the file name. In fact, the system first finds the corresponding inode number according to the file name, then obtains the inode information through the inode number, and then finds the address of the data block according to the inode information, and reads the data.
  • the inode records the attributes of the file and the actual storage location of the file, that is, the data block number (block number), each block (common size is 4KB), and the file can be searched and located through the inode.
  • Inode is in Linux and vnode in Unix.
  • the information contained in the inode is at least the following: (1) the type of the file; (2) the file access permission; (3) the owner and group of the file; (4) the size of the file; (5) the number of links, which point to The total number of file names of the inode; (6) the state change time (ctime), the latest access time (atime) and the latest modification time (mtime) of the file; (7) the special attributes of the file, SUID, SGID and SBIT; (8) the file The actual pointer of the content.
  • FIG. 8 it is an existing data block index format, which does not support scalability, for example, does not support additional writing, block reservation, truncate, and the like.
  • appending refers to adding new content on the basis of the original file without deleting the content in the original file.
  • Block reservation refers to the fact that the file system considers in advance where the disk blocks can be allocated if the file grows, and reserves these disk blocks.
  • Truncate refers to modifying files, such as deleting, adding, and so on.
  • the data block index is referred to by blk entry, which is abbreviated as blk for ease of description.
  • blk 1 is the index of the compressed data block 1
  • blk2 is an index of the compressed data block 2
  • the address of the compressed data block 2 on the storage device is stored in blk2.
  • blk3 is an index of the compressed data block 3
  • the address of the compressed data block 3 on the storage device is stored in blk3.
  • blk4 is an index of the compressed data block 4, and the address of the compressed data block 4 on the storage device is stored in blk4. Therefore, the location of the compressed data block on the storage device can be determined according to the address stored in the blk.
  • the data compression method provided by the embodiment of the present application further includes: establishing j data blocks corresponding to the i-th compressed data block in the n compressed data
  • the first index of each data block in and record the mapping relationship between the first index and j data blocks.
  • i is a positive integer greater than or equal to 1 and less than or equal to n.
  • j is a positive integer greater than or equal to 1 and less than or equal to m.
  • the first index is used to identify the storage location of each data block contained in the j data blocks in the storage medium, and the attribute information contained in each data block contained in the j data blocks.
  • the first index is also used to identify the storage location of the i-th compressed data block in the storage medium, and the attribute information contained in each of the j data blocks.
  • At least one of the following attribute information is:
  • the first attribute is used to indicate whether the storage location of the compressed data block where the data block is compressed is pre-allocated.
  • the second attribute is used to represent whether the data page of the data block is valid; that is, whether it is a normal data page or a hole data page, where the hole data page can be understood as a blank data page.
  • the third attribute is used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block.
  • the fourth attribute is used to represent whether the data page of the data block is included in the compressed data pages of the two compressed blocks.
  • the fifth attribute is used to indicate whether the data page of the data block is a compressed page of the compressed data block after the data block is compressed.
  • the sixth attribute is used to represent the index address of the compressed data block where the data page of the data block is located.
  • the seventh attribute is used to represent that when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the offset of the data block in the set corresponding to the compressed data block; when When the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the distance between the data page of the data block and the first compressed page of the compressed data block.
  • the first index of the data block includes: blk entry, which corresponds to storing the address of the data block or compressed data block; and extent entry, which corresponds to storing extended attribute information.
  • each extent entry corresponds to a blk entry one by one, and each data page has a corresponding extent entry and blk entry.
  • Extent entry members are as follows:
  • the members included in the data block index may be shown in set A, and it should be noted that each data page has a corresponding set A.
  • first_page is the above-mentioned third attribute.
  • cross_block is the fourth attribute mentioned above.
  • is_compress is the fifth attribute mentioned above.
  • blkidx is the sixth attribute mentioned above.
  • FIG. 9 is a schematic flowchart of a data compression method provided by an embodiment of the present application. As shown in Figure 9, the method includes:
  • the m data blocks can be understood as data blocks that need to be written back.
  • write-back may refer to writing data into memory first for caching during a write operation, but not immediately writing data into a storage device (for example: disk).
  • the data cached in the memory will be written to the storage device only under some specific conditions or operations (for example: a refresh mechanism, a synchronization (sync) operation, etc.).
  • the preset compression algorithm may be the LZ4 compression algorithm, and of course, other fixed-output compression algorithms may also be used, which are not specifically limited in this embodiment of the present application.
  • n can be any positive integer.
  • m is 4, m is 10, or m is 20.
  • S902 can be specifically implemented as:
  • the preset sequence can store addresses consecutively. That is, a contiguous sequence of m data blocks.
  • This first set may be referred to as the smallest compressible unit (cluster).
  • the first set is the smallest compressible set of data blocks.
  • a 6kb data block set, a 7kb data block set, and a 5kb data block set are shown in FIG. 6 .
  • m data blocks are mapped to a continuous address in the storage medium. Assuming a data block as the starting point, according to the order of the address of the data block mapped in the storage medium, divide the fixed-size data set in sequence, as shown in Figure 6, the data block 0 and 1/2 of the data block 1 form 6kb 1/2 data of data block 1, 3/4 data of data block 2, and blank data pages form a 7kb data set; 1/4 data of data block 2 and data block 3 form a 5kb data set collection of data.
  • S9022 Determine whether the data capacity of the j data blocks in the first set is equal to the rated capacity of the first set, where j is a positive integer greater than or equal to 1 and less than or equal to m. If the data capacity of the j data blocks is not equal to the rated capacity of the first set, execute S9021; if the data capacity of the j data blocks is equal to the rated capacity of the first set, execute S9023.
  • a compression threshold is set to characterize the compression rate.
  • the expression formula for setting the compression threshold may be:
  • S9024 Determine whether the total data length of the j data blocks is greater than the sum of the header data of the ith compressed data block, the total data length of the compressed data, and the set compression threshold. If yes, execute S903. Otherwise, commit the source data page to flash.
  • the first index is used to identify the storage location of each data block contained in the j data blocks in the storage medium, and the attribute information contained in each data block contained in the j data blocks.
  • the first index format of the data block on F2fs can be:
  • the data structure of the first index containing attribute information may be:
  • Entry data structure For example, Entry data structure:
  • the attribute information may include at least one of the following:
  • the first attribute (is_reserved) is used to represent whether the storage location of the compressed data block where the data block is compressed is pre-allocated;
  • the second attribute (is_valid) is used to represent whether the data page of the data block is valid
  • the third attribute (first_page) is used to represent whether the data page of the data block is the first compressed page of the compressed data block of the data block;
  • cross_block is used to represent whether the data page of the data block is included in the compressed data pages of the two compressed blocks
  • the fifth attribute is used to represent whether the data page of the data block is a compressed page of the compressed data block after the data block is compressed;
  • the sixth attribute (blkidx) is used to represent the index address of the compressed data block where the data page of the data block is located.
  • the seventh attribute (ofs) is used to represent when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the offset of the data block in the set corresponding to the compressed data block When the data page of the data block does not belong to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is the distance between the data page of the data block and the first compressed page of the compressed data block.
  • the attribute information may include a third attribute (first_page) and a seventh attribute (ofs), and S903 may specifically be implemented as:
  • attribute information may also include a fourth attribute (cross_block), and S103 may be specifically implemented as:
  • attribute information can also include the second attribute (is_valid), and S103 can be specifically implemented as:
  • attribute information may also include a sixth attribute (blkidx), and S103 may be specifically implemented as:
  • the compression is performed with the size of the smallest fixed compression unit (such as the first set).
  • the data pages of the complete data block corresponding to the first compressed data block are all at the index positions of the first compressed data block.
  • the data blocks corresponding to the first compressed data block include partial data of data block 0, data block 1, data block 2, and data block 3.
  • the complete data blocks corresponding to the first compressed data block are data block 0, data block 1 and data block 2. Therefore, the data pages of data block 0, data block 1, and data block 2 are at the index position of the first compressed data block.
  • the data block index update processes corresponding to other attributes are independent of each other.
  • m data blocks include data block 0 (ie, block0), data block 1 (ie, block1), data block 2 (ie, block2), and data block 3 (ie, block3).
  • block0, block1, block2, and block3 map a continuous address in memory.
  • the compression is performed with the size of the smallest fixed compression unit (such as the first set).
  • the data page of block0 falls on the first compressed page of the first compressed data block, so first_page is assigned a value of 1.
  • the data page of block0 only falls on the first compressed page of the first compressed data block, so the assignment value of cross_block is 0.
  • the index address of the data page of block0 in the first compressed data block is the serial number of the first compressed data block (compress blk0), so the assignment value of blkidx is 0.
  • the data page of block0 falls on the first compressed page of the first compressed data block, and the offset of block0 in its corresponding first set is 0, so ofs is assigned a value of 0.
  • the data page of block0 is a valid data page, so the assignment of is_valid is 1.
  • the data page of block1 falls on the first compressed page of the second compressed data block, so first_page is assigned a value of 1.
  • the data page of Block1 falls on the compressed pages of the first compressed data block and the second compressed data block, so cross_block is assigned a value of 1.
  • the index address of the data page of Block1 in the second compressed data block is the serial number of the second compressed data block (compress blk1), so the assignment value of blkidx is 1.
  • the data page of Block1 falls on the first compressed page of the second compressed data block, and the offset of block1 in the set of data blocks is Ofs1, so ofs is assigned the value Ofs1.
  • the data page of Block1 is a valid data page, so the value of is_valid is 1.
  • the data page of block2 does not fall on the first compressed page of the second compressed data block, so first_page is assigned a value of 0.
  • the data page of Block2 only falls on the compressed page of the second compressed data block, so the assignment value of cross_block is 0.
  • the index address of the data page of Block2 in the second compressed data block is the serial number of the second compressed data block (compress blk1), so the assignment value of blkidx is 1.
  • the data page of Block2 does not fall on the first compressed page of the second compressed data block, and the distance between the data page of block2 and the first compressed page of the first compressed data block is 1, so ofs is assigned a value of 1.
  • the data page of Block2 is a valid data page, so the value of is_valid is 1.
  • the third compression is performed to obtain the third compressed data block (compress blk2).
  • the data block index of block3 is established, as shown in Table 5:
  • the data page of block3 falls on the first compressed page of the third compressed data block, so first_page is assigned a value of 1.
  • the data page of Block1 falls on the compressed pages of the second compressed data block and the third compressed data block, so cross_block is assigned a value of 1.
  • the index address of the data page of Block1 in the third compressed data block is the serial number of the second compressed data block (compress blk2), so the assignment value of blkidx is 2.
  • the data page of Block3 falls on the first compressed page of the third compressed data block, and the offset of block3 in the set of data blocks is Ofs2, so ofs is assigned a value of Ofs2.
  • the data page of Block3 is a valid data page, so the value of is_valid is 1.
  • FIG. 12 it is a schematic flowchart of a data compression method provided in the embodiment of the present application. As shown in FIG. 12, before performing S902, the data compression method provided in the embodiment of the present application further includes:
  • the second set may include p compressed data blocks, and p is a positive integer greater than or equal to 1.
  • the index address of each compressed data block in the second set may be read, and the compressed data block is decompressed to obtain each data block corresponding to the compressed data block. Then determine the position offset of each data block among the q data blocks.
  • FIG. 14 it is a schematic flow chart of data reading provided by the embodiment of the present application. As shown in Figure 14, the data reading process is as follows:
  • the attribute information of the first data block may include at least one of the first attribute to the seventh attribute in the foregoing embodiment.
  • the attribute information of the first data block including the third attribute (first_page), the fourth attribute (cross_block), the sixth attribute (blkidx) and the seventh attribute (ofs) as an example:
  • the first data block is data block 2 (block2) shown in FIG. 13 .
  • Read attribute information such as ofs, cross_block, blkidx of block2, and obtain the index address of the first compressed data block corresponding to block2.
  • the value of ofs read in block2 is 1, it can be determined that the data page of Block2 does not fall on the first compressed page of the second compressed data block, and the distance between the data page of block2 and the first compressed page can also be obtained.
  • the distance of the first compressed page of the compressed data block is 1.
  • the first compressed data block is found on the device according to the index of the first compressed data block. After the first compressed data block is found, the first compressed data block is parsed, and multiple parsed data blocks are obtained. For example, as shown in Figure 13, the first compressed data block is copress blk1, and after copress blk1 is parsed, it is obtained: part of the data of data block 1 (block1), part of data of data block 2 (block2), and part of data of data block 3 (block3) .
  • the first data block is data block 2 (block2) shown in FIG. 13 .
  • the expression of the offset (dstofs) of block2 in multiple data blocks parsed by copress blk1 is:
  • dstofs represents the offset of block2 in the multiple data blocks parsed by copress blk1; block_size represents the length of the data block; ofs1 represents the attribute value of the seventh attribute; ofs1% block_size represents the remainder.
  • the first data block is block2, as shown in FIG. 13 and Table 4, and the data of block2 can be obtained.
  • the communication system described in this possible design is used to perform the functions of each device in the data compression method shown in FIG. 9 , so the same effect as the above data compression method can be achieved.
  • Fig. 15 is a data compression device provided by the embodiment of the present application.
  • the data compression device 1500 may include: a first acquisition unit 1501, configured to acquire m data blocks in the data area of the writable and writable file system, where m is greater than or equal to A positive integer of 1.
  • the compression unit 1502 is configured to compress m data blocks using a preset compression algorithm to obtain n compressed data blocks in turn, wherein the first capacity of each compressed data block is the same, and the first capacity represents the compression process that the compressed data block can contain The number of bytes of the subsequent data, n is a positive integer greater than or equal to 1.
  • the updating unit 1503 is configured to establish the first index of each data block in the j data blocks corresponding to the i-th compressed data block in the n compressed data, and record the mapping relationship between the first index and the j data blocks.
  • i is a positive integer greater than or equal to 1 and less than or equal to n
  • j is a positive integer greater than or equal to 1 and less than or equal to m.
  • the first index is used to identify the storage location of each data block contained in the j data blocks in the storage medium, and the attribute information contained in each data block contained in the j data blocks.
  • the compression unit 1502 is configured to: sequentially allocate each data block in the m data blocks to the first set in a preset order. When the data capacity of the j data blocks in the first set is equal to the rated capacity of the first set, perform the compression operation on the j data blocks according to the set compression threshold, and obtain the ith compressed data block.
  • the updating unit 1503 is configured to: when the sum of the header data and the total data length of the compressed data of the ith compressed data block and the set compression threshold is less than or equal to the total data length of the j data blocks When the length is long, the first index of each data block in the j data blocks is established.
  • the attribute information includes at least one of the following: the first attribute is used to indicate whether the storage location of the compressed data block where the data block is compressed is pre-allocated; the second attribute is used to indicate Whether the data page of the data block is valid; the third attribute is used to indicate whether the data page of the data block is the first compressed page of the compressed data block of the data block; the fourth attribute is used to indicate whether the data page of the data block is included in Among the compressed data pages of the two compressed blocks; the fifth attribute is used to represent whether the data page of the data block is the compressed page of the compressed data block after the data block is compressed; the sixth attribute is used to represent the location of the data page of the data block The index address of the compressed data block; the seventh attribute is used to represent that when the data page of the data block belongs to the first compressed page of the compressed data block of the data block, the attribute value of the seventh attribute is corresponding to the compressed data block of the data block The offset within the collection; when the data page of the data block does not belong to the first
  • the attribute information includes a third attribute
  • the updating unit 1503 is further configured to: when the data page of each data block in the j data blocks is the first compressed page of the i-th compressed data block , the attribute value of the third attribute is assigned a value of 1. When the data page of each data block in the j data blocks is not the first compressed page of the ith compressed data block, the attribute value of the third attribute is assigned a value of 0.
  • the attribute information includes a seventh attribute
  • the updating unit 1503 is further configured to: when the attribute value of the third attribute is 1, update the attribute value of the seventh attribute in the data block corresponding to the compressed data block The offset within the set.
  • the attribute value of the third attribute is 0, the distance between the data page of the data block and the first compressed page of the compressed data block is updated when the attribute value of the seventh attribute is 0.
  • the attribute information includes a fourth attribute
  • the updating unit 1503 is further configured to: when the data page of each data block in the j data blocks is included in the compressed data pages of two compressed blocks, The attribute value of the fourth attribute is assigned a value of 1. When the data page of each data block in the j data blocks is not included in the compressed data pages of the two compressed blocks, the attribute value of the fourth attribute is assigned a value of 0.
  • the attribute information includes a second attribute
  • the updating unit is further configured to: assign a value of 1 to the attribute value of the second attribute when the data page of each data block in the j data blocks is valid.
  • the attribute value of the second attribute is assigned a value of 0.
  • a second obtaining unit 1504 configured to obtain a second set of data to be overwritten and written, the second set includes p compressed data blocks, and p is a positive integer greater than or equal to 1.
  • the third obtaining unit 1505 is configured to obtain the compressed page of the first target compressed data in the p compressed data blocks, and q data blocks corresponding to the compressed page of the first target compressed data block, where q is a positive integer greater than or equal to 1.
  • the first determining unit 1506 is configured to determine the position offset of the first target data block among the q data blocks among the q data blocks.
  • the second determining unit 1507 is configured to determine that the data page of the first target data block is the data page to be overwritten with data.
  • it further includes: a first reading unit, configured to read the first index of the first data block, and obtain the index address of the first compressed data block corresponding to the first data block, wherein the first index It includes attribute information of the first data block.
  • the second reading unit is configured to read the index of the first compressed data block corresponding to the first data block.
  • the decompression unit is configured to decompress the first compressed data block according to the index of the first compressed data block to obtain a plurality of data blocks corresponding to the first compressed data block, the plurality of data blocks including the first data block.
  • the third determining unit is configured to determine the offset of the first data block in the decompressed multiple data blocks.
  • the third obtaining unit is configured to obtain the data of the first data block according to the offset of the first data block in the decompressed multiple data blocks.
  • the first index is used to identify the storage location of the i-th compressed data block in the storage medium, and the attribute information contained in each of the j data blocks.
  • the data compression method provided by the embodiment of the present application can effectively improve the reading efficiency when reading a data block, and can ensure that the random reading scenario completes data reading with a small read amplification factor.
  • the attributes contained in the index of the data block can be modified, so that the compressed file on the storage device can be modified. It can be seen that the embodiment of the present application solves the problem of random read amplification in the existing read-write file system compression scheme, and at the same time solves the problem that the existing file system with fixed output compression mode cannot support data and metadata update.
  • An embodiment of the present application also provides a device, which includes: a unit for performing the steps described in any one of the foregoing, or a unit for performing the steps described in any one of the foregoing.
  • An embodiment of the present application also provides a computer-readable storage medium, including instructions, which, when run on a computer, cause the computer to execute any one of the above methods.
  • the embodiment of the present application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to execute any one of the above methods.
  • the embodiment of the present application also provides a chip, the chip includes a processor and an interface circuit, the interface circuit is coupled to the processor, the processor is used to run computer programs or instructions to implement the above method, and the interface circuit is used to communicate with other modules outside the chip to communicate.
  • words such as “exemplary” or “for example” are used to mean an example, illustration or illustration. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as “exemplary” or “such as” is intended to present related concepts in a concrete manner.
  • the disclosed devices and methods may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places . Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请公开了一种数据压缩方法及装置，该方法包括：获取可读写文件系统中数据区的m个数据块，采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块，其中，每个压缩数据块的第一容量相同，第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数。建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录第一索引与j个数据块的映射关系。其中，第一索引用于标识j个数据块包含中每个数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。当读取数据块时，可以有效提高读取效率，可以保证随机读场景以很小的读放大系数完成数据的读取。

Description

一种数据压缩方法及装置
本申请要求于2021年06月16日提交中国专利局、申请号为202110667882.7、申请名称为“一种数据压缩方法及装”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据压缩技术领域，尤其涉及一种数据压缩方法及装置。
背景技术
为了提升存储系统的整体的输入输出(input output，IO)读写性能，需要对内存中的文件进行压缩。目前，Linux的读写文件系统，例如：F2FS、第二代闪存文件系统(journalling Flash file system version2，JFFS2)、B树文件系统(B-tree file system，BTRFS)等，Windows的读写文件系统，例如：NTFS等。由于文件系统中元数据区占整个文件系统比例较小，数据区往往占据较高的设备存储容量。所以，压缩数据区的数据可以减小输入输出IO的大小，提升IO整体的读写性能。
现有的数据压缩方法通常是将需要压缩的原始文件数据(或称源数据)按照固定大小的最小可压缩单元进行压缩，压缩后的文件数据(或称压缩数据)可以包括头部数据和压缩数据。其中，头部数据用于表征文件数据的属性信息；压缩数据用于表征文件数据的内容。再将压缩后的文件数据保存到存储介质上。但是，现有的读写文件系统的压缩方案存在随机读放大的问题，读取效率低下。
发明内容
本申请实施例提供一种数据压缩方法及装置，可以解决读写文件系统的随机读放大的问题，提高读取效率。
第一方面，本申请实施例提供一种数据压缩方法，该方法的执行主体可以是电子设备，也可以是位于电子设备中的部件(例如，芯片，芯片系统或处理器等)，下面以执行主体是电子设备为例进行描述。该方法包括：电子设备获取可读写文件系统中数据区的m个数据块，m为大于等于1的正整数。电子设备采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块，其中，每个压缩数据块的第一容量相同，第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数，n为大于等于1的正整数。电子设备建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录第一索引与j个数据块的映射关系。其中，i为大于等于1，且小于等于n的正整数；j为大于等于1，且小于等于m的正整数。其中，第一索引用于标识j个数据块包含中每个数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
因此，本申请实施例提供的数据压缩方法，当读取数据块时，可以有效提高读取效率，可以保证随机读场景以很小的读放大系数完成数据的读取。另外，数据块的索引中包含的属性可以进行修改，从而可以对存储设备上的压缩文件进行修改。可见，本申请实施例解决已有读写文件系统压缩方案随机读放大的问题，同时解决已有的固 定输出压缩方式的文件系统无法支持数据和元数据更新的问题。
在一种具体可实现方式中，采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块，具体为：将m个数据块中的各数据块以预设顺序依次分配至第一集合。当第一集合中的j个数据块的数据容量等于第一集合的额定容量时，按照设定压缩的阈值，对j个数据块执行压缩操作，并得到第i个压缩数据块。
在一种具体可实现方式中，建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，具体为：当第i个压缩数据块的头部数据和压缩数据的数据总长度与设定压缩的阈值之和小于等于j个数据块的数据总长度时，建立j个数据块中每个数据块的第一索引。
在一种具体可实现方式中，属性信息包括以下至少一项：第一属性，用于表征数据块被压缩后所在的压缩数据块的存储位置是否是预先分配的。第二属性，用于表征数据块的数据页是否有效。第三属性，用于表征数据块的数据页是否是数据块的压缩数据块的第一个压缩页。第四属性，用于表征数据块的数据页是否包含在两个压缩块的压缩数据页中。第五属性，用于表征数据块的数据页是否是数据块被压缩后的压缩数据块的压缩页。第六属性，用于表征数据块的数据页所在的压缩数据块的索引地址。第七属性，用于表征当数据块的数据页属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块在压缩数据块对应的集合内的偏移；当数据块的数据页不属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
在一种具体可实现方式中，属性信息包括第三属性，建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，具体为：当j个数据块中每个数据块的数据页是第i个压缩数据块的第一个压缩页时，第三属性的属性值赋值为1。当j个数据块中每个数据块的数据页不是第i个压缩数据块的第一个压缩页时，第三属性的属性值赋值为0。
在一种具体可实现方式中，属性信息包括第七属性，还包括：当第三属性的属性值为1时，更新第七属性的属性值为数据块在压缩数据块对应的集合内的偏移。当第三属性的属性值为0时，更新第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
在一种具体可实现方式中，属性信息包括第四属性，建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，具体为：当j个数据块中的每个数据块的数据页包含在两个压缩块的压缩数据页中时，第四属性的属性值赋值为1。当j个数据块中的每个数据块的数据页不包含在两个压缩块的压缩数据页中时，第四属性的属性值赋值为0。
在一种具体可实现方式中，属性信息包括第二属性，建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，具体为：当j个数据块中的每个数据块的数据页为有效时，第二属性的属性值赋值为1。当j个数据块中的每个数据块的数据页为无效时，第二属性的属性值赋值为0。
在一些可实现方式中，在采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块之前，还包括：获取待覆盖写入数据的第二集合，第二集合包括p个压缩数据 块，p为大于等于1的正整数。获取p个压缩数据块中第一目标压缩数据的压缩页，及第一目标压缩数据块的压缩页对应的q个数据块，q为大于等于1的正整数。确定q个数据块中的第一目标数据块在q个数据块中的位置偏移。确定第一目标数据块的数据页为待覆盖写入数据的数据页。
在一种具体可实现方式中，第一索引用于标识第i个压缩数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
在一些可实现方式中，还包括：读取第一数据块的第一索引，得到第一数据块对应的第一压缩数据块的索引地址，其中，第一索引包括第一数据块的属性信息。读取第一数据块对应的第一压缩数据块的索引。根据第一压缩数据块的索引，解压第一压缩数据块，得到第一压缩数据块对应的多个数据块，该多个数据块中包括第一数据块。确定第一数据块在解压后的多个数据块内的偏移。根据第一数据块在解压后的多个数据块内的偏移，可得到第一数据块的数据。
第二方面，本申请实施例提供一种数据压缩装置，该装置包括：第一获取单元，用于获取可读写文件系统中数据区的m个数据块，m为大于等于1的正整数。压缩单元，用于采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块，其中，每个压缩数据块的第一容量相同，第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数，n为大于等于1的正整数。更新单元，用于建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录第一索引与j个数据块的映射关系。其中，i为大于等于1，且小于等于n的正整数；j为大于等于1，且小于等于m的正整数。其中，第一索引用于标识j个数据块包含中每个数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
因此，本申请实施例提供的数据压缩方法，当读取数据块时，可以有效提高读取效率，可以保证随机读场景以很小的读放大系数完成数据的读取。另外，数据块的索引中包含的属性可以进行修改，从而可以对存储设备上的压缩文件进行修改。可见，本申请实施例解决已有读写文件系统压缩方案随机读放大的问题，同时解决已有的固定输出压缩方式的文件系统无法支持数据和元数据更新的问题。
在一种具体可实现方式中，压缩单元用于：将m个数据块中的各数据块以预设顺序依次分配至第一集合。当第一集合中的j个数据块的数据容量等于第一集合的额定容量时，按照设定压缩的阈值，对j个数据块执行压缩操作，并得到第i个压缩数据块。
在一种具体可实现方式中，更新单元用于：当第i个压缩数据块的头部数据和压缩数据的数据总长度与设定压缩的阈值之和小于等于j个数据块的数据总长度时，建立j个数据块中每个数据块的第一索引。
在一种具体可实现方式中，属性信息包括以下至少一项：第一属性，用于表征数据块被压缩后所在的压缩数据块的存储位置是否是预先分配的。第二属性，用于表征数据块的数据页是否有效。第三属性，用于表征数据块的数据页是否是数据块的压缩数据块的第一个压缩页。第四属性，用于表征数据块的数据页是否包含在两个压缩块的压缩数据页中。第五属性，用于表征数据块的数据页是否是数据块被压缩后的压缩数据块的压缩页。第六属性，用于表征数据块的数据页所在的压缩数据块的索引地址。第七属性，用于表征当数据块的数据页属于数据块的压缩数据块的第一个压缩页时， 第七属性的属性值为数据块在压缩数据块对应的集合内的偏移。当数据块的数据页不属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
在一种具体可实现方式中，属性信息包括第三属性，更新单元还用于：当j个数据块中每个数据块的数据页是第i个压缩数据块的第一个压缩页时，第三属性的属性值赋值为1。当j个数据块中每个数据块的数据页不是第i个压缩数据块的第一个压缩页时，第三属性的属性值赋值为0。
在一种具体可实现方式中，属性信息包括第七属性，更新单元还用于：当第三属性的属性值为1时，更新第七属性的属性值为数据块在压缩数据块对应的集合内的偏移。当第三属性的属性值为0时，更新第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
在一种具体可实现方式中，属性信息包括第四属性，更新单元还用于：当j个数据块中的每个数据块的数据页包含在两个压缩块的压缩数据页中时，第四属性的属性值赋值为1。当j个数据块中的每个数据块的数据页不包含在两个压缩块的压缩数据页中时，第四属性的属性值赋值为0。
在一种具体可实现方式中，属性信息包括第二属性，更新单元还用于：当j个数据块中的每个数据块的数据页为有效时，第二属性的属性值赋值为1。当j个数据块中的每个数据块的数据页为无效时，第二属性的属性值赋值为0。
在一些可实现方式中，该装置还包括：第二获取单元，用于获取待覆盖写入数据的第二集合，第二集合包括p个压缩数据块，p为大于等于1的正整数。第三获取单元，用于获取p个压缩数据块中第一目标压缩数据的压缩页，及第一目标压缩数据块的压缩页对应的q个数据块，q为大于等于1的正整数。第一确定单元，用于确定q个数据块中的第一目标数据块在q个数据块中的位置偏移。第二确定单元，用于确定第一目标数据块的数据页为待覆盖写入数据的数据页。
在一种具体可实现方式中，第一索引用于标识第i个压缩数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
在一些可实现方式中，还包括：第一读取单元，用于读取第一数据块的第一索引，得到第一数据块对应的第一压缩数据块的索引地址，其中，第一索引包括第一数据块的属性信息。第二读取单元，用于读取第一数据块对应的第一压缩数据块的索引。解压单元，用于根据第一压缩数据块的索引，解压第一压缩数据块，得到第一压缩数据块对应的多个数据块，该多个数据块中包括第一数据块。第三确定单元，用于确定第一数据块在解压后的多个数据块内的偏移。第三获取单元，用于根据第一数据块在解压后的多个数据块内的偏移，可得到第一数据块的数据。
第三方面，本申请实施例提供一种设备，其包括：用于执行第一方面的数据压缩方法。
第四方面，本申请实施例提供一种计算机可读存储介质，所述计算机可读存储介质包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备执行第一方面的数据压缩方法。
第五方面，本申请实施例提供一种计算机程序，当所述程序被处理器调用时，第 一方面的数据压缩方法被执行。
第六方面，本申请实施例提供一种芯片系统，其包括一个或多个处理器，当所述一个或多个处理器执行指令时，所述一个或多个处理器执行第一方面的数据压缩方法。
应当理解的是，本申请中对技术特征、技术方案、有益效果或类似语言的描述并不是暗示在任意的单个实施例中可以实现所有的特点和优点。相反，可以理解的是对于特征或有益效果的描述意味着在至少一个实施例中包括特定的技术特征、技术方案或有益效果。因此，本说明书中对于技术特征、技术方案或有益效果的描述并不一定是指相同的实施例。进而，还可以任何适当的方式组合本实施例中所描述的技术特征、技术方案和有益效果。本领域技术人员将会理解，无需特定实施例的一个或多个特定的技术特征、技术方案或有益效果即可实现实施例。在其他实施例中，还可在没有体现所有实施例的特定实施例中识别出额外的技术特征和有益效果。
附图说明
图1a为本申请实施例提供的一种操作系统的组成框图；
图1b为本申请实施例提供的一种存储系统的结构示意图；
图2为图1b中存储系统的固态硬盘的结构示意图；
图3为图2中固态硬盘的闪存芯片的结构示意图；
图4为图3中闪存芯片对应的闪存翻译层的示意图；
图5为固定输入压缩模式示意图；
图6为本申请实施例提供的固定输出压缩模式示意图；
图7为本申请实施例提供的数据块索引示意图；
图8为现有可扩展只读文件系统中数据块索引示意图；
图9为本申请实施例提供的一种数据压缩方法的流程示意图；
图10为本申请实施例提供的一种数据块索引更新的流程示意图；
图11为本申请实施例提供的一种数据压缩时数据块索引关系示意图；
图12为本申请实施例提供的又一种数据压缩方法的流程示意图；
图13为本申请实施例提供的一种覆盖写入或读取流程时数据块索引关系示意图；
图14为本申请实施例提供的一种数据读取的流程示意图；
图15为本申请实施例提供的一种数据压缩装置的结构示意图。
具体实施方式
本申请的描述中所提到的术语“包括”和“具有”以及它们的任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元，而是可选地还包括其他没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。
需要说明的是，本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
在本申请的描述中，除非另有说明，“多个”的含义是指两个或两个以上。本文中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如， A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。
为便于理解，下面先对本申请实施例可能涉及到的相关术语和概念进行介绍。
如图1a所示的一种操作系统的组成框图。
操作系统(operation system，简称OS)是管理计算机硬件与软件资源的计算机程序。例如，unix操作系统、windows操作系统、linux操作系统等。操作系统需要处理如管理与配置内存、决定系统资源供需的优先次序、控制输入设备与输出设备、操作网络与管理文件系统等基本事务。操作系统也提供一个让用户与系统交互的操作界面。
操作系统内核是指大多数操作系统的核心部分。它由操作系统中用于管理存储器、文件、外设和系统资源的那些部分组成，负责管理系统的进程、内存、设备驱动程序、文件和网络系统，决定着操作系统的性能和稳定性。操作系统内核是一个提供硬件抽象层、磁盘以及文件系统控制、多任务等功能的系统软件。它为众多应用程序提供对计算机硬件的安全访问，它可以决定一个应用程序在什么时候对计算机的某部分硬件操作多长时间。由于直接对计算机硬件操作是非常复杂的，因此操作系统内核可以提供一套硬件抽象的方法来完成这些操作。
文件系统是操作系统内核的核心模块，也即主要组成部分，文件系统是在存储设备上组织文件的方法，负责管理和存储文件信息，主要是为用户建立文件，存入、读出、修改、转储文件，控制文件的存取，当用户不再使用时撤销文件等。
文件系统提供文件在内核的抽象表示，完成文件到实体存储设备(例如：磁盘、硬盘等)的映射，将文件在存储设备上的物理地址映射成用户可见的路径名、文件名，方便文件数据的快速读取、修改和持久化等。
文件系统包括读写文件系统以及只读文件系统。读写文件系统是既可以写文件到存储设备，也可以从存储设备读取文件的文件系统，例如：档案配置表(file allocation table，FAT)、高效能文件系统(high performance file system，HPFS)、新技术文件系统(new technology file system，NTFS)、第四代扩展文件系统(fourth extended file system，EXT4)、闪存友好文件系统(flash friendly file system，F2FS)等。只读文件系统是只能从存储设备读取文件，但不能向存储设备写文件的文件系统，例如：可扩展只读文件系统(extendable read-only file system，EROFS)。
为了使得本申请更加的清楚，首先对本申请的一种应用场景进行说明。
图1b所示的一种存储系统的结构示意图。
在图1b所示的应用场景中，用户通过应用程序来存取数据。运行这些应用程序的计算机被称为“应用服务器”。应用服务器100可以是物理机，也可以是虚拟机。物理应用服务器包括但不限于桌面电脑、服务器、笔记本电脑以及移动设备。应用服务器通过光纤交换机110访问存储系统以存取数据。然而，交换机110只是一个可选设备，应用服务器100也可以直接通过网络与存储系统120通信。或者，光纤交换机110也可以替换成以太网交换机、InfiniBand交换机、RoCE(RDMA over Converged Ethernet)交换机等。
图1b所示的存储系统120是一个集中式存储系统。所谓集中式存储系统就是指由一台或多台主设备组成中心节点，数据集中存储于这个中心节点中，并且整个系统的所有数据处理业务都集中部署在这个中心节点上。换言之，集中式存储系统中，终端 或客户端仅负责数据的录入和输出，而数据的存储与控制处理完全交由中心节点来完成。集中式存储系统的特点是有一个统一的入口，所有从外部设备来的数据都要经过这个入口，这个入口就是集中式存储系统的引擎121。引擎121是集中式存储系统中最为核心的部件，许多存储系统的高级功能都在其中实现。
如图1b所示，引擎121中有一个或多个控制器，图1b以引擎包含两个控制器为例予以说明。控制器0与控制器1之间具有镜像通道，那么当控制器0将一份数据写入其内存124后，可以通过所述镜像通道将所述数据的副本发送给控制器1，控制器1将所述副本存储在自己本地的内存124中。由此，控制器0和控制器1互为备份，当控制器0发生故障时，控制器1可以接管控制器0的业务，当控制器1发生故障时，控制器0可以接管控制器1的业务，从而避免硬件故障导致整个存储系统120的不可用。当引擎121中部署有4个控制器时，任意两个控制器之间都具有镜像通道，因此任意两个控制器互为备份。
引擎121还包含前端接口125和后端接口126，其中前端接口125用于与应用服务器100通信，从而为应用服务器100提供存储服务。而后端接口126用于与硬盘134通信，以扩充存储系统的容量。通过后端接口126，引擎121可以连接更多的硬盘134，从而形成一个非常大的存储资源池。
按照引擎121与硬盘框130之间通信协议的类型，硬盘框130可能是SAS硬盘框，也可能是NVMe硬盘框，IP硬盘框以及其他类型的硬盘框。SAS硬盘框，采用SAS3.0协议，每个框支持25块SAS硬盘。引擎121通过板载SAS接口或者SAS接口模块与硬盘框130连接。NVMe硬盘框，更像一个完整的计算机系统，NVMe硬盘插在NVMe硬盘框内。NVMe硬盘框再通过RDMA端口与引擎121连接。
在硬件上，如图1b所示，控制器0至少包括处理器123、内存124。处理器123是一个中央处理器(central processing unit，CPU)，用于处理来自存储系统外部(服务器或者其他存储系统)的数据访问请求，也用于处理存储系统内部生成的请求。示例性的，处理器123通过前端端口125接收应用服务器100发送的写数据请求时，会将这些写数据请求中的数据暂时保存在内存124中。当内存124中的数据总量达到一定阈值时，处理器123通过后端端口将内存124中存储的数据发送给硬盘134进行持久化存储。
内存124是指与处理器直接交换数据的内部存储器，它可以随时读写数据，而且速度很快，作为操作系统或其他正在运行中的程序的临时数据存储器。内存包括至少两种存储器，例如内存既可以是随机存取存储器。举例来说，随机存取存储器是动态随机存取存储器(Dynamic Random Access Memory,DRAM)，或者存储级存储器(Storage Class Memory,SCM)。DRAM是一种半导体存储器，与大部分随机存取存储器(Random Access Memory，RAM)一样，属于一种易失性存储器(volatile memory)设备。SCM是一种同时结合传统储存装置与存储器特性的复合型储存技术，存储级存储器能够提供比硬盘更快速的读写速度，但存取速度上比DRAM慢，在成本上也比DRAM更为便宜。然而，DRAM和SCM在本实施例中只是示例性的说明，内存还可以包括其他随机存取存储器，例如静态随机存取存储器(Static Random Access Memory，SRAM)等。另外，内存124还可以是双列直插式存储器模块或双线存储器模块 (Dual In-line Memory Module，简称DIMM)，即由动态随机存取存储器(DRAM)组成的模块，还可以是固态硬盘(Solid State Disk,SSD)。实际应用中，控制器0中可配置多个内存124，以及不同类型的内存124。本实施例不对内存113的数量和类型进行限定。此外，可对内存124进行配置使其具有保电功能。保电功能是指系统发生掉电又重新上电时，内存124中存储的数据也不会丢失。具有保电功能的内存被称为非易失性存储器。
示例性的，内存124和硬盘134均可以是固态硬盘(英语：Solid-state drive或Solid-state disk，简称SSD)是一种主要以闪存(NAND Flash)作为永久性存储器的存储设备。如图2所示，SSD 200包括NAND闪存和主控制器(简称：主控)201。NAND闪存包括多个闪存芯片205，用于存储数据。主控201是SSD的大脑中枢，负责一些复杂的任务，如管理数据存储、维护SSD性能和使用寿命等。主控201是一种嵌入式微芯片，它包括处理器202，其功能就像命令中心，发出SSD的所有操作请求。例如，处理器202可通过缓冲区中的固件来执行读取/写入数据，垃圾回收以及磨损均衡等功能。
SSD主控201还包括主机接口204和若干个通道控制器。其中，主机接口204用于与主机通信。这里的主机可以指服务器、个人电脑或者阵列控制器等任何设备。通过若干个通道控制器，主控201可以并行操作多块闪存芯片205，从而提高底层的带宽。举个例子，假设主控201与FLASH颗粒之间有8个通道，那么主控201通过这8通道并行地向8个闪存芯片205读写数据。
如图3所示，die是一个或多个闪存芯片的封装。一个die可包含多个区域(panel)，多Plane NAND是一种能够有效提升性能的设计。如图3所示，一个die内部分成了2个Plane，而且2个Plane内的Block编号是单双交叉的，因此在操作时，可以进行一单一双交叉操作来提升性能。一个panel包含多个块(block)。而一个block由若干个页(page)组成。以一个16GB容量的闪存芯片为例，每4314*8＝34512个cell逻辑上形成一个page，每个page中可以存放4KB的内容和218B的ECC校验数据，page也是IO操作的最小单位。每128个page组成一个block，每个2048个block组成一个panel，一整片闪存芯片由两个panel组成，一个panel存储奇数序号的block，另一个则存储偶数序号的block，两个plane可以并行操作。这只是一个示例，page的尺寸，block的容量，闪存芯片的容量都可以有不同的规格，本实施例不予限定。
主机往block中写入数据，当一个Block写满的时候，SSD主控201会挑选下一个Block继续写入。页是数据写入的最小单位。换言之，主控201是以页为粒度往block里面写入数据的。Block是数据擦除的最小单位。主控在擦除数据时，每次只能擦除整个block。
主机通过逻辑块地址(Logical Block Address，LBA)访问SSD，每个LBA代表着一个扇区(以512B为例)，而在SSD内部，主控以页(以4KB为例)为单位访问SSD。因此，应用服务器每次写入一个数据时,SSD主控会找一个Page将该数据写入，page的地址被称为物理块地址(Physical Block Address，PBA)。SSD内部记录了一条LBA到PBA的映射。有了这样一个映射后，下次主机需要读某个LBA的数据时，SSD就知道从闪存芯片的哪个位置把数据读取上来。图4是闪存翻译层(Flash  Translation Layer,FTL)的示意图，FTL位于处理器202的固件中。如图4所示，主机每写入一个新数据，就会产生一个新的映射关系，这个映射关系会加入(第一次写)或者更改(覆盖写)FTL。当读取某个数据时，SSD首先查找FTL中该数据的LBA所对应的PBA，然后再根据PBA读取相应的数据。
闪存芯片不能支持覆盖写，这意味着当主机对某个LBA上的数据进行修改时，不能直接在这个LBA所对应的PBA上更改，必须写到一个新的PBA，并在FTL中增加一条映射。如，FTL中原来有LBA D与PBA D之间的映射关系，当主机发送一个IO请求，要求修改LBA D的数据时，SSD寻找一个新的位置(PBA E)写入该数据，并在FTL中增加LBA D与PBA E之间的映射关系。就导致PBA D上的数据变成无效数据。无效数据(也称垃圾数据)，是指没有任何映射关系指向的数据。对于这部分数据，用户不会访问这些FLASH空间，因为它们被新的映射关系所取代。随着应用服务器的持续写入，FLASH存储空间慢慢变小，直到耗尽。如果不及时清除这些垃圾数据，主机就无法写入。SSD内部都有垃圾回收机制，它的基本原理是把几个Block中的有效数据集中搬到一个新的Block上面去，然后再把这几个Block擦除掉，这样就产生新的可用Block了。
另外，内存124中还存储有软件程序，处理器123运行内存124中的软件程序可实现对硬盘的管理。例如将硬盘抽象化为存储资源池，然后划分为LUN提供给服务器使用等。这里的LUN其实就是在服务器上看到的硬盘。当然，一些集中式存储系统本身也是文件服务器，可以为服务器提供共享文件服务。
内存124中存储的数据可以通过文件系统呈现。该文件系统是一个结构化的数据文件存储和组织形式。我们知道，计算机中所有的数据都是0和1，存储在硬件介质上的一连串的01组合对我们来说完全无法去分辨以及管理。因此我们用“文件”这个概念对这些数据进行组织，用于同一用途的数据，按照不同应用程序要求的结构方式组成不同类型的文件。通常用不同的后缀来指代不同的类型，然后我们给每个文件起一个方便理解记忆的名字。而当文件很多的时候，我们按照某种划分方式给这些文件分组，每一组文件放在同一个目录(或者叫文件夹)里面。而且目录下面除了文件还可以有下一级目录(称之为子目录或者子文件夹)，所有的文件、目录形成一个树状结构。这个树状结构有一个专用的名字：文件系统(File System)。文件系统有很多类型，常见的有Windows的FAT/FAT32/NTFS，Linux的EXT2/EXT3/EXT4/XFS/BtrFS等。为了方便查找，从根节点开始逐级目录往下，一直到文件本身，把这些目录、子目录、文件的名字用特殊的字符(例如Windows/DOS用“\”，类Unix系统用“/”)拼接起来，这样的一串字符称之为文件路径，例如Linux中的“/etc/systemd/system.conf”或者Windows中的“C:\Windows\System32\taskmgr.exe”。路径是访问某个具体的文件的唯一标识。例如，Windows下的D:\data\file.exe就是一个文件的路径，它表示D分区下的data目录下的file.exe文件。
文件系统是建立在块设备上面的，文件系统不但记录文件路径，还记录哪些块组成一个文件，哪些块记录的是目录/子目录信息。不同的文件系统有不同的组织结构。为了方便管理，硬盘这样的块设备通常可以划分为多个逻辑块设备，也就是硬盘分区(Partition)。反过来，单个介质的容量、性能有限，可以通过某些技术手段把多个物 理块设备组合成一个逻辑块设备，例如各种级别的RAID，JBOD等。文件系统也可以建立在这些逻辑块设备之上。无论如何，应用服务器应用并不需要关心所要访问的文件位于底层的块设备的具体位置，只需要该文件的文件名/ID发送给文件系统，由文件系统根据所述文件名/ID查询出文件路径即可。
比较常见的文件访问协议是NFS、CIFS或者SMB等，本实施例不对此进行任何限定。
本申请中的文件系统为读写文件系统。读写文件系统是既可以写文件到存储设备，也可以从存储设备读取文件的文件系统，例如：FAT、HPFS、NTFS、EXT4、F2FS等等。
文件系统中一般包括元数据区和数据区，元数据区包括超级块和索引节点(inode)区。元数据区的超级块内可以包括文件系统的控制信息、数据结构等内容，元数据区的inode区可以包括文件的描述信息，例如文件长度、文件类型等，文件类型例如为普通文件(regular inode)、目录文件(directory inode)、软链接(symbol link inode)、特殊文件(special inode)等。数据区中存储的数据可以是基于无损压缩技术进行文件级别的压缩处理后得到的数据。数据区中的数据在存储介质(例如为磁盘、闪存等)的物理存储空间上按照磁盘块的集合来进行存储。其中，同一文件的数据可以在连续的磁盘块中存储，或者，也可以在不连续的磁盘块中交叉存储。
应理解，本申请中引入磁盘块的概念并不意味着存储介质仅限定为磁盘，磁盘块可以用来表示存储介质的物理存储空间经划分后得到的小的物理存储空间。
当然，本申请中的存储系统还可以包括分布式存储系统。所谓分布式存储系统是指将数据分散存储在多台独立的存储节点上的系统。传统的网络存储系统采用集中式的存储阵列存放所有数据，存储阵列的性能既是系统性能的瓶颈，也是可靠性和安全性的焦点，不能满足大规模存储应用的需要。
以上，为本申请的一种应用场景的简单介绍。
在上述存储系统中，按数据读写能力对器件的速率进行排序，由强到弱的顺序如下：中央处理器(central processing unit，CPU)>>双倍数据速率同步动态随机存储器(double data rate synchronous dynamic random access memory，DDR SDRAM)>闪存芯片falsh。可见，存储系统内数据访问瓶颈是数据在内存与flash之间的IO(输入输出)时间开销。
为了提升存储系统的整体的IO读写性能，需要对内存中的文件进行压缩。由于文件系统中元数据区占整个文件系统比例较小，数据区往往占据较高的设备存储容量。所以，当向falsh写入数据时，压缩数据，并将压缩后的数据写入falsh中，可以减少falsh的存储容量占用，延长falsh的使用寿命。
目前，Linux的读写文件系统，例如：F2FS、第二代闪存文件系统(journalling Flash file system version2，JFFS2)、B树文件系统(B-tree file system，BTRFS)等，Windows的读写文件系统，例如：NTFS等，均可以采用如下数据压缩方式：
将需要压缩的原始文件数据(或称源数据)按照固定大小的最小可压缩单元
(cluster)进行压缩，压缩后的文件数据(或称压缩数据)包括头部数据和压缩数据。其中，头部数据用于表征文件数据的属性信息；压缩数据用于表征文件数据的内容。 再将压缩后的文件数据保存到falsh上，并以4kb的大小对齐。
示例性的，如图5所示的固定输入压缩模式示意图，将4个地址连续的数据块(block)作为一个cluster0进行压缩，得到由头部数据(header)+压缩数据(compressed data)组成的压缩文件数据。若压缩文件数据小于4kb，则将压缩文件数据以4kb大小存储在falsh上。
假设，图5所示的原始文件数据(或称源数据)大小为4个block，每个block大小为4kb，一个block为一个逻辑页，对原始文件数据的逻辑页进行编号：0、1、2、3。将原始文件数据按照75％的压缩率压缩成压缩文件数据，该压缩文件数据大小为12kb。故压缩文件数据大小为3个block，所以，压缩文件数据的实际页为3页，即读取单个逻辑页需要读取的falsh实际页的大小如表1所示。
表1
Figure PCTCN2022085621-appb-000001
在将压缩后的文件数据保存到falsh上之后。若需要在falsh上读取原始文件数据的目标逻辑页，则需要读取压缩文件数据的3页数据，并将压缩文件数据解压后才能读取到目标逻辑页。例如，在随机读取的场景下，若需要在falsh上读取0号逻辑页的原始文件数据，则需要将压缩文件数据的3页数据都读取出来，并解压压缩文件数据之后，才可以成功读取到0号逻辑页的原始文件数据。因此，数据读取效率为：
Figure PCTCN2022085621-appb-000002
可见，经过图5所示的数据压缩方式得到的压缩文件数据，在随机读取的场景下，读取效率较低。
为了解决上述技术问题，本申请实施例提供了一种数据压缩方法，该方法通过获取可读写文件系统中数据区的m个数据块。采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块，其中，每个压缩数据块的第一容量相同，第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数。其中，m，n均为大于等于1的正整数。
其中，预设压缩算法可以为采用固定输出压缩模式对应的压缩算法，如，(lempel-ziv 4，LZ4)压缩算法。当然，预设压缩算法还可以为其他压缩算法，本申请实施例不做具体限定。
示例性的，在一种应用场景中，如图6所示，假设源数据大小为16kb，以4kb的数据是一个数据块，亦是一个逻辑页为例，对源数据的逻辑页进行编号：0、1、2、3，如表2的第一行所示。
假设将逻辑页连续的16kb的源数据划分为3份，分别为：6kb、7kb、5kb。对这3份数据采用预设压缩算法(如，LZ4)对其进行压缩，直到压缩数据块中压缩后的每份数据大小为4kb为止。
压缩数据块的数据页为3个页，并分别编号为：如图6所示的压缩页4、压缩页5和压缩页6。
表2
Figure PCTCN2022085621-appb-000003
可见，逻辑页0的源数据都压缩在压缩页4中，故压缩成1页。逻辑页1的一部分源数据压缩在压缩页4中，逻辑页1的另一部分源数据压缩在压缩页5中，故压缩成2页。逻辑页2的一部分源数据压缩在压缩页5中，逻辑页2的另一部分源数据压缩在压缩页6中，故压缩成2页。逻辑页3的源数据都压缩在压缩页6中，故压缩成1页。
因此，在随机读取场景下，可能读取上述任意一个或者多个逻辑页。示例性的，当读取逻辑页0时，只需读取一个压缩页，如表2第二行第二列所示。解压之后可以获得逻辑页0的全部数据。
此时，读取效率可按照下列式2计算得出：
Figure PCTCN2022085621-appb-000004
读取逻辑页3的读取效率同读取逻辑页0的读取效率。
当读取逻辑页1时，需要读取2个压缩页，如表2的第二行第三列所示。将压缩页4和压缩页5的数据解压之后才可以获得逻辑页1的全部数据。
此时，读取效率可按照下列式3计算得出：
Figure PCTCN2022085621-appb-000005
读取逻辑页2的读取效率同读取逻辑页1的读取效率。
另外，平均4个逻辑页的读取效率可按照下列式4计算得出：
Figure PCTCN2022085621-appb-000006
从式2、式3、式4计算得出的读取效率可以看出，在随机读取场景下，图6所示的数据压缩方法的读取效率要远远大于图5所示的数据压缩方法的读取效率的。
可见，本申请实施例通过将可读写文件系统中数据区的m个数据块采用固定输出压缩模式对应的压缩算法进行压缩，依次得到n个相同字节数的数据的压缩数据块，使得输出的每个压缩数据块均是固定大小。当读取数据块时，可以有效提高读取效率，可以保证随机读场景以很小的读放大系数完成数据的读取。
另外，图8所示的现有可扩展只读文件系统(extendable read-only file system，erofs)中数据块索引方式。在数据块地址数组data_addr中，存取的都是block地址，其指向实际数据块的地址。当erofs在制作镜像时，采用图5所示的方法压缩数据时，由于存储设备(例如：磁盘)的结构和文件内容已经固定，无法支持对文件的修改。 但是在用户实际操作的场景下，对于存储设备上的许多压缩文件可能需要经常修改，erofs无法支持这种诉求。
上述数据块索引可以查找到对应的数据块，可以理解为，该数据块索引也即inode，也即元数据。其中，inode就是用来存储元数据的区域，也即用来存储文件相关属性信息的区域，例如：文件的创建者、创建日期、大小、数据块的位置等等。其中，每个inode都有一个号码，操作系统用不同的inode号码来识别不同的文件。示例性的，表面上，用户通过文件名，打开文件。实际上，系统先根据文件名找到对应的inode号码，然后通过inode号码获取到inode信息，再根据inode信息查找到数据块的地址，读取该数据。
也就是说，inode记录了文件的属性以及该文件实际存储位置，即数据块号(block number)，每一个block(常见大小4KB)，通过inode可以实现文件的查找定位。inode是Linux中的，Unix中是vnode。基本上，inode包含的信息至少有如下这些：(1)文件的类型(2)文件访问权限；(3)文件的所有者与组；(4)文件的大小；(5)链接数，即指向该inode的文件名总数；(6)文件的状态改变时间(ctime)、最近访问时间(atime)和最近修改时间(mtime)；(7)文件特殊属性，SUID、SGID和SBIT；(8)文件内容的真正指向(pointer)。
如图8所示，为现有的数据块索引格式，该数据块索引格式是不支持可扩展的，例如：不支持追加写、块预留、删节(truncate)等等。其中，追加写指的是在原有文件的基础上，添加新内容，不删除原有文件中的内容。块预留指的是文件系统提前考虑如果文件增长，可以从哪块空间分配磁盘块，并将这些磁盘块预留下来。truncate指的是修改文件，例如：删除、新增等等。
示例性的，如图8所示，将数据块索引用blk entry代指，为便于描述，简写为blk。其中blk 1，为压缩数据块1的索引，在blk1中存放有压缩数据块1在存储设备上的地址。blk2，为压缩数据块2的索引，在blk2中存放有压缩数据块2在存储设备上的地址。blk3，为压缩数据块3的索引，在blk3中存放有压缩数据块3在存储设备上的地址。blk4，为压缩数据块4的索引，在blk4中存放有压缩数据块4在存储设备上的地址。因此，根据blk中存放的地址可以确定压缩数据块在存储设备上的位置。
为了使得读写文件系统支持写入、覆盖写、预分配、truncate等，本申请实施例提供的数据压缩方法还包括：建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录第一索引与j个数据块的映射关系。其中，i为大于等于1，且小于等于n的正整数。j为大于等于1，且小于等于m的正整数。其中，第一索引用于标识j个数据块包含中每个数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。当然，第一索引还用于标识第i个压缩数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
其中，属性信息以下至少一项：
第一属性，用于表征数据块被压缩后所在的压缩数据块的存储位置是否是预先分配的。
第二属性，用于表征数据块的数据页是否有效；也即，是正常数据页还是空洞数据页，其中空洞数据页，可以理解为空白数据页。
第三属性，用于表征数据块的数据页是否是数据块的压缩数据块的第一个压缩页。
第四属性，用于表征数据块的数据页是否包含在两个压缩块的压缩数据页中。
第五属性，用于表征数据块的数据页是否是数据块被压缩后的压缩数据块的压缩页。
第六属性，用于表征数据块的数据页所在的压缩数据块的索引地址。
第七属性，用于表征当数据块的数据页属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块在压缩数据块对应的集合内的偏移；当数据块的数据页不属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
示例性的，如图7所示，数据块的第一索引包括：blk entry，其对应存放数据块或者压缩数据块的地址；extent entry，其对应存放扩展的属性信息。其中，每个extent entry与blk entry一一对应，每个数据页都有对应的一个extent entry和blk entry。
其中，Extent entry成员的集合表示方式如下：
示例性的，如集合A所示，数据块索引中包括的成员可以如集合A所示，需要说明的是，每个数据页都有对应的一个集合A。
Figure PCTCN2022085621-appb-000007
集合成员含义解释如下：
is_reserved为上述第一属性。
is_valid为上述第二属性。
first_page为上述第三属性。
cross_block为上述第四属性。
is_compress为上述第五属性。
blkidx为上述第六属性。
ofs为上述第七属性。
可见，在读写文件系统中应用图6所示的方法压缩数据时，可以对图7所示的数据块的索引中包含的属性进行修改，从而可以对存储设备上的压缩文件进行修改。
下面结合具体的示例，对本申请实施例提供的数据压缩方法进行说明。
图9为本申请实施例提供的一种数据压缩方法的流程示意图。如图9所示，该方法包括：
S901、获取可读写文件系统中数据区的m个数据块，m为大于等于1的正整数。
该m个数据块可以理解为需要回写的数据块。其中，回写可以是指在写操作时，先将数据先写入内存中进行缓存，但不会立即写入存储设备(例如：磁盘)中。当在一些特定条件或操作下(例如：回刷机制、同步(sync)操作等)，才会将内存中缓存的数据写入存储设备。
S902、采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块，其中，每个压缩数据块的第一容量相同，第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数，n为大于等于1的正整数。
其中，预设压缩算法可以为LZ4压缩算法，当然，也可以其他固定输出的压缩算法，本申请实施例不做具体限定。
其中，m可以为任意的正整数。如，m为4，m为10，或m为20。
S902具体可实现为：
S9021、将m个数据块中的各数据块以预设顺序依次分配至第一集合。
该预设顺序可以连续的存储地址的顺序。也就是说，m个数据块的连续顺序。
该第一集合可以称为最小可压缩单元(cluster)。换句话说，第一集合为最小可压缩的数据块集合。如，图6中所示的6kb的数据块集合、7kb的数据块集合和5kb的数据块集合。
示例性的，m个数据块映射在存储介质中的一段连续的地址。假设以一个数据块为起点，按照数据块映射在存储介质中的地址的顺序，依次划分固定大小的数据集合，如图6所示，将数据块0及数据块1的1/2数据组成6kb的数据集合；将数据块1的1/2数据，数据块2的3/4数据，及空白数据页组成7kb的数据集合；将数据块2的1/4数据，及数据块3组成5kb的数据集合。
S9022、确定第一集合中的j个数据块的数据容量是否等于第一集合的额定容量，j为大于等于1，且小于等于m的正整数。若j个数据块的数据容量不等于第一集合的额定容量，则执行S9021；若j个数据块的数据容量等于第一集合的额定容量，则执行S9023。
S9023、按照设定压缩的阈值，对第一集合中的j个数据块执行固定压缩操作，并得到第i个压缩数据块。
其中，设定压缩的阈值，用于表征压缩率。示例性的，设定压缩的阈值的表达公式可以为：
设定压缩的阈值＝数据总长度-数据总长度*压缩率
S9024、确定j个数据块的数据总长度是否大于第i个压缩数据块的头部数据和压缩数据的数据总长度与设定压缩的阈值之和。若是，则执行S903。否则，将源数据页提交至闪存。
S903、建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录所述第一索引与j个数据块的映射关系；其中，i为大于等于1，且小于等于n的正整数；j为大于等于1，且小于等于m的正整数。
应理解为，当压缩完一个压缩数据块时，建立该压缩数据块对应的各数据块中每个数据块的索引。
其中，第一索引用于标识j个数据块包含中每个数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
示例性的，以linux的f2fs读写文件系统为例，数据块在F2fs上的第一索引格式可以为：
第一索引包含属性信息的数据结构可以为：
如，Entry数据结构：
Figure PCTCN2022085621-appb-000008
其中，属性信息可以包括以下至少一项：
第一属性(is_reserved)，用于表征数据块被压缩后所在的压缩数据块的存储位置是否是预先分配的；
第二属性(is_valid)，用于表征数据块的数据页是否有效；
第三属性(first_page)，用于表征数据块的数据页是否是数据块的压缩数据块的第一个压缩页；
第四属性(cross_block)，用于表征数据块的数据页是否包含在两个压缩块的压缩数据页中；
第五属性(is_compress)，用于表征数据块的数据页是否是数据块被压缩后的压缩数据块的压缩页；
第六属性(blkidx)，用于表征数据块的数据页所在的压缩数据块的索引地址。
第七属性(ofs)，用于表征当数据块的数据页属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块在压缩数据块对应的集合内的偏移；当数据块的数据页不属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
在一种具体可实现方式中，如图10所示，为本申请实施例提供的一种数据块索引更新的流程示意图。如图10所示，属性信息可以包括第三属性(first_page)和第七属性(ofs)，S903具体可实现为：
S1031、确定j个数据块中每个数据块的数据页是否是第i个压缩数据块的第一个压缩页。若是，则第三属性的属性值赋值为1；若否，则第三属性的属性值赋值为0。
S1032、当第三属性的属性值为1时，更新第七属性的属性值为在数据块内对应的第一集合内的偏移。
S1033、当第三属性的属性值为0时，更新第七属性的属性值为数据块的数据页距离第i个压缩数据块的第一个压缩页的距离。
当然，属性信息还可以包括第四属性(cross_block)，S103具体可实现为：
S1034、确定j个数据块中的每个数据块的数据页是否包含在两个压缩块的压缩数据页中。若是，则第四属性的属性值赋值为1；若否，则第四属性的属性值赋值为0。
当然，属性信息还可以包括第二属性(is_valid)，S103具体可实现为：
S1035、确定j个数据块中的每个数据块的数据页是否有效。若是，则第二属性的属性值赋值为1；若否，则第二属性的属性值赋值为0。
当然，属性信息还可以包括第六属性(blkidx)，S103具体可实现为：
S1036、确定m个数据块中的每个数据块的数据页所在的压缩数据块的索引地址。
其中，按照m个数据块在内存中的存储位置的顺序，以最小固定压缩单元(如第一集合)的大小进行压缩。当完成第一次压缩(即压缩出第1个压缩数据块)时，第1个压缩数据块对应的完整数据块的数据页均在第1压缩块的索引位置上。如，第1个压缩数据块对应的数据块有：数据块0、数据块1、数据块2、数据块3的部分数据。第1个压缩数据块对应的完整数据块为数据块0、数据块1和数据块2。故，数据块0、数据块1和数据块2的数据页在第1个压缩数据块的索引位置上。
这里需要说明的是，除了上述第七属性需要依附于第三属性之外，其他的属性对应的数据块索引更新流程是相互独立的。本申请实施例中对各个第一属性、第二属性、第四属性、第五属性和第六属性对应的数据块索引更新流程的顺序不做具体限定。
示例性的，假设，如图11所示，m个数据块包括数据块0(即block0)、数据块1(即block1)、数据块2(即block2)、数据块3(即block3)。其中，block0、block1、block2、block3在内存中映射一段连续的地址。当按照block0、block1、block2、block3在内存中映射一段连续的地址的顺序(如图11中由左向右的压缩方向)，以最小固定压缩单元(如第一集合)的大小进行压缩。在压缩过程中:
当取block0，及block1的一部分达到最小固定压缩单元(如4kb)时，执行第一 次压缩，得到第1个压缩数据块(compress blk0)。此时，建立block0的数据块索引，如表3所示：
表3
Figure PCTCN2022085621-appb-000009
结合图10可得，block0的数据页落在了第一压缩数据块的第一压缩页上，故first_page赋值为1。block0的数据页只落在了第一压缩数据块的第一压缩页上，故cross_block的赋值为0。block0的数据页在第一压缩数据块的索引地址为第一压缩数据块(compress blk0)的序号，故blkidx的赋值为0。block0的数据页落在了第一压缩数据块的第一压缩页上，block0在其对应的第一集合内的偏移为0，故ofs赋值为0。block0的数据页是有效数据页，故is_valid的赋值为1。
当取block1的剩余部分，block2，及block3的一部分达到最小固定压缩单元(如4kb)时，执行第二次压缩，得到第2个压缩数据块(compress blk1)。此时，建立block1和block2的数据块索引，如表4所示：
表4
Figure PCTCN2022085621-appb-000010
结合图10可得，block1的数据页落在了第二压缩数据块的第一压缩页上，故first_page赋值为1。Block1的数据页落在了第一压缩数据块和第二压缩数据块的压缩页上，故cross_block的赋值为1。Block1的数据页在第二压缩数据块的索引地址为第二压缩数据块(compress blk1)的序号，故blkidx的赋值为1。Block1的数据页落在了第二压缩数据块的第一压缩页上，block1在数据块的集合内的偏移为Ofs1，故ofs赋值为Ofs1。Block1的数据页是有效数据页，故is_valid的赋值为1。
同理，block2的数据页未落在了第二压缩数据块的第一压缩页上，故first_page赋值为0。Block2的数据页只落在了第二压缩数据块的压缩页上，故cross_block的赋值为0。Block2的数据页在第二压缩数据块的索引地址为第二压缩数据块(compress blk1)的序号，故blkidx的赋值为1。Block2的数据页未落在了第二压缩数据块的第一压缩页上，block2的数据页距离第一压缩数据块的第一压缩页的距离为1，故ofs赋值为1。Block2的数据页是有效数据页，故is_valid的赋值为1。
当取block3的剩余部分达到最小固定压缩单元(如4kb)时，执行第三次压缩，得到第3个压缩数据块(compress blk2)。此时，建立block3的数据块索引，如表5所示：
表5
Figure PCTCN2022085621-appb-000011
结合图10可得，block3的数据页落在了第三压缩数据块的第一压缩页上，故first_page赋值为1。Block1的数据页落在了第二压缩数据块和第三压缩数据块的压缩页上，故cross_block的赋值为1。Block1的数据页在第三压缩数据块的索引地址为第二压缩数据块(compress blk2)的序号，故blkidx的赋值为2。Block3的数据页落在了第三压缩数据块的第一压缩页上，block3在数据块的集合内的偏移为Ofs2，故ofs赋值为Ofs2。Block3的数据页是有效数据页，故is_valid的赋值为1。
S904、确定m个数据是否压缩完毕。若压缩完毕，则将压缩数据块的压缩页提交至设备。若否，则执行S902。
在一些实施例中，如图12所示，为本申请实施例提供的一种数据压缩方法的流程示意图。如图12所示，在执行S902之前，本申请实施例提供的数据压缩方法还包括：
S905、获取待覆盖写入数据的第二集合。
S906、确定待覆盖写入数据的第二集合中是否包含压缩数据块。若是，则执行S907；若否，则执行现有的数据覆盖写入流程。
其中，第二集合可以包括p个压缩数据块，p为大于等于1的正整数。
S907、获取第二集合中的p个压缩数据块中第一目标压缩数据的压缩页，及第一目标压缩数据块的压缩页对应的q个数据块，q为大于等于1的正整数。
S908、确定q个数据块中的第一目标数据块在q个数据块中的位置偏移。
具体可以为，读取第二集合中各压缩数据块的索引地址，并解压上述压缩数据块，得到上述压缩数据块对应的各数据块。再确定各个数据块在q个数据块中的位置偏移。
S909、根据第一压缩页和第一数据块在压缩页内
的偏移位置，获取第一数据块的数据页。
S910、确定第一目标数据块的数据页为待覆盖写入数据的数据页。
S911、将第二数据块覆盖写入第一数据块的数据页。
S912、将第二数据块分配给第一集合。
综上，在读写文件系统f2fs中，采用本申请实施提供的固定输出压缩模式压缩指定so文件、vdex文件和odex文件等，可达到如下收益效果：如，在电子设备上安装40个应用的过程中，平均每个应用可获得12％的时间收益。应用安装过程so文件存在追加写，vdex文件和odex文件存在覆盖写流程。40个应用的平均启动收益，相比于固定输入压缩模式压缩数据可获得额外8％的启动收益。
上述为本申请实施例提供的数据压缩方法，应用上述数据压缩方法对数据进行压缩之后，需要读取数据。如图14所示，为本申请实施例提供的一种数据读取的流程示意图。如图14所示，该数据读取流程如下：
S141、读取第一数据块的第一索引，得到第一数据块对应的第一压缩数据块的索引地址，其中，第一索引包括第一数据块的属性信息。
其中，第一数据块的属性信息可以包括上述实施例中的第一属性至第七属性中的至少一个。示例性的，以第一数据块的属性信息包括第三属性(first_page)、第四属性(cross_block)、第六属性(blkidx)和第七属性(ofs)为例：
在覆盖写时和只读场景中，假设第一数据块为图13中所示的数据块2(block2)。读取block2的ofs、cross_block、blkidx等属性信息，得到block2对应的第一压缩数据 块的索引地址。具体为，沿用表4，读取到block2的ofs的赋值为1，可确定Block2的数据页未落在了第二压缩数据块的第一压缩页上，亦可得到block2的数据页距离第一压缩数据块的第一压缩页的距离为1。再读取到block2的cross_block的赋值为0，可确定Block2的数据页只落在了第一压缩数据块的压缩页上，并为落在其他的压缩数据块的压缩页上。再读取到block2的blkidx的赋值为1，可确定Block2的数据页在第一压缩数据块的索引地址可以为第一压缩数据块的序号，即可得到第一压缩数据块的索引地址为1。
S142、读取第一数据块对应的第一压缩数据块的索引。
S143、根据第一压缩数据块的索引，解压第一压缩数据块，得到第一压缩数据块对应的多个数据块，该多个数据块中包括第一数据块。
具体为，根据第一压缩数据块的索引在设备上找到第一压缩数据块。找到第一压缩数据块之后，解析第一压缩数据块，并得到解析后的多个数据块。例如，如图13所示，第一压缩数据块为copress blk1，copress blk1解析后得到：数据块1(block1)的部分数据，数据块2(block2)，及数据块3(block3)的部分数据。
S144、确定第一数据块在解压后的多个数据块内的偏移。
具体为，根据第一数据块的属性信息可得出，第一数据块为图13中所示的数据块2(block2)。如图13所示，block2在copress blk1解析后的多个数据块中的偏移(dstofs)的表达式为：
dstofs＝block_size-ofs1％block_size
其中，dstofs表示block2在copress blk1解析后的多个数据块中的偏移；block_size表示数据块的长度；ofs1表示第七属性的属性值；ofs1％block_size表示取余数。
S145、根据第一数据块在解压后的多个数据块内的偏移，可得到第一数据块的数据。
沿用上述示例，第一数据块即为block2，如图13和表4所示，即可得到block2的数据。
具体的，该可能的设计中所述的通信系统用于执行图9所示数据压缩方法中各个设备的功能，因此可以达到与上述数据压缩方法相同的效果。
图15为本申请实施例提供的一种数据压缩装置，该数据压缩装置1500可以包括：第一获取单元1501，用于获取可读写文件系统中数据区的m个数据块，m为大于等于1的正整数。压缩单元1502，用于采用预设压缩算法压缩m个数据块，依次得到n个压缩数据块，其中，每个压缩数据块的第一容量相同，第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数，n为大于等于1的正整数。更新单元1503，用于建立n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录第一索引与j个数据块的映射关系。其中，i为大于等于1，且小于等于n的正整数；j为大于等于1，且小于等于m的正整数。其中，第一索引用于标识j个数据块包含中每个数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
在一种具体可实现方式中，压缩单元1502用于：将m个数据块中的各数据块以预设顺序依次分配至第一集合。当第一集合中的j个数据块的数据容量等于第一集合 的额定容量时，按照设定压缩的阈值，对j个数据块执行压缩操作，并得到第i个压缩数据块。
在一种具体可实现方式中，更新单元1503用于：当第i个压缩数据块的头部数据和压缩数据的数据总长度与设定压缩的阈值之和小于等于j个数据块的数据总长度时，建立j个数据块中每个数据块的第一索引。
在一种具体可实现方式中，属性信息包括以下至少一项：第一属性，用于表征数据块被压缩后所在的压缩数据块的存储位置是否是预先分配的；第二属性，用于表征数据块的数据页是否有效；第三属性，用于表征数据块的数据页是否是数据块的压缩数据块的第一个压缩页；第四属性，用于表征数据块的数据页是否包含在两个压缩块的压缩数据页中；第五属性，用于表征数据块的数据页是否是数据块被压缩后的压缩数据块的压缩页；第六属性，用于表征数据块的数据页所在的压缩数据块的索引地址；第七属性，用于表征当数据块的数据页属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块在压缩数据块对应的集合内的偏移；当数据块的数据页不属于数据块的压缩数据块的第一个压缩页时，第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
在一种具体可实现方式中，属性信息包括第三属性，更新单元1503还用于：当j个数据块中每个数据块的数据页是第i个压缩数据块的第一个压缩页时，第三属性的属性值赋值为1。当j个数据块中每个数据块的数据页不是第i个压缩数据块的第一个压缩页时，第三属性的属性值赋值为0。
在一种具体可实现方式中，属性信息包括第七属性，更新单元1503还用于：当第三属性的属性值为1时，更新第七属性的属性值为数据块在压缩数据块对应的集合内的偏移。当第三属性的属性值为0时，更新第七属性的属性值为数据块的数据页距离压缩数据块的第一个压缩页的距离。
在一种具体可实现方式中，属性信息包括第四属性，更新单元1503还用于：当j个数据块中的每个数据块的数据页包含在两个压缩块的压缩数据页中时，第四属性的属性值赋值为1。当j个数据块中的每个数据块的数据页不包含在两个压缩块的压缩数据页中时，第四属性的属性值赋值为0。
在一种具体可实现方式中，属性信息包括第二属性，更新单元还用于：当j个数据块中的每个数据块的数据页为有效时，第二属性的属性值赋值为1。当j个数据块中的每个数据块的数据页为无效时，第二属性的属性值赋值为0。
在一些可实现方式中，还包括：第二获取单元1504，用于获取待覆盖写入数据的第二集合，第二集合包括p个压缩数据块，p为大于等于1的正整数。第三获取单元1505，用于获取p个压缩数据块中第一目标压缩数据的压缩页，及第一目标压缩数据块的压缩页对应的q个数据块，q为大于等于1的正整数。第一确定单元1506，用于确定q个数据块中的第一目标数据块在q个数据块中的位置偏移。第二确定单元1507，用于确定第一目标数据块的数据页为待覆盖写入数据的数据页。
在一些可实现方式中，还包括：第一读取单元，用于读取第一数据块的第一索引，得到第一数据块对应的第一压缩数据块的索引地址，其中，第一索引包括第一数据块的属性信息。第二读取单元，用于读取第一数据块对应的第一压缩数据块的索引。解 压单元，用于根据第一压缩数据块的索引，解压第一压缩数据块，得到第一压缩数据块对应的多个数据块，该多个数据块中包括第一数据块。第三确定单元，用于确定第一数据块在解压后的多个数据块内的偏移。第三获取单元，用于根据第一数据块在解压后的多个数据块内的偏移，可得到第一数据块的数据。
在一种具体可实现方式中，第一索引用于标识第i个压缩数据块在存储介质中的存储位置，及j个数据块中每个数据块包含的属性信息。
因此，本申请实施例提供的数据压缩方法，当读取数据块时，可以有效提高读取效率，可以保证随机读场景以很小的读放大系数完成数据的读取。另外，数据块的索引中包含的属性可以进行修改，从而可以对存储设备上的压缩文件进行修改。可见，本申请实施例解决已有读写文件系统压缩方案随机读放大的问题，同时解决已有的固定输出压缩方式的文件系统无法支持数据和元数据更新的问题。
本申请实施例还提供了一种设备，该设备包括：用于执行上述任一项所述的各个步骤的单元，或者，用于执行上述任一项所述的各个步骤的单元。
本申请实施例还提供了一种计算机可读存储介质，包括指令，当其在计算机上运行时，使得计算机执行上述任一方法。
本申请实施例还提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述任一方法。
本申请实施例还提供了一种芯片，该芯片包括处理器和接口电路，该接口电路和该处理器耦合，该处理器用于运行计算机程序或指令，以实现上述方法，该接口电路用于与该芯片之外的其它模块进行通信。
在本申请的描述中，除非另有说明，“/”表示“或”的意思，例如，A/B可以表示A或B。本文中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。此外，“至少一个”是指一个或多个，“多个”是指两个或两个以上。“第一”、“第二”等字样并不对数量和执行次序进行限定，并且“第一”、“第二”等字样也并不限定一定不同。
在本申请的描述中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
尽管结合具体特征及其实施例对本申请进行了描述，显而易见的，在不脱离本申请的精神和范围的情况下，可对其进行各种修改和组合。相应地，本说明书和附图仅仅是所附权利要求所界定的本申请的示例性说明，且视为已覆盖本申请范围内的任意和所有修改、变化、组合或等同物。显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。
需要说明的是：以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (24)

  1. 一种数据压缩方法，其特征在于，所述方法包括：
    获取可读写文件系统中数据区的m个数据块，m为大于等于1的正整数；
    采用预设压缩算法压缩所述m个数据块，依次得到n个压缩数据块，其中，每个压缩数据块的第一容量相同，所述第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数，n为大于等于1的正整数；
    建立所述n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录所述第一索引与j个数据块的映射关系；其中，i为大于等于1，且小于等于n的正整数；j为大于等于1，且小于等于m的正整数；
    其中，所述第一索引用于标识所述j个数据块包含中每个数据块在存储介质中的存储位置，及所述j个数据块中每个数据块包含的属性信息。
  2. 根据权利要求1所述的方法，其特征在于，所述采用预设压缩算法压缩所述m个数据块，依次得到n个压缩数据块，包括：
    将所述m个数据块中的各数据块以预设顺序依次分配至第一集合；
    当所述第一集合中的所述j个数据块的数据容量等于所述第一集合的额定容量时，按照设定压缩的阈值，对所述j个数据块执行压缩操作，并得到所述第i个压缩数据块。
  3. 根据权利要求2所述的方法，其特征在于，所述建立所述n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，包括：
    当所述第i个压缩数据块的头部数据和压缩数据的数据总长度与所述设定压缩的阈值之和小于等于所述j个数据块的数据总长度时，建立所述j个数据块中每个数据块的第一索引。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述属性信息包括以下至少一项：
    第一属性，用于表征数据块被压缩后所在的压缩数据块的存储位置是否是预先分配的；
    第二属性，用于表征数据块的数据页是否有效；
    第三属性，用于表征数据块的数据页是否是所述数据块的压缩数据块的第一个压缩页；
    第四属性，用于表征数据块的数据页是否包含在两个压缩块的压缩数据页中；
    第五属性，用于表征数据块的数据页是否是所述数据块被压缩后的压缩数据块的压缩页；
    第六属性，用于表征数据块的数据页所在的压缩数据块的索引地址；
    第七属性，用于表征当数据块的数据页属于所述数据块的压缩数据块的第一个压缩页时，所述第七属性的属性值为所述数据块在所述压缩数据块对应的集合内的偏移；当所述数据块的数据页不属于所述数据块的压缩数据块的第一个压缩页时，所述第七属性的属性值为所述数据块的数据页距离压缩数据块的第一个压缩页的距离。
  5. 根据权利要求4所述的方法，其特征在于，所述属性信息包括第三属性，所述建立所述n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，包括：
    当所述j个数据块中每个数据块的数据页是所述第i个压缩数据块的第一个压缩页时，所述第三属性的属性值赋值为1；
    当所述j个数据块中每个数据块的数据页不是所述第i个压缩数据块的第一个压缩页时，所述第三属性的属性值赋值为0。
  6. 根据权利要求4或5所述的方法，其特征在于，所述属性信息包括第七属性，还包括：
    当所述第三属性的属性值为1时，更新所述第七属性的属性值为所述数据块在所述压缩数据块对应的集合内的偏移；
    当所述第三属性的属性值为0时，更新所述第七属性的属性值为所述数据块的数据页距离压缩数据块的第一个压缩页的距离。
  7. 根据权利要求4-6中任一项所述的方法，其特征在于，所述属性信息包括第四属性，所述建立所述n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，包括：
    当所述j个数据块中的每个数据块的数据页包含在两个压缩块的压缩数据页中时，所述第四属性的属性值赋值为1；
    当所述j个数据块中的每个数据块的数据页不包含在两个压缩块的压缩数据页中时，所述第四属性的属性值赋值为0。
  8. 根据权利要求4-7中任一项所述的方法，其特征在于，所述属性信息包括第二属性，所述建立所述n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，包括：
    当所述j个数据块中的每个数据块的数据页为有效时，所述第二属性的属性值赋值为1；
    当所述j个数据块中的每个数据块的数据页为无效时，所述第二属性的属性值赋值为0。
  9. 根据权利要求1-8中任一项所述的方法，其特征在于，在所述采用预设压缩算法压缩所述m个数据块，依次得到n个压缩数据块之前，还包括：
    获取待覆盖写入数据的第二集合，所述第二集合包括p个压缩数据块，p为大于等于1的正整数；
    获取所述p个压缩数据块中第一目标压缩数据的压缩页，及所述第一目标压缩数据块的压缩页对应的q个数据块，q为大于等于1的正整数；
    确定所述q个数据块中的第一目标数据块在所述q个数据块中的位置偏移；
    确定所述第一目标数据块的数据页为待覆盖写入数据的数据页。
  10. 根据权利要求1-9中任一项所述的方法，其特征在于，所述第一索引用于标识所述第i个压缩数据块在存储介质中的存储位置，及所述j个数据块中每个数据块包含的属性信息。
  11. 一种数据压缩装置，其特征在于，所述装置包括：
    第一获取单元，用于获取可读写文件系统中数据区的m个数据块，m为大于等于1的正整数；
    压缩单元，用于采用预设压缩算法压缩所述m个数据块，依次得到n个压缩数据 块，其中，每个压缩数据块的第一容量相同，所述第一容量表征压缩数据块能够包含的压缩处理后的数据的字节数，n为大于等于1的正整数；
    更新单元，用于建立所述n个压缩数据中的第i个压缩数据块对应的j个数据块中每个数据块的第一索引，并记录所述第一索引与j个数据块的映射关系；其中，i为大于等于1，且小于等于n的正整数；j为大于等于1，且小于等于m的正整数；
    其中，所述第一索引用于标识所述j个数据块包含中每个数据块在存储介质中的存储位置，及所述j个数据块中每个数据块包含的属性信息。
  12. 根据权利要求11所述的装置，其特征在于，所述压缩单元用于：
    将所述m个数据块中的各数据块以预设顺序依次分配至第一集合；
    当所述第一集合中的所述j个数据块的数据容量等于所述第一集合的额定容量时，按照设定压缩的阈值，对所述j个数据块执行压缩操作，并得到所述第i个压缩数据块。
  13. 根据权利要求12所述的装置，其特征在于，所述更新单元用于：
    当所述第i个压缩数据块的头部数据和压缩数据的数据总长度与所述设定压缩的阈值之和小于等于所述j个数据块的数据总长度时，建立所述j个数据块中每个数据块的第一索引。
  14. 根据权利要求11-13中任一项所述的装置，其特征在于，所述属性信息包括以下至少一项：
    第一属性，用于表征数据块被压缩后所在的压缩数据块的存储位置是否是预先分配的；
    第二属性，用于表征数据块的数据页是否有效；
    第三属性，用于表征数据块的数据页是否是所述数据块的压缩数据块的第一个压缩页；
    第四属性，用于表征数据块的数据页是否包含在两个压缩块的压缩数据页中；
    第五属性，用于表征数据块的数据页是否是所述数据块被压缩后的压缩数据块的压缩页；
    第六属性，用于表征数据块的数据页所在的压缩数据块的索引地址；
    第七属性，用于表征当数据块的数据页属于所述数据块的压缩数据块的第一个压缩页时，所述第七属性的属性值为所述数据块在所述压缩数据块对应的集合内的偏移；当所述数据块的数据页不属于所述数据块的压缩数据块的第一个压缩页时，所述第七属性的属性值为所述数据块的数据页距离压缩数据块的第一个压缩页的距离。
  15. 根据权利要求14所述的装置，其特征在于，所述属性信息包括第三属性，所述更新单元还用于：
    当所述j个数据块中每个数据块的数据页是所述第i个压缩数据块的第一个压缩页时，所述第三属性的属性值赋值为1；
    当所述j个数据块中每个数据块的数据页不是所述第i个压缩数据块的第一个压缩页时，所述第三属性的属性值赋值为0。
  16. 根据权利要求14或15所述的装置，其特征在于，所述属性信息包括第七属性，所述更新单元还用于：
    当所述第三属性的属性值为1时，更新所述第七属性的属性值为所述数据块在所 述压缩数据块对应的集合内的偏移；
    当所述第三属性的属性值为0时，更新所述第七属性的属性值为所述数据块的数据页距离压缩数据块的第一个压缩页的距离。
  17. 根据权利要求14-16中任一项所述的装置，其特征在于，所述属性信息包括第四属性，所述更新单元还用于：
    当所述j个数据块中的每个数据块的数据页包含在两个压缩块的压缩数据页中时，所述第四属性的属性值赋值为1；
    当所述j个数据块中的每个数据块的数据页不包含在两个压缩块的压缩数据页中时，所述第四属性的属性值赋值为0。
  18. 根据权利要求14-17中任一项所述的装置，其特征在于，所述属性信息包括第二属性，所述更新单元还用于：
    当所述j个数据块中的每个数据块的数据页为有效时，所述第二属性的属性值赋值为1；
    当所述j个数据块中的每个数据块的数据页为无效时，所述第二属性的属性值赋值为0。
  19. 根据权利要求11-18中任一项所述的装置，其特征在于，还包括：
    第二获取单元，用于获取待覆盖写入数据的第二集合，所述第二集合包括p个压缩数据块，p为大于等于1的正整数；
    第三获取单元，用于获取所述p个压缩数据块中第一目标压缩数据的压缩页，及所述第一目标压缩数据块的压缩页对应的q个数据块，q为大于等于1的正整数；
    第一确定单元，用于确定所述q个数据块中的第一目标数据块在所述q个数据块中的位置偏移；
    第二确定单元，用于确定所述第一目标数据块的数据页为待覆盖写入数据的数据页。
  20. 根据权利要求11-19中任一项所述的装置，其特征在于，所述第一索引用于标识所述第i个压缩数据块在存储介质中的存储位置，及所述j个数据块中每个数据块包含的属性信息。
  21. 一种设备，其特征在于，包括：用于执行权利要求1至10中任一项所述的数据压缩方法。
  22. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备执行如权利要求1至10中任一项所述的数据压缩方法。
  23. 一种计算机程序，其特征在于，当所述程序被处理器调用时，权利要求1至10中任一项所述的数据压缩方法被执行。
  24. 一种芯片系统，其特征在于，包括一个或多个处理器，当所述一个或多个处理器执行指令时，所述一个或多个处理器执行如权利要求1至10中任一项所述的数据压缩方法。
PCT/CN2022/085621 2021-06-16 2022-04-07 一种数据压缩方法及装置 WO2022262381A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP22823871.3A EP4336336A1 (en) 2021-06-16 2022-04-07 Data compression method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110667882.7 2021-06-16
CN202110667882.7A CN115480692A (zh) 2021-06-16 2021-06-16 一种数据压缩方法及装置

Publications (1)

Publication Number Publication Date
WO2022262381A1 true WO2022262381A1 (zh) 2022-12-22

Family

ID=84419764

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/085621 WO2022262381A1 (zh) 2021-06-16 2022-04-07 一种数据压缩方法及装置

Country Status (3)

Country Link
EP (1) EP4336336A1 (zh)
CN (1) CN115480692A (zh)
WO (1) WO2022262381A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117389484A (zh) * 2023-12-12 2024-01-12 深圳大普微电子股份有限公司 数据存储处理方法、装置、设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101727298A (zh) * 2009-11-04 2010-06-09 北京东方广视科技股份有限公司 实现独立磁盘冗余阵列的方法和装置
CN103020205A (zh) * 2012-12-05 2013-04-03 北京普泽天玑数据技术有限公司 一种分布式文件系统上基于硬件加速卡的压缩解压缩方法
CN103516369A (zh) * 2013-06-20 2014-01-15 易乐天 一种自适应数据压缩和解压缩的方法和系统及存储装置
CN107947799A (zh) * 2017-11-28 2018-04-20 郑州云海信息技术有限公司 一种数据压缩方法及装置
US20190339911A1 (en) * 2018-05-04 2019-11-07 EMC IP Holding Company LLC Reporting of space savings due to compression in storage systems
CN110557124A (zh) * 2018-05-30 2019-12-10 华为技术有限公司 一种数据压缩方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101727298A (zh) * 2009-11-04 2010-06-09 北京东方广视科技股份有限公司 实现独立磁盘冗余阵列的方法和装置
CN103020205A (zh) * 2012-12-05 2013-04-03 北京普泽天玑数据技术有限公司 一种分布式文件系统上基于硬件加速卡的压缩解压缩方法
CN103516369A (zh) * 2013-06-20 2014-01-15 易乐天 一种自适应数据压缩和解压缩的方法和系统及存储装置
CN107947799A (zh) * 2017-11-28 2018-04-20 郑州云海信息技术有限公司 一种数据压缩方法及装置
US20190339911A1 (en) * 2018-05-04 2019-11-07 EMC IP Holding Company LLC Reporting of space savings due to compression in storage systems
CN110557124A (zh) * 2018-05-30 2019-12-10 华为技术有限公司 一种数据压缩方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117389484A (zh) * 2023-12-12 2024-01-12 深圳大普微电子股份有限公司 数据存储处理方法、装置、设备及存储介质
CN117389484B (zh) * 2023-12-12 2024-04-26 深圳大普微电子股份有限公司 数据存储处理方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN115480692A (zh) 2022-12-16
EP4336336A1 (en) 2024-03-13

Similar Documents

Publication Publication Date Title
Kwon et al. Strata: A cross media file system
JP6664218B2 (ja) 効率的なデータオブジェクトストレージ及び検索
US8285967B1 (en) Method for on-demand block map generation for direct mapped LUN
TWI778157B (zh) 固態驅動器、分散式資料儲存系統和利用鍵值儲存的方法
US9342256B2 (en) Epoch based storage management for a storage device
US8578128B1 (en) Virtual block mapping for relocating compressed and/or encrypted file data block blocks
CN107180092B (zh) 一种文件系统的控制方法、装置及终端
US11061770B1 (en) Reconstruction of logical pages in a storage system
US9075754B1 (en) Managing cache backup and restore
US11256678B2 (en) Reconstruction of links between logical pages in a storage system
US9021222B1 (en) Managing incremental cache backup and restore
WO2022095346A1 (zh) 一种区块链数据存储方法、系统、设备及可读存储介质
US11099940B1 (en) Reconstruction of links to orphaned logical pages in a storage system
US11334523B2 (en) Finding storage objects of a snapshot group pointing to a logical page in a logical address space of a storage system
US11625169B2 (en) Efficient token management in a storage system
US11269547B2 (en) Reusing overwritten portion of write buffer of a storage system
CN114860163A (zh) 一种存储系统、内存管理方法和管理节点
CN115427941A (zh) 数据管理系统和控制的方法
WO2021208239A1 (zh) 一种低延迟的文件系统地址空间管理方法、系统及介质
US11210230B2 (en) Cache retention for inline deduplication based on number of physical blocks with common fingerprints among multiple cache entries
WO2022262381A1 (zh) 一种数据压缩方法及装置
US11232043B2 (en) Mapping virtual block addresses to portions of a logical address space that point to the virtual block addresses
US11868256B2 (en) Techniques for metadata updating and retrieval
CN115904255A (zh) 一种数据请求方法、装置、设备及存储介质
US11620062B1 (en) Resource allocation techniques using a metadata log

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22823871

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18567644

Country of ref document: US

Ref document number: 2022823871

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022823871

Country of ref document: EP

Effective date: 20231206

ENP Entry into the national phase

Ref document number: 2023577669

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE