WO2021169388A1 - 一种压缩数据落盘的方法、系统、设备及可读存储介质 - Google Patents

一种压缩数据落盘的方法、系统、设备及可读存储介质 Download PDF

Info

Publication number
WO2021169388A1
WO2021169388A1 PCT/CN2020/126353 CN2020126353W WO2021169388A1 WO 2021169388 A1 WO2021169388 A1 WO 2021169388A1 CN 2020126353 W CN2020126353 W CN 2020126353W WO 2021169388 A1 WO2021169388 A1 WO 2021169388A1
Authority
WO
WIPO (PCT)
Prior art keywords
compressed data
linked list
length
disk
data
Prior art date
Application number
PCT/CN2020/126353
Other languages
English (en)
French (fr)
Inventor
亓国强
Original Assignee
苏州浪潮智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏州浪潮智能科技有限公司 filed Critical 苏州浪潮智能科技有限公司
Priority to US17/907,796 priority Critical patent/US12026371B2/en
Publication of WO2021169388A1 publication Critical patent/WO2021169388A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0676—Magnetic disk device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks

Definitions

  • This application relates to the field of compressed data placement, and in particular, to a method, system, device, and readable storage medium for compressing data placement.
  • Data compression refers to reducing the amount of data without losing useful information to reduce storage space, improving its transmission, storage and processing efficiency, or reorganizing data according to a certain algorithm to reduce data redundancy and storage space A technical approach.
  • data compression technology With the development of data compression technology, the amount of data in the storage system has been greatly reduced, thereby reducing storage space and improving system transmission, storage and processing efficiency.
  • the purpose of this application is to provide a method, system, device, and readable storage medium for compressing data to a disk, which are used to reduce the impact of compressed data on IO performance when the compressed data is dropped to a disk.
  • this application provides a method for compressing data to disk.
  • the method includes:
  • the method further includes:
  • the method further includes:
  • the method further includes:
  • This application also provides a system for compressing data to disk, and the system includes:
  • An obtaining module for obtaining compressed data and determining the length of the compressed data
  • a data storage module for storing the compressed data in a linked list, and adding the length of the compressed data to the length of the linked list; wherein the length of the linked list is the total length of all compressed data in the linked list;
  • a judging module for judging whether the length of the linked list exceeds a threshold
  • the disk placement module is configured to allocate idle threads to merge all compressed data in the linked list when the length of the linked list exceeds the threshold, and perform a disk placement operation on the combined compressed data.
  • it also includes:
  • the delete module is used for allocating idle threads to merge all the compressed data in the linked list, and after the merged compressed data is placed in the disk, delete all the compressed data in the linked list, and delete all compressed data in the linked list.
  • the length of the linked list is set to zero.
  • it also includes:
  • the return module is used to return to the acquisition module to execute the steps of acquiring compressed data and determining the length of the compressed data when the length of the linked list does not exceed the threshold.
  • it also includes:
  • the output module is used for outputting a prompt message indicating completion of the compressed data placement after all the compressed data in the linked list is combined by allocating idle threads, and the combined compressed data is placed on the disk.
  • This application also provides a compressed data storage device, the compressed data storage device includes:
  • Memory used to store computer programs
  • the processor is configured to implement the steps of the method for compressing data to a disk as described in any one of the foregoing when executing the computer program.
  • the present application also provides a readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for compressing data to a disk as described in any one of the above are implemented.
  • the method for placing compressed data includes: obtaining compressed data and determining the length of the compressed data; storing the compressed data in a linked list, and adding the length of the compressed data to the length of the linked list; wherein the length of the linked list is the linked list The total length of all compressed data in the list; determine whether the length of the linked list exceeds the threshold; if so, allocate idle threads to merge all the compressed data in the linked list, and perform a disk placement operation on the merged compressed data.
  • the technical solution provided by this application stores compressed data in a linked list, and adds the length of the compressed data to the length of the linked list, and then allocates idle threads to merge all compressed data in the linked list when the length of the linked list exceeds the threshold.
  • the merging process of compressed data is decoupled from the IO path. There is no need to merge compressed data during the IO process of compressing the disk. Instead, idle threads are specifically allocated to merge the compressed data, thereby avoiding the impact on IO performance. Influence, make full use of CPU resources and improve storage performance.
  • This application also provides a system, device and readable storage medium for compressing data to disk, which have the above-mentioned beneficial effects, and will not be repeated here.
  • FIG. 1 is a flowchart of a method for compressing data to disk according to an embodiment of the application
  • FIG. 2 is a structural diagram of a system for compressing data to disk provided by an embodiment of the application
  • FIG. 3 is a structural diagram of another system for compressing data to disk provided by an embodiment of the application.
  • FIG. 4 is a structural diagram of a compressed data disk placement device provided by an embodiment of the application.
  • the core of this application is to provide a method, system, device, and readable storage medium for compressing data to a disk, which are used to reduce the impact of compressed data on IO performance when the compressed data is dropped to a disk.
  • FIG. 1 is a flowchart of a method for compressing data to disk according to an embodiment of the application.
  • the compressed data mentioned here can be compressed data obtained after the compression card compresses the data, or compressed data obtained after the data is compressed by a compression program.
  • This application does not specifically limit the source of the compressed data.
  • the length of the compressed data mentioned here is the length of the compressed data.
  • S102 Store the compressed data in the linked list, and add the length of the compressed data to the length of the linked list;
  • the length of the linked list mentioned here is the total length of all compressed data in the linked list. After the compressed data is stored in the linked list, the length of the compressed data is added to the length of the linked list to ensure the length of the linked list and the total length of all compressed data in the linked list. It is the same.
  • the present application can determine the amount of compressed data stored in the linked list by the length of the linked list, and then determine whether the current compressed data needs to be placed in a disk.
  • step S104 If yes, go to step S104;
  • the threshold mentioned here is the maximum value allowed by the total length of all compressed data in the linked list. When the length of the linked list exceeds the threshold, it indicates that there are enough compressed data in the linked list, and merging and placing operations are required. Go to step S104;
  • the steps of obtaining compressed data and determining the length of the compressed data can be returned to, until the length of the linked list exceeds the threshold, then the compressed data in the linked list is merged and placed.
  • S104 Allocate idle threads to merge all compressed data in the linked list, and perform a disk placement operation on the merged compressed data.
  • the following steps may also be performed:
  • the following steps may also be performed :
  • a method for compressing data to disk is to store the compressed data in a linked list, and add the length of the compressed data to the length of the linked list, and then allocate free space when the length of the linked list exceeds the threshold.
  • the thread merges all the compressed data in the linked list, and decouples the merging process of the compressed data from the IO path. It does not need to merge the compressed data during the IO process of compressing the disk. Instead, it allocates an idle thread to compress the data.
  • the merging work of IO thus avoiding the impact on IO performance, making full use of CPU resources, and improving storage performance.
  • FIG. 2 is a structural diagram of a system for compressing data to disk provided by an embodiment of the application.
  • the system can include:
  • the obtaining module 100 is used to obtain compressed data and determine the length of the compressed data
  • the data storage module 200 is used for storing compressed data in a linked list, and adding the length of the compressed data to the length of the linked list; wherein the length of the linked list is the total length of all compressed data in the linked list;
  • the judging module 300 is used to judge whether the length of the linked list exceeds a threshold
  • the disk placement module 400 is used to allocate idle threads to merge all compressed data in the linked list when the length of the linked list exceeds the threshold, and perform disk placement operations on the combined compressed data.
  • FIG. 3 is a structural diagram of another system for compressing data to disk provided by an embodiment of the application.
  • the system can also include:
  • the delete module is used to allocate idle threads to merge all the compressed data in the linked list, and after the merged compressed data is placed on the disk, delete all the compressed data in the linked list and set the length of the linked list to zero.
  • the system can also include:
  • the return module is used to return to the acquisition module to execute the steps of acquiring compressed data and determining the length of the compressed data when the length of the linked list does not exceed the threshold.
  • the system can also include:
  • the output module is used for outputting a prompt message indicating that the compressed data is placed on the disk after all the compressed data in the linked list is combined by allocating idle threads, and the combined compressed data is placed on the disk.
  • the acquisition module obtains the compressed data and determines the length of the compressed data; the data storage module stores the compressed data in the linked list, and adds the length of the compressed data to the length of the linked list; the judgment module determines whether the length of the linked list exceeds the threshold; when the length of the linked list does not exceed When the threshold is reached, return to the obtaining module to perform the steps of obtaining compressed data and determining the length of the compressed data.
  • the placing module allocates an idle thread to merge all the compressed data in the linked list, and performs the placing operation on the combined compressed data; in the allocation of the idle thread, all the compressed data in the linked list is merged, and After performing the disk placement operation on the combined compressed data, the delete module deletes all the compressed data in the linked list and sets the length of the linked list to zero; the output module outputs a prompt message that the compressed data placement is completed.
  • FIG. 4 is a structural diagram of a compressed data disk placement device provided by an embodiment of the application.
  • the compressed data placement device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 622 (for example, one or more processors) and a memory 632.
  • processors central processing units, CPU
  • One or more storage media 630 for storing application programs 652 or data 655 for example, one or one storage device with a large amount of storage.
  • the memory 632 and the storage medium 630 may be short-term storage or persistent storage.
  • the program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the device.
  • the processor 622 may be configured to communicate with the storage medium 630, and execute a series of instruction operations in the storage medium 630 on the compressed data storage device 600.
  • the compressed data storage device 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and/or one or more operating systems 651, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • operating systems 651 such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • the steps in the compressed data disk placement method described in FIG. 1 are implemented by the compressed data disk placement device based on the structure shown in FIG. 4.
  • the disclosed devices, equipment, and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules or components can be combined or integrated. To another system, or some features can be ignored, or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or modules, and may be in electrical, mechanical or other forms.
  • the modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional modules in the various embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or software functional modules.
  • the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to make a computer device (which can be a personal computer, a function calling device, or a network device, etc.) execute all or part of the steps of the methods in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种压缩数据落盘的方法、系统、设备及可读存储介质，所述方法包括：获取压缩数据，并确定压缩数据的长度；将压缩数据存入链表中，并将压缩数据的长度加入到链表长度中；其中，链表长度为链表中所有压缩数据的总长度；判断链表长度是否超过阈值；若是，则分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作。本申请将压缩数据的合并过程从IO路径中解耦出来，不需要在压缩落盘的IO过程中进行压缩数据合并，而是专门分配了空闲线程进行压缩数据的合并工作，进而避免了对IO性能的影响，充分的利用了CPU资源，提升了存储性能。

Description

一种压缩数据落盘的方法、系统、设备及可读存储介质
本申请要求于2020年02月28日提交中国专利局、申请号为202010131190.6、发明名称为“一种压缩数据落盘的方法、系统、设备及可读存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及压缩数据落盘领域，特别涉及一种压缩数据落盘的方法、系统、设备及可读存储介质。
背景技术
数据压缩是指在不丢失有用信息的前提下，缩减数据量以减少存储空间，提高其传输、存储和处理效率，或按照一定的算法对数据进行重新组织，减少数据的冗余和存储的空间的一种技术方法。随着数据压缩技术的发展，较大地缩减了存储系统中的数据量，从而减少了存储空间，提高了系统传输、存储和处理效率。
然而，随着数据业务的日益膨胀，对存储设备高负载、低时延的要求也越来越高，这就要求存储设备的具备足够高的读写速度，现有的数据压缩过程中，需要将压缩后的数据合并到一起再进行落盘操作，该过程涉及内存内字节级数据的拷贝，而压缩数据的合并动作是嵌在IO路径上的，导致IO性能受到影响。
因此，如何降低压缩数据落盘时对IO性能的影响是本领域技术人员目前需要解决的技术问题。
发明内容
本申请的目的是提供一种压缩数据落盘的方法、系统、设备及可读存储介质，用于降低压缩数据落盘时对IO性能的影响。
为解决上述技术问题，本申请提供一种压缩数据落盘的方法，该方法包括：
获取压缩数据，并确定所述压缩数据的长度；
将所述压缩数据存入链表中，并将所述压缩数据的长度加入到链表长度中；其中，所述链表长度为所述链表中所有压缩数据的总长度；
判断所述链表长度是否超过阈值；
若是，则分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作。
可选的，在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，还包括：
将所述链表中的所有的压缩数据删除，并将所述链表长度置为零。
可选的，当所述链表长度未超过所述阈值时，还包括：
返回执行获取压缩数据，并确定所述压缩数据的长度的步骤。
可选的，在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，还包括：
输出压缩数据落盘完成的提示信息。
本申请还提供一种压缩数据落盘的系统，该系统包括：
获取模块，用于获取压缩数据，并确定所述压缩数据的长度；
数据存放模块，用于将所述压缩数据存入链表中，并将所述压缩数据的长度加入到链表长度中；其中，所述链表长度为所述链表中所有压缩数据的总长度；
判断模块，用于判断所述链表长度是否超过阈值；
落盘模块，用于当所述链表长度超过所述阈值时，分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作。
可选的，还包括：
删除模块，用于在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，将所述链表中的所有的压缩数据删除，并将所述链表长度置为零。
可选的，还包括：
返回模块，用于当所述链表长度未超过所述阈值时，返回所述获取模 块执行获取压缩数据，并确定所述压缩数据的长度的步骤。
可选的，还包括：
输出模块，用于在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，输出压缩数据落盘完成的提示信息。
本申请还提供一种压缩数据落盘设备，该压缩数据落盘设备包括：
存储器，用于存储计算机程序；
处理器，用于执行所述计算机程序时实现如上述任一项所述压缩数据落盘的方法的步骤。
本申请还提供一种可读存储介质，所述可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如上述任一项所述压缩数据落盘的方法的步骤。
本申请所提供压缩数据落盘的方法，包括：获取压缩数据，并确定压缩数据的长度；将压缩数据存入链表中，并将压缩数据的长度加入到链表长度中；其中，链表长度为链表中所有压缩数据的总长度；判断链表长度是否超过阈值；若是，则分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作。
本申请所提供的技术方案，通过将压缩数据存入链表中，并将压缩数据的长度加入到链表长度中，然后在链表长度超过阈值时，分配空闲线程将链表中所有的压缩数据进行合并，将压缩数据的合并过程从IO路径中解耦出来，不需要在压缩落盘的IO过程中进行压缩数据合并，而是专门分配了空闲线程进行压缩数据的合并工作，进而避免了对IO性能的影响，充分的利用了CPU资源，提升了存储性能。本申请同时还提供了一种压缩数据落盘的系统、设备及可读存储介质，具有上述有益效果，在此不再赘述。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地， 下面描述中的附图仅仅是本申请的实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的附图获得其他的附图。
图1为本申请实施例所提供的一种压缩数据落盘的方法的流程图；
图2为本申请实施例所提供的一种压缩数据落盘的系统的结构图；
图3为本申请实施例所提供的另一种压缩数据落盘的系统的结构图；
图4为本申请实施例所提供的一种压缩数据落盘设备的结构图。
具体实施方式
本申请的核心是提供一种压缩数据落盘的方法、系统、设备及可读存储介质，用于降低压缩数据落盘时对IO性能的影响。
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
请参考图1，图1为本申请实施例所提供的一种压缩数据落盘的方法的流程图。
其具体包括如下步骤：
S101：获取压缩数据，并确定压缩数据的长度；
基于现有的数据压缩过程中，需要将压缩后的数据合并到一起再进行落盘操作，该过程涉及内存内字节级数据的拷贝，而压缩数据的合并动作是嵌在IO路径上的，导致IO性能受到影响；故本申请提供了一种压缩数据落盘的方法，用于解决上述问题；
可选的，这里提到的压缩数据可以为压缩卡对数据进行压缩后得到的压缩数据，也可以为压缩程序对数据进行压缩后得到的压缩数据，本申请对压缩数据的来源不作具体限定，这里提到的压缩数据的长度即为压缩后数据的长度。
S102：将压缩数据存入链表中，并将压缩数据的长度加入到链表长度中；
这里提到的链表长度为链表中所有压缩数据的总长度，在将压缩数据存入链表中后，将压缩数据的长度加入到链表长度中，以保证链表长度与链表中所有压缩数据的总长度是相同的，本申请可以通过该链表长度来确定链表中已经存储的压缩数据的数量，进而判断当前的压缩数据的是否需要进行落盘操作。
S103：判断链表长度是否超过阈值；
若是，则进入步骤S104；
这里提到的阈值即为链表中所有压缩数据的总长度所允许的最大值，当链表长度超过阈值时，则表明链表中的压缩数据已经足够多了，需要进行合并及落盘操作，此时执行步骤S104；
可选的，当链表长度未超过阈值时，还可以返回执行获取压缩数据，并确定压缩数据的长度的步骤，直到链表长度超过阈值后，再对链表中的压缩数据进行合并及落盘操作。
S104：分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作。
优选的，为使链表能继续存储下一次合并前的压缩数据，在分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作之后，还可以执行如下步骤：
将链表中的所有的压缩数据删除，并将链表长度置为零。
优选的，为使用户能够及时对落盘的数据进行进一步的操作，在分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作之后，还可以执行如下步骤：
输出压缩数据落盘完成的提示信息。
基于上述技术方案，本申请所提供的一种压缩数据落盘的方法，通过将压缩数据存入链表中，并将压缩数据的长度加入到链表长度中，然后在链表长度超过阈值时，分配空闲线程将链表中所有的压缩数据进行合并，将压缩数据的合并过程从IO路径中解耦出来，不需要在压缩落盘的IO过程中进行压缩数据合并，而是专门分配了空闲线程进行压缩数据的合并工作，进而避免了对IO性能的影响，充分的利用了CPU资源，提升了存储性能。
请参考图2，图2为本申请实施例所提供的一种压缩数据落盘的系统的结构图。
该系统可以包括：
获取模块100，用于获取压缩数据，并确定压缩数据的长度；
数据存放模块200，用于将压缩数据存入链表中，并将压缩数据的长度加入到链表长度中；其中，链表长度为链表中所有压缩数据的总长度；
判断模块300，用于判断链表长度是否超过阈值；
落盘模块400，用于当链表长度超过阈值时，分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作。
请参考图3，图3为本申请实施例所提供的另一种压缩数据落盘的系统的结构图。
该系统还可以包括：
删除模块，用于在分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作之后，将链表中的所有的压缩数据删除，并将链表长度置为零。
该系统还可以包括：
返回模块，用于当链表长度未超过阈值时，返回获取模块执行获取压缩数据，并确定压缩数据的长度的步骤。
该系统还可以包括：
输出模块，用于在分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作之后，输出压缩数据落盘完成的提示信息。
以上系统中的各个组成部分可实际应用于以下的实施例中：
获取模块获取压缩数据，并确定压缩数据的长度；数据存放模块将压缩数据存入链表中，并将压缩数据的长度加入到链表长度中；判断模块判断链表长度是否超过阈值；当链表长度未超过阈值时，返回获取模块执行获取压缩数据，并确定压缩数据的长度的步骤。
当链表长度超过阈值时，落盘模块分配空闲线程将链表中所有的压缩 数据进行合并，并对合并后的压缩数据进行落盘操作；在分配空闲线程将链表中所有的压缩数据进行合并，并对合并后的压缩数据进行落盘操作之后，删除模块将链表中的所有的压缩数据删除，并将链表长度置为零；输出模块输出压缩数据落盘完成的提示信息。
请参考图4，图4为本申请实施例所提供的一种压缩数据落盘设备的结构图。
该压缩数据落盘设备600可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上处理器(central processing units，CPU)622(例如，一个或一个以上处理器)和存储器632，一个或一个以上存储应用程序652或数据655的存储介质630(例如一个或一个以上海量存储设备)。其中，存储器632和存储介质630可以是短暂存储或持久存储。存储在存储介质630的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对装置中的一系列指令操作。更进一步地，处理器622可以设置为与存储介质630通信，在压缩数据落盘设备600上执行存储介质630中的一系列指令操作。
压缩数据落盘设备600还可以包括一个或一个以上电源626，一个或一个以上有线或无线网络接口650，一个或一个以上输入输出接口658，和/或，一个或一个以上操作系统651，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
上述图1所描述的压缩数据落盘的方法中的步骤由压缩数据落盘设备基于该图4所示的结构实现。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置、设备和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个模块或组件可以结合或者可以集成到另 一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或模块的间接耦合或通信连接，可以是电性，机械或其它的形式。
作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络模块上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，功能调用装置，或者网络设备等)执行本申请各个实施例方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上对本申请所提供的一种压缩数据落盘的方法、系统、设备及可读存储介质进行了详细介绍。本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想。应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以对本申请进行若干改进和修饰，这些改进和修饰也落入本申请权利要求的保护范围内。
还需要说明的是，在本说明书中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且， 术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括要素的过程、方法、物品或者设备中还存在另外的相同要素。

Claims (10)

  1. 一种压缩数据落盘的方法，其特征在于，包括：
    获取压缩数据，并确定所述压缩数据的长度；
    将所述压缩数据存入链表中，并将所述压缩数据的长度加入到链表长度中；其中，所述链表长度为所述链表中所有压缩数据的总长度；
    判断所述链表长度是否超过阈值；
    若是，则分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作。
  2. 根据权利要求1所述的方法，其特征在于，在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，还包括：
    将所述链表中的所有的压缩数据删除，并将所述链表长度置为零。
  3. 根据权利要求1所述的方法，其特征在于，当所述链表长度未超过所述阈值时，还包括：
    返回执行获取压缩数据，并确定所述压缩数据的长度的步骤。
  4. 根据权利要求1所述的方法，其特征在于，在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，还包括：
    输出压缩数据落盘完成的提示信息。
  5. 一种压缩数据落盘的系统，其特征在于，包括：
    获取模块，用于获取压缩数据，并确定所述压缩数据的长度；
    数据存放模块，用于将所述压缩数据存入链表中，并将所述压缩数据的长度加入到链表长度中；其中，所述链表长度为所述链表中所有压缩数据的总长度；
    判断模块，用于判断所述链表长度是否超过阈值；
    落盘模块，用于当所述链表长度超过所述阈值时，分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作。
  6. 根据权利要求5所述的系统，其特征在于，还包括：
    删除模块，用于在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，将所述链表中的所有的压缩数据删除，并将所述链表长度置为零。
  7. 根据权利要求5所述的系统，其特征在于，还包括：
    返回模块，用于当所述链表长度未超过所述阈值时，返回所述获取模块执行获取压缩数据，并确定所述压缩数据的长度的步骤。
  8. 根据权利要求5所述的系统，其特征在于，还包括：
    输出模块，用于在分配空闲线程将所述链表中所有的压缩数据进行合并，并对所述合并后的压缩数据进行落盘操作之后，输出压缩数据落盘完成的提示信息。
  9. 一种压缩数据落盘设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述计算机程序时实现如权利要求1至4任一项所述压缩数据落盘的方法的步骤。
  10. 一种可读存储介质，其特征在于，所述可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1至4任一项所述压缩数据落盘的方法的步骤。
PCT/CN2020/126353 2020-02-28 2020-11-04 一种压缩数据落盘的方法、系统、设备及可读存储介质 WO2021169388A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/907,796 US12026371B2 (en) 2020-02-28 2020-11-04 Method, system, and device for writing compressed data to disk, and readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010131190.6 2020-02-28
CN202010131190.6A CN111309269B (zh) 2020-02-28 2020-02-28 一种压缩数据落盘的方法、系统、设备及可读存储介质

Publications (1)

Publication Number Publication Date
WO2021169388A1 true WO2021169388A1 (zh) 2021-09-02

Family

ID=71149302

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/126353 WO2021169388A1 (zh) 2020-02-28 2020-11-04 一种压缩数据落盘的方法、系统、设备及可读存储介质

Country Status (3)

Country Link
US (1) US12026371B2 (zh)
CN (1) CN111309269B (zh)
WO (1) WO2021169388A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111309269B (zh) * 2020-02-28 2021-12-17 苏州浪潮智能科技有限公司 一种压缩数据落盘的方法、系统、设备及可读存储介质
CN113360470B (zh) * 2021-05-25 2024-10-11 深圳市珍爱捷云信息技术有限公司 基于MySQL的分库分表同步方法、装置、电子设备和计算机可读存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105515842A (zh) * 2015-12-01 2016-04-20 成都科来软件有限公司 一种基于网络数据的通用查询系统及查询方法
CN109062502A (zh) * 2018-07-10 2018-12-21 郑州云海信息技术有限公司 一种数据压缩方法、装置、设备及计算机可读存储介质
US10216654B1 (en) * 2016-07-25 2019-02-26 EMC IP Holding Company LLC Data service-aware input/output scheduling
CN109460193A (zh) * 2018-11-15 2019-03-12 郑州云海信息技术有限公司 一种存储系统中io处理方法、装置及终端
CN109634528A (zh) * 2018-12-12 2019-04-16 浪潮(北京)电子信息产业有限公司 一种目标数据的落盘方法、装置、设备及存储介质
CN110727685A (zh) * 2019-10-09 2020-01-24 苏州浪潮智能科技有限公司 一种基于Cassandra数据库的数据压缩方法、设备以及存储介质
CN111309269A (zh) * 2020-02-28 2020-06-19 苏州浪潮智能科技有限公司 一种压缩数据落盘的方法、系统、设备及可读存储介质

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6523102B1 (en) * 2000-04-14 2003-02-18 Interactive Silicon, Inc. Parallel compression/decompression system and method for implementation of in-memory compressed cache improving storage density and access speed for industry standard memory subsystems and in-line memory modules
US8566286B1 (en) * 2009-05-15 2013-10-22 Idera, Inc. System and method for high speed database backup using rapidly adjusted dynamic compression ratios controlled by a feedback loop
KR20120090194A (ko) * 2011-02-07 2012-08-17 삼성전자주식회사 데이터 처리 장치 및 이를 포함하는 데이터 처리 시스템
US9218142B2 (en) * 2013-08-22 2015-12-22 International Business Machines Corporation Log data store that stores data across a plurality of storage devices using non-disjoint layers
CN103744627A (zh) * 2014-01-26 2014-04-23 武汉英泰斯特电子技术有限公司 对实时采集的数据压缩存储的方法及系统
CN103984528A (zh) * 2014-05-15 2014-08-13 中国人民解放军国防科学技术大学 基于飞腾处理器平台的多线程并发数据压缩方法
CN106569750A (zh) * 2016-11-09 2017-04-19 郑州云海信息技术有限公司 一种数据压缩方法及装置
US10783119B2 (en) * 2017-08-08 2020-09-22 Seagate Technology Llc Fixed record media conversion with data compression and encryption
CN109725840B (zh) * 2017-10-30 2022-04-05 伊姆西Ip控股有限责任公司 利用异步冲刷对写入进行节流
CN108363719B (zh) * 2018-01-02 2022-10-21 中科边缘智慧信息科技（苏州）有限公司 分布式文件系统中可配置的透明压缩方法
WO2020175720A1 (ko) * 2019-02-28 2020-09-03 엘지전자 주식회사 디지털 디바이스 및 그 제어 방법
CN110767265A (zh) * 2019-10-23 2020-02-07 中国科学院计算技术研究所 一种针对大数据基因组比对文件排序的并行加速方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105515842A (zh) * 2015-12-01 2016-04-20 成都科来软件有限公司 一种基于网络数据的通用查询系统及查询方法
US10216654B1 (en) * 2016-07-25 2019-02-26 EMC IP Holding Company LLC Data service-aware input/output scheduling
CN109062502A (zh) * 2018-07-10 2018-12-21 郑州云海信息技术有限公司 一种数据压缩方法、装置、设备及计算机可读存储介质
CN109460193A (zh) * 2018-11-15 2019-03-12 郑州云海信息技术有限公司 一种存储系统中io处理方法、装置及终端
CN109634528A (zh) * 2018-12-12 2019-04-16 浪潮(北京)电子信息产业有限公司 一种目标数据的落盘方法、装置、设备及存储介质
CN110727685A (zh) * 2019-10-09 2020-01-24 苏州浪潮智能科技有限公司 一种基于Cassandra数据库的数据压缩方法、设备以及存储介质
CN111309269A (zh) * 2020-02-28 2020-06-19 苏州浪潮智能科技有限公司 一种压缩数据落盘的方法、系统、设备及可读存储介质

Also Published As

Publication number Publication date
CN111309269B (zh) 2021-12-17
US20230221865A1 (en) 2023-07-13
US12026371B2 (en) 2024-07-02
CN111309269A (zh) 2020-06-19

Similar Documents

Publication Publication Date Title
WO2022062537A1 (zh) 数据压缩方法、装置及计算机可读存储介质
WO2021073635A1 (zh) 一种数据存储方法及装置
WO2021169388A1 (zh) 一种压缩数据落盘的方法、系统、设备及可读存储介质
US10649905B2 (en) Method and apparatus for storing data
WO2021098267A1 (zh) 一种磁盘处理的方法、系统、设备及可读存储介质
CN103294710A (zh) 一种数据存取方法和装置
US9632562B2 (en) Systems and methods for reducing volatile memory standby power in a portable computing device
WO2021223468A1 (zh) 一种基于ssd的日志数据保存方法、装置、设备和介质
US9569381B2 (en) Scheduler for memory
WO2024119775A1 (zh) 一种raid卡的节点更新方法、系统及相关装置
CN114564457B (zh) 一种针对数据库文件的存储空间优化方法及系统
CN109558456A (zh) 一种文件迁移方法、装置、设备及可读存储介质
US20220253252A1 (en) Data processing method and apparatus
CN103984528A (zh) 基于飞腾处理器平台的多线程并发数据压缩方法
WO2023151373A1 (zh) 一种实现bios选项设置的方法、系统、设备和存储介质
WO2024055571A1 (zh) 一种namespace设置方法、装置及可读存储介质
WO2024109489A1 (zh) 一种固态硬盘及其数据写方法、装置、主机和存储介质
CN114579055B (zh) 磁盘存储方法、装置、设备及介质
JP6309621B2 (ja) メモリシステムにおける電力消費を節約するためのシステムおよび方法
WO2023070462A1 (zh) 一种文件去重方法、装置和设备
US20240070120A1 (en) Data processing method and apparatus
WO2024187779A1 (zh) 业务数据存储方法、装置、计算机设备和存储介质
WO2024027140A1 (zh) 一种数据处理方法、装置、设备、系统及可读存储介质
US20230048813A1 (en) Method of storing data and method of reading data
CN105630697A (zh) 一种利用mram存储小文件的存储结构

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20921909

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20921909

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20921909

Country of ref document: EP

Kind code of ref document: A1