WO2018171296A1 - 一种文件合并方法及控制器 - Google Patents

一种文件合并方法及控制器 Download PDF

Info

Publication number
WO2018171296A1
WO2018171296A1 PCT/CN2018/071929 CN2018071929W WO2018171296A1 WO 2018171296 A1 WO2018171296 A1 WO 2018171296A1 CN 2018071929 W CN2018071929 W CN 2018071929W WO 2018171296 A1 WO2018171296 A1 WO 2018171296A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
file
controller
keyword
hard disk
Prior art date
Application number
PCT/CN2018/071929
Other languages
English (en)
French (fr)
Inventor
任波
黄飞腾
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2018171296A1 publication Critical patent/WO2018171296A1/zh
Priority to US16/579,349 priority Critical patent/US11403021B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1724—Details of de-fragmentation performed by the file system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2246—Trees, e.g. B+trees
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/062—Securing storage systems
    • G06F3/0622—Securing storage systems in relation to access
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0658—Controller construction arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G06F3/0649—Lifecycle management

Definitions

  • Embodiments of the present invention relate to the field of storage technologies, and in particular, to a file merging method and a controller.
  • the log structured merge tree (LSMtree) is a data structure corresponding to the B+ tree. It is mainly to overcome the problem of low performance caused by a large number of random IOs in the B+ tree. The essence is to achieve balance and read before reading and writing. Read performance to dramatically improve write performance.
  • KV storage engines implemented in the industry based on the LSMtree principle. These storage engines are widely used in various data storage systems.
  • the core idea of these applications to implement the KV storage engine is to convert random IO into sequential IO writes, and then store the data into a file system through local memory, and finally merge the files in multiple file systems. Sorting improves read performance.
  • how to improve the efficiency of reading data of these files is a problem that needs to be solved currently.
  • a first aspect of the present invention provides a file merging method. This method is applied to a storage system.
  • the storage system includes a controller, a solid state hard disk, and a mechanical hard disk.
  • the controller reads the first file and the second file in the solid state hard disk.
  • the first file stores a first data and a keyword of the first data
  • the second file stores a second data and a keyword of the second data.
  • the controller determines whether the keyword of the first data is the same as the keyword of the second data, and when the keyword of the first data is the same as the keyword of the second data, the controller Create a third file on the mechanical hard disk.
  • the controller then merges the first data and the second data and writes the merged data to the third file.
  • the files before merging are stored in a solid state hard disk, and the merged files are stored in a mechanical hard disk. Since the data in the file before the merge belongs to the new data, it is highly likely to be read, and storing it in the solid state hard disk can improve the efficiency of reading the data. The data in the merged file belongs to the old data, and it is less likely to be read. It is stored in the mechanical hard disk, which can reduce the cost.
  • the merged file that is, the third file, may also be stored in the solid state hard disk.
  • the efficiency of reading the data can be further improved.
  • At least two immutable hop tables and one variable tempo table are saved in a cache of the controller, where the first data is from the at least two Data in an immutable hop table of the immutable table, the second data is from data in another immutable hop table of the at least two immutable tables, the variable hop table is used to receive the host The data sent.
  • the controller monitors whether the number of files in the SSD reaches a preset threshold, when the number of files in the SSD has reached
  • the controller reads the first file and the second file in the solid state drive when the threshold is preset.
  • a trigger condition for file merging that is, when the number of files in the SSD reaches a preset threshold, file merging is started.
  • all the files in the solid state hard disk may be used as a file to be merged, or a file to be merged by a user may be specified.
  • the storage system further includes a storage level storage, and the storage level storage further stores a log file.
  • the controller receives a first write data instruction sent by the host, and writes the first write data instruction into the log file, where the first write data instruction includes the first data and a key of the first data word.
  • the controller receives a second write data instruction sent by the host, and writes the second write data instruction into the log file, where the second write data instruction includes the second data and the second data Keyword.
  • the controller plays back the log file to obtain keywords of the first data and the first data, and keywords of the second data and the second data.
  • the controller creates the first file in the solid state hard disk, writes the first data and a keyword of the first data into the first file, and creates the first file in the solid state hard disk And a second file, the second data and the keyword of the second data are written into the second file. Since the storage level memory is a new type of storage medium with high read and write efficiency, the log file is saved in the storage level memory, and the data obtained by the log file playback is more efficient.
  • the first write data instruction is written into the log file, specifically by using an additional write manner.
  • the use of additional writes can improve the efficiency of writing log files.
  • the log file further includes another write data instruction
  • the method further includes: the controller All write data instructions in the log file are sorted according to the keywords carried in each write data command. After sorting the log files, after the file data is generated by playback, the keywords of the data are also sorted in order from small to large or from large to small, which is convenient for searching.
  • a second aspect of the invention provides a controller.
  • the controller is located in a storage system including the controller, a solid state drive, and a mechanical hard disk.
  • the controller includes a communication interface and a processor.
  • the communication interface is configured to communicate with the solid state hard disk and the mechanical hard disk.
  • the processor is for performing the method of any of the implementations of the first aspect.
  • FIG. 1 is a structural diagram of a storage system according to an embodiment of the present invention
  • FIG. 2 is a schematic structural diagram of a controller 11 according to an embodiment of the present invention.
  • FIG. 3 is a structural diagram of organizing data of a controller 11 according to an embodiment of the present invention.
  • FIG. 4 is a flowchart of a file merging method according to an embodiment of the present invention.
  • LSM-Tree converts random writes of hard disks into sequential writes, which increases the speed of writing data.
  • the embodiment of the present invention applies LSM-Tree to a storage system including a solid state hard disk. The storage system described in this embodiment will be described below with reference to FIG. 1 and FIG. 2.
  • the storage system includes at least one controller 11, at least one solid state drive 44, and at least one mechanical hard disk 45.
  • the controller 11 is connected to a host (not shown) through a storage area network (SAN).
  • Controller 11 can be a computing device such as a server, desktop computer, or the like.
  • a writing system and an application are installed on the controller 11.
  • the controller 11 can receive input and output (I/O) requests from the host.
  • the controller 11 can also store data carried in the I/O request and write the data into the solid state hard disk 44.
  • the controller 11 belongs to a system controller which is a separate device, unlike the control chip in the solid state hard disk 44. This embodiment refers to the control chip of the solid state hard disk 44 as a solid state hard disk controller. Unless otherwise specified, the controller in this embodiment refers to a system controller.
  • SSD Solid State Disk
  • flash memory English: flash memory
  • SSD Solid State Drive
  • the controller 11 includes an interface card 110, a processor 112, a memory 111, and an interface card 113.
  • the interface card 110 is configured to communicate with the host, and the controller 11 can receive the write data command of the host through the interface card 110.
  • the write data instruction includes a keyword (English: key) and a value (English: value), the value refers to data to be written into the storage system (hereinafter referred to as data), and the keyword is the value.
  • the value may be various information of the student
  • the keyword may be the student's student number, or other identifier indicating a certain aspect of the student's attributes.
  • the interface card 113 is configured to communicate with the solid state hard disk 44, and the controller 11 can write a data request through the interface card 113 (including the keywords and data in the write data command described above, and the logic that the controller 11 allocates for the data. The address) is sent to the SSD 44 for storage.
  • the processor 112 may be a central processing unit (CPU). In an embodiment of the invention, the processor 112 can be configured to receive write data instructions or read instructions from the host and process the instructions. The processor 112 can also send the data in the write data command to the solid state hard disk 44. In addition, the processor 112 can also be used to perform a merge operation of files under certain conditions.
  • the memory 111, 1 includes volatile memory, non-volatile memory, or a combination thereof.
  • the volatile memory is, for example, a random access memory (RAM).
  • Non-volatile memory such as a floppy disk, a hard disk, a solid state disk (SSD), an optical disk, or the like, and various machine readable media that can store program code.
  • the memory 111 has a power-saving function, and the power-preserving function means that the data stored in the memory 111 is not lost when the system is powered off and re-powered.
  • the memory 111 may have one or more for temporarily storing data received from the host or data read from the solid state hard disk 44. For example, when the controller 11 receives a plurality of write data requests sent by the host, the plurality of writes may be performed. The data in the data request is temporarily stored in the memory 111.
  • the memory 111 also holds a variable storage table (English: memtable) and an immutable storage table (immutable memtable). According to the LSM-Tree mechanism, memtable is used to receive write data instructions sent by the host for a period of time.
  • the write data instructions stored in the memtable can be as shown in Table 1:
  • the version information refers to the time information written by the data, or other information used to identify the order in which the data is written. It can be understood that the version information in this embodiment may be accurate to the month, or may be accurate to the day, hour, minute, and second. This embodiment does not limit the form and accuracy of the version information. In addition, as long as the data written during this time will be saved in the memtable.
  • the version information is allocated for the write data instruction when the controller 11 receives the write data instruction.
  • the memtable when the amount of data of all data stored in the memtable reaches a preset threshold, the memtable is converted into an immutable memtable. Immutable memtable will no longer receive new data, and a new memtable will be created in memory 111. The new memtable will continue to receive data. Therefore, one or more immutable memtables may be stored in the memory 111, but only one memtable.
  • the data in each immutable memtable is solidified, and memtable can continuously receive newly written data.
  • the data in immutable memtable needs to be sorted from small to large or from big to small, as shown in Table 2:
  • each immutable memtable is dumped (English: dump) in a Sorted String Table (SST) file (hereinafter referred to as a file).
  • SST Sorted String Table
  • the storage system includes a plurality of files, and each file corresponds to an immutable memtable for receiving data dumped by the immutable memtable.
  • a file directly dumped by an immutable memtable that is, a file that has not been merged
  • the controller 11 includes a file system, so data can be organized in the form of files. Specifically, the structure of the controller 11 organization data is as shown in FIG. 3. As shown in FIG. 3, the key-value data is stored in an erase block (English: block) of the solid state drive 44, and the actual address in which the data is stored in the erase block is the physical address of the data.
  • the controller 11 maps one or more erase blocks into one logical space, which may be, for example, a logical unit (LU). Both the controller 11 and the solid state hard disk 44 hold the logical address of the data and the correspondence between the logical address and the physical address.
  • LU logical unit
  • the logical address of the data refers to the identity of the logical unit in which the data is located (eg, LU ID), the starting location (eg, Logical block Address) and length of the data within the logical unit.
  • the controller 11 since the controller 11 also has a file system on the upper layer of the logical space, the user can access the data in the form of a file.
  • the controller 11 also stores a correspondence between the logical address and the identifier of the file, and a correspondence between the physical address and the identifier of the file.
  • each immutable memtable contains data of different keywords, and accordingly, the files corresponding to the immutable memtable are also stored with data of different keywords.
  • the data corresponding to the same keyword will be distributed in different files. Therefore, when receiving the read command and requesting to read the data, the SSD 44 needs to obtain each data corresponding to the keyword from different files according to the keyword carried by the read command, and confirm the latest data return according to the version information. To the user. Such a process can affect the efficiency of reading data, so it is necessary to merge the files.
  • the purpose of the merge is to merge the data with the same keyword in different files. After the merge, the same keyword only corresponds to one data or a small amount of data (according to system settings or user requirements).
  • the files in the first level need to be merged.
  • the method may be that the files in the first level are merged, or the files in the first level are merged, or the files to be merged in the first level are specified by the user.
  • the new file generated after the merge is the second level file.
  • it can be re-specified to establish a third level file.
  • Combining files can reduce duplicate data or invalid data and improve the efficiency of reading data. For example, in a time period, data in the key range [1-10] is written in file1, file4 updates the data in the key range [5-20], and file7 deletes part of the data in [10-20].
  • each data is a key value (Key-Value, KV), and each key in the file is identified by the keyword in the key value, and all the keywords stored in each file are ordered. Arranged, the key range of each file is identified by a combination of a starting keyword and a terminating keyword.
  • the data in the file of the first level is more likely to be read because the data is newly received, and the data in other levels of the file belongs to the old data, and the possibility of being read is relatively high. small. Therefore, in the present embodiment, the files of the first level are deployed in the solid state hard disk 44, and files of other levels than the first level are deployed in the mechanical hard disk 45. Since the SSD reads and writes data more efficiently than the mechanical hard disk, the use of such file deployment can further improve the efficiency of reading data.
  • the storage system shown in FIG. 1 may further include a storage class memory (SCM) 46 for storing the log file, and the controller 11 first writes the information carried by the write data instruction after receiving the write data instruction.
  • the log file further writes the write data instruction to the memtable.
  • SCM storage class memory
  • this embodiment also provides another way of storing data, that is, generating an SST file directly according to the records in the log file. Specifically, after receiving a plurality of write data instructions sent by the host, the controller 11 writes each write data instruction into the log file by using an additional write manner. Therefore, the log file stores a plurality of write data instructions, each of which carries a keyword.
  • the log files are sorted according to the keywords of each write data instruction.
  • the log file further includes control information, such as an operation type, a global serial number, and the like. This information is used to generate data and the keywords of the data after the log file is played back, as well as to ensure the sequence and operational reliability during playback.
  • the operation type is used to indicate operations such as writing a data instruction or deleting an instruction.
  • the global serial number can be a log sequence number (LSN).
  • the storage system also stores one or more management files (not shown in FIG. 3) for recording the level of each file, the largest keyword of the single file, and the smallest keyword, the identifier of each file, The range of keywords included in each file and other metadata information.
  • Other metadata information includes, but is not limited to, the actual address of the data in the solid state hard disk 44, the logical address and the write record for writing the data, and the correspondence between the actual address and the identifier of the file to which the file belongs, and the logical address of the data may be Is the identifier of the file to which the data belongs, or the combination of the identifier of the file to which the data belongs and the offset position of the data in the file, or the identifier of the file to which the data belongs, the offset position of the data in the file, and the data.
  • the write record includes the time and time at which the user writes the data, and the time at which the data is written by the version information.
  • the management file may be stored in the solid state drive 44, or may be stored in the mechanical hard disk 45, or may be stored in the storage level memory 46.
  • FIG. 4 is a flowchart of a file merging method according to an embodiment of the present invention. As shown in FIG. 4, the storage method of the data is executed by the processor 112 in the controller 11, and specifically includes the following steps.
  • the controller 11 reads the first file and the second file in the solid state hard disk 44.
  • the data in the first file and the data in the second file are both from the memory 111 of the controller 11.
  • the first file is dumped by an immutable memtable in the memory 111
  • the second file is dumped by another immutable memtable in the memory 111. Since the data in the immutable memtable is sorted in order of keywords from small to large or from large to small, the data in the first file and the second file are also sorted by keyword. This is to improve the efficiency of reading data.
  • the data in the first file and the data in the second file are both played back from the log file.
  • the log file contains multiple write data instructions, as well as information such as operation type and global serial number. Therefore, the data of the first file and the data of the second file can be generated by the log file playback.
  • the size of each file is pre-set, can be the same size as an immutable memtable, or different from the size of an immutable memtable.
  • the log files are typically stored in storage level memory to take advantage of faster retrieval of data from the log files.
  • the controller 11 reads the first file, and the controller 11 queries the correspondence between the identifier of the file and the logical address of the data included in the file according to the identifier of the first file, and acquires data included in the first file (for example, a logical address of the first data); the controller 11 sends the logical address to the solid state hard disk 44, and the solid state hard disk 44 acquires data from the physical address corresponding to the logical address according to the correspondence between the logical address and the physical address And sent to the controller 11. Similarly, the controller 11 also reads the second file in this manner.
  • the first file includes the first data and the keyword of the first data
  • the second file includes the second data and the keyword of the second data
  • the controller 11 determines whether the keyword of the first data is the same as the keyword of the second data.
  • the keyword of the data is used to identify the data, and whether the keyword of the first data is the same as the keyword of the second data is to determine whether the first file and the second file need to be merged.
  • the controller 11 creates a third file on the mechanical hard disk 45.
  • the controller 11 If the keyword of the first data is the same as the keyword of the second data, it indicates that the first file and the second file need to be merged, and the merged file is the third file. Since the data in the merged file belongs to the old data, the possibility of being read is smaller than that of the first file and the second file, so from the viewpoint of cost saving, the controller 11 writes the merged data to the machine. Hard disk 45. Specifically, the controller 11 creates a third file, assigns an identifier to the third file, and allocates a logical address interval for the third file, the logical address interval corresponding to a physical address interval of the mechanical hard disk 45. The controller 11 stores a correspondence between the identifier of the third file and the logical address interval, and a correspondence between the logical address interval and the physical address interval.
  • the controller 11 merges the first data and the second data, and writes the merged data into the third file.
  • the controller 11 writes the merged data into a physical address corresponding to the allocated logical address interval.
  • the controller 11 creates a third file on the mechanical hard disk 45, and writes the data contained in the first file and the data contained in the second file into the third file, respectively.
  • the process of writing the third file in detail is the same as the process of writing the merged data into the third file, and details are not described herein again.
  • the file before merging is stored in a solid state hard disk, and the merged file is stored in a mechanical hard disk. Since the data in the file before the merge belongs to the new data, it is highly likely to be read, and storing it in the solid state hard disk can improve the efficiency of reading the data. The data in the merged file belongs to the old data, and it is less likely to be read. It is stored in the mechanical hard disk, which can reduce the cost.
  • the disclosed systems, devices, and methods may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the modules is only a logical function division, and may be implemented in another manner, for example, multiple modules or components may be combined or may be Integrate into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or otherwise.
  • the modules described as separate components may or may not be physically separated.
  • the components displayed as modules may or may not be physical modules, that is, may be located in one place, or may be distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
  • the above integrated modules can be implemented in the form of hardware or in the form of hardware plus software function modules.
  • the above-described integrated modules implemented in the form of software function modules can be stored in a computer readable storage medium.
  • the software functional modules described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform some of the steps of the methods described in various embodiments of the present invention.
  • the foregoing storage medium includes: a removable hard disk, a read only memory, a random access memory, a magnetic disk, or an optical disk, and the like, which can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种文件合并方法，该方法应用于存储系统中，所述存储系统包括控制器、固态硬盘和机械硬盘。所述控制器（11）读取所述固态硬盘（44）中的第一文件和第二文件（S301）；第一文件中保存有第一数据以及所述第一数据的关键字，第二文件中保存有第二数据以及所述第二数据的关键字；所述控制器（11）判断所述第一数据的关键字与所述第二数据的关键字是否相同（S302），当所述第一数据的关键字与所述第二数据的关键字相同时，所述控制器（11）在所述机械硬盘（45）上创建第三文件（S303）；然后，所述控制器（11）合并所述第一数据和所述第二数据，并将所述合并后的数据写入所述第三文件（S304）。上述方法可以提高读文件的效率。

Description

一种文件合并方法及控制器 技术领域
本发明实施例涉及存储技术领域，尤其涉及一种文件合并方法及控制器。
背景技术
日志结构合并树(log structured merge tree,LSMtree)是相对应B+树的一种数据结构，其主要是为了克服B+树的大量随机IO导致性能低的问题，本质是实现读写之前取得平衡，牺牲读性能来大幅度提高写性能。
业界有很多基于LSMtree原理实现的KV存储引擎，这些存储引擎被广泛应用各种数据存储系统中。这些应用实现KV存储引擎的核心思想都是将随机IO转化为顺序IO写入，数据再通过内存整理为局部有序的文件存储到文件系统，最后再对多个文件系统中的文件进行合并和排序提升读取性能。然而，如何提高这些文件的读数据的效率是当前需要解决的问题。
发明内容
本发明第一方面提供了一种文件合并方法。该方法应用于存储系统中。所述存储系统包括控制器、固态硬盘和机械硬盘。所述控制器读取所述固态硬盘中的第一文件和第二文件。第一文件中保存有第一数据以及所述第一数据的关键字，第二文件中保存有第二数据以及所述第二数据的关键字。所述控制器判断所述第一数据的关键字与所述第二数据的关键字是否相同，当所述第一数据的关键字与所述第二数据的关键字相同时，所述控制器在所述机械硬盘上创建第三文件。然后，所述控制器合并所述第一数据和所述第二数据，并将所述合并后的数据写入所述第三文件。
按照本发明第一方面提供的文件合并方法，将合并前的文件存储在固态硬盘中，将合并后的文件存储在机械硬盘中。由于合并前的文件中的数据属于新数据，被读取的可能性较大，将其存储在固态硬盘中可以提高读数据的效率。而合并后的文件中的数据属于旧数据，被读取的可能性较小，将其存储在机械硬盘中，可以降低成本。
在第一方面中，也可以将合并后的文件，即第三文件保存在固态硬盘中，虽然不能达到降低成本的目的，但可以进一步提高读数据的效率。
基于第一方面，在第一方面的第一种实现方式中，所述控制器的缓存中保存有至少两个不可变跳表和一个可变调表，所述第一数据来自于所述至少两个不可变调表中一个不可变跳表中的数据，所述第二数据来自于所述至少两个不可变调表中另一个不可变跳表中的数据，所述可变跳表用于接收主机发送的数据。
基于第一方面，在第一方面的第二种实现方式中，所述控制器监控所述固态硬盘中的文件的数量是否达到预设阈值，当所述固态硬盘中的文件的数量已达到所述预设阈值时，所述控制器读取所述固态硬盘中的所述第一文件和所述第二文件。这里是给文件合并设置了一个触发条件，即，当固态硬盘中的文件的数量达到预设阈值时，开始执行文件合并。关于如何选择待合并的文件，可以是将所述固态硬盘中的所有文件作为待合并文件，也可以是由用户指定待合并的文件等方式。
基于第一方面，在第一方面的第三种实现方式中，所述存储系统还包括储存级存储器，所述储存级存储器中还保存有日志文件。所述控制器接收主机发送的第一写数据指令，将 所述第一写数据指令写入所述日志文件，所述第一写数据指令包括所述第一数据以及所述第一数据的关键字。所述控制器接收主机发送的第二写数据指令，将所述第二写数据指令写入所述日志文件中，所述第二写数据指令包括所述第二数据以及所述第二数据的关键字。所述控制器回放所述日志文件，得到所述第一数据以及所述第一数据的关键字以及所述第二数据以及所述第二数据的关键字。所述控制器在所述固态硬盘中创建所述第一文件，将所述第一数据以及所述第一数据的关键字写入所述第一文件，在所述固态硬盘中创建所述第二文件，将所述第二数据以及所述第二数据的关键字写入所述第二文件。由于储存级存储器是一种读写效率较高的新型存储介质，将日志文件保存在储存级存储器中，通过所述日志文件回放得到数据的效率会更高
结合第一方面的第三种实现方式，在第一方面的第四种实现方式中将所述第一写数据指令写入所述日志文件中具体是采用追加写的方式。采用追加写的方式可以提高写日志文件的效率。
结合第一方面的第三种或第四种实现方式，在第一方面的第五种实现方式中，所述日志文件还包括其他写数据指令，所述方法还包括：所述控制器对所述日志文件中的所有写数据指令按照每个写数据指令中携带的关键字进行排序。经过排序后的日志文件，经回放生成文件数据之后，数据的关键字也是按照从小到大或者从大到小的顺序进行排序的，方便查找。
本发明第二方面提供了一种控制器。所述控制器位于存储系统中，所述存储系统包括所述控制器、固态硬盘和机械硬盘。所述控制器包括通信接口和处理器。其中，通信接口，用于与所述固态硬盘，以及所述机械硬盘通信。所述处理器用于执行第一方面任意一种实现方式的方法。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍。
图1为本发明实施例提供的一种存储系统的组成图；
图2为本发明实施例提供的一种控制器11的结构示例图；
图3为本发明实施例提供的一种控制器11组织数据的结构图；
图4为本发明实施例提供的一种文件合并方法的流程图。
具体实施方式
下面将结合附图，对本发明实施例中的技术方案进行清楚、完整地描述。
KV(key-value)存储系统的数据存储采用LSM-Tree的结构实现。LSM-Tree将硬盘的随机写转化为顺序写，从而提高了写数据的速度。本发明实施例将LSM-Tree应用于包含固态硬盘的存储系统中。下面结合图1和图2，对本实施例所描述的存储系统进行介绍。
如图1所示，存储系统包括至少一个控制器11、至少一个固态硬盘44和至少一个机械硬盘45。
控制器11通过存储区域网络(英文：storage area network，SAN)与主机(图中未示出)连接。控制器11可以是一种计算设备，如服务器、台式计算机等等。在控制器11上安装有写入系统以及应用程序。控制器11可以接收来自主机的输入输出(I/O)请求。控 制器11还可以存储I/O请求中携带的数据，并且将该数据写入固态硬盘44中。控制器11属于系统控制器，系统控制器是独立的设备，不同于固态硬盘44中的控制芯片。本实施例将固态硬盘44的控制芯片称为固态硬盘控制器。如无特别指明，本实施例中的控制器都是指系统控制器。
固态硬盘44(英文：Solid State Disk，SSD)是以闪存(英文：flash memory)芯片为存储介质的存储器，又名固态驱动器(Solid State Drive，SSD)。
图2是控制器11的结构示例图，如图2所示，控制器11包括接口卡110、处理器112、存储器111和接口卡113。
接口卡110，用于和主机通信，控制器11可以通过接口卡110接收主机的写数据指令。例如，写数据指令包括关键字(英文：key)和值(英文：value)，所述值是指待写入所述存储系统的数据(以下简称为数据)，所述关键字是所述值的标识。举一个具体的例子，所述值可以是学生的各种信息，所述关键字可以是该学生的学号，或者其他表示该学生某一方面属性的标识。
接口卡113，用于和固态硬盘44通信，控制器11可以通过接口卡113将写数据请求(包括上面描述的写数据指令中的关键字和数据，以及控制器11为所述数据分配的逻辑地址)发送给固态硬盘44存储。
处理器112，可能是一个中央处理器(英文：central processing unit，CPU)。在本发明实施例中，处理器112可以用于接收来自主机的写数据指令或者读取指令并且处理这些指令。处理器112还可以将所述写数据指令中的数据发送给固态硬盘44。此外，处理器112，还可以用于在一定条件时，执行文件的合并操作。存储器111，1包括易失性存储器，非易失性存储器或其组合。易失性存储器例如为随机访问存储器(英文：random-access memory，RAM)。非易失性存储器例如软盘、硬盘、固态硬盘44(solid state disk，SSD)、光盘等各种可以存储程序代码的机器可读介质。存储器111具有保电功能，保电功能是指系统发生掉电又重新上电时，存储器111中存储的数据也不会丢失。存储器111可以有一个或者多个，用于临时存储从主机接收的数据或从固态硬盘44读取的数据，例如控制器11接收主机发送的多个写数据请求时，可以将所述多个写数据请求中的数据暂时保存在存储器111中。此外，存储器111还保存有可变存储表(英文：memtable)和不可变存储表(immutable memtable)。根据LSM-Tree机制，memtable用于接收一段时间内主机发送的写数据指令。示例性的，memtable中存储的写数据指令可以如表1所示：
Key Value Version
0001 张三 2017.3
0003 李四 2017.3
…… …… ……
0002 王五 2017.3
表1
如表1所示，memtable中保存有多组数据，每组数据都包括关键字、值和版本信息。关键字和值已在上面的段落中进行了介绍，这里不再赘述。版本信息是指数据写入的时间 信息，或者其他用以标识数据写入的先后顺序的信息。可以理解的是，本实施例中版本信息可以精确到月，也可以精确到日、时、分、秒。本实施例并不对版本信息的形式和精确度进行任何限定。另外，只要在这段时间内写入的数据都会保存在所述memtable中。所述版本信息是控制器11接收写数据指令时为所述写数据指令分配的。
根据LSM-Tree机制，当memtable中保存的所有数据的数据量达到预设阈值时，所述memtable将转化为immutable memtable。immutable memtable会不再接收新的数据，同时存储器111中会再创建一个新的memtable。所述新的memtable将继续接收数据。因此存储器111中可能保存有一个或多个immutable memtable，但只有一个memtable。每个immutable memtable中的数据是固化的，而memtable可以不断接收新写入的数据。immutable memtable中的数据需要按照关键字从小到大或者从大到小进行排序，如表2所示：
Key Value Version
0001 张三 2017.3
0001 张三 2017.4
0002 王五 2017.3
0002 王五 2017.4
…… …… ……
0003 李四 2017.3
在LSM-Tree机制中，每个immutable memtable中的数据被倾倒(英文：dump)在一个分类线表(Sorted String Table，SST)文件(以下简称为文件)中。可以理解的是，所述存储系统中包含多个文件，每个文件与一个immutable memtable对应，用于接收由所述immutable memtable倾倒而来的数据。本实施例中将直接由immutable memtable倾倒而来的文件(也就是未经过合并的文件)作为第一层级的文件。
本实施例中，控制器11包含文件系统，因此可以以文件的形式组织数据。具体的，控制器11组织数据的结构如图3所示。如图3所示，key-value数据保存在固态硬盘的44的擦除块(英文：block)中，该数据保存在擦除块中的实际地址为该数据的物理地址。控制器11将一个或多个擦除块映射为一个逻辑空间，举例来说，所述逻辑空间可以是逻辑单元(logical unit，LU)。控制器11和固态硬盘44都保存有该数据的逻辑地址，以及所述逻辑地址与物理地址之间的对应关系。该数据的逻辑地址是指该数据所在的逻辑单元的标识(例如，LU ID)、该数据位于该逻辑单元内的起始位置(例如，Logical block Address)和长度。另外，由于控制器11在逻辑空间的上层还具有文件系统，因此用户可以以文件的形式访问数据。控制器11还保存有所述逻辑地址与文件的标识之间的对应关系，以及所述物理地址与文件的标识之间的对应关系。
由上面的描述可知，每个immutable memtable中包含不同关键字的数据，那么，相应地，与immutable memtable对应的文件中保存的也是不同关键字的数据。也就是说，同样的关键字所对应的数据会分布在不同的文件中。因此，当接收读取指令要求读取数据时，固态硬盘44需要根据读取指令携带的关键字从不同的文件中获取所述关键字对应的各个 数据，并根据版本信息确认出最新的数据返回给用户。这样的过程会影响读取数据的效率，因此有必要对文件进行合并。合并的目的是对不同文件中，关键字相同的数据进行合并，合并后同一个关键字只对应一个数据或少量的几个数据(根据系统设置或者用户需求设定)。
具体的，当第一层级的文件的总量达到一定阈值时，需要将第一层级中的文件进行合并。其合并方式可以是，对于第一层级中的所有文件进行合并，也可以是第一层级中的部分文件进行合并，还可以是由用户指定第一层级中待合并的文件。合并后生成的新的文件，就是第二层级的文件。依此类推，当第二层级的文件的总量达到一定阈值时，可以对其进行再次规定，建立第三层级的文件。对文件进行合并可以减少重复的数据或者无效的数据，提高读数据的效率。例如，在一个时间段内file1中写入了键范围为[1-10]的数据，file4更新了键范围为[5-20]的数据，file7又删除了[10-20]的部分数据。
另外，在SST文件中，每个数据的存储格式为键值(Key-Value，KV),通过键值中的关键字标识文件中的每个数据，每个文件保存的所有关键字是有序排列的，通过起始关键字与终止关键字的组合标识每个文件的键范围。
一般来说，第一层级的文件中的数据被读取的可能性更大，因为这些数据是新接收的数据，而其他层级的文件中的数据属于旧数据，被读取的可能性相对较小。因此，在本实施例中，将第一层级的文件部署在固态硬盘44中，将除第一层级以外的其他层级的文件部署在机械硬盘45中。由于固态硬盘读写数据的效率高于机械硬盘，因此采用这样的文件部署，可以进一步提高读数据的效率。
另外，图1所示的存储系统还可以包含储存级存储器(storage class memory，SCM)46，用于保存日志文件，控制器11在接收写数据指令之后，首先将写数据指令携带的信息写入所述日志文件，再将所述写数据指令写入memtable。上面介绍了将immutable memtable中的数据倾倒在固态硬盘44中，生成一个SST文件。另外，本实施例还提供了另一种数据存储的方式，即，直接根据所述日志文件中的记录生成SST文件。具体的，控制器11在接收主机发送的多个写数据指令之后，采用追加写的方式将每个写数据指令写入所述日志文件。因此，所述日志文件中保存有多个写数据指令，其中每个写数据指令都携带有关键字。所述日志文件按照每个写数据指令的关键字进行排序。另外，所述日志文件中还包括控制信息，例如操作类型、全局序列号等信息。这些信息用于日志文件回放后生成数据以及数据的关键字，以及保证回放时的顺序和操作的可靠性。操作类型用于指示写数据指令或者删除指令等操作。全局序列号可以是日志序列号(log sequence number，LSN)。
另外，存储系统中还保存有一个或多个管理文件(图3中未示出)，用于记录各个文件所在的层级，单个文件最大的关键字以及最小的关键字，每个文件的标识，每个文件所包含的关键字的区间以及其他元数据信息。其他元数据信息包括但不限于：数据在固态硬盘44中的实际地址，逻辑地址和对数据进行写入的写入记录，以及实际地址与所属文件的标识的对应关系等，数据的逻辑地址可以是数据所属的文件的标识，也可以是数据所属的文件的标识和数据在文件中的偏移位置的组合，也可以是数据所属的文件的标识，数据在文件中的偏移位置和数据的长度的组合。写入记录包括对用户对数据进行写入的方式和时间，通过版本信息对数据进行写入的时间。所述管理文件可以保存在固态硬盘44中，也可以保 存在机械硬盘45中，还可以保存在储存级存储器46中。
下面结合图1所示的存储系统，以及图2所示的控制器11的结构，对本发明实施例公开的文件合并方法进行具体的描述。请参见图4，图4为本发明实施例提供的一种文件合并方法的流程图。如图4所示，所述数据的存储方法由控制器11中的处理器112执行，具体包括如下步骤。
S301，控制器11读取固态硬盘44中的第一文件和第二文件。
一种情况下，第一文件中的数据和第二文件中的数据均来自控制器11的存储器111。具体的，第一文件是由存储器111中的一个immutable memtable倾倒而来的，第二文件是由存储器111中的另一个immutable memtable倾倒而来的。由于immutable memtable中的数据是按照关键字从小到大或者从大到小的顺序进行排序的，因此第一文件和第二文件中的数据也是按照关键字排序的。这是为了提高读数据的效率。
另一种情况下，第一文件中的数据和第二文件中的数据均是由日志文件回放而来的。由于日志文件中保存有多个写数据指令，以及操作类型、全局序列号等信息。因此，可以由日志文件回放生成第一文件的数据和第二文件的数据。其中，每个文件的尺寸是预先设置的，可以和一个immutable memtable的尺寸相同，也可以和一个immutable memtable的尺寸不同。在这种情况下，日志文件通常保存在储存级存储器中，以利用更加快速地从所述日志文件获取数据。
无论第一文件的数据和第二文件的数据是来自存储器111还是来自日志文件，当所述第一文件的数据和所述第二文件的数据在固态硬盘44中进行持久化存储之后，可以将日志文件中对应的内容删掉，以节省日志文件所占用的存储空间。
以控制器11读取第一文件为例，控制器11根据第一文件的标识在文件的标识与文件所包含的数据的逻辑地址的对应关系中查询，获取第一文件所包含的数据(例如，第一数据)的逻辑地址；控制器11将所述逻辑地址发送给固态硬盘44，固态硬盘44根据所述逻辑地址和物理地址的对应关系，从所述逻辑地址对应的物理地址中获取数据并发送给控制器11。同理，控制器11也按照这样的方式读取第二文件。
为了方便描述，本实施例中第一文件包含第一数据以及第一数据的关键字，第二文件包含第二数据以及第二数据的关键字。
S302，控制器11判断所述第一数据的关键字与所述第二数据的关键字是否相同。
数据的关键字是用来标识所述数据的，判断第一数据的关键字与所述第二数据的关键字是否相同是为了确定是否需要对第一文件和第二文件进行合并。
S303，当所述第一数据的关键字与所述第二数据的关键字相同时，控制器11在所述机械硬盘45上创建第三文件。
如果第一数据的关键字与所述第二数据的关键字相同，则说明第一文件和第二文件需要合并，合并后的文件为第三文件。由于合并后的文件中的数据属于旧数据，被读取的可能性比第一文件、第二文件的可能性小，因此从节省成本的角度考虑，控制器11将合并后的数据写入机械硬盘45。具体的，控制器11创建第三文件，为所述第三文件分配标识，并且为所述第三文件分配一段逻辑地址区间，所述逻辑地址区间与机械硬盘45的一段物理地 址区间对应。控制器11保存所述第三文件的标识与所述逻辑地址区间的对应关系，以及所述逻辑地址区间与所述物理地址区间之间的对应关系。
S304，所述控制器11合并所述第一数据和所述第二数据，并且将所述合并后的数据写入所述第三文件。
具体的，控制器11将所述合并后的数据写入所述分配的逻辑地址区间所对应的物理地址中。
如果第一数据的关键字与第二数据的关键字不相同，则不需要对所述第一文件与所述第二文件进行合并。那么控制器11在机械硬盘45上创建第三文件，分别将第一文件所包含的数据和第二文件所包含的数据写入第三文件。其具体的写入第三文件的过程与将合并后的数据写入所述第三文件的过程相同，这里不再赘述。
本发明实施例提供的文件合并方法，将合并前的文件存储在固态硬盘中，将合并后的文件存储在机械硬盘中。由于合并前的文件中的数据属于新数据，被读取的可能性较大，将其存储在固态硬盘中可以提高读数据的效率。而合并后的文件中的数据属于旧数据，被读取的可能性较小，将其存储在机械硬盘中，可以降低成本。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，设备和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块的划分，仅仅为一种逻辑功能划分，实现时可以有另外的划分方式，例如多个模块或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或模块的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络模块上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用硬件加软件功能模块的形式实现。
上述以软件功能模块的形式实现的集成的模块，可以存储在一个计算机可读取存储介质中。上述软件功能模块存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的部分步骤。而前述的存储介质包括：移动硬盘、只读存储器、随机存取存储器、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的保护范围。

Claims (12)

  1. 一种文件合并方法，其特征在于，所述方法应用于存储系统中，所述存储系统包括控制器、固态硬盘和机械硬盘，所述方法包括：
    所述控制器读取所述固态硬盘中的第一文件和第二文件，其中，所述第一文件中保存有第一数据以及所述第一数据的关键字，所述第二文件中保存有第二数据以及所述第二数据的关键字；
    判断所述第一数据的关键字与所述第二数据的关键字是否相同；
    当所述第一数据的关键字与所述第二数据的关键字相同时，所述控制器在所述机械硬盘上创建第三文件；
    所述控制器合并所述第一数据和所述第二数据，并将所述合并后的数据写入所述第三文件。
  2. 根据权利要求1所述的方法，其特征在于，所述控制器的缓存中保存有至少两个不可变跳表和一个可变调表，所述第一数据来自于所述至少两个不可变调表中一个不可变跳表中的数据，所述第二数据来自于所述至少两个不可变调表中另一个不可变跳表中的数据，所述可变跳表用于接收主机发送的数据。
  3. 根据权利要求1所述的方法，其特征在于，还包括：所述控制器监控所述固态硬盘中的文件的数量是否达到预设阈值，当所述固态硬盘中的文件的数量已达到所述预设阈值时，所述控制器读取所述固态硬盘中的所述第一文件和所述第二文件。
  4. 根据权利要求1所述的方法，其特征在于，所述存储系统还包括储存级存储器，所述储存级存储器中还保存有日志文件，所述方法还包括：
    所述控制器接收主机发送的第一写数据指令，将所述第一写数据指令写入所述日志文件中，所述第一写数据指令包括所述第一数据以及所述第一数据的关键字；
    所述控制器接收主机发送的第二写数据指令，将所述第二写数据指令写入所述日志文件中，所述第二写数据指令包括所述第二数据以及所述第二数据的关键字；
    所述控制器回放所述日志文件，得到所述第一数据以及所述第一数据的关键字以及所述第二数据以及所述第二数据的关键字；
    所述控制器在所述固态硬盘中创建所述第一文件，将所述第一数据以及所述第一数据的关键字写入所述第一文件；
    所述控制器在所述固态硬盘中创建所述第二文件，将所述第二数据以及所述第二数据的关键字写入所述第二文件。
  5. 根据权利要求4所述的方法，其特征在于，所述将所述第一写数据指令写入所述日志文件中包括采用追加写的方式将所述第一写数据指令写入所述日志文件。
  6. 根据权利要求4或5所述的方法，其特征在于，所述日志文件还包括其他写数据指令，所述方法还包括：所述控制器对所述日志文件中的所有写数据指令按照每个写数据指 令中携带的关键字进行排序。
  7. 一种控制器，其特征在于，所述控制器位于存储系统中，所述存储系统包括所述控制器、固态硬盘和机械硬盘，所述控制器包括：
    通信接口，用于与所述固态硬盘通信，以及用于与所述机械硬盘通信；
    所述处理器，用于所述固态硬盘中的第一文件和第二文件，其中，所述第一文件中保存有第一数据以及所述第一数据的关键字，所述第二文件中保存有第二数据以及所述第二数据的关键字；
    判断所述第一数据的关键字与所述第二数据的关键字是否相同；
    当所述第一数据的关键字与所述第二数据的关键字相同时，在所述机械硬盘上创建第三文件；
    合并所述第一数据和所述第二数据，并将所述合并后的数据写入所述第三文件。
  8. 根据权利要求7所述的控制器，其特征在于，所述控制器还包括缓存，所述缓存中保存有至少两个不可变跳表和一个可变调表，所述第一数据来自于所述至少两个不可变调表中一个不可变跳表中的数据，所述第二数据来自于所述至少两个不可变调表中另一个不可变跳表中的数据，所述可变跳表用于接收主机发送的数据。
  9. 根据权利要求7所述的控制器，其特征在于，所述处理器还用于监控所述固态硬盘中的文件的数量是否达到预设阈值，当所述固态硬盘中的文件的数量已达到所述预设阈值时，所述控制器读取所述固态硬盘中的所述第一文件和所述第二文件。
  10. 根据权利要求7所述的控制器，其特征在于，所述存储系统还包括储存级存储器，所述储存级存储器中还保存有日志文件，
    所述处理器，还用于接收主机发送的第一写数据指令，将所述第一写数据指令写入所述日志文件中，所述第一写数据指令包括所述第一数据以及所述第一数据的关键字；
    接收主机发送的第二写数据指令，将所述第二写数据指令写入所述日志文件中，所述第二写数据指令包括所述第二数据以及所述第二数据的关键字；
    回放所述日志文件，得到所述第一数据以及所述第一数据的关键字以及所述第二数据以及所述第二数据的关键字；
    在所述固态硬盘中创建所述第一文件，将所述第一数据以及所述第一数据的关键字写入所述第一文件；
    在所述固态硬盘中创建所述第二文件，将所述第二数据以及所述第二数据的关键字写入所述第二文件。
  11. 根据权利要求10所述的控制器，其特征在于，所述将所述第一写数据指令写入所述日志文件中包括采用追加写的方式将所述第一写数据指令写入所述日志文件。
  12. 根据权利要求10或11所述的控制器，其特征在于，所述日志文件还包括其他写数据指令，所述方法还包括：所述控制器对所述日志文件中的所有写数据指令按照每个写数据指令中携带的关键字进行排序。
PCT/CN2018/071929 2017-03-22 2018-01-09 一种文件合并方法及控制器 WO2018171296A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/579,349 US11403021B2 (en) 2017-03-22 2019-09-23 File merging method and controller

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710175180.0 2017-03-22
CN201710175180.0A CN108628542B (zh) 2017-03-22 2017-03-22 一种文件合并方法及控制器

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/579,349 Continuation US11403021B2 (en) 2017-03-22 2019-09-23 File merging method and controller

Publications (1)

Publication Number Publication Date
WO2018171296A1 true WO2018171296A1 (zh) 2018-09-27

Family

ID=63586223

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/071929 WO2018171296A1 (zh) 2017-03-22 2018-01-09 一种文件合并方法及控制器

Country Status (3)

Country Link
US (1) US11403021B2 (zh)
CN (1) CN108628542B (zh)
WO (1) WO2018171296A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114691039A (zh) * 2022-03-22 2022-07-01 北京达佳互联信息技术有限公司 数据处理方法、装置、电子设备及存储介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112115521B (zh) * 2019-06-19 2023-02-07 华为技术有限公司 数据访问方法及装置
CN111352908B (zh) * 2020-02-28 2023-10-10 北京奇艺世纪科技有限公司 基于lsm的数据存储方法、装置、存储介质及计算机设备
CN113495871B (zh) * 2020-04-04 2023-06-23 厦门网宿有限公司 一种基于LSM-Tree存储引擎的文件管理方法和装置
CN112667644B (zh) * 2021-01-20 2023-02-21 浪潮云信息技术股份公司 一种混合索引内存数据库存储引擎管理方法
CN113010107B (zh) * 2021-02-26 2023-08-29 深圳忆联信息系统有限公司 数据存储管理方法、装置、计算机设备及存储介质
KR102512571B1 (ko) * 2021-12-06 2023-03-22 성균관대학교산학협력단 메모리 시스템 및 메모리 시스템의 동작 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013025864A1 (en) * 2011-08-16 2013-02-21 Pure Storage, Inc. Mapping in a storage system
CN103186350A (zh) * 2011-12-31 2013-07-03 北京快网科技有限公司 混合存储系统及热点数据块的迁移方法
CN103838681A (zh) * 2012-11-27 2014-06-04 联想(北京)有限公司 存储装置和数据文件存取方法
CN104809237A (zh) * 2015-05-12 2015-07-29 百度在线网络技术（北京）有限公司 LSM-tree索引的优化方法和装置
CN105447059A (zh) * 2014-09-29 2016-03-30 华为技术有限公司 一种数据处理方法及装置

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6912549B2 (en) * 2001-09-05 2005-06-28 Siemens Medical Solutions Health Services Corporation System for processing and consolidating records
US20080177994A1 (en) * 2003-01-12 2008-07-24 Yaron Mayer System and method for improving the efficiency, comfort, and/or reliability in Operating Systems, such as for example Windows
US20070128899A1 (en) * 2003-01-12 2007-06-07 Yaron Mayer System and method for improving the efficiency, comfort, and/or reliability in Operating Systems, such as for example Windows
US7873607B1 (en) * 2005-07-07 2011-01-18 Knutson Stanley G Model driven consolidator of database information
US8776038B2 (en) * 2008-08-07 2014-07-08 Code Systems Corporation Method and system for configuration of virtualized software applications
US8370835B2 (en) * 2009-03-12 2013-02-05 Arend Erich Dittmer Method for dynamically generating a configuration for a virtual machine with a virtual hard disk in an external storage device
US8983983B2 (en) * 2010-02-04 2015-03-17 Network State, LLC State operating system
CN103678405B (zh) * 2012-09-21 2016-12-21 阿里巴巴集团控股有限公司 邮件索引建立方法及系统、邮件搜索方法及系统
CN103593436B (zh) * 2013-11-12 2017-02-08 华为技术有限公司 文件合并方法和装置
CN103744628B (zh) * 2014-01-27 2016-09-28 北京奇虎科技有限公司 SSTable文件存储方法及装置
CN103812877B (zh) * 2014-03-12 2016-10-12 西安电子科技大学 基于Bigtable分布式存储系统的数据压缩方法
US9658924B2 (en) * 2014-12-12 2017-05-23 Schneider Electric Software, Llc Event data merge system in an event historian
CN106156070B (zh) * 2015-03-31 2019-07-12 华为技术有限公司 一种查询方法、文件合并方法与相关装置
US9324367B1 (en) * 2015-05-05 2016-04-26 Futurewei Technologies, Inc. SMR-aware append-only file system
US20160350302A1 (en) * 2015-05-27 2016-12-01 Hedvig, Inc. Dynamically splitting a range of a node in a distributed hash table
CN106407224B (zh) * 2015-07-31 2019-09-13 华为技术有限公司 一种键值存储系统中文件压实的方法和装置
US20170193041A1 (en) * 2016-01-05 2017-07-06 Sqrrl Data, Inc. Document-partitioned secondary indexes in a sorted, distributed key/value data store

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013025864A1 (en) * 2011-08-16 2013-02-21 Pure Storage, Inc. Mapping in a storage system
CN103186350A (zh) * 2011-12-31 2013-07-03 北京快网科技有限公司 混合存储系统及热点数据块的迁移方法
CN103838681A (zh) * 2012-11-27 2014-06-04 联想(北京)有限公司 存储装置和数据文件存取方法
CN105447059A (zh) * 2014-09-29 2016-03-30 华为技术有限公司 一种数据处理方法及装置
CN104809237A (zh) * 2015-05-12 2015-07-29 百度在线网络技术（北京）有限公司 LSM-tree索引的优化方法和装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114691039A (zh) * 2022-03-22 2022-07-01 北京达佳互联信息技术有限公司 数据处理方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN108628542B (zh) 2021-08-03
US11403021B2 (en) 2022-08-02
CN108628542A (zh) 2018-10-09
US20200019331A1 (en) 2020-01-16

Similar Documents

Publication Publication Date Title
WO2018171296A1 (zh) 一种文件合并方法及控制器
CN108733306B (zh) 一种文件合并方法及装置
US9996542B2 (en) Cache management in a computerized system
US9069673B2 (en) Memory system configured to perform segment cleaning and related method of operation
US9996421B2 (en) Data storage method, data storage apparatus, and storage device
CN106445405B (zh) 一种面向闪存存储的数据访问方法及其装置
US8694563B1 (en) Space recovery for thin-provisioned storage volumes
US10740187B1 (en) Systems and methods of managing and creating snapshots in a cache-based storage system
WO2017113213A1 (zh) 访问请求处理方法、装置及计算机系统
US9430492B1 (en) Efficient scavenging of data and metadata file system blocks
US11144508B2 (en) Region-integrated data deduplication implementing a multi-lifetime duplicate finder
US10664193B2 (en) Storage system for improved efficiency of parity generation and minimized processor load
WO2017113211A1 (zh) 访问请求处理方法、装置及计算机系统
CN118051179A (zh) 用于使用多个分区的分区命名空间存储的技术
WO2020057479A1 (zh) 地址映射表项页管理
US10346077B2 (en) Region-integrated data deduplication
US11662949B2 (en) Storage server, a method of operating the same storage server and a data center including the same storage server
CN108021562B (zh) 应用于分布式文件系统的存盘方法、装置及分布式文件系统
CN115878580A (zh) 一种日志管理方法及装置
CN110019092B (zh) 数据存储的方法、控制器和系统
US20240192860A1 (en) Method and device for log structured merge-tree based key-value data storage
JP2008134777A (ja) ファイル割当テーブルのキャッシュ方法
WO2020052216A1 (zh) 一种系统垃圾回收方法和固态硬盘中的垃圾回收方法
CN118245447A (zh) 容灾复制方法、装置、存储设备和系统
KR20160095963A (ko) 데이터 베이스 테이블 이름에 따른 플래시 메모리의 데이터 저장 블록 배치방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18772133

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18772133

Country of ref document: EP

Kind code of ref document: A1