WO2018153051A1 - 一种存储系统的控制装置 - Google Patents

一种存储系统的控制装置 Download PDF

Info

Publication number
WO2018153051A1
WO2018153051A1 PCT/CN2017/100422 CN2017100422W WO2018153051A1 WO 2018153051 A1 WO2018153051 A1 WO 2018153051A1 CN 2017100422 W CN2017100422 W CN 2017100422W WO 2018153051 A1 WO2018153051 A1 WO 2018153051A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
module
interface
storage
control device
Prior art date
Application number
PCT/CN2017/100422
Other languages
English (en)
French (fr)
Inventor
杨靖
何海波
杨庆
Original Assignee
深圳大普微电子科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳大普微电子科技有限公司 filed Critical 深圳大普微电子科技有限公司
Publication of WO2018153051A1 publication Critical patent/WO2018153051A1/zh
Priority to US16/440,979 priority Critical patent/US11567696B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0862—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with prefetch
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/16—Handling requests for interconnection or transfer for access to memory bus
    • G06F13/1668—Details of memory controller
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/16—Handling requests for interconnection or transfer for access to memory bus
    • G06F13/1668—Details of memory controller
    • G06F13/1694—Configuration of memory controller to different memory types
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0658—Controller construction arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/547—Remote procedure calls [RPC]; Web services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/12—Replacement control
    • G06F12/121—Replacement control using replacement algorithms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/14—Protection against unauthorised use of memory or access to memory
    • G06F12/1408—Protection against unauthorised use of memory or access to memory by using cryptography
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/10—Providing a specific technical effect
    • G06F2212/1041—Resource optimization
    • G06F2212/1044—Space efficiency improvement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/10—Providing a specific technical effect
    • G06F2212/1052—Security improvement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/60—Details of cache memory
    • G06F2212/6026—Prefetching based on access pattern detection, e.g. stride based prefetch

Definitions

  • the present invention relates to the field of computers, and in particular to a control device for a data storage system.
  • the current data storage system needs to process data with an increasing amount of data, and the efficiency requirements for data processing are also increasing.
  • the data storage system can control the access and management of data of the storage device of the entire storage system through a control device provided in a specific storage server.
  • the technical problem to be solved by the present invention is to improve the performance and efficiency of the data storage system, specifically:
  • An embodiment of the present invention provides a control device for a data storage system, including a host interface, a peer interface, a storage unit interface, a processor, and a data management module.
  • the host interface is in communication with the storage server, and is configured to implement data interaction with the storage server; the data interaction includes receiving a data management instruction of the storage server;
  • the peer interface is configured to implement a data communication connection with a storage unit of an adjacent control device in the data storage system
  • the storage unit interface is configured to connect to the storage unit
  • the data management module is configured to implement data management for the storage unit according to the data management instruction by using the processor;
  • the host interface sends result data of data management to the storage server.
  • the host interface includes one or any combination of a PCIe interface, a SAS interface, a SATA interface, a RAPID-IO interface, and an NVMe interface;
  • the storage unit interface includes one or any combination of a PCIe interface, a SAS interface, a SCSI interface, and a SATA interface;
  • the peer interface includes one or any combination of an Ethernet interface, an FC interface, an iSCSI interface, and a SAN interface.
  • the method further includes a write control unit and a write cache module;
  • the write cache module is configured to cache data to be written into the solid state hard disk
  • the write control unit includes a sorting module and a write module, and the sorting module is configured to:
  • the write module is configured to write the replacement data to a flash memory of the solid state hard disk.
  • the write control unit further includes a grouping module and a location determining module
  • the grouping module is configured to group the data to be written; the grouping rule includes: grouping according to a rewriting interval value of the to-be-written data;
  • the location determining module is configured to determine a target data block in the SSD flash memory according to the data group information of the replacement data.
  • the method further includes a prefetch unit, where the prefetch unit includes a hash table maintenance module, a data acquisition module, a first learning model, and a prefetch module.
  • the prefetch unit includes a hash table maintenance module, a data acquisition module, a first learning model, and a prefetch module.
  • a hash table maintenance module configured to preset a hash table, the hash table includes one or more hash values; the hash value is used to record and characterize data characteristics of data pages in the solid state hard disk;
  • the data feature includes access history data of the data page, a current I/O operation type and address of the data page, an access frequency of the data page, a re-reference interval of the data page, and a data page a data category, and a file name, a file type, and a file size of the file to which the data page belongs;
  • a data acquisition module configured to acquire an I/O data stream of the SSD, and obtain a hash value corresponding to the I/O data stream in the hash table
  • the first learning model for predicting a data page that is about to be accessed;
  • the first learning model is first
  • the data feature in the data feature set is an input;
  • the first data feature set is a set of data features required to obtain a first learning model prediction result; and the data feature in the first data feature set is based on the hash value obtain;
  • a prefetching module configured to perform data prefetching in the solid state hard disk according to an output result of the first learning model.
  • the security unit is further included, and the security unit includes a sensitive data acquiring module and an encryption and decryption module;
  • the sensitive data obtaining module is configured to acquire sensitive data during data interaction with the storage server; the sensitive data includes data that meets preset characteristics;
  • the encryption and decryption module includes a data security module that is adapted to the encryption and decryption module in the storage server.
  • the processor accepts the service request of the storage server for data management through a remote procedure call protocol RPC interface protocol.
  • a data storage system including: a control device of the data storage system, a storage device corresponding to the control device, and a storage server;
  • the control device includes:
  • Host interface peer interface, storage unit interface, processor and data management module
  • the host interface is in communication with the storage server, and is configured to implement data interaction with the storage server; the data interaction includes receiving a data management instruction of the storage server;
  • the peer interface is configured to implement a data communication connection with a storage unit of an adjacent control device in the data storage system
  • the storage unit interface is configured to connect to the storage unit
  • the data management module is configured to implement data management for the storage unit according to the data management instruction by using the processor;
  • the host interface sends result data of data management to the storage server.
  • the control device is connected to the storage server via network data.
  • control device includes two or more.
  • the storage device includes a SATA hard disk and/or a solid state hard disk.
  • a dedicated control device is set up for the storage device in the data storage system, and the control is controlled.
  • the device includes a processor and a local data management module. Through the connection with the storage server, the storage device can perform local data processing according to the data management instruction of the storage server; thus, the entire data storage system can calculate and process the data management.
  • the process is designed to be distributed, that is, in addition to the storage server having the processing capability of data management, the control device can also implement part of the processing function in the data management, and the data in the storage device is filtered, searched, classified, and analyzed by the control device.
  • the data transmitted to the storage server is data after local pre-processing (ie, local data management), instead of transmitting all data to the storage server for data management as in the prior art.
  • the processing so can also effectively reduce the load of network data transmission of the entire data storage system, and further improve the performance of data management of the data storage system.
  • control device in the embodiment of the present invention includes a storage unit interface, which can be adapted to different storage units, so that the compatibility of the data storage system can be effectively improved.
  • FIG. 1 is a schematic structural diagram of a data storage system according to the present application.
  • FIG. 2 is another schematic structural diagram of a control device of the data storage system according to the present application.
  • FIG. 3 is another schematic structural diagram of a control device of the data storage system according to the present application.
  • FIG. 4 is another schematic structural diagram of a control device of the data storage system in the present application.
  • FIG. 5 is still another schematic structural diagram of a control device of the data storage system in the present application.
  • the inventors have found through research that in the prior art, as the amount of data continues to increase, the performance and efficiency of the data storage system will continue to decrease. The reason is that the increasing amount of data will increase the data storage accordingly.
  • the processing load of the storage server in the system in addition, the increasing amount of data will correspondingly increase the number of storage units and the data transmission load of the entire data storage system network, thereby reducing the performance of data management of the entire data storage system. .
  • the embodiment of the present invention provides a control device 01 for a data storage system.
  • the control device 01 includes a host interface 11 and a peer interface 12 , the storage unit interface 13, the processor 14 and the local data management module 15;
  • the host interface 11 is in communication with the storage server 02 for implementing data interaction with the storage server 02; the data interaction includes receiving data management instructions of the storage server 02;
  • the peer interface 12 is configured to implement a data communication connection with a storage unit of an adjacent control device in the data storage system;
  • the storage unit interface 13 is used to connect the storage unit 03;
  • the local data management module 15 is configured to implement local data management for the storage unit 03 according to the data management instruction by the processor 14; the local data management includes one of searching, classifying, analyzing, hashing, and converting the data or a combination of any kind of operation;
  • the host interface 11 transmits the result data of the local data management to the storage server 02.
  • the core idea is to design the data management process of the entire data storage system to be implemented in two parts: a server end and a terminal. Specifically, a part of the processing process may be performed by the storage server 02 (as The server side) is completed, and another part can be completed by the control device 01 (as a terminal); for example, local data management that can be performed by the control device 01 can include searching, classifying, analyzing, hashing, and converting data, etc. That is to say, the operation and processing of these data management are not completed by the storage server 02.
  • the data storage system in the implementation of the present invention includes a storage server 02, a storage unit 03, and a control device 01; wherein the control device 01 passes through a storage unit interface (eg, may be a PCIe interface, a SAS interface, and/or a SATA interface)
  • the storage server 02 can be remotely connected to the control unit 01 through a network (the specific connection may be through a PCIe interface or an NVMe interface); the entire data storage system includes a plurality of control devices 01,
  • each control device 01 is directly connected to the storage device 03 to which it belongs, and on the other hand, it can also be connected to two adjacent control devices via two peer interfaces (for example, a SAN interface), so that each control device The device 01 can quickly obtain the required data from a plurality of storage units 03 in the entire data storage system.
  • a dedicated control device is set up for the storage device in the data storage system, and the control device includes a processor and a local data management module, and can be managed according to the data of the storage server through connection with the storage server.
  • the instruction can perform local data processing on the storage device; thus, the entire data storage system can design the data processing calculation process as distributed, that is, the control device can implement the data in addition to the storage server having the data management processing capability.
  • the control device may load corresponding to a certain data management processing, it does not make the data storage server because the arithmetic processing load caused by excessive performance degradation.
  • the data transmitted to the storage server is data after local pre-processing (ie, local data management), instead of transmitting all data to the storage server for data management as in the prior art.
  • the processing that is to say, can reduce the amount of data transmitted through the network, so it can effectively reduce the load of network data transmission of the entire data storage system, and further improve the performance of data management of the data storage system.
  • control device in the embodiment of the present invention includes a storage unit interface, which can be adapted to different storage units, so that the compatibility of the data storage system can be effectively improved.
  • a write control unit 16 and a write cache module 17 may be further included;
  • the write cache module 16 is configured to cache data to be written into the solid state hard disk;
  • the write control unit 17 includes a sorting module (not shown), a replacement unit (not shown), and a write module (not shown in the figure)
  • the sorting module is configured to sort the data to be written in the write cache module 17 according to the least recently used page LRU algorithm; when the data storage amount in the write cache module 17 reaches a preset value, according to a preset cache replacement algorithm A preset number of replacement data is determined in the unusable data to be written; the write module is used to write the replacement data to the flash memory 04 of the solid state hard disk.
  • a write cache module 17 for buffering write data is pre-configured, that is, data written to the solid state hard disk (ie, data to be written) is not directly written into the flash memory 04 of the solid state hard disk, but The write cache module 16 is written first.
  • the sorting unit sorts the generation write data of the write cache module 16 according to the LRU algorithm, so that the data that is not commonly used in the write cache can be precipitated and distinguished.
  • the inventive idea of the embodiment of the present invention is that, in general, the frequency of use of active data pages (that is, the frequency of rewriting) tends to be high, so the data is kept in accordance with the LRU algorithm after being written to the write cache module.
  • Pages, which are data pages to which data to be written is relatively inactive. Since the LRU algorithm is based on whether data is frequently used, the relatively inactive data pages to be written in the data can also be called no. Commonly used data to be written.
  • the capacity of the write cache module 16 is limited, and the data to be written cannot be stored in an unlimited manner. Therefore, by replacing the replacement mechanism set by the unit, it is determined which data needs to be written to the flash memory; The less active data pages are transferred from the write cache module 16 to the flash memory 04, which makes room for subsequent data writes; in addition, only the transfer from the write cache module 16 to the flash memory 04 is the completion of the data in the solid state. The final write of the hard disk.
  • a preset value may be set according to actual needs, and the preset value may be a specific value, or may be a ratio that occupies the overall capacity of the write buffer module, and is not specifically limited herein; Taking the preset value of 80% as an example, when the data occupancy in the write cache module reaches the write cache When 80% of the volume is used, the data replacement is started according to the cache replacement algorithm, so that the inactive data pages in the write cache module can be cleared out of the write cache module, that is, which data pages in the write cache module belong to the replacement data; It should be noted that the amount of data cleared each time may also be specifically set by a person skilled in the art according to actual needs.
  • the data can be written to the flash memory 04 of the solid state hard disk by the writing unit. Since these replacement data are less active, these data pages are written after being written into the flash memory 04. The possibility of being rewritten will also be lower, which will reduce the number of rewrites of the flash memory 04 as a whole, thereby reducing the write amplification problem of the solid state hard disk during data access.
  • the data to be written is first written into a preset write cache, and the data to be written successively acquired is sorted by LRU; thus, the unused write is to be written.
  • the data is deposited to the bottom of the sorting sequence; thus, it is possible to select infrequent write data from the write cache and write it to the flash memory of the SSD; the possibility of frequent rewriting of these unusable data to be written is subsequently followed. It is relatively small, so it can effectively reduce the number of rewrites of the flash memory in the SSD, which in turn can effectively reduce the write operation to the flash memory and reduce the write amplification problem of the SSD in the data access process.
  • the write control device 16 in the embodiment of the present invention may further include:
  • a grouping unit for grouping the data to be written into the preset write cache module 17 after the obtained data to be written is cached;
  • the grouping rule includes, according to the weight of the data to be written Writing interval values for grouping; recording unit (not shown) for recording page information of data to be written; page information including group information of data to be written; position determining unit (not shown), For determining the target data block of 04 in the solid state hard disk flash memory based on the data group information of the replacement data before writing the replacement data to the flash memory 04 of the solid state hard disk.
  • the group flag is 127 data page 128K, and the group flag is 002 data page 640K; at this time, the storage mode is, the group flag is The data pages of 001 are respectively stored in two data blocks, and the data pages whose group flags are 002 are respectively stored in 10 data blocks, so that the data pages in each data block belong to the same group, that is, both have The same or approximate rewrite interval.
  • the data to be written is also grouped according to the rewriting interval of the data to be written, and the data synchronized by the rewriting interval is written into the same data block in the flash memory.
  • the amount of all data pages in each data block in the flash has the same rewrite frequency, thereby reducing the occurrence of both the valid page and the invalid page in the data block; due to the process of performing block deletion, when the data is When the pages in the block are all invalid pages, there is no need to perform an additional write operation, that is, there is no need to increase the write operation by dumping the valid pages in the data block, so the write enlargement problem can be further reduced.
  • the prefetching unit 18 is further included, and the prefetching unit 18 includes a hash table maintenance module (not shown), a data acquisition module (not shown), and a first learning model. (not shown in the figure) and prefetch module (not shown), where:
  • the hash table maintenance module 81 is configured to preset a hash table, the hash table includes more than one hash value; the hash value is used to record and characterize data features of the data page in the solid state hard disk; and the data feature includes access to the data page Historical data, the current I/O operation type and address of the data page, the access frequency of the data page, the re-reference interval of the data page, and the data type of the data page, and the file name, file type, and file of the file to which the data page belongs.
  • the data acquisition module is configured to obtain an I/O data stream of the solid state hard disk, and obtain a hash value corresponding to the I/O data stream in the hash table; the first learning model is used to predict the data page to be accessed and/or Or a data page sequence; the first learning model takes as input the data features in the first data feature set; the first data feature set is a set of data features required to obtain the first learning model prediction result; the first data feature set The data feature is obtained according to the hash value; the prefetch module is configured to perform data prefetching in the solid state hard disk according to the output result of the first learning model.
  • the hash table maintenance module can record the data characteristics of each data page in the solid state hard disk by maintaining the hash table; when the data acquisition unit obtains the real-time data stream, it can analyze the data page that is predicted to be accessed by the first learning model. Therefore, the prefetch unit can perform prefetching according to the prediction result of the first learning model. Since the pre-fetched data is predicted by the embodiment of the present invention, learning can be performed in real time to adapt to different application types and access modes, so that better data prefetching performance can be obtained.
  • a second learning model (not shown in the figure) and a determining unit (not shown) may be further included, specifically:
  • a second learning model for grouping the data to be written by grouping the data to be written into different groups according to whether the rewriting time between the data to be written is similar;
  • the second learning model is second The data feature in the data feature set is an input;
  • the second data feature set is a set of data features required to obtain the second learning model prediction result;
  • the data feature in the second data feature set is obtained according to the hash value;
  • the data characteristics of each data page recorded in the solid state hard disk recorded by the hash table maintenance unit are utilized; when the data acquisition unit acquires the real-time data stream, the data to be written can be grouped by the second learning model, thereby The data block of only one group is included in a single data block in the solid state hard disk, and since the data page rewriting time of the same group is similar, the situation that the valid data page and the invalid data page are included in the single data block can be effectively reduced. Occurs, and thus can effectively reduce the write amplification problem of the solid state drive.
  • the security unit 19 is further included, and the security unit 19 includes a sensitive data acquisition module (not shown) and an encryption and decryption module (not shown);
  • the sensitive data acquisition module is configured to acquire sensitive data during data interaction with the storage server; the sensitive data includes data conforming to preset features; and the encryption and decryption module includes a data security module adapted to the encryption and decryption module included at one end of the storage server. .
  • the processor in the embodiment of the present invention accepts a service request of the storage server for data management through a remote procedure call protocol RPC interface protocol.
  • the control device may further include a security unit 19; in the embodiment of the present invention, the front end of the storage device storing the data is further provided with a control device, so The unit is used to improve the security of the data to avoid illegal intrusion of data during the transmission of the network; in practical applications, it is possible to protect only specific data by judging whether the data stream includes sensitive data. Therefore, it is not necessary to perform encryption and decryption operations on all the data, so that the influence of the encryption and decryption process on the data transmission performance can be effectively reduced.
  • the security unit 19 includes a sensitive data acquisition module and an encryption and decryption module; the security unit included at one end of the corresponding storage server also has an adapted module for identifying sensitive data and an encryption and decryption module; thus, the control device
  • the storage server can perform the corresponding encryption and decryption operations on the data of the sensitive data before and after the first transmission, so as to minimize the impact on the data transmission and improve the security of the data.
  • a data storage system is further provided.
  • the data storage system includes a control device 01, a storage device 03 corresponding to the control device, and a storage device.
  • the control device 01 includes a host interface 11, a peer interface 12, a storage unit interface 13, a processor 14 and a data management module 15;
  • the host interface 11 is in communication with the storage server 02 for data interaction with the storage server 02; the data interaction includes receiving data management instructions of the storage server 02; the peer interface 12 is for implementing storage with adjacent control devices in the data storage system.
  • the control device 01 is connected to the storage server 02 via network data.
  • the data storage system in the embodiment of the present invention is mainly configured to have a control device at the front end of the storage unit to implement a preset data management budget and processing function, so that the entire data storage system can calculate the data processing process.
  • the control device can also implement partial processing functions in the data management, and the data in the storage device can be filtered, searched, classified, analyzed, and Operation such as hash calculation and data conversion, so that the storage server can receive the data as the data preprocessed by the control device; by the embodiment of the invention, all the data need not be transmitted to the storage server for data management processing, so
  • the operation load of the storage service can be effectively alleviated; even if the storage device of the data storage system is of a large magnitude, since the control device corresponding to the storage device can bear a certain data management processing work, the data storage server is not made. Because of the processing load Large cause performance degradation.
  • control device may include two or more; thus, a data storage system may be composed of a plurality of controllers distributedly controlled by a storage server, and then Greatly improve the overall capacity of the storage system.
  • the storage device in the embodiment of the present invention specifically includes a SATA hard disk and a solid state hard disk, that is, a storage unit interface included in the control device can be adapted to different storage units, so that the data storage system can be effectively improved. compatibility.
  • the disclosed system and device And methods can be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • the technical solution of the present invention which is essential or contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product stored in a storage medium.
  • a number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.
  • the foregoing storage medium includes: a USB flash drive, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a ReRAM, an MRAM, a PCM, a NAND Flash, a NOR Flash, and a Memristor.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • ReRAM ReRAM
  • MRAM Magnetoresistive RAM
  • PCM Peripheral Component Interconnect Memory
  • NAND Flash a Flash
  • NOR Flash NOR Flash
  • Memristor a variety of media that can store program code, such as a disk or a disc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种数据存储系统的控制装置（01），包括主机接口（11）、同级接口（12）、存储单元接口（13）、处理器（14）和本地数据管理模块（15）；主机接口（11）与存储服务器（02）通讯连接，用于实现与存储服务器（02）数据交互；同级接口（12）用于实现与数据存储系统中相邻控制装置的存储单元的数据通信连接；存储单元接口（13）用于连接存储单元（03）；本地数据管理模块（15）用于通过处理器（14）根据数据管理指令实现对于存储单元（03）中数据的本地数据管理；主机接口（11）向存储服务器（02）发送本地数据管理的结果数据。所述控制装置可以有效地减轻存储服务的负荷；即使数据存储系统的存储装置的量级很大，由于该存储装置所对应的控制装置可以负担一定的数据管理的处理工作，所以不会使数据存储服务器因为负荷过大造成性能下降。

Description

一种存储系统的控制装置
本申请要求于2017年02月21日提交中国专利局，申请号为201710093959.8、发明名称为“一种存储系统的控制装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及计算机领域，特别是涉及一种数据存储系统的控制装置。
背景技术
随着互联网的普及，以及涉及各个领域的物联网和大数据运算等科技技术的快速发展，数据的产生也在呈爆发式的增长；从而使得现有技术中数据存储系统的性能和效率越来越无法满足当前的需求。
具体来说，当前的数据存储系统需要处理数据的数据量越来越大，而且，对于数据处理的效率要求也越来高。现有技术中，数据存储系统可以通过在一个特定的存储服务器中设有的控制装置，来对整个存储系统的存储设备进行数据的存取和管理的控制。
发明人经过研究发现，现有技术中，至少还存在以下缺陷：
随着数据量的不断增长，数据存储系统的性能和效率会不断地下降。
发明内容
本发明所要解决的技术问题是提高数据存储系统的性能和效率，具体的：
本发明实施例提供了一种数据存储系统的控制装置，包括主机接口、同级接口、存储单元接口、处理器和数据管理模块；
所述主机接口与存储服务器通讯连接，用于实现与所述存储服务器数据交互；所述数据交互包括接收所述存储服务器的数据管理指令；
所述同级接口用于实现与所述数据存储系统中相邻控制装置的存储单元的数据通信连接；
所述存储单元接口用于连接存储单元；
所述数据管理模块用于通过所述处理器根据所述数据管理指令实现对于所述存储单元的数据管理；
所述主机接口向所述存储服务器发送数据管理的结果数据。
优选的，在本发明实施例中，所述主机接口包括PCIe接口、SAS接口、SATA接口、RAPID－IO接口和NVMe接口中的一种或任意组合；
所述存储单元接口包括PCIe接口、SAS接口、SCSI接口和SATA接口中的一种或任意组合；
所述同级接口包括Ethernet接口、FC接口、iSCSI接口和SAN接口中的一种或任意组合。
优选的，在本发明实施例中，还包括写控制单元和写缓存模块；
所述写缓存模块用于缓存待写入固态硬盘的数据；
所述写控制单元包括排序模块和写入模块，所述排序模块用于：
根据最近最少使用页面LRU算法对所述写缓存模块中的所述待写入数据进行排序；当所述写缓存模块中的数据存储量达到预设值时，根据预设的缓存替换算法在不常用的待写入数据中确定预设数量的替换数据；
所述写入模块用于将所述的替换数据写入所述固态硬盘的闪存。
优选的，在本发明实施例中，所述写控制单元还包括分组模块和位置确定模块；
所述分组模块用于对所述待写入数据进行分组；分组规则包括，根据所述待写入数据的重写间隔值进行分组；
所述位置确定模块用于根据所述替换数据的数据组别信息确定所述固态硬盘闪存中的目标数据块。
优选的，在本发明实施例中，还包括预取单元，所述预取单元包括哈希表维护模块、数据获取模块、第一学习模型和预取模块；
哈希表维护模块，用于预设哈希表，所述哈希表包括一个以上的哈希值；所述哈希值用于记录和表征所述固态硬盘中数据页的数据特征；所述数据特征包括所述数据页的访问历史数据、所述数据页的当前I/O操作类型和地址、所述数据页的访问频率、所述数据页的重新引用间隔，和，所述数据页的数据类别，以及，所述数据页所属文件的文件名、文件类型和文件大小；
数据获取模块，用于获取所述固态硬盘的I/O数据流，在所述哈希表中获取与所述I/O数据流对应的哈希值；
第一学习模型，用于预测即将发生访问的数据页；所述第一学习模型以第一 数据特征集中的数据特征为输入；所述第一数据特征集为用于获取第一学习模型预测结果所需的数据特征的集合；所述第一数据特征集中的数据特征根据所述哈希值获得；
预取模块，用于根据所述第一学习模型的输出结果在所述固态硬盘中进行数据预取。
优选的，在本发明实施例中，还包安全单元，所述安全单元包括敏感数据获取模块和加解密模块；
所述敏感数据获取模块用于获取与所述存储服务器进行数据交互过程中的敏感数据；所述敏感数据包括符合预设特征的数据；
所述加解密模块包括与所述存储服务器中的加解密模块适配的数据安全模块。
优选的，在本发明实施例中，所述处理器通过远程过程调用协议RPC接口协议接受所述存储服务器用于数据管理的服务请求。
在本发明实施例的另一面，还提供了一种数据存储系统，包括数据存储系统的控制装置，与所述控制装置对应的存储装置，以及，存储服务器；
控制装置包括：
主机接口、同级接口、存储单元接口、处理器和数据管理模块；
所述主机接口与存储服务器通讯连接，用于实现与所述存储服务器数据交互；所述数据交互包括接收所述存储服务器的数据管理指令；
所述同级接口用于实现与所述数据存储系统中相邻控制装置的存储单元的数据通信连接；
所述存储单元接口用于连接存储单元；
所述数据管理模块用于通过所述处理器根据所述数据管理指令实现对于所述存储单元的数据管理；
所述主机接口向所述存储服务器发送数据管理的结果数据。
所述控制装置与所述存储服务器通过网络数据连接。
优选的，在本发明实施例中，所述控制装置包括有两个及两个以上。
优选的，在本发明实施例中，述存储装置包括SATA硬盘和/或固态硬盘。
本发明实施例中，为数据存储系统中的存储装置设立专用的控制装置，控制 装置包括有处理器和本地数据管理模块，通过与存储服务器的连接，可以根据存储服务器的数据管理指令可以对存储装置进行本地的数据处理；这样，可以将整个数据存储系统将数据管理的计算处理过程设计为分布式，即，除了存储服务器具有数据管理的处理能力以外，控制装置也可以实现数据管理中的部分处理功能，通过控制装置来对存储装置中的数据进行筛选、查找、分类、分析、哈希计算和数据转换等操作，这样可以使存储服务器接收到的数据为经过控制装置预处理后的数据；通过本发明实施例，无需将所有的数据均传输至存储服务器进行数据管理的处理，所以，可以有效地减轻存储服务的负荷；即使数据存储系统的存储装置的量级很大，由于该存储装置所对应的控制装置可以负担一定的数据管理的处理工作，所以不会使数据存储服务器因为负荷过大造成性能下降。
此外，本发明实施例，向存储服务器传输的数据为经过本地的预处理(即，本地数据管理)后的数据，而不是像现有技术中那样将所有的数据均传输至存储服务器进行数据管理的处理，所以还可以有效的降低整个数据存储系统的网络数据传输的负荷，进一步的提高数据存储系统的数据管理的性能。
此外，本发明实施例中的控制装置包括有存储单元接口，可以适配不同的存储单元，所以可以有效地提高数据存储系统的兼容性。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请中记载的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请中所述数据存储系统的结构示意图；
图2为本申请中所述数据存储系统的控制装置的又一结构示意图；
图3为本申请中所述数据存储系统的控制装置的又一结构示意图；
图4为本申请中所述数据存储系统的控制装置的又一结构示意图；
图5为本申请中所述数据存储系统的控制装置的又一结构示意图。
具体实施方式
为了使本技术领域的人员更好地理解本发明方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述 的实施例仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
发明人经过研究发现，现有技术中，随着数据量的不断增长，数据存储系统的性能和效率会不断地下降，究其原因，是因为数据量的不断增大也会相应的增加数据存储系统中存储服务器的处理负荷；此外，数据量的不断增大也会相应的增加存储单元的数量和整个数据存储系统网络的数据传输负荷，从而也会降低整个数据存储系统的数据管理的执行效能。
基于上述认知，为了提高数据存储系统的性能和效率，本发明实施例提供了一种数据存储系统的控制装置01，参考图1和图2，控制装置01包括主机接口11、同级接口12、存储单元接口13、处理器14和本地数据管理模块15；
主机接口11与存储服务器02通讯连接，用于实现与存储服务器02数据交互；数据交互包括接收存储服务器02的数据管理指令；
同级接口12用于实现与数据存储系统中相邻控制装置的存储单元的数据通信连接；
存储单元接口13用于连接存储单元03；
本地数据管理模块15用于通过处理器14根据数据管理指令实现对于存储单元03的本地数据管理；本地数据管理包括对于数据的查找、分类、分析、哈希计算和数据转换中的一种操作或任意种操作的组合；
主机接口11向存储服务器02发送本地数据管理的结果数据。
在本发明实施例中，其核心思想是将整个数据存储系统的数据管理的处理过程分布式设计为有服务器端和终端两部分来实现，具体来说，一部分处理过程可以由存储服务器02(作为服务器端)来完成，另一部分可以由控制装置01(作为终端)来完成；比如，可以由控制装置01完成的本地数据管理可以包括对于数据的查找、分类、分析、哈希计算和数据转换等；也就是说，这些数据管理的运算和处理不是有存储服务器02完成的。
需要说明的是，本发明实施例中所提及的本地数据管理的几种数据处理方式(查找、分类、分析、哈希计算和数据转换)的只是控制装置01进行数据管理处理所涉及的具体应用的典型举例，而非限定，本领域技术人员可以根据实际的 需要对本地数据管理模块15进行相应的处理功能的设计，这些设计并不超出本发明实施例的保护范围。
在实际应用中，本发明实施中的数据存储系统包括存储服务器02、存储单元03和控制装置01；其中控制装置01通过存储单元接口(如，可以是PCIe接口、SAS接口和/或SATA接口)与存储单元02近场连接，存储服务器02可以与控制装置01通过网络远程的连接(具体的连接方式可以是通过PCIe接口或NVMe接口连接)；整个数据存储系统中包括有多个控制装置01，每个控制装置01一方面与其所属的存储装置03直接连接，另一方面还可以通过两个同级接口(如，可以是SAN接口)分别与相邻的两个控制装置连接，这样每个控制装置01就可以从整数据存储系统中的多个存储单元03中快捷的获取所需的数据。
由上可知，本发明实施例中，为数据存储系统中的存储装置设立专用的控制装置，控制装置包括有处理器和本地数据管理模块，通过与存储服务器的连接，可以根据存储服务器的数据管理指令可以对存储装置进行本地的数据处理；这样，可以将整个数据存储系统将数据管理的计算处理过程设计为分布式，即，除了存储服务器具有数据管理的处理能力以外，控制装置也可以实现数据管理中的部分处理功能，通过控制装置来对存储装置中的数据进行筛选、查找、分类、分析、哈希计算和数据转换等操作，这样可以使存储服务器接收到数据为经过控制装置预处理后的数据；通过本发明实施例，无需将所有的数据均传输至存储服务器进行数据管理的处理，所以，可以有效地减轻存储服务的运算负荷；即使数据存储系统的存储装置的量级很大，由于该存储装置所对应的控制装置可以负担一定的数据管理的处理工作，所以不会使数据存储服务器因为运算处理负荷过大造成性能下降。
此外，本发明实施例，向存储服务器传输的数据为经过本地的预处理(即，本地数据管理)后的数据，而不是像现有技术中那样将所有的数据均传输至存储服务器进行数据管理的处理，也就是说可以减少通过网络进行数据传输的数据量，所以还可以有效的降低整个数据存储系统的网络数据传输的负荷，进一步的提高数据存储系统的数据管理的性能。
此外，本发明实施例中的控制装置包括有存储单元接口，可以适配不同的存储单元，所以可以有效地提高数据存储系统的兼容性。
优选的，在本发明实施例中，如图3所示，还可以包括有写控制单元16和写缓存模块17；
写缓存模块16用于缓存待写入固态硬盘的数据；写控制单17元包括排序模块(图中未示出)、替换单元(图中未示出)和写入模块(图中未示出)；排序模块用于根据最近最少使用页面LRU算法对写缓存模块17中的待写入数据进行排序；当写缓存模块17中的数据存储量达到预设值时，根据预设的缓存替换算法在不常用的待写入数据中确定预设数量的替换数据；写入模块用于将的替换数据写入固态硬盘的闪存04。
在本发明实施例中，预设有用于缓存写入数据的写缓存模块17，即，写入固态硬盘的数据(即，待写入数据)不直接写入固态硬盘的闪存04中，而是先写入写缓存模块16。
排序单元将写缓存模块16的代写入数据根据LRU算法进行排序，从而可以将写缓存中不常用的数据进行沉淀区分。
本发明实施例的发明思路为，一般情况下，活跃数据页的使用频率(即，重写频率)往往会较高，所以这些数据在写入写缓存模块后就会一直保持在根据LRU算法进行排序的序列的靠前的位置；在该序列的靠后的位置一般都是重写频率较低的数据页，所以通过写缓存模块并进行排序，可以区分出哪些待写入数据中是活跃数据页，哪些是待写入数据相对不活跃的数据页，由于LRU算法是以数据是否经常使用为依据进行的排序，所以待写入数据中的这些相对不活跃的数据页也可以称之为不常用的待写入数据。
写缓存模块16的容量是有限的，不能无限制的存储待写入数据，因此通过替换单元所设置的替换机制，来确定哪些数据是需要写入闪存的；从而可以后需通过写入单元将活跃程度较低的数据页从写缓存模块16中转移到闪存04中，这样可以为后续的数据写入腾出空间；此外，只有从写缓存模块16中转入闪存04才是完成数据在固态硬盘的最终写入。
首先，本领域技术人员可以根据实际需要设定一个预设值，该预设值可以是一个具体的数值，也可以是一个占用写缓存模块整体容量的比例值，在此并不作具体的限定；以预设值是80％为例，当写缓存模块中的数据占用量达到写缓存整 体容量的80％时，开始根据缓存替换算法进行数据的替换，从而可以将写缓存模块中不活跃的数据页清出写缓存模块，即，确定写缓存模块中的数据页哪些属于替换数据；需要说明的是，每次清出的数据量也可以是本领域技术人员根据实际需要具体设定的。
在确定了替换数据以后，就可以通过写入单元正式的将数据先写入固态硬盘的闪存04了，由于这些替换数据的活跃程度较低，所以在将其写进闪存04后，这些数据页被重写的可能性也会较低，这样也就从整体上降低了对闪存04的重写次数，进而也就减少固态硬盘在数据存取过程中的写放大问题。
综上所述，本发明实施例中，首先将待写入数据写入一个预设的写缓存中，并将陆续获取的待写入数据进行LRU排序；这样，通过将不常用的待写入数据沉淀至排序序列的底部；从而可以从写缓存中选择出不常用的代写入数据后在写入固态硬盘的闪存中；由于这些不常用的待写入数据后续被频繁重写的可能性比较小，所以就可以有效的减少对固态硬盘中闪存的重写次数，进而也就可以有效的降低对闪存的写操作，减少固态硬盘在数据存取过程中的写放大问题。
优选的，在本发明实施例中的写控制装置16还可以包括：
分组单元(图中未示出)，用于在将获取到的待写入数据缓存至预设的写缓存模块17后，对待写入数据进行分组；分组规则包括，根据待写入数据的重写间隔值进行分组；记录单元(图中未示出)，用于记录待写入数据的页面信息；页面信息包括待写入数据的组别信息；位置确定单元(图中未示出)，用于在将替换数据写入所述固态硬盘的闪存04前，根据替换数据的数据组别信息确定固态硬盘闪存中04的目标数据块。
举例说明，以闪存中每个数据块的大小为64K为例，组别标志为001的数据页128K，组别标志为002的数据页640K；此时，存储的方式为，将组别标志为001的数据页分别存入两个数据块中，将组别标志为002的数据页分别存入10个数据块中，这样，每个数据块中的数据页都属于同一组，即，都具有相同或是近似的重写间隔。
由上可以看出，在本发明实施例中，还根据待写入数据的重写间隔对待写入数据进行了分组，通过将重写间隔同步的数据写入闪存中同一数据块的方式，尽 量使闪存中的每个数据块中的所有数据页都具有相同的重写频率，从而减少了数据块中同时包括有效页和无效页的情况的发生；由于进行块删除的过程中，当数据块中的页面均为无效页时，就无需进行额外的写操作，即，无需因为转存数据块中的有效页而增加写操作，所以可以进一步的减少写放大问题。
优选的，在本发明实施例中，还包括预取单元18，预取单元18包括哈希表维护模块(图中未示出)、数据获取模块(图中未示出)、第一学习模型(图中未示出)和预取模块(图中未示出)，其中：
哈希表维护模块81，用于预设哈希表，哈希表包括一个以上的哈希值；哈希值用于记录和表征固态硬盘中数据页的数据特征；数据特征包括数据页的访问历史数据、数据页的当前I/O操作类型和地址、数据页的访问频率、数据页的重新引用间隔，和，数据页的数据类别，以及，数据页所属文件的文件名、文件类型和文件大小；数据获取模块用于获取固态硬盘的I/O数据流，在哈希表中获取与I/O数据流对应的哈希值；第一学习模型用于预测即将发生访问的数据页和/或数据页序列；第一学习模型以第一数据特征集中的数据特征为输入；第一数据特征集为用于获取第一学习模型预测结果所需的数据特征的集合；第一数据特征集中的数据特征根据哈希值获得；预取模块用于根据第一学习模型的输出结果在固态硬盘中进行数据预取。
哈希表维护模块通过维护哈希表可以记录固态硬盘中各个数据页的数据特征；当数据获取单元获取到实时的数据流时，就可以分析通过第一学习模型预测出即将发生访问的数据页，从而预取单元可以根据第一学习模型的预测结果进行预取。由于通过本发明实施例，在预测预取的数据时，可以实时的进行学习以适应不同的应用种类和访问模式，从而可以能够获得更好的数据预取效能。
进一步的，在本发明实施例中，还可以包括有第二学习模型(图中未示出)和确定单元(图中未示出)，具体的：
第二学习模型，用于对待写入数据的进行分组，分组的方式为根据待写入数据间的重写时间是否相近，将待写入数据划分为不同组别；第二学习模型以第二数据特征集中的数据特征为输入；第二数据特征集为用于获取第二学习模型预测结果所需的数据特征的集合；第二数据特征集中的数据特征根据哈希值获得；确 定单元，用于根据待写入数据的组别信息，为待写入数据确定在固态硬盘中的目标数据块。
同样的，利用哈希表维护单元所记录的记录固态硬盘中各个数据页的数据特征；当数据获取单元获取到实时的数据流时，通过第二学习模型可以待写入数据的进行分组，从而使固态硬盘中单个数据块中只包括一个组别的数据页，由于相同组别的数据页重写时间相近，因此可以有效的减少单个数据块中同时包括有效数据页和无效数据页的情况的发生，进而也就可以有效的降低固态硬盘的写放大问题。
优选的，在本发明实施例中，还包安全单元19，安全单元19包括敏感数据获取模块(图中未示出)和加解密模块(图中未示出)；
敏感数据获取模块用于获取与存储服务器进行数据交互过程中的敏感数据；敏感数据包括符合预设特征的数据；加解密模块包括与存储服务器的一端所包括的加解密模块适配的数据安全模块。
在实际应用中，本发明实施例中的处理器通过远程过程调用协议RPC接口协议接受存储服务器用于数据管理的服务请求。
在本发明实施例中，为了提高数据的安全性，控制装置还可以包括有安全单元19；由于本发明实施例中，存放数据的存储设备的前端还设置有控制装置，所以可以通过设有安全单元来提高数据的安全性，以避免数据在网络的传输过程中受到非法的入侵；在实际应用中，可以是通过判断数据流中是否包括有敏感数据的方式来只对特定的数据进行保护，从而可以无需对所有的数据进行加解密的操作，从而可以有效的减少加解密过程对于数据传输效能的影响。具体的，安全单元19包括有敏感数据获取模块和加解密模块；相应的存储服务器的一端所包括的安全单元也会有适配的用于敏感数据识别的模块和加解密模块；这样，控制装置和存储服务器可以将共识为敏感数据的数据在首发前后进行相应的加解密操作，从而来尽量减少对于数据传输影响的情况下提高数据的安全性。
在本发明实施例的另一面，还提供了一种数据存储系统，参考图1至图5，数据存储系统包括控制装置01，与控制装置对应的存储装置03，以及，存储服 务器02；
控制装置01包括主机接口11、同级接口12、存储单元接口13、处理器14和数据管理模块15；
主机接口11与存储服务器02通讯连接，用于实现与存储服务器02数据交互；数据交互包括接收存储服务器02的数据管理指令；同级接口12用于实现与数据存储系统中相邻控制装置的存储单元02的数据通信连接；存储单元接口13用于连接存储单元03；数据管理模块15用于通过处理器14根据数据管理指令实现对于存储单元03的数据管理；主机接口11向存储服务器02发送数据管理的结果数据。控制装置01与存储服务器02通过网络数据连接。
本发明实施例中的数据存储系统构成，主要是在存储单元的前端设置有一个控制装置来实现预设的数据管理的预算和处理功能，从而可以将整个数据存储系统将数据管理的计算处理过程设计为分布式，即，除了存储服务器具有数据管理的处理能力以外，控制装置也可以实现数据管理中的部分处理功能，通过控制装置来对存储装置中的数据进行筛选、查找、分类、分析、哈希计算和数据转换等操作，这样可以使存储服务器接收到数据为经过控制装置预处理后的数据；通过本发明实施例，无需将所有的数据均传输至存储服务器进行数据管理的处理，所以，可以有效地减轻存储服务的运算负荷；即使数据存储系统的存储装置的量级很大，由于该存储装置所对应的控制装置可以负担一定的数据管理的处理工作，所以不会使数据存储服务器因为运算处理负荷过大造成性能下降。
本发明实施例中的技术方案、工作原理和能达到的有益效果在图1所对应的数据存储系统的控制装置的实施例中已经进行了记载和描述，在此就不再赘述。
为了能够适应大容量的存储需求，在实际应用中，控制装置可以包括有两个及两个以上；这样，一个数据存储系统可以通过一个存储服务器来分布式的控制多个控制器来组成，进而大大的提高存储系统的整体容量。
在实际应用中，本发明实施例中的存储装置具体包括SATA硬盘和固态硬盘，即，通过控制装置所包括的存储单元接口，可以适配不同的存储单元，所以可以有效地提高数据存储系统的兼容性。
在本发明实施例所提供的几个实施例中，应该理解到，所揭露的系统，装置 和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、ReRAM、MRAM、PCM、NAND Flash，NOR Flash，Memristor、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围。

Claims (10)

  1. 一种数据存储系统的控制装置，其特征在于，包括主机接口、同级接口、存储单元接口、处理器和本地数据管理模块；
    所述主机接口与存储服务器通讯连接，用于实现与所述存储服务器数据交互；所述数据交互包括接收所述存储服务器的数据管理指令；
    所述同级接口用于实现与所述数据存储系统中相邻控制装置的存储单元的数据通信连接；
    所述存储单元接口用于连接存储单元；
    所述本地数据管理模块用于通过所述处理器根据所述数据管理指令实现对于所述存储单元中数据的本地数据管理；所述本地数据管理包括对于数据的查找、分类、分析、哈希计算和数据转换中的一种操作或任意种操作的组合；
    所述主机接口向所述存储服务器发送本地数据管理的结果数据。
  2. 根据权利要求1所述的数据存储系统的控制装置，其特征在于，
    所述主机接口包括PCIe接口、SAS接口、SATA接口、RAPID－IO接口和NVMe接口中的一种或任意组合；
    所述存储单元接口包括PCIe接口、SAS接口、SCSI接口和SATA接口中的一种或任意组合；
    所述同级接口包括Ethernet接口、FC接口、iSCSI接口和SAN接口中的一种或任意组合。
  3. 根据权利要求1所述的数据存储系统的控制装置，其特征在于，还包括写控制单元和写缓存模块；
    所述写缓存模块用于缓存待写入固态硬盘的数据；
    所述写控制单元包括排序模块和写入模块，所述排序模块用于：
    根据最近最少使用页面LRU和/或LFU算法对所述写缓存模块中的所述待写入数据进行排序；当所述写缓存模块中的数据存储量达到预设值时，根据预设的缓存替换算法在不常用的待写入数据中确定预设数量的替换数据；
    所述写入模块用于将所述的替换数据写入所述固态硬盘的闪存。
  4. 根据权利要求3所述的数据存储系统的控制装置，其特征在于，所述写控制单元还包括分组模块和位置确定模块；
    所述分组模块用于对所述待写入数据进行分组；分组规则包括，根据所述待写入数据的重写间隔值进行分组；
    所述位置确定模块用于根据所述替换数据的数据组别信息确定所述固态硬盘闪存中的目标数据块。
  5. 根据权利要求1所述的数据存储系统的控制装置，其特征在于，还包括预取单元，所述预取单元包括哈希表维护模块、数据获取模块、第一学习模型和预取模块；
    哈希表维护模块，用于预设哈希表，所述哈希表包括一个以上的哈希值；所述哈希值用于记录和表征所述固态硬盘中数据页的数据特征；所述数据特征包括所述数据页的访问历史数据、所述数据页的当前I/O操作类型和地址、所述数据页的访问频率、所述数据页的重新引用间隔，和，所述数据页的数据类别，以及，所述数据页所属文件的文件名、文件类型和文件大小，若数据是图像或视频数据，其特征值包括其图像特征例如人物、风景、机动车、车牌号、人脸、建筑等等；
    在I/O读操作时，本发明在传统的基于I/O地址寻找数据的基础上，开创了基于数据特征值来寻找/查找数据的存取方法。
    数据获取模块，用于获取所述固态硬盘的I/O数据流，在所述哈希表中获取与所述I/O数据流对应的哈希值；
    第一学习模型，用于预测即将发生访问的数据页；所述第一学习模型以第一数据特征集中的数据特征为输入；所述第一数据特征集为用于获取第一学习模型预测结果所需的数据特征的集合；所述第一数据特征集中的数据特征根据所述哈希值获得；
    预取模块，用于根据所述第一学习模型的输出结果在所述固态硬盘中进行数据预取。
  6. 根据权利要求1所述的数据存储系统的控制装置，其特征在于，还包安全单元，所述安全单元包括敏感数据获取模块和加解密模块；
    所述敏感数据获取模块用于获取与所述存储服务器进行数据交互过程中的敏感数据；所述敏感数据包括符合预设特征的数据；
    所述加解密模块包括与所述存储服务器中的加解密模块适配的数据安全模块。
  7. 根据权利要求1所述的数据存储系统的控制装置，其特征在于，所述处理器通过远程过程调用协议RPC接口协议接受所述存储服务器用于数据管理的服务请求。
  8. 一种数据存储系统，其特征在于，包括如权利要求1至7中任一所述的数据存储系统的控制装置，与所述控制装置对应的存储装置，以及，存储服务器；
    所述控制装置与所述存储服务器通过网络数据连接。
  9. 根据权利要求8中所述数据存储系统，其特征在于，所述控制装置包括有两个及两个以上。
  10. 根据权利要求8中所述数据存储系统，其特征在于，所述存储装置包括SATA硬盘和/或固态硬盘。
PCT/CN2017/100422 2017-02-21 2017-09-04 一种存储系统的控制装置 WO2018153051A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/440,979 US11567696B2 (en) 2017-02-21 2019-06-13 Control device of storage system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710093959.8 2017-02-21
CN201710093959.8A CN106951181A (zh) 2017-02-21 2017-02-21 一种数据存储系统的控制装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/440,979 Continuation US11567696B2 (en) 2017-02-21 2019-06-13 Control device of storage system

Publications (1)

Publication Number Publication Date
WO2018153051A1 true WO2018153051A1 (zh) 2018-08-30

Family

ID=59467644

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/100422 WO2018153051A1 (zh) 2017-02-21 2017-09-04 一种存储系统的控制装置

Country Status (3)

Country Link
US (1) US11567696B2 (zh)
CN (1) CN106951181A (zh)
WO (1) WO2018153051A1 (zh)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106951181A (zh) 2017-02-21 2017-07-14 深圳大普微电子科技有限公司 一种数据存储系统的控制装置
CN107169075A (zh) * 2017-05-10 2017-09-15 深圳大普微电子科技有限公司 基于特征分析的数据存取方法、存储设备及存储系统
CN108073363B (zh) * 2017-12-28 2021-10-01 深圳市得一微电子有限责任公司 数据存储方法、存储设备及计算机可读存储介质
CN108509652B (zh) * 2018-04-17 2021-01-22 山东大众益康网络科技有限公司 数据处理系统及方法
CN112445725A (zh) * 2019-08-27 2021-03-05 华为技术有限公司 预读取文件页的方法、装置和终端设备
CN116724299A (zh) 2021-03-01 2023-09-08 美光科技公司 支持写入命令的两阶段缓冲操作

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105426119A (zh) * 2015-10-28 2016-03-23 上海新储集成电路有限公司 一种存储设备及数据处理方法
CN105446895A (zh) * 2014-08-25 2016-03-30 北京云巢动脉科技有限公司 一种存储系统非同源数据运行时io去重的方法及系统
CN106227471A (zh) * 2016-08-19 2016-12-14 深圳大普微电子科技有限公司 固态硬盘和应用于固态硬盘的数据存取方法
CN106250064A (zh) * 2016-08-19 2016-12-21 深圳大普微电子科技有限公司 固态硬盘控制装置和基于学习的固态硬盘数据存取方法
CN106951181A (zh) * 2017-02-21 2017-07-14 深圳大普微电子科技有限公司 一种数据存储系统的控制装置

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7870317B2 (en) * 2005-04-29 2011-01-11 Network Appliance, Inc. Storage processor for handling disparate requests to transmit in a storage appliance
US7376034B2 (en) * 2005-12-15 2008-05-20 Stec, Inc. Parallel data storage system
US7500050B2 (en) * 2006-03-20 2009-03-03 International Business Machines Corporation Wise ordering for writes—combining spatial and temporal locality in write caches for multi-rank storage
US20110040771A1 (en) * 2008-06-18 2011-02-17 Petascan Ltd. Distributed hardware-based data querying
US9338139B2 (en) * 2008-09-15 2016-05-10 Vaultive Ltd. System, apparatus and method for encryption and decryption of data transmitted over a network
JP5341112B2 (ja) * 2008-11-25 2013-11-13 株式会社日立製作所 ライト完了を検知する機能を有するストレージシステム
KR20100082185A (ko) * 2009-01-08 2010-07-16 삼성전자주식회사 플래시 메모리, 캐시 메모리, 그리고 제어기를 포함하는 사용자 장치
CN101916295B (zh) * 2010-08-27 2011-12-14 董方 基于点对点网络的互联网搜索系统和方法
US8930364B1 (en) * 2012-03-29 2015-01-06 Amazon Technologies, Inc. Intelligent data integration
CN103608762B (zh) * 2013-05-09 2016-03-09 华为技术有限公司 存储设备、存储系统及数据发送方法
US9747214B2 (en) * 2013-05-22 2017-08-29 Schwegman Lundberg & Woessner, P.A. Forecast modeling cache prefetch system and method
JP6106028B2 (ja) * 2013-05-28 2017-03-29 株式会社日立製作所 サーバ及びキャッシュ制御方法
KR102074329B1 (ko) * 2013-09-06 2020-02-06 삼성전자주식회사 데이터 저장 장치 및 그것의 데이터 처리 방법
US9614925B2 (en) * 2014-03-31 2017-04-04 Futurewei Technologies, Inc. Intelligent file pre-fetch based on access patterns
US10120618B2 (en) * 2014-06-24 2018-11-06 Vmware, Inc. Systems and methods for adaptive offloading of mass storage data movement
US10296368B2 (en) * 2016-03-09 2019-05-21 Commvault Systems, Inc. Hypervisor-independent block-level live browse for access to backed up virtual machine (VM) data and hypervisor-free file-level recovery (block-level pseudo-mount)
US10275180B2 (en) * 2017-05-08 2019-04-30 Samsung Electronics Co., Ltd. Ethernet SSD system including storage offload engine (SoE) controller and ethernet switch
US11151137B2 (en) * 2017-09-25 2021-10-19 Splunk Inc. Multi-partition operation in combination operations
US20190102675A1 (en) * 2017-09-29 2019-04-04 Coupa Software Incorporated Generating and training machine learning systems using stored training datasets
US10649927B2 (en) * 2018-08-20 2020-05-12 Intel Corporation Dual in-line memory module (DIMM) programmable accelerator card

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105446895A (zh) * 2014-08-25 2016-03-30 北京云巢动脉科技有限公司 一种存储系统非同源数据运行时io去重的方法及系统
CN105426119A (zh) * 2015-10-28 2016-03-23 上海新储集成电路有限公司 一种存储设备及数据处理方法
CN106227471A (zh) * 2016-08-19 2016-12-14 深圳大普微电子科技有限公司 固态硬盘和应用于固态硬盘的数据存取方法
CN106250064A (zh) * 2016-08-19 2016-12-21 深圳大普微电子科技有限公司 固态硬盘控制装置和基于学习的固态硬盘数据存取方法
CN106951181A (zh) * 2017-02-21 2017-07-14 深圳大普微电子科技有限公司 一种数据存储系统的控制装置

Also Published As

Publication number Publication date
CN106951181A (zh) 2017-07-14
US20190294381A1 (en) 2019-09-26
US11567696B2 (en) 2023-01-31

Similar Documents

Publication Publication Date Title
WO2018153051A1 (zh) 一种存储系统的控制装置
US11960726B2 (en) Method and apparatus for SSD storage access
EP3118745B1 (en) A heuristic interface for enabling a computer device to utilize data property-based data placement inside a nonvolatile memory device
US10795586B2 (en) System and method for optimization of global data placement to mitigate wear-out of write cache and NAND flash
US9128855B1 (en) Flash cache partitioning
CN108268219B (zh) 一种处理io请求的方法及装置
US10635329B2 (en) Method and apparatus for performing transparent mass storage backups and snapshots
KR20170054299A (ko) 메모리 관리 시의 중복 제거를 위해서 기준 세트로 기준 블록을 취합하는 기법
TWI771933B (zh) 借助命令相關過濾器來進行重複資料刪除管理的方法、主裝置以及儲存伺服器
US9684665B2 (en) Storage apparatus and data compression method
WO2017041570A1 (zh) 向缓存写入数据的方法及装置
US10067683B2 (en) Systems and methods for classifying data in solid state drives
KR20170085951A (ko) 버저닝 저장 장치 및 방법
US20220019527A1 (en) Temperature-Based Data Storage Processing
US9720608B2 (en) Storage system
US10152242B1 (en) Host based hints
CN109783019B (zh) 一种数据智能存储管理方法与装置
US11327929B2 (en) Method and system for reduced data movement compression using in-storage computing and a customized file system
CN115470157A (zh) 预取方法、电子设备、存储介质及程序产品
US11249666B2 (en) Storage control apparatus
CN106155583B (zh) 缓存固态设备读取请求结果的系统和方法
WO2021248863A1 (zh) 数据处理方法及存储设备
CN116360673A (zh) 存储装置、存储装置的操作方法及使用其的存储系统
Hwang Towards even lower total cost of ownership of data center it infrastructure
KR102149468B1 (ko) 통합 캐시를 하나 또는 복수의 논리 유닛에 동적 할당하는 시스템 및 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17898205

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17898205

Country of ref document: EP

Kind code of ref document: A1