WO2022179294A1 - 数据预取的方法、设备及系统 - Google Patents

数据预取的方法、设备及系统 Download PDF

Info

Publication number
WO2022179294A1
WO2022179294A1 PCT/CN2021/142647 CN2021142647W WO2022179294A1 WO 2022179294 A1 WO2022179294 A1 WO 2022179294A1 CN 2021142647 W CN2021142647 W CN 2021142647W WO 2022179294 A1 WO2022179294 A1 WO 2022179294A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
partitions
feature
memory
partition
Prior art date
Application number
PCT/CN2021/142647
Other languages
English (en)
French (fr)
Inventor
涂剑洪
董如良
王海昕
吴沛
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2022179294A1 publication Critical patent/WO2022179294A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0644—Management of space entities, e.g. partitions, extents, pools
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0688—Non-volatile semiconductor memory arrays

Definitions

  • the present application relates to the field of computers, and in particular, to a method, device and system for data prefetching.
  • the main frequency of the central processing unit has been developed from MHz to GHz, but the access rate of common hard disks is less than 100M/S.
  • the difference in access speed between the CPU and different storage media causes the processor to spend a lot of time waiting for the return of memory data, which limits the access efficiency of the storage system.
  • the industry usually uses cache to speed up storage performance, and the controller of the cache is usually provided with a prefetcher (prefetching), which can predict from the data stream of the low-speed storage medium the upcoming access to the application. Prefetch data, then obtain the prefetched data from the data stream and store it in the cache in advance, thereby improving the access efficiency of the storage system.
  • prefetching a prefetcher
  • the current prefetching algorithm can usually only be performed for sequential streams. Prefetching, and various types of data streams will interfere with each other, resulting in limited prefetching capability of the prefetcher.
  • the present application provides a method, device and system for data prefetching, which are used to solve the problem that the prefetching capability of a prefetcher is limited.
  • a data prefetching method is provided, the method is applied to a prefetching system including multiple data streams, the prefetching system includes a first memory and a second memory, and the method includes the following steps: obtaining the data of the first memory. Statistical data of multiple partitions, the statistical data is used to indicate the historical situation of the data stream included in each partition of the multiple partitions, and the prefetching strategy is determined according to the statistical data of the multiple partitions, and the prefetching strategy is used to indicate the The location and the size of the prefetched data, the prefetch operation is performed according to the prefetch policy, and the prefetched data is prefetched from the first memory to the second memory.
  • the method divides the first memory into multiple partitions, and then determines the prefetching strategy of each partition in a targeted manner according to the characteristics of the data streams in each partition, so as to avoid the prefetching effect caused by mutual interference between multiple data streams Poor problem, improve cache hit rate.
  • the statistical data may include data flow characteristics, wherein the data flow characteristics include one of the number of data flows, the length of the data flow, the type of the data flow, the size of the data flow IO, and the IO interval of the data flow, or variety.
  • the statistical data may further include data flow information of the data flow included in each partition, and the above-mentioned data flow characteristics are obtained according to the data flow information.
  • the data stream includes multiple IOs delivered by upper-layer applications.
  • a data stream is a collection of IOs generated by the same application or a class of applications and used for data transmission or notification.
  • the data flow information includes but is not limited to one or more of the IO offset, IO length, and IO time of each IO.
  • the IO offset can be the address information where the data accessed by the application is located, such as the access address or the data stream.
  • the offset value in the data stream, the IO length can be the data length of the data
  • the IO time refers to the access time
  • the data stream characteristics can include but are not limited to the number of IOs in the data stream, the type of the data stream, the length of the data stream, the data stream The size of each IO in the stream, the data stream IO interval, etc.
  • the data flow feature may be determined according to data flow information.
  • the above statistical data may be represented in the form of a feature vector (x 1 , x 2 ,...,x m ), wherein each element in the feature vector may represent a data flow feature, which may be specifically
  • the numerical value of the data stream feature may also be the level, gear, code, etc. corresponding to the data stream feature, which is not specifically limited in this application.
  • the statistical data is represented by a five-dimensional vector (x 1 , x 2 , x 3 , x 4 , x 5 ), where x 1 is used to represent the number of data streams. If the number of data streams is less than or equal to 100, the corresponding The dimension is set to 1, the dimension of more than 100 less than or equal to 200 is set to 2, and so on.
  • x 2 is used to indicate the number of types of data streams, the number is not higher than 5, the corresponding value is 1, the number is between 5-10, the corresponding value is 2, and so on.
  • x 3 is used to indicate the length of the data stream.
  • the corresponding dimension of less than 1MB is set to 1, and the corresponding dimension of more than 1MB and less than 2MB is set to 2, and so on.
  • x 4 is used for the IO size of the data stream
  • the dimension less than 64KB is set to 1
  • the dimension greater than 64KB and less than 128KB is set to 2, and so on.
  • x 5 is used to represent the IO interval of the data stream.
  • the dimension less than 1ms is set to 1
  • the dimension greater than 1ms and less than 2ms is set to 2, and so on.
  • the statistical data of the above statistical partition 1 is: (2,1,2,2,1). It should be understood that the above examples are for illustration, and are not specifically limited in the present application.
  • the multiple partitions may be adjusted according to the statistical data of the multiple partitions to obtain multiple characteristic partitions, wherein multiple Each feature partition in the feature partition includes at least one type of data stream, and then, according to the statistical data of the multiple feature partitions, a prefetch strategy for each feature partition in the multiple feature partitions is determined, wherein each feature partition corresponds to a Prefetch strategy.
  • the storage space of the first memory can be divided according to a preset division rule to obtain the above-mentioned multiple partitions, and then the multiple partitions can be adjusted according to the type of the data stream included in each partition to obtain: Multiple feature partitions.
  • the preset division rules may include, but are not limited to, partitioning by disk, partitioning by size of storage space, etc. For example, if the first memory 110 includes multiple disks, then one disk may be divided into one partition. The memory 110 includes 1 disk, then each 64MB of continuous storage space of the disk can be divided into one partition, or every 138GB of continuous storage space can be divided into one partition, etc.
  • the division rules can be preset according to the actual situation, which is not made in this application. Specific restrictions.
  • a feature partition is used to separate the sequence flow from the mixed data flow.
  • multiple feature partitions are prefetched. According to the single data flow characteristics in each feature partition, the prefetching strategy can be determined in a targeted manner to avoid Mutual interference between multiple data streams leads to the problem of poor prefetching effect and improves the cache hit rate.
  • the multiple partitions are adjusted according to the statistical data of the multiple partitions, and when multiple characteristic partitions are obtained, the type of the data stream in the multiple partitions can be calculated according to the statistical data of the multiple partitions.
  • the number of partitions not less than the first threshold is split to obtain multiple feature partitions.
  • the partition with the type and number of data streams higher than the first threshold when the partition with the type and number of data streams higher than the first threshold is split, it can be split into two partitions, or it can be split into more than two partitions, and the splitting rule can be random splitting.
  • Splitting can also be average splitting, or splitting according to the distribution ratios of different types of data streams, and this application does not limit the splitting rules. For example, if the number of types of data streams in partition 1 is 5, the number of types of data streams in partition 2 is 2, and the first threshold is 3, then partition 1 can be split to obtain feature partition 11 and feature partition 12, Partition 2 is not split, so partition 1 and partition 2 are split into feature partition 11, feature partition 12, and feature partition 2. It should be understood that the above examples are for illustration, and are not limited in the present application.
  • the number of types of data streams in the characteristic partition can be made to be single, so that the sequence stream can be removed from the mixed data stream. It is separated to avoid the problem of poor prefetching effect caused by mutual interference between multiple data streams, and improve the cache hit rate.
  • the multiple partitions are adjusted according to the statistical data of the multiple partitions, and when multiple characteristic partitions are obtained, the data flow characteristics of the multiple partitions can be determined according to the statistical data of the multiple partitions. and then merge the partitions whose similarity is not lower than the second threshold to obtain multiple feature partitions.
  • the above-mentioned two partitions are two adjacent partitions that are logically continuous, that is to say, two partitions whose addressing modes of logical addresses are continuous can be merged, for example, the logical block addresses of adjacent partitions.
  • the physical address corresponding to partition 1 and the physical address corresponding to partition 2 may be continuous or discontinuous, which is not specifically limited in this application.
  • the similarity of statistical data can be determined according to the distance between feature vectors, such as Eucledian distance, Manhattan distance, cosine similarity, and minkowski distance. etc., which are not specifically limited in this application.
  • Merging the partitions whose similarity between data stream features is not lower than the second threshold can prevent the same type of data stream from being scattered in multiple partitions, so that the same type of data stream can uniformly determine the prefetch strategy and reduce computing resources. consumption and improve processing efficiency.
  • the partitions in which the number of types of data streams in the multiple partitions exceeds the first threshold may be split, and the partitions in which the feature similarity of the data streams in the multiple partitions is lower than the second threshold may be split.
  • Merge to obtain multiple feature partitions For example, multiple partitions include partition 1, partition 2, and partition 3. The number of data stream types in partition 1 exceeds the first threshold, and the number of data streams in partition 2 and partition 3 is lower than the first threshold.
  • partition 1 can be split into features Partition 11, feature partition 12, partition 2 and partition 3 are combined into feature partition 23.
  • splitting may be average splitting or random splitting, which is not specifically limited in this application, and the above examples are used to illustrate , which is not specifically limited in this application.
  • the IO contained in the data stream is the IO issued by the application, and the time of the IO issued by the application is not fixed, it may occur that after multiple adjustment cycles, the characteristics of the data stream in multiple feature partitions tend to be single.
  • the data flow in individual feature partitions has changed, resulting in data flow characteristics in individual feature partitions no longer tending to be single. Therefore, before determining the prefetching strategy corresponding to each feature partition, the Multiple feature partitions are adjusted to avoid including multiple data streams in the feature partition due to changes in the data stream, thereby improving the prefetching accuracy and improving the cache hit rate.
  • the prefetching benefit of each feature partition can be predicted according to the historical data of the prefetching operation performed by each feature partition in the multiple feature partitions, and the selection Prefetching a set of feature partitions whose revenue is higher than the third threshold, the set of feature partitions includes at least one feature partition, and a prefetch operation of each feature partition in the set of feature partitions is performed according to the prefetching strategy.
  • the prefetch strategy may include a prefetch offset value and a prefetch length of the prefetch data in the data stream, and may also include other relevant information for the processor to read the prefetch data from the data stream, For example, the reading time of the prefetched data, the address of the prefetched data, etc. are not limited in this application.
  • the processor can read the prefetched data from the first memory and store it in the second memory. in memory.
  • the prefetching strategy can be obtained by using the prefetching algorithm according to the statistical data of each feature partition.
  • the prefetching algorithm can be the Tap prefetching algorithm (Table-based Prefetching for Storage Caches), AMP prefetching algorithm (Adaptive Multi-stream Prefetching) in a Shared Cache), etc., this application does not limit which prefetching algorithm is used.
  • the prefetch benefit of each feature partition may include one or more of the prefetch hit rate, the prefetch waste rate, and the number of prefetch IOs, where the prefetch hit rate x includes the prefetch data being applied from The probability of reading from the second memory, the prefetch waste rate y includes the probability that the prefetched data is not read from the second memory by the application, and the prefetched IO number z is the IO number of the prefetched number.
  • the prefetch benefit can be determined according to the statistical data of each feature partition, and the historical prefetch hit rate, historical IO data volume, historical prefetch waste rate, etc. of each feature partition can be calculated according to the statistical data.
  • the above formula is used as an example, and is not specifically limited in this application.
  • the number of prefetch IOs may be obtained according to statistical data, and the above prefetch hit rate x and prefetch waste rate y may be determined according to historical data of each feature partition in combination with statistical data.
  • the historical data of each feature partition includes historical statistical data of each feature partition, the historical statistical data may include historical data flow information and historical data flow characteristics in all periods before the current period, and the historical data flow information may include but not Limited to IO offset, IO length and IO time, historical data flow characteristics may include but are not limited to IO number, data flow length, IO interval, etc., and the historical statistical data may be the historical statistical data of each feature partition in the previous cycle , or the historical statistical data of the previous two cycles, or the historical statistical data of all cycles before the current cycle, which is not specifically limited in this application.
  • the statistics collected in the current cycle show that the data stream of partition 1 includes 1 IO delivered by the application, and the IO offset of this IO is 100 and the length is 50.
  • the data actually accessed has an offset value of 0 and a length of 100, that is, a segment of data with an offset value from 0 to 100 in the data stream is read, which is determined according to the prefetch algorithm (such as the AMP prefetch algorithm).
  • the prefetch strategy is: the prefetch offset value is 100 and the prefetch length is 100.
  • this prefetch measurement a segment of data with an offset value from 100 to 200 will be read, and the statistics collected in the current cycle show that,
  • the actual demand is a piece of data with an offset value from 100 to 150, then it can be predicted that the prefetch hit rate is 50%, the prefetch waste rate is 50%, and the number of prefetch IOs is 1.
  • the prefetch income ⁇ x + ⁇ y + ⁇ z can obtain the prefetching profit, and the feature partition set is determined according to the size of the prefetching profit and the third threshold.
  • the prefetch hit rate can be further improved, and the access efficiency of the storage system can be improved.
  • the read/write speed of the first memory is lower than the read/write speed of the second memory; if the first memory is a memory, the second memory is a cache; or, if the first memory is a In the case of a magnetic disk, the second storage is memory.
  • the method provided by the present application can prefetch the data in the memory to the cache in advance, and the upper-layer application can quickly access the data from the cache, thereby improving the access efficiency of the memory;
  • the method provided by the present application can prefetch the data in the disk into the memory in advance, and the upper-layer application can quickly access the data from the memory, thereby improving the access efficiency of the disk.
  • a prefetching system in a second aspect, includes a processor, a first memory and a second memory, wherein the processor is configured to acquire statistical data of multiple partitions of the first memory, and the statistical data is used for In order to indicate the historical situation of the data stream included in each partition in the multiple partitions; the processor is used to determine the prefetch strategy according to the statistical data of the multiple partitions, and the prefetch strategy is used to indicate the prefetch data in the first memory. the location and the size of the prefetch data; and, performing a prefetch operation according to the prefetch policy, and storing the prefetch data to the second memory.
  • the system divides the first memory into multiple partitions, so that the CPU can determine the prefetch strategy of each partition in a targeted manner according to the characteristics of the data flow in each partition, and prefetch the prefetched data from the first memory to
  • the second memory the system can avoid the problem of poor prefetching effect caused by mutual interference between various data streams, and improve the cache hit rate.
  • the statistical data of the multiple partitions includes a type of data stream included in each of the multiple partitions
  • the processor is configured to, according to the statistical data of the multiple partitions of the first memory, perform an analysis on the multiple partitions. Perform adjustment to obtain multiple feature partitions, wherein each feature partition in the multiple feature partitions includes at least one type of data stream; the processor is configured to determine each of the multiple feature partitions according to the statistical data of the multiple feature partitions. Prefetch strategy for feature partitions, where one feature partition corresponds to one prefetch strategy.
  • the processor is configured to, according to the statistical data of the multiple partitions, split the partitions in the multiple partitions whose type and quantity of the data stream is not lower than the first threshold to obtain multiple characteristic partitions.
  • the statistical data includes data flow characteristics, wherein the data flow characteristics include one of the number of data flows, the length of the data flow, the type of the data flow, the size of the data flow IO, and the IO interval of the data flow, or variety.
  • the processor is configured to determine the similarity between the data flow features of the multiple partitions according to the statistical data of the multiple partitions of the first memory; the processor is configured to determine the similarity not lower than the second The thresholded partitions are merged to obtain multiple feature partitions.
  • the processor is used to predict the prefetch gain of each feature partition according to the historical data of the prefetch operation performed by each feature partition in the multiple feature partitions; the processor is used to select a prefetch gain higher than the A feature partition set with three thresholds, the feature partition set includes at least one feature partition; the processor is configured to perform a prefetch operation for each feature partition in the feature partition set according to the prefetch strategy, and store the prefetch data in the second memory.
  • the read/write speed of the first memory is lower than the read/write speed of the second memory; if the first memory is a memory, the second memory is a cache; or, if the first memory is a In the case of a magnetic disk, the second storage is memory.
  • a processor is provided, the processor is applied in a prefetch system including multiple data streams, the prefetch system includes a processor, a first memory and a second memory, wherein the processor includes a processor for Each module of the data prefetching method in the first aspect or any possible implementation manner of the first aspect is executed.
  • a computer program product which, when run on a computer, causes the computer to perform the methods described in the above aspects.
  • a computer-readable storage medium is provided, and instructions are stored in the computer-readable storage medium, which, when executed on a computer, cause the computer to perform the methods described in the above aspects.
  • a computing device in a sixth aspect, includes a processor configured to execute the methods described in the above aspects.
  • the present application may further combine to provide more implementation manners.
  • FIG. 1 is a schematic structural diagram of a prefetching system provided by the application.
  • FIG. 2 is a schematic structural diagram of a prefetching system provided by the present application in a distributed scenario
  • FIG. 3 is a schematic flowchart of steps of a prefetching method provided by the present application.
  • FIG. 4 is a schematic flowchart of adjusting multiple partitions into multiple feature partitions in an application scenario provided by the present application
  • FIG. 5 is a schematic flowchart of steps for performing a prefetch operation in an application scenario provided by the present application
  • FIG. 6 is a schematic structural diagram of a processor provided by the present application.
  • FIG. 7 is a schematic structural diagram of a computing device provided by the present application.
  • the main frequency of the central processing unit has been developed from MHz to GHz, but the access rate of common hard disks is less than 100M/S.
  • the performance of the processor can double every two years, while the access rate of mechanical and electronic devices such as disks only increases by about 8% per year, resulting in an ever-increasing gap in the access speed between the CPU and different storage media.
  • the larger the value the processor needs to spend a lot of time waiting for the return of memory data, and the memory performance severely limits the performance of the CPU.
  • caches are commonly used in the industry to accelerate storage performance.
  • the access speed of the cache is high but the capacity is small, so the controller of the cache is usually provided with a prefetcher (prefetching).
  • prefetching The prefetched data is read from the low-speed storage medium with the capacity to the cache, so that the upper-layer application can quickly read the prefetched data from the cache. It should be understood that an excellent prefetcher can accurately predict the prefetched data to be accessed by the application, which can effectively improve the cache hit rate and greatly improve the access speed of the memory.
  • the data stream in the low-speed storage medium is a mixed data stream, wherein the mixed data stream includes multiple upper-layer applications delivered.
  • the mixed data stream includes various types of data streams, such as sequence stream, random stream, etc., and the sequence stream can further include long sequence stream, short sequence stream and so on.
  • the sequence flow may include multiple data requests of the same time sequence request, and the data volume of the long sequence flow is much larger than that of the short sequence flow.
  • the sequence flow may include a large number of query IOs performed by the database, streaming media service IOs, etc.
  • a long sequence flow can include multiple IOs that scan and query the entire table
  • a short sequence flow can include multiple IOs for user-specified queries, such as half a table or a filtered part that matches Query the table items of the conditions.
  • the random stream may include application IO randomly delivered by multiple upper-layer applications, such as world wide web (Web) service requests, mailbox (mail) service requests, and so on.
  • Web world wide web
  • email mailbox
  • FIG. 1 is a prefetching system 100 provided by the present application.
  • the prefetching system 100 The first memory including the mixed data stream can be divided into multiple partitions, and the multiple partitions can be adjusted according to the characteristics of the data flow in each partition, so that the characteristics of the data flow in each partition tend to be single, and multiple partitions can be obtained.
  • Feature partitioning so as to separate the sequence stream from the mixed data stream.
  • prefetching multiple feature partitions can avoid the problem of poor prefetching effect caused by mutual interference between multiple data streams, thereby improving the cache hit rate. .
  • the prefetching system 100 can be deployed inside a computing device, and the computing device can specifically be a physical server, such as an X86 server, an ARM server, etc.; the computing device can also be based on a general physical server combined with network functions virtualization (NFV)
  • NFV network functions virtualization
  • a virtual machine (VM) implemented by technology, a virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment, such as a virtual device in cloud computing; computing devices can also It is a storage class device used to store data, such as a storage array, all-flash high-end storage, etc., which is not specifically limited in this application.
  • the prefetching system 100 can also be deployed in a distributed storage system, such as Fusion storage, Hadoop distributed file system (hadoop distributed file dystem, HDFS), etc. Specifically, the prefetching system 100 can be deployed in a distributed storage system. In each storage node, some units of the data area system 100 may be deployed on the storage node 1, and some unit modules may be deployed on the storage node 2, which is not specifically limited in this application.
  • a distributed storage system such as Fusion storage, Hadoop distributed file system (hadoop distributed file dystem, HDFS), etc.
  • HDFS Hadoop distributed file dystem
  • the prefetching system 100 may be divided in various manners.
  • the prefetching system 100 may include a first memory 110 , a second memory 120 and a processor 130 .
  • the first memory 110, the second memory 120 and the processor 130 may be connected to each other through a bus, such as a peripheral component interconnect express (PCIe) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, etc., communication can also be realized by other means such as wireless transmission, such as Ethernet (Ethernet), which is not specifically limited in this application.
  • PCIe peripheral component interconnect express
  • EISA extended industry standard architecture
  • FIG. 1 is only an exemplary division manner, and each module unit may be merged or split into more or less module units, which is not specifically limited in this application, and the system shown in FIG. 1 and The positional relationship between modules also does not constitute any limitation.
  • Both the first memory 110 and the second memory 120 are memories.
  • the data in the first memory 110 may be prefetched to the second memory 120 in advance, so that the processor 130 reads the data from the second memory 120 .
  • the data refers to the data to be accessed by the application determined by the prefetching algorithm, and the data may be one piece of data, or may be a data set formed by a group of data.
  • the access rate of the first memory 110 may be lower than that of the second memory 120.
  • the prefetched data in the first memory 110 is prefetched to the second memory 120, and the application can access the second memory from the higher speed.
  • the prefetched data is read in 120 , thereby increasing the access rate of the first memory 110 .
  • the memory types of the first memory 110 and the second memory 120 are different. Exemplarily, the following describes the memory types of the first memory 110 and the second memory 120 in different application scenarios.
  • the first memory 110 may be an external memory
  • the second memory 120 may be a main memory in the computing device, wherein , the external memory refers to the memory other than the memory and the processor's cache.
  • the prefetched data X to be accessed by the application is prefetched from the first storage 110 to the second storage 120 in advance, and the application can read the prefetched data X from the second storage 120.
  • the access rate is much higher than the external memory, so using the prefetching system provided by this application can improve the access rate of the CPU.
  • the first memory may be any one of a magnetic disk (disk), a compact disc (CD), a U disk (universal serial bus, USB), a memory card (memory card), etc., wherein the above-mentioned magnetic disk includes But not limited to hard disk drive (HDD), solid-state drive (SSD), hybrid hard disk (solid state hybrid drive, SSHD), redundant arrays of independent disks (RAID), etc. etc., the first memory 110 may also be a combination of the above-mentioned various storage media, which is not specifically limited in this application.
  • the second memory 120 may be a read-only memory (ROM), a random access memory (RAM), a dynamic random-access memory (DRAM), a double rate synchronous dynamic random access memory (double Data rate (SDRAM, DDR), etc., the second memory 120 may also be a combination of the above-mentioned various storage media, which is not specifically limited in this application.
  • the first memory 110 may be a memory
  • the second memory is a cache (cache) of the computing device.
  • the data X to be accessed by the application is prefetched from the first memory 110 to the second memory 120 in advance, and the application can read the data X from the second memory 120, because the access speed of the cache is much higher than Therefore, using the prefetching system provided by this application can improve the access rate of the CPU.
  • the first memory may be any one of ROM, RAM, DRAM, DDR, etc.
  • the second memory 120 may be a static random-access memory (SRAM), an asynchronous SRAM (synchronous SRAM, Async SRAM), multi-threaded SRAM (synchronous SRAM, Sync SRAM), synchronous SRAM (pipelined brst SRAM, PB SRAM), etc., which are not specifically limited in this application.
  • SRAM static random-access memory
  • asynchronous SRAM synchronous SRAM, Async SRAM
  • multi-threaded SRAM synchronous SRAM, Sync SRAM
  • synchronous SRAM pipelined brst SRAM, PB SRAM
  • the prefetching system 100 includes multiple computing devices, and the first memory and the second memory may be respectively deployed on different computing devices.
  • FIG. 2 is a distributed scenario
  • the following is a schematic structural diagram of the prefetch system 100 provided by the present application, wherein the first memory 110 may be the external memory of the storage node 1, the second memory may be the memory of the storage node 2, and the data X is stored in the first memory of the storage node 1.
  • the statistics module 112 and the partition adjustment module 113 are deployed in the processor 1301 of the storage node 1
  • the prefetch module 121 and the storage module 122 are deployed in the processor 1302 of the storage node 2 .
  • the data X to be accessed by the application is prefetched from the first memory of the storage node 1 to the second memory 120 of the storage node 2 in advance.
  • the prefetching system provided by the present application can read the data X from the second memory 120 (the local memory), and can Increase the access rate of the CPU.
  • the first memory in the storage node 1 may be a magnetic disk, CD, U disk, memory card, etc.
  • the above-mentioned disks include but are not limited to HDD, SSD, SSHD, RAID, etc., and may also be the above-mentioned various external memory The combination is not specifically limited in this application.
  • the second memory 120 may be ROM, RAM, DRAM, DDR, etc., or a combination of the above-mentioned various memories, which is not specifically limited in this application.
  • the first memory may also be the memory of storage node 1
  • the second memory may be the cache of storage node 2 or
  • the first memory may also be the memory of storage node 1
  • the second memory may be the memory of storage node 1.
  • the data X is prefetched from the first memory 110 of the storage node 1 to the memory of the storage node 2, so the processor 1301 and the processor 1302 display the part that participates in the prefetching in this application scenario Unit modules, in a specific implementation, the storage node 1 and the storage node 2 in FIG.
  • the processor 1301 of the storage node 1 may also include a prefetch module and a storage module.
  • the processor 1302 of the storage node 2 may also include a statistics module and a partition adjustment module, which are used for partition adjustment and acquisition of statistical data for the external memory in the storage node 2, This application does not limit this.
  • the first memory 110 and the processor 130 may be divided in various manners.
  • the first memory 110 may include multiple partitions 111
  • the processor 130 may include a statistics module 112 and a partition adjustment module. 113 , a prefetch module 121 and a storage module 122 .
  • a plurality of partitions 111 are used to store data.
  • the plurality of partitions 111 may be shown as partitions 1 to N shown in FIG. 1 and FIG. 2 , where N is a positive integer.
  • the statistics module 112 is configured to acquire statistical data of the multiple partitions of the first memory, where the statistical data is used to indicate the historical situation of the data flow in each of the multiple partitions.
  • the data stream includes multiple application IOs issued by the upper-layer application.
  • the historical situation of the data stream can be the historical information of each IO in the data stream, such as the access offset, access time and access size of each IO, etc.
  • the application IO in the data stream may be that the database executes a large number of query requests, and the history of the data stream may include the request address, request time, and size of the requested data for each query request.
  • a data stream is a collection of IOs generated by the same application or a class of applications to transmit data or notifications.
  • the partition adjustment module 113 is configured to adjust the multiple partitions to obtain multiple feature partitions according to the above statistical data, so that each feature partition in the adjusted multiple feature partitions includes at least one type of data stream, wherein the data
  • the types of streams include at least long sequence streams, short sequence streams and random streams.
  • the partition adjustment module 113 adjusts the multiple partitions 111, it can split the partitions whose type and number of data streams are higher than the first threshold, or can also divide the similarity of the data stream features higher than the second threshold.
  • the adjacent partitions are merged to obtain multiple feature partitions, where the adjacent partitions refer to two logically consecutive partitions, for example, two consecutive partitions whose logical addresses are addressed in a continuous manner can be merged.
  • the partition adjustment module 113 can adjust the above-mentioned multiple partitions 111
  • the statistics module 112 obtains the statistical data of the above-mentioned multiple characteristic partitions again, and the partition adjustment module 113 adjusts the above-mentioned multiple characteristic partitions again.
  • the data flow features in each feature partition tend to be single.
  • the multiple partitions 111 can be divided according to preset division rules, for example, a disk is divided into one partition, or every 64MB of continuous storage space is divided into one partition, or every 138GB of continuous storage space is divided into one partition etc.
  • the division rules may be preset according to the actual situation, which is not specifically limited in this application. For example, first, according to the division rule that a disk is divided into one partition, the first storage 110 is divided into three partitions, partition 1 is disk A, partition 2 is disk B, and partition 3 is disk C, which is obtained by the statistics module 112.
  • the partition adjustment module 113 can Adjust each partition to obtain the characteristic partition 1' including the storage space for storing long sequence streams in disk A and disk B, and the characteristic partition 2' including the storage space for storing short sequence streams in disk A and disk C, and the characteristic partition 3' includes the storage space in disk B and disk C for storing random streams. It should be understood that the above examples are for illustration, and are not specifically limited in the present application.
  • the partition adjustment module 113 may perform multiple adjustments on multiple partitions before the prefetching system 100 is enabled, so that the data flow characteristics in each feature partition tend to be single, and then enable the prefetching system 100 .
  • the statistics module 112 obtains the statistical data of each feature partition. If the data streams in multiple feature partitions change at this time, the data flow characteristics in some feature partitions will no longer tend to be single.
  • the partition adjustment module 113 can adjust these feature partitions again, and after splitting or merging the feature partitions whose data flow characteristics no longer tend to be single, the prefetching module 121 determines the prefetching strategy for each feature partition, so as to avoid Due to the change of the data stream, the feature partition includes multiple data streams, thereby improving the prefetching accuracy and the cache hit rate.
  • the prefetching module 121 is used to determine the prefetching strategy of each feature partition according to the data flow characteristics of the above-mentioned multiple feature partitions, and the prefetching strategy is used to indicate the position of the prefetched data and the size of the prefetched data, so that the storage module 122 performs a prefetch operation on each feature partition according to the prefetch policy, reads the prefetch data from the feature partition, and stores the data in the storage module of the second memory 120 .
  • the prefetching module 121 may include multiple prefetching submodules, and each prefetching submodule corresponds to a feature partition.
  • the feature partition 1' includes a long sequence stream
  • the feature partition 2' includes Short sequence flow
  • feature partition 3' includes random flow
  • prefetch module 121 may include prefetch submodule 1, prefetch submodule 2 and prefetch submodule 3, wherein prefetch submodule 1 may be based on feature partition 1'
  • the data characteristics of the medium and long sequence streams determine the long sequence stream prefetching strategy, so that the storage module 122 reads the prefetched data from the feature partition 1' and stores it in the storage module, and so on, the prefetching sub-module 2 can The prefetching strategy of the feature partition 2' is determined, and the prefetching sub-module 3 can determine the prefetching strategy of the feature partition 3', which will not be described in detail here.
  • each prefetch sub-module in the prefetch module 121 may first predict the prefetch value of each feature partition according to the historical data of the prefetch operation performed by each feature partition. Take the profit, select the feature partition set with the prefetch profit higher than the third threshold, and perform the prefetch operation of each feature partition in the feature partition set according to the prefetch strategy, wherein the feature partition set includes at least one feature partition. If the prefetching benefit is less than the threshold value of the characteristic partition, the prefetching strategy is not used to prefetch the partition, thereby further improving the prefetching hit rate and the access rate of the storage system.
  • the prefetch benefit of each feature partition above may include one or more of the prefetch hit rate, the prefetch waste rate, and the number of prefetch IOs.
  • the prefetch profit can be determined according to the statistical data of each feature partition. According to the statistical data, the historical prefetch hit rate, historical IO data volume, historical prefetch waste rate, etc. of each feature partition can be calculated.
  • the weight of y, ⁇ is the weight of the prefetch IO number z, ⁇ , ⁇ , and ⁇ can be empirical values, which can also be adjusted according to the accuracy of the prefetching benefits.
  • the above formula is used as an example, and is not specifically limited in this application.
  • the processor 130 may be composed of at least one general-purpose processor, such as a central processing unit, or a combination of a CPU and a hardware chip.
  • the above-mentioned hardware chip may be an ASIC, a PLD, or a combination thereof, and the above-mentioned PLD may be a CPLD, an FPGA, or a GAL. or any combination thereof, which is not specifically limited in this application.
  • the processor 130 when the prefetching system 100 is deployed on a single computing device, the processor 130 may be the CPU of the computing device; when the prefetching system 100 is deployed in a distributed system, such as the distributed storage system shown in FIG. 2 , the processor 130 may be the processor 1302 in the storage node 2, that is, the processor 130 is the CPU in the storage node where the second memory 120 is located.
  • FIG. 2 is for illustration, and is not specifically limited in this application.
  • the processor 130 may execute various types of program codes to enable the prefetch system 100 to implement various functions.
  • the first memory 110 may store the program codes of the statistics module 112 and the partition adjustment module 113
  • the second memory 120 may store the program codes of the prefetch module 121 and the storage module 122 .
  • the processor 130 can call the program code of the statistics module 112 in the first memory 110 to obtain the statistical data of the multiple partitions 111, and then call the program code of the partition adjustment module 113 to realize the adjustment of the multiple partitions 111, so as to obtain the multiple partitions 111.
  • the above prefetching strategy can use the applicable prefetching algorithm to determine the prefetching strategy of each feature partition according to the data flow characteristics in the feature partition, for example, using the Tap prefetching algorithm (Table-based Prefetching for Storage Caches). ), AMP prefetching algorithm (Adaptive Multi-stream Prefetching in a Shared Cache), etc., which are not specifically limited in this application.
  • the Tap prefetching algorithm Table-based Prefetching for Storage Caches.
  • AMP prefetching algorithm Adaptive Multi-stream Prefetching in a Shared Cache
  • the prefetching system provided by the present application can divide the first memory including the mixed data stream into multiple partitions, and adjust the multiple partitions according to the characteristics of the data stream in each partition, so that each partition The characteristics of the data flow in the data flow tend to be single, and multiple feature partitions are obtained, so as to separate the sequence flow from the mixed data flow.
  • multiple feature partitions are prefetched, and the single data flow in each feature partition can be used It can avoid the problem of poor prefetching effect caused by mutual interference between various data streams and improve the cache hit rate.
  • the above adjustment process may be performed once before data prefetching to adjust multiple partitions; or multiple partitions may be dynamically adjusted by using the above adjustment process multiple times according to data flow conditions during the prefetching process.
  • FIG. 3 is a schematic flowchart of steps of a data prefetching method provided by the present application, wherein the data prefetching method can be applied to the prefetching system 100 shown in FIG. 1 and FIG. 2 , and the prefetching system 100 includes a variety of Data flow (ie mixed data flow), as shown in Figure 3, the method may include the following steps:
  • S310 Acquire statistical data of the multiple partitions 111 of the first memory 110, where the statistical data is used to indicate the historical situation of the data stream included in each partition.
  • the above-mentioned multiple partitions 111 may first divide the storage space of the first memory according to a preset division rule. For example, if the first memory 110 includes multiple disks, one disk may be divided into one partition. If the storage 110 includes one disk, then every 64MB of continuous storage space of the disk can be divided into one partition, or every 138GB of continuous storage space can be divided into one partition, etc. Specifically, the division rules can be preset according to the actual situation. Without making a specific limitation, it should be understood that for the detailed description of the multiple partitions 111, reference may be made to the relevant descriptions in the foregoing embodiments of FIG. 1 and FIG. 2 , which will not be repeated here.
  • the above-mentioned statistical data may include data flow information and data flow characteristics of the data flow included in each partition, wherein the data flow includes multiple IOs delivered by an upper-layer application, and the data flow information includes but Not limited to one or more of the IO offset, IO length, and IO time of each IO, the IO offset can be the address information where the data accessed by the application is located, such as the access address or the offset value in the data stream , the IO length can be the data length of the data, the IO time refers to the access time, and the data flow characteristics can include but are not limited to the number of IOs in the data flow, the type of data flow, the length of the data flow, and each IO in the data flow. size, data stream IO interval, etc.
  • the data flow feature may be determined according to data flow information.
  • the above statistical data can be represented in the form of a feature vector (x 1 , x 2 ,...,x m ), wherein each element in the feature vector can represent a data flow feature, which can be specifically
  • the numerical value of the data stream feature may also be the level, gear, code, etc. corresponding to the data stream feature, which is not specifically limited in this application.
  • the statistical data is represented by a five-dimensional vector (x 1 , x 2 , x 3 , x 4 , x 5 ), where x 1 is used to represent the number of data streams. If the number of data streams is less than or equal to 100, the corresponding The dimension is set to 1, the dimension of more than 100 less than or equal to 200 is set to 2, and so on.
  • x 2 is used to indicate the number of types of data streams, the number is not higher than 5, the corresponding value is 1, the number is between 5-10, the corresponding value is 2, and so on.
  • x 3 is used to indicate the length of the data stream.
  • the corresponding dimension of less than 1MB is set to 1, and the corresponding dimension of more than 1MB and less than 2MB is set to 2, and so on.
  • x 4 is used for the IO size of the data stream
  • the dimension less than 64KB is set to 1
  • the dimension greater than 64KB and less than 128KB is set to 2, and so on.
  • x 5 is used to represent the IO interval of the data stream.
  • the dimension less than 1ms is set to 1
  • the dimension greater than 1ms and less than 2ms is set to 2, and so on.
  • the statistical data of the above statistical partition 1 is: (2,1,2,2,1). It should be understood that the above examples are for illustration, and are not specifically limited in the present application.
  • step S310 may be the statistical data of each partition obtained after the processor 130 invokes the code of the statistics module 112 in the first memory 110 .
  • the processor 130 may periodically acquire the statistical data of each partition, and the statistical data is the statistical data of each partition in the current cycle. For example, the current statistical data of each partition is acquired every 2 seconds. time is limited.
  • S320 Determine a prefetch policy according to the statistical data of the multiple partitions, where the prefetch policy is used to indicate the position of the prefetched data and the size of the prefetched data.
  • the above-mentioned multiple partitions 111 may be adjusted according to the statistical data of the multiple partitions to obtain multiple characteristic partitions, and step S320 may determine the prefetching strategy according to the statistical data of the multiple characteristic partitions.
  • each feature partition is used to store at least one type of data stream, and then the prefetching strategy of each feature partition is determined.
  • three feature partitions can be obtained, wherein the data stream in feature partition 1 is a long sequence stream, the data stream in feature partition 2 is a short sequence stream, and the data stream in feature partition 3 is a short sequence stream.
  • the data stream is a random stream.
  • the partitions whose type quantity of the data stream is higher than the first threshold may be split to obtain multiple feature partitions.
  • the partitions with the type and number of data streams not less than the first threshold can be split into two partitions or more than two partitions when splitting, and the splitting rules can be random Splitting can also be average splitting, 3
  • This application does not limit the splitting rules. For example, if the number of types of data streams in partition 1 is 5, the number of types of data streams in partition 2 is 2, and the first threshold is 3, then partition 1 can be split to obtain feature partition 11 and feature partition 12, Partition 2 is not split, so partition 1 and partition 2 are split into feature partition 11, feature partition 12, and feature partition 2. It should be understood that the above examples are for illustration, and are not limited in the present application.
  • the similarity calculation may be performed on the data flow characteristics of the two phase partitions according to the statistical data of the two partitions. If the similarity is higher than the second threshold, the data flow characteristics of the two partitions are indicated. To be consistent, two partitions can be merged. The above two partitions are two adjacent partitions that are logically continuous.
  • partitions with consecutive logical addresses can be combined, such as adjacent
  • the physical address corresponding to partition 1 and the physical address corresponding to partition 2 may be continuous or discontinuous, which is not specifically limited in this application.
  • step S310 it can be known that the statistical data of each partition can be represented in the form of feature vectors (x 1 , x 2 ,..., x m ). Therefore, the similarity of the statistical data can be determined according to the The distance is determined, such as Euclidian distance, Manhattan distance, cosine similarity, Minkowski distance, etc., which are not specifically limited in this application.
  • the partitions in the multiple partitions 111 whose types and number of data streams exceeds the first threshold may be split, and the feature similarity of the data streams in the multiple partitions 111 whose feature similarity is lower than the second threshold may be split.
  • the partitions are merged to obtain multiple feature partitions.
  • multiple partitions include partition 1, partition 2, and partition 3.
  • the number of data stream types in partition 1 exceeds the first threshold, and the number of data streams in partition 2 and partition 3 is lower than the first threshold.
  • partition 1 can be split into features Partition 11, feature partition 12, partition 2 and partition 3 are combined into feature partition 23. It should be understood that the above-mentioned splitting can be average splitting, random splitting, or splitting according to the distribution ratio of different types of data streams. point.
  • the processor 130 may adjust the multiple partitions before determining the prefetch strategy in step S320, and may also periodically obtain statistical data of each partition before step S310, and periodically adjust each partition to obtain multiple characteristic partitions , when the number of data flow types in each feature partition is lower than the first threshold, and the similarity of data flow features between each feature partition is higher than the second threshold, that is, the data flow features in each feature partition tend to be single. , and then perform steps S310 to S330 to realize the process of data prefetching.
  • the processor 130 can call the code of the statistics module 112 to adjust the first adjustment cycle
  • the obtained multiple feature partitions are counted, and the statistical data of the multiple feature partitions are obtained, and then the code of the partition adjustment module 113 is called to further adjust the multiple feature partitions.
  • the number of types of data streams in the data stream will be lower than the first threshold, and the similarity between the statistical data of the feature partitions will be lower than the second threshold, so that the data stream features in each feature partition tend to be single, different types
  • the sequence stream can be separated from the mixed data stream.
  • steps S310 to S330 are executed to prefetch data from the first memory to the second memory, which can avoid mutual interference between multiple data streams and cause poor prefetching effect. problem, thereby increasing the cache hit rate.
  • FIG. 4 is a schematic flowchart of adjusting multiple partitions into multiple feature partitions under an application scenario provided by the present application.
  • the multiple partitions include partition 1, partition 2 and Partition 3, where Partition 1, Partition 2, and Partition 3 are logically continuous three partitions.
  • the CPU calls the code of the statistics module 112 every cycle T to collect the historical conditions of the data streams in multiple partitions, and obtains more information.
  • Statistical data of each partition, and call the code of the partition adjustment module 113 to adjust the multiple partitions once, then at time t T, according to the statistical characteristics of partition 1, partition 2 and partition 3, the number of types of data streams can be adjusted to a higher value.
  • Partition 1 at the first threshold is split into feature partition 11 and feature partition 12 , and partitions 2 and 3 whose statistical data similarity is higher than the second threshold are merged into feature partition 23 .
  • feature partition 11 is split into feature partition 111 and feature partition 112
  • feature partition 12 and feature partition 13 are combined into feature partition 1223 .
  • FIG. 4 is used for illustration, and is not specifically limited in the present application.
  • step S320 determine the corresponding Before the prefetching strategy, multiple feature partitions can be adjusted to avoid multiple data streams in the feature partition due to changes in the data stream, thereby improving the prefetching accuracy and the cache hit rate.
  • the processor 130 may call the code of the prefetch module 121 in the second memory 120 to determine each feature partition according to the data flow characteristics in each feature partition The corresponding prefetch strategy. It should be understood that the prefetch policy is used to indicate the location of the prefetched data and the size of the prefetched data, and the processor 130 may acquire the prefetched data from the data stream of the first memory 110 according to the prefetch policy.
  • the prefetch strategy may include a prefetch offset value and a prefetch length of the prefetch data in the data stream, and may also include other relevant information for the processor 130 to read the prefetch data from the data stream, such as The read time of the prefetched data, the address of the prefetched data, etc., are not limited in this application.
  • the processor 130 may read the prefetched data from the first memory and store it in the second memory.
  • the prefetching strategy may be obtained by using a prefetching algorithm according to the statistical data of each feature partition.
  • the prefetching algorithm may be a Tap prefetching algorithm, an AMP prefetching algorithm, etc. algorithm is limited.
  • the processor 130 may use the prefetch strategy corresponding to each feature partition to prefetch each feature partition, read the prefetched data from the data stream of the first memory 110, and store it in the second In the memory 120, the function of data prefetching is implemented to improve the access speed of the memory.
  • the prefetching benefit of each feature partition can be predicted according to the historical data of the prefetching operation performed by each feature partition in the multiple feature partitions. Then, a set of feature partitions with a prefetching profit higher than the third threshold is selected, and a prefetch operation of each feature partition in the set of feature partitions is performed according to the prefetching strategy, wherein the set of feature partitions includes at least one feature partition.
  • the processor 130 may determine the prefetching revenue after data prefetching is performed for each partition using the corresponding prefetching strategy, and then call the storage module 122 to prefetch the feature partitions whose prefetching revenue is not lower than the third threshold, Take feature partitions whose revenue is lower than the third threshold and give up prefetching, thereby further improving the prefetching hit rate and avoiding resource waste caused by prefetching failures.
  • the processor 130 may periodically acquire statistical data of multiple partitions, periodically adjust the multiple partitions into multiple characteristic partitions, and then periodically prefetch the first memory, still referring to FIG. 4 .
  • the prefetch benefit of each feature partition described above may include one or more of the prefetch hit rate x, the prefetch waste rate y, and the number of prefetch IOs z, where the prefetch hit rate
  • the rate x includes the probability that the prefetch data is read from the second memory 120 by the application
  • the prefetch waste rate y includes the probability that the prefetch data is not read from the second memory 120 by the application
  • the prefetch IO number z is the prefetch rate. Get the number of IOs. Specifically, different weights can be set for the prefetch hit rate, the prefetch waste rate, and the number of prefetch IOs.
  • prefetch profit ⁇ x+ ⁇ y+ ⁇ z
  • ⁇ is the weight of the prefetch hit rate x
  • ⁇ is the weight of the prefetch waste rate y
  • ⁇ is the weight of the prefetch IO quantity z
  • ⁇ , ⁇ and ⁇ can be empirical values, which can also be adjusted according to the accuracy of the prefetch benefits.
  • the above formula is used as an example, and is not specifically limited in this application.
  • the number of prefetch IOs can be obtained according to the statistical data collected in step S310, and the prefetch hit rate x and the prefetch waste rate y can be determined according to the historical data of each feature partition in combination with the statistical data collected in step S310.
  • the historical data of each feature partition includes historical statistical data of each feature partition
  • the historical statistical data may include historical data flow information and historical data flow characteristics in all periods before the current period
  • the historical data flow information may include but not Limited to IO offset, IO length and IO time
  • historical data flow characteristics may include but are not limited to IO number, data flow length, IO interval, etc., wherein, for the specific description of historical statistical data, refer to the description of statistical data in step S310 , which will not be repeated here, and the historical statistical data can be the historical statistical data of each feature partition in the previous cycle, the historical statistical data of the previous two cycles, or the historical statistical data of all cycles before this cycle. , which is not specifically limited in this application.
  • the statistics collected in the current cycle show that the data stream of partition 1 includes 1 IO delivered by the application, and the IO offset of this IO is 100 and the length is 50.
  • the data actually accessed has an offset value of 0 and a length of 100, that is, a segment of data with an offset value from 0 to 100 in the data stream is read, which is determined according to the prefetch algorithm (such as the AMP prefetch algorithm).
  • the prefetch strategy is: the prefetch offset value is 100 and the prefetch length is 100.
  • this prefetch measurement a segment of data with an offset value from 100 to 200 will be read, and the statistics collected in the current cycle show that,
  • the actual demand is a piece of data with an offset value from 100 to 150, then it can be predicted that the prefetch hit rate is 50%, the prefetch waste rate is 50%, and the number of prefetch IOs is 1.
  • the prefetch income ⁇ x + ⁇ y + ⁇ z can obtain the prefetching profit, and the feature partition set is determined according to the size of the prefetching profit and the third threshold.
  • the historical statistical data may also include the current load situation of the prefetching system 100, the accuracy of historical prefetching revenue estimates, and other relevant information.
  • the accuracy of historical prefetching revenue estimates may also include other relevant information.
  • the processor 130 calls the prefetching module to determine the prefetching strategy and the prefetching profit of the feature partition, and when it is determined that the prefetching profit is not lower than the third threshold, the processor 130 calls the storage module Before using the prefetch strategy to perform the prefetch operation on the feature partition, it is also possible to query whether there is prefetch data corresponding to the prefetch strategy in the second memory 120. If it exists, the prefetch operation can be cancelled. The characteristic partition is then prefetched according to the prefetching strategy, and the prefetched data is read from the first memory to the second memory, so as to avoid the occurrence of repeated reading, thereby avoiding waste of resources.
  • each feature partition may change, which may lead to a sudden increase in the number of data flow types in some feature partitions.
  • prefetch this type of feature partition There will still be a situation where the prefetch hit rate is low.
  • the feature partitions with changes in this type of data stream can be screened out, thereby improving the prefetch hit rate, thereby improving the memory access efficiency.
  • the prefetching operation can be performed on the feature partition whose prefetching revenue is not lower than the third threshold in the current cycle, or the prefetching operation can be performed in the next cycle. , which is not specifically limited in this application.
  • the partition adjustment module 113 can be called to adjust the feature partitions of this type, thereby improving the prefetch hit rate;
  • the prefetching income is still low, and the weight parameters in the prefetching income can be adjusted to avoid the prefetching operation being affected by the unreasonable setting of the weighting parameters of the prefetching income.
  • the prefetching strategy of each feature partition can be determined first, and the prefetching strategy 111 of the feature partition 111 and the prefetching strategy 112 of the feature partition 112 can be obtained,
  • the prefetch strategy 1223 of the feature partition 1223 and then determine the prefetch revenue of each feature partition, determine the prefetch revenue 111 according to the historical data of the feature partition 111, determine the prefetch revenue 112 according to the historical data of the feature partition 112, and determine the prefetch revenue 112 according to the feature partition 1223
  • the historical data determines the prefetch profit 1223, and then compares the prefetch profit of each feature partition with the third threshold.
  • the feature partition 1223 lower than the third threshold does not perform prefetching operations in this cycle, and the feature partition 1223 higher than the third threshold
  • the data prefetching method provided by the present application can divide the first memory for storing prefetched data into multiple partitions, and adjust the multiple partitions according to the characteristics of the data flow in each partition, so that the The data stream features in each partition tend to be single, and multiple feature partitions are obtained, so as to separate the sequence stream from the mixed data stream.
  • multiple feature partitions are prefetched to avoid the need for multiple data streams. The mutual interference between the two leads to the problem of poor prefetching effect, which in turn improves the cache hit rate.
  • FIG. 6 is a schematic structural diagram of a processor 130 provided by the present application.
  • the processor 130 is applied to the prefetching system 100 shown in FIG. 1 .
  • the prefetching system 100 includes a processor 130 , a first memory 110 and a second memory 120.
  • the processor 130 may include a statistics module 112, a partition adjustment module 113, a prefetch module 121, and a storage module 122, wherein,
  • the statistics module 112 is configured to acquire statistical data of multiple partitions of the first memory, where the statistical data is used to indicate the historical situation of data streams included in each of the multiple partitions.
  • the prefetching module 121 is configured to determine a prefetching strategy according to statistical data of multiple partitions, where the prefetching strategy is used to indicate the location of the prefetched data in the first memory 110 and the size of the prefetched data.
  • the storage module 122 is configured to perform a prefetch operation according to a prefetch policy, and store the prefetched data in the second memory 120 .
  • the statistical data of the plurality of partitions includes the type of data streams included in each of the plurality of partitions
  • the partition adjustment module 113 is configured to perform an analysis on the plurality of partitions according to the statistical data of the plurality of partitions of the first memory 110. Adjust to obtain multiple feature partitions, wherein each feature partition in the multiple feature partitions includes at least one type of data stream; the partition adjustment module 113 is configured to determine the Prefetch strategy for each feature partition, where one feature partition corresponds to one prefetch strategy.
  • the partition adjustment module 113 is configured to, according to the statistical data of the multiple partitions, divide the partitions whose type and quantity of data streams in the multiple partitions are not lower than the first threshold, to obtain multiple characteristic partitions.
  • the statistical data includes data stream characteristics, wherein the data stream characteristics include one or more of the number of data streams, the length of the data streams, the type of the data streams, the IO size of the data streams, and the IO interval of the data streams.
  • the partition adjustment module 113 is configured to determine the similarity between the data flow features of the multiple partitions according to the statistical data of the multiple partitions of the first memory 110; the processor 130 is configured to adjust the similarity not lower than the second The thresholded partitions are merged to obtain multiple feature partitions.
  • the prefetching module 121 is used to predict the prefetching profit of each feature partition according to the historical data of the prefetching operation performed by each feature partition in the multiple feature partitions; the prefetching module 121 is used to select a prefetching profit higher than the A set of feature partitions with three thresholds, the set of feature partitions includes at least one feature partition; the storage module 122 is configured to perform a prefetch operation for each feature partition in the set of feature partitions according to the prefetch strategy, and store the prefetched data in the second memory 120 .
  • the read/write speed of the first memory 110 is lower than the read/write speed of the second memory 120; if the first memory 110 is a memory, the second memory 120 is a cache; or, if the first memory 110 is a magnetic disk In the case of time, the second memory 120 is a memory.
  • the processor 130 may correspond to executing the methods described in the embodiments of the present application, and each module and/or function in the processor 130 is to implement the corresponding flow of each method in FIG. 1 to FIG. 5, respectively, For brevity, details are not repeated here.
  • the processor provided by the present application can divide the first memory for storing prefetched data into multiple partitions, and adjust the multiple partitions according to the characteristics of the data flow in each partition, so that each The characteristics of the data flow in the partition tend to be single, and multiple characteristic partitions are obtained, so as to separate the sequential flow from the mixed data flow.
  • prefetching the multiple characteristic partitions can avoid the mutual interaction between the various data flows. Interference leads to the problem of poor prefetching, which in turn increases the cache hit rate.
  • FIG. 7 is a schematic structural diagram of a computing device 700 provided by the present application.
  • the computing device 700 may be the prefetching system 100 in FIGS. 1-6 .
  • the computing device 700 includes: a processor 130 , a communication interface 140 , a first memory 110 and a second memory 120 .
  • the processor 130 , the communication interface 140 , the first memory 110 and the second memory 120 may be connected to each other through the internal bus 150 , and may also communicate through other means such as wireless transmission.
  • the embodiments of the present application take the connection through a bus as an example, and the bus may be a PCIe bus or an EISA bus, or the like.
  • the bus 150 may be classified into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 7, but it does not mean that there is only one bus or one type of bus.
  • the processor 130 may be constituted by at least one general-purpose processor, such as a CPU, or a combination of a CPU and a hardware chip.
  • the above-mentioned hardware chip may be an ASIC, a PLD or a combination thereof.
  • the above PLD can be CPLD, FPGA, GAL or any combination thereof.
  • the processor 130 executes various types of digitally stored instructions, such as software or firmware programs stored in the first memory 110 or the second memory 120, which enable the computing device 700 to provide various services.
  • the first memory 110 is used for storing data and program codes, and the program codes can be controlled and executed by the processor 130 .
  • the program code may include one or more software modules, such as the code of the statistics module, the code of the partition adjustment module, etc., wherein the statistics module is used to obtain statistical data of multiple partitions of the first memory 110; the partition adjustment module is used to According to the statistical data of multiple partitions, adjust multiple partitions to obtain multiple feature partitions. Specifically, it can be used to perform step S310 and its optional steps in the embodiment of FIG. 3 , which will not be repeated here.
  • the second memory 120 is used to store data and program codes, and the program codes can be controlled and executed by the processor 130 .
  • the program code may include one or more software modules, such as the code of the prefetch module, the code of the storage module, etc., wherein the prefetch module is used to determine the prefetch strategy according to the statistical data of the multiple partitions; the storage module is used to determine the prefetch strategy according to the The prefetch strategy performs a prefetch operation and stores the prefetched data in the second memory, which can be specifically used to perform steps S320 to S330 in the embodiment of FIG. 3 and optional steps thereof, which will not be repeated here.
  • the first storage 110 may be any one of a magnetic disk, CD, USB, memory card, etc., wherein, the above-mentioned magnetic disk includes but is not limited to HDD, SSD, SSHD, RAID, etc., and the first storage 110 also It may be a combination of the above-mentioned various storage media, which is not specifically limited in this application.
  • the second memory 120 may be any one of ROM, RAM, DRAM, DDR, etc. The second memory 120 may also be a combination of the above-mentioned various storage media, which is not specifically limited in this application.
  • the first memory 110 may be any one of ROM, RAM, DRAM, DDR, etc.
  • the second memory 120 may be any one of SRAM, asynchronous SRAM, multi-threaded SRAM, synchronous SRAM, etc. species, which are not specifically limited in this application.
  • the communication interface 140 may be a wired interface (such as an Ethernet interface), an internal interface (such as a high-speed serial computer expansion bus (peripheral component interconnect express, PCIe) bus interface), a wired interface (such as an Ethernet interface), or a wireless interface (such as a cellular network interface or using a wireless local area network interface) to communicate with other devices or modules.
  • a wired interface such as an Ethernet interface
  • an internal interface such as a high-speed serial computer expansion bus (peripheral component interconnect express, PCIe) bus interface
  • PCIe peripheral component interconnect express
  • Ethernet interface such as an Ethernet interface
  • a wireless interface such as a cellular network interface or using a wireless local area network interface
  • this embodiment can be implemented by a general physical server, for example, an ARM server or an X86 server, or can be implemented by a virtual machine based on a general physical server combined with NFV technology.
  • a complete computer system with complete hardware system functions running in a completely isolated environment, for example, in this embodiment, can be implemented on a cloud computing infrastructure.
  • FIG. 7 is only a possible implementation manner of the embodiment of the present application.
  • the computing device 700 may further include more or less components, which is not limited here.
  • the computing device shown in FIG. 7 may also be a computer cluster composed of at least one physical server.
  • the first storage 110 and the second storage 120 are located on different physical servers.
  • the present application also provides a storage array.
  • the storage array may include one or more storage nodes, and the structure of each storage node may be as shown in FIG. 1 .
  • the storage array is specifically redundant arrays of independent disks. , RAID) as an example, the processor 1301 may be a RAID controller, the first storage 110 may be a logical hard disk block controlled by RAID, such as a logical solid state disk (LSSD), and the second storage 120 may be a RAID controller
  • the storage array may also be other types of storage arrays, which are not specifically limited in this application.
  • the structure of the storage array may also be as shown in FIG. 2 , and the storage array is used to implement the operation steps of the flow of the method shown in FIG. 3 to FIG. 5 .
  • the present application further provides a prefetching system.
  • the structure of the prefetching system is shown in FIG. 1 or FIG. 2 , and the prefetching system is used to implement the operation steps of the flow of the method shown in FIG. 3 to FIG. 5 .
  • Embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a processor, the method flow shown in FIGS. 3-5 is implemented.
  • the embodiment of the present application further provides a computer program product, when the computer program product runs on the processor, the method flow shown in FIG. 1 to FIG. 5 is realized.
  • the above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination.
  • the above-described embodiments may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes at least one computer instruction.
  • computer program instructions When computer program instructions are loaded or executed on a computer, the procedures or functions according to the embodiments of the present invention result in whole or in part.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • Computer instructions may be stored on or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server, or data center over a wire (e.g.
  • Coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless means to transmit to another website site, computer, server or data center.
  • the computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage node such as a server, a data center, or the like containing at least one set of available media.
  • Useful media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, high density digital video discs (DVDs), or semiconductor media.
  • the semiconductor media may be SSDs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Memory System Of A Hierarchy Structure (AREA)

Abstract

本申请提供了一种数据预取的方法、设备及系统，该方法应用于包括多种数据流的预取系统，该预取系统包括第一存储器和第二存储器，该方法包括以下步骤：获取第一存储器的多个分区的统计数据，统计数据用于指示多个分区中每个分区所包括的数据流的历史情况，根据多个分区的统计数据确定预取策略，预取策略用于指示预取数据的位置和预取数据的大小，根据预取策略执行预取操作。本申请提供的数据预取方法可根据每个分区内数据流的特征，针对性确定每个分区的预取策略，可以避免多种数据流之间互相干扰导致预取效果差的问题，提高缓存命中率。

Description

数据预取的方法、设备及系统 技术领域
本申请涉及计算机领域，尤其涉及一种数据预取的方法、设备及系统。
背景技术
随着计算机硬件的不断发展，中央处理器(central processing unit,CPU)主频已由MHz发展到了GHz，但是常用硬盘的存取速率不到100M/S。CPU和不同存储介质之间的访问速度差距，导致处理器需要花费大量的时间等待存储器数据的返回，使得存储系统的访问效率受限。
为了解决上述问题，业内通常利用高速缓存来加速存储性能，而高速缓存的控制器上通常设置有预取器(prefetching)，预取器可以从低速存储介质的数据流中，预测应用即将访问的预取数据，然后从数据流中获取该预取数据并将其提前存储至高速缓存中，从而提高存储系统的访问效率。但是，由于低速存储介质中的数据流的类型繁多，包括顺序流和随机流，其中，顺序流又包括长顺序流和短顺序流等等，但是当前的预取算法通常只能针对顺序流进行预取，并且各种类型的数据流之间还会互相干扰，导致预取器的预取能力受限。
发明内容
本申请提供了一种数据预取的方法、设备及系统，用于解决预取器的预取能力受限的问题。
第一方面，提供了一种数据预取方法，该方法应用于包括多种数据流的预取系统，预取系统包括第一存储器和第二存储器，该方法包括以下步骤：获取第一存储器的多个分区的统计数据，统计数据用于指示多个分区中每个分区所包括的数据流的历史情况，根据多个分区的统计数据确定预取策略，预取策略用于指示预取数据的位置和预取数据的大小，根据预取策略执行预取操作，将预取数据从第一存储器中预取至第二存储器。
该方法通过将第一存储器划分为多个分区，然后根据每个分区内数据流的特征，针对性地确定每个分区的预取策略，可以避免多种数据流之间互相干扰导致预取效果差的问题，提高缓存命中率。
在一种可能的实现方式中，统计数据可包括数据流特征，其中，数据流特征包括数据流数目、数据流长度、数据流的类型、数据流IO大小、数据流IO间隔中的一种或者多种。
可选地，统计数据还可包括每个分区所包括的数据流的数据流信息，且上述数据流特征是根据数据流信息获得的。其中，数据流包括上层应用下发的多个IO，通常地，一个数据流是同一个应用或一类应用生成的用于传输数据或通知的IO的集合。数据流信息包括但不限于每个IO的IO偏移、IO长度和IO时间中的一种或者多种，IO偏移可以是应用所访问的数据所在的地址信息，比如访问地址或者在数据流中的偏移值，IO长度可以是该数据的数据长度，IO时间指的是访问时间，数据流特征可包括但不限于数据流中IO的数目、数据流的类型、数据流的长度、数据流中每个IO的大小、数据流IO间隔等等。其中，数据流特征可以是根据数据流信息确定的。
可选地，上述统计数据可以以特征向量(x 1,x 2,...,x m)的形式进行表示，其中，该特征向量 中的每个元素可代表一个数据流特征，具体可以是该数据流特征的数值，也可以是该数据流特征对应的等级、档位、代码等等，本申请不作具体限定。比如x 1可以表示数据流的数目，若数据流数目为1000，那么x 1＝100。比如x 1表示数据流的数目所处的区间，且数据流的数目小于100时所处的区间为1，数据流的数目在100～200之间时对应的区间为2，以此类推，那么对于数据流数目为150的数据流来说，其特征向量中的x 1＝2。应理解，上述举例用于说明，本申请不作具体限定。
举例来说，假设统计数据用五维向量(x 1,x 2,x 3,x 4,x 5)表示，其中，x 1用于表示数据流的数目，如数据流数目小于等于100，对应维度设为1，超过100小于等于200维度设为2，以此类推。x 2用于表示数据流的类型数量，数量不高于5对应值为1，数量在5-10之间对应值为2，以此类推。x 3用于表示数据流长度，小于1MB对应维度设为1，超过1MB小于2MB对应维度设为2，一次类推。x 4用于数据流IO大小，小于64KB对应维度设为1，大于64KB小于128KB对应维度设为2，以此类推。x 5用于表示数据流IO间隔，小于1ms对应维度设为1，大于1ms小于2ms对应维度设为2，以此类推。假设统计分区1的数据流数目为150，数据流的类型有3种，数据流长度为1.5MB，数据流IO大小为120KB，数据流IO间隔为0.5ms，那么上述统计分区1的统计数据为(2,1,2,2,1)。应理解，上述举例用于说明，本申请不作具体限定。
多个分区的统计数据包括的内容越丰富，可以提高分区调整的效率和预取策略的准确度，同时，为了避免大量的统计数据将会占用系统资源，降低处理效率，统计数据用特征向量进行表示，便于后续数据流的类型的统计、数据流的特征相似度的计算等等，从而提高处理效率。
在另一种可能的实现方式中，根据多个分区的统计数据确定预取策略时，可以先根据多个分区的统计数据，对多个分区进行调整，获得多个特征分区，其中，多个特征分区中的每个特征分区包括至少一种类型的数据流，再根据多个特征分区的统计数据，确定多个特征分区中每个特征分区的预取策略，其中，每个特征分区对应一个预取策略。
可选地，可以先根据预设的划分规则对第一存储器的存储空间进行划分，获得上述多个分区，然后再根据每个分区所包括的数据流的类型，对多个分区进行调整，获得多个特征分区。预设的划分规可包括但不限于按磁盘进行分区、按存储空间大小进行分区等等，举例来说，如果第一存储器110包括多块磁盘，那么可以一块盘分为一个分区，如果第一存储器110包括1块磁盘，那么可以将该磁盘每64MB的连续存储空间划分为一个分区，或者每138GB的连续存储空间划分为一个分区等等，具体可根据实际情况预设划分规则，本申请不作具体限定。
将包括混合数据流的第一存储器划分为多个分区，并根据每个分区内数据流的特征，对多个分区进行调整，可以使得每个特征分区内的数据流特征趋于单一，获得多个特征分区，从而将顺序流从混合数据流中分离出来，此时再对多个特征分区进行预取，可以根据每个特征分区内单一的数据流特征，针对性地确定预取策略，避免多种数据流之间互相干扰导致预取效果差的问题，提高缓存命中率。
在另一种可能的实现方式中，根据多个分区的统计数据，对多个分区进行调整，获得多个特征分区时，可根据多个分区的统计数据，将多个分区中数据流的类型数量不低于第一阈值的分区进行拆分，获得多个特征分区。
可选地，数据流的类型数量高于第一阈值的分区在进行拆分时，可以被拆分为2个分区，也可以被拆分为2个以上的分区，拆分规则可以是随机拆分，也可以是平均拆分，还可以是 根据不同类型的数据流的分布比例拆分，本申请不对拆分规则进行限定。举例来说，若分区1的数据流的类型数量为5，分区2的数据流的类型数量为2，第一阈值为3，那么可以将分区1进行拆分获得特征分区11和特征分区12，分区2不进行拆分，从而将分区1和分区2拆分为特征分区11、特征分区12和特征分区2。应理解，上述举例用于说明，本申请不对此进行限定。
在多次将多个分区中数据流的类型数量不低于第一阈值的分区进行拆分后，可以使得特征分区内的数据流的类型数量趋于单一，从而将顺序流从混合数据流中分离出来，避免多种数据流之间互相干扰导致预取效果差的问题，提高缓存命中率。
在另一种可能的实现方式中，根据多个分区的统计数据，对多个分区进行调整，获得多个特征分区时，可根据多个分区的统计数据，确定多个分区的数据流特征之间的相似度，然后将相似度不低于第二阈值的分区进行合并，获得多个特征分区。
可选地，上述两个分区是逻辑性连续的两个相邻分区，也就是说，逻辑地址的编址方式连续的两个分区可以合并，比如相邻分区的逻辑区块地址(logical block address,LBA)可以是连续的，即LBA＝0的逻辑区块为分区1，LBA＝2的逻辑区块为分区2，那么分区1和分区2即为逻辑性连续的两个相邻分区，而分区1对应的物理地址和分区2对应的物理地址可以是连续的，也可以是不连续的，本申请不作具体限定。
可选地，统计数据的相似度可以根据特征向量之间的距离确定，比如欧氏距离(eucledian distance)、曼哈顿距离(manhattan distance)、余弦相似度(cosine similarity)、明式距离(minkowski distance)等等，本申请不作具体限定。
将数据流特征之间的相似度不低于第二阈值的分区进行合并，可以避免同一类型的数据流被分散在多个分区，使得同一类型的数据流可以统一确定预取策略，降低计算资源消耗，提高处理效率。
在另一种可能的实现方式中，可将多个分区中数据流的类型数量超过第一阈值的分区进行拆分，同时对多个分区中数据流的特征相似度低于第二阈值的分区进行合并，从而获得多个特征分区。比如多个分区包括分区1、分区2和分区3，分区1的数据流的类型数量超过第一阈值，分区2和分区3的数据流的类型数量低于第一阈值，分区1和分区2的统计数据相似度低于第二阈值，分区1和分区3的数据相似度低于第二阈值，分区2和分区3的统计数据相似度高于第二阈值，那么可以将分区1拆分为特征分区11和特征分区12，分区2和分区3合并为特征分区23，应理解，上述拆分可以是平均拆分，也可以是随机拆分，本申请不作具体限定，并且，上述举例用于说明，本申请不作具体限定。
可选地，在获取第一存储器的多个分区的统计数据之前，可先周期性的对多个分区进行调整，使得多个分区内的数据流类型趋于单一后，再获取第一存储器的多个特征分区的统计数据，若此时多个特征分区内的数据流发生了变化，可以再次调整多个特征分区后，确定每个特征分区的预取策略，执行预取操作。
由于数据流中包含的IO是应用下发的IO，应用下发IO的时间并不是固定的，因此可能会出现经过多个调整周期后，多个特征分区内的数据流特征已趋于单一，但是开始数据预取时，个别特征分区内的数据流发生了变化，导致个别特征分区内的数据流特征不再趋于单一，因此在确定每个特征分区对应的预取策略之前，可再对多个特征分区进行调整，避免由于数据流发生变化导致特征分区内包括多种数据流，从而提高预取准确率，提高缓存命中率。
在另一种可能的实现方式中，根据预取策略执行预取操作时，可先根据多个特征分区中每个特征分区执行预取操作的历史数据预测每个特征分区的预取收益，选择预取收益高于第 三阈值的特征分区集合，特征分区集合包括至少一个特征分区，按照预取策略分别执行特征分区集合中每个特征分区的预取操作。
可选地，该预取策略可包括预取数据在数据流中的预取偏移值以及预取长度，还可包括其他用以供处理器从数据流中读取预取数据的相关信息，比如预取数据的读取时间，预取数据的地址等等，本申请不对此进行限定，根据预取策略，处理器可以从第一存储器中读取预取数据，并将其存储至第二存储器中。预取策略可以是根据每个特征分区的统计数据，使用预取算法获得的，预取算法可以是Tap预取算法(Table-based Prefetching for Storage Caches)、AMP预取算法(Adaptive Multi-stream Prefetching in a Shared Cache)等等，本申请不对采用何种预取算法进行限定。
可选地，每个特征分区的预取收益可包括预取命中率、预取浪费率和预取IO数量中的一种或者多种，其中，预取命中率x包括预取数据被应用从第二存储器中读取的概率，预取浪费率y包括预取数据未被应用从第二存储器中读取的概率，预取IO数量z即为预取数量的IO数目。具体实现中，预取收益可根据每个特征分区的统计数据确定，根据统计数据可计算每个特征分区的历史预取命中率、历史IO数据量、历史预取浪费率等等，基于每个特征分区的历史预取命中率、历史IO数据量、历史预取浪费率可确定预取收益，比如预取收益＝αx+βy+γz，其中，α是预取命中率x的权重，β是预取浪费率y的权重，γ是预取IO数量z的权重，α、β和γ可以是经验值，该经验值还可以根据预取收益的准确度进行调整。上述公式用于举例，本申请不作具体限定。
具体实现中，上述预取IO数量可以根据统计数据获得，上述预取命中率x和预取浪费率y可以根据每个特征分区的历史数据，结合统计数据确定。其中，每个特征分区的历史数据包括每个特征分区的历史统计数据，历史统计数据可包括当前周期之前的所有周期内的历史数据流信息和历史数据流特征，历史数据流信息可包括但不限于IO偏移、IO长度和IO时间，历史数据流特征可包括但不限于IO数量、数据流长度、IO间隔等等，并且，历史统计数据可以是每个特征分区上一周期的历史统计数据，也可以是上两个周期的历史统计数据，还可以是本周期之前所有周期的历史统计数据，本申请不作具体限定。
举例来说，当前周期采集的统计信息显示分区1的数据流包括应用下发的1个IO，且该IO的IO偏移为100，长度为50，若历史统计数据显示上一周期该应用下发的历史IO，其实际访问的数据偏移值为0，长度为100，即读取了数据流中偏移值从0～100的一段数据，根据预取算法(比如AMP预取算法)确定预取策略为：预取偏移值为100，预取长度为100，若使用该预取测量，将会读取偏移值从100～200的一段数据，而当前周期采集的统计信息显示，实际需求为偏移值从100～150的一段数据，那么即可预测得到预取命中率为50％，预取浪费率为50％，预取IO数量为1，根据预取收益＝αx+βy+γz可以获得预取收益，根据预取收益与第三阈值的大小，确定特征分区集合。应理解，上述举例用于说明，本申请不作具体限定。
通过在执行预取操作之前，确定每个特征分区的预取收益，对预取收益较低的分区取消预取操作，可以进一步提高预取的命中率，提高存储系统的访问效率。
在另一种可能的实现方式中，第一存储器的读写速度低于第二存储器的读写速度；在第一存储器为内存的情况下，第二存储器为缓存；或者，在第一存储器为磁盘时的情况下，第二存储器为内存。
在第一存储器为内存，第二存储器为缓存时，本申请提供的方法可以将内存中的数据提前预取至缓存中，上层应用可从缓存中快速访问该数据，提高内存的访问效率；在第一存储器为磁盘，第二存储器为内存的情况下，本申请提供的方法可以将磁盘中的数据提前预取至 内存中，上层应用可从内存中快速访问该数据，提高磁盘的访问效率。
第二方面，提供了一种预取系统，该预取系统包括处理器、第一存储器和第二存储器，其中，处理器，用于获取第一存储器的多个分区的统计数据，统计数据用于指示多个分区中每个分区所包括的数据流的历史情况；处理器，用于根据多个分区的统计数据确定预取策略，预取策略用于指示预取数据在第一存储器中的位置和预取数据的大小；以及，根据预取策略执行预取操作，将预取数据存储至第二存储器。
该系统通过将第一存储器划分为多个分区，使得CPU可根据每个分区内数据流的特征，针对性地确定每个分区的预取策略，将预取数据从第一存储器中预取至第二存储器，该系统可以避免多种数据流之间互相干扰导致预取效果差的问题，提高缓存命中率。
在一种可能的实现方式中，多个分区的统计数据包括多个分区中每个分区所包括的数据流的类型，处理器用于根据第一存储器的多个分区的统计数据，对多个分区进行调整，获得多个特征分区，其中，多个特征分区中的每个特征分区包括至少一种类型的数据流；处理器用于根据多个特征分区的统计数据，确定多个特征分区中每个特征分区的预取策略，其中，一个特征分区对应一个预取策略。
在另一种可能的实现方式中，处理器用于根据多个分区的统计数据，将多个分区中数据流的类型数量不低于第一阈值的分区进行拆分，获得多个特征分区。
在另一种可能的实现方式中，统计数据包括数据流特征，其中，数据流特征包括数据流数目、数据流长度、数据流的类型、数据流IO大小、数据流IO间隔中的一种或者多种。
在另一种可能的实现方式中，处理器用于根据第一存储器的多个分区的统计数据，确定多个分区的数据流特征之间的相似度；处理器用于将相似度不低于第二阈值的分区进行合并，获得多个特征分区。
在另一种可能的实现方式中，处理器用于根据多个特征分区中每个特征分区执行预取操作的历史数据预测每个特征分区的预取收益；处理器用于选择预取收益高于第三阈值的特征分区集合，特征分区集合包括至少一个特征分区；处理器用于按照预取策略分别执行特征分区集合中每个特征分区的预取操作，将预取数据存储至第二存储器。
在另一种可能的实现方式中，第一存储器的读写速度低于第二存储器的读写速度；在第一存储器为内存的情况下，第二存储器为缓存；或者，在第一存储器为磁盘时的情况下，第二存储器为内存。
第三方面，提供了一种处理器，该处理器应用于包括多种数据流的预取系统中，该预取系统包括处理器、第一存储器和第二存储器，其中，处理器包括用于执行第一方面或第一方面任一种可能实现方式中的数据预取方法的各个模块。
第四方面，提供了一种计算机程序产品，当其在计算机上运行时，使得计算机执行上述各方面所述的方法。
第五方面，提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述各方面所述的方法。
第六方面，提供了一种计算设备，该计算设备包括处理器，该处理器用于执行上述各方面描述的方法。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1是本申请提供的一种预取系统的结构示意图；
图2是本申请提供的一种预取系统在分布式场景中的结构示意图；
图3是本申请提供的一种预取方法的步骤流程示意图；
图4是本申请提供的一种应用场景下的多个分区调整为多个特征分区的流程示意图；
图5是本申请提供的一种应用场景下执行预取操作的步骤流程示意图；
图6是本申请提供的一种处理器的结构示意图；
图7是本申请提供的一种计算设备的结构示意图。
具体实施方式
为了便于理解本发明的技术方案，首先，对本发明涉及的“预取”的应用场景进行解释说明。
随着计算机硬件的不断发展，中央处理器(central processing unit,CPU)主频已由MHz发展到了GHz，但是常用硬盘的存取速率不到100M/S。并且根据摩尔定律，处理器的性能每隔两年可以翻一倍，而磁盘这类机械电子设备，存取速率每年仅增加约8％，导致CPU和不同存储介质之间的访问速度差距越来越大，处理器需要花费大量的时间等待存储器数据的返回，存储器性能严重限制了CPU性能的发挥。
为了解决上述问题，业内通常利用高速缓存来加速存储性能。其中，高速缓存的访问速度高但是容量小，因此高速缓存的控制器上通常设置有预取器(prefetching)，预取器可以在上层应用访问预取数据之前，提前将该预取数据从大容量的低速存储介质中读取至高速缓存中，使得上层应用可以从高速缓存中快速读取该预取数据。应理解，一个优秀的预取器能够准确预测出应用即将访问的预取数据，有效提高缓存命中率，极大提升存储器的访问速度。
由于上层应用下发的应用输入/输出(input/output,IO)通常具备不同的IO特征，因此低速存储介质中的数据流为混合数据流，其中，该混合数据流包括多个上层应用下发的应用IO操作，根据应用IO的不同IO特征，混合数据流包括多种类型的数据流，比如顺序流、随机流等等，顺序流还可进一步包括长顺序流、短顺序流等等。具体地，顺序流可包括多个同一时间顺序请求的数据请求，长顺序流的数据量远大于短顺序流，示例性的，顺序流可以包括数据库执行大量的查询IO、流媒体服务IO等等，以数据库中的查询IO为例，长顺序流可以包括对整个表进行扫描查询的多个IO，短顺序流可包括用户指定查询的多个IO，比如对半个表或者筛选出的部分符合条件的表项进行查询。随机流可以包括多个上层应用随机下发的应用IO，比如万维网(world wide web,Web)服务请求，邮箱(mail)服务请求等。而在上述预取应用场景中，当前的预取算法通常只能针对混合数据流中的顺序流实现预取操作，而各种类型的数据流之间互相干扰，导致预取器的预取能力受限。
为了解决上述预取器的预取能力受限的问题，本申请提供了一种预取系统，如图1所示，图1是本申请提供的一种预取系统100，该预取系统100可以将包括混合数据流的第一存储器划分为多个分区，并根据每个分区内数据流的特征，对多个分区进行调整，使得每个分区内的数据流特征趋于单一，获得多个特征分区，从而将顺序流从混合数据流中分离出来，此时再对多个特征分区进行预取，可以避免多种数据流之间互相干扰导致预取效果差的问题，进而提高缓存命中率。
预取系统100可以部署于计算设备内部，计算设备具体可以是物理服务器，比如X86服务器、ARM服务器等等；计算设备也可以是基于通用的物理服务器结合网络功能虚拟化(network functions virtualization,NFV)技术实现的虚拟机(virtual machine,VM)，虚拟机指通过 软件模拟的具有完整硬件系统功能的、运行在一个完全隔离环境中的完整计算机系统，比如云计算中的虚拟设备；计算设备还可以是用于存储数据的存储类设备，比如存储阵列、全闪存高端存储等等，本申请不作具体限定。
预取系统100还可以部署在分布式存储系统中，比如Fusion storage、Hadoop分布式文件系统(hadoop distributed file dystem,HDFS)等，具体可以是将预取系统100部署在分布式存储系统中的某个存储节点中，也可以是数据区域系统100的部分单元部署在存储节点1，部分单元模块部署在存储节点2，本申请不作具体限定。
其中，预取系统100存在多种划分方式，示例性地，如图1所示，该预取系统100可包括第一存储器110、第二存储器120以及处理器130。其中，第一存储器110、第二存储器120以及处理器130之间可以通过总线相互连接，比如快捷外围部件互连标准(peripheral component interconnect express,PCIe)总线或扩展工业标准结构(extended industry standard architecture,EISA)总线等，也可以通过无线传输等其他手段实现通信，比如以太网(Ethernet)，本申请不作具体限定。应理解，图1仅为一种示例性的划分方式，各个模块单元之间可以合并或者拆分为更多或更少的模块单元，本申请不作具体限定，且图1中所示的系统和模块之间的位置关系也不构成任何限制。
第一存储器110和第二存储器120均为存储器。其中，第一存储器110中的数据可被提前预取至第二存储器120，以使处理器130从第二存储器120中读取该数据。其中，该数据指的是通过预取算法确定的应用即将访问的数据，该数据可以是一个数据，也可以是一组数据构成的数据集合。可选地，第一存储器110的访问速率可以低于第二存储器120，这样，将第一存储器110中的预取数据预取至第二存储器120，应用可从访问速率更高的第二存储器120中读取该预取数据，从而提高第一存储器110的访问速率。
应理解，在不同的应用场景下，第一存储器110和第二存储器120的存储器类型不同。示例性地，下面对不同应用场景下第一存储器110和第二存储器120的存储器类型进行举例说明。
在一应用场景中，当图1所示的预取系统100为一台计算设备时，第一存储器110可以是外存，第二存储器120可以是该计算设备中的内存(main memory)，其中，外存指的是除内存和处理器的缓存外的存储器。在该应用场景下，将应用即将访问的预取数据X从第一存储器110中提前预取至第二存储器120中，应用可以从第二存储器120中读取该预取数据X，由于内存的访问速率远高于外存，因此使用本申请提供的预取系统可以提高CPU的访问速率。具体实现中，第一存储器可以是磁盘(disk)、光盘(compact disc,CD)、U盘(universal serial bus,USB)、存储卡(memory card)等等中任意一种，其中，上述磁盘包括但不限于机械硬盘(hard disk drive,HDD)、固态硬盘(solid-state drive,SSD)、混合硬盘(solid state hybrid drive,SSHD)、独立冗余磁盘阵列(redundant arrays of independent disks,RAID)等等，第一存储器110还可以是上述各种存储介质的组合，本申请不作具体限定。第二存储器120可以是只读存储器(read-only memory,ROM)、随机存储器(random access memory,RAM)、动态随机存储器(dynamic random-access memory,DRAM)、双倍速率同步动态随机存储器(double data rate SDRAM,DDR)等等中任意一种，第二存储器120还可以是上述各种存储介质的组合，本申请不作具体限定。
在另一应用场景下，当图1所示的预取系统100为一台计算设备时，第一存储器110可以是内存，第二存储器是该计算设备的缓存(cache)。在该应用场景下，将应用即将访问的数据X从第一存储器110提前预取至第二存储器120中，应用可以从第二存储器120中读取该 数据X，由于缓存的访问速度远高于内存，因此使用本申请提供的预取系统可以提高CPU的访问速率。具体实现中，第一存储器可以是ROM、RAM、DRAM、DDR等等中任意一种，第二存储器120可以是静态随机存取存储器(static random-access memory,SRAM)、异步SRAM(asynchronous SRAM,Async SRAM)、多线程SRAM(synchronous SRAM,Sync SRAM)，同歩SRAM(pipelined brst SRAM,PB SRAM)等等中任意一种，本申请不作具体限定。
在再一应用场景下，预取系统100包括多个计算设备，第一存储器和第二存储器可以分别部署于不同的计算设备上，示例性的，如图2所示，图2是分布式场景下本申请提供的预取系统100的结构示意图，其中，第一存储器110可以是存储节点1的外存，第二存储器可以是存储节点2的内存，数据X存储于存储节点1的第一存储器110中，统计模块112和分区调整模块113部署于存储节点1的处理器1301中，预取模块121和存储模块122部署于存储节点2的处理器1302中。在该应用场景下，将应用即将访问的数据X从存储节点1的第一存储器提前预取至存储节点2的第二存储器120中，这样，当存储节点2中的应用访问数据X时，相比于通过网络从第一存储节点的外存(第一存储器110)中远程读取数据X，本申请提供的预取系统可以从第二存储器120(本地内存)中读取该数据X，能够提高CPU的访问速率。具体实现中，存储节点1中的第一存储器可以是磁盘、CD、U盘、存储卡等等，上述磁盘包括但不限于HDD、SSD、SSHD、RAID等等，还可以是上述各种外存的组合，本申请不作具体限定。第二存储器120可以是ROM、RAM、DRAM、DDR等等，还可以是上述各种内存的组合，本申请不作具体限定。
应理解，图2用于举例说明，第一存储器还可以是存储节点1的内存，第二存储器是存储节点2的缓存，或者，第一存储器还可以是存储节点1中的内存，第二存储器是存储节点2的内存。并且，在该应用场景下，数据X从存储节点1的第一存储器110中预取至了存储节点2的内存中，因此处理器1301和处理器1302显示了该应用场景下参与预取的部分单元模块，具体实现中，图2中的存储节点1和存储节点2还可包括更多或更少的单元模块，比如存储节点1的处理器1301中也可包括预取模块和存储模块，用于将预取数据存储至存储节点1的内存中，存储节点2的处理器1302也可包括统计模块和分区调整模块，用于对存储节点2中的外存进行分区调整和统计数据的获取，本申请不对此进行限定。
进一步地，第一存储器110和处理器130存在多种划分方式，示例性的，如图1所示，第一存储器110可包括多个分区111，处理器130可包括统计模块112、分区调整模块113、预取模块121以及存储模块122。
多个分区111用于存储数据。其中，多个分区111可以如图1和图2所示的分区1～分区N所示，N为正整数。
统计模块112用于获取第一存储器的多个分区的统计数据，该统计数据用于指示多个分区中每个分区中数据流的历史情况。参考前述内容可知，数据流包括上层应用下发的多个应用IO，数据流的历史情况可以是数据流中每个IO的历史信息，比如每个IO的访问偏移、访问时间和访问大小等等，举例来说，数据流中的应用IO可以是数据库执行大量的查询请求，该数据流的历史情况可以包括每个查询请求的请求地址、请求时间以及请求数据的大小等等。通常地，一个数据流是同一个应用或一类应用生成的用于传输数据或通知的IO的集合。
分区调整模块113用于根据上述统计数据，对多个分区进行调整获得多个特征分区，以使调整后的多个特征分区中的每个特征分区包括至少一种类型的数据流，其中，数据流的类型至少包括长顺序流、短顺序流和随机流。可选地，分区调整模块113在对多个分区111进行调整时，可以将数据流的类型数量高于第一阈值的分区进行拆分，也可以将数据流特征的 相似度高于第二阈值的相邻分区进行合并，获得多个特征分区，其中，相邻分区指的是逻辑性连续的两个分区，例如，逻辑地址的编址方式连续的两个分区可以合并。进一步地，分区调整模块113可对上述多个分区111进行调整后，统计模块112再次获取上述多个特征分区的统计数据，分区调整模块113再次对上述多个特征分区进行调整，循环往复，直至每个特征分区中的数据流特征趋于单一。
具体实现中，多个分区111可以先根据预设的划分规则进行划分，比如一块盘分为一个分区，或者每64MB的连续存储空间划分为一个分区，或者每138GB的连续存储空间划分为一个分区等等，具体可根据实际情况预设划分规则，本申请不作具体限定。举例来说，先根据一块盘分为一个分区的划分规则，将第一存储器110划分为3个分区，分区1为磁盘A，分区2为磁盘B，分区3为磁盘C，通过统计模块112获得是3个分区的统计数据后，确定磁盘A包括长顺序流和短顺序流，磁盘B包括长顺序流和随机流，磁盘C包括短顺序流和随机流，那么分区调整模块113可以对上述3个分区进行调整，获得特征分区1’包括磁盘A和磁盘B中用于存储长顺序流的存储空间，特征分区2’包括磁盘A和磁盘C中用于存储短顺序流的存储空间，特征分区3’包括磁盘B和磁盘C中用于存储随机流的存储空间。应理解，上述举例用于说明，本申请不作具体限定。
可选地，分区调整模块113可在预取系统100启用之前，对多个分区进行多次调整，使得每个特征分区中的数据流特征趋于单一后，再启用该预取系统100。而在预取系统100启用后，统计模块112获取每个特征分区的统计数据，若此时多个特征分区内的数据流发生了变化，导致一些特征分区内的数据流特征不再趋于单一，此时分区调整模块113可再次调整这些特征分区，对数据流特征不再趋于单一的特征分区进行拆分或者合并后，预取模块121再确定每个特征分区的预取策略，从而避免由于数据流发生变化导致特征分区内包括多种数据流，从而提高预取准确率，提高缓存命中率。
预取模块121用于根据上述多个特征分区的数据流特征，确定每个特征分区的预取策略，该预取策略用于指示预取数据的位置和预取数据的大小，以使存储模块122根据该预取策略对每个特征分区执行预取操作，从特征分区中读取预取数据，并存储至第二存储器120的存储模块中。
具体实现中，该预取模块121可包括多个预取子模块，每个预取子模块对应一个特征分区，仍以上述例子为例，特征分区1’包括长顺序流，特征分区2’包括短顺序流，特征分区3’包括随机流，那么预取模块121可包括预取子模块1、预取子模块2以及预取子模块3，其中，预取子模块1可根据特征分区1’中长顺序流的数据特征，确定长顺序流预取策略，以使存储模块122从特征分区1’中读取预取数据，并存储至存储模块中，以此类推，预取子模块2可确定特征分区2’的预取策略，预取子模块3可确定特征分区3’的预取策略，这里不一一展开赘述。并且，多个预取子模块可并行确定各个特征分区的预取策略，提高预取效率。
可选地，该预取模块121中的每个预取子模块在确定每个特征分区的预取策略之后，可先根据每个特征分区执行预取操作的历史数据预测每个特征分区的预取收益，选择预取收益高于第三阈值的特征分区集合，按照预取策略分别执行特征分区集合中每个特征分区的预取操作，其中，特征分区集合包括至少一个特征分区。若预取收益小于阈值的特征分区，则不使用预取策略对该分区进行预取，从而进一步提高预取命中率，提高存储系统的访问速率。
其中，上述每个特征分区的预取收益可包括预取命中率、预取浪费率和预取IO数量中的一种或者多种。预取收益可根据每个特征分区的统计数据确定，根据统计数据可计算每个特 征分区的历史预取命中率、历史IO数据量、历史预取浪费率等等，基于每个特征分区的历史预取命中率、历史IO数据量、历史预取浪费率可确定预取收益，比如预取收益＝αx+βy+γz，其中，α是预取命中率x的权重，β是预取浪费率y的权重，γ是预取IO数量z的权重，α、β和γ可以是经验值，该经验值还可以根据预取收益的准确度进行调整。上述公式用于举例，本申请不作具体限定。
具体实现中，处理器130可以由至少一个通用处理器构成，例如中央处理器，或者CPU和硬件芯片的组合，上述硬件芯片可以是ASIC、PLD或其组合，上述PLD可以是CPLD、FPGA、GAL或其任一组合，本申请不作具体限定。具体实现中，当预取系统100部署于单个计算设备上时，处理器130可以是该计算设备的CPU；当预取系统100部署于分布式系统时，例如图2所示的分布式存储系统，处理器130可以是存储节点2中的处理器1302，即处理器130为第二存储器120所在的存储节点中的CPU，图2的举例用于说明，本申请不作具体限定。
处理器130可执行各种类型的程序代码，以使预取系统100实现各种功能。具体地，第一存储器110可存储有统计模块112和分区调整模块113的程序代码，第二存储器120可存储有预取模块121和存储模块122的程序代码。处理器130可调用第一存储器110中统计模块112的程序代码，以获取多个分区111的统计数据，然后调用分区调整模块113的程序代码，以实现对多个分区111的调整，从而获得多个特征分区，然后调用第二存储器120中预取模块121的代码，确定多个特征分区的预取策略，然后调用存储模块122的代码，根据预取策略对从多个特征分区中读取预取数据，并将预取数据存储至第二存储器120的存储模块中。
可以理解的是，上述预取策略可以根据特征分区内的数据流特征使用适用的预取算法确定出每个特征分区的预取策略，例如，使用Tap预取算法(Table-based Prefetching for Storage Caches)、AMP预取算法(Adaptive Multi-stream Prefetching in a Shared Cache)等等，本申请不作具体限定。
综上可知，本申请提供的预取系统，可以将包括混合数据流的第一存储器划分为多个分区，并根据每个分区内数据流的特征，对多个分区进行调整，使得每个分区内的数据流特征趋于单一，获得多个特征分区，从而将顺序流从混合数据流中分离出来，此时再对多个特征分区进行预取，可以根据每个特征分区内单一的数据流特征，针对性地确定预取策略，可以避免多种数据流之间互相干扰导致预取效果差的问题，提高缓存命中率。上述调整过程可以在执行数据预取前执行一次上述调整过程，对多个分区进行调整；也可以在预取过程中根据数据流的情况采用多次上述调整过程，对多个分区实现动态调整。
图3是本申请提供的一种数据预取方法的步骤流程示意图，其中，该数据预取方法可应用于图1和图2所示的预取系统100中，该预取系统100包括多种数据流(即混合数据流)，如图3所示，该方法可包括以下步骤：
S310：获取第一存储器110的多个分区111的统计数据，其中，该统计数据用于指示每个分区所包括的数据流的历史情况。
其中，上述多个分区111可以先根据预设的划分规则对第一存储器的存储空间进行划分，举例来说，如果第一存储器110包括多块磁盘，那么可以一块盘分为一个分区，如果第一存储器110包括1块磁盘，那么可以将该磁盘每64MB的连续存储空间划分为一个分区，或者每138GB的连续存储空间划分为一个分区等等，具体可根据实际情况预设划分规则，本申请不作具体限定，应理解，多个分区111的详细描述可参见前述图1和图2实施例中的相关描 述，这里不再重复赘述。
在一种可能的实现方式中，上述统计数据可包括每个分区所包括的数据流的数据流信息和数据流特征，其中，数据流包括上层应用下发的多个IO，数据流信息包括但不限于每个IO的IO偏移、IO长度和IO时间中的一种或者多种，IO偏移可以是应用所访问的数据所在的地址信息，比如访问地址或者在数据流中的偏移值，IO长度可以是该数据的数据长度，IO时间指的是访问时间，数据流特征可包括但不限于数据流中IO的数目、数据流的类型、数据流的长度、数据流中每个IO的大小、数据流IO间隔等等。其中，数据流特征可以是根据数据流信息确定的。
具体实现中，上述统计数据可以以特征向量(x 1,x 2,...,x m)的形式进行表示，其中，该特征向量中的每个元素可代表一个数据流特征，具体可以是该数据流特征的数值，也可以是该数据流特征对应的等级、档位、代码等等，本申请不作具体限定。比如x 1可以表示数据流的数目，若数据流数目为1000，那么x 1＝100。比如x 1表示数据流的数目所处的区间，且数据流的数目小于100时所处的区间为1，数据流的数目在100～200之间时对应的区间为2，以此类推，那么对于数据流数目为150的数据流来说，其特征向量中的x 1＝2。应理解，上述举例用于说明，本申请不作具体限定。
举例来说，假设统计数据用五维向量(x 1,x 2,x 3,x 4,x 5)表示，其中，x 1用于表示数据流的数目，如数据流数目小于等于100，对应维度设为1，超过100小于等于200维度设为2，以此类推。x 2用于表示数据流的类型数量，数量不高于5对应值为1，数量在5-10之间对应值为2，以此类推。x 3用于表示数据流长度，小于1MB对应维度设为1，超过1MB小于2MB对应维度设为2，一次类推。x 4用于数据流IO大小，小于64KB对应维度设为1，大于64KB小于128KB对应维度设为2，以此类推。x 5用于表示数据流IO间隔，小于1ms对应维度设为1，大于1ms小于2ms对应维度设为2，以此类推。假设统计分区1的数据流数目为150，数据流的类型有3种，数据流长度为1.5MB，数据流IO大小为120KB，数据流IO间隔为0.5ms，那么上述统计分区1的统计数据为(2,1,2,2,1)。应理解，上述举例用于说明，本申请不作具体限定。
在一种可能的实现方式中，步骤S310可以是处理器130调用第一存储器110中的统计模块112的代码后，获得的各个分区的统计数据。具体实现中，处理器130可以周期性的获取各个分区统计数据，该统计数据是当前周期下各个分区的统计数据，比如每隔2秒获取一次当前各个分区的统计数据，本申请不对周期的具体时间进行限定。
S320：根据多个分区的统计数据确定预取策略，该预取策略用于指示预取数据的位置和预取数据的大小。
在一种可能的实现方式中，可先根据多个分区的统计数据，对上述多个分区111进行调整，获得多个特征分区，步骤S320可以根据多个特征分区的统计数据确定预取策略。其中，每个特征分区用于存储至少一种类型的数据流，然后再确定每个特征分区的预取策略。举例来说，对多个分区111进行调整后可获得3个特征分区，其中，特征分区1内的数据流为长顺序流，特征分区2内的数据流为短顺序流，特征分区3内的数据流为随机流。上述举例用于说明，本申请不作具体限定。
在一种可能的实现方式中，可将数据流的类型数量高于第一阈值的分区进行拆分，获得多个特征分区。具体实现中，数据流的类型数量不低于第一阈值的分区在进行拆分时，可以被拆分为2个分区，也可以被拆分为2个以上的分区，拆分规则可以是随机拆分，也可以是平均拆分，3本申请不对拆分规则进行限定。举例来说，若分区1的数据流的类型数量为5， 分区2的数据流的类型数量为2，第一阈值为3，那么可以将分区1进行拆分获得特征分区11和特征分区12，分区2不进行拆分，从而将分区1和分区2拆分为特征分区11、特征分区12和特征分区2。应理解，上述举例用于说明，本申请不对此进行限定。
在一种可能的实现方式中，可根据两个分区的统计数据，对两个相分区的数据流特征进行相似度计算，若相似度高于第二阈值，则说明两个分区的数据流特征趋于一致，可将两个分区进行合并，其中，上述两个分区是逻辑性连续的两个相邻分区，换句话说，逻辑地址的编址方式连续的两个分区可以合并，比如相邻分区的LBA可以是连续的，即LBA＝0的逻辑区块为分区1，LBA＝2的逻辑区块为分区2，那么分区1和分区2即为逻辑性连续的两个相邻分区，而分区1对应的物理地址和分区2对应的物理地址可以是连续的，也可以是不连续的，本申请不作具体限定。
参考步骤S310的描述可知，每个分区的统计数据可以是特征向量(x 1,x 2,...,x m)的形式进行表示，因此，统计数据的相似度可以根据特征向量之间的距离确定，比如欧氏距离(eucledian distance)、曼哈顿距离(manhattan distance)、余弦相似度(cosine similarity)、明式距离(Minkowski distance)等等，本申请不作具体限定。
在一种可能的实现方式中，可将多个分区111中数据流的类型数量超过第一阈值的分区进行拆分，同时对多个分区111中数据流的特征相似度低于第二阈值的分区进行合并，从而获得多个特征分区。比如多个分区包括分区1、分区2和分区3，分区1的数据流的类型数量超过第一阈值，分区2和分区3的数据流的类型数量低于第一阈值，分区1和分区2的统计数据相似度低于第二阈值，分区1和分区3的数据相似度低于第二阈值，分区2和分区3的统计数据相似度高于第二阈值，那么可以将分区1拆分为特征分区11和特征分区12，分区2和分区3合并为特征分区23，应理解，上述拆分可以是平均拆分，也可以是随机拆分，还可以是根据不同类型的数据流的分布比例拆分。
具体实现中，处理器130可以在步骤S320确定预取策略之前，对多个分区进行调整，还可以在步骤S310之前，周期性的获取各个分区统计数据，周期性调整各个分区获得多个特征分区，每个特征分区中数据流的类型数量低于第一阈值，且各个特征分区之间的数据流特征相似度高于第二阈值时，即每个特征分区中的数据流特征趋于单一后，再执行步骤S310～步骤S330实现数据预取的过程。换句话说，在第一个调整周期内对上述多个分区111进行调整获得多个特征分区后，在第二个调整周期内处理器130可调用统计模块112的代码，对第一个调整周期获得的多个特征分区进行统计，获得多个特征分区的统计数据，然后再调用分区调整模块113的代码对多个特征分区进行进一步的调整，循环往复，多个调整周期后，每个特征分区中的数据流的类型数量将会低于第一阈值，且特征分区的统计数据之间的相似度将会低于第二阈值，使得每个特征分区中的数据流特征趋于单一，不同类型的顺序流可以从混合数据流中分离出来，此时再执行步骤S310～步骤S330将数据从第一存储器预取至第二存储器，能够避免多种数据流之间互相干扰导致预取效果差的问题，进而提高缓存命中率。
例如，图4是本申请提供的一种应用场景下的多个分区调整为多个特征分区的流程示意图，如图4所示，在t＝0时刻，多个分区包括分区1、分区2和分区3，其中，分区1、分区2和分区3是逻辑性连续的3个分区，假设CPU每隔周期T调用统计模块112的代码对多个分区内的数据流的历史情况进行采集，获得多个分区的统计数据，并调用分区调整模块113的代码对多个分区进行一次调整，那么在t＝T时刻，可以根据分区1、分区2以及分区3的统计特征，将数据流的类型数量高于第一阈值的分区1拆分为特征分区11和特征分区12，将统计数据之间的相似度高于第二阈值的分区2和分区3合并为特征分区23。同理，在t＝2T 时刻，将特征分区11拆分为特征分区111和特征分区112，将特征分区12和特征分区13合并为特征分区1223。应理解，图4用于举例说明，本申请不作具体限定。
可以理解的，由于数据流中包含的IO是应用下发的IO，应用下发IO的时间并不是固定的，因此可能会出现在步骤S310之前经过多个调整周期后，多个特征分区内的数据流特征已趋于单一，但是步骤S310之后，个别特征分区内的数据流发生了变化，导致个别特征分区内的数据流特征不再趋于单一，因此在步骤S320确定每个特征分区对应的预取策略之前，可再对多个特征分区进行调整，避免由于数据流发生变化导致特征分区内包括多种数据流，从而提高预取准确率，提高缓存命中率。
在一种可能的实现方式中，获得多个特征分区后，处理器130可调用第二存储器120中的预取模块121的代码，根据每个特征分区内的数据流特征，确定每个特征分区对应的预取策略。应理解，预取策略用于指示预取数据的位置和预取数据的大小，处理器130可根据预取策略从第一存储器110的数据流中获取预取数据。
其中，该预取策略可包括预取数据在数据流中的预取偏移值以及预取长度，还可包括其他用以供处理器130从数据流中读取预取数据的相关信息，比如预取数据的读取时间，预取数据的地址等等，本申请不对此进行限定。根据预取策略，处理器130可以从第一存储器中读取预取数据，并将其存储至第二存储器中。具体实现中，预取策略可以是根据每个特征分区的统计数据，使用预取算法获得的，预取算法可以是Tap预取算法、AMP预取算法等等，本申请不对采用何种预取算法进行限定。
S330：根据预取策略执行预取操作。
具体实现中，处理器130可使用每个特征分区对应的预取策略，对每个特征分区进行预取，将预取数据从第一存储器110的数据流中读取出来，并存储至第二存储器120中，以实现数据预取的功能，提高存储器的访问速度。
在一种可能的实现方式中，获得每个特征分区的预取策略之后，可先个根据多个特征分区中每个特征分区执行预取操作的历史数据预测每个特征分区的预取收益，然后选择预取收益高于第三阈值的特征分区集合，按照预取策略分别执行特征分区集合中每个特征分区的预取操作，其中，特征分区集合包括至少一个特征分区。处理器130可确定每个分区使用对应的预取策略进行数据预取后的预取收益，然后再调用存储模块122，对预取收益不低于第三阈值的特征分区进行预取，对预取收益低于第三阈值的特征分区放弃预取，从而进一步提高预取命中率，避免预取失效带来的资源浪费。
具体实现中，处理器130可周期性的获取多个分区的统计数据，并周期性的将多个分区调整为多个特征分区，然后周期性的对第一存储器进行预取，仍以图4所示的例子为例，t＝T时刻获得特征分区11、特征分区12以及特征分区23之后，处理器130可调用预取模块121，确定特征分区11的预取策略11，特征分区12的预取策略12以及特征分区23的预取策略23，然后确定特征分区11使用预取策略11进行预取后的预取收益11，特征分区12使用预取策略12进行预取后的预取收收益12，特征分区23使用预取策略23进行预取后的预取收益23，若预取收益11和预取收益12都不低于第三阈值，但是预取收益23低于第三阈值，那么在t＝T时刻下，处理器130可调用存储模块122对特征分区11和特征分区12进行预取，而不对特征分区23进行预取，同理，可确定的t＝2T时刻下每个特征分区的预取收益，这里不再展开赘述。应理解，上述举例用于说明，本申请不作具体限定。
在一种可能的实现方式中，上述每个特征分区的预取收益可包括预取命中率x、预取浪费率y和预取IO数量z中的一种或者多种，其中，预取命中率x包括预取数据被应用从第二 存储器120中读取的概率，预取浪费率y包括预取数据未被应用从第二存储器120中读取的概率，预取IO数量z即为预取数量的IO数目。具体地，可以为预取命中率、预取浪费率和预取IO数量设置不同的权重，示例性的，预取收益＝αx+βy+γz，其中，α是预取命中率x的权重，β是预取浪费率y的权重，γ是预取预取IO数量z的权重，α、β和γ可以是经验值，该经验值还可以根据预取收益的准确度进行调整。上述公式用于举例，本申请不作具体限定。
具体实现中，上述预取IO数量可以根据步骤S310采集的统计数据获得，上述预取命中率x和预取浪费率y可以根据每个特征分区的历史数据，结合步骤S310采集的统计数据确定。其中，每个特征分区的历史数据包括每个特征分区的历史统计数据，历史统计数据可包括当前周期之前的所有周期内的历史数据流信息和历史数据流特征，历史数据流信息可包括但不限于IO偏移、IO长度和IO时间，历史数据流特征可包括但不限于IO数量、数据流长度、IO间隔等等，其中，历史统计数据的具体描述可参考步骤S310中关于统计数据的描述，这里不再重复赘述，并且，历史统计数据可以是每个特征分区上一周期的历史统计数据，也可以是上两个周期的历史统计数据，还可以是本周期之前所有周期的历史统计数据，本申请不作具体限定。
举例来说，当前周期采集的统计信息显示分区1的数据流包括应用下发的1个IO，且该IO的IO偏移为100，长度为50，若历史统计数据显示上一周期该应用下发的历史IO，其实际访问的数据偏移值为0，长度为100，即读取了数据流中偏移值从0～100的一段数据，根据预取算法(比如AMP预取算法)确定预取策略为：预取偏移值为100，预取长度为100，若使用该预取测量，将会读取偏移值从100～200的一段数据，而当前周期采集的统计信息显示，实际需求为偏移值从100～150的一段数据，那么即可预测得到预取命中率为50％，预取浪费率为50％，预取IO数量为1，根据预取收益＝αx+βy+γz可以获得预取收益，根据预取收益与第三阈值的大小，确定特征分区集合。应理解，上述举例用于说明，本申请不作具体限定。
可选地，该历史统计数据还可包括预取系统100当前的负载情况、历史预取收益估计的准确率等等其他相关信息，可以理解的，历史统计数据包含的内容越丰富，预取收益的准确度越高，但是对于预取系统100的处理压力将会增大，因此具体可根据预取系统100的处理能力确定历史数据的其他相关信息，本申请不对此进行限定。
在一种可能的实现方式中，处理器130调用预取模块确定特征分区的预取策略和预取收益，并确定该预取收益不低于第三阈值的情况下，处理器130调用存储模块使用预取策略对该特征分区进行预取操作之前，还可先查询第二存储器120中是否存在该预取策略对应的预取数据，若存在，可以取消本次预取操作，若不存在，再根据预取策略对该特征分区进行预取，将预取数据从第一存储器中读取至第二存储器，从而避免重复读取这一情况的发生，进而避免资源浪费。
应理解，由于每个周期内，每个特征分区内的数据流特征可能会发生变化，可能会导致部分特征分区内的数据流的类型数量突然增加，此时对该类特征分区进行预取，仍会出现预取命中率低的情况，本申请通过在预取前确定每个特征分区的预取收益，可以筛选出该类数据流发生变化的特征分区，从而提高预取命中率，进而提高存储器的访问效率。值得注意的是，在当前周期确定每个特征分区的预取收益后，可以在当前周期对预取收益不低于第三阈值的特征分区执行预取操作，也可以在下一周期执行预取操作，本申请不作具体限定。
进一步地，对于部分长期保持低预取收益的特征分区，可以调用分区调整模块113对该类特征分区进行调整，从而提高预取命中率；若该特征分区内的数据流已趋于单一，但是预取收益仍旧较低，可以对预取收益中的权重参数进行调整，避免由于预取收益的权重参数设 置不合理对预取操作产生影响。
举例来说，仍以图4所示的应用场景为例，假设在t＝2T时刻获得了特征分区111、特征分区112以及特征分区1223后，根据预取策略执行预取操作的步骤流程可以如图5所示，在t＝2T与t＝2T这一时间周期内，可以先确定每个特征分区的预取策略，获得特征分区111的预取策略111，特征分区112的预取策略112，特征分区1223的预取策略1223，然后确定每个特征分区的预取收益，根据特征分区111的历史数据确定预取收益111，根据特征分区112的历史数据确定预取收益112，根据特征分区1223的历史数据确定预取收益1223，接着将每个特征分区的预取收益与第三阈值进行比较，低于第三阈值的特征分区1223在本周期不进行预取操作，高于第三阈值的特征分区111和特征分区112则根据各自对应的预取策略，在下一周期开始时，即t＝3T时执行预取操作。应理解，图5用于举例说明，特征分区111好特征分区112也可以在当前周期执行预取操作，本申请不作具体限定。
综上可知，本申请提供的数据预取方法，可以将用于存储预取数据的第一存储器划分为多个分区，并根据每个分区内数据流的特征，对多个分区进行调整，使得每个分区内的数据流特征趋于单一，获得多个特征分区，从而将顺序流从混合数据流中分离出来，此时再对多个特征分区进行预取，可以避免多种数据流之间的互相干扰导致预取效果差的问题，进而提高缓存命中率。
上面详细阐述了本申请实施例的方法，为了便于更好的实施本申请实施例上述方案，相应地，下面还提供用于配合实施上述方案的相关设备。
图6是本申请提供的一种处理器130的结构示意图，该处理器130应用于图1所示的预取系统100中，该预取系统100包括处理器130、第一存储器110和第二存储器120。如图6所示，处理器130可包括统计模块112、分区调整模块113、预取模块121以及存储模块122，其中，
统计模块112，用于获取第一存储器的多个分区的统计数据，统计数据用于指示多个分区中每个分区所包括的数据流的历史情况。
预取模块121，用于根据多个分区的统计数据确定预取策略，预取策略用于指示预取数据在第一存储器110中的位置和预取数据的大小。
存储模块122用于根据预取策略执行预取操作，将预取数据存储至第二存储器120。
可选地，多个分区的统计数据包括多个分区中每个分区所包括的数据流的类型，分区调整模块113用于根据第一存储器110的多个分区的统计数据，对多个分区进行调整，获得多个特征分区，其中，多个特征分区中的每个特征分区包括至少一种类型的数据流；分区调整模块113用于根据多个特征分区的统计数据，确定多个特征分区中每个特征分区的预取策略，其中，一个特征分区对应一个预取策略。
可选地，分区调整模块113用于根据多个分区的统计数据，将多个分区中数据流的类型数量不低于第一阈值的分区进行拆分，获得多个特征分区。
可选地，统计数据包括数据流特征，其中，数据流特征包括数据流数目、数据流长度、数据流的类型、数据流IO大小、数据流IO间隔中的一种或者多种。
可选地，分区调整模块113用于根据第一存储器110的多个分区的统计数据，确定多个分区的数据流特征之间的相似度；处理器130用于将相似度不低于第二阈值的分区进行合并，获得多个特征分区。
可选地，预取模块121用于根据多个特征分区中每个特征分区执行预取操作的历史数据预测每个特征分区的预取收益；预取模块121用于选择预取收益高于第三阈值的特征分区集 合，特征分区集合包括至少一个特征分区；存储模块122用于按照预取策略分别执行特征分区集合中每个特征分区的预取操作，将预取数据存储至第二存储器120。
可选地，第一存储器110的读写速度低于第二存储器120的读写速度；在第一存储器110为内存的情况下，第二存储器120为缓存；或者，在第一存储器110为磁盘时的情况下，第二存储器120为内存。
根据本申请实施例的处理器130可对应于执行本申请实施例中描述的方法，并且处理器130中的各个模块和/或功能分别为了实现图1至图5中的各个方法的相应流程，为了简洁，在此不再赘述。
综上可知，本申请提供的处理器，可以将用于存储预取数据的第一存储器划分为多个分区，并根据每个分区内数据流的特征，对多个分区进行调整，使得每个分区内的数据流特征趋于单一，获得多个特征分区，从而将顺序流从混合数据流中分离出来，此时再对多个特征分区进行预取，可以避免多种数据流之间的互相干扰导致预取效果差的问题，进而提高缓存命中率。
图7是本申请提供的一种计算设备700的结构示意图。其中，计算设备700可以是图1-图6中的预取系统100。如图7所示，计算设备700包括：处理器130、通信接口140、第一存储器110以及第二存储器120。其中，处理器130、通信接口140、第一存储器110以及第二存储器120可以通过内部总线150相互连接，也可通过无线传输等其他手段实现通信。本申请实施例以通过总线连接为例，总线可以是PCIe总线或EISA总线等。总线150可以分为地址总线、数据总线、控制总线等。为便于表示，图7中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
处理器130可以由至少一个通用处理器构成，例如CPU，或者CPU和硬件芯片的组合。上述硬件芯片可以是ASIC、PLD或其组合。上述PLD可以是CPLD、FPGA、GAL或其任意组合。处理器130执行各种类型的数字存储指令，例如存储在第一存储器110或者第二存储器120中的软件或者固件程序，它能使计算设备700提供多种服务。
第一存储器110用于存储数据和程序代码，该程序代码可由处理器130来控制执行。程序代码中可以包括一个或多个软件模块，如统计模块的代码、分区调整模块的代码等等，其中，统计模块用于获取第一存储器110的多个分区的统计数据；分区调整模块用于根据多个分区的统计数据，对多个分区进行调整，获得多个特征分区。具体可用于执行图3实施例中的步骤S310及其可选步骤，这里不再进行赘述。
第二存储器120用于存储数据和程序代码，该程序代码可由处理器130来控制执行。程序代码中可以包括一个或多个软件模块，如预取模块的代码、存储模块的代码等等，其中，预取模块用于根据多个分区的统计数据确定预取策略；存储模块用于根据预取策略执行预取操作，将预取数据存储至第二存储器，具体可用于执行图3实施例中的步骤S320～步骤S330及其可选步骤，这里不再进行赘述。
在一应用场景中，第一存储器110可以是磁盘、CD、USB、存储卡等等中任意一种，其中，上述磁盘包括但不限于HDD、SSD、SSHD、RAID等等，第一存储器110还可以是上述各种存储介质的组合，本申请不作具体限定。第二存储器120可以是ROM、RAM、DRAM、DDR等等中任意一种，第二存储器120还可以是上述各种存储介质的组合，本申请不作具体限定。
在另一应用场景中，第一存储器110可以是ROM、RAM、DRAM、DDR等等中任意一种，第二存储器120可以是SRAM、异步SRAM、多线程SRAM，同歩SRAM等等中任意一 种，本申请不作具体限定。
通信接口140可以为有线接口(例如以太网接口)，可以为内部接口(例如高速串行计算机扩展总线(peripheral component interconnect express,PCIe)总线接口)、有线接口(例如以太网接口)或无线接口(例如蜂窝网络接口或使用无线局域网接口)，用于与与其他设备或模块进行通信。
需要说明的是，本实施例可以是通用的物理服务器实现的，例如，ARM服务器或者X86服务器，也可以是基于通用的物理服务器结合NFV技术实现的虚拟机实现的，虚拟机指通过软件模拟的具有完整硬件系统功能的、运行在一个完全隔离环境中的完整计算机系统，比如在本实施例可以在云计算基础设施上实现。
需要说明的，图7仅仅是本申请实施例的一种可能的实现方式，实际应用中，计算设备700还可以包括更多或更少的部件，这里不作限制。关于本申请实施例中未示出或未描述的内容，可参见前述图1-图6实施例中的相关阐述，这里不再赘述。
应理解，图7所示的计算设备还可以是至少一个物理服务器构成的计算机集群，比如第一存储器110和第二存储器120分别位于不同的物理服务器上，具体可参考图2实施例，为了避免重复，此处不再赘述。
本申请还提供一种存储阵列，该存储阵列可包括一个或多个存储节点，每个存储节点的结构可以如图1所示，以存储阵列具体为独立冗余磁盘阵列(redundant arrays of independent disks,RAID)为例，处理器1301可以是RAID控制器，第一存储器110可以是RAID控制的逻辑硬盘块，比如逻辑固态硬盘(logical solid state disk，LSSD)，第二存储器120可以是RAID控制器的内存，当然，存储阵列还可以是其他类型的存储阵列，本申请不作具体限定。示例性的，以该存储阵列包括两个存储节点为例，该存储阵列的结构还可以如图2所示，该存储阵列用于实现上述图3-图5所示方法的流程的操作步骤。
本申请还提供一种预取系统，该预取系统的结构如图1或图2所示，该预取系统用于实现上述图3-图5所示方法的流程的操作步骤。
本申请实施例还提供一种计算机可读存储介质，计算机可读存储介质中存储有指令，当其在处理器上运行时，图3-图5所示的方法流程得以实现。
本申请实施例还提供一种计算机程序产品，当计算机程序产品在处理器上运行时，图1-图5所示的方法流程得以实现。
上述实施例，可以全部或部分地通过软件、硬件、固件或其他任意组合来实现。当使用软件实现时，上述实施例可以全部或部分地以计算机程序产品的形式实现。计算机程序产品包括至少一个计算机指令。在计算机上加载或执行计算机程序指令时，全部或部分地产生按照本发明实施例的流程或功能。计算机可以为通用计算机、专用计算机、计算机网络、或者其他可编程装置。计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line,DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含至少一个可用介质集合的服务器、数据中心等数据存储节点。可用介质可以是磁性介质(例如，软盘、硬盘、磁带)、光介质(例如，高密度数字视频光盘(digital video disc,DVD)、或者半导体介质。半导体介质可以是SSD。
以上，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技 术领域的技术人员在本发明揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应以权利要求的保护范围为准。

Claims (15)

  1. 一种数据预取方法，其特征在于，所述方法应用于包括多种数据流的预取系统，所述预取系统包括第一存储器和第二存储器，所述方法包括：
    获取所述第一存储器的多个分区的统计数据，所述统计数据用于指示所述多个分区中每个分区所包括的数据流的历史情况；
    根据所述多个分区的统计数据确定预取策略，所述预取策略用于指示预取数据的位置和预取数据的大小；
    根据所述预取策略执行预取操作。
  2. 根据权利要求1所述的方法，其特征在于，所述多个分区的统计数据包括所述多个分区中每个分区所包括的数据流的类型，所述根据所述多个分区的统计数据确定预取策略包括：
    根据所述多个分区的统计数据，对所述多个分区进行调整，获得多个特征分区，其中，所述多个特征分区中的每个特征分区包括至少一种类型的数据流；
    根据所述多个特征分区的统计数据，确定所述多个特征分区中每个特征分区的预取策略，其中，所述每个特征分区对应一个预取策略。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述多个分区的统计数据，对所述多个分区进行调整，获得多个特征分区包括：
    根据所述多个分区的统计数据，将所述多个分区中数据流的类型数量不低于第一阈值的分区进行拆分，获得所述多个特征分区。
  4. 根据权利要求2或3所述的方法，其特征在于，所述统计数据包括数据流特征，其中，所述数据流特征包括数据流数目、数据流长度、数据流的类型、数据流IO大小、数据流IO间隔中的一种或者多种。
  5. 根据权利要求4所述的方法，其特征在于，所述根据所述多个分区的统计数据，对所述多个分区进行调整，获得多个特征分区包括：
    根据所述多个分区的统计数据，确定所述多个分区的数据流特征之间的相似度；
    将所述相似度不低于第二阈值的分区进行合并，获得所述多个特征分区。
  6. 根据权利要求2至5任一权利要求所述的方法，其特征在于，所述根据所述预取策略执行预取操作包括：
    根据所述多个特征分区中每个特征分区执行预取操作的历史数据预测所述每个特征分区的预取收益；
    选择预取收益高于第三阈值的特征分区集合，所述特征分区集合包括至少一个特征分区；
    按照所述预取策略分别执行所述特征分区集合中每个特征分区的预取操作。
  7. 根据权利要求1至6任一权利要求所述的方法，其特征在于，所述第一存储器的读写速度低于所述第二存储器的读写速度；
    在所述第一存储器为内存的情况下，所述第二存储器为缓存；或者，
    在所述第一存储器为磁盘时的情况下，所述第二存储器为内存。
  8. 一种预取系统，其特征在于，所述预取系统包括处理器、第一存储器和第二存储器，其中，
    所述处理器，用于获取所述第一存储器的多个分区的统计数据，所述统计数据用于指示所述多个分区中每个分区所包括的数据流的历史情况；
    所述处理器，用于根据所述多个分区的统计数据确定预取策略，所述预取策略用于指示预取数据在所述第一存储器中的位置和预取数据的大小；以及，根据所述预取策略执行预取操作，将所述预取数据存储至所述第二存储器。
  9. 根据权利要求8所述的系统，其特征在于，所述多个分区的统计数据包括所述多个分区中每个分区所包括的数据流的类型，
    所述处理器用于根据所述第一存储器的多个分区的统计数据，对所述多个分区进行调整，获得多个特征分区，其中，所述多个特征分区中的每个特征分区包括至少一种类型的数据流；
    所述处理器用于根据所述多个特征分区的统计数据，确定所述多个特征分区中每个特征分区的预取策略，其中，一个特征分区对应一个预取策略。
  10. 根据权利要求9所述的系统，其特征在于，所述处理器用于根据所述多个分区的统计数据，将所述多个分区中数据流的类型数量不低于第一阈值的分区进行拆分，获得所述多个特征分区。
  11. 根据权利要求9或10所述的系统，其特征在于，所述统计数据包括数据流特征，其中，所述数据流特征包括数据流数目、数据流长度、数据流的类型、数据流IO大小、数据流IO间隔中的一种或者多种。
  12. 根据权利要求11所述的系统，其特征在于，
    所述处理器用于根据所述第一存储器的多个分区的统计数据，确定所述多个分区的数据流特征之间的相似度；
    所述处理器用于将所述相似度不低于第二阈值的分区进行合并，获得所述多个特征分区。
  13. 根据权利要求9至12任一权利要求所述的系统，其特征在于，
    所述处理器用于根据所述多个特征分区中每个特征分区执行预取操作的历史数据预测所述每个特征分区的预取收益；
    所述处理器用于选择预取收益高于第三阈值的特征分区集合，所述特征分区集合包括至少一个特征分区；
    所述处理器用于按照所述预取策略分别执行所述特征分区集合中每个特征分区的预取操作，将所述预取数据存储至所述第二存储器。
  14. 根据权利要求10至13任一权利要求所述的系统，其特征在于，所述第一存储器的读写速度低于所述第二存储器的读写速度；
    在所述第一存储器为内存的情况下，所述第二存储器为缓存；或者，
    在所述第一存储器为磁盘时的情况下，所述第二存储器为内存。
  15. 一种计算设备，其特征在于，包括处理器和存储器，所述存储器存储有代码，所述处理器执行所述代码实现如权利要求1至9任一权利要求所述方法的操作步骤。
PCT/CN2021/142647 2021-02-27 2021-12-29 数据预取的方法、设备及系统 WO2022179294A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110221620.8A CN114968073A (zh) 2021-02-27 2021-02-27 数据预取的方法、设备及系统
CN202110221620.8 2021-02-27

Publications (1)

Publication Number Publication Date
WO2022179294A1 true WO2022179294A1 (zh) 2022-09-01

Family

ID=82973665

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/142647 WO2022179294A1 (zh) 2021-02-27 2021-12-29 数据预取的方法、设备及系统

Country Status (2)

Country Link
CN (1) CN114968073A (zh)
WO (1) WO2022179294A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116910100A (zh) * 2023-09-08 2023-10-20 湖南立人科技有限公司 用于低代码平台的缓存数据处理方法

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101390047A (zh) * 2006-02-27 2009-03-18 Nxp股份有限公司 数据处理系统和用于预取数据和/或指令的方法
CN101853303A (zh) * 2010-06-02 2010-10-06 深圳市迪菲特科技股份有限公司 一种基于语义智能存储方法及系统
CN102857483A (zh) * 2011-06-30 2013-01-02 国际商业机器公司 预取数据的方法、设备和装置
US20170091104A1 (en) * 2015-09-30 2017-03-30 Arm Limited Apparatus and method for prefetching
CN109032965A (zh) * 2017-06-12 2018-12-18 华为技术有限公司 一种数据读取方法、主机及存储设备
US20190361811A1 (en) * 2018-05-24 2019-11-28 Hitachi, Ltd. Data processing apparatus and prefetch method
CN110688062A (zh) * 2019-08-26 2020-01-14 华为技术有限公司 一种缓存空间的管理方法及装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101390047A (zh) * 2006-02-27 2009-03-18 Nxp股份有限公司 数据处理系统和用于预取数据和/或指令的方法
CN101853303A (zh) * 2010-06-02 2010-10-06 深圳市迪菲特科技股份有限公司 一种基于语义智能存储方法及系统
CN102857483A (zh) * 2011-06-30 2013-01-02 国际商业机器公司 预取数据的方法、设备和装置
US20170091104A1 (en) * 2015-09-30 2017-03-30 Arm Limited Apparatus and method for prefetching
CN109032965A (zh) * 2017-06-12 2018-12-18 华为技术有限公司 一种数据读取方法、主机及存储设备
US20190361811A1 (en) * 2018-05-24 2019-11-28 Hitachi, Ltd. Data processing apparatus and prefetch method
CN110688062A (zh) * 2019-08-26 2020-01-14 华为技术有限公司 一种缓存空间的管理方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116910100A (zh) * 2023-09-08 2023-10-20 湖南立人科技有限公司 用于低代码平台的缓存数据处理方法
CN116910100B (zh) * 2023-09-08 2023-11-28 湖南立人科技有限公司 用于低代码平台的缓存数据处理方法

Also Published As

Publication number Publication date
CN114968073A (zh) 2022-08-30

Similar Documents

Publication Publication Date Title
KR101761301B1 (ko) 메모리 자원 최적화 방법 및 장치
US20030065886A1 (en) Dynamic cache partitioning
JP6146087B2 (ja) ストレージ制御プログラム，ストレージ制御方法，ストレージシステム及びその階層制御装置
CN104503703B (zh) 缓存的处理方法和装置
US11093399B2 (en) Selecting resources to make available in local queues for processors to use
CN106534308A (zh) 一种分布式存储系统中解决数据块访问热点的方法及装置
CN111338579B (zh) 基于存储池的读写缓存优化方法、系统、终端及存储介质
WO2018022175A1 (en) Techniques to allocate regions of a multi level, multitechnology system memory to appropriate memory access initiators
CN111737168A (zh) 一种缓存系统、缓存处理方法、装置、设备及介质
WO2023000536A1 (zh) 一种数据处理方法、系统、设备以及介质
WO2022179294A1 (zh) 数据预取的方法、设备及系统
CN107783720A (zh) 一种数据均衡方法及存储设备
US11048631B2 (en) Maintaining cache hit ratios for insertion points into a cache list to optimize memory allocation to a cache
CN112379841A (zh) 数据处理方法、装置和电子设备
CN115421924A (zh) 一种内存分配方法、装置及设备
CN115033185A (zh) 访存处理方法和装置、存储装置、芯片、板卡、电子设备
US11782624B2 (en) Worflow-based partition allocation
Zhou et al. Hierarchical consistent hashing for heterogeneous object-based storage
CN115495433A (zh) 一种分布式存储系统、数据迁移方法及存储装置
CN115794366A (zh) 一种内存预取方法及装置
CN116910314A (zh) 基于键值分离的键值存储系统中范围查询优化方法及装置
CN112445794B (zh) 一种大数据系统的缓存方法
US9354820B2 (en) VSAM data set tier management
CN118276771A (zh) 一种numa架构下nvm文件并行访问方法
US9652373B2 (en) Adaptive statistics for a linear address space

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21927718

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21927718

Country of ref document: EP

Kind code of ref document: A1