WO2018121242A1

WO2018121242A1 - Multiple buffer-based data elimination method and device

Info

Publication number: WO2018121242A1
Application number: PCT/CN2017/115616
Authority: WO
Inventors: 王文铎; 陈宗志; 彭信东; 王康
Original assignee: 北京奇虎科技有限公司
Priority date: 2016-12-29
Filing date: 2017-12-12
Publication date: 2018-07-05
Also published as: CN106649139A; CN106649139B

Abstract

A multiple buffer-based data elimination method and device. The method comprises: classifying, according to a preconfigured level classification rule, buffers into a plurality of buffer levels to create thread pools matching the respective buffer levels, wherein each of the thread pools contains a plurality of threads (S110); employing the plurality of threads in the respective thread pools to scan respective buffers, and determining, according to a scan result and the classification rule, the buffer levels of the respective buffers (S120); and employing the plurality of threads in the respective thread pools to eliminate data in the buffers matching the respective buffer levels and thread pools (S130).

Description

Data elimination method and device based on multiple caches

Cross-reference to related applications

This application claims priority to Chinese Patent Application No. 201611246005.8, entitled "A Multi-Cache Based Data Elimination Method and Apparatus", filed on December 29, 2016, the entire contents of which are incorporated by reference. In this application.

Technical field

The present disclosure relates to the field of computer technologies, and in particular, to a data elimination method and apparatus based on multiple caches.

Background technique

Cache is an important technology used to solve the speed mismatch between high and low speed devices. It is widely used in various fields such as storage systems, databases, web servers, processors, file systems, Disk systems, etc., can reduce application response time and improve efficiency. However, the storage medium used in the Cache technology, such as RAM and SSD, has higher performance and is more expensive. For the sake of cost performance, the capacity of the Cache is limited, so it is necessary to effectively manage the Cache space. Therefore, a variety of Cache elimination algorithms have emerged, such as: Least Recently Used (LRU) elimination algorithm; Recently Least Frequently Used (LFU) elimination algorithm; Most recently used (Most Recently Used) MRU) elimination algorithm; adaptive Cache (Adaptive Replacement Cache, ARC) elimination algorithm.

However, in the process of implementing the present disclosure, the inventors have found that at least the following problems exist in the prior art: the prior art elimination algorithm is generally a single-threaded processing method, and the processing efficiency is low, and the lower processing efficiency sometimes causes After the limited cache space is used up, it cannot be vacated in time, and the subsequent data cannot be stored in time.

Summary of the invention

In view of the above problems, the present disclosure has been made in order to provide a multi-cache based data elimination method and corresponding apparatus that overcomes the above problems or at least partially solves the above problems.

According to an aspect of the present disclosure, a data culling method based on multiple caches is provided, including:

Dividing a plurality of cache levels according to a preset level division rule, respectively creating a matching thread pool for each cache level; wherein each thread pool includes multiple threads;

Each cache is scanned by multiple threads in each thread pool, and the cache level of each cache is determined according to the scan result and the level division rule;

The data in the cache with the cache level matching the thread pool is eliminated by using multiple threads in each thread pool.

According to another aspect of the present disclosure, a data culling device based on a plurality of caches is provided, including:

a dividing module, configured to divide a plurality of cache levels according to a preset level dividing rule, and respectively create a matching thread pool for each cache level; wherein each thread pool includes multiple threads;

a scanning module, configured to scan each cache separately by using multiple threads in each thread pool, and determine a cache level of each cache according to the scan result and the level division rule;

The culling module is configured to use multiple threads in each thread pool to eliminate data in a cache whose cache level matches the thread pool.

According to a third aspect of the present disclosure, a computer program is provided, comprising:

Computer readable code, when the computer readable code is run on a computing device, causes the computing device to perform the plurality of cache based data culling methods described above.

According to a fourth aspect of the present disclosure, a computer readable medium comprising:

A computer program for performing the above-described multi-cache based data elimination method is stored.

According to the disclosure, a method and an apparatus for eliminating data based on multiple caches may divide multiple cache levels according to a preset level division rule, respectively create matching thread pools for each cache level; and utilize multiple threads in each thread pool. Each cache is scanned separately, and the cache level of each cache is determined according to the scan result and the level division rule; and the data in the cache whose cache level matches the thread pool is eliminated by using multiple threads in each thread pool. It can be seen that by dividing the cache into multiple cache levels and respectively creating corresponding thread pools for each cache level, the number of threads in the thread pool can be better adjusted according to the cache level; and, through multiple thread pools, parallel processing The way to greatly improve the efficiency of data elimination processing.

The above description is only an overview of the technical solutions of the present disclosure, in order to be able to understand the technology of the present disclosure more clearly. The above and other objects, features, and advantages of the present invention will be apparent from the description and appended claims.

BRIEF abstract

Various other advantages and benefits will become apparent to those skilled in the art from a The drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting. Throughout the drawings, the same reference numerals are used to refer to the same parts. In the drawing:

1 is a schematic flow chart of a data elimination method based on multiple caches according to Embodiment 1 of the present disclosure;

2 is a schematic flowchart of a data elimination method based on multiple caches according to Embodiment 2 of the present disclosure;

3 is a schematic structural diagram of a data elimination device based on multiple caches according to Embodiment 3 of the present disclosure;

4 is a schematic structural diagram of a data elimination device based on multiple caches according to Embodiment 4 of the present disclosure;

FIG. 5 schematically illustrates a block diagram of a computing device for performing a multiple cache based data retirement method in accordance with an embodiment of the present disclosure;

FIG. 6 schematically illustrates a storage unit for maintaining or carrying program code that implements a plurality of cache-based data retirement methods in accordance with an embodiment of the present disclosure.

Preferred embodiment of the invention

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While the embodiments of the present invention have been shown in the drawings, the embodiments Rather, these embodiments are provided so that this disclosure will be more fully understood and the scope of the disclosure will be fully disclosed.

Embodiment 1

FIG. 1 is a schematic flowchart of a method for eliminating data based on multiple caches according to Embodiment 1 of the present disclosure. As shown in the figure, the method includes:

Step S110: Divide a plurality of cache levels according to a preset level division rule, and respectively create matching thread pools for each cache level.

The preset level division rule is used to divide each cache into different levels according to different usage conditions, and each cache in the same level has similar usage. This level is artificially defined by the technician. For the specific content of the preset grading rule, the first embodiment of the present disclosure does not specifically limit this, and those skilled in the art can flexibly set according to actual conditions.

In order to improve the processing efficiency of data elimination, a matching thread pool is created for each cache level, and each thread pool contains multiple threads. Multiple threads in each thread pool are used for data elimination processing of the corresponding level of cache. Because different levels of cache usage are different, in order to optimize resource allocation as much as possible, the number of threads in the thread pool corresponding to different levels may also be different.

The reason why the thread pool technology is adopted is that if the corresponding thread is set for each cache, it will consume a lot of system resources and has no practical operation.

Step S120: Scan each cache separately by using multiple threads in each thread pool, and determine the cache level of each cache according to the scan result and the level division rule.

Because when multiple threads perform data elimination processing on all caches, if the matching relationship between the threads and the cache is not limited, two threads may simultaneously process the same cache. At this time, the two threads will Conflicts lead to a series of problems. Therefore, the embodiment of the present disclosure optimizes the workflow by classifying all caches and specifying the corresponding processing relationship between each thread and a different level of cache, thereby effectively avoiding the occurrence of the above conflict situation.

Specifically, each cache is scanned by using multiple threads in each thread pool, and the cache level is determined for each scanned cache according to the scan result and the level division rule, for subsequent targeted processing.

Step S130: Using multiple threads in each thread pool to eliminate data in the cache whose cache level matches the thread pool.

Specifically, according to the cache level determined in step S120, the data having the corresponding cache level is subjected to data elimination processing by using a plurality of threads in the thread pool matching the respective cache levels. For the specific method of the data elimination processing, the first embodiment of the present disclosure does not specifically limit this, and those skilled in the art can flexibly set according to actual conditions.

Therefore, the data elimination method based on multiple caches provided by the embodiment of the present disclosure may divide multiple cache levels according to a preset level division rule, and respectively create a match for each cache level. Thread pool; using multiple threads in each thread pool to scan each cache separately, determining the cache level of each cache according to the scan result and the hierarchical rule; using multiple threads in each thread pool to match the cache level with the thread pool The data in the cache is eliminated. It can be seen that by dividing the cache into multiple cache levels and respectively creating corresponding thread pools for each cache level, the number of threads in the thread pool can be better adjusted according to the cache level; and, through multiple thread pools, parallel processing The way to greatly improve the efficiency of data elimination processing.

Embodiment 2

FIG. 2 is a schematic flowchart of a method for eliminating data based on multiple caches according to Embodiment 2 of the present disclosure. As shown in the figure, the method includes:

Step S210: Divide a plurality of cache levels according to a preset level division rule, and respectively create matching thread pools for each cache level.

The preset level division rule is used to divide each cache into different levels according to different usage conditions, and each cache in the same level has similar usage. In the embodiment of the present disclosure, the level dividing rule includes: dividing a cache level according to a ratio between a remaining storage space of the cache and a total storage space, wherein a ratio between the remaining storage space and the total storage space is larger, and the cache level is higher. High; the smaller the ratio between the remaining storage space and the total storage space, the lower the cache level. For example, suppose the cache level is divided into three levels, which are high (HIGH) level, low (LOW) level, and idle (IDLE) level, where the ratio of the remaining storage space of the cache to the total storage space is above 60%. The cache is determined to be the HIGH level; the cache between the remaining storage space of the cache and the total storage space is determined to be LOW level between 30% and 60%; the ratio of the remaining storage space of the cache to the total storage space The cache below 30% is determined to be the IDLE level.

In order to improve the processing efficiency of data elimination, a matching thread pool is created for each cache level, and each thread pool contains multiple threads. Multiple threads in each thread pool are used for data elimination processing of the corresponding level of cache. Because the usage of different levels of cache is different, in order to optimize the resource configuration as much as possible, the number of threads in the thread pool corresponding to different levels is also different.

Step S220: respectively set corresponding weight values for each thread pool, and set the number of threads included in each thread pool according to the weight values of the respective thread pools.

For the specific setting method of the weight value, for each thread pool, the weight value corresponding to the thread pool is set according to the level of the cache level matched with the thread pool, wherein the cache level matching the thread pool is higher. The thread pool has a higher weight value; instead, the cache matches the thread pool The lower the level, the smaller the weight value of the thread pool. The greater the weight value of the thread pool, the greater the number of threads included in the thread pool; the smaller the weight value of the thread pool, the fewer the number of threads contained in the thread pool. Therefore, the number of threads contained in each thread pool is dynamically changing.

Step S230: Scan each cache separately by using multiple threads in each thread pool, and determine the cache level of each cache according to the scan result and the level division rule.

Because when multiple threads perform data elimination processing on all caches, if the matching relationship between the threads and the cache is not limited, two threads will process the same cache. At this time, two threads will be generated. Conflicts lead to a series of problems. Therefore, the embodiment of the present disclosure optimizes the workflow by classifying all caches and specifying the corresponding processing relationship between each thread and a different level of cache, thereby effectively avoiding the occurrence of the above conflict situation.

Correspondingly, the setting method of the weight value of the thread pool may further include: periodically acquiring the scan result of each thread pool, determining the number of caches corresponding to each cache level according to the scan result; and then adjusting each thread according to the number of caches corresponding to each cache level. The weight value of the pool, and adjust the number of threads included in each thread pool according to the adjusted weight value of each thread pool. The greater the number of caches corresponding to the cache level, the greater the weight of the thread pool matching the cache level. Conversely, the smaller the cache level corresponding to the cache level, the smaller the weight value of the thread pool matching the cache level. . Determining the weight value of the thread pool by the number of caches, thereby determining the number of threads included in each thread pool, so that the number of threads in each thread pool can accurately satisfy the processing operations of each cache in the corresponding cache level, so that the resources are obtained. Reasonable use and cost saving.

In other embodiments, the weight value setting method of the thread pool provided in step S220 and step S230 may be comprehensively used to set a more reasonable weight value of the thread pool. In addition, the weight value of the thread pool can be further determined according to various factors such as the type and importance of the cache of the corresponding level.

Step S240: Using multiple threads in each thread pool to eliminate data in the cache whose cache level matches the thread pool.

Specifically, according to the cache level determined in the above step, the data having the corresponding cache level is subjected to data elimination processing by using a plurality of threads in the thread pool matching the respective cache levels. among them, Each thread pool can process only one cache level cache. For example, in step S210, the cache is divided into three levels: a HIGH level, a LOW level, and an IDLE level, so only three thread pools are required. Specifically, the thread pool 1 corresponds to the HIGH level, the thread pool 2 corresponds to the LOW level, and the thread pool 3 corresponds to the IDLE level. In this case, all the threads in the thread pool 1 only process all the caches in the HIGH level, and the thread pool 2 All threads only process all caches in the LOW level, and all threads in thread pool 3 only process all caches in the IDLE level. Of course, when there are more cache levels, each thread pool can also be used to handle multiple cache level caches. For example, when the cache level includes six levels, it can also be processed by three thread pools, each of which handles two levels of cache.

In short, through the division of the cache level and the application of the thread pool technology, the cache scanning and data elimination work can be more flexibly implemented. In addition, the above steps S230 and S240 may be repeatedly performed multiple times. For example, step S230 may be performed once every preset first time interval, and step S240 may be performed once every preset second time interval. The first time interval and the second time interval may be equal or may not be equal. In addition, the first time interval and the second time interval may be either fixed values or dynamically changing values. For example, the first time interval may be dynamically adjusted according to the scan result: when the number of caches of the HIGH level is large in the scan result, the first time interval is reduced; when the number of caches of the HIGH level in the scan result is small, the first time is increased. interval. In addition, in each execution process of step S240, each thread pool may perform the phase-out operation on the cache of the corresponding level according to the same execution cycle, or perform the phase-out operation on the cache of the corresponding level according to different execution cycles. For example, a thread pool for handling HIGH-level caches can perform data elimination operations with a shorter execution cycle to prevent insufficient free space for the HIGH-level cache; thread pools for handling IDLE-level caches can be longer. The execution cycle performs data elimination operations to save system overhead. In summary, the number of executions and the execution timing of the above-mentioned step S230 and step S240 can be determined in a variety of manners by a person skilled in the art according to actual needs, which is not limited by the disclosure. It can be seen that the division of the cache level and the application of the thread pool technology provide more flexibility and controllability for the data elimination operation, and can meet the needs of various scenarios.

In the embodiment of the present disclosure, a specific method for performing data elimination may be flexibly set by a person skilled in the art, which is not limited in the disclosure. For example, it can be eliminated based on various factors such as data write time, number of data writes, data temperature attributes, and data types. In this embodiment, data mining The method may be: calculating a temperature attribute value of each data in the cache according to a total number of times of writing each data in the cache and a preset temperature attribute calculation rule, and determining an elimination order of each data in the cache according to the temperature attribute value. .

The preset temperature attribute calculation rule is a rule for calculating the popularity degree of each cached data set by a person skilled in the art according to actual conditions. Here, the popularity of the cached data can be determined by factors such as the total number of times the cached data is written, and/or the storage period of the cached data. Specifically, when calculating the temperature attribute value of each cached data, the temperature attribute value of each cached data may be separately calculated according to the total number of writes of each cached data; the temperature attribute value of each cached data may be further calculated in combination with other factors. The present disclosure does not limit the specific calculation rule of the temperature attribute value, as long as it can meet the actual needs of the user.

After calculating the temperature attribute values of the respective cache data, the cached data having the lowest temperature attribute value is sequentially eliminated according to the calculated temperature attribute values from low to high, thereby achieving the effect of eliminating the data according to the popularity of the cached data. And release the cache space in a timely and effective manner.

In addition, various modifications and changes may be made to the above described embodiments. For example, when the temperature attribute is determined according to the total number of times of writing, in addition to directly determining the value according to the total number of times of writing, the total number of times of writing may be divided into a plurality of numerical intervals in advance, and corresponding intervals are respectively set for each numerical interval. The score is determined and the temperature attribute value is determined based on the interval score. For example, when the total number of writes belongs to the value range [0, 10], the interval score is 1; when the total number of writes belongs to the value range [10, 50], the interval score is 5; When the total number of entries is in the range of [50,100], the interval score is 10. The interval score can be used to more flexibly determine the data whose total number of writes is within a certain interval as hot data. Moreover, in order to make the data elimination mode more flexible, the preset temperature attribute calculation rule may further include: further dividing the cache duration corresponding to the cache into a plurality of buffer periods, and respectively setting corresponding period weight values for each buffer period; For each cached data, the temperature attribute value of the cached data is determined according to the period weight value of the corresponding cache period when the cache data is written each time. The buffer duration may be: a length of time defined by a first data write time corresponding to the data with the oldest write time in the cache and a second data write time corresponding to the latest data of the write time. Moreover, the buffer duration can also be a preset length of time. For example, suppose a cache is dedicated to storing cached data within the last three hours, and is automatically deleted once the write cached data is written for more than three hours. , the cache cache time is 3 hours. When the cache duration is divided into multiple cache periods, the entire cache duration can be divided into multiple An equal cache period can also divide the entire cache duration into multiple cache periods. In order to facilitate the calculation of the temperature attribute of the cache data according to the buffer period, after performing the above division, optionally, a period data table corresponding to the buffer period may be separately set for each buffer period, wherein each period data table is used for recording. The cached data written during the corresponding cache period. In this embodiment, it is also required to set a corresponding time slot weight value for each of the divided buffer periods in the present embodiment, and the setting manner thereof is also diverse. Specifically, the weight values of the respective time periods may be set to be equal, so that the temperature attribute value of the cached data is calculated from the aspect of the number of occurrences of each cached data; or, the time period may be followed according to the cache time period. The subsequent order correspondingly sets an incremented (or decremented) time slot weight value, such that the focus is on combining the number of occurrences of the cached data with the write time to calculate the temperature attribute value of each cached data. Here, the weight value setting of each time period is determined by a person skilled in the art according to actual conditions, and the disclosure does not limit this. In short, the setting of the buffer period and the time weight value enables the user to prioritize the elimination of data in non-critical time periods according to actual needs, so that the elimination scheme is more flexible.

In addition, the foregoing preset level division rules may be further divided according to other types of data stored in the cache, in addition to the storage space, and in summary, the manner of dividing the cache level and the weight setting of the thread pool in the present disclosure. The method is not limited.

In the embodiment of the present disclosure, each of the above thread pools runs in parallel with each other, thereby further improving data processing efficiency.

Therefore, the data elimination method based on multiple caches provided in Embodiment 2 of the present disclosure may divide multiple cache levels according to a preset level division rule, respectively create matching thread pools for each cache level; use each thread pool. Each of the multiple threads scans each cache, determines the cache level of each cache according to the scan result and the level division rule; and uses multiple threads in each thread pool to eliminate the data in the cache whose cache level matches the thread pool. Therefore, the problem of low efficiency of single-thread processing in the prior art is solved, and multi-thread parallel operation of data elimination operations on different weighted cache sets is realized, which ensures the consistency of data elimination and greatly improves the efficiency of data elimination processing. Through the improvement of the thread pool, the elimination priority of the cache set can be guaranteed in parallel.

Embodiment 3

FIG. 3 shows a data eliminator based on multiple caches provided by Embodiment 3 of the present disclosure. Schematic diagram of the structure, as shown, the apparatus includes: a partitioning module 310, a scanning module 320, and a culling module 330.

The dividing module 310 is configured to divide a plurality of cache levels according to a preset level dividing rule, and respectively create matching thread pools for each cache level.

The preset level division rule is used to divide each cache into different levels according to different usage conditions, and each cache in the same level has similar usage. This level is artificially defined by the technician. For the specific content of the preset level division rule, the third embodiment of the present disclosure does not specifically limit this, and those skilled in the art can flexibly set according to actual conditions.

In order to improve the processing efficiency of data elimination, the partitioning module 310 respectively creates matching thread pools for each cache level, and each thread pool contains multiple threads. Multiple threads in each thread pool are used for data elimination processing of the corresponding level of cache. Because different levels of cache usage are different, in order to optimize resource allocation as much as possible, the number of threads in the thread pool corresponding to different levels may also be different.

The scanning module 320 is configured to separately scan each cache by using multiple threads in each thread pool, and determine the cache level of each cache according to the scan result and the level division rule.

Specifically, the scanning module 320 scans each cache by using multiple threads in each thread pool, and determines a cache level for each scanned cache according to the scan result and the level division rule, for subsequent targeted processing.

The eliminating module 330 is configured to use multiple threads in each thread pool to eliminate data in a cache whose cache level matches the thread pool.

Specifically, according to the cache level determined by the scanning module 320, the culling module 330 performs data elimination processing on the cache having the corresponding cache level by using a plurality of threads in the thread pool matching the respective cache levels. For the specific method of the data elimination processing, the third embodiment of the present disclosure does not specifically limit this. Those skilled in the art can flexibly set according to actual conditions.

For the specific structure and working principle of each module mentioned above, reference may be made to the description of corresponding parts in the method embodiments, and details are not described herein again.

It can be seen that the data culling device based on multiple caches according to the embodiment of the present disclosure can divide multiple cache levels according to a preset level division rule, and respectively create matching thread pools for each cache level; The plurality of threads respectively scan each cache, determine the cache level of each cache according to the scan result and the level division rule; and use multiple threads in each thread pool to eliminate the data in the cache whose cache level matches the thread pool. It can be seen that by dividing the cache into multiple cache levels and respectively creating corresponding thread pools for each cache level, the number of threads in the thread pool can be better adjusted according to the cache level; and, through multiple thread pools, parallel processing The way to greatly improve the efficiency of data elimination processing.

Embodiment 4

FIG. 4 is a schematic structural diagram of a data eliminator device based on multiple caches according to Embodiment 4 of the present disclosure. As shown, the device includes: a partitioning module 410, a weighting module 420, a scanning module 430, and a culling module 440. .

The dividing module 410 is configured to divide a plurality of cache levels according to a preset level dividing rule, and respectively create matching thread pools for each cache level.

In order to improve the processing efficiency of data elimination, the partitioning module 410 creates matching thread pools for each cache level, and each thread pool contains multiple threads. Multiple threads in each thread pool are used for data elimination processing of the corresponding level of cache. Because different levels of cache usage is different In order to optimize resource allocation as much as possible, the number of threads in the thread pool corresponding to different levels is also different.

The weight module 420 is configured to separately set a corresponding weight value for each thread pool, and set the number of threads included in each thread pool according to the weight value of each thread pool.

For the specific setting method of the weight value, for each thread pool, the weight module 420 sets the weight value corresponding to the thread pool according to the level of the cache level matched with the thread pool, wherein the cache level matching the thread pool The higher the value of the thread pool, the greater the weight value; conversely, the lower the cache level that matches the thread pool, the smaller the weight value of the thread pool. The greater the weight value of the thread pool, the greater the number of threads included in the thread pool; the smaller the weight value of the thread pool, the fewer the number of threads contained in the thread pool. Therefore, the number of threads contained in each thread pool is dynamically changing.

The scanning module 430 is configured to separately scan each cache by using multiple threads in each thread pool, and determine a cache level of each cache according to the scan result and the level division rule.

Specifically, the scanning module 430 scans each cache by using multiple threads in each thread pool, and determines a cache level for each scanned cache according to the scan result and the level division rule, for subsequent targeted processing.

In other embodiments, the weight value setting method of the thread pool provided in the weight module 420 and the scanning module 430 may be comprehensively combined to set a more reasonable weight value of the thread pool. In addition, the weight value of the thread pool can be further determined according to various factors such as the type and importance of the cache of the corresponding level.

The elimination module 440: utilizes multiple threads in each thread pool to eliminate data in the cache whose cache level matches the thread pool.

Specifically, according to the cache level determined by the above module, the elimination module 440 performs data elimination processing on the cache with the corresponding cache level by using multiple threads in the thread pool matching the respective cache levels. Each thread pool can process only one cache level cache. For example, the partition module 410 divides the cache into three levels of a HIGH level, a LOW level, and an IDLE level, so only three thread pools are required. Specifically, the thread pool 1 corresponds to the HIGH level, the thread pool 2 corresponds to the LOW level, and the thread pool 3 corresponds to the IDLE level. In this case, all the threads in the thread pool 1 only process all the caches in the HIGH level, and the thread pool 2 All threads only process all caches in the LOW level, and all threads in thread pool 3 only process all caches in the IDLE level. Of course, when there are more cache levels, each thread pool can also be used to handle multiple cache level caches. For example, when the cache level includes six levels, it can also be processed by three thread pools, each of which handles two levels of cache.

In short, through the division of the cache level and the application of the thread pool technology, the cache scanning and data elimination work can be more flexibly implemented. In addition, the scanning module 430 and the eliminating module 440 can be repeatedly operated multiple times. For example, the scanning module 430 can be run once every preset first time interval, and the eliminating module 440 can be operated every preset second time interval. once. The first time interval and the second time interval may be equal or may not be equal. In addition, the first time interval and the second time interval may be either fixed values or dynamically changing values. For example, the first time interval may be dynamically adjusted according to the scan result: when the number of caches of the HIGH level is large in the scan result, the first time interval is reduced; when the number of caches of the HIGH level in the scan result is small, the first time is increased. interval. In addition, during each operation of the elimination module 440, each thread pool can perform the elimination operation on the cache of the corresponding level according to the same execution cycle, or perform the elimination operation on the cache of the corresponding level according to different execution cycles. For example, a thread pool for handling HIGH-level caches can perform data elimination operations with a shorter execution cycle to prevent insufficient free space for the HIGH-level cache; the thread pool for handling IDLE-level caches can be compared A long execution cycle performs data elimination operations to save system overhead. In summary, the number of times of running and the running time of the scanning module 430 and the eliminating module 440 can be determined in a variety of manners by a person skilled in the art according to actual needs, which is not limited by the disclosure. It can be seen that the division of the cache level and the application of the thread pool technology provide more flexibility and controllability for the data elimination operation, and can meet the needs of various scenarios.

In the embodiment of the present disclosure, the specific method for the data elimination by the eliminating module 440 can be flexibly set by a person skilled in the art, which is not limited by the disclosure. For example, it can be eliminated based on various factors such as data write time, number of data writes, data temperature attributes, and data types. In this embodiment, the data elimination method may be: calculating a temperature attribute value of each data in the cache according to a total number of writes of each data in the cache and a preset temperature attribute calculation rule, and determining a cache according to the temperature attribute value. The order of elimination of each data within.

In addition, various modifications and changes may be made to the above described embodiments. For example, when the temperature attribute is determined according to the total number of times of writing, in addition to directly determining the value according to the total number of times of writing, the total number of times of writing may be divided into a plurality of numerical intervals in advance, and corresponding intervals are respectively set for each numerical interval. The score is determined and the temperature attribute value is determined based on the interval score. For example, when the total number of writes belongs to the value range [0, 10], the interval score is 1; when the total number of writes belongs to the value range [10, 50], the interval score is 5; When the total number of entries is in the range of [50,100], the interval score is 10. The interval score can be used to more flexibly determine the data whose total number of writes is within a certain interval as hot data. Moreover, in order to make the data elimination mode more flexible, the foregoing preset temperature attribute calculation rule may further include: pre-setting the cache duration corresponding to the cache into one Steps are divided into a plurality of buffer periods, and corresponding period weight values are respectively set for each buffer period; for each cache data, a temperature attribute of the cache data is determined according to a period weight value of the corresponding buffer period when the cache data is written each time value. The buffer duration may be: a length of time defined by a first data write time corresponding to the data with the oldest write time in the cache and a second data write time corresponding to the latest data of the write time. Moreover, the buffer duration can also be a preset length of time. For example, suppose a cache is dedicated to storing cached data within the last three hours, and is automatically deleted once the write cached data is written for more than three hours. , the cache cache time is 3 hours. When the cache duration is divided into multiple cache periods, the entire cache duration may be divided into multiple equal cache periods, or the entire cache duration may be divided into multiple cache periods. In order to facilitate the calculation of the temperature attribute of the cache data according to the buffer period, after performing the above division, optionally, a period data table corresponding to the buffer period may be separately set for each buffer period, wherein each period data table is used for recording. The cached data written during the corresponding cache period. In this embodiment, it is also required to set a corresponding time slot weight value for each of the divided buffer periods in the present embodiment, and the setting manner thereof is also diverse. Specifically, the weight values of the respective time periods may be set to be equal, so that the temperature attribute value of the cached data is calculated from the aspect of the number of occurrences of each cached data; or, the time period may be followed according to the cache time period. The subsequent order correspondingly sets an incremented (or decremented) time slot weight value, such that the focus is on combining the number of occurrences of the cached data with the write time to calculate the temperature attribute value of each cached data. Here, the weight value setting of each time period is determined by a person skilled in the art according to actual conditions, and the disclosure does not limit this. In short, the setting of the buffer period and the time weight value enables the user to prioritize the elimination of data in non-critical time periods according to actual needs, so that the elimination scheme is more flexible.

It can be seen that the data elimination device based on multiple caches provided by Embodiment 4 of the present disclosure may be Dividing a plurality of cache levels according to a preset level division rule, respectively creating a matching thread pool for each cache level; scanning each cache separately by using multiple threads in each thread pool, and determining each cache according to the scan result and the level division rule Cache level; utilizes multiple threads in each thread pool to eliminate data in the cache with a cache level that matches the thread pool. Therefore, the problem of low efficiency of single-thread processing in the prior art is solved, and multi-thread parallel operation of data elimination operations on different weighted cache sets is realized, which ensures the consistency of data elimination and greatly improves the efficiency of data elimination processing. Through the improvement of the thread pool, the elimination priority of the cache set can be guaranteed in parallel.

FIG. 5 schematically illustrates a block diagram of a computing device for performing a multiple cache based data retirement method in accordance with an embodiment of the present disclosure. The computing device conventionally includes a processor 510 and a computer program product or computer readable medium in the form of a storage device 520. Storage device 520 can be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Storage device 520 has a storage space 530 that stores program code 531 for performing any of the method steps described above. For example, storage space 530 storing program code may include various program code 531 for implementing various steps in the above methods, respectively. The program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as a hard disk, a compact disk (CD), a memory card, or a floppy disk. Such a computer program product is typically a portable or fixed storage unit such as that shown in FIG. The storage unit may have storage segments, storage spaces, and the like that are similarly arranged to storage device 520 in the computing device of FIG. The program code can be compressed, for example, in an appropriate form. Typically, the storage unit includes computer readable code 531' for performing the steps of the method in accordance with the present disclosure, ie, code that can be read by a processor, such as 510, which when executed by the computing device causes the computing device Perform the various steps in the method described above.

The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general purpose systems can also be used with the teaching based on the teachings herein. The structure required to construct such a system is apparent from the above description. Moreover, the present disclosure is not directed to any particular programming language. It is to be understood that the subject matter of the present disclosure, which is described herein, may be described in a particular language.

In the description provided herein, numerous specific details are set forth. However, can understand, this The disclosed embodiments may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of the description.

In the description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, Figure, or a description of it. However, the method disclosed is not to be interpreted as reflecting the intention that the claimed invention requires more features than those recited in the claims. Rather, as disclosed in the following claims, the disclosed aspects are less than all features of the single embodiments disclosed herein. Therefore, the claims following the specific embodiments are hereby explicitly incorporated into the specific embodiments, and each of the claims as a separate embodiment of the present disclosure.

Those skilled in the art will appreciate that the modules in the devices of the embodiments can be adaptively changed and placed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and further they may be divided into a plurality of sub-modules or sub-units or sub-components. In addition to such features and/or at least some of the processes or units being mutually exclusive, any combination of the features disclosed in the specification, including the accompanying claims, the abstract and the drawings, and any methods so disclosed, or All processes or units of the device are combined. Each feature disclosed in this specification (including the accompanying claims, the abstract and the drawings) may be replaced by alternative features that provide the same, equivalent or similar purpose.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features that are included in other embodiments and not in other features, combinations of features of different embodiments are intended to be within the scope of the present disclosure. Different embodiments are formed and formed. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

Various component embodiments of the present disclosure may be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some of some or all of the components of a plurality of cache-based data elimining devices in accordance with embodiments of the present disclosure. Or all features. The present disclosure may also be implemented as a device or device program (eg, a computer program and a computer program product) for performing some or all of the methods described herein. Such a program implementing the present disclosure may be stored on a computer readable medium or may be in the form of one or more signals. Such signals may be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

It should be noted that the above-described embodiments are illustrative of the present disclosure and are not intended to limit the scope of the disclosure, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as a limitation. The word "comprising" does not exclude the presence of the elements or steps that are not recited in the claims. The word "a" or "an" The present disclosure can be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means can be embodied by the same hardware item. The use of the words first, second, and third does not indicate any order. These words can be interpreted as names.

Claims

A data culling method based on multiple caches, including:

Dividing a plurality of cache levels according to a preset level division rule, respectively creating a matching thread pool for each cache level; wherein each thread pool includes multiple threads;

Each cache is scanned by multiple threads in each thread pool, and the cache level of each cache is determined according to the scan result and the level division rule;

The data in the cache with the cache level matching the thread pool is eliminated by using multiple threads in each thread pool.
The method of claim 1, further comprising: setting a corresponding weight value for each thread pool, and setting a number of threads included in each thread pool according to a weight value of each thread pool; wherein, the weight value of the thread pool is higher Large, the number of threads contained in the thread pool is larger.
The method of claim 2, wherein the step of setting a corresponding weight value for each thread pool, and setting the number of threads included in each thread pool according to the weight value of each thread pool comprises:

Regularly obtaining scan results of each thread pool, and determining the number of caches corresponding to each cache level according to the scan result;

Adjusting the weight values of the respective thread pools according to the number of caches corresponding to the respective cache levels, and adjusting the number of threads included in each thread pool according to the adjusted weight values of the respective thread pools;

The more the cache level corresponding to the cache level, the larger the weight value of the thread pool matching the cache level.
The method of claim 2 or 3, wherein the step of separately setting a corresponding weight value for each thread pool further comprises:

For each thread pool, the weight value corresponding to the thread pool is set according to the level of the cache level matched with the thread pool; wherein the higher the cache level matching the thread pool, the greater the weight value of the thread pool.
The method according to any one of claims 1 to 4, wherein the preset ranking rule The method includes: dividing a cache level according to a ratio between a remaining storage space of the cache and a total storage space, wherein a larger ratio between the remaining storage space and the total storage space is, and a cache level is higher.
The method according to any one of claims 1-5, wherein the step of using the plurality of threads in each thread pool to eliminate the data in the cache matching the cache level and the thread pool comprises:

Calculating a temperature attribute value of each data in the cache according to a total number of writes of each data in the cache and a preset temperature attribute calculation rule, and determining each data in the cache according to the temperature attribute value The order of elimination.
The method of any of claims 1-6, wherein the respective thread pools run in parallel with each other.
A data culling device based on multiple caches, comprising:

a dividing module, configured to divide a plurality of cache levels according to a preset level dividing rule, and respectively create a matching thread pool for each cache level; wherein each thread pool includes multiple threads;

a scanning module, configured to scan each cache separately by using multiple threads in each thread pool, and determine a cache level of each cache according to the scan result and the level division rule;

The culling module is configured to use multiple threads in each thread pool to eliminate data in a cache whose cache level matches the thread pool.
The device according to claim 8, further comprising: a weighting module, configured to respectively set a corresponding weight value for each thread pool, and set a number of threads included in each thread pool according to a weight value of each thread pool; wherein, the thread The larger the weight value of the pool, the greater the number of threads contained in the thread pool.
The apparatus according to claim 9, wherein the weight module is specifically configured to:

Regularly obtaining scan results of each thread pool, and determining the number of caches corresponding to each cache level according to the scan result;

Adjusting the weight values of the respective thread pools according to the number of caches corresponding to the respective cache levels, and adjusting the number of threads included in each thread pool according to the adjusted weight values of the respective thread pools;

The more the cache level corresponding to the cache level, the larger the weight value of the thread pool matching the cache level.
The apparatus of claim 9 or 10, wherein the weighting module is further configured to:

For each thread pool, the weight value corresponding to the thread pool is set according to the level of the cache level matched with the thread pool; wherein the higher the cache level matching the thread pool, the greater the weight value of the thread pool.
The device according to any one of claims 8-11, wherein the preset level dividing rule comprises: dividing a cache level according to a ratio between a cached remaining storage space and a total storage space, wherein the remaining storage space and the total The larger the ratio between storage spaces, the higher the cache level.
The apparatus according to any one of claims 8 to 12, wherein the elimination module is specifically configured to:

Calculating a temperature attribute value of each data in the cache according to a total number of writes of each data in the cache and a preset temperature attribute calculation rule, and determining each data in the cache according to the temperature attribute value The order of elimination.
The apparatus of any of claims 8-13, wherein the respective thread pools run in parallel with each other.
A computer program comprising computer readable code, when the computer readable code is run on a computing device, causing the computing device to perform a plurality of cache-based according to any one of claims 1-7 Data elimination method.
A computer readable medium storing the computer program of claim 15.