CN112948444A - Management method and device for cache data - Google Patents

Management method and device for cache data Download PDF

Info

Publication number
CN112948444A
CN112948444A CN202110371767.5A CN202110371767A CN112948444A CN 112948444 A CN112948444 A CN 112948444A CN 202110371767 A CN202110371767 A CN 202110371767A CN 112948444 A CN112948444 A CN 112948444A
Authority
CN
China
Prior art keywords
partition
target data
linked list
data
level
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110371767.5A
Other languages
Chinese (zh)
Inventor
谢凯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Original Assignee
Beijing Dianzhi Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dianzhi Technology Co ltd filed Critical Beijing Dianzhi Technology Co ltd
Priority to CN202110371767.5A priority Critical patent/CN112948444A/en
Publication of CN112948444A publication Critical patent/CN112948444A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/901Indexing; Data structures therefor; Storage structures
    • G06F16/9024Graphs; Linked lists

Abstract

The invention discloses a method and a device for managing cache data, and relates to the technical field of computers. One embodiment of the method comprises: receiving an access request for target data; when the target data exist in the LRU linked list of the buffer pool, determining the level of the current partition of the target data in the LRU linked list and reading the accessed frequency of the target data; acquiring a grading frequency threshold corresponding to the grade of the current partition from a parameter configuration library to judge whether the accessed frequency is greater than the grading frequency threshold; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition. According to the implementation method, only a part of space in the linked list can be used for loading new data, and the data with high hit rate exists at the head of the linked list as much as possible and is not easy to eliminate, so that the use effect of the LRU linked list is optimized, and the reading efficiency of the cache data is optimized.

Description

Management method and device for cache data
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for managing cache data.
Background
When data in a system is read, a buffer pool (buffer pool) mechanism is usually used to put the data which is accessed frequently into the buffer pool, so as to avoid accessing the disk each time, and further accelerate the access of the data. An LRU (Least Recently Used) linked list is often Used in the buffer pool to manage the data in the buffer pool.
In the process of implementing the invention, the prior art at least has the following problems:
1) when a certain data page is loaded into the LRU linked list from the disk, other data pages adjacent to the data page are always pre-read and loaded into the linked list, the adjacent data pages cannot be used until being eliminated, and other data pages with higher hit rate than the pre-read invalid data pages are eliminated; the condition of the pre-reading failure of the adjacent data pages can occupy the space of the LRU linked list, influence the use effect of the LRU linked list and further influence the reading efficiency of the cache data;
2) when a large number of data pages exceeding the storage space of the linked list are required to be loaded from the disk to the LRU linked list, all original data pages in the linked list are replaced by new data pages, a plurality of data pages with high hit rate often exist in the replaced data pages, and the data pages with high hit rate are usually read into the linked list quickly; the pollution of the buffer pool can also affect the use effect of the LR U linked list, and further affect the reading efficiency of the buffer data.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for managing cached data, which can divide an LRU linked list into different levels and set a forward or progressive manner when data is accessed, so that data with a true high hit rate exists at the head of the linked list as much as possible and is less likely to be eliminated, and only a part of space in the linked list can be used for loading new data, thereby optimizing the use effect of the LRU linked list and optimizing the reading efficiency of the cached data as a whole.
In order to achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a method for managing cache data, including:
receiving an access request for target data;
when the target data exist in an LRU linked list of a buffer pool, determining the level of a current partition of the target data in the LRU linked list, and reading the accessed frequency of the target data; wherein the LRU linked list comprises at least three partitions of different levels;
acquiring a ranking frequency threshold corresponding to the level of the current partition from a parameter configuration library to judge whether the accessed frequency is greater than the ranking frequency threshold; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition.
Optionally, after receiving the access request for the target data, the method further includes:
when the target data does not exist in the LRU linked list of the buffer pool, reading the target data from the disk so as to place the target data into the head of the lowest-level partition in the LRU linked list of the buffer pool.
Optionally, the method for managing cache data provided by the present invention further includes:
all the partitions in the LRU linked list are sequentially arranged from the tail part of the linked list to the head part of the linked list according to the sequence of the levels from low to high; wherein, for two adjacent partitions, the head of the lower-level partition is connected with the tail of the higher-level partition.
Optionally, the method for managing cache data provided by the present invention further includes:
in the LRU linked list, the occupation ratio of each partition with the level from low to high is increased based on a power law.
Optionally, before obtaining the ranking frequency threshold corresponding to the level of the current partition from the parameter configuration library, the method further includes determining the ranking frequency threshold in the parameter configuration library according to the following method:
acquiring the access quantity information of the data in the LRU linked list in a preset time period based on a preset frequency;
calling a threshold analysis model, calculating a progressive frequency threshold corresponding to each partition based on the access amount information, and storing the progressive frequency threshold into the parameter configuration library; wherein the content of the first and second substances,
and the advance frequency threshold corresponding to the highest-level partition in the LRU linked list is an upper limit threshold calculated according to the access amount information, and the upper limit threshold is greater than the accessed frequency of all data in the LRU linked list in the preset time period.
Optionally, when the frequency of access is greater than the advance frequency threshold, moving the target data to a partition one level higher than the current partition includes:
when the current partition where the target data is located is not the highest-level partition in the LRU linked list, acquiring the accessed frequency of each data in the higher-level partition;
determining a position to be inserted of the target data by adopting a preset insertion algorithm based on the accessed frequency of each data;
and moving the target data to a position to be inserted in the higher-level partition.
Optionally, the method for managing cache data provided by the present invention further includes:
when the current partition in which the target data is located is the highest-level partition in the LRU linked list and the accessed frequency of the target data is judged to be greater than the advanced frequency threshold, calculating the advanced frequency threshold corresponding to each partition again according to the method provided by the invention to update the parameter configuration library, and moving the target data to the head of the current partition.
According to still another aspect of the embodiments of the present invention, there is provided a management apparatus for caching data, including:
the receiving module is used for receiving an access request for target data;
the reading module is used for determining the level of a current partition of the target data in an LRU linked list of a buffer pool when the target data exists in the LRU linked list, and reading the accessed frequency of the target data; wherein the LRU linked list comprises at least three partitions of different levels;
the mobile module is used for acquiring a ranking frequency threshold value corresponding to the level of the current partition from a parameter configuration library so as to judge whether the accessed frequency is greater than the ranking frequency threshold value; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition.
Optionally, after the receiving module receives the access request for the target data, the reading module is further configured to: when the target data does not exist in the LRU linked list of the buffer pool, reading the target data from a disk; and the number of the first and second groups,
the mobile module is further configured to: and putting the target data into the head of the lowest-level partition in the LRU linked list of the buffer pool.
Optionally, in the management apparatus for caching data provided by the invention,
all the partitions in the LRU linked list are sequentially arranged from the tail part of the linked list to the head part of the linked list according to the sequence of the levels from low to high; wherein, for two adjacent partitions, the head of the lower-level partition is connected with the tail of the higher-level partition.
Optionally, in the management apparatus for caching data provided by the present invention, in the LRU linked list, the ratio of each partition whose level is from low to high is increased based on a power law.
Optionally, the management apparatus for cache data provided by the present invention further includes: a configuration module;
before the mobile module obtains the ranking frequency threshold corresponding to the level of the current partition from the parameter configuration library, the configuration module is configured to determine the ranking frequency threshold in the parameter configuration library according to the following method:
acquiring the access quantity information of the data in the LRU linked list in a preset time period based on a preset frequency;
calling a threshold analysis model, calculating a progressive frequency threshold corresponding to each partition based on the access amount information, and storing the progressive frequency threshold into the parameter configuration library; wherein the content of the first and second substances,
and the advance frequency threshold corresponding to the highest-level partition in the LRU linked list is an upper limit threshold calculated according to the access amount information, and the upper limit threshold is greater than the accessed frequency of all data in the LRU linked list in the preset time period.
Optionally, when the frequency of access is greater than the advance frequency threshold, the moving module moves the target data to a partition higher than the current partition by one level, including:
when the current partition where the target data is located is not the highest-level partition in the LRU linked list, acquiring the accessed frequency of each data in the higher-level partition;
determining a position to be inserted of the target data by adopting a preset insertion algorithm based on the accessed frequency of each data;
and moving the target data to a position to be inserted in the higher-level partition.
Optionally, when the current partition in which the target data is located is the highest-level partition in the LRU linked list, and it is determined that the frequency of accessing the target data is greater than the ranking frequency threshold,
the configuration module is further configured to calculate the ranking frequency threshold corresponding to each partition again according to the method provided by the present invention to update the parameter configuration library, and the moving module is further configured to move the target data to the head of the current partition.
According to another aspect of the embodiments of the present invention, there is provided an electronic device for managing cache data, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the management method of the cache data provided by the invention.
According to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing the method for managing cached data provided by the present invention.
One embodiment of the above invention has the following advantages or benefits: because the method of dividing the LRU linked list into different levels and setting the forward or progressive mode when the data is accessed is adopted, the condition that the use effect of the LRU linked list is poor due to pre-reading failure or buffer pool pollution in the prior art is avoided, the data with the real high hit rate can exist at the head of the linked list as much as possible and is not easy to be eliminated, and only a part of space in the linked list can be used for loading new data, thereby optimizing the use effect of the LRU linked list and integrally optimizing the reading efficiency of the cache data.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a management method for cache data according to a first embodiment of the present invention;
fig. 2(a) is a schematic structural diagram of an LR U linked list in a management method for cached data according to a second embodiment of the present invention;
FIG. 2(b) is a diagram of an LRU linked list in an application scenario according to a second embodiment of the present invention;
FIG. 2(c) is a diagram of an LRU linked list in another application scenario according to the second embodiment of the present invention;
fig. 3 is a flowchart illustrating a method for managing cache data according to a third embodiment of the present invention;
fig. 4 is a schematic diagram of main modules of a management apparatus for caching data according to a fourth embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 6 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a management method for cache data according to a first embodiment of the present invention, as shown in fig. 1, including:
step S101, receiving an access request for target data;
step S102, when the target data exists in an LRU linked list of a buffer pool, determining the level of a current partition of the target data in the LRU linked list, and reading the accessed frequency of the target data; wherein the LRU linked list comprises at least three partitions of different levels;
step S103, acquiring a progressive frequency threshold corresponding to the level of the current partition from a parameter configuration library to judge whether the accessed frequency is greater than the progressive frequency threshold; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition.
The LRU linked list in the buffer pool may be divided into at least three parts, such as: advanced (sensor-sublist), middle-sublist, primary (primary-sublist), based on the idea of the LR U linked list, data can be advanced when accessed in the linked list: the data can be moved from the original position in the partition to the head of the partition, or moved to a higher-level partition; the manner of shifting forward may be determined based on the frequency of accessing the data within a preset time period, wherein the frequency of accessing the data may effectively reflect the used condition of the data,
according to the method, the LRU linked list is divided into different levels, when the data are accessed, whether the data meet the ranking condition is judged according to the access frequency of the data, and then the mode of advancing the data is determined, so that the data with the real high hit rate exist at the head of the linked list as much as possible and are not easy to be eliminated, the use effect of the LRU linked list is optimized, and the reading efficiency of the cache data is integrally optimized.
In some embodiments, after receiving the access request for the target data, the method further comprises:
when the target data does not exist in the LRU linked list of the buffer pool, reading the target data from the disk so as to place the target data into the head of the lowest-level partition in the LRU linked list of the buffer pool.
A portion of the space in the linked list (e.g., the primary partition as described above) may be used for loading new data, and when new data is loaded to the head of the primary partition, a corresponding amount of data at the tail of the primary partition is eliminated;
therefore, when a large number of data pages exceeding the storage space of the whole linked list are required to be loaded into the LRU linked list from the disk, only all the data pages in the primary partition are replaced by new data pages, the data in the higher-level partition cannot be influenced, and the pollution condition of the buffer pool is avoided, so that the use effect of the LRU linked list is optimized, and the reading efficiency of the cached data is integrally optimized.
In some embodiments, the method for managing cached data provided by the present invention further includes:
all the partitions in the LRU linked list are sequentially arranged from the tail part of the linked list to the head part of the linked list according to the sequence of the levels from low to high; wherein, for two adjacent partitions, the head of the lower-level partition is connected with the tail of the higher-level partition.
In some embodiments, the method for managing cached data provided by the present invention further includes:
in the LRU linked list, the occupation ratio of each partition with the level from low to high is increased based on a power law.
In the LRU linked list, high-level, medium-level and primary partitions can be set end to end, that is: the tail (tail) of the high-level partition is connected with the head (head) of the middle-level partition, and the tail (tail) of the middle-level partition is connected with the head (head) of the primary partition;
it can also be set that the ratio of each partition with the level from low to high is increased based on the power law, such as: the high-level, middle-level and primary partition ratio is 4:2:1, so that a small part of space in the whole LRU linked list is used for replacing new data, a large part of space is used for storing data with high access rate, the data with high access rate is eliminated later, and the using effect of the LRU linked list is optimized;
in some practical applications, the ratio of each partition in the LRU linked list can be increased or decreased to adapt to different environments.
Fig. 2(a) is a schematic structural diagram of an LR U linked list in a management method for cached data according to a second embodiment of the present invention; as shown in fig. 2 (a):
the LRU linked list comprises a high-level (sensor-sublist), a medium-level (middle-sublist) and a primary-level (primary-sublist), wherein data 1-5 are located in a high-level partition and occupy 5 spatial positions; 6-12 data are located in the middle-level partition and occupy 7 spatial positions; data 13-15 are located in the primary partition, occupying 3 spatial positions.
FIG. 2(b) is a diagram of an LRU linked list in an application scenario according to a second embodiment of the present invention; as shown in fig. 2 (b):
when data 50 needs to be loaded from disk into the linked list, data 50 will be loaded to the head of the primary partition, while data 15 at the tail of the primary partition will be evicted from the linked list.
FIG. 2(c) is a diagram of an LRU linked list in another application scenario according to the second embodiment of the present invention;
when a large amount of data 51-55 needs to be loaded from disk into the list, the data 51-55 will not occupy as much space as possible from the head of the LRU list, but from the head of the primary partition of the LRU list, as shown in fig. 2 (c): data 51-55 will be inserted from the head of the primary partition; after loading is finished, 13-15 in the original primary partition are eliminated, 51-53 in the data 51-55 are loaded into the primary partition, and therefore the situation that the space of the LRU linked list is excessively occupied by new data and pollution of a buffer pool is avoided.
In some embodiments, before obtaining the ranking frequency threshold corresponding to the level of the current partition from the parameter configuration library, the method further includes determining the ranking frequency threshold in the parameter configuration library according to the following method:
acquiring the access quantity information of the data in the LRU linked list in a preset time period based on a preset frequency;
calling a threshold analysis model, calculating a progressive frequency threshold corresponding to each partition based on the access amount information, and storing the progressive frequency threshold into the parameter configuration library; wherein the content of the first and second substances,
and the advance frequency threshold corresponding to the highest-level partition in the LRU linked list is an upper limit threshold calculated according to the access amount information, and the upper limit threshold is greater than the accessed frequency of all data in the LRU linked list in the preset time period.
The threshold analysis model can be trained in advance, and can be trained through the cache utilization rate or QPS value recorded in different stages by the system; calculating to obtain a grading frequency threshold value corresponding to each partition suitable for the current environment based on the access amount information through a threshold value analysis model; typically, the rank threshold derived by the threshold analysis model is adapted to the overall visited frequency distribution of the data in the corresponding partition, such as: if the average value of the accessed frequency of the data in a certain partition is a, the difference value of the advanced frequency b and the advanced frequency a corresponding to the partition is within a preset range;
because the data in the highest-level partition in the linked list cannot be subjected to ranking, for the highest-level partition, the ranking frequency threshold corresponding to the highest-level partition can be set as an upper limit threshold calculated according to the access amount information, and the upper limit threshold can be larger than the accessed frequency of all the data in the LRU linked list; when data in the highest level partition is accessed, the frequency with which the data is accessed will typically be less than the corresponding advance frequency, and it will be decided to move the data to the head of the highest level partition.
In some embodiments, when the frequency of access is greater than the rank frequency threshold, moving the target data to a higher rank partition than the current partition comprises:
when the current partition where the target data is located is not the highest-level partition in the LRU linked list, acquiring the accessed frequency of each data in the higher-level partition;
determining a position to be inserted of the target data by adopting a preset insertion algorithm based on the accessed frequency of each data;
and moving the target data to a position to be inserted in the higher-level partition.
When the target data moves forward in a progressive mode, determining a position to be inserted of the target data based on the accessed frequency of each data in the partition to be progressive; such as: determining the position to be inserted according to the distribution condition of the access frequency of each data in the partition to be graded and the access frequency of the target data; or determining a frequency interval according to the maximum value and the minimum value of the accessed frequency of the data in the partition to be graded, determining the position of the accessed frequency of the target data in the frequency interval, and further determining the position to be inserted of the target data; when the target data moves forward to a higher-level partition, the data originally positioned at the tail part in the higher-level partition is moved forward to the head part of the partition where the target data is originally positioned;
the insertion algorithm can be flexibly adjusted according to different environments.
In some embodiments, the method for managing cached data provided by the present invention further includes:
when the current partition in which the target data is located is the highest-level partition in the LRU linked list and the accessed frequency of the target data is judged to be greater than the advanced frequency threshold, calculating the advanced frequency threshold corresponding to each partition again according to the method provided by the invention to update the parameter configuration library, and moving the target data to the head of the current partition.
When the visited frequency of the data in the highest-level partition is greater than the corresponding ranking frequency threshold (i.e. the upper threshold), the obtained ranking frequency threshold is not suitable for the current situation, so that the ranking frequency threshold corresponding to each partition more suitable for the current situation can be immediately calculated again to update the parameter configuration library, and the target data is moved to the head of the current partition; by the mode, the cache data management method is more reasonable and has stronger adaptability.
Fig. 3 is a flowchart illustrating a method for managing cache data according to a third embodiment of the present invention; as shown in fig. 3, includes:
step S301, receiving an access request for target data;
step S302, judging whether the target data exists in an LRU linked list in the buffer pool or not; if not, executing step S303; if yes, go to step S304;
step S303, loading the target data to the head of the lowest-level partition in the LRU linked list;
step S304, determining the level of the current partition of the target data in the LRU linked list, so as to obtain a ranking frequency threshold value corresponding to the level of the current partition from the parameter configuration library, and reading the accessed frequency of the target data;
step S305, judging whether the accessed frequency is greater than a progressive frequency threshold value; if not, executing step S306; if yes, go to step S307;
step S306, moving the target data to the head of the current subarea;
step S307, judging whether the current partition is the highest-level partition; if not, go to step S308; if yes, go to step S309;
step S308, moving the target data to a partition with a level higher than that of the current partition;
step S309, calculating the advance frequency threshold corresponding to each partition again to update the parameter configuration library, and moving the target data to the head of the current partition (i.e., the highest-level partition).
Fig. 4 is a schematic diagram of main modules of a management apparatus for caching data according to a fourth embodiment of the present invention, and as shown in fig. 4, the management apparatus 400 for caching data includes:
a receiving module 401, configured to receive an access request for target data;
a reading module 402, configured to determine, when the target data exists in an LRU linked list of a buffer pool, a level of a current partition of the target data in the LRU linked list, and read a frequency of being accessed of the target data; wherein the LRU linked list comprises at least three partitions of different levels;
a moving module 403, configured to obtain a ranking frequency threshold corresponding to the level of the current partition from a parameter configuration library, so as to determine whether the visited frequency is greater than the ranking frequency threshold; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition.
The LRU linked list in the buffer pool may be divided into at least three parts, such as: high-level (sensor-sublist), medium-level (middle-sublist), primary-level (primary-sublist), based on the idea of the LR U-linked list, data can be moved forward when accessed: the data can be moved from the original position in the partition to the head of the partition, or moved to a higher-level partition; the shifting mode of the data can be specifically judged based on the accessed frequency of the data in a preset time period, wherein the accessed frequency of the data can effectively reflect the used condition of the data;
according to the method, the LRU linked list is divided into different levels, when the data are accessed, whether the data meet the ranking condition is judged according to the access frequency of the data, and then the mode of advancing the data is determined, so that the data with the real high hit rate exist at the head of the linked list as much as possible and are not easy to be eliminated, the use effect of the LRU linked list is optimized, and the reading efficiency of the cache data is integrally optimized.
In some embodiments, after the receiving module receives the access request for the target data, the reading module is further configured to: when the target data does not exist in the LRU linked list of the buffer pool, reading the target data from a disk; and the number of the first and second groups,
the mobile module is further configured to: and putting the target data into the head of the lowest-level partition in the LRU linked list of the buffer pool.
A portion of the space in the linked list (e.g., the primary partition as described above) may be used for loading new data, and when new data is loaded to the head of the primary partition, a corresponding amount of data at the tail of the primary partition is eliminated;
therefore, when a large number of data pages exceeding the storage space of the whole linked list are required to be loaded into the LRU linked list from the disk, only all the data pages in the primary partition are replaced by new data pages, the data in the higher-level partition is not affected, the condition of buffer pool pollution is avoided, and the using effect of the LRU linked list is optimized.
In some embodiments, in the management device for caching data provided by the invention,
all the partitions in the LRU linked list are sequentially arranged from the tail part of the linked list to the head part of the linked list according to the sequence of the levels from low to high; wherein, for two adjacent partitions, the head of the lower-level partition is connected with the tail of the higher-level partition.
In some embodiments, in the management apparatus for caching data provided by the present invention, in the LRU linked list, the partition occupancy of the level from low to high is increased based on a power law.
In the LRU linked list, high-level, medium-level and primary partitions can be set end to end, that is: the tail (tail) of the high-level partition is connected with the head (head) of the middle-level partition, and the tail (tail) of the middle-level partition is connected with the head (head) of the primary partition;
it can also be set that the ratio of each partition with the level from low to high is increased based on the power law, such as: the high-level, middle-level and primary partition ratio is 4:2:1, so that a small part of space in the whole LRU linked list is used for replacing new data, a large part of space is used for storing data with high access rate, the data with high access rate is eliminated later, and the using effect of the LRU linked list is optimized;
in some practical applications, the ratio of each partition in the LRU linked list can be increased or decreased to adapt to different environments.
In some embodiments, the management apparatus for caching data provided by the present invention further includes: a configuration module 404;
before the mobile module obtains the ranking frequency threshold corresponding to the level of the current partition from the parameter configuration library, the configuration module is configured to determine the ranking frequency threshold in the parameter configuration library according to the following method:
acquiring the access quantity information of the data in the LRU linked list in a preset time period based on a preset frequency;
calling a threshold analysis model, calculating a progressive frequency threshold corresponding to each partition based on the access amount information, and storing the progressive frequency threshold into the parameter configuration library; wherein the content of the first and second substances,
and the advance frequency threshold corresponding to the highest-level partition in the LRU linked list is an upper limit threshold calculated according to the access amount information, and the upper limit threshold is greater than the accessed frequency of all data in the LRU linked list in the preset time period.
The threshold analysis model can be trained in advance, and can be trained through the cache utilization rate or QPS value recorded in different stages by the system; calculating to obtain a grading frequency threshold value corresponding to each partition suitable for the current environment based on the access amount information through a threshold value analysis model; typically, the rank threshold derived by the threshold analysis model is adapted to the overall visited frequency distribution of the data in the corresponding partition, such as: if the average value of the accessed frequency of the data in a certain partition is a, the difference value of the advanced frequency b and the advanced frequency a corresponding to the partition is within a preset range;
because the data in the highest-level partition in the linked list cannot be subjected to ranking, for the highest-level partition, the ranking frequency threshold corresponding to the highest-level partition can be set as an upper limit threshold calculated according to the access amount information, and the upper limit threshold can be larger than the accessed frequency of all the data in the LRU linked list; when data in the highest level partition is accessed, the frequency with which the data is accessed will typically be less than the corresponding advance frequency, and it will be decided to move the data to the head of the highest level partition.
In some embodiments, when the frequency of access is greater than the advance frequency threshold, the moving module moves the target data to a partition one level higher than the current partition, including:
when the current partition where the target data is located is not the highest-level partition in the LRU linked list, acquiring the accessed frequency of each data in the higher-level partition;
determining a position to be inserted of the target data by adopting a preset insertion algorithm based on the accessed frequency of each data;
and moving the target data to a position to be inserted in the higher-level partition.
When the target data moves forward in a progressive mode, determining a position to be inserted of the target data based on the accessed frequency of each data in the partition to be progressive; such as: determining the position to be inserted according to the distribution condition of the access frequency of each data in the partition to be graded and the access frequency of the target data; or determining a frequency interval according to the maximum value and the minimum value of the accessed frequency of the data in the partition to be graded, determining the position of the accessed frequency of the target data in the frequency interval, and further determining the position to be inserted of the target data; when the target data moves forward to a higher-level partition, the data originally positioned at the tail part in the higher-level partition is moved forward to the head part of the partition where the target data is originally positioned;
the insertion algorithm can be flexibly adjusted according to different environments.
In some embodiments, when the current partition in which the target data is located is the highest-level partition in the LRU linked list and it is determined that the target data is accessed more frequently than the ranking frequency threshold,
the configuration module is further configured to calculate the ranking frequency threshold corresponding to each partition again according to the method provided by the present invention to update the parameter configuration library, and the moving module is further configured to move the target data to the head of the current partition.
When the visited frequency of the data in the highest-level partition is greater than the corresponding ranking frequency threshold (i.e. the upper threshold), the obtained ranking frequency threshold is not suitable for the current situation, so that the ranking frequency threshold corresponding to each partition more suitable for the current situation can be immediately calculated again to update the parameter configuration library, and the target data is moved to the head of the current partition; by the mode, the cache data management method is more reasonable and has stronger adaptability.
Fig. 5 shows an exemplary system architecture 500 to which the management method of cache data or the management apparatus of cache data of the embodiments of the present invention may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 serves to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The terminal devices 501, 502, 503 may have various applications required for reading data installed thereon, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like.
The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server providing support for data read by the user using the terminal device 501, 502, 503. The background management server can analyze and process the received access data request and feed back the processing result to the terminal equipment.
It should be noted that the management method for cache data provided in the embodiment of the present invention is generally executed by the server 505, and accordingly, the management apparatus for cache data is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, a block diagram of a computer system 600 suitable for use with a terminal device implementing an embodiment of the invention is shown. The terminal device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the system of the present invention when executed by the central processing unit (CP U) 601.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a receiving module, a reading module, a moving module, and a configuration module. Wherein the names of the modules do not in some cases constitute a limitation of the module itself.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: step S101, receiving an access request for target data; step S102, when the target data exists in an LRU linked list of a buffer pool, determining the level of the current partition of the target data in the LRU linked list, and reading the accessed frequency of the target data; wherein the LRU linked list comprises at least three partitions of different levels; step S103, acquiring a progressive frequency threshold corresponding to the level of the current partition from a parameter configuration library to judge whether the accessed frequency is greater than the progressive frequency threshold; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition.
According to the technical scheme of the embodiment of the invention, the method for dividing the LRU linked list into different levels and setting the forward or progressive mode when the data is accessed is adopted, so that the condition that the use effect of the LRU linked list is poor due to pre-reading failure or buffer pool pollution in the prior art is avoided, the data with the real high hit rate can exist at the head of the linked list as much as possible and is not easy to eliminate, and only a part of space in the linked list can be used for loading new data, so that the use effect of the LRU linked list is optimized, and the reading efficiency of the cache data is optimized integrally.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (11)

1. A method for managing cache data, comprising:
receiving an access request for target data;
when the target data exist in an LRU linked list of a buffer pool, determining the level of a current partition of the target data in the LRU linked list, and reading the accessed frequency of the target data; wherein the LRU linked list comprises at least three partitions of different levels;
acquiring a ranking frequency threshold corresponding to the level of the current partition from a parameter configuration library to judge whether the accessed frequency is greater than the ranking frequency threshold; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition.
2. The method of claim 1, after receiving the request for access to the target data, further comprising:
when the target data does not exist in the LRU linked list of the buffer pool, reading the target data from the disk so as to place the target data into the head of the lowest-level partition in the LRU linked list of the buffer pool.
3. The method of claim 1, further comprising:
all the partitions in the LRU linked list are sequentially arranged from the tail part of the linked list to the head part of the linked list according to the sequence of the levels from low to high; wherein, for two adjacent partitions, the head of the lower-level partition is connected with the tail of the higher-level partition.
4. The method of claim 3, further comprising:
in the LRU linked list, the occupation ratio of each partition with the level from low to high is increased based on a power law.
5. The method of claim 1, further comprising, before obtaining the ranking frequency threshold corresponding to the level of the current partition from a parameter configuration repository, determining the ranking frequency threshold in the parameter configuration repository according to the following method:
acquiring the access quantity information of the data in the LRU linked list in a preset time period based on a preset frequency;
calling a threshold analysis model, calculating a progressive frequency threshold corresponding to each partition based on the access amount information, and storing the progressive frequency threshold into the parameter configuration library; wherein the content of the first and second substances,
and the advance frequency threshold corresponding to the highest-level partition in the LRU linked list is an upper limit threshold calculated according to the access amount information, and the upper limit threshold is greater than the accessed frequency of all data in the LRU linked list in the preset time period.
6. The method of claim 5, wherein moving the target data to a partition one level higher than the current partition when the frequency of access is greater than the advance frequency threshold comprises:
when the current partition where the target data is located is not the highest-level partition in the LRU linked list, acquiring the accessed frequency of each data in the higher-level partition;
determining a position to be inserted of the target data by adopting a preset insertion algorithm based on the accessed frequency of each data;
and moving the target data to a position to be inserted in the higher-level partition.
7. The method of claim 6, further comprising:
when the current partition in which the target data is located is the highest-level partition in the LRU linked list and it is determined that the frequency of accessing the target data is greater than the ranking frequency threshold, the ranking frequency threshold corresponding to each partition is calculated again according to the method of claim 5 to update the parameter configuration library, and the target data is moved to the head of the current partition.
8. A management apparatus for caching data, comprising:
the receiving module is used for receiving an access request for target data;
the reading module is used for determining the level of a current partition of the target data in an LRU linked list of a buffer pool when the target data exists in the LRU linked list, and reading the accessed frequency of the target data; wherein the LRU linked list comprises at least three partitions of different levels;
the mobile module is used for acquiring a ranking frequency threshold value corresponding to the level of the current partition from a parameter configuration library so as to judge whether the accessed frequency is greater than the ranking frequency threshold value; if yes, moving the target data to a partition with a level higher than that of the current partition; if not, moving the target data to the head of the current partition.
9. The apparatus of claim 8, wherein after the receiving module receives the request for access to the target data, the reading module is further configured to: when the target data does not exist in the LRU linked list of the buffer pool, reading the target data from a disk; and the number of the first and second groups,
the mobile module is further configured to: and putting the target data into the head of the lowest-level partition in the LRU linked list of the buffer pool.
10. A managed electronic device that caches data, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
CN202110371767.5A 2021-04-07 2021-04-07 Management method and device for cache data Pending CN112948444A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110371767.5A CN112948444A (en) 2021-04-07 2021-04-07 Management method and device for cache data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110371767.5A CN112948444A (en) 2021-04-07 2021-04-07 Management method and device for cache data

Publications (1)

Publication Number Publication Date
CN112948444A true CN112948444A (en) 2021-06-11

Family

ID=76232355

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110371767.5A Pending CN112948444A (en) 2021-04-07 2021-04-07 Management method and device for cache data

Country Status (1)

Country Link
CN (1) CN112948444A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115277243A (en) * 2022-08-04 2022-11-01 陈锦汇 Network communication scheme based on LRU cache
CN116560585A (en) * 2023-07-05 2023-08-08 支付宝(杭州)信息技术有限公司 Data hierarchical storage method and system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108334460A (en) * 2017-05-25 2018-07-27 中兴通讯股份有限公司 data cache method and device
CN109359095A (en) * 2018-09-11 2019-02-19 东华大学 A kind of DLK method that big data is quickly read
CN111159066A (en) * 2020-01-07 2020-05-15 杭州电子科技大学 Dynamically-adjusted cache data management and elimination method
CN111309650A (en) * 2020-02-11 2020-06-19 广州市百果园信息技术有限公司 Cache control method, device, storage medium and equipment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108334460A (en) * 2017-05-25 2018-07-27 中兴通讯股份有限公司 data cache method and device
CN109359095A (en) * 2018-09-11 2019-02-19 东华大学 A kind of DLK method that big data is quickly read
CN111159066A (en) * 2020-01-07 2020-05-15 杭州电子科技大学 Dynamically-adjusted cache data management and elimination method
CN111309650A (en) * 2020-02-11 2020-06-19 广州市百果园信息技术有限公司 Cache control method, device, storage medium and equipment

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115277243A (en) * 2022-08-04 2022-11-01 陈锦汇 Network communication scheme based on LRU cache
CN115277243B (en) * 2022-08-04 2024-03-08 陈锦汇 Network communication method based on LRU cache
CN116560585A (en) * 2023-07-05 2023-08-08 支付宝(杭州)信息技术有限公司 Data hierarchical storage method and system
CN116560585B (en) * 2023-07-05 2024-04-09 支付宝(杭州)信息技术有限公司 Data hierarchical storage method and system

Similar Documents

Publication Publication Date Title
WO2020119485A1 (en) Page display method and device, apparatus, and storage medium
US11017152B2 (en) Optimizing loading of web page based on aggregated user preferences for web page elements of web page
CN107656768B (en) Method and system for controlling page jump
CN110933140B (en) CDN storage allocation method, system and electronic equipment
CN107729570B (en) Data migration method and device for server
CN112948444A (en) Management method and device for cache data
CN108011949B (en) Method and apparatus for acquiring data
CN113391765A (en) Data storage method, device, equipment and medium based on distributed storage system
CN109697019B (en) Data writing method and system based on FAT file system
CN112068765A (en) Method, apparatus and computer program product for managing a storage system
CN110427394B (en) Data operation method and device
CN107480269B (en) Object display method and system, medium and computing equipment
CN112395437A (en) 3D model loading method and device, electronic equipment and storage medium
CN111859225B (en) Program file access method, apparatus, computing device and medium
CN112711572B (en) Online capacity expansion method and device suitable for database and table division
CN110888583B (en) Page display method, system and device and electronic equipment
CN109213815B (en) Method, device, server terminal and readable medium for controlling execution times
CN112100553A (en) Webpage configuration method and device, electronic equipment and storage medium
CN113760972B (en) Data searching method and device
CN113760982B (en) Data processing method and device
CN112422342B (en) Method and device for acquiring service data
CN113688160A (en) Data processing method, processing device, electronic device and storage medium
CN113779450A (en) Page access method and page access device
CN117156147A (en) Video transcoding method, device, equipment and storage medium
CN113449215A (en) Method and device for loading picture

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20220318

Address after: 100176 Room 201, 2 / F, block C, 18 Kechuang 11th Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing

Applicant after: BEIJING JINGDONG CENTURY TRADING Co.,Ltd.

Address before: 100176 room 703, 7th floor, building 1, yard 18, Kechuang 11th Street, Beijing Economic and Technological Development Zone, Beijing

Applicant before: Beijing Dianzhi Technology Co.,Ltd.