CN117235088B - Cache updating method, device, equipment, medium and platform of storage system - Google Patents
Cache updating method, device, equipment, medium and platform of storage system Download PDFInfo
- Publication number
- CN117235088B CN117235088B CN202311483095.2A CN202311483095A CN117235088B CN 117235088 B CN117235088 B CN 117235088B CN 202311483095 A CN202311483095 A CN 202311483095A CN 117235088 B CN117235088 B CN 117235088B
- Authority
- CN
- China
- Prior art keywords
- node
- cache
- nodes
- target
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 235000019633 pungent taste Nutrition 0.000 claims abstract description 26
- 238000012549 training Methods 0.000 claims description 79
- 238000004590 computer program Methods 0.000 claims description 22
- 230000008030 elimination Effects 0.000 claims description 14
- 238000003379 elimination reaction Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 14
- 238000012360 testing method Methods 0.000 claims description 14
- 238000010276 construction Methods 0.000 abstract description 4
- 238000012217 deletion Methods 0.000 abstract description 2
- 230000037430 deletion Effects 0.000 abstract description 2
- 230000006399 behavior Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 4
- PCTMTFRHKVHKIS-BMFZQQSSSA-N (1s,3r,4e,6e,8e,10e,12e,14e,16e,18s,19r,20r,21s,25r,27r,30r,31r,33s,35r,37s,38r)-3-[(2r,3s,4s,5s,6r)-4-amino-3,5-dihydroxy-6-methyloxan-2-yl]oxy-19,25,27,30,31,33,35,37-octahydroxy-18,20,21-trimethyl-23-oxo-22,39-dioxabicyclo[33.3.1]nonatriaconta-4,6,8,10 Chemical compound C1C=C2C[C@@H](OS(O)(=O)=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H]([C@H](C)CCCC(C)C)[C@@]1(C)CC2.O[C@H]1[C@@H](N)[C@H](O)[C@@H](C)O[C@H]1O[C@H]1/C=C/C=C/C=C/C=C/C=C/C=C/C=C/[C@H](C)[C@@H](O)[C@@H](C)[C@H](C)OC(=O)C[C@H](O)C[C@H](O)CC[C@@H](O)[C@H](O)C[C@H](O)C[C@](O)(C[C@H](O)[C@H]2C(O)=O)O[C@H]2C1 PCTMTFRHKVHKIS-BMFZQQSSSA-N 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000001052 transient effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Landscapes
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a cache updating method, a device, equipment, a medium and a platform of a storage system in the technical field of computers. Under the condition of cache miss, N samples are obtained through construction of M recently accessed nodes and each node in the cache, so that each sample covers recently-occurring actual access behavior information; predicting hit probability of each target node in each sample based on real-time access hotness of M nodes, selecting eliminated nodes for deletion according to the hit probability, and adding the nodes read from the disk at the current moment into a cache, thereby finishing cache updating of the storage system. According to the scheme, the current actual access behavior and the actual access heat are considered to predict the hit probability of each node in the cache, so that the cache update of the storage system is combined with the actual access behavior of the service, and the prediction accuracy and the cache hit rate can be improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, a medium, and a platform for updating a cache of a storage system.
Background
At present, cache updating of a storage system is performed by adopting a heuristic fixed rule mode, and although the updating principle is better understood and simple to implement, the method only performs data elimination from the use time of data in the cache, cannot mine the internal rule of the service system for accessing the storage system, and has limited cache hit rate.
Therefore, how to mine the intrinsic law of the service system accessing the storage system to improve the cache hit rate is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the present invention is directed to a method, apparatus, device, medium and platform for updating a cache of a storage system, so as to mine the internal rule of a service system accessing the storage system to increase the cache hit rate. The specific scheme is as follows:
in a first aspect, the present invention provides a method for updating a cache of a storage system, where the storage system includes: cache and disk, the method comprising:
acquiring M accessed nodes before the current moment at the current moment of the cache miss;
taking each node in the cache as a target node respectively, and combining the M nodes with each target node to obtain N samples;
Predicting hit probability of each target node in each sample according to the real-time access hotness of the M nodes, and selecting eliminated nodes from all target nodes according to the hit probability;
and deleting the eliminated node from the cache, and adding the node read from the disk at the current moment into the cache.
Optionally, the acquiring the M nodes accessed before the current time includes:
querying key value pairs accessed in a last period of time;
arranging each key value pair into a key value pair sequence according to the sequence of access time;
querying and recording nodes comprising elements in the key-value pair sequence in the cache; the same node which is repeatedly searched is recorded once;
and taking each recorded node as the M nodes.
Optionally, the querying the key-value pair accessed in the last period of time includes:
and inquiring the key value pair accessed from the previous cache miss time to the current time in the log record module.
Optionally, the combining the M nodes with each target node to obtain N samples includes:
arranging the M nodes into a node sequence according to the sequence of the access time;
And arranging each target node at the tail part of the node sequence respectively to obtain the N samples.
Optionally, the access frequency of the M nodes in the last period of time is determined as the real-time access hotness of the M nodes.
Optionally, the determining the access frequency of the M nodes in the last period of time as the real-time access hotness of the M nodes includes:
and determining the access frequency of the M nodes from the previous cache miss time to the current time as the real-time access heat of the M nodes.
Optionally, predicting the hit probability of each target node in each sample according to the real-time access hotness of the M nodes includes:
inputting each sample into a preset prediction model respectively, so that the prediction model predicts a probability value for each sample;
the probability value obtained by the prediction model for each sample is used as the hit probability of each target node in each sample.
Optionally, the training process of the prediction model includes:
the key value pairs accessed in a preset time period are arranged into a target sequence according to the sequence of the access time;
determining hit nodes in the cache according to the target sequence;
Intercepting a node group in the hit node;
combining the node group with each target node respectively to obtain N training data;
determining a probability label corresponding to each training data, and adding a corresponding probability label for each training data;
and training by using N training data with probability labels to obtain the prediction model.
Optionally, the determining a hit node in the cache according to the target sequence includes:
querying and recording nodes comprising elements in the target sequence in the cache; wherein the same node that is repeatedly searched is recorded once.
Optionally, the determining the probability label corresponding to each training data includes:
after the node group is accessed, the probability that the target node in each training data is accessed is the probability label of the corresponding training data.
Optionally, the combining the node group with each target node respectively to obtain N training data includes:
and arranging each target node at the tail of the node group respectively to obtain the N training data.
Optionally, before the key value pairs accessed in the preset time period are arranged into the target sequence according to the sequence of the access time, the method further includes:
And inquiring the key value pairs accessed in the preset time period in a log record module.
Optionally, the method further comprises:
obtaining an increment log in a log recording module;
judging whether the data volume of the increment log exceeds a preset first threshold value or not;
and if the data quantity exceeds the preset first threshold value, retraining the prediction model, and replacing the prediction model by the model obtained by retraining.
Optionally, the method further comprises:
counting the hit rate of the cache;
judging whether the hit rate is lower than a preset second threshold value or not;
and if the hit rate is lower than the preset second threshold value, retraining the prediction model, and replacing the prediction model by the model obtained by retraining.
Optionally, the method further comprises:
testing the accuracy of the prediction model by using a test data set;
judging whether the accuracy is lower than a preset third threshold value or not;
and if the accuracy rate is lower than the preset third threshold value, retraining the prediction model, and replacing the prediction model by the model obtained by retraining.
Optionally, selecting the obsolete node from all the target nodes according to the hit probability includes:
And selecting the node with the smallest hit probability from all the target nodes as the elimination node.
Optionally, selecting the obsolete node from all the target nodes according to the hit probability includes:
selecting a node with the smallest hit probability from all target nodes as a first node to be selected;
selecting a second node to be selected from the cache according to a plurality of preset rules; the plurality of preset rules include: the least recently unused rule, the first in first out rule, and the least used rule;
and taking the repeated nodes in the first node to be selected and the second node to be selected as the elimination nodes.
Optionally, predicting the hit probability of each target node in each sample according to the real-time access hotness of the M nodes includes:
and calculating the probability that each target node is accessed at the future moment after the M nodes are accessed according to the real-time access hotness of the M nodes, and taking the probability as the hit probability of each target node in each sample.
Optionally, the cached data structure is: each target node is arranged in the cache according to the size of the node address; or each target node is arranged in the cache according to the real-time access heat; or each target node is arranged in the cache according to the time sequence of adding the target node into the cache; each target node comprises: a plurality of key-value pairs; each key-value pair is keyed by a logical block address and is valued by a physical block address.
In a second aspect, the present invention provides a cache updating apparatus of a storage system, where the storage system includes: cache and disk, the apparatus comprising:
the acquisition module is used for acquiring M accessed nodes before the current moment at the current moment of the cache miss;
the combination module is used for taking each node in the cache as a target node respectively, and combining the M nodes with each target node to obtain N samples;
the prediction module is used for predicting the hit probability of each target node in each sample according to the real-time access heat of the M nodes and selecting the obsolete node from all the target nodes according to the hit probability;
and the updating module is used for deleting the eliminated node from the cache and adding the node read from the disk at the current moment into the cache.
Optionally, the acquiring module is specifically configured to:
querying key value pairs accessed in a last period of time;
arranging each key value pair into a key value pair sequence according to the sequence of access time;
querying and recording nodes comprising elements in the key-value pair sequence in the cache; the same node which is repeatedly searched is recorded once;
And taking each recorded node as the M nodes.
Optionally, the acquiring module is specifically configured to:
and inquiring the key value pair accessed from the previous cache miss time to the current time in the log record module.
Optionally, the combination module is specifically configured to:
arranging the M nodes into a node sequence according to the sequence of the access time;
and arranging each target node at the tail part of the node sequence respectively to obtain the N samples.
Optionally, the access frequency of the M nodes in the last period of time is determined as the real-time access hotness of the M nodes.
Optionally, the determining the access frequency of the M nodes in the last period of time as the real-time access hotness of the M nodes includes:
and determining the access frequency of the M nodes from the previous cache miss time to the current time as the real-time access heat of the M nodes.
Optionally, the prediction module is specifically configured to:
inputting each sample into a preset prediction model respectively, so that the prediction model predicts a probability value for each sample;
the probability value obtained by the prediction model for each sample is used as the hit probability of each target node in each sample.
Optionally, a training module for training the predictive model is also included; the training module is specifically used for:
the key value pairs accessed in a preset time period are arranged into a target sequence according to the sequence of the access time;
determining hit nodes in the cache according to the target sequence;
intercepting a node group in the hit node;
combining the node group with each target node respectively to obtain N training data;
determining a probability label corresponding to each training data, and adding a corresponding probability label for each training data;
and training by using N training data with probability labels to obtain the prediction model.
Optionally, the training module is specifically configured to:
querying and recording nodes comprising elements in the target sequence in the cache; wherein the same node that is repeatedly searched is recorded once.
Optionally, the training module is specifically configured to:
after the node group is accessed, the probability that the target node in each training data is accessed is the probability label of the corresponding training data.
Optionally, the training module is specifically configured to:
and arranging each target node at the tail of the node group respectively to obtain the N training data.
Optionally, the training module is specifically configured to:
and inquiring the key value pairs accessed in the preset time period in a log record module.
Optionally, the method further comprises:
the module updating module is used for acquiring the increment log in the log recording module; judging whether the data volume of the increment log exceeds a preset first threshold value or not; and if the data quantity exceeds the preset first threshold value, retraining the prediction model, and replacing the prediction model by the model obtained by retraining.
Optionally, the module updating module is further configured to:
counting the hit rate of the cache;
judging whether the hit rate is lower than a preset second threshold value or not;
and if the hit rate is lower than the preset second threshold value, retraining the prediction model, and replacing the prediction model by the model obtained by retraining.
Optionally, the module updating module is further configured to:
testing the accuracy of the prediction model by using a test data set;
judging whether the accuracy is lower than a preset third threshold value or not;
and if the accuracy rate is lower than the preset third threshold value, retraining the prediction model, and replacing the prediction model by the model obtained by retraining.
Optionally, the prediction module is specifically configured to:
and selecting the node with the smallest hit probability from all the target nodes as the elimination node.
Optionally, the prediction module is specifically configured to:
selecting a node with the smallest hit probability from all target nodes as a first node to be selected;
selecting a second node to be selected from the cache according to a plurality of preset rules; the plurality of preset rules include: the least recently unused rule, the first in first out rule, and the least used rule;
and taking the repeated nodes in the first node to be selected and the second node to be selected as the elimination nodes.
Optionally, the prediction module is specifically configured to:
and calculating the probability that each target node is accessed at the future moment after the M nodes are accessed according to the real-time access hotness of the M nodes, and taking the probability as the hit probability of each target node in each sample.
Optionally, the cached data structure is: each target node is arranged in the cache according to the size of the node address; or each target node is arranged in the cache according to the real-time access heat; or each target node is arranged in the cache according to the time sequence of adding the target node into the cache; each target node comprises: a plurality of key-value pairs; each key-value pair is keyed by a logical block address and is valued by a physical block address.
In a third aspect, the present invention provides an electronic device, comprising:
a memory for storing a computer program;
and a processor for executing the computer program to implement the cache updating method of the storage system disclosed above.
In a fourth aspect, the present invention provides a readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the cache update method of the storage system disclosed above.
In a fifth aspect, the present invention provides a storage platform comprising: a client and a storage system; the storage system comprises a cache and a disk; the client is used for sending an access request to the storage system; the storage system is configured to respond to the access request and perform the method of any one of the preceding claims.
As can be seen from the above scheme, the present invention provides a method for updating a cache of a storage system, including: at the current moment of cache miss, M accessed nodes before the current moment are acquired; taking each node in the cache as a target node respectively, and combining the M nodes with each target node to obtain N samples; predicting hit probability of each target node in each sample according to the real-time access hotness of the M nodes, and selecting eliminated nodes from all target nodes according to the hit probability; and deleting the eliminated node from the cache, and adding the node read from the disk at the current moment into the cache.
The beneficial effects of the invention are as follows: under the condition of cache miss, N samples are obtained through construction of M recently accessed nodes and each node in a cache, so that each sample covers recently occurring actual access behavior information; predicting hit probability of each target node in each sample based on real-time access hotness of M nodes, selecting eliminated nodes for deletion according to the hit probability, and adding the nodes read from the disk at the current moment into a cache, thereby finishing cache updating of the storage system. According to the scheme, the hit probability of each node in the cache is predicted by considering the current actual access behavior and the actual access heat, so that the prediction accuracy can be improved, the nodes in the cache are updated, the cache update of the storage system is combined with the actual access behavior of the service, the internal rule of the service system for accessing the storage system is mined, and the cache hit rate can be improved.
Correspondingly, the cache updating device, the cache updating equipment, the readable storage medium and the cache updating platform of the storage system have the technical effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for updating a cache of a storage system according to the present invention;
FIG. 2 is a schematic diagram illustrating an arrangement of nodes in a cache according to the present disclosure;
FIG. 3 is a schematic diagram illustrating a cache update process of a storage system according to the present disclosure;
FIG. 4 is a schematic diagram of a cache update apparatus of a storage system according to the present disclosure;
FIG. 5 is a schematic diagram of an electronic device according to the present disclosure;
FIG. 6 is a diagram of a server according to the present invention;
FIG. 7 is a diagram of a terminal structure according to the present invention;
fig. 8 is a schematic diagram of a storage platform according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
At present, cache updating of a storage system is performed by adopting a heuristic fixed rule mode, and although the updating principle is better understood and simple to implement, the method only performs data elimination from the use time of data in the cache, cannot mine the internal rule of the service system for accessing the storage system, and has limited cache hit rate. Therefore, the invention provides a cache updating scheme of the storage system, which can mine the internal rule of the service system for accessing the storage system to improve the cache hit rate.
Referring to fig. 1, an embodiment of the present invention discloses a method for updating a cache of a storage system, where the storage system includes: cache and disk, the method comprising:
s101, at the current time of the cache miss, M accessed nodes before the current time are acquired.
The present embodiment applies to a storage system including a disk and a cache, which may be provided by any type of memory. Wherein, the cache hit refers to: information (such as metadata information) corresponding to the access request is searched in the cache according to the access request sent by the client; cache misses refer to: information (such as metadata information) corresponding to the access request cannot be found in the cache according to the access request sent by the client. It can be seen that metadata information can be stored in the cache.
Wherein the storage system may be DAS (Decimal Adjust for Subtraction)) a direct connection storage system, NAS (Network Attached Storage) network file system, or SAN (Storage Area Network) storage network. In the SAN storage network, the difference of various storage media at the bottom layer can be shielded through intermediate management, and a unified storage view is provided for users, so that the advanced characteristics of dynamic capacity expansion, automatic backup, repeated deleting compression and the like which are not perceived by the users are realized. The basis for achieving this is the separation of the logical space seen by the user from the actual physical space, and SAN systems achieve this goal by building and managing a mapping of logical space to physical storage space. In SAN systems, this function is typically implemented with metadata management. In metadata management, in order to accelerate metadata query, cache of metadata can be constructed by reducing user query times, partial high-frequency accessed metadata is loaded into a memory, the stored query of the metadata is realized, the query efficiency of the metadata is improved, the disk reading times are reduced, and the IOPS and response efficiency of the system are improved.
The metadata is simply understood as the mapping of LBAs (Logical Block Address, logical block addresses) to PBAs (Physical Block Address, physical block addresses), which are typically expressed in terms of kv pairs (key-value pairs), i.e. key-value pairs (LBAs, PBAs). There is of course also a kv pair of the PBA to LBA mapping in reverse. Metadata caching is to cache a large number of kv pairs in memory. Meanwhile, in order to efficiently inquire the KV pairs, the KV pairs can be organized and stored in a cache in a B+ tree mode. Each node of the B+ tree stores a plurality of kv pairs, and the node searches downwards from the root node according to the key value in sequence until the node reaches the corresponding leaf node.
The method is limited by the size limitation of the memory, the kv pairs in the cache are only partial kv pairs of the whole system, and the cache data cannot be unchanged all the time, and the cache data needs to be updated and adjusted in real time according to the change of the access space of the user so as to improve the hit rate of the cache.
All the disks in the SAN storage network may be solid state disks.
In this embodiment, the node may specifically be: b+ tree nodes, each node is equal in size and comprises: a plurality of key-value pairs; each key-value pair is keyed by a logical block address (key) and is valued by a physical block address (value). Accordingly, the cache includes N target nodes, each of which includes: a plurality of key-value pairs; each key-value pair takes a logical block address as a key and takes a physical block address as a value; each target node is of equal size, so that each target node includes an equal number of key-value pairs. The cached data structure is thus: each target node is arranged in the cache according to the size of the node address; or each target node is arranged in the cache according to the real-time access heat; or each target node is arranged in the cache according to the time sequence of adding the target node into the cache; each target node comprises: a plurality of key-value pairs; each key-value pair is keyed by a logical block address and is valued by a physical block address. Any node in this embodiment is: a data block comprising a plurality of key-value pairs.
Of course, each target node in the cache may also be arranged according to the last time the node was used. Referring to fig. 2, a queue or linked list is maintained, in which the last time of use of each node is recorded, t0 is the current time, and t6 is a time farthest from the current time. When the cache is full of data, nodes that have not been accessed the longest should be eliminated. The node newly inserted into the cache is placed at the head of the queue, and the node hit in the cache is also moved to the head of the queue, so that the node at the tail of the queue is naturally the longest unused node. When the queue is full, nodes at the tail of the queue are eliminated.
In one embodiment, obtaining M nodes accessed prior to a current time includes: querying key value pairs accessed in a last period of time; arranging each key value pair into a key value pair sequence according to the sequence of access time; querying and recording nodes comprising elements in the key-value pair sequence in a cache; the same node which is repeatedly searched is recorded once; each node recorded is taken as M nodes. Because a node includes a plurality of key value pairs, the same node may be repeatedly accessed to check the key value pairs in the node within a period of time, and the embodiment makes the repeatedly checked same node record once.
In one embodiment, querying key-value pairs accessed during a recent period of time includes: and inquiring the key value pair accessed from the previous cache miss time to the current time in the log record module. It can be seen that the last period of time may be the time between the last cache miss time to the current time. Of course, the last period of time may be any time in history to the current time, so the M nodes accessed before the current time include: cache hit nodes and cache miss nodes.
S102, respectively taking each node in the cache as a target node, and combining M nodes with each target node to obtain N samples.
In this embodiment, N samples are obtained through the M nodes that are accessed recently and each node in the cache, so that each sample covers the actual access behavior information that occurs recently. In one embodiment, combining M nodes with each target node results in N samples, including: arranging M nodes into a node sequence according to the sequence of the access time; and arranging each target node at the tail part of the node sequence respectively to obtain N samples. Wherein M, N is natural number, M is less than or equal to N.
In one example, if the node sequence obtained by arranging the M nodes is ABCD and all the target nodes in the cache are abcde, the N samples obtained by combining are respectively: ABCDa, ABCDb, ABCDc, ABCDd, ABCDe. Of course, in each sample, the target node may be located at any position in the node sequence, such as: at the beginning of the sequence of nodes.
S103, predicting hit probability of each target node in each sample according to the real-time access hotness of the M nodes, and selecting eliminated nodes from all the target nodes according to the hit probability.
In one embodiment, the access frequency of the M nodes during the last period of time is determined as the real-time access hotness of the M nodes. The method for determining the access frequency of the M nodes in the latest time period as the real-time access heat of the M nodes comprises the following steps: and determining the access frequency of the M nodes from the previous cache miss time to the current time as the real-time access heat of the M nodes.
In this embodiment, predicting the hit probability of each target node in each sample according to the real-time access hotness of M nodes includes: according to the real-time access heat of the M nodes, the probability that each target node is accessed at the future moment after the M nodes are accessed is calculated as the hit probability of each target node in each sample. For example: the hit probability of the target node a in the sample ABCDa is: the probability that target node a is accessed at a future time after ABCD is accessed; the hit probability of the target node b in the sample ABCDb is: the probability that target node b will be visited at a future time after ABCD is visited, and so on.
S104, deleting the eliminated node from the cache, and adding the node read from the disk at the current moment into the cache.
In one embodiment, selecting the obsolete node among all target nodes based on the hit probability comprises: and selecting the node with the smallest hit probability from all the target nodes as the obsolete node.
According to the embodiment, the cache updating of the storage system is performed by combining the internal rule of the service system access storage system, the future access behavior can be accurately predicted according to the current access behavior, and meanwhile, node elimination can be performed based on the service time of the nodes in the cache, so that node elimination is performed by comprehensively considering more comprehensive information. In one embodiment, selecting the obsolete node among all target nodes based on the hit probability comprises: selecting a node with the smallest hit probability from all target nodes as a first node to be selected; selecting a second node to be selected in the cache according to a plurality of preset rules; the plurality of preset rules include: the least recently unused rule, the first in first out rule, and the least used rule; and taking the repeated nodes in the first to-be-selected node and the second to-be-selected node as the elimination nodes.
According to the embodiment, the obsolete node is selected to delete according to the hit probability of each target node in each sample, and the node read from the disk at the current moment is added into the cache, so that the cache update of the storage system is completed. According to the scheme, the hit probability of each node in the cache is predicted by considering the current actual access behavior and the actual access heat, so that the prediction accuracy can be improved, the nodes in the cache are updated, the cache update of the storage system is combined with the actual access behavior of the service, the internal rule of the service system for accessing the storage system is mined, and the cache hit rate can be improved.
In one embodiment, predicting the hit probability of each target node in each sample based on the real-time access hotness of the M nodes includes: inputting each sample into a preset prediction model respectively, so that the prediction model predicts a probability value for each sample; the probability value obtained by the prediction model for each sample is taken as the hit probability of each target node in each sample. The prediction model may be an XGBoost or the like model. The probability value predicted by the prediction model for each sample is: the prediction model calculates the probability that each target node is accessed at a future time after the M nodes are accessed based on the real-time access hotness of the M nodes.
In one embodiment, the training process of the predictive model includes: the key value pairs accessed in a preset time period are arranged into a target sequence according to the sequence of the access time; determining hit nodes in the cache according to the target sequence; intercepting a node group in the hit node; combining the node group with each target node respectively to obtain N training data; determining a probability label corresponding to each training data, and adding a corresponding probability label for each training data; and training by using N training data with probability labels to obtain a prediction model. There are a plurality of hit nodes.
It should be noted that the construction process of the training data is similar to that of the foregoing sample. In one embodiment, therefore, determining hit nodes in the cache based on the target sequence includes: querying and recording nodes comprising elements in the target sequence in a cache; wherein the same node that is repeatedly searched is recorded once. In one embodiment, determining the probability tag corresponding to each training data includes: after the node group is accessed, the probability of the target node in each training data being accessed (which is calculated based on the historical access record in the log data) is a probability tag for the corresponding training data. In one embodiment, combining the node group with each target node respectively to obtain N training data includes: and arranging each target node at the tail part of the node group respectively to obtain the N training data. In one embodiment, before arranging the key value pairs accessed in the preset time period into the target sequence according to the order of the access time, the method further comprises: and inquiring the key value pairs accessed in a preset time period in the log record module.
In order to ensure the accuracy of the prediction model, the embodiment can update the prediction model in time in a timed task mode or a conditional triggering mode. The prediction model is updated in three ways, and the three ways can be freely combined or selected.
Mode one: obtaining an increment log in a log recording module; judging whether the data volume of the increment log exceeds a preset first threshold value or not; if the data quantity exceeds the preset first threshold value, retraining the prediction model, and replacing the prediction model by the model obtained through retraining. The increment log includes the access behavior of the last-looking-up cache, that is, the last-accessed key value pair in the cache.
Mode two: counting hit rate of the cache; judging whether the hit rate is lower than a preset second threshold value; if the hit rate is lower than a preset second threshold value, retraining the prediction model, and replacing the prediction model with the model obtained by retraining.
Mode three: testing the accuracy of the prediction model by using the test data set; judging whether the accuracy is lower than a preset third threshold value or not; if the accuracy is lower than a preset third threshold, retraining the prediction model, and replacing the prediction model with the model obtained by retraining. Wherein each test data in the test data set is implemented according to a construction process of the training data.
In one example, XGBoost (eXtreme Gradient Boosting, limit gradient lifting) model is used to predict nodes that need to be eliminated. XGBoost is a predictive model for classification and regression in the machine learning field, enabling GBDT (Gradient Boosting Decision Tree, gradient-lifting decision tree) and many improvements in algorithms and engineering. The biggest difference between XGBoost and GBDT is: XGBoost obtains the weight of leaf nodes of a tree to be fitted in the next step (the structure of the tree needs to be determined first) by performing second-order Taylor expansion on an objective function, obtains the size of loss reduction of each splitting node according to a loss function, and selects proper attributes according to the splitting loss to split. The objective function is:
The method comprises the steps of carrying out a first treatment on the surface of the Wherein, the first item->Fitting data representing model, second termThen it is a regularization term to prevent overfitting the control model complexity.
And the strong prediction capability of XGBoost is utilized, and the historical access data of the storage system is used as model input data to train a prediction model, so that the prediction model predicts which node in the cache should be eliminated most according to the current access behavior, thereby realizing intelligent updating of the cache.
Referring to fig. 3, the present embodiment is implemented with several parts, such as an access log recording module, a cache data structure, a prediction model, and disk data. The access log recording module is responsible for recording access history data of the external service system for accessing the storage system. A cache is a data structure that can store metadata nodes of a current hotspot in the form of an array, a linked list, or the like. The prediction model comprises a training engine, an reasoning engine and an updating engine, when the cache needs to be updated, the reasoning engine predicts the optimal nodes needing to be eliminated according to the access context behaviors aiming at the cache recently. Referring to fig. 3, for a metadata request sent from a client of a service system, if corresponding data is found in a cache, a cache hit directly responds; if the cache is not hit, corresponding data is read from the disk data to respond, meanwhile, the prediction model predicts obsolete nodes based on the log data, the nodes needing obsolete are deleted from the cache, and the data read from the disk data are added into the cache to update the cache. Of course, the data currently accessed in the process (i.e., the data read from the disk data) is recorded into the log data.
And in the initial stage of the operation of the storage system, the access log recording module continuously records the access log of the service system, and when the access log records reach a certain number, the model training is triggered. The model training is to perform cleaning conversion and feature extraction on log data under the control of a training engine, construct input data suitable for training the XGBoost model, and then train to obtain a prediction model based on the obtained input data.
And then, when the metadata is queried for cache miss in the running process of the system, calling a prediction model when the cache is required to be updated, predicting nodes which are most likely to be eliminated based on the access behavior of the current user by the prediction model to eliminate, and inserting new nodes read from a disk due to the cache miss into the cache.
In the continuous operation process of the whole system, the access log recording module continuously records the access behaviors of users, detects the cache hit results, starts a model updating engine when the log newly increases data to reach a certain amount and the cache hit rate is reduced, and continuously trains the newly increased data in an incremental mode by utilizing the last training to obtain a new prediction model so as to ensure that the model adapts to the newly changed data. The model training and updating process in this embodiment follows the general model training and updating process.
The logging module records the access behavior of the service system in a time sequence form, and the unlabeled data cannot be directly used for training a model, so that feature selection and feature conversion are required. First, the key sequence (k 1, k2, … …, kt) of user access recorded in the logging module for a period of time is converted into a node sequence (P1, P2, … …, pt), wherein all hit nodes are included. Since each node comprises several kv pairs, different keys accessed by the user may belong to the same node, and thus are counted only once if they belong to the same node n. For the node sequences (P1, P2, … …, pt), based on the principle of sliding window, M adjacent nodes are selected from the node sequences each time as a sub-sequence (i.e. a node group) which records the access history behavior of the user during the last period of time.
Assuming a total of N nodes in the buffer, then for each sub-sequence, it is combined with each node in the buffer, resulting in N sample combinations. If there are Y subsequences, there may be Y N sample combinations. The labels for each sample combination are: the probability that a node in the cache in the sample combination is subsequently accessed (i.e., hit probability). In order to obtain the probability that a node in a cache in a sample combination is subsequently accessed, the frequency of the node being accessed can be counted, and the frequency is converted into the probability of the node being accessed. After the labeled sample combination is obtained, the labeled sample combination is divided according to the proportion of 70%, 20% and 10% to obtain three parts of a training data set, a verification data set and a test data set. The training data set is used for training the model, and the verification data set is used for carrying out parameter selection on the trained model to obtain the optimal training parameters. The test dataset is used to evaluate the accuracy of the model on unknown data.
As mentioned previously, XGBoost is selected for model training in this embodiment, and is suitable for both classification tasks and regression tasks. This embodiment is intended to predict which node is most likely to be eliminated from the dynamically changing cache using a predictive model, and this objective to be achieved can be translated into a regression problem, namely: predicting the reserved probability of each node, and selecting the node with the lowest reserved probability for elimination.
Model prediction process: when the cache is not hit and the cache needs to be updated at a certain moment, a prediction model is called to predict nodes needing to be eliminated. And sequentially combining the M currently accessed nodes with each node in the cache to form a prediction sample, inputting the prediction sample into an XGBoost prediction model, giving the retention probability of each node in the cache by the prediction model, and finally selecting smaller nodes for elimination.
Model updating: in order to prevent model drift caused by too long time, the accuracy of the prediction model used on the line and the hit rate of the cache are monitored in real time according to log data, and when the log incremental data is accumulated to a certain data volume, an incremental model updating engine triggers model updating and uses the incremental data to update the model.
According to the embodiment, the cache updating of the storage system is performed based on the machine learning model, so that the accuracy and the cache hit rate of the cache updating can be directly improved, the IOPS (Input Output Operations Per Second read-write times per second) of the storage system can be further improved, the response time of the system can be shortened, and the comprehensive performance of the system can be improved. Meanwhile, data features and prediction tasks are designed in a targeted manner, the data features are extracted based on a sliding window method, meanwhile, the extracted sample data are automatically calibrated, and manual intervention is not needed in the whole process.
The following describes a cache updating apparatus of a storage system according to an embodiment of the present invention, and the cache updating apparatus of a storage system described below may refer to other embodiments described herein.
Referring to fig. 4, an embodiment of the present invention discloses a cache updating apparatus of a storage system, where the storage system includes: cache and disk, the apparatus comprising:
an obtaining module 401, configured to obtain, at a current time of a cache miss, M nodes accessed before the current time;
a combining module 402, configured to take each node in the cache as a target node, and combine M nodes with each target node to obtain N samples;
The prediction module 403 is configured to predict a hit probability of each target node in each sample according to the real-time access hotness of the M nodes, and select a obsolete node from all the target nodes according to the hit probability;
and the updating module is used for deleting the eliminated node from the cache and adding the node read from the disk at the current moment into the cache.
In one embodiment, the obtaining module is specifically configured to:
querying key value pairs accessed in a last period of time;
arranging each key value pair into a key value pair sequence according to the sequence of access time;
querying and recording nodes comprising elements in the key-value pair sequence in a cache; the same node which is repeatedly searched is recorded once;
each node recorded is taken as M nodes.
In one embodiment, the obtaining module is specifically configured to:
and inquiring the key value pair accessed from the previous cache miss time to the current time in the log record module.
In one embodiment, the combination module is specifically configured to:
arranging M nodes into a node sequence according to the sequence of the access time;
and arranging each target node at the tail part of the node sequence respectively to obtain N samples.
In one embodiment, the access frequency of the M nodes during the last period of time is determined as the real-time access hotness of the M nodes.
In one embodiment, determining the access frequency of the M nodes in the last period of time as the real-time access hotness of the M nodes includes:
and determining the access frequency of the M nodes from the previous cache miss time to the current time as the real-time access heat of the M nodes.
In one embodiment, the prediction module is specifically configured to:
inputting each sample into a preset prediction model respectively, so that the prediction model predicts a probability value for each sample;
the probability value obtained by the prediction model for each sample is taken as the hit probability of each target node in each sample.
In one embodiment, the method further comprises a training module for training the predictive model; the training module is specifically used for:
the key value pairs accessed in a preset time period are arranged into a target sequence according to the sequence of the access time;
determining hit nodes in the cache according to the target sequence;
intercepting a node group in the hit node;
combining the node group with each target node respectively to obtain N training data;
determining a probability label corresponding to each training data, and adding a corresponding probability label for each training data;
and training by using N training data with probability labels to obtain a prediction model.
In one embodiment, the training module is specifically configured to:
querying and recording nodes comprising elements in the target sequence in a cache; wherein the same node that is repeatedly searched is recorded once.
In one embodiment, the training module is specifically configured to:
after the node group is accessed, the probability that the target node in each training data is accessed is the probability label of the corresponding training data.
In one embodiment, the training module is specifically configured to:
and arranging each target node at the tail part of the node group respectively to obtain the N training data.
In one embodiment, the training module is specifically configured to:
and inquiring the key value pairs accessed in a preset time period in the log record module.
In one embodiment, the method further comprises:
the module updating module is used for acquiring the increment log in the log recording module; judging whether the data volume of the increment log exceeds a preset first threshold value or not; if the data quantity exceeds the preset first threshold value, retraining the prediction model, and replacing the prediction model by the model obtained through retraining.
In one embodiment, the module update module is further to:
counting hit rate of the cache;
judging whether the hit rate is lower than a preset second threshold value;
If the hit rate is lower than a preset second threshold value, retraining the prediction model, and replacing the prediction model with the model obtained by retraining.
In one embodiment, the module update module is further to:
testing the accuracy of the prediction model by using the test data set;
judging whether the accuracy is lower than a preset third threshold value or not;
if the accuracy is lower than a preset third threshold, retraining the prediction model, and replacing the prediction model with the model obtained by retraining.
In one embodiment, the prediction module is specifically configured to:
and selecting the node with the smallest hit probability from all the target nodes as the obsolete node.
In one embodiment, the prediction module is specifically configured to:
selecting a node with the smallest hit probability from all target nodes as a first node to be selected;
selecting a second node to be selected in the cache according to a plurality of preset rules; the plurality of preset rules include: the least recently unused rule, the first in first out rule, and the least used rule;
and taking the repeated nodes in the first to-be-selected node and the second to-be-selected node as the elimination nodes.
In one embodiment, the prediction module is specifically configured to:
according to the real-time access heat of the M nodes, the probability that each target node is accessed at the future moment after the M nodes are accessed is calculated as the hit probability of each target node in each sample.
In one embodiment, the cached data structure is: each target node is arranged in the cache according to the size of the node address; or each target node is arranged in the cache according to the real-time access heat; or each target node is arranged in the cache according to the time sequence of adding the target node into the cache; each target node comprises: a plurality of key-value pairs; each key-value pair is keyed by a logical block address and is valued by a physical block address.
The more specific working process of each module and unit in this embodiment may refer to the corresponding content disclosed in the foregoing embodiment, and will not be described herein.
Therefore, the embodiment provides a cache updating device of a storage system, which can mine the internal rule of a service system accessing the storage system to improve the cache hit rate.
An electronic device provided in the embodiments of the present invention is described below, and an electronic device described below may refer to other embodiments described herein.
Referring to fig. 5, an embodiment of the present invention discloses an electronic device, including:
a memory 501 for storing a computer program;
a processor 502 for executing the computer program to implement the method disclosed in any of the embodiments above.
Further, the embodiment of the invention also provides electronic equipment. The electronic device may be a server as shown in fig. 6 or a terminal as shown in fig. 7. Fig. 6 and 7 are structural diagrams of electronic devices according to an exemplary embodiment, and the contents of the drawings should not be construed as any limitation on the scope of use of the present invention.
Fig. 6 is a schematic structural diagram of a server according to an embodiment of the present invention. The server specifically may include: at least one processor, at least one memory, a power supply, a communication interface, an input-output interface, and a communication bus. The memory is used for storing a computer program, and the computer program is loaded and executed by the processor to implement relevant steps in cache updating of the storage system disclosed in any of the foregoing embodiments.
In this embodiment, the power supply is configured to provide a working voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external equipment, and the communication protocol to be followed by the communication interface is any communication protocol applicable to the technical scheme of the invention, and the communication protocol is not particularly limited; the input/output interface is used for acquiring external input data or outputting data to the external, and the specific interface type can be selected according to the specific application requirement, and is not limited in detail herein.
In addition, the memory may be a read-only memory, a random access memory, a magnetic disk, an optical disk, or the like as a carrier for storing resources, where the resources stored include an operating system, a computer program, data, and the like, and the storage mode may be transient storage or permanent storage.
The operating system is used for managing and controlling each hardware device and computer program on the Server to realize the operation and processing of the processor on the data in the memory, and the operation and processing can be Windows Server, netware, unix, linux and the like. The computer program may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the cache update method of the storage system disclosed in any of the foregoing embodiments. The data may include data such as information on a developer of the application program in addition to data such as update information of the application program.
Fig. 7 is a schematic structural diagram of a terminal according to an embodiment of the present invention, where the terminal may specifically include, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like.
Generally, the terminal in this embodiment includes: a processor and a memory.
The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor may also include a main processor, which is a processor for processing data in an awake state, also called a CPU (Central Processing Unit ), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor may incorporate a GPU (Graphics Processing Unit, image processor) for rendering and rendering of content required to be displayed by the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.
The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory is at least configured to store a computer program, where the computer program, after being loaded and executed by the processor, is capable of implementing relevant steps in the cache updating method of the storage system executed by the terminal side disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory can also comprise an operating system, data and the like, and the storage mode can be short-term storage or permanent storage. The operating system may include Windows, unix, linux, among others. The data may include, but is not limited to, update information for the application.
In some embodiments, the terminal may further include a display screen, an input-output interface, a communication interface, a sensor, a power supply, and a communication bus.
Those skilled in the art will appreciate that the structure shown in fig. 7 is not limiting of the terminal and may include more or fewer components than shown.
A readable storage medium provided by embodiments of the present invention is described below, and the readable storage medium described below may be referred to with respect to other embodiments described herein.
A readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the cache update method of the storage system disclosed in the foregoing embodiment. The readable storage medium is a computer readable storage medium, and can be used as a carrier for storing resources, such as read-only memory, random access memory, magnetic disk or optical disk, wherein the resources stored on the readable storage medium comprise an operating system, a computer program, data and the like, and the storage mode can be transient storage or permanent storage.
A storage platform provided by embodiments of the present invention is described below, and a storage platform described below may refer to other embodiments described herein.
Referring to fig. 8, an embodiment of the present invention provides a storage platform, including: a client and a storage system; the storage system comprises a cache and a disk; the client is used for sending an access request to the storage system; the storage system is configured to respond to the access request and perform the method of any of the foregoing embodiments.
In this embodiment, the storage system is specifically a DAS direct-connection storage system, NAS network file system, or SAN storage network.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of readable storage medium known in the art.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (18)
1. A method for cache update of a storage system, the storage system comprising: cache and disk, the method comprising:
inquiring a key value pair accessed from the previous cache miss time to the current time in a log record module at the current time of the cache miss;
arranging each key value pair into a key value pair sequence according to the sequence of access time;
querying and recording nodes comprising elements in the key-value pair sequence in the cache; the same node which is repeatedly searched is recorded once;
taking each recorded node as M nodes accessed before the current moment;
taking each node in the cache as a target node respectively, and combining the M nodes with each target node to obtain N samples;
Predicting hit probability of each target node in each sample according to the real-time access hotness of the M nodes, and selecting eliminated nodes from all target nodes according to the hit probability;
and deleting the eliminated node from the cache, and adding the node read from the disk at the current moment into the cache.
2. The method of claim 1, wherein combining the M nodes with each target node results in N samples, comprising:
arranging the M nodes into a node sequence according to the sequence of the access time;
and arranging each target node at the tail part of the node sequence respectively to obtain the N samples.
3. The method of claim 1, wherein the frequency of access of the M nodes over the last period of time is determined as the real-time access hotness of the M nodes.
4. The method of claim 3, wherein said determining the access frequency of the M nodes over the last period of time as the real-time access hotness of the M nodes comprises:
and determining the access frequency of the M nodes from the previous cache miss time to the current time as the real-time access heat of the M nodes.
5. The method of claim 1, wherein predicting the hit probability of each target node in each sample based on the real-time access hotness of the M nodes comprises:
inputting each sample into a preset prediction model respectively, so that the prediction model predicts a probability value for each sample;
the probability value obtained by the prediction model for each sample is used as the hit probability of each target node in each sample.
6. The method of claim 5, wherein the training process of the predictive model comprises:
the key value pairs accessed in a preset time period are arranged into a target sequence according to the sequence of the access time;
determining hit nodes in the cache according to the target sequence;
intercepting a node group in the hit node;
combining the node group with each target node respectively to obtain N training data;
determining a probability label corresponding to each training data, and adding a corresponding probability label for each training data;
and training by using N training data with probability labels to obtain the prediction model.
7. The method of claim 6, wherein said determining hit nodes in said cache from said target sequence comprises:
Querying and recording nodes comprising elements in the target sequence in the cache; wherein the same node that is repeatedly searched is recorded once.
8. The method of claim 6, wherein determining the probability label for each training data comprises:
after the node group is accessed, the probability that the target node in each training data is accessed is the probability label of the corresponding training data.
9. The method of claim 6, wherein combining the node group with each target node separately results in N training data, comprising:
and arranging each target node at the tail of the node group respectively to obtain the N training data.
10. The method as recited in claim 5, further comprising:
obtaining an increment log in a log recording module; judging whether the data volume of the increment log exceeds a preset first threshold value or not; if the data quantity exceeds the preset first threshold value, retraining the prediction model, and replacing the prediction model with the model obtained by retraining; and/or counting hit rate of the cache; judging whether the hit rate is lower than a preset second threshold value or not; if the hit rate is lower than the preset second threshold value, retraining the prediction model, and replacing the prediction model with the model obtained by retraining; and/or testing the accuracy of the predictive model using a test dataset; judging whether the accuracy is lower than a preset third threshold value or not; and if the accuracy rate is lower than the preset third threshold value, retraining the prediction model, and replacing the prediction model by the model obtained by retraining.
11. The method according to any one of claims 1 to 10, wherein selecting a culled node among all target nodes according to the hit probability comprises:
and selecting the node with the smallest hit probability from all the target nodes as the elimination node.
12. The method according to any one of claims 1 to 10, wherein selecting a culled node among all target nodes according to the hit probability comprises:
selecting a node with the smallest hit probability from all target nodes as a first node to be selected;
selecting a second node to be selected from the cache according to a plurality of preset rules; the plurality of preset rules include: the least recently unused rule, the first in first out rule, and the least used rule;
and taking the repeated nodes in the first node to be selected and the second node to be selected as the elimination nodes.
13. The method according to any one of claims 1 to 10, wherein predicting the hit probability of each target node in each sample according to the real-time access hotness of the M nodes comprises:
and calculating the probability that each target node is accessed at the future moment after the M nodes are accessed according to the real-time access hotness of the M nodes, and taking the probability as the hit probability of each target node in each sample.
14. The method according to any one of claims 1 to 10, wherein the cached data structure is: each target node is arranged in the cache according to the size of the node address; or each target node is arranged in the cache according to the real-time access heat; or each target node is arranged in the cache according to the time sequence of adding the target node into the cache; each target node comprises: a plurality of key-value pairs; each key-value pair is keyed by a logical block address and is valued by a physical block address.
15. A cache update apparatus for a storage system, the storage system comprising: cache and disk, the apparatus comprising:
the acquisition module is used for inquiring the key value pair accessed from the previous cache miss time to the current time in the log recording module at the current time of the cache miss; arranging each key value pair into a key value pair sequence according to the sequence of access time; querying and recording nodes comprising elements in the key-value pair sequence in the cache; the same node which is repeatedly searched is recorded once; taking each recorded node as M nodes accessed before the current moment;
The combination module is used for taking each node in the cache as a target node respectively, and combining the M nodes with each target node to obtain N samples;
the prediction module is used for predicting the hit probability of each target node in each sample according to the real-time access heat of the M nodes and selecting the obsolete node from all the target nodes according to the hit probability;
and the updating module is used for deleting the eliminated node from the cache and adding the node read from the disk at the current moment into the cache.
16. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the method of any one of claims 1 to 14.
17. A readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the method of any one of claims 1 to 14.
18. A storage platform, comprising: a client and a storage system; the storage system comprises a cache and a disk;
the client is used for sending an access request to the storage system;
The storage system is configured to respond to the access request and to perform the method of any one of claims 1 to 14.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311483095.2A CN117235088B (en) | 2023-11-09 | 2023-11-09 | Cache updating method, device, equipment, medium and platform of storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311483095.2A CN117235088B (en) | 2023-11-09 | 2023-11-09 | Cache updating method, device, equipment, medium and platform of storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117235088A CN117235088A (en) | 2023-12-15 |
| CN117235088B true CN117235088B (en) | 2024-02-13 |
Family
ID=89089657
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311483095.2A Active CN117235088B (en) | 2023-11-09 | 2023-11-09 | Cache updating method, device, equipment, medium and platform of storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117235088B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118035130A (en) * | 2024-03-07 | 2024-05-14 | 北京龙创悦动网络科技有限公司 | Game server data storage method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019095402A1 (en) * | 2017-11-15 | 2019-05-23 | 东南大学 | Content popularity prediction-based edge cache system and method therefor |
| CN111858405A (en) * | 2020-07-10 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Data caching method and device for distributed storage |
| CN115168244A (en) * | 2022-07-29 | 2022-10-11 | 苏州浪潮智能科技有限公司 | Data updating method, device, equipment and readable storage medium |
| CN115237825A (en) * | 2022-05-31 | 2022-10-25 | 杭州祺鲸科技有限公司 | Intelligent cache replacement method based on machine learning |
| WO2022233272A1 (en) * | 2021-05-06 | 2022-11-10 | 北京奥星贝斯科技有限公司 | Method and apparatus for eliminating cache memory block, and electronic device |
| CN116931838A (en) * | 2023-08-02 | 2023-10-24 | 武汉汇迪森信息技术有限公司 | Solid-state disk cache management method, system, electronic equipment and storage medium |
-
2023
- 2023-11-09 CN CN202311483095.2A patent/CN117235088B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019095402A1 (en) * | 2017-11-15 | 2019-05-23 | 东南大学 | Content popularity prediction-based edge cache system and method therefor |
| CN111858405A (en) * | 2020-07-10 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Data caching method and device for distributed storage |
| WO2022233272A1 (en) * | 2021-05-06 | 2022-11-10 | 北京奥星贝斯科技有限公司 | Method and apparatus for eliminating cache memory block, and electronic device |
| CN115237825A (en) * | 2022-05-31 | 2022-10-25 | 杭州祺鲸科技有限公司 | Intelligent cache replacement method based on machine learning |
| CN115168244A (en) * | 2022-07-29 | 2022-10-11 | 苏州浪潮智能科技有限公司 | Data updating method, device, equipment and readable storage medium |
| CN116931838A (en) * | 2023-08-02 | 2023-10-24 | 武汉汇迪森信息技术有限公司 | Solid-state disk cache management method, system, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于多查询特性的搜索引擎缓存替换策略研究;房耘耘;;现代计算机(专业版)(23);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117235088A (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8521986B2 (en) | Allocating storage memory based on future file size or use estimates | |
| WO2021120789A1 (en) | Data writing method and apparatus, and storage server and computer-readable storage medium | |
| US9355112B1 (en) | Optimizing compression based on data activity | |
| CN111143243B (en) | Cache prefetching method and system based on NVM hybrid memory | |
| CN117235088B (en) | Cache updating method, device, equipment, medium and platform of storage system | |
| CN104657286B (en) | A kind of classification storage method and device | |
| CN112667528A (en) | Data prefetching method and related equipment | |
| CN111917882B (en) | File caching method and device and electronic equipment | |
| WO2022057749A1 (en) | Method and apparatus for handling missing memory page abnomality, and device and storage medium | |
| JP6394231B2 (en) | Data arrangement control program, data arrangement control apparatus, and data arrangement control method | |
| CN113742131B (en) | Method, electronic device and computer program product for storage management | |
| CN111858469A (en) | Self-adaptive hierarchical storage method based on time sliding window | |
| CN111913913A (en) | Access request processing method and device | |
| CN117111834A (en) | Memory and computing system including memory | |
| CN109189696B (en) | SSD (solid State disk) caching system and caching method | |
| CN113435601A (en) | Data prefetching method and device and storage device | |
| WO2022152086A1 (en) | Data caching method and apparatus, and device and computer-readable storage medium | |
| CN114461590A (en) | Database file page prefetching method and device based on association rule | |
| CN117677941A (en) | Data compression and deduplication aware layering in a storage system | |
| CN112446490A (en) | Network training data set caching method, device, equipment and storage medium | |
| US20230342300A1 (en) | Data eviction method and apparatus, cache node, and cache system | |
| CN117331514B (en) | Solid-state disk data compression system and method based on region division | |
| US20240192880A1 (en) | Data processing method, apparatus, and system | |
| US20240111713A1 (en) | File Management Method and Apparatus, Device, and System | |
| US20240232140A1 (en) | Heat score-based tiering of data between different storage tiers of a file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |